Chao Yu <chao@kernel.org> <yuchao0@huawei.com>
Chris Chiu <chris.chiu@canonical.com> <chiu@endlessm.com>
Chris Chiu <chris.chiu@canonical.com> <chiu@endlessos.org>
+Christian Borntraeger <borntraeger@linux.ibm.com> <borntraeger@de.ibm.com>
+Christian Borntraeger <borntraeger@linux.ibm.com> <cborntra@de.ibm.com>
+Christian Borntraeger <borntraeger@linux.ibm.com> <borntrae@de.ibm.com>
Christophe Ricard <christophe.ricard@gmail.com>
Christoph Hellwig <hch@lst.de>
+Colin Ian King <colin.king@intel.com> <colin.king@canonical.com>
+Colin Ian King <colin.king@intel.com> <colin.i.king@gmail.com>
Corey Minyard <minyard@acm.org>
Damian Hobson-Garcia <dhobsong@igel.co.jp>
Daniel Borkmann <daniel@iogearbox.net> <danborkmann@googlemail.com>
This attribute indicates the mode that the irq vector named by
the file is in (msi vs. msix)
+What: /sys/bus/pci/devices/.../irq
+Date: August 2021
+Contact: Linux PCI developers <linux-pci@vger.kernel.org>
+Description:
+ If a driver has enabled MSI (not MSI-X), "irq" contains the
+ IRQ of the first MSI vector. Otherwise "irq" contains the
+ IRQ of the legacy INTx interrupt.
+
+ "irq" being set to 0 indicates that the device isn't
+ capable of generating legacy INTx interrupts.
+
What: /sys/bus/pci/devices/.../remove
Date: January 2009
Contact: Linux PCI developers <linux-pci@vger.kernel.org>
--- /dev/null
+What: /sys/class/fc/fc_udev_device/appid_store
+Date: Aug 2021
+Contact: Muneendra Kumar <muneendra.kumar@broadconm.com>
+Description:
+ This interface allows an admin to set an FC application
+ identifier in the blkcg associated with a cgroup id. The
+ identifier is typically a UUID that is associated with
+ an application or logical entity such as a virtual
+ machine or container group. The application or logical
+ entity utilizes a block device via the cgroup id.
+ FC adapter drivers may query the identifier and tag FC
+ traffic based on the identifier. FC host and FC fabric
+ entities can utilize the application id and FC traffic
+ tag to identify traffic sources.
+
+ The interface expects a string "<cgroupid>:<appid>" where:
+ <cgroupid> is inode of the cgroup in hexadecimal
+ <appid> is user provided string upto 128 characters
+ in length.
+
+ If an appid_store is done for a cgroup id that already
+ has an appid set, the new value will override the
+ previous value.
+
+ If an admin wants to remove an FC application identifier
+ from a cgroup, an appid_store should be done with the
+ following string: "<cgroupid>:"
What: /sys/class/scsi_device/*/device/dyn_cap_needed
Date: February 2018
Contact: Stanislav Nijnikov <stanislav.nijnikov@wdc.com>
-Description: This file shows the The amount of physical memory needed
+Description: This file shows the amount of physical memory needed
to be removed from the physical memory resources pool of
the particular logical unit. The full information about
the attribute could be found at UFS specifications 2.1.
Contact: "Daeho Jeong" <daehojeong@google.com>
Description: You can control the multiplier value of bdi device readahead window size
between 2 (default) and 256 for POSIX_FADV_SEQUENTIAL advise option.
+
+What: /sys/fs/f2fs/<disk>/max_fragment_chunk
+Date: August 2021
+Contact: "Daeho Jeong" <daehojeong@google.com>
+Description: With "mode=fragment:block" mount options, we can scatter block allocation.
+ f2fs will allocate 1..<max_fragment_chunk> blocks in a chunk and make a hole
+ in the length of 1..<max_fragment_hole> by turns. This value can be set
+ between 1..512 and the default value is 4.
+
+What: /sys/fs/f2fs/<disk>/max_fragment_hole
+Date: August 2021
+Contact: "Daeho Jeong" <daehojeong@google.com>
+Description: With "mode=fragment:block" mount options, we can scatter block allocation.
+ f2fs will allocate 1..<max_fragment_chunk> blocks in a chunk and make a hole
+ in the length of 1..<max_fragment_hole> by turns. This value can be set
+ between 1..512 and the default value is 4.
From now on, any pages on zram are idle pages. The idle mark
will be removed until someone requests access of the block.
IOW, unless there is access request, those pages are still idle pages.
+Additionally, when CONFIG_ZRAM_MEMORY_TRACKING is enabled pages can be
+marked as idle based on how long (in seconds) it's been since they were
+last accessed::
+
+ echo 86400 > /sys/block/zramX/idle
+
+In this example all pages which haven't been accessed in more than 86400
+seconds (one day) will be marked idle.
Admin can request writeback of those idle pages at right timing via::
memory.oom_control set/show oom controls.
memory.numa_stat show the number of memory usage per numa
node
- memory.kmem.limit_in_bytes set/show hard limit for kernel memory
- This knob is deprecated and shouldn't be
- used. It is planned that this be removed in
- the foreseeable future.
+ memory.kmem.limit_in_bytes This knob is deprecated and writing to
+ it will return -ENOTSUPP.
memory.kmem.usage_in_bytes show current kernel memory allocation
memory.kmem.failcnt show the number of kernel memory usage
hits limits
charged file caches. Some out-of-use page caches may keep charged until
memory pressure happens. If you want to avoid that, force_empty will be useful.
- Also, note that when memory.kmem.limit_in_bytes is set the charges due to
- kernel pages will still be seen. This is not considered a failure and the
- write will still return success. In this case, it is expected that
- memory.kmem.usage_in_bytes == memory.usage_in_bytes.
-
5.2 stat file
-------------
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0
+
+====================================
+File system Monitoring with fanotify
+====================================
+
+File system Error Reporting
+===========================
+
+Fanotify supports the FAN_FS_ERROR event type for file system-wide error
+reporting. It is meant to be used by file system health monitoring
+daemons, which listen for these events and take actions (notify
+sysadmin, start recovery) when a file system problem is detected.
+
+By design, a FAN_FS_ERROR notification exposes sufficient information
+for a monitoring tool to know a problem in the file system has happened.
+It doesn't necessarily provide a user space application with semantics
+to verify an IO operation was successfully executed. That is out of
+scope for this feature. Instead, it is only meant as a framework for
+early file system problem detection and reporting recovery tools.
+
+When a file system operation fails, it is common for dozens of kernel
+errors to cascade after the initial failure, hiding the original failure
+log, which is usually the most useful debug data to troubleshoot the
+problem. For this reason, FAN_FS_ERROR tries to report only the first
+error that occurred for a file system since the last notification, and
+it simply counts additional errors. This ensures that the most
+important pieces of information are never lost.
+
+FAN_FS_ERROR requires the fanotify group to be setup with the
+FAN_REPORT_FID flag.
+
+At the time of this writing, the only file system that emits FAN_FS_ERROR
+notifications is Ext4.
+
+A FAN_FS_ERROR Notification has the following format::
+
+ ::
+
+ [ Notification Metadata (Mandatory) ]
+ [ Generic Error Record (Mandatory) ]
+ [ FID record (Mandatory) ]
+
+The order of records is not guaranteed, and new records might be added
+in the future. Therefore, applications must not rely on the order and
+must be prepared to skip over unknown records. Please refer to
+``samples/fanotify/fs-monitor.c`` for an example parser.
+
+Generic error record
+--------------------
+
+The generic error record provides enough information for a file system
+agnostic tool to learn about a problem in the file system, without
+providing any additional details about the problem. This record is
+identified by ``struct fanotify_event_info_header.info_type`` being set
+to FAN_EVENT_INFO_TYPE_ERROR.
+
+ ::
+
+ struct fanotify_event_info_error {
+ struct fanotify_event_info_header hdr;
+ __s32 error;
+ __u32 error_count;
+ };
+
+The `error` field identifies the type of error using errno values.
+`error_count` tracks the number of errors that occurred and were
+suppressed to preserve the original error information, since the last
+notification.
+
+FID record
+----------
+
+The FID record can be used to uniquely identify the inode that triggered
+the error through the combination of fsid and file handle. A file system
+specific application can use that information to attempt a recovery
+procedure. Errors that are not related to an inode are reported with an
+empty file handle of type FILEID_INVALID.
``pid`` of the task for which the operation applies.
arg4:
- ``pid_type`` for which the operation applies. It is of type ``enum pid_type``.
- For example, if arg4 is ``PIDTYPE_TGID``, then the operation of this command
+ ``pid_type`` for which the operation applies. It is one of
+ ``PR_SCHED_CORE_SCOPE_``-prefixed macro constants. For example, if arg4
+ is ``PR_SCHED_CORE_SCOPE_THREAD_GROUP``, then the operation of this command
will be performed for all tasks in the task group of ``pid``.
arg5:
edid
efi-stub
ext4
+ filesystem-monitoring
nfs/index
gpio/index
highuid
registers. Default set by CONFIG_HPET_MMAP_DEFAULT.
hugetlb_cma= [HW,CMA] The size of a CMA area used for allocation
- of gigantic hugepages.
- Format: nn[KMGTPE]
+ of gigantic hugepages. Or using node format, the size
+ of a CMA area per node can be specified.
+ Format: nn[KMGTPE] or (node format)
+ <node>:nn[KMGTPE][,<node>:nn[KMGTPE]]
Reserve a CMA area of given size and allocate gigantic
hugepages using the CMA allocator. If enabled, the
the number of pages of hugepagesz to be allocated.
If this is the first HugeTLB parameter on the command
line, it specifies the number of pages to allocate for
- the default huge page size. See also
- Documentation/admin-guide/mm/hugetlbpage.rst.
- Format: <integer>
+ the default huge page size. If using node format, the
+ number of pages to allocate per-node can be specified.
+ See also Documentation/admin-guide/mm/hugetlbpage.rst.
+ Format: <integer> or (node format)
+ <node>:<integer>[,<node>:<integer>]
hugepagesz=
[HW] The size of the HugeTLB pages. This is used in
driver. A non-zero value sets the minimum interval
in seconds between layoutstats transmissions.
+ nfsd.inter_copy_offload_enable =
+ [NFSv4.2] When set to 1, the server will support
+ server-to-server copies for which this server is
+ the destination of the copy.
+
+ nfsd.nfsd4_ssc_umount_timeout =
+ [NFSv4.2] When used as the destination of a
+ server-to-server copy, knfsd temporarily mounts
+ the source server. It caches the mount in case
+ it will be needed again, and discards it if not
+ used for the number of milliseconds specified by
+ this parameter.
+
nfsd.nfs4_disable_idmapping=
[NFSv4] When set to the default of '1', the NFSv4
server will return only numeric uids and gids to
and gids from such clients. This is intended to ease
migration from NFSv2/v3.
+
nmi_backtrace.backtrace_idle [KNL]
Dump stacks even of idle CPUs in response to an
NMI stack-backtrace request.
an IOTLB flush. Default is lazy flushing before reuse,
which is faster.
+ s390_iommu_aperture= [KNL,S390]
+ Specifies the size of the per device DMA address space
+ accessible through the DMA and IOMMU APIs as a decimal
+ factor of the size of main memory.
+ The default is 1 meaning that one can concurrently use
+ as many DMA addresses as physical memory is installed,
+ if supported by hardware, and thus map all of memory
+ once. With a value of 2 one can map all of memory twice
+ and so on. As a special case a factor of 0 imposes no
+ restrictions other than those given by hardware at the
+ cost of significant additional memory use for tables.
+
sa1100ir [NET]
See drivers/net/irda/sa1100_ir.c.
improve timer resolution at the expense of processing
more timer interrupts.
+ xen.balloon_boot_timeout= [XEN]
+ The time (in seconds) to wait before giving up to boot
+ in case initial ballooning fails to free enough memory.
+ Applies only when running as HVM or PVH guest and
+ started with less memory configured than allowed at
+ max. Default is 180.
+
xen.event_eoi_delay= [XEN]
How long to delay EOI handling in case of event
storms (jiffies). Default is 10.
Lenovo X1 Carbon 2nd gen (2014)'s adaptive keyboard. The value can be read
and set.
-- 1 = Home mode
-- 2 = Web-browser mode
-- 3 = Web-conference mode
-- 4 = Function mode
-- 5 = Layflat mode
+- 0 = Home mode
+- 1 = Web-browser mode
+- 2 = Web-conference mode
+- 3 = Function mode
+- 4 = Layflat mode
For more details about which buttons will appear depending on the mode, please
review the laptop's user guide:
-http://www.lenovo.com/shop/americas/content/user_guides/x1carbon_2_ug_en.pdf
+https://download.lenovo.com/ibmdl/pub/pc/pccbbs/mobiles_pdf/x1carbon_2_ug_en.pdf
Battery charge control
----------------------
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================
+DAMON-based Reclamation
+=======================
+
+DAMON-based Reclamation (DAMON_RECLAIM) is a static kernel module that aimed to
+be used for proactive and lightweight reclamation under light memory pressure.
+It doesn't aim to replace the LRU-list based page_granularity reclamation, but
+to be selectively used for different level of memory pressure and requirements.
+
+Where Proactive Reclamation is Required?
+========================================
+
+On general memory over-committed systems, proactively reclaiming cold pages
+helps saving memory and reducing latency spikes that incurred by the direct
+reclaim of the process or CPU consumption of kswapd, while incurring only
+minimal performance degradation [1]_ [2]_ .
+
+Free Pages Reporting [3]_ based memory over-commit virtualization systems are
+good example of the cases. In such systems, the guest VMs reports their free
+memory to host, and the host reallocates the reported memory to other guests.
+As a result, the memory of the systems are fully utilized. However, the
+guests could be not so memory-frugal, mainly because some kernel subsystems and
+user-space applications are designed to use as much memory as available. Then,
+guests could report only small amount of memory as free to host, results in
+memory utilization drop of the systems. Running the proactive reclamation in
+guests could mitigate this problem.
+
+How It Works?
+=============
+
+DAMON_RECLAIM finds memory regions that didn't accessed for specific time
+duration and page out. To avoid it consuming too much CPU for the paging out
+operation, a speed limit can be configured. Under the speed limit, it pages
+out memory regions that didn't accessed longer time first. System
+administrators can also configure under what situation this scheme should
+automatically activated and deactivated with three memory pressure watermarks.
+
+Interface: Module Parameters
+============================
+
+To use this feature, you should first ensure your system is running on a kernel
+that is built with ``CONFIG_DAMON_RECLAIM=y``.
+
+To let sysadmins enable or disable it and tune for the given system,
+DAMON_RECLAIM utilizes module parameters. That is, you can put
+``damon_reclaim.<parameter>=<value>`` on the kernel boot command line or write
+proper values to ``/sys/modules/damon_reclaim/parameters/<parameter>`` files.
+
+Note that the parameter values except ``enabled`` are applied only when
+DAMON_RECLAIM starts. Therefore, if you want to apply new parameter values in
+runtime and DAMON_RECLAIM is already enabled, you should disable and re-enable
+it via ``enabled`` parameter file. Writing of the new values to proper
+parameter values should be done before the re-enablement.
+
+Below are the description of each parameter.
+
+enabled
+-------
+
+Enable or disable DAMON_RECLAIM.
+
+You can enable DAMON_RCLAIM by setting the value of this parameter as ``Y``.
+Setting it as ``N`` disables DAMON_RECLAIM. Note that DAMON_RECLAIM could do
+no real monitoring and reclamation due to the watermarks-based activation
+condition. Refer to below descriptions for the watermarks parameter for this.
+
+min_age
+-------
+
+Time threshold for cold memory regions identification in microseconds.
+
+If a memory region is not accessed for this or longer time, DAMON_RECLAIM
+identifies the region as cold, and reclaims it.
+
+120 seconds by default.
+
+quota_ms
+--------
+
+Limit of time for the reclamation in milliseconds.
+
+DAMON_RECLAIM tries to use only up to this time within a time window
+(quota_reset_interval_ms) for trying reclamation of cold pages. This can be
+used for limiting CPU consumption of DAMON_RECLAIM. If the value is zero, the
+limit is disabled.
+
+10 ms by default.
+
+quota_sz
+--------
+
+Limit of size of memory for the reclamation in bytes.
+
+DAMON_RECLAIM charges amount of memory which it tried to reclaim within a time
+window (quota_reset_interval_ms) and makes no more than this limit is tried.
+This can be used for limiting consumption of CPU and IO. If this value is
+zero, the limit is disabled.
+
+128 MiB by default.
+
+quota_reset_interval_ms
+-----------------------
+
+The time/size quota charge reset interval in milliseconds.
+
+The charget reset interval for the quota of time (quota_ms) and size
+(quota_sz). That is, DAMON_RECLAIM does not try reclamation for more than
+quota_ms milliseconds or quota_sz bytes within quota_reset_interval_ms
+milliseconds.
+
+1 second by default.
+
+wmarks_interval
+---------------
+
+Minimal time to wait before checking the watermarks, when DAMON_RECLAIM is
+enabled but inactive due to its watermarks rule.
+
+wmarks_high
+-----------
+
+Free memory rate (per thousand) for the high watermark.
+
+If free memory of the system in bytes per thousand bytes is higher than this,
+DAMON_RECLAIM becomes inactive, so it does nothing but only periodically checks
+the watermarks.
+
+wmarks_mid
+----------
+
+Free memory rate (per thousand) for the middle watermark.
+
+If free memory of the system in bytes per thousand bytes is between this and
+the low watermark, DAMON_RECLAIM becomes active, so starts the monitoring and
+the reclaiming.
+
+wmarks_low
+----------
+
+Free memory rate (per thousand) for the low watermark.
+
+If free memory of the system in bytes per thousand bytes is lower than this,
+DAMON_RECLAIM becomes inactive, so it does nothing but periodically checks the
+watermarks. In the case, the system falls back to the LRU-list based page
+granularity reclamation logic.
+
+sample_interval
+---------------
+
+Sampling interval for the monitoring in microseconds.
+
+The sampling interval of DAMON for the cold memory monitoring. Please refer to
+the DAMON documentation (:doc:`usage`) for more detail.
+
+aggr_interval
+-------------
+
+Aggregation interval for the monitoring in microseconds.
+
+The aggregation interval of DAMON for the cold memory monitoring. Please
+refer to the DAMON documentation (:doc:`usage`) for more detail.
+
+min_nr_regions
+--------------
+
+Minimum number of monitoring regions.
+
+The minimal number of monitoring regions of DAMON for the cold memory
+monitoring. This can be used to set lower-bound of the monitoring quality.
+But, setting this too high could result in increased monitoring overhead.
+Please refer to the DAMON documentation (:doc:`usage`) for more detail.
+
+max_nr_regions
+--------------
+
+Maximum number of monitoring regions.
+
+The maximum number of monitoring regions of DAMON for the cold memory
+monitoring. This can be used to set upper-bound of the monitoring overhead.
+However, setting this too low could result in bad monitoring quality. Please
+refer to the DAMON documentation (:doc:`usage`) for more detail.
+
+monitor_region_start
+--------------------
+
+Start of target memory region in physical address.
+
+The start physical address of memory region that DAMON_RECLAIM will do work
+against. That is, DAMON_RECLAIM will find cold memory regions in this region
+and reclaims. By default, biggest System RAM is used as the region.
+
+monitor_region_end
+------------------
+
+End of target memory region in physical address.
+
+The end physical address of memory region that DAMON_RECLAIM will do work
+against. That is, DAMON_RECLAIM will find cold memory regions in this region
+and reclaims. By default, biggest System RAM is used as the region.
+
+kdamond_pid
+-----------
+
+PID of the DAMON thread.
+
+If DAMON_RECLAIM is enabled, this becomes the PID of the worker thread. Else,
+-1.
+
+Example
+=======
+
+Below runtime example commands make DAMON_RECLAIM to find memory regions that
+not accessed for 30 seconds or more and pages out. The reclamation is limited
+to be done only up to 1 GiB per second to avoid DAMON_RECLAIM consuming too
+much CPU time for the paging out operation. It also asks DAMON_RECLAIM to do
+nothing if the system's free memory rate is more than 50%, but start the real
+works if it becomes lower than 40%. If DAMON_RECLAIM doesn't make progress and
+therefore the free memory rate becomes lower than 20%, it asks DAMON_RECLAIM to
+do nothing again, so that we can fall back to the LRU-list based page
+granularity reclamation. ::
+
+ # cd /sys/modules/damon_reclaim/parameters
+ # echo 30000000 > min_age
+ # echo $((1 * 1024 * 1024 * 1024)) > quota_sz
+ # echo 1000 > quota_reset_interval_ms
+ # echo 500 > wmarks_high
+ # echo 400 > wmarks_mid
+ # echo 200 > wmarks_low
+ # echo Y > enabled
+
+.. [1] https://research.google/pubs/pub48551/
+.. [2] https://lwn.net/Articles/787611/
+.. [3] https://www.kernel.org/doc/html/latest/vm/free_page_reporting.html
This document briefly describes how you can use DAMON by demonstrating its
default user space tool. Please note that this document describes only a part
-of its features for brevity. Please refer to :doc:`usage` for more details.
-
-
-TL; DR
-======
-
-Follow the commands below to monitor and visualize the memory access pattern of
-your workload. ::
-
- # # build the kernel with CONFIG_DAMON_*=y, install it, and reboot
- # mount -t debugfs none /sys/kernel/debug/
- # git clone https://github.com/awslabs/damo
- # ./damo/damo record $(pidof <your workload>)
- # ./damo/damo report heat --plot_ascii
-
-The final command draws the access heatmap of ``<your workload>``. The heatmap
-shows which memory region (x-axis) is accessed when (y-axis) and how frequently
-(number; the higher the more accesses have been observed). ::
-
- 111111111111111111111111111111111111111111111111111111110000
- 111121111111111111111111111111211111111111111111111111110000
- 000000000000000000000000000000000000000000000000001555552000
- 000000000000000000000000000000000000000000000222223555552000
- 000000000000000000000000000000000000000011111677775000000000
- 000000000000000000000000000000000000000488888000000000000000
- 000000000000000000000000000000000177888400000000000000000000
- 000000000000000000000000000046666522222100000000000000000000
- 000000000000000000000014444344444300000000000000000000000000
- 000000000000000002222245555510000000000000000000000000000000
- # access_frequency: 0 1 2 3 4 5 6 7 8 9
- # x-axis: space (140286319947776-140286426374096: 101.496 MiB)
- # y-axis: time (605442256436361-605479951866441: 37.695430s)
- # resolution: 60x10 (1.692 MiB and 3.770s for each character)
+of its features for brevity. Please refer to the usage `doc
+<https://github.com/awslabs/damo/blob/next/USAGE.md>`_ of the tool for more
+details.
Prerequisites
Visualizing Recorded Patterns
=============================
-The following three commands visualize the recorded access patterns and save
-the results as separate image files. ::
-
- $ damo report heats --heatmap access_pattern_heatmap.png
- $ damo report wss --range 0 101 1 --plot wss_dist.png
- $ damo report wss --range 0 101 1 --sortby time --plot wss_chron_change.png
-
-- ``access_pattern_heatmap.png`` will visualize the data access pattern in a
- heatmap, showing which memory region (y-axis) got accessed when (x-axis)
- and how frequently (color).
-- ``wss_dist.png`` will show the distribution of the working set size.
-- ``wss_chron_change.png`` will show how the working set size has
- chronologically changed.
-
-You can view the visualizations of this example workload at [1]_.
-Visualizations of other realistic workloads are available at [2]_ [3]_ [4]_.
-
-.. [1] https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/start.html#visualizing-recorded-patterns
-.. [2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
-.. [3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
-.. [4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
+You can visualize the pattern in a heatmap, showing which memory region
+(x-axis) got accessed when (y-axis) and how frequently (number).::
+
+ $ sudo damo report heats --heatmap stdout
+ 22222222222222222222222222222222222222211111111111111111111111111111111111111100
+ 44444444444444444444444444444444444444434444444444444444444444444444444444443200
+ 44444444444444444444444444444444444444433444444444444444444444444444444444444200
+ 33333333333333333333333333333333333333344555555555555555555555555555555555555200
+ 33333333333333333333333333333333333344444444444444444444444444444444444444444200
+ 22222222222222222222222222222222222223355555555555555555555555555555555555555200
+ 00000000000000000000000000000000000000288888888888888888888888888888888888888400
+ 00000000000000000000000000000000000000288888888888888888888888888888888888888400
+ 33333333333333333333333333333333333333355555555555555555555555555555555555555200
+ 88888888888888888888888888888888888888600000000000000000000000000000000000000000
+ 88888888888888888888888888888888888888600000000000000000000000000000000000000000
+ 33333333333333333333333333333333333333444444444444444444444444444444444444443200
+ 00000000000000000000000000000000000000288888888888888888888888888888888888888400
+ [...]
+ # access_frequency: 0 1 2 3 4 5 6 7 8 9
+ # x-axis: space (139728247021568-139728453431248: 196.848 MiB)
+ # y-axis: time (15256597248362-15326899978162: 1 m 10.303 s)
+ # resolution: 80x40 (2.461 MiB and 1.758 s for each character)
+
+You can also visualize the distribution of the working set size, sorted by the
+size.::
+
+ $ sudo damo report wss --range 0 101 10
+ # <percentile> <wss>
+ # target_id 18446632103789443072
+ # avr: 107.708 MiB
+ 0 0 B | |
+ 10 95.328 MiB |**************************** |
+ 20 95.332 MiB |**************************** |
+ 30 95.340 MiB |**************************** |
+ 40 95.387 MiB |**************************** |
+ 50 95.387 MiB |**************************** |
+ 60 95.398 MiB |**************************** |
+ 70 95.398 MiB |**************************** |
+ 80 95.504 MiB |**************************** |
+ 90 190.703 MiB |********************************************************* |
+ 100 196.875 MiB |***********************************************************|
+
+Using ``--sortby`` option with the above command, you can show how the working
+set size has chronologically changed.::
+
+ $ sudo damo report wss --range 0 101 10 --sortby time
+ # <percentile> <wss>
+ # target_id 18446632103789443072
+ # avr: 107.708 MiB
+ 0 3.051 MiB | |
+ 10 190.703 MiB |***********************************************************|
+ 20 95.336 MiB |***************************** |
+ 30 95.328 MiB |***************************** |
+ 40 95.387 MiB |***************************** |
+ 50 95.332 MiB |***************************** |
+ 60 95.320 MiB |***************************** |
+ 70 95.398 MiB |***************************** |
+ 80 95.398 MiB |***************************** |
+ 90 95.340 MiB |***************************** |
+ 100 95.398 MiB |***************************** |
+
+
+Data Access Pattern Aware Memory Management
+===========================================
+
+Below three commands make every memory region of size >=4K that doesn't
+accessed for >=60 seconds in your workload to be swapped out. ::
+
+ $ echo "#min-size max-size min-acc max-acc min-age max-age action" > test_scheme
+ $ echo "4K max 0 0 60s max pageout" >> test_scheme
+ $ damo schemes -c test_scheme <pid of your workload>
This is for privileged people such as system administrators who want a
just-working human-friendly interface. Using this, users can use the DAMON’s
major features in a human-friendly way. It may not be highly tuned for
- special cases, though. It supports only virtual address spaces monitoring.
+ special cases, though. It supports both virtual and physical address spaces
+ monitoring.
- *debugfs interface.*
This is for privileged user space programmers who want more optimized use of
DAMON. Using this, users can use DAMON’s major features by reading
from and writing to special debugfs files. Therefore, you can write and use
your personalized DAMON debugfs wrapper programs that reads/writes the
debugfs files instead of you. The DAMON user space tool is also a reference
- implementation of such programs. It supports only virtual address spaces
- monitoring.
+ implementation of such programs. It supports both virtual and physical
+ address spaces monitoring.
- *Kernel Space Programming Interface.*
This is for kernel space programmers. Using this, users can utilize every
feature of DAMON most flexibly and efficiently by writing kernel space
debugfs Interface
=================
-DAMON exports three files, ``attrs``, ``target_ids``, and ``monitor_on`` under
-its debugfs directory, ``<debugfs>/damon/``.
+DAMON exports five files, ``attrs``, ``target_ids``, ``init_regions``,
+``schemes`` and ``monitor_on`` under its debugfs directory,
+``<debugfs>/damon/``.
Attributes
# cat target_ids
42 4242
+Users can also monitor the physical memory address space of the system by
+writing a special keyword, "``paddr\n``" to the file. Because physical address
+space monitoring doesn't support multiple targets, reading the file will show a
+fake value, ``42``, as below::
+
+ # cd <debugfs>/damon
+ # echo paddr > target_ids
+ # cat target_ids
+ 42
+
Note that setting the target ids doesn't start the monitoring.
+Initial Monitoring Target Regions
+---------------------------------
+
+In case of the virtual address space monitoring, DAMON automatically sets and
+updates the monitoring target regions so that entire memory mappings of target
+processes can be covered. However, users can want to limit the monitoring
+region to specific address ranges, such as the heap, the stack, or specific
+file-mapped area. Or, some users can know the initial access pattern of their
+workloads and therefore want to set optimal initial regions for the 'adaptive
+regions adjustment'.
+
+In contrast, DAMON do not automatically sets and updates the monitoring target
+regions in case of physical memory monitoring. Therefore, users should set the
+monitoring target regions by themselves.
+
+In such cases, users can explicitly set the initial monitoring target regions
+as they want, by writing proper values to the ``init_regions`` file. Each line
+of the input should represent one region in below form.::
+
+ <target id> <start address> <end address>
+
+The ``target id`` should already in ``target_ids`` file, and the regions should
+be passed in address order. For example, below commands will set a couple of
+address ranges, ``1-100`` and ``100-200`` as the initial monitoring target
+region of process 42, and another couple of address ranges, ``20-40`` and
+``50-100`` as that of process 4242.::
+
+ # cd <debugfs>/damon
+ # echo "42 1 100
+ 42 100 200
+ 4242 20 40
+ 4242 50 100" > init_regions
+
+Note that this sets the initial monitoring target regions only. In case of
+virtual memory monitoring, DAMON will automatically updates the boundary of the
+regions after one ``regions update interval``. Therefore, users should set the
+``regions update interval`` large enough in this case, if they don't want the
+update.
+
+
+Schemes
+-------
+
+For usual DAMON-based data access aware memory management optimizations, users
+would simply want the system to apply a memory management action to a memory
+region of a specific size having a specific access frequency for a specific
+time. DAMON receives such formalized operation schemes from the user and
+applies those to the target processes. It also counts the total number and
+size of regions that each scheme is applied. This statistics can be used for
+online analysis or tuning of the schemes.
+
+Users can get and set the schemes by reading from and writing to ``schemes``
+debugfs file. Reading the file also shows the statistics of each scheme. To
+the file, each of the schemes should be represented in each line in below form:
+
+ min-size max-size min-acc max-acc min-age max-age action
+
+Note that the ranges are closed interval. Bytes for the size of regions
+(``min-size`` and ``max-size``), number of monitored accesses per aggregate
+interval for access frequency (``min-acc`` and ``max-acc``), number of
+aggregate intervals for the age of regions (``min-age`` and ``max-age``), and a
+predefined integer for memory management actions should be used. The supported
+numbers and their meanings are as below.
+
+ - 0: Call ``madvise()`` for the region with ``MADV_WILLNEED``
+ - 1: Call ``madvise()`` for the region with ``MADV_COLD``
+ - 2: Call ``madvise()`` for the region with ``MADV_PAGEOUT``
+ - 3: Call ``madvise()`` for the region with ``MADV_HUGEPAGE``
+ - 4: Call ``madvise()`` for the region with ``MADV_NOHUGEPAGE``
+ - 5: Do nothing but count the statistics
+
+You can disable schemes by simply writing an empty string to the file. For
+example, below commands applies a scheme saying "If a memory region of size in
+[4KiB, 8KiB] is showing accesses per aggregate interval in [0, 5] for aggregate
+interval in [10, 20], page out the region", check the entered scheme again, and
+finally remove the scheme. ::
+
+ # cd <debugfs>/damon
+ # echo "4096 8192 0 5 10 20 2" > schemes
+ # cat schemes
+ 4096 8192 0 5 10 20 2 0 0
+ # echo > schemes
+
+The last two integers in the 4th line of above example is the total number and
+the total size of the regions that the scheme is applied.
+
+
Turning On/Off
--------------
implicitly specifies the number of huge pages of default size to
allocate. If the number of huge pages of default size is implicitly
specified, it can not be overwritten by a hugepagesz,hugepages
- parameter pair for the default size.
+ parameter pair for the default size. This parameter also has a
+ node format. The node format specifies the number of huge pages
+ to allocate on specific nodes.
For example, on an architecture with 2M default huge page size::
indicating that the hugepages=512 parameter is ignored. If a hugepages
parameter is preceded by an invalid hugepagesz parameter, it will
be ignored.
+
+ Node format example::
+
+ hugepagesz=2M hugepages=0:1,1:2
+
+ It will allocate 1 2M hugepage on node0 and 2 2M hugepages on node1.
+ If the node number is invalid, the parameter will be ignored.
+
default_hugepagesz
Specify the default huge page size. This parameter can
only be specified once on the command line. default_hugepagesz can
hugepages-${size}kB
-Inside each of these directories, the same set of files will exist::
+Inside each of these directories, the set of files contained in ``/proc``
+will exist. In addition, two additional interfaces for demoting huge
+pages may exist::
+ demote
+ demote_size
nr_hugepages
nr_hugepages_mempolicy
nr_overcommit_hugepages
resv_hugepages
surplus_hugepages
-which function as described above for the default huge page-sized case.
+The demote interfaces provide the ability to split a huge page into
+smaller huge pages. For example, the x86 architecture supports both
+1GB and 2MB huge pages sizes. A 1GB huge page can be split into 512
+2MB huge pages. Demote interfaces are not available for the smallest
+huge page size. The demote interfaces are:
+
+demote_size
+ is the size of demoted pages. When a page is demoted a corresponding
+ number of huge pages of demote_size will be created. By default,
+ demote_size is set to the next smaller huge page size. If there are
+ multiple smaller huge page sizes, demote_size can be set to any of
+ these smaller sizes. Only huge page sizes less than the current huge
+ pages size are allowed.
+
+demote
+ is used to demote a number of huge pages. A user with root privileges
+ can write to this file. It may not be possible to demote the
+ requested number of huge pages. To determine how many pages were
+ actually demoted, compare the value of nr_hugepages before and after
+ writing to the demote interface. demote is a write only interface.
+
+The interfaces which are the same as in ``/proc`` (all except demote and
+demote_size) function as described above for the default huge page-sized case.
.. _mem_policy_and_hp_alloc:
numaperf
pagemap
soft-dirty
+ swap_numa
transhuge
userfaultfd
+ zswap
% echo 1 > /sys/devices/system/memory/memoryXXX/online
-The kernel will select the target zone automatically, usually defaulting to
-``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel
-command line or if the memory block would intersect the ZONE_MOVABLE already.
+The kernel will select the target zone automatically, depending on the
+configured ``online_policy``.
One can explicitly request to associate an offline memory block with
ZONE_MOVABLE by::
% echo online > /sys/devices/system/memory/auto_online_blocks
+Similarly to manual onlining, with ``online`` the kernel will select the
+target zone automatically, depending on the configured ``online_policy``.
+
Modifying the auto-online behavior will only affect all subsequently added
memory blocks only.
======================== =======================================================
``memhp_default_state`` configure auto-onlining by essentially setting
``/sys/devices/system/memory/auto_online_blocks``.
-``movablecore`` configure automatic zone selection of the kernel. When
- set, the kernel will default to ZONE_MOVABLE, unless
- other zones can be kept contiguous.
+``movable_node`` configure automatic zone selection in the kernel when
+ using the ``contig-zones`` online policy. When
+ set, the kernel will default to ZONE_MOVABLE when
+ onlining a memory block, unless other zones can be kept
+ contiguous.
======================== =======================================================
+See Documentation/admin-guide/kernel-parameters.txt for a more generic
+description of these command line parameters.
+
Module Parameters
------------------
and they can be observed (and some even modified at runtime) via::
- /sys/modules/memory_hotplug/parameters/
+ /sys/module/memory_hotplug/parameters/
The following module parameters are currently defined:
-======================== =======================================================
-``memmap_on_memory`` read-write: Allocate memory for the memmap from the
- added memory block itself. Even if enabled, actual
- support depends on various other system properties and
- should only be regarded as a hint whether the behavior
- would be desired.
-
- While allocating the memmap from the memory block
- itself makes memory hotplug less likely to fail and
- keeps the memmap on the same NUMA node in any case, it
- can fragment physical memory in a way that huge pages
- in bigger granularity cannot be formed on hotplugged
- memory.
-======================== =======================================================
+================================ ===============================================
+``memmap_on_memory`` read-write: Allocate memory for the memmap from
+ the added memory block itself. Even if enabled,
+ actual support depends on various other system
+ properties and should only be regarded as a
+ hint whether the behavior would be desired.
+
+ While allocating the memmap from the memory
+ block itself makes memory hotplug less likely
+ to fail and keeps the memmap on the same NUMA
+ node in any case, it can fragment physical
+ memory in a way that huge pages in bigger
+ granularity cannot be formed on hotplugged
+ memory.
+``online_policy`` read-write: Set the basic policy used for
+ automatic zone selection when onlining memory
+ blocks without specifying a target zone.
+ ``contig-zones`` has been the kernel default
+ before this parameter was added. After an
+ online policy was configured and memory was
+ online, the policy should not be changed
+ anymore.
+
+ When set to ``contig-zones``, the kernel will
+ try keeping zones contiguous. If a memory block
+ intersects multiple zones or no zone, the
+ behavior depends on the ``movable_node`` kernel
+ command line parameter: default to ZONE_MOVABLE
+ if set, default to the applicable kernel zone
+ (usually ZONE_NORMAL) if not set.
+
+ When set to ``auto-movable``, the kernel will
+ try onlining memory blocks to ZONE_MOVABLE if
+ possible according to the configuration and
+ memory device details. With this policy, one
+ can avoid zone imbalances when eventually
+ hotplugging a lot of memory later and still
+ wanting to be able to hotunplug as much as
+ possible reliably, very desirable in
+ virtualized environments. This policy ignores
+ the ``movable_node`` kernel command line
+ parameter and isn't really applicable in
+ environments that require it (e.g., bare metal
+ with hotunpluggable nodes) where hotplugged
+ memory might be exposed via the
+ firmware-provided memory map early during boot
+ to the system instead of getting detected,
+ added and onlined later during boot (such as
+ done by virtio-mem or by some hypervisors
+ implementing emulated DIMMs). As one example, a
+ hotplugged DIMM will be onlined either
+ completely to ZONE_MOVABLE or completely to
+ ZONE_NORMAL, not a mixture.
+ As another example, as many memory blocks
+ belonging to a virtio-mem device will be
+ onlined to ZONE_MOVABLE as possible,
+ special-casing units of memory blocks that can
+ only get hotunplugged together. *This policy
+ does not protect from setups that are
+ problematic with ZONE_MOVABLE and does not
+ change the zone of memory blocks dynamically
+ after they were onlined.*
+``auto_movable_ratio`` read-write: Set the maximum MOVABLE:KERNEL
+ memory ratio in % for the ``auto-movable``
+ online policy. Whether the ratio applies only
+ for the system across all NUMA nodes or also
+ per NUMA nodes depends on the
+ ``auto_movable_numa_aware`` configuration.
+
+ All accounting is based on present memory pages
+ in the zones combined with accounting per
+ memory device. Memory dedicated to the CMA
+ allocator is accounted as MOVABLE, although
+ residing on one of the kernel zones. The
+ possible ratio depends on the actual workload.
+ The kernel default is "301" %, for example,
+ allowing for hotplugging 24 GiB to a 8 GiB VM
+ and automatically onlining all hotplugged
+ memory to ZONE_MOVABLE in many setups. The
+ additional 1% deals with some pages being not
+ present, for example, because of some firmware
+ allocations.
+
+ Note that ZONE_NORMAL memory provided by one
+ memory device does not allow for more
+ ZONE_MOVABLE memory for a different memory
+ device. As one example, onlining memory of a
+ hotplugged DIMM to ZONE_NORMAL will not allow
+ for another hotplugged DIMM to get onlined to
+ ZONE_MOVABLE automatically. In contrast, memory
+ hotplugged by a virtio-mem device that got
+ onlined to ZONE_NORMAL will allow for more
+ ZONE_MOVABLE memory within *the same*
+ virtio-mem device.
+``auto_movable_numa_aware`` read-write: Configure whether the
+ ``auto_movable_ratio`` in the ``auto-movable``
+ online policy also applies per NUMA
+ node in addition to the whole system across all
+ NUMA nodes. The kernel default is "Y".
+
+ Disabling NUMA awareness can be helpful when
+ dealing with NUMA nodes that should be
+ completely hotunpluggable, onlining the memory
+ completely to ZONE_MOVABLE automatically if
+ possible.
+
+ Parameter availability depends on CONFIG_NUMA.
+================================ ===============================================
ZONE_MOVABLE
============
====================================
0 - LOCKED
- page is being locked for exclusive access, e.g. by undergoing read/write IO
+ The page is being locked for exclusive access, e.g. by undergoing read/write
+ IO.
7 - SLAB
- page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator
+ The page is managed by the SLAB/SLOB/SLUB/SLQB kernel memory allocator.
When compound page is used, SLUB/SLQB will only set this flag on the head
page; SLOB will not flag it at all.
10 - BUDDY
- a free memory block managed by the buddy system allocator
+ A free memory block managed by the buddy system allocator.
The buddy system organizes free memory in blocks of various orders.
An order N block has 2^N physically contiguous pages, with the BUDDY flag
set for and _only_ for the first page.
16 - COMPOUND_TAIL
A compound page tail (see description above).
17 - HUGE
- this is an integral part of a HugeTLB page
+ This is an integral part of a HugeTLB page.
19 - HWPOISON
- hardware detected memory corruption on this page: don't touch the data!
+ Hardware detected memory corruption on this page: don't touch the data!
20 - NOPAGE
- no page frame exists at the requested address
+ No page frame exists at the requested address.
21 - KSM
- identical memory pages dynamically shared between one or more processes
+ Identical memory pages dynamically shared between one or more processes.
22 - THP
- contiguous pages which construct transparent hugepages
+ Contiguous pages which construct transparent hugepages.
23 - OFFLINE
- page is logically offline
+ The page is logically offline.
24 - ZERO_PAGE
- zero page for pfn_zero or huge_zero page
+ Zero page for pfn_zero or huge_zero page.
25 - IDLE
- page has not been accessed since it was marked idle (see
+ The page has not been accessed since it was marked idle (see
:ref:`Documentation/admin-guide/mm/idle_page_tracking.rst <idle_page_tracking>`).
Note that this flag may be stale in case the page was accessed via
a PTE. To make sure the flag is up-to-date one has to read
``/sys/kernel/mm/page_idle/bitmap`` first.
26 - PGTABLE
- page is in use as a page table
+ The page is in use as a page table.
IO related page flags
---------------------
1 - ERROR
- IO error occurred
+ IO error occurred.
3 - UPTODATE
- page has up-to-date data
+ The page has up-to-date data.
ie. for file backed page: (in-memory data revision >= on-disk one)
4 - DIRTY
- page has been written to, hence contains new data
+ The page has been written to, hence contains new data.
i.e. for file backed page: (in-memory data revision > on-disk one)
8 - WRITEBACK
- page is being synced to disk
+ The page is being synced to disk.
LRU related page flags
----------------------
5 - LRU
- page is in one of the LRU lists
+ The page is in one of the LRU lists.
6 - ACTIVE
- page is in the active LRU list
+ The page is in the active LRU list.
18 - UNEVICTABLE
- page is in the unevictable (non-)LRU list It is somehow pinned and
+ The page is in the unevictable (non-)LRU list It is somehow pinned and
not a candidate for LRU page reclaims, e.g. ramfs pages,
- shmctl(SHM_LOCK) and mlock() memory segments
+ shmctl(SHM_LOCK) and mlock() memory segments.
2 - REFERENCED
- page has been referenced since last LRU list enqueue/requeue
+ The page has been referenced since last LRU list enqueue/requeue.
9 - RECLAIM
- page will be reclaimed soon after its pageout IO completed
+ The page will be reclaimed soon after its pageout IO completed.
11 - MMAP
- a memory mapped page
+ A memory mapped page.
12 - ANON
- a memory mapped page that is not part of a file
+ A memory mapped page that is not part of a file.
13 - SWAPCACHE
- page is mapped to swap space, i.e. has an associated swap entry
+ The page is mapped to swap space, i.e. has an associated swap entry.
14 - SWAPBACKED
- page is backed by swap/RAM
+ The page is backed by swap/RAM.
The page-types tool in the tools/vm directory can be used to query the
above flags.
===============
Enables/disables task delay accounting (see
-:doc:`accounting/delay-accounting.rst`). Enabling this feature incurs
+Documentation/accounting/delay-accounting.rst. Enabling this feature incurs
a small amount of overhead in the scheduler but is useful for debugging
and performance tuning. It is required by some tools such as iotop.
Not supported by the Linux kernel.
+ Homepage:
+ https://web.archive.org/web/20110924171043/http://www.marvell.com/embedded-processors/discovery-innovation/
Core:
Feroceon 88fr571-vd ARMv5 compatible
- 88F6707
- 88F6W11
+ - Product infos: https://web.archive.org/web/20141002083258/http://www.marvell.com/embedded-processors/armada-370/
- Product Brief: https://web.archive.org/web/20121115063038/http://www.marvell.com/embedded-processors/armada-300/assets/Marvell_ARMADA_370_SoC.pdf
- Hardware Spec: https://web.archive.org/web/20140617183747/http://www.marvell.com/embedded-processors/armada-300/assets/ARMADA370-datasheet.pdf
- Functional Spec: https://web.archive.org/web/20140617183701/http://www.marvell.com/embedded-processors/armada-300/assets/ARMADA370-FunctionalSpec-datasheet.pdf
Core:
Sheeva ARMv7 compatible PJ4B
+ Armada XP Flavors:
+ - MV78230
+ - MV78260
+ - MV78460
+
+ NOTE:
+ not to be confused with the non-SMP 78xx0 SoCs
+
+ - Product infos: https://web.archive.org/web/20150101215721/http://www.marvell.com/embedded-processors/armada-xp/
+ - Product Brief: https://web.archive.org/web/20121021173528/http://www.marvell.com/embedded-processors/armada-xp/assets/Marvell-ArmadaXP-SoC-product%20brief.pdf
+ - Functional Spec: https://web.archive.org/web/20180829171131/http://www.marvell.com/embedded-processors/armada-xp/assets/ARMADA-XP-Functional-SpecDatasheet.pdf
+ - Hardware Specs:
+ - https://web.archive.org/web/20141127013651/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78230_OS.PDF
+ - https://web.archive.org/web/20141222000224/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78260_OS.PDF
+ - https://web.archive.org/web/20141222000230/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78460_OS.PDF
+
+ Core:
+ Sheeva ARMv7 compatible Dual-core or Quad-core PJ4B-MP
+
Armada 375 Flavors:
- 88F6720
+ - Product infos: https://web.archive.org/web/20140108032402/http://www.marvell.com/embedded-processors/armada-375/
- Product Brief: https://web.archive.org/web/20131216023516/http://www.marvell.com/embedded-processors/armada-300/assets/ARMADA_375_SoC-01_product_brief.pdf
Core:
Core:
ARM Cortex-A9
- Armada XP Flavors:
- - MV78230
- - MV78260
- - MV78460
-
- NOTE:
- not to be confused with the non-SMP 78xx0 SoCs
-
- Product Brief:
- https://web.archive.org/web/20121021173528/http://www.marvell.com/embedded-processors/armada-xp/assets/Marvell-ArmadaXP-SoC-product%20brief.pdf
-
- Functional Spec:
- https://web.archive.org/web/20180829171131/http://www.marvell.com/embedded-processors/armada-xp/assets/ARMADA-XP-Functional-SpecDatasheet.pdf
-
- - Hardware Specs:
-
- - https://web.archive.org/web/20141127013651/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78230_OS.PDF
- - https://web.archive.org/web/20141222000224/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78260_OS.PDF
- - https://web.archive.org/web/20141222000230/http://www.marvell.com/embedded-processors/armada-xp/assets/HW_MV78460_OS.PDF
-
- Core:
- Sheeva ARMv7 compatible Dual-core or Quad-core PJ4B-MP
-
Linux kernel mach directory:
arch/arm/mach-mvebu
Linux kernel plat directory:
- Flavors:
- 88DE3010, Armada 1000 (no Linux support)
- Core: Marvell PJ1 (ARMv5TE), Dual-core
- - Product Brief: http://www.marvell.com.cn/digital-entertainment/assets/armada_1000_pb.pdf
+ - Product Brief: https://web.archive.org/web/20131103162620/http://www.marvell.com/digital-entertainment/assets/armada_1000_pb.pdf
- 88DE3005, Armada 1500 Mini
- Design name: BG2CD
- Core: ARM Cortex-A9, PL310 L2CC
libbpf
======
-Documentation/bpf/libbpf/libbpf.rst is a userspace library for loading and interacting with bpf programs.
+Documentation/bpf/libbpf/index.rst is a userspace library for loading and interacting with bpf programs.
BPF Type Format (BTF)
=====================
unsigned long start_pfn;
unsigned long nr_pages;
int status_change_nid_normal;
- int status_change_nid_high;
int status_change_nid;
}
- nr_pages is # of pages of online/offline memory.
- status_change_nid_normal is set node id when N_NORMAL_MEMORY of nodemask
is (will be) set/clear, if this is -1, then nodemask status is not changed.
-- status_change_nid_high is set node id when N_HIGH_MEMORY of nodemask
- is (will be) set/clear, if this is -1, then nodemask status is not changed.
- status_change_nid is set node id when N_MEMORY of nodemask is (will be)
set/clear. It means a new(memoryless) node gets new memory by online and a
node loses all memory. If this is -1, then nodemask status is not changed.
#include <sys/mman.h>
#include <unistd.h>
#include <fcntl.h>
+ #include <linux/types.h>
#define KCOV_INIT_TRACE _IOR('c', 1, unsigned long)
#define KCOV_ENABLE _IO('c', 100)
/* Read number of comparisons collected. */
n = __atomic_load_n(&cover[0], __ATOMIC_RELAXED);
for (i = 0; i < n; i++) {
+ uint64_t ip;
+
type = cover[i * KCOV_WORDS_PER_CMP + 1];
/* arg1 and arg2 - operands of the comparison. */
arg1 = cover[i * KCOV_WORDS_PER_CMP + 2];
.. code-block:: c
+ /* Same includes and defines as above. */
+
struct kcov_remote_arg {
__u32 trace_mode;
__u32 area_size;
of the sample interval, the next allocation through the main allocator (SLAB or
SLUB) returns a guarded allocation from the KFENCE object pool (allocation
sizes up to PAGE_SIZE are supported). At this point, the timer is reset, and
-the next allocation is set up after the expiration of the interval. To "gate" a
-KFENCE allocation through the main allocator's fast-path without overhead,
-KFENCE relies on static branches via the static keys infrastructure. The static
-branch is toggled to redirect the allocation to KFENCE.
+the next allocation is set up after the expiration of the interval.
+
+When using ``CONFIG_KFENCE_STATIC_KEYS=y``, KFENCE allocations are "gated"
+through the main allocator's fast-path by relying on static branches via the
+static keys infrastructure. The static branch is toggled to redirect the
+allocation to KFENCE. Depending on sample interval, target workloads, and
+system architecture, this may perform better than the simple dynamic branch.
+Careful benchmarking is recommended.
KFENCE objects each reside on a dedicated page, at either the left or right
page boundaries selected at random. The pages to the left and right of the
first, and the chances of detecting use-after-frees of recently freed objects
is increased.
+If pool utilization reaches 75% (default) or above, to reduce the risk of the
+pool eventually being fully occupied by allocated objects yet ensure diverse
+coverage of allocations, KFENCE limits currently covered allocations of the
+same source from further filling up the pool. The "source" of an allocation is
+based on its partial allocation stack trace. A side-effect is that this also
+limits frequent long-lived allocations (e.g. pagecache) of the same source
+filling up the pool permanently, which is the most common risk for the pool
+becoming full and the sampled allocation rate dropping to zero. The threshold
+at which to start limiting currently covered allocations can be configured via
+the boot parameter ``kfence.skip_covered_thresh`` (pool usage%).
+
Interface
---------
title: ST STi Platforms Device Tree Bindings
maintainers:
- - Patrice Chotard <patrice.chotard@st.com>
+ - Patrice Chotard <patrice.chotard@foss.st.com>
properties:
$nodename:
title: STMicroelectronics STM32 ML-AHB interconnect bindings
maintainers:
- - Fabien Dessenne <fabien.dessenne@st.com>
- - Arnaud Pouliquen <arnaud.pouliquen@st.com>
+ - Fabien Dessenne <fabien.dessenne@foss.st.com>
+ - Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
description: |
These bindings describe the STM32 SoCs ML-AHB interconnect bus which connects
title: STMicroelectronics STM32 Platforms System Controller bindings
maintainers:
- - Alexandre Torgue <alexandre.torgue@st.com>
- - Christophe Roullier <christophe.roullier@st.com>
+ - Alexandre Torgue <alexandre.torgue@foss.st.com>
+ - Christophe Roullier <christophe.roullier@foss.st.com>
properties:
compatible:
title: STMicroelectronics STM32 Platforms Device Tree Bindings
maintainers:
- - Alexandre Torgue <alexandre.torgue@st.com>
+ - Alexandre Torgue <alexandre.torgue@foss.st.com>
properties:
$nodename:
properties:
compatible:
- const: holtek,ht16k33
+ oneOf:
+ - items:
+ - enum:
+ - adafruit,3108 # 0.56" 4-Digit 7-Segment FeatherWing Display (Red)
+ - adafruit,3130 # 0.54" Quad Alphanumeric FeatherWing Display (Red)
+ - const: holtek,ht16k33
+
+ - const: holtek,ht16k33 # Generic 16*8 LED controller with dot-matrix display
reg:
maxItems: 1
refresh-rate-hz:
maxItems: 1
- description: Display update interval in Hertz
+ description: Display update interval in Hertz for dot-matrix displays
interrupts:
maxItems: 1
default: 16
description: Initial brightness level
+ led:
+ type: object
+ $ref: /schemas/leds/common.yaml#
+ unevaluatedProperties: false
+
required:
- compatible
- reg
- - refresh-rate-hz
+
+if:
+ properties:
+ compatible:
+ const: holtek,ht16k33
+then:
+ required:
+ - refresh-rate-hz
additionalProperties: false
- |
#include <dt-bindings/interrupt-controller/irq.h>
#include <dt-bindings/input/input.h>
+ #include <dt-bindings/leds/common.h>
i2c1 {
#address-cells = <1>;
#size-cells = <0>;
<MATRIX_KEY(4, 1, KEY_F9)>,
<MATRIX_KEY(5, 1, KEY_F3)>,
<MATRIX_KEY(6, 1, KEY_F1)>;
+
+ led {
+ color = <LED_COLOR_ID_RED>;
+ function = LED_FUNCTION_BACKLIGHT;
+ linux,default-trigger = "backlight";
+ };
};
};
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
cgu: clock-controller@10000000 {
compatible = "ingenic,jz4770-cgu", "simple-mfd";
reg = <0x10000000 0x100>;
max77686: max77686@9 {
compatible = "maxim,max77686";
interrupt-parent = <&wakeup_eint>;
- interrupts = <26 0>;
+ interrupts = <26 IRQ_TYPE_LEVEL_LOW>;
reg = <0x09>;
#clock-cells = <1>;
max77802: max77802@9 {
compatible = "maxim,max77802";
interrupt-parent = <&wakeup_eint>;
- interrupts = <26 0>;
+ interrupts = <26 IRQ_TYPE_LEVEL_LOW>;
reg = <0x09>;
#clock-cells = <1>;
"#clock-cells":
const: 1
+ "#reset-cells":
+ const: 1
+
required:
- compatible
- reg
reg = <0x10000000 0x1000>;
clocks = <&hfclk>, <&rtcclk>;
#clock-cells = <1>;
+ #reset-cells = <1>;
};
Reference
[1] Si5351A/B/C Data Sheet
- https://www.silabs.com/Support%20Documents/TechnicalDocs/Si5351.pdf
+ https://www.skyworksinc.com/-/media/Skyworks/SL/documents/public/data-sheets/Si5351-B.pdf
The Si5351a/b/c are programmable i2c clock generators with up to 8 output
clocks. Si5351a also has a reduced pin-count package (MSOP10) where only
- socionext,uniphier-ld11-clock
- socionext,uniphier-ld20-clock
- socionext,uniphier-pxs3-clock
+ - socionext,uniphier-nx1-clock
- description: Media I/O (MIO) clock, SD clock
enum:
- socionext,uniphier-ld4-mio-clock
- socionext,uniphier-ld11-mio-clock
- socionext,uniphier-ld20-sd-clock
- socionext,uniphier-pxs3-sd-clock
+ - socionext,uniphier-nx1-sd-clock
- description: Peripheral clock
enum:
- socionext,uniphier-ld4-peri-clock
- socionext,uniphier-ld11-peri-clock
- socionext,uniphier-ld20-peri-clock
- socionext,uniphier-pxs3-peri-clock
+ - socionext,uniphier-nx1-peri-clock
+ - description: SoC-glue clock
+ enum:
+ - socionext,uniphier-pro4-sg-clock
"#clock-cells":
const: 1
title: Reset Clock Controller Binding
maintainers:
- - Gabriel Fernandez <gabriel.fernandez@st.com>
+ - Gabriel Fernandez <gabriel.fernandez@foss.st.com>
description: |
The RCC IP is both a reset and a clock controller.
title: STMicroelectronics STM32 CRC bindings
maintainers:
- - Lionel Debieve <lionel.debieve@st.com>
+ - Lionel Debieve <lionel.debieve@foss.st.com>
properties:
compatible:
title: STMicroelectronics STM32 CRYP bindings
maintainers:
- - Lionel Debieve <lionel.debieve@st.com>
+ - Lionel Debieve <lionel.debieve@foss.st.com>
properties:
compatible:
title: STMicroelectronics STM32 HASH bindings
maintainers:
- - Lionel Debieve <lionel.debieve@st.com>
+ - Lionel Debieve <lionel.debieve@foss.st.com>
properties:
compatible:
properties:
port@0:
- $ref: /schemas/graph.yaml#/properties/port
+ $ref: /schemas/graph.yaml#/$defs/port-base
description: |
For LVDS encoders, port 0 is the parallel input
For LVDS decoders, port 0 is the LVDS input
+ properties:
+ endpoint:
+ $ref: /schemas/media/video-interfaces.yaml#
+ unevaluatedProperties: false
+
+ properties:
+ data-mapping:
+ enum:
+ - jeida-18
+ - jeida-24
+ - vesa-24
+ description: |
+ The color signals mapping order. See details in
+ Documentation/devicetree/bindings/display/panel/lvds.yaml
+
port@1:
$ref: /schemas/graph.yaml#/properties/port
description: |
power-supply: true
+if:
+ not:
+ properties:
+ compatible:
+ contains:
+ const: lvds-decoder
+then:
+ properties:
+ ports:
+ properties:
+ port@0:
+ properties:
+ endpoint:
+ properties:
+ data-mapping: false
+
required:
- compatible
- ports
vdd33-supply:
description: Regulator for 3.3V digital core power.
+ aux-bus:
+ $ref: /schemas/display/dp-aux-bus.yaml#
+
ports:
$ref: /schemas/graph.yaml#/properties/ports
reg = <1>;
ps8640_out: endpoint {
remote-endpoint = <&panel_in>;
- };
+ };
+ };
+ };
+
+ aux-bus {
+ panel {
+ compatible = "boe,nv133fhm-n62";
+ power-supply = <&pp3300_dx_edp>;
+ backlight = <&backlight>;
+
+ port {
+ panel_in: endpoint {
+ remote-endpoint = <&ps8640_out>;
+ };
+ };
};
};
};
title: Synopsys DesignWare MIPI DSI host controller
maintainers:
- - Philippe CORNU <philippe.cornu@st.com>
+ - Philippe CORNU <philippe.cornu@foss.st.com>
description: |
This document defines device tree properties for the Synopsys DesignWare MIPI
+++ /dev/null
-Toshiba TC358767 eDP bridge bindings
-
-Required properties:
- - compatible: "toshiba,tc358767"
- - reg: i2c address of the bridge, 0x68 or 0x0f, depending on bootstrap pins
- - clock-names: should be "ref"
- - clocks: OF device-tree clock specification for refclk input. The reference
- clock rate must be 13 MHz, 19.2 MHz, 26 MHz, or 38.4 MHz.
-
-Optional properties:
- - shutdown-gpios: OF device-tree gpio specification for SD pin
- (active high shutdown input)
- - reset-gpios: OF device-tree gpio specification for RSTX pin
- (active low system reset)
- - toshiba,hpd-pin: TC358767 GPIO pin number to which HPD is connected to (0 or 1)
- - ports: the ports node can contain video interface port nodes to connect
- to a DPI/DSI source and to an eDP/DP sink according to [1][2]:
- - port@0: DSI input port
- - port@1: DPI input port
- - port@2: eDP/DP output port
-
-[1]: Documentation/devicetree/bindings/graph.txt
-[2]: Documentation/devicetree/bindings/media/video-interfaces.txt
-
-Example:
- edp-bridge@68 {
- compatible = "toshiba,tc358767";
- reg = <0x68>;
- shutdown-gpios = <&gpio3 23 GPIO_ACTIVE_HIGH>;
- reset-gpios = <&gpio3 24 GPIO_ACTIVE_LOW>;
- clock-names = "ref";
- clocks = <&edp_refclk>;
-
- ports {
- #address-cells = <1>;
- #size-cells = <0>;
-
- port@1 {
- reg = <1>;
-
- bridge_in: endpoint {
- remote-endpoint = <&dpi_out>;
- };
- };
-
- port@2 {
- reg = <2>;
-
- bridge_out: endpoint {
- remote-endpoint = <&panel_in>;
- };
- };
- };
- };
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/display/bridge/toshiba,tc358767.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Toshiba TC358767 eDP bridge bindings
+
+maintainers:
+ - Andrey Gusakov <andrey.gusakov@cogentembedded.com>
+
+description: The TC358767 is bridge device which converts DSI/DPI to eDP/DP
+
+properties:
+ compatible:
+ const: toshiba,tc358767
+
+ reg:
+ enum:
+ - 0x68
+ - 0x0f
+ description: |
+ i2c address of the bridge, 0x68 or 0x0f, depending on bootstrap pins
+
+ clock-names:
+ const: "ref"
+
+ clocks:
+ maxItems: 1
+ description: |
+ OF device-tree clock specification for refclk input. The reference.
+ clock rate must be 13 MHz, 19.2 MHz, 26 MHz, or 38.4 MHz.
+
+ shutdown-gpios:
+ maxItems: 1
+ description: |
+ OF device-tree gpio specification for SD pin(active high shutdown input)
+
+ reset-gpios:
+ maxItems: 1
+ description: |
+ OF device-tree gpio specification for RSTX pin(active low system reset)
+
+ toshiba,hpd-pin:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ enum:
+ - 0
+ - 1
+ description: TC358767 GPIO pin number to which HPD is connected to (0 or 1)
+
+ ports:
+ $ref: /schemas/graph.yaml#/properties/ports
+
+ properties:
+ port@0:
+ $ref: /schemas/graph.yaml#/properties/port
+ description: |
+ DSI input port. The remote endpoint phandle should be a
+ reference to a valid DSI output endpoint node
+
+ port@1:
+ $ref: /schemas/graph.yaml#/properties/port
+ description: |
+ DPI input port. The remote endpoint phandle should be a
+ reference to a valid DPI output endpoint node
+
+ port@2:
+ $ref: /schemas/graph.yaml#/properties/port
+ description: |
+ eDP/DP output port. The remote endpoint phandle should be a
+ reference to a valid eDP panel input endpoint node. This port is
+ optional, treated as DP panel if not defined
+
+ oneOf:
+ - required:
+ - port@0
+ - required:
+ - port@1
+
+
+required:
+ - compatible
+ - reg
+ - clock-names
+ - clocks
+ - ports
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+
+ /* DPI input and eDP output */
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ edp-bridge@68 {
+ compatible = "toshiba,tc358767";
+ reg = <0x68>;
+ shutdown-gpios = <&gpio3 23 GPIO_ACTIVE_HIGH>;
+ reset-gpios = <&gpio3 24 GPIO_ACTIVE_LOW>;
+ clock-names = "ref";
+ clocks = <&edp_refclk>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@1 {
+ reg = <1>;
+
+ bridge_in_0: endpoint {
+ remote-endpoint = <&dpi_out>;
+ };
+ };
+
+ port@2 {
+ reg = <2>;
+
+ bridge_out: endpoint {
+ remote-endpoint = <&panel_in>;
+ };
+ };
+ };
+ };
+ };
+ - |
+ /* DPI input and DP output */
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ edp-bridge@68 {
+ compatible = "toshiba,tc358767";
+ reg = <0x68>;
+ shutdown-gpios = <&gpio3 23 GPIO_ACTIVE_HIGH>;
+ reset-gpios = <&gpio3 24 GPIO_ACTIVE_LOW>;
+ clock-names = "ref";
+ clocks = <&edp_refclk>;
+
+ ports {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ port@1 {
+ reg = <1>;
+
+ bridge_in_1: endpoint {
+ remote-endpoint = <&dpi_out>;
+ };
+ };
+ };
+ };
+ };
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
ipu@13080000 {
compatible = "ingenic,jz4770-ipu", "ingenic,jz4760-ipu";
reg = <0x13080000 0x800>;
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
lcd-controller@13050000 {
compatible = "ingenic,jz4740-lcd";
reg = <0x13050000 0x1000>;
};
- |
- #include <dt-bindings/clock/jz4725b-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4725b-cgu.h>
lcd-controller@13050000 {
compatible = "ingenic,jz4725b-lcd";
reg = <0x13050000 0x1000>;
title: Orise Tech OTM8009A 3.97" 480x800 TFT LCD panel (MIPI-DSI video mode)
maintainers:
- - Philippe CORNU <philippe.cornu@st.com>
+ - Philippe CORNU <philippe.cornu@foss.st.com>
description: |
The Orise Tech OTM8009A is a 3.97" 480x800 TFT LCD panel connected using
- innolux,at070tn92
# Innolux G070Y2-L01 7" WVGA (800x480) TFT LCD panel
- innolux,g070y2-l01
+ # Innolux G070Y2-T02 7" WVGA (800x480) TFT LCD TTL panel
+ - innolux,g070y2-t02
# Innolux Corporation 10.1" G101ICE-L01 WXGA (1280x800) LVDS panel
- innolux,g101ice-l01
# Innolux Corporation 12.1" WXGA (1280x800) TFT LCD panel
- urt,umsh-8596md-11t
- urt,umsh-8596md-19t
- urt,umsh-8596md-20t
+ # Vivax TPC-9150 tablet 9.0" WSVGA TFT LCD panel
+ - vivax,tpc9150-panel
# VXT 800x480 color TFT LCD panel
- vxt,vl050-8048nt-c01
# Winstar Display Corporation 3.5" QVGA (320x240) TFT LCD panel
- yes-optoelectronics,ytc700tlag-05-201c
backlight: true
+ ddc-i2c-bus: true
enable-gpios: true
port: true
power-supply: true
title: Raydium Semiconductor Corporation RM68200 5.5" 720p MIPI-DSI TFT LCD panel
maintainers:
- - Philippe CORNU <philippe.cornu@st.com>
+ - Philippe CORNU <philippe.cornu@foss.st.com>
description: |
The Raydium Semiconductor Corporation RM68200 is a 5.5" 720x1280 TFT LCD
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/display/panel/sharp,ls060t1sx01.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Sharp Microelectronics 6.0" FullHD TFT LCD panel
+
+maintainers:
+ - Dmitry Baryskov <dmitry.baryshkov@linaro.org>
+
+allOf:
+ - $ref: panel-common.yaml#
+
+properties:
+ compatible:
+ const: sharp,ls060t1sx01
+
+ reg: true
+ backlight: true
+ reset-gpios: true
+ port: true
+
+ avdd-supply:
+ description: handle of the regulator that provides the positive supply voltage
+ avee-supply:
+ description: handle of the regulator that provides the negative supply voltage
+ vddi-supply:
+ description: handle of the regulator that provides the I/O supply voltage
+ vddh-supply:
+ description: handle of the regulator that provides the analog supply voltage
+
+required:
+ - compatible
+ - reg
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+
+ dsi {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ panel@0 {
+ compatible = "sharp,ls060t1sx01";
+ reg = <0>;
+ avdd-supply = <&pm8941_l22>;
+ backlight = <&backlight>;
+ reset-gpios = <&pm8916_gpios 25 GPIO_ACTIVE_LOW>;
+ };
+ };
+
+...
title: STMicroelectronics STM32 DSI host controller
maintainers:
- - Philippe Cornu <philippe.cornu@st.com>
- - Yannick Fertre <yannick.fertre@st.com>
+ - Philippe Cornu <philippe.cornu@foss.st.com>
+ - Yannick Fertre <yannick.fertre@foss.st.com>
description:
The STMicroelectronics STM32 DSI controller uses the Synopsys DesignWare MIPI-DSI host controller.
title: STMicroelectronics STM32 lcd-tft display controller
maintainers:
- - Philippe Cornu <philippe.cornu@st.com>
- - Yannick Fertre <yannick.fertre@st.com>
+ - Philippe Cornu <philippe.cornu@foss.st.com>
+ - Yannick Fertre <yannick.fertre@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
dma: dma-controller@13420000 {
compatible = "ingenic,jz4780-dma";
reg = <0x13420000 0x400>, <0x13421000 0x40>;
the secure world.
- qcom,controlled-remotely : optional, indicates that the bam is controlled by
remote proccessor i.e. execution environment.
+- qcom,powered-remotely : optional, indicates that the bam is powered up by
+ a remote processor but must be initialized by the local processor.
- num-channels : optional, indicates supported number of DMA channels in a
remotely controlled bam.
- qcom,num-ees : optional, indicates supported number of Execution Environments
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
allOf:
- $ref: "dma-controller.yaml#"
title: STMicroelectronics STM32 DMA MUX (DMA request router) bindings
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
allOf:
- $ref: "dma-router.yaml#"
if no HW ack signal is used by the MDMA client
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
allOf:
- $ref: "dma-controller.yaml#"
maintainers:
- Daniel Baluta <daniel.baluta@nxp.com>
+ - Shengjiu Wang <shengjiu.wang@nxp.com>
description: |
Some boards from i.MX8 family contain a DSP core used for
- fsl,imx8qxp-dsp
- fsl,imx8qm-dsp
- fsl,imx8mp-dsp
+ - fsl,imx8qxp-hifi4
+ - fsl,imx8qm-hifi4
+ - fsl,imx8mp-hifi4
+ - fsl,imx8ulp-hifi4
reg:
maxItems: 1
- description: ipg clock
- description: ocram clock
- description: core clock
+ - description: debug interface clock
+ - description: message unit clock
+ minItems: 3
clock-names:
items:
- const: ipg
- const: ocram
- const: core
+ - const: debug
+ - const: mu
+ minItems: 3
power-domains:
description:
List of phandle and PM domain specifier as documented in
Documentation/devicetree/bindings/power/power_domain.txt
+ minItems: 1
maxItems: 4
mboxes:
description:
List of <&phandle type channel> - 2 channels for TXDB, 2 channels for RXDB
+ or - 1 channel for TX, 1 channel for RX, 1 channel for RXDB
(see mailbox/fsl,mu.txt)
+ minItems: 3
maxItems: 4
mbox-names:
- items:
- - const: txdb0
- - const: txdb1
- - const: rxdb0
- - const: rxdb1
+ minItems: 3
+ maxItems: 4
memory-region:
description:
phandle to a node describing reserved memory (System RAM memory)
used by DSP (see bindings/reserved-memory/reserved-memory.txt)
- maxItems: 1
+ minItems: 1
+ maxItems: 4
+
+ firmware-name:
+ description: |
+ Default name of the firmware to load to the remote processor.
+
+ fsl,dsp-ctrl:
+ $ref: /schemas/types.yaml#/definitions/phandle
+ description:
+ Phandle to syscon block which provide access for processor enablement
required:
- compatible
- mbox-names
- memory-region
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - fsl,imx8qxp-dsp
+ - fsl,imx8qm-dsp
+ - fsl,imx8qxp-hifi4
+ - fsl,imx8qm-hifi4
+ then:
+ properties:
+ power-domains:
+ minItems: 4
+ else:
+ properties:
+ power-domains:
+ maxItems: 1
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - fsl,imx8qxp-hifi4
+ - fsl,imx8qm-hifi4
+ - fsl,imx8mp-hifi4
+ - fsl,imx8ulp-hifi4
+ then:
+ properties:
+ memory-region:
+ minItems: 4
+ mboxes:
+ maxItems: 3
+ mbox-names:
+ items:
+ - const: tx
+ - const: rx
+ - const: rxdb
+ else:
+ properties:
+ memory-region:
+ maxItems: 1
+ mboxes:
+ minItems: 4
+ mbox-names:
+ items:
+ - const: txdb0
+ - const: txdb1
+ - const: rxdb0
+ - const: rxdb1
+
additionalProperties: false
examples:
mboxes = <&lsio_mu13 2 0>, <&lsio_mu13 2 1>, <&lsio_mu13 3 0>, <&lsio_mu13 3 1>;
memory-region = <&dsp_reserved>;
};
+ - |
+ #include <dt-bindings/clock/imx8mp-clock.h>
+ dsp_reserved: dsp@92400000 {
+ reg = <0x92400000 0x1000000>;
+ no-map;
+ };
+ dsp_vdev0vring0: vdev0vring0@942f0000 {
+ reg = <0x942f0000 0x8000>;
+ no-map;
+ };
+ dsp_vdev0vring1: vdev0vring1@942f8000 {
+ reg = <0x942f8000 0x8000>;
+ no-map;
+ };
+ dsp_vdev0buffer: vdev0buffer@94300000 {
+ compatible = "shared-dma-pool";
+ reg = <0x94300000 0x100000>;
+ no-map;
+ };
+
+ dsp: dsp@3b6e8000 {
+ compatible = "fsl,imx8mp-hifi4";
+ reg = <0x3b6e8000 0x88000>;
+ clocks = <&audio_blk_ctrl IMX8MP_CLK_AUDIOMIX_DSP_ROOT>,
+ <&audio_blk_ctrl IMX8MP_CLK_AUDIOMIX_OCRAMA_IPG>,
+ <&audio_blk_ctrl IMX8MP_CLK_AUDIOMIX_DSP_ROOT>,
+ <&audio_blk_ctrl IMX8MP_CLK_AUDIOMIX_DSPDBG_ROOT>;
+ clock-names = "ipg", "ocram", "core", "debug";
+ firmware-name = "imx/dsp/hifi4.bin";
+ power-domains = <&audiomix_pd>;
+ mbox-names = "tx", "rx", "rxdb";
+ mboxes = <&mu2 0 0>,
+ <&mu2 1 0>,
+ <&mu2 3 0>;
+ memory-region = <&dsp_vdev0buffer>, <&dsp_vdev0vring0>,
+ <&dsp_vdev0vring1>, <&dsp_reserved>;
+ fsl,dsp-ctrl = <&audio_blk_ctrl>;
+ };
- const: nxp,se97b
- const: atmel,24c02
- items:
+ - const: onnn,cat24c04
+ - const: atmel,24c04
+ - items:
+ - const: onnn,cat24c05
+ - const: atmel,24c04
+ - items:
- const: renesas,r1ex24002
- const: atmel,24c02
- items:
+++ /dev/null
-AXP209 GPIO & pinctrl controller
-
-This driver follows the usual GPIO bindings found in
-Documentation/devicetree/bindings/gpio/gpio.txt
-
-This driver follows the usual pinctrl bindings found in
-Documentation/devicetree/bindings/pinctrl/pinctrl-bindings.txt
-
-This driver employs the per-pin muxing pattern.
-
-Required properties:
-- compatible: Should be one of:
- - "x-powers,axp209-gpio"
- - "x-powers,axp813-gpio"
-- #gpio-cells: Should be two. The first cell is the pin number and the
- second is the GPIO flags.
-- gpio-controller: Marks the device node as a GPIO controller.
-
-This node must be a subnode of the axp20x PMIC, documented in
-Documentation/devicetree/bindings/mfd/axp20x.txt
-
-Example:
-
-axp209: pmic@34 {
- compatible = "x-powers,axp209";
- reg = <0x34>;
- interrupt-parent = <&nmi_intc>;
- interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
- interrupt-controller;
- #interrupt-cells = <1>;
-
- axp_gpio: gpio {
- compatible = "x-powers,axp209-gpio";
- gpio-controller;
- #gpio-cells = <2>;
- };
-};
-
-The GPIOs can be muxed to other functions and therefore, must be a subnode of
-axp_gpio.
-
-Example:
-
-&axp_gpio {
- gpio0_adc: gpio0-adc {
- pins = "GPIO0";
- function = "adc";
- };
-};
-
-&example_node {
- pinctrl-names = "default";
- pinctrl-0 = <&gpio0_adc>;
-};
-
-GPIOs and their functions
--------------------------
-
-Each GPIO is independent from the other (i.e. GPIO0 in gpio_in function does
-not force GPIO1 and GPIO2 to be in gpio_in function as well).
-
-axp209
-------
-GPIO | Functions
-------------------------
-GPIO0 | gpio_in, gpio_out, ldo, adc
-GPIO1 | gpio_in, gpio_out, ldo, adc
-GPIO2 | gpio_in, gpio_out
-
-axp813
-------
-GPIO | Functions
-------------------------
-GPIO0 | gpio_in, gpio_out, ldo, adc
-GPIO1 | gpio_in, gpio_out, ldo
+++ /dev/null
-Netlogic XLP Family GPIO
-========================
-
-This GPIO driver is used for following Netlogic XLP SoCs:
- XLP832, XLP316, XLP208, XLP980, XLP532
-This GPIO driver is also compatible with GPIO controller found on
-Broadcom Vulcan ARM64.
-
-Required properties:
--------------------
-
-- compatible: Should be one of the following:
- - "netlogic,xlp832-gpio": For Netlogic XLP832
- - "netlogic,xlp316-gpio": For Netlogic XLP316
- - "netlogic,xlp208-gpio": For Netlogic XLP208
- - "netlogic,xlp980-gpio": For Netlogic XLP980
- - "netlogic,xlp532-gpio": For Netlogic XLP532
- - "brcm,vulcan-gpio": For Broadcom Vulcan ARM64
-- reg: Physical base address and length of the controller's registers.
-- #gpio-cells: Should be two. The first cell is the pin number and the second
- cell is used to specify optional parameters (currently unused).
-- gpio-controller: Marks the device node as a GPIO controller.
-- nr-gpios: Number of GPIO pins supported by the controller.
-- interrupt-cells: Should be two. The first cell is the GPIO Number. The
- second cell is used to specify flags. The following subset of flags is
- supported:
- - trigger type:
- 1 = low to high edge triggered.
- 2 = high to low edge triggered.
- 4 = active high level-sensitive.
- 8 = active low level-sensitive.
-- interrupts: Interrupt number for this device.
-- interrupt-controller: Identifies the node as an interrupt controller.
-
-Example:
-
- gpio: xlp_gpio@34000 {
- compatible = "netlogic,xlp316-gpio";
- reg = <0 0x34100 0x1000
- 0 0x35100 0x1000>;
- #gpio-cells = <2>;
- gpio-controller;
- nr-gpios = <57>;
-
- #interrupt-cells = <2>;
- interrupt-parent = <&pic>;
- interrupts = <39>;
- interrupt-controller;
- };
gpio-controller: true
+ gpio-line-names: true
+
"#gpio-cells":
const: 2
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/gpio/x-powers,axp209-gpio.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: X-Powers AXP209 GPIO Device Tree Bindings
+
+maintainers:
+ - Chen-Yu Tsai <wens@csie.org>
+
+properties:
+ "#gpio-cells":
+ const: 2
+ description: >
+ The first cell is the pin number and the second is the GPIO flags.
+
+ compatible:
+ oneOf:
+ - enum:
+ - x-powers,axp209-gpio
+ - x-powers,axp813-gpio
+ - items:
+ - const: x-powers,axp803-gpio
+ - const: x-powers,axp813-gpio
+
+ gpio-controller: true
+
+patternProperties:
+ "^.*-pins?$":
+ $ref: /schemas/pinctrl/pinmux-node.yaml#
+
+ properties:
+ pins:
+ items:
+ enum:
+ - GPIO0
+ - GPIO1
+ - GPIO2
+
+ function:
+ enum:
+ - adc
+ - ldo
+ - gpio_in
+ - gpio_out
+
+required:
+ - compatible
+ - "#gpio-cells"
+ - gpio-controller
+
+additionalProperties: false
+
+...
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/gpio/xlnx,zynqmp-gpio-modepin.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: ZynqMP Mode Pin GPIO controller
+
+description:
+ PS_MODE is 4-bits boot mode pins sampled on POR deassertion. Mode Pin
+ GPIO controller with configurable from numbers of pins (from 0 to 3 per
+ PS_MODE). Every pin can be configured as input/output.
+
+maintainers:
+ - Piyush Mehta <piyush.mehta@xilinx.com>
+
+properties:
+ compatible:
+ const: xlnx,zynqmp-gpio-modepin
+
+ gpio-controller: true
+
+ "#gpio-cells":
+ const: 2
+
+required:
+ - compatible
+ - gpio-controller
+ - "#gpio-cells"
+
+additionalProperties: false
+
+examples:
+ - |
+ zynqmp-firmware {
+ gpio {
+ compatible = "xlnx,zynqmp-gpio-modepin";
+ gpio-controller;
+ #gpio-cells = <2>;
+ };
+ };
+
+...
title: STMicroelectronics STM32 Hardware Spinlock bindings
maintainers:
- - Benjamin Gaignard <benjamin.gaignard@st.com>
- - Fabien Dessenne <fabien.dessenne@st.com>
+ - Fabien Dessenne <fabien.dessenne@foss.st.com>
properties:
"#hwlock-cells":
#size-cells = <0>;
axp221: pmic@68 {
- compatible = "x-powers,axp221";
+ /* compatible = "x-powers,axp221"; */
reg = <0x68>;
};
};
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/i2c/apple,i2c.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Apple/PASemi I2C controller
+
+maintainers:
+ - Sven Peter <sven@svenpeter.dev>
+
+description: |
+ Apple SoCs such as the M1 come with a I2C controller based on the one found
+ in machines with P. A. Semi's PWRficient processors.
+ The bus is used to communicate with e.g. USB PD chips or the speaker
+ amp.
+
+allOf:
+ - $ref: /schemas/i2c/i2c-controller.yaml#
+
+properties:
+ compatible:
+ enum:
+ - apple,t8103-i2c
+ - apple,i2c
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ items:
+ - description: I2C bus reference clock
+
+ interrupts:
+ maxItems: 1
+
+ clock-frequency:
+ description: |
+ Desired I2C bus clock frequency in Hz. If not specified, 100 kHz will be
+ used. This frequency is generated by dividing the reference clock.
+ Allowed values are between ref_clk/(16*4) and ref_clk/(16*255).
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - interrupts
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ i2c@35010000 {
+ compatible = "apple,t8103-i2c";
+ reg = <0x35010000 0x4000>;
+ interrupt-parent = <&aic>;
+ interrupts = <0 627 4>;
+ clocks = <&ref_clk>;
+ #address-cells = <1>;
+ #size-cells = <0>;
+ };
const: ipg
clock-frequency:
- enum: [ 100000, 400000 ]
+ minimum: 1
+ default: 100000
+ maximum: 400000
dmas:
items:
+++ /dev/null
-Device tree configuration for the I2C controller on the XLP9xx/5xx SoC
-
-Required properties:
-- compatible : should be "netlogic,xlp980-i2c"
-- reg : bus address start and address range size of device
-- interrupts : interrupt number
-
-Optional properties:
-- clock-frequency : frequency of bus clock in Hz
- Defaults to 100 KHz when the property is not specified
-
-Example:
-
-i2c0: i2c@113100 {
- compatible = "netlogic,xlp980-i2c";
- #address-cells = <1>;
- #size-cells = <0>;
- reg = <0 0x113100 0x100>;
- clock-frequency = <400000>;
- interrupts = <30>;
- interrupt-parent = <&pic>;
-};
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include <dt-bindings/dma/jz4780-dma.h>
#include <dt-bindings/interrupt-controller/irq.h>
i2c@10054000 {
title: I2C controller embedded in STMicroelectronics STM32 I2C platform
maintainers:
- - Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
+ - Pierre-Yves MORDRET <pierre-yves.mordret@foss.st.com>
allOf:
- $ref: /schemas/i2c/i2c-controller.yaml#
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
#include <dt-bindings/iio/adc/ingenic,adc.h>
adc@10070000 {
title: Device-Tree bindings for sigma delta modulator
maintainers:
- - Arnaud Pouliquen <arnaud.pouliquen@st.com>
+ - Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
properties:
compatible:
Each STM32 ADC block can have up to 3 ADC instances.
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
properties:
compatible:
title: STMicroelectronics STM32 DFSDM ADC device driver
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
- - Olivier Moysan <olivier.moysan@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
+ - Olivier Moysan <olivier.moysan@foss.st.com>
description: |
STM32 DFSDM ADC is a sigma delta analog-to-digital converter dedicated to
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/iio/adc/ti,am3359-adc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI AM3359 ADC
+
+maintainers:
+ - Miquel Raynal <miquel.raynal@bootlin.com>
+
+properties:
+ compatible:
+ enum:
+ - ti,am3359-adc
+ - ti,am4372-adc
+
+ '#io-channel-cells':
+ const: 1
+
+ ti,adc-channels:
+ description: List of analog inputs available for ADC. AIN0 = 0, AIN1 = 1 and
+ so on until AIN7 = 7.
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 1
+ maxItems: 8
+
+ ti,chan-step-opendelay:
+ description: List of open delays for each channel of ADC in the order of
+ ti,adc-channels. The value corresponds to the number of ADC clock cycles
+ to wait after applying the step configuration registers and before sending
+ the start of ADC conversion. Maximum value is 0x3FFFF.
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 1
+ maxItems: 8
+
+ ti,chan-step-sampledelay:
+ description: List of sample delays for each channel of ADC in the order of
+ ti,adc-channels. The value corresponds to the number of ADC clock cycles
+ to sample (to hold start of conversion high). Maximum value is 0xFF.
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 1
+ maxItems: 8
+
+ ti,chan-step-avg:
+ description: Number of averages to be performed for each channel of ADC. If
+ average is 16 (this is also the maximum) then input is sampled 16 times
+ and averaged to get more accurate value. This increases the time taken by
+ ADC to generate a sample. Maximum value is 16.
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 1
+ maxItems: 8
+
+required:
+ - compatible
+ - '#io-channel-cells'
+ - ti,adc-channels
+
+additionalProperties: false
+
+examples:
+ - |
+ adc {
+ compatible = "ti,am3359-adc";
+ #io-channel-cells = <1>;
+ ti,adc-channels = <4 5 6 7>;
+ ti,chan-step-opendelay = <0x098 0x3ffff 0x098 0x0>;
+ ti,chan-step-sampledelay = <0xff 0x0 0xf 0x0>;
+ ti,chan-step-avg = <16 2 4 8>;
+ };
current.
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
properties:
compatible:
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/input/cypress-sf.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Cypress StreetFighter touchkey controller
+
+maintainers:
+ - Yassine Oudjana <y.oudjana@protonmail.com>
+
+allOf:
+ - $ref: input.yaml#
+
+properties:
+ compatible:
+ const: cypress,sf3155
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ avdd-supply:
+ description: Regulator for AVDD analog voltage
+
+ vdd-supply:
+ description: Regulator for VDD digital voltage
+
+ linux,keycodes:
+ minItems: 1
+ maxItems: 8
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - avdd-supply
+ - vdd-supply
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/input/input.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ touchkey@28 {
+ compatible = "cypress,sf3155";
+ reg = <0x28>;
+ interrupt-parent = <&msmgpio>;
+ interrupts = <77 IRQ_TYPE_EDGE_FALLING>;
+ avdd-supply = <&vreg_l6a_1p8>;
+ vdd-supply = <&vdd_3v2_tp>;
+ linux,keycodes = <KEY_BACK KEY_MENU>;
+ };
+ };
- vdd-supply: phandle of the regulator that provides the supply voltage.
- post-power-on-delay-ms: time required by the device after enabling its regulators
or powering it on, before it is ready for communication.
+- touchscreen-inverted-x: See touchscreen.txt
+- touchscreen-inverted-y: See touchscreen.txt
Example:
- microchip,cap1106
- microchip,cap1126
- microchip,cap1188
+ - microchip,cap1206
reg:
maxItems: 1
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/input/touchscreen/ti,am3359-tsc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI AM3359 Touchscreen controller
+
+maintainers:
+ - Miquel Raynal <miquel.raynal@bootlin.com>
+
+properties:
+ compatible:
+ const: ti,am3359-tsc
+
+ ti,wires:
+ description: Wires refer to application modes i.e. 4/5/8 wire touchscreen
+ support on the platform.
+ $ref: /schemas/types.yaml#/definitions/uint32
+ enum: [4, 5, 8]
+
+ ti,x-plate-resistance:
+ description: X plate resistance
+ $ref: /schemas/types.yaml#/definitions/uint32
+
+ ti,coordinate-readouts:
+ description: The sequencer supports a total of 16 programmable steps. Each
+ step is used to read a single coordinate. A single readout is enough but
+ multiple reads can increase the quality. A value of 5 means, 5 reads for
+ X, 5 for Y and 2 for Z (always). This utilises 12 of the 16 software steps
+ available. The remaining 4 can be used by the ADC.
+ $ref: /schemas/types.yaml#/definitions/uint32
+ minimum: 1
+ maximum: 6
+
+ ti,wire-config:
+ description: Different boards could have a different order for connecting
+ wires on touchscreen. We need to provide an 8-bit number where the
+ first four bits represent the analog lines and the next 4 bits represent
+ positive/negative terminal on that input line. Notations to represent the
+ input lines and terminals respectively are as follows, AIN0 = 0, AIN1 = 1
+ and so on until AIN7 = 7. XP = 0, XN = 1, YP = 2, YN = 3.
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 4
+ maxItems: 8
+
+ ti,charge-delay:
+ description: Length of touch screen charge delay step in terms of ADC clock
+ cycles. Charge delay value should be large in order to avoid false pen-up
+ events. This value effects the overall sampling speed, hence need to be
+ kept as low as possible, while avoiding false pen-up event. Start from a
+ lower value, say 0x400, and increase value until false pen-up events are
+ avoided. The pen-up detection happens immediately after the charge step,
+ so this does in fact function as a hardware knob for adjusting the amount
+ of "settling time".
+ $ref: /schemas/types.yaml#/definitions/uint32
+
+required:
+ - compatible
+ - ti,wires
+ - ti,x-plate-resistance
+ - ti,coordinate-readouts
+ - ti,wire-config
+
+additionalProperties: false
+
+examples:
+ - |
+ tsc {
+ compatible = "ti,am3359-tsc";
+ ti,wires = <4>;
+ ti,x-plate-resistance = <200>;
+ ti,coordinate-readouts = <5>;
+ ti,wire-config = <0x00 0x11 0x22 0x33>;
+ ti,charge-delay = <0x400>;
+ };
+++ /dev/null
-* TI - TSC ADC (Touschscreen and analog digital converter)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Required properties:
-- mfd
- compatible: Should be
- "ti,am3359-tscadc" for AM335x/AM437x SoCs
- "ti,am654-tscadc", "ti,am3359-tscadc" for AM654 SoCs
-- child "tsc"
- compatible: Should be "ti,am3359-tsc".
- ti,wires: Wires refer to application modes i.e. 4/5/8 wire touchscreen
- support on the platform.
- ti,x-plate-resistance: X plate resistance
- ti,coordinate-readouts: The sequencer supports a total of 16
- programmable steps each step is used to
- read a single coordinate. A single
- readout is enough but multiple reads can
- increase the quality.
- A value of 5 means, 5 reads for X, 5 for
- Y and 2 for Z (always). This utilises 12
- of the 16 software steps available. The
- remaining 4 can be used by the ADC.
- ti,wire-config: Different boards could have a different order for
- connecting wires on touchscreen. We need to provide an
- 8 bit number where in the 1st four bits represent the
- analog lines and the next 4 bits represent positive/
- negative terminal on that input line. Notations to
- represent the input lines and terminals resoectively
- is as follows:
- AIN0 = 0, AIN1 = 1 and so on till AIN7 = 7.
- XP = 0, XN = 1, YP = 2, YN = 3.
-- child "adc"
- compatible: Should be
- "ti,am3359-adc" for AM335x/AM437x SoCs
- "ti,am654-adc", "ti,am3359-adc" for AM654 SoCs
- ti,adc-channels: List of analog inputs available for ADC.
- AIN0 = 0, AIN1 = 1 and so on till AIN7 = 7.
-
-Optional properties:
-- child "tsc"
- ti,charge-delay: Length of touch screen charge delay step in terms of
- ADC clock cycles. Charge delay value should be large
- in order to avoid false pen-up events. This value
- effects the overall sampling speed, hence need to be
- kept as low as possible, while avoiding false pen-up
- event. Start from a lower value, say 0x400, and
- increase value until false pen-up events are avoided.
- The pen-up detection happens immediately after the
- charge step, so this does in fact function as a
- hardware knob for adjusting the amount of "settling
- time".
-
-- child "adc"
- ti,chan-step-opendelay: List of open delays for each channel of
- ADC in the order of ti,adc-channels. The
- value corresponds to the number of ADC
- clock cycles to wait after applying the
- step configuration registers and before
- sending the start of ADC conversion.
- Maximum value is 0x3FFFF.
- ti,chan-step-sampledelay: List of sample delays for each channel
- of ADC in the order of ti,adc-channels.
- The value corresponds to the number of
- ADC clock cycles to sample (to hold
- start of conversion high).
- Maximum value is 0xFF.
- ti,chan-step-avg: Number of averages to be performed for each
- channel of ADC. If average is 16 then input
- is sampled 16 times and averaged to get more
- accurate value. This increases the time taken
- by ADC to generate a sample. Valid range is 0
- average to 16 averages. Maximum value is 16.
-
-Example:
- tscadc: tscadc@44e0d000 {
- compatible = "ti,am3359-tscadc";
- tsc {
- ti,wires = <4>;
- ti,x-plate-resistance = <200>;
- ti,coordiante-readouts = <5>;
- ti,wire-config = <0x00 0x11 0x22 0x33>;
- ti,charge-delay = <0x400>;
- };
-
- adc {
- ti,adc-channels = <4 5 6 7>;
- ti,chan-step-opendelay = <0x098 0x3ffff 0x098 0x0>;
- ti,chan-step-sampledelay = <0xff 0x0 0xf 0x0>;
- ti,chan-step-avg = <16 2 4 8>;
- };
- }
title: STM32 External Interrupt Controller Device Tree Bindings
maintainers:
- - Alexandre Torgue <alexandre.torgue@st.com>
- - Ludovic Barre <ludovic.barre@st.com>
+ - Alexandre Torgue <alexandre.torgue@foss.st.com>
+ - Ludovic Barre <ludovic.barre@foss.st.com>
properties:
compatible:
channels (N) can be read from a dedicated register.
maintainers:
- - Fabien Dessenne <fabien.dessenne@st.com>
- - Arnaud Pouliquen <arnaud.pouliquen@st.com>
+ - Fabien Dessenne <fabien.dessenne@foss.st.com>
+ - Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
properties:
compatible:
power-domain-names:
minItems: 2
- maxItems: 3
items:
- const: venus
- const: vcodec0
title: STMicroelectronics STM32 CEC bindings
maintainers:
- - Benjamin Gaignard <benjamin.gaignard@st.com>
- - Yannick Fertre <yannick.fertre@st.com>
+ - Yannick Fertre <yannick.fertre@foss.st.com>
properties:
compatible:
title: STMicroelectronics STM32 Digital Camera Memory Interface (DCMI) binding
maintainers:
- - Hugues Fruchet <hugues.fruchet@st.com>
+ - Hugues Fruchet <hugues.fruchet@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include <dt-bindings/gpio/gpio.h>
nemc: memory-controller@13410000 {
compatible = "ingenic,jz4780-nemc";
Select. The FMC2 performs only one access at a time to an external device.
maintainers:
- - Christophe Kerello <christophe.kerello@st.com>
+ - Christophe Kerello <christophe.kerello@foss.st.com>
properties:
compatible:
+++ /dev/null
-X-Powers AC100 Codec/RTC IC Device Tree bindings
-
-AC100 is a audio codec and RTC subsystem combo IC. The 2 parts are
-separated, including power supplies and interrupt lines, but share
-a common register address space and host interface.
-
-Required properties:
-- compatible: "x-powers,ac100"
-- reg: The I2C slave address or RSB hardware address for the chip
-- sub-nodes:
- - codec
- - compatible: "x-powers,ac100-codec"
- - interrupts: SoC NMI / GPIO interrupt connected to the
- IRQ_AUDIO pin
- - #clock-cells: Shall be 0
- - clock-output-names: "4M_adda"
-
- - see clock/clock-bindings.txt for common clock bindings
-
- - rtc
- - compatible: "x-powers,ac100-rtc"
- - clocks: A phandle to the codec's "4M_adda" clock
- - #clock-cells: Shall be 1
- - clock-output-names: "cko1_rtc", "cko2_rtc", "cko3_rtc"
-
- - see clock/clock-bindings.txt for common clock bindings
-
-Example:
-
-ac100: codec@e89 {
- compatible = "x-powers,ac100";
- reg = <0xe89>;
-
- ac100_codec: codec {
- compatible = "x-powers,ac100-codec";
- interrupt-parent = <&r_pio>;
- interrupts = <0 9 IRQ_TYPE_LEVEL_LOW>; /* PL9 */
- #clock-cells = <0>;
- clock-output-names = "4M_adda";
- };
-
- ac100_rtc: rtc {
- compatible = "x-powers,ac100-rtc";
- interrupt-parent = <&nmi_intc>;
- interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
- clocks = <&ac100_codec>;
- #clock-cells = <1>;
- clock-output-names = "cko1_rtc", "cko2_rtc", "cko3_rtc";
- };
-};
+++ /dev/null
-AXP family PMIC device tree bindings
-
-The axp20x family current members :
-axp152 (X-Powers)
-axp202 (X-Powers)
-axp209 (X-Powers)
-axp221 (X-Powers)
-axp223 (X-Powers)
-axp803 (X-Powers)
-axp806 (X-Powers)
-axp809 (X-Powers)
-axp813 (X-Powers)
-
-The AXP813 is 2 chips packaged into 1. The 2 chips do not share anything
-other than the packaging. Pins are routed separately. As such they should
-be treated as separate entities. The other half is an AC100 RTC/codec
-combo chip. Please see ./ac100.txt for its bindings.
-
-Required properties:
-- compatible: should be one of:
- * "x-powers,axp152"
- * "x-powers,axp202"
- * "x-powers,axp209"
- * "x-powers,axp221"
- * "x-powers,axp223"
- * "x-powers,axp803"
- * "x-powers,axp806"
- * "x-powers,axp805", "x-powers,axp806"
- * "x-powers,axp305", "x-powers,axp805", "x-powers,axp806"
- * "x-powers,axp809"
- * "x-powers,axp813"
-- reg: The I2C slave address or RSB hardware address for the AXP chip
-- interrupt-controller: The PMIC has its own internal IRQs
-- #interrupt-cells: Should be set to 1
-
-Supported common regulator properties, see ../regulator/regulator.txt for
-more information:
-- regulator-ramp-delay: sets the ramp up delay in uV/us
- AXP20x/DCDC2: 1600, 800
- AXP20x/LDO3: 1600, 800
-- regulator-soft-start: enable the output at the lowest possible voltage and
- only then set the desired voltage
- AXP20x/LDO3: software-based implementation
-
-Optional properties:
-- interrupts: SoC NMI / GPIO interrupt connected to the PMIC's IRQ pin
-- x-powers,dcdc-freq: defines the work frequency of DC-DC in KHz
- AXP152/20X: range: 750-1875, Default: 1.5 MHz
- AXP22X/8XX: range: 1800-4050, Default: 3 MHz
-
-- x-powers,drive-vbus-en: boolean, set this when the N_VBUSEN pin is
- used as an output pin to control an external
- regulator to drive the OTG VBus, rather then
- as an input pin which signals whether the
- board is driving OTG VBus or not.
- (axp221 / axp223 / axp803/ axp813 only)
-
-- x-powers,self-working-mode and
- x-powers,master-mode: Boolean (axp806 only). Set either of these when the
- PMIC is wired for self-working mode or master mode.
- If neither is set then slave mode is assumed.
- This corresponds to how the MODESET pin is wired.
-
-- <input>-supply: a phandle to the regulator supply node. May be omitted if
- inputs are unregulated, such as using the IPSOUT output
- from the PMIC.
-
-- regulators: A node that houses a sub-node for each regulator. Regulators
- not used but preferred to be managed by the OS should be
- listed as well.
- See Documentation/devicetree/bindings/regulator/regulator.txt
- for more information on standard regulator bindings.
-
-Optional properties for DCDC regulators:
-- x-powers,dcdc-workmode: 1 for PWM mode, 0 for AUTO (PWM/PFM) mode
- Default: Current hardware setting
- The DCDC regulators work in a mixed PWM/PFM mode,
- using PFM under light loads and switching to PWM
- for heavier loads. Forcing PWM mode trades efficiency
- under light loads for lower output noise. This
- probably makes sense for HiFi audio related
- applications that aren't battery constrained.
-
-AXP202/AXP209 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDC2 : DC-DC buck : vin2-supply
-DCDC3 : DC-DC buck : vin3-supply
-LDO1 : LDO : acin-supply : always on
-LDO2 : LDO : ldo24in-supply : shared supply
-LDO3 : LDO : ldo3in-supply
-LDO4 : LDO : ldo24in-supply : shared supply
-LDO5 : LDO : ldo5in-supply
-
-AXP221/AXP223 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDC1 : DC-DC buck : vin1-supply
-DCDC2 : DC-DC buck : vin2-supply
-DCDC3 : DC-DC buck : vin3-supply
-DCDC4 : DC-DC buck : vin4-supply
-DCDC5 : DC-DC buck : vin5-supply
-DC1SW : On/Off Switch : : DCDC1 secondary output
-DC5LDO : LDO : : input from DCDC5
-ALDO1 : LDO : aldoin-supply : shared supply
-ALDO2 : LDO : aldoin-supply : shared supply
-ALDO3 : LDO : aldoin-supply : shared supply
-DLDO1 : LDO : dldoin-supply : shared supply
-DLDO2 : LDO : dldoin-supply : shared supply
-DLDO3 : LDO : dldoin-supply : shared supply
-DLDO4 : LDO : dldoin-supply : shared supply
-ELDO1 : LDO : eldoin-supply : shared supply
-ELDO2 : LDO : eldoin-supply : shared supply
-ELDO3 : LDO : eldoin-supply : shared supply
-LDO_IO0 : LDO : ips-supply : GPIO 0
-LDO_IO1 : LDO : ips-supply : GPIO 1
-RTC_LDO : LDO : ips-supply : always on
-DRIVEVBUS : Enable output : drivevbus-supply : external regulator
-
-AXP803 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDC1 : DC-DC buck : vin1-supply
-DCDC2 : DC-DC buck : vin2-supply : poly-phase capable
-DCDC3 : DC-DC buck : vin3-supply : poly-phase capable
-DCDC4 : DC-DC buck : vin4-supply
-DCDC5 : DC-DC buck : vin5-supply : poly-phase capable
-DCDC6 : DC-DC buck : vin6-supply : poly-phase capable
-DC1SW : On/Off Switch : : DCDC1 secondary output
-ALDO1 : LDO : aldoin-supply : shared supply
-ALDO2 : LDO : aldoin-supply : shared supply
-ALDO3 : LDO : aldoin-supply : shared supply
-DLDO1 : LDO : dldoin-supply : shared supply
-DLDO2 : LDO : dldoin-supply : shared supply
-DLDO3 : LDO : dldoin-supply : shared supply
-DLDO4 : LDO : dldoin-supply : shared supply
-ELDO1 : LDO : eldoin-supply : shared supply
-ELDO2 : LDO : eldoin-supply : shared supply
-ELDO3 : LDO : eldoin-supply : shared supply
-FLDO1 : LDO : fldoin-supply : shared supply
-FLDO2 : LDO : fldoin-supply : shared supply
-LDO_IO0 : LDO : ips-supply : GPIO 0
-LDO_IO1 : LDO : ips-supply : GPIO 1
-RTC_LDO : LDO : ips-supply : always on
-DRIVEVBUS : Enable output : drivevbus-supply : external regulator
-
-AXP806 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDCA : DC-DC buck : vina-supply : poly-phase capable
-DCDCB : DC-DC buck : vinb-supply : poly-phase capable
-DCDCC : DC-DC buck : vinc-supply : poly-phase capable
-DCDCD : DC-DC buck : vind-supply : poly-phase capable
-DCDCE : DC-DC buck : vine-supply : poly-phase capable
-ALDO1 : LDO : aldoin-supply : shared supply
-ALDO2 : LDO : aldoin-supply : shared supply
-ALDO3 : LDO : aldoin-supply : shared supply
-BLDO1 : LDO : bldoin-supply : shared supply
-BLDO2 : LDO : bldoin-supply : shared supply
-BLDO3 : LDO : bldoin-supply : shared supply
-BLDO4 : LDO : bldoin-supply : shared supply
-CLDO1 : LDO : cldoin-supply : shared supply
-CLDO2 : LDO : cldoin-supply : shared supply
-CLDO3 : LDO : cldoin-supply : shared supply
-SW : On/Off Switch : swin-supply
-
-Additionally, the AXP806 DC-DC regulators support poly-phase arrangements
-for higher output current. The possible groupings are: A+B, A+B+C, D+E.
-
-AXP809 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDC1 : DC-DC buck : vin1-supply
-DCDC2 : DC-DC buck : vin2-supply
-DCDC3 : DC-DC buck : vin3-supply
-DCDC4 : DC-DC buck : vin4-supply
-DCDC5 : DC-DC buck : vin5-supply
-DC1SW : On/Off Switch : : DCDC1 secondary output
-DC5LDO : LDO : : input from DCDC5
-ALDO1 : LDO : aldoin-supply : shared supply
-ALDO2 : LDO : aldoin-supply : shared supply
-ALDO3 : LDO : aldoin-supply : shared supply
-DLDO1 : LDO : dldoin-supply : shared supply
-DLDO2 : LDO : dldoin-supply : shared supply
-ELDO1 : LDO : eldoin-supply : shared supply
-ELDO2 : LDO : eldoin-supply : shared supply
-ELDO3 : LDO : eldoin-supply : shared supply
-LDO_IO0 : LDO : ips-supply : GPIO 0
-LDO_IO1 : LDO : ips-supply : GPIO 1
-RTC_LDO : LDO : ips-supply : always on
-SW : On/Off Switch : swin-supply
-
-AXP813 regulators, type, and corresponding input supply names:
-
-Regulator Type Supply Name Notes
---------- ---- ----------- -----
-DCDC1 : DC-DC buck : vin1-supply
-DCDC2 : DC-DC buck : vin2-supply : poly-phase capable
-DCDC3 : DC-DC buck : vin3-supply : poly-phase capable
-DCDC4 : DC-DC buck : vin4-supply
-DCDC5 : DC-DC buck : vin5-supply : poly-phase capable
-DCDC6 : DC-DC buck : vin6-supply : poly-phase capable
-DCDC7 : DC-DC buck : vin7-supply
-ALDO1 : LDO : aldoin-supply : shared supply
-ALDO2 : LDO : aldoin-supply : shared supply
-ALDO3 : LDO : aldoin-supply : shared supply
-DLDO1 : LDO : dldoin-supply : shared supply
-DLDO2 : LDO : dldoin-supply : shared supply
-DLDO3 : LDO : dldoin-supply : shared supply
-DLDO4 : LDO : dldoin-supply : shared supply
-ELDO1 : LDO : eldoin-supply : shared supply
-ELDO2 : LDO : eldoin-supply : shared supply
-ELDO3 : LDO : eldoin-supply : shared supply
-FLDO1 : LDO : fldoin-supply : shared supply
-FLDO2 : LDO : fldoin-supply : shared supply
-FLDO3 : LDO : fldoin-supply : shared supply
-LDO_IO0 : LDO : ips-supply : GPIO 0
-LDO_IO1 : LDO : ips-supply : GPIO 1
-RTC_LDO : LDO : ips-supply : always on
-SW : On/Off Switch : swin-supply
-DRIVEVBUS : Enable output : drivevbus-supply : external regulator
-
-Example:
-
-axp209: pmic@34 {
- compatible = "x-powers,axp209";
- reg = <0x34>;
- interrupt-parent = <&nmi_intc>;
- interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
- interrupt-controller;
- #interrupt-cells = <1>;
-
- regulators {
- x-powers,dcdc-freq = <1500>;
-
- vdd_cpu: dcdc2 {
- regulator-always-on;
- regulator-min-microvolt = <1000000>;
- regulator-max-microvolt = <1450000>;
- regulator-name = "vdd-cpu";
- };
-
- vdd_int_dll: dcdc3 {
- regulator-always-on;
- regulator-min-microvolt = <1000000>;
- regulator-max-microvolt = <1400000>;
- regulator-name = "vdd-int-dll";
- };
-
- vdd_rtc: ldo1 {
- regulator-always-on;
- regulator-min-microvolt = <1200000>;
- regulator-max-microvolt = <1400000>;
- regulator-name = "vdd-rtc";
- };
-
- avcc: ldo2 {
- regulator-always-on;
- regulator-min-microvolt = <2700000>;
- regulator-max-microvolt = <3300000>;
- regulator-name = "avcc";
- };
-
- ldo3 {
- /* unused but preferred to be managed by OS */
- };
- };
-};
'^clock-controller@[a-f0-9]+$':
$ref: ../clock/brcm,iproc-clocks.yaml
+ '^phy@[a-f0-9]+$':
+ $ref: ../phy/bcm-ns-usb2-phy.yaml
+
'^pin-controller@[a-f0-9]+$':
$ref: ../pinctrl/brcm,ns-pinmux.yaml
+ '^syscon@[a-f0-9]+$':
+ $ref: syscon.yaml
+
'^thermal@[a-f0-9]+$':
$ref: ../thermal/brcm,ns-thermal.yaml
examples:
- |
+ #include <dt-bindings/clock/bcm-nsp.h>
cru-bus@1800c100 {
compatible = "brcm,ns-cru", "simple-mfd";
reg = <0x1800c100 0x1d0>;
"iprocfast", "sata1", "sata2";
};
+ phy@164 {
+ compatible = "brcm,ns-usb2-phy";
+ reg = <0x164 0x4>;
+ brcm,syscon-clkset = <&clkset>;
+ clocks = <&genpll BCM_NSP_GENPLL_USB_PHY_REF_CLK>;
+ clock-names = "phy-ref-clk";
+ #phy-cells = <0>;
+ };
+
+ clkset: syscon@180 {
+ compatible = "brcm,cru-clkset", "syscon";
+ reg = <0x180 0x4>;
+ };
+
pin-controller@1c0 {
compatible = "brcm,bcm4708-pinmux";
reg = <0x1c0 0x24>;
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/brcm,misc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Broadcom's MISC block
+
+maintainers:
+ - Rafał Miłecki <rafal@milecki.pl>
+
+description: |
+ Broadcom's MISC is a hardware block used on some SoCs (e.g. bcm63xx and
+ bcm4908). It's used to implement some simple functions like a watchdog, PCIe
+ reset, UniMAC control and more.
+
+properties:
+ compatible:
+ items:
+ - const: brcm,misc
+ - const: simple-mfd
+
+ reg:
+ description: MISC block registers
+
+ ranges: true
+
+ "#address-cells":
+ const: 1
+
+ "#size-cells":
+ const: 1
+
+patternProperties:
+ '^reset-controller@[a-f0-9]+$':
+ $ref: ../reset/brcm,bcm4908-misc-pcie-reset.yaml
+
+additionalProperties: false
+
+required:
+ - reg
+ - '#address-cells'
+ - '#size-cells'
+
+examples:
+ - |
+ misc@ff802600 {
+ compatible = "brcm,misc", "simple-mfd";
+ reg = <0xff802600 0xe4>;
+
+ #address-cells = <1>;
+ #size-cells = <1>;
+ ranges = <0x0 0x0 0xe4>;
+
+ reset-controller@44 {
+ compatible = "brcm,bcm4908-misc-pcie-reset";
+ reg = <0x44 0x4>;
+ #reset-cells = <1>;
+ };
+ };
compatible = "maxim,max14577";
reg = <0x25>;
interrupt-parent = <&gpx1>;
- interrupts = <5 IRQ_TYPE_NONE>;
+ interrupts = <5 IRQ_TYPE_LEVEL_LOW>;
muic: max14577-muic {
compatible = "maxim,max14577-muic";
compatible = "maxim,max77836";
reg = <0x25>;
interrupt-parent = <&gpx1>;
- interrupts = <5 IRQ_TYPE_NONE>;
+ interrupts = <5 IRQ_TYPE_LEVEL_LOW>;
muic: max77836-muic {
compatible = "maxim,max77836-muic";
max77686: pmic@9 {
compatible = "maxim,max77686";
interrupt-parent = <&wakeup_eint>;
- interrupts = <26 0>;
+ interrupts = <26 IRQ_TYPE_LEVEL_LOW>;
reg = <0x09>;
};
compatible = "maxim,max77693";
reg = <0x66>;
interrupt-parent = <&gpx1>;
- interrupts = <5 2>;
+ interrupts = <5 IRQ_TYPE_LEVEL_LOW>;
regulators {
esafeout@1 {
Required properties:
- compatible: Should contain one of:
- "qcom,pm8941",
- "qcom,pm8841",
- "qcom,pma8084",
+ "qcom,pm660",
+ "qcom,pm660l",
+ "qcom,pm7325",
+ "qcom,pm8004",
+ "qcom,pm8005",
"qcom,pm8019",
- "qcom,pm8226",
+ "qcom,pm8028",
"qcom,pm8110",
- "qcom,pma8084",
- "qcom,pmi8962",
- "qcom,pmd9635",
- "qcom,pm8994",
- "qcom,pmi8994",
- "qcom,pm8916",
- "qcom,pm8004",
+ "qcom,pm8150",
+ "qcom,pm8150b",
+ "qcom,pm8150c",
+ "qcom,pm8150l",
+ "qcom,pm8226",
+ "qcom,pm8350c",
+ "qcom,pm8841",
+ "qcom,pm8901",
"qcom,pm8909",
+ "qcom,pm8916",
+ "qcom,pm8941",
"qcom,pm8950",
- "qcom,pmi8950",
+ "qcom,pm8994",
"qcom,pm8998",
+ "qcom,pma8084",
+ "qcom,pmd9635",
+ "qcom,pmi8950",
+ "qcom,pmi8962",
+ "qcom,pmi8994",
"qcom,pmi8998",
- "qcom,pm8005",
- "qcom,pm8350c",
+ "qcom,pmk8002",
"qcom,pmk8350",
- "qcom,pm7325",
"qcom,pmr735a",
+ "qcom,smb2351",
or generalized "qcom,spmi-pmic".
- reg: Specifies the SPMI USID slave address for this device.
For more information see:
Required properties:
- compatible: Should contain:
+ "qcom,tcsr-ipq6018", "syscon", "simple-mfd" for IPQ6018
"qcom,tcsr-ipq8064", "syscon" for IPQ8064
"qcom,tcsr-apq8064", "syscon" for APQ8064
"qcom,tcsr-msm8660", "syscon" for MSM8660
properties:
compatible:
enum:
+ - qcom,pm8018
- qcom,pm8058
- qcom,pm8821
- qcom,pm8921
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/samsung,s2mpa01.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Samsung S2MPA01 Power Management IC
+
+maintainers:
+ - Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
+
+description: |
+ This is a part of device tree bindings for S2M and S5M family of Power
+ Management IC (PMIC).
+
+ The Samsung S2MPA01 is a Power Management IC which includes voltage
+ and current regulators, RTC, clock outputs and other sub-blocks.
+
+properties:
+ compatible:
+ const: samsung,s2mpa01-pmic
+
+ interrupts:
+ maxItems: 1
+
+ reg:
+ maxItems: 1
+
+ regulators:
+ $ref: ../regulator/samsung,s2mpa01.yaml
+ description:
+ List of child nodes that specify the regulators.
+
+ wakeup-source: true
+
+required:
+ - compatible
+ - reg
+ - regulators
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@66 {
+ compatible = "samsung,s2mpa01-pmic";
+ reg = <0x66>;
+
+ regulators {
+ ldo1_reg: LDO1 {
+ regulator-name = "VDD_ALIVE";
+ regulator-min-microvolt = <1000000>;
+ regulator-max-microvolt = <1000000>;
+ };
+
+ ldo2_reg: LDO2 {
+ regulator-name = "VDDQ_MMC2";
+ regulator-min-microvolt = <2800000>;
+ regulator-max-microvolt = <2800000>;
+ regulator-always-on;
+ };
+
+ // ...
+
+ buck1_reg: BUCK1 {
+ regulator-name = "vdd_mif";
+ regulator-min-microvolt = <950000>;
+ regulator-max-microvolt = <1350000>;
+ regulator-always-on;
+ regulator-boot-on;
+ };
+
+ buck2_reg: BUCK2 {
+ regulator-name = "vdd_arm";
+ regulator-min-microvolt = <950000>;
+ regulator-max-microvolt = <1350000>;
+ regulator-always-on;
+ regulator-boot-on;
+ regulator-ramp-delay = <50000>;
+ };
+
+ // ...
+ };
+ };
+ };
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/samsung,s2mps11.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Samsung S2MPS11/13/14/15 and S2MPU02 Power Management IC
+
+maintainers:
+ - Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
+
+description: |
+ This is a part of device tree bindings for S2M and S5M family of Power
+ Management IC (PMIC).
+
+ The Samsung S2MPS11/13/14/15 and S2MPU02 is a family of Power Management IC
+ which include voltage and current regulators, RTC, clock outputs and other
+ sub-blocks.
+
+properties:
+ compatible:
+ enum:
+ - samsung,s2mps11-pmic
+ - samsung,s2mps13-pmic
+ - samsung,s2mps14-pmic
+ - samsung,s2mps15-pmic
+ - samsung,s2mpu02-pmic
+
+ clocks:
+ $ref: ../clock/samsung,s2mps11.yaml
+ description:
+ Child node describing clock provider.
+
+ interrupts:
+ maxItems: 1
+
+ reg:
+ maxItems: 1
+
+ regulators:
+ type: object
+ description:
+ List of child nodes that specify the regulators.
+
+ samsung,s2mps11-acokb-ground:
+ description: |
+ Indicates that ACOKB pin of S2MPS11 PMIC is connected to the ground so
+ the PMIC must manually set PWRHOLD bit in CTRL1 register to turn off the
+ power. Usually the ACOKB is pulled up to VBATT so when PWRHOLD pin goes
+ low, the rising ACOKB will trigger power off.
+ type: boolean
+
+ samsung,s2mps11-wrstbi-ground:
+ description: |
+ Indicates that WRSTBI pin of PMIC is pulled down. When the system is
+ suspended it will always go down thus triggerring unwanted buck warm
+ reset (setting buck voltages to default values).
+ type: boolean
+
+ wakeup-source: true
+
+required:
+ - compatible
+ - reg
+ - regulators
+
+additionalProperties: false
+
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: samsung,s2mps11-pmic
+ then:
+ properties:
+ regulators:
+ $ref: ../regulator/samsung,s2mps11.yaml
+ samsung,s2mps11-wrstbi-ground: false
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: samsung,s2mps13-pmic
+ then:
+ properties:
+ regulators:
+ $ref: ../regulator/samsung,s2mps13.yaml
+ samsung,s2mps11-acokb-ground: false
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: samsung,s2mps14-pmic
+ then:
+ properties:
+ regulators:
+ $ref: ../regulator/samsung,s2mps14.yaml
+ samsung,s2mps11-acokb-ground: false
+ samsung,s2mps11-wrstbi-ground: false
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: samsung,s2mps15-pmic
+ then:
+ properties:
+ regulators:
+ $ref: ../regulator/samsung,s2mps15.yaml
+ samsung,s2mps11-acokb-ground: false
+ samsung,s2mps11-wrstbi-ground: false
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: samsung,s2mpu02-pmic
+ then:
+ properties:
+ regulators:
+ $ref: ../regulator/samsung,s2mpu02.yaml
+ samsung,s2mps11-acokb-ground: false
+ samsung,s2mps11-wrstbi-ground: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@66 {
+ compatible = "samsung,s2mps11-pmic";
+ reg = <0x66>;
+
+ interrupt-parent = <&gpx0>;
+ interrupts = <4 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&s2mps11_irq>;
+ samsung,s2mps11-acokb-ground;
+ wakeup-source;
+
+ clocks {
+ compatible = "samsung,s2mps11-clk";
+ #clock-cells = <1>;
+ clock-output-names = "s2mps11_ap", "s2mps11_cp", "s2mps11_bt";
+ };
+
+ regulators {
+ LDO1 {
+ regulator-name = "vdd_ldo1";
+ regulator-min-microvolt = <1000000>;
+ regulator-max-microvolt = <1000000>;
+ regulator-always-on;
+ };
+
+ LDO4 {
+ regulator-name = "vdd_adc";
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+
+ regulator-state-mem {
+ regulator-off-in-suspend;
+ };
+ };
+
+ // ....
+
+ BUCK1 {
+ regulator-name = "vdd_mif";
+ regulator-min-microvolt = <800000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-always-on;
+ regulator-boot-on;
+
+ regulator-state-mem {
+ regulator-off-in-suspend;
+ };
+ };
+
+ BUCK2 {
+ regulator-name = "vdd_arm";
+ regulator-min-microvolt = <800000>;
+ regulator-max-microvolt = <1500000>;
+ regulator-always-on;
+ regulator-boot-on;
+ regulator-coupled-with = <&buck3_reg>;
+ regulator-coupled-max-spread = <300000>;
+
+ regulator-state-mem {
+ regulator-off-in-suspend;
+ };
+ };
+
+ BUCK3 {
+ regulator-name = "vdd_int";
+ regulator-min-microvolt = <800000>;
+ regulator-max-microvolt = <1400000>;
+ regulator-always-on;
+ regulator-boot-on;
+ regulator-coupled-with = <&buck2_reg>;
+ regulator-coupled-max-spread = <300000>;
+
+ regulator-state-mem {
+ regulator-off-in-suspend;
+ };
+ };
+
+ // ...
+ };
+ };
+ };
+
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@66 {
+ compatible = "samsung,s2mps14-pmic";
+ reg = <0x66>;
+
+ interrupt-parent = <&gpx0>;
+ interrupts = <7 IRQ_TYPE_LEVEL_LOW>;
+ wakeup-source;
+
+ clocks {
+ compatible = "samsung,s2mps14-clk";
+ #clock-cells = <1>;
+ clock-output-names = "s2mps14_ap", "unused", "s2mps14_bt";
+ };
+
+ regulators {
+ LDO1 {
+ regulator-name = "VLDO1_1.0V";
+ regulator-min-microvolt = <1000000>;
+ regulator-max-microvolt = <1000000>;
+ regulator-always-on;
+
+ regulator-state-mem {
+ regulator-on-in-suspend;
+ };
+ };
+
+ // ...
+
+ BUCK1 {
+ regulator-name = "VBUCK1_1.0V";
+ regulator-min-microvolt = <800000>;
+ regulator-max-microvolt = <1000000>;
+ regulator-always-on;
+
+ regulator-state-mem {
+ regulator-off-in-suspend;
+ };
+ };
+
+ // ...
+ };
+ };
+ };
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/samsung,s5m8767.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Samsung S5M8767 Power Management IC
+
+maintainers:
+ - Krzysztof Kozlowski <krzysztof.kozlowski@canonical.com>
+
+description: |
+ This is a part of device tree bindings for S2M and S5M family of Power
+ Management IC (PMIC).
+
+ The Samsung S5M8767 is a Power Management IC which includes voltage
+ and current regulators, RTC, clock outputs and other sub-blocks.
+
+properties:
+ compatible:
+ const: samsung,s5m8767-pmic
+
+ clocks:
+ $ref: ../clock/samsung,s2mps11.yaml
+ description:
+ Child node describing clock provider.
+
+ interrupts:
+ maxItems: 1
+
+ reg:
+ maxItems: 1
+
+ regulators:
+ $ref: ../regulator/samsung,s5m8767.yaml
+ description:
+ List of child nodes that specify the regulators.
+
+ s5m8767,pmic-buck2-dvs-voltage:
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 8
+ maxItems: 8
+ description: |
+ A set of 8 voltage values in micro-volt (uV) units for buck2 when
+ changing voltage using gpio dvs.
+
+ s5m8767,pmic-buck3-dvs-voltage:
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 8
+ maxItems: 8
+ description: |
+ A set of 8 voltage values in micro-volt (uV) units for buck3 when
+ changing voltage using gpio dvs.
+
+ s5m8767,pmic-buck4-dvs-voltage:
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minItems: 8
+ maxItems: 8
+ description: |
+ A set of 8 voltage values in micro-volt (uV) units for buck4 when
+ changing voltage using gpio dvs.
+
+ s5m8767,pmic-buck-ds-gpios:
+ minItems: 3
+ maxItems: 3
+ description: |
+ GPIO specifiers for three host gpio's used for selecting GPIO DVS lines.
+ It is one-to-one mapped to dvs gpio lines.
+
+ s5m8767,pmic-buck2-uses-gpio-dvs:
+ type: boolean
+ description: buck2 can be controlled by gpio dvs.
+
+ s5m8767,pmic-buck3-uses-gpio-dvs:
+ type: boolean
+ description: buck3 can be controlled by gpio dvs.
+
+ s5m8767,pmic-buck4-uses-gpio-dvs:
+ type: boolean
+ description: buck4 can be controlled by gpio dvs.
+
+ s5m8767,pmic-buck-default-dvs-idx:
+ $ref: /schemas/types.yaml#/definitions/uint32-array
+ minimum: 0
+ maximum: 7
+ default: 0
+ description: |
+ Default voltage setting selected from the possible 8 options selectable
+ by the dvs gpios. The value of this property should be between 0 and 7.
+ If not specified or if out of range, the default value of this property
+ is set to 0.
+
+ s5m8767,pmic-buck-dvs-gpios:
+ minItems: 3
+ maxItems: 3
+ description: |
+ GPIO specifiers for three host gpio's used for dvs.
+
+ vinb1-supply:
+ description: Power supply for buck1
+ vinb2-supply:
+ description: Power supply for buck2
+ vinb3-supply:
+ description: Power supply for buck3
+ vinb4-supply:
+ description: Power supply for buck4
+ vinb5-supply:
+ description: Power supply for buck5
+ vinb6-supply:
+ description: Power supply for buck6
+ vinb7-supply:
+ description: Power supply for buck7
+ vinb8-supply:
+ description: Power supply for buck8
+ vinb9-supply:
+ description: Power supply for buck9
+
+ vinl1-supply:
+ description: Power supply for LDO3, LDO10, LDO26, LDO27
+ vinl2-supply:
+ description: Power supply for LDO13, LDO16, LDO25, LDO28
+ vinl3-supply:
+ description: Power supply for LDO11, LDO14
+ vinl4-supply:
+ description: Power supply for LDO4, LDO9
+ vinl5-supply:
+ description: Power supply for LDO12, LDO17, LDO19, LDO23
+ vinl6-supply:
+ description: Power supply for LDO18, LDO20, LDO21, LDO24
+ vinl7-supply:
+ description: Power supply for LDO5, LDO22
+ vinl8-supply:
+ description: Power supply for LDO1, LDO6, LDO7, LDO8, LDO15
+ vinl9-supply:
+ description: Power supply for LDO2
+
+ wakeup-source: true
+
+required:
+ - compatible
+ - reg
+ - regulators
+ - s5m8767,pmic-buck-ds-gpios
+
+dependencies:
+ s5m8767,pmic-buck2-dvs-voltage: [ 's5m8767,pmic-buck-dvs-gpios' ]
+ s5m8767,pmic-buck3-dvs-voltage: [ 's5m8767,pmic-buck-dvs-gpios' ]
+ s5m8767,pmic-buck4-dvs-voltage: [ 's5m8767,pmic-buck-dvs-gpios' ]
+ s5m8767,pmic-buck2-uses-gpio-dvs: [ 's5m8767,pmic-buck-dvs-gpios', 's5m8767,pmic-buck2-dvs-voltage' ]
+ s5m8767,pmic-buck3-uses-gpio-dvs: [ 's5m8767,pmic-buck-dvs-gpios', 's5m8767,pmic-buck3-dvs-voltage' ]
+ s5m8767,pmic-buck4-uses-gpio-dvs: [ 's5m8767,pmic-buck-dvs-gpios', 's5m8767,pmic-buck4-dvs-voltage' ]
+
+additionalProperties: false
+
+allOf:
+ - if:
+ required:
+ - s5m8767,pmic-buck2-uses-gpio-dvs
+ then:
+ properties:
+ s5m8767,pmic-buck3-uses-gpio-dvs: false
+ s5m8767,pmic-buck4-uses-gpio-dvs: false
+
+ - if:
+ required:
+ - s5m8767,pmic-buck3-uses-gpio-dvs
+ then:
+ properties:
+ s5m8767,pmic-buck2-uses-gpio-dvs: false
+ s5m8767,pmic-buck4-uses-gpio-dvs: false
+
+ - if:
+ required:
+ - s5m8767,pmic-buck4-uses-gpio-dvs
+ then:
+ properties:
+ s5m8767,pmic-buck2-uses-gpio-dvs: false
+ s5m8767,pmic-buck3-uses-gpio-dvs: false
+
+examples:
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@66 {
+ compatible = "samsung,s5m8767-pmic";
+ reg = <0x66>;
+
+ interrupt-parent = <&gpx3>;
+ interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&s5m8767_irq &s5m8767_dvs &s5m8767_ds>;
+ wakeup-source;
+
+ s5m8767,pmic-buck-default-dvs-idx = <3>;
+ s5m8767,pmic-buck2-uses-gpio-dvs;
+
+ s5m8767,pmic-buck-dvs-gpios = <&gpd1 0 GPIO_ACTIVE_LOW>,
+ <&gpd1 1 GPIO_ACTIVE_LOW>,
+ <&gpd1 2 GPIO_ACTIVE_LOW>;
+
+ s5m8767,pmic-buck-ds-gpios = <&gpx2 3 GPIO_ACTIVE_LOW>,
+ <&gpx2 4 GPIO_ACTIVE_LOW>,
+ <&gpx2 5 GPIO_ACTIVE_LOW>;
+
+ s5m8767,pmic-buck2-dvs-voltage = <1350000>, <1300000>,
+ <1250000>, <1200000>,
+ <1150000>, <1100000>,
+ <1000000>, <950000>;
+
+ s5m8767,pmic-buck3-dvs-voltage = <1100000>, <1100000>,
+ <1100000>, <1100000>,
+ <1000000>, <1000000>,
+ <1000000>, <1000000>;
+
+ s5m8767,pmic-buck4-dvs-voltage = <1200000>, <1200000>,
+ <1200000>, <1200000>,
+ <1200000>, <1200000>,
+ <1200000>, <1200000>;
+
+ clocks {
+ compatible = "samsung,s5m8767-clk";
+ #clock-cells = <1>;
+ clock-output-names = "en32khz_ap", "en32khz_cp", "en32khz_bt";
+ };
+
+ regulators {
+ LDO1 {
+ regulator-name = "VDD_ALIVE";
+ regulator-min-microvolt = <1100000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ regulator-boot-on;
+ op_mode = <1>; /* Normal Mode */
+ };
+
+ // ...
+
+ BUCK1 {
+ regulator-name = "VDD_MIF";
+ regulator-min-microvolt = <950000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ regulator-boot-on;
+ op_mode = <1>; /* Normal Mode */
+ };
+
+ BUCK2 {
+ regulator-name = "VDD_ARM";
+ regulator-min-microvolt = <900000>;
+ regulator-max-microvolt = <1350000>;
+ regulator-always-on;
+ regulator-boot-on;
+ op_mode = <1>; /* Normal Mode */
+ };
+
+ // ...
+ };
+ };
+ };
+
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@66 {
+ compatible = "samsung,s5m8767-pmic";
+ reg = <0x66>;
+
+ interrupt-parent = <&gpx3>;
+ interrupts = <2 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&s5m8767_irq &s5m8767_dvs &s5m8767_ds>;
+ wakeup-source;
+
+ s5m8767,pmic-buck-ds-gpios = <&gpx2 3 GPIO_ACTIVE_LOW>,
+ <&gpx2 4 GPIO_ACTIVE_LOW>,
+ <&gpx2 5 GPIO_ACTIVE_LOW>;
+
+ clocks {
+ compatible = "samsung,s5m8767-clk";
+ #clock-cells = <1>;
+ clock-output-names = "en32khz_ap", "en32khz_cp", "en32khz_bt";
+ };
+
+ regulators {
+ LDO1 {
+ regulator-name = "VDD_ALIVE";
+ regulator-min-microvolt = <1100000>;
+ regulator-max-microvolt = <1100000>;
+ regulator-always-on;
+ regulator-boot-on;
+ op_mode = <1>; /* Normal Mode */
+ };
+
+ // ...
+ };
+ };
+ };
+++ /dev/null
-Binding for Samsung S2M and S5M family multi-function device
-============================================================
-
-This is a part of device tree bindings for S2M and S5M family multi-function
-devices.
-
-The Samsung S2MPA01, S2MPS11/13/14/15, S2MPU02 and S5M8767 is a family
-of multi-function devices which include voltage and current regulators, RTC,
-charger controller, clock outputs and other sub-blocks. It is interfaced
-to the host controller using an I2C interface. Each sub-block is usually
-addressed by the host system using different I2C slave addresses.
-
-
-This document describes bindings for main device node. Optional sub-blocks
-must be a sub-nodes to it. Bindings for them can be found in:
- - bindings/regulator/samsung,s2mpa01.txt
- - bindings/regulator/samsung,s2mps11.txt
- - bindings/regulator/samsung,s5m8767.txt
- - bindings/clock/samsung,s2mps11.txt
-
-
-Required properties:
- - compatible: Should be one of the following
- - "samsung,s2mpa01-pmic",
- - "samsung,s2mps11-pmic",
- - "samsung,s2mps13-pmic",
- - "samsung,s2mps14-pmic",
- - "samsung,s2mps15-pmic",
- - "samsung,s2mpu02-pmic",
- - "samsung,s5m8767-pmic".
- - reg: Specifies the I2C slave address of the pmic block. It should be 0x66.
-
-Optional properties:
- - interrupts: Interrupt specifiers for interrupt sources.
- - samsung,s2mps11-wrstbi-ground: Indicates that WRSTBI pin of PMIC is pulled
- down. When the system is suspended it will always go down thus triggerring
- unwanted buck warm reset (setting buck voltages to default values).
- - samsung,s2mps11-acokb-ground: Indicates that ACOKB pin of S2MPS11 PMIC is
- connected to the ground so the PMIC must manually set PWRHOLD bit in CTRL1
- register to turn off the power. Usually the ACOKB is pulled up to VBATT so
- when PWRHOLD pin goes low, the rising ACOKB will trigger power off.
-
-Example:
-
- s2mps11_pmic@66 {
- compatible = "samsung,s2mps11-pmic";
- reg = <0x66>;
-
- s2m_osc: clocks {
- compatible = "samsung,s2mps11-clk";
- #clock-cells = <1>;
- clock-output-names = "xx", "yy", "zz";
- };
-
- regulators {
- ldo1_reg: LDO1 {
- regulator-name = "VDD_ABB_3.3V";
- regulator-min-microvolt = <3300000>;
- regulator-max-microvolt = <3300000>;
- };
-
- ldo2_reg: LDO2 {
- regulator-name = "VDD_ALIVE_1.1V";
- regulator-min-microvolt = <1100000>;
- regulator-max-microvolt = <1100000>;
- regulator-always-on;
- };
-
- buck1_reg: BUCK1 {
- regulator-name = "vdd_mif";
- regulator-min-microvolt = <950000>;
- regulator-max-microvolt = <1350000>;
- regulator-always-on;
- regulator-boot-on;
- };
-
- buck2_reg: BUCK2 {
- regulator-name = "vdd_arm";
- regulator-min-microvolt = <950000>;
- regulator-max-microvolt = <1350000>;
- regulator-always-on;
- regulator-boot-on;
- regulator-ramp-delay = <50000>;
- };
- };
- };
- simple counter from IN1 input signal.
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
properties:
compatible:
programmable prescaler.
maintainers:
- - Benjamin Gaignard <benjamin.gaignard@st.com>
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
properties:
compatible:
through VDD) and resistive touchscreen controller.
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
properties:
compatible:
description: STMicroelectronics STPMIC1 Power Management IC
maintainers:
- - pascal Paillet <p.paillet@st.com>
+ - pascal Paillet <p.paillet@foss.st.com>
properties:
compatible:
- allwinner,sun8i-h3-system-controller
- allwinner,sun8i-v3s-system-controller
- allwinner,sun50i-a64-system-controller
+ - brcm,cru-clkset
- hisilicon,dsa-subctrl
- hisilicon,hi6220-sramctrl
- hisilicon,pcie-sas-subctrl
- rockchip,rk3066-qos
- rockchip,rk3228-qos
- rockchip,rk3288-qos
+ - rockchip,rk3368-qos
- rockchip,rk3399-qos
- rockchip,rk3568-qos
- samsung,exynos3-sysreg
- samsung,exynos4-sysreg
- samsung,exynos5-sysreg
- samsung,exynos5433-sysreg
+ - samsung,exynosautov9-sysreg
- const: syscon
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/ti,am3359-tscadc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: TI AM3359 Touchscreen controller/ADC
+
+maintainers:
+ - Miquel Raynal <miquel.raynal@bootlin.com>
+
+properties:
+ compatible:
+ oneOf:
+ - const: ti,am3359-tscadc
+ - items:
+ - const: ti,am654-tscadc
+ - const: ti,am3359-tscadc
+ - const: ti,am4372-magadc
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ clock-names:
+ const: fck
+
+ dmas:
+ items:
+ - description: DMA controller phandle and request line for FIFO0
+ - description: DMA controller phandle and request line for FIFO1
+
+ dma-names:
+ items:
+ - const: fifo0
+ - const: fifo1
+
+ adc:
+ type: object
+ description: ADC child
+
+ tsc:
+ type: object
+ description: Touchscreen controller child
+
+ mag:
+ type: object
+ description: Magnetic reader
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+ - dmas
+ - dma-names
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+
+ tscadc@0 {
+ compatible = "ti,am3359-tscadc";
+ reg = <0x0 0x1000>;
+ interrupts = <GIC_SPI 16 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&adc_tsc_fck>;
+ clock-names = "fck";
+ dmas = <&edma 53 0>, <&edma 57 0>;
+ dma-names = "fifo0", "fifo1";
+
+ tsc {
+ };
+
+ adc {
+ };
+ };
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/mfd/x-powers,ac100.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: X-Powers AC100 Device Tree Bindings
+
+maintainers:
+ - Chen-Yu Tsai <wens@csie.org>
+
+properties:
+ compatible:
+ const: x-powers,ac100
+
+ reg:
+ maxItems: 1
+
+ codec:
+ type: object
+
+ properties:
+ "#clock-cells":
+ const: 0
+
+ compatible:
+ const: x-powers,ac100-codec
+
+ interrupts:
+ maxItems: 1
+
+ clock-output-names:
+ maxItems: 1
+ description: >
+ Name of the 4M_adda clock exposed by the codec
+
+ required:
+ - "#clock-cells"
+ - compatible
+ - interrupts
+ - clock-output-names
+
+ additionalProperties: false
+
+ rtc:
+ type: object
+
+ properties:
+ "#clock-cells":
+ const: 1
+
+ compatible:
+ const: x-powers,ac100-rtc
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+ description: >
+ A phandle to the codec's "4M_adda" clock
+
+ clock-output-names:
+ maxItems: 3
+ description: >
+ Name of the cko1, cko2 and cko3 clocks exposed by the codec
+
+ required:
+ - "#clock-cells"
+ - compatible
+ - interrupts
+ - clocks
+ - clock-output-names
+
+ additionalProperties: false
+
+required:
+ - compatible
+ - reg
+ - codec
+ - rtc
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ rsb {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ codec@e89 {
+ compatible = "x-powers,ac100";
+ reg = <0xe89>;
+
+ ac100_codec: codec {
+ compatible = "x-powers,ac100-codec";
+ interrupt-parent = <&r_pio>;
+ interrupts = <0 9 IRQ_TYPE_LEVEL_LOW>; /* PL9 */
+ #clock-cells = <0>;
+ clock-output-names = "4M_adda";
+ };
+
+ ac100_rtc: rtc {
+ compatible = "x-powers,ac100-rtc";
+ interrupt-parent = <&nmi_intc>;
+ interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
+ clocks = <&ac100_codec>;
+ #clock-cells = <1>;
+ clock-output-names = "cko1_rtc", "cko2_rtc", "cko3_rtc";
+ };
+ };
+ };
+
+...
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/mfd/x-powers,axp152.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: X-Powers AXP PMIC Device Tree Bindings
+
+maintainers:
+ - Chen-Yu Tsai <wens@csie.org>
+
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - x-powers,axp152
+ - x-powers,axp202
+ - x-powers,axp209
+
+ then:
+ properties:
+ regulators:
+ properties:
+ x-powers,dcdc-freq:
+ minimum: 750
+ maximum: 1875
+ default: 1500
+
+ else:
+ properties:
+ regulators:
+ properties:
+ x-powers,dcdc-freq:
+ minimum: 1800
+ maximum: 4050
+ default: 3000
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - x-powers,axp152
+ - x-powers,axp202
+ - x-powers,axp209
+
+ then:
+ not:
+ required:
+ - x-powers,drive-vbus-en
+
+ - if:
+ not:
+ properties:
+ compatible:
+ contains:
+ const: x-powers,axp806
+
+ then:
+ allOf:
+ - not:
+ required:
+ - x-powers,self-working-mode
+
+ - not:
+ required:
+ - x-powers,master-mode
+
+ - if:
+ not:
+ properties:
+ compatible:
+ contains:
+ const: x-powers,axp305
+
+ then:
+ required:
+ - interrupts
+
+properties:
+ compatible:
+ oneOf:
+ - enum:
+ - x-powers,axp152
+ - x-powers,axp202
+ - x-powers,axp209
+ - x-powers,axp221
+ - x-powers,axp223
+ - x-powers,axp803
+ - x-powers,axp806
+ - x-powers,axp809
+ - x-powers,axp813
+ - items:
+ - const: x-powers,axp805
+ - const: x-powers,axp806
+ - items:
+ - const: x-powers,axp305
+ - const: x-powers,axp805
+ - const: x-powers,axp806
+ - items:
+ - const: x-powers,axp818
+ - const: x-powers,axp813
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ interrupt-controller: true
+
+ "#interrupt-cells":
+ const: 1
+
+ x-powers,drive-vbus-en:
+ type: boolean
+ description: >
+ Set this when the N_VBUSEN pin is used as an output pin to control an
+ external regulator to drive the OTG VBus, rather then as an input pin
+ which signals whether the board is driving OTG VBus or not.
+
+ x-powers,self-working-mode:
+ type: boolean
+ description: >
+ Set this when the PMIC is wired for self-working mode through the MODESET
+ pin.
+
+ x-powers,master-mode:
+ type: boolean
+ description: >
+ Set this when the PMIC is wired for master mode through the MODESET pin.
+
+ vin1-supply:
+ description: >
+ DCDC1 power supply node, if present.
+
+ vin2-supply:
+ description: >
+ DCDC2 power supply node, if present.
+
+ vin3-supply:
+ description: >
+ DCDC3 power supply node, if present.
+
+ vin4-supply:
+ description: >
+ DCDC4 power supply node, if present.
+
+ vin5-supply:
+ description: >
+ DCDC5 power supply node, if present.
+
+ vin6-supply:
+ description: >
+ DCDC6 power supply node, if present.
+
+ vin7-supply:
+ description: >
+ DCDC7 power supply node, if present.
+
+ vina-supply:
+ description: >
+ DCDCA power supply node, if present.
+
+ vinb-supply:
+ description: >
+ DCDCB power supply node, if present.
+
+ vinc-supply:
+ description: >
+ DCDCC power supply node, if present.
+
+ vind-supply:
+ description: >
+ DCDCD power supply node, if present.
+
+ vine-supply:
+ description: >
+ DCDCE power supply node, if present.
+
+ acin-supply:
+ description: >
+ LDO1 power supply node, if present.
+
+ ldo24in-supply:
+ description: >
+ LDO2 and LDO4 power supply node, if present.
+
+ ldo3in-supply:
+ description: >
+ LDO3 power supply node, if present.
+
+ ldo5in-supply:
+ description: >
+ LDO5 power supply node, if present.
+
+ aldoin-supply:
+ description: >
+ ALDO* power supply node, if present.
+
+ bldoin-supply:
+ description: >
+ BLDO* power supply node, if present.
+
+ cldoin-supply:
+ description: >
+ CLDO* power supply node, if present.
+
+ dldoin-supply:
+ description: >
+ DLDO* power supply node, if present.
+
+ eldoin-supply:
+ description: >
+ ELDO* power supply node, if present.
+
+ fldoin-supply:
+ description: >
+ FLDO* power supply node, if present.
+
+ ips-supply:
+ description: >
+ LDO_IO0, LDO_IO1 and RTC_LDO power supply node, if present.
+
+ drivevbus-supply:
+ description: >
+ DRIVEVBUS power supply node, if present.
+
+ swin-supply:
+ description: >
+ SW power supply node, if present.
+
+ adc:
+ $ref: /schemas/iio/adc/x-powers,axp209-adc.yaml#
+
+ gpio:
+ $ref: /schemas/gpio/x-powers,axp209-gpio.yaml#
+
+ ac-power:
+ $ref: /schemas/power/supply/x-powers,axp20x-ac-power-supply.yaml#
+
+ battery-power:
+ $ref: /schemas/power/supply/x-powers,axp20x-battery-power-supply.yaml#
+
+ usb-power:
+ $ref: /schemas/power/supply/x-powers,axp20x-usb-power-supply.yaml#
+
+ regulators:
+ type: object
+
+ properties:
+ x-powers,dcdc-freq:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: >
+ Defines the work frequency of DC-DC in kHz.
+
+ patternProperties:
+ "^(([a-f])?ldo[0-9]|dcdc[0-7a-e]|ldo(_|-)io(0|1)|(dc1)?sw|rtc(_|-)ldo|drivevbus|dc5ldo)$":
+ $ref: /schemas/regulator/regulator.yaml#
+ type: object
+
+ properties:
+ regulator-ramp-delay:
+ description: >
+ Only 800 and 1600 are valid for the DCDC2 and LDO3 regulators on
+ the AXP209.
+
+ regulator-soft-start:
+ description: >
+ Only valid for the LDO3 regulator.
+
+ x-powers,dcdc-workmode:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ enum: [0, 1]
+ description: >
+ Only valid for DCDC regulators. Setup 1 for PWM mode, 0
+ for AUTO (PWM/PFM) mode. The DCDC regulators work in a
+ mixed PWM/PFM mode, using PFM under light loads and
+ switching to PWM for heavier loads. Forcing PWM mode
+ trades efficiency under light loads for lower output
+ noise. This probably makes sense for HiFi audio related
+ applications that aren't battery constrained.
+
+ additionalProperties: false
+
+required:
+ - compatible
+ - reg
+ - "#interrupt-cells"
+ - interrupt-controller
+
+additionalProperties: false
+
+examples:
+ - |
+ i2c0 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@30 {
+ compatible = "x-powers,axp152";
+ reg = <0x30>;
+ interrupts = <0>;
+ interrupt-controller;
+ #interrupt-cells = <1>;
+ };
+ };
+
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c0 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ pmic@34 {
+ compatible = "x-powers,axp209";
+ reg = <0x34>;
+ interrupt-parent = <&nmi_intc>;
+ interrupts = <0 IRQ_TYPE_LEVEL_LOW>;
+ interrupt-controller;
+ #interrupt-cells = <1>;
+
+ ac_power_supply: ac-power {
+ compatible = "x-powers,axp202-ac-power-supply";
+ };
+
+ axp_adc: adc {
+ compatible = "x-powers,axp209-adc";
+ #io-channel-cells = <1>;
+ };
+
+ axp_gpio: gpio {
+ compatible = "x-powers,axp209-gpio";
+ gpio-controller;
+ #gpio-cells = <2>;
+
+ gpio0-adc-pin {
+ pins = "GPIO0";
+ function = "adc";
+ };
+ };
+
+ battery_power_supply: battery-power {
+ compatible = "x-powers,axp209-battery-power-supply";
+ };
+
+ regulators {
+ /* Default work frequency for buck regulators */
+ x-powers,dcdc-freq = <1500>;
+
+ reg_dcdc2: dcdc2 {
+ regulator-always-on;
+ regulator-min-microvolt = <1000000>;
+ regulator-max-microvolt = <1450000>;
+ regulator-name = "vdd-cpu";
+ };
+
+ reg_dcdc3: dcdc3 {
+ regulator-always-on;
+ regulator-min-microvolt = <1000000>;
+ regulator-max-microvolt = <1400000>;
+ regulator-name = "vdd-int-dll";
+ };
+
+ reg_ldo1: ldo1 {
+ /* LDO1 is a fixed output regulator */
+ regulator-always-on;
+ regulator-min-microvolt = <1300000>;
+ regulator-max-microvolt = <1300000>;
+ regulator-name = "vdd-rtc";
+ };
+
+ reg_ldo2: ldo2 {
+ regulator-always-on;
+ regulator-min-microvolt = <3000000>;
+ regulator-max-microvolt = <3000000>;
+ regulator-name = "avcc";
+ };
+
+ reg_ldo3: ldo3 {
+ regulator-name = "ldo3";
+ };
+
+ reg_ldo4: ldo4 {
+ regulator-name = "ldo4";
+ };
+
+ reg_ldo5: ldo5 {
+ regulator-name = "ldo5";
+ };
+ };
+
+ usb_power_supply: usb-power {
+ compatible = "x-powers,axp202-usb-power-supply";
+ };
+ };
+ };
"^gpio@[0-9a-f]+$":
$ref: /schemas/gpio/xylon,logicvc-gpio.yaml#
+ "^display@[0-9a-f]+$":
+ $ref: /schemas/display/xylon,logicvc-display.yaml#
+
required:
- compatible
- reg
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
cpus {
#address-cells = <1>;
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include <dt-bindings/dma/jz4780-dma.h>
mmc0: mmc@13450000 {
compatible = "ingenic,jz4780-mmc";
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
memory-controller@13410000 {
compatible = "ingenic,jz4780-nemc";
reg = <0x13410000 0x10000>;
title: STMicroelectronics Flexible Memory Controller 2 (FMC2) Bindings
maintainers:
- - Christophe Kerello <christophe.kerello@st.com>
+ - Christophe Kerello <christophe.kerello@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/x1000-cgu.h>
+ #include <dt-bindings/clock/ingenic,x1000-cgu.h>
mac: ethernet@134b0000 {
compatible = "ingenic,x1000-mac";
title: Synopsys DesignWare MAC Device Tree Bindings
maintainers:
- - Alexandre Torgue <alexandre.torgue@st.com>
+ - Alexandre Torgue <alexandre.torgue@foss.st.com>
- Giuseppe Cavallaro <peppe.cavallaro@st.com>
- Jose Abreu <joabreu@synopsys.com>
title: STMicroelectronics STM32 / MCU DWMAC glue layer controller
maintainers:
- - Alexandre Torgue <alexandre.torgue@st.com>
- - Christophe Roullier <christophe.roullier@st.com>
+ - Alexandre Torgue <alexandre.torgue@foss.st.com>
+ - Christophe Roullier <christophe.roullier@foss.st.com>
description:
This file documents platform glue layer for stmmac.
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
efuse@134100d0 {
compatible = "ingenic,jz4780-efuse";
internal vref (VREFIN_CAL), unique device ID...
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
allOf:
- $ref: "nvmem.yaml#"
type: boolean
patternProperties:
- '^opp-?[0-9]+$':
+ '^opp(-?[0-9]+)*$':
type: object
description:
One or more OPP nodes describing voltage-current-frequency combinations.
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/mediatek,mt7621-pcie.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: MediaTek MT7621 PCIe controller
+
+maintainers:
+ - Sergio Paracuellos <sergio.paracuellos@gmail.com>
+
+description: |+
+ MediaTek MT7621 PCIe subsys supports a single Root Complex (RC)
+ with 3 Root Ports. Each Root Port supports a Gen1 1-lane Link
+
+allOf:
+ - $ref: /schemas/pci/pci-bus.yaml#
+
+properties:
+ compatible:
+ const: mediatek,mt7621-pci
+
+ reg:
+ items:
+ - description: host-pci bridge registers
+ - description: pcie port 0 RC control registers
+ - description: pcie port 1 RC control registers
+ - description: pcie port 2 RC control registers
+
+ ranges:
+ maxItems: 2
+
+patternProperties:
+ 'pcie@[0-2],0':
+ type: object
+ $ref: /schemas/pci/pci-bus.yaml#
+
+ properties:
+ resets:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ phys:
+ maxItems: 1
+
+ required:
+ - "#interrupt-cells"
+ - interrupt-map-mask
+ - interrupt-map
+ - resets
+ - clocks
+ - phys
+ - phy-names
+ - ranges
+
+ unevaluatedProperties: false
+
+required:
+ - compatible
+ - reg
+ - ranges
+ - "#interrupt-cells"
+ - interrupt-map-mask
+ - interrupt-map
+ - reset-gpios
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/gpio/gpio.h>
+ #include <dt-bindings/interrupt-controller/mips-gic.h>
+
+ pcie: pcie@1e140000 {
+ compatible = "mediatek,mt7621-pci";
+ reg = <0x1e140000 0x100>,
+ <0x1e142000 0x100>,
+ <0x1e143000 0x100>,
+ <0x1e144000 0x100>;
+
+ #address-cells = <3>;
+ #size-cells = <2>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&pcie_pins>;
+ device_type = "pci";
+ ranges = <0x02000000 0 0x60000000 0x60000000 0 0x10000000>, /* pci memory */
+ <0x01000000 0 0x1e160000 0x1e160000 0 0x00010000>; /* io space */
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0xF800 0 0 0>;
+ interrupt-map = <0x0000 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
+ <0x0800 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
+ <0x1000 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
+ reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>;
+
+ pcie@0,0 {
+ reg = <0x0000 0 0 0 0>;
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 0>;
+ interrupt-map = <0 0 0 0 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&rstctrl 24>;
+ clocks = <&clkctrl 24>;
+ phys = <&pcie0_phy 1>;
+ phy-names = "pcie-phy0";
+ ranges;
+ };
+
+ pcie@1,0 {
+ reg = <0x0800 0 0 0 0>;
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 0>;
+ interrupt-map = <0 0 0 0 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&rstctrl 25>;
+ clocks = <&clkctrl 25>;
+ phys = <&pcie0_phy 1>;
+ phy-names = "pcie-phy1";
+ ranges;
+ };
+
+ pcie@2,0 {
+ reg = <0x1000 0 0 0 0>;
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 0>;
+ interrupt-map = <0 0 0 0 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&rstctrl 26>;
+ clocks = <&clkctrl 26>;
+ phys = <&pcie2_phy 0>;
+ phy-names = "pcie-phy2";
+ ranges;
+ };
+ };
+...
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/qcom,pcie-ep.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm PCIe Endpoint Controller binding
+
+maintainers:
+ - Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
+
+allOf:
+ - $ref: "pci-ep.yaml#"
+
+properties:
+ compatible:
+ const: qcom,sdx55-pcie-ep
+
+ reg:
+ items:
+ - description: Qualcomm-specific PARF configuration registers
+ - description: DesignWare PCIe registers
+ - description: External local bus interface registers
+ - description: Address Translation Unit (ATU) registers
+ - description: Memory region used to map remote RC address space
+ - description: BAR memory region
+
+ reg-names:
+ items:
+ - const: parf
+ - const: dbi
+ - const: elbi
+ - const: atu
+ - const: addr_space
+ - const: mmio
+
+ clocks:
+ items:
+ - description: PCIe Auxiliary clock
+ - description: PCIe CFG AHB clock
+ - description: PCIe Master AXI clock
+ - description: PCIe Slave AXI clock
+ - description: PCIe Slave Q2A AXI clock
+ - description: PCIe Sleep clock
+ - description: PCIe Reference clock
+
+ clock-names:
+ items:
+ - const: aux
+ - const: cfg
+ - const: bus_master
+ - const: bus_slave
+ - const: slave_q2a
+ - const: sleep
+ - const: ref
+
+ qcom,perst-regs:
+ description: Reference to a syscon representing TCSR followed by the two
+ offsets within syscon for Perst enable and Perst separation
+ enable registers
+ $ref: "/schemas/types.yaml#/definitions/phandle-array"
+ items:
+ minItems: 3
+ maxItems: 3
+
+ interrupts:
+ items:
+ - description: PCIe Global interrupt
+ - description: PCIe Doorbell interrupt
+
+ interrupt-names:
+ items:
+ - const: global
+ - const: doorbell
+
+ reset-gpios:
+ description: GPIO used as PERST# input signal
+ maxItems: 1
+
+ wake-gpios:
+ description: GPIO used as WAKE# output signal
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ reset-names:
+ const: core
+
+ power-domains:
+ maxItems: 1
+
+ phys:
+ maxItems: 1
+
+ phy-names:
+ const: pciephy
+
+ num-lanes:
+ default: 2
+
+required:
+ - compatible
+ - reg
+ - reg-names
+ - clocks
+ - clock-names
+ - qcom,perst-regs
+ - interrupts
+ - interrupt-names
+ - reset-gpios
+ - resets
+ - reset-names
+ - power-domains
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/qcom,gcc-sdx55.h>
+ #include <dt-bindings/gpio/gpio.h>
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ pcie_ep: pcie-ep@40000000 {
+ compatible = "qcom,sdx55-pcie-ep";
+ reg = <0x01c00000 0x3000>,
+ <0x40000000 0xf1d>,
+ <0x40000f20 0xc8>,
+ <0x40001000 0x1000>,
+ <0x40002000 0x1000>,
+ <0x01c03000 0x3000>;
+ reg-names = "parf", "dbi", "elbi", "atu", "addr_space",
+ "mmio";
+
+ clocks = <&gcc GCC_PCIE_AUX_CLK>,
+ <&gcc GCC_PCIE_CFG_AHB_CLK>,
+ <&gcc GCC_PCIE_MSTR_AXI_CLK>,
+ <&gcc GCC_PCIE_SLV_AXI_CLK>,
+ <&gcc GCC_PCIE_SLV_Q2A_AXI_CLK>,
+ <&gcc GCC_PCIE_SLEEP_CLK>,
+ <&gcc GCC_PCIE_0_CLKREF_CLK>;
+ clock-names = "aux", "cfg", "bus_master", "bus_slave",
+ "slave_q2a", "sleep", "ref";
+
+ qcom,perst-regs = <&tcsr 0xb258 0xb270>;
+
+ interrupts = <GIC_SPI 140 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 145 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-names = "global", "doorbell";
+ reset-gpios = <&tlmm 57 GPIO_ACTIVE_LOW>;
+ wake-gpios = <&tlmm 53 GPIO_ACTIVE_LOW>;
+ resets = <&gcc GCC_PCIE_BCR>;
+ reset-names = "core";
+ power-domains = <&gcc PCIE_GDSC>;
+ phys = <&pcie0_lane>;
+ phy-names = "pciephy";
+ max-link-speed = <3>;
+ num-lanes = <2>;
+ };
- "qcom,pcie-ipq4019" for ipq4019
- "qcom,pcie-ipq8074" for ipq8074
- "qcom,pcie-qcs404" for qcs404
+ - "qcom,pcie-sc8180x" for sc8180x
- "qcom,pcie-sdm845" for sdm845
- "qcom,pcie-sm8250" for sm8250
- "qcom,pcie-ipq6018" for ipq6018
- "pipe" PIPE clock
- clock-names:
- Usage: required for sm8250
+ Usage: required for sc8180x and sm8250
Value type: <stringlist>
Definition: Should contain the following entries
- "aux" Auxiliary clock
- "ahb" AHB reset
- reset-names:
- Usage: required for sdm845 and sm8250
+ Usage: required for sc8180x, sdm845 and sm8250
Value type: <stringlist>
Definition: Should contain the following entries
- "pci" PCIe core reset
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/pci/rockchip-dw-pcie.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: DesignWare based PCIe controller on Rockchip SoCs
+
+maintainers:
+ - Shawn Lin <shawn.lin@rock-chips.com>
+ - Simon Xue <xxm@rock-chips.com>
+ - Heiko Stuebner <heiko@sntech.de>
+
+description: |+
+ RK3568 SoC PCIe host controller is based on the Synopsys DesignWare
+ PCIe IP and thus inherits all the common properties defined in
+ designware-pcie.txt.
+
+allOf:
+ - $ref: /schemas/pci/pci-bus.yaml#
+
+# We need a select here so we don't match all nodes with 'snps,dw-pcie'
+select:
+ properties:
+ compatible:
+ contains:
+ const: rockchip,rk3568-pcie
+ required:
+ - compatible
+
+properties:
+ compatible:
+ items:
+ - const: rockchip,rk3568-pcie
+ - const: snps,dw-pcie
+
+ reg:
+ items:
+ - description: Data Bus Interface (DBI) registers
+ - description: Rockchip designed configuration registers
+ - description: Config registers
+
+ reg-names:
+ items:
+ - const: dbi
+ - const: apb
+ - const: config
+
+ clocks:
+ items:
+ - description: AHB clock for PCIe master
+ - description: AHB clock for PCIe slave
+ - description: AHB clock for PCIe dbi
+ - description: APB clock for PCIe
+ - description: Auxiliary clock for PCIe
+
+ clock-names:
+ items:
+ - const: aclk_mst
+ - const: aclk_slv
+ - const: aclk_dbi
+ - const: pclk
+ - const: aux
+
+ msi-map: true
+
+ num-lanes: true
+
+ phys:
+ maxItems: 1
+
+ phy-names:
+ const: pcie-phy
+
+ power-domains:
+ maxItems: 1
+
+ ranges:
+ maxItems: 2
+
+ resets:
+ maxItems: 1
+
+ reset-names:
+ const: pipe
+
+ vpcie3v3-supply: true
+
+required:
+ - compatible
+ - reg
+ - reg-names
+ - clocks
+ - clock-names
+ - msi-map
+ - num-lanes
+ - phys
+ - phy-names
+ - power-domains
+ - resets
+ - reset-names
+
+unevaluatedProperties: false
+
+examples:
+ - |
+
+ bus {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ pcie3x2: pcie@fe280000 {
+ compatible = "rockchip,rk3568-pcie", "snps,dw-pcie";
+ reg = <0x3 0xc0800000 0x0 0x390000>,
+ <0x0 0xfe280000 0x0 0x10000>,
+ <0x3 0x80000000 0x0 0x100000>;
+ reg-names = "dbi", "apb", "config";
+ bus-range = <0x20 0x2f>;
+ clocks = <&cru 143>, <&cru 144>,
+ <&cru 145>, <&cru 146>,
+ <&cru 147>;
+ clock-names = "aclk_mst", "aclk_slv",
+ "aclk_dbi", "pclk",
+ "aux";
+ device_type = "pci";
+ linux,pci-domain = <2>;
+ max-link-speed = <2>;
+ msi-map = <0x2000 &its 0x2000 0x1000>;
+ num-lanes = <2>;
+ phys = <&pcie30phy>;
+ phy-names = "pcie-phy";
+ power-domains = <&power 15>;
+ ranges = <0x81000000 0x0 0x80800000 0x3 0x80800000 0x0 0x100000>,
+ <0x83000000 0x0 0x80900000 0x3 0x80900000 0x0 0x3f700000>;
+ resets = <&cru 193>;
+ reset-names = "pipe";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ };
+ };
+...
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
otg_phy: usb-phy@3c {
compatible = "ingenic,jz4770-phy";
reg = <0x3c 0x10>;
|_ UTMI switch_______| OTG controller
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
properties:
compatible:
title: STM32 GPIO and Pin Mux/Config controller
maintainers:
- - Alexandre TORGUE <alexandre.torgue@st.com>
+ - Alexandre TORGUE <alexandre.torgue@foss.st.com>
description: |
STMicroelectronics's STM32 MCUs intregrate a GPIO and Pin mux/config hardware
- renesas,tpu-r8a7794 # R-Car E2
- renesas,tpu-r8a7795 # R-Car H3
- renesas,tpu-r8a7796 # R-Car M3-W
+ - renesas,tpu-r8a77961 # R-Car M3-W+
- renesas,tpu-r8a77965 # R-Car M3-N
- renesas,tpu-r8a77970 # R-Car V3M
- renesas,tpu-r8a77980 # R-Car V3H
+ - renesas,tpu-r8a779a0 # R-Car V3U
- const: renesas,tpu
reg:
max77686: pmic@9 {
compatible = "maxim,max77686";
interrupt-parent = <&wakeup_eint>;
- interrupts = <26 IRQ_TYPE_NONE>;
+ interrupts = <26 IRQ_TYPE_LEVEL_LOW>;
reg = <0x09>;
voltage-regulators {
title: STMicroelectronics STM32 booster for ADC analog input switches bindings
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
description: |
Some STM32 devices embed a 3.3V booster supplied by Vdda, that can be used
components through the dedicated VREF+ pin.
maintainers:
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
allOf:
- $ref: "regulator.yaml#"
title: STM32MP1 PWR voltage regulators
maintainers:
- - Pascal Paillet <p.paillet@st.com>
+ - Pascal Paillet <p.paillet@foss.st.com>
properties:
compatible:
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: "http://devicetree.org/schemas/remoteproc/amlogic,meson-mx-ao-arc.yaml#"
+$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+
+title: Amlogic Meson AO ARC Remote Processor bindings
+
+description:
+ Amlogic Meson6, Meson8, Meson8b and Meson8m2 SoCs embed an ARC core
+ controller for always-on operations, typically used for managing
+ system suspend. Meson6 and older use a ARC core based on the ARCv1
+ ISA, while Meson8, Meson8b and Meson8m2 use an ARC EM4 (ARCv2 ISA)
+ core.
+
+maintainers:
+ - Martin Blumenstingl <martin.blumenstingl@googlemail.com>
+
+properties:
+ compatible:
+ items:
+ - enum:
+ - amlogic,meson8-ao-arc
+ - amlogic,meson8b-ao-arc
+ - const: amlogic,meson-mx-ao-arc
+
+ firmware-name:
+ $ref: /schemas/types.yaml#/definitions/string
+ description:
+ The name of the firmware which should be loaded for this remote
+ processor.
+
+ reg:
+ description:
+ Address ranges of the remap and CPU control addresses for the
+ remote processor.
+ minItems: 2
+
+ reg-names:
+ items:
+ - const: remap
+ - const: cpu
+
+ resets:
+ minItems: 1
+
+ clocks:
+ minItems: 1
+
+ sram:
+ $ref: /schemas/types.yaml#/definitions/phandle
+ description:
+ phandles to a reserved SRAM region which is used as the memory of
+ the ARC core. The region should be defined as child nodes of the
+ AHB SRAM node as per the generic bindings in
+ Documentation/devicetree/bindings/sram/sram.yaml
+
+ amlogic,secbus2:
+ $ref: /schemas/types.yaml#/definitions/phandle
+ description:
+ A phandle to the SECBUS2 region which contains some configuration
+ bits of this remote processor
+
+required:
+ - compatible
+ - reg
+ - reg-names
+ - resets
+ - clocks
+ - sram
+ - amlogic,secbus2
+
+additionalProperties: false
+
+examples:
+ - |
+ remoteproc@1c {
+ compatible= "amlogic,meson8-ao-arc", "amlogic,meson-mx-ao-arc";
+ reg = <0x1c 0x8>, <0x38 0x8>;
+ reg-names = "remap", "cpu";
+ resets = <&media_cpu_reset>;
+ clocks = <&media_cpu_clock>;
+ sram = <&ahb_sram_ao_arc>;
+ amlogic,secbus2 = <&secbus2>;
+ };
+
+...
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
vpu: video-decoder@132a0000 {
compatible = "ingenic,jz4770-vpu-rproc";
+++ /dev/null
-Mediatek SCP Bindings
-----------------------------------------
-
-This binding provides support for ARM Cortex M4 Co-processor found on some
-Mediatek SoCs.
-
-Required properties:
-- compatible Should be "mediatek,mt8183-scp"
-- reg Should contain the address ranges for memory regions:
- SRAM, CFG, and L1TCM.
-- reg-names Contains the corresponding names for the memory regions:
- "sram", "cfg", and "l1tcm".
-- clocks Clock for co-processor (See: ../clock/clock-bindings.txt)
-- clock-names Contains the corresponding name for the clock. This
- should be named "main".
-
-Subnodes
---------
-
-Subnodes of the SCP represent rpmsg devices. The names of the devices are not
-important. The properties of these nodes are defined by the individual bindings
-for the rpmsg devices - but must contain the following property:
-
-- mtk,rpmsg-name Contains the name for the rpmsg device. Used to match
- the subnode to rpmsg device announced by SCP.
-
-Example:
-
- scp: scp@10500000 {
- compatible = "mediatek,mt8183-scp";
- reg = <0 0x10500000 0 0x80000>,
- <0 0x105c0000 0 0x5000>;
- reg-names = "sram", "cfg";
- clocks = <&infracfg CLK_INFRA_SCPSYS>;
- clock-names = "main";
- };
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/remoteproc/mtk,scp.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Mediatek SCP Bindings
+
+maintainers:
+ - Tinghan Shen <tinghan.shen@mediatek.com>
+
+description:
+ This binding provides support for ARM Cortex M4 Co-processor found on some
+ Mediatek SoCs.
+
+properties:
+ compatible:
+ enum:
+ - mediatek,mt8183-scp
+ - mediatek,mt8192-scp
+ - mediatek,mt8195-scp
+
+ reg:
+ description:
+ Should contain the address ranges for memory regions SRAM, CFG, and
+ L1TCM.
+ maxItems: 3
+
+ reg-names:
+ items:
+ - const: sram
+ - const: cfg
+ - const: l1tcm
+
+ clocks:
+ description:
+ Clock for co-processor (see ../clock/clock-bindings.txt).
+ Required by mt8183 and mt8192.
+ maxItems: 1
+
+ clock-names:
+ const: main
+
+required:
+ - compatible
+ - reg
+ - reg-names
+
+if:
+ properties:
+ compatible:
+ enum:
+ - mediatek,mt8183-scp
+ - mediatek,mt8192-scp
+then:
+ required:
+ - clocks
+ - clock-names
+
+additionalProperties:
+ type: object
+ description:
+ Subnodes of the SCP represent rpmsg devices. The names of the devices
+ are not important. The properties of these nodes are defined by the
+ individual bindings for the rpmsg devices.
+ properties:
+ mediatek,rpmsg-name:
+ $ref: /schemas/types.yaml#/definitions/string-array
+ description:
+ Contains the name for the rpmsg device. Used to match
+ the subnode to rpmsg device announced by SCP.
+
+ required:
+ - mediatek,rpmsg-name
+
+examples:
+ - |
+ #include <dt-bindings/clock/mt8183-clk.h>
+
+ scp@10500000 {
+ compatible = "mediatek,mt8183-scp";
+ reg = <0x10500000 0x80000>,
+ <0x10700000 0x8000>,
+ <0x10720000 0xe0000>;
+ reg-names = "sram", "cfg", "l1tcm";
+ clocks = <&infracfg CLK_INFRA_SCPSYS>;
+ clock-names = "main";
+
+ cros_ec {
+ mediatek,rpmsg-name = "cros-ec-rpmsg";
+ };
+ };
- qcom,qcs404-cdsp-pas
- qcom,qcs404-wcss-pas
- qcom,sc7180-mpss-pas
+ - qcom,sc7280-mpss-pas
- qcom,sc8180x-adsp-pas
- qcom,sc8180x-cdsp-pas
- qcom,sc8180x-mpss-pas
maxItems: 1
description: Reference to the reserved-memory for the Hexagon core
+ qcom,qmp:
+ $ref: /schemas/types.yaml#/definitions/phandle
+ description: Reference to the AOSS side-channel message RAM.
+
qcom,smem-states:
$ref: /schemas/types.yaml#/definitions/phandle-array
description: States used by the AP to signal the Hexagon core
- qcom,msm8998-adsp-pas
- qcom,qcs404-adsp-pas
- qcom,qcs404-wcss-pas
+ - qcom,sc7280-mpss-pas
- qcom,sc8180x-adsp-pas
- qcom,sc8180x-cdsp-pas
- qcom,sc8180x-mpss-pas
contains:
enum:
- qcom,sc7180-mpss-pas
+ - qcom,sc7280-mpss-pas
- qcom,sc8180x-mpss-pas
- qcom,sdx55-mpss-pas
- qcom,sm8150-mpss-pas
properties:
power-domains:
items:
- - description: Load State power domain
- description: CX power domain
- description: MX power domain
- description: MSS power domain
power-domain-names:
items:
- - const: load_state
- const: cx
- const: mx
- const: mss
properties:
power-domains:
items:
- - description: Load State power domain
- description: CX power domain
- power-domain-names:
- items:
- - const: load_state
- - const: cx
- if:
properties:
compatible:
contains:
enum:
+ - qcom,sc7280-mpss-pas
+ - qcom,sdx55-mpss-pas
- qcom,sm8150-mpss-pas
- qcom,sm8350-mpss-pas
then:
properties:
power-domains:
items:
- - description: Load State power domain
- - description: CX power domain
- - description: MSS power domain
- power-domain-names:
- items:
- - const: load_state
- - const: cx
- - const: mss
-
- - if:
- properties:
- compatible:
- contains:
- enum:
- - qcom,sdx55-mpss-pas
- then:
- properties:
- power-domains:
- items:
- description: CX power domain
- description: MSS power domain
power-domain-names:
properties:
power-domains:
items:
- - description: Load State power domain
- description: LCX power domain
- description: LMX power domain
power-domain-names:
items:
- - const: load_state
- const: lcx
- const: lmx
properties:
power-domains:
items:
- - description: Load State power domain
- description: CX power domain
- description: MXC power domain
power-domain-names:
items:
- - const: load_state
- const: cx
- const: mxc
contains:
enum:
- qcom,sc7180-mpss-pas
+ - qcom,sc7280-mpss-pas
then:
properties:
resets:
- const: mss_restart
- const: pdc_reset
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - qcom,msm8974-adsp-pil
+ - qcom,msm8996-adsp-pil
+ - qcom,msm8996-slpi-pil
+ - qcom,msm8998-adsp-pas
+ - qcom,msm8998-slpi-pas
+ - qcom,qcs404-adsp-pas
+ - qcom,qcs404-cdsp-pas
+ - qcom,qcs404-wcss-pas
+ - qcom,sdm660-adsp-pas
+ - qcom,sdx55-mpss-pas
+ then:
+ properties:
+ qcom,qmp: false
+
examples:
- |
#include <dt-bindings/clock/qcom,rpmcc.h>
"qcom,msm8996-mss-pil"
"qcom,msm8998-mss-pil"
"qcom,sc7180-mss-pil"
+ "qcom,sc7280-mss-pil"
"qcom,sdm845-mss-pil"
- reg:
qcom,msm8996-mss-pil:
qcom,msm8998-mss-pil:
qcom,sc7180-mss-pil:
+ qcom,sc7280-mss-pil:
qcom,sdm845-mss-pil:
must be "wdog", "fatal", "ready", "handover", "stop-ack",
"shutdown-ack"
qcom,sc7180-mss-pil:
must be "iface", "bus", "xo", "snoc_axi", "mnoc_axi",
"nav"
+ qcom,sc7280-mss-pil:
+ must be "iface", "xo", "snoc_axi", "offline", "pka"
qcom,sdm845-mss-pil:
must be "iface", "bus", "mem", "xo", "gpll0_mss",
"snoc_axi", "mnoc_axi", "prng"
reference to the list of 3 reset-controllers for the
wcss sub-system
reference to the list of 2 reset-controllers for the modem
- sub-system on SC7180, SDM845 SoCs
+ sub-system on SC7180, SC7280, SDM845 SoCs
- reset-names:
Usage: required
must be "wcss_aon_reset", "wcss_reset", "wcss_q6_reset"
for the wcss sub-system
must be "mss_restart", "pdc_reset" for the modem
- sub-system on SC7180, SDM845 SoCs
+ sub-system on SC7180, SC7280, SDM845 SoCs
For devices where the mba and mpss sub-nodes are not specified, mba/mpss region
should be referenced as follows:
qcom,msm8998-mss-pil:
must be "cx", "mx"
qcom,sc7180-mss-pil:
+ must be "cx", "mx", "mss"
+ qcom,sc7280-mss-pil:
+ must be "cx", "mss"
qcom,sdm845-mss-pil:
- must be "cx", "mx", "mss", "load_state"
+ must be "cx", "mx", "mss"
+
+- qcom,qmp:
+ Usage: optional
+ Value type: <phandle>
+ Definition: reference to the AOSS side-channel message RAM.
- qcom,smem-states:
Usage: required
Definition: a phandle reference to a syscon representing TCSR followed
by the three offsets within syscon for q6, modem and nc
halt registers.
+ a phandle reference to a syscon representing TCSR followed
+ by the four offsets within syscon for q6, modem, nc and vq6
+ halt registers on SC7280 SoCs.
For the compatible strings below the following phandle references are required:
"qcom,sc7180-mss-pil"
by the offset within syscon for conn_box_spare0 register
used by the modem sub-system running on SC7180 SoC.
+For the compatible strings below the following phandle references are required:
+ "qcom,sc7280-mss-pil"
+- qcom,ext-regs:
+ Usage: required
+ Value type: <prop-encoded-array>
+ Definition: two phandle references to syscons representing TCSR_REG and
+ TCSR register space followed by the two offsets within the syscon
+ to force_clk_en/rscc_disable and axim1_clk_off/crypto_clk_off
+ registers respectively.
+
+- qcom,qaccept-regs:
+ Usage: required
+ Value type: <prop-encoded-array>
+ Definition: a phandle reference to a syscon representing TCSR followed
+ by the three offsets within syscon for mdm, cx and axi
+ qaccept registers used by the modem sub-system running on
+ SC7280 SoC.
+
The Hexagon node must contain iommus property as described in ../iommu/iommu.txt
on platforms which do not have TrustZone.
boots firmwares on the ST32MP family chipset.
maintainers:
- - Fabien Dessenne <fabien.dessenne@st.com>
- - Arnaud Pouliquen <arnaud.pouliquen@st.com>
+ - Fabien Dessenne <fabien.dessenne@foss.st.com>
+ - Arnaud Pouliquen <arnaud.pouliquen@foss.st.com>
properties:
compatible:
examples:
- |
- / {
- model = "Texas Instruments K3 J721E SoC";
- compatible = "ti,j721e";
+ soc {
#address-cells = <2>;
#size-cells = <2>;
examples:
- |
- / {
- model = "Texas Instruments K3 AM654 SoC";
- compatible = "ti,am654-evm", "ti,am654";
+ soc {
#address-cells = <2>;
#size-cells = <2>;
examples:
- |
- #include <dt-bindings/clock/x1830-cgu.h>
+ #include <dt-bindings/clock/ingenic,x1830-cgu.h>
dtrng: trng@10072000 {
compatible = "ingenic,x1830-dtrng";
IP and is fully separated from other crypto functions.
maintainers:
- - Lionel Debieve <lionel.debieve@st.com>
+ - Lionel Debieve <lionel.debieve@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
rtc_dev: rtc@10003000 {
compatible = "ingenic,jz4740-rtc";
reg = <0x10003000 0x40>;
--- /dev/null
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/rtc/mstar,msc313-rtc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Mstar MSC313e RTC Device Tree Bindings
+
+allOf:
+ - $ref: "rtc.yaml#"
+
+maintainers:
+ - Daniel Palmer <daniel@0x0f.com>
+ - Romain Perier <romain.perier@gmail.com>
+
+properties:
+ compatible:
+ enum:
+ - mstar,msc313-rtc
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ start-year: true
+
+ clocks:
+ maxItems: 1
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ rtc@2400 {
+ compatible = "mstar,msc313-rtc";
+ reg = <0x2400 0x40>;
+ clocks = <&xtal_div2>;
+ interrupts-extended = <&intc_irq GIC_SPI 44 IRQ_TYPE_LEVEL_HIGH>;
+ };
+...
expressed in femto Farad (fF). Valid values are 7000 and 12500.
Default value (if no value is specified) is 7000fF.
+Optional child node:
+- clock: Provide this if the square wave pin is used as boot-enabled fixed clock.
+
Example:
pcf85063: rtc@51 {
compatible = "nxp,pcf85063";
reg = <0x51>;
quartz-load-femtofarads = <12500>;
+
+ clock {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ clock-frequency = <32768>;
+ };
};
title: STMicroelectronics STM32 Real Time Clock Bindings
maintainers:
- - Gabriel Fernandez <gabriel.fernandez@st.com>
+ - Gabriel Fernandez <gabriel.fernandez@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4780-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include <dt-bindings/dma/jz4780-dma.h>
#include <dt-bindings/gpio/gpio.h>
serial@10032000 {
$schema: http://devicetree.org/meta-schemas/core.yaml#
maintainers:
- - Erwan Le Ray <erwan.leray@st.com>
+ - Erwan Le Ray <erwan.leray@foss.st.com>
title: STMicroelectronics STM32 USART bindings
title: CS42L51 audio codec DT bindings
maintainers:
- - Olivier Moysan <olivier.moysan@st.com>
+ - Olivier Moysan <olivier.moysan@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
aic: audio-controller@10020000 {
compatible = "ingenic,jz4740-i2s";
reg = <0x10020000 0x38>;
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
codec: audio-codec@10020080 {
compatible = "ingenic,jz4740-codec";
reg = <0x10020080 0x8>;
title: STMicroelectronics STM32 SPI/I2S Controller
maintainers:
- - Olivier Moysan <olivier.moysan@st.com>
+ - Olivier Moysan <olivier.moysan@foss.st.com>
description:
The SPI/I2S block supports I2S/PCM protocols when configured on I2S mode.
title: STMicroelectronics STM32 Serial Audio Interface (SAI)
maintainers:
- - Olivier Moysan <olivier.moysan@st.com>
+ - Olivier Moysan <olivier.moysan@foss.st.com>
description:
The SAI interface (Serial Audio Interface) offers a wide set of audio
title: STMicroelectronics STM32 S/PDIF receiver (SPDIFRX)
maintainers:
- - Olivier Moysan <olivier.moysan@st.com>
+ - Olivier Moysan <olivier.moysan@foss.st.com>
description: |
The SPDIFRX peripheral, is designed to receive an S/PDIF flow compliant with
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
spi@10043000 {
compatible = "ingenic,jz4770-spi", "ingenic,jz4750-spi";
reg = <0x10043000 0x1c>;
+++ /dev/null
-SPI Master controller for Netlogic XLP MIPS64 SOCs
-==================================================
-
-Currently this SPI controller driver is supported for the following
-Netlogic XLP SoCs:
- XLP832, XLP316, XLP208, XLP980, XLP532
-
-Required properties:
-- compatible : Should be "netlogic,xlp832-spi".
-- #address-cells : Number of cells required to define a chip select address
- on the SPI bus.
-- #size-cells : Should be zero.
-- reg : Should contain register location and length.
-- clocks : Phandle of the spi clock
-- interrupts : Interrupt number used by this controller.
-
-SPI slave nodes must be children of the SPI master node and can contain
-properties described in Documentation/devicetree/bindings/spi/spi-bus.txt.
-
-Example:
-
- spi: xlp_spi@3a100 {
- compatible = "netlogic,xlp832-spi";
- #address-cells = <1>;
- #size-cells = <0>;
- reg = <0 0x3a100 0x100>;
- clocks = <&spi_clk>;
- interrupts = <34>;
- interrupt-parent = <&pic>;
-
- spi_nor@1 {
- compatible = "spansion,s25sl12801";
- #address-cells = <1>;
- #size-cells = <1>;
- reg = <1>; /* Chip Select */
- spi-max-frequency = <40000000>;
- };
-};
title: STMicroelectronics STM32 Quad Serial Peripheral Interface (QSPI) bindings
maintainers:
- - Christophe Kerello <christophe.kerello@st.com>
- - Patrice Chotard <patrice.chotard@st.com>
+ - Christophe Kerello <christophe.kerello@foss.st.com>
+ - Patrice Chotard <patrice.chotard@foss.st.com>
allOf:
- $ref: "spi-controller.yaml#"
from 4 to 32-bit data size.
maintainers:
- - Erwan Leray <erwan.leray@st.com>
- - Fabrice Gasnier <fabrice.gasnier@st.com>
+ - Erwan Leray <erwan.leray@foss.st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
allOf:
- $ref: "spi-controller.yaml#"
title: STMicroelectronics STM32 digital thermal sensor (DTS) binding
maintainers:
- - David Hernandez Sanchez <david.hernandezsanchez@st.com>
+ - Pascal Paillet <p.paillet@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/x1000-cgu.h>
+ #include <dt-bindings/clock/ingenic,x1000-cgu.h>
ost: timer@12000000 {
compatible = "ingenic,x1000-ost";
examples:
- |
- #include <dt-bindings/clock/jz4770-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4770-cgu.h>
#include <dt-bindings/clock/ingenic,tcu.h>
tcu: timer@10002000 {
compatible = "ingenic,jz4770-tcu", "ingenic,jz4760-tcu", "simple-mfd";
title: STMicroelectronics STM32 general-purpose 16 and 32 bits timers bindings
maintainers:
- - Benjamin Gaignard <benjamin.gaignard@st.com>
+ - Fabrice Gasnier <fabrice.gasnier@foss.st.com>
+ - Patrice Chotard <patrice.chotard@foss.st.com>
properties:
compatible:
examples:
- |
- #include <dt-bindings/clock/jz4740-cgu.h>
+ #include <dt-bindings/clock/ingenic,jz4740-cgu.h>
usb_phy: usb-phy {
compatible = "usb-nop-xceiv";
#phy-cells = <0>;
title: STMicroelectronics STUSB160x Type-C controller bindings
maintainers:
- - Amelie Delaunay <amelie.delaunay@st.com>
+ - Amelie Delaunay <amelie.delaunay@foss.st.com>
properties:
compatible:
description: Vitesse Semiconductor Corporation
"^vivante,.*":
description: Vivante Corporation
+ "^vivax,.*":
+ description: Vivax brand by M SAN Grupa d.o.o.
"^vocore,.*":
description: VoCore Studio
"^voipac,.*":
- allwinner,sun50i-a100-wdt
- allwinner,sun50i-h6-wdt
- allwinner,sun50i-h616-wdt
+ - allwinner,sun50i-r329-wdt
+ - allwinner,sun50i-r329-wdt-reset
- const: allwinner,sun6i-a31-wdt
- items:
- const: allwinner,suniv-f1c100s-wdt
- const: allwinner,sun4i-a10-wdt
+ - const: allwinner,sun20i-d1-wdt
+ - items:
+ - const: allwinner,sun20i-d1-wdt-reset
+ - const: allwinner,sun20i-d1-wdt
reg:
maxItems: 1
clocks:
- maxItems: 1
+ minItems: 1
+ items:
+ - description: High-frequency oscillator input, divided internally
+ - description: Low-frequency oscillator input, only found on some variants
+
+ clock-names:
+ minItems: 1
+ items:
+ - const: hosc
+ - const: losc
interrupts:
maxItems: 1
- clocks
- interrupts
+if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - allwinner,sun20i-d1-wdt
+ - allwinner,sun20i-d1-wdt-reset
+ - allwinner,sun50i-r329-wdt
+ - allwinner,sun50i-r329-wdt-reset
+
+then:
+ properties:
+ clocks:
+ minItems: 2
+
+ clock-names:
+ minItems: 2
+
+ required:
+ - clock-names
+
+else:
+ properties:
+ clocks:
+ maxItems: 1
+
+ clock-names:
+ maxItems: 1
+
unevaluatedProperties: false
examples:
- reg : Specifies base physical address and size of the registers.
Optional properties:
+- mediatek,disable-extrst: disable send output reset signal
- interrupts: Watchdog pre-timeout (bark) interrupt.
- timeout-sec: contains the watchdog timeout in seconds.
- #reset-cells: Should be 1.
watchdog: watchdog@10007000 {
compatible = "mediatek,mt8183-wdt",
"mediatek,mt6589-wdt";
+ mediatek,disable-extrst;
reg = <0 0x10007000 0 0x100>;
interrupts = <GIC_SPI 139 IRQ_TYPE_NONE>;
timeout-sec = <10>;
title: STMicroelectronics STM32 Independent WatchDoG (IWDG) bindings
maintainers:
- - Yannick Fertre <yannick.fertre@st.com>
- - Christophe Roullier <christophe.roullier@st.com>
+ - Yannick Fertre <yannick.fertre@foss.st.com>
+ - Christophe Roullier <christophe.roullier@foss.st.com>
allOf:
- $ref: "watchdog.yaml#"
==============
The ReST markups currently used by the Documentation/ files are meant to be
-built with ``Sphinx`` version 1.3 or higher.
+built with ``Sphinx`` version 1.7 or higher.
There's a script that checks for the Sphinx requirements. Please see
:ref:`sphinx-pre-install` for further details.
.. note::
- #) Sphinx versions below 1.5 don't work properly with Python's
- docutils version 0.13.1 or higher. So, if you're willing to use
- those versions, you should run ``pip install 'docutils==0.12'``.
-
#) It is recommended to use the RTD theme for html output. Depending
on the Sphinx version, it should be installed separately,
with ``pip install sphinx_rtd_theme``.
those expressions are written using LaTeX notation. It needs texlive
installed with amsfonts and amsmath in order to evaluate them.
-In summary, if you want to install Sphinx version 1.7.9, you should do::
+In summary, if you want to install Sphinx version 2.4.4, you should do::
- $ virtualenv sphinx_1.7.9
- $ . sphinx_1.7.9/bin/activate
- (sphinx_1.7.9) $ pip install -r Documentation/sphinx/requirements.txt
+ $ virtualenv sphinx_2.4.4
+ $ . sphinx_2.4.4/bin/activate
+ (sphinx_2.4.4) $ pip install -r Documentation/sphinx/requirements.txt
-After running ``. sphinx_1.7.9/bin/activate``, the prompt will change,
+After running ``. sphinx_2.4.4/bin/activate``, the prompt will change,
in order to indicate that you're using the new environment. If you
open a new shell, you need to rerun this command to enter again at
the virtual environment before building the documentation.
PDF and LaTeX builds
--------------------
-Such builds are currently supported only with Sphinx versions 1.4 and higher.
+Such builds are currently supported only with Sphinx versions 2.4 and higher.
For PDF and LaTeX output, you'll also need ``XeLaTeX`` version 3.14159265.
You should run:
sudo dnf install -y texlive-luatex85
- /usr/bin/virtualenv sphinx_1.7.9
- . sphinx_1.7.9/bin/activate
+ /usr/bin/virtualenv sphinx_2.4.4
+ . sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
.. kernel-doc:: drivers/cxl/core/bus.c
:doc: cxl core
+.. kernel-doc:: drivers/cxl/core/bus.c
+ :identifiers:
+
.. kernel-doc:: drivers/cxl/core/pmem.c
:doc: cxl pmem
.. kernel-doc:: drivers/cxl/core/regs.c
:doc: cxl registers
+.. kernel-doc:: drivers/cxl/core/mbox.c
+ :doc: cxl mbox
+
External Interfaces
===================
required with any user-space program. Subsequent text refers to this
as the "automount daemon" or simply "the daemon".
-"autofs" is a Linux kernel module with provides the "autofs"
+"autofs" is a Linux kernel module which provides the "autofs"
filesystem type. Several "autofs" filesystems can be mounted and they
can each be managed separately, or all managed by the same daemon.
FAULT_DISCARD 0x000002000
FAULT_WRITE_IO 0x000004000
FAULT_SLAB_ALLOC 0x000008000
+ FAULT_DQUOT_INIT 0x000010000
=================== ===========
mode=%s Control block allocation mode which supports "adaptive"
and "lfs". In "lfs" mode, there should be no random
writes towards main area.
+ "fragment:segment" and "fragment:block" are newly added here.
+ These are developer options for experiments to simulate filesystem
+ fragmentation/after-GC situation itself. The developers use these
+ modes to understand filesystem fragmentation/after-GC condition well,
+ and eventually get some insights to handle them better.
+ In "fragment:segment", f2fs allocates a new segment in ramdom
+ position. With this, we can simulate the after-GC condition.
+ In "fragment:block", we can scatter block allocation with
+ "max_fragment_chunk" and "max_fragment_hole" sysfs nodes.
+ We added some randomness to both chunk and hole size to make
+ it close to realistic IO pattern. So, in this mode, f2fs will allocate
+ 1..<max_fragment_chunk> blocks in a chunk and make a hole in the
+ length of 1..<max_fragment_hole> by turns. With this, the newly
+ allocated blocks will be scattered throughout the whole partition.
+ Note that "fragment:block" implicitly enables "fragment:segment"
+ option for more randomness.
+ Please, use these options for your experiments and we strongly
+ recommend to re-format the filesystem after using these options.
io_bits=%u Set the bit size of write IO requests. It should be set
with "mode=lfs".
usrquota Enable plain user disk quota accounting.
rpc-server-gss
nfs41-server
knfsd-stats
+ reexport
--- /dev/null
+Reexporting NFS filesystems
+===========================
+
+Overview
+--------
+
+It is possible to reexport an NFS filesystem over NFS. However, this
+feature comes with a number of limitations. Before trying it, we
+recommend some careful research to determine whether it will work for
+your purposes.
+
+A discussion of current known limitations follows.
+
+"fsid=" required, crossmnt broken
+---------------------------------
+
+We require the "fsid=" export option on any reexport of an NFS
+filesystem. You can use "uuidgen -r" to generate a unique argument.
+
+The "crossmnt" export does not propagate "fsid=", so it will not allow
+traversing into further nfs filesystems; if you wish to export nfs
+filesystems mounted under the exported filesystem, you'll need to export
+them explicitly, assigning each its own unique "fsid= option.
+
+Reboot recovery
+---------------
+
+The NFS protocol's normal reboot recovery mechanisms don't work for the
+case when the reexport server reboots. Clients will lose any locks
+they held before the reboot, and further IO will result in errors.
+Closing and reopening files should clear the errors.
+
+Filehandle limits
+-----------------
+
+If the original server uses an X byte filehandle for a given object, the
+reexport server's filehandle for the reexported object will be X+22
+bytes, rounded up to the nearest multiple of four bytes.
+
+The result must fit into the RFC-mandated filehandle size limits:
+
++-------+-----------+
+| NFSv2 | 32 bytes |
++-------+-----------+
+| NFSv3 | 64 bytes |
++-------+-----------+
+| NFSv4 | 128 bytes |
++-------+-----------+
+
+So, for example, you will only be able to reexport a filesystem over
+NFSv2 if the original server gives you filehandles that fit in 10
+bytes--which is unlikely.
+
+In general there's no way to know the maximum filehandle size given out
+by an NFS server without asking the server vendor.
+
+But the following table gives a few examples. The first column is the
+typical length of the filehandle from a Linux server exporting the given
+filesystem, the second is the length after that nfs export is reexported
+by another Linux host:
+
++--------+-------------------+----------------+
+| | filehandle length | after reexport |
++========+===================+================+
+| ext4: | 28 bytes | 52 bytes |
++--------+-------------------+----------------+
+| xfs: | 32 bytes | 56 bytes |
++--------+-------------------+----------------+
+| btrfs: | 40 bytes | 64 bytes |
++--------+-------------------+----------------+
+
+All will therefore fit in an NFSv3 or NFSv4 filehandle after reexport,
+but none are reexportable over NFSv2.
+
+Linux server filehandles are a bit more complicated than this, though;
+for example:
+
+ - The (non-default) "subtreecheck" export option generally
+ requires another 4 to 8 bytes in the filehandle.
+ - If you export a subdirectory of a filesystem (instead of
+ exporting the filesystem root), that also usually adds 4 to 8
+ bytes.
+ - If you export over NFSv2, knfsd usually uses a shorter
+ filesystem identifier that saves 8 bytes.
+ - The root directory of an export uses a filehandle that is
+ shorter.
+
+As you can see, the 128-byte NFSv4 filehandle is large enough that
+you're unlikely to have trouble using NFSv4 to reexport any filesystem
+exported from a Linux server. In general, if the original server is
+something that also supports NFSv3, you're *probably* OK. Re-exporting
+over NFSv3 may be dicier, and reexporting over NFSv2 will probably
+never work.
+
+For more details of Linux filehandle structure, the best reference is
+the source code and comments; see in particular:
+
+ - include/linux/exportfs.h:enum fid_type
+ - include/uapi/linux/nfsd/nfsfh.h:struct nfs_fhbase_new
+ - fs/nfsd/nfsfh.c:set_version_and_fsid_type
+ - fs/nfs/export.c:nfs_encode_fh
+
+Open DENY bits ignored
+----------------------
+
+NFS since NFSv4 supports ALLOW and DENY bits taken from Windows, which
+allow you, for example, to open a file in a mode which forbids other
+read opens or write opens. The Linux client doesn't use them, and the
+server's support has always been incomplete: they are enforced only
+against other NFS users, not against processes accessing the exported
+filesystem locally. A reexport server will also not pass them along to
+the original server, so they will not be enforced between clients of
+different reexport servers.
acpi-lid
lpit
video_extension
+ non-d0-probe
extcon-intel-int3496
intel-pmc-mux
--- /dev/null
+.. SPDX-License-Identifier: GPL-2.0
+
+========================================
+Probing devices in other D states than 0
+========================================
+
+Introduction
+============
+
+In some cases it may be preferred to leave certain devices powered off for the
+entire system bootup if powering on these devices has adverse side effects,
+beyond just powering on the said device.
+
+How it works
+============
+
+The _DSC (Device State for Configuration) object that evaluates to an integer
+may be used to tell Linux the highest allowed D state for a device during
+probe. The support for _DSC requires support from the kernel bus type if the
+bus driver normally sets the device in D0 state for probe.
+
+The downside of using _DSC is that as the device is not powered on, even if
+there's a problem with the device, the driver likely probes just fine but the
+first user will find out the device doesn't work, instead of a failure at probe
+time. This feature should thus be used sparingly.
+
+I²C
+---
+
+If an I²C driver indicates its support for this by setting the
+I2C_DRV_ACPI_WAIVE_D0_PROBE flag in struct i2c_driver.flags field and the
+_DSC object evaluates to integer higher than the D state of the device,
+the device will not be powered on (put in D0 state) for probe.
+
+D states
+--------
+
+The D states and thus also the allowed values for _DSC are listed below. Refer
+to [1] for more information on device power states.
+
+.. code-block:: text
+
+ Number State Description
+ 0 D0 Device fully powered on
+ 1 D1
+ 2 D2
+ 3 D3hot
+ 4 D3cold Off
+
+References
+==========
+
+[1] https://uefi.org/specifications/ACPI/6.4/02_Definition_of_Terms/Definition_of_Terms.html#device-power-state-definitions
+
+Example
+=======
+
+An ASL example describing an ACPI device using _DSC object to tell Operating
+System the device should remain powered off during probe looks like this. Some
+objects not relevant from the example point of view have been omitted.
+
+.. code-block:: text
+
+ Device (CAM0)
+ {
+ Name (_HID, "SONY319A")
+ Name (_UID, Zero)
+ Name (_CRS, ResourceTemplate ()
+ {
+ I2cSerialBus(0x0020, ControllerInitiated, 0x00061A80,
+ AddressingMode7Bit, "\\_SB.PCI0.I2C0",
+ 0x00, ResourceConsumer)
+ })
+ Method (_DSC, 0, NotSerialized)
+ {
+ Return (0x4)
+ }
+ }
Garbage collect fbdev scrolling acceleration
--------------------------------------------
-Scroll acceleration is disabled in fbcon by hard-wiring p->scrollmode =
-SCROLL_REDRAW. There's a ton of code this will allow us to remove:
+Scroll acceleration has been disabled in fbcon. Now it works as the old
+SCROLL_REDRAW mode. A ton of code was removed in fbcon.c and the hook bmove was
+removed from fbcon_ops.
+Remaining tasks:
-- lots of code in fbcon.c
-
-- a bunch of the hooks in fbcon_ops, maybe the remaining hooks could be called
+- a bunch of the hooks in fbcon_ops could be removed or simplified by calling
directly instead of the function table (with a switch on p->rotate)
- fb_copyarea is unused after this, and can be deleted from all drivers
+- after that, fb_copyarea can be deleted from fb_ops in include/linux/fb.h as
+ well as cfb_copyarea
+
Note that not all acceleration code can be deleted, since clearing and cursor
support is still accelerated, which might be good candidates for further
deletion projects.
The above assignment instructs kbuild to descend down in the
directory compressed/ when "make clean" is executed.
-To support the clean infrastructure in the Makefiles that build the
-final bootimage there is an optional target named archclean:
-
- Example::
-
- #arch/x86/Makefile
- archclean:
- $(Q)$(MAKE) $(clean)=arch/x86/boot
-
-When "make clean" is executed, make will descend down in arch/x86/boot,
-and clean as usual. The Makefile located in arch/x86/boot/ may use
-the subdir- trick to descend further down.
-
Note 1: arch/$(SRCARCH)/Makefile cannot use "subdir-", because that file is
-included in the top level makefile, and the kbuild infrastructure
-is not operational at that point.
+included in the top level makefile. Instead, arch/$(SRCARCH)/Kbuild can use
+"subdir-".
Note 2: All directories listed in core-y, libs-y, drivers-y and net-y will
be visited during "make clean".
udp_mem - vector of 3 INTEGERs: min, pressure, max
Number of pages allowed for queueing by all UDP sockets.
- min: Below this number of pages UDP is not bothered about its
- memory appetite. When amount of memory allocated by UDP exceeds
- this number, UDP starts to moderate memory usage.
+ min: Number of pages allowed for queueing by all UDP sockets.
pressure: This value was introduced to follow format of tcp_mem.
- max: Number of pages allowed for queueing by all UDP sockets.
+ max: This value was introduced to follow format of tcp_mem.
Default is calculated at boot time from amount of available memory.
2.2 Registration of performance domains
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Registration of 'advanced' EM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The 'advanced' EM gets it's name due to the fact that the driver is allowed
+to provide more precised power model. It's not limited to some implemented math
+formula in the framework (like it's in 'simple' EM case). It can better reflect
+the real power measurements performed for each performance state. Thus, this
+registration method should be preferred in case considering EM static power
+(leakage) is important.
+
Drivers are expected to register performance domains into the EM framework by
calling the following API::
See Section 3. for an example of driver implementing this
callback, or Section 2.4 for further documentation on this API
+Registration of 'simple' EM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The 'simple' EM is registered using the framework helper function
+cpufreq_register_em_with_opp(). It implements a power model which is tight to
+math formula::
+
+ Power = C * V^2 * f
+
+The EM which is registered using this method might not reflect correctly the
+physics of a real device, e.g. when static power (leakage) is important.
+
2.3 Accessing performance domains
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3. Example driver
-----------------
+The CPUFreq framework supports dedicated callback for registering
+the EM for a given CPU(s) 'policy' object: cpufreq_driver::register_em().
+That callback has to be implemented properly for a given driver,
+because the framework would call it at the right time during setup.
This section provides a simple example of a CPUFreq driver registering a
performance domain in the Energy Model framework using the (fake) 'foo'
protocol. The driver implements an est_power() function to be provided to the
20 return 0;
21 }
22
- 23 static int foo_cpufreq_init(struct cpufreq_policy *policy)
+ 23 static void foo_cpufreq_register_em(struct cpufreq_policy *policy)
24 {
25 struct em_data_callback em_cb = EM_DATA_CB(est_power);
26 struct device *cpu_dev;
- 27 int nr_opp, ret;
+ 27 int nr_opp;
28
29 cpu_dev = get_cpu_device(cpumask_first(policy->cpus));
30
- 31 /* Do the actual CPUFreq init work ... */
- 32 ret = do_foo_cpufreq_init(policy);
- 33 if (ret)
- 34 return ret;
- 35
- 36 /* Find the number of OPPs for this policy */
- 37 nr_opp = foo_get_nr_opp(policy);
+ 31 /* Find the number of OPPs for this policy */
+ 32 nr_opp = foo_get_nr_opp(policy);
+ 33
+ 34 /* And register the new performance domain */
+ 35 em_dev_register_perf_domain(cpu_dev, nr_opp, &em_cb, policy->cpus,
+ 36 true);
+ 37 }
38
- 39 /* And register the new performance domain */
- 40 em_dev_register_perf_domain(cpu_dev, nr_opp, &em_cb, policy->cpus,
- 41 true);
- 42
- 43 return 0;
- 44 }
+ 39 static struct cpufreq_driver foo_cpufreq_driver = {
+ 40 .register_em = foo_cpufreq_register_em,
+ 41 };
iptables 1.4.2 iptables -V
openssl & libcrypto 1.0.0 openssl version
bc 1.06.95 bc --version
-Sphinx\ [#f1]_ 1.3 sphinx-build --version
+Sphinx\ [#f1]_ 1.7 sphinx-build --version
====================== =============== ========================================
.. [#f1] Sphinx is needed only to build the Kernel documentation
easier.
Some subsystems and maintainer trees have additional information about
-their workflow and expectations, see :ref:`Documentation/process/maintainer
-handbooks <maintainer_handbooks_main>`.
+their workflow and expectations, see
+:ref:`Documentation/process/maintainer-handbooks.rst <maintainer_handbooks_main>`.
Obtain a current source tree
----------------------------
security_sctp_assoc_request()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Passes the ``@ep`` and ``@chunk->skb`` of the association INIT packet to the
+Passes the ``@asoc`` and ``@chunk->skb`` of the association INIT packet to the
security module. Returns 0 on success, error on failure.
::
- @ep - pointer to sctp endpoint structure.
+ @asoc - pointer to sctp association structure.
@skb - pointer to skbuff of association packet.
calls **sctp_peeloff**\(3).
::
- @ep - pointer to current sctp endpoint structure.
+ @asoc - pointer to current sctp association structure.
@sk - pointer to current sock structure.
- @sk - pointer to new sock structure.
+ @newsk - pointer to new sock structure.
security_inet_conn_established()
INIT --------------------------------------------->
sctp_sf_do_5_1B_init()
Respond to an INIT chunk.
- SCTP peer endpoint "A" is
- asking for an association. Call
- security_sctp_assoc_request()
+ SCTP peer endpoint "A" is asking
+ for a temporary association.
+ Call security_sctp_assoc_request()
to set the peer label if first
association.
If not first association, check
| discard the packet.
|
COOKIE ECHO ------------------------------------------>
- |
- |
- |
+ sctp_sf_do_5_1D_ce()
+ Respond to an COOKIE ECHO chunk.
+ Confirm the cookie and create a
+ permanent association.
+ Call security_sctp_assoc_request() to
+ do the same as for INIT chunk Response.
<------------------------------------------- COOKIE ACK
| |
sctp_sf_do_5_1E_ca |
security_sctp_assoc_request()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Passes the ``@ep`` and ``@chunk->skb`` of the association INIT packet to the
+Passes the ``@asoc`` and ``@chunk->skb`` of the association INIT packet to the
security module. Returns 0 on success, error on failure.
::
- @ep - pointer to sctp endpoint structure.
+ @asoc - pointer to sctp association structure.
@skb - pointer to skbuff of association packet.
The security module performs the following operations:
- IF this is the first association on ``@ep->base.sk``, then set the peer
+ IF this is the first association on ``@asoc->base.sk``, then set the peer
sid to that in ``@skb``. This will ensure there is only one peer sid
- assigned to ``@ep->base.sk`` that may support multiple associations.
+ assigned to ``@asoc->base.sk`` that may support multiple associations.
- ELSE validate the ``@ep->base.sk peer_sid`` against the ``@skb peer sid``
+ ELSE validate the ``@asoc->base.sk peer_sid`` against the ``@skb peer sid``
to determine whether the association should be allowed or denied.
- Set the sctp ``@ep sid`` to socket's sid (from ``ep->base.sk``) with
+ Set the sctp ``@asoc sid`` to socket's sid (from ``asoc->base.sk``) with
MLS portion taken from ``@skb peer sid``. This will be used by SCTP
TCP style sockets and peeled off connections as they cause a new socket
to be generated.
Called whenever a new socket is created by **accept**\(2) (i.e. a TCP style
socket) or when a socket is 'peeled off' e.g userspace calls
**sctp_peeloff**\(3). ``security_sctp_sk_clone()`` will set the new
-sockets sid and peer sid to that contained in the ``@ep sid`` and
-``@ep peer sid`` respectively.
+sockets sid and peer sid to that contained in the ``@asoc sid`` and
+``@asoc peer sid`` respectively.
::
- @ep - pointer to current sctp endpoint structure.
+ @asoc - pointer to current sctp association structure.
@sk - pointer to current sock structure.
- @sk - pointer to new sock structure.
+ @newsk - pointer to new sock structure.
security_inet_conn_established()
#!/bin/bash
tracefs=`sed -ne 's/^tracefs \(.*\) tracefs.*/\1/p' /proc/mounts`
- echo nop > $tracefs/tracing/current_tracer
- echo 0 > $tracefs/tracing/tracing_on
- echo $$ > $tracefs/tracing/set_ftrace_pid
- echo function > $tracefs/tracing/current_tracer
- echo 1 > $tracefs/tracing/tracing_on
+ echo 0 > $tracefs/tracing_on
+ echo $$ > $tracefs/set_ftrace_pid
+ echo function > $tracefs/current_tracer
+ echo 1 > $tracefs/tracing_on
exec "$@"
====================
I marcatori ReST utilizzati nei file in Documentation/ sono pensati per essere
-processati da ``Sphinx`` nella versione 1.3 o superiore.
+processati da ``Sphinx`` nella versione 1.7 o superiore.
Esiste uno script che verifica i requisiti Sphinx. Per ulteriori dettagli
consultate :ref:`it_sphinx-pre-install`.
.. note::
- #) Le versioni di Sphinx inferiori alla 1.5 non funzionano bene
- con il pacchetto Python docutils versione 0.13.1 o superiore.
- Se volete usare queste versioni, allora dovere eseguire
- ``pip install 'docutils==0.12'``.
-
#) Viene raccomandato l'uso del tema RTD per la documentazione in HTML.
A seconda della versione di Sphinx, potrebbe essere necessaria
l'installazione tramite il comando ``pip install sphinx_rtd_theme``.
utilizzando LaTeX. Per una corretta interpretazione, è necessario aver
installato texlive con i pacchetti amdfonts e amsmath.
-Riassumendo, se volete installare la versione 1.7.9 di Sphinx dovete eseguire::
+Riassumendo, se volete installare la versione 2.4.4 di Sphinx dovete eseguire::
- $ virtualenv sphinx_1.7.9
- $ . sphinx_1.7.9/bin/activate
- (sphinx_1.7.9) $ pip install -r Documentation/sphinx/requirements.txt
+ $ virtualenv sphinx_2.4.4
+ $ . sphinx_2.4.4/bin/activate
+ (sphinx_2.4.4) $ pip install -r Documentation/sphinx/requirements.txt
-Dopo aver eseguito ``. sphinx_1.7.9/bin/activate``, il prompt cambierà per
+Dopo aver eseguito ``. sphinx_2.4.4/bin/activate``, il prompt cambierà per
indicare che state usando il nuovo ambiente. Se aprite un nuova sessione,
prima di generare la documentazione, dovrete rieseguire questo comando per
rientrare nell'ambiente virtuale.
--------------------------
Al momento, la generazione di questi documenti è supportata solo dalle
-versioni di Sphinx superiori alla 1.4.
+versioni di Sphinx superiori alla 2.4.
Per la generazione di PDF e LaTeX, avrete bisogno anche del pacchetto
``XeLaTeX`` nella versione 3.14159265
You should run:
sudo dnf install -y texlive-luatex85
- /usr/bin/virtualenv sphinx_1.7.9
- . sphinx_1.7.9/bin/activate
+ /usr/bin/virtualenv sphinx_2.4.4
+ . sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
iptables 1.4.2 iptables -V
openssl & libcrypto 1.0.0 openssl version
bc 1.06.95 bc --version
-Sphinx\ [#f1]_ 1.3 sphinx-build --version
+Sphinx\ [#f1]_ 1.7 sphinx-build --version
====================== ================= ========================================
.. [#f1] Sphinx è necessario solo per produrre la documentazione del Kernel
unsigned long start_pfn;
unsigned long nr_pages;
int status_change_nid_normal;
- int status_change_nid_high;
int status_change_nid;
}
- status_change_nid_normal是当nodemask的N_NORMAL_MEMORY被设置/清除时设置节
点id，如果是-1，则nodemask状态不改变。
-- status_change_nid_high是当nodemask的N_HIGH_MEMORY被设置/清除时设置的节点
- id，如果这个值为-1，那么nodemask状态不会改变。
-
- status_change_nid是当nodemask的N_MEMORY被（将）设置/清除时设置的节点id。这
意味着一个新的（没上线的）节点通过联机获得新的内存，而一个节点失去了所有的内
存。如果这个值为-1，那么nodemask的状态就不会改变。
安装Sphinx
==========
-Documentation/ 下的ReST文件现在使用sphinx1.3或更高版本构建。
+Documentation/ 下的ReST文件现在使用sphinx1.7或更高版本构建。
这有一个脚本可以检查Sphinx的依赖项。更多详细信息见
:ref:`sphinx-pre-install_zh` 。
.. note::
- #) 低于1.5版本的Sphinx无法与Python的0.13.1或更高版本docutils一起正常工作。
- 如果您想使用这些版本，那么应该运行 ``pip install 'docutils==0.12'`` 。
-
#) html输出建议使用RTD主题。根据Sphinx版本的不同，它应该用
``pip install sphinx_rtd_theme`` 单独安装。
#) 一些ReST页面包含数学表达式。由于Sphinx的工作方式，这些表达式是使用 LaTeX
编写的。它需要安装amsfonts和amsmath宏包，以便显示。
-总之，如您要安装Sphinx 1.7.9版本，应执行::
+总之，如您要安装Sphinx 2.4.4版本，应执行::
- $ virtualenv sphinx_1.7.9
- $ . sphinx_1.7.9/bin/activate
- (sphinx_1.7.9) $ pip install -r Documentation/sphinx/requirements.txt
+ $ virtualenv sphinx_2.4.4
+ $ . sphinx_2.4.4/bin/activate
+ (sphinx_2.4.4) $ pip install -r Documentation/sphinx/requirements.txt
-在运行 ``. sphinx_1.7.9/bin/activate`` 之后，提示符将变化，以指示您正在使用新
+在运行 ``. sphinx_2.4.4/bin/activate`` 之后，提示符将变化，以指示您正在使用新
环境。如果您打开了一个新的shell，那么在构建文档之前，您需要重新运行此命令以再
次进入虚拟环境中。
PDF和LaTeX构建
--------------
-目前只有Sphinx 1.4及更高版本才支持这种构建。
+目前只有Sphinx 2.4及更高版本才支持这种构建。
对于PDF和LaTeX输出，还需要 ``XeLaTeX`` 3.14159265版本。（译注：此版本号真实
存在）
You should run:
sudo dnf install -y texlive-luatex85
- /usr/bin/virtualenv sphinx_1.7.9
- . sphinx_1.7.9/bin/activate
+ /usr/bin/virtualenv sphinx_2.4.4
+ . sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
每个人都认为管理者做决定，而且决策很重要。决定越大越痛苦，管理者就必须越高级。
这很明显，但事实并非如此。
-游æ\88\8fç\9a\84å\90\8då\97是 **避免** 做出决定。尤其是，如果有人告诉你“选择（a）或（b），
+æ\9c\80é\87\8dè¦\81ç\9a\84是 **避免** 做出决定。尤其是，如果有人告诉你“选择（a）或（b），
我们真的需要你来做决定”，你就是陷入麻烦的管理者。你管理的人比你更了解细节，
所以如果他们来找你做技术决策，你完蛋了。你显然没有能力为他们做这个决定。
（推论：如果你管理的人不比你更了解细节，你也会被搞砸，尽管原因完全不同。
也就是说，你的工作是错的，他们应该管理你的才智）
-æ\89\80以游æ\88\8fç\9a\84å\90\8då\97是 **避免** 做出决定，至少是那些大而痛苦的决定。做一些小的
+æ\89\80以æ\9c\80é\87\8dè¦\81ç\9a\84是 **避免** 做出决定，至少是那些大而痛苦的决定。做一些小的
和非结果性的决定是很好的，并且使您看起来好像知道自己在做什么，所以内核管理者
需要做的是将那些大的和痛苦的决定变成那些没有人真正关心的小事情。
When enabled the VMM may make use of the ``KVM_ARM_MTE_COPY_TAGS`` ioctl to
perform a bulk copy of tags to/from the guest.
+7.29 KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM
+-------------------------------------
+
+Architectures: x86 SEV enabled
+Type: vm
+Parameters: args[0] is the fd of the source vm
+Returns: 0 on success
+
+This capability enables userspace to migrate the encryption context from the VM
+indicated by the fd to the VM this is called on.
+
+This is intended to support intra-host migration of VMs between userspace VMMs,
+upgrading the VMM process without interrupting the guest.
+
8. Other capabilities.
======================
1. Identification of the monitoring target address range for the address space.
2. Access check of specific address range in the target space.
-DAMON currently provides the implementation of the primitives for only the
-virtual address spaces. Below two subsections describe how it works.
+DAMON currently provides the implementations of the primitives for the physical
+and virtual address spaces. Below two subsections describe how those work.
VMA-based Target Address Range Construction
-------------------------------------------
+This is only for the virtual address space primitives implementation. That for
+the physical address space simply asks users to manually set the monitoring
+target address ranges.
+
Only small parts in the super-huge virtual address space of the processes are
mapped to the physical memory and accessed. Thus, tracking the unmapped
address regions is just wasteful. However, because DAMON can deal with some
PTE Accessed-bit Based Access Check
-----------------------------------
-The implementation for the virtual address space uses PTE Accessed-bit for
-basic access checks. It finds the relevant PTE Accessed bit from the address
-by walking the page table for the target task of the address. In this way, the
-implementation finds and clears the bit for next sampling target address and
-checks whether the bit set again after one sampling period. This could disturb
-other kernel subsystems using the Accessed bits, namely Idle page tracking and
-the reclaim logic. To avoid such disturbances, DAMON makes it mutually
-exclusive with Idle page tracking and uses ``PG_idle`` and ``PG_young`` page
-flags to solve the conflict with the reclaim logic, as Idle page tracking does.
+Both of the implementations for physical and virtual address spaces use PTE
+Accessed-bit for basic access checks. Only one difference is the way of
+finding the relevant PTE Accessed bit(s) from the address. While the
+implementation for the virtual address walks the page table for the target task
+of the address, the implementation for the physical address walks every page
+table having a mapping to the address. In this way, the implementations find
+and clear the bit(s) for next sampling target address and checks whether the
+bit(s) set again after one sampling period. This could disturb other kernel
+subsystems using the Accessed bits, namely Idle page tracking and the reclaim
+logic. To avoid such disturbances, DAMON makes it mutually exclusive with Idle
+page tracking and uses ``PG_idle`` and ``PG_young`` page flags to solve the
+conflict with the reclaim logic, as Idle page tracking does.
Address Space Independent Core Mechanisms
DAMON core by the users. In this way, DAMON users can monitor any address
space with any access check technique.
-Nonetheless, DAMON provides vma tracking and PTE Accessed bit check based
+Nonetheless, DAMON provides vma/rmap tracking and PTE Accessed bit check based
implementations of the address space dependent functions for the virtual memory
-by default, for a reference and convenient use. In near future, we will
-provide those for physical memory address space.
+and the physical memory by default, for a reference and convenient use.
Can I simply monitor page granularity?
system memory page, locks the page with ``lock_page()``, and fills in the
``dst`` array entry with::
- dst[i] = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED;
+ dst[i] = migrate_pfn(page_to_pfn(dpage));
Now that the driver knows that this page is being migrated, it can
invalidate device private MMU mappings and copy device private memory
=====================================
This is a collection of documents about the Linux memory management (mm)
-subsystem. If you are looking for advice on simply allocating memory,
-see the :ref:`memory_allocation`.
-
-User guides for MM features
-===========================
-
-The following documents provide guides for controlling and tuning
-various features of the Linux memory management
-
-.. toctree::
- :maxdepth: 1
-
- swap_numa
- zswap
-
-Kernel developers MM documentation
-==================================
-
-The below documents describe MM internals with different level of
-details ranging from notes and mailing list responses to elaborate
-descriptions of data structures and algorithms.
+subsystem internals with different level of details ranging from notes and
+mailing list responses for elaborating descriptions of data structures and
+algorithms. If you are looking for advice on simply allocating memory, see the
+:ref:`memory_allocation`. For controlling and tuning guides, see the
+:doc:`admin guide <../admin-guide/mm/index>`.
.. toctree::
:maxdepth: 1
cat /sys/kernel/debug/page_owner > page_owner_full.txt
./page_owner_sort page_owner_full.txt sorted_page_owner.txt
+ The general output of ``page_owner_full.txt`` is as follows:
+
+ Page allocated via order XXX, ...
+ PFN XXX ...
+ // Detailed stack
+
+ Page allocated via order XXX, ...
+ PFN XXX ...
+ // Detailed stack
+
+ The ``page_owner_sort`` tool ignores ``PFN`` rows, puts the remaining rows
+ in buf, uses regexp to extract the page order value, counts the times
+ and pages of buf, and finally sorts them according to the times.
+
See the result about who allocated each page
- in the ``sorted_page_owner.txt``.
+ in the ``sorted_page_owner.txt``. General output:
+
+ XXX times, XXX pages:
+ Page allocated via order XXX, ...
+ // Detailed stack
+
+ By default, ``page_owner_sort`` is sorted according to the times of buf.
+ If you want to sort by the pages nums of buf, use the ``-m`` parameter.
the handler allocates a larger xstate buffer for the task so the large
state can be context switched. In the unlikely cases that the allocation
fails, the kernel sends SIGSEGV.
+
+Dynamic features in signal frames
+---------------------------------
+
+Dynamcally enabled features are not written to the signal frame upon signal
+entry if the feature is in its initial configuration. This differs from
+non-dynamic features which are always written regardless of their
+configuration. Signal handlers can examine the XSAVE buffer's XSTATE_BV
+field to determine if a features was written.
ALLWINNER HARDWARE SPINLOCK SUPPORT
M: Wilken Gottwalt <wilken.gottwalt@posteo.net>
S: Maintained
-F: Documentation/devicetree/bindings/hwlock/allwinner,sun6i-hwspinlock.yaml
+F: Documentation/devicetree/bindings/hwlock/allwinner,sun6i-a31-hwspinlock.yaml
F: drivers/hwspinlock/sun6i_hwspinlock.c
ALLWINNER THERMAL DRIVER
F: drivers/thermal/thermal_mmio.c
AMAZON ETHERNET DRIVERS
-M: Netanel Belgazal <netanel@amazon.com>
+M: Shay Agroskin <shayagr@amazon.com>
M: Arthur Kiyanovski <akiyano@amazon.com>
-R: Guy Tzalik <gtzalik@amazon.com>
+R: David Arinzon <darinzon@amazon.com>
+R: Noam Dagan <ndagan@amazon.com>
R: Saeed Bishara <saeedb@amazon.com>
L: netdev@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/iommu/apple,dart.yaml
F: drivers/iommu/apple-dart.c
+APPLE PCIE CONTROLLER DRIVER
+M: Alyssa Rosenzweig <alyssa@rosenzweig.io>
+M: Marc Zyngier <maz@kernel.org>
+L: linux-pci@vger.kernel.org
+S: Maintained
+F: drivers/pci/controller/pcie-apple.c
+
APPLE SMC DRIVER
M: Henrik Rydberg <rydberg@bitmath.org>
L: linux-hwmon@vger.kernel.org
F: arch/arm/mach-mstar/
F: drivers/clk/mstar/
F: drivers/gpio/gpio-msc313.c
+F: drivers/rtc/rtc-msc313.c
F: drivers/watchdog/msc313e_wdt.c
F: include/dt-bindings/clock/mstar-*
F: include/dt-bindings/gpio/msc313-gpio.h
F: Documentation/devicetree/bindings/net/toshiba,visconti-dwmac.yaml
F: Documentation/devicetree/bindings/gpio/toshiba,gpio-visconti.yaml
F: Documentation/devicetree/bindings/pci/toshiba,visconti-pcie.yaml
-F: Documentation/devicetree/bindings/pinctrl/toshiba,tmpv7700-pinctrl.yaml
+F: Documentation/devicetree/bindings/pinctrl/toshiba,visconti-pinctrl.yaml
F: Documentation/devicetree/bindings/watchdog/toshiba,visconti-wdt.yaml
F: arch/arm64/boot/dts/toshiba/
F: drivers/net/ethernet/stmicro/stmmac/dwmac-visconti.c
AUXILIARY DISPLAY DRIVERS
M: Miguel Ojeda <ojeda@kernel.org>
S: Maintained
+F: Documentation/devicetree/bindings/auxdisplay/
F: drivers/auxdisplay/
F: include/linux/cfag12864b.h
BROADCOM BNX2X 10 GIGABIT ETHERNET DRIVER
M: Ariel Elior <aelior@marvell.com>
M: Sudarsana Kalluru <skalluru@marvell.com>
-M: GR-everest-linux-l2@marvell.com
+M: Manish Chopra <manishc@marvell.com>
L: netdev@vger.kernel.org
S: Supported
F: drivers/net/ethernet/broadcom/bnx2x/
M: Hans de Goede <hdegoede@redhat.com>
L: linux-input@vger.kernel.org
S: Maintained
-F: Documentation/devicetree/bindings/input/touchscreen/chipone_icn8318.txt
+F: Documentation/devicetree/bindings/input/touchscreen/chipone,icn8318.yaml
F: drivers/input/touchscreen/chipone_icn8318.c
CHIPONE ICN8505 I2C TOUCHSCREEN DRIVER
CHROME HARDWARE PLATFORM SUPPORT
M: Benson Leung <bleung@chromium.org>
-M: Enric Balletbo i Serra <enric.balletbo@collabora.com>
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/chrome-platform/linux.git
F: drivers/platform/chrome/
CHROMEOS EC CODEC DRIVER
M: Cheng-Yi Chiang <cychiang@chromium.org>
-R: Enric Balletbo i Serra <enric.balletbo@collabora.com>
R: Guenter Roeck <groeck@chromium.org>
S: Maintained
F: Documentation/devicetree/bindings/sound/google,cros-ec-codec.yaml
CHROMEOS EC SUBDRIVERS
M: Benson Leung <bleung@chromium.org>
-M: Enric Balletbo i Serra <enric.balletbo@collabora.com>
R: Guenter Roeck <groeck@chromium.org>
S: Maintained
F: drivers/power/supply/cros_usbpd-charger.c
N: cros_ec
N: cros-ec
+CHROMEOS EC USB TYPE-C DRIVER
+M: Prashant Malani <pmalani@chromium.org>
+S: Maintained
+F: drivers/platform/chrome/cros_ec_typec.c
+
+CHROMEOS EC USB PD NOTIFY DRIVER
+M: Prashant Malani <pmalani@chromium.org>
+S: Maintained
+F: drivers/platform/chrome/cros_usbpd_notify.c
+F: include/linux/platform_data/cros_usbpd_notify.h
+
CHRONTEL CH7322 CEC DRIVER
M: Joe Tessler <jrt@google.com>
L: linux-media@vger.kernel.org
M: Julia Lawall <Julia.Lawall@inria.fr>
M: Gilles Muller <Gilles.Muller@inria.fr>
M: Nicolas Palix <nicolas.palix@imag.fr>
-M: Michal Marek <michal.lkml@markovi.net>
-L: cocci@systeme.lip6.fr (moderated for non-subscribers)
+L: cocci@inria.fr (moderated for non-subscribers)
S: Supported
-W: http://coccinelle.lip6.fr/
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild.git misc
+W: https://coccinelle.gitlabpages.inria.fr/website/
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/jlawall/linux.git
F: Documentation/dev-tools/coccinelle.rst
F: scripts/coccicheck
F: scripts/coccinelle/
S: Maintained
F: drivers/input/touchscreen/cy8ctma140.c
+CYPRESS STREETFIGHTER TOUCHKEYS DRIVER
+M: Yassine Oudjana <y.oudjana@protonmail.com>
+L: linux-input@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/input/cypress-sf.yaml
+F: drivers/input/keyboard/cypress-sf.c
+
CYTTSP TOUCHSCREEN DRIVER
M: Linus Walleij <linus.walleij@linaro.org>
L: linux-input@vger.kernel.org
F: net/ax25/sysctl_net_ax25.c
DATA ACCESS MONITOR
-M: SeongJae Park <sjpark@amazon.de>
+M: SeongJae Park <sj@kernel.org>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/admin-guide/mm/damon/
F: include/uapi/linux/mii.h
F: net/core/of_net.c
+EXEC & BINFMT API
+R: Eric Biederman <ebiederm@xmission.com>
+R: Kees Cook <keescook@chromium.org>
+F: arch/alpha/kernel/binfmt_loader.c
+F: arch/x86/ia32/ia32_aout.c
+F: fs/*binfmt_*.c
+F: fs/exec.c
+F: include/linux/binfmts.h
+F: include/linux/elf.h
+F: include/uapi/linux/binfmts.h
+F: tools/testing/selftests/exec/
+N: asm/elf.h
+N: binfmt
+
EXFAT FILE SYSTEM
M: Namjae Jeon <linkinjeon@kernel.org>
M: Sungjong Seo <sj1557.seo@samsung.com>
GOODIX TOUCHSCREEN
M: Bastien Nocera <hadess@hadess.net>
+M: Hans de Goede <hdegoede@redhat.com>
L: linux-input@vger.kernel.org
S: Maintained
-F: drivers/input/touchscreen/goodix.c
+F: drivers/input/touchscreen/goodix*
GOOGLE ETHERNET DRIVERS
M: Jeroen de Borst <jeroendb@google.com>
L: linux-kernel@vger.kernel.org
S: Maintained
F: drivers/misc/hisi_hikey_usb.c
-F: Documentation/devicetree/bindings/misc/hisilicon-hikey-usb.yaml
HISILICON PMU DRIVER
M: Shaokun Zhang <zhangshaokun@hisilicon.com>
Q: http://patchwork.ozlabs.org/project/linux-mtd/list/
C: irc://irc.oftc.net/mtd
T: git git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git cfi/next
-F: Documentation/devicetree/bindings/mtd/cypress,hyperflash.txt
-F: Documentation/devicetree/bindings/mtd/ti,am654-hbmc.txt
+F: Documentation/devicetree/bindings/mtd/ti,am654-hbmc.yaml
F: drivers/mtd/hyperbus/
F: include/linux/mtd/hyperbus.h
M: Anitha Chrisanthus <anitha.chrisanthus@intel.com>
M: Edmund Dea <edmund.j.dea@intel.com>
S: Maintained
-F: Documentation/devicetree/bindings/display/intel,kmb_display.yaml
+F: Documentation/devicetree/bindings/display/intel,keembay-display.yaml
F: drivers/gpu/drm/kmb/
INTEL KEEM BAY OCS AES/SM4 CRYPTO DRIVER
F: arch/riscv/kvm/
KERNEL VIRTUAL MACHINE for s390 (KVM/s390)
-M: Christian Borntraeger <borntraeger@de.ibm.com>
+M: Christian Borntraeger <borntraeger@linux.ibm.com>
M: Janosch Frank <frankja@linux.ibm.com>
R: David Hildenbrand <david@redhat.com>
R: Claudio Imbrenda <imbrenda@linux.ibm.com>
F: include/linux/ata.h
F: include/linux/libata.h
-LIBLOCKDEP
-M: Sasha Levin <alexander.levin@microsoft.com>
-S: Maintained
-F: tools/lib/lockdep/
-
LIBNVDIMM BLK: MMIO-APERTURE DRIVER
M: Dan Williams <dan.j.williams@intel.com>
M: Vishal Verma <vishal.l.verma@intel.com>
F: Documentation/devicetree/bindings/i2c/i2c-mt7621.txt
F: drivers/i2c/busses/i2c-mt7621.c
+MEDIATEK MT7621 PCIE CONTROLLER DRIVER
+M: Sergio Paracuellos <sergio.paracuellos@gmail.com>
+S: Maintained
+F: Documentation/devicetree/bindings/pci/mediatek,mt7621-pcie.yaml
+F: drivers/pci/controller/pcie-mt7621.c
+
MEDIATEK MT7621 PHY PCI DRIVER
M: Sergio Paracuellos <sergio.paracuellos@gmail.com>
S: Maintained
M: Deep Shah <sdeep@vmware.com>
M: "VMware, Inc." <pv-drivers@vmware.com>
L: virtualization@lists.linux-foundation.org
+L: x86@kernel.org
S: Supported
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/core
F: Documentation/virt/paravirt_ops.rst
F: arch/*/include/asm/paravirt*.h
F: arch/*/kernel/paravirt*
R: Krzysztof Wilczyński <kw@linux.com>
L: linux-pci@vger.kernel.org
S: Supported
+Q: https://patchwork.kernel.org/project/linux-pci/list/
+B: https://bugzilla.kernel.org
+C: irc://irc.oftc.net/linux-pci
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git
F: Documentation/PCI/endpoint/*
F: Documentation/misc-devices/pci-endpoint-test.rst
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/kishon/pci-endpoint.git
F: drivers/misc/pci_endpoint_test.c
F: drivers/pci/endpoint/
F: tools/pci/
R: Krzysztof Wilczyński <kw@linux.com>
L: linux-pci@vger.kernel.org
S: Supported
-Q: http://patchwork.ozlabs.org/project/linux-pci/list/
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git/
+Q: https://patchwork.kernel.org/project/linux-pci/list/
+B: https://bugzilla.kernel.org
+C: irc://irc.oftc.net/linux-pci
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/lpieralisi/pci.git
F: drivers/pci/controller/
+F: drivers/pci/pci-bridge-emul.c
+F: drivers/pci/pci-bridge-emul.h
PCI SUBSYSTEM
M: Bjorn Helgaas <bhelgaas@google.com>
L: linux-pci@vger.kernel.org
S: Supported
-Q: http://patchwork.ozlabs.org/project/linux-pci/list/
+Q: https://patchwork.kernel.org/project/linux-pci/list/
+B: https://bugzilla.kernel.org
+C: irc://irc.oftc.net/linux-pci
T: git git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git
F: Documentation/PCI/
F: Documentation/devicetree/bindings/pci/
L: linux-pci@vger.kernel.org
L: linux-arm-msm@vger.kernel.org
S: Maintained
-F: drivers/pci/controller/dwc/*qcom*
+F: drivers/pci/controller/dwc/pcie-qcom.c
+
+PCIE ENDPOINT DRIVER FOR QUALCOMM
+M: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
+L: linux-pci@vger.kernel.org
+L: linux-arm-msm@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/pci/qcom,pcie-ep.yaml
+F: drivers/pci/controller/dwc/pcie-qcom-ep.c
PCIE DRIVER FOR ROCKCHIP
M: Shawn Lin <shawn.lin@rock-chips.com>
QLOGIC QL4xxx ETHERNET DRIVER
M: Ariel Elior <aelior@marvell.com>
-M: GR-everest-linux-l2@marvell.com
+M: Manish Chopra <manishc@marvell.com>
L: netdev@vger.kernel.org
S: Supported
F: drivers/net/ethernet/qlogic/qed/
F: Documentation/devicetree/bindings/regulator/vqmmc-ipq4019-regulator.yaml
F: drivers/regulator/vqmmc-ipq4019-regulator.c
+QUALCOMM NAND CONTROLLER DRIVER
+M: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
+L: linux-mtd@lists.infradead.org
+L: linux-arm-msm@vger.kernel.org
+S: Maintained
+F: Documentation/devicetree/bindings/mtd/qcom,nandc.yaml
+F: drivers/mtd/nand/raw/qcom_nandc.c
+
QUALCOMM RMNET DRIVER
M: Subash Abhinov Kasiviswanathan <subashab@codeaurora.org>
M: Sean Tranchetti <stranche@codeaurora.org>
M: Mathieu Poirier <mathieu.poirier@linaro.org>
L: linux-remoteproc@vger.kernel.org
S: Maintained
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/andersson/remoteproc.git rproc-next
+T: git https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git rproc-next
F: Documentation/ABI/testing/sysfs-class-remoteproc
F: Documentation/devicetree/bindings/remoteproc/
F: Documentation/staging/remoteproc.rst
M: Mathieu Poirier <mathieu.poirier@linaro.org>
L: linux-remoteproc@vger.kernel.org
S: Maintained
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/andersson/remoteproc.git rpmsg-next
+T: git https://git.kernel.org/pub/scm/linux/kernel/git/remoteproc/linux.git rpmsg-next
F: Documentation/ABI/testing/sysfs-bus-rpmsg
F: Documentation/staging/rpmsg.rst
F: drivers/rpmsg/
S390
M: Heiko Carstens <hca@linux.ibm.com>
M: Vasily Gorbik <gor@linux.ibm.com>
-M: Christian Borntraeger <borntraeger@de.ibm.com>
+M: Christian Borntraeger <borntraeger@linux.ibm.com>
R: Alexander Gordeev <agordeev@linux.ibm.com>
L: linux-s390@vger.kernel.org
S: Supported
L: linux-samsung-soc@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/clock/samsung,s2mps11.yaml
-F: Documentation/devicetree/bindings/mfd/samsung,sec-core.txt
+F: Documentation/devicetree/bindings/mfd/samsung,s2m*.yaml
+F: Documentation/devicetree/bindings/mfd/samsung,s5m*.yaml
F: Documentation/devicetree/bindings/regulator/samsung,s2m*.yaml
F: Documentation/devicetree/bindings/regulator/samsung,s5m*.yaml
F: drivers/clk/clk-s2mps11.c
Q: http://patchwork.ozlabs.org/project/linux-mtd/list/
C: irc://irc.oftc.net/mtd
T: git git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git spi-nor/next
+F: Documentation/devicetree/bindings/mtd/jedec,spi-nor.yaml
F: drivers/mtd/spi-nor/
F: include/linux/mtd/spi-nor.h
F: sound/virtio/*
VIRTIO I2C DRIVER
-M: Jie Deng <jie.deng@intel.com>
+M: Conghui Chen <conghui.chen@intel.com>
M: Viresh Kumar <viresh.kumar@linaro.org>
L: linux-i2c@vger.kernel.org
L: virtualization@lists.linux-foundation.org
F: arch/x86/kernel/cpu/vmware.c
VMWARE PVRDMA DRIVER
-M: Adit Ranadive <aditr@vmware.com>
+M: Bryan Tan <bryantan@vmware.com>
+M: Vishnu Dasa <vdasa@vmware.com>
M: VMware PV-Drivers <pv-drivers@vmware.com>
L: linux-rdma@vger.kernel.org
S: Maintained
F: include/linux/zsmalloc.h
F: mm/zsmalloc.c
+ZSTD
+M: Nick Terrell <terrelln@fb.com>
+S: Maintained
+B: https://github.com/facebook/zstd/issues
+T: git git://github.com/terrelln/linux.git
+F: include/linux/zstd*
+F: lib/zstd/
+F: lib/decompress_unzstd.c
+F: crypto/zstd.c
+N: zstd
+K: zstd
+
ZSWAP COMPRESSED SWAP CACHING
M: Seth Jennings <sjenning@redhat.com>
M: Dan Streetman <ddstreet@ieee.org>
# SPDX-License-Identifier: GPL-2.0
VERSION = 5
-PATCHLEVEL = 15
+PATCHLEVEL = 16
SUBLEVEL = 0
-EXTRAVERSION =
+EXTRAVERSION = -rc2
NAME = Trick or Treat
# *DOCUMENTATION*
KBUILD_CFLAGS += $(stackp-flags-y)
KBUILD_CFLAGS-$(CONFIG_WERROR) += -Werror
-KBUILD_CFLAGS += $(KBUILD_CFLAGS-y)
+KBUILD_CFLAGS += $(KBUILD_CFLAGS-y) $(CONFIG_CC_IMPLICIT_FALLTHROUGH)
ifdef CONFIG_CC_IS_CLANG
KBUILD_CPPFLAGS += -Qunused-arguments
KBUILD_CFLAGS += -mno-global-merge
else
-# Warn about unmarked fall-throughs in switch statement.
-# Disabled for clang while comment to attribute conversion happens and
-# https://github.com/ClangBuiltLinux/linux/issues/636 is discussed.
-KBUILD_CFLAGS += $(call cc-option,-Wimplicit-fallthrough=5,)
# gcc inanely warns about local variables called 'main'
KBUILD_CFLAGS += -Wno-main
endif
KBUILD_CFLAGS += -fzero-call-used-regs=used-gpr
endif
-DEBUG_CFLAGS :=
-
-ifdef CONFIG_DEBUG_INFO
-
-ifdef CONFIG_DEBUG_INFO_SPLIT
-DEBUG_CFLAGS += -gsplit-dwarf
-else
-DEBUG_CFLAGS += -g
-endif
-
-ifndef CONFIG_AS_IS_LLVM
-KBUILD_AFLAGS += -Wa,-gdwarf-2
-endif
-
-ifndef CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
-dwarf-version-$(CONFIG_DEBUG_INFO_DWARF4) := 4
-dwarf-version-$(CONFIG_DEBUG_INFO_DWARF5) := 5
-DEBUG_CFLAGS += -gdwarf-$(dwarf-version-y)
-endif
-
-ifdef CONFIG_DEBUG_INFO_REDUCED
-DEBUG_CFLAGS += -fno-var-tracking
-ifdef CONFIG_CC_IS_GCC
-DEBUG_CFLAGS += -femit-struct-debug-baseonly
-endif
-endif
-
-ifdef CONFIG_DEBUG_INFO_COMPRESSED
-DEBUG_CFLAGS += -gz=zlib
-KBUILD_AFLAGS += -gz=zlib
-KBUILD_LDFLAGS += --compress-debug-sections=zlib
-endif
-
-endif # CONFIG_DEBUG_INFO
-
-KBUILD_CFLAGS += $(DEBUG_CFLAGS)
-export DEBUG_CFLAGS
-
ifdef CONFIG_FUNCTION_TRACER
ifdef CONFIG_FTRACE_MCOUNT_USE_CC
CC_FLAGS_FTRACE += -mrecord-mcount
endif
# arch Makefile may override CC so keep this after arch Makefile is included
-NOSTDINC_FLAGS += -nostdinc -isystem $(shell $(CC) -print-file-name=include)
+NOSTDINC_FLAGS += -nostdinc
# warn about C99 declaration after statement
KBUILD_CFLAGS += -Wdeclaration-after-statement
KBUILD_CFLAGS += -Wno-maybe-uninitialized
endif
+ifdef CONFIG_CC_IS_GCC
+# The allocators already balk at large sizes, so silence the compiler
+# warnings for bounds checks involving those possible values. While
+# -Wno-alloc-size-larger-than would normally be used here, earlier versions
+# of gcc (<9.1) weirdly don't handle the option correctly when _other_
+# warnings are produced (?!). Using -Walloc-size-larger-than=SIZE_MAX
+# doesn't work (as it is documented to), silently resolving to "0" prior to
+# version 9.1 (and producing an error more recently). Numeric values larger
+# than PTRDIFF_MAX also don't work prior to version 9.1, which are silently
+# ignored, continuing to default to PTRDIFF_MAX. So, left with no other
+# choice, we must perform a versioned check to disable this warning.
+# https://lore.kernel.org/lkml/20210824115859.187f272f@canb.auug.org.au
+KBUILD_CFLAGS += $(call cc-ifversion, -ge, 0901, -Wno-alloc-size-larger-than)
+endif
+
# disable invalid "can't wrap" optimizations for signed / pointers
KBUILD_CFLAGS += -fno-strict-overflow
# include additional Makefiles when needed
include-y := scripts/Makefile.extrawarn
+include-$(CONFIG_DEBUG_INFO) += scripts/Makefile.debug
include-$(CONFIG_KASAN) += scripts/Makefile.kasan
include-$(CONFIG_KCSAN) += scripts/Makefile.kcsan
include-$(CONFIG_UBSAN) += scripts/Makefile.ubsan
# SPDX-License-Identifier: GPL-2.0-only
obj-y += kernel/ mm/
obj-$(CONFIG_MATHEMU) += math-emu/
+
+# for cleaning
+subdir- += boot
bootimage bootpfile bootpzfile: vmlinux
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
archheaders:
$(Q)$(MAKE) $(build)=arch/alpha/kernel/syscalls all
unsigned long size;
size = initrd_end - initrd_start;
- memblock_free(__pa(initrd_start), PAGE_ALIGN(size));
+ memblock_free((void *)initrd_start, PAGE_ALIGN(size));
if (!move_initrd(pci_mem))
printk("irongate_init_arch: initrd too big "
"(%ldK)\ndisabling initrd\n",
extern char _stext[], _etext[];
unsigned long tmp = *sp;
sp++;
- if (tmp < (unsigned long) &_stext)
- continue;
- if (tmp >= (unsigned long) &_etext)
+ if (!is_kernel_text(tmp))
continue;
printk("%s[<%lx>] %pSR\n", loglvl, tmp, (void *)tmp);
if (i > 40) {
# SPDX-License-Identifier: GPL-2.0
obj-y += kernel/
obj-y += mm/
+
+# for cleaning
+subdir- += boot
@$(kecho) ' Image $(boot)/uImage is ready'
CLEAN_FILES += $(boot)/uImage
-
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
eflags = x->e_flags;
if ((eflags & EF_ARC_OSABI_MSK) != EF_ARC_OSABI_CURRENT) {
pr_err("ABI mismatch - you need newer toolchain\n");
- force_sigsegv(SIGSEGV);
+ force_fatal_sig(SIGSEGV);
return 0;
}
low_mem_sz = size;
in_use = 1;
- memblock_add_node(base, size, 0);
+ memblock_add_node(base, size, 0, MEMBLOCK_NONE);
} else {
#ifdef CONFIG_HIGHMEM
high_mem_start = base;
high_mem_sz = size;
in_use = 1;
- memblock_add_node(base, size, 1);
+ memblock_add_node(base, size, 1, MEMBLOCK_NONE);
memblock_reserve(base, size);
#endif
}
#ifdef CONFIG_HIGHMEM
unsigned long tmp;
- memblock_free(high_mem_start, high_mem_sz);
+ memblock_phys_free(high_mem_start, high_mem_sz);
for (tmp = min_high_pfn; tmp < max_high_pfn; tmp++)
free_highmem_page(pfn_to_page(tmp));
#endif
obj-y += probes/
obj-y += net/
obj-y += crypto/
+
+# for cleaning
+subdir- += boot
bool "High Memory Support"
depends on MMU
select KMAP_LOCAL
+ select KMAP_LOCAL_NON_LINEAR_PTE_ARRAY
help
The address space of ARM processors is only 4 Gigabytes large
and it has to accommodate user address space, kernel address
# Note that GCC does not numerically define an architecture version
# macro, but instead defines a whole series of macros which makes
# testing for a specific architecture or later rather impossible.
-arch-$(CONFIG_CPU_32v7M) =-D__LINUX_ARM_ARCH__=7 -march=armv7-m -Wa,-march=armv7-m
-arch-$(CONFIG_CPU_32v7) =-D__LINUX_ARM_ARCH__=7 $(call cc-option,-march=armv7-a,-march=armv5t -Wa$(comma)-march=armv7-a)
-arch-$(CONFIG_CPU_32v6) =-D__LINUX_ARM_ARCH__=6 $(call cc-option,-march=armv6,-march=armv5t -Wa$(comma)-march=armv6)
+arch-$(CONFIG_CPU_32v7M) =-D__LINUX_ARM_ARCH__=7 -march=armv7-m
+arch-$(CONFIG_CPU_32v7) =-D__LINUX_ARM_ARCH__=7 -march=armv7-a
+arch-$(CONFIG_CPU_32v6) =-D__LINUX_ARM_ARCH__=6 -march=armv6
# Only override the compiler option if ARMv6. The ARMv6K extensions are
# always available in ARMv7
ifeq ($(CONFIG_CPU_32v6),y)
-arch-$(CONFIG_CPU_32v6K) =-D__LINUX_ARM_ARCH__=6 $(call cc-option,-march=armv6k,-march=armv5t -Wa$(comma)-march=armv6k)
+arch-$(CONFIG_CPU_32v6K) =-D__LINUX_ARM_ARCH__=6 -march=armv6k
endif
-arch-$(CONFIG_CPU_32v5) =-D__LINUX_ARM_ARCH__=5 $(call cc-option,-march=armv5te,-march=armv4t)
+arch-$(CONFIG_CPU_32v5) =-D__LINUX_ARM_ARCH__=5 -march=armv5te
arch-$(CONFIG_CPU_32v4T) =-D__LINUX_ARM_ARCH__=4 -march=armv4t
arch-$(CONFIG_CPU_32v4) =-D__LINUX_ARM_ARCH__=4 -march=armv4
arch-$(CONFIG_CPU_32v3) =-D__LINUX_ARM_ARCH__=3 -march=armv3m
tune-$(CONFIG_CPU_ARM740T) =-mtune=arm7tdmi
tune-$(CONFIG_CPU_ARM9TDMI) =-mtune=arm9tdmi
tune-$(CONFIG_CPU_ARM940T) =-mtune=arm9tdmi
-tune-$(CONFIG_CPU_ARM946E) =$(call cc-option,-mtune=arm9e,-mtune=arm9tdmi)
+tune-$(CONFIG_CPU_ARM946E) =-mtune=arm9e
tune-$(CONFIG_CPU_ARM920T) =-mtune=arm9tdmi
tune-$(CONFIG_CPU_ARM922T) =-mtune=arm9tdmi
tune-$(CONFIG_CPU_ARM925T) =-mtune=arm9tdmi
tune-$(CONFIG_CPU_FA526) =-mtune=arm9tdmi
tune-$(CONFIG_CPU_SA110) =-mtune=strongarm110
tune-$(CONFIG_CPU_SA1100) =-mtune=strongarm1100
-tune-$(CONFIG_CPU_XSCALE) =$(call cc-option,-mtune=xscale,-mtune=strongarm110) -Wa,-mcpu=xscale
-tune-$(CONFIG_CPU_XSC3) =$(call cc-option,-mtune=xscale,-mtune=strongarm110) -Wa,-mcpu=xscale
-tune-$(CONFIG_CPU_FEROCEON) =$(call cc-option,-mtune=marvell-f,-mtune=xscale)
-tune-$(CONFIG_CPU_V6) =$(call cc-option,-mtune=arm1136j-s,-mtune=strongarm)
-tune-$(CONFIG_CPU_V6K) =$(call cc-option,-mtune=arm1136j-s,-mtune=strongarm)
+tune-$(CONFIG_CPU_XSCALE) =-mtune=xscale
+tune-$(CONFIG_CPU_XSC3) =-mtune=xscale
+tune-$(CONFIG_CPU_FEROCEON) =-mtune=xscale
+tune-$(CONFIG_CPU_V6) =-mtune=arm1136j-s
+tune-$(CONFIG_CPU_V6K) =-mtune=arm1136j-s
# Evaluate tune cc-option calls now
tune-y := $(tune-y)
$(Q)$(MAKE) $(build)=arch/arm/vdso $@
endif
-# We use MRPROPER_FILES and CLEAN_FILES now
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
# My testing targets (bypasses dependencies)
bp:; $(Q)$(MAKE) $(build)=$(boot) MACHINE=$(MACHINE) $(boot)/bootpImage
memcpy(args, ®s->ARM_r0 + 1, 5 * sizeof(args[0]));
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->ARM_ORIG_r0 = args[0];
- args++;
-
- memcpy(®s->ARM_r0 + 1, args, 5 * sizeof(args[0]));
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
/* ARM tasks don't change audit architectures on the fly. */
err_sysctrl:
iounmap(relocation);
err_reloc:
- memblock_free(hip04_boot_method[0], hip04_boot_method[1]);
+ memblock_phys_free(hip04_boot_method[0], hip04_boot_method[1]);
err:
return ret;
}
panic("Failed to steal %pa bytes at %pS\n",
&size, (void *)_RET_IP_);
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
return phys;
static __init void *kasan_alloc_block(size_t size)
{
return memblock_alloc_try_nid(size, size, __pa(MAX_DMA_ADDRESS),
- MEMBLOCK_ALLOC_KASAN, NUMA_NO_NODE);
+ MEMBLOCK_ALLOC_NOLEAKTRACE, NUMA_NO_NODE);
}
static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr,
BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) < FIXADDR_START);
BUG_ON(idx >= __end_of_fixed_addresses);
- /* we only support device mappings until pgprot_kernel has been set */
+ /* We support only device mappings before pgprot_kernel is set. */
if (WARN_ON(pgprot_val(prot) != pgprot_val(FIXMAP_PAGE_IO) &&
- pgprot_val(pgprot_kernel) == 0))
+ pgprot_val(prot) && pgprot_val(pgprot_kernel) == 0))
return;
if (pgprot_val(prot))
EXPORT_SYMBOL_GPL(HYPERVISOR_memory_op);
EXPORT_SYMBOL_GPL(HYPERVISOR_physdev_op);
EXPORT_SYMBOL_GPL(HYPERVISOR_vcpu_op);
-EXPORT_SYMBOL_GPL(HYPERVISOR_tmem_op);
EXPORT_SYMBOL_GPL(HYPERVISOR_platform_op_raw);
EXPORT_SYMBOL_GPL(HYPERVISOR_multicall);
EXPORT_SYMBOL_GPL(HYPERVISOR_vm_assist);
HYPERCALL2(memory_op);
HYPERCALL2(physdev_op);
HYPERCALL3(vcpu_op);
-HYPERCALL1(tmem_op);
HYPERCALL1(platform_op_raw);
HYPERCALL2(multicall);
HYPERCALL2(vm_assist);
obj-$(CONFIG_XEN) += xen/
obj-$(subst m,y,$(CONFIG_HYPERV)) += hyperv/
obj-$(CONFIG_CRYPTO) += crypto/
+
+# for cleaning
+subdir- += boot
def_bool y
depends on NUMA
+config NEED_PER_CPU_PAGE_FIRST_CHUNK
+ def_bool y
+ depends on NUMA
+
source "kernel/Kconfig.hz"
config ARCH_SPARSEMEM_ENABLE
endif
endif
-
-# We use MRPROPER_FILES and CLEAN_FILES now
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
- $(Q)$(MAKE) $(clean)=arch/arm64/kernel/vdso
- $(Q)$(MAKE) $(clean)=arch/arm64/kernel/vdso32
-
ifeq ($(KBUILD_EXTMOD),)
# We need to generate vdso-offsets.h before compiling certain files in kernel/.
# In order to do that, we should use the archprepare target, but we can't since
#define ESR_ELx_EC_MAX (0x3F)
#define ESR_ELx_EC_SHIFT (26)
+#define ESR_ELx_EC_WIDTH (6)
#define ESR_ELx_EC_MASK (UL(0x3F) << ESR_ELx_EC_SHIFT)
#define ESR_ELx_EC(esr) (((esr) & ESR_ELx_EC_MASK) >> ESR_ELx_EC_SHIFT)
u64 exits;
};
-int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init);
+void kvm_vcpu_preferred_target(struct kvm_vcpu_init *init);
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu);
int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);
int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
* page table entry, taking care of 52-bit addresses.
*/
#ifdef CONFIG_ARM64_PA_BITS_52
-#define __pte_to_phys(pte) \
- ((pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << 36))
-#define __phys_to_pte_val(phys) (((phys) | ((phys) >> 36)) & PTE_ADDR_MASK)
+static inline phys_addr_t __pte_to_phys(pte_t pte)
+{
+ return (pte_val(pte) & PTE_ADDR_LOW) |
+ ((pte_val(pte) & PTE_ADDR_HIGH) << 36);
+}
+static inline pteval_t __phys_to_pte_val(phys_addr_t phys)
+{
+ return (phys | (phys >> 36)) & PTE_ADDR_MASK;
+}
#else
#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK)
#define __phys_to_pte_val(phys) (phys)
memcpy(args, ®s->regs[1], 5 * sizeof(args[0]));
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->orig_x0 = args[0];
- args++;
-
- memcpy(®s->regs[1], args, 5 * sizeof(args[0]));
-}
-
/*
* We don't care about endianness (__AUDIT_ARCH_LE bit) here because
* AArch64 has the same system calls both on little- and big- endian.
ifeq ($(CONFIG_DEBUG_EFI),y)
AFLAGS_head.o += -DVMLINUX_PATH="\"$(realpath $(objtree)/vmlinux)\""
endif
+
+# for cleaning
+subdir- += vdso vdso32
ARM64_FTR_END,
};
-#define ARM64_FTR_REG_OVERRIDE(id, table, ovr) { \
+#define __ARM64_FTR_REG_OVERRIDE(id_str, id, table, ovr) { \
.sys_id = id, \
.reg = &(struct arm64_ftr_reg){ \
- .name = #id, \
+ .name = id_str, \
.override = (ovr), \
.ftr_bits = &((table)[0]), \
}}
-#define ARM64_FTR_REG(id, table) ARM64_FTR_REG_OVERRIDE(id, table, &no_override)
+#define ARM64_FTR_REG_OVERRIDE(id, table, ovr) \
+ __ARM64_FTR_REG_OVERRIDE(#id, id, table, ovr)
+
+#define ARM64_FTR_REG(id, table) \
+ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override)
struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override;
struct arm64_ftr_override __ro_after_init id_aa64pfr1_override;
return false;
}
+EXPORT_SYMBOL_GPL(this_cpu_has_cap);
/*
* This helper function is used in a narrow window when,
# potential future proofing if we end up with internal calls to the exported
# routines, as x86 does (see 6f121e548f83 ("x86, vdso: Reimplement vdso.so
# preparation in build-time C")).
-ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \
+ldflags-y := -shared -soname=linux-vdso.so.1 --hash-style=sysv \
-Bsymbolic --build-id=sha1 -n $(btildflags-y) -T
ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18
# From arm vDSO Makefile
VDSO_LDFLAGS += -Bsymbolic --no-undefined -soname=linux-vdso.so.1
VDSO_LDFLAGS += -z max-page-size=4096 -z common-page-size=4096
-VDSO_LDFLAGS += -nostdlib -shared --hash-style=sysv --build-id=sha1
+VDSO_LDFLAGS += -shared --hash-style=sysv --build-id=sha1
# Borrow vdsomunge.c from the arm vDSO
r = 1;
break;
case KVM_CAP_NR_VCPUS:
- r = num_online_cpus();
+ /*
+ * ARM64 treats KVM_CAP_NR_CPUS differently from all other
+ * architectures, as it does not always bound it to
+ * KVM_CAP_MAX_VCPUS. It should not matter much because
+ * this is just an advisory value.
+ */
+ r = min_t(unsigned int, num_online_cpus(),
+ kvm_arm_default_max_vcpus());
break;
case KVM_CAP_MAX_VCPUS:
case KVM_CAP_MAX_VCPU_ID:
return kvm_vm_ioctl_set_device_addr(kvm, &dev_addr);
}
case KVM_ARM_PREFERRED_TARGET: {
- int err;
struct kvm_vcpu_init init;
- err = kvm_vcpu_preferred_target(&init);
- if (err)
- return err;
+ kvm_vcpu_preferred_target(&init);
if (copy_to_user(argp, &init, sizeof(init)))
return -EFAULT;
return KVM_ARM_TARGET_GENERIC_V8;
}
-int kvm_vcpu_preferred_target(struct kvm_vcpu_init *init)
+void kvm_vcpu_preferred_target(struct kvm_vcpu_init *init)
{
u32 target = kvm_target_cpu();
- if (target < 0)
- return -ENODEV;
-
memset(init, 0, sizeof(*init));
/*
* target type.
*/
init->target = (__u32)target;
-
- return 0;
}
int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
el1_sync: // Guest trapped into EL2
mrs x0, esr_el2
- lsr x0, x0, #ESR_ELx_EC_SHIFT
+ ubfx x0, x0, #ESR_ELx_EC_SHIFT, #ESR_ELx_EC_WIDTH
cmp x0, #ESR_ELx_EC_HVC64
ccmp x0, #ESR_ELx_EC_HVC32, #4, ne
b.ne el1_trap
.L__vect_start\@:
stp x0, x1, [sp, #-16]!
mrs x0, esr_el2
- lsr x0, x0, #ESR_ELx_EC_SHIFT
+ ubfx x0, x0, #ESR_ELx_EC_SHIFT, #ESR_ELx_EC_WIDTH
cmp x0, #ESR_ELx_EC_HVC64
b.eq __host_hvc
b __host_exit
phys = kvm_pte_to_phys(pte);
if (!addr_is_memory(phys))
- return 0;
+ return -EINVAL;
/*
* Adjust the host stage-2 mappings to match the ownership attributes
.cb = finalize_host_mappings_walker,
.flags = KVM_PGTABLE_WALK_LEAF,
};
+ int i, ret;
+
+ for (i = 0; i < hyp_memblock_nr; i++) {
+ struct memblock_region *reg = &hyp_memory[i];
+ u64 start = (u64)hyp_phys_to_virt(reg->base);
+
+ ret = kvm_pgtable_walk(&pkvm_pgtable, start, reg->size, &walker);
+ if (ret)
+ return ret;
+ }
- return kvm_pgtable_walk(&pkvm_pgtable, 0, BIT(pkvm_pgtable.ia_bits), &walker);
+ return 0;
}
void __noreturn __pkvm_init_finalise(void)
return true;
}
-/**
+/*
* Handler for protected VM restricted exceptions.
*
* Inject an undefined exception into the guest and return true to indicate that
obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o
CFLAGS_REMOVE_xor-neon.o += -mgeneral-regs-only
CFLAGS_xor-neon.o += -ffreestanding
+# Enable <arm_neon.h>
+CFLAGS_xor-neon.o += -isystem $(shell $(CC) -print-file-name=include)
endif
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
{
void *p = memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE,
__pa(MAX_DMA_ADDRESS),
- MEMBLOCK_ALLOC_KASAN, node);
+ MEMBLOCK_ALLOC_NOLEAKTRACE, node);
if (!p)
panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%llx\n",
__func__, PAGE_SIZE, PAGE_SIZE, node,
{
void *p = memblock_alloc_try_nid_raw(PAGE_SIZE, PAGE_SIZE,
__pa(MAX_DMA_ADDRESS),
- MEMBLOCK_ALLOC_KASAN, node);
+ MEMBLOCK_ALLOC_NOLEAKTRACE,
+ node);
if (!p)
panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%llx\n",
__func__, PAGE_SIZE, PAGE_SIZE, node,
init_task.kasan_depth = 0;
}
+#ifdef CONFIG_KASAN_VMALLOC
+void __init kasan_populate_early_vm_area_shadow(void *start, unsigned long size)
+{
+ unsigned long shadow_start, shadow_end;
+
+ if (!is_vmalloc_or_module_addr(start))
+ return;
+
+ shadow_start = (unsigned long)kasan_mem_to_shadow(start);
+ shadow_start = ALIGN_DOWN(shadow_start, PAGE_SIZE);
+ shadow_end = (unsigned long)kasan_mem_to_shadow(start + size);
+ shadow_end = ALIGN(shadow_end, PAGE_SIZE);
+ kasan_map_populate(shadow_start, shadow_end, NUMA_NO_NODE);
+}
+#endif
+
void __init kasan_init(void)
{
kasan_init_shadow();
kasan_init_depth();
#if defined(CONFIG_KASAN_GENERIC)
/* CONFIG_KASAN_SW_TAGS also requires kasan_init_sw_tags(). */
- pr_info("KernelAddressSanitizer initialized\n");
+ pr_info("KernelAddressSanitizer initialized (generic)\n");
#endif
}
phys_addr_t phys;
void *ptr;
- phys = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE);
+ phys = memblock_phys_alloc_range(PAGE_SIZE, PAGE_SIZE, 0,
+ MEMBLOCK_ALLOC_NOLEAKTRACE);
if (!phys)
panic("Failed to allocate page table page\n");
cpu_replace_ttbr1(lm_alias(swapper_pg_dir));
init_mm.pgd = swapper_pg_dir;
- memblock_free(__pa_symbol(init_pg_dir),
- __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir));
+ memblock_phys_free(__pa_symbol(init_pg_dir),
+ __pa_symbol(init_pg_end) - __pa_symbol(init_pg_dir));
memblock_allow_resize();
}
HYPERCALL2(memory_op);
HYPERCALL2(physdev_op);
HYPERCALL3(vcpu_op);
-HYPERCALL1(tmem_op);
HYPERCALL1(platform_op_raw);
HYPERCALL2(multicall);
HYPERCALL2(vm_assist);
# SPDX-License-Identifier: GPL-2.0-only
+
+# for cleaning
+subdir- += boot
zImage Image uImage: vmlinux
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
define archhelp
echo '* zImage - Compressed kernel image (arch/$(ARCH)/boot/zImage)'
echo ' Image - Uncompressed kernel image (arch/$(ARCH)/boot/Image)'
memcpy(args, ®s->a1, 5 * sizeof(args[0]));
}
-static inline void
-syscall_set_arguments(struct task_struct *task, struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->orig_a0 = args[0];
- args++;
- memcpy(®s->a1, args, 5 * sizeof(regs->a1));
-}
-
static inline int
syscall_get_arch(struct task_struct *task)
{
# SPDX-License-Identifier: GPL-2.0-only
obj-y += kernel/ mm/ boot/dts/
+
+# for cleaning
+subdir- += boot
boot := arch/h8300/boot
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
vmlinux.srec vmlinux.bin zImage uImage.bin: vmlinux
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Timer support for Hexagon
- *
- * Copyright (c) 2010-2011, The Linux Foundation. All rights reserved.
- */
-
-#ifndef _ASM_TIMER_REGS_H
-#define _ASM_TIMER_REGS_H
-
-/* This stuff should go into a platform specific file */
-#define TCX0_CLK_RATE 19200
-#define TIMER_ENABLE 0
-#define TIMER_CLR_ON_MATCH 1
-
-/*
- * 8x50 HDD Specs 5-8. Simulator co-sim not fixed until
- * release 1.1, and then it's "adjustable" and probably not defaulted.
- */
-#define RTOS_TIMER_INT 3
-#ifdef CONFIG_HEXAGON_COMET
-#define RTOS_TIMER_REGS_ADDR 0xAB000000UL
-#endif
-#define SLEEP_CLK_RATE 32000
-
-#endif
#define _ASM_TIMEX_H
#include <asm-generic/timex.h>
-#include <asm/timer-regs.h>
#include <asm/hexagon_vm.h>
/* Using TCX0 as our clock. CLOCK_TICK_RATE scheduled to be removed. */
-#define CLOCK_TICK_RATE TCX0_CLK_RATE
+#define CLOCK_TICK_RATE 19200
#define ARCH_HAS_READ_CURRENT_TIMER
--- /dev/null
+vmlinux.lds
#include <linux/of_irq.h>
#include <linux/module.h>
-#include <asm/timer-regs.h>
#include <asm/hexagon_vm.h>
+#define TIMER_ENABLE BIT(0)
+
/*
* For the clocksource we need:
* pcycle frequency (600MHz)
cycles_t thread_freq_mhz;
cycles_t sleep_clk_freq;
+/*
+ * 8x50 HDD Specs 5-8. Simulator co-sim not fixed until
+ * release 1.1, and then it's "adjustable" and probably not defaulted.
+ */
+#define RTOS_TIMER_INT 3
+#define RTOS_TIMER_REGS_ADDR 0xAB000000UL
+
static struct resource rtos_timer_resources[] = {
{
.start = RTOS_TIMER_REGS_ADDR,
iowrite32(0, &rtos_timer->clear);
iowrite32(delta, &rtos_timer->match);
- iowrite32(1 << TIMER_ENABLE, &rtos_timer->enable);
+ iowrite32(TIMER_ENABLE, &rtos_timer->enable);
return 0;
}
*dst++ = *src;
}
+EXPORT_SYMBOL(__raw_readsw);
/*
* __raw_writesw - read words a short at a time
}
+EXPORT_SYMBOL(__raw_writesw);
/* Pretty sure len is pre-adjusted for the length of the access already */
void __raw_readsl(const void __iomem *addr, void *data, int len)
}
+EXPORT_SYMBOL(__raw_readsl);
void __raw_writesl(void __iomem *addr, const void *data, int len)
{
}
+EXPORT_SYMBOL(__raw_writesl);
unwcheck: vmlinux
-$(Q)READELF=$(READELF) $(PYTHON3) $(srctree)/arch/ia64/scripts/unwcheck.py $<
-archclean:
-
archheaders:
$(Q)$(MAKE) $(build)=arch/ia64/kernel/syscalls all
}
}
-extern void ia64_syscall_get_set_arguments(struct task_struct *task,
- struct pt_regs *regs, unsigned long *args, int rw);
-static inline void syscall_get_arguments(struct task_struct *task,
- struct pt_regs *regs,
- unsigned long *args)
-{
- ia64_syscall_get_set_arguments(task, regs, args, 0);
-}
-
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- unsigned long *args)
-{
- ia64_syscall_get_set_arguments(task, regs, args, 1);
-}
+extern void syscall_get_arguments(struct task_struct *task,
+ struct pt_regs *regs, unsigned long *args);
static inline int syscall_get_arch(struct task_struct *task)
{
return &user_ia64_view;
}
-struct syscall_get_set_args {
+struct syscall_get_args {
unsigned int i;
unsigned int n;
unsigned long *args;
struct pt_regs *regs;
- int rw;
};
-static void syscall_get_set_args_cb(struct unw_frame_info *info, void *data)
+static void syscall_get_args_cb(struct unw_frame_info *info, void *data)
{
- struct syscall_get_set_args *args = data;
+ struct syscall_get_args *args = data;
struct pt_regs *pt = args->regs;
unsigned long *krbs, cfm, ndirty, nlocals, nouts;
int i, count;
/* Iterate over outs. */
for (i = 0; i < count; i++) {
int j = ndirty + nlocals + i + args->i;
- if (args->rw)
- *ia64_rse_skip_regs(krbs, j) = args->args[i];
- else
- args->args[i] = *ia64_rse_skip_regs(krbs, j);
+ args->args[i] = *ia64_rse_skip_regs(krbs, j);
}
- if (!args->rw) {
- while (i < args->n) {
- args->args[i] = 0;
- i++;
- }
+ while (i < args->n) {
+ args->args[i] = 0;
+ i++;
}
}
-void ia64_syscall_get_set_arguments(struct task_struct *task,
- struct pt_regs *regs, unsigned long *args, int rw)
+void syscall_get_arguments(struct task_struct *task,
+ struct pt_regs *regs, unsigned long *args)
{
- struct syscall_get_set_args data = {
+ struct syscall_get_args data = {
.i = 0,
.n = 6,
.args = args,
.regs = regs,
- .rw = rw,
};
if (task == current)
- unw_init_running(syscall_get_set_args_cb, &data);
+ unw_init_running(syscall_get_args_cb, &data);
else {
struct unw_frame_info ufi;
memset(&ufi, 0, sizeof(ufi));
unw_init_from_blocked_task(&ufi, task);
- syscall_get_set_args_cb(&ufi, &data);
+ syscall_get_args_cb(&ufi, &data);
}
}
efi_memmap_walk(find_max_min_low_pfn, NULL);
max_pfn = max_low_pfn;
- memblock_add_node(0, PFN_PHYS(max_low_pfn), 0);
+ memblock_add_node(0, PFN_PHYS(max_low_pfn), 0, MEMBLOCK_NONE);
find_initrd();
#endif
if (start < end)
- memblock_add_node(__pa(start), end - start, nid);
+ memblock_add_node(__pa(start), end - start, nid, MEMBLOCK_NONE);
return 0;
}
System-On-Chip devices (eg 68328, 68302, etc). It does not contain
a paging MMU.
-config MCPU32
- bool
- select CPU_HAS_NO_BITFIELDS
- select CPU_HAS_NO_CAS
- select CPU_HAS_NO_UNALIGNED
- select CPU_NO_EFFICIENT_FFS
- help
- The Freescale (was then Motorola) CPU32 is a CPU core that is
- based on the 68020 processor. For the most part it is used in
- System-On-Chip parts, and does not contain a paging MMU.
-
config M68020
bool "68020 support"
depends on MMU
config MEMORY_RESERVE
int "Memory reservation (MiB)"
depends on (UCSIMM || UCDIMM)
+ default 0
help
Reserve certain memory regions on 68x328 based boards.
# m68k/Makefile
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
#
# This file is subject to the terms and conditions of the GNU General Public
# License. See the file "COPYING" in the main directory of this archive
* generic functions for those.
*/
#if (defined(__mcfisaaplus__) || defined(__mcfisac__)) && \
- !defined(CONFIG_M68000) && !defined(CONFIG_MCPU32)
+ !defined(CONFIG_M68000)
static inline unsigned long __ffs(unsigned long x)
{
__asm__ __volatile__ ("bitrev %0; ff1 %0"
*/
asmlinkage void fpsp040_die(void)
{
- force_sigsegv(SIGSEGV);
+ force_exit_sig(SIGSEGV);
}
#ifdef CONFIG_M68KFPU_EMU
m68k_memory[0].addr = _rambase;
m68k_memory[0].size = _ramend - _rambase;
- memblock_add_node(m68k_memory[0].addr, m68k_memory[0].size, 0);
+ memblock_add_node(m68k_memory[0].addr, m68k_memory[0].size, 0,
+ MEMBLOCK_NONE);
/* compute total pages in system */
num_pages = PFN_DOWN(_ramend - _rambase);
min_addr = m68k_memory[0].addr;
max_addr = min_addr + m68k_memory[0].size;
- memblock_add_node(m68k_memory[0].addr, m68k_memory[0].size, 0);
+ memblock_add_node(m68k_memory[0].addr, m68k_memory[0].size, 0,
+ MEMBLOCK_NONE);
for (i = 1; i < m68k_num_memory;) {
if (m68k_memory[i].addr < min_addr) {
printk("Ignoring memory chunk at 0x%lx:0x%lx before the first chunk\n",
(m68k_num_memory - i) * sizeof(struct m68k_mem_info));
continue;
}
- memblock_add_node(m68k_memory[i].addr, m68k_memory[i].size, i);
+ memblock_add_node(m68k_memory[i].addr, m68k_memory[i].size, i,
+ MEMBLOCK_NONE);
addr = m68k_memory[i].addr + m68k_memory[i].size;
if (addr > max_addr)
max_addr = addr;
obj-y += mm/
obj-$(CONFIG_PCI) += pci/
obj-y += boot/dts/
+
+# for cleaning
+subdir- += boot
all: linux.bin
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
archheaders:
$(Q)$(MAKE) $(build)=arch/microblaze/kernel/syscalls all
return ~0;
}
-static inline void microblaze_set_syscall_arg(struct pt_regs *regs,
- unsigned int n,
- unsigned long val)
-{
- switch (n) {
- case 5:
- regs->r10 = val;
- case 4:
- regs->r9 = val;
- case 3:
- regs->r8 = val;
- case 2:
- regs->r7 = val;
- case 1:
- regs->r6 = val;
- case 0:
- regs->r5 = val;
- default:
- BUG();
- }
-}
-
static inline void syscall_get_arguments(struct task_struct *task,
struct pt_regs *regs,
unsigned long *args)
*args++ = microblaze_get_syscall_arg(regs, i++);
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- unsigned int i = 0;
- unsigned int n = 6;
-
- while (n--)
- microblaze_set_syscall_arg(regs, i++, *args++);
-}
-
asmlinkage unsigned long do_syscall_trace_enter(struct pt_regs *regs);
asmlinkage void do_syscall_trace_leave(struct pt_regs *regs);
#include <linux/mm_types.h>
#include <linux/pgtable.h>
#include <linux/memblock.h>
+#include <linux/kallsyms.h>
#include <asm/pgalloc.h>
#include <linux/io.h>
for (s = 0; s < lowmem_size; s += PAGE_SIZE) {
f = _PAGE_PRESENT | _PAGE_ACCESSED |
_PAGE_SHARED | _PAGE_HWEXEC;
- if ((char *) v < _stext || (char *) v >= _etext)
+ if (!is_kernel_text(v))
f |= _PAGE_WRENABLE;
else
/* On the MicroBlaze, no user access
}
DECLARE_PCI_FIXUP_HEADER(PCI_ANY_ID, PCI_ANY_ID, pcibios_fixup_resources);
-int pcibios_add_device(struct pci_dev *dev)
+int pcibios_device_add(struct pci_dev *dev)
{
dev->irq = of_irq_parse_and_map_pci(dev, 0, 0);
return 0;
}
-EXPORT_SYMBOL(pcibios_add_device);
/*
* Reparent resource children of pr that conflict with res
ifdef CONFIG_KVM
obj-y += kvm/
endif
+
+# for cleaning
+subdir- += boot
platform-$(CONFIG_MACH_VR41XX) += vr41xx/
# include the platform specific files
-include $(patsubst %, $(srctree)/arch/mips/%/Platform, $(platform-y))
+include $(patsubst %/, $(srctree)/arch/mips/%/Platform, $(platform-y))
select USB_OHCI_BIG_ENDIAN_DESC if CPU_BIG_ENDIAN
select USB_OHCI_BIG_ENDIAN_MMIO if CPU_BIG_ENDIAN
select HARDIRQS_SW_RESEND
+ select HAVE_PCI
+ select PCI_DRIVERS_GENERIC
help
Build a generic DT-based kernel image that boots on select
BCM33xx cable modem chips, BCM63xx DSL chips, and BCM7xxx set-top
select SYS_SUPPORTS_32BIT_KERNEL
select SYS_SUPPORTS_BIG_ENDIAN
select SYS_HAS_EARLY_PRINTK
+ select SYS_HAS_CPU_BMIPS32_3300
+ select SYS_HAS_CPU_BMIPS4350
+ select SYS_HAS_CPU_BMIPS4380
select SWAP_IO_SPACE
select GPIOLIB
select MIPS_L1_CACHE_SHIFT_4
# Copyright (C) 2002, 2003, 2004 Maciej W. Rozycki
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" cleaning up for this architecture.
+# architecture-specific flags and dependencies.
#
archscripts: scripts_basic
#
# Board-dependent options and extra files
#
+ifdef need-compiler
include $(srctree)/arch/mips/Kbuild.platforms
+endif
ifdef CONFIG_PHYSICAL_START
load-y = $(CONFIG_PHYSICAL_START)
$(Q)install -D -m 644 .config $(INSTALL_PATH)/config-$(KERNELRELEASE)
$(Q)install -D -m 644 System.map $(INSTALL_PATH)/System.map-$(KERNELRELEASE)
-archclean:
- $(Q)$(MAKE) $(clean)=arch/mips/boot
- $(Q)$(MAKE) $(clean)=arch/mips/boot/compressed
- $(Q)$(MAKE) $(clean)=arch/mips/boot/tools
-
archheaders:
$(Q)$(MAKE) $(build)=arch/mips/kernel/syscalls all
EXPORT_SYMBOL(clk_disable);
+struct clk *clk_get_parent(struct clk *clk)
+{
+ return NULL;
+}
+EXPORT_SYMBOL(clk_get_parent);
+
unsigned long clk_get_rate(struct clk *clk)
{
if (!clk)
$(obj)/vmlinux.%.itb: $(obj)/vmlinux.%.its $(obj)/vmlinux.bin.% FORCE
$(call if_changed,itb-image,$<)
+
+# for cleaning
+subdir- += compressed tools
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0-only
-ashldi3.c
-bswapsi.c
vmlinuzobjs-$(CONFIG_ATH79) += $(obj)/uart-ath79.o
endif
-extra-y += uart-ath79.c
-$(obj)/uart-ath79.c: $(srctree)/arch/mips/ath79/early_printk.c
- $(call cmd,shipped)
-
vmlinuzobjs-$(CONFIG_KERNEL_XZ) += $(obj)/ashldi3.o
-extra-y += ashldi3.c
-$(obj)/ashldi3.c: $(obj)/%.c: $(srctree)/lib/%.c FORCE
- $(call if_changed,shipped)
-
-extra-y += bswapsi.c
-$(obj)/bswapsi.c: $(obj)/%.c: $(srctree)/arch/mips/lib/%.c FORCE
- $(call if_changed,shipped)
+vmlinuzobjs-$(CONFIG_KERNEL_ZSTD) += $(obj)/bswapdi.o
targets := $(notdir $(vmlinuzobjs-y))
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../../../lib/ashldi3.c"
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../lib/bswapdi.c"
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../lib/bswapsi.c"
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../ath79/early_printk.c"
// SPDX-License-Identifier: GPL-2.0
-#include <dt-bindings/clock/jz4725b-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4725b-cgu.h>
#include <dt-bindings/clock/ingenic,tcu.h>
/ {
// SPDX-License-Identifier: GPL-2.0
-#include <dt-bindings/clock/jz4740-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4740-cgu.h>
#include <dt-bindings/clock/ingenic,tcu.h>
/ {
// SPDX-License-Identifier: GPL-2.0
-#include <dt-bindings/clock/jz4770-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4770-cgu.h>
#include <dt-bindings/clock/ingenic,tcu.h>
/ {
// SPDX-License-Identifier: GPL-2.0
-#include <dt-bindings/clock/jz4780-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include <dt-bindings/clock/ingenic,tcu.h>
#include <dt-bindings/dma/jz4780-dma.h>
// SPDX-License-Identifier: GPL-2.0
#include <dt-bindings/clock/ingenic,tcu.h>
-#include <dt-bindings/clock/x1000-cgu.h>
+#include <dt-bindings/clock/ingenic,x1000-cgu.h>
#include <dt-bindings/dma/x1000-dma.h>
/ {
// SPDX-License-Identifier: GPL-2.0
#include <dt-bindings/clock/ingenic,tcu.h>
-#include <dt-bindings/clock/x1830-cgu.h>
+#include <dt-bindings/clock/ingenic,x1830-cgu.h>
#include <dt-bindings/dma/x1830-dma.h>
/ {
# CONFIG_LOCALVERSION_AUTO is not set
# CONFIG_SWAP is not set
CONFIG_NO_HZ=y
+CONFIG_HZ=1000
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_BMIPS_GENERIC=y
CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_HIGHMEM=y
+CONFIG_HIGH_RES_TIMERS=y
CONFIG_SMP=y
CONFIG_NR_CPUS=4
+CONFIG_CC_STACKPROTECTOR_STRONG=y
# CONFIG_SECCOMP is not set
CONFIG_MIPS_O32_FP64_SUPPORT=y
+# CONFIG_RD_GZIP is not set
+# CONFIG_RD_BZIP2 is not set
+# CONFIG_RD_LZMA is not set
+CONFIG_RD_XZ=y
+# CONFIG_RD_LZO is not set
+# CONFIG_RD_LZ4 is not set
+# CONFIG_IOSCHED_DEADLINE is not set
+# CONFIG_IOSCHED_CFQ is not set
+CONFIG_PCI=y
+CONFIG_PCI_MSI=y
+CONFIG_PCIEASPM_POWERSAVE=y
+CONFIG_PCIEPORTBUS=y
+CONFIG_PCIE_BRCMSTB=y
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_STAT=y
+CONFIG_CPU_FREQ_STAT_DETAILS=y
+CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
+CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
-CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
CONFIG_BMIPS_CPUFREQ=y
# CONFIG_BLK_DEV_BSG is not set
CONFIG_NET=y
# CONFIG_INET_DIAG is not set
CONFIG_CFG80211=y
CONFIG_NL80211_TESTMODE=y
+CONFIG_WIRELESS=y
CONFIG_MAC80211=y
+CONFIG_NL80211=y
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
# CONFIG_STANDALONE is not set
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
+CONFIG_BRCMSTB_GISB_ARB=y
+CONFIG_MODULES=y
+CONFIG_MODULE_FORCE_LOAD=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+CONFIG_IP_MULTICAST=y
+CONFIG_IP_PNP=y
+CONFIG_IP_PNP_DHCP=y
+CONFIG_IP_PNP_BOOTP=y
+CONFIG_IP_PNP_RARP=y
+CONFIG_IP_MROUTE=y
+CONFIG_IP_PIMSM_V1=y
+CONFIG_IP_PIMSM_V2=y
+# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
+# CONFIG_INET_XFRM_MODE_TUNNEL is not set
+# CONFIG_INET_XFRM_MODE_BEET is not set
+# CONFIG_INET_LRO is not set
+CONFIG_INET_UDP_DIAG=y
+CONFIG_TCP_CONG_ADVANCED=y
+CONFIG_TCP_CONG_BIC=y
+# CONFIG_TCP_CONG_WESTWOOD is not set
+# CONFIG_TCP_CONG_HTCP is not set
+# CONFIG_IPV6 is not set
+CONFIG_IP_NF_IPTABLES=y
+CONFIG_IP_NF_FILTER=y
+CONFIG_NETFILTER=y
+CONFIG_NETFILTER_XTABLES=y
+CONFIG_BRIDGE=y
+CONFIG_BRIDGE_NETFILTER=m
+CONFIG_BRIDGE_NF_EBTABLES=m
+CONFIG_BRIDGE_EBT_BROUTE=m
+CONFIG_NET_DSA=y
+CONFIG_NET_SWITCHDEV=y
+CONFIG_DMA_CMA=y
+CONFIG_CMA_ALIGNMENT=12
+CONFIG_SPI=y
+CONFIG_SPI_BRCMSTB=y
CONFIG_MTD=y
+CONFIG_MTD_CMDLINE_PARTS=y
+CONFIG_MTD_BLOCK=y
CONFIG_MTD_CFI=y
+CONFIG_MTD_JEDECPROBE=y
CONFIG_MTD_CFI_INTELEXT=y
CONFIG_MTD_CFI_AMDSTD=y
-CONFIG_MTD_PHYSMAP=y
+CONFIG_MTD_CFI_STAA=y
+CONFIG_MTD_ROM=y
+CONFIG_MTD_ABSENT=y
+CONFIG_MTD_PHYSMAP_OF=y
+CONFIG_MTD_M25P80=y
+CONFIG_MTD_NAND=y
+CONFIG_MTD_NAND_BRCMNAND=y
+CONFIG_MTD_SPI_NOR=y
+# CONFIG_MTD_SPI_NOR_USE_4K_SECTORS is not set
+CONFIG_MTD_UBI=y
+CONFIG_MTD_UBI_GLUEBI=y
+CONFIG_PROC_DEVICETREE=y
+CONFIG_BLK_DEV_LOOP=y
+CONFIG_BLK_DEV_RAM=y
+CONFIG_BLK_DEV_RAM_SIZE=8192
# CONFIG_BLK_DEV is not set
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
+CONFIG_CHR_DEV_SG=y
+CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_NETDEVICES=y
+CONFIG_VLAN_8021Q=y
+CONFIG_MACVLAN=y
CONFIG_BCMGENET=y
CONFIG_USB_USBNET=y
-# CONFIG_INPUT is not set
+CONFIG_INPUT_EVDEV=y
+# CONFIG_INPUT_KEYBOARD is not set
+# CONFIG_INPUT_MOUSE is not set
+CONFIG_INPUT_MISC=y
+CONFIG_INPUT_UINPUT=y
# CONFIG_SERIO is not set
-# CONFIG_VT is not set
+CONFIG_VT=y
+CONFIG_VT_HW_CONSOLE_BINDING=y
+# CONFIG_DEVKMEM is not set
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_OF_PLATFORM=y
# CONFIG_HW_RANDOM is not set
CONFIG_POWER_RESET=y
+CONFIG_POWER_RESET_BRCMSTB=y
CONFIG_POWER_RESET_SYSCON=y
CONFIG_POWER_SUPPLY=y
# CONFIG_HWMON is not set
CONFIG_USB_OHCI_HCD_PLATFORM=y
CONFIG_USB_STORAGE=y
CONFIG_SOC_BRCMSTB=y
+CONFIG_MMC=y
+CONFIG_MMC_BLOCK_MINORS=16
+CONFIG_MMC_SDHCI=y
+CONFIG_MMC_SDHCI_PLTFM=y
CONFIG_EXT4_FS=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_DNOTIFY is not set
+CONFIG_PROC_KCORE=y
+CONFIG_CIFS=y
+CONFIG_JBD2_DEBUG=y
CONFIG_FUSE_FS=y
+CONFIG_FHANDLE=y
+CONFIG_CGROUPS=y
+CONFIG_CUSE=y
+CONFIG_ISO9660_FS=y
+CONFIG_JOLIET=y
+CONFIG_ZISOFS=y
+CONFIG_UDF_FS=y
+CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
-CONFIG_PROC_KCORE=y
CONFIG_TMPFS=y
+CONFIG_JFFS2_FS=y
+CONFIG_UBIFS_FS=y
+CONFIG_SQUASHFS=y
+CONFIG_SQUASHFS_LZO=y
+CONFIG_SQUASHFS_XZ=y
CONFIG_NFS_FS=y
-CONFIG_CIFS=y
+CONFIG_NFS_V3_ACL=y
+CONFIG_NFS_V4=y
+CONFIG_NFS_V4_1=y
+CONFIG_NFS_V4_2=y
+CONFIG_ROOT_NFS=y
CONFIG_NLS_CODEPAGE_437=y
-CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
-# CONFIG_CRYPTO_HW is not set
CONFIG_PRINTK_TIME=y
+CONFIG_DYNAMIC_DEBUG=y
+# CONFIG_DEBUG_INFO is not set
+# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_FS=y
CONFIG_MAGIC_SYSRQ=y
+CONFIG_LOCKUP_DETECTOR=y
+CONFIG_DEBUG_USER=y
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="earlycon"
+# CONFIG_MIPS_CMDLINE_FROM_DTB is not set
+CONFIG_MIPS_CMDLINE_DTB_EXTEND=y
+# CONFIG_MIPS_CMDLINE_FROM_BOOTLOADER is not set
+# CONFIG_CRYPTO_HW is not set
+CONFIG_DT_BCM974XX=y
+CONFIG_FW_CFE=y
+CONFIG_ATA=y
+CONFIG_SATA_AHCI_PLATFORM=y
+CONFIG_AHCI_BRCMSTB=y
+CONFIG_GENERIC_PHY=y
+CONFIG_GPIOLIB=y
+CONFIG_GPIO_SYSFS=y
+CONFIG_PHY_BRCM_USB=y
+CONFIG_PHY_BRCM_SATA=y
+CONFIG_PM_RUNTIME=y
+CONFIG_PM_DEBUG=y
+CONFIG_SYSVIPC=y
+CONFIG_FUNCTION_GRAPH_TRACER=y
+CONFIG_DYNAMIC_FTRACE=y
+CONFIG_FUNCTION_TRACER=y
+CONFIG_FUNCTION_PROFILER=y
+CONFIG_IRQSOFF_TRACER=y
+CONFIG_SCHED_TRACER=y
+CONFIG_BLK_DEV_IO_TRACE=y
+CONFIG_FTRACE_SYSCALLS=y
+CONFIG_TRACER_SNAPSHOT=y
+CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y
+CONFIG_STACK_TRACER=y
{
switch (mips_machtype) {
case MACH_DS23100: /* DS2100/DS3100 Pmin/Pmax */
- board_be_handler = dec_kn01_be_handler;
+ mips_set_be_handler(dec_kn01_be_handler);
busirq_handler = dec_kn01_be_interrupt;
busirq_flags |= IRQF_SHARED;
dec_kn01_be_init();
break;
case MACH_DS5000_1XX: /* DS5000/1xx 3min */
case MACH_DS5000_XX: /* DS5000/xx Maxine */
- board_be_handler = dec_kn02xa_be_handler;
+ mips_set_be_handler(dec_kn02xa_be_handler);
busirq_handler = dec_kn02xa_be_interrupt;
dec_kn02xa_be_init();
break;
case MACH_DS5000_200: /* DS5000/200 3max */
case MACH_DS5000_2X0: /* DS5000/240 3max+ */
case MACH_DS5900: /* DS5900 bigmax */
- board_be_handler = dec_ecc_be_handler;
+ mips_set_be_handler(dec_ecc_be_handler);
busirq_handler = dec_ecc_be_interrupt;
dec_ecc_be_init();
break;
__init int yamon_dt_append_memory(void *fdt,
const struct yamon_mem_region *regions)
{
- unsigned long phys_memsize, memsize;
+ unsigned long phys_memsize = 0, memsize;
__be32 mem_array[2 * MAX_MEM_ARRAY_ENTRIES];
unsigned int mem_entries;
int i, err, mem_off;
#define MIPS_BE_FATAL 2 /* treat as an unrecoverable error */
extern void (*board_be_init)(void);
-extern int (*board_be_handler)(struct pt_regs *regs, int is_fixup);
+void mips_set_be_handler(int (*handler)(struct pt_regs *reg, int is_fixup));
extern void (*board_nmi_handler_setup)(void);
extern void (*board_ejtag_handler_setup)(void);
#define EX2(a,b) \
9: a,##b; \
.section __ex_table,"a"; \
- PTR 9b,bad_stack; \
- PTR 9b+4,bad_stack; \
+ PTR 9b,fault; \
+ PTR 9b+4,fault; \
.previous
.set mips1
{
return -ENOSYS;
}
-
-/*
- * If we ever come here the user sp is bad. Zap the process right away.
- * Due to the bad stack signaling wouldn't work.
- */
-asmlinkage void bad_stack(void)
-{
- do_exit(SIGSEGV);
-}
446 n32 landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 n32 process_mrelease sys_process_mrelease
+449 n32 futex_waitv sys_futex_waitv
446 n64 landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 n64 process_mrelease sys_process_mrelease
+449 n64 futex_waitv sys_futex_waitv
446 o32 landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 o32 process_mrelease sys_process_mrelease
+449 o32 futex_waitv sys_futex_waitv
extern void tlb_do_page_fault_0(void);
void (*board_be_init)(void);
-int (*board_be_handler)(struct pt_regs *regs, int is_fixup);
+static int (*board_be_handler)(struct pt_regs *regs, int is_fixup);
void (*board_nmi_handler_setup)(void);
void (*board_ejtag_handler_setup)(void);
void (*board_bind_eic_interrupt)(int irq, int regset);
void (*board_ebase_setup)(void);
void(*board_cache_error_setup)(void);
+void mips_set_be_handler(int (*handler)(struct pt_regs *regs, int is_fixup))
+{
+ board_be_handler = handler;
+}
+EXPORT_SYMBOL_GPL(mips_set_be_handler);
+
static void show_raw_backtrace(unsigned long reg29, const char *loglvl,
bool user)
{
r = 1;
break;
case KVM_CAP_NR_VCPUS:
- r = num_online_cpus();
+ r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
break;
case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
}
EXPORT_SYMBOL(clk_deactivate);
+struct clk *clk_get_parent(struct clk *clk)
+{
+ return NULL;
+}
+EXPORT_SYMBOL(clk_get_parent);
+
static inline u32 get_counter_resolution(void)
{
u32 res;
(u32)node_id, mem_type, mem_start, mem_size);
pr_info(" start_pfn:0x%llx, end_pfn:0x%llx, num_physpages:0x%lx\n",
start_pfn, end_pfn, num_physpages);
- memblock_add_node(PFN_PHYS(start_pfn), PFN_PHYS(node_psize), node);
+ memblock_add_node(PFN_PHYS(start_pfn),
+ PFN_PHYS(node_psize), node,
+ MEMBLOCK_NONE);
break;
case SYSTEM_RAM_RESERVED:
pr_info("Node%d: mem_type:%d, mem_start:0x%llx, mem_size:0x%llx MB\n",
static void __init pcpu_fc_free(void *ptr, size_t size)
{
- memblock_free_early(__pa(ptr), size);
+ memblock_free(ptr, size);
}
void __init setup_per_cpu_areas(void)
select SYS_SUPPORTS_HIGHMEM
select MIPS_GIC
select CLKSRC_MIPS_GIC
- select HAVE_PCI if PCI_MT7621
+ select HAVE_PCI
+ select PCI_DRIVERS_GENERIC
select SOC_BUS
endchoice
void __init ip22_be_init(void)
{
- board_be_handler = ip22_be_handler;
+ mips_set_be_handler(ip22_be_handler);
}
void __init ip22_be_init(void)
{
- board_be_handler = ip28_be_handler;
+ mips_set_be_handler(ip28_be_handler);
}
int ip28_show_be_info(struct seq_file *m)
int cpu = LOCAL_HUB_L(PI_CPU_NUM);
int cpuoff = cpu << 8;
- board_be_handler = ip27_be_handler;
+ mips_set_be_handler(ip27_be_handler);
LOCAL_HUB_S(PI_ERR_INT_PEND,
cpu ? PI_ERR_CLEAR_ALL_B : PI_ERR_CLEAR_ALL_A);
continue;
}
memblock_add_node(PFN_PHYS(slot_getbasepfn(node, slot)),
- PFN_PHYS(slot_psize), node);
+ PFN_PHYS(slot_psize), node,
+ MEMBLOCK_NONE);
}
}
}
total_mem += size;
if (addr >= IP30_REAL_MEMORY_START)
- memblock_free(addr, size);
+ memblock_phys_free(addr, size);
else if ((addr + size) > IP30_REAL_MEMORY_START)
- memblock_free(IP30_REAL_MEMORY_START,
- size - IP30_MAX_PROM_MEMORY);
+ memblock_phys_free(IP30_REAL_MEMORY_START,
+ size - IP30_MAX_PROM_MEMORY);
}
pr_info("Detected %luMB of physical memory.\n", MEM_SHIFT(total_mem));
}
void __init ip32_be_init(void)
{
- board_be_handler = ip32_be_handler;
+ mips_set_be_handler(ip32_be_handler);
}
#error invalid SiByte board configuration
#endif
- board_be_handler = swarm_be_handler;
+ mips_set_be_handler(swarm_be_handler);
if (xicor_probe())
swarm_rtc_type = RTC_XICOR;
}
static void __init tx4927_be_init(void)
{
- board_be_handler = tx4927_be_handler;
+ mips_set_be_handler(tx4927_be_handler);
}
static struct resource tx4927_sdram_resource[4];
}
static void __init tx4938_be_init(void)
{
- board_be_handler = tx4938_be_handler;
+ mips_set_be_handler(tx4938_be_handler);
}
static struct resource tx4938_sdram_resource[4];
}
static void __init tx4939_be_init(void)
{
- board_be_handler = tx4939_be_handler;
+ mips_set_be_handler(tx4939_be_handler);
}
static struct resource tx4939_sdram_resource[4];
# VDSO linker flags.
ldflags-y := -Bsymbolic --no-undefined -soname=linux-vdso.so.1 \
- $(filter -E%,$(KBUILD_CFLAGS)) -nostdlib -shared \
+ $(filter -E%,$(KBUILD_CFLAGS)) -shared \
-G 0 --eh-frame-hdr --hash-style=sysv --build-id=sha1 -T
CFLAGS_REMOVE_vdso.o = $(CC_FLAGS_FTRACE)
# SPDX-License-Identifier: GPL-2.0-only
+
+# for cleaning
+subdir- += boot
# Avoid generating FPU instructions
arch-y += -mno-ext-fpu-sp -mno-ext-fpu-dp -mfloat-abi=soft
+# Enable <nds32_intrinsic.h>
+KBUILD_CFLAGS += -isystem $(shell $(CC) -print-file-name=include)
KBUILD_CFLAGS += $(call cc-option, -mno-sched-prolog-epilog)
KBUILD_CFLAGS += -mcmodel=large
vdso_prepare: prepare0
$(Q)$(MAKE) $(build)=arch/nds32/kernel/vdso include/generated/vdso-offsets.h
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
define archhelp
echo ' Image - kernel image (arch/$(ARCH)/boot/Image)'
endef
memcpy(args, ®s->uregs[0] + 1, 5 * sizeof(args[0]));
}
-/**
- * syscall_set_arguments - change system call parameter value
- * @task: task of interest, must be in system call entry tracing
- * @regs: task_pt_regs() of @task
- * @args: array of argument values to store
- *
- * Changes 6 arguments to the system call. The first argument gets value
- * @args[0], and so on.
- *
- * It's only valid to call this when @task is stopped for tracing on
- * entry to a system call, due to %TIF_SYSCALL_TRACE or %TIF_SYSCALL_AUDIT.
- */
-static inline void
-syscall_set_arguments(struct task_struct *task, struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->orig_r0 = args[0];
- args++;
-
- memcpy(®s->uregs[0] + 1, args, 5 * sizeof(args[0]));
-}
-
static inline int
syscall_get_arch(struct task_struct *task)
{
/*
* This function is protected against re-entrancy.
*/
-void die(const char *str, struct pt_regs *regs, int err)
+void __noreturn die(const char *str, struct pt_regs *regs, int err)
{
struct task_struct *tsk = current;
static int die_counter;
#include <asm/tlbflush.h>
-extern void die(const char *str, struct pt_regs *regs, long err);
+extern void __noreturn die(const char *str, struct pt_regs *regs, long err);
/*
* This is useful to dump out the page tables associated with
show_pte(mm, addr);
die("Oops", regs, error_code);
- bust_spinlocks(0);
- do_exit(SIGKILL);
-
- return;
/*
* We ran out of memory, or some other thing happened to us that made
# SPDX-License-Identifier: GPL-2.0-only
+
+# for cleaning
+subdir- += boot
# Written by Fredrik Markstrom
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" cleaning up for this architecture.
+# architecture-specific flags and dependencies.
#
# Nios2 port by Wind River Systems Inc trough:
# fredrik.markstrom@gmail.com and ivarholmqvist@gmail.com
all: vmImage
-archclean:
- $(Q)$(MAKE) $(clean)=$(nios2-boot)
-
$(BOOT_TARGETS): vmlinux
$(Q)$(MAKE) $(build)=$(nios2-boot) $(nios2-boot)/$@
install:
- $(Q)$(MAKE) $(build)=$(nios2-boot) BOOTIMAGE=$(KBUILD_IMAGE) install
+ sh $(srctree)/$(nios2-boot)/install.sh $(KERNELRELEASE) \
+ $(KBUILD_IMAGE) System.map "$(INSTALL_PATH)"
define archhelp
echo '* vmImage - Kernel-only image for U-Boot ($(KBUILD_IMAGE))'
$(obj)/compressed/vmlinux: $(obj)/vmlinux.gz FORCE
$(Q)$(MAKE) $(build)=$(obj)/compressed $@
-
-install:
- sh $(srctree)/$(src)/install.sh $(KERNELRELEASE) $(BOOTIMAGE) System.map "$(INSTALL_PATH)"
*args = regs->r9;
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs, const unsigned long *args)
-{
- regs->r4 = *args++;
- regs->r5 = *args++;
- regs->r6 = *args++;
- regs->r7 = *args++;
- regs->r8 = *args++;
- regs->r9 = *args;
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
return AUDIT_ARCH_NIOS2;
# SPDX-License-Identifier: GPL-2.0
obj-y += lib/ kernel/ mm/
obj-y += boot/dts/
+
+# for cleaning
+subdir- += boot
# BK Id: %F% %I% %G% %U% %#%
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
#
# This file is subject to the terms and conditions of the GNU General Public
# License. See the file "COPYING" in the main directory of this archive
vmlinux.bin: vmlinux
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
-
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
memcpy(args, ®s->gpr[3], 6 * sizeof(args[0]));
}
-static inline void
-syscall_set_arguments(struct task_struct *task, struct pt_regs *regs,
- const unsigned long *args)
-{
- memcpy(®s->gpr[3], args, 6 * sizeof(args[0]));
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
return AUDIT_ARCH_OPENRISC;
* Flush the page out of the TLB so that the new page flags get
* picked up next time there's an access
*/
- flush_tlb_page(NULL, addr);
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
/* Flush page out of dcache */
for (cl = __pa(addr); cl < __pa(next); cl += cpuinfo->dcache_block_size)
* Flush the page out of the TLB so that the new page flags get
* picked up next time there's an access
*/
- flush_tlb_page(NULL, addr);
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
return 0;
}
#include <asm/ucontext.h>
#include <linux/uaccess.h>
-#define DEBUG_SIG 0
-
struct rt_sigframe {
struct siginfo info;
struct ucontext uc;
local_flush_tlb_range(NULL, fd->addr1, fd->addr2);
}
-static void smp_flush_tlb_range(struct cpumask *cmask, unsigned long start,
+static void smp_flush_tlb_range(const struct cpumask *cmask, unsigned long start,
unsigned long end)
{
unsigned int cpuid;
void flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
- smp_flush_tlb_range(mm_cpumask(vma->vm_mm), start, end);
+ const struct cpumask *cmask = vma ? mm_cpumask(vma->vm_mm)
+ : cpu_online_mask;
+ smp_flush_tlb_range(cmask, start, end);
}
/* Instruction cache invalidate - performed on each cpu */
return IRQ_HANDLED;
}
-/**
+/*
* Clocksource: Based on OpenRISC timer/counter
*
* This sets up the OpenRISC Tick Timer as a clock source. The tick timer
}
/* This is normally the 'Oops' routine */
-void die(const char *str, struct pt_regs *regs, long err)
+void __noreturn die(const char *str, struct pt_regs *regs, long err)
{
console_verbose();
*/
volatile pgd_t *current_pgd[NR_CPUS];
-extern void die(char *, struct pt_regs *, long);
+extern void __noreturn die(char *, struct pt_regs *, long);
/*
* This routine handles page faults. It determines the address,
die("Oops", regs, write_acc);
- do_exit(SIGKILL);
-
/*
* We ran out of memory, or some other thing happened to us that made
* us unable to handle the page fault gracefully.
# SPDX-License-Identifier: GPL-2.0-only
obj-y += mm/ kernel/ math-emu/
+
+# for cleaning
+subdir- += boot
# parisc/Makefile
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
#
# This file is subject to the terms and conditions of the GNU General Public
# License. See the file "COPYING" in the main directory of this archive
@echo ' zinstall - Install compressed vmlinuz kernel'
endef
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
archheaders:
$(Q)$(MAKE) $(build)=arch/parisc/kernel/syscalls all
CONFIG_CRC_CCITT=m
CONFIG_CRC_T10DIF=y
CONFIG_FONTS=y
+CONFIG_PRINTK_TIME=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_MEMORY_INIT=y
* Copyright (C) 1999 Hewlett-Packard (Frank Rowand)
* Copyright (C) 1999 Philipp Rumpf <prumpf@tux.org>
* Copyright (C) 1999 SuSE GmbH
+ * Copyright (C) 2021 Helge Deller <deller@gmx.de>
*/
#ifndef _PARISC_ASSEMBLY_H
#define _PARISC_ASSEMBLY_H
-#define CALLEE_FLOAT_FRAME_SIZE 80
-
#ifdef CONFIG_64BIT
-#define LDREG ldd
-#define STREG std
-#define LDREGX ldd,s
-#define LDREGM ldd,mb
-#define STREGM std,ma
-#define SHRREG shrd
-#define SHLREG shld
-#define ANDCM andcm,*
-#define COND(x) * ## x
#define RP_OFFSET 16
#define FRAME_SIZE 128
#define CALLEE_REG_FRAME_SIZE 144
#define REG_SZ 8
#define ASM_ULONG_INSN .dword
#else /* CONFIG_64BIT */
-#define LDREG ldw
-#define STREG stw
-#define LDREGX ldwx,s
-#define LDREGM ldwm
-#define STREGM stwm
-#define SHRREG shr
-#define SHLREG shlw
-#define ANDCM andcm
-#define COND(x) x
#define RP_OFFSET 20
#define FRAME_SIZE 64
#define CALLEE_REG_FRAME_SIZE 128
/* Frame alignment for 32- and 64-bit */
#define FRAME_ALIGN 64
+#define CALLEE_FLOAT_FRAME_SIZE 80
#define CALLEE_SAVE_FRAME_SIZE (CALLEE_REG_FRAME_SIZE + CALLEE_FLOAT_FRAME_SIZE)
#ifdef CONFIG_PA20
#ifdef __ASSEMBLY__
#ifdef CONFIG_64BIT
+#define LDREG ldd
+#define STREG std
+#define LDREGX ldd,s
+#define LDREGM ldd,mb
+#define STREGM std,ma
+#define SHRREG shrd
+#define SHLREG shld
+#define ANDCM andcm,*
+#define COND(x) * ## x
+#else /* CONFIG_64BIT */
+#define LDREG ldw
+#define STREG stw
+#define LDREGX ldwx,s
+#define LDREGM ldwm
+#define STREGM stwm
+#define SHRREG shr
+#define SHLREG shlw
+#define ANDCM andcm
+#define COND(x) x
+#endif
+
+#ifdef CONFIG_64BIT
/* the 64-bit pa gnu assembler unfortunately defaults to .level 1.1 or 2.0 so
* work around that for now... */
.level 2.0w
#ifndef __ASSEMBLY__
#include <linux/types.h>
+#include <linux/stringify.h>
#include <asm/assembly.h>
#define JUMP_LABEL_NOP_SIZE 4
purge_tlb_end(flags);
}
+extern void __update_cache(pte_t pte);
+
/* Certain architectures need to do special things when PTEs
* within a page table are directly modified. Thus, the following
* hook is made available.
#define set_pte(pteptr, pteval) \
do { \
*(pteptr) = (pteval); \
- barrier(); \
+ mb(); \
} while(0)
#define set_pte_at(mm, addr, pteptr, pteval) \
do { \
+ if (pte_present(pteval) && \
+ pte_user(pteval)) \
+ __update_cache(pteval); \
*(pteptr) = (pteval); \
purge_tlb_entries(mm, addr); \
} while (0)
#define pte_none(x) (pte_val(x) == 0)
#define pte_present(x) (pte_val(x) & _PAGE_PRESENT)
+#define pte_user(x) (pte_val(x) & _PAGE_USER)
#define pte_clear(mm, addr, xp) set_pte_at(mm, addr, xp, __pte(0))
#define pmd_flag(x) (pmd_val(x) & PxD_FLAG_MASK)
#define PG_dcache_dirty PG_arch_1
-extern void update_mmu_cache(struct vm_area_struct *, unsigned long, pte_t *);
+#define update_mmu_cache(vms,addr,ptep) __update_cache(*ptep)
/* Encode and de-code a swap entry */
#ifndef _ASM_PARISC_RT_SIGFRAME_H
#define _ASM_PARISC_RT_SIGFRAME_H
-#define SIGRETURN_TRAMP 3
+#define SIGRETURN_TRAMP 4
#define SIGRESTARTBLOCK_TRAMP 5
#define TRAMP_SIZE (SIGRETURN_TRAMP + SIGRESTARTBLOCK_TRAMP)
#define pfn_va(pfn) __va(PFN_PHYS(pfn))
void
-update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *ptep)
+__update_cache(pte_t pte)
{
- unsigned long pfn = pte_pfn(*ptep);
+ unsigned long pfn = pte_pfn(pte);
struct page *page;
/* We don't have pte special. As a result, we can be called with
/* Are we being ptraced? */
LDREG TASK_TI_FLAGS(%r1),%r19
- ldi _TIF_SYSCALL_TRACE_MASK,%r2
+ ldi _TIF_SINGLESTEP|_TIF_BLOCKSTEP,%r2
and,COND(=) %r19,%r2,%r0
b,n syscall_restore_rfi
already in userspace. The first words of tramp are used to
save the previous sigrestartblock trampoline that might be
on the stack. We start the sigreturn trampoline at
- SIGRESTARTBLOCK_TRAMP. */
+ SIGRESTARTBLOCK_TRAMP+X. */
err |= __put_user(in_syscall ? INSN_LDI_R25_1 : INSN_LDI_R25_0,
&frame->tramp[SIGRESTARTBLOCK_TRAMP+0]);
- err |= __put_user(INSN_BLE_SR2_R0,
+ err |= __put_user(INSN_LDI_R20,
&frame->tramp[SIGRESTARTBLOCK_TRAMP+1]);
- err |= __put_user(INSN_LDI_R20,
+ err |= __put_user(INSN_BLE_SR2_R0,
&frame->tramp[SIGRESTARTBLOCK_TRAMP+2]);
+ err |= __put_user(INSN_NOP, &frame->tramp[SIGRESTARTBLOCK_TRAMP+3]);
- start = (unsigned long) &frame->tramp[SIGRESTARTBLOCK_TRAMP+0];
- end = (unsigned long) &frame->tramp[SIGRESTARTBLOCK_TRAMP+3];
+ start = (unsigned long) &frame->tramp[0];
+ end = (unsigned long) &frame->tramp[TRAMP_SIZE];
flush_user_dcache_range_asm(start, end);
flush_user_icache_range_asm(start, end);
/* TRAMP Words 0-4, Length 5 = SIGRESTARTBLOCK_TRAMP
- * TRAMP Words 5-7, Length 3 = SIGRETURN_TRAMP
+ * TRAMP Words 5-9, Length 4 = SIGRETURN_TRAMP
* So the SIGRETURN_TRAMP is at the end of SIGRESTARTBLOCK_TRAMP
*/
rp = (unsigned long) &frame->tramp[SIGRESTARTBLOCK_TRAMP];
compat_int_t rf_sar;
};
-#define COMPAT_SIGRETURN_TRAMP 3
+#define COMPAT_SIGRETURN_TRAMP 4
#define COMPAT_SIGRESTARTBLOCK_TRAMP 5
#define COMPAT_TRAMP_SIZE (COMPAT_SIGRETURN_TRAMP + \
COMPAT_SIGRESTARTBLOCK_TRAMP)
*
* TODO: Userspace stacktrace (CONFIG_USER_STACKTRACE_SUPPORT)
*/
+#include <linux/kernel.h>
#include <linux/stacktrace.h>
#include <asm/unwind.h>
446 common landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 common process_mrelease sys_process_mrelease
+449 common futex_waitv sys_futex_waitv
{
. = KERNEL_BINARY_TEXT_START;
+ _stext = .; /* start of kernel text, includes init code & data */
+
__init_begin = .;
HEAD_TEXT_SECTION
MLONGCALL_DISCARD(INIT_TEXT_SECTION(8))
/* freed after init ends here */
_text = .; /* Text and read-only data */
- _stext = .;
MLONGCALL_KEEP(INIT_TEXT_SECTION(8))
.text ALIGN(PAGE_SIZE) : {
TEXT_TEXT
obj-$(CONFIG_PERF_EVENTS) += perf/
obj-$(CONFIG_KEXEC_CORE) += kexec/
obj-$(CONFIG_KEXEC_FILE) += purgatory/
+
+# for cleaning
+subdir- += boot
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture.
+# architecture-specific flags and dependencies.
#
# This file is subject to the terms and conditions of the GNU General Public
# License. See the file "COPYING" in the main directory of this archive
sh -x $(srctree)/$(boot)/install.sh "$(KERNELRELEASE)" vmlinux \
System.map "$(INSTALL_PATH)"
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
ifeq ($(KBUILD_EXTMOD),)
# We need to generate vdso-offsets.h before compiling certain files in kernel/.
# In order to do that, we should use the archprepare target, but we can't since
CONFIG_ENCRYPTED_KEYS=y
CONFIG_SECURITY=y
CONFIG_HARDENED_USERCOPY=y
-# CONFIG_HARDENED_USERCOPY_FALLBACK is not set
CONFIG_HARDENED_USERCOPY_PAGESPAN=y
CONFIG_FORTIFY_SOURCE=y
CONFIG_SECURITY_LOCKDOWN_LSM=y
#ifdef CONFIG_PM
void (*iommu_restore)(void);
#endif
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
unsigned long (*memory_block_size)(void);
#endif
#endif /* CONFIG_PPC64 */
void eeh_sysfs_add_device(struct pci_dev *pdev);
void eeh_sysfs_remove_device(struct pci_dev *pdev);
-static inline const char *eeh_driver_name(struct pci_dev *pdev)
-{
- return (pdev && pdev->driver) ? pdev->driver->name : "<null>";
-}
-
#endif /* CONFIG_EEH */
#define PCI_BUSNO(bdfn) ((bdfn >> 8) & 0xff)
#include <linux/elf.h>
#include <linux/uaccess.h>
-#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
-
#include <asm-generic/sections.h>
-extern bool init_mem_is_free;
-
-static inline int arch_is_kernel_initmem_freed(unsigned long addr)
-{
- if (!init_mem_is_free)
- return 0;
-
- return addr >= (unsigned long)__init_begin &&
- addr < (unsigned long)__init_end;
-}
-
extern char __head_end[];
#ifdef __powerpc64__
}
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- memcpy(®s->gpr[3], args, 6 * sizeof(args[0]));
-
- /* Also copy the first argument into orig_gpr3 */
- regs->orig_gpr3 = args[0];
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
if (is_32bit_task())
# Force dependency (incbin is bad)
$(obj)/vdso32_wrapper.o : $(obj)/vdso32/vdso32.so.dbg
$(obj)/vdso64_wrapper.o : $(obj)/vdso64/vdso64.so.dbg
+
+# for cleaning
+subdir- += vdso32 vdso64
cpufeatures_setup_finished();
- memblock_free(__pa(dt_cpu_features),
- sizeof(struct dt_cpu_feature)*nr_dt_cpu_features);
+ memblock_free(dt_cpu_features,
+ sizeof(struct dt_cpu_feature) * nr_dt_cpu_features);
return 0;
}
return ret;
}
+static inline const char *eeh_driver_name(struct pci_dev *pdev)
+{
+ if (pdev)
+ return dev_driver_string(&pdev->dev);
+
+ return "<null>";
+}
+
/**
* eeh_dev_check_failure - Check if all 1's data is due to EEH slot freeze
* @edev: eeh device
*/
static inline struct pci_driver *eeh_pcid_get(struct pci_dev *pdev)
{
- if (!pdev || !pdev->driver)
+ if (!pdev || !pdev->dev.driver)
return NULL;
- if (!try_module_get(pdev->driver->driver.owner))
+ if (!try_module_get(pdev->dev.driver->owner))
return NULL;
- return pdev->driver;
+ return to_pci_driver(pdev->dev.driver);
}
/**
*/
static inline void eeh_pcid_put(struct pci_dev *pdev)
{
- if (!pdev || !pdev->driver)
+ if (!pdev || !pdev->dev.driver)
return;
- module_put(pdev->driver->driver.owner);
+ module_put(pdev->dev.driver->owner);
}
/**
#ifdef CONFIG_PIN_TLB_DATA
LOAD_REG_IMMEDIATE(r6, PAGE_OFFSET)
LOAD_REG_IMMEDIATE(r7, MI_SVALID | MI_PS8MEG | _PMD_ACCESSED)
+ li r8, 0
#ifdef CONFIG_PIN_TLB_IMMR
li r0, 3
#else
mtctr r0
cmpwi r4, 0
beq 4f
- LOAD_REG_IMMEDIATE(r8, 0xf0 | _PAGE_RO | _PAGE_SPS | _PAGE_SH | _PAGE_PRESENT)
LOAD_REG_ADDR(r9, _sinittext)
2: ori r0, r6, MD_EVALID
+ ori r12, r8, 0xf0 | _PAGE_RO | _PAGE_SPS | _PAGE_SH | _PAGE_PRESENT
mtspr SPRN_MD_CTR, r5
mtspr SPRN_MD_EPN, r0
mtspr SPRN_MD_TWC, r7
- mtspr SPRN_MD_RPN, r8
+ mtspr SPRN_MD_RPN, r12
addi r5, r5, 0x100
addis r6, r6, SZ_8M@h
addis r8, r8, SZ_8M@h
cmplw r6, r9
bdnzt lt, 2b
-
-4: LOAD_REG_IMMEDIATE(r8, 0xf0 | _PAGE_DIRTY | _PAGE_SPS | _PAGE_SH | _PAGE_PRESENT)
+4:
2: ori r0, r6, MD_EVALID
+ ori r12, r8, 0xf0 | _PAGE_DIRTY | _PAGE_SPS | _PAGE_SH | _PAGE_PRESENT
mtspr SPRN_MD_CTR, r5
mtspr SPRN_MD_EPN, r0
mtspr SPRN_MD_TWC, r7
- mtspr SPRN_MD_RPN, r8
+ mtspr SPRN_MD_RPN, r12
addi r5, r5, 0x100
addis r6, r6, SZ_8M@h
addis r8, r8, SZ_8M@h
#endif
#if defined(CONFIG_PIN_TLB_IMMR) || defined(CONFIG_PIN_TLB_DATA)
lis r0, (MD_RSV4I | MD_TWAM)@h
- mtspr SPRN_MI_CTR, r0
+ mtspr SPRN_MD_CTR, r0
#endif
mtspr SPRN_SRR1, r10
mtspr SPRN_SRR0, r11
new_ptrs_size = sizeof(struct paca_struct *) * nr_cpu_ids;
if (new_ptrs_size < paca_ptrs_size)
- memblock_free(__pa(paca_ptrs) + new_ptrs_size,
- paca_ptrs_size - new_ptrs_size);
+ memblock_phys_free(__pa(paca_ptrs) + new_ptrs_size,
+ paca_ptrs_size - new_ptrs_size);
paca_nr_cpu_ids = nr_cpu_ids;
paca_ptrs_size = new_ptrs_size;
#ifdef CONFIG_PPC_BOOK3S_64
if (early_radix_enabled()) {
/* Ugly fixup, see new_slb_shadow() */
- memblock_free(__pa(paca_ptrs[boot_cpuid]->slb_shadow_ptr),
- sizeof(struct slb_shadow));
+ memblock_phys_free(__pa(paca_ptrs[boot_cpuid]->slb_shadow_ptr),
+ sizeof(struct slb_shadow));
paca_ptrs[boot_cpuid]->slb_shadow_ptr = NULL;
}
#endif
ppc_md.pcibios_bus_add_device(dev);
}
-int pcibios_add_device(struct pci_dev *dev)
+int pcibios_device_add(struct pci_dev *dev)
{
struct irq_domain *d;
set_hard_smp_processor_id(cpu, cpu_to_phys_id[cpu]);
}
- memblock_free(__pa(cpu_to_phys_id), nr_cpu_ids * sizeof(u32));
+ memblock_free(cpu_to_phys_id, nr_cpu_ids * sizeof(u32));
cpu_to_phys_id = NULL;
}
#endif
static void __init pcpu_free_bootmem(void *ptr, size_t size)
{
- memblock_free(__pa(ptr), size);
+ memblock_free(ptr, size);
}
static int pcpu_cpu_distance(unsigned int from, unsigned int to)
}
#endif
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
unsigned long memory_block_size_bytes(void)
{
if (ppc_md.memory_block_size)
return __get_user(dst->sig[0], (u64 __user *)&src->sig[0]);
}
-#define unsafe_get_user_sigset(dst, src, label) \
- unsafe_get_user((dst)->sig[0], (u64 __user *)&(src)->sig[0], label)
+#define unsafe_get_user_sigset(dst, src, label) do { \
+ sigset_t *__dst = dst; \
+ const sigset_t __user *__src = src; \
+ int i; \
+ \
+ for (i = 0; i < _NSIG_WORDS; i++) \
+ unsafe_get_user(__dst->sig[i], &__src->sig[i], label); \
+} while (0)
#ifdef CONFIG_VSX
extern unsigned long copy_vsx_to_user(void __user *to,
* or if another thread unmaps the region containing the context.
* We kill the task with a SIGSEGV in this situation.
*/
- if (do_setcontext(new_ctx, regs, 0))
- do_exit(SIGSEGV);
+ if (do_setcontext(new_ctx, regs, 0)) {
+ force_exit_sig(SIGSEGV);
+ return -EFAULT;
+ }
set_thread_flag(TIF_RESTOREALL);
return 0;
* We kill the task with a SIGSEGV in this situation.
*/
- if (__get_user_sigset(&set, &new_ctx->uc_sigmask))
- do_exit(SIGSEGV);
+ if (__get_user_sigset(&set, &new_ctx->uc_sigmask)) {
+ force_exit_sig(SIGSEGV);
+ return -EFAULT;
+ }
set_current_blocked(&set);
if (!user_read_access_begin(new_ctx, ctx_size))
return -EFAULT;
if (__unsafe_restore_sigcontext(current, NULL, 0, &new_ctx->uc_mcontext)) {
user_read_access_end();
- do_exit(SIGSEGV);
+ force_exit_sig(SIGSEGV);
+ return -EFAULT;
}
user_read_access_end();
if (sysctl_hardlockup_all_cpu_backtrace)
trigger_allbutself_cpu_backtrace();
+ /*
+ * Force flush any remote buffers that might be stuck in IRQ context
+ * and therefore could not run their irq_work.
+ */
+ printk_trigger_flush();
+
if (hardlockup_panic)
nmi_panic(NULL, "Hard LOCKUP");
.globl hcall_real_table_end
hcall_real_table_end:
-_GLOBAL(kvmppc_h_set_xdabr)
+_GLOBAL_TOC(kvmppc_h_set_xdabr)
EXPORT_SYMBOL_GPL(kvmppc_h_set_xdabr)
andi. r0, r5, DABRX_USER | DABRX_KERNEL
beq 6f
6: li r3, H_PARAMETER
blr
-_GLOBAL(kvmppc_h_set_dabr)
+_GLOBAL_TOC(kvmppc_h_set_dabr)
EXPORT_SYMBOL_GPL(kvmppc_h_set_dabr)
li r5, DABRX_USER | DABRX_KERNEL
3:
gpa, 0, page_shift);
if (ret == U_SUCCESS)
- *mig.dst = migrate_pfn(pfn) | MIGRATE_PFN_LOCKED;
+ *mig.dst = migrate_pfn(pfn);
else {
unlock_page(dpage);
__free_page(dpage);
}
}
- *mig.dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED;
+ *mig.dst = migrate_pfn(page_to_pfn(dpage));
migrate_vma_pages(&mig);
out_finalize:
migrate_vma_finalize(&mig);
* implementations just count online CPUs.
*/
if (hv_enabled)
- r = num_present_cpus();
+ r = min_t(unsigned int, num_present_cpus(), KVM_MAX_VCPUS);
else
- r = num_online_cpus();
+ r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
break;
case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o
CFLAGS_xor_vmx.o += -maltivec $(call cc-option,-mabi=altivec)
+# Enable <altivec.h>
+CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include)
obj-$(CONFIG_PPC64) += $(obj64-y)
m->hstate = hstate;
return 1;
}
+
+bool __init hugetlb_node_alloc_supported(void)
+{
+ return false;
+}
#endif
-int __init alloc_bootmem_huge_page(struct hstate *h)
+int __init alloc_bootmem_huge_page(struct hstate *h, int nid)
{
#ifdef CONFIG_PPC_BOOK3S_64
if (firmware_has_feature(FW_FEATURE_LPAR) && !radix_enabled())
return pseries_alloc_bootmem_huge_page(h);
#endif
- return __alloc_bootmem_huge_page(h);
+ return __alloc_bootmem_huge_page(h, nid);
}
#ifndef CONFIG_PPC_BOOK3S_64
pr_warn("KASLR: No safe seed for randomizing the kernel base.\n");
ram = min_t(phys_addr_t, __max_low_memory, size);
- ram = map_mem_in_cams(ram, CONFIG_LOWMEM_CAM_NUM, true, false);
+ ram = map_mem_in_cams(ram, CONFIG_LOWMEM_CAM_NUM, true, true);
linear_sz = min_t(unsigned long, ram, SZ_512M);
/* If the linear size is smaller than 64M, do not randmize */
if (map)
linear_map_top = map_mem_in_cams(linear_map_top,
- num_cams, true, true);
+ num_cams, false, true);
}
#endif
num_cams = (mfspr(SPRN_TLB1CFG) & TLBnCFG_N_ENTRY) / 4;
linear_sz = map_mem_in_cams(first_memblock_size, num_cams,
- false, true);
+ true, true);
ppc64_rma_size = min_t(u64, linear_sz, 0x40000000);
} else
{
int i, j;
struct device_node *root;
- const __u8 *numa_dist_table;
+ const __u8 *form2_distances;
const __be32 *numa_lookup_index;
- int numa_dist_table_length;
+ int form2_distances_length;
int max_numa_index, distance_index;
if (firmware_has_feature(FW_FEATURE_OPAL))
max_numa_index = of_read_number(&numa_lookup_index[0], 1);
/* first element of the array is the size and is encode-int */
- numa_dist_table = of_get_property(root, "ibm,numa-distance-table", NULL);
- numa_dist_table_length = of_read_number((const __be32 *)&numa_dist_table[0], 1);
+ form2_distances = of_get_property(root, "ibm,numa-distance-table", NULL);
+ form2_distances_length = of_read_number((const __be32 *)&form2_distances[0], 1);
/* Skip the size which is encoded int */
- numa_dist_table += sizeof(__be32);
+ form2_distances += sizeof(__be32);
- pr_debug("numa_dist_table_len = %d, numa_dist_indexes_len = %d\n",
- numa_dist_table_length, max_numa_index);
+ pr_debug("form2_distances_len = %d, numa_dist_indexes_len = %d\n",
+ form2_distances_length, max_numa_index);
for (i = 0; i < max_numa_index; i++)
/* +1 skip the max_numa_index in the property */
numa_id_index_table[i] = of_read_number(&numa_lookup_index[i + 1], 1);
- if (numa_dist_table_length != max_numa_index * max_numa_index) {
+ if (form2_distances_length != max_numa_index * max_numa_index) {
WARN(1, "Wrong NUMA distance information\n");
- /* consider everybody else just remote. */
- for (i = 0; i < max_numa_index; i++) {
- for (j = 0; j < max_numa_index; j++) {
- int nodeA = numa_id_index_table[i];
- int nodeB = numa_id_index_table[j];
-
- if (nodeA == nodeB)
- numa_distance_table[nodeA][nodeB] = LOCAL_DISTANCE;
- else
- numa_distance_table[nodeA][nodeB] = REMOTE_DISTANCE;
- }
- }
+ form2_distances = NULL; // don't use it
}
-
distance_index = 0;
for (i = 0; i < max_numa_index; i++) {
for (j = 0; j < max_numa_index; j++) {
int nodeA = numa_id_index_table[i];
int nodeB = numa_id_index_table[j];
-
- numa_distance_table[nodeA][nodeB] = numa_dist_table[distance_index++];
- pr_debug("dist[%d][%d]=%d ", nodeA, nodeB, numa_distance_table[nodeA][nodeB]);
+ int dist;
+
+ if (form2_distances)
+ dist = form2_distances[distance_index++];
+ else if (nodeA == nodeB)
+ dist = LOCAL_DISTANCE;
+ else
+ dist = REMOTE_DISTANCE;
+ numa_distance_table[nodeA][nodeB] = dist;
+ pr_debug("dist[%d][%d]=%d ", nodeA, nodeB, dist);
}
}
+
of_node_put(root);
}
#include <mm/mmu_decl.h>
-extern char etext[], _stext[], _sinittext[], _einittext[];
-
static u8 early_fixmap_pagetable[FIXMAP_PTE_SIZE] __page_aligned_data;
notrace void __init early_ioremap_init(void)
{
unsigned long v, s;
phys_addr_t p;
- int ktext;
+ bool ktext;
s = offset;
v = PAGE_OFFSET + s;
p = memstart_addr + s;
for (; s < top; s += PAGE_SIZE) {
- ktext = ((char *)v >= _stext && (char *)v < etext) ||
- ((char *)v >= _sinittext && (char *)v < _einittext);
+ ktext = core_kernel_text(v);
map_kernel_page(v, p, ktext ? PAGE_KERNEL_TEXT : PAGE_KERNEL);
v += PAGE_SIZE;
p += PAGE_SIZE;
static int mcu_remove(struct i2c_client *client)
{
struct mcu *mcu = i2c_get_clientdata(client);
- int ret;
kthread_stop(shutdown_thread);
int pos;
u32 val;
- pos = find_dvsec_from_pos(dev, OCXL_DVSEC_FUNC_ID, 0);
+ pos = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_IBM,
+ OCXL_DVSEC_FUNC_ID);
if (!pos)
return -ESRCH;
if (!phb->hose) {
pr_err(" Can't allocate PCI controller for %pOF\n",
np);
- memblock_free(__pa(phb), sizeof(struct pnv_phb));
+ memblock_free(phb, sizeof(struct pnv_phb));
return;
}
* to "new_size", calculated above. Implementing this is a convoluted process
* which requires several hooks in the PCI core:
*
- * 1. In pcibios_add_device() we call pnv_pci_ioda_fixup_iov().
+ * 1. In pcibios_device_add() we call pnv_pci_ioda_fixup_iov().
*
* At this point the device has been probed and the device's BARs are sized,
* but no resource allocations have been done. The SR-IOV BARs are sized
}
#endif /* CONFIG_KEXEC_CORE */
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
static unsigned long pnv_memory_block_size(void)
{
/*
#ifdef CONFIG_KEXEC_CORE
.kexec_cpu_down = pnv_kexec_cpu_down,
#endif
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
.memory_block_size = pnv_memory_block_size,
#endif
};
phys_addr_t max_addr = memory_hotplug_max();
struct device_node *memory;
- /*
- * The "ibm,pmemory" can appear anywhere in the address space.
- * Assuming it is still backed by page structs, set the upper limit
- * for the huge DMA window as MAX_PHYSMEM_BITS.
- */
- if (of_find_node_by_type(NULL, "ibm,pmemory"))
- return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
- (phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
-
for_each_node_by_type(memory, "memory") {
unsigned long start, size;
int n_mem_addr_cells, n_mem_size_cells, len;
u32 ddw_avail[DDW_APPLICABLE_SIZE];
struct dma_win *window;
struct property *win64;
- bool ddw_enabled = false;
struct failed_ddw_pdn *fpdn;
bool default_win_removed = false, direct_mapping = false;
bool pmem_present;
if (find_existing_ddw(pdn, &dev->dev.archdata.dma_offset, &len)) {
direct_mapping = (len >= max_ram_len);
- ddw_enabled = true;
goto out_unlock;
}
len = order_base_2(query.largest_available_block << page_shift);
win_name = DMA64_PROPNAME;
} else {
- direct_mapping = true;
- win_name = DIRECT64_PROPNAME;
+ direct_mapping = !default_win_removed ||
+ (len == MAX_PHYSMEM_BITS) ||
+ (!pmem_present && (len == max_ram_len));
+ win_name = direct_mapping ? DIRECT64_PROPNAME : DMA64_PROPNAME;
}
ret = create_ddw(dev, ddw_avail, &create, page_shift, len);
dev_info(&dev->dev, "failed to map DMA window for %pOF: %d\n",
dn, ret);
- /* Make sure to clean DDW if any TCE was set*/
- clean_dma_window(pdn, win64->value);
+ /* Make sure to clean DDW if any TCE was set*/
+ clean_dma_window(pdn, win64->value);
goto out_del_list;
}
} else {
spin_unlock(&dma_win_list_lock);
dev->dev.archdata.dma_offset = win_addr;
- ddw_enabled = true;
goto out_unlock;
out_del_list:
* as RAM, then we failed to create a window to cover persistent
* memory and need to set the DMA limit.
*/
- if (pmem_present && ddw_enabled && direct_mapping && len == max_ram_len)
+ if (pmem_present && direct_mapping && len == max_ram_len)
dev->dev.bus_dma_limit = dev->dev.archdata.dma_offset + (1ULL << len);
- return ddw_enabled && direct_mapping;
+ return direct_mapping;
}
static void pci_dma_dev_setup_pSeriesLP(struct pci_dev *dev)
.machine_kexec = pSeries_machine_kexec,
.kexec_cpu_down = pseries_kexec_cpu_down,
#endif
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
.memory_block_size = pseries_memory_block_size,
#endif
};
return;
- memblock_free_early(__pa(vstart),
- PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT));
+ memblock_free(vstart, PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT));
panic("SVM: Cannot allocate SWIOTLB buffer");
}
bool
select PPC_SMP_MUXED_IPI
select HARDIRQS_SW_RESEND
- select IRQ_DOMAIN_NOMAP
config PPC_XIVE_NATIVE
bool
static void __init xive_init_host(struct device_node *np)
{
- xive_irq_domain = irq_domain_add_nomap(np, XIVE_MAX_IRQ,
- &xive_irq_domain_ops, NULL);
+ xive_irq_domain = irq_domain_add_tree(np, &xive_irq_domain_ops, NULL);
if (WARN_ON(xive_irq_domain == NULL))
return;
irq_set_default_host(xive_irq_domain);
obj-y += kernel/ mm/ net/
obj-$(CONFIG_BUILTIN_DTB) += boot/dts/
+
+# for cleaning
+subdir- += boot
select GENERIC_SCHED_CLOCK
select GENERIC_SMP_IDLE_THREAD
select GENERIC_TIME_VSYSCALL if MMU && 64BIT
+ select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL
select HAVE_ARCH_JUMP_LABEL_RELATIVE if !XIP_KERNEL
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
#
# This file is subject to the terms and conditions of the GNU General Public
# License. See the file "COPYING" in the main directory of this archive
vdso_install:
$(Q)$(MAKE) $(build)=arch/riscv/kernel/vdso $@
+ifeq ($(KBUILD_EXTMOD),)
ifeq ($(CONFIG_MMU),y)
prepare: vdso_prepare
vdso_prepare: prepare0
$(Q)$(MAKE) $(build)=arch/riscv/kernel/vdso include/generated/vdso-offsets.h
endif
+endif
ifneq ($(CONFIG_XIP_KERNEL),y)
ifeq ($(CONFIG_RISCV_M_MODE)$(CONFIG_SOC_CANAAN),yy)
$(CONFIG_SHELL) $(srctree)/$(boot)/install.sh $(KERNELRELEASE) \
$(boot)/$(install-image) System.map "$(INSTALL_PATH)"
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
+PHONY += rv32_randconfig
+rv32_randconfig:
+ $(Q)$(MAKE) KCONFIG_ALLCONFIG=$(srctree)/arch/riscv/configs/32-bit.config \
+ -f $(srctree)/Makefile randconfig
+
+PHONY += rv64_randconfig
+rv64_randconfig:
+ $(Q)$(MAKE) KCONFIG_ALLCONFIG=$(srctree)/arch/riscv/configs/64-bit.config \
+ -f $(srctree)/Makefile randconfig
#define RTCCLK_FREQ 1000000
/ {
- #address-cells = <2>;
- #size-cells = <2>;
model = "Microchip PolarFire-SoC Icicle Kit";
- compatible = "microchip,mpfs-icicle-kit";
+ compatible = "microchip,mpfs-icicle-kit", "microchip,mpfs";
aliases {
ethernet0 = &emac1;
reg = <0x0 0x80000000 0x0 0x40000000>;
clocks = <&clkcfg 26>;
};
-
- soc {
- };
};
&serial0 {
status = "okay";
};
-&sdcard {
+&mmc {
status = "okay";
+
+ bus-width = <4>;
+ disable-wp;
+ cap-sd-highspeed;
+ card-detect-delay = <200>;
+ sd-uhs-sdr12;
+ sd-uhs-sdr25;
+ sd-uhs-sdr50;
+ sd-uhs-sdr104;
};
&emac0 {
/ {
#address-cells = <2>;
#size-cells = <2>;
- model = "Microchip MPFS Icicle Kit";
- compatible = "microchip,mpfs-icicle-kit";
+ model = "Microchip PolarFire SoC";
+ compatible = "microchip,mpfs";
chosen {
};
};
clint@2000000 {
- compatible = "sifive,clint0";
+ compatible = "sifive,fu540-c000-clint", "sifive,clint0";
reg = <0x0 0x2000000 0x0 0xC000>;
interrupts-extended = <&cpu0_intc 3 &cpu0_intc 7
&cpu1_intc 3 &cpu1_intc 7
plic: interrupt-controller@c000000 {
#interrupt-cells = <1>;
- compatible = "sifive,plic-1.0.0";
+ compatible = "sifive,fu540-c000-plic", "sifive,plic-1.0.0";
reg = <0x0 0xc000000 0x0 0x4000000>;
riscv,ndev = <186>;
interrupt-controller;
status = "disabled";
};
- emmc: mmc@20008000 {
- compatible = "cdns,sd4hc";
+ /* Common node entry for emmc/sd */
+ mmc: mmc@20008000 {
+ compatible = "microchip,mpfs-sd4hc", "cdns,sd4hc";
reg = <0x0 0x20008000 0x0 0x1000>;
interrupt-parent = <&plic>;
interrupts = <88 89>;
- pinctrl-names = "default";
clocks = <&clkcfg 6>;
- bus-width = <4>;
- cap-mmc-highspeed;
- mmc-ddr-3_3v;
- max-frequency = <200000000>;
- non-removable;
- no-sd;
- no-sdio;
- voltage-ranges = <3300 3300>;
- status = "disabled";
- };
-
- sdcard: sdhc@20008000 {
- compatible = "cdns,sd4hc";
- reg = <0x0 0x20008000 0x0 0x1000>;
- interrupt-parent = <&plic>;
- interrupts = <88>;
- pinctrl-names = "default";
- clocks = <&clkcfg 6>;
- bus-width = <4>;
- disable-wp;
- cap-sd-highspeed;
- card-detect-delay = <200>;
- sd-uhs-sdr12;
- sd-uhs-sdr25;
- sd-uhs-sdr50;
- sd-uhs-sdr104;
max-frequency = <200000000>;
status = "disabled";
};
ranges;
plic0: interrupt-controller@c000000 {
#interrupt-cells = <1>;
- compatible = "sifive,plic-1.0.0";
+ compatible = "sifive,fu540-c000-plic", "sifive,plic-1.0.0";
reg = <0x0 0xc000000 0x0 0x4000000>;
riscv,ndev = <53>;
interrupt-controller;
#define RTCCLK_FREQ 1000000
/ {
- #address-cells = <2>;
- #size-cells = <2>;
model = "SiFive HiFive Unleashed A00";
- compatible = "sifive,hifive-unleashed-a00", "sifive,fu540-c000";
+ compatible = "sifive,hifive-unleashed-a00", "sifive,fu540-c000",
+ "sifive,fu540";
chosen {
stdout-path = "serial0";
reg = <0x0 0x80000000 0x2 0x00000000>;
};
- soc {
- };
-
hfclk: hfclk {
#clock-cells = <0>;
compatible = "fixed-clock";
&qspi0 {
status = "okay";
flash@0 {
- compatible = "issi,is25wp256", "jedec,spi-nor";
+ compatible = "jedec,spi-nor";
reg = <0>;
spi-max-frequency = <50000000>;
m25p,fast-read;
#define RTCCLK_FREQ 1000000
/ {
- #address-cells = <2>;
- #size-cells = <2>;
model = "SiFive HiFive Unmatched A00";
compatible = "sifive,hifive-unmatched-a00", "sifive,fu740-c000",
"sifive,fu740";
reg = <0x0 0x80000000 0x4 0x00000000>;
};
- soc {
- };
-
hfclk: hfclk {
#clock-cells = <0>;
compatible = "fixed-clock";
&qspi0 {
status = "okay";
flash@0 {
- compatible = "issi,is25wp256", "jedec,spi-nor";
+ compatible = "jedec,spi-nor";
reg = <0>;
spi-max-frequency = <50000000>;
m25p,fast-read;
--- /dev/null
+CONFIG_ARCH_RV32I=y
+CONFIG_32BIT=y
--- /dev/null
+CONFIG_ARCH_RV64I=y
+CONFIG_64BIT=y
CONFIG_SOC_MICROCHIP_POLARFIRE=y
CONFIG_SMP=y
CONFIG_HOTPLUG_CPU=y
+CONFIG_VIRTUALIZATION=y
+CONFIG_KVM=m
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_GPIO_SIFIVE=y
# CONFIG_PTP_1588_CLOCK is not set
CONFIG_POWER_RESET=y
-CONFIG_DRM=y
-CONFIG_DRM_RADEON=y
-CONFIG_DRM_VIRTIO_GPU=y
+CONFIG_DRM=m
+CONFIG_DRM_RADEON=m
+CONFIG_DRM_NOUVEAU=m
+CONFIG_DRM_VIRTIO_GPU=m
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_USB=y
CONFIG_USB_XHCI_HCD=y
CONFIG_ARCH_RV32I=y
CONFIG_SMP=y
CONFIG_HOTPLUG_CPU=y
+CONFIG_VIRTUALIZATION=y
+CONFIG_KVM=m
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
#define page_to_bus(page) (page_to_phys(page))
#define phys_to_page(paddr) (pfn_to_page(phys_to_pfn(paddr)))
+#define sym_to_pfn(x) __phys_to_pfn(__pa_symbol(x))
+
#ifdef CONFIG_FLATMEM
#define pfn_valid(pfn) \
(((pfn) >= ARCH_PFN_OFFSET) && (((pfn) - ARCH_PFN_OFFSET) < max_mapnr))
#endif
#ifdef CONFIG_XIP_KERNEL
-#define XIP_OFFSET SZ_8M
+#define XIP_OFFSET SZ_32M
+#define XIP_OFFSET_MASK (SZ_32M - 1)
#else
#define XIP_OFFSET 0
#endif
#ifdef CONFIG_XIP_KERNEL
#define XIP_FIXUP(addr) ({ \
uintptr_t __a = (uintptr_t)(addr); \
- (__a >= CONFIG_XIP_PHYS_ADDR && __a < CONFIG_XIP_PHYS_ADDR + SZ_16M) ? \
+ (__a >= CONFIG_XIP_PHYS_ADDR && \
+ __a < CONFIG_XIP_PHYS_ADDR + XIP_OFFSET * 2) ? \
__a - CONFIG_XIP_PHYS_ADDR + CONFIG_PHYS_RAM_BASE - XIP_OFFSET :\
__a; \
})
memcpy(args, ®s->a1, 5 * sizeof(args[0]));
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->orig_a0 = args[0];
- args++;
- memcpy(®s->a1, args, 5 * sizeof(regs->a1));
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
#ifdef CONFIG_64BIT
#ifndef _ASM_RISCV_VDSO_H
#define _ASM_RISCV_VDSO_H
-
-/*
- * All systems with an MMU have a VDSO, but systems without an MMU don't
- * support shared libraries and therefor don't have one.
- */
-#ifdef CONFIG_MMU
-
-#include <linux/types.h>
/*
* All systems with an MMU have a VDSO, but systems without an MMU don't
* support shared libraries and therefor don't have one.
*/
#ifdef CONFIG_MMU
-#define __VVAR_PAGES 1
+#define __VVAR_PAGES 2
#ifndef __ASSEMBLY__
#include <generated/vdso-offsets.h>
#define VDSO_SYMBOL(base, name) \
(void __user *)((unsigned long)(base) + __vdso_##name##_offset)
-
-#endif /* CONFIG_MMU */
-
#endif /* !__ASSEMBLY__ */
#endif /* CONFIG_MMU */
return _vdso_data;
}
+#ifdef CONFIG_TIME_NS
+static __always_inline
+const struct vdso_data *__arch_get_timens_vdso_data(const struct vdso_data *vd)
+{
+ return _timens_data;
+}
+#endif
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_VDSO_GETTIMEOFDAY_H */
REG_L t0, _xip_fixup
add \reg, \reg, t0
.endm
+.macro XIP_FIXUP_FLASH_OFFSET reg
+ la t1, __data_loc
+ li t0, XIP_OFFSET_MASK
+ and t1, t1, t0
+ li t1, XIP_OFFSET
+ sub t0, t0, t1
+ sub \reg, \reg, t0
+.endm
_xip_fixup: .dword CONFIG_PHYS_RAM_BASE - CONFIG_XIP_PHYS_ADDR - XIP_OFFSET
#else
.macro XIP_FIXUP_OFFSET reg
.endm
+.macro XIP_FIXUP_FLASH_OFFSET reg
+.endm
#endif /* CONFIG_XIP_KERNEL */
__HEAD
la a3, hart_lottery
mv a2, a3
XIP_FIXUP_OFFSET a2
+ XIP_FIXUP_FLASH_OFFSET a3
lw t1, (a3)
amoswap.w t0, t1, (a2)
/* first time here if hart_lottery in RAM is not set */
XIP_FIXUP_OFFSET sp
#ifdef CONFIG_BUILTIN_DTB
la a0, __dtb_start
+ XIP_FIXUP_OFFSET a0
#else
mv a0, s1
#endif /* CONFIG_BUILTIN_DTB */
wait_for_interrupt();
}
-void (*pm_power_off)(void) = default_power_off;
+void (*pm_power_off)(void) = NULL;
EXPORT_SYMBOL(pm_power_off);
void machine_restart(char *cmd)
void machine_halt(void)
{
- pm_power_off();
+ if (pm_power_off != NULL)
+ pm_power_off();
+ else
+ default_power_off();
}
void machine_power_off(void)
{
- pm_power_off();
+ if (pm_power_off != NULL)
+ pm_power_off();
+ else
+ default_power_off();
}
/* Clean-up any unused pre-allocated resources */
if (res_idx >= 0)
- memblock_free(__pa(mem_res), (res_idx + 1) * sizeof(*mem_res));
+ memblock_free(mem_res, (res_idx + 1) * sizeof(*mem_res));
return;
error:
/* Better an empty resource tree than an inconsistent one */
release_child_resources(&iomem_resource);
- memblock_free(__pa(mem_res), mem_res_sz);
+ memblock_free(mem_res, mem_res_sz);
}
#include <linux/err.h>
#include <asm/page.h>
#include <asm/vdso.h>
+#include <linux/time_namespace.h>
#ifdef CONFIG_GENERIC_TIME_VSYSCALL
#include <vdso/datapage.h>
enum vvar_pages {
VVAR_DATA_PAGE_OFFSET,
+ VVAR_TIMENS_PAGE_OFFSET,
VVAR_NR_PAGES,
};
#define VVAR_SIZE (VVAR_NR_PAGES << PAGE_SHIFT)
-static unsigned int vdso_pages __ro_after_init;
-static struct page **vdso_pagelist __ro_after_init;
-
/*
* The vDSO data page.
*/
} vdso_data_store __page_aligned_data;
struct vdso_data *vdso_data = &vdso_data_store.data;
-static int __init vdso_init(void)
+struct __vdso_info {
+ const char *name;
+ const char *vdso_code_start;
+ const char *vdso_code_end;
+ unsigned long vdso_pages;
+ /* Data Mapping */
+ struct vm_special_mapping *dm;
+ /* Code Mapping */
+ struct vm_special_mapping *cm;
+};
+
+static struct __vdso_info vdso_info __ro_after_init = {
+ .name = "vdso",
+ .vdso_code_start = vdso_start,
+ .vdso_code_end = vdso_end,
+};
+
+static int vdso_mremap(const struct vm_special_mapping *sm,
+ struct vm_area_struct *new_vma)
+{
+ current->mm->context.vdso = (void *)new_vma->vm_start;
+
+ return 0;
+}
+
+static int __init __vdso_init(void)
{
unsigned int i;
+ struct page **vdso_pagelist;
+ unsigned long pfn;
- vdso_pages = (vdso_end - vdso_start) >> PAGE_SHIFT;
- vdso_pagelist =
- kcalloc(vdso_pages + VVAR_NR_PAGES, sizeof(struct page *), GFP_KERNEL);
- if (unlikely(vdso_pagelist == NULL)) {
- pr_err("vdso: pagelist allocation failed\n");
- return -ENOMEM;
+ if (memcmp(vdso_info.vdso_code_start, "\177ELF", 4)) {
+ pr_err("vDSO is not a valid ELF object!\n");
+ return -EINVAL;
}
- for (i = 0; i < vdso_pages; i++) {
- struct page *pg;
+ vdso_info.vdso_pages = (
+ vdso_info.vdso_code_end -
+ vdso_info.vdso_code_start) >>
+ PAGE_SHIFT;
+
+ vdso_pagelist = kcalloc(vdso_info.vdso_pages,
+ sizeof(struct page *),
+ GFP_KERNEL);
+ if (vdso_pagelist == NULL)
+ return -ENOMEM;
+
+ /* Grab the vDSO code pages. */
+ pfn = sym_to_pfn(vdso_info.vdso_code_start);
+
+ for (i = 0; i < vdso_info.vdso_pages; i++)
+ vdso_pagelist[i] = pfn_to_page(pfn + i);
+
+ vdso_info.cm->pages = vdso_pagelist;
+
+ return 0;
+}
+
+#ifdef CONFIG_TIME_NS
+struct vdso_data *arch_get_vdso_data(void *vvar_page)
+{
+ return (struct vdso_data *)(vvar_page);
+}
+
+/*
+ * The vvar mapping contains data for a specific time namespace, so when a task
+ * changes namespace we must unmap its vvar data for the old namespace.
+ * Subsequent faults will map in data for the new namespace.
+ *
+ * For more details see timens_setup_vdso_data().
+ */
+int vdso_join_timens(struct task_struct *task, struct time_namespace *ns)
+{
+ struct mm_struct *mm = task->mm;
+ struct vm_area_struct *vma;
+
+ mmap_read_lock(mm);
- pg = virt_to_page(vdso_start + (i << PAGE_SHIFT));
- vdso_pagelist[i] = pg;
+ for (vma = mm->mmap; vma; vma = vma->vm_next) {
+ unsigned long size = vma->vm_end - vma->vm_start;
+
+ if (vma_is_special_mapping(vma, vdso_info.dm))
+ zap_page_range(vma, vma->vm_start, size);
}
- vdso_pagelist[i] = virt_to_page(vdso_data);
+ mmap_read_unlock(mm);
return 0;
}
+
+static struct page *find_timens_vvar_page(struct vm_area_struct *vma)
+{
+ if (likely(vma->vm_mm == current->mm))
+ return current->nsproxy->time_ns->vvar_page;
+
+ /*
+ * VM_PFNMAP | VM_IO protect .fault() handler from being called
+ * through interfaces like /proc/$pid/mem or
+ * process_vm_{readv,writev}() as long as there's no .access()
+ * in special_mapping_vmops.
+ * For more details check_vma_flags() and __access_remote_vm()
+ */
+ WARN(1, "vvar_page accessed remotely");
+
+ return NULL;
+}
+#else
+static struct page *find_timens_vvar_page(struct vm_area_struct *vma)
+{
+ return NULL;
+}
+#endif
+
+static vm_fault_t vvar_fault(const struct vm_special_mapping *sm,
+ struct vm_area_struct *vma, struct vm_fault *vmf)
+{
+ struct page *timens_page = find_timens_vvar_page(vma);
+ unsigned long pfn;
+
+ switch (vmf->pgoff) {
+ case VVAR_DATA_PAGE_OFFSET:
+ if (timens_page)
+ pfn = page_to_pfn(timens_page);
+ else
+ pfn = sym_to_pfn(vdso_data);
+ break;
+#ifdef CONFIG_TIME_NS
+ case VVAR_TIMENS_PAGE_OFFSET:
+ /*
+ * If a task belongs to a time namespace then a namespace
+ * specific VVAR is mapped with the VVAR_DATA_PAGE_OFFSET and
+ * the real VVAR page is mapped with the VVAR_TIMENS_PAGE_OFFSET
+ * offset.
+ * See also the comment near timens_setup_vdso_data().
+ */
+ if (!timens_page)
+ return VM_FAULT_SIGBUS;
+ pfn = sym_to_pfn(vdso_data);
+ break;
+#endif /* CONFIG_TIME_NS */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+
+ return vmf_insert_pfn(vma, vmf->address, pfn);
+}
+
+enum rv_vdso_map {
+ RV_VDSO_MAP_VVAR,
+ RV_VDSO_MAP_VDSO,
+};
+
+static struct vm_special_mapping rv_vdso_maps[] __ro_after_init = {
+ [RV_VDSO_MAP_VVAR] = {
+ .name = "[vvar]",
+ .fault = vvar_fault,
+ },
+ [RV_VDSO_MAP_VDSO] = {
+ .name = "[vdso]",
+ .mremap = vdso_mremap,
+ },
+};
+
+static int __init vdso_init(void)
+{
+ vdso_info.dm = &rv_vdso_maps[RV_VDSO_MAP_VVAR];
+ vdso_info.cm = &rv_vdso_maps[RV_VDSO_MAP_VDSO];
+
+ return __vdso_init();
+}
arch_initcall(vdso_init);
-int arch_setup_additional_pages(struct linux_binprm *bprm,
- int uses_interp)
+static int __setup_additional_pages(struct mm_struct *mm,
+ struct linux_binprm *bprm,
+ int uses_interp)
{
- struct mm_struct *mm = current->mm;
- unsigned long vdso_base, vdso_len;
- int ret;
+ unsigned long vdso_base, vdso_text_len, vdso_mapping_len;
+ void *ret;
BUILD_BUG_ON(VVAR_NR_PAGES != __VVAR_PAGES);
- vdso_len = (vdso_pages + VVAR_NR_PAGES) << PAGE_SHIFT;
+ vdso_text_len = vdso_info.vdso_pages << PAGE_SHIFT;
+ /* Be sure to map the data page */
+ vdso_mapping_len = vdso_text_len + VVAR_SIZE;
- if (mmap_write_lock_killable(mm))
- return -EINTR;
-
- vdso_base = get_unmapped_area(NULL, 0, vdso_len, 0, 0);
+ vdso_base = get_unmapped_area(NULL, 0, vdso_mapping_len, 0, 0);
if (IS_ERR_VALUE(vdso_base)) {
- ret = vdso_base;
- goto end;
+ ret = ERR_PTR(vdso_base);
+ goto up_fail;
}
- mm->context.vdso = NULL;
- ret = install_special_mapping(mm, vdso_base, VVAR_SIZE,
- (VM_READ | VM_MAYREAD), &vdso_pagelist[vdso_pages]);
- if (unlikely(ret))
- goto end;
+ ret = _install_special_mapping(mm, vdso_base, VVAR_SIZE,
+ (VM_READ | VM_MAYREAD | VM_PFNMAP), vdso_info.dm);
+ if (IS_ERR(ret))
+ goto up_fail;
+ vdso_base += VVAR_SIZE;
+ mm->context.vdso = (void *)vdso_base;
ret =
- install_special_mapping(mm, vdso_base + VVAR_SIZE,
- vdso_pages << PAGE_SHIFT,
+ _install_special_mapping(mm, vdso_base, vdso_text_len,
(VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC),
- vdso_pagelist);
+ vdso_info.cm);
- if (unlikely(ret))
- goto end;
+ if (IS_ERR(ret))
+ goto up_fail;
- /*
- * Put vDSO base into mm struct. We need to do this before calling
- * install_special_mapping or the perf counter mmap tracking code
- * will fail to recognise it as a vDSO (since arch_vma_name fails).
- */
- mm->context.vdso = (void *)vdso_base + VVAR_SIZE;
+ return 0;
-end:
- mmap_write_unlock(mm);
- return ret;
+up_fail:
+ mm->context.vdso = NULL;
+ return PTR_ERR(ret);
}
-const char *arch_vma_name(struct vm_area_struct *vma)
+int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
{
- if (vma->vm_mm && (vma->vm_start == (long)vma->vm_mm->context.vdso))
- return "[vdso]";
- if (vma->vm_mm && (vma->vm_start ==
- (long)vma->vm_mm->context.vdso - VVAR_SIZE))
- return "[vdso_data]";
- return NULL;
+ struct mm_struct *mm = current->mm;
+ int ret;
+
+ if (mmap_write_lock_killable(mm))
+ return -EINTR;
+
+ ret = __setup_additional_pages(mm, bprm, uses_interp);
+ mmap_write_unlock(mm);
+
+ return ret;
}
SECTIONS
{
PROVIDE(_vdso_data = . - __VVAR_PAGES * PAGE_SIZE);
+#ifdef CONFIG_TIME_NS
+ PROVIDE(_timens_data = _vdso_data + PAGE_SIZE);
+#endif
. = SIZEOF_HEADERS;
.hash : { *(.hash) } :text
/*
* From this point, stuff is considered writable and will be copied to RAM
*/
- __data_loc = ALIGN(16); /* location in file */
- . = LOAD_OFFSET + XIP_OFFSET; /* location in memory */
+ __data_loc = ALIGN(PAGE_SIZE); /* location in file */
+ . = KERNEL_LINK_ADDR + XIP_OFFSET; /* location in memory */
+
+#undef LOAD_OFFSET
+#define LOAD_OFFSET (KERNEL_LINK_ADDR + XIP_OFFSET - (__data_loc & XIP_OFFSET_MASK))
_sdata = .; /* Start of data section */
_data = .;
KEEP(*(__soc_builtin_dtb_table))
__soc_builtin_dtb_table_end = .;
}
- PERCPU_SECTION(L1_CACHE_BYTES)
. = ALIGN(8);
.alternative : {
BSS_SECTION(PAGE_SIZE, PAGE_SIZE, 0)
+ PERCPU_SECTION(L1_CACHE_BYTES)
+
.rel.dyn : AT(ADDR(.rel.dyn) - LOAD_OFFSET) {
*(.rel.dyn*)
}
* Ensure we set mode to IN_GUEST_MODE after we disable
* interrupts and before the final VCPU requests check.
* See the comment in kvm_vcpu_exiting_guest_mode() and
- * Documentation/virtual/kvm/vcpu-requests.rst
+ * Documentation/virt/kvm/vcpu-requests.rst
*/
vcpu->mode = IN_GUEST_MODE;
// SPDX-License-Identifier: GPL-2.0
-/**
+/*
* Copyright (c) 2019 Western Digital Corporation or its affiliates.
*
* Authors:
r = 1;
break;
case KVM_CAP_NR_VCPUS:
- r = num_online_cpus();
+ r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
break;
case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
*/
#include <linux/delay.h>
+#include <linux/math.h>
#include <linux/param.h>
#include <linux/timex.h>
+#include <linux/types.h>
#include <linux/export.h>
+#include <asm/processor.h>
+
/*
* This is copies from arch/arm/include/asm/delay.h
*
local_flush_tlb_all();
/* Pre-compute ASID details */
- num_asids = 1 << asid_bits;
- asid_mask = num_asids - 1;
+ if (asid_bits) {
+ num_asids = 1 << asid_bits;
+ asid_mask = num_asids - 1;
+ }
/*
* Use ASID allocator only if number of HW ASIDs are
pr_info("ASID allocator using %lu bits (%lu entries)\n",
asid_bits, num_asids);
} else {
- pr_info("ASID allocator disabled\n");
+ pr_info("ASID allocator disabled (%lu bits)\n", asid_bits);
}
return 0;
#include <linux/module.h>
#include <linux/uaccess.h>
-#ifdef CONFIG_BPF_JIT
+#if defined(CONFIG_BPF_JIT) && defined(CONFIG_ARCH_RV64I)
int rv_bpf_fixup_exception(const struct exception_table_entry *ex, struct pt_regs *regs);
#endif
if (!fixup)
return 0;
-#ifdef CONFIG_BPF_JIT
+#if defined(CONFIG_BPF_JIT) && defined(CONFIG_ARCH_RV64I)
if (regs->epc >= BPF_JIT_REGION_START && regs->epc < BPF_JIT_REGION_END)
return rv_bpf_fixup_exception(fixup, regs);
#endif
EXPORT_SYMBOL(phys_ram_base);
#ifdef CONFIG_XIP_KERNEL
-extern char _xiprom[], _exiprom[];
+extern char _xiprom[], _exiprom[], __data_loc;
#endif
unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]
/* called from head.S with MMU off */
asmlinkage void __init __copy_data(void)
{
- void *from = (void *)(&_sdata);
- void *end = (void *)(&_end);
+ void *from = (void *)(&__data_loc);
void *to = (void *)CONFIG_PHYS_RAM_BASE;
- size_t sz = (size_t)(end - from + 1);
+ size_t sz = (size_t)((uintptr_t)(&_end) - (uintptr_t)(&_sdata));
memcpy(to, from, sz);
}
#define BPF_FIXUP_REG_MASK GENMASK(31, 27)
int rv_bpf_fixup_exception(const struct exception_table_entry *ex,
+ struct pt_regs *regs);
+int rv_bpf_fixup_exception(const struct exception_table_entry *ex,
struct pt_regs *regs)
{
off_t offset = FIELD_GET(BPF_FIXUP_OFFSET_MASK, ex->fixup);
obj-y += net/
obj-$(CONFIG_PCI) += pci/
obj-$(CONFIG_ARCH_HAS_KEXEC_PURGATORY) += purgatory/
+
+# for cleaning
+subdir- += boot tools
config KASAN_SHADOW_OFFSET
hex
depends on KASAN
- default 0x18000000000000
+ default 0x1C000000000000
config S390
def_bool y
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_CONTIGUOUS
select HAVE_DYNAMIC_FTRACE
+ select HAVE_DYNAMIC_FTRACE_WITH_ARGS
+ select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
select HAVE_DYNAMIC_FTRACE_WITH_REGS
select HAVE_EBPF_JIT if PACK_STACK && HAVE_MARCH_Z196_FEATURES
select HAVE_EFFICIENT_UNALIGNED_ACCESS
select HAVE_FAST_GUP
select HAVE_FENTRY
select HAVE_FTRACE_MCOUNT_RECORD
+ select HAVE_FUNCTION_ARG_ACCESS_API
select HAVE_FUNCTION_ERROR_INJECTION
select HAVE_FUNCTION_GRAPH_TRACER
select HAVE_FUNCTION_TRACER
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RELIABLE_STACKTRACE
select HAVE_RSEQ
+ select HAVE_SAMPLE_FTRACE_DIRECT
+ select HAVE_SAMPLE_FTRACE_DIRECT_MULTI
select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING
config 64BIT
def_bool y
+config COMMAND_LINE_SIZE
+ int "Maximum size of kernel command line"
+ default 4096
+ range 896 1048576
+ help
+ This allows you to specify the maximum length of the kernel command
+ line.
+
config COMPAT
def_bool y
prompt "Kernel support for 31 bit emulation"
config S390_UNWIND_SELFTEST
def_tristate n
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
prompt "Test unwind functions"
help
This option enables s390 specific stack unwinder testing kernel
Say N if you are unsure.
+config S390_KPROBES_SANITY_TEST
+ def_tristate n
+ prompt "Enable s390 specific kprobes tests"
+ depends on KPROBES
+ depends on KUNIT
+ help
+ This option enables an s390 specific kprobes test module. This option
+ is not useful for distributions or general kernels, but only for kernel
+ developers working on architecture code.
+
+ Say N if you are unsure.
+
endmenu
# s390/Makefile
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
#
# Copyright (C) 1994 by Linus Torvalds
#
KBUILD_CFLAGS_DECOMPRESSOR += $(cflags-y)
ifneq ($(call cc-option,-mstack-size=8192 -mstack-guard=128),)
-cflags-$(CONFIG_CHECK_STACK) += -mstack-size=$(STACK_SIZE)
-ifeq ($(call cc-option,-mstack-size=8192),)
-cflags-$(CONFIG_CHECK_STACK) += -mstack-guard=$(CONFIG_STACK_GUARD)
-endif
+ CC_FLAGS_CHECK_STACK := -mstack-size=$(STACK_SIZE)
+ ifeq ($(call cc-option,-mstack-size=8192),)
+ CC_FLAGS_CHECK_STACK += -mstack-guard=$(CONFIG_STACK_GUARD)
+ endif
+ export CC_FLAGS_CHECK_STACK
+ cflags-$(CONFIG_CHECK_STACK) += $(CC_FLAGS_CHECK_STACK)
endif
ifdef CONFIG_EXPOLINE
vdso_install:
$(Q)$(MAKE) $(build)=arch/$(ARCH)/kernel/vdso64 $@
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
- $(Q)$(MAKE) $(clean)=$(tools)
-
archheaders:
$(Q)$(MAKE) $(build)=$(syscalls) uapi
unsigned long dynsym_start;
unsigned long rela_dyn_start;
unsigned long rela_dyn_end;
+ unsigned long amode31_size;
};
/* Symbols defined by linker scripts */
bas %r14,.Lloader # load parameter file
ltr %r2,%r2 # got anything ?
bz .Lnopf
- chi %r2,895
- bnh .Lnotrunc
- la %r2,895
+ l %r3,MAX_COMMAND_LINE_SIZE+ARCH_OFFSET-PARMAREA(%r12)
+ ahi %r3,-1
+ clr %r2,%r3
+ bl .Lnotrunc
+ lr %r2,%r3
.Lnotrunc:
l %r4,.Linitrd
clc 0(3,%r4),.L_hdr # if it is HDRx
bz .Lagain1 # skip dataset header
clc 0(3,%r4),.L_eof # if it is EOFx
bz .Lagain1 # skip dateset trailer
- la %r5,0(%r4,%r2)
- lr %r3,%r2
- la %r3,COMMAND_LINE-PARMAREA(%r12) # load adr. of command line
- mvc 0(256,%r3),0(%r4)
- mvc 256(256,%r3),256(%r4)
- mvc 512(256,%r3),512(%r4)
- mvc 768(122,%r3),768(%r4)
- slr %r0,%r0
- b .Lcntlp
-.Ldelspc:
- ic %r0,0(%r2,%r3)
- chi %r0,0x20 # is it a space ?
- be .Lcntlp
- ahi %r2,1
- b .Leolp
-.Lcntlp:
- brct %r2,.Ldelspc
-.Leolp:
- slr %r0,%r0
- stc %r0,0(%r2,%r3) # terminate buffer
+
+ lr %r5,%r2
+ la %r6,COMMAND_LINE-PARMAREA(%r12)
+ lr %r7,%r2
+ ahi %r7,1
+ mvcl %r6,%r4
.Lnopf:
#
xc 0x300(256),0x300
xc 0xe00(256),0xe00
xc 0xf00(256),0xf00
+ lctlg %c0,%c15,.Lctl-.LPG0(%r13) # load control registers
stcke __LC_BOOT_CLOCK
mvc __LC_LAST_UPDATE_CLOCK(8),__LC_BOOT_CLOCK+1
spt 6f-.LPG0(%r13)
.quad 0x0000000180000000,startup_pgm_check_handler
.Lio_new_psw:
.quad 0x0002000180000000,0x1f0 # disabled wait
+.Lctl: .quad 0x04040000 # cr0: AFP registers & secondary space
+ .quad 0 # cr1: primary space segment table
+ .quad 0 # cr2: dispatchable unit control table
+ .quad 0 # cr3: instruction authorization
+ .quad 0xffff # cr4: instruction authorization
+ .quad 0 # cr5: primary-aste origin
+ .quad 0 # cr6: I/O interrupts
+ .quad 0 # cr7: secondary space segment table
+ .quad 0x0000000000008000 # cr8: access registers translation
+ .quad 0 # cr9: tracing off
+ .quad 0 # cr10: tracing off
+ .quad 0 # cr11: tracing off
+ .quad 0 # cr12: tracing off
+ .quad 0 # cr13: home space segment table
+ .quad 0xc0000000 # cr14: machine check handling off
+ .quad 0 # cr15: linkage stack operations
#include "head_kdump.S"
.quad 0 # OLDMEM_BASE
.quad 0 # OLDMEM_SIZE
.quad kernel_version # points to kernel version string
+ .quad COMMAND_LINE_SIZE
.org COMMAND_LINE
.byte "root=/dev/ram0 ro"
.byte 0
.org PARMAREA+__PARMAREA_SIZE
SYM_DATA_END(parmarea)
-
- .org HEAD_END
void setup_boot_command_line(void)
{
- parmarea.command_line[ARCH_COMMAND_LINE_SIZE - 1] = 0;
+ parmarea.command_line[COMMAND_LINE_SIZE - 1] = 0;
/* convert arch command line to ascii if necessary */
if (has_ebcdic_char(parmarea.command_line))
- EBCASC(parmarea.command_line, ARCH_COMMAND_LINE_SIZE);
+ EBCASC(parmarea.command_line, COMMAND_LINE_SIZE);
/* copy arch command line */
strcpy(early_command_line, strim(parmarea.command_line));
gpregs[12], gpregs[13], gpregs[14], gpregs[15]);
print_stacktrace();
decompressor_printk("Last Breaking-Event-Address:\n");
- decompressor_printk(" [<%016lx>] %pS\n", (unsigned long)S390_lowcore.breaking_event_addr,
- (void *)S390_lowcore.breaking_event_addr);
+ decompressor_printk(" [<%016lx>] %pS\n", (unsigned long)S390_lowcore.pgm_last_break,
+ (void *)S390_lowcore.pgm_last_break);
}
#include "uv.h"
unsigned long __bootdata_preserved(__kaslr_offset);
+unsigned long __bootdata(__amode31_base);
unsigned long __bootdata_preserved(VMALLOC_START);
unsigned long __bootdata_preserved(VMALLOC_END);
struct page *__bootdata_preserved(vmemmap);
static void setup_kernel_memory_layout(void)
{
- bool vmalloc_size_verified = false;
- unsigned long vmemmap_off;
- unsigned long vspace_left;
+ unsigned long vmemmap_start;
unsigned long rte_size;
unsigned long pages;
- unsigned long vmax;
pages = ident_map_size / PAGE_SIZE;
/* vmemmap contains a multiple of PAGES_PER_SECTION struct pages */
vmemmap_size = SECTION_ALIGN_UP(pages) * sizeof(struct page);
/* choose kernel address space layout: 4 or 3 levels. */
- vmemmap_off = round_up(ident_map_size, _REGION3_SIZE);
+ vmemmap_start = round_up(ident_map_size, _REGION3_SIZE);
if (IS_ENABLED(CONFIG_KASAN) ||
vmalloc_size > _REGION2_SIZE ||
- vmemmap_off + vmemmap_size + vmalloc_size + MODULES_LEN > _REGION2_SIZE)
- vmax = _REGION1_SIZE;
- else
- vmax = _REGION2_SIZE;
-
- /* keep vmemmap_off aligned to a top level region table entry */
- rte_size = vmax == _REGION1_SIZE ? _REGION2_SIZE : _REGION3_SIZE;
- MODULES_END = vmax;
- if (is_prot_virt_host()) {
- /*
- * forcing modules and vmalloc area under the ultravisor
- * secure storage limit, so that any vmalloc allocation
- * we do could be used to back secure guest storage.
- */
- adjust_to_uv_max(&MODULES_END);
- }
-
-#ifdef CONFIG_KASAN
- if (MODULES_END < vmax) {
- /* force vmalloc and modules below kasan shadow */
- MODULES_END = min(MODULES_END, KASAN_SHADOW_START);
+ vmemmap_start + vmemmap_size + vmalloc_size + MODULES_LEN >
+ _REGION2_SIZE) {
+ MODULES_END = _REGION1_SIZE;
+ rte_size = _REGION2_SIZE;
} else {
- /*
- * leave vmalloc and modules above kasan shadow but make
- * sure they don't overlap with it
- */
- vmalloc_size = min(vmalloc_size, vmax - KASAN_SHADOW_END - MODULES_LEN);
- vmalloc_size_verified = true;
- vspace_left = KASAN_SHADOW_START;
+ MODULES_END = _REGION2_SIZE;
+ rte_size = _REGION3_SIZE;
}
+ /*
+ * forcing modules and vmalloc area under the ultravisor
+ * secure storage limit, so that any vmalloc allocation
+ * we do could be used to back secure guest storage.
+ */
+ adjust_to_uv_max(&MODULES_END);
+#ifdef CONFIG_KASAN
+ /* force vmalloc and modules below kasan shadow */
+ MODULES_END = min(MODULES_END, KASAN_SHADOW_START);
#endif
MODULES_VADDR = MODULES_END - MODULES_LEN;
VMALLOC_END = MODULES_VADDR;
- if (vmalloc_size_verified) {
- VMALLOC_START = VMALLOC_END - vmalloc_size;
- } else {
- vmemmap_off = round_up(ident_map_size, rte_size);
+ /* allow vmalloc area to occupy up to about 1/2 of the rest virtual space left */
+ vmalloc_size = min(vmalloc_size, round_down(VMALLOC_END / 2, _REGION3_SIZE));
+ VMALLOC_START = VMALLOC_END - vmalloc_size;
- if (vmemmap_off + vmemmap_size > VMALLOC_END ||
- vmalloc_size > VMALLOC_END - vmemmap_off - vmemmap_size) {
- /*
- * allow vmalloc area to occupy up to 1/2 of
- * the rest virtual space left.
- */
- vmalloc_size = min(vmalloc_size, VMALLOC_END / 2);
- }
- VMALLOC_START = VMALLOC_END - vmalloc_size;
- vspace_left = VMALLOC_START;
- }
-
- pages = vspace_left / (PAGE_SIZE + sizeof(struct page));
+ /* split remaining virtual space between 1:1 mapping & vmemmap array */
+ pages = VMALLOC_START / (PAGE_SIZE + sizeof(struct page));
pages = SECTION_ALIGN_UP(pages);
- vmemmap_off = round_up(vspace_left - pages * sizeof(struct page), rte_size);
- /* keep vmemmap left most starting from a fresh region table entry */
- vmemmap_off = min(vmemmap_off, round_up(ident_map_size, rte_size));
- /* take care that identity map is lower then vmemmap */
- ident_map_size = min(ident_map_size, vmemmap_off);
+ /* keep vmemmap_start aligned to a top level region table entry */
+ vmemmap_start = round_down(VMALLOC_START - pages * sizeof(struct page), rte_size);
+ /* vmemmap_start is the future VMEM_MAX_PHYS, make sure it is within MAX_PHYSMEM */
+ vmemmap_start = min(vmemmap_start, 1UL << MAX_PHYSMEM_BITS);
+ /* make sure identity map doesn't overlay with vmemmap */
+ ident_map_size = min(ident_map_size, vmemmap_start);
vmemmap_size = SECTION_ALIGN_UP(ident_map_size / PAGE_SIZE) * sizeof(struct page);
- VMALLOC_START = max(vmemmap_off + vmemmap_size, VMALLOC_START);
- vmemmap = (struct page *)vmemmap_off;
+ /* make sure vmemmap doesn't overlay with vmalloc area */
+ VMALLOC_START = max(vmemmap_start + vmemmap_size, VMALLOC_START);
+ vmemmap = (struct page *)vmemmap_start;
}
/*
vmlinux.dynsym_start += offset;
}
+static unsigned long reserve_amode31(unsigned long safe_addr)
+{
+ __amode31_base = PAGE_ALIGN(safe_addr);
+ return safe_addr + vmlinux.amode31_size;
+}
+
void startup_kernel(void)
{
unsigned long random_lma;
setup_lpp();
store_ipl_parmblock();
safe_addr = mem_safe_offset();
+ safe_addr = reserve_amode31(safe_addr);
safe_addr = read_ipl_report(safe_addr);
uv_query_info();
rescue_initrd(safe_addr);
CONFIG_CMM=m
CONFIG_APPLDATA_BASE=y
CONFIG_KVM=m
-CONFIG_S390_UNWIND_SELFTEST=y
+CONFIG_S390_UNWIND_SELFTEST=m
+CONFIG_S390_KPROBES_SANITY_TEST=m
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
CONFIG_STATIC_KEYS_SELFTEST=y
CONFIG_RANDOM32_SELFTEST=y
CONFIG_DMA_CMA=y
CONFIG_CMA_SIZE_MBYTES=0
-CONFIG_DMA_API_DEBUG=y
CONFIG_PRINTK_TIME=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_HIST_TRIGGERS=y
CONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_EVENT_TRACE_STARTUP_TEST is not set
+CONFIG_SAMPLES=y
+CONFIG_SAMPLE_TRACE_PRINTK=m
+CONFIG_SAMPLE_FTRACE_DIRECT=m
CONFIG_DEBUG_ENTRY=y
CONFIG_CIO_INJECT=y
+CONFIG_KUNIT=m
+CONFIG_KUNIT_DEBUGFS=y
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_NETDEV_NOTIFIER_ERROR_INJECT=m
CONFIG_FAULT_INJECTION=y
CONFIG_APPLDATA_BASE=y
CONFIG_KVM=m
CONFIG_S390_UNWIND_SELFTEST=m
+CONFIG_S390_KPROBES_SANITY_TEST=m
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_GCC_PLUGINS is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_HIST_TRIGGERS=y
+CONFIG_SAMPLES=y
+CONFIG_SAMPLE_TRACE_PRINTK=m
+CONFIG_SAMPLE_FTRACE_DIRECT=m
+CONFIG_KUNIT=m
+CONFIG_KUNIT_DEBUGFS=y
CONFIG_LKDTM=m
CONFIG_PERCPU_TEST=m
CONFIG_ATOMIC64_SELFTEST=y
#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
/* Fast-BCR without checkpoint synchronization */
-#define __ASM_BARRIER "bcr 14,0\n"
+#define __ASM_BCR_SERIALIZE "bcr 14,0\n"
#else
-#define __ASM_BARRIER "bcr 15,0\n"
+#define __ASM_BCR_SERIALIZE "bcr 15,0\n"
#endif
-#define mb() do { asm volatile(__ASM_BARRIER : : : "memory"); } while (0)
+static __always_inline void bcr_serialize(void)
+{
+ asm volatile(__ASM_BCR_SERIALIZE : : : "memory");
+}
-#define rmb() barrier()
-#define wmb() barrier()
-#define dma_rmb() mb()
-#define dma_wmb() mb()
-#define __smp_mb() mb()
-#define __smp_rmb() rmb()
-#define __smp_wmb() wmb()
+#define mb() bcr_serialize()
+#define rmb() barrier()
+#define wmb() barrier()
+#define dma_rmb() mb()
+#define dma_wmb() mb()
+#define __smp_mb() mb()
+#define __smp_rmb() rmb()
+#define __smp_wmb() wmb()
#define __smp_store_release(p, v) \
do { \
volatile unsigned long *ptr)
{
if (arch_test_bit(nr, ptr))
- return 1;
+ return true;
return arch_test_and_set_bit(nr, ptr);
}
#ifndef __ASSEMBLY__
#include <linux/types.h>
+#include <linux/jump_label.h>
struct cpuid
{
unsigned int unused : 16;
} __attribute__ ((packed, aligned(8)));
+DECLARE_STATIC_KEY_FALSE(cpu_has_bear);
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_S390_CPU_H */
*
* @var: Name of debug_info_t variable
* @name: Name of debug log (e.g. used for debugfs entry)
- * @pages_per_area: Number of pages per area
+ * @pages: Number of pages per area
* @nr_areas: Number of debug areas
* @buf_size: Size of data area in each debug entry
* @view: Pointer to debug view struct
#define __ASM_FACILITY_H
#include <asm/facility-defs.h>
+
+#include <linux/minmax.h>
#include <linux/string.h>
+#include <linux/types.h>
#include <linux/preempt.h>
+
#include <asm/lowcore.h>
#define MAX_FACILITY_BIT (sizeof(stfle_fac_list) * 8)
void ftrace_caller(void);
-extern char ftrace_graph_caller_end;
extern void *ftrace_func;
struct dyn_arch_ftrace { };
return addr;
}
+struct ftrace_regs {
+ struct pt_regs regs;
+};
+
+static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *fregs)
+{
+ return &fregs->regs;
+}
+
+static __always_inline void ftrace_instruction_pointer_set(struct ftrace_regs *fregs,
+ unsigned long ip)
+{
+ struct pt_regs *regs = arch_ftrace_get_regs(fregs);
+
+ regs->psw.addr = ip;
+}
+
+/*
+ * When an ftrace registered caller is tracing a function that is
+ * also set by a register_ftrace_direct() call, it needs to be
+ * differentiated in the ftrace_caller trampoline. To do this,
+ * place the direct caller in the ORIG_GPR2 part of pt_regs. This
+ * tells the ftrace_caller that there's a direct caller.
+ */
+static inline void arch_ftrace_set_direct_caller(struct pt_regs *regs, unsigned long addr)
+{
+ regs->orig_gpr2 = addr;
+}
+
/*
* Even though the system call numbers are identical for s390/s390x a
* different system call table is used for compat tasks. This may lead
}
#endif /* __ASSEMBLY__ */
+
+#ifdef CONFIG_FUNCTION_TRACER
+
+#define FTRACE_NOP_INSN .word 0xc004, 0x0000, 0x0000 /* brcl 0,0 */
+
+#ifndef CC_USING_HOTPATCH
+
+#define FTRACE_GEN_MCOUNT_RECORD(name) \
+ .section __mcount_loc, "a", @progbits; \
+ .quad name; \
+ .previous;
+
+#else /* !CC_USING_HOTPATCH */
+
+#define FTRACE_GEN_MCOUNT_RECORD(name)
+
+#endif /* !CC_USING_HOTPATCH */
+
+#define FTRACE_GEN_NOP_ASM(name) \
+ FTRACE_GEN_MCOUNT_RECORD(name) \
+ FTRACE_NOP_INSN
+
+#else /* CONFIG_FUNCTION_TRACER */
+
+#define FTRACE_GEN_NOP_ASM(name)
+
+#endif /* CONFIG_FUNCTION_TRACER */
+
#endif /* _ASM_S390_FTRACE_H */
#ifndef _ASM_S390_JUMP_LABEL_H
#define _ASM_S390_JUMP_LABEL_H
+#define HAVE_JUMP_LABEL_BATCH
+
#ifndef __ASSEMBLY__
#include <linux/types.h>
DIE_NMI_IPI,
};
-extern void die(struct pt_regs *, const char *);
+extern void __noreturn die(struct pt_regs *, const char *);
#endif
int arch_kexec_do_relocs(int r_type, void *loc, unsigned long val,
unsigned long addr);
+#define ARCH_HAS_KIMAGE_ARCH
+
+struct kimage_arch {
+ void *ipl_buf;
+};
+
extern const struct kexec_file_ops s390_kexec_image_ops;
extern const struct kexec_file_ops s390_kexec_elf_ops;
static inline void klp_arch_set_pc(struct ftrace_regs *fregs, unsigned long ip)
{
- struct pt_regs *regs = ftrace_get_regs(fregs);
-
- regs->psw.addr = ip;
+ ftrace_instruction_pointer_set(fregs, ip);
}
#endif
__u32 external_damage_code; /* 0x00f4 */
__u64 failing_storage_address; /* 0x00f8 */
__u8 pad_0x0100[0x0110-0x0100]; /* 0x0100 */
- __u64 breaking_event_addr; /* 0x0110 */
+ __u64 pgm_last_break; /* 0x0110 */
__u8 pad_0x0118[0x0120-0x0118]; /* 0x0118 */
psw_t restart_old_psw; /* 0x0120 */
psw_t external_old_psw; /* 0x0130 */
psw_t return_psw; /* 0x0290 */
psw_t return_mcck_psw; /* 0x02a0 */
+ __u64 last_break; /* 0x02b0 */
+
/* CPU accounting and timing values. */
- __u64 sys_enter_timer; /* 0x02b0 */
- __u8 pad_0x02b8[0x02c0-0x02b8]; /* 0x02b8 */
+ __u64 sys_enter_timer; /* 0x02b8 */
__u64 mcck_enter_timer; /* 0x02c0 */
__u64 exit_timer; /* 0x02c8 */
__u64 user_timer; /* 0x02d0 */
__u32 tod_progreg_save_area; /* 0x1324 */
__u32 cpu_timer_save_area[2]; /* 0x1328 */
__u32 clock_comp_save_area[2]; /* 0x1330 */
- __u8 pad_0x1338[0x1340-0x1338]; /* 0x1338 */
+ __u64 last_break_save_area; /* 0x1338 */
__u32 access_regs_save_area[16]; /* 0x1340 */
__u64 cregs_save_area[16]; /* 0x1380 */
__u8 pad_0x1400[0x1800-0x1400]; /* 0x1400 */
void nospec_auto_detect(void);
void nospec_revert(s32 *start, s32 *end);
+static inline bool nospec_uses_trampoline(void)
+{
+ return __is_defined(CC_USING_EXPOLINE) && !nospec_disable;
+}
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_S390_EXPOLINE_H */
void zpci_device_reserved(struct zpci_dev *zdev);
bool zpci_is_device_configured(struct zpci_dev *zdev);
+int zpci_hot_reset_device(struct zpci_dev *zdev);
int zpci_register_ioat(struct zpci_dev *, u8, u64, u64, u64);
int zpci_unregister_ioat(struct zpci_dev *, u8);
void zpci_remove_reserved_devices(void);
+void zpci_update_fh(struct zpci_dev *zdev, u32 fh);
/* CLP */
int clp_setup_writeback_mio(void);
void zpci_debug_init_device(struct zpci_dev *, const char *);
void zpci_debug_exit_device(struct zpci_dev *);
-/* Error reporting */
+/* Error handling */
int zpci_report_error(struct pci_dev *, struct zpci_report_error_header *);
+int zpci_clear_error_state(struct zpci_dev *zdev);
+int zpci_reset_load_store_blocked(struct zpci_dev *zdev);
#ifdef CONFIG_NUMA
#define CRDTE_DTT_REGION1 0x1cUL
static inline void crdte(unsigned long old, unsigned long new,
- unsigned long table, unsigned long dtt,
+ unsigned long *table, unsigned long dtt,
unsigned long address, unsigned long asce)
{
union register_pair r1 = { .even = old, .odd = new, };
- union register_pair r2 = { .even = table | dtt, .odd = address, };
+ union register_pair r2 = { .even = __pa(table) | dtt, .odd = address, };
asm volatile(".insn rrf,0xb98f0000,%[r1],%[r2],%[asce],0"
: [r1] "+&d" (r1.pair)
unsigned long opt, unsigned long asce,
int local)
{
- unsigned long pto = (unsigned long) ptep;
+ unsigned long pto = __pa(ptep);
if (__builtin_constant_p(opt) && opt == 0) {
/* Invalidation + TLB flush for the pte */
static __always_inline void __ptep_ipte_range(unsigned long address, int nr,
pte_t *ptep, int local)
{
- unsigned long pto = (unsigned long) ptep;
+ unsigned long pto = __pa(ptep);
/* Invalidate a range of ptes + TLB flush of the ptes */
do {
{
unsigned long sto;
- sto = (unsigned long) pmdp - pmd_index(addr) * sizeof(pmd_t);
+ sto = __pa(pmdp) - pmd_index(addr) * sizeof(pmd_t);
if (__builtin_constant_p(opt) && opt == 0) {
/* flush without guest asce */
asm volatile(
{
unsigned long r3o;
- r3o = (unsigned long) pudp - pud_index(addr) * sizeof(pud_t);
+ r3o = __pa(pudp) - pud_index(addr) * sizeof(pud_t);
r3o |= _ASCE_TYPE_REGION3;
if (__builtin_constant_p(opt) && opt == 0) {
/* flush without guest asce */
* The pt_regs struct defines the way the registers are stored on
* the stack during a system call.
*/
-struct pt_regs
-{
+struct pt_regs {
union {
user_pt_regs user_regs;
struct {
};
unsigned long flags;
unsigned long cr1;
+ unsigned long last_break;
};
/*
unsigned long regs_get_register(struct pt_regs *regs, unsigned int offset);
unsigned long regs_get_kernel_stack_nth(struct pt_regs *regs, unsigned int n);
+/**
+ * regs_get_kernel_argument() - get Nth function argument in kernel
+ * @regs: pt_regs of that context
+ * @n: function argument number (start from 0)
+ *
+ * regs_get_kernel_argument() returns @n th argument of the function call.
+ */
+static inline unsigned long regs_get_kernel_argument(struct pt_regs *regs,
+ unsigned int n)
+{
+ unsigned int argoffset = STACK_FRAME_OVERHEAD / sizeof(long);
+
+#define NR_REG_ARGUMENTS 5
+ if (n < NR_REG_ARGUMENTS)
+ return regs_get_register(regs, 2 + n);
+ n -= NR_REG_ARGUMENTS;
+ return regs_get_kernel_stack_nth(regs, argoffset + n);
+}
+
static inline unsigned long kernel_stack_pointer(struct pt_regs *regs)
{
return regs->gprs[15];
extern char *sclp_early_sccb;
+void sclp_early_adjust_va(void);
void sclp_early_set_buffer(void *sccb);
int sclp_early_read_info(void);
int sclp_early_read_storage_info(void);
#ifndef _S390_SECTIONS_H
#define _S390_SECTIONS_H
-#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
-
#include <asm-generic/sections.h>
-extern bool initmem_freed;
-
-static inline int arch_is_kernel_initmem_freed(unsigned long addr)
-{
- if (!initmem_freed)
- return 0;
- return addr >= (unsigned long)__init_begin &&
- addr < (unsigned long)__init_end;
-}
-
/*
* .boot.data section contains variables "shared" between the decompressor and
* the decompressed kernel. The decompressor will store values in them, and
#include <linux/build_bug.h>
#define PARMAREA 0x10400
-#define HEAD_END 0x11000
+#define COMMAND_LINE_SIZE CONFIG_COMMAND_LINE_SIZE
/*
* Machine features detected in early.c
*/
#define STARTUP_NORMAL_OFFSET 0x10000
#define STARTUP_KDUMP_OFFSET 0x10010
+#define LEGACY_COMMAND_LINE_SIZE 896
+
#ifndef __ASSEMBLY__
#include <asm/lowcore.h>
unsigned long oldmem_base; /* 0x10418 */
unsigned long oldmem_size; /* 0x10420 */
unsigned long kernel_version; /* 0x10428 */
- char pad1[0x10480 - 0x10430]; /* 0x10430 - 0x10480 */
- char command_line[ARCH_COMMAND_LINE_SIZE]; /* 0x10480 */
+ unsigned long max_command_line_size; /* 0x10430 */
+ char pad1[0x10480-0x10438]; /* 0x10438 - 0x10480 */
+ char command_line[COMMAND_LINE_SIZE]; /* 0x10480 */
};
extern struct parmarea parmarea;
#define __HAVE_ARCH_STRCMP /* arch function */
#define __HAVE_ARCH_STRCPY /* inline & arch function */
#define __HAVE_ARCH_STRLCAT /* arch function */
-#define __HAVE_ARCH_STRLCPY /* arch function */
#define __HAVE_ARCH_STRLEN /* inline & arch function */
#define __HAVE_ARCH_STRNCAT /* arch function */
#define __HAVE_ARCH_STRNCPY /* arch function */
#define __HAVE_ARCH_STRNLEN /* inline & arch function */
-#define __HAVE_ARCH_STRRCHR /* arch function */
#define __HAVE_ARCH_STRSTR /* arch function */
/* Prototypes for non-inlined arch strings functions. */
int memcmp(const void *s1, const void *s2, size_t n);
int strcmp(const char *s1, const char *s2);
size_t strlcat(char *dest, const char *src, size_t n);
-size_t strlcpy(char *dest, const char *src, size_t size);
char *strncat(char *dest, const char *src, size_t n);
char *strncpy(char *dest, const char *src, size_t n);
-char *strrchr(const char *s, int c);
char *strstr(const char *s1, const char *s2);
#endif /* !CONFIG_KASAN */
args[0] = regs->orig_gpr2 & mask;
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- unsigned int n = 6;
-
- while (n-- > 0)
- if (n > 0)
- regs->gprs[2 + n] = args[n];
- regs->orig_gpr2 = args[0];
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
#ifdef CONFIG_COMPAT
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_S390_TEXT_PATCHING_H
+#define _ASM_S390_TEXT_PATCHING_H
+
+#include <asm/barrier.h>
+
+static __always_inline void sync_core(void)
+{
+ bcr_serialize();
+}
+
+void text_poke_sync(void);
+void text_poke_sync_lock(void);
+
+#endif /* _ASM_S390_TEXT_PATCHING_H */
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-/*
- * S390 version
- * Copyright IBM Corp. 1999, 2010
- */
-
-#ifndef _UAPI_ASM_S390_SETUP_H
-#define _UAPI_ASM_S390_SETUP_H
-
-#define COMMAND_LINE_SIZE 4096
-
-#define ARCH_COMMAND_LINE_SIZE 896
-
-#endif /* _UAPI_ASM_S390_SETUP_H */
// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>
+#include <linux/cpu.h>
+#include <linux/smp.h>
+#include <asm/text-patching.h>
#include <asm/alternative.h>
#include <asm/facility.h>
#include <asm/nospec-branch.h>
{
apply_alternatives(__alt_instructions, __alt_instructions_end);
}
+
+static void do_sync_core(void *info)
+{
+ sync_core();
+}
+
+void text_poke_sync(void)
+{
+ on_each_cpu(do_sync_core, NULL, 1);
+}
+
+void text_poke_sync_lock(void)
+{
+ cpus_read_lock();
+ text_poke_sync();
+ cpus_read_unlock();
+}
OFFSET(__PT_ORIG_GPR2, pt_regs, orig_gpr2);
OFFSET(__PT_FLAGS, pt_regs, flags);
OFFSET(__PT_CR1, pt_regs, cr1);
+ OFFSET(__PT_LAST_BREAK, pt_regs, last_break);
DEFINE(__PT_SIZE, sizeof(struct pt_regs));
BLANK();
/* stack_frame offsets */
OFFSET(__SF_SIE_SAVEAREA, stack_frame, empty1[2]);
OFFSET(__SF_SIE_REASON, stack_frame, empty1[3]);
OFFSET(__SF_SIE_FLAGS, stack_frame, empty1[4]);
+ DEFINE(STACK_FRAME_OVERHEAD, sizeof(struct stack_frame));
BLANK();
/* idle data offsets */
OFFSET(__CLOCK_IDLE_ENTER, s390_idle_data, clock_idle_enter);
OFFSET(__LC_MCCK_CODE, lowcore, mcck_interruption_code);
OFFSET(__LC_EXT_DAMAGE_CODE, lowcore, external_damage_code);
OFFSET(__LC_MCCK_FAIL_STOR_ADDR, lowcore, failing_storage_address);
- OFFSET(__LC_LAST_BREAK, lowcore, breaking_event_addr);
+ OFFSET(__LC_PGM_LAST_BREAK, lowcore, pgm_last_break);
OFFSET(__LC_RETURN_LPSWE, lowcore, return_lpswe);
OFFSET(__LC_RETURN_MCCK_LPSWE, lowcore, return_mcck_lpswe);
OFFSET(__LC_RST_OLD_PSW, lowcore, restart_old_psw);
OFFSET(__LC_PREEMPT_COUNT, lowcore, preempt_count);
OFFSET(__LC_GMAP, lowcore, gmap);
OFFSET(__LC_BR_R1, lowcore, br_r1_trampoline);
+ OFFSET(__LC_LAST_BREAK, lowcore, last_break);
/* software defined ABI-relevant lowcore locations 0xe00 - 0xe20 */
OFFSET(__LC_DUMP_REIPL, lowcore, ipib);
/* hardware defined lowcore locations 0x1000 - 0x18ff */
OFFSET(__LC_TOD_PROGREG_SAVE_AREA, lowcore, tod_progreg_save_area);
OFFSET(__LC_CPU_TIMER_SAVE_AREA, lowcore, cpu_timer_save_area);
OFFSET(__LC_CLOCK_COMP_SAVE_AREA, lowcore, clock_comp_save_area);
+ OFFSET(__LC_LAST_BREAK_SAVE_AREA, lowcore, last_break_save_area);
OFFSET(__LC_AREGS_SAVE_AREA, lowcore, access_regs_save_area);
OFFSET(__LC_CREGS_SAVE_AREA, lowcore, cregs_save_area);
OFFSET(__LC_PGM_TDB, lowcore, pgm_tdb);
DEFINE(OLDMEM_BASE, PARMAREA + offsetof(struct parmarea, oldmem_base));
DEFINE(OLDMEM_SIZE, PARMAREA + offsetof(struct parmarea, oldmem_size));
DEFINE(COMMAND_LINE, PARMAREA + offsetof(struct parmarea, command_line));
+ DEFINE(MAX_COMMAND_LINE_SIZE, PARMAREA + offsetof(struct parmarea, max_command_line_size));
return 0;
}
asm volatile(
" diag %[rx],%[ry],0x8\n"
: [ry] "+&d" (cmdlen)
- : [rx] "d" ((addr_t) cpcmd_buf)
+ : [rx] "d" (__pa(cpcmd_buf))
: "cc");
return cmdlen;
}
union register_pair rx, ry;
int cc;
- rx.even = (addr_t) cpcmd_buf;
- rx.odd = (addr_t) response;
+ rx.even = __pa(cpcmd_buf);
+ rx.odd = __pa(response);
ry.even = cmdlen | 0x40000000L;
ry.odd = *rlen;
asm volatile(
return rc;
} else {
/* Check for swapped kdump oldmem areas */
- if (oldmem_data.start && from - oldmem_data.size < oldmem_data.size) {
- from -= oldmem_data.size;
+ if (oldmem_data.start && from - oldmem_data.start < oldmem_data.size) {
+ from -= oldmem_data.start;
len = min(count, oldmem_data.size - from);
} else if (oldmem_data.start && from < oldmem_data.size) {
len = min(count, oldmem_data.size - from);
static void show_last_breaking_event(struct pt_regs *regs)
{
printk("Last Breaking-Event-Address:\n");
- printk(" [<%016lx>] %pSR\n", regs->args[0], (void *)regs->args[0]);
+ printk(" [<%016lx>] %pSR\n", regs->last_break, (void *)regs->last_break);
}
void show_registers(struct pt_regs *regs)
static DEFINE_SPINLOCK(die_lock);
-void die(struct pt_regs *regs, const char *str)
+void __noreturn die(struct pt_regs *regs, const char *str)
{
static int die_counter;
static void __init setup_boot_command_line(void)
{
/* copy arch command line */
- strlcpy(boot_command_line, early_command_line, ARCH_COMMAND_LINE_SIZE);
+ strlcpy(boot_command_line, early_command_line, COMMAND_LINE_SIZE);
}
static void __init check_image_bootable(void)
void __init startup_init(void)
{
+ sclp_early_adjust_va();
reset_tod_clock();
check_image_bootable();
time_early_init();
_LPP_OFFSET = __LC_LPP
+ .macro STBEAR address
+ ALTERNATIVE "", ".insn s,0xb2010000,\address", 193
+ .endm
+
+ .macro LBEAR address
+ ALTERNATIVE "", ".insn s,0xb2000000,\address", 193
+ .endm
+
+ .macro LPSWEY address,lpswe
+ ALTERNATIVE "b \lpswe", ".insn siy,0xeb0000000071,\address,0", 193
+ .endm
+
+ .macro MBEAR reg
+ ALTERNATIVE "", __stringify(mvc __PT_LAST_BREAK(8,\reg),__LC_LAST_BREAK), 193
+ .endm
+
.macro CHECK_STACK savearea
#ifdef CONFIG_CHECK_STACK
tml %r15,STACK_SIZE - CONFIG_STACK_GUARD
BPOFF
lghi %r14,0
.Lsysc_per:
+ STBEAR __LC_LAST_BREAK
lctlg %c1,%c1,__LC_KERNEL_ASCE
lg %r12,__LC_CURRENT
lg %r15,__LC_KERNEL_STACK
xgr %r11,%r11
la %r2,STACK_FRAME_OVERHEAD(%r15) # pointer to pt_regs
mvc __PT_R8(64,%r2),__LC_SAVE_AREA_SYNC
+ MBEAR %r2
lgr %r3,%r14
brasl %r14,__do_syscall
lctlg %c1,%c1,__LC_USER_ASCE
mvc __LC_RETURN_PSW(16),STACK_FRAME_OVERHEAD+__PT_PSW(%r15)
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
+ LBEAR STACK_FRAME_OVERHEAD+__PT_LAST_BREAK(%r15)
lmg %r0,%r15,STACK_FRAME_OVERHEAD+__PT_R0(%r15)
stpt __LC_EXIT_TIMER
- b __LC_RETURN_LPSWE
+ LPSWEY __LC_RETURN_PSW,__LC_RETURN_LPSWE
ENDPROC(system_call)
#
lctlg %c1,%c1,__LC_USER_ASCE
mvc __LC_RETURN_PSW(16),STACK_FRAME_OVERHEAD+__PT_PSW(%r15)
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
+ LBEAR STACK_FRAME_OVERHEAD+__PT_LAST_BREAK(%r15)
lmg %r0,%r15,STACK_FRAME_OVERHEAD+__PT_R0(%r15)
stpt __LC_EXIT_TIMER
- b __LC_RETURN_LPSWE
+ LPSWEY __LC_RETURN_PSW,__LC_RETURN_LPSWE
ENDPROC(ret_from_fork)
/*
xc __SF_BACKCHAIN(8,%r15),__SF_BACKCHAIN(%r15)
stmg %r0,%r7,__PT_R0(%r11)
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
+ mvc __PT_LAST_BREAK(8,%r11),__LC_PGM_LAST_BREAK
stmg %r8,%r9,__PT_PSW(%r11)
# clear user controlled registers to prevent speculative use
stpt __LC_EXIT_TIMER
.Lpgm_exit_kernel:
mvc __LC_RETURN_PSW(16),STACK_FRAME_OVERHEAD+__PT_PSW(%r15)
+ LBEAR STACK_FRAME_OVERHEAD+__PT_LAST_BREAK(%r15)
lmg %r0,%r15,STACK_FRAME_OVERHEAD+__PT_R0(%r15)
- b __LC_RETURN_LPSWE
+ LPSWEY __LC_RETURN_PSW,__LC_RETURN_LPSWE
#
# single stepped system call
larl %r14,.Lsysc_per
stg %r14,__LC_RETURN_PSW+8
lghi %r14,1
- lpswe __LC_RETURN_PSW # branch to .Lsysc_per
+ LBEAR __LC_PGM_LAST_BREAK
+ LPSWEY __LC_RETURN_PSW,__LC_RETURN_LPSWE # branch to .Lsysc_per
ENDPROC(pgm_check_handler)
/*
ENTRY(\name)
STCK __LC_INT_CLOCK
stpt __LC_SYS_ENTER_TIMER
+ STBEAR __LC_LAST_BREAK
BPOFF
stmg %r8,%r15,__LC_SAVE_AREA_ASYNC
lg %r12,__LC_CURRENT
xgr %r10,%r10
xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
+ MBEAR %r11
stmg %r8,%r9,__PT_PSW(%r11)
tm %r8,0x0001 # coming from user space?
jno 1f
lctlg %c1,%c1,__LC_USER_ASCE
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
stpt __LC_EXIT_TIMER
-2: lmg %r0,%r15,__PT_R0(%r11)
- b __LC_RETURN_LPSWE
+2: LBEAR __PT_LAST_BREAK(%r11)
+ lmg %r0,%r15,__PT_R0(%r11)
+ LPSWEY __LC_RETURN_PSW,__LC_RETURN_LPSWE
ENDPROC(\name)
.endm
BPOFF
la %r1,4095 # validate r1
spt __LC_CPU_TIMER_SAVE_AREA-4095(%r1) # validate cpu timer
+ LBEAR __LC_LAST_BREAK_SAVE_AREA-4095(%r1) # validate bear
lmg %r0,%r15,__LC_GPREGS_SAVE_AREA-4095(%r1)# validate gprs
lg %r12,__LC_CURRENT
lmg %r8,%r9,__LC_MCK_OLD_PSW
jno 0f
BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
stpt __LC_EXIT_TIMER
-0: lmg %r11,%r15,__PT_R11(%r11)
- b __LC_RETURN_MCCK_LPSWE
+0: ALTERNATIVE "", __stringify(lghi %r12,__LC_LAST_BREAK_SAVE_AREA),193
+ LBEAR 0(%r12)
+ lmg %r11,%r15,__PT_R11(%r11)
+ LPSWEY __LC_RETURN_MCCK_PSW,__LC_RETURN_MCCK_LPSWE
.Lmcck_panic:
/*
#define __amode31_data __section(".amode31.data")
#define __amode31_ref __section(".amode31.refs")
extern long _start_amode31_refs[], _end_amode31_refs[];
+extern unsigned long __amode31_base;
#endif /* _ENTRY_H */
#include <linux/kprobes.h>
#include <trace/syscall.h>
#include <asm/asm-offsets.h>
+#include <asm/text-patching.h>
#include <asm/cacheflush.h>
#include <asm/ftrace.lds.h>
#include <asm/nospec-branch.h>
#ifdef CONFIG_MODULES
static char *ftrace_plt;
-
-asm(
- " .data\n"
- "ftrace_plt_template:\n"
- " basr %r1,%r0\n"
- " lg %r1,0f-.(%r1)\n"
- " br %r1\n"
- "0: .quad ftrace_caller\n"
- "ftrace_plt_template_end:\n"
- " .previous\n"
-);
#endif /* CONFIG_MODULES */
static const char *ftrace_shared_hotpatch_trampoline(const char **end)
bool ftrace_need_init_nop(void)
{
- return ftrace_shared_hotpatch_trampoline(NULL);
+ return true;
}
int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec)
return 0;
}
-static void ftrace_generate_nop_insn(struct ftrace_insn *insn)
-{
- /* brcl 0,0 */
- insn->opc = 0xc004;
- insn->disp = 0;
-}
-
-static void ftrace_generate_call_insn(struct ftrace_insn *insn,
- unsigned long ip)
-{
- unsigned long target;
-
- /* brasl r0,ftrace_caller */
- target = FTRACE_ADDR;
-#ifdef CONFIG_MODULES
- if (is_module_addr((void *)ip))
- target = (unsigned long)ftrace_plt;
-#endif /* CONFIG_MODULES */
- insn->opc = 0xc005;
- insn->disp = (target - ip) / 2;
-}
-
static void brcl_disable(void *brcl)
{
u8 op = 0x04; /* set mask field to zero */
int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec,
unsigned long addr)
{
- struct ftrace_insn orig, new, old;
-
- if (ftrace_shared_hotpatch_trampoline(NULL)) {
- brcl_disable((void *)rec->ip);
- return 0;
- }
-
- if (copy_from_kernel_nofault(&old, (void *) rec->ip, sizeof(old)))
- return -EFAULT;
- /* Replace ftrace call with a nop. */
- ftrace_generate_call_insn(&orig, rec->ip);
- ftrace_generate_nop_insn(&new);
-
- /* Verify that the to be replaced code matches what we expect. */
- if (memcmp(&orig, &old, sizeof(old)))
- return -EINVAL;
- s390_kernel_write((void *) rec->ip, &new, sizeof(new));
+ brcl_disable((void *)rec->ip);
return 0;
}
int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
{
- struct ftrace_insn orig, new, old;
-
- if (ftrace_shared_hotpatch_trampoline(NULL)) {
- brcl_enable((void *)rec->ip);
- return 0;
- }
-
- if (copy_from_kernel_nofault(&old, (void *) rec->ip, sizeof(old)))
- return -EFAULT;
- /* Replace nop with an ftrace call. */
- ftrace_generate_nop_insn(&orig);
- ftrace_generate_call_insn(&new, rec->ip);
-
- /* Verify that the to be replaced code matches what we expect. */
- if (memcmp(&orig, &old, sizeof(old)))
- return -EINVAL;
- s390_kernel_write((void *) rec->ip, &new, sizeof(new));
+ brcl_enable((void *)rec->ip);
return 0;
}
void arch_ftrace_update_code(int command)
{
- if (ftrace_shared_hotpatch_trampoline(NULL))
- ftrace_modify_all_code(command);
- else
- ftrace_run_stop_machine(command);
-}
-
-static void __ftrace_sync(void *dummy)
-{
+ ftrace_modify_all_code(command);
}
int ftrace_arch_code_modify_post_process(void)
{
- if (ftrace_shared_hotpatch_trampoline(NULL)) {
- /* Send SIGP to the other CPUs, so they see the new code. */
- smp_call_function(__ftrace_sync, NULL, 1);
- }
+ /*
+ * Flush any pre-fetched instructions on all
+ * CPUs to make the new code visible.
+ */
+ text_poke_sync_lock();
return 0;
}
panic("cannot allocate ftrace plt\n");
start = ftrace_shared_hotpatch_trampoline(&end);
- if (!start) {
- start = ftrace_plt_template;
- end = ftrace_plt_template_end;
- }
memcpy(ftrace_plt, start, end - start);
set_memory_ro((unsigned long)ftrace_plt, 1);
return 0;
int ftrace_enable_ftrace_graph_caller(void)
{
brcl_disable(ftrace_graph_caller);
+ text_poke_sync_lock();
return 0;
}
int ftrace_disable_ftrace_graph_caller(void)
{
brcl_enable(ftrace_graph_caller);
+ text_poke_sync_lock();
return 0;
}
ENTRY(startup_continue)
larl %r1,tod_clock_base
mvc 0(16,%r1),__LC_BOOT_CLOCK
- larl %r13,.LPG1 # get base
- lctlg %c0,%c15,.Lctl-.LPG1(%r13) # load control registers
#
# Setup stack
#
.align 16
.LPG1:
.Ldw: .quad 0x0002000180000000,0x0000000000000000
-.Lctl: .quad 0x04040000 # cr0: AFP registers & secondary space
- .quad 0 # cr1: primary space segment table
- .quad 0 # cr2: dispatchable unit control table
- .quad 0 # cr3: instruction authorization
- .quad 0xffff # cr4: instruction authorization
- .quad 0 # cr5: primary-aste origin
- .quad 0 # cr6: I/O interrupts
- .quad 0 # cr7: secondary space segment table
- .quad 0x0000000000008000 # cr8: access registers translation
- .quad 0 # cr9: tracing off
- .quad 0 # cr10: tracing off
- .quad 0 # cr11: tracing off
- .quad 0 # cr12: tracing off
- .quad 0 # cr13: home space segment table
- .quad 0xc0000000 # cr14: machine check handling off
- .quad 0 # cr15: linkage stack operations
buf = vzalloc(report->size);
if (!buf)
- return ERR_PTR(-ENOMEM);
+ goto out;
ptr = buf;
memcpy(ptr, report->ipib, report->ipib->hdr.len);
}
BUG_ON(ptr > buf + report->size);
+out:
return buf;
}
irq_enter();
- if (user_mode(regs))
+ if (user_mode(regs)) {
update_timer_sys();
+ if (static_branch_likely(&cpu_has_bear))
+ current->thread.last_break = regs->last_break;
+ }
from_idle = !user_mode(regs) && regs->psw.addr == (unsigned long)psw_idle_exit;
if (from_idle)
irq_enter();
- if (user_mode(regs))
+ if (user_mode(regs)) {
update_timer_sys();
+ if (static_branch_likely(&cpu_has_bear))
+ current->thread.last_break = regs->last_break;
+ }
regs->int_code = S390_lowcore.ext_int_code_addr;
regs->int_parm = S390_lowcore.ext_params;
* Author(s): Jan Glauber <jang@linux.vnet.ibm.com>
*/
#include <linux/uaccess.h>
-#include <linux/stop_machine.h>
#include <linux/jump_label.h>
+#include <linux/module.h>
+#include <asm/text-patching.h>
#include <asm/ipl.h>
struct insn {
.offset = JUMP_LABEL_NOP_OFFSET >> 1,
};
-static void __jump_label_transform(struct jump_entry *entry,
- enum jump_label_type type,
- int init)
+static void jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type,
+ int init)
{
void *code = (void *)jump_entry_code(entry);
struct insn old, new;
s390_kernel_write(code, &new, sizeof(new));
}
-static void __jump_label_sync(void *dummy)
+void arch_jump_label_transform(struct jump_entry *entry,
+ enum jump_label_type type)
{
+ jump_label_transform(entry, type, 0);
+ text_poke_sync();
}
-void arch_jump_label_transform(struct jump_entry *entry,
- enum jump_label_type type)
+bool arch_jump_label_transform_queue(struct jump_entry *entry,
+ enum jump_label_type type)
+{
+ jump_label_transform(entry, type, 0);
+ return true;
+}
+
+void arch_jump_label_transform_apply(void)
{
- __jump_label_transform(entry, type, 0);
- smp_call_function(__jump_label_sync, NULL, 1);
+ text_poke_sync();
}
-void arch_jump_label_transform_static(struct jump_entry *entry,
- enum jump_label_type type)
+void __init_or_module arch_jump_label_transform_static(struct jump_entry *entry,
+ enum jump_label_type type)
{
- __jump_label_transform(entry, type, 1);
+ jump_label_transform(entry, type, 1);
+ text_poke_sync();
}
}
NOKPROBE_SYMBOL(s390_free_insn_slot);
+/* Check if paddr is at an instruction boundary */
+static bool can_probe(unsigned long paddr)
+{
+ unsigned long addr, offset = 0;
+ kprobe_opcode_t insn;
+ struct kprobe *kp;
+
+ if (paddr & 0x01)
+ return false;
+
+ if (!kallsyms_lookup_size_offset(paddr, NULL, &offset))
+ return false;
+
+ /* Decode instructions */
+ addr = paddr - offset;
+ while (addr < paddr) {
+ if (copy_from_kernel_nofault(&insn, (void *)addr, sizeof(insn)))
+ return false;
+
+ if (insn >> 8 == 0) {
+ if (insn != BREAKPOINT_INSTRUCTION) {
+ /*
+ * Note that QEMU inserts opcode 0x0000 to implement
+ * software breakpoints for guests. Since the size of
+ * the original instruction is unknown, stop following
+ * instructions and prevent setting a kprobe.
+ */
+ return false;
+ }
+ /*
+ * Check if the instruction has been modified by another
+ * kprobe, in which case the original instruction is
+ * decoded.
+ */
+ kp = get_kprobe((void *)addr);
+ if (!kp) {
+ /* not a kprobe */
+ return false;
+ }
+ insn = kp->opcode;
+ }
+ addr += insn_length(insn >> 8);
+ }
+ return addr == paddr;
+}
+
int arch_prepare_kprobe(struct kprobe *p)
{
- if ((unsigned long) p->addr & 0x01)
+ if (!can_probe((unsigned long)p->addr))
return -EINVAL;
/* Make sure the probe isn't going on a difficult instruction */
if (probe_is_prohibited_opcode(p->addr))
#include <linux/kexec.h>
#include <linux/module_signature.h>
#include <linux/verification.h>
+#include <linux/vmalloc.h>
#include <asm/boot_data.h>
#include <asm/ipl.h>
#include <asm/setup.h>
struct kexec_buf buf;
unsigned long addr;
void *ptr, *end;
+ int ret;
buf.image = image;
ptr += len;
}
+ ret = -ENOMEM;
buf.buffer = ipl_report_finish(data->report);
+ if (!buf.buffer)
+ goto out;
buf.bufsz = data->report->size;
buf.memsz = buf.bufsz;
+ image->arch.ipl_buf = buf.buffer;
data->memsz += buf.memsz;
data->kernel_buf + offsetof(struct lowcore, ipl_parmblock_ptr);
*lc_ipl_parmblock_ptr = (__u32)buf.mem;
- return kexec_add_buffer(&buf);
+ ret = kexec_add_buffer(&buf);
+out:
+ return ret;
}
void *kexec_file_add_components(struct kimage *image,
int (*add_kernel)(struct kimage *image,
struct s390_load_data *data))
{
+ unsigned long max_command_line_size = LEGACY_COMMAND_LINE_SIZE;
struct s390_load_data data = {0};
+ unsigned long minsize;
int ret;
data.report = ipl_report_init(&ipl_block);
if (ret)
goto out;
- if (image->cmdline_buf_len >= ARCH_COMMAND_LINE_SIZE) {
- ret = -EINVAL;
+ ret = -EINVAL;
+ minsize = PARMAREA + offsetof(struct parmarea, command_line);
+ if (image->kernel_buf_len < minsize)
goto out;
- }
+
+ if (data.parm->max_command_line_size)
+ max_command_line_size = data.parm->max_command_line_size;
+
+ if (minsize + max_command_line_size < minsize)
+ goto out;
+
+ if (image->kernel_buf_len < minsize + max_command_line_size)
+ goto out;
+
+ if (image->cmdline_buf_len >= max_command_line_size)
+ goto out;
+
memcpy(data.parm->command_line, image->cmdline_buf,
image->cmdline_buf_len);
return 0;
}
-int arch_kexec_kernel_image_probe(struct kimage *image, void *buf,
- unsigned long buf_len)
+int arch_kimage_file_post_load_cleanup(struct kimage *image)
{
- /* A kernel must be at least large enough to contain head.S. During
- * load memory in head.S will be accessed, e.g. to register the next
- * command line. If the next kernel were smaller the current kernel
- * will panic at load.
- */
- if (buf_len < HEAD_END)
- return -ENOEXEC;
-
- return kexec_image_probe_default(image, buf, buf_len);
+ vfree(image->arch.ipl_buf);
+ image->arch.ipl_buf = NULL;
+
+ return kexec_image_post_load_cleanup_default(image);
}
BR_EX %r14
ENDPROC(ftrace_stub)
-#define STACK_FRAME_SIZE (STACK_FRAME_OVERHEAD + __PT_SIZE)
-#define STACK_PTREGS (STACK_FRAME_OVERHEAD)
-#define STACK_PTREGS_GPRS (STACK_PTREGS + __PT_GPRS)
-#define STACK_PTREGS_PSW (STACK_PTREGS + __PT_PSW)
+#define STACK_FRAME_SIZE (STACK_FRAME_OVERHEAD + __PT_SIZE)
+#define STACK_PTREGS (STACK_FRAME_OVERHEAD)
+#define STACK_PTREGS_GPRS (STACK_PTREGS + __PT_GPRS)
+#define STACK_PTREGS_PSW (STACK_PTREGS + __PT_PSW)
+#define STACK_PTREGS_ORIG_GPR2 (STACK_PTREGS + __PT_ORIG_GPR2)
#ifdef __PACK_STACK
/* allocate just enough for r14, r15 and backchain */
#define TRACED_FUNC_FRAME_SIZE 24
#define TRACED_FUNC_FRAME_SIZE STACK_FRAME_OVERHEAD
#endif
-ENTRY(ftrace_caller)
- .globl ftrace_regs_caller
- .set ftrace_regs_caller,ftrace_caller
+ .macro ftrace_regs_entry, allregs=0
stg %r14,(__SF_GPRS+8*8)(%r15) # save traced function caller
+
+ .if \allregs == 1
lghi %r14,0 # save condition code
ipm %r14 # don't put any instructions
sllg %r14,%r14,16 # clobbering CC before this point
+ .endif
+
lgr %r1,%r15
# allocate stack frame for ftrace_caller to contain traced function
aghi %r15,-TRACED_FUNC_FRAME_SIZE
# allocate pt_regs and stack frame for ftrace_trace_function
aghi %r15,-STACK_FRAME_SIZE
stg %r1,(STACK_PTREGS_GPRS+15*8)(%r15)
+ xc STACK_PTREGS_ORIG_GPR2(8,%r15),STACK_PTREGS_ORIG_GPR2(%r15)
+
+ .if \allregs == 1
stg %r14,(STACK_PTREGS_PSW)(%r15)
- lg %r14,(__SF_GPRS+8*8)(%r1) # restore original return address
stosm (STACK_PTREGS_PSW)(%r15),0
+ .endif
+
+ lg %r14,(__SF_GPRS+8*8)(%r1) # restore original return address
aghi %r1,-TRACED_FUNC_FRAME_SIZE
stg %r1,__SF_BACKCHAIN(%r15)
stg %r0,(STACK_PTREGS_PSW+8)(%r15)
stmg %r2,%r14,(STACK_PTREGS_GPRS+2*8)(%r15)
+ .endm
+
+SYM_CODE_START(ftrace_regs_caller)
+ ftrace_regs_entry 1
+ j ftrace_common
+SYM_CODE_END(ftrace_regs_caller)
+
+SYM_CODE_START(ftrace_caller)
+ ftrace_regs_entry 0
+ j ftrace_common
+SYM_CODE_END(ftrace_caller)
+
+SYM_CODE_START(ftrace_common)
#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
aghik %r2,%r0,-MCOUNT_INSN_SIZE
lgrl %r4,function_trace_op
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
# The j instruction gets runtime patched to a nop instruction.
# See ftrace_enable_ftrace_graph_caller.
- .globl ftrace_graph_caller
-ftrace_graph_caller:
- j ftrace_graph_caller_end
+SYM_INNER_LABEL(ftrace_graph_caller, SYM_L_GLOBAL)
+ j .Lftrace_graph_caller_end
lmg %r2,%r3,(STACK_PTREGS_GPRS+14*8)(%r15)
lg %r4,(STACK_PTREGS_PSW+8)(%r15)
brasl %r14,prepare_ftrace_return
stg %r2,(STACK_PTREGS_GPRS+14*8)(%r15)
-ftrace_graph_caller_end:
- .globl ftrace_graph_caller_end
+.Lftrace_graph_caller_end:
+#endif
+ lg %r0,(STACK_PTREGS_PSW+8)(%r15)
+#ifdef CONFIG_HAVE_MARCH_Z196_FEATURES
+ ltg %r1,STACK_PTREGS_ORIG_GPR2(%r15)
+ locgrz %r1,%r0
+#else
+ lg %r1,STACK_PTREGS_ORIG_GPR2(%r15)
+ ltgr %r1,%r1
+ jnz 0f
+ lgr %r1,%r0
#endif
- lg %r1,(STACK_PTREGS_PSW+8)(%r15)
- lmg %r2,%r15,(STACK_PTREGS_GPRS+2*8)(%r15)
+0: lmg %r2,%r15,(STACK_PTREGS_GPRS+2*8)(%r15)
BR_EX %r1
-ENDPROC(ftrace_caller)
+SYM_CODE_END(ftrace_common)
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
-ENTRY(return_to_handler)
+SYM_FUNC_START(return_to_handler)
stmg %r2,%r5,32(%r15)
lgr %r1,%r15
aghi %r15,-STACK_FRAME_OVERHEAD
lgr %r14,%r2
lmg %r2,%r5,32(%r15)
BR_EX %r14
-ENDPROC(return_to_handler)
+SYM_FUNC_END(return_to_handler)
#endif
{
if (test_facility(156))
pr_info("Spectre V2 mitigation: etokens\n");
- if (__is_defined(CC_USING_EXPOLINE) && !nospec_disable)
+ if (nospec_uses_trampoline())
pr_info("Spectre V2 mitigation: execute trampolines\n");
if (__test_facility(82, alt_stfle_fac_list))
pr_info("Spectre V2 mitigation: limited branch prediction\n");
{
if (test_facility(156))
return sprintf(buf, "Mitigation: etokens\n");
- if (__is_defined(CC_USING_EXPOLINE) && !nospec_disable)
+ if (nospec_uses_trampoline())
return sprintf(buf, "Mitigation: execute trampolines\n");
if (__test_facility(82, alt_stfle_fac_list))
return sprintf(buf, "Mitigation: limited branch prediction\n");
false);
if (cfdiag_diffctr(cpuhw, event->hw.config_base))
cfdiag_push_sample(event, cpuhw);
- } else
+ } else if (cpuhw->flags & PMU_F_RESERVED) {
+ /* Only update when PMU not hotplugged off */
hw_perf_event_update(event);
+ }
hwc->state |= PERF_HES_UPTODATE;
}
}
* counter set via normal file operations.
*/
-static atomic_t cfset_opencnt = ATOMIC_INIT(0); /* Excl. access */
+static atomic_t cfset_opencnt = ATOMIC_INIT(0); /* Access count */
static DEFINE_MUTEX(cfset_ctrset_mutex);/* Synchronize access to hardware */
struct cfset_call_on_cpu_parm { /* Parm struct for smp_call_on_cpu */
unsigned int sets; /* Counter set bit mask */
atomic_t cpus_ack; /* # CPUs successfully executed func */
};
-static struct cfset_request { /* CPUs and counter set bit mask */
+static struct cfset_session { /* CPUs and counter set bit mask */
+ struct list_head head; /* Head of list of active processes */
+} cfset_session = {
+ .head = LIST_HEAD_INIT(cfset_session.head)
+};
+
+struct cfset_request { /* CPUs and counter set bit mask */
unsigned long ctrset; /* Bit mask of counter set to read */
cpumask_t mask; /* CPU mask to read from */
-} cfset_request;
+ struct list_head node; /* Chain to cfset_session.head */
+};
+
+static void cfset_session_init(void)
+{
+ INIT_LIST_HEAD(&cfset_session.head);
+}
+
+/* Remove current request from global bookkeeping. Maintain a counter set bit
+ * mask on a per CPU basis.
+ * Done in process context under mutex protection.
+ */
+static void cfset_session_del(struct cfset_request *p)
+{
+ list_del(&p->node);
+}
-static void cfset_ctrset_clear(void)
+/* Add current request to global bookkeeping. Maintain a counter set bit mask
+ * on a per CPU basis.
+ * Done in process context under mutex protection.
+ */
+static void cfset_session_add(struct cfset_request *p)
{
- cpumask_clear(&cfset_request.mask);
- cfset_request.ctrset = 0;
+ list_add(&p->node, &cfset_session.head);
}
/* The /dev/hwctr device access uses PMU_F_IN_USE to mark the device access
struct cfset_call_on_cpu_parm *p = parm;
int rc;
- cpuhw->dev_state = 0;
+ /* Check if any counter set used by /dev/hwc */
for (rc = CPUMF_CTR_SET_BASIC; rc < CPUMF_CTR_SET_MAX; ++rc)
- if ((p->sets & cpumf_ctr_ctl[rc]))
- atomic_dec(&cpuhw->ctr_set[rc]);
- rc = lcctl(cpuhw->state); /* Keep perf_event_open counter sets */
+ if ((p->sets & cpumf_ctr_ctl[rc])) {
+ if (!atomic_dec_return(&cpuhw->ctr_set[rc])) {
+ ctr_set_disable(&cpuhw->dev_state,
+ cpumf_ctr_ctl[rc]);
+ ctr_set_stop(&cpuhw->dev_state,
+ cpumf_ctr_ctl[rc]);
+ }
+ }
+ /* Keep perf_event_open counter sets */
+ rc = lcctl(cpuhw->dev_state | cpuhw->state);
if (rc)
pr_err("Counter set stop %#llx of /dev/%s failed rc=%i\n",
cpuhw->state, S390_HWCTR_DEVICE, rc);
- cpuhw->flags &= ~PMU_F_IN_USE;
+ if (!cpuhw->dev_state)
+ cpuhw->flags &= ~PMU_F_IN_USE;
debug_sprintf_event(cf_dbg, 4, "%s rc %d state %#llx dev_state %#llx\n",
__func__, rc, cpuhw->state, cpuhw->dev_state);
}
debug_sprintf_event(cf_dbg, 4, "%s state %#llx dev_state %#llx\n",
__func__, cpuhw->state, cpuhw->dev_state);
+ cpuhw->dev_state = 0;
rc = lcctl(cpuhw->state); /* Keep perf_event_open counter sets */
if (rc)
pr_err("Counter set release %#llx of /dev/%s failed rc=%i\n",
cpuhw->state, S390_HWCTR_DEVICE, rc);
- cpuhw->dev_state = 0;
+}
+
+/* This modifies the process CPU mask to adopt it to the currently online
+ * CPUs. Offline CPUs can not be addresses. This call terminates the access
+ * and is usually followed by close() or a new iotcl(..., START, ...) which
+ * creates a new request structure.
+ */
+static void cfset_all_stop(struct cfset_request *req)
+{
+ struct cfset_call_on_cpu_parm p = {
+ .sets = req->ctrset,
+ };
+
+ cpumask_and(&req->mask, &req->mask, cpu_online_mask);
+ on_each_cpu_mask(&req->mask, cfset_ioctl_off, &p, 1);
}
/* Release function is also called when application gets terminated without
*/
static int cfset_release(struct inode *inode, struct file *file)
{
- on_each_cpu(cfset_release_cpu, NULL, 1);
+ mutex_lock(&cfset_ctrset_mutex);
+ /* Open followed by close/exit has no private_data */
+ if (file->private_data) {
+ cfset_all_stop(file->private_data);
+ cfset_session_del(file->private_data);
+ kfree(file->private_data);
+ file->private_data = NULL;
+ }
+ if (!atomic_dec_return(&cfset_opencnt))
+ on_each_cpu(cfset_release_cpu, NULL, 1);
+ mutex_unlock(&cfset_ctrset_mutex);
+
hw_perf_event_destroy(NULL);
- cfset_ctrset_clear();
- atomic_set(&cfset_opencnt, 0);
return 0;
}
{
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- /* Only one user space program can open /dev/hwctr */
- if (atomic_xchg(&cfset_opencnt, 1))
- return -EBUSY;
+ mutex_lock(&cfset_ctrset_mutex);
+ if (atomic_inc_return(&cfset_opencnt) == 1)
+ cfset_session_init();
+ mutex_unlock(&cfset_ctrset_mutex);
cpumf_hw_inuse();
file->private_data = NULL;
return nonseekable_open(inode, file);
}
-static int cfset_all_stop(void)
-{
- struct cfset_call_on_cpu_parm p = {
- .sets = cfset_request.ctrset,
- };
- cpumask_var_t mask;
-
- if (!alloc_cpumask_var(&mask, GFP_KERNEL))
- return -ENOMEM;
- cpumask_and(mask, &cfset_request.mask, cpu_online_mask);
- on_each_cpu_mask(mask, cfset_ioctl_off, &p, 1);
- free_cpumask_var(mask);
- return 0;
-}
-
-static int cfset_all_start(void)
+static int cfset_all_start(struct cfset_request *req)
{
struct cfset_call_on_cpu_parm p = {
- .sets = cfset_request.ctrset,
+ .sets = req->ctrset,
.cpus_ack = ATOMIC_INIT(0),
};
cpumask_var_t mask;
if (!alloc_cpumask_var(&mask, GFP_KERNEL))
return -ENOMEM;
- cpumask_and(mask, &cfset_request.mask, cpu_online_mask);
+ cpumask_and(mask, &req->mask, cpu_online_mask);
on_each_cpu_mask(mask, cfset_ioctl_on, &p, 1);
if (atomic_read(&p.cpus_ack) != cpumask_weight(mask)) {
on_each_cpu_mask(mask, cfset_ioctl_off, &p, 1);
cpuhw->sets, cpuhw->used);
}
-static int cfset_all_read(unsigned long arg)
+static int cfset_all_read(unsigned long arg, struct cfset_request *req)
{
struct cfset_call_on_cpu_parm p;
cpumask_var_t mask;
if (!alloc_cpumask_var(&mask, GFP_KERNEL))
return -ENOMEM;
- p.sets = cfset_request.ctrset;
- cpumask_and(mask, &cfset_request.mask, cpu_online_mask);
+ p.sets = req->ctrset;
+ cpumask_and(mask, &req->mask, cpu_online_mask);
on_each_cpu_mask(mask, cfset_cpu_read, &p, 1);
rc = cfset_all_copy(arg, mask);
free_cpumask_var(mask);
return rc;
}
-static long cfset_ioctl_read(unsigned long arg)
+static long cfset_ioctl_read(unsigned long arg, struct cfset_request *req)
{
struct s390_ctrset_read read;
- int ret = 0;
+ int ret = -ENODATA;
- if (copy_from_user(&read, (char __user *)arg, sizeof(read)))
- return -EFAULT;
- ret = cfset_all_read(arg);
+ if (req && req->ctrset) {
+ if (copy_from_user(&read, (char __user *)arg, sizeof(read)))
+ return -EFAULT;
+ ret = cfset_all_read(arg, req);
+ }
return ret;
}
-static long cfset_ioctl_stop(void)
+static long cfset_ioctl_stop(struct file *file)
{
- int ret = ENXIO;
-
- if (cfset_request.ctrset) {
- ret = cfset_all_stop();
- cfset_ctrset_clear();
+ struct cfset_request *req = file->private_data;
+ int ret = -ENXIO;
+
+ if (req) {
+ cfset_all_stop(req);
+ cfset_session_del(req);
+ kfree(req);
+ file->private_data = NULL;
+ ret = 0;
}
return ret;
}
-static long cfset_ioctl_start(unsigned long arg)
+static long cfset_ioctl_start(unsigned long arg, struct file *file)
{
struct s390_ctrset_start __user *ustart;
struct s390_ctrset_start start;
+ struct cfset_request *preq;
void __user *umask;
unsigned int len;
int ret = 0;
size_t need;
- if (cfset_request.ctrset)
+ if (file->private_data)
return -EBUSY;
ustart = (struct s390_ctrset_start __user *)arg;
if (copy_from_user(&start, ustart, sizeof(start)))
return -EINVAL; /* Invalid counter set */
if (!start.counter_sets)
return -EINVAL; /* No counter set at all? */
- cpumask_clear(&cfset_request.mask);
+
+ preq = kzalloc(sizeof(*preq), GFP_KERNEL);
+ if (!preq)
+ return -ENOMEM;
+ cpumask_clear(&preq->mask);
len = min_t(u64, start.cpumask_len, cpumask_size());
umask = (void __user *)start.cpumask;
- if (copy_from_user(&cfset_request.mask, umask, len))
+ if (copy_from_user(&preq->mask, umask, len)) {
+ kfree(preq);
return -EFAULT;
- if (cpumask_empty(&cfset_request.mask))
+ }
+ if (cpumask_empty(&preq->mask)) {
+ kfree(preq);
return -EINVAL;
+ }
need = cfset_needspace(start.counter_sets);
- if (put_user(need, &ustart->data_bytes))
- ret = -EFAULT;
- if (ret)
- goto out;
- cfset_request.ctrset = start.counter_sets;
- ret = cfset_all_start();
-out:
- if (ret)
- cfset_ctrset_clear();
- debug_sprintf_event(cf_dbg, 4, "%s sets %#lx need %ld ret %d\n",
- __func__, cfset_request.ctrset, need, ret);
+ if (put_user(need, &ustart->data_bytes)) {
+ kfree(preq);
+ return -EFAULT;
+ }
+ preq->ctrset = start.counter_sets;
+ ret = cfset_all_start(preq);
+ if (!ret) {
+ cfset_session_add(preq);
+ file->private_data = preq;
+ debug_sprintf_event(cf_dbg, 4, "%s set %#lx need %ld ret %d\n",
+ __func__, preq->ctrset, need, ret);
+ } else {
+ kfree(preq);
+ }
return ret;
}
* counter set keeps running until explicitly stopped. Returns the number
* of bytes needed to store the counter values. If another S390_HWCTR_START
* ioctl subcommand is called without a previous S390_HWCTR_STOP stop
- * command, -EBUSY is returned.
+ * command on the same file descriptor, -EBUSY is returned.
* S390_HWCTR_READ: Read the counter set values from specified CPU list given
* with the S390_HWCTR_START command.
* S390_HWCTR_STOP: Stops the counter sets on the CPU list given with the
mutex_lock(&cfset_ctrset_mutex);
switch (cmd) {
case S390_HWCTR_START:
- ret = cfset_ioctl_start(arg);
+ ret = cfset_ioctl_start(arg, file);
break;
case S390_HWCTR_STOP:
- ret = cfset_ioctl_stop();
+ ret = cfset_ioctl_stop(file);
break;
case S390_HWCTR_READ:
- ret = cfset_ioctl_read(arg);
+ ret = cfset_ioctl_read(arg, file->private_data);
break;
default:
ret = -ENOTTY;
.fops = &cfset_fops,
};
+/* Hotplug add of a CPU. Scan through all active processes and add
+ * that CPU to the list of CPUs supplied with ioctl(..., START, ...).
+ */
int cfset_online_cpu(unsigned int cpu)
{
struct cfset_call_on_cpu_parm p;
+ struct cfset_request *rp;
mutex_lock(&cfset_ctrset_mutex);
- if (cfset_request.ctrset) {
- p.sets = cfset_request.ctrset;
- cfset_ioctl_on(&p);
- cpumask_set_cpu(cpu, &cfset_request.mask);
+ if (!list_empty(&cfset_session.head)) {
+ list_for_each_entry(rp, &cfset_session.head, node) {
+ p.sets = rp->ctrset;
+ cfset_ioctl_on(&p);
+ cpumask_set_cpu(cpu, &rp->mask);
+ }
}
mutex_unlock(&cfset_ctrset_mutex);
return 0;
}
+/* Hotplug remove of a CPU. Scan through all active processes and clear
+ * that CPU from the list of CPUs supplied with ioctl(..., START, ...).
+ */
int cfset_offline_cpu(unsigned int cpu)
{
struct cfset_call_on_cpu_parm p;
+ struct cfset_request *rp;
mutex_lock(&cfset_ctrset_mutex);
- if (cfset_request.ctrset) {
- p.sets = cfset_request.ctrset;
- cfset_ioctl_off(&p);
- cpumask_clear_cpu(cpu, &cfset_request.mask);
+ if (!list_empty(&cfset_session.head)) {
+ list_for_each_entry(rp, &cfset_session.head, node) {
+ p.sets = rp->ctrset;
+ cfset_ioctl_off(&p);
+ cpumask_clear_cpu(cpu, &rp->mask);
+ }
}
mutex_unlock(&cfset_ctrset_mutex);
return 0;
frame->childregs.gprs[10] = arg;
frame->childregs.gprs[11] = (unsigned long)do_exit;
frame->childregs.orig_gpr2 = -1;
-
+ frame->childregs.last_break = 1;
return 0;
}
frame->childregs = *current_pt_regs();
* relocated above 2 GB, because it has to use 31 bit addresses.
* Such code and data is part of the .amode31 section.
*/
-unsigned long __amode31_ref __samode31 = __pa(&_samode31);
-unsigned long __amode31_ref __eamode31 = __pa(&_eamode31);
-unsigned long __amode31_ref __stext_amode31 = __pa(&_stext_amode31);
-unsigned long __amode31_ref __etext_amode31 = __pa(&_etext_amode31);
+unsigned long __amode31_ref __samode31 = (unsigned long)&_samode31;
+unsigned long __amode31_ref __eamode31 = (unsigned long)&_eamode31;
+unsigned long __amode31_ref __stext_amode31 = (unsigned long)&_stext_amode31;
+unsigned long __amode31_ref __etext_amode31 = (unsigned long)&_etext_amode31;
struct exception_table_entry __amode31_ref *__start_amode31_ex_table = _start_amode31_ex_table;
struct exception_table_entry __amode31_ref *__stop_amode31_ex_table = _stop_amode31_ex_table;
struct initrd_data __bootdata(initrd_data);
unsigned long __bootdata_preserved(__kaslr_offset);
+unsigned long __bootdata(__amode31_base);
unsigned int __bootdata_preserved(zlib_dfltcc_support);
EXPORT_SYMBOL(zlib_dfltcc_support);
u64 __bootdata_preserved(stfle_fac_list[16]);
struct lowcore *lowcore_ptr[NR_CPUS];
EXPORT_SYMBOL(lowcore_ptr);
+DEFINE_STATIC_KEY_FALSE(cpu_has_bear);
+
/*
* The Write Back bit position in the physaddr is given by the SLPC PCI.
* Leaving the mask zero always uses write through which is safe
* part of the System RAM resource.
*/
if (crashk_res.end) {
- memblock_add_node(crashk_res.start, resource_size(&crashk_res), 0);
+ memblock_add_node(crashk_res.start, resource_size(&crashk_res),
+ 0, MEMBLOCK_NONE);
memblock_reserve(crashk_res.start, resource_size(&crashk_res));
insert_resource(&iomem_resource, &crashk_res);
}
static void __init setup_memory_end(void)
{
- memblock_remove(ident_map_size, ULONG_MAX);
+ memblock_remove(ident_map_size, PHYS_ADDR_MAX - ident_map_size);
max_pfn = max_low_pfn = PFN_DOWN(ident_map_size);
pr_notice("The maximum memory size is %luMB\n", ident_map_size >> 20);
}
#endif
/*
- * Make sure that the area above identity mapping is protected
- */
-static void __init reserve_above_ident_map(void)
-{
- memblock_reserve(ident_map_size, ULONG_MAX);
-}
-
-/*
* Reserve memory for kdump kernel to be loaded with kexec
*/
static void __init reserve_crashkernel(void)
}
if (register_memory_notifier(&kdump_mem_nb)) {
- memblock_free(crash_base, crash_size);
+ memblock_phys_free(crash_base, crash_size);
return;
}
#ifdef CONFIG_BLK_DEV_INITRD
if (!initrd_data.start || !initrd_data.size)
return;
- initrd_start = initrd_data.start;
+ initrd_start = (unsigned long)__va(initrd_data.start);
initrd_end = initrd_start + initrd_data.size;
memblock_reserve(initrd_data.start, initrd_data.size);
#endif
get_mem_detect_reserved(&start, &size);
if (size)
- memblock_free(start, size);
+ memblock_phys_free(start, size);
}
static const char * __init get_mem_info_source(void)
}
memblock_set_bottom_up(false);
memblock_set_node(0, ULONG_MAX, &memblock.memory, 0);
- memblock_dump_all();
}
/*
if (initrd_data.start && initrd_data.size &&
!memblock_is_region_memory(initrd_data.start, initrd_data.size)) {
pr_err("The initial RAM disk does not fit into the memory\n");
- memblock_free(initrd_data.start, initrd_data.size);
+ memblock_phys_free(initrd_data.start, initrd_data.size);
initrd_start = initrd_end = 0;
}
#endif
*/
static void __init reserve_kernel(void)
{
- unsigned long start_pfn = PFN_UP(__pa(_end));
-
memblock_reserve(0, STARTUP_NORMAL_OFFSET);
- memblock_reserve((unsigned long)sclp_early_sccb, EXT_SCCB_READ_SCP);
- memblock_reserve((unsigned long)_stext, PFN_PHYS(start_pfn)
- - (unsigned long)_stext);
+ memblock_reserve(__amode31_base, __eamode31 - __samode31);
+ memblock_reserve(__pa(sclp_early_sccb), EXT_SCCB_READ_SCP);
+ memblock_reserve(__pa(_stext), _end - _stext);
}
static void __init setup_memory(void)
storage_key_init_range(start, end);
psw_set_key(PAGE_DEFAULT_KEY);
-
- /* Only cosmetics */
- memblock_enforce_memory_limit(memblock_end_of_DRAM());
}
static void __init relocate_amode31_section(void)
{
- unsigned long amode31_addr, amode31_size;
- long amode31_offset;
+ unsigned long amode31_size = __eamode31 - __samode31;
+ long amode31_offset = __amode31_base - __samode31;
long *ptr;
- /* Allocate a new AMODE31 capable memory region */
- amode31_size = __eamode31 - __samode31;
pr_info("Relocating AMODE31 section of size 0x%08lx\n", amode31_size);
- amode31_addr = (unsigned long)memblock_alloc_low(amode31_size, PAGE_SIZE);
- if (!amode31_addr)
- panic("Failed to allocate memory for AMODE31 section\n");
- amode31_offset = amode31_addr - __samode31;
/* Move original AMODE31 section to the new one */
- memmove((void *)amode31_addr, (void *)__samode31, amode31_size);
+ memmove((void *)__amode31_base, (void *)__samode31, amode31_size);
/* Zero out the old AMODE31 section to catch invalid accesses within it */
memset((void *)__samode31, 0, amode31_size);
{
struct sysinfo_3_2_2 *vmms;
- vmms = (struct sysinfo_3_2_2 *) memblock_phys_alloc(PAGE_SIZE,
- PAGE_SIZE);
+ vmms = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
if (!vmms)
panic("Failed to allocate memory for sysinfo structure\n");
-
if (stsi(vmms, 3, 2, 2) == 0 && vmms->count)
add_device_randomness(&vmms->vm, sizeof(vmms->vm[0]) * vmms->count);
- memblock_free((unsigned long) vmms, PAGE_SIZE);
+ memblock_free(vmms, PAGE_SIZE);
}
/*
setup_control_program_code();
/* Do some memory reservations *before* memory is added to memblock */
- reserve_above_ident_map();
reserve_kernel();
reserve_initrd();
reserve_certificate_list();
reserve_mem_detect_info();
+ memblock_set_current_limit(ident_map_size);
memblock_allow_resize();
/* Get information about *all* installed memory */
memblock_add_mem_detect_info();
free_mem_detect_info();
+ setup_memory_end();
+ memblock_dump_all();
+ setup_memory();
relocate_amode31_section();
setup_cr();
-
setup_uv();
- setup_memory_end();
- setup_memory();
dma_contiguous_reserve(ident_map_size);
vmcp_cma_reserve();
if (MACHINE_HAS_EDAT2)
smp_detect_cpus();
topology_init_early();
+ if (test_facility(193))
+ static_branch_enable(&cpu_has_bear);
+
/*
* Create kernel page tables and switch to virtual addressing.
*/
/* Get the CPU registers */
smp_save_cpu_regs(sa, addr, is_boot_cpu, page);
}
- memblock_free(page, PAGE_SIZE);
+ memblock_phys_free(page, PAGE_SIZE);
diag_amode31_ops.diag308_reset();
pcpu_set_smt(0);
}
/* Add CPUs present at boot */
__smp_rescan_cpus(info, true);
- memblock_free_early((unsigned long)info, sizeof(*info));
+ memblock_phys_free((unsigned long)info, sizeof(*info));
}
/*
regs->psw = S390_lowcore.svc_old_psw;
regs->int_code = S390_lowcore.svc_int_code;
update_timer_sys();
+ if (static_branch_likely(&cpu_has_bear))
+ current->thread.last_break = regs->last_break;
local_irq_enable();
regs->orig_gpr2 = regs->gprs[2];
446 common landlock_restrict_self sys_landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 common process_mrelease sys_process_mrelease sys_process_mrelease
+449 common futex_waitv sys_futex_waitv sys_futex_waitv
{
if (user_mode(regs)) {
report_user_fault(regs, SIGSEGV, 0);
- do_exit(SIGSEGV);
+ force_exit_sig(SIGSEGV);
} else
die(regs, "Unknown program exception");
}
void noinstr __do_pgm_check(struct pt_regs *regs)
{
- unsigned long last_break = S390_lowcore.breaking_event_addr;
unsigned int trapnr;
irqentry_state_t state;
if (user_mode(regs)) {
update_timer_sys();
- if (last_break < 4096)
- last_break = 1;
- current->thread.last_break = last_break;
- regs->args[0] = last_break;
+ if (!static_branch_likely(&cpu_has_bear)) {
+ if (regs->last_break < 4096)
+ regs->last_break = 1;
+ }
+ current->thread.last_break = regs->last_break;
}
if (S390_lowcore.pgm_code & 0x0200) {
}
if (uv_init(uv_stor_base, uv_info.uv_base_stor_len)) {
- memblock_free(uv_stor_base, uv_info.uv_base_stor_len);
+ memblock_phys_free(uv_stor_base, uv_info.uv_base_stor_len);
goto fail;
}
KBUILD_CFLAGS_32 := $(filter-out -m64,$(KBUILD_CFLAGS))
KBUILD_CFLAGS_32 += -m31 -fPIC -shared -fno-common -fno-builtin
-LDFLAGS_vdso32.so.dbg += -fPIC -shared -nostdlib -soname=linux-vdso32.so.1 \
+LDFLAGS_vdso32.so.dbg += -fPIC -shared -soname=linux-vdso32.so.1 \
--hash-style=both --build-id=sha1 -melf_s390 -T
$(targets:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_32)
include $(srctree)/lib/vdso/Makefile
obj-vdso64 = vdso_user_wrapper.o note.o
obj-cvdso64 = vdso64_generic.o getcpu.o
-CFLAGS_REMOVE_getcpu.o = -pg $(CC_FLAGS_FTRACE) $(CC_FLAGS_EXPOLINE)
-CFLAGS_REMOVE_vdso64_generic.o = -pg $(CC_FLAGS_FTRACE) $(CC_FLAGS_EXPOLINE)
+VDSO_CFLAGS_REMOVE := -pg $(CC_FLAGS_FTRACE) $(CC_FLAGS_EXPOLINE) $(CC_FLAGS_CHECK_STACK)
+CFLAGS_REMOVE_getcpu.o = $(VDSO_CFLAGS_REMOVE)
+CFLAGS_REMOVE_vdso64_generic.o = $(VDSO_CFLAGS_REMOVE)
# Build rules
KBUILD_CFLAGS_64 := $(filter-out -m64,$(KBUILD_CFLAGS))
KBUILD_CFLAGS_64 += -m64 -fPIC -shared -fno-common -fno-builtin
-ldflags-y := -fPIC -shared -nostdlib -soname=linux-vdso64.so.1 \
+ldflags-y := -fPIC -shared -soname=linux-vdso64.so.1 \
--hash-style=both --build-id=sha1 -T
$(targets:%=$(obj)/%.dbg): KBUILD_CFLAGS = $(KBUILD_CFLAGS_64)
QUAD(__dynsym_start) /* dynsym_start */
QUAD(__rela_dyn_start) /* rela_dyn_start */
QUAD(__rela_dyn_end) /* rela_dyn_end */
+ QUAD(_eamode31 - _samode31) /* amode31_size */
} :NONE
/* Debugging sections. */
/* bit 1+2 of the target are the ilc, so we can directly use ilen */
rc |= put_guest_lc(vcpu, ilen, (u16 *) __LC_PGM_ILC);
rc |= put_guest_lc(vcpu, vcpu->arch.sie_block->gbea,
- (u64 *) __LC_LAST_BREAK);
+ (u64 *) __LC_PGM_LAST_BREAK);
rc |= put_guest_lc(vcpu, pgm_info.code,
(u16 *)__LC_PGM_INT_CODE);
rc |= write_guest_lc(vcpu, __LC_PGM_OLD_PSW,
r = KVM_MAX_VCPUS;
else if (sclp.has_esca && sclp.has_64bscao)
r = KVM_S390_ESCA_CPU_SLOTS;
+ if (ext == KVM_CAP_NR_VCPUS)
+ r = min_t(unsigned int, num_online_cpus(), r);
break;
case KVM_CAP_S390_COW:
r = MACHINE_HAS_ESOP;
obj-y += mem.o xor.o
lib-$(CONFIG_KPROBES) += probes.o
lib-$(CONFIG_UPROBES) += probes.o
+obj-$(CONFIG_S390_KPROBES_SANITY_TEST) += test_kprobes_s390.o
+test_kprobes_s390-objs += test_kprobes_asm.o test_kprobes.o
# Instrumenting memory accesses to __user data (in different address space)
# produce false positives
}
early_initcall(spin_retry_init);
-/**
+/*
* spin_retry= parameter
*/
static int __init spin_retry_setup(char *str)
#endif
/**
- * strlcpy - Copy a %NUL terminated string into a sized buffer
- * @dest: Where to copy the string to
- * @src: Where to copy the string from
- * @size: size of destination buffer
- *
- * Compatible with *BSD: the result is always a valid
- * NUL-terminated string that fits in the buffer (unless,
- * of course, the buffer size is zero). It does not pad
- * out the result like strncpy() does.
- */
-#ifdef __HAVE_ARCH_STRLCPY
-size_t strlcpy(char *dest, const char *src, size_t size)
-{
- size_t ret = __strend(src) - src;
-
- if (size) {
- size_t len = (ret >= size) ? size-1 : ret;
- dest[len] = '\0';
- memcpy(dest, src, len);
- }
- return ret;
-}
-EXPORT_SYMBOL(strlcpy);
-#endif
-
-/**
* strncpy - Copy a length-limited, %NUL-terminated string
* @dest: Where to copy the string to
* @src: Where to copy the string from
EXPORT_SYMBOL(strcmp);
#endif
-/**
- * strrchr - Find the last occurrence of a character in a string
- * @s: The string to be searched
- * @c: The character to search for
- */
-#ifdef __HAVE_ARCH_STRRCHR
-char *strrchr(const char *s, int c)
-{
- ssize_t len = __strend(s) - s;
-
- do {
- if (s[len] == (char)c)
- return (char *)s + len;
- } while (--len >= 0);
- return NULL;
-}
-EXPORT_SYMBOL(strrchr);
-#endif
-
static inline int clcle(const char *s1, unsigned long l1,
const char *s2, unsigned long l2)
{
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+
+
+#include <linux/kernel.h>
+#include <linux/kprobes.h>
+#include <linux/random.h>
+#include <kunit/test.h>
+#include "test_kprobes.h"
+
+static struct kprobe kp;
+
+static void setup_kprobe(struct kunit *test, struct kprobe *kp,
+ const char *symbol, int offset)
+{
+ kp->offset = offset;
+ kp->addr = NULL;
+ kp->symbol_name = symbol;
+}
+
+static void test_kprobe_offset(struct kunit *test, struct kprobe *kp,
+ const char *target, int offset)
+{
+ int ret;
+
+ setup_kprobe(test, kp, target, 0);
+ ret = register_kprobe(kp);
+ if (!ret)
+ unregister_kprobe(kp);
+ KUNIT_EXPECT_EQ(test, 0, ret);
+ setup_kprobe(test, kp, target, offset);
+ ret = register_kprobe(kp);
+ KUNIT_EXPECT_EQ(test, -EINVAL, ret);
+ if (!ret)
+ unregister_kprobe(kp);
+}
+
+static void test_kprobe_odd(struct kunit *test)
+{
+ test_kprobe_offset(test, &kp, "kprobes_target_odd",
+ kprobes_target_odd_offs);
+}
+
+static void test_kprobe_in_insn4(struct kunit *test)
+{
+ test_kprobe_offset(test, &kp, "kprobes_target_in_insn4",
+ kprobes_target_in_insn4_offs);
+}
+
+static void test_kprobe_in_insn6_lo(struct kunit *test)
+{
+ test_kprobe_offset(test, &kp, "kprobes_target_in_insn6_lo",
+ kprobes_target_in_insn6_lo_offs);
+}
+
+static void test_kprobe_in_insn6_hi(struct kunit *test)
+{
+ test_kprobe_offset(test, &kp, "kprobes_target_in_insn6_hi",
+ kprobes_target_in_insn6_hi_offs);
+}
+
+static struct kunit_case kprobes_testcases[] = {
+ KUNIT_CASE(test_kprobe_odd),
+ KUNIT_CASE(test_kprobe_in_insn4),
+ KUNIT_CASE(test_kprobe_in_insn6_lo),
+ KUNIT_CASE(test_kprobe_in_insn6_hi),
+ {}
+};
+
+static struct kunit_suite kprobes_test_suite = {
+ .name = "kprobes_test_s390",
+ .test_cases = kprobes_testcases,
+};
+
+kunit_test_suites(&kprobes_test_suite);
+
+MODULE_LICENSE("GPL");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+#ifndef TEST_KPROBES_H
+#define TEST_KPROBES_H
+
+extern unsigned long kprobes_target_odd_offs;
+extern unsigned long kprobes_target_in_insn4_offs;
+extern unsigned long kprobes_target_in_insn6_lo_offs;
+extern unsigned long kprobes_target_in_insn6_hi_offs;
+
+#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ */
+
+#include <linux/linkage.h>
+#include <asm/ftrace.h>
+
+#define KPROBES_TARGET_START(name) \
+ SYM_FUNC_START(name); \
+ FTRACE_GEN_NOP_ASM(name)
+
+#define KPROBES_TARGET_END(name) \
+ SYM_FUNC_END(name); \
+ SYM_DATA(name##_offs, .quad 1b - name)
+
+KPROBES_TARGET_START(kprobes_target_in_insn4)
+ .word 0x4700 // bc 0,0
+1: .word 0x0000
+ br %r14
+KPROBES_TARGET_END(kprobes_target_in_insn4)
+
+KPROBES_TARGET_START(kprobes_target_in_insn6_lo)
+ .word 0xe310 // ly 1,0
+1: .word 0x0000
+ .word 0x0058
+ br %r14
+KPROBES_TARGET_END(kprobes_target_in_insn6_lo)
+
+KPROBES_TARGET_START(kprobes_target_in_insn6_hi)
+ .word 0xe310 // ly 1,0
+ .word 0x0000
+1: .word 0x0058
+ br %r14
+KPROBES_TARGET_END(kprobes_target_in_insn6_hi)
+
+KPROBES_TARGET_START(kprobes_target_bp)
+ nop
+ .word 0x0000
+ nop
+1: br %r14
+KPROBES_TARGET_END(kprobes_target_bp)
+
+KPROBES_TARGET_START(kprobes_target_odd)
+ .byte 0x07
+1: .byte 0x07
+ br %r14
+KPROBES_TARGET_END(kprobes_target_odd)
* Test module for unwind_for_each_frame
*/
-#define pr_fmt(fmt) "test_unwind: " fmt
+#include <kunit/test.h>
#include <asm/unwind.h>
#include <linux/completion.h>
#include <linux/kallsyms.h>
#include <linux/wait.h>
#include <asm/irq.h>
+struct kunit *current_test;
+
#define BT_BUF_SIZE (PAGE_SIZE * 4)
/*
p = strsep(&bt, "\n");
if (!p)
break;
- pr_err("%s\n", p);
+ kunit_err(current_test, "%s\n", p);
}
}
bt = kmalloc(BT_BUF_SIZE, GFP_ATOMIC);
if (!bt) {
- pr_err("failed to allocate backtrace buffer\n");
+ kunit_err(current_test, "failed to allocate backtrace buffer\n");
return -ENOMEM;
}
/* Unwind. */
if (frame_count++ == max_frames)
break;
if (state.reliable && !addr) {
- pr_err("unwind state reliable but addr is 0\n");
+ kunit_err(current_test, "unwind state reliable but addr is 0\n");
ret = -EINVAL;
break;
}
stack_type_name(state.stack_info.type),
(void *)state.sp, (void *)state.ip);
if (bt_pos >= BT_BUF_SIZE)
- pr_err("backtrace buffer is too small\n");
+ kunit_err(current_test, "backtrace buffer is too small\n");
}
frame_count += 1;
if (prev_is_func2 && str_has_prefix(sym, "unwindme_func1"))
/* Check the results. */
if (unwind_error(&state)) {
- pr_err("unwind error\n");
+ kunit_err(current_test, "unwind error\n");
ret = -EINVAL;
}
if (!seen_func2_func1) {
- pr_err("unwindme_func2 and unwindme_func1 not found\n");
+ kunit_err(current_test, "unwindme_func2 and unwindme_func1 not found\n");
ret = -EINVAL;
}
if (frame_count == max_frames) {
- pr_err("Maximum number of frames exceeded\n");
+ kunit_err(current_test, "Maximum number of frames exceeded\n");
ret = -EINVAL;
}
if (ret)
kp.pre_handler = pgm_pre_handler;
ret = register_kprobe(&kp);
if (ret < 0) {
- pr_err("register_kprobe failed %d\n", ret);
+ kunit_err(current_test, "register_kprobe failed %d\n", ret);
return -EINVAL;
}
}
/* Spawns a task and passes it to test_unwind(). */
-static int test_unwind_task(struct unwindme *u)
+static int test_unwind_task(struct kunit *test, struct unwindme *u)
{
struct task_struct *task;
int ret;
*/
task = kthread_run(unwindme_func1, u, "%s", __func__);
if (IS_ERR(task)) {
- pr_err("kthread_run() failed\n");
+ kunit_err(test, "kthread_run() failed\n");
return PTR_ERR(task);
}
/*
return ret;
}
-static int test_unwind_flags(int flags)
+struct test_params {
+ int flags;
+ char *name;
+};
+
+/*
+ * Create required parameter list for tests
+ */
+static const struct test_params param_list[] = {
+ {.flags = UWM_DEFAULT, .name = "UWM_DEFAULT"},
+ {.flags = UWM_SP, .name = "UWM_SP"},
+ {.flags = UWM_REGS, .name = "UWM_REGS"},
+ {.flags = UWM_SWITCH_STACK,
+ .name = "UWM_SWITCH_STACK"},
+ {.flags = UWM_SP | UWM_REGS,
+ .name = "UWM_SP | UWM_REGS"},
+ {.flags = UWM_CALLER | UWM_SP,
+ .name = "WM_CALLER | UWM_SP"},
+ {.flags = UWM_CALLER | UWM_SP | UWM_REGS,
+ .name = "UWM_CALLER | UWM_SP | UWM_REGS"},
+ {.flags = UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK,
+ .name = "UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK"},
+ {.flags = UWM_THREAD, .name = "UWM_THREAD"},
+ {.flags = UWM_THREAD | UWM_SP,
+ .name = "UWM_THREAD | UWM_SP"},
+ {.flags = UWM_THREAD | UWM_CALLER | UWM_SP,
+ .name = "UWM_THREAD | UWM_CALLER | UWM_SP"},
+ {.flags = UWM_IRQ, .name = "UWM_IRQ"},
+ {.flags = UWM_IRQ | UWM_SWITCH_STACK,
+ .name = "UWM_IRQ | UWM_SWITCH_STACK"},
+ {.flags = UWM_IRQ | UWM_SP,
+ .name = "UWM_IRQ | UWM_SP"},
+ {.flags = UWM_IRQ | UWM_REGS,
+ .name = "UWM_IRQ | UWM_REGS"},
+ {.flags = UWM_IRQ | UWM_SP | UWM_REGS,
+ .name = "UWM_IRQ | UWM_SP | UWM_REGS"},
+ {.flags = UWM_IRQ | UWM_CALLER | UWM_SP,
+ .name = "UWM_IRQ | UWM_CALLER | UWM_SP"},
+ {.flags = UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS,
+ .name = "UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS"},
+ {.flags = UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK,
+ .name = "UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK"},
+ #ifdef CONFIG_KPROBES
+ {.flags = UWM_PGM, .name = "UWM_PGM"},
+ {.flags = UWM_PGM | UWM_SP,
+ .name = "UWM_PGM | UWM_SP"},
+ {.flags = UWM_PGM | UWM_REGS,
+ .name = "UWM_PGM | UWM_REGS"},
+ {.flags = UWM_PGM | UWM_SP | UWM_REGS,
+ .name = "UWM_PGM | UWM_SP | UWM_REGS"},
+ #endif
+};
+
+/*
+ * Parameter description generator: required for KUNIT_ARRAY_PARAM()
+ */
+static void get_desc(const struct test_params *params, char *desc)
+{
+ strscpy(desc, params->name, KUNIT_PARAM_DESC_SIZE);
+}
+
+/*
+ * Create test_unwind_gen_params
+ */
+KUNIT_ARRAY_PARAM(test_unwind, param_list, get_desc);
+
+static void test_unwind_flags(struct kunit *test)
{
struct unwindme u;
+ const struct test_params *params;
- u.flags = flags;
+ current_test = test;
+ params = (const struct test_params *)test->param_value;
+ u.flags = params->flags;
if (u.flags & UWM_THREAD)
- return test_unwind_task(&u);
+ KUNIT_EXPECT_EQ(test, 0, test_unwind_task(test, &u));
else if (u.flags & UWM_IRQ)
- return test_unwind_irq(&u);
+ KUNIT_EXPECT_EQ(test, 0, test_unwind_irq(&u));
else
- return unwindme_func1(&u);
+ KUNIT_EXPECT_EQ(test, 0, unwindme_func1(&u));
}
-static int test_unwind_init(void)
-{
- int failed = 0;
- int total = 0;
-
-#define TEST(flags) \
-do { \
- pr_info("[ RUN ] " #flags "\n"); \
- total++; \
- if (!test_unwind_flags((flags))) { \
- pr_info("[ OK ] " #flags "\n"); \
- } else { \
- pr_err("[ FAILED ] " #flags "\n"); \
- failed++; \
- } \
-} while (0)
-
- pr_info("running stack unwinder tests");
- TEST(UWM_DEFAULT);
- TEST(UWM_SP);
- TEST(UWM_REGS);
- TEST(UWM_SWITCH_STACK);
- TEST(UWM_SP | UWM_REGS);
- TEST(UWM_CALLER | UWM_SP);
- TEST(UWM_CALLER | UWM_SP | UWM_REGS);
- TEST(UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK);
- TEST(UWM_THREAD);
- TEST(UWM_THREAD | UWM_SP);
- TEST(UWM_THREAD | UWM_CALLER | UWM_SP);
- TEST(UWM_IRQ);
- TEST(UWM_IRQ | UWM_SWITCH_STACK);
- TEST(UWM_IRQ | UWM_SP);
- TEST(UWM_IRQ | UWM_REGS);
- TEST(UWM_IRQ | UWM_SP | UWM_REGS);
- TEST(UWM_IRQ | UWM_CALLER | UWM_SP);
- TEST(UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS);
- TEST(UWM_IRQ | UWM_CALLER | UWM_SP | UWM_REGS | UWM_SWITCH_STACK);
-#ifdef CONFIG_KPROBES
- TEST(UWM_PGM);
- TEST(UWM_PGM | UWM_SP);
- TEST(UWM_PGM | UWM_REGS);
- TEST(UWM_PGM | UWM_SP | UWM_REGS);
-#endif
-#undef TEST
- if (failed) {
- pr_err("%d of %d stack unwinder tests failed", failed, total);
- WARN(1, "%d of %d stack unwinder tests failed", failed, total);
- } else {
- pr_info("all %d stack unwinder tests passed", total);
- }
+static struct kunit_case unwind_test_cases[] = {
+ KUNIT_CASE_PARAM(test_unwind_flags, test_unwind_gen_params),
+ {}
+};
- return failed ? -EINVAL : 0;
-}
+static struct kunit_suite test_unwind_suite = {
+ .name = "test_unwind",
+ .test_cases = unwind_test_cases,
+};
-static void test_unwind_exit(void)
-{
-}
+kunit_test_suites(&test_unwind_suite);
-module_init(test_unwind_init);
-module_exit(test_unwind_exit);
MODULE_LICENSE("GPL");
#include <linux/moduleparam.h>
#include <linux/gfp.h>
#include <linux/sched.h>
+#include <linux/string_helpers.h>
#include <linux/sysctl.h>
-#include <linux/ctype.h>
#include <linux/swap.h>
#include <linux/kthread.h>
#include <linux/oom.h>
goto out_sysctl;
#ifdef CONFIG_CMM_IUCV
/* convert sender to uppercase characters */
- if (sender) {
- int len = strlen(sender);
- while (len--)
- sender[len] = toupper(sender[len]);
- } else {
+ if (sender)
+ string_upper(sender, sender);
+ else
sender = cmm_default_sender;
- }
rc = smsg_register_callback(SMSG_PREFIX, cmm_smsg_target);
if (rc < 0)
#include <linux/kasan.h>
#include <asm/ptdump.h>
#include <asm/kasan.h>
+#include <asm/nospec-branch.h>
#include <asm/sections.h>
static unsigned long max_addr;
return;
if (st->current_prot & _PAGE_NOEXEC)
return;
- /* The first lowcore page is currently still W+X. */
- if (addr == PAGE_SIZE)
+ /*
+ * The first lowcore page is W+X if spectre mitigations are using
+ * trampolines or the BEAR enhancements facility is not installed,
+ * in which case we have two lpswe instructions in lowcore that need
+ * to be executable.
+ */
+ if (addr == PAGE_SIZE && (nospec_uses_trampoline() || !static_key_enabled(&cpu_has_bear)))
return;
WARN_ONCE(1, "s390/mm: Found insecure W+X mapping at address %pS\n",
(void *)st->start_address);
if (st.wx_pages)
pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found\n", st.wx_pages);
else
- pr_info("Checked W+X mappings: passed, no unexpected W+X pages found\n");
+ pr_info("Checked W+X mappings: passed, no %sW+X pages found\n",
+ (nospec_uses_trampoline() || !static_key_enabled(&cpu_has_bear)) ?
+ "unexpected " : "");
}
#endif /* CONFIG_DEBUG_WX */
" in virtual user address space\n");
dump_fault_info(regs);
die(regs, "Oops");
- do_exit(SIGKILL);
}
static noinline void do_low_address(struct pt_regs *regs)
if (regs->psw.mask & PSW_MASK_PSTATE) {
/* Low-address protection hit in user mode 'cannot happen'. */
die (regs, "Low-address protection");
- do_exit(SIGKILL);
}
do_no_context(regs);
EXPORT_SYMBOL(empty_zero_page);
EXPORT_SYMBOL(zero_page_mask);
-bool initmem_freed;
-
static void __init setup_zero_pages(void)
{
unsigned int order;
void free_initmem(void)
{
- initmem_freed = true;
__set_memory((unsigned long)_sinittext,
(unsigned long)(_einittext - _sinittext) >> PAGE_SHIFT,
SET_MEMORY_RW | SET_MEMORY_NX);
void __init kasan_free_early_identity(void)
{
- memblock_free(pgalloc_pos, pgalloc_freeable - pgalloc_pos);
+ memblock_phys_free(pgalloc_pos, pgalloc_freeable - pgalloc_pos);
}
static void pgt_set(unsigned long *old, unsigned long new, unsigned long addr,
unsigned long dtt)
{
- unsigned long table, mask;
+ unsigned long *table, mask;
mask = 0;
if (MACHINE_HAS_EDAT2) {
mask = ~(PTRS_PER_PTE * sizeof(pte_t) - 1);
break;
}
- table = (unsigned long)old & mask;
+ table = (unsigned long *)((unsigned long)old & mask);
crdte(*old, new, table, dtt, addr, S390_lowcore.kernel_asce);
} else if (MACHINE_HAS_IDTE) {
cspg(old, *old, new);
#include <linux/hugetlb.h>
#include <linux/slab.h>
#include <asm/cacheflush.h>
+#include <asm/nospec-branch.h>
#include <asm/pgalloc.h>
#include <asm/setup.h>
#include <asm/tlbflush.h>
__set_memory(__stext_amode31, (__etext_amode31 - __stext_amode31) >> PAGE_SHIFT,
SET_MEMORY_RO | SET_MEMORY_X);
- /* we need lowcore executable for our LPSWE instructions */
- set_memory_x(0, 1);
+ if (nospec_uses_trampoline() || !static_key_enabled(&cpu_has_bear)) {
+ /*
+ * Lowcore must be executable for LPSWE
+ * and expoline trampoline branch instructions.
+ */
+ set_memory_x(0, 1);
+ }
pr_info("Write protected kernel read-only data: %luk\n",
(unsigned long)(__end_rodata - _stext) >> 10);
EMIT4(0xb9040000, REG_2, BPF_REG_0);
/* Restore registers */
save_restore_regs(jit, REGS_RESTORE, stack_depth);
- if (__is_defined(CC_USING_EXPOLINE) && !nospec_disable) {
+ if (nospec_uses_trampoline()) {
jit->r14_thunk_ip = jit->prg;
/* Generate __s390_indirect_jump_r14 thunk */
if (test_facility(35)) {
/* br %r14 */
_EMIT2(0x07fe);
- if (__is_defined(CC_USING_EXPOLINE) && !nospec_disable &&
+ if ((nospec_uses_trampoline()) &&
(is_first_pass(jit) || (jit->seen & SEEN_FUNC))) {
jit->r1_thunk_ip = jit->prg;
/* Generate __s390_indirect_jump_r1 thunk */
jit->seen |= SEEN_FUNC;
/* lgrl %w1,func */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, _EMIT_CONST_U64(func));
- if (__is_defined(CC_USING_EXPOLINE) && !nospec_disable) {
+ if (nospec_uses_trampoline()) {
/* brasl %r14,__s390_indirect_jump_r1 */
EMIT6_PCREL_RILB(0xc0050000, REG_14, jit->r1_thunk_ip);
} else {
spin_unlock(&zpci_iomap_lock);
}
+static void zpci_do_update_iomap_fh(struct zpci_dev *zdev, u32 fh)
+{
+ int bar, idx;
+
+ spin_lock(&zpci_iomap_lock);
+ for (bar = 0; bar < PCI_STD_NUM_BARS; bar++) {
+ if (!zdev->bars[bar].size)
+ continue;
+ idx = zdev->bars[bar].map_idx;
+ if (!zpci_iomap_start[idx].count)
+ continue;
+ WRITE_ONCE(zpci_iomap_start[idx].fh, zdev->fh);
+ }
+ spin_unlock(&zpci_iomap_lock);
+}
+
+void zpci_update_fh(struct zpci_dev *zdev, u32 fh)
+{
+ if (!fh || zdev->fh == fh)
+ return;
+
+ zdev->fh = fh;
+ if (zpci_use_mio(zdev))
+ return;
+ if (zdev->has_resources && zdev_enabled(zdev))
+ zpci_do_update_iomap_fh(zdev, fh);
+}
+
static struct resource *__alloc_res(struct zpci_dev *zdev, unsigned long start,
unsigned long size, unsigned long flags)
{
zdev->has_resources = 0;
}
-int pcibios_add_device(struct pci_dev *pdev)
+int pcibios_device_add(struct pci_dev *pdev)
{
struct zpci_dev *zdev = to_zpci(pdev);
struct resource *res;
if (clp_enable_fh(zdev, &fh, ZPCI_NR_DMA_SPACES))
rc = -EIO;
else
- zdev->fh = fh;
+ zpci_update_fh(zdev, fh);
return rc;
}
cc = clp_disable_fh(zdev, &fh);
if (!cc) {
- zdev->fh = fh;
+ zpci_update_fh(zdev, fh);
} else if (cc == CLP_RC_SETPCIFN_ALRDY) {
pr_info("Disabling PCI function %08x had no effect as it was already disabled\n",
zdev->fid);
/* Function is already disabled - update handle */
rc = clp_refresh_fh(zdev->fid, &fh);
if (!rc) {
- zdev->fh = fh;
+ zpci_update_fh(zdev, fh);
rc = -EINVAL;
}
} else {
}
/**
+ * zpci_hot_reset_device - perform a reset of the given zPCI function
+ * @zdev: the slot which should be reset
+ *
+ * Performs a low level reset of the zPCI function. The reset is low level in
+ * the sense that the zPCI function can be reset without detaching it from the
+ * common PCI subsystem. The reset may be performed while under control of
+ * either DMA or IOMMU APIs in which case the existing DMA/IOMMU translation
+ * table is reinstated at the end of the reset.
+ *
+ * After the reset the functions internal state is reset to an initial state
+ * equivalent to its state during boot when first probing a driver.
+ * Consequently after reset the PCI function requires re-initialization via the
+ * common PCI code including re-enabling IRQs via pci_alloc_irq_vectors()
+ * and enabling the function via e.g.pci_enablde_device_flags().The caller
+ * must guard against concurrent reset attempts.
+ *
+ * In most cases this function should not be called directly but through
+ * pci_reset_function() or pci_reset_bus() which handle the save/restore and
+ * locking.
+ *
+ * Return: 0 on success and an error value otherwise
+ */
+int zpci_hot_reset_device(struct zpci_dev *zdev)
+{
+ int rc;
+
+ zpci_dbg(3, "rst fid:%x, fh:%x\n", zdev->fid, zdev->fh);
+ if (zdev_enabled(zdev)) {
+ /* Disables device access, DMAs and IRQs (reset state) */
+ rc = zpci_disable_device(zdev);
+ /*
+ * Due to a z/VM vs LPAR inconsistency in the error state the
+ * FH may indicate an enabled device but disable says the
+ * device is already disabled don't treat it as an error here.
+ */
+ if (rc == -EINVAL)
+ rc = 0;
+ if (rc)
+ return rc;
+ }
+
+ rc = zpci_enable_device(zdev);
+ if (rc)
+ return rc;
+
+ if (zdev->dma_table)
+ rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+ (u64)zdev->dma_table);
+ else
+ rc = zpci_dma_init_device(zdev);
+ if (rc) {
+ zpci_disable_device(zdev);
+ return rc;
+ }
+
+ return 0;
+}
+
+/**
* zpci_create_device() - Create a new zpci_dev and add it to the zbus
* @fid: Function ID of the device to be created
* @fh: Current Function Handle of the device to be created
{
int rc;
- zdev->fh = fh;
+ zpci_update_fh(zdev, fh);
/* the PCI function will be scanned once function 0 appears */
if (!zdev->zbus->bus)
return 0;
}
EXPORT_SYMBOL(zpci_report_error);
+/**
+ * zpci_clear_error_state() - Clears the zPCI error state of the device
+ * @zdev: The zdev for which the zPCI error state should be reset
+ *
+ * Clear the zPCI error state of the device. If clearing the zPCI error state
+ * fails the device is left in the error state. In this case it may make sense
+ * to call zpci_io_perm_failure() on the associated pdev if it exists.
+ *
+ * Returns: 0 on success, -EIO otherwise
+ */
+int zpci_clear_error_state(struct zpci_dev *zdev)
+{
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_RESET_ERROR);
+ struct zpci_fib fib = {0};
+ u8 status;
+ int cc;
+
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc) {
+ zpci_dbg(3, "ces fid:%x, cc:%d, status:%x\n", zdev->fid, cc, status);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+/**
+ * zpci_reset_load_store_blocked() - Re-enables L/S from error state
+ * @zdev: The zdev for which to unblock load/store access
+ *
+ * Re-enables load/store access for a PCI function in the error state while
+ * keeping DMA blocked. In this state drivers can poke MMIO space to determine
+ * if error recovery is possible while catching any rogue DMA access from the
+ * device.
+ *
+ * Returns: 0 on success, -EIO otherwise
+ */
+int zpci_reset_load_store_blocked(struct zpci_dev *zdev)
+{
+ u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_RESET_BLOCK);
+ struct zpci_fib fib = {0};
+ u8 status;
+ int cc;
+
+ cc = zpci_mod_fc(req, &fib, &status);
+ if (cc) {
+ zpci_dbg(3, "rls fid:%x, cc:%d, status:%x\n", zdev->fid, cc, status);
+ return -EIO;
+ }
+
+ return 0;
+}
+
static int zpci_mem_init(void)
{
BUILD_BUG_ON(!is_power_of_2(__alignof__(struct zpci_fmb)) ||
static struct kmem_cache *dma_region_table_cache;
static struct kmem_cache *dma_page_table_cache;
static int s390_iommu_strict;
+static u64 s390_iommu_aperture;
+static u32 s390_iommu_aperture_factor = 1;
static int zpci_refresh_global(struct zpci_dev *zdev)
{
/*
* Restrict the iommu bitmap size to the minimum of the following:
- * - main memory size
+ * - s390_iommu_aperture which defaults to high_memory
* - 3-level pagetable address limit minus start_dma offset
* - DMA address range allowed by the hardware (clp query pci fn)
*
* Also set zdev->end_dma to the actual end address of the usable
* range, instead of the theoretical maximum as reported by hardware.
+ *
+ * This limits the number of concurrently usable DMA mappings since
+ * for each DMA mapped memory address we need a DMA address including
+ * extra DMA addresses for multiple mappings of the same memory address.
*/
zdev->start_dma = PAGE_ALIGN(zdev->start_dma);
- zdev->iommu_size = min3((u64) high_memory,
+ zdev->iommu_size = min3(s390_iommu_aperture,
ZPCI_TABLE_SIZE_RT - zdev->start_dma,
zdev->end_dma - zdev->start_dma + 1);
zdev->end_dma = zdev->start_dma + zdev->iommu_size - 1;
int __init zpci_dma_init(void)
{
+ s390_iommu_aperture = (u64)high_memory;
+ if (!s390_iommu_aperture_factor)
+ s390_iommu_aperture = ULONG_MAX;
+ else
+ s390_iommu_aperture *= s390_iommu_aperture_factor;
+
return dma_alloc_cpu_table_caches();
}
}
__setup("s390_iommu=", s390_iommu_setup);
+
+static int __init s390_iommu_aperture_setup(char *str)
+{
+ if (kstrtou32(str, 10, &s390_iommu_aperture_factor))
+ s390_iommu_aperture_factor = 1;
+ return 1;
+}
+
+__setup("s390_iommu_aperture=", s390_iommu_aperture_setup);
u16 pec; /* PCI event code */
} __packed;
+static inline bool ers_result_indicates_abort(pci_ers_result_t ers_res)
+{
+ switch (ers_res) {
+ case PCI_ERS_RESULT_CAN_RECOVER:
+ case PCI_ERS_RESULT_RECOVERED:
+ case PCI_ERS_RESULT_NEED_RESET:
+ return false;
+ default:
+ return true;
+ }
+}
+
+static bool is_passed_through(struct zpci_dev *zdev)
+{
+ return zdev->s390_domain;
+}
+
+static bool is_driver_supported(struct pci_driver *driver)
+{
+ if (!driver || !driver->err_handler)
+ return false;
+ if (!driver->err_handler->error_detected)
+ return false;
+ if (!driver->err_handler->slot_reset)
+ return false;
+ if (!driver->err_handler->resume)
+ return false;
+ return true;
+}
+
+static pci_ers_result_t zpci_event_notify_error_detected(struct pci_dev *pdev,
+ struct pci_driver *driver)
+{
+ pci_ers_result_t ers_res = PCI_ERS_RESULT_DISCONNECT;
+
+ ers_res = driver->err_handler->error_detected(pdev, pdev->error_state);
+ if (ers_result_indicates_abort(ers_res))
+ pr_info("%s: Automatic recovery failed after initial reporting\n", pci_name(pdev));
+ else if (ers_res == PCI_ERS_RESULT_NEED_RESET)
+ pr_debug("%s: Driver needs reset to recover\n", pci_name(pdev));
+
+ return ers_res;
+}
+
+static pci_ers_result_t zpci_event_do_error_state_clear(struct pci_dev *pdev,
+ struct pci_driver *driver)
+{
+ pci_ers_result_t ers_res = PCI_ERS_RESULT_DISCONNECT;
+ struct zpci_dev *zdev = to_zpci(pdev);
+ int rc;
+
+ pr_info("%s: Unblocking device access for examination\n", pci_name(pdev));
+ rc = zpci_reset_load_store_blocked(zdev);
+ if (rc) {
+ pr_err("%s: Unblocking device access failed\n", pci_name(pdev));
+ /* Let's try a full reset instead */
+ return PCI_ERS_RESULT_NEED_RESET;
+ }
+
+ if (driver->err_handler->mmio_enabled) {
+ ers_res = driver->err_handler->mmio_enabled(pdev);
+ if (ers_result_indicates_abort(ers_res)) {
+ pr_info("%s: Automatic recovery failed after MMIO re-enable\n",
+ pci_name(pdev));
+ return ers_res;
+ } else if (ers_res == PCI_ERS_RESULT_NEED_RESET) {
+ pr_debug("%s: Driver needs reset to recover\n", pci_name(pdev));
+ return ers_res;
+ }
+ }
+
+ pr_debug("%s: Unblocking DMA\n", pci_name(pdev));
+ rc = zpci_clear_error_state(zdev);
+ if (!rc) {
+ pdev->error_state = pci_channel_io_normal;
+ } else {
+ pr_err("%s: Unblocking DMA failed\n", pci_name(pdev));
+ /* Let's try a full reset instead */
+ return PCI_ERS_RESULT_NEED_RESET;
+ }
+
+ return ers_res;
+}
+
+static pci_ers_result_t zpci_event_do_reset(struct pci_dev *pdev,
+ struct pci_driver *driver)
+{
+ pci_ers_result_t ers_res = PCI_ERS_RESULT_DISCONNECT;
+
+ pr_info("%s: Initiating reset\n", pci_name(pdev));
+ if (zpci_hot_reset_device(to_zpci(pdev))) {
+ pr_err("%s: The reset request failed\n", pci_name(pdev));
+ return ers_res;
+ }
+ pdev->error_state = pci_channel_io_normal;
+ ers_res = driver->err_handler->slot_reset(pdev);
+ if (ers_result_indicates_abort(ers_res)) {
+ pr_info("%s: Automatic recovery failed after slot reset\n", pci_name(pdev));
+ return ers_res;
+ }
+
+ return ers_res;
+}
+
+/* zpci_event_attempt_error_recovery - Try to recover the given PCI function
+ * @pdev: PCI function to recover currently in the error state
+ *
+ * We follow the scheme outlined in Documentation/PCI/pci-error-recovery.rst.
+ * With the simplification that recovery always happens per function
+ * and the platform determines which functions are affected for
+ * multi-function devices.
+ */
+static pci_ers_result_t zpci_event_attempt_error_recovery(struct pci_dev *pdev)
+{
+ pci_ers_result_t ers_res = PCI_ERS_RESULT_DISCONNECT;
+ struct pci_driver *driver;
+
+ /*
+ * Ensure that the PCI function is not removed concurrently, no driver
+ * is unbound or probed and that userspace can't access its
+ * configuration space while we perform recovery.
+ */
+ pci_dev_lock(pdev);
+ if (pdev->error_state == pci_channel_io_perm_failure) {
+ ers_res = PCI_ERS_RESULT_DISCONNECT;
+ goto out_unlock;
+ }
+ pdev->error_state = pci_channel_io_frozen;
+
+ if (is_passed_through(to_zpci(pdev))) {
+ pr_info("%s: Cannot be recovered in the host because it is a pass-through device\n",
+ pci_name(pdev));
+ goto out_unlock;
+ }
+
+ driver = to_pci_driver(pdev->dev.driver);
+ if (!is_driver_supported(driver)) {
+ if (!driver)
+ pr_info("%s: Cannot be recovered because no driver is bound to the device\n",
+ pci_name(pdev));
+ else
+ pr_info("%s: The %s driver bound to the device does not support error recovery\n",
+ pci_name(pdev),
+ driver->name);
+ goto out_unlock;
+ }
+
+ ers_res = zpci_event_notify_error_detected(pdev, driver);
+ if (ers_result_indicates_abort(ers_res))
+ goto out_unlock;
+
+ if (ers_res == PCI_ERS_RESULT_CAN_RECOVER) {
+ ers_res = zpci_event_do_error_state_clear(pdev, driver);
+ if (ers_result_indicates_abort(ers_res))
+ goto out_unlock;
+ }
+
+ if (ers_res == PCI_ERS_RESULT_NEED_RESET)
+ ers_res = zpci_event_do_reset(pdev, driver);
+
+ if (ers_res != PCI_ERS_RESULT_RECOVERED) {
+ pr_err("%s: Automatic recovery failed; operator intervention is required\n",
+ pci_name(pdev));
+ goto out_unlock;
+ }
+
+ pr_info("%s: The device is ready to resume operations\n", pci_name(pdev));
+ if (driver->err_handler->resume)
+ driver->err_handler->resume(pdev);
+out_unlock:
+ pci_dev_unlock(pdev);
+
+ return ers_res;
+}
+
+/* zpci_event_io_failure - Report PCI channel failure state to driver
+ * @pdev: PCI function for which to report
+ * @es: PCI channel failure state to report
+ */
+static void zpci_event_io_failure(struct pci_dev *pdev, pci_channel_state_t es)
+{
+ struct pci_driver *driver;
+
+ pci_dev_lock(pdev);
+ pdev->error_state = es;
+ /**
+ * While vfio-pci's error_detected callback notifies user-space QEMU
+ * reacts to this by freezing the guest. In an s390 environment PCI
+ * errors are rarely fatal so this is overkill. Instead in the future
+ * we will inject the error event and let the guest recover the device
+ * itself.
+ */
+ if (is_passed_through(to_zpci(pdev)))
+ goto out;
+ driver = to_pci_driver(pdev->dev.driver);
+ if (driver && driver->err_handler && driver->err_handler->error_detected)
+ driver->err_handler->error_detected(pdev, pdev->error_state);
+out:
+ pci_dev_unlock(pdev);
+}
+
static void __zpci_event_error(struct zpci_ccdf_err *ccdf)
{
struct zpci_dev *zdev = get_zdev_by_fid(ccdf->fid);
struct pci_dev *pdev = NULL;
+ pci_ers_result_t ers_res;
+ zpci_dbg(3, "err fid:%x, fh:%x, pec:%x\n",
+ ccdf->fid, ccdf->fh, ccdf->pec);
zpci_err("error CCDF:\n");
zpci_err_hex(ccdf, sizeof(*ccdf));
- if (zdev)
- pdev = pci_get_slot(zdev->zbus->bus, zdev->devfn);
+ if (zdev) {
+ zpci_update_fh(zdev, ccdf->fh);
+ if (zdev->zbus->bus)
+ pdev = pci_get_slot(zdev->zbus->bus, zdev->devfn);
+ }
pr_err("%s: Event 0x%x reports an error for PCI function 0x%x\n",
pdev ? pci_name(pdev) : "n/a", ccdf->pec, ccdf->fid);
if (!pdev)
return;
- pdev->error_state = pci_channel_io_perm_failure;
+ switch (ccdf->pec) {
+ case 0x003a: /* Service Action or Error Recovery Successful */
+ ers_res = zpci_event_attempt_error_recovery(pdev);
+ if (ers_res != PCI_ERS_RESULT_RECOVERED)
+ zpci_event_io_failure(pdev, pci_channel_io_perm_failure);
+ break;
+ default:
+ /*
+ * Mark as frozen not permanently failed because the device
+ * could be subsequently recovered by the platform.
+ */
+ zpci_event_io_failure(pdev, pci_channel_io_frozen);
+ break;
+ }
pci_dev_put(pdev);
}
static void zpci_event_hard_deconfigured(struct zpci_dev *zdev, u32 fh)
{
- zdev->fh = fh;
+ zpci_update_fh(zdev, fh);
/* Give the driver a hint that the function is
* already unusable.
*/
struct zpci_dev *zdev = get_zdev_by_fid(ccdf->fid);
enum zpci_state state;
+ zpci_dbg(3, "avl fid:%x, fh:%x, pec:%x\n",
+ ccdf->fid, ccdf->fh, ccdf->pec);
zpci_err("avail CCDF:\n");
zpci_err_hex(ccdf, sizeof(*ccdf));
if (!zdev)
zpci_create_device(ccdf->fid, ccdf->fh, ZPCI_FN_STATE_STANDBY);
else
- zdev->fh = ccdf->fh;
+ zpci_update_fh(zdev, ccdf->fh);
break;
case 0x0303: /* Deconfiguration requested */
if (zdev) {
*/
if (zdev->state != ZPCI_FN_STATE_CONFIGURED)
break;
- zdev->fh = ccdf->fh;
+ zpci_update_fh(zdev, ccdf->fh);
zpci_deconfigure_device(zdev);
}
break;
unsigned long len)
{
struct zpci_iomap_entry *entry = &zpci_iomap_start[ZPCI_IDX(addr)];
- u64 req = ZPCI_CREATE_REQ(entry->fh, entry->bar, len);
+ u64 req = ZPCI_CREATE_REQ(READ_ONCE(entry->fh), entry->bar, len);
return __zpci_load(data, req, ZPCI_OFFSET(addr));
}
unsigned long len)
{
struct zpci_iomap_entry *entry = &zpci_iomap_start[ZPCI_IDX(addr)];
- u64 req = ZPCI_CREATE_REQ(entry->fh, entry->bar, len);
+ u64 req = ZPCI_CREATE_REQ(READ_ONCE(entry->fh), entry->bar, len);
return __zpci_store(data, req, ZPCI_OFFSET(addr));
}
airq_iv_free(zpci_ibv[0], zdev->msi_first_bit, zdev->msi_nr_irqs);
}
+void arch_restore_msi_irqs(struct pci_dev *pdev)
+{
+ struct zpci_dev *zdev = to_zpci(pdev);
+
+ if (!zdev->irqs_registered)
+ zpci_set_irq(zdev);
+ default_restore_msi_irqs(pdev);
+}
+
static struct airq_struct zpci_airq = {
.handler = zpci_floating_irq_handler,
.isc = PCI_ISC,
if (zdev_enabled(zdev)) {
ret = zpci_disable_device(zdev);
+ /*
+ * Due to a z/VM vs LPAR inconsistency in the error
+ * state the FH may indicate an enabled device but
+ * disable says the device is already disabled don't
+ * treat it as an error here.
+ */
+ if (ret == -EINVAL)
+ ret = 0;
if (ret)
goto out;
}
obj-y += kernel/ mm/ boards/
obj-$(CONFIG_SH_FPU_EMU) += math-emu/
obj-$(CONFIG_USE_BUILTIN_DTB) += boot/dts/
+
+# for cleaning
+subdir- += boot
select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
select IRQ_FORCED_THREADING
- select MAY_HAVE_SPARSE_IRQ
select MODULES_USE_ELF_RELA
select NEED_SG_DMA_LENGTH
select NO_DMA if !MMU && !DMA_COHERENT
config DWARF_UNWINDER
bool "Enable the DWARF unwinder for stacktraces"
+ depends on DEBUG_KERNEL
select FRAME_POINTER
default n
help
archprepare:
$(Q)$(MAKE) $(build)=arch/sh/tools include/generated/machtypes.h
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
- $(Q)$(MAKE) $(clean)=arch/sh/kernel/vsyscall
-
archheaders:
$(Q)$(MAKE) $(build)=arch/sh/kernel/syscalls all
if (!phys)
panic("Failed to allocate CEU memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu_dma_membase = phys;
if (!phys)
panic("Failed to allocate CEU0 memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu0_dma_membase = phys;
if (!phys)
panic("Failed to allocate CEU1 memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu1_dma_membase = phys;
}
if (!phys)
panic("Failed to allocate CEU memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu_dma_membase = phys;
PCI_INTD, /* PCI int D */
ATA, /* ATA */
FATA, /* CF */
- POWER, /* Power swtich */
- BUTTON, /* Button swtich */
+ POWER, /* Power switch */
+ BUTTON, /* Button switch */
};
/* Vectors for LANDISK */
if (!phys)
panic("Failed to allocate CEU memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu_dma_membase = phys;
if (!phys)
panic("Failed to allocate CEU0 memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu0_dma_membase = phys;
if (!phys)
panic("Failed to allocate CEU1 memory\n");
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
memblock_remove(phys, size);
ceu1_dma_membase = phys;
}
suffix-$(CONFIG_KERNEL_LZO) := lzo
targets := zImage vmlinux.srec romImage uImage uImage.srec uImage.gz \
- uImage.bz2 uImage.lzma uImage.xz uImage.lzo uImage.bin
-extra-y += vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma \
+ uImage.bz2 uImage.lzma uImage.xz uImage.lzo uImage.bin \
+ vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma \
vmlinux.bin.xz vmlinux.bin.lzo
subdir- := compressed romimage
# SPDX-License-Identifier: GPL-2.0-only
-ashiftrt.S
-ashldi3.c
-ashlsi3.S
-ashrsi3.S
-lshrsi3.S
vmlinux.bin.*
# create a compressed vmlinux image from the original vmlinux
#
-targets := vmlinux vmlinux.bin vmlinux.bin.gz \
- vmlinux.bin.bz2 vmlinux.bin.lzma \
- vmlinux.bin.xz vmlinux.bin.lzo \
- head_32.o misc.o piggy.o
+OBJECTS := head_32.o misc.o cache.o piggy.o \
+ ashiftrt.o ashldi3.o ashrsi3.o ashlsi3.o lshrsi3.o
+
+# These were previously generated files. When you are building the kernel
+# with O=, make sure to remove the stale files in the output tree. Otherwise,
+# the build system wrongly compiles the stale ones.
+ifdef building_out_of_srctree
+$(shell rm -f $(addprefix $(obj)/, ashiftrt.S ashldi3.c ashrsi3.S ashlsi3.S lshrsi3.S))
+endif
-OBJECTS = $(obj)/head_32.o $(obj)/misc.o $(obj)/cache.o
+targets := vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 \
+ vmlinux.bin.lzma vmlinux.bin.xz vmlinux.bin.lzo $(OBJECTS)
GCOV_PROFILE := n
LDFLAGS_vmlinux := --oformat $(ld-bfd) -Ttext $(IMAGE_OFFSET) -e startup \
-T $(obj)/../../kernel/vmlinux.lds
-#
-# Pull in the necessary libgcc bits from the in-kernel implementation.
-#
-lib1funcs-y := ashiftrt.S ashldi3.c ashrsi3.S ashlsi3.S lshrsi3.S
-lib1funcs-obj := \
- $(addsuffix .o, $(basename $(addprefix $(obj)/, $(lib1funcs-y))))
-
-lib1funcs-dir := $(srctree)/arch/$(SRCARCH)/lib
-
-KBUILD_CFLAGS += -I$(lib1funcs-dir) -DDISABLE_BRANCH_PROFILING
-
-$(addprefix $(obj)/,$(lib1funcs-y)): $(obj)/%: $(lib1funcs-dir)/% FORCE
- $(call cmd,shipped)
+KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
-$(obj)/vmlinux: $(OBJECTS) $(obj)/piggy.o $(lib1funcs-obj) FORCE
+$(obj)/vmlinux: $(addprefix $(obj)/, $(OBJECTS)) FORCE
$(call if_changed,ld)
$(obj)/vmlinux.bin: vmlinux FORCE
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include "../../lib/ashiftrt.S"
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../lib/ashldi3.c"
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include "../../lib/ashlsi3.S"
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include "../../lib/ashrsi3.S"
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include "../../lib/lshrsi3.S"
*/
static inline __sum16 ip_fast_csum(const void *iph, unsigned int ihl)
{
- unsigned int sum, __dummy0, __dummy1;
+ __wsum sum;
+ unsigned int __dummy0, __dummy1;
__asm__ __volatile__(
"mov.l @%1+, %0\n\t"
{
if (!access_ok(dst, len))
return 0;
- return csum_partial_copy_generic((__force const void *)src, dst, len);
+ return csum_partial_copy_generic(src, (__force void *)dst, len);
}
#endif /* __ASM_SH_CHECKSUM_H */
#include <asm/machvec.h>
/*
- * Only legacy non-sparseirq platforms have to set a reasonably sane
- * value here. sparseirq platforms allocate their irq_descs on the fly,
- * so will expand automatically based on the number of registered IRQs.
- */
-#ifdef CONFIG_SPARSE_IRQ
-# define NR_IRQS 8
-#else
-# define NR_IRQS 512
-#endif
-
-/*
* This is a special IRQ number for indicating that no IRQ has been
* triggered and to simply ignore the IRQ dispatch. This is a special
* case that can happen with IRQ auto-distribution when multiple CPUs
#ifndef _SFP_MACHINE_H
#define _SFP_MACHINE_H
+#ifdef __BIG_ENDIAN__
+#define __BYTE_ORDER __BIG_ENDIAN
+#define __LITTLE_ENDIAN 0
+#else
+#define __BYTE_ORDER __LITTLE_ENDIAN
+#define __BIG_ENDIAN 0
+#endif
+
#define _FP_W_TYPE_SIZE 32
#define _FP_W_TYPE unsigned long
#define _FP_WS_TYPE signed long
args[0] = regs->regs[4];
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- regs->regs[1] = args[5];
- regs->regs[0] = args[4];
- regs->regs[7] = args[3];
- regs->regs[6] = args[2];
- regs->regs[5] = args[1];
- regs->regs[4] = args[0];
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
int arch = AUDIT_ARCH_SH;
({ \
long __gu_err = -EFAULT; \
unsigned long __gu_val = 0; \
- const __typeof__(*(ptr)) *__gu_addr = (ptr); \
+ const __typeof__(*(ptr)) __user *__gu_addr = (ptr); \
if (likely(access_ok(__gu_addr, (size)))) \
__get_user_size(__gu_val, __gu_addr, (size), __gu_err); \
(x) = (__force __typeof__(*(ptr)))__gu_val; \
* Clear the area and return remaining number of bytes
* (on failure. Usually it's 0.)
*/
-__kernel_size_t __clear_user(void *addr, __kernel_size_t size);
+__kernel_size_t __clear_user(void __user *addr, __kernel_size_t size);
#define clear_user(addr,n) \
({ \
}
if (!tsk_used_math(tsk)) {
- local_irq_enable();
+ int ret;
/*
* does a slab alloc which can sleep
*/
- if (init_fpu(tsk)) {
+ local_irq_enable();
+ ret = init_fpu(tsk);
+ local_irq_disable();
+ if (ret) {
/*
* ran out of memory!
*/
- do_group_exit(SIGKILL);
+ force_sig(SIGKILL);
return;
}
- local_irq_disable();
}
grab_fpu(regs);
BUILD_BUG_ON(SMP_MSG_NR >= 8);
for (i = 0; i < SMP_MSG_NR; i++)
- request_irq(104 + i, ipi_interrupt_handler,
- IRQF_PERCPU, "IPI", (void *)(long)i);
+ if (request_irq(104 + i, ipi_interrupt_handler,
+ IRQF_PERCPU, "IPI", (void *)(long)i))
+ pr_err("Failed to request irq %d\n", i);
for (i = 0; i < max_cpus; i++)
set_cpu_present(i, true);
ssize_t copy_oldmem_page(unsigned long pfn, char *buf,
size_t csize, unsigned long offset, int userbuf)
{
- void *vaddr;
+ void __iomem *vaddr;
if (!csize)
return 0;
vaddr = ioremap(pfn << PAGE_SHIFT, PAGE_SIZE);
if (userbuf) {
- if (copy_to_user(buf, (vaddr + offset), csize)) {
+ if (copy_to_user((void __user *)buf, (vaddr + offset), csize)) {
iounmap(vaddr);
return -EFAULT;
}
static DEFINE_SPINLOCK(die_lock);
-void die(const char *str, struct pt_regs *regs, long err)
+void __noreturn die(const char *str, struct pt_regs *regs, long err)
{
static int die_counter;
inc_unaligned_user_access();
oldfs = force_uaccess_begin();
- if (copy_from_user(&instruction, (insn_size_t *)(regs->pc & ~1),
+ if (copy_from_user(&instruction, (insn_size_t __user *)(regs->pc & ~1),
sizeof(instruction))) {
force_uaccess_end(oldfs);
goto uspace_segv;
unsigned short inst = 0;
int err;
- get_user(inst, (unsigned short*)regs->pc);
+ get_user(inst, (unsigned short __user *)regs->pc);
err = do_fpu_inst(inst, regs);
if (!err) {
return;
#ifdef CONFIG_SH_FPU_EMU
- get_user(inst, (unsigned short *)regs->pc + 1);
+ get_user(inst, (unsigned short __user *)regs->pc + 1);
if (!do_fpu_inst(inst, regs)) {
- get_user(inst, (unsigned short *)regs->pc);
+ get_user(inst, (unsigned short __user *)regs->pc);
if (!emulate_branch(inst, regs))
return;
/* fault in branch.*/
#define Rn (regs->regs[n])
#define Rm (regs->regs[m])
-#define WRITE(d,a) ({if(put_user(d, (typeof (d)*)a)) return -EFAULT;})
-#define READ(d,a) ({if(get_user(d, (typeof (d)*)a)) return -EFAULT;})
+#define MWRITE(d,a) ({if(put_user(d, (typeof (d) __user *)a)) return -EFAULT;})
+#define MREAD(d,a) ({if(get_user(d, (typeof (d) __user *)a)) return -EFAULT;})
#define PACK_S(r,f) FP_PACK_SP(&r,f)
#define UNPACK_S(f,r) FP_UNPACK_SP(f,&r)
{
if (FPSCR_SZ) {
FMOV_EXT(n);
- READ(FRn, Rm + R0 + 4);
+ MREAD(FRn, Rm + R0 + 4);
n++;
- READ(FRn, Rm + R0);
+ MREAD(FRn, Rm + R0);
} else {
- READ(FRn, Rm + R0);
+ MREAD(FRn, Rm + R0);
}
return 0;
{
if (FPSCR_SZ) {
FMOV_EXT(n);
- READ(FRn, Rm + 4);
+ MREAD(FRn, Rm + 4);
n++;
- READ(FRn, Rm);
+ MREAD(FRn, Rm);
} else {
- READ(FRn, Rm);
+ MREAD(FRn, Rm);
}
return 0;
{
if (FPSCR_SZ) {
FMOV_EXT(n);
- READ(FRn, Rm + 4);
+ MREAD(FRn, Rm + 4);
n++;
- READ(FRn, Rm);
+ MREAD(FRn, Rm);
Rm += 8;
} else {
- READ(FRn, Rm);
+ MREAD(FRn, Rm);
Rm += 4;
}
{
if (FPSCR_SZ) {
FMOV_EXT(m);
- WRITE(FRm, Rn + R0 + 4);
+ MWRITE(FRm, Rn + R0 + 4);
m++;
- WRITE(FRm, Rn + R0);
+ MWRITE(FRm, Rn + R0);
} else {
- WRITE(FRm, Rn + R0);
+ MWRITE(FRm, Rn + R0);
}
return 0;
{
if (FPSCR_SZ) {
FMOV_EXT(m);
- WRITE(FRm, Rn + 4);
+ MWRITE(FRm, Rn + 4);
m++;
- WRITE(FRm, Rn);
+ MWRITE(FRm, Rn);
} else {
- WRITE(FRm, Rn);
+ MWRITE(FRm, Rn);
}
return 0;
if (FPSCR_SZ) {
FMOV_EXT(m);
Rn -= 8;
- WRITE(FRm, Rn + 4);
+ MWRITE(FRm, Rn + 4);
m++;
- WRITE(FRm, Rn);
+ MWRITE(FRm, Rn);
} else {
Rn -= 4;
- WRITE(FRm, Rn);
+ MWRITE(FRm, Rn);
}
return 0;
case 0x4052:
case 0x4062:
Rn -= 4;
- WRITE(*reg, Rn);
+ MWRITE(*reg, Rn);
break;
case 0x4056:
case 0x4066:
- READ(*reg, Rn);
+ MREAD(*reg, Rn);
Rn += 4;
break;
default:
}
/**
- * denormal_to_double - Given denormalized float number,
- * store double float
- *
- * @fpu: Pointer to sh_fpu_soft structure
- * @n: Index to FP register
- */
-static void denormal_to_double(struct sh_fpu_soft_struct *fpu, int n)
-{
- unsigned long du, dl;
- unsigned long x = fpu->fpul;
- int exp = 1023 - 126;
-
- if (x != 0 && (x & 0x7f800000) == 0) {
- du = (x & 0x80000000);
- while ((x & 0x00800000) == 0) {
- x <<= 1;
- exp--;
- }
- x &= 0x007fffff;
- du |= (exp << 20) | (x >> 3);
- dl = x << 29;
-
- fpu->fp_regs[n] = du;
- fpu->fp_regs[n+1] = dl;
- }
-}
-
-/**
- * ieee_fpe_handler - Handle denormalized number exception
- *
- * @regs: Pointer to register structure
- *
- * Returns 1 when it's handled (should not cause exception).
- */
-static int ieee_fpe_handler(struct pt_regs *regs)
-{
- unsigned short insn = *(unsigned short *)regs->pc;
- unsigned short finsn;
- unsigned long nextpc;
- int nib[4] = {
- (insn >> 12) & 0xf,
- (insn >> 8) & 0xf,
- (insn >> 4) & 0xf,
- insn & 0xf};
-
- if (nib[0] == 0xb ||
- (nib[0] == 0x4 && nib[2] == 0x0 && nib[3] == 0xb)) /* bsr & jsr */
- regs->pr = regs->pc + 4;
-
- if (nib[0] == 0xa || nib[0] == 0xb) { /* bra & bsr */
- nextpc = regs->pc + 4 + ((short) ((insn & 0xfff) << 4) >> 3);
- finsn = *(unsigned short *) (regs->pc + 2);
- } else if (nib[0] == 0x8 && nib[1] == 0xd) { /* bt/s */
- if (regs->sr & 1)
- nextpc = regs->pc + 4 + ((char) (insn & 0xff) << 1);
- else
- nextpc = regs->pc + 4;
- finsn = *(unsigned short *) (regs->pc + 2);
- } else if (nib[0] == 0x8 && nib[1] == 0xf) { /* bf/s */
- if (regs->sr & 1)
- nextpc = regs->pc + 4;
- else
- nextpc = regs->pc + 4 + ((char) (insn & 0xff) << 1);
- finsn = *(unsigned short *) (regs->pc + 2);
- } else if (nib[0] == 0x4 && nib[3] == 0xb &&
- (nib[2] == 0x0 || nib[2] == 0x2)) { /* jmp & jsr */
- nextpc = regs->regs[nib[1]];
- finsn = *(unsigned short *) (regs->pc + 2);
- } else if (nib[0] == 0x0 && nib[3] == 0x3 &&
- (nib[2] == 0x0 || nib[2] == 0x2)) { /* braf & bsrf */
- nextpc = regs->pc + 4 + regs->regs[nib[1]];
- finsn = *(unsigned short *) (regs->pc + 2);
- } else if (insn == 0x000b) { /* rts */
- nextpc = regs->pr;
- finsn = *(unsigned short *) (regs->pc + 2);
- } else {
- nextpc = regs->pc + 2;
- finsn = insn;
- }
-
- if ((finsn & 0xf1ff) == 0xf0ad) { /* fcnvsd */
- struct task_struct *tsk = current;
-
- if ((tsk->thread.xstate->softfpu.fpscr & (1 << 17))) {
- /* FPU error */
- denormal_to_double (&tsk->thread.xstate->softfpu,
- (finsn >> 8) & 0xf);
- tsk->thread.xstate->softfpu.fpscr &=
- ~(FPSCR_CAUSE_MASK | FPSCR_FLAG_MASK);
- task_thread_info(tsk)->status |= TS_USEDFPU;
- } else {
- force_sig_fault(SIGFPE, FPE_FLTINV,
- (void __user *)regs->pc);
- }
-
- regs->pc = nextpc;
- return 1;
- }
-
- return 0;
-}
-
-/**
* fpu_init - Initialize FPU registers
* @fpu: Pointer to software emulated FPU registers.
*/
show_fault_oops(regs, address);
die("Oops", regs, error_code);
- bust_spinlocks(0);
- do_exit(SIGKILL);
}
static void
return 0;
}
-__kernel_size_t __clear_user(void *to, __kernel_size_t n)
+__kernel_size_t __clear_user(void __user *to, __kernel_size_t n)
{
- memset(to, 0, n);
+ memset((__force void *)to, 0, n);
return 0;
}
obj-y += net/
obj-y += crypto/
obj-$(CONFIG_SPARC64) += vdso/
+
+# for cleaning
+subdir- += boot
def_bool !64BIT
select ARCH_32BIT_OFF_T
select ARCH_HAS_SYNC_DMA_FOR_CPU
- select GENERIC_ATOMIC64
select CLZ_TAB
+ select DMA_DIRECT_REMAP
+ select GENERIC_ATOMIC64
select HAVE_UID16
select OLD_SIGACTION
select ZONE_DMA
sh $(srctree)/$(boot)/install.sh $(KERNELRELEASE) $(KBUILD_IMAGE) \
System.map "$(INSTALL_PATH)"
-archclean:
- $(Q)$(MAKE) $(clean)=$(boot)
-
archheaders:
$(Q)$(MAKE) $(build)=arch/sparc/kernel/syscalls all
# Actual linking
-$(obj)/zImage: $(obj)/image
+$(obj)/zImage: $(obj)/image FORCE
$(call if_changed,gzip)
@echo ' kernel: $@ is ready'
@echo ' kernel: $@ is ready'
else
-$(obj)/zImage: $(obj)/image
+$(obj)/zImage: $(obj)/image FORCE
$(call if_changed,strip)
@echo ' kernel: $@ is ready'
$(obj)/image.bin: $(obj)/image FORCE
$(call if_changed,objcopy)
-$(obj)/image.gz: $(obj)/image.bin
+$(obj)/image.gz: $(obj)/image.bin FORCE
$(call if_changed,gzip)
UIMAGE_LOADADDR = $(CONFIG_UBOOT_LOAD_ADDR)
-r -b binary $@ -o $@.o
targets += uImage
-$(obj)/uImage: $(obj)/image.gz
+$(obj)/uImage: $(obj)/image.gz FORCE
$(call if_changed,uimage)
$(call if_changed,uimage.o)
@echo ' Image $@ is ready'
}
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- unsigned int i;
-
- for (i = 0; i < 6; i++)
- regs->u_regs[UREG_I0 + i] = args[i];
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
#if defined(CONFIG_SPARC64) && defined(CONFIG_COMPAT)
#include <asm/io-unit.h>
#include <asm/leon.h>
-/* This function must make sure that caches and memory are coherent after DMA
- * On LEON systems without cache snooping it flushes the entire D-CACHE.
- */
-static inline void dma_make_coherent(unsigned long pa, unsigned long len)
-{
- if (sparc_cpu_model == sparc_leon) {
- if (!sparc_leon3_snooping_enabled())
- leon_flush_dcache_all();
- }
-}
-
static void __iomem *_sparc_ioremap(struct resource *res, u32 bus, u32 pa, int sz);
static void __iomem *_sparc_alloc_io(unsigned int busno, unsigned long phys,
unsigned long size, char *name);
#endif /* CONFIG_SBUS */
-
-/* Allocate and map kernel buffer using consistent mode DMA for a device.
- * hwdev should be valid struct pci_dev pointer for PCI devices.
- */
-void *arch_dma_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle,
- gfp_t gfp, unsigned long attrs)
-{
- unsigned long addr;
- void *va;
-
- if (!size || size > 256 * 1024) /* __get_free_pages() limit */
- return NULL;
-
- size = PAGE_ALIGN(size);
- va = (void *) __get_free_pages(gfp | __GFP_ZERO, get_order(size));
- if (!va) {
- printk("%s: no %zd pages\n", __func__, size >> PAGE_SHIFT);
- return NULL;
- }
-
- addr = sparc_dma_alloc_resource(dev, size);
- if (!addr)
- goto err_nomem;
-
- srmmu_mapiorange(0, virt_to_phys(va), addr, size);
-
- *dma_handle = virt_to_phys(va);
- return (void *)addr;
-
-err_nomem:
- free_pages((unsigned long)va, get_order(size));
- return NULL;
-}
-
-/* Free and unmap a consistent DMA buffer.
- * cpu_addr is what was returned arch_dma_alloc, size must be the same as what
- * was passed into arch_dma_alloc, and likewise dma_addr must be the same as
- * what *dma_ndler was set to.
+/*
+ * IIep is write-through, not flushing on cpu to device transfer.
*
- * References to the memory and mappings associated with cpu_addr/dma_addr
- * past this call are illegal.
+ * On LEON systems without cache snooping, the entire D-CACHE must be flushed to
+ * make DMA to cacheable memory coherent.
*/
-void arch_dma_free(struct device *dev, size_t size, void *cpu_addr,
- dma_addr_t dma_addr, unsigned long attrs)
-{
- size = PAGE_ALIGN(size);
-
- if (!sparc_dma_free_resource(cpu_addr, size))
- return;
-
- dma_make_coherent(dma_addr, size);
- srmmu_unmapiorange((unsigned long)cpu_addr, size);
- free_pages((unsigned long)phys_to_virt(dma_addr), get_order(size));
-}
-
-/* IIep is write-through, not flushing on cpu to device transfer. */
-
void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size,
enum dma_data_direction dir)
{
- if (dir != PCI_DMA_TODEVICE)
- dma_make_coherent(paddr, PAGE_ALIGN(size));
+ if (dir != PCI_DMA_TODEVICE &&
+ sparc_cpu_model == sparc_leon &&
+ !sparc_leon3_snooping_enabled())
+ leon_flush_dcache_all();
}
#ifdef CONFIG_PROC_FS
}
#ifdef CONFIG_PCI_IOV
-int pcibios_add_device(struct pci_dev *dev)
+int pcibios_device_add(struct pci_dev *dev)
{
struct pci_dev *pdev;
get_sigframe(ksig, regs, sigframe_size);
if (invalid_frame_pointer(sf, sigframe_size)) {
- do_exit(SIGILL);
+ force_exit_sig(SIGILL);
return -EINVAL;
}
sf = (struct rt_signal_frame __user *)
get_sigframe(ksig, regs, sigframe_size);
if (invalid_frame_pointer(sf, sigframe_size)) {
- do_exit(SIGILL);
+ force_exit_sig(SIGILL);
return -EINVAL;
}
static void __init pcpu_free_bootmem(void *ptr, size_t size)
{
- memblock_free(__pa(ptr), size);
+ memblock_free(ptr, size);
}
static int __init pcpu_cpu_distance(unsigned int from, unsigned int to)
if ((sp & 7) ||
copy_to_user((char __user *) sp, &tp->reg_window[window],
- sizeof(struct reg_window32)))
- do_exit(SIGILL);
+ sizeof(struct reg_window32))) {
+ force_exit_sig(SIGILL);
+ return;
+ }
}
tp->w_saved = 0;
}
}
unhandled_fault(address, tsk, regs);
- do_exit(SIGKILL);
/*
* We ran out of memory, or some other thing happened to us that made
default:
printk(KERN_ERR "TSB[%s:%d]: Impossible TSB size %lu, killing process.\n",
current->comm, current->pid, tsb_bytes);
- do_exit(SIGSEGV);
+ BUG();
}
tte |= pte_sz_bits(page_sz);
*args = UPT_SYSCALL_ARG6(r);
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- struct uml_pt_regs *r = ®s->regs;
-
- UPT_SYSCALL_ARG1(r) = *args++;
- UPT_SYSCALL_ARG2(r) = *args++;
- UPT_SYSCALL_ARG3(r) = *args++;
- UPT_SYSCALL_ARG4(r) = *args++;
- UPT_SYSCALL_ARG5(r) = *args++;
- UPT_SYSCALL_ARG6(r) = *args;
-}
-
/* See arch/x86/um/asm/syscall.h for syscall_get_arch() definition. */
#endif /* __UM_SYSCALL_GENERIC_H */
*/
brk_end = (unsigned long) UML_ROUND_UP(sbrk(0));
map_memory(brk_end, __pa(brk_end), uml_reserved - brk_end, 1, 1, 0);
- memblock_free(__pa(brk_end), uml_reserved - brk_end);
+ memblock_free((void *)brk_end, uml_reserved - brk_end);
uml_reserved = brk_end;
/* this will put all low memory onto the freelists */
void fatal_sigsegv(void)
{
- force_sigsegv(SIGSEGV);
+ force_fatal_sig(SIGSEGV);
do_signal(¤t->thread.regs);
/*
* This is to tell gcc that we're not returning - do_signal
obj-y += net/
obj-$(CONFIG_KEXEC_FILE) += purgatory/
+
+# for cleaning
+subdir- += boot tools
select ARCH_CLOCKSOURCE_INIT
select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE
select ARCH_ENABLE_HUGEPAGE_MIGRATION if X86_64 && HUGETLB_PAGE && MIGRATION
- select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64 || (X86_32 && HIGHMEM)
+ select ARCH_ENABLE_MEMORY_HOTPLUG if X86_64
select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if (PGTABLE_LEVELS > 2) && (X86_64 || X86_PAE)
select ARCH_ENABLE_THP_MIGRATION if X86_64 && TRANSPARENT_HUGEPAGE
select HAVE_DYNAMIC_FTRACE_WITH_REGS
select HAVE_DYNAMIC_FTRACE_WITH_ARGS if X86_64
select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
+ select HAVE_SAMPLE_FTRACE_DIRECT if X86_64
+ select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64
select HAVE_EBPF_JIT
select HAVE_EFFICIENT_UNALIGNED_ACCESS
select HAVE_EISA
config ARCH_MEMORY_PROBE
bool "Enable sysfs memory/probe interface"
- depends on X86_64 && MEMORY_HOTPLUG
+ depends on MEMORY_HOTPLUG
help
This option enables a sysfs memory/probe interface for testing.
See Documentation/admin-guide/mm/memory-hotplug.rst for more information.
config ARCH_HAS_ADD_PAGES
def_bool y
- depends on X86_64 && ARCH_ENABLE_MEMORY_HOTPLUG
+ depends on ARCH_ENABLE_MEMORY_HOTPLUG
config ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE
def_bool y
archclean:
$(Q)rm -rf $(objtree)/arch/i386
$(Q)rm -rf $(objtree)/arch/x86_64
- $(Q)$(MAKE) $(clean)=$(boot)
- $(Q)$(MAKE) $(clean)=arch/x86/tools
define archhelp
echo '* bzImage - Compressed kernel image (arch/x86/boot/bzImage)'
if ((!tmp && regs->orig_ax != syscall_nr) || regs->ip != address) {
warn_bad_vsyscall(KERN_DEBUG, regs,
"seccomp tried to change syscall nr or ip");
- do_exit(SIGSYS);
+ force_exit_sig(SIGSYS);
+ return true;
}
regs->orig_ax = -1;
if (tmp)
/* must not have branches... */
local_irq_save(flags);
__intel_pmu_disable_all(false); /* we don't care about BTS */
- __intel_pmu_pebs_disable_all();
__intel_pmu_lbr_disable();
/* ... until here */
return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
/* must not have branches... */
local_irq_save(flags);
__intel_pmu_disable_all(false); /* we don't care about BTS */
- __intel_pmu_pebs_disable_all();
__intel_pmu_arch_lbr_disable();
/* ... until here */
return __intel_pmu_snapshot_branch_stack(entries, cnt, flags);
{
struct event_constraint *c = &vlbr_constraint;
- if (unlikely(constraint_match(c, event->hw.config)))
+ if (unlikely(constraint_match(c, event->hw.config))) {
+ event->hw.flags |= c->flags;
return c;
+ }
return NULL;
}
cpuc->last_task_ctx = NULL;
cpuc->last_log_id = 0;
+ if (!static_cpu_has(X86_FEATURE_ARCH_LBR) && cpuc->lbr_select)
+ wrmsrl(MSR_LBR_SELECT, 0);
}
/*
* PCI slot and func to indicate the uncore box.
*/
if (id->driver_data & ~0xffff) {
- struct pci_driver *pci_drv = pdev->driver;
+ struct pci_driver *pci_drv = to_pci_driver(pdev->dev.driver);
pmu = uncore_pci_find_dev_pmu(pdev, pci_drv->id_table);
if (pmu == NULL)
struct hw_perf_event_extra *reg1 = &event->hw.extra_reg;
struct extra_reg *er;
int idx = 0;
+ /* Any of the CHA events may be filtered by Thread/Core-ID.*/
+ if (event->hw.config & SNBEP_CBO_PMON_CTL_TID_EN)
+ idx = SKX_CHA_MSR_PMON_BOX_FILTER_TID;
for (er = skx_uncore_cha_extra_regs; er->msr; er++) {
if (er->event != (event->hw.config & er->config_mask))
UNCORE_EVENT_CONSTRAINT(0xc0, 0xc),
UNCORE_EVENT_CONSTRAINT(0xc5, 0xc),
UNCORE_EVENT_CONSTRAINT(0xd4, 0xc),
+ UNCORE_EVENT_CONSTRAINT(0xd5, 0xc),
EVENT_CONSTRAINT_END
};
pmu_iio_cleanup_mapping(type, &snr_iio_mapping_group);
}
+static struct event_constraint snr_uncore_iio_constraints[] = {
+ UNCORE_EVENT_CONSTRAINT(0x83, 0x3),
+ UNCORE_EVENT_CONSTRAINT(0xc0, 0xc),
+ UNCORE_EVENT_CONSTRAINT(0xd5, 0xc),
+ EVENT_CONSTRAINT_END
+};
+
static struct intel_uncore_type snr_uncore_iio = {
.name = "iio",
.num_counters = 4,
.event_mask_ext = SNR_IIO_PMON_RAW_EVENT_MASK_EXT,
.box_ctl = SNR_IIO_MSR_PMON_BOX_CTL,
.msr_offset = SNR_IIO_MSR_OFFSET,
+ .constraints = snr_uncore_iio_constraints,
.ops = &ivbep_uncore_msr_ops,
.format_group = &snr_uncore_iio_format_group,
.attr_update = snr_iio_attr_update,
return;
}
+ if (!hv_vp_index)
+ return;
+
hv_reenlightenment_cb = cb;
/* Make sure callback is registered before we write to MSRs */
*/
void __init hyperv_init(void)
{
- u64 guest_id, required_msrs;
+ u64 guest_id;
union hv_x64_msr_hypercall_contents hypercall_msr;
int cpuhp;
if (x86_hyper_type != X86_HYPER_MS_HYPERV)
return;
- /* Absolutely required MSRs */
- required_msrs = HV_MSR_HYPERCALL_AVAILABLE |
- HV_MSR_VP_INDEX_AVAILABLE;
-
- if ((ms_hyperv.features & required_msrs) != required_msrs)
- return;
-
if (hv_common_init())
return;
#define _ASM_X86_FPU_XCR_H
#define XCR_XFEATURE_ENABLED_MASK 0x00000000
+#define XCR_XFEATURE_IN_USE_MASK 0x00000001
static inline u64 xgetbv(u32 index)
{
asm volatile("xsetbv" :: "a" (eax), "d" (edx), "c" (index));
}
+/*
+ * Return a mask of xfeatures which are currently being tracked
+ * by the processor as being in the initial configuration.
+ *
+ * Callers should check X86_FEATURE_XGETBV1.
+ */
+static inline u64 xfeatures_in_use(void)
+{
+ return xgetbv(XCR_XFEATURE_IN_USE_MASK);
+}
+
#endif /* _ASM_X86_FPU_XCR_H */
#define XFEATURE_MASK_FPSTATE (XFEATURE_MASK_USER_RESTORE | \
XFEATURE_MASK_SUPERVISOR_SUPPORTED)
+/*
+ * Features in this mask have space allocated in the signal frame, but may not
+ * have that space initialized when the feature is in its init state.
+ */
+#define XFEATURE_MASK_SIGFRAME_INITOPT (XFEATURE_MASK_XTILE | \
+ XFEATURE_MASK_USER_DYNAMIC)
+
extern u64 xstate_fx_sw_bytes[USER_XSTATE_FX_SW_WORDS];
extern void __init update_regset_xstate_info(unsigned int size,
#define INTEL_FAM6_ALDERLAKE 0x97 /* Golden Cove / Gracemont */
#define INTEL_FAM6_ALDERLAKE_L 0x9A /* Golden Cove / Gracemont */
+#define INTEL_FAM6_RAPTOR_LAKE 0xB7
+
/* "Small Core" Processors (Atom) */
#define INTEL_FAM6_ATOM_BONNELL 0x1C /* Diamondville, Pineview */
#define __KVM_HAVE_ARCH_VCPU_DEBUGFS
#define KVM_MAX_VCPUS 1024
-#define KVM_SOFT_MAX_VCPUS 710
/*
* In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
unsigned int cr4_smap:1;
unsigned int cr4_smep:1;
unsigned int cr4_la57:1;
+ unsigned int efer_lma:1;
};
};
int cpuid_nent;
struct kvm_cpuid_entry2 *cpuid_entries;
+ u32 kvm_cpuid_base;
u64 reserved_gpa_bits;
int maxphyaddr;
u8 preempted;
u64 msr_val;
u64 last_steal;
- struct gfn_to_pfn_cache cache;
+ struct gfn_to_hva_cache cache;
} st;
u64 l1_tsc_offset;
#define APICV_INHIBIT_REASON_IRQWIN 3
#define APICV_INHIBIT_REASON_PIT_REINJ 4
#define APICV_INHIBIT_REASON_X2APIC 5
+#define APICV_INHIBIT_REASON_BLOCKIRQ 6
struct kvm_arch {
unsigned long n_used_mmu_pages;
int (*mem_enc_reg_region)(struct kvm *kvm, struct kvm_enc_region *argp);
int (*mem_enc_unreg_region)(struct kvm *kvm, struct kvm_enc_region *argp);
int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
+ int (*vm_move_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
int (*get_msr_feature)(struct kvm_msr_entry *entry);
return ret;
}
+static inline long kvm_sev_hypercall3(unsigned int nr, unsigned long p1,
+ unsigned long p2, unsigned long p3)
+{
+ long ret;
+
+ asm volatile("vmmcall"
+ : "=a"(ret)
+ : "a"(nr), "b"(p1), "c"(p2), "d"(p3)
+ : "memory");
+ return ret;
+}
+
#ifdef CONFIG_KVM_GUEST
void kvmclock_init(void);
void kvmclock_disable(void);
int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size);
int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size);
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages,
+ bool enc);
void __init mem_encrypt_free_decrypted_mem(void);
early_set_memory_decrypted(unsigned long vaddr, unsigned long size) { return 0; }
static inline int __init
early_set_memory_encrypted(unsigned long vaddr, unsigned long size) { return 0; }
+static inline void __init
+early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages, bool enc) {}
static inline void mem_encrypt_free_decrypted_mem(void) { }
PVOP_VCALL1(mmu.exit_mmap, mm);
}
+static inline void notify_page_enc_status_changed(unsigned long pfn,
+ int npages, bool enc)
+{
+ PVOP_VCALL3(mmu.notify_page_enc_status_changed, pfn, npages, enc);
+}
+
#ifdef CONFIG_PARAVIRT_XXL
static inline void load_sp0(unsigned long sp0)
{
/* Hook for intercepting the destruction of an mm_struct. */
void (*exit_mmap)(struct mm_struct *mm);
+ void (*notify_page_enc_status_changed)(unsigned long pfn, int npages, bool enc);
#ifdef CONFIG_PARAVIRT_XXL
struct paravirt_callee_save read_cr2;
void paravirt_flush_lazy_mmu(void);
void _paravirt_nop(void);
+void paravirt_BUG(void);
u64 _paravirt_ident_64(u64);
+unsigned long paravirt_ret0(void);
#define paravirt_nop ((void *)_paravirt_nop)
static inline u32 amd_get_highest_perf(void) { return 0; }
#endif
+#define for_each_possible_hypervisor_cpuid_base(function) \
+ for (function = 0x40000000; function < 0x40010000; function += 0x100)
+
static inline uint32_t hypervisor_cpuid_base(const char *sig, uint32_t leaves)
{
uint32_t base, eax, signature[3];
- for (base = 0x40000000; base < 0x40010000; base += 0x100) {
+ for_each_possible_hypervisor_cpuid_base(base) {
cpuid(base, &eax, &signature[0], &signature[1], &signature[2]);
if (!memcmp(sig, signature, 12) &&
int set_direct_map_invalid_noflush(struct page *page);
int set_direct_map_default_noflush(struct page *page);
bool kernel_page_present(struct page *page);
+void notify_range_enc_status_changed(unsigned long vaddr, int npages, bool enc);
extern int kernel_set_to_readonly;
void cpu_disable_common(void);
void native_smp_prepare_boot_cpu(void);
+void smp_prepare_cpus_common(void);
void native_smp_prepare_cpus(unsigned int max_cpus);
void calculate_max_logical_packages(void);
void native_smp_cpus_done(unsigned int max_cpus);
".globl " STATIC_CALL_TRAMP_STR(name) " \n" \
STATIC_CALL_TRAMP_STR(name) ": \n" \
insns " \n" \
+ ".byte 0x53, 0x43, 0x54 \n" \
".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \
".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \
".popsection \n")
memcpy(args, ®s->bx, 6 * sizeof(args[0]));
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- unsigned int i, unsigned int n,
- const unsigned long *args)
-{
- BUG_ON(i + n > 6);
- memcpy(®s->bx + i, args, n * sizeof(args[0]));
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
return AUDIT_ARCH_I386;
}
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
-# ifdef CONFIG_IA32_EMULATION
- if (task->thread_info.status & TS_COMPAT) {
- regs->bx = *args++;
- regs->cx = *args++;
- regs->dx = *args++;
- regs->si = *args++;
- regs->di = *args++;
- regs->bp = *args;
- } else
-# endif
- {
- regs->di = *args++;
- regs->si = *args++;
- regs->dx = *args++;
- regs->r10 = *args++;
- regs->r8 = *args++;
- regs->r9 = *args;
- }
-}
-
static inline int syscall_get_arch(struct task_struct *task)
{
/* x32 tasks should be considered AUDIT_ARCH_X86_64. */
return res;
}
+#ifdef CONFIG_XEN_PV
static inline int
HYPERVISOR_set_trap_table(struct trap_info *table)
{
}
static inline int
+HYPERVISOR_set_debugreg(int reg, unsigned long value)
+{
+ return _hypercall2(int, set_debugreg, reg, value);
+}
+
+static inline unsigned long
+HYPERVISOR_get_debugreg(int reg)
+{
+ return _hypercall1(unsigned long, get_debugreg, reg);
+}
+
+static inline int
+HYPERVISOR_update_descriptor(u64 ma, u64 desc)
+{
+ return _hypercall2(int, update_descriptor, ma, desc);
+}
+
+static inline int
+HYPERVISOR_update_va_mapping(unsigned long va, pte_t new_val,
+ unsigned long flags)
+{
+ return _hypercall3(int, update_va_mapping, va, new_val.pte, flags);
+}
+
+static inline int
+HYPERVISOR_set_segment_base(int reg, unsigned long value)
+{
+ return _hypercall2(int, set_segment_base, reg, value);
+}
+
+static inline void
+MULTI_fpu_taskswitch(struct multicall_entry *mcl, int set)
+{
+ mcl->op = __HYPERVISOR_fpu_taskswitch;
+ mcl->args[0] = set;
+
+ trace_xen_mc_entry(mcl, 1);
+}
+
+static inline void
+MULTI_update_va_mapping(struct multicall_entry *mcl, unsigned long va,
+ pte_t new_val, unsigned long flags)
+{
+ mcl->op = __HYPERVISOR_update_va_mapping;
+ mcl->args[0] = va;
+ mcl->args[1] = new_val.pte;
+ mcl->args[2] = flags;
+
+ trace_xen_mc_entry(mcl, 3);
+}
+
+static inline void
+MULTI_update_descriptor(struct multicall_entry *mcl, u64 maddr,
+ struct desc_struct desc)
+{
+ mcl->op = __HYPERVISOR_update_descriptor;
+ mcl->args[0] = maddr;
+ mcl->args[1] = *(unsigned long *)&desc;
+
+ trace_xen_mc_entry(mcl, 2);
+}
+
+static inline void
+MULTI_mmu_update(struct multicall_entry *mcl, struct mmu_update *req,
+ int count, int *success_count, domid_t domid)
+{
+ mcl->op = __HYPERVISOR_mmu_update;
+ mcl->args[0] = (unsigned long)req;
+ mcl->args[1] = count;
+ mcl->args[2] = (unsigned long)success_count;
+ mcl->args[3] = domid;
+
+ trace_xen_mc_entry(mcl, 4);
+}
+
+static inline void
+MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count,
+ int *success_count, domid_t domid)
+{
+ mcl->op = __HYPERVISOR_mmuext_op;
+ mcl->args[0] = (unsigned long)op;
+ mcl->args[1] = count;
+ mcl->args[2] = (unsigned long)success_count;
+ mcl->args[3] = domid;
+
+ trace_xen_mc_entry(mcl, 4);
+}
+
+static inline void
+MULTI_stack_switch(struct multicall_entry *mcl,
+ unsigned long ss, unsigned long esp)
+{
+ mcl->op = __HYPERVISOR_stack_switch;
+ mcl->args[0] = ss;
+ mcl->args[1] = esp;
+
+ trace_xen_mc_entry(mcl, 2);
+}
+#endif
+
+static inline int
HYPERVISOR_sched_op(int cmd, void *arg)
{
return _hypercall2(int, sched_op, cmd, arg);
return _hypercall1(int, platform_op, op);
}
-static __always_inline int
-HYPERVISOR_set_debugreg(int reg, unsigned long value)
-{
- return _hypercall2(int, set_debugreg, reg, value);
-}
-
-static __always_inline unsigned long
-HYPERVISOR_get_debugreg(int reg)
-{
- return _hypercall1(unsigned long, get_debugreg, reg);
-}
-
-static inline int
-HYPERVISOR_update_descriptor(u64 ma, u64 desc)
-{
- if (sizeof(u64) == sizeof(long))
- return _hypercall2(int, update_descriptor, ma, desc);
- return _hypercall4(int, update_descriptor, ma, ma>>32, desc, desc>>32);
-}
-
static inline long
HYPERVISOR_memory_op(unsigned int cmd, void *arg)
{
}
static inline int
-HYPERVISOR_update_va_mapping(unsigned long va, pte_t new_val,
- unsigned long flags)
-{
- if (sizeof(new_val) == sizeof(long))
- return _hypercall3(int, update_va_mapping, va,
- new_val.pte, flags);
- else
- return _hypercall4(int, update_va_mapping, va,
- new_val.pte, new_val.pte >> 32, flags);
-}
-
-static inline int
HYPERVISOR_event_channel_op(int cmd, void *arg)
{
return _hypercall2(int, event_channel_op, cmd, arg);
return _hypercall3(int, vcpu_op, cmd, vcpuid, extra_args);
}
-#ifdef CONFIG_X86_64
-static inline int
-HYPERVISOR_set_segment_base(int reg, unsigned long value)
-{
- return _hypercall2(int, set_segment_base, reg, value);
-}
-#endif
-
static inline int
HYPERVISOR_suspend(unsigned long start_info_mfn)
{
}
static inline int
-HYPERVISOR_tmem_op(
- struct tmem_op *op)
-{
- return _hypercall1(int, tmem_op, op);
-}
-
-static inline int
HYPERVISOR_xenpmu_op(unsigned int op, void *arg)
{
return _hypercall2(int, xenpmu_op, op, arg);
return ret;
}
-static inline void
-MULTI_fpu_taskswitch(struct multicall_entry *mcl, int set)
-{
- mcl->op = __HYPERVISOR_fpu_taskswitch;
- mcl->args[0] = set;
-
- trace_xen_mc_entry(mcl, 1);
-}
-
-static inline void
-MULTI_update_va_mapping(struct multicall_entry *mcl, unsigned long va,
- pte_t new_val, unsigned long flags)
-{
- mcl->op = __HYPERVISOR_update_va_mapping;
- mcl->args[0] = va;
- if (sizeof(new_val) == sizeof(long)) {
- mcl->args[1] = new_val.pte;
- mcl->args[2] = flags;
- } else {
- mcl->args[1] = new_val.pte;
- mcl->args[2] = new_val.pte >> 32;
- mcl->args[3] = flags;
- }
-
- trace_xen_mc_entry(mcl, sizeof(new_val) == sizeof(long) ? 3 : 4);
-}
-
-static inline void
-MULTI_update_descriptor(struct multicall_entry *mcl, u64 maddr,
- struct desc_struct desc)
-{
- mcl->op = __HYPERVISOR_update_descriptor;
- if (sizeof(maddr) == sizeof(long)) {
- mcl->args[0] = maddr;
- mcl->args[1] = *(unsigned long *)&desc;
- } else {
- u32 *p = (u32 *)&desc;
-
- mcl->args[0] = maddr;
- mcl->args[1] = maddr >> 32;
- mcl->args[2] = *p++;
- mcl->args[3] = *p;
- }
-
- trace_xen_mc_entry(mcl, sizeof(maddr) == sizeof(long) ? 2 : 4);
-}
-
-static inline void
-MULTI_mmu_update(struct multicall_entry *mcl, struct mmu_update *req,
- int count, int *success_count, domid_t domid)
-{
- mcl->op = __HYPERVISOR_mmu_update;
- mcl->args[0] = (unsigned long)req;
- mcl->args[1] = count;
- mcl->args[2] = (unsigned long)success_count;
- mcl->args[3] = domid;
-
- trace_xen_mc_entry(mcl, 4);
-}
-
-static inline void
-MULTI_mmuext_op(struct multicall_entry *mcl, struct mmuext_op *op, int count,
- int *success_count, domid_t domid)
-{
- mcl->op = __HYPERVISOR_mmuext_op;
- mcl->args[0] = (unsigned long)op;
- mcl->args[1] = count;
- mcl->args[2] = (unsigned long)success_count;
- mcl->args[3] = domid;
-
- trace_xen_mc_entry(mcl, 4);
-}
-
-static inline void
-MULTI_stack_switch(struct multicall_entry *mcl,
- unsigned long ss, unsigned long esp)
-{
- mcl->op = __HYPERVISOR_stack_switch;
- mcl->args[0] = ss;
- mcl->args[1] = esp;
-
- trace_xen_mc_entry(mcl, 2);
-}
-
#endif /* _ASM_X86_XEN_HYPERCALL_H */
void xen_arch_unregister_cpu(int num);
#endif
+#ifdef CONFIG_PVH
+void __init xen_pvh_init(struct boot_params *boot_params);
+#endif
+
#endif /* _ASM_X86_XEN_HYPERVISOR_H */
return -1;
}
#endif
-#ifdef CONFIG_XEN_DOM0
-int xen_find_device_domain_owner(struct pci_dev *dev);
-int xen_register_device_domain_owner(struct pci_dev *dev, uint16_t domain);
-int xen_unregister_device_domain_owner(struct pci_dev *dev);
-#else
-static inline int xen_find_device_domain_owner(struct pci_dev *dev)
-{
- return -1;
-}
-static inline int xen_register_device_domain_owner(struct pci_dev *dev,
- uint16_t domain)
-{
- return -1;
-}
-static inline int xen_unregister_device_domain_owner(struct pci_dev *dev)
-{
- return -1;
-}
-#endif
#if defined(CONFIG_PCI_MSI)
#if defined(CONFIG_PCI_XEN)
* should be used to determine that a VM is running under KVM.
*/
#define KVM_CPUID_SIGNATURE 0x40000000
+#define KVM_SIGNATURE "KVMKVMKVM\0\0\0"
/* This CPUID returns two feature bitmaps in eax, edx. Before enabling
* a particular paravirtualization, the appropriate feature bit should
(pfn >= aperture_pfn_start + aperture_page_count));
}
+#ifdef CONFIG_PROC_VMCORE
+static bool gart_oldmem_pfn_is_ram(struct vmcore_cb *cb, unsigned long pfn)
+{
+ return !!gart_mem_pfn_is_ram(pfn);
+}
+
+static struct vmcore_cb gart_vmcore_cb = {
+ .pfn_is_ram = gart_oldmem_pfn_is_ram,
+};
+#endif
+
static void __init exclude_from_core(u64 aper_base, u32 aper_order)
{
aperture_pfn_start = aper_base >> PAGE_SHIFT;
aperture_page_count = (32 * 1024 * 1024) << aper_order >> PAGE_SHIFT;
#ifdef CONFIG_PROC_VMCORE
- WARN_ON(register_oldmem_pfn_is_ram(&gart_mem_pfn_is_ram));
+ register_vmcore_cb(&gart_vmcore_cb);
#endif
#ifdef CONFIG_PROC_KCORE
WARN_ON(register_mem_pfn_is_ram(&gart_mem_pfn_is_ram));
{ X86_FEATURE_SGX1, X86_FEATURE_SGX },
{ X86_FEATURE_SGX2, X86_FEATURE_SGX1 },
{ X86_FEATURE_XFD, X86_FEATURE_XSAVES },
+ { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 },
{ X86_FEATURE_AMX_TILE, X86_FEATURE_XFD },
{}
};
{
struct cpuinfo_x86 *c = &boot_cpu_data;
- /* MCE errata HSD131, HSM142, HSW131, BDM48, and HSM142 */
+ /* MCE errata HSD131, HSM142, HSW131, BDM48, HSM142 and SKX37 */
if ((c->x86 == 6) &&
((c->x86_model == INTEL_FAM6_HASWELL) ||
(c->x86_model == INTEL_FAM6_HASWELL_L) ||
(c->x86_model == INTEL_FAM6_BROADWELL) ||
- (c->x86_model == INTEL_FAM6_HASWELL_G)) &&
+ (c->x86_model == INTEL_FAM6_HASWELL_G) ||
+ (c->x86_model == INTEL_FAM6_SKYLAKE_X)) &&
(m->bank == 0) &&
((m->status & 0xa0000000ffffffff) == 0x80000000000f0005))
return true;
cpuid(HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS,
&eax, &hyp_signature[0], &hyp_signature[1], &hyp_signature[2]);
- if (eax >= HYPERV_CPUID_MIN &&
- eax <= HYPERV_CPUID_MAX &&
- !memcmp("Microsoft Hv", hyp_signature, 12))
- return HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS;
+ if (eax < HYPERV_CPUID_MIN || eax > HYPERV_CPUID_MAX ||
+ memcmp("Microsoft Hv", hyp_signature, 12))
+ return 0;
- return 0;
+ /* HYPERCALL and VP_INDEX MSRs are mandatory for all features. */
+ eax = cpuid_eax(HYPERV_CPUID_FEATURES);
+ if (!(eax & HV_MSR_HYPERCALL_AVAILABLE)) {
+ pr_warn("x86/hyperv: HYPERCALL MSR not available.\n");
+ return 0;
+ }
+ if (!(eax & HV_MSR_VP_INDEX_AVAILABLE)) {
+ pr_warn("x86/hyperv: VP_INDEX MSR not available.\n");
+ return 0;
+ }
+
+ return HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS;
}
static unsigned char hv_get_nmi_reason(void)
static LIST_HEAD(sgx_active_page_list);
static DEFINE_SPINLOCK(sgx_reclaimer_lock);
-/* The free page list lock protected variables prepend the lock. */
-static unsigned long sgx_nr_free_pages;
+static atomic_long_t sgx_nr_free_pages = ATOMIC_LONG_INIT(0);
/* Nodes with one or more EPC sections. */
static nodemask_t sgx_numa_mask;
spin_lock(&node->lock);
list_add_tail(&epc_page->list, &node->free_page_list);
- sgx_nr_free_pages++;
spin_unlock(&node->lock);
+ atomic_long_inc(&sgx_nr_free_pages);
}
}
static bool sgx_should_reclaim(unsigned long watermark)
{
- return sgx_nr_free_pages < watermark && !list_empty(&sgx_active_page_list);
+ return atomic_long_read(&sgx_nr_free_pages) < watermark &&
+ !list_empty(&sgx_active_page_list);
}
static int ksgxd(void *p)
page = list_first_entry(&node->free_page_list, struct sgx_epc_page, list);
list_del_init(&page->list);
- sgx_nr_free_pages--;
spin_unlock(&node->lock);
+ atomic_long_dec(&sgx_nr_free_pages);
return page;
}
spin_lock(&node->lock);
list_add_tail(&page->list, &node->free_page_list);
- sgx_nr_free_pages++;
spin_unlock(&node->lock);
+ atomic_long_inc(&sgx_nr_free_pages);
}
static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,
* some way to reconstruct CR3. We could make a credible guess based
* on cpu_tlbstate, but that would be racy and would not account for
* PTI.
- *
- * Instead, don't bother. We can return through
- * rewind_stack_do_exit() instead.
*/
panic("cannot return from double fault\n");
}
#include <asm/cpufeature.h>
#include <asm/fpu/xstate.h>
+#include <asm/fpu/xcr.h>
#ifdef CONFIG_X86_64
DECLARE_PER_CPU(u64, xfd_state);
}
/*
+ * XSAVE itself always writes all requested xfeatures. Removing features
+ * from the request bitmap reduces the features which are written.
+ * Generate a mask of features which must be written to a sigframe. The
+ * unset features can be optimized away and not written.
+ *
+ * This optimization is user-visible. Only use for states where
+ * uninitialized sigframe contents are tolerable, like dynamic features.
+ *
+ * Users of buffers produced with this optimization must check XSTATE_BV
+ * to determine which features have been optimized out.
+ */
+static inline u64 xfeatures_need_sigframe_write(void)
+{
+ u64 xfeaures_to_write;
+
+ /* In-use features must be written: */
+ xfeaures_to_write = xfeatures_in_use();
+
+ /* Also write all non-optimizable sigframe features: */
+ xfeaures_to_write |= XFEATURE_MASK_USER_SUPPORTED &
+ ~XFEATURE_MASK_SIGFRAME_INITOPT;
+
+ return xfeaures_to_write;
+}
+
+/*
* Save xstate to user space xsave area.
*
* We don't use modified optimization because xrstor/xrstors might track
*/
struct fpstate *fpstate = current->thread.fpu.fpstate;
u64 mask = fpstate->user_xfeatures;
- u32 lmask = mask;
- u32 hmask = mask >> 32;
+ u32 lmask;
+ u32 hmask;
int err;
+ /* Optimize away writing unnecessary xfeatures: */
+ if (fpu_state_size_dynamic())
+ mask &= xfeatures_need_sigframe_write();
+
+ lmask = mask;
+ hmask = mask >> 32;
xfd_validate_state(fpstate, mask, false);
stac();
#include <linux/swait.h>
#include <linux/syscore_ops.h>
#include <linux/cc_platform.h>
+#include <linux/efi.h>
#include <asm/timer.h>
#include <asm/cpu.h>
#include <asm/traps.h>
#include <asm/ptrace.h>
#include <asm/reboot.h>
#include <asm/svm.h>
+#include <asm/e820/api.h>
DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled);
kvm_disable_steal_time();
if (kvm_para_has_feature(KVM_FEATURE_PV_EOI))
wrmsrl(MSR_KVM_PV_EOI_EN, 0);
+ if (kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL))
+ wrmsrl(MSR_KVM_MIGRATION_CONTROL, 0);
kvm_pv_disable_apf();
if (!shutdown)
apf_task_wake_all();
__send_ipi_mask(local_mask, vector);
}
+static int __init setup_efi_kvm_sev_migration(void)
+{
+ efi_char16_t efi_sev_live_migration_enabled[] = L"SevLiveMigrationEnabled";
+ efi_guid_t efi_variable_guid = AMD_SEV_MEM_ENCRYPT_GUID;
+ efi_status_t status;
+ unsigned long size;
+ bool enabled;
+
+ if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) ||
+ !kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL))
+ return 0;
+
+ if (!efi_enabled(EFI_BOOT))
+ return 0;
+
+ if (!efi_enabled(EFI_RUNTIME_SERVICES)) {
+ pr_info("%s : EFI runtime services are not enabled\n", __func__);
+ return 0;
+ }
+
+ size = sizeof(enabled);
+
+ /* Get variable contents into buffer */
+ status = efi.get_variable(efi_sev_live_migration_enabled,
+ &efi_variable_guid, NULL, &size, &enabled);
+
+ if (status == EFI_NOT_FOUND) {
+ pr_info("%s : EFI live migration variable not found\n", __func__);
+ return 0;
+ }
+
+ if (status != EFI_SUCCESS) {
+ pr_info("%s : EFI variable retrieval failed\n", __func__);
+ return 0;
+ }
+
+ if (enabled == 0) {
+ pr_info("%s: live migration disabled in EFI\n", __func__);
+ return 0;
+ }
+
+ pr_info("%s : live migration enabled in EFI\n", __func__);
+ wrmsrl(MSR_KVM_MIGRATION_CONTROL, KVM_MIGRATION_READY);
+
+ return 1;
+}
+
+late_initcall(setup_efi_kvm_sev_migration);
+
/*
* Set the IPI entry points
*/
return 0; /* So we don't blow up on old processors */
if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
- return hypervisor_cpuid_base("KVMKVMKVM\0\0\0", 0);
+ return hypervisor_cpuid_base(KVM_SIGNATURE, 0);
return 0;
}
return kvm_para_has_feature(KVM_FEATURE_MSI_EXT_DEST_ID);
}
+static void kvm_sev_hc_page_enc_status(unsigned long pfn, int npages, bool enc)
+{
+ kvm_sev_hypercall3(KVM_HC_MAP_GPA_RANGE, pfn << PAGE_SHIFT, npages,
+ KVM_MAP_GPA_RANGE_ENC_STAT(enc) | KVM_MAP_GPA_RANGE_PAGE_SZ_4K);
+}
+
static void __init kvm_init_platform(void)
{
+ if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+ kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL)) {
+ unsigned long nr_pages;
+ int i;
+
+ pv_ops.mmu.notify_page_enc_status_changed =
+ kvm_sev_hc_page_enc_status;
+
+ /*
+ * Reset the host's shared pages list related to kernel
+ * specific page encryption status settings before we load a
+ * new kernel by kexec. Reset the page encryption status
+ * during early boot intead of just before kexec to avoid SMP
+ * races during kvm_pv_guest_cpu_reboot().
+ * NOTE: We cannot reset the complete shared pages list
+ * here as we need to retain the UEFI/OVMF firmware
+ * specific settings.
+ */
+
+ for (i = 0; i < e820_table->nr_entries; i++) {
+ struct e820_entry *entry = &e820_table->entries[i];
+
+ if (entry->type != E820_TYPE_RAM)
+ continue;
+
+ nr_pages = DIV_ROUND_UP(entry->size, PAGE_SIZE);
+
+ kvm_sev_hypercall3(KVM_HC_MAP_GPA_RANGE, entry->addr,
+ nr_pages,
+ KVM_MAP_GPA_RANGE_ENCRYPTED | KVM_MAP_GPA_RANGE_PAGE_SZ_4K);
+ }
+
+ /*
+ * Ensure that _bss_decrypted section is marked as decrypted in the
+ * shared pages list.
+ */
+ nr_pages = DIV_ROUND_UP(__end_bss_decrypted - __start_bss_decrypted,
+ PAGE_SIZE);
+ early_set_mem_enc_dec_hypercall((unsigned long)__start_bss_decrypted,
+ nr_pages, 0);
+
+ /*
+ * If not booted using EFI, enable Live migration support.
+ */
+ if (!efi_enabled(EFI_BOOT))
+ wrmsrl(MSR_KVM_MIGRATION_CONTROL,
+ KVM_MIGRATION_READY);
+ }
kvmclock_init();
x86_platform.apic_post_init = kvm_apic_init;
}
".type _paravirt_nop, @function\n\t"
".popsection");
+/* stub always returning 0. */
+asm (".pushsection .entry.text, \"ax\"\n"
+ ".global paravirt_ret0\n"
+ "paravirt_ret0:\n\t"
+ "xor %" _ASM_AX ", %" _ASM_AX ";\n\t"
+ "ret\n\t"
+ ".size paravirt_ret0, . - paravirt_ret0\n\t"
+ ".type paravirt_ret0, @function\n\t"
+ ".popsection");
+
+
void __init default_banner(void)
{
printk(KERN_INFO "Booting paravirtualized kernel on %s\n",
}
/* Undefined instruction for dealing with missing ops pointers. */
-static void paravirt_BUG(void)
+noinstr void paravirt_BUG(void)
{
BUG();
}
(void (*)(struct mmu_gather *, void *))tlb_remove_page,
.mmu.exit_mmap = paravirt_nop,
+ .mmu.notify_page_enc_status_changed = paravirt_nop,
#ifdef CONFIG_PARAVIRT_XXL
.mmu.read_cr2 = __PV_IS_CALLEE_SAVE(pv_native_read_cr2),
*/
static bool match_id(struct pci_dev *pdev, unsigned short vendor, unsigned short device)
{
- struct pci_driver *drv = pdev->driver;
+ struct pci_driver *drv = to_pci_driver(pdev->dev.driver);
const struct pci_device_id *id;
if (pdev->vendor == vendor && pdev->device == device)
struct unwind_state state;
unsigned long addr = 0;
+ if (!try_get_task_stack(p))
+ return 0;
+
for (unwind_start(&state, p, NULL, NULL); !unwind_done(&state);
unwind_next_frame(&state)) {
addr = unwind_get_return_address(&state);
break;
}
+ put_task_stack(p);
+
return addr;
}
relocate_initrd();
- memblock_free(ramdisk_image, ramdisk_end - ramdisk_image);
+ memblock_phys_free(ramdisk_image, ramdisk_end - ramdisk_image);
}
#else
}
if (crash_base >= (1ULL << 32) && reserve_crashkernel_low()) {
- memblock_free(crash_base, crash_size);
+ memblock_phys_free(crash_base, crash_size);
return;
}
return 0;
}
+static char *prepare_command_line(void)
+{
+#ifdef CONFIG_CMDLINE_BOOL
+#ifdef CONFIG_CMDLINE_OVERRIDE
+ strlcpy(boot_command_line, builtin_cmdline, COMMAND_LINE_SIZE);
+#else
+ if (builtin_cmdline[0]) {
+ /* append boot loader cmdline to builtin */
+ strlcat(builtin_cmdline, " ", COMMAND_LINE_SIZE);
+ strlcat(builtin_cmdline, boot_command_line, COMMAND_LINE_SIZE);
+ strlcpy(boot_command_line, builtin_cmdline, COMMAND_LINE_SIZE);
+ }
+#endif
+#endif
+
+ strlcpy(command_line, boot_command_line, COMMAND_LINE_SIZE);
+
+ parse_early_param();
+
+ return command_line;
+}
+
/*
* Determine if we were loaded by an EFI loader. If so, then we have also been
* passed the efi memmap, systab, etc., so we should use these data structures
x86_init.oem.arch_setup();
/*
+ * x86_configure_nx() is called before parse_early_param() (called by
+ * prepare_command_line()) to detect whether hardware doesn't support
+ * NX (so that the early EHCI debug console setup can safely call
+ * set_fixmap()). It may then be called again from within noexec_setup()
+ * during parsing early parameters to honor the respective command line
+ * option.
+ */
+ x86_configure_nx();
+
+ /*
+ * This parses early params and it needs to run before
+ * early_reserve_memory() because latter relies on such settings
+ * supplied as early params.
+ */
+ *cmdline_p = prepare_command_line();
+
+ /*
* Do some memory reservations *before* memory is added to memblock, so
* memblock allocations won't overwrite it.
*
bss_resource.start = __pa_symbol(__bss_start);
bss_resource.end = __pa_symbol(__bss_stop)-1;
-#ifdef CONFIG_CMDLINE_BOOL
-#ifdef CONFIG_CMDLINE_OVERRIDE
- strlcpy(boot_command_line, builtin_cmdline, COMMAND_LINE_SIZE);
-#else
- if (builtin_cmdline[0]) {
- /* append boot loader cmdline to builtin */
- strlcat(builtin_cmdline, " ", COMMAND_LINE_SIZE);
- strlcat(builtin_cmdline, boot_command_line, COMMAND_LINE_SIZE);
- strlcpy(boot_command_line, builtin_cmdline, COMMAND_LINE_SIZE);
- }
-#endif
-#endif
-
- strlcpy(command_line, boot_command_line, COMMAND_LINE_SIZE);
- *cmdline_p = command_line;
-
- /*
- * x86_configure_nx() is called before parse_early_param() to detect
- * whether hardware doesn't support NX (so that the early EHCI debug
- * console setup can safely call set_fixmap()). It may then be called
- * again from within noexec_setup() during parsing early parameters
- * to honor the respective command line option.
- */
- x86_configure_nx();
-
- parse_early_param();
-
#ifdef CONFIG_MEMORY_HOTPLUG
/*
* Memory used by the kernel cannot be hot-removed because Linux
static void __init pcpu_fc_free(void *ptr, size_t size)
{
- memblock_free_ptr(ptr, size);
+ memblock_free(ptr, size);
}
static int __init pcpu_cpu_distance(unsigned int from, unsigned int to)
cpu0_logical_apicid = GET_APIC_LOGICAL_ID(apic_read(APIC_LDR));
}
-/*
- * Prepare for SMP bootup.
- * @max_cpus: configured maximum number of CPUs, It is a legacy parameter
- * for common interface support.
- */
-void __init native_smp_prepare_cpus(unsigned int max_cpus)
+void __init smp_prepare_cpus_common(void)
{
unsigned int i;
set_sched_topology(x86_topology);
set_cpu_sibling_map(0);
+}
+
+/*
+ * Prepare for SMP bootup.
+ * @max_cpus: configured maximum number of CPUs, It is a legacy parameter
+ * for common interface support.
+ */
+void __init native_smp_prepare_cpus(unsigned int max_cpus)
+{
+ smp_prepare_cpus_common();
+
init_freq_invariance(false, false);
smp_sanity_check();
text_poke_bp(insn, code, size, emulate);
}
-static void __static_call_validate(void *insn, bool tail)
+static void __static_call_validate(void *insn, bool tail, bool tramp)
{
u8 opcode = *(u8 *)insn;
+ if (tramp && memcmp(insn+5, "SCT", 3)) {
+ pr_err("trampoline signature fail");
+ BUG();
+ }
+
if (tail) {
if (opcode == JMP32_INSN_OPCODE ||
opcode == RET_INSN_OPCODE)
/*
* If we ever trigger this, our text is corrupt, we'll probably not live long.
*/
- WARN_ONCE(1, "unexpected static_call insn opcode 0x%x at %pS\n", opcode, insn);
+ pr_err("unexpected static_call insn opcode 0x%x at %pS\n", opcode, insn);
+ BUG();
}
static inline enum insn_type __sc_insn(bool null, bool tail)
mutex_lock(&text_mutex);
if (tramp) {
- __static_call_validate(tramp, true);
+ __static_call_validate(tramp, true, true);
__static_call_transform(tramp, __sc_insn(!func, true), func);
}
if (IS_ENABLED(CONFIG_HAVE_STATIC_CALL_INLINE) && site) {
- __static_call_validate(site, tail);
+ __static_call_validate(site, tail, false);
__static_call_transform(site, __sc_insn(!func, tail), func);
}
}
/* vmlinux .init slow lookup: */
- if (init_kernel_text(ip))
+ if (is_kernel_inittext(ip))
return __orc_find(__start_orc_unwind_ip, __start_orc_unwind,
__stop_orc_unwind_ip - __start_orc_unwind_ip, ip);
*/
local_irq_enable();
- if (!vm86 || !vm86->user_vm86) {
- pr_alert("no user_vm86: BAD\n");
- do_exit(SIGSEGV);
- }
+ BUG_ON(!vm86);
+
set_flags(regs->pt.flags, VEFLAGS, X86_EFLAGS_VIF | vm86->veflags_mask);
user = vm86->user_vm86;
user_access_end();
+exit_vm86:
preempt_disable();
tsk->thread.sp0 = vm86->saved_sp0;
tsk->thread.sysenter_cs = __KERNEL_CS;
user_access_end();
Efault:
pr_alert("could not access userspace vm86 info\n");
- do_exit(SIGSEGV);
+ force_exit_sig(SIGSEGV);
+ goto exit_vm86;
}
static int do_vm86_irq_handling(int subfunction, int irqnumber);
return 0;
}
-void kvm_update_pv_runtime(struct kvm_vcpu *vcpu)
+static void kvm_update_kvm_cpuid_base(struct kvm_vcpu *vcpu)
{
- struct kvm_cpuid_entry2 *best;
+ u32 function;
+ struct kvm_cpuid_entry2 *entry;
+
+ vcpu->arch.kvm_cpuid_base = 0;
+
+ for_each_possible_hypervisor_cpuid_base(function) {
+ entry = kvm_find_cpuid_entry(vcpu, function, 0);
- best = kvm_find_cpuid_entry(vcpu, KVM_CPUID_FEATURES, 0);
+ if (entry) {
+ u32 signature[3];
+
+ signature[0] = entry->ebx;
+ signature[1] = entry->ecx;
+ signature[2] = entry->edx;
+
+ BUILD_BUG_ON(sizeof(signature) > sizeof(KVM_SIGNATURE));
+ if (!memcmp(signature, KVM_SIGNATURE, sizeof(signature))) {
+ vcpu->arch.kvm_cpuid_base = function;
+ break;
+ }
+ }
+ }
+}
+
+static struct kvm_cpuid_entry2 *kvm_find_kvm_cpuid_features(struct kvm_vcpu *vcpu)
+{
+ u32 base = vcpu->arch.kvm_cpuid_base;
+
+ if (!base)
+ return NULL;
+
+ return kvm_find_cpuid_entry(vcpu, base | KVM_CPUID_FEATURES, 0);
+}
+
+void kvm_update_pv_runtime(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpuid_entry2 *best = kvm_find_kvm_cpuid_features(vcpu);
/*
* save the feature bitmap to avoid cpuid lookup for every PV
cpuid_entry_has(best, X86_FEATURE_XSAVEC)))
best->ebx = xstate_required_size(vcpu->arch.xcr0, true);
- best = kvm_find_cpuid_entry(vcpu, KVM_CPUID_FEATURES, 0);
+ best = kvm_find_kvm_cpuid_features(vcpu);
if (kvm_hlt_in_guest(vcpu->kvm) && best &&
(best->eax & (1 << KVM_FEATURE_PV_UNHALT)))
best->eax &= ~(1 << KVM_FEATURE_PV_UNHALT);
return rsvd_bits(cpuid_maxphyaddr(vcpu), 63);
}
+static int kvm_set_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid_entry2 *e2,
+ int nent)
+{
+ int r;
+
+ r = kvm_check_cpuid(e2, nent);
+ if (r)
+ return r;
+
+ kvfree(vcpu->arch.cpuid_entries);
+ vcpu->arch.cpuid_entries = e2;
+ vcpu->arch.cpuid_nent = nent;
+
+ kvm_update_kvm_cpuid_base(vcpu);
+ kvm_update_cpuid_runtime(vcpu);
+ kvm_vcpu_after_set_cpuid(vcpu);
+
+ return 0;
+}
+
/* when an old userspace process fills a new kernel module */
int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu,
struct kvm_cpuid *cpuid,
e2[i].padding[2] = 0;
}
- r = kvm_check_cpuid(e2, cpuid->nent);
- if (r) {
+ r = kvm_set_cpuid(vcpu, e2, cpuid->nent);
+ if (r)
kvfree(e2);
- goto out_free_cpuid;
- }
-
- kvfree(vcpu->arch.cpuid_entries);
- vcpu->arch.cpuid_entries = e2;
- vcpu->arch.cpuid_nent = cpuid->nent;
-
- kvm_update_cpuid_runtime(vcpu);
- kvm_vcpu_after_set_cpuid(vcpu);
out_free_cpuid:
kvfree(e);
return PTR_ERR(e2);
}
- r = kvm_check_cpuid(e2, cpuid->nent);
- if (r) {
+ r = kvm_set_cpuid(vcpu, e2, cpuid->nent);
+ if (r)
kvfree(e2);
- return r;
- }
- kvfree(vcpu->arch.cpuid_entries);
- vcpu->arch.cpuid_entries = e2;
- vcpu->arch.cpuid_nent = cpuid->nent;
-
- kvm_update_cpuid_runtime(vcpu);
- kvm_vcpu_after_set_cpuid(vcpu);
-
- return 0;
+ return r;
}
int kvm_vcpu_ioctl_get_cpuid2(struct kvm_vcpu *vcpu,
}
break;
case KVM_CPUID_SIGNATURE: {
- static const char signature[12] = "KVMKVMKVM\0\0";
- const u32 *sigptr = (const u32 *)signature;
+ const u32 *sigptr = (const u32 *)KVM_SIGNATURE;
entry->eax = KVM_CPUID_FEATURES;
entry->ebx = sigptr[0];
entry->ecx = sigptr[1];
if (!(data & HV_X64_MSR_VP_ASSIST_PAGE_ENABLE)) {
hv_vcpu->hv_vapic = data;
- if (kvm_lapic_enable_pv_eoi(vcpu, 0, 0))
+ if (kvm_lapic_set_pv_eoi(vcpu, 0, 0))
return 1;
break;
}
return 1;
hv_vcpu->hv_vapic = data;
kvm_vcpu_mark_page_dirty(vcpu, gfn);
- if (kvm_lapic_enable_pv_eoi(vcpu,
+ if (kvm_lapic_set_pv_eoi(vcpu,
gfn_to_gpa(gfn) | KVM_MSR_ENABLED,
sizeof(struct hv_vp_assist_page)))
return 1;
{
bool longmode;
- longmode = is_64_bit_mode(vcpu);
+ longmode = is_64_bit_hypercall(vcpu);
if (longmode)
kvm_rax_write(vcpu, result);
else {
}
#ifdef CONFIG_X86_64
- if (is_64_bit_mode(vcpu)) {
+ if (is_64_bit_hypercall(vcpu)) {
hc.param = kvm_rcx_read(vcpu);
hc.ingpa = kvm_rdx_read(vcpu);
hc.outgpa = kvm_r8_read(vcpu);
return 0;
}
-int kvm_lapic_enable_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)
+int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len)
{
u64 addr = data & ~KVM_MSR_ENABLED;
struct gfn_to_hva_cache *ghc = &vcpu->arch.pv_eoi.data;
unsigned long new_len;
+ int ret;
if (!IS_ALIGNED(addr, 4))
return 1;
- vcpu->arch.pv_eoi.msr_val = data;
- if (!pv_eoi_enabled(vcpu))
- return 0;
+ if (data & KVM_MSR_ENABLED) {
+ if (addr == ghc->gpa && len <= ghc->len)
+ new_len = ghc->len;
+ else
+ new_len = len;
- if (addr == ghc->gpa && len <= ghc->len)
- new_len = ghc->len;
- else
- new_len = len;
+ ret = kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, addr, new_len);
+ if (ret)
+ return ret;
+ }
+
+ vcpu->arch.pv_eoi.msr_val = data;
- return kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, addr, new_len);
+ return 0;
}
int kvm_apic_accept_events(struct kvm_vcpu *vcpu)
int kvm_hv_vapic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data);
int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data);
-int kvm_lapic_enable_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);
+int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long len);
void kvm_lapic_exit(void);
#define VEC_POS(v) ((v) & (32 - 1))
new_spte |= PT_WRITABLE_MASK;
/*
- * Do not fix write-permission on the large spte. Since
- * we only dirty the first page into the dirty-bitmap in
+ * Do not fix write-permission on the large spte when
+ * dirty logging is enabled. Since we only dirty the
+ * first page into the dirty-bitmap in
* fast_pf_fix_direct_spte(), other pages are missed
* if its slot has dirty logging enabled.
*
* Instead, we let the slow page fault path create a
* normal spte to fix the access.
- *
- * See the comments in kvm_arch_commit_memory_region().
*/
- if (sp->role.level > PG_LEVEL_4K)
+ if (sp->role.level > PG_LEVEL_4K &&
+ kvm_slot_dirty_track_enabled(fault->slot))
break;
}
/* PKEY and LA57 are active iff long mode is active. */
ext.cr4_pke = ____is_efer_lma(regs) && ____is_cr4_pke(regs);
ext.cr4_la57 = ____is_efer_lma(regs) && ____is_cr4_la57(regs);
+ ext.efer_lma = ____is_efer_lma(regs);
}
ext.valid = 1;
struct kvm_page_fault *fault,
struct tdp_iter *iter)
{
- struct kvm_mmu_page *sp = sptep_to_sp(iter->sptep);
+ struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(iter->sptep));
u64 new_spte;
int ret = RET_PF_FIXED;
bool wrprot = false;
}
/* check if idx is a valid index to access PMU */
-int kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
+bool kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
{
return kvm_x86_ops.pmu_ops->is_valid_rdpmc_ecx(vcpu, idx);
}
struct kvm_pmc *(*rdpmc_ecx_to_pmc)(struct kvm_vcpu *vcpu,
unsigned int idx, u64 *mask);
struct kvm_pmc *(*msr_idx_to_pmc)(struct kvm_vcpu *vcpu, u32 msr);
- int (*is_valid_rdpmc_ecx)(struct kvm_vcpu *vcpu, unsigned int idx);
+ bool (*is_valid_rdpmc_ecx)(struct kvm_vcpu *vcpu, unsigned int idx);
bool (*is_valid_msr)(struct kvm_vcpu *vcpu, u32 msr);
int (*get_msr)(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
int (*set_msr)(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu);
void kvm_pmu_handle_event(struct kvm_vcpu *vcpu);
int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data);
-int kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx);
+bool kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx);
bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr);
int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info);
BIT(APICV_INHIBIT_REASON_NESTED) |
BIT(APICV_INHIBIT_REASON_IRQWIN) |
BIT(APICV_INHIBIT_REASON_PIT_REINJ) |
- BIT(APICV_INHIBIT_REASON_X2APIC);
+ BIT(APICV_INHIBIT_REASON_X2APIC) |
+ BIT(APICV_INHIBIT_REASON_BLOCKIRQ);
return supported & BIT(bit);
}
return get_gp_pmc_amd(pmu, base + pmc_idx, PMU_TYPE_COUNTER);
}
-/* returns 0 if idx's corresponding MSR exists; otherwise returns 1. */
-static int amd_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
+static bool amd_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
{
struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
idx &= ~(3u << 30);
- return (idx >= pmu->nr_arch_gp_counters);
+ return idx < pmu->nr_arch_gp_counters;
}
/* idx is the ECX register of RDPMC instruction */
return true;
}
+static int sev_misc_cg_try_charge(struct kvm_sev_info *sev)
+{
+ enum misc_res_type type = sev->es_active ? MISC_CG_RES_SEV_ES : MISC_CG_RES_SEV;
+ return misc_cg_try_charge(type, sev->misc_cg, 1);
+}
+
+static void sev_misc_cg_uncharge(struct kvm_sev_info *sev)
+{
+ enum misc_res_type type = sev->es_active ? MISC_CG_RES_SEV_ES : MISC_CG_RES_SEV;
+ misc_cg_uncharge(type, sev->misc_cg, 1);
+}
+
static int sev_asid_new(struct kvm_sev_info *sev)
{
int asid, min_asid, max_asid, ret;
bool retry = true;
- enum misc_res_type type;
- type = sev->es_active ? MISC_CG_RES_SEV_ES : MISC_CG_RES_SEV;
WARN_ON(sev->misc_cg);
sev->misc_cg = get_current_misc_cg();
- ret = misc_cg_try_charge(type, sev->misc_cg, 1);
+ ret = sev_misc_cg_try_charge(sev);
if (ret) {
put_misc_cg(sev->misc_cg);
sev->misc_cg = NULL;
return asid;
e_uncharge:
- misc_cg_uncharge(type, sev->misc_cg, 1);
+ sev_misc_cg_uncharge(sev);
put_misc_cg(sev->misc_cg);
sev->misc_cg = NULL;
return ret;
{
struct svm_cpu_data *sd;
int cpu;
- enum misc_res_type type;
mutex_lock(&sev_bitmap_lock);
mutex_unlock(&sev_bitmap_lock);
- type = sev->es_active ? MISC_CG_RES_SEV_ES : MISC_CG_RES_SEV;
- misc_cg_uncharge(type, sev->misc_cg, 1);
+ sev_misc_cg_uncharge(sev);
put_misc_cg(sev->misc_cg);
sev->misc_cg = NULL;
}
static int sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp)
{
struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
- bool es_active = argp->id == KVM_SEV_ES_INIT;
int asid, ret;
if (kvm->created_vcpus)
if (unlikely(sev->active))
return ret;
- sev->es_active = es_active;
+ sev->active = true;
+ sev->es_active = argp->id == KVM_SEV_ES_INIT;
asid = sev_asid_new(sev);
if (asid < 0)
goto e_no_asid;
if (ret)
goto e_free;
- sev->active = true;
- sev->asid = asid;
INIT_LIST_HEAD(&sev->regions_list);
return 0;
sev->asid = 0;
e_no_asid:
sev->es_active = false;
+ sev->active = false;
return ret;
}
* traditional VMSA as it has been built so far (in prep
* for LAUNCH_UPDATE_VMSA) to be the initial SEV-ES state.
*/
- memcpy(svm->vmsa, save, sizeof(*save));
+ memcpy(svm->sev_es.vmsa, save, sizeof(*save));
return 0;
}
* the VMSA memory content (i.e it will write the same memory region
* with the guest's key), so invalidate it first.
*/
- clflush_cache_range(svm->vmsa, PAGE_SIZE);
+ clflush_cache_range(svm->sev_es.vmsa, PAGE_SIZE);
vmsa.reserved = 0;
vmsa.handle = to_kvm_svm(kvm)->sev_info.handle;
- vmsa.address = __sme_pa(svm->vmsa);
+ vmsa.address = __sme_pa(svm->sev_es.vmsa);
vmsa.len = PAGE_SIZE;
ret = sev_issue_cmd(kvm, SEV_CMD_LAUNCH_UPDATE_VMSA, &vmsa, error);
if (ret)
return sev_issue_cmd(kvm, SEV_CMD_RECEIVE_FINISH, &data, &argp->error);
}
-static bool cmd_allowed_from_miror(u32 cmd_id)
+static bool is_cmd_allowed_from_mirror(u32 cmd_id)
{
/*
* Allow mirrors VM to call KVM_SEV_LAUNCH_UPDATE_VMSA to enable SEV-ES
return false;
}
+static int sev_lock_for_migration(struct kvm *kvm)
+{
+ struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+
+ /*
+ * Bail if this VM is already involved in a migration to avoid deadlock
+ * between two VMs trying to migrate to/from each other.
+ */
+ if (atomic_cmpxchg_acquire(&sev->migration_in_progress, 0, 1))
+ return -EBUSY;
+
+ mutex_lock(&kvm->lock);
+
+ return 0;
+}
+
+static void sev_unlock_after_migration(struct kvm *kvm)
+{
+ struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
+
+ mutex_unlock(&kvm->lock);
+ atomic_set_release(&sev->migration_in_progress, 0);
+}
+
+
+static int sev_lock_vcpus_for_migration(struct kvm *kvm)
+{
+ struct kvm_vcpu *vcpu;
+ int i, j;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (mutex_lock_killable(&vcpu->mutex))
+ goto out_unlock;
+ }
+
+ return 0;
+
+out_unlock:
+ kvm_for_each_vcpu(j, vcpu, kvm) {
+ if (i == j)
+ break;
+
+ mutex_unlock(&vcpu->mutex);
+ }
+ return -EINTR;
+}
+
+static void sev_unlock_vcpus_for_migration(struct kvm *kvm)
+{
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ mutex_unlock(&vcpu->mutex);
+ }
+}
+
+static void sev_migrate_from(struct kvm_sev_info *dst,
+ struct kvm_sev_info *src)
+{
+ dst->active = true;
+ dst->asid = src->asid;
+ dst->handle = src->handle;
+ dst->pages_locked = src->pages_locked;
+
+ src->asid = 0;
+ src->active = false;
+ src->handle = 0;
+ src->pages_locked = 0;
+
+ INIT_LIST_HEAD(&dst->regions_list);
+ list_replace_init(&src->regions_list, &dst->regions_list);
+}
+
+static int sev_es_migrate_from(struct kvm *dst, struct kvm *src)
+{
+ int i;
+ struct kvm_vcpu *dst_vcpu, *src_vcpu;
+ struct vcpu_svm *dst_svm, *src_svm;
+
+ if (atomic_read(&src->online_vcpus) != atomic_read(&dst->online_vcpus))
+ return -EINVAL;
+
+ kvm_for_each_vcpu(i, src_vcpu, src) {
+ if (!src_vcpu->arch.guest_state_protected)
+ return -EINVAL;
+ }
+
+ kvm_for_each_vcpu(i, src_vcpu, src) {
+ src_svm = to_svm(src_vcpu);
+ dst_vcpu = kvm_get_vcpu(dst, i);
+ dst_svm = to_svm(dst_vcpu);
+
+ /*
+ * Transfer VMSA and GHCB state to the destination. Nullify and
+ * clear source fields as appropriate, the state now belongs to
+ * the destination.
+ */
+ memcpy(&dst_svm->sev_es, &src_svm->sev_es, sizeof(src_svm->sev_es));
+ dst_svm->vmcb->control.ghcb_gpa = src_svm->vmcb->control.ghcb_gpa;
+ dst_svm->vmcb->control.vmsa_pa = src_svm->vmcb->control.vmsa_pa;
+ dst_vcpu->arch.guest_state_protected = true;
+
+ memset(&src_svm->sev_es, 0, sizeof(src_svm->sev_es));
+ src_svm->vmcb->control.ghcb_gpa = INVALID_PAGE;
+ src_svm->vmcb->control.vmsa_pa = INVALID_PAGE;
+ src_vcpu->arch.guest_state_protected = false;
+ }
+ to_kvm_svm(src)->sev_info.es_active = false;
+ to_kvm_svm(dst)->sev_info.es_active = true;
+
+ return 0;
+}
+
+int svm_vm_migrate_from(struct kvm *kvm, unsigned int source_fd)
+{
+ struct kvm_sev_info *dst_sev = &to_kvm_svm(kvm)->sev_info;
+ struct kvm_sev_info *src_sev, *cg_cleanup_sev;
+ struct file *source_kvm_file;
+ struct kvm *source_kvm;
+ bool charged = false;
+ int ret;
+
+ ret = sev_lock_for_migration(kvm);
+ if (ret)
+ return ret;
+
+ if (sev_guest(kvm)) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
+ source_kvm_file = fget(source_fd);
+ if (!file_is_kvm(source_kvm_file)) {
+ ret = -EBADF;
+ goto out_fput;
+ }
+
+ source_kvm = source_kvm_file->private_data;
+ ret = sev_lock_for_migration(source_kvm);
+ if (ret)
+ goto out_fput;
+
+ if (!sev_guest(source_kvm)) {
+ ret = -EINVAL;
+ goto out_source;
+ }
+
+ src_sev = &to_kvm_svm(source_kvm)->sev_info;
+ dst_sev->misc_cg = get_current_misc_cg();
+ cg_cleanup_sev = dst_sev;
+ if (dst_sev->misc_cg != src_sev->misc_cg) {
+ ret = sev_misc_cg_try_charge(dst_sev);
+ if (ret)
+ goto out_dst_cgroup;
+ charged = true;
+ }
+
+ ret = sev_lock_vcpus_for_migration(kvm);
+ if (ret)
+ goto out_dst_cgroup;
+ ret = sev_lock_vcpus_for_migration(source_kvm);
+ if (ret)
+ goto out_dst_vcpu;
+
+ if (sev_es_guest(source_kvm)) {
+ ret = sev_es_migrate_from(kvm, source_kvm);
+ if (ret)
+ goto out_source_vcpu;
+ }
+ sev_migrate_from(dst_sev, src_sev);
+ kvm_vm_dead(source_kvm);
+ cg_cleanup_sev = src_sev;
+ ret = 0;
+
+out_source_vcpu:
+ sev_unlock_vcpus_for_migration(source_kvm);
+out_dst_vcpu:
+ sev_unlock_vcpus_for_migration(kvm);
+out_dst_cgroup:
+ /* Operates on the source on success, on the destination on failure. */
+ if (charged)
+ sev_misc_cg_uncharge(cg_cleanup_sev);
+ put_misc_cg(cg_cleanup_sev->misc_cg);
+ cg_cleanup_sev->misc_cg = NULL;
+out_source:
+ sev_unlock_after_migration(source_kvm);
+out_fput:
+ if (source_kvm_file)
+ fput(source_kvm_file);
+out_unlock:
+ sev_unlock_after_migration(kvm);
+ return ret;
+}
+
int svm_mem_enc_op(struct kvm *kvm, void __user *argp)
{
struct kvm_sev_cmd sev_cmd;
/* Only the enc_context_owner handles some memory enc operations. */
if (is_mirroring_enc_context(kvm) &&
- !cmd_allowed_from_miror(sev_cmd.id)) {
+ !is_cmd_allowed_from_mirror(sev_cmd.id)) {
r = -EINVAL;
goto out;
}
mutex_unlock(&source_kvm->lock);
mutex_lock(&kvm->lock);
- if (sev_guest(kvm)) {
+ /*
+ * Disallow out-of-band SEV/SEV-ES init if the target is already an
+ * SEV guest, or if vCPUs have been created. KVM relies on vCPUs being
+ * created after SEV/SEV-ES initialization, e.g. to init intercepts.
+ */
+ if (sev_guest(kvm) || kvm->created_vcpus) {
ret = -EINVAL;
goto e_mirror_unlock;
}
svm = to_svm(vcpu);
if (vcpu->arch.guest_state_protected)
- sev_flush_guest_memory(svm, svm->vmsa, PAGE_SIZE);
- __free_page(virt_to_page(svm->vmsa));
+ sev_flush_guest_memory(svm, svm->sev_es.vmsa, PAGE_SIZE);
+ __free_page(virt_to_page(svm->sev_es.vmsa));
- if (svm->ghcb_sa_free)
- kfree(svm->ghcb_sa);
+ if (svm->sev_es.ghcb_sa_free)
+ kfree(svm->sev_es.ghcb_sa);
}
static void dump_ghcb(struct vcpu_svm *svm)
{
- struct ghcb *ghcb = svm->ghcb;
+ struct ghcb *ghcb = svm->sev_es.ghcb;
unsigned int nbits;
/* Re-use the dump_invalid_vmcb module parameter */
static void sev_es_sync_to_ghcb(struct vcpu_svm *svm)
{
struct kvm_vcpu *vcpu = &svm->vcpu;
- struct ghcb *ghcb = svm->ghcb;
+ struct ghcb *ghcb = svm->sev_es.ghcb;
/*
* The GHCB protocol so far allows for the following data
{
struct vmcb_control_area *control = &svm->vmcb->control;
struct kvm_vcpu *vcpu = &svm->vcpu;
- struct ghcb *ghcb = svm->ghcb;
+ struct ghcb *ghcb = svm->sev_es.ghcb;
u64 exit_code;
/*
struct ghcb *ghcb;
u64 exit_code = 0;
- ghcb = svm->ghcb;
+ ghcb = svm->sev_es.ghcb;
/* Only GHCB Usage code 0 is supported */
if (ghcb->ghcb_usage)
void sev_es_unmap_ghcb(struct vcpu_svm *svm)
{
- if (!svm->ghcb)
+ if (!svm->sev_es.ghcb)
return;
- if (svm->ghcb_sa_free) {
+ if (svm->sev_es.ghcb_sa_free) {
/*
* The scratch area lives outside the GHCB, so there is a
* buffer that, depending on the operation performed, may
* need to be synced, then freed.
*/
- if (svm->ghcb_sa_sync) {
+ if (svm->sev_es.ghcb_sa_sync) {
kvm_write_guest(svm->vcpu.kvm,
- ghcb_get_sw_scratch(svm->ghcb),
- svm->ghcb_sa, svm->ghcb_sa_len);
- svm->ghcb_sa_sync = false;
+ ghcb_get_sw_scratch(svm->sev_es.ghcb),
+ svm->sev_es.ghcb_sa,
+ svm->sev_es.ghcb_sa_len);
+ svm->sev_es.ghcb_sa_sync = false;
}
- kfree(svm->ghcb_sa);
- svm->ghcb_sa = NULL;
- svm->ghcb_sa_free = false;
+ kfree(svm->sev_es.ghcb_sa);
+ svm->sev_es.ghcb_sa = NULL;
+ svm->sev_es.ghcb_sa_free = false;
}
- trace_kvm_vmgexit_exit(svm->vcpu.vcpu_id, svm->ghcb);
+ trace_kvm_vmgexit_exit(svm->vcpu.vcpu_id, svm->sev_es.ghcb);
sev_es_sync_to_ghcb(svm);
- kvm_vcpu_unmap(&svm->vcpu, &svm->ghcb_map, true);
- svm->ghcb = NULL;
+ kvm_vcpu_unmap(&svm->vcpu, &svm->sev_es.ghcb_map, true);
+ svm->sev_es.ghcb = NULL;
}
void pre_sev_run(struct vcpu_svm *svm, int cpu)
static bool setup_vmgexit_scratch(struct vcpu_svm *svm, bool sync, u64 len)
{
struct vmcb_control_area *control = &svm->vmcb->control;
- struct ghcb *ghcb = svm->ghcb;
+ struct ghcb *ghcb = svm->sev_es.ghcb;
u64 ghcb_scratch_beg, ghcb_scratch_end;
u64 scratch_gpa_beg, scratch_gpa_end;
void *scratch_va;
return false;
}
- scratch_va = (void *)svm->ghcb;
+ scratch_va = (void *)svm->sev_es.ghcb;
scratch_va += (scratch_gpa_beg - control->ghcb_gpa);
} else {
/*
* the vCPU next time (i.e. a read was requested so the data
* must be written back to the guest memory).
*/
- svm->ghcb_sa_sync = sync;
- svm->ghcb_sa_free = true;
+ svm->sev_es.ghcb_sa_sync = sync;
+ svm->sev_es.ghcb_sa_free = true;
}
- svm->ghcb_sa = scratch_va;
- svm->ghcb_sa_len = len;
+ svm->sev_es.ghcb_sa = scratch_va;
+ svm->sev_es.ghcb_sa_len = len;
return true;
}
return -EINVAL;
}
- if (kvm_vcpu_map(vcpu, ghcb_gpa >> PAGE_SHIFT, &svm->ghcb_map)) {
+ if (kvm_vcpu_map(vcpu, ghcb_gpa >> PAGE_SHIFT, &svm->sev_es.ghcb_map)) {
/* Unable to map GHCB from guest */
vcpu_unimpl(vcpu, "vmgexit: error mapping GHCB [%#llx] from guest\n",
ghcb_gpa);
return -EINVAL;
}
- svm->ghcb = svm->ghcb_map.hva;
- ghcb = svm->ghcb_map.hva;
+ svm->sev_es.ghcb = svm->sev_es.ghcb_map.hva;
+ ghcb = svm->sev_es.ghcb_map.hva;
trace_kvm_vmgexit_enter(vcpu->vcpu_id, ghcb);
ret = kvm_sev_es_mmio_read(vcpu,
control->exit_info_1,
control->exit_info_2,
- svm->ghcb_sa);
+ svm->sev_es.ghcb_sa);
break;
case SVM_VMGEXIT_MMIO_WRITE:
if (!setup_vmgexit_scratch(svm, false, control->exit_info_2))
ret = kvm_sev_es_mmio_write(vcpu,
control->exit_info_1,
control->exit_info_2,
- svm->ghcb_sa);
+ svm->sev_es.ghcb_sa);
break;
case SVM_VMGEXIT_NMI_COMPLETE:
ret = svm_invoke_exit_handler(vcpu, SVM_EXIT_IRET);
if (!setup_vmgexit_scratch(svm, in, bytes))
return -EINVAL;
- return kvm_sev_es_string_io(&svm->vcpu, size, port, svm->ghcb_sa, count, in);
+ return kvm_sev_es_string_io(&svm->vcpu, size, port, svm->sev_es.ghcb_sa,
+ count, in);
}
void sev_es_init_vmcb(struct vcpu_svm *svm)
* VMCB page. Do not include the encryption mask on the VMSA physical
* address since hardware will access it using the guest key.
*/
- svm->vmcb->control.vmsa_pa = __pa(svm->vmsa);
+ svm->vmcb->control.vmsa_pa = __pa(svm->sev_es.vmsa);
/* Can't intercept CR register access, HV can't modify CR registers */
svm_clr_intercept(svm, INTERCEPT_CR0_READ);
struct vcpu_svm *svm = to_svm(vcpu);
/* First SIPI: Use the values as initially set by the VMM */
- if (!svm->received_first_sipi) {
- svm->received_first_sipi = true;
+ if (!svm->sev_es.received_first_sipi) {
+ svm->sev_es.received_first_sipi = true;
return;
}
* the guest will set the CS and RIP. Set SW_EXIT_INFO_2 to a
* non-zero value.
*/
- if (!svm->ghcb)
+ if (!svm->sev_es.ghcb)
return;
- ghcb_set_sw_exit_info_2(svm->ghcb, 1);
+ ghcb_set_sw_exit_info_2(svm->sev_es.ghcb, 1);
}
svm_switch_vmcb(svm, &svm->vmcb01);
if (vmsa_page)
- svm->vmsa = page_address(vmsa_page);
+ svm->sev_es.vmsa = page_address(vmsa_page);
svm->guest_state_loaded = false;
static int svm_complete_emulated_msr(struct kvm_vcpu *vcpu, int err)
{
struct vcpu_svm *svm = to_svm(vcpu);
- if (!err || !sev_es_guest(vcpu->kvm) || WARN_ON_ONCE(!svm->ghcb))
+ if (!err || !sev_es_guest(vcpu->kvm) || WARN_ON_ONCE(!svm->sev_es.ghcb))
return kvm_complete_insn_gp(vcpu, err);
- ghcb_set_sw_exit_info_1(svm->ghcb, 1);
- ghcb_set_sw_exit_info_2(svm->ghcb,
+ ghcb_set_sw_exit_info_1(svm->sev_es.ghcb, 1);
+ ghcb_set_sw_exit_info_2(svm->sev_es.ghcb,
X86_TRAP_GP |
SVM_EVTINJ_TYPE_EXEPT |
SVM_EVTINJ_VALID);
type = svm->vmcb->control.exit_info_2;
gva = svm->vmcb->control.exit_info_1;
- if (type > 3) {
- kvm_inject_gp(vcpu, 0);
- return 1;
- }
-
return kvm_handle_invpcid(vcpu, type, gva);
}
.mem_enc_unreg_region = svm_unregister_enc_region,
.vm_copy_enc_context_from = svm_vm_copy_asid_from,
+ .vm_move_enc_context_from = svm_vm_migrate_from,
.can_emulate_instruction = svm_can_emulate_instruction,
u64 ap_jump_table; /* SEV-ES AP Jump Table address */
struct kvm *enc_context_owner; /* Owner of copied encryption context */
struct misc_cg *misc_cg; /* For misc cgroup accounting */
+ atomic_t migration_in_progress;
};
struct kvm_svm {
bool initialized;
};
+struct vcpu_sev_es_state {
+ /* SEV-ES support */
+ struct vmcb_save_area *vmsa;
+ struct ghcb *ghcb;
+ struct kvm_host_map ghcb_map;
+ bool received_first_sipi;
+
+ /* SEV-ES scratch area support */
+ void *ghcb_sa;
+ u32 ghcb_sa_len;
+ bool ghcb_sa_sync;
+ bool ghcb_sa_free;
+};
+
struct vcpu_svm {
struct kvm_vcpu vcpu;
/* vmcb always points at current_vmcb->ptr, it's purely a shorthand. */
DECLARE_BITMAP(write, MAX_DIRECT_ACCESS_MSRS);
} shadow_msr_intercept;
- /* SEV-ES support */
- struct vmcb_save_area *vmsa;
- struct ghcb *ghcb;
- struct kvm_host_map ghcb_map;
- bool received_first_sipi;
-
- /* SEV-ES scratch area support */
- void *ghcb_sa;
- u32 ghcb_sa_len;
- bool ghcb_sa_sync;
- bool ghcb_sa_free;
+ struct vcpu_sev_es_state sev_es;
bool guest_state_loaded;
};
#ifdef CONFIG_KVM_AMD_SEV
struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
- return sev_guest(kvm) && sev->es_active;
+ return sev->es_active && !WARN_ON_ONCE(!sev->active);
#else
return false;
#endif
int svm_unregister_enc_region(struct kvm *kvm,
struct kvm_enc_region *range);
int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd);
+int svm_vm_migrate_from(struct kvm *kvm, unsigned int source_fd);
void pre_sev_run(struct vcpu_svm *svm, int cpu);
void __init sev_set_cpu_caps(void);
void __init sev_hardware_setup(void);
}
/*
- * Check if MSR is intercepted for L01 MSR bitmap.
+ * For x2APIC MSRs, ignore the vmcs01 bitmap. L1 can enable x2APIC without L1
+ * itself utilizing x2APIC. All MSRs were previously set to be intercepted,
+ * only the "disable intercept" case needs to be handled.
*/
-static bool msr_write_intercepted_l01(struct kvm_vcpu *vcpu, u32 msr)
+static void nested_vmx_disable_intercept_for_x2apic_msr(unsigned long *msr_bitmap_l1,
+ unsigned long *msr_bitmap_l0,
+ u32 msr, int type)
{
- unsigned long *msr_bitmap;
- int f = sizeof(unsigned long);
+ if (type & MSR_TYPE_R && !vmx_test_msr_bitmap_read(msr_bitmap_l1, msr))
+ vmx_clear_msr_bitmap_read(msr_bitmap_l0, msr);
- if (!cpu_has_vmx_msr_bitmap())
- return true;
-
- msr_bitmap = to_vmx(vcpu)->vmcs01.msr_bitmap;
-
- if (msr <= 0x1fff) {
- return !!test_bit(msr, msr_bitmap + 0x800 / f);
- } else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff)) {
- msr &= 0x1fff;
- return !!test_bit(msr, msr_bitmap + 0xc00 / f);
- }
-
- return true;
-}
-
-/*
- * If a msr is allowed by L0, we should check whether it is allowed by L1.
- * The corresponding bit will be cleared unless both of L0 and L1 allow it.
- */
-static void nested_vmx_disable_intercept_for_msr(unsigned long *msr_bitmap_l1,
- unsigned long *msr_bitmap_nested,
- u32 msr, int type)
-{
- int f = sizeof(unsigned long);
-
- /*
- * See Intel PRM Vol. 3, 20.6.9 (MSR-Bitmap Address). Early manuals
- * have the write-low and read-high bitmap offsets the wrong way round.
- * We can control MSRs 0x00000000-0x00001fff and 0xc0000000-0xc0001fff.
- */
- if (msr <= 0x1fff) {
- if (type & MSR_TYPE_R &&
- !test_bit(msr, msr_bitmap_l1 + 0x000 / f))
- /* read-low */
- __clear_bit(msr, msr_bitmap_nested + 0x000 / f);
-
- if (type & MSR_TYPE_W &&
- !test_bit(msr, msr_bitmap_l1 + 0x800 / f))
- /* write-low */
- __clear_bit(msr, msr_bitmap_nested + 0x800 / f);
-
- } else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff)) {
- msr &= 0x1fff;
- if (type & MSR_TYPE_R &&
- !test_bit(msr, msr_bitmap_l1 + 0x400 / f))
- /* read-high */
- __clear_bit(msr, msr_bitmap_nested + 0x400 / f);
-
- if (type & MSR_TYPE_W &&
- !test_bit(msr, msr_bitmap_l1 + 0xc00 / f))
- /* write-high */
- __clear_bit(msr, msr_bitmap_nested + 0xc00 / f);
-
- }
+ if (type & MSR_TYPE_W && !vmx_test_msr_bitmap_write(msr_bitmap_l1, msr))
+ vmx_clear_msr_bitmap_write(msr_bitmap_l0, msr);
}
static inline void enable_x2apic_msr_intercepts(unsigned long *msr_bitmap)
}
}
+#define BUILD_NVMX_MSR_INTERCEPT_HELPER(rw) \
+static inline \
+void nested_vmx_set_msr_##rw##_intercept(struct vcpu_vmx *vmx, \
+ unsigned long *msr_bitmap_l1, \
+ unsigned long *msr_bitmap_l0, u32 msr) \
+{ \
+ if (vmx_test_msr_bitmap_##rw(vmx->vmcs01.msr_bitmap, msr) || \
+ vmx_test_msr_bitmap_##rw(msr_bitmap_l1, msr)) \
+ vmx_set_msr_bitmap_##rw(msr_bitmap_l0, msr); \
+ else \
+ vmx_clear_msr_bitmap_##rw(msr_bitmap_l0, msr); \
+}
+BUILD_NVMX_MSR_INTERCEPT_HELPER(read)
+BUILD_NVMX_MSR_INTERCEPT_HELPER(write)
+
+static inline void nested_vmx_set_intercept_for_msr(struct vcpu_vmx *vmx,
+ unsigned long *msr_bitmap_l1,
+ unsigned long *msr_bitmap_l0,
+ u32 msr, int types)
+{
+ if (types & MSR_TYPE_R)
+ nested_vmx_set_msr_read_intercept(vmx, msr_bitmap_l1,
+ msr_bitmap_l0, msr);
+ if (types & MSR_TYPE_W)
+ nested_vmx_set_msr_write_intercept(vmx, msr_bitmap_l1,
+ msr_bitmap_l0, msr);
+}
+
/*
* Merge L0's and L1's MSR bitmap, return false to indicate that
* we do not use the hardware.
static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
int msr;
unsigned long *msr_bitmap_l1;
- unsigned long *msr_bitmap_l0 = to_vmx(vcpu)->nested.vmcs02.msr_bitmap;
- struct kvm_host_map *map = &to_vmx(vcpu)->nested.msr_bitmap_map;
+ unsigned long *msr_bitmap_l0 = vmx->nested.vmcs02.msr_bitmap;
+ struct kvm_host_map *map = &vmx->nested.msr_bitmap_map;
/* Nothing to do if the MSR bitmap is not in use. */
if (!cpu_has_vmx_msr_bitmap() ||
/*
* To keep the control flow simple, pay eight 8-byte writes (sixteen
* 4-byte writes on 32-bit systems) up front to enable intercepts for
- * the x2APIC MSR range and selectively disable them below.
+ * the x2APIC MSR range and selectively toggle those relevant to L2.
*/
enable_x2apic_msr_intercepts(msr_bitmap_l0);
}
}
- nested_vmx_disable_intercept_for_msr(
+ nested_vmx_disable_intercept_for_x2apic_msr(
msr_bitmap_l1, msr_bitmap_l0,
X2APIC_MSR(APIC_TASKPRI),
MSR_TYPE_R | MSR_TYPE_W);
if (nested_cpu_has_vid(vmcs12)) {
- nested_vmx_disable_intercept_for_msr(
+ nested_vmx_disable_intercept_for_x2apic_msr(
msr_bitmap_l1, msr_bitmap_l0,
X2APIC_MSR(APIC_EOI),
MSR_TYPE_W);
- nested_vmx_disable_intercept_for_msr(
+ nested_vmx_disable_intercept_for_x2apic_msr(
msr_bitmap_l1, msr_bitmap_l0,
X2APIC_MSR(APIC_SELF_IPI),
MSR_TYPE_W);
}
}
- /* KVM unconditionally exposes the FS/GS base MSRs to L1. */
+ /*
+ * Always check vmcs01's bitmap to honor userspace MSR filters and any
+ * other runtime changes to vmcs01's bitmap, e.g. dynamic pass-through.
+ */
#ifdef CONFIG_X86_64
- nested_vmx_disable_intercept_for_msr(msr_bitmap_l1, msr_bitmap_l0,
- MSR_FS_BASE, MSR_TYPE_RW);
+ nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
+ MSR_FS_BASE, MSR_TYPE_RW);
- nested_vmx_disable_intercept_for_msr(msr_bitmap_l1, msr_bitmap_l0,
- MSR_GS_BASE, MSR_TYPE_RW);
+ nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
+ MSR_GS_BASE, MSR_TYPE_RW);
- nested_vmx_disable_intercept_for_msr(msr_bitmap_l1, msr_bitmap_l0,
- MSR_KERNEL_GS_BASE, MSR_TYPE_RW);
+ nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
+ MSR_KERNEL_GS_BASE, MSR_TYPE_RW);
#endif
+ nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
+ MSR_IA32_SPEC_CTRL, MSR_TYPE_RW);
- /*
- * Checking the L0->L1 bitmap is trying to verify two things:
- *
- * 1. L0 gave a permission to L1 to actually passthrough the MSR. This
- * ensures that we do not accidentally generate an L02 MSR bitmap
- * from the L12 MSR bitmap that is too permissive.
- * 2. That L1 or L2s have actually used the MSR. This avoids
- * unnecessarily merging of the bitmap if the MSR is unused. This
- * works properly because we only update the L01 MSR bitmap lazily.
- * So even if L0 should pass L1 these MSRs, the L01 bitmap is only
- * updated to reflect this when L1 (or its L2s) actually write to
- * the MSR.
- */
- if (!msr_write_intercepted_l01(vcpu, MSR_IA32_SPEC_CTRL))
- nested_vmx_disable_intercept_for_msr(
- msr_bitmap_l1, msr_bitmap_l0,
- MSR_IA32_SPEC_CTRL,
- MSR_TYPE_R | MSR_TYPE_W);
-
- if (!msr_write_intercepted_l01(vcpu, MSR_IA32_PRED_CMD))
- nested_vmx_disable_intercept_for_msr(
- msr_bitmap_l1, msr_bitmap_l0,
- MSR_IA32_PRED_CMD,
- MSR_TYPE_W);
+ nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0,
+ MSR_IA32_PRED_CMD, MSR_TYPE_W);
- kvm_vcpu_unmap(vcpu, &to_vmx(vcpu)->nested.msr_bitmap_map, false);
+ kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false);
return true;
}
static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
- struct kvm_host_map map;
- struct vmcs12 *shadow;
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+ struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
if (!nested_cpu_has_shadow_vmcs(vmcs12) ||
vmcs12->vmcs_link_pointer == INVALID_GPA)
return;
- shadow = get_shadow_vmcs12(vcpu);
-
- if (kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->vmcs_link_pointer), &map))
+ if (ghc->gpa != vmcs12->vmcs_link_pointer &&
+ kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc,
+ vmcs12->vmcs_link_pointer, VMCS12_SIZE))
return;
- memcpy(shadow, map.hva, VMCS12_SIZE);
- kvm_vcpu_unmap(vcpu, &map, false);
+ kvm_read_guest_cached(vmx->vcpu.kvm, ghc, get_shadow_vmcs12(vcpu),
+ VMCS12_SIZE);
}
static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
+ struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
if (!nested_cpu_has_shadow_vmcs(vmcs12) ||
vmcs12->vmcs_link_pointer == INVALID_GPA)
return;
- kvm_write_guest(vmx->vcpu.kvm, vmcs12->vmcs_link_pointer,
- get_shadow_vmcs12(vcpu), VMCS12_SIZE);
+ if (ghc->gpa != vmcs12->vmcs_link_pointer &&
+ kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc,
+ vmcs12->vmcs_link_pointer, VMCS12_SIZE))
+ return;
+
+ kvm_write_guest_cached(vmx->vcpu.kvm, ghc, get_shadow_vmcs12(vcpu),
+ VMCS12_SIZE);
}
/*
return 0;
}
+static int nested_vmx_check_address_space_size(struct kvm_vcpu *vcpu,
+ struct vmcs12 *vmcs12)
+{
+#ifdef CONFIG_X86_64
+ if (CC(!!(vmcs12->vm_exit_controls & VM_EXIT_HOST_ADDR_SPACE_SIZE) !=
+ !!(vcpu->arch.efer & EFER_LMA)))
+ return -EINVAL;
+#endif
+ return 0;
+}
+
static int nested_vmx_check_host_state(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
return -EINVAL;
#ifdef CONFIG_X86_64
- ia32e = !!(vcpu->arch.efer & EFER_LMA);
+ ia32e = !!(vmcs12->vm_exit_controls & VM_EXIT_HOST_ADDR_SPACE_SIZE);
#else
ia32e = false;
#endif
if (ia32e) {
- if (CC(!(vmcs12->vm_exit_controls & VM_EXIT_HOST_ADDR_SPACE_SIZE)) ||
- CC(!(vmcs12->host_cr4 & X86_CR4_PAE)))
+ if (CC(!(vmcs12->host_cr4 & X86_CR4_PAE)))
return -EINVAL;
} else {
- if (CC(vmcs12->vm_exit_controls & VM_EXIT_HOST_ADDR_SPACE_SIZE) ||
- CC(vmcs12->vm_entry_controls & VM_ENTRY_IA32E_MODE) ||
+ if (CC(vmcs12->vm_entry_controls & VM_ENTRY_IA32E_MODE) ||
CC(vmcs12->host_cr4 & X86_CR4_PCIDE) ||
CC((vmcs12->host_rip) >> 32))
return -EINVAL;
static int nested_vmx_check_vmcs_link_ptr(struct kvm_vcpu *vcpu,
struct vmcs12 *vmcs12)
{
- int r = 0;
- struct vmcs12 *shadow;
- struct kvm_host_map map;
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+ struct gfn_to_hva_cache *ghc = &vmx->nested.shadow_vmcs12_cache;
+ struct vmcs_hdr hdr;
if (vmcs12->vmcs_link_pointer == INVALID_GPA)
return 0;
if (CC(!page_address_valid(vcpu, vmcs12->vmcs_link_pointer)))
return -EINVAL;
- if (CC(kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->vmcs_link_pointer), &map)))
- return -EINVAL;
+ if (ghc->gpa != vmcs12->vmcs_link_pointer &&
+ CC(kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc,
+ vmcs12->vmcs_link_pointer, VMCS12_SIZE)))
+ return -EINVAL;
- shadow = map.hva;
+ if (CC(kvm_read_guest_offset_cached(vcpu->kvm, ghc, &hdr,
+ offsetof(struct vmcs12, hdr),
+ sizeof(hdr))))
+ return -EINVAL;
- if (CC(shadow->hdr.revision_id != VMCS12_REVISION) ||
- CC(shadow->hdr.shadow_vmcs != nested_cpu_has_shadow_vmcs(vmcs12)))
- r = -EINVAL;
+ if (CC(hdr.revision_id != VMCS12_REVISION) ||
+ CC(hdr.shadow_vmcs != nested_cpu_has_shadow_vmcs(vmcs12)))
+ return -EINVAL;
- kvm_vcpu_unmap(vcpu, &map, false);
- return r;
+ return 0;
}
/*
if (nested_vmx_check_controls(vcpu, vmcs12))
return nested_vmx_fail(vcpu, VMXERR_ENTRY_INVALID_CONTROL_FIELD);
+ if (nested_vmx_check_address_space_size(vcpu, vmcs12))
+ return nested_vmx_fail(vcpu, VMXERR_ENTRY_INVALID_HOST_STATE_FIELD);
+
if (nested_vmx_check_host_state(vcpu, vmcs12))
return nested_vmx_fail(vcpu, VMXERR_ENTRY_INVALID_HOST_STATE_FIELD);
return 1;
if (vmx->nested.current_vmptr != vmptr) {
- struct kvm_host_map map;
- struct vmcs12 *new_vmcs12;
+ struct gfn_to_hva_cache *ghc = &vmx->nested.vmcs12_cache;
+ struct vmcs_hdr hdr;
- if (kvm_vcpu_map(vcpu, gpa_to_gfn(vmptr), &map)) {
+ if (ghc->gpa != vmptr &&
+ kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, vmptr, VMCS12_SIZE)) {
/*
* Reads from an unbacked page return all 1s,
* which means that the 32 bits located at the
VMXERR_VMPTRLD_INCORRECT_VMCS_REVISION_ID);
}
- new_vmcs12 = map.hva;
+ if (kvm_read_guest_offset_cached(vcpu->kvm, ghc, &hdr,
+ offsetof(struct vmcs12, hdr),
+ sizeof(hdr))) {
+ return nested_vmx_fail(vcpu,
+ VMXERR_VMPTRLD_INCORRECT_VMCS_REVISION_ID);
+ }
- if (new_vmcs12->hdr.revision_id != VMCS12_REVISION ||
- (new_vmcs12->hdr.shadow_vmcs &&
+ if (hdr.revision_id != VMCS12_REVISION ||
+ (hdr.shadow_vmcs &&
!nested_cpu_has_vmx_shadow_vmcs(vcpu))) {
- kvm_vcpu_unmap(vcpu, &map, false);
return nested_vmx_fail(vcpu,
VMXERR_VMPTRLD_INCORRECT_VMCS_REVISION_ID);
}
* Load VMCS12 from guest memory since it is not already
* cached.
*/
- memcpy(vmx->nested.cached_vmcs12, new_vmcs12, VMCS12_SIZE);
- kvm_vcpu_unmap(vcpu, &map, false);
+ if (kvm_read_guest_cached(vcpu->kvm, ghc, vmx->nested.cached_vmcs12,
+ VMCS12_SIZE)) {
+ return nested_vmx_fail(vcpu,
+ VMXERR_VMPTRLD_INCORRECT_VMCS_REVISION_ID);
+ }
set_current_vmptr(vmx, vmptr);
}
struct {
u64 eptp, gpa;
} operand;
- int i, r;
+ int i, r, gpr_index;
if (!(vmx->nested.msrs.secondary_ctls_high &
SECONDARY_EXEC_ENABLE_EPT) ||
return 1;
vmx_instruction_info = vmcs_read32(VMX_INSTRUCTION_INFO);
- type = kvm_register_read(vcpu, (vmx_instruction_info >> 28) & 0xf);
+ gpr_index = vmx_get_instr_info_reg2(vmx_instruction_info);
+ type = kvm_register_read(vcpu, gpr_index);
types = (vmx->nested.msrs.ept_caps >> VMX_EPT_EXTENT_SHIFT) & 6;
u64 gla;
} operand;
u16 vpid02;
- int r;
+ int r, gpr_index;
if (!(vmx->nested.msrs.secondary_ctls_high &
SECONDARY_EXEC_ENABLE_VPID) ||
return 1;
vmx_instruction_info = vmcs_read32(VMX_INSTRUCTION_INFO);
- type = kvm_register_read(vcpu, (vmx_instruction_info >> 28) & 0xf);
+ gpr_index = vmx_get_instr_info_reg2(vmx_instruction_info);
+ type = kvm_register_read(vcpu, gpr_index);
types = (vmx->nested.msrs.vpid_caps &
VMX_VPID_EXTENT_SUPPORTED_MASK) >> 8;
}
}
-/* returns 0 if idx's corresponding MSR exists; otherwise returns 1. */
-static int intel_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
+static bool intel_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx)
{
struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);
bool fixed = idx & (1u << 30);
idx &= ~(3u << 30);
- return (!fixed && idx >= pmu->nr_arch_gp_counters) ||
- (fixed && idx >= pmu->nr_arch_fixed_counters);
+ return fixed ? idx < pmu->nr_arch_fixed_counters
+ : idx < pmu->nr_arch_gp_counters;
}
static struct kvm_pmc *intel_rdpmc_ecx_to_pmc(struct kvm_vcpu *vcpu,
/*
* Check if MSR is intercepted for currently loaded MSR bitmap.
*/
-static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr)
+static bool msr_write_intercepted(struct vcpu_vmx *vmx, u32 msr)
{
- unsigned long *msr_bitmap;
- int f = sizeof(unsigned long);
-
- if (!cpu_has_vmx_msr_bitmap())
+ if (!(exec_controls_get(vmx) & CPU_BASED_USE_MSR_BITMAPS))
return true;
- msr_bitmap = to_vmx(vcpu)->loaded_vmcs->msr_bitmap;
-
- if (msr <= 0x1fff) {
- return !!test_bit(msr, msr_bitmap + 0x800 / f);
- } else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff)) {
- msr &= 0x1fff;
- return !!test_bit(msr, msr_bitmap + 0xc00 / f);
- }
-
- return true;
+ return vmx_test_msr_bitmap_write(vmx->loaded_vmcs->msr_bitmap,
+ MSR_IA32_SPEC_CTRL);
}
static void clear_atomic_switch_msr_special(struct vcpu_vmx *vmx,
spin_unlock(&vmx_vpid_lock);
}
-static void vmx_clear_msr_bitmap_read(ulong *msr_bitmap, u32 msr)
-{
- int f = sizeof(unsigned long);
-
- if (msr <= 0x1fff)
- __clear_bit(msr, msr_bitmap + 0x000 / f);
- else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff))
- __clear_bit(msr & 0x1fff, msr_bitmap + 0x400 / f);
-}
-
-static void vmx_clear_msr_bitmap_write(ulong *msr_bitmap, u32 msr)
-{
- int f = sizeof(unsigned long);
-
- if (msr <= 0x1fff)
- __clear_bit(msr, msr_bitmap + 0x800 / f);
- else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff))
- __clear_bit(msr & 0x1fff, msr_bitmap + 0xc00 / f);
-}
-
-static void vmx_set_msr_bitmap_read(ulong *msr_bitmap, u32 msr)
-{
- int f = sizeof(unsigned long);
-
- if (msr <= 0x1fff)
- __set_bit(msr, msr_bitmap + 0x000 / f);
- else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff))
- __set_bit(msr & 0x1fff, msr_bitmap + 0x400 / f);
-}
-
-static void vmx_set_msr_bitmap_write(ulong *msr_bitmap, u32 msr)
-{
- int f = sizeof(unsigned long);
-
- if (msr <= 0x1fff)
- __set_bit(msr, msr_bitmap + 0x800 / f);
- else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff))
- __set_bit(msr & 0x1fff, msr_bitmap + 0xc00 / f);
-}
-
void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
u64 pcid;
u64 gla;
} operand;
+ int gpr_index;
if (!guest_cpuid_has(vcpu, X86_FEATURE_INVPCID)) {
kvm_queue_exception(vcpu, UD_VECTOR);
}
vmx_instruction_info = vmcs_read32(VMX_INSTRUCTION_INFO);
- type = kvm_register_read(vcpu, (vmx_instruction_info >> 28) & 0xf);
-
- if (type > 3) {
- kvm_inject_gp(vcpu, 0);
- return 1;
- }
+ gpr_index = vmx_get_instr_info_reg2(vmx_instruction_info);
+ type = kvm_register_read(vcpu, gpr_index);
/* According to the Intel instruction reference, the memory operand
* is read even if it isn't needed (e.g., for type==all)
* If the L02 MSR bitmap does not intercept the MSR, then we need to
* save it.
*/
- if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)))
+ if (unlikely(!msr_write_intercepted(vmx, MSR_IA32_SPEC_CTRL)))
vmx->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL);
x86_spec_ctrl_restore_host(vmx->spec_ctrl, 0);
static bool vmx_check_apicv_inhibit_reasons(ulong bit)
{
ulong supported = BIT(APICV_INHIBIT_REASON_DISABLE) |
- BIT(APICV_INHIBIT_REASON_HYPERV);
+ BIT(APICV_INHIBIT_REASON_HYPERV) |
+ BIT(APICV_INHIBIT_REASON_BLOCKIRQ);
return supported & BIT(bit);
}
struct vmcs12 *cached_shadow_vmcs12;
/*
+ * GPA to HVA cache for accessing vmcs12->vmcs_link_pointer
+ */
+ struct gfn_to_hva_cache shadow_vmcs12_cache;
+
+ /*
+ * GPA to HVA cache for VMCS12
+ */
+ struct gfn_to_hva_cache vmcs12_cache;
+
+ /*
* Indicates if the shadow vmcs or enlightened vmcs must be updated
* with the data held by struct vmcs12.
*/
void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu);
+/*
+ * Note, early Intel manuals have the write-low and read-high bitmap offsets
+ * the wrong way round. The bitmaps control MSRs 0x00000000-0x00001fff and
+ * 0xc0000000-0xc0001fff. The former (low) uses bytes 0-0x3ff for reads and
+ * 0x800-0xbff for writes. The latter (high) uses 0x400-0x7ff for reads and
+ * 0xc00-0xfff for writes. MSRs not covered by either of the ranges always
+ * VM-Exit.
+ */
+#define __BUILD_VMX_MSR_BITMAP_HELPER(rtype, action, bitop, access, base) \
+static inline rtype vmx_##action##_msr_bitmap_##access(unsigned long *bitmap, \
+ u32 msr) \
+{ \
+ int f = sizeof(unsigned long); \
+ \
+ if (msr <= 0x1fff) \
+ return bitop##_bit(msr, bitmap + base / f); \
+ else if ((msr >= 0xc0000000) && (msr <= 0xc0001fff)) \
+ return bitop##_bit(msr & 0x1fff, bitmap + (base + 0x400) / f); \
+ return (rtype)true; \
+}
+#define BUILD_VMX_MSR_BITMAP_HELPERS(ret_type, action, bitop) \
+ __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0x0) \
+ __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 0x800)
+
+BUILD_VMX_MSR_BITMAP_HELPERS(bool, test, test)
+BUILD_VMX_MSR_BITMAP_HELPERS(void, clear, __clear)
+BUILD_VMX_MSR_BITMAP_HELPERS(void, set, __set)
+
static inline u8 vmx_get_rvi(void)
{
return vmcs_read16(GUEST_INTR_STATUS) & 0xff;
void dump_vmcs(struct kvm_vcpu *vcpu);
+static inline int vmx_get_instr_info_reg2(u32 vmx_instr_info)
+{
+ return (vmx_instr_info >> 28) & 0xf;
+}
+
#endif /* __KVM_X86_VMX_H */
static void record_steal_time(struct kvm_vcpu *vcpu)
{
- struct kvm_host_map map;
- struct kvm_steal_time *st;
+ struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
+ struct kvm_steal_time __user *st;
+ struct kvm_memslots *slots;
+ u64 steal;
+ u32 version;
if (kvm_xen_msr_enabled(vcpu->kvm)) {
kvm_xen_runstate_set_running(vcpu);
if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED))
return;
- /* -EAGAIN is returned in atomic context so we can just return. */
- if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT,
- &map, &vcpu->arch.st.cache, false))
+ if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
return;
- st = map.hva +
- offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS);
+ slots = kvm_memslots(vcpu->kvm);
+
+ if (unlikely(slots->generation != ghc->generation ||
+ kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+ gfn_t gfn = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
+
+ /* We rely on the fact that it fits in a single page. */
+ BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS);
+
+ if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gfn, sizeof(*st)) ||
+ kvm_is_error_hva(ghc->hva) || !ghc->memslot)
+ return;
+ }
+ st = (struct kvm_steal_time __user *)ghc->hva;
/*
* Doing a TLB flush here, on the guest's behalf, can avoid
* expensive IPIs.
*/
if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) {
- u8 st_preempted = xchg(&st->preempted, 0);
+ u8 st_preempted = 0;
+ int err = -EFAULT;
+
+ if (!user_access_begin(st, sizeof(*st)))
+ return;
+
+ asm volatile("1: xchgb %0, %2\n"
+ "xor %1, %1\n"
+ "2:\n"
+ _ASM_EXTABLE_UA(1b, 2b)
+ : "+q" (st_preempted),
+ "+&r" (err),
+ "+m" (st->preempted));
+ if (err)
+ goto out;
+
+ user_access_end();
+
+ vcpu->arch.st.preempted = 0;
trace_kvm_pv_tlb_flush(vcpu->vcpu_id,
st_preempted & KVM_VCPU_FLUSH_TLB);
if (st_preempted & KVM_VCPU_FLUSH_TLB)
kvm_vcpu_flush_tlb_guest(vcpu);
+
+ if (!user_access_begin(st, sizeof(*st)))
+ goto dirty;
} else {
- st->preempted = 0;
- }
+ if (!user_access_begin(st, sizeof(*st)))
+ return;
- vcpu->arch.st.preempted = 0;
+ unsafe_put_user(0, &st->preempted, out);
+ vcpu->arch.st.preempted = 0;
+ }
- if (st->version & 1)
- st->version += 1; /* first time write, random junk */
+ unsafe_get_user(version, &st->version, out);
+ if (version & 1)
+ version += 1; /* first time write, random junk */
- st->version += 1;
+ version += 1;
+ unsafe_put_user(version, &st->version, out);
smp_wmb();
- st->steal += current->sched_info.run_delay -
+ unsafe_get_user(steal, &st->steal, out);
+ steal += current->sched_info.run_delay -
vcpu->arch.st.last_steal;
vcpu->arch.st.last_steal = current->sched_info.run_delay;
+ unsafe_put_user(steal, &st->steal, out);
- smp_wmb();
+ version += 1;
+ unsafe_put_user(version, &st->version, out);
- st->version += 1;
-
- kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, false);
+ out:
+ user_access_end();
+ dirty:
+ mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
}
int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI))
return 1;
- if (kvm_lapic_enable_pv_eoi(vcpu, data, sizeof(u8)))
+ if (kvm_lapic_set_pv_eoi(vcpu, data, sizeof(u8)))
return 1;
break;
r = !static_call(kvm_x86_cpu_has_accelerated_tpr)();
break;
case KVM_CAP_NR_VCPUS:
- r = KVM_SOFT_MAX_VCPUS;
+ r = min_t(unsigned int, num_online_cpus(), KVM_MAX_VCPUS);
break;
case KVM_CAP_MAX_VCPUS:
r = KVM_MAX_VCPUS;
static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
{
- struct kvm_host_map map;
- struct kvm_steal_time *st;
+ struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
+ struct kvm_steal_time __user *st;
+ struct kvm_memslots *slots;
+ static const u8 preempted = KVM_VCPU_PREEMPTED;
if (!(vcpu->arch.st.msr_val & KVM_MSR_ENABLED))
return;
if (vcpu->arch.st.preempted)
return;
- if (kvm_map_gfn(vcpu, vcpu->arch.st.msr_val >> PAGE_SHIFT, &map,
- &vcpu->arch.st.cache, true))
+ /* This happens on process exit */
+ if (unlikely(current->mm != vcpu->kvm->mm))
+ return;
+
+ slots = kvm_memslots(vcpu->kvm);
+
+ if (unlikely(slots->generation != ghc->generation ||
+ kvm_is_error_hva(ghc->hva) || !ghc->memslot))
return;
- st = map.hva +
- offset_in_page(vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS);
+ st = (struct kvm_steal_time __user *)ghc->hva;
+ BUILD_BUG_ON(sizeof(st->preempted) != sizeof(preempted));
- st->preempted = vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED;
+ if (!copy_to_user_nofault(&st->preempted, &preempted, sizeof(preempted)))
+ vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED;
- kvm_unmap_gfn(vcpu, &map, &vcpu->arch.st.cache, true, true);
+ mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa));
}
void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
if (kvm_x86_ops.vm_copy_enc_context_from)
r = kvm_x86_ops.vm_copy_enc_context_from(kvm, cap->args[0]);
return r;
+ case KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM:
+ r = -EINVAL;
+ if (kvm_x86_ops.vm_move_enc_context_from)
+ r = kvm_x86_ops.vm_move_enc_context_from(
+ kvm, cap->args[0]);
+ return r;
case KVM_CAP_EXIT_HYPERCALL:
if (cap->args[0] & ~KVM_EXIT_HYPERCALL_VALID_MASK) {
r = -EINVAL;
static int emulator_check_pmc(struct x86_emulate_ctxt *ctxt,
u32 pmc)
{
- return kvm_pmu_is_valid_rdpmc_ecx(emul_to_vcpu(ctxt), pmc);
+ if (kvm_pmu_is_valid_rdpmc_ecx(emul_to_vcpu(ctxt), pmc))
+ return 0;
+ return -EINVAL;
}
static int emulator_read_pmc(struct x86_emulate_ctxt *ctxt,
trace_kvm_hypercall(nr, a0, a1, a2, a3);
- op_64_bit = is_64_bit_mode(vcpu);
+ op_64_bit = is_64_bit_hypercall(vcpu);
if (!op_64_bit) {
nr &= 0xFFFFFFFF;
a0 &= 0xFFFFFFFF;
if (!kvm_apic_hw_enabled(vcpu->arch.apic))
return;
- if (to_hv_vcpu(vcpu))
+ if (to_hv_vcpu(vcpu)) {
bitmap_or((ulong *)eoi_exit_bitmap,
vcpu->arch.ioapic_handled_vectors,
to_hv_synic(vcpu)->vec_bitmap, 256);
+ static_call(kvm_x86_load_eoi_exitmap)(vcpu, eoi_exit_bitmap);
+ return;
+ }
- static_call(kvm_x86_load_eoi_exitmap)(vcpu, eoi_exit_bitmap);
+ static_call(kvm_x86_load_eoi_exitmap)(
+ vcpu, (u64 *)vcpu->arch.ioapic_handled_vectors);
}
void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm,
}
if (kvm_request_pending(vcpu)) {
- if (kvm_check_request(KVM_REQ_VM_BUGGED, vcpu)) {
+ if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu)) {
r = -EIO;
goto out;
}
return ret;
}
+static void kvm_arch_vcpu_guestdbg_update_apicv_inhibit(struct kvm *kvm)
+{
+ bool inhibit = false;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ down_write(&kvm->arch.apicv_update_lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (vcpu->guest_debug & KVM_GUESTDBG_BLOCKIRQ) {
+ inhibit = true;
+ break;
+ }
+ }
+ __kvm_request_apicv_update(kvm, !inhibit, APICV_INHIBIT_REASON_BLOCKIRQ);
+ up_write(&kvm->arch.apicv_update_lock);
+}
+
int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
struct kvm_guest_debug *dbg)
{
static_call(kvm_x86_update_exception_bitmap)(vcpu);
+ kvm_arch_vcpu_guestdbg_update_apicv_inhibit(vcpu->kvm);
+
r = 0;
out:
void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
{
- struct gfn_to_pfn_cache *cache = &vcpu->arch.st.cache;
int idx;
- kvm_release_pfn(cache->pfn, cache->dirty, cache);
-
kvmclock_reset(vcpu);
static_call(kvm_x86_vcpu_free)(vcpu);
return kvm_skip_emulated_instruction(vcpu);
default:
- BUG(); /* We have already checked above that type <= 3 */
+ kvm_inject_gp(vcpu, 0);
+ return 1;
}
}
EXPORT_SYMBOL_GPL(kvm_handle_invpcid);
{
int cs_db, cs_l;
+ WARN_ON_ONCE(vcpu->arch.guest_state_protected);
+
if (!is_long_mode(vcpu))
return false;
static_call(kvm_x86_get_cs_db_l_bits)(vcpu, &cs_db, &cs_l);
return cs_l;
}
+static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu)
+{
+ /*
+ * If running with protected guest state, the CS register is not
+ * accessible. The hypercall register values will have had to been
+ * provided in 64-bit mode, so assume the guest is in 64-bit.
+ */
+ return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu);
+}
+
static inline bool x86_exception_has_error_code(unsigned int vector)
{
static u32 exception_has_error_code = BIT(DF_VECTOR) | BIT(TS_VECTOR) |
state_entry_time = vx->runstate_entry_time;
state_entry_time |= XEN_RUNSTATE_UPDATE;
- BUILD_BUG_ON(sizeof(((struct vcpu_runstate_info *)0)->state_entry_time) !=
+ BUILD_BUG_ON(sizeof_field(struct vcpu_runstate_info, state_entry_time) !=
sizeof(state_entry_time));
- BUILD_BUG_ON(sizeof(((struct compat_vcpu_runstate_info *)0)->state_entry_time) !=
+ BUILD_BUG_ON(sizeof_field(struct compat_vcpu_runstate_info, state_entry_time) !=
sizeof(state_entry_time));
if (kvm_write_guest_offset_cached(v->kvm, &v->arch.xen.runstate_cache,
*/
BUILD_BUG_ON(offsetof(struct vcpu_runstate_info, state) !=
offsetof(struct compat_vcpu_runstate_info, state));
- BUILD_BUG_ON(sizeof(((struct vcpu_runstate_info *)0)->state) !=
+ BUILD_BUG_ON(sizeof_field(struct vcpu_runstate_info, state) !=
sizeof(vx->current_runstate));
- BUILD_BUG_ON(sizeof(((struct compat_vcpu_runstate_info *)0)->state) !=
+ BUILD_BUG_ON(sizeof_field(struct compat_vcpu_runstate_info, state) !=
sizeof(vx->current_runstate));
if (kvm_write_guest_offset_cached(v->kvm, &v->arch.xen.runstate_cache,
offsetof(struct vcpu_runstate_info, time) - sizeof(u64));
BUILD_BUG_ON(offsetof(struct compat_vcpu_runstate_info, state_entry_time) !=
offsetof(struct compat_vcpu_runstate_info, time) - sizeof(u64));
- BUILD_BUG_ON(sizeof(((struct vcpu_runstate_info *)0)->time) !=
- sizeof(((struct compat_vcpu_runstate_info *)0)->time));
- BUILD_BUG_ON(sizeof(((struct vcpu_runstate_info *)0)->time) !=
+ BUILD_BUG_ON(sizeof_field(struct vcpu_runstate_info, time) !=
+ sizeof_field(struct compat_vcpu_runstate_info, time));
+ BUILD_BUG_ON(sizeof_field(struct vcpu_runstate_info, time) !=
sizeof(vx->runstate_times));
if (kvm_write_guest_offset_cached(v->kvm, &v->arch.xen.runstate_cache,
BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=
offsetof(struct compat_vcpu_info, evtchn_upcall_pending));
BUILD_BUG_ON(sizeof(rc) !=
- sizeof(((struct vcpu_info *)0)->evtchn_upcall_pending));
+ sizeof_field(struct vcpu_info, evtchn_upcall_pending));
BUILD_BUG_ON(sizeof(rc) !=
- sizeof(((struct compat_vcpu_info *)0)->evtchn_upcall_pending));
+ sizeof_field(struct compat_vcpu_info, evtchn_upcall_pending));
/*
* For efficiency, this mirrors the checks for using the valid
break;
case KVM_XEN_ATTR_TYPE_SHARED_INFO:
- data->u.shared_info.gfn = gpa_to_gfn(kvm->arch.xen.shinfo_gfn);
+ data->u.shared_info.gfn = kvm->arch.xen.shinfo_gfn;
r = 0;
break;
kvm_hv_hypercall_enabled(vcpu))
return kvm_hv_hypercall(vcpu);
- longmode = is_64_bit_mode(vcpu);
+ longmode = is_64_bit_hypercall(vcpu);
if (!longmode) {
params[0] = (u32)kvm_rbx_read(vcpu);
params[1] = (u32)kvm_rcx_read(vcpu);
*/
addr = memblock_phys_alloc_range(PMD_SIZE, PMD_SIZE, map_start,
map_end);
- memblock_free(addr, PMD_SIZE);
+ memblock_phys_free(addr, PMD_SIZE);
real_end = addr + PMD_SIZE;
/* step_size need to be small so pgt_buf from BRK could cover it */
}
}
-/*
- * The <linux/kallsyms.h> already defines is_kernel_text,
- * using '__' prefix not to get in conflict.
- */
-static inline int __is_kernel_text(unsigned long addr)
+static inline int is_x86_32_kernel_text(unsigned long addr)
{
if (addr >= (unsigned long)_text && addr <= (unsigned long)__init_end)
return 1;
addr2 = (pfn + PTRS_PER_PTE-1) * PAGE_SIZE +
PAGE_OFFSET + PAGE_SIZE-1;
- if (__is_kernel_text(addr) ||
- __is_kernel_text(addr2))
+ if (is_x86_32_kernel_text(addr) ||
+ is_x86_32_kernel_text(addr2))
prot = PAGE_KERNEL_LARGE_EXEC;
pages_2m++;
*/
pgprot_t init_prot = __pgprot(PTE_IDENT_ATTR);
- if (__is_kernel_text(addr))
+ if (is_x86_32_kernel_text(addr))
prot = PAGE_KERNEL_EXEC;
pages_4k++;
test_wp_bit();
}
-#ifdef CONFIG_MEMORY_HOTPLUG
-int arch_add_memory(int nid, u64 start, u64 size,
- struct mhp_params *params)
-{
- unsigned long start_pfn = start >> PAGE_SHIFT;
- unsigned long nr_pages = size >> PAGE_SHIFT;
- int ret;
-
- /*
- * The page tables were already mapped at boot so if the caller
- * requests a different mapping type then we must change all the
- * pages with __set_memory_prot().
- */
- if (params->pgprot.pgprot != PAGE_KERNEL.pgprot) {
- ret = __set_memory_prot(start, nr_pages, params->pgprot);
- if (ret)
- return ret;
- }
-
- return __add_pages(nid, start_pfn, nr_pages, params);
-}
-
-void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
-{
- unsigned long start_pfn = start >> PAGE_SHIFT;
- unsigned long nr_pages = size >> PAGE_SHIFT;
-
- __remove_pages(start_pfn, nr_pages, altmap);
-}
-#endif
-
int kernel_set_to_readonly __read_mostly;
static void mark_nxdata_nx(void)
*/
unsigned long start = PFN_ALIGN(_etext);
/*
- * This comes from __is_kernel_text upper limit. Also HPAGE where used:
+ * This comes from is_x86_32_kernel_text upper limit. Also HPAGE where used:
*/
unsigned long size = (((unsigned long)__init_end + HPAGE_SIZE) & HPAGE_MASK) - start;
p = early_alloc(PMD_SIZE, nid, false);
if (p && pmd_set_huge(pmd, __pa(p), PAGE_KERNEL))
return;
- memblock_free_ptr(p, PMD_SIZE);
+ memblock_free(p, PMD_SIZE);
}
p = early_alloc(PAGE_SIZE, nid, true);
p = early_alloc(PUD_SIZE, nid, false);
if (p && pud_set_huge(pud, __pa(p), PAGE_KERNEL))
return;
- memblock_free_ptr(p, PUD_SIZE);
+ memblock_free(p, PUD_SIZE);
}
p = early_alloc(PAGE_SIZE, nid, true);
swiotlb_adjust_size(size);
}
-static void __init __set_clr_pte_enc(pte_t *kpte, int level, bool enc)
+static unsigned long pg_level_to_pfn(int level, pte_t *kpte, pgprot_t *ret_prot)
{
- pgprot_t old_prot, new_prot;
- unsigned long pfn, pa, size;
- pte_t new_pte;
+ unsigned long pfn = 0;
+ pgprot_t prot;
switch (level) {
case PG_LEVEL_4K:
pfn = pte_pfn(*kpte);
- old_prot = pte_pgprot(*kpte);
+ prot = pte_pgprot(*kpte);
break;
case PG_LEVEL_2M:
pfn = pmd_pfn(*(pmd_t *)kpte);
- old_prot = pmd_pgprot(*(pmd_t *)kpte);
+ prot = pmd_pgprot(*(pmd_t *)kpte);
break;
case PG_LEVEL_1G:
pfn = pud_pfn(*(pud_t *)kpte);
- old_prot = pud_pgprot(*(pud_t *)kpte);
+ prot = pud_pgprot(*(pud_t *)kpte);
break;
default:
- return;
+ WARN_ONCE(1, "Invalid level for kpte\n");
+ return 0;
+ }
+
+ if (ret_prot)
+ *ret_prot = prot;
+
+ return pfn;
+}
+
+void notify_range_enc_status_changed(unsigned long vaddr, int npages, bool enc)
+{
+#ifdef CONFIG_PARAVIRT
+ unsigned long sz = npages << PAGE_SHIFT;
+ unsigned long vaddr_end = vaddr + sz;
+
+ while (vaddr < vaddr_end) {
+ int psize, pmask, level;
+ unsigned long pfn;
+ pte_t *kpte;
+
+ kpte = lookup_address(vaddr, &level);
+ if (!kpte || pte_none(*kpte)) {
+ WARN_ONCE(1, "kpte lookup for vaddr\n");
+ return;
+ }
+
+ pfn = pg_level_to_pfn(level, kpte, NULL);
+ if (!pfn)
+ continue;
+
+ psize = page_level_size(level);
+ pmask = page_level_mask(level);
+
+ notify_page_enc_status_changed(pfn, psize >> PAGE_SHIFT, enc);
+
+ vaddr = (vaddr & pmask) + psize;
}
+#endif
+}
+
+static void __init __set_clr_pte_enc(pte_t *kpte, int level, bool enc)
+{
+ pgprot_t old_prot, new_prot;
+ unsigned long pfn, pa, size;
+ pte_t new_pte;
+
+ pfn = pg_level_to_pfn(level, kpte, &old_prot);
+ if (!pfn)
+ return;
new_prot = old_prot;
if (enc)
static int __init early_set_memory_enc_dec(unsigned long vaddr,
unsigned long size, bool enc)
{
- unsigned long vaddr_end, vaddr_next;
+ unsigned long vaddr_end, vaddr_next, start;
unsigned long psize, pmask;
int split_page_size_mask;
int level, ret;
pte_t *kpte;
+ start = vaddr;
vaddr_next = vaddr;
vaddr_end = vaddr + size;
ret = 0;
+ notify_range_enc_status_changed(start, PAGE_ALIGN(size) >> PAGE_SHIFT, enc);
out:
__flush_tlb_all();
return ret;
return early_set_memory_enc_dec(vaddr, size, true);
}
+void __init early_set_mem_enc_dec_hypercall(unsigned long vaddr, int npages, bool enc)
+{
+ notify_range_enc_status_changed(vaddr, npages, enc);
+}
+
/* Override for DMA direct allocation check - ARCH_HAS_FORCE_DMA_UNENCRYPTED */
bool force_dma_unencrypted(struct device *dev)
{
/* numa_distance could be 1LU marking allocation failure, test cnt */
if (numa_distance_cnt)
- memblock_free_ptr(numa_distance, size);
+ memblock_free(numa_distance, size);
numa_distance_cnt = 0;
numa_distance = NULL; /* enable table creation */
}
}
/* free the copied physical distance table */
- memblock_free_ptr(phys_dist, phys_size);
+ memblock_free(phys_dist, phys_size);
return;
no_emu:
*/
cpa_flush(&cpa, 0);
+ /*
+ * Notify hypervisor that a given memory range is mapped encrypted
+ * or decrypted.
+ */
+ notify_range_enc_status_changed(addr, numpages, enc);
+
return ret;
}
pdev->hotplug_user_indicators = 1;
}
-int pcibios_add_device(struct pci_dev *dev)
+int pcibios_device_add(struct pci_dev *dev)
{
struct pci_setup_rom *rom;
struct irq_domain *msidom;
#include <xen/features.h>
#include <xen/events.h>
+#include <xen/pci.h>
#include <asm/xen/pci.h>
#include <asm/xen/cpuid.h>
#include <asm/apic.h>
}
#endif
-#ifdef CONFIG_XEN_DOM0
-
-struct xen_device_domain_owner {
- domid_t domain;
- struct pci_dev *dev;
- struct list_head list;
-};
-
-static DEFINE_SPINLOCK(dev_domain_list_spinlock);
-static struct list_head dev_domain_list = LIST_HEAD_INIT(dev_domain_list);
-
-static struct xen_device_domain_owner *find_device(struct pci_dev *dev)
-{
- struct xen_device_domain_owner *owner;
-
- list_for_each_entry(owner, &dev_domain_list, list) {
- if (owner->dev == dev)
- return owner;
- }
- return NULL;
-}
-
-int xen_find_device_domain_owner(struct pci_dev *dev)
-{
- struct xen_device_domain_owner *owner;
- int domain = -ENODEV;
-
- spin_lock(&dev_domain_list_spinlock);
- owner = find_device(dev);
- if (owner)
- domain = owner->domain;
- spin_unlock(&dev_domain_list_spinlock);
- return domain;
-}
-EXPORT_SYMBOL_GPL(xen_find_device_domain_owner);
-
-int xen_register_device_domain_owner(struct pci_dev *dev, uint16_t domain)
-{
- struct xen_device_domain_owner *owner;
-
- owner = kzalloc(sizeof(struct xen_device_domain_owner), GFP_KERNEL);
- if (!owner)
- return -ENODEV;
-
- spin_lock(&dev_domain_list_spinlock);
- if (find_device(dev)) {
- spin_unlock(&dev_domain_list_spinlock);
- kfree(owner);
- return -EEXIST;
- }
- owner->domain = domain;
- owner->dev = dev;
- list_add_tail(&owner->list, &dev_domain_list);
- spin_unlock(&dev_domain_list_spinlock);
- return 0;
-}
-EXPORT_SYMBOL_GPL(xen_register_device_domain_owner);
-
-int xen_unregister_device_domain_owner(struct pci_dev *dev)
-{
- struct xen_device_domain_owner *owner;
-
- spin_lock(&dev_domain_list_spinlock);
- owner = find_device(dev);
- if (!owner) {
- spin_unlock(&dev_domain_list_spinlock);
- return -ENODEV;
- }
- list_del(&owner->list);
- spin_unlock(&dev_domain_list_spinlock);
- kfree(owner);
- return 0;
-}
-EXPORT_SYMBOL_GPL(xen_unregister_device_domain_owner);
-#endif /* CONFIG_XEN_DOM0 */
* Pointer to the xen_vcpu_info structure or
* &HYPERVISOR_shared_info->vcpu_info[cpu]. See xen_hvm_init_shared_info
* and xen_vcpu_setup for details. By default it points to share_info->vcpu_info
- * but if the hypervisor supports VCPUOP_register_vcpu_info then it can point
- * to xen_vcpu_info. The pointer is used in __xen_evtchn_do_upcall to
- * acknowledge pending events.
- * Also more subtly it is used by the patched version of irq enable/disable
- * e.g. xen_irq_enable_direct and xen_iret in PV mode.
- *
- * The desire to be able to do those mask/unmask operations as a single
- * instruction by using the per-cpu offset held in %gs is the real reason
- * vcpu info is in a per-cpu pointer and the original reason for this
- * hypercall.
- *
+ * but during boot it is switched to point to xen_vcpu_info.
+ * The pointer is used in __xen_evtchn_do_upcall to acknowledge pending events.
*/
DEFINE_PER_CPU(struct vcpu_info *, xen_vcpu);
-
-/*
- * Per CPU pages used if hypervisor supports VCPUOP_register_vcpu_info
- * hypercall. This can be used both in PV and PVHVM mode. The structure
- * overrides the default per_cpu(xen_vcpu, cpu) value.
- */
DEFINE_PER_CPU(struct vcpu_info, xen_vcpu_info);
/* Linux <-> Xen vCPU id mapping */
*/
struct shared_info *HYPERVISOR_shared_info = &xen_dummy_shared_info;
-/*
- * Flag to determine whether vcpu info placement is available on all
- * VCPUs. We assume it is to start with, and then set it to zero on
- * the first failure. This is because it can succeed on some VCPUs
- * and not others, since it can involve hypervisor memory allocation,
- * or because the guest failed to guarantee all the appropriate
- * constraints on all VCPUs (ie buffer can't cross a page boundary).
- *
- * Note that any particular CPU may be using a placed vcpu structure,
- * but we can only optimise if the all are.
- *
- * 0: not available, 1: available
- */
-int xen_have_vcpu_info_placement = 1;
-
static int xen_cpu_up_online(unsigned int cpu)
{
xen_init_lock_cpu(cpu);
return rc >= 0 ? 0 : rc;
}
-static int xen_vcpu_setup_restore(int cpu)
+static void xen_vcpu_setup_restore(int cpu)
{
- int rc = 0;
-
/* Any per_cpu(xen_vcpu) is stale, so reset it */
xen_vcpu_info_reset(cpu);
* be handled by hotplug.
*/
if (xen_pv_domain() ||
- (xen_hvm_domain() && cpu_online(cpu))) {
- rc = xen_vcpu_setup(cpu);
- }
-
- return rc;
+ (xen_hvm_domain() && cpu_online(cpu)))
+ xen_vcpu_setup(cpu);
}
/*
*/
void xen_vcpu_restore(void)
{
- int cpu, rc;
+ int cpu;
for_each_possible_cpu(cpu) {
bool other_cpu = (cpu != smp_processor_id());
if (xen_pv_domain() || xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_runstate_info(cpu);
- rc = xen_vcpu_setup_restore(cpu);
- if (rc)
- pr_emerg_once("vcpu restore failed for cpu=%d err=%d. "
- "System will hang.\n", cpu, rc);
- /*
- * In case xen_vcpu_setup_restore() fails, do not bring up the
- * VCPU. This helps us avoid the resulting OOPS when the VCPU
- * accesses pvclock_vcpu_time via xen_vcpu (which is NULL.)
- * Note that this does not improve the situation much -- now the
- * VM hangs instead of OOPSing -- with the VCPUs that did not
- * fail, spinning in stop_machine(), waiting for the failed
- * VCPUs to come up.
- */
- if (other_cpu && is_up && (rc == 0) &&
+ xen_vcpu_setup_restore(cpu);
+
+ if (other_cpu && is_up &&
HYPERVISOR_vcpu_op(VCPUOP_up, xen_vcpu_nr(cpu), NULL))
BUG();
}
}
}
-int xen_vcpu_setup(int cpu)
+void xen_vcpu_setup(int cpu)
{
struct vcpu_register_vcpu_info info;
int err;
*/
if (xen_hvm_domain()) {
if (per_cpu(xen_vcpu, cpu) == &per_cpu(xen_vcpu_info, cpu))
- return 0;
+ return;
}
- if (xen_have_vcpu_info_placement) {
- vcpup = &per_cpu(xen_vcpu_info, cpu);
- info.mfn = arbitrary_virt_to_mfn(vcpup);
- info.offset = offset_in_page(vcpup);
-
- /*
- * Check to see if the hypervisor will put the vcpu_info
- * structure where we want it, which allows direct access via
- * a percpu-variable.
- * N.B. This hypercall can _only_ be called once per CPU.
- * Subsequent calls will error out with -EINVAL. This is due to
- * the fact that hypervisor has no unregister variant and this
- * hypercall does not allow to over-write info.mfn and
- * info.offset.
- */
- err = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info,
- xen_vcpu_nr(cpu), &info);
-
- if (err) {
- pr_warn_once("register_vcpu_info failed: cpu=%d err=%d\n",
- cpu, err);
- xen_have_vcpu_info_placement = 0;
- } else {
- /*
- * This cpu is using the registered vcpu info, even if
- * later ones fail to.
- */
- per_cpu(xen_vcpu, cpu) = vcpup;
- }
- }
+ vcpup = &per_cpu(xen_vcpu_info, cpu);
+ info.mfn = arbitrary_virt_to_mfn(vcpup);
+ info.offset = offset_in_page(vcpup);
- if (!xen_have_vcpu_info_placement)
- xen_vcpu_info_reset(cpu);
+ /*
+ * N.B. This hypercall can _only_ be called once per CPU.
+ * Subsequent calls will error out with -EINVAL. This is due to
+ * the fact that hypervisor has no unregister variant and this
+ * hypercall does not allow to over-write info.mfn and
+ * info.offset.
+ */
+ err = HYPERVISOR_vcpu_op(VCPUOP_register_vcpu_info, xen_vcpu_nr(cpu),
+ &info);
+ if (err)
+ panic("register_vcpu_info failed: cpu=%d err=%d\n", cpu, err);
- return ((per_cpu(xen_vcpu, cpu) == NULL) ? -ENODEV : 0);
+ per_cpu(xen_vcpu, cpu) = vcpup;
}
void __init xen_banner(void)
per_cpu(xen_vcpu_id, cpu) = cpu_acpi_id(cpu);
else
per_cpu(xen_vcpu_id, cpu) = cpu;
- rc = xen_vcpu_setup(cpu);
- if (rc || !xen_have_vector_callback)
- return rc;
+ xen_vcpu_setup(cpu);
+ if (!xen_have_vector_callback)
+ return 0;
if (xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_timer(cpu);
#include <linux/export.h>
#include <linux/mm.h>
#include <linux/page-flags.h>
-#include <linux/highmem.h>
#include <linux/pci.h>
#include <linux/gfp.h>
#include <linux/edd.h>
for_each_possible_cpu(cpu) {
/* Set up direct vCPU id mapping for PV guests. */
per_cpu(xen_vcpu_id, cpu) = cpu;
-
- /*
- * xen_vcpu_setup(cpu) can fail -- in which case it
- * falls back to the shared_info version for cpus
- * where xen_vcpu_nr(cpu) < MAX_VIRT_CPUS.
- *
- * xen_cpu_up_prepare_pv() handles the rest by failing
- * them in hotplug.
- */
- (void) xen_vcpu_setup(cpu);
+ xen_vcpu_setup(cpu);
}
- /*
- * xen_vcpu_setup managed to place the vcpu_info within the
- * percpu area for all cpus, so make use of it.
- */
- if (xen_have_vcpu_info_placement) {
- pv_ops.irq.save_fl = __PV_IS_CALLEE_SAVE(xen_save_fl_direct);
- pv_ops.irq.irq_disable =
- __PV_IS_CALLEE_SAVE(xen_irq_disable_direct);
- pv_ops.irq.irq_enable =
- __PV_IS_CALLEE_SAVE(xen_irq_enable_direct);
- pv_ops.mmu.read_cr2 =
- __PV_IS_CALLEE_SAVE(xen_read_cr2_direct);
- }
+ pv_ops.irq.save_fl = __PV_IS_CALLEE_SAVE(xen_save_fl_direct);
+ pv_ops.irq.irq_disable = __PV_IS_CALLEE_SAVE(xen_irq_disable_direct);
+ pv_ops.irq.irq_enable = __PV_IS_CALLEE_SAVE(xen_irq_enable_direct);
+ pv_ops.mmu.read_cr2 = __PV_IS_CALLEE_SAVE(xen_read_cr2_direct);
}
static const struct pv_info xen_info __initconst = {
__supported_pte_mask &= ~_PAGE_GLOBAL;
__default_kernel_pte_mask &= ~_PAGE_GLOBAL;
- /*
- * Prevent page tables from being allocated in highmem, even
- * if CONFIG_HIGHPTE is enabled.
- */
- __userpte_alloc_gfp &= ~__GFP_HIGHMEM;
-
/* Get mfn list */
xen_build_dynamic_phys_to_machine();
(void)HYPERVISOR_xen_version(0, NULL);
}
-asmlinkage __visible noinstr unsigned long xen_save_fl(void)
-{
- struct vcpu_info *vcpu;
- unsigned long flags;
-
- vcpu = this_cpu_read(xen_vcpu);
-
- /* flag has opposite sense of mask */
- flags = !vcpu->evtchn_upcall_mask;
-
- /* convert to IF type flag
- -0 -> 0x00000000
- -1 -> 0xffffffff
- */
- return (-flags) & X86_EFLAGS_IF;
-}
-__PV_CALLEE_SAVE_REGS_THUNK(xen_save_fl, ".noinstr.text");
-
-asmlinkage __visible noinstr void xen_irq_disable(void)
-{
- /* There's a one instruction preempt window here. We need to
- make sure we're don't switch CPUs between getting the vcpu
- pointer and updating the mask. */
- preempt_disable();
- this_cpu_read(xen_vcpu)->evtchn_upcall_mask = 1;
- preempt_enable_no_resched();
-}
-__PV_CALLEE_SAVE_REGS_THUNK(xen_irq_disable, ".noinstr.text");
-
-asmlinkage __visible noinstr void xen_irq_enable(void)
-{
- struct vcpu_info *vcpu;
-
- /*
- * We may be preempted as soon as vcpu->evtchn_upcall_mask is
- * cleared, so disable preemption to ensure we check for
- * events on the VCPU we are still running on.
- */
- preempt_disable();
-
- vcpu = this_cpu_read(xen_vcpu);
- vcpu->evtchn_upcall_mask = 0;
-
- /* Doesn't matter if we get preempted here, because any
- pending event will get dealt with anyway. */
-
- barrier(); /* unmask then check (avoid races) */
- if (unlikely(vcpu->evtchn_upcall_pending))
- xen_force_evtchn_callback();
-
- preempt_enable();
-}
-__PV_CALLEE_SAVE_REGS_THUNK(xen_irq_enable, ".noinstr.text");
-
static void xen_safe_halt(void)
{
/* Blocking includes an implicit local_irq_enable(). */
static const typeof(pv_ops) xen_irq_ops __initconst = {
.irq = {
-
- .save_fl = PV_CALLEE_SAVE(xen_save_fl),
- .irq_disable = PV_CALLEE_SAVE(xen_irq_disable),
- .irq_enable = PV_CALLEE_SAVE(xen_irq_enable),
+ /* Initial interrupt flag handling only called while interrupts off. */
+ .save_fl = __PV_IS_CALLEE_SAVE(paravirt_ret0),
+ .irq_disable = __PV_IS_CALLEE_SAVE(paravirt_nop),
+ .irq_enable = __PV_IS_CALLEE_SAVE(paravirt_BUG),
.safe_halt = xen_safe_halt,
.halt = xen_halt,
#ifdef CONFIG_PROC_VMCORE
/*
- * This function is used in two contexts:
- * - the kdump kernel has to check whether a pfn of the crashed kernel
- * was a ballooned page. vmcore is using this function to decide
- * whether to access a pfn of the crashed kernel.
- * - the kexec kernel has to check whether a pfn was ballooned by the
- * previous kernel. If the pfn is ballooned, handle it properly.
- * Returns 0 if the pfn is not backed by a RAM page, the caller may
+ * The kdump kernel has to check whether a pfn of the crashed kernel
+ * was a ballooned page. vmcore is using this function to decide
+ * whether to access a pfn of the crashed kernel.
+ * Returns "false" if the pfn is not backed by a RAM page, the caller may
* handle the pfn special in this case.
*/
-static int xen_oldmem_pfn_is_ram(unsigned long pfn)
+static bool xen_vmcore_pfn_is_ram(struct vmcore_cb *cb, unsigned long pfn)
{
struct xen_hvm_get_mem_type a = {
.domid = DOMID_SELF,
.pfn = pfn,
};
- int ram;
- if (HYPERVISOR_hvm_op(HVMOP_get_mem_type, &a))
- return -ENXIO;
-
- switch (a.mem_type) {
- case HVMMEM_mmio_dm:
- ram = 0;
- break;
- case HVMMEM_ram_rw:
- case HVMMEM_ram_ro:
- default:
- ram = 1;
- break;
+ if (HYPERVISOR_hvm_op(HVMOP_get_mem_type, &a)) {
+ pr_warn_once("Unexpected HVMOP_get_mem_type failure\n");
+ return true;
}
-
- return ram;
+ return a.mem_type != HVMMEM_mmio_dm;
}
+static struct vmcore_cb xen_vmcore_cb = {
+ .pfn_is_ram = xen_vmcore_pfn_is_ram,
+};
#endif
static void xen_hvm_exit_mmap(struct mm_struct *mm)
if (is_pagetable_dying_supported())
pv_ops.mmu.exit_mmap = xen_hvm_exit_mmap;
#ifdef CONFIG_PROC_VMCORE
- WARN_ON(register_oldmem_pfn_is_ram(&xen_oldmem_pfn_is_ram));
+ register_vmcore_cb(&xen_vmcore_cb);
#endif
}
* Jeremy Fitzhardinge <jeremy@xensource.com>, XenSource Inc, 2007
*/
#include <linux/sched/mm.h>
-#include <linux/highmem.h>
#include <linux/debugfs.h>
#include <linux/bug.h>
#include <linux/vmalloc.h>
#include "mmu.h"
#include "debugfs.h"
+#ifdef CONFIG_X86_VSYSCALL_EMULATION
/* l3 pud for userspace vsyscall mapping */
static pud_t level3_user_vsyscall[PTRS_PER_PUD] __page_aligned_bss;
+#endif
/*
* Protects atomic reservation decrease/increase against concurrent increases.
* Associate a virtual page frame with a given physical page frame
* and protection flags for that frame.
*/
-void set_pte_mfn(unsigned long vaddr, unsigned long mfn, pgprot_t flags)
+void __init set_pte_mfn(unsigned long vaddr, unsigned long mfn, pgprot_t flags)
{
- set_pte_vaddr(vaddr, mfn_pte(mfn, flags));
+ if (HYPERVISOR_update_va_mapping(vaddr, mfn_pte(mfn, flags),
+ UVMF_INVLPG))
+ BUG();
}
static bool xen_batched_set_pte(pte_t *ptep, pte_t pteval)
static void __init xen_after_bootmem(void)
{
static_branch_enable(&xen_struct_pages_ready);
+#ifdef CONFIG_X86_VSYSCALL_EMULATION
SetPagePinned(virt_to_page(level3_user_vsyscall));
+#endif
xen_pgd_walk(&init_mm, xen_mark_pinned, FIXADDR_TOP);
}
for (; vaddr < vaddr_end; vaddr += PAGE_SIZE)
make_lowmem_page_readwrite(vaddr);
- memblock_free(paddr, size);
+ memblock_phys_free(paddr, size);
}
static void __init xen_cleanmfnmap_free_pgtbl(void *pgtbl, bool unpin)
xen_cleanhighmap(addr, addr + size);
size = PAGE_ALIGN(xen_start_info->nr_pages *
sizeof(unsigned long));
- memblock_free(__pa(addr), size);
+ memblock_free((void *)addr, size);
} else {
xen_cleanmfnmap(addr);
}
static void __init xen_pagetable_init(void)
{
+ /*
+ * The majority of further PTE writes is to pagetables already
+ * announced as such to Xen. Hence it is more efficient to use
+ * hypercalls for these updates.
+ */
+ pv_ops.mmu.set_pte = __xen_set_pte;
+
paging_init();
xen_post_allocator_init();
*
* Many of these PTE updates are done on unpinned and writable pages
* and doing a hypercall for these is unnecessary and expensive. At
- * this point it is not possible to tell if a page is pinned or not,
- * so always write the PTE directly and rely on Xen trapping and
+ * this point it is rarely possible to tell if a page is pinned, so
+ * mostly write the PTE directly and rely on Xen trapping and
* emulating any updates as necessary.
*/
+static void __init xen_set_pte_init(pte_t *ptep, pte_t pte)
+{
+ if (unlikely(is_early_ioremap_ptep(ptep)))
+ __xen_set_pte(ptep, pte);
+ else
+ native_set_pte(ptep, pte);
+}
+
__visible pte_t xen_make_pte_init(pteval_t pte)
{
unsigned long pfn;
}
PV_CALLEE_SAVE_REGS_THUNK(xen_make_pte_init);
-static void __init xen_set_pte_init(pte_t *ptep, pte_t pte)
-{
- __xen_set_pte(ptep, pte);
-}
-
/* Early in boot, while setting up the initial pagetable, assume
everything is pinned. */
static void __init xen_alloc_pte_init(struct mm_struct *mm, unsigned long pfn)
set_page_prot(init_top_pgt, PAGE_KERNEL_RO);
set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
set_page_prot(level2_ident_pgt, PAGE_KERNEL_RO);
set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
/* Unpin Xen-provided one */
pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
+#ifdef CONFIG_X86_VSYSCALL_EMULATION
+ /* Pin user vsyscall L3 */
+ set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
+ pin_pagetable_pfn(MMUEXT_PIN_L3_TABLE,
+ PFN_DOWN(__pa_symbol(level3_user_vsyscall)));
+#endif
+
/*
* At this stage there can be no user pgd, and no page structure to
* attach it to, so make sure we just set kernel pgd.
pfn_end = p2m_pfn_end;
}
- memblock_free(PFN_PHYS(pfn), PAGE_SIZE * (pfn_end - pfn));
+ memblock_phys_free(PFN_PHYS(pfn), PAGE_SIZE * (pfn_end - pfn));
while (pfn < pfn_end) {
if (pfn == p2m_pfn) {
pfn = p2m_pfn_end;
static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
{
pte_t pte;
+ unsigned long vaddr;
phys >>= PAGE_SHIFT;
break;
}
- __native_set_fixmap(idx, pte);
+ vaddr = __fix_to_virt(idx);
+ if (HYPERVISOR_update_va_mapping(vaddr, pte, UVMF_INVLPG))
+ BUG();
#ifdef CONFIG_X86_VSYSCALL_EMULATION
/* Replicate changes to map the vsyscall page into the user
pagetable vsyscall mapping. */
- if (idx == VSYSCALL_PAGE) {
- unsigned long vaddr = __fix_to_virt(idx);
+ if (idx == VSYSCALL_PAGE)
set_pte_vaddr_pud(level3_user_vsyscall, vaddr, pte);
- }
#endif
}
static void __ref free_p2m_page(void *p)
{
if (unlikely(!slab_is_available())) {
- memblock_free((unsigned long)p, PAGE_SIZE);
+ memblock_free(p, PAGE_SIZE);
return;
}
break;
}
}
- memblock_free(PFN_PHYS(start_pfn), PFN_PHYS(n_pfns));
+ memblock_phys_free(PFN_PHYS(start_pfn), PFN_PHYS(n_pfns));
}
/*
BUG();
}
- /* Update kernel mapping, but not for highmem. */
- if (pfn >= PFN_UP(__pa(high_memory - 1)))
- return;
-
if (HYPERVISOR_update_va_mapping((unsigned long)__va(pfn << PAGE_SHIFT),
mfn_pte(mfn, PAGE_KERNEL), 0)) {
WARN(1, "Failed to update kernel mapping for mfn=%ld pfn=%ld\n",
}
/*
- * If the PFNs are currently mapped, the VA mapping also needs
- * to be updated to be 1:1.
+ * If the PFNs are currently mapped, their VA mappings need to be
+ * zapped.
*/
for (pfn = start_pfn; pfn <= max_pfn_mapped && pfn < end_pfn; pfn++)
(void)HYPERVISOR_update_va_mapping(
(unsigned long)__va(pfn << PAGE_SHIFT),
- mfn_pte(pfn, PAGE_KERNEL_IO), 0);
+ native_make_pte(0), 0);
return remap_pfn;
}
return;
xen_relocate_p2m();
- memblock_free(start, size);
+ memblock_phys_free(start, size);
}
/**
xen_phys_memcpy(new_area, start, size);
pr_info("initrd moved from [mem %#010llx-%#010llx] to [mem %#010llx-%#010llx]\n",
start, start + size, new_area, new_area + size);
- memblock_free(start, size);
+ memblock_phys_free(start, size);
boot_params.hdr.ramdisk_image = new_area;
boot_params.ext_ramdisk_image = new_area >> 32;
}
void __init xen_smp_cpus_done(unsigned int max_cpus)
{
- int cpu, rc, count = 0;
-
if (xen_hvm_domain())
native_smp_cpus_done(max_cpus);
else
calculate_max_logical_packages();
-
- if (xen_have_vcpu_info_placement)
- return;
-
- for_each_online_cpu(cpu) {
- if (xen_vcpu_nr(cpu) < MAX_VIRT_CPUS)
- continue;
-
- rc = remove_cpu(cpu);
-
- if (rc == 0) {
- /*
- * Reset vcpu_info so this cpu cannot be onlined again.
- */
- xen_vcpu_info_reset(cpu);
- count++;
- } else {
- pr_warn("%s: failed to bring CPU %d down, error %d\n",
- __func__, cpu, rc);
- }
- }
- WARN(count, "%s: brought %d CPUs offline\n", __func__, count);
}
void xen_smp_send_reschedule(int cpu)
static irqreturn_t xen_call_function_interrupt(int irq, void *dev_id)
{
- irq_enter();
generic_smp_call_function_interrupt();
inc_irq_stat(irq_call_count);
- irq_exit();
return IRQ_HANDLED;
}
static irqreturn_t xen_call_function_single_interrupt(int irq, void *dev_id)
{
- irq_enter();
generic_smp_call_function_single_interrupt();
inc_irq_stat(irq_call_count);
- irq_exit();
return IRQ_HANDLED;
}
static void __init xen_pv_smp_prepare_cpus(unsigned int max_cpus)
{
unsigned cpu;
- unsigned int i;
if (skip_ioapic_setup) {
char *m = (max_cpus == 0) ?
}
xen_init_lock_cpu(0);
- smp_store_boot_cpu_info();
- cpu_data(0).x86_max_cores = 1;
+ smp_prepare_cpus_common();
- for_each_possible_cpu(i) {
- zalloc_cpumask_var(&per_cpu(cpu_sibling_map, i), GFP_KERNEL);
- zalloc_cpumask_var(&per_cpu(cpu_core_map, i), GFP_KERNEL);
- zalloc_cpumask_var(&per_cpu(cpu_die_map, i), GFP_KERNEL);
- zalloc_cpumask_var(&per_cpu(cpu_llc_shared_map, i), GFP_KERNEL);
- }
- set_cpu_sibling_map(0);
+ cpu_data(0).x86_max_cores = 1;
speculative_store_bypass_ht_init();
static irqreturn_t xen_irq_work_interrupt(int irq, void *dev_id)
{
- irq_enter();
irq_work_run();
inc_irq_stat(apic_irq_work_irqs);
- irq_exit();
return IRQ_HANDLED;
}
/* Clear .bss */
xor %eax,%eax
- mov $__bss_start, %_ASM_DI
- mov $__bss_stop, %_ASM_CX
- sub %_ASM_DI, %_ASM_CX
- shr $__ASM_SEL(2, 3), %_ASM_CX
- rep __ASM_SIZE(stos)
+ mov $__bss_start, %rdi
+ mov $__bss_stop, %rcx
+ sub %rdi, %rcx
+ shr $3, %rcx
+ rep stosq
- mov %_ASM_SI, xen_start_info
+ mov %rsi, xen_start_info
mov initial_stack(%rip), %rsp
/* Set up %gs.
bool xen_vcpu_stolen(int vcpu);
-extern int xen_have_vcpu_info_placement;
-
-int xen_vcpu_setup(int cpu);
+void xen_vcpu_setup(int cpu);
void xen_vcpu_info_reset(int cpu);
void xen_setup_vcpu_info_placement(void);
# Copyright (C) 2014 Cadence Design Systems Inc.
#
# This file is included by the global makefile so that you can add your own
-# architecture-specific flags and dependencies. Remember to do have actions
-# for "archclean" and "archdep" for cleaning up and making dependencies for
-# this architecture
+# architecture-specific flags and dependencies.
# Core configuration.
# (Use VAR=<xtensa_config> to use another default compiler.)
.align 4
_SetupMMU:
+#if XCHAL_HAVE_WINDOWED
movi a0, 0
wsr a0, windowbase
rsync
movi a0, 1
wsr a0, windowstart
rsync
+#endif
movi a0, 0x1F
wsr a0, ps
rsync
#include <asm/regs.h>
#include <asm/asmmacro.h>
#include <asm/cacheasm.h>
+#include <asm/processor.h>
/*
* RB-Data: RedBoot data/bss
* P: Boot-Parameters
.globl __start
/* this must be the first byte of the loader! */
__start:
- entry sp, 32 # we do not intend to return
+ abi_entry(32) # we do not intend to return
_call0 _start
__start_a0:
.align 4
movi a4, 1
wsr a4, ps
rsync
-
+#if XCHAL_HAVE_WINDOWED
rsr a5, windowbase
ssl a5
sll a4, a4
wsr a4, windowstart
rsync
-
- movi a4, 0x00040000
+#endif
+ movi a4, KERNEL_PS_WOE_MASK
wsr a4, ps
rsync
+KABI_C0 mov abi_saved0, abi_arg0
+
/* copy the loader to its address
* Note: The loader itself is a very small piece, so we assume we
* don't partially overlap. We also assume (even more important)
movi a3, __image_load
sub a4, a3, a4
- add a8, a0, a4
+ add abi_arg2, a0, a4
# a1 Stack
# a8(a4) Load address of the image
- movi a6, _image_start
- movi a10, _image_end
- movi a7, 0x1000000
- sub a11, a10, a6
- movi a9, complen
- s32i a11, a9, 0
+ movi abi_arg0, _image_start
+ movi abi_arg4, _image_end
+ movi abi_arg1, 0x1000000
+ sub abi_tmp0, abi_arg4, abi_arg0
+ movi abi_arg3, complen
+ s32i abi_tmp0, abi_arg3, 0
movi a0, 0
- # a6 destination
- # a7 maximum size of destination
- # a8 source
- # a9 ptr to length
+ # abi_arg0 destination
+ # abi_arg1 maximum size of destination
+ # abi_arg2 source
+ # abi_arg3 ptr to length
.extern gunzip
- movi a4, gunzip
- beqz a4, 1f
+ movi abi_tmp0, gunzip
+ beqz abi_tmp0, 1f
- callx4 a4
+ abi_callx abi_tmp0
j 2f
- # a6 destination start
- # a7 maximum size of destination
- # a8 source start
- # a9 ptr to length
- # a10 destination end
+ # abi_arg0 destination start
+ # abi_arg1 maximum size of destination
+ # abi_arg2 source start
+ # abi_arg3 ptr to length
+ # abi_arg4 destination end
1:
- l32i a9, a8, 0
- l32i a11, a8, 4
- s32i a9, a6, 0
- s32i a11, a6, 4
- l32i a9, a8, 8
- l32i a11, a8, 12
- s32i a9, a6, 8
- s32i a11, a6, 12
- addi a6, a6, 16
- addi a8, a8, 16
- blt a6, a10, 1b
+ l32i abi_tmp0, abi_arg2, 0
+ l32i abi_tmp1, abi_arg2, 4
+ s32i abi_tmp0, abi_arg0, 0
+ s32i abi_tmp1, abi_arg0, 4
+ l32i abi_tmp0, abi_arg2, 8
+ l32i abi_tmp1, abi_arg2, 12
+ s32i abi_tmp0, abi_arg0, 8
+ s32i abi_tmp1, abi_arg0, 12
+ addi abi_arg0, abi_arg0, 16
+ addi abi_arg2, abi_arg2, 16
+ blt abi_arg0, abi_arg4, 1b
/* jump to the kernel */
# a2 Boot parameter list
+KABI_C0 mov abi_arg0, abi_saved0
movi a0, _image_start
jx a0
#define XTENSA_STACK_ALIGNMENT 16
#if defined(__XTENSA_WINDOWED_ABI__)
+
+/* Assembly instructions for windowed kernel ABI. */
+#define KABI_W
+/* Assembly instructions for call0 kernel ABI (will be ignored). */
+#define KABI_C0 #
+
#define XTENSA_FRAME_SIZE_RESERVE 16
#define XTENSA_SPILL_STACK_RESERVE 32
#define abi_ret(frame_size) retw
#define abi_ret_default retw
+ /* direct call */
+#define abi_call call4
+ /* indirect call */
+#define abi_callx callx4
+ /* outgoing call argument registers */
+#define abi_arg0 a6
+#define abi_arg1 a7
+#define abi_arg2 a8
+#define abi_arg3 a9
+#define abi_arg4 a10
+#define abi_arg5 a11
+ /* return value */
+#define abi_rv a6
+ /* registers preserved across call */
+#define abi_saved0 a2
+#define abi_saved1 a3
+
+ /* none of the above */
+#define abi_tmp0 a4
+#define abi_tmp1 a5
+
#elif defined(__XTENSA_CALL0_ABI__)
+/* Assembly instructions for windowed kernel ABI (will be ignored). */
+#define KABI_W #
+/* Assembly instructions for call0 kernel ABI. */
+#define KABI_C0
+
#define XTENSA_SPILL_STACK_RESERVE 0
#define abi_entry(frame_size) __abi_entry (frame_size)
#define abi_ret_default ret
+ /* direct call */
+#define abi_call call0
+ /* indirect call */
+#define abi_callx callx0
+ /* outgoing call argument registers */
+#define abi_arg0 a2
+#define abi_arg1 a3
+#define abi_arg2 a4
+#define abi_arg3 a5
+#define abi_arg4 a6
+#define abi_arg5 a7
+ /* return value */
+#define abi_rv a2
+ /* registers preserved across call */
+#define abi_saved0 a12
+#define abi_saved1 a13
+
+ /* none of the above */
+#define abi_tmp0 a8
+#define abi_tmp1 a9
+
#else
#error Unsupported Xtensa ABI
#endif
+#if defined(USER_SUPPORT_WINDOWED)
+/* Assembly instructions for windowed user ABI. */
+#define UABI_W
+/* Assembly instructions for call0 user ABI (will be ignored). */
+#define UABI_C0 #
+#else
+/* Assembly instructions for windowed user ABI (will be ignored). */
+#define UABI_W #
+/* Assembly instructions for call0 user ABI. */
+#define UABI_C0
+#endif
+
#define __XTENSA_HANDLER .section ".exception.text", "ax"
#endif /* _XTENSA_ASMMACRO_H */
*
* Locking interrupts looks like this:
*
- * rsil a15, TOPLEVEL
+ * rsil a14, TOPLEVEL
* <code>
- * wsr a15, PS
+ * wsr a14, PS
* rsync
*
- * Note that a15 is used here because the register allocation
+ * Note that a14 is used here because the register allocation
* done by the compiler is not guaranteed and a window overflow
* may not occur between the rsil and wsr instructions. By using
- * a15 in the rsil, the machine is guaranteed to be in a state
+ * a14 in the rsil, the machine is guaranteed to be in a state
* where no register reference will cause an overflow.
*/
unsigned int vval; \
\
__asm__ __volatile__( \
- " rsil a15, "__stringify(TOPLEVEL)"\n" \
+ " rsil a14, "__stringify(TOPLEVEL)"\n" \
" l32i %[result], %[mem]\n" \
" " #op " %[result], %[result], %[i]\n" \
" s32i %[result], %[mem]\n" \
- " wsr a15, ps\n" \
+ " wsr a14, ps\n" \
" rsync\n" \
: [result] "=&a" (vval), [mem] "+m" (*v) \
: [i] "a" (i) \
- : "a15", "memory" \
+ : "a14", "memory" \
); \
} \
unsigned int vval; \
\
__asm__ __volatile__( \
- " rsil a15,"__stringify(TOPLEVEL)"\n" \
+ " rsil a14,"__stringify(TOPLEVEL)"\n" \
" l32i %[result], %[mem]\n" \
" " #op " %[result], %[result], %[i]\n" \
" s32i %[result], %[mem]\n" \
- " wsr a15, ps\n" \
+ " wsr a14, ps\n" \
" rsync\n" \
: [result] "=&a" (vval), [mem] "+m" (*v) \
: [i] "a" (i) \
- : "a15", "memory" \
+ : "a14", "memory" \
); \
\
return vval; \
unsigned int tmp, vval; \
\
__asm__ __volatile__( \
- " rsil a15,"__stringify(TOPLEVEL)"\n" \
+ " rsil a14,"__stringify(TOPLEVEL)"\n" \
" l32i %[result], %[mem]\n" \
" " #op " %[tmp], %[result], %[i]\n" \
" s32i %[tmp], %[mem]\n" \
- " wsr a15, ps\n" \
+ " wsr a14, ps\n" \
" rsync\n" \
: [result] "=&a" (vval), [tmp] "=&a" (tmp), \
[mem] "+m" (*v) \
: [i] "a" (i) \
- : "a15", "memory" \
+ : "a14", "memory" \
); \
\
return vval; \
return new;
#else
__asm__ __volatile__(
- " rsil a15, "__stringify(TOPLEVEL)"\n"
+ " rsil a14, "__stringify(TOPLEVEL)"\n"
" l32i %[old], %[mem]\n"
" bne %[old], %[cmp], 1f\n"
" s32i %[new], %[mem]\n"
"1:\n"
- " wsr a15, ps\n"
+ " wsr a14, ps\n"
" rsync\n"
: [old] "=&a" (old), [mem] "+m" (*p)
: [cmp] "a" (old), [new] "r" (new)
- : "a15", "memory");
+ : "a14", "memory");
return old;
#endif
}
/*
* xchg_u32
*
- * Note that a15 is used here because the register allocation
+ * Note that a14 is used here because the register allocation
* done by the compiler is not guaranteed and a window overflow
* may not occur between the rsil and wsr instructions. By using
- * a15 in the rsil, the machine is guaranteed to be in a state
+ * a14 in the rsil, the machine is guaranteed to be in a state
* where no register reference will cause an overflow.
*/
#else
unsigned long tmp;
__asm__ __volatile__(
- " rsil a15, "__stringify(TOPLEVEL)"\n"
+ " rsil a14, "__stringify(TOPLEVEL)"\n"
" l32i %[tmp], %[mem]\n"
" s32i %[val], %[mem]\n"
- " wsr a15, ps\n"
+ " wsr a14, ps\n"
" rsync\n"
: [tmp] "=&a" (tmp), [mem] "+m" (*m)
: [val] "a" (val)
- : "a15", "memory");
+ : "a14", "memory");
return tmp;
#endif
}
#define XCHAL_SPANNING_WAY 0
#endif
+#if XCHAL_HAVE_WINDOWED
+#if defined(CONFIG_USER_ABI_DEFAULT) || defined(CONFIG_USER_ABI_CALL0_PROBE)
+/* Whether windowed ABI is supported in userspace. */
+#define USER_SUPPORT_WINDOWED
+#endif
+#if defined(__XTENSA_WINDOWED_ABI__) || defined(USER_SUPPORT_WINDOWED)
+/* Whether windowed ABI is supported either in userspace or in the kernel. */
+#define SUPPORT_WINDOWED
+#endif
+#endif
+
#endif
#include <asm/types.h>
#include <asm/regs.h>
-/* Assertions. */
-
-#if (XCHAL_HAVE_WINDOWED != 1)
-# error Linux requires the Xtensa Windowed Registers Option.
-#endif
-
/* Xtensa ABI requires stack alignment to be at least 16 */
#define STACK_ALIGN (XCHAL_DATA_WIDTH > 16 ? XCHAL_DATA_WIDTH : 16)
#define WSBITS (XCHAL_NUM_AREGS / 4) /* width of WINDOWSTART in bits */
#define WBBITS (XCHAL_NUM_AREGS_LOG2 - 2) /* width of WINDOWBASE in bits */
+#if defined(__XTENSA_WINDOWED_ABI__)
+#define KERNEL_PS_WOE_MASK PS_WOE_MASK
+#elif defined(__XTENSA_CALL0_ABI__)
+#define KERNEL_PS_WOE_MASK 0
+#else
+#error Unsupported xtensa ABI
+#endif
+
#ifndef __ASSEMBLY__
+#if defined(__XTENSA_WINDOWED_ABI__)
+
/* Build a valid return address for the specified call winsize.
* winsize must be 1 (call4), 2 (call8), or 3 (call12)
*/
*/
#define MAKE_PC_FROM_RA(ra,sp) (((ra) & 0x3fffffff) | ((sp) & 0xc0000000))
+#elif defined(__XTENSA_CALL0_ABI__)
+
+/* Build a valid return address for the specified call winsize.
+ * winsize must be 1 (call4), 2 (call8), or 3 (call12)
+ */
+#define MAKE_RA_FOR_CALL(ra, ws) (ra)
+
+/* Convert return address to a valid pc
+ * Note: We assume that the stack pointer is in the same 1GB ranges as the ra
+ */
+#define MAKE_PC_FROM_RA(ra, sp) (ra)
+
+#else
+#error Unsupported Xtensa ABI
+#endif
+
/* Spill slot location for the register reg in the spill area under the stack
* pointer sp. reg must be in the range [0..4).
*/
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _XTENSA_SECTIONS_H
+#define _XTENSA_SECTIONS_H
+
+#include <asm-generic/sections.h>
+
+#ifdef CONFIG_VECTORS_ADDR
+extern char _WindowVectors_text_start[];
+extern char _WindowVectors_text_end[];
+extern char _DebugInterruptVector_text_start[];
+extern char _DebugInterruptVector_text_end[];
+extern char _KernelExceptionVector_text_start[];
+extern char _KernelExceptionVector_text_end[];
+extern char _UserExceptionVector_text_start[];
+extern char _UserExceptionVector_text_end[];
+extern char _DoubleExceptionVector_text_start[];
+extern char _DoubleExceptionVector_text_end[];
+extern char _exception_text_start[];
+extern char _exception_text_end[];
+extern char _Level2InterruptVector_text_start[];
+extern char _Level2InterruptVector_text_end[];
+extern char _Level3InterruptVector_text_start[];
+extern char _Level3InterruptVector_text_end[];
+extern char _Level4InterruptVector_text_start[];
+extern char _Level4InterruptVector_text_end[];
+extern char _Level5InterruptVector_text_start[];
+extern char _Level5InterruptVector_text_end[];
+extern char _Level6InterruptVector_text_start[];
+extern char _Level6InterruptVector_text_end[];
+#endif
+#ifdef CONFIG_SMP
+extern char _SecondaryResetVector_text_start[];
+extern char _SecondaryResetVector_text_end[];
+#endif
+#ifdef CONFIG_XIP_KERNEL
+extern char _xip_start[];
+extern char _xip_end[];
+#endif
+
+#endif
args[i] = regs->areg[reg[i]];
}
-static inline void syscall_set_arguments(struct task_struct *task,
- struct pt_regs *regs,
- const unsigned long *args)
-{
- static const unsigned int reg[] = XTENSA_SYSCALL_ARGUMENT_REGS;
- unsigned int i;
-
- for (i = 0; i < 6; ++i)
- regs->areg[reg[i]] = args[i];
-}
-
asmlinkage long xtensa_rt_sigreturn(void);
asmlinkage long xtensa_shmat(int, char __user *, int);
asmlinkage long xtensa_fadvise64_64(int, int,
static inline void spill_registers(void)
{
+#if defined(__XTENSA_WINDOWED_ABI__)
#if XCHAL_NUM_AREGS > 16
__asm__ __volatile__ (
" call8 1f\n"
" mov a12, a12\n"
: : : "memory");
#endif
+#endif
}
struct debug_table {
* BE shift left / mask 0 0 X X
*/
+#if XCHAL_HAVE_WINDOWED
#define UNALIGNED_USER_EXCEPTION
+#endif
#if XCHAL_HAVE_BE
/* Rotate ws so that the current windowbase is at bit0. */
/* Assume ws = xxwww1yyyy. Rotate ws right, so that a2 = yyyyxxwww1 */
+#if defined(USER_SUPPORT_WINDOWED)
rsr a2, windowbase
rsr a3, windowstart
ssr a2
src a2, a3, a2
srli a2, a2, 32-WSBITS
s32i a2, a1, PT_WMASK # needed for restoring registers
+#else
+ movi a2, 0
+ movi a3, 1
+ s32i a2, a1, PT_WINDOWBASE
+ s32i a3, a1, PT_WINDOWSTART
+ s32i a3, a1, PT_WMASK
+#endif
/* Save only live registers. */
- _bbsi.l a2, 1, 1f
+UABI_W _bbsi.l a2, 1, 1f
s32i a4, a1, PT_AREG4
s32i a5, a1, PT_AREG5
s32i a6, a1, PT_AREG6
s32i a7, a1, PT_AREG7
- _bbsi.l a2, 2, 1f
+UABI_W _bbsi.l a2, 2, 1f
s32i a8, a1, PT_AREG8
s32i a9, a1, PT_AREG9
s32i a10, a1, PT_AREG10
s32i a11, a1, PT_AREG11
- _bbsi.l a2, 3, 1f
+UABI_W _bbsi.l a2, 3, 1f
s32i a12, a1, PT_AREG12
s32i a13, a1, PT_AREG13
s32i a14, a1, PT_AREG14
s32i a15, a1, PT_AREG15
+
+#if defined(USER_SUPPORT_WINDOWED)
_bnei a2, 1, 1f # only one valid frame?
/* Only one valid frame, skip saving regs. */
rsync
/* We are back to the original stack pointer (a1) */
-
+#endif
2: /* Now, jump to the common exception handler. */
j common_exception
s32i a3, a1, PT_SAR
s32i a2, a1, PT_ICOUNTLEVEL
+#if defined(__XTENSA_WINDOWED_ABI__)
/* Rotate ws so that the current windowbase is at bit0. */
/* Assume ws = xxwww1yyyy. Rotate ws right, so that a2 = yyyyxxwww1 */
src a2, a3, a2
srli a2, a2, 32-WSBITS
s32i a2, a1, PT_WMASK # needed for kernel_exception_exit
+#endif
/* Save only the live window-frame */
- _bbsi.l a2, 1, 1f
+KABI_W _bbsi.l a2, 1, 1f
s32i a4, a1, PT_AREG4
s32i a5, a1, PT_AREG5
s32i a6, a1, PT_AREG6
s32i a7, a1, PT_AREG7
- _bbsi.l a2, 2, 1f
+KABI_W _bbsi.l a2, 2, 1f
s32i a8, a1, PT_AREG8
s32i a9, a1, PT_AREG9
s32i a10, a1, PT_AREG10
s32i a11, a1, PT_AREG11
- _bbsi.l a2, 3, 1f
+KABI_W _bbsi.l a2, 3, 1f
s32i a12, a1, PT_AREG12
s32i a13, a1, PT_AREG13
s32i a14, a1, PT_AREG14
s32i a15, a1, PT_AREG15
+#ifdef __XTENSA_WINDOWED_ABI__
_bnei a2, 1, 1f
-
/* Copy spill slots of a0 and a1 to imitate movsp
* in order to keep exception stack continuous
*/
l32i a0, a1, PT_SIZE + 4
s32e a3, a1, -16
s32e a0, a1, -12
+#endif
1:
l32i a0, a1, PT_AREG0 # restore saved a0
wsr a0, depc
movi a3, LOCKLEVEL
.Lexception:
- movi a0, PS_WOE_MASK
- or a3, a3, a0
+KABI_W movi a0, PS_WOE_MASK
+KABI_W or a3, a3, a0
#else
addi a2, a2, -EXCCAUSE_LEVEL1_INTERRUPT
movi a0, LOCKLEVEL
extui a3, a3, PS_INTLEVEL_SHIFT, PS_INTLEVEL_WIDTH
# a3 = PS.INTLEVEL
moveqz a3, a0, a2 # a3 = LOCKLEVEL iff interrupt
- movi a2, PS_WOE_MASK
- or a3, a3, a2
+KABI_W movi a2, PS_WOE_MASK
+KABI_W or a3, a3, a2
rsr a2, exccause
#endif
*/
rsr a4, excsave1
- mov a6, a1 # pass stack frame
- mov a7, a2 # pass EXCCAUSE
addx4 a4, a2, a4
l32i a4, a4, EXC_TABLE_DEFAULT # load handler
+ mov abi_arg1, a2 # pass EXCCAUSE
+ mov abi_arg0, a1 # pass stack frame
/* Call the second-level handler */
- callx4 a4
+ abi_callx a4
/* Jump here for exception exit */
.global common_exception_return
1:
irq_save a2, a3
#ifdef CONFIG_TRACE_IRQFLAGS
- call4 trace_hardirqs_off
+ abi_call trace_hardirqs_off
#endif
/* Jump if we are returning from kernel exceptions. */
- l32i a3, a1, PT_PS
+ l32i abi_saved1, a1, PT_PS
GET_THREAD_INFO(a2, a1)
l32i a4, a2, TI_FLAGS
- _bbci.l a3, PS_UM_BIT, 6f
+ _bbci.l abi_saved1, PS_UM_BIT, 6f
/* Specific to a user exception exit:
* We need to check some flags for signal handling and rescheduling,
/* Call do_signal() */
#ifdef CONFIG_TRACE_IRQFLAGS
- call4 trace_hardirqs_on
+ abi_call trace_hardirqs_on
#endif
rsil a2, 0
- mov a6, a1
- call4 do_notify_resume # int do_notify_resume(struct pt_regs*)
+ mov abi_arg0, a1
+ abi_call do_notify_resume # int do_notify_resume(struct pt_regs*)
j 1b
3: /* Reschedule */
#ifdef CONFIG_TRACE_IRQFLAGS
- call4 trace_hardirqs_on
+ abi_call trace_hardirqs_on
#endif
rsil a2, 0
- call4 schedule # void schedule (void)
+ abi_call schedule # void schedule (void)
j 1b
#ifdef CONFIG_PREEMPTION
l32i a4, a2, TI_PRE_COUNT
bnez a4, 4f
- call4 preempt_schedule_irq
+ abi_call preempt_schedule_irq
j 4f
#endif
#if XTENSA_FAKE_NMI
.LNMIexit:
- l32i a3, a1, PT_PS
- _bbci.l a3, PS_UM_BIT, 4f
+ l32i abi_saved1, a1, PT_PS
+ _bbci.l abi_saved1, PS_UM_BIT, 4f
#endif
5:
#ifdef CONFIG_HAVE_HW_BREAKPOINT
_bbci.l a4, TIF_DB_DISABLED, 7f
- call4 restore_dbreak
+ abi_call restore_dbreak
7:
#endif
#ifdef CONFIG_DEBUG_TLB_SANITY
l32i a4, a1, PT_DEPC
bgeui a4, VALID_DOUBLE_EXCEPTION_ADDRESS, 4f
- call4 check_tlb_sanity
+ abi_call check_tlb_sanity
#endif
6:
4:
#ifdef CONFIG_TRACE_IRQFLAGS
- extui a4, a3, PS_INTLEVEL_SHIFT, PS_INTLEVEL_WIDTH
+ extui a4, abi_saved1, PS_INTLEVEL_SHIFT, PS_INTLEVEL_WIDTH
bgei a4, LOCKLEVEL, 1f
- call4 trace_hardirqs_on
+ abi_call trace_hardirqs_on
1:
#endif
/* Restore optional registers. */
l32i a2, a1, PT_SCOMPARE1
wsr a2, scompare1
#endif
- wsr a3, ps /* disable interrupts */
+ wsr abi_saved1, ps /* disable interrupts */
- _bbci.l a3, PS_UM_BIT, kernel_exception_exit
+ _bbci.l abi_saved1, PS_UM_BIT, kernel_exception_exit
user_exception_exit:
/* Restore the state of the task and return from the exception. */
+#if defined(USER_SUPPORT_WINDOWED)
/* Switch to the user thread WINDOWBASE. Save SP temporarily in DEPC */
l32i a2, a1, PT_WINDOWBASE
* frame where we had loaded a2), or at least the lower 4 bits
* (if we have restored WSBITS-1 frames).
*/
-
2:
+#else
+ movi a2, 1
+#endif
#if XCHAL_HAVE_THREADPTR
l32i a3, a1, PT_THREADPTR
wur a3, threadptr
kernel_exception_exit:
+#if defined(__XTENSA_WINDOWED_ABI__)
/* Check if we have to do a movsp.
*
* We only have to do a movsp if the previous window-frame has
*
* Note: We expect a2 to hold PT_WMASK
*/
+#else
+ movi a2, 1
+#endif
common_exception_exit:
ENTRY(unrecoverable_exception)
+#if XCHAL_HAVE_WINDOWED
movi a0, 1
movi a1, 0
wsr a0, windowstart
wsr a1, windowbase
rsync
+#endif
- movi a1, PS_WOE_MASK | LOCKLEVEL
+ movi a1, KERNEL_PS_WOE_MASK | LOCKLEVEL
wsr a1, ps
rsync
movi a0, 0
addi a1, a1, PT_REGS_OFFSET
- movi a6, unrecoverable_text
- call4 panic
+ movi abi_arg0, unrecoverable_text
+ abi_call panic
1: j 1b
__XTENSA_HANDLER
.literal_position
+#ifdef SUPPORT_WINDOWED
/*
* Fast-handler for alloca exceptions
*
8: j _WindowUnderflow8
4: j _WindowUnderflow4
ENDPROC(fast_alloca)
+#endif
#ifdef CONFIG_USER_ABI_CALL0_PROBE
/*
* Note: We assume the stack pointer is EXC_TABLE_KSTK in the fixup handler.
*/
-#ifdef CONFIG_FAST_SYSCALL_SPILL_REGISTERS
+#if defined(CONFIG_FAST_SYSCALL_SPILL_REGISTERS) && \
+ defined(USER_SUPPORT_WINDOWED)
ENTRY(fast_syscall_spill_registers)
rsr a3, excsave1
l32i a1, a3, EXC_TABLE_KSTK
- movi a4, PS_WOE_MASK | LOCKLEVEL
+ movi a4, KERNEL_PS_WOE_MASK | LOCKLEVEL
wsr a4, ps
rsync
- movi a6, SIGSEGV
- call4 do_exit
+ movi abi_arg0, SIGSEGV
+ abi_call do_exit
/* shouldn't return, so panic */
ENTRY(system_call)
+#if defined(__XTENSA_WINDOWED_ABI__)
abi_entry_default
+#elif defined(__XTENSA_CALL0_ABI__)
+ abi_entry(12)
+
+ s32i a0, sp, 0
+ s32i abi_saved0, sp, 4
+ s32i abi_saved1, sp, 8
+ mov abi_saved0, a2
+#else
+#error Unsupported Xtensa ABI
+#endif
/* regs->syscall = regs->areg[2] */
- l32i a7, a2, PT_AREG2
- s32i a7, a2, PT_SYSCALL
+ l32i a7, abi_saved0, PT_AREG2
+ s32i a7, abi_saved0, PT_SYSCALL
GET_THREAD_INFO(a4, a1)
- l32i a3, a4, TI_FLAGS
+ l32i abi_saved1, a4, TI_FLAGS
movi a4, _TIF_WORK_MASK
- and a3, a3, a4
- beqz a3, 1f
+ and abi_saved1, abi_saved1, a4
+ beqz abi_saved1, 1f
- mov a6, a2
- call4 do_syscall_trace_enter
- beqz a6, .Lsyscall_exit
- l32i a7, a2, PT_SYSCALL
+ mov abi_arg0, abi_saved0
+ abi_call do_syscall_trace_enter
+ beqz abi_rv, .Lsyscall_exit
+ l32i a7, abi_saved0, PT_SYSCALL
1:
/* syscall = sys_call_table[syscall_nr] */
movi a4, sys_call_table
movi a5, __NR_syscalls
- movi a6, -ENOSYS
+ movi abi_rv, -ENOSYS
bgeu a7, a5, 1f
addx4 a4, a7, a4
- l32i a4, a4, 0
+ l32i abi_tmp0, a4, 0
/* Load args: arg0 - arg5 are passed via regs. */
- l32i a6, a2, PT_AREG6
- l32i a7, a2, PT_AREG3
- l32i a8, a2, PT_AREG4
- l32i a9, a2, PT_AREG5
- l32i a10, a2, PT_AREG8
- l32i a11, a2, PT_AREG9
+ l32i abi_arg0, abi_saved0, PT_AREG6
+ l32i abi_arg1, abi_saved0, PT_AREG3
+ l32i abi_arg2, abi_saved0, PT_AREG4
+ l32i abi_arg3, abi_saved0, PT_AREG5
+ l32i abi_arg4, abi_saved0, PT_AREG8
+ l32i abi_arg5, abi_saved0, PT_AREG9
- callx4 a4
+ abi_callx abi_tmp0
1: /* regs->areg[2] = return_value */
- s32i a6, a2, PT_AREG2
- bnez a3, 1f
+ s32i abi_rv, abi_saved0, PT_AREG2
+ bnez abi_saved1, 1f
.Lsyscall_exit:
+#if defined(__XTENSA_WINDOWED_ABI__)
abi_ret_default
+#elif defined(__XTENSA_CALL0_ABI__)
+ l32i a0, sp, 0
+ l32i abi_saved0, sp, 4
+ l32i abi_saved1, sp, 8
+ abi_ret(12)
+#else
+#error Unsupported Xtensa ABI
+#endif
1:
- mov a6, a2
- call4 do_syscall_trace_leave
- abi_ret_default
+ mov abi_arg0, abi_saved0
+ abi_call do_syscall_trace_leave
+ j .Lsyscall_exit
ENDPROC(system_call)
ENTRY(_switch_to)
+#if defined(__XTENSA_WINDOWED_ABI__)
abi_entry(XTENSA_SPILL_STACK_RESERVE)
+#elif defined(__XTENSA_CALL0_ABI__)
+ abi_entry(16)
+ s32i a12, sp, 0
+ s32i a13, sp, 4
+ s32i a14, sp, 8
+ s32i a15, sp, 12
+#else
+#error Unsupported Xtensa ABI
+#endif
mov a11, a3 # and 'next' (a3)
l32i a4, a2, TASK_THREAD_INFO
/* Flush register file. */
+#if defined(__XTENSA_WINDOWED_ABI__)
spill_registers_kernel
+#endif
/* Set kernel stack (and leave critical section)
* Note: It's save to set it here. The stack will not be overwritten
wsr a14, ps
rsync
+#if defined(__XTENSA_WINDOWED_ABI__)
abi_ret(XTENSA_SPILL_STACK_RESERVE)
+#elif defined(__XTENSA_CALL0_ABI__)
+ l32i a12, sp, 0
+ l32i a13, sp, 4
+ l32i a14, sp, 8
+ l32i a15, sp, 12
+ abi_ret(16)
+#else
+#error Unsupported Xtensa ABI
+#endif
ENDPROC(_switch_to)
ENTRY(ret_from_fork)
/* void schedule_tail (struct task_struct *prev)
- * Note: prev is still in a6 (return value from fake call4 frame)
+ * Note: prev is still in abi_arg0 (return value from fake call frame)
*/
- call4 schedule_tail
-
- mov a6, a1
- call4 do_syscall_trace_leave
+ abi_call schedule_tail
- j common_exception_return
+ mov abi_arg0, a1
+ abi_call do_syscall_trace_leave
+ j common_exception_return
ENDPROC(ret_from_fork)
/*
* Kernel thread creation helper
- * On entry, set up by copy_thread: a2 = thread_fn, a3 = thread_fn arg
- * left from _switch_to: a6 = prev
+ * On entry, set up by copy_thread: abi_saved0 = thread_fn,
+ * abi_saved1 = thread_fn arg. Left from _switch_to: abi_arg0 = prev
*/
ENTRY(ret_from_kernel_thread)
- call4 schedule_tail
- mov a6, a3
- callx4 a2
- j common_exception_return
+ abi_call schedule_tail
+ mov abi_arg0, abi_saved1
+ abi_callx abi_saved0
+ j common_exception_return
ENDPROC(ret_from_kernel_thread)
* Kevin Chea
*/
+#include <asm/asmmacro.h>
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/cacheasm.h>
* xt-gdb to single step via DEBUG exceptions received directly
* by ocd.
*/
+#if XCHAL_HAVE_WINDOWED
movi a1, 1
movi a0, 0
wsr a1, windowstart
wsr a0, windowbase
rsync
+#endif
movi a1, LOCKLEVEL
wsr a1, ps
movi a1, start_info
l32i a1, a1, 0
- movi a2, PS_WOE_MASK | LOCKLEVEL
- # WOE=1, INTLEVEL=LOCKLEVEL, UM=0
- wsr a2, ps # (enable reg-windows; progmode stack)
+ /* Disable interrupts. */
+ /* Enable window exceptions if kernel is built with windowed ABI. */
+ movi a2, KERNEL_PS_WOE_MASK | LOCKLEVEL
+ wsr a2, ps
rsync
#ifdef CONFIG_SMP
l32i a1, a1, 0
#endif
- movi a6, 0
- xsr a6, excsave1
+ movi abi_arg0, 0
+ xsr abi_arg0, excsave1
/* init_arch kick-starts the linux kernel */
- call4 init_arch
- call4 start_kernel
+ abi_call init_arch
+ abi_call start_kernel
should_never_return:
j should_never_return
s32i a3, a2, 0
memw
- movi a6, 0
- wsr a6, excsave1
+ movi abi_arg0, 0
+ wsr abi_arg0, excsave1
- call4 secondary_start_kernel
+ abi_call secondary_start_kernel
j should_never_return
#endif /* CONFIG_SMP */
/*
* Entry condition:
*
- * a2: a0 of the caller
+ * a2: a0 of the caller in windowed ABI
+ * a10: a0 of the caller in call0 ABI
+ *
+ * In call0 ABI the function _mcount is called with the special ABI:
+ * its argument is in a10 and all the usual argument registers (a2 - a7)
+ * must be preserved in addition to callee-saved a12 - a15.
*/
ENTRY(_mcount)
-
+#if defined(__XTENSA_WINDOWED_ABI__)
abi_entry_default
movi a4, ftrace_trace_function
callx4 a4
abi_ret_default
+#elif defined(__XTENSA_CALL0_ABI__)
+ abi_entry_default
+
+ movi a9, ftrace_trace_function
+ l32i a9, a9, 0
+ movi a11, ftrace_stub
+ bne a9, a11, 1f
+ abi_ret_default
+1: abi_entry(28)
+ s32i a0, sp, 0
+ s32i a2, sp, 4
+ s32i a3, sp, 8
+ s32i a4, sp, 12
+ s32i a5, sp, 16
+ s32i a6, sp, 20
+ s32i a7, sp, 24
+ addi a2, a10, -MCOUNT_INSN_SIZE
+ callx0 a9
+ l32i a0, sp, 0
+ l32i a2, sp, 4
+ l32i a3, sp, 8
+ l32i a4, sp, 12
+ l32i a5, sp, 16
+ l32i a6, sp, 20
+ l32i a7, sp, 24
+ abi_ret(28)
+#else
+#error Unsupported Xtensa ABI
+#endif
ENDPROC(_mcount)
ENTRY(ftrace_stub)
struct thread_info *ti;
#endif
+#if defined(__XTENSA_WINDOWED_ABI__)
/* Create a call4 dummy-frame: a0 = 0, a1 = childregs. */
SPILL_SLOT(childregs, 1) = (unsigned long)childregs;
SPILL_SLOT(childregs, 0) = 0;
p->thread.sp = (unsigned long)childregs;
+#elif defined(__XTENSA_CALL0_ABI__)
+ /* Reserve 16 bytes for the _switch_to stack frame. */
+ p->thread.sp = (unsigned long)childregs - 16;
+#else
+#error Unsupported Xtensa ABI
+#endif
if (!(p->flags & (PF_KTHREAD | PF_IO_WORKER))) {
struct pt_regs *regs = current_pt_regs();
p->thread.ra = MAKE_RA_FOR_CALL(
(unsigned long)ret_from_kernel_thread, 1);
- /* pass parameters to ret_from_kernel_thread:
- * a2 = thread_fn, a3 = thread_fn arg
+ /* pass parameters to ret_from_kernel_thread: */
+#if defined(__XTENSA_WINDOWED_ABI__)
+ /*
+ * a2 = thread_fn, a3 = thread_fn arg.
+ * Window underflow will load registers from the
+ * spill slots on the stack on return from _switch_to.
*/
- SPILL_SLOT(childregs, 3) = thread_fn_arg;
SPILL_SLOT(childregs, 2) = usp_thread_fn;
+ SPILL_SLOT(childregs, 3) = thread_fn_arg;
+#elif defined(__XTENSA_CALL0_ABI__)
+ /*
+ * a12 = thread_fn, a13 = thread_fn arg.
+ * _switch_to epilogue will load registers from the stack.
+ */
+ ((unsigned long *)p->thread.sp)[0] = usp_thread_fn;
+ ((unsigned long *)p->thread.sp)[1] = thread_fn_arg;
+#else
+#error Unsupported Xtensa ABI
+#endif
/* Childregs are only used when we're going to userspace
* in which case start_thread will set them up.
#include <asm/bootparam.h>
#include <asm/kasan.h>
#include <asm/mmu_context.h>
-#include <asm/processor.h>
-#include <asm/timex.h>
-#include <asm/platform.h>
#include <asm/page.h>
-#include <asm/setup.h>
#include <asm/param.h>
+#include <asm/platform.h>
+#include <asm/processor.h>
+#include <asm/sections.h>
+#include <asm/setup.h>
#include <asm/smp.h>
#include <asm/sysmem.h>
+#include <asm/timex.h>
#if defined(CONFIG_VGA_CONSOLE) || defined(CONFIG_DUMMY_CONSOLE)
struct screen_info screen_info = {
* Initialize system. Setup memory and reserve regions.
*/
-extern char _end[];
-extern char _stext[];
-extern char _WindowVectors_text_start;
-extern char _WindowVectors_text_end;
-extern char _DebugInterruptVector_text_start;
-extern char _DebugInterruptVector_text_end;
-extern char _KernelExceptionVector_text_start;
-extern char _KernelExceptionVector_text_end;
-extern char _UserExceptionVector_text_start;
-extern char _UserExceptionVector_text_end;
-extern char _DoubleExceptionVector_text_start;
-extern char _DoubleExceptionVector_text_end;
-extern char _exception_text_start;
-extern char _exception_text_end;
-#if XCHAL_EXCM_LEVEL >= 2
-extern char _Level2InterruptVector_text_start;
-extern char _Level2InterruptVector_text_end;
-#endif
-#if XCHAL_EXCM_LEVEL >= 3
-extern char _Level3InterruptVector_text_start;
-extern char _Level3InterruptVector_text_end;
-#endif
-#if XCHAL_EXCM_LEVEL >= 4
-extern char _Level4InterruptVector_text_start;
-extern char _Level4InterruptVector_text_end;
-#endif
-#if XCHAL_EXCM_LEVEL >= 5
-extern char _Level5InterruptVector_text_start;
-extern char _Level5InterruptVector_text_end;
-#endif
-#if XCHAL_EXCM_LEVEL >= 6
-extern char _Level6InterruptVector_text_start;
-extern char _Level6InterruptVector_text_end;
-#endif
-#ifdef CONFIG_SMP
-extern char _SecondaryResetVector_text_start;
-extern char _SecondaryResetVector_text_end;
-#endif
-#ifdef CONFIG_XIP_KERNEL
-extern char _xip_start[];
-extern char _xip_end[];
-#endif
-
static inline int __init_memblock mem_reserve(unsigned long start,
unsigned long end)
{
#endif
#ifdef CONFIG_VECTORS_ADDR
- mem_reserve(__pa(&_WindowVectors_text_start),
- __pa(&_WindowVectors_text_end));
+#ifdef SUPPORT_WINDOWED
+ mem_reserve(__pa(_WindowVectors_text_start),
+ __pa(_WindowVectors_text_end));
+#endif
- mem_reserve(__pa(&_DebugInterruptVector_text_start),
- __pa(&_DebugInterruptVector_text_end));
+ mem_reserve(__pa(_DebugInterruptVector_text_start),
+ __pa(_DebugInterruptVector_text_end));
- mem_reserve(__pa(&_KernelExceptionVector_text_start),
- __pa(&_KernelExceptionVector_text_end));
+ mem_reserve(__pa(_KernelExceptionVector_text_start),
+ __pa(_KernelExceptionVector_text_end));
- mem_reserve(__pa(&_UserExceptionVector_text_start),
- __pa(&_UserExceptionVector_text_end));
+ mem_reserve(__pa(_UserExceptionVector_text_start),
+ __pa(_UserExceptionVector_text_end));
- mem_reserve(__pa(&_DoubleExceptionVector_text_start),
- __pa(&_DoubleExceptionVector_text_end));
+ mem_reserve(__pa(_DoubleExceptionVector_text_start),
+ __pa(_DoubleExceptionVector_text_end));
- mem_reserve(__pa(&_exception_text_start),
- __pa(&_exception_text_end));
+ mem_reserve(__pa(_exception_text_start),
+ __pa(_exception_text_end));
#if XCHAL_EXCM_LEVEL >= 2
- mem_reserve(__pa(&_Level2InterruptVector_text_start),
- __pa(&_Level2InterruptVector_text_end));
+ mem_reserve(__pa(_Level2InterruptVector_text_start),
+ __pa(_Level2InterruptVector_text_end));
#endif
#if XCHAL_EXCM_LEVEL >= 3
- mem_reserve(__pa(&_Level3InterruptVector_text_start),
- __pa(&_Level3InterruptVector_text_end));
+ mem_reserve(__pa(_Level3InterruptVector_text_start),
+ __pa(_Level3InterruptVector_text_end));
#endif
#if XCHAL_EXCM_LEVEL >= 4
- mem_reserve(__pa(&_Level4InterruptVector_text_start),
- __pa(&_Level4InterruptVector_text_end));
+ mem_reserve(__pa(_Level4InterruptVector_text_start),
+ __pa(_Level4InterruptVector_text_end));
#endif
#if XCHAL_EXCM_LEVEL >= 5
- mem_reserve(__pa(&_Level5InterruptVector_text_start),
- __pa(&_Level5InterruptVector_text_end));
+ mem_reserve(__pa(_Level5InterruptVector_text_start),
+ __pa(_Level5InterruptVector_text_end));
#endif
#if XCHAL_EXCM_LEVEL >= 6
- mem_reserve(__pa(&_Level6InterruptVector_text_start),
- __pa(&_Level6InterruptVector_text_end));
+ mem_reserve(__pa(_Level6InterruptVector_text_start),
+ __pa(_Level6InterruptVector_text_end));
#endif
#endif /* CONFIG_VECTORS_ADDR */
#ifdef CONFIG_SMP
- mem_reserve(__pa(&_SecondaryResetVector_text_start),
- __pa(&_SecondaryResetVector_text_end));
+ mem_reserve(__pa(_SecondaryResetVector_text_start),
+ __pa(_SecondaryResetVector_text_end));
#endif
parse_early_param();
bootmem_init();
unsigned int window[4];
};
-/*
+#if defined(USER_SUPPORT_WINDOWED)
+/*
* Flush register windows stored in pt_regs to stack.
* Returns 1 for errors.
*/
-int
+static int
flush_window_regs_user(struct pt_regs *regs)
{
const unsigned long ws = regs->windowstart;
errout:
return err;
}
+#else
+static int
+flush_window_regs_user(struct pt_regs *regs)
+{
+ return 0;
+}
+#endif
/*
* Note: We don't copy double exception 'regs', we have to finish double exc.
/* EXCCAUSE_INSTRUCTION_FETCH unhandled */
/* EXCCAUSE_LOAD_STORE_ERROR unhandled*/
{ EXCCAUSE_LEVEL1_INTERRUPT, 0, do_interrupt },
+#ifdef SUPPORT_WINDOWED
{ EXCCAUSE_ALLOCA, USER|KRNL, fast_alloca },
+#endif
/* EXCCAUSE_INTEGER_DIVIDE_BY_ZERO unhandled */
/* EXCCAUSE_PRIVILEGED unhandled */
#if XCHAL_UNALIGNED_LOAD_EXCEPTION || XCHAL_UNALIGNED_STORE_EXCEPTION
void show_regs(struct pt_regs * regs)
{
- int i, wmask;
+ int i;
show_regs_print_info(KERN_DEFAULT);
- wmask = regs->wmask & ~1;
-
for (i = 0; i < 16; i++) {
if ((i % 8) == 0)
pr_info("a%02d:", i);
DEFINE_SPINLOCK(die_lock);
-void die(const char * str, struct pt_regs * regs, long err)
+void __noreturn die(const char * str, struct pt_regs * regs, long err)
{
static int die_counter;
const char *pr = "";
xsr a0, depc # get DEPC, save a0
+#ifdef SUPPORT_WINDOWED
movi a2, WINDOW_VECTORS_VADDR
_bltu a0, a2, .Lfixup
addi a2, a2, WINDOW_VECTORS_SIZE
l32i a0, a0, EXC_TABLE_FAST_USER
jx a0
+#else
+ j .Lfixup
+#endif
+
/*
* We only allow the ITLB miss exception if we are in kernel space.
* All other exceptions are unexpected and thus unrecoverable!
l32i a0, a0, EXC_TABLE_FAST_USER
jx a0
+#ifdef SUPPORT_WINDOWED
/*
* Restart window OVERFLOW exception.
* Currently:
rsr a0, depc
rotw -3
j 1b
+#endif
ENDPROC(_DoubleExceptionVector)
+#ifdef SUPPORT_WINDOWED
+
/*
* Fixup handler for TLB miss in double exception handler for window owerflow.
* We get here with windowbase set to the window that was being spilled and
ENDPROC(window_overflow_restore_a0_fixup)
+#endif
+
/*
* Debug interrupt vector
*
irq_entry_level 5
irq_entry_level 6
+#if XCHAL_EXCM_LEVEL >= 2
+ /*
+ * Continuation of medium priority interrupt dispatch code.
+ * On entry here, a0 contains PS, and EPC2 contains saved a0:
+ */
+ __XTENSA_HANDLER
+ .align 4
+_SimulateUserKernelVectorException:
+ addi a0, a0, (1 << PS_EXCM_BIT)
+#if !XTENSA_FAKE_NMI
+ wsr a0, ps
+#endif
+ bbsi.l a0, PS_UM_BIT, 1f # branch if user mode
+ xsr a0, excsave2 # restore a0
+ j _KernelExceptionVector # simulate kernel vector exception
+1: xsr a0, excsave2 # restore a0
+ j _UserExceptionVector # simulate user vector exception
+#endif
+
/* Window overflow and underflow handlers.
* The handlers must be 64 bytes apart, first starting with the underflow
.section .WindowVectors.text, "ax"
+#ifdef SUPPORT_WINDOWED
+
/* 4-Register Window Overflow Vector (Handler) */
ENTRY_ALIGN64(_WindowOverflow4)
ENDPROC(_WindowOverflow4)
-
-#if XCHAL_EXCM_LEVEL >= 2
- /* Not a window vector - but a convenient location
- * (where we know there's space) for continuation of
- * medium priority interrupt dispatch code.
- * On entry here, a0 contains PS, and EPC2 contains saved a0:
- */
- .align 4
-_SimulateUserKernelVectorException:
- addi a0, a0, (1 << PS_EXCM_BIT)
-#if !XTENSA_FAKE_NMI
- wsr a0, ps
-#endif
- bbsi.l a0, PS_UM_BIT, 1f # branch if user mode
- xsr a0, excsave2 # restore a0
- j _KernelExceptionVector # simulate kernel vector exception
-1: xsr a0, excsave2 # restore a0
- j _UserExceptionVector # simulate user vector exception
-#endif
-
-
/* 4-Register Window Underflow Vector (Handler) */
ENTRY_ALIGN64(_WindowUnderflow4)
ENDPROC(_WindowUnderflow12)
+#endif
+
.text
. = ALIGN(PAGE_SIZE);
_vecbase = .;
+#ifdef SUPPORT_WINDOWED
SECTION_VECTOR2 (.WindowVectors.text, WINDOW_VECTORS_VADDR)
+#endif
#if XCHAL_EXCM_LEVEL >= 2
SECTION_VECTOR2 (.Level2InterruptVector.text, INTLEVEL2_VECTOR_VADDR)
#endif
__boot_reloc_table_start = ABSOLUTE(.);
#if !MERGED_VECTORS
+#ifdef SUPPORT_WINDOWED
RELOCATE_ENTRY(_WindowVectors_text,
.WindowVectors.text);
+#endif
#if XCHAL_EXCM_LEVEL >= 2
RELOCATE_ENTRY(_Level2InterruptVector_text,
.Level2InterruptVector.text);
#if !MERGED_VECTORS
/* The vectors are relocated to the real position at startup time */
+#ifdef SUPPORT_WINDOWED
SECTION_VECTOR4 (_WindowVectors_text,
.WindowVectors.text,
WINDOW_VECTORS_VADDR,
- .dummy)
+ LAST)
+#undef LAST
+#define LAST .WindowVectors.text
+#endif
SECTION_VECTOR4 (_DebugInterruptVector_text,
.DebugInterruptVector.text,
DEBUG_VECTOR_VADDR,
- .WindowVectors.text)
+ LAST)
#undef LAST
#define LAST .DebugInterruptVector.text
#if XCHAL_EXCM_LEVEL >= 2
# a9/ tmp
# a10/ tmp
# a11/ dst
-# a12/ tmp
.text
ENTRY(__strncpy_user)
bbsi.l a3, 0, .Lsrc1mod2 # if only 8-bit aligned
bbsi.l a3, 1, .Lsrc2mod4 # if only 16-bit aligned
.Lsrcaligned: # return here when src is word-aligned
- srli a12, a4, 2 # number of loop iterations with 4B per loop
+ srli a10, a4, 2 # number of loop iterations with 4B per loop
movi a9, 3
bnone a11, a9, .Laligned
j .Ldstunaligned
.byte 0 # (0 mod 4 alignment for LBEG)
.Laligned:
#if XCHAL_HAVE_LOOPS
- loopnez a12, .Loop1done
+ loopnez a10, .Loop1done
#else
- beqz a12, .Loop1done
- slli a12, a12, 2
- add a12, a12, a11 # a12 = end of last 4B chunck
+ beqz a10, .Loop1done
+ slli a10, a10, 2
+ add a10, a10, a11 # a10 = end of last 4B chunck
#endif
.Loop1:
EX(11f) l32i a9, a3, 0 # get word from src
bnone a9, a8, .Lz3 # if byte 3 is zero
addi a11, a11, 4 # advance dst pointer
#if !XCHAL_HAVE_LOOPS
- blt a11, a12, .Loop1
+ blt a11, a10, .Loop1
#endif
.Loop1done:
loopnez a4, .Lunalignedend
#else
beqz a4, .Lunalignedend
- add a12, a11, a4 # a12 = ending address
+ add a10, a11, a4 # a10 = ending address
#endif /* XCHAL_HAVE_LOOPS */
.Lnextbyte:
EX(11f) l8ui a9, a3, 0
beqz a9, .Lunalignedend
addi a11, a11, 1
#if !XCHAL_HAVE_LOOPS
- blt a11, a12, .Lnextbyte
+ blt a11, a10, .Lnextbyte
#endif
.Lunalignedend:
.text
ENTRY(__xtensa_copy_user)
- abi_entry_default
+#if !XCHAL_HAVE_LOOPS && defined(__XTENSA_CALL0_ABI__)
+#define STACK_SIZE 4
+#else
+#define STACK_SIZE 0
+#endif
+ abi_entry(STACK_SIZE)
# a2/ dst, a3/ src, a4/ len
mov a5, a2 # copy dst so that a2 is return value
mov a11, a4 # preserve original len for error case
__ssa8 a3 # set shift amount from byte offset
bnez a4, .Lsrcunaligned
movi a2, 0 # return success for len==0
- abi_ret_default
+ abi_ret(STACK_SIZE)
/*
* Destination is unaligned
#endif /* !XCHAL_HAVE_LOOPS */
.Lbytecopydone:
movi a2, 0 # return success for len bytes copied
- abi_ret_default
+ abi_ret(STACK_SIZE)
/*
* Destination and source are word-aligned.
EX(10f) s8i a6, a5, 0
.L5:
movi a2, 0 # return success for len bytes copied
- abi_ret_default
+ abi_ret(STACK_SIZE)
/*
* Destination is aligned, Source is unaligned
loopnez a7, .Loop2done
#else /* !XCHAL_HAVE_LOOPS */
beqz a7, .Loop2done
+#if defined(__XTENSA_CALL0_ABI__)
+ s32i a10, a1, 0
+ slli a10, a7, 4
+ add a10, a10, a3 # a10 = end of last 16B source chunk
+#else
slli a12, a7, 4
add a12, a12, a3 # a12 = end of last 16B source chunk
+#endif
#endif /* !XCHAL_HAVE_LOOPS */
.Loop2:
EX(10f) l32i a7, a3, 4
EX(10f) s32i a9, a5, 12
addi a5, a5, 16
#if !XCHAL_HAVE_LOOPS
+#if defined(__XTENSA_CALL0_ABI__)
+ blt a3, a10, .Loop2
+ l32i a10, a1, 0
+#else
blt a3, a12, .Loop2
+#endif
#endif /* !XCHAL_HAVE_LOOPS */
.Loop2done:
bbci.l a4, 3, .L12
EX(10f) s8i a6, a5, 0
.L15:
movi a2, 0 # return success for len bytes copied
- abi_ret_default
+ abi_ret(STACK_SIZE)
ENDPROC(__xtensa_copy_user)
10:
sub a2, a5, a2 /* a2 <-- bytes copied */
sub a2, a11, a2 /* a2 <-- bytes not copied */
- abi_ret_default
+ abi_ret(STACK_SIZE)
void
bad_page_fault(struct pt_regs *regs, unsigned long address, int sig)
{
- extern void die(const char*, struct pt_regs*, long);
+ extern void __noreturn die(const char*, struct pt_regs*, long);
const struct exception_table_entry *entry;
/* Are we prepared to handle this kernel fault? */
"address %08lx\n pc = %08lx, ra = %08lx\n",
address, regs->pc, regs->areg[0]);
die("Oops", regs, sig);
- do_exit(sig);
}
*/
ret = blk_queue_enter(q, 0);
if (ret)
- return ret;
+ goto fail;
rcu_read_lock();
spin_lock_irq(&q->queue_lock);
new_blkg = blkg_alloc(pos, q, GFP_KERNEL);
if (unlikely(!new_blkg)) {
ret = -ENOMEM;
- goto fail;
+ goto fail_exit_queue;
}
if (radix_tree_preload(GFP_KERNEL)) {
blkg_free(new_blkg);
ret = -ENOMEM;
- goto fail;
+ goto fail_exit_queue;
}
rcu_read_lock();
fail_unlock:
spin_unlock_irq(&q->queue_lock);
rcu_read_unlock();
+fail_exit_queue:
+ blk_queue_exit(q);
fail:
blkdev_put_no_open(bdev);
- blk_queue_exit(q);
/*
* If queue was bypassing, we should retry. Do so after a
* short msleep(). It isn't strictly necessary but queue
blk_queue_flag_set(QUEUE_FLAG_DEAD, q);
blk_sync_queue(q);
- if (queue_is_mq(q))
+ if (queue_is_mq(q)) {
+ blk_mq_cancel_work_sync(q);
blk_mq_exit_queue(q);
+ }
/*
* In theory, request pool of sched_tags belongs to request queue.
}
EXPORT_SYMBOL(blk_cleanup_queue);
-static bool blk_try_enter_queue(struct request_queue *q, bool pm)
-{
- rcu_read_lock();
- if (!percpu_ref_tryget_live_rcu(&q->q_usage_counter))
- goto fail;
-
- /*
- * The code that increments the pm_only counter must ensure that the
- * counter is globally visible before the queue is unfrozen.
- */
- if (blk_queue_pm_only(q) &&
- (!pm || queue_rpm_status(q) == RPM_SUSPENDED))
- goto fail_put;
-
- rcu_read_unlock();
- return true;
-
-fail_put:
- blk_queue_exit(q);
-fail:
- rcu_read_unlock();
- return false;
-}
-
/**
* blk_queue_enter() - try to increase q->q_usage_counter
* @q: request queue pointer
return 0;
}
-static inline int bio_queue_enter(struct bio *bio)
+int __bio_queue_enter(struct request_queue *q, struct bio *bio)
{
- struct request_queue *q = bdev_get_queue(bio->bi_bdev);
-
while (!blk_try_enter_queue(q, false)) {
struct gendisk *disk = bio->bi_bdev->bd_disk;
return BLK_STS_OK;
}
-static noinline_for_stack bool submit_bio_checks(struct bio *bio)
+noinline_for_stack bool submit_bio_checks(struct bio *bio)
{
struct block_device *bdev = bio->bi_bdev;
struct request_queue *q = bdev_get_queue(bdev);
if (unlikely(!current->io_context))
create_task_io_context(current, GFP_ATOMIC, q->node);
- if (blk_throtl_bio(bio)) {
- blkcg_bio_issue_init(bio);
+ if (blk_throtl_bio(bio))
return false;
- }
blk_cgroup_bio_start(bio);
blkcg_bio_issue_init(bio);
return false;
}
-static void __submit_bio(struct bio *bio)
+static void __submit_bio_fops(struct gendisk *disk, struct bio *bio)
{
- struct gendisk *disk = bio->bi_bdev->bd_disk;
-
if (unlikely(bio_queue_enter(bio) != 0))
return;
+ if (submit_bio_checks(bio) && blk_crypto_bio_prep(&bio))
+ disk->fops->submit_bio(bio);
+ blk_queue_exit(disk->queue);
+}
- if (!submit_bio_checks(bio) || !blk_crypto_bio_prep(&bio))
- goto queue_exit;
- if (!disk->fops->submit_bio) {
+static void __submit_bio(struct bio *bio)
+{
+ struct gendisk *disk = bio->bi_bdev->bd_disk;
+
+ if (!disk->fops->submit_bio)
blk_mq_submit_bio(bio);
- return;
- }
- disk->fops->submit_bio(bio);
-queue_exit:
- blk_queue_exit(disk->queue);
+ else
+ __submit_bio_fops(disk, bio);
}
/*
flush_plug_callbacks(plug, from_schedule);
if (!rq_list_empty(plug->mq_list))
blk_mq_flush_plug_list(plug, from_schedule);
- if (unlikely(!from_schedule && plug->cached_rq))
+ /*
+ * Unconditionally flush out cached requests, even if the unplug
+ * event came from schedule. Since we know hold references to the
+ * queue for cached requests, we don't want a blocked task holding
+ * up a queue freeze/quiesce event.
+ */
+ if (unlikely(!rq_list_empty(plug->cached_rq)))
blk_mq_free_plug_rqs(plug);
}
* @rq is being submitted. Analyze what needs to be done and put it on the
* right queue.
*/
-bool blk_insert_flush(struct request *rq)
+void blk_insert_flush(struct request *rq)
{
struct request_queue *q = rq->q;
unsigned long fflags = q->queue_flags; /* may change, cache */
*/
if (!policy) {
blk_mq_end_request(rq, 0);
- return true;
+ return;
}
BUG_ON(rq->bio != rq->biotail); /*assumes zero or single bio rq */
* for normal execution.
*/
if ((policy & REQ_FSEQ_DATA) &&
- !(policy & (REQ_FSEQ_PREFLUSH | REQ_FSEQ_POSTFLUSH)))
- return false;
+ !(policy & (REQ_FSEQ_PREFLUSH | REQ_FSEQ_POSTFLUSH))) {
+ blk_mq_request_bypass_insert(rq, false, true);
+ return;
+ }
/*
* @rq should go through flush machinery. Mark it part of flush
spin_lock_irq(&fq->mq_flush_lock);
blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0);
spin_unlock_irq(&fq->mq_flush_lock);
-
- return true;
}
/**
};
/**
- * disk_register_ia_ranges - register with sysfs a set of independent
- * access ranges
+ * disk_register_independent_access_ranges - register with sysfs a set of
+ * independent access ranges
* @disk: Target disk
* @new_iars: New set of independent access ranges
*
* the same queue, there should be only one such rq in a queue
*/
*same_queue_rq = true;
+
+ if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) ==
+ BIO_MERGE_OK)
+ return true;
}
- if (blk_attempt_bio_merge(q, rq, bio, nr_segs, false) == BIO_MERGE_OK)
- return true;
return false;
}
RQF_NAME(SPECIAL_PAYLOAD),
RQF_NAME(ZONE_WRITE_LOCKED),
RQF_NAME(MQ_POLL_SLEPT),
+ RQF_NAME(ELV),
};
#undef RQF_NAME
bool ret = false;
enum hctx_type type;
- if (e && e->type->ops.bio_merge)
- return e->type->ops.bio_merge(q, bio, nr_segs);
+ if (e && e->type->ops.bio_merge) {
+ ret = e->type->ops.bio_merge(q, bio, nr_segs);
+ goto out_put;
+ }
ctx = blk_mq_get_ctx(q);
hctx = blk_mq_map_queue(q, bio->bi_opf, ctx);
type = hctx->type;
if (!(hctx->flags & BLK_MQ_F_SHOULD_MERGE) ||
list_empty_careful(&ctx->rq_lists[type]))
- return false;
+ goto out_put;
/* default per sw-queue merge */
spin_lock(&ctx->lock);
ret = true;
spin_unlock(&ctx->lock);
+out_put:
return ret;
}
* busy in case of 'none' scheduler, and this way may save
* us one extra enqueue & dequeue to sw queue.
*/
- if (!hctx->dispatch_busy && !e && !run_queue_async) {
+ if (!hctx->dispatch_busy && !run_queue_async) {
blk_mq_try_issue_list_directly(hctx, list);
if (list_empty(list))
goto out;
EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_nowait);
/**
- * blk_mq_quiesce_queue() - wait until all ongoing dispatches have finished
+ * blk_mq_wait_quiesce_done() - wait until in-progress quiesce is done
* @q: request queue.
*
- * Note: this function does not prevent that the struct request end_io()
- * callback function is invoked. Once this function is returned, we make
- * sure no dispatch can happen until the queue is unquiesced via
- * blk_mq_unquiesce_queue().
+ * Note: it is driver's responsibility for making sure that quiesce has
+ * been started.
*/
-void blk_mq_quiesce_queue(struct request_queue *q)
+void blk_mq_wait_quiesce_done(struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
unsigned int i;
bool rcu = false;
- blk_mq_quiesce_queue_nowait(q);
-
queue_for_each_hw_ctx(q, hctx, i) {
if (hctx->flags & BLK_MQ_F_BLOCKING)
synchronize_srcu(hctx->srcu);
if (rcu)
synchronize_rcu();
}
+EXPORT_SYMBOL_GPL(blk_mq_wait_quiesce_done);
+
+/**
+ * blk_mq_quiesce_queue() - wait until all ongoing dispatches have finished
+ * @q: request queue.
+ *
+ * Note: this function does not prevent that the struct request end_io()
+ * callback function is invoked. Once this function is returned, we make
+ * sure no dispatch can happen until the queue is unquiesced via
+ * blk_mq_unquiesce_queue().
+ */
+void blk_mq_quiesce_queue(struct request_queue *q)
+{
+ blk_mq_quiesce_queue_nowait(q);
+ blk_mq_wait_quiesce_done(q);
+}
EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue);
/*
for (i = 0; tag_mask; i++) {
if (!(tag_mask & (1UL << i)))
continue;
- prefetch(tags->static_rqs[tag]);
tag = tag_offset + i;
+ prefetch(tags->static_rqs[tag]);
tag_mask &= ~(1UL << i);
rq = blk_mq_rq_ctx_init(data, tags, tag, alloc_time_ns);
rq_list_add(data->cached_rq, rq);
+ nr++;
}
+ /* caller already holds a reference, add for remainder */
+ percpu_ref_get_many(&data->q->q_usage_counter, nr - 1);
data->nr_tags -= nr;
return rq_list_pop(data->cached_rq);
static struct request *__blk_mq_alloc_requests(struct blk_mq_alloc_data *data)
{
struct request_queue *q = data->q;
- struct elevator_queue *e = q->elevator;
u64 alloc_time_ns = 0;
struct request *rq;
unsigned int tag;
if (data->cmd_flags & REQ_NOWAIT)
data->flags |= BLK_MQ_REQ_NOWAIT;
- if (e) {
+ if (q->elevator) {
+ struct elevator_queue *e = q->elevator;
+
+ data->rq_flags |= RQF_ELV;
+
/*
* Flush/passthrough requests are special and go directly to the
* dispatch list. Don't include reserved tags in the
retry:
data->ctx = blk_mq_get_ctx(q);
data->hctx = blk_mq_map_queue(q, data->cmd_flags, data->ctx);
- if (!e)
+ if (!(data->rq_flags & RQF_ELV))
blk_mq_tag_busy(data->hctx);
/*
.q = q,
.flags = flags,
.cmd_flags = op,
- .rq_flags = q->elevator ? RQF_ELV : 0,
.nr_tags = 1,
};
struct request *rq;
.q = q,
.flags = flags,
.cmd_flags = op,
- .rq_flags = q->elevator ? RQF_ELV : 0,
.nr_tags = 1,
};
u64 alloc_time_ns = 0;
if (!q->elevator)
blk_mq_tag_busy(data.hctx);
+ else
+ data.rq_flags |= RQF_ELV;
ret = -EWOULDBLOCK;
tag = blk_mq_get_tag(&data);
{
struct request *rq;
- while ((rq = rq_list_pop(&plug->cached_rq)) != NULL) {
- percpu_ref_get(&rq->q->q_usage_counter);
+ while ((rq = rq_list_pop(&plug->cached_rq)) != NULL)
blk_mq_free_request(rq);
- }
}
static void req_bio_endio(struct request *rq, struct bio *bio,
{
struct request_queue *q = hctx->queue;
+ /*
+ * All requests should have been marked as RQF_MQ_INFLIGHT, so
+ * update hctx->nr_active in batch
+ */
+ if (hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
+ __blk_mq_sub_active_requests(hctx, nr_tags);
+
blk_mq_put_tags(hctx->tags, tag_array, nr_tags);
percpu_ref_put_many(&q->q_usage_counter, nr_tags);
}
plug->rq_count = 0;
if (!plug->multiple_queues && !plug->has_elevator && !from_schedule) {
- blk_mq_plug_issue_direct(plug, from_schedule);
+ blk_mq_plug_issue_direct(plug, false);
if (rq_list_empty(plug->mq_list))
return;
}
return BLK_MAX_REQUEST_COUNT;
}
+static bool blk_mq_attempt_bio_merge(struct request_queue *q,
+ struct bio *bio, unsigned int nr_segs,
+ bool *same_queue_rq)
+{
+ if (!blk_queue_nomerges(q) && bio_mergeable(bio)) {
+ if (blk_attempt_plug_merge(q, bio, nr_segs, same_queue_rq))
+ return true;
+ if (blk_mq_sched_bio_merge(q, bio, nr_segs))
+ return true;
+ }
+ return false;
+}
+
+static struct request *blk_mq_get_new_requests(struct request_queue *q,
+ struct blk_plug *plug,
+ struct bio *bio,
+ unsigned int nsegs,
+ bool *same_queue_rq)
+{
+ struct blk_mq_alloc_data data = {
+ .q = q,
+ .nr_tags = 1,
+ .cmd_flags = bio->bi_opf,
+ };
+ struct request *rq;
+
+ if (blk_mq_attempt_bio_merge(q, bio, nsegs, same_queue_rq))
+ return NULL;
+
+ rq_qos_throttle(q, bio);
+
+ if (plug) {
+ data.nr_tags = plug->nr_ios;
+ plug->nr_ios = 1;
+ data.cached_rq = &plug->cached_rq;
+ }
+
+ rq = __blk_mq_alloc_requests(&data);
+ if (rq)
+ return rq;
+
+ rq_qos_cleanup(q, bio);
+ if (bio->bi_opf & REQ_NOWAIT)
+ bio_wouldblock_error(bio);
+
+ return NULL;
+}
+
+static inline bool blk_mq_can_use_cached_rq(struct request *rq, struct bio *bio)
+{
+ if (blk_mq_get_hctx_type(bio->bi_opf) != rq->mq_hctx->type)
+ return false;
+
+ if (op_is_flush(rq->cmd_flags) != op_is_flush(bio->bi_opf))
+ return false;
+
+ return true;
+}
+
+static inline struct request *blk_mq_get_request(struct request_queue *q,
+ struct blk_plug *plug,
+ struct bio *bio,
+ unsigned int nsegs,
+ bool *same_queue_rq)
+{
+ struct request *rq;
+ bool checked = false;
+
+ if (plug) {
+ rq = rq_list_peek(&plug->cached_rq);
+ if (rq && rq->q == q) {
+ if (unlikely(!submit_bio_checks(bio)))
+ return NULL;
+ if (blk_mq_attempt_bio_merge(q, bio, nsegs,
+ same_queue_rq))
+ return NULL;
+ checked = true;
+ if (!blk_mq_can_use_cached_rq(rq, bio))
+ goto fallback;
+ rq->cmd_flags = bio->bi_opf;
+ plug->cached_rq = rq_list_next(rq);
+ INIT_LIST_HEAD(&rq->queuelist);
+ rq_qos_throttle(q, bio);
+ return rq;
+ }
+ }
+
+fallback:
+ if (unlikely(bio_queue_enter(bio)))
+ return NULL;
+ if (unlikely(!checked && !submit_bio_checks(bio)))
+ goto out_put;
+ rq = blk_mq_get_new_requests(q, plug, bio, nsegs, same_queue_rq);
+ if (rq)
+ return rq;
+out_put:
+ blk_queue_exit(q);
+ return NULL;
+}
+
/**
* blk_mq_submit_bio - Create and send a request to block device.
* @bio: Bio pointer.
unsigned int nr_segs = 1;
blk_status_t ret;
+ if (unlikely(!blk_crypto_bio_prep(&bio)))
+ return;
+
blk_queue_bounce(q, &bio);
if (blk_may_split(q, bio))
__blk_queue_split(q, &bio, &nr_segs);
if (!bio_integrity_prep(bio))
- goto queue_exit;
-
- if (!blk_queue_nomerges(q) && bio_mergeable(bio)) {
- if (blk_attempt_plug_merge(q, bio, nr_segs, &same_queue_rq))
- goto queue_exit;
- if (blk_mq_sched_bio_merge(q, bio, nr_segs))
- goto queue_exit;
- }
-
- rq_qos_throttle(q, bio);
+ return;
plug = blk_mq_plug(q, bio);
- if (plug && plug->cached_rq) {
- rq = rq_list_pop(&plug->cached_rq);
- INIT_LIST_HEAD(&rq->queuelist);
- } else {
- struct blk_mq_alloc_data data = {
- .q = q,
- .nr_tags = 1,
- .cmd_flags = bio->bi_opf,
- .rq_flags = q->elevator ? RQF_ELV : 0,
- };
-
- if (plug) {
- data.nr_tags = plug->nr_ios;
- plug->nr_ios = 1;
- data.cached_rq = &plug->cached_rq;
- }
- rq = __blk_mq_alloc_requests(&data);
- if (unlikely(!rq)) {
- rq_qos_cleanup(q, bio);
- if (bio->bi_opf & REQ_NOWAIT)
- bio_wouldblock_error(bio);
- goto queue_exit;
- }
- }
+ rq = blk_mq_get_request(q, plug, bio, nr_segs, &same_queue_rq);
+ if (unlikely(!rq))
+ return;
trace_block_getrq(bio);
return;
}
- if (op_is_flush(bio->bi_opf) && blk_insert_flush(rq))
+ if (op_is_flush(bio->bi_opf)) {
+ blk_insert_flush(rq);
return;
+ }
if (plug && (q->nr_hw_queues == 1 ||
blk_mq_is_shared_tags(rq->mq_hctx->flags) ||
/* Default case. */
blk_mq_sched_insert_request(rq, false, true, true);
}
-
- return;
-queue_exit:
- blk_queue_exit(q);
}
static size_t order_to_size(unsigned int order)
struct blk_mq_hw_ctx *hctx = hctxs[j];
if (hctx) {
- __blk_mq_free_map_and_rqs(set, j);
blk_mq_exit_hctx(q, set, hctx, j);
hctxs[j] = NULL;
}
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_realloc_hw_ctxs(set, q);
if (q->nr_hw_queues != set->nr_hw_queues) {
+ int i = prev_nr_hw_queues;
+
pr_warn("Increasing nr_hw_queues to %d fails, fallback to %d\n",
nr_hw_queues, prev_nr_hw_queues);
+ for (; i < set->nr_hw_queues; i++)
+ __blk_mq_free_map_and_rqs(set, i);
+
set->nr_hw_queues = prev_nr_hw_queues;
blk_mq_map_queues(&set->map[HCTX_TYPE_DEFAULT]);
goto fallback;
}
EXPORT_SYMBOL(blk_mq_rq_cpu);
+void blk_mq_cancel_work_sync(struct request_queue *q)
+{
+ if (queue_is_mq(q)) {
+ struct blk_mq_hw_ctx *hctx;
+ int i;
+
+ cancel_delayed_work_sync(&q->requeue_work);
+
+ queue_for_each_hw_ctx(q, hctx, i)
+ cancel_delayed_work_sync(&hctx->run_work);
+ }
+}
+
static int __init blk_mq_init(void)
{
int i;
return q->queue_hw_ctx[q->tag_set->map[type].mq_map[cpu]];
}
-/*
- * blk_mq_map_queue() - map (cmd_flags,type) to hardware queue
- * @q: request queue
- * @flags: request command flags
- * @ctx: software queue cpu ctx
- */
-static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q,
- unsigned int flags,
- struct blk_mq_ctx *ctx)
+static inline enum hctx_type blk_mq_get_hctx_type(unsigned int flags)
{
enum hctx_type type = HCTX_TYPE_DEFAULT;
type = HCTX_TYPE_POLL;
else if ((flags & REQ_OP_MASK) == REQ_OP_READ)
type = HCTX_TYPE_READ;
-
- return ctx->hctxs[type];
+ return type;
+}
+
+/*
+ * blk_mq_map_queue() - map (cmd_flags,type) to hardware queue
+ * @q: request queue
+ * @flags: request command flags
+ * @ctx: software queue cpu ctx
+ */
+static inline struct blk_mq_hw_ctx *blk_mq_map_queue(struct request_queue *q,
+ unsigned int flags,
+ struct blk_mq_ctx *ctx)
+{
+ return ctx->hctxs[blk_mq_get_hctx_type(flags)];
}
/*
void blk_mq_free_plug_rqs(struct blk_plug *plug);
void blk_mq_flush_plug_list(struct blk_plug *plug, bool from_schedule);
+void blk_mq_cancel_work_sync(struct request_queue *q);
+
void blk_mq_release(struct request_queue *q);
static inline struct blk_mq_ctx *__blk_mq_get_ctx(struct request_queue *q,
blk_mq_req_flags_t flags;
unsigned int shallow_depth;
unsigned int cmd_flags;
- unsigned int rq_flags;
+ req_flags_t rq_flags;
/* allocate multiple requests/tags in one go */
unsigned int nr_tags;
atomic_inc(&hctx->nr_active);
}
-static inline void __blk_mq_dec_active_requests(struct blk_mq_hw_ctx *hctx)
+static inline void __blk_mq_sub_active_requests(struct blk_mq_hw_ctx *hctx,
+ int val)
{
if (blk_mq_is_shared_tags(hctx->flags))
- atomic_dec(&hctx->queue->nr_active_requests_shared_tags);
+ atomic_sub(val, &hctx->queue->nr_active_requests_shared_tags);
else
- atomic_dec(&hctx->nr_active);
+ atomic_sub(val, &hctx->nr_active);
+}
+
+static inline void __blk_mq_dec_active_requests(struct blk_mq_hw_ctx *hctx)
+{
+ __blk_mq_sub_active_requests(hctx, 1);
}
static inline int __blk_mq_active_requests(struct blk_mq_hw_ctx *hctx)
blk_free_queue_stats(q->stats);
- if (queue_is_mq(q)) {
- struct blk_mq_hw_ctx *hctx;
- int i;
-
- cancel_delayed_work_sync(&q->requeue_work);
-
- queue_for_each_hw_ctx(q, hctx, i)
- cancel_delayed_work_sync(&hctx->run_work);
- }
-
blk_exit_queue(q);
blk_queue_free_zone_bitmaps(q);
op = REQ_OP_ZONE_RESET;
/* Invalidate the page cache, including dirty pages. */
+ filemap_invalidate_lock(bdev->bd_inode->i_mapping);
ret = blkdev_truncate_zone_range(bdev, mode, &zrange);
if (ret)
- return ret;
+ goto fail;
break;
case BLKOPENZONE:
op = REQ_OP_ZONE_OPEN;
ret = blkdev_zone_mgmt(bdev, op, zrange.sector, zrange.nr_sectors,
GFP_KERNEL);
- /*
- * Invalidate the page cache again for zone reset: writes can only be
- * direct for zoned devices so concurrent writes would not add any page
- * to the page cache after/during reset. The page cache may be filled
- * again due to concurrent reads though and dropping the pages for
- * these is fine.
- */
- if (!ret && cmd == BLKRESETZONE)
- ret = blkdev_truncate_zone_range(bdev, mode, &zrange);
+fail:
+ if (cmd == BLKRESETZONE)
+ filemap_invalidate_unlock(bdev->bd_inode->i_mapping);
return ret;
}
void blk_freeze_queue(struct request_queue *q);
void __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic);
void blk_queue_start_drain(struct request_queue *q);
+int __bio_queue_enter(struct request_queue *q, struct bio *bio);
+bool submit_bio_checks(struct bio *bio);
+
+static inline bool blk_try_enter_queue(struct request_queue *q, bool pm)
+{
+ rcu_read_lock();
+ if (!percpu_ref_tryget_live_rcu(&q->q_usage_counter))
+ goto fail;
+
+ /*
+ * The code that increments the pm_only counter must ensure that the
+ * counter is globally visible before the queue is unfrozen.
+ */
+ if (blk_queue_pm_only(q) &&
+ (!pm || queue_rpm_status(q) == RPM_SUSPENDED))
+ goto fail_put;
+
+ rcu_read_unlock();
+ return true;
+
+fail_put:
+ blk_queue_exit(q);
+fail:
+ rcu_read_unlock();
+ return false;
+}
+
+static inline int bio_queue_enter(struct bio *bio)
+{
+ struct request_queue *q = bdev_get_queue(bio->bi_bdev);
+
+ if (blk_try_enter_queue(q, false))
+ return 0;
+ return __bio_queue_enter(q, bio);
+}
#define BIO_INLINE_VECS 4
struct bio_vec *bvec_alloc(mempool_t *pool, unsigned short *nr_vecs,
*/
#define ELV_ON_HASH(rq) ((rq)->rq_flags & RQF_HASHED)
-bool blk_insert_flush(struct request *rq);
+void blk_insert_flush(struct request *rq);
int elevator_switch_mq(struct request_queue *q,
struct elevator_type *new_e);
if (!e)
return;
+ /*
+ * We are called before adding disk, when there isn't any FS I/O,
+ * so freezing queue plus canceling dispatch work is enough to
+ * drain any dispatch activities originated from passthrough
+ * requests, then no need to quiesce queue which may add long boot
+ * latency, especially when lots of disks are involved.
+ */
blk_mq_freeze_queue(q);
- blk_mq_quiesce_queue(q);
+ blk_mq_cancel_work_sync(q);
err = blk_mq_init_sched(q, e);
- blk_mq_unquiesce_queue(q);
blk_mq_unfreeze_queue(q);
if (err) {
{
struct block_device *bdev = iocb->ki_filp->private_data;
struct inode *bd_inode = bdev->bd_inode;
- loff_t size = i_size_read(bd_inode);
+ loff_t size = bdev_nr_bytes(bdev);
struct blk_plug plug;
size_t shorted = 0;
ssize_t ret;
static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to)
{
struct block_device *bdev = iocb->ki_filp->private_data;
- loff_t size = i_size_read(bdev->bd_inode);
+ loff_t size = bdev_nr_bytes(bdev);
loff_t pos = iocb->ki_pos;
size_t shorted = 0;
ssize_t ret;
* @major: the requested major device number [1..BLKDEV_MAJOR_MAX-1]. If
* @major = 0, try to allocate any unused major number.
* @name: the name of the new block device as a zero terminated string
- * @probe: allback that is called on access to any minor number of @major
+ * @probe: pre-devtmpfs / pre-udev callback used to create disks when their
+ * pre-created device node is accessed. When a probe call uses
+ * add_disk() and it fails the driver must cleanup resources. This
+ * interface may soon be removed.
*
* The @name must be unique within the system.
*
* This function registers the partitioning information in @disk
* with the kernel.
*/
-int device_add_disk(struct device *parent, struct gendisk *disk,
- const struct attribute_group **groups)
+int __must_check device_add_disk(struct device *parent, struct gendisk *disk,
+ const struct attribute_group **groups)
{
struct device *ddev = disk_to_dev(disk);
disk->part0->bd_holder_dir =
kobject_create_and_add("holders", &ddev->kobj);
- if (!disk->part0->bd_holder_dir)
+ if (!disk->part0->bd_holder_dir) {
+ ret = -ENOMEM;
goto out_del_integrity;
+ }
disk->slave_dir = kobject_create_and_add("slaves", &ddev->kobj);
- if (!disk->slave_dir)
+ if (!disk->slave_dir) {
+ ret = -ENOMEM;
goto out_put_holder_dir;
+ }
ret = bd_register_pending_holders(disk);
if (ret < 0)
out_free_ext_minor:
if (disk->major == BLOCK_EXT_MAJOR)
blk_free_ext_minor(disk->first_minor);
- return WARN_ON_ONCE(ret); /* keep until all callers handle errors */
+ return ret;
}
EXPORT_SYMBOL(device_add_disk);
might_sleep();
WARN_ON_ONCE(disk_live(disk));
+ blk_mq_cancel_work_sync(disk->queue);
+
disk_release_events(disk);
kfree(disk->random);
xa_destroy(&disk->part_tbl);
uint64_t range[2];
uint64_t start, len;
struct request_queue *q = bdev_get_queue(bdev);
+ struct inode *inode = bdev->bd_inode;
int err;
if (!(mode & FMODE_WRITE))
if (start + len > bdev_nr_bytes(bdev))
return -EINVAL;
+ filemap_invalidate_lock(inode->i_mapping);
err = truncate_bdev_range(bdev, mode, start, start + len - 1);
if (err)
- return err;
+ goto fail;
- return blkdev_issue_discard(bdev, start >> 9, len >> 9,
- GFP_KERNEL, flags);
+ err = blkdev_issue_discard(bdev, start >> 9, len >> 9,
+ GFP_KERNEL, flags);
+
+fail:
+ filemap_invalidate_unlock(inode->i_mapping);
+ return err;
}
static int blk_ioctl_zeroout(struct block_device *bdev, fmode_t mode,
{
uint64_t range[2];
uint64_t start, end, len;
+ struct inode *inode = bdev->bd_inode;
int err;
if (!(mode & FMODE_WRITE))
return -EINVAL;
/* Invalidate the page cache, including dirty pages */
+ filemap_invalidate_lock(inode->i_mapping);
err = truncate_bdev_range(bdev, mode, start, end);
if (err)
- return err;
+ goto fail;
+
+ err = blkdev_issue_zeroout(bdev, start >> 9, len >> 9, GFP_KERNEL,
+ BLKDEV_ZERO_NOUNMAP);
- return blkdev_issue_zeroout(bdev, start >> 9, len >> 9, GFP_KERNEL,
- BLKDEV_ZERO_NOUNMAP);
+fail:
+ filemap_invalidate_unlock(inode->i_mapping);
+ return err;
}
static int put_ushort(unsigned short __user *argp, unsigned short val)
switch (class) {
case IOPRIO_CLASS_RT:
- if (!capable(CAP_SYS_NICE) && !capable(CAP_SYS_ADMIN))
+ /*
+ * Originally this only checked for CAP_SYS_ADMIN,
+ * which was implicitly allowed for pid 0 by security
+ * modules such as SELinux. Make sure we check
+ * CAP_SYS_ADMIN first to avoid a denial/avc for
+ * possibly missing CAP_SYS_NICE permission.
+ */
+ if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_NICE))
return -EPERM;
fallthrough;
/* rt has prio field too */
CFLAGS_REMOVE_aegis128-neon-inner.o += -mgeneral-regs-only
aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
endif
+# Enable <arm_neon.h>
+CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)
obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o
obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o
if (larval)
list_add(&larval->alg.cra_list, &crypto_alg_list);
+ else
+ alg->cra_flags |= CRYPTO_ALG_TESTED;
crypto_stats_init(alg);
#define ZSTD_DEF_LEVEL 3
struct zstd_ctx {
- ZSTD_CCtx *cctx;
- ZSTD_DCtx *dctx;
+ zstd_cctx *cctx;
+ zstd_dctx *dctx;
void *cwksp;
void *dwksp;
};
-static ZSTD_parameters zstd_params(void)
+static zstd_parameters zstd_params(void)
{
- return ZSTD_getParams(ZSTD_DEF_LEVEL, 0, 0);
+ return zstd_get_params(ZSTD_DEF_LEVEL, 0);
}
static int zstd_comp_init(struct zstd_ctx *ctx)
{
int ret = 0;
- const ZSTD_parameters params = zstd_params();
- const size_t wksp_size = ZSTD_CCtxWorkspaceBound(params.cParams);
+ const zstd_parameters params = zstd_params();
+ const size_t wksp_size = zstd_cctx_workspace_bound(¶ms.cParams);
ctx->cwksp = vzalloc(wksp_size);
if (!ctx->cwksp) {
goto out;
}
- ctx->cctx = ZSTD_initCCtx(ctx->cwksp, wksp_size);
+ ctx->cctx = zstd_init_cctx(ctx->cwksp, wksp_size);
if (!ctx->cctx) {
ret = -EINVAL;
goto out_free;
static int zstd_decomp_init(struct zstd_ctx *ctx)
{
int ret = 0;
- const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+ const size_t wksp_size = zstd_dctx_workspace_bound();
ctx->dwksp = vzalloc(wksp_size);
if (!ctx->dwksp) {
goto out;
}
- ctx->dctx = ZSTD_initDCtx(ctx->dwksp, wksp_size);
+ ctx->dctx = zstd_init_dctx(ctx->dwksp, wksp_size);
if (!ctx->dctx) {
ret = -EINVAL;
goto out_free;
{
size_t out_len;
struct zstd_ctx *zctx = ctx;
- const ZSTD_parameters params = zstd_params();
+ const zstd_parameters params = zstd_params();
- out_len = ZSTD_compressCCtx(zctx->cctx, dst, *dlen, src, slen, params);
- if (ZSTD_isError(out_len))
+ out_len = zstd_compress_cctx(zctx->cctx, dst, *dlen, src, slen, ¶ms);
+ if (zstd_is_error(out_len))
return -EINVAL;
*dlen = out_len;
return 0;
size_t out_len;
struct zstd_ctx *zctx = ctx;
- out_len = ZSTD_decompressDCtx(zctx->dctx, dst, *dlen, src, slen);
- if (ZSTD_isError(out_len))
+ out_len = zstd_decompress_dctx(zctx->dctx, dst, *dlen, src, slen);
+ if (zstd_is_error(out_len))
return -EINVAL;
*dlen = out_len;
return 0;
}
EXPORT_SYMBOL_GPL(acpi_storage_d3);
+/**
+ * acpi_dev_state_d0 - Tell if the device is in D0 power state
+ * @dev: Physical device the ACPI power state of which to check
+ *
+ * On a system without ACPI, return true. On a system with ACPI, return true if
+ * the current ACPI power state of the device is D0, or false otherwise.
+ *
+ * Note that the power state of a device is not well-defined after it has been
+ * passed to acpi_device_set_power() and before that function returns, so it is
+ * not valid to ask for the ACPI power state of the device in that time frame.
+ *
+ * This function is intended to be used in a driver's probe or remove
+ * function. See Documentation/firmware-guide/acpi/low-power-probe.rst for
+ * more information.
+ */
+bool acpi_dev_state_d0(struct device *dev)
+{
+ struct acpi_device *adev = ACPI_COMPANION(dev);
+
+ if (!adev)
+ return true;
+
+ return adev->power.state == ACPI_STATE_D0;
+}
+EXPORT_SYMBOL_GPL(acpi_dev_state_d0);
+
#endif /* CONFIG_PM */
module_param(ec_storm_threshold, uint, 0644);
MODULE_PARM_DESC(ec_storm_threshold, "Maxim false GPE numbers not considered as GPE storm");
-static bool ec_freeze_events __read_mostly = false;
+static bool ec_freeze_events __read_mostly;
module_param(ec_freeze_events, bool, 0644);
MODULE_PARM_DESC(ec_freeze_events, "Disabling event handling during suspend/resume");
EXPORT_SYMBOL(first_ec);
static struct acpi_ec *boot_ec;
-static bool boot_ec_is_ecdt = false;
+static bool boot_ec_is_ecdt;
static struct workqueue_struct *ec_wq;
static struct workqueue_struct *ec_query_wq;
DMI_MATCH(DMI_PRODUCT_FAMILY, "ThinkPad X1 Yoga 3rd"),
},
},
+ {
+ .ident = "HP ZHAN 66 Pro",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "HP"),
+ DMI_MATCH(DMI_PRODUCT_FAMILY, "103C_5336AN HP ZHAN 66 Pro"),
+ },
+ },
{ },
};
acpi_unbind_one(dev);
}
-
-int acpi_dev_turn_off_if_unused(struct device *dev, void *not_used)
-{
- struct acpi_device *adev = to_acpi_device(dev);
-
- /*
- * Skip device objects with device IDs, because they may be in use even
- * if they are not companions of any physical device objects.
- */
- if (adev->pnp.type.hardware_id)
- return 0;
-
- mutex_lock(&adev->physical_node_lock);
-
- /*
- * Device objects without device IDs are not in use if they have no
- * corresponding physical device objects.
- */
- if (list_empty(&adev->physical_node_list))
- acpi_device_set_power(adev, ACPI_STATE_D3_COLD);
-
- mutex_unlock(&adev->physical_node_lock);
-
- return 0;
-}
bool acpi_device_is_first_physical_node(struct acpi_device *adev,
const struct device *dev);
int acpi_bus_register_early_device(int type);
-int acpi_dev_turn_off_if_unused(struct device *dev, void *not_used);
/* --------------------------------------------------------------------------
Device Matching and Notification
acpi_status status;
u32 result, capbuf[3];
- support &= OSC_PCI_SUPPORT_MASKS;
support |= root->osc_support_set;
capbuf[OSC_QUERY_DWORD] = OSC_QUERY_ENABLE;
capbuf[OSC_SUPPORT_DWORD] = support;
- if (control) {
- *control &= OSC_PCI_CONTROL_MASKS;
- capbuf[OSC_CONTROL_DWORD] = *control | root->osc_control_set;
- } else {
- /* Run _OSC query only with existing controls. */
- capbuf[OSC_CONTROL_DWORD] = root->osc_control_set;
- }
+ capbuf[OSC_CONTROL_DWORD] = *control | root->osc_control_set;
status = acpi_pci_run_osc(root->device->handle, capbuf, &result);
if (ACPI_SUCCESS(status)) {
root->osc_support_set = support;
- if (control)
- *control = result;
+ *control = result;
}
return status;
}
-static acpi_status acpi_pci_osc_support(struct acpi_pci_root *root, u32 flags)
-{
- return acpi_pci_query_osc(root, flags, NULL);
-}
-
struct acpi_pci_root *acpi_pci_find_root(acpi_handle handle)
{
struct acpi_pci_root *root;
* _OSC bits the BIOS has granted control of, but its contents are meaningless
* on failure.
**/
-static acpi_status acpi_pci_osc_control_set(acpi_handle handle, u32 *mask, u32 req)
+static acpi_status acpi_pci_osc_control_set(acpi_handle handle, u32 *mask, u32 support)
{
+ u32 req = OSC_PCI_EXPRESS_CAPABILITY_CONTROL;
struct acpi_pci_root *root;
acpi_status status;
u32 ctrl, capbuf[3];
if (!mask)
return AE_BAD_PARAMETER;
- ctrl = *mask & OSC_PCI_CONTROL_MASKS;
- if ((ctrl & req) != req)
- return AE_TYPE;
-
root = acpi_pci_find_root(handle);
if (!root)
return AE_NOT_EXIST;
- *mask = ctrl | root->osc_control_set;
- /* No need to evaluate _OSC if the control was already granted. */
- if ((root->osc_control_set & ctrl) == ctrl)
- return AE_OK;
+ ctrl = *mask;
+ *mask |= root->osc_control_set;
/* Need to check the available controls bits before requesting them. */
- while (*mask) {
- status = acpi_pci_query_osc(root, root->osc_support_set, mask);
+ do {
+ status = acpi_pci_query_osc(root, support, mask);
if (ACPI_FAILURE(status))
return status;
if (ctrl == *mask)
decode_osc_control(root, "platform does not support",
ctrl & ~(*mask));
ctrl = *mask;
- }
+ } while (*mask);
+
+ /* No need to request _OSC if the control was already granted. */
+ if ((root->osc_control_set & ctrl) == ctrl)
+ return AE_OK;
if ((ctrl & req) != req) {
decode_osc_control(root, "not requesting control; platform does not support",
return AE_OK;
}
-static void negotiate_os_control(struct acpi_pci_root *root, int *no_aspm,
- bool is_pcie)
+static u32 calculate_support(void)
{
- u32 support, control, requested;
- acpi_status status;
- struct acpi_device *device = root->device;
- acpi_handle handle = device->handle;
-
- /*
- * Apple always return failure on _OSC calls when _OSI("Darwin") has
- * been called successfully. We know the feature set supported by the
- * platform, so avoid calling _OSC at all
- */
- if (x86_apple_machine) {
- root->osc_control_set = ~OSC_PCI_EXPRESS_PME_CONTROL;
- decode_osc_control(root, "OS assumes control of",
- root->osc_control_set);
- return;
- }
+ u32 support;
/*
* All supported architectures that use ACPI have support for
if (IS_ENABLED(CONFIG_PCIE_EDR))
support |= OSC_PCI_EDR_SUPPORT;
- decode_osc_support(root, "OS supports", support);
- status = acpi_pci_osc_support(root, support);
- if (ACPI_FAILURE(status)) {
- *no_aspm = 1;
-
- /* _OSC is optional for PCI host bridges */
- if ((status == AE_NOT_FOUND) && !is_pcie)
- return;
-
- dev_info(&device->dev, "_OSC: platform retains control of PCIe features (%s)\n",
- acpi_format_exception(status));
- return;
- }
-
- if (pcie_ports_disabled) {
- dev_info(&device->dev, "PCIe port services disabled; not requesting _OSC control\n");
- return;
- }
+ return support;
+}
- if ((support & ACPI_PCIE_REQ_SUPPORT) != ACPI_PCIE_REQ_SUPPORT) {
- decode_osc_support(root, "not requesting OS control; OS requires",
- ACPI_PCIE_REQ_SUPPORT);
- return;
- }
+static u32 calculate_control(void)
+{
+ u32 control;
control = OSC_PCI_EXPRESS_CAPABILITY_CONTROL
| OSC_PCI_EXPRESS_PME_CONTROL;
if (IS_ENABLED(CONFIG_PCIE_DPC) && IS_ENABLED(CONFIG_PCIE_EDR))
control |= OSC_PCI_EXPRESS_DPC_CONTROL;
- requested = control;
- status = acpi_pci_osc_control_set(handle, &control,
- OSC_PCI_EXPRESS_CAPABILITY_CONTROL);
+ return control;
+}
+
+static bool os_control_query_checks(struct acpi_pci_root *root, u32 support)
+{
+ struct acpi_device *device = root->device;
+
+ if (pcie_ports_disabled) {
+ dev_info(&device->dev, "PCIe port services disabled; not requesting _OSC control\n");
+ return false;
+ }
+
+ if ((support & ACPI_PCIE_REQ_SUPPORT) != ACPI_PCIE_REQ_SUPPORT) {
+ decode_osc_support(root, "not requesting OS control; OS requires",
+ ACPI_PCIE_REQ_SUPPORT);
+ return false;
+ }
+
+ return true;
+}
+
+static void negotiate_os_control(struct acpi_pci_root *root, int *no_aspm,
+ bool is_pcie)
+{
+ u32 support, control = 0, requested = 0;
+ acpi_status status;
+ struct acpi_device *device = root->device;
+ acpi_handle handle = device->handle;
+
+ /*
+ * Apple always return failure on _OSC calls when _OSI("Darwin") has
+ * been called successfully. We know the feature set supported by the
+ * platform, so avoid calling _OSC at all
+ */
+ if (x86_apple_machine) {
+ root->osc_control_set = ~OSC_PCI_EXPRESS_PME_CONTROL;
+ decode_osc_control(root, "OS assumes control of",
+ root->osc_control_set);
+ return;
+ }
+
+ support = calculate_support();
+
+ decode_osc_support(root, "OS supports", support);
+
+ if (os_control_query_checks(root, support))
+ requested = control = calculate_control();
+
+ status = acpi_pci_osc_control_set(handle, &control, support);
if (ACPI_SUCCESS(status)) {
- decode_osc_control(root, "OS now controls", control);
+ if (control)
+ decode_osc_control(root, "OS now controls", control);
+
if (acpi_gbl_FADT.boot_flags & ACPI_FADT_NO_ASPM) {
/*
* We have ASPM control, but the FADT indicates that
*no_aspm = 1;
}
} else {
- decode_osc_control(root, "OS requested", requested);
- decode_osc_control(root, "platform willing to grant", control);
- dev_info(&device->dev, "_OSC: platform retains control of PCIe features (%s)\n",
- acpi_format_exception(status));
-
/*
* We want to disable ASPM here, but aspm_disabled
* needs to remain in its state from boot so that we
* root scan.
*/
*no_aspm = 1;
+
+ /* _OSC is optional for PCI host bridges */
+ if ((status == AE_NOT_FOUND) && !is_pcie)
+ return;
+
+ if (control) {
+ decode_osc_control(root, "OS requested", requested);
+ decode_osc_control(root, "platform willing to grant", control);
+ }
+
+ dev_info(&device->dev, "_OSC: platform retains control of PCIe features (%s)\n",
+ acpi_format_exception(status));
}
}
void *handler_context, void *region_context)
{
struct intel_pmic_opregion *opregion = region_context;
- int result = 0;
+ int result = -EINVAL;
+
+ if (function == ACPI_WRITE) {
+ switch (address) {
+ case 0:
+ return AE_OK;
+ case 1:
+ opregion->ctx.addr |= (*value64 & 0xff) << 8;
+ return AE_OK;
+ case 2:
+ opregion->ctx.addr |= *value64 & 0xff;
+ return AE_OK;
+ case 3:
+ opregion->ctx.val = *value64 & 0xff;
+ return AE_OK;
+ case 4:
+ if (*value64) {
+ result = regmap_write(opregion->regmap, opregion->ctx.addr,
+ opregion->ctx.val);
+ } else {
+ result = regmap_read(opregion->regmap, opregion->ctx.addr,
+ &opregion->ctx.val);
+ }
+ opregion->ctx.addr = 0;
+ }
+ }
- switch (address) {
- case 0:
- return AE_OK;
- case 1:
- opregion->ctx.addr |= (*value64 & 0xff) << 8;
- return AE_OK;
- case 2:
- opregion->ctx.addr |= *value64 & 0xff;
+ if (function == ACPI_READ && address == 3) {
+ *value64 = opregion->ctx.val;
return AE_OK;
- case 3:
- opregion->ctx.val = *value64 & 0xff;
- return AE_OK;
- case 4:
- if (*value64) {
- result = regmap_write(opregion->regmap, opregion->ctx.addr,
- opregion->ctx.val);
- } else {
- result = regmap_read(opregion->regmap, opregion->ctx.addr,
- &opregion->ctx.val);
- if (result == 0)
- *value64 = opregion->ctx.val;
- }
- memset(&opregion->ctx, 0x00, sizeof(opregion->ctx));
}
if (result < 0) {
mutex_lock(&acpi_device_lock);
- if (dev->wakeup.prepare_count > 1) {
- dev->wakeup.prepare_count--;
+ /* Do nothing if wakeup power has not been enabled for this device. */
+ if (dev->wakeup.prepare_count <= 0)
goto out;
- }
- /* Do nothing if wakeup power has not been enabled for this device. */
- if (!dev->wakeup.prepare_count)
+ if (--dev->wakeup.prepare_count > 0)
goto out;
err = acpi_device_sleep_wake(dev, 0, 0, 0);
static void acpi_bus_get_power_flags(struct acpi_device *device)
{
+ unsigned long long dsc = ACPI_STATE_D0;
u32 i;
/* Presence of _PS0|_PR0 indicates 'power manageable' */
if (acpi_has_method(device->handle, "_DSW"))
device->power.flags.dsw_present = 1;
+ acpi_evaluate_integer(device->handle, "_DSC", NULL, &dsc);
+ device->power.state_for_enumeration = dsc;
+
/*
* Enumerate supported power management states
*/
}
}
- /*
- * Make sure that power management resources are not blocked by ACPI
- * device objects with no users.
- */
- bus_for_each_dev(&acpi_bus_type, NULL, NULL, acpi_dev_turn_off_if_unused);
-
acpi_turn_off_unused_power_resources();
acpi_scan_initialized = true;
*/
{
.callback = video_detect_force_vendor,
- .ident = "X360",
+ /* X360 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME, "X360"),
},
{
.callback = video_detect_force_vendor,
- .ident = "Asus UL30VT",
+ /* Asus UL30VT */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK Computer Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "UL30VT"),
},
{
.callback = video_detect_force_vendor,
- .ident = "Asus UL30A",
+ /* Asus UL30A */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK Computer Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "UL30A"),
},
{
.callback = video_detect_force_vendor,
- .ident = "GIGABYTE GB-BXBT-2807",
+ /* GIGABYTE GB-BXBT-2807 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "GIGABYTE"),
DMI_MATCH(DMI_PRODUCT_NAME, "GB-BXBT-2807"),
},
{
.callback = video_detect_force_vendor,
- .ident = "Sony VPCEH3U1E",
+ /* Sony VPCEH3U1E */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Sony Corporation"),
DMI_MATCH(DMI_PRODUCT_NAME, "VPCEH3U1E"),
},
},
+ {
+ .callback = video_detect_force_vendor,
+ /* Xiaomi Mi Pad 2 */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Xiaomi Inc"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Mipad2"),
+ },
+ },
/*
* These models have a working acpi_video backlight control, and using
*/
{
.callback = video_detect_force_video,
- .ident = "ThinkPad T420",
+ /* ThinkPad T420 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T420"),
},
{
.callback = video_detect_force_video,
- .ident = "ThinkPad T520",
+ /* ThinkPad T520 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T520"),
},
{
.callback = video_detect_force_video,
- .ident = "ThinkPad X201s",
+ /* ThinkPad X201s */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad X201s"),
},
{
.callback = video_detect_force_video,
- .ident = "ThinkPad X201T",
+ /* ThinkPad X201T */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad X201T"),
{
/* https://bugs.freedesktop.org/show_bug.cgi?id=81515 */
.callback = video_detect_force_video,
- .ident = "HP ENVY 15 Notebook",
+ /* HP ENVY 15 Notebook */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Hewlett-Packard"),
DMI_MATCH(DMI_PRODUCT_NAME, "HP ENVY 15 Notebook PC"),
},
{
.callback = video_detect_force_video,
- .ident = "SAMSUNG 870Z5E/880Z5E/680Z5E",
+ /* SAMSUNG 870Z5E/880Z5E/680Z5E */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME, "870Z5E/880Z5E/680Z5E"),
},
{
.callback = video_detect_force_video,
- .ident = "SAMSUNG 370R4E/370R4V/370R5E/3570RE/370R5V",
+ /* SAMSUNG 370R4E/370R4V/370R5E/3570RE/370R5V */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME,
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1186097 */
.callback = video_detect_force_video,
- .ident = "SAMSUNG 3570R/370R/470R/450R/510R/4450RV",
+ /* SAMSUNG 3570R/370R/470R/450R/510R/4450RV */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME,
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1557060 */
.callback = video_detect_force_video,
- .ident = "SAMSUNG 670Z5E",
+ /* SAMSUNG 670Z5E */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME, "670Z5E"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1094948 */
.callback = video_detect_force_video,
- .ident = "SAMSUNG 730U3E/740U3E",
+ /* SAMSUNG 730U3E/740U3E */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME, "730U3E/740U3E"),
{
/* https://bugs.freedesktop.org/show_bug.cgi?id=87286 */
.callback = video_detect_force_video,
- .ident = "SAMSUNG 900X3C/900X3D/900X3E/900X4C/900X4D",
+ /* SAMSUNG 900X3C/900X3D/900X3E/900X4C/900X4D */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME,
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1272633 */
.callback = video_detect_force_video,
- .ident = "Dell XPS14 L421X",
+ /* Dell XPS14 L421X */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "XPS L421X"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1163574 */
.callback = video_detect_force_video,
- .ident = "Dell XPS15 L521X",
+ /* Dell XPS15 L521X */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "XPS L521X"),
{
/* https://bugzilla.kernel.org/show_bug.cgi?id=108971 */
.callback = video_detect_force_video,
- .ident = "SAMSUNG 530U4E/540U4E",
+ /* SAMSUNG 530U4E/540U4E */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "SAMSUNG ELECTRONICS CO., LTD."),
DMI_MATCH(DMI_PRODUCT_NAME, "530U4E/540U4E"),
/* https://bugs.launchpad.net/bugs/1894667 */
{
.callback = video_detect_force_video,
- .ident = "HP 635 Notebook",
+ /* HP 635 Notebook */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Hewlett-Packard"),
DMI_MATCH(DMI_PRODUCT_NAME, "HP 635 Notebook PC"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1201530 */
.callback = video_detect_force_native,
- .ident = "Lenovo Ideapad S405",
+ /* Lenovo Ideapad S405 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_BOARD_NAME, "Lenovo IdeaPad S405"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1187004 */
.callback = video_detect_force_native,
- .ident = "Lenovo Ideapad Z570",
+ /* Lenovo Ideapad Z570 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_NAME, "102434U"),
},
{
.callback = video_detect_force_native,
- .ident = "Lenovo E41-25",
+ /* Lenovo E41-25 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_NAME, "81FS"),
},
{
.callback = video_detect_force_native,
- .ident = "Lenovo E41-45",
+ /* Lenovo E41-45 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
DMI_MATCH(DMI_PRODUCT_NAME, "82BK"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1217249 */
.callback = video_detect_force_native,
- .ident = "Apple MacBook Pro 12,1",
+ /* Apple MacBook Pro 12,1 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Apple Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "MacBookPro12,1"),
},
{
.callback = video_detect_force_native,
- .ident = "Dell Vostro V131",
+ /* Dell Vostro V131 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "Vostro V131"),
{
/* https://bugzilla.redhat.com/show_bug.cgi?id=1123661 */
.callback = video_detect_force_native,
- .ident = "Dell XPS 17 L702X",
+ /* Dell XPS 17 L702X */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "Dell System XPS L702X"),
},
{
.callback = video_detect_force_native,
- .ident = "Dell Precision 7510",
+ /* Dell Precision 7510 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "Precision 7510"),
},
{
.callback = video_detect_force_native,
- .ident = "Acer Aspire 5738z",
+ /* Acer Aspire 5738z */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Acer"),
DMI_MATCH(DMI_PRODUCT_NAME, "Aspire 5738"),
{
/* https://bugzilla.kernel.org/show_bug.cgi?id=207835 */
.callback = video_detect_force_native,
- .ident = "Acer TravelMate 5735Z",
+ /* Acer TravelMate 5735Z */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Acer"),
DMI_MATCH(DMI_PRODUCT_NAME, "TravelMate 5735Z"),
},
{
.callback = video_detect_force_native,
- .ident = "ASUSTeK COMPUTER INC. GA401",
+ /* ASUSTeK COMPUTER INC. GA401 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
DMI_MATCH(DMI_PRODUCT_NAME, "GA401"),
},
{
.callback = video_detect_force_native,
- .ident = "ASUSTeK COMPUTER INC. GA502",
+ /* ASUSTeK COMPUTER INC. GA502 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
DMI_MATCH(DMI_PRODUCT_NAME, "GA502"),
},
{
.callback = video_detect_force_native,
- .ident = "ASUSTeK COMPUTER INC. GA503",
+ /* ASUSTeK COMPUTER INC. GA503 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
DMI_MATCH(DMI_PRODUCT_NAME, "GA503"),
*/
{
.callback = video_detect_force_none,
- .ident = "Dell OptiPlex 9020M",
+ /* Dell OptiPlex 9020M */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc."),
DMI_MATCH(DMI_PRODUCT_NAME, "OptiPlex 9020M"),
},
{
.callback = video_detect_force_none,
- .ident = "MSI MS-7721",
+ /* MSI MS-7721 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "MSI"),
DMI_MATCH(DMI_PRODUCT_NAME, "MS-7721"),
/* AMD */
{ PCI_VDEVICE(AMD, 0x7800), board_ahci }, /* AMD Hudson-2 */
{ PCI_VDEVICE(AMD, 0x7900), board_ahci }, /* AMD CZ */
+ { PCI_VDEVICE(AMD, 0x7901), board_ahci_mobile }, /* AMD Green Sardine */
/* AMD is using RAID class only for ahci controllers */
{ PCI_VENDOR_ID_AMD, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
PCI_CLASS_STORAGE_RAID << 8, 0xffffff, board_ahci },
extern int ahci_ignore_sss;
-extern struct device_attribute *ahci_shost_attrs[];
-extern struct device_attribute *ahci_sdev_attrs[];
+extern const struct attribute_group *ahci_shost_groups[];
+extern const struct attribute_group *ahci_sdev_groups[];
/*
* This must be instantiated by the edge drivers. Read the comments
.can_queue = AHCI_MAX_CMDS, \
.sg_tablesize = AHCI_MAX_SG, \
.dma_boundary = AHCI_DMA_BOUNDARY, \
- .shost_attrs = ahci_shost_attrs, \
- .sdev_attrs = ahci_sdev_attrs, \
+ .shost_groups = ahci_shost_groups, \
+ .sdev_groups = ahci_sdev_groups, \
.change_queue_depth = ata_scsi_change_queue_depth, \
.tag_alloc_policy = BLK_TAG_ALLOC_RR, \
.slave_configure = ata_scsi_slave_config
.set_dmamode = ich_set_dmamode,
};
-static struct device_attribute *piix_sidpr_shost_attrs[] = {
- &dev_attr_link_power_management_policy,
+static struct attribute *piix_sidpr_shost_attrs[] = {
+ &dev_attr_link_power_management_policy.attr,
NULL
};
+ATTRIBUTE_GROUPS(piix_sidpr_shost);
+
static struct scsi_host_template piix_sidpr_sht = {
ATA_BMDMA_SHT(DRV_NAME),
- .shost_attrs = piix_sidpr_shost_attrs,
+ .shost_groups = piix_sidpr_shost_groups,
};
static struct ata_port_operations piix_sidpr_sata_ops = {
ahci_read_em_buffer, ahci_store_em_buffer);
static DEVICE_ATTR(em_message_supported, S_IRUGO, ahci_show_em_supported, NULL);
-struct device_attribute *ahci_shost_attrs[] = {
- &dev_attr_link_power_management_policy,
- &dev_attr_em_message_type,
- &dev_attr_em_message,
- &dev_attr_ahci_host_caps,
- &dev_attr_ahci_host_cap2,
- &dev_attr_ahci_host_version,
- &dev_attr_ahci_port_cmd,
- &dev_attr_em_buffer,
- &dev_attr_em_message_supported,
+static struct attribute *ahci_shost_attrs[] = {
+ &dev_attr_link_power_management_policy.attr,
+ &dev_attr_em_message_type.attr,
+ &dev_attr_em_message.attr,
+ &dev_attr_ahci_host_caps.attr,
+ &dev_attr_ahci_host_cap2.attr,
+ &dev_attr_ahci_host_version.attr,
+ &dev_attr_ahci_port_cmd.attr,
+ &dev_attr_em_buffer.attr,
+ &dev_attr_em_message_supported.attr,
NULL
};
-EXPORT_SYMBOL_GPL(ahci_shost_attrs);
-struct device_attribute *ahci_sdev_attrs[] = {
- &dev_attr_sw_activity,
- &dev_attr_unload_heads,
- &dev_attr_ncq_prio_supported,
- &dev_attr_ncq_prio_enable,
+static const struct attribute_group ahci_shost_attr_group = {
+ .attrs = ahci_shost_attrs
+};
+
+const struct attribute_group *ahci_shost_groups[] = {
+ &ahci_shost_attr_group,
+ NULL
+};
+EXPORT_SYMBOL_GPL(ahci_shost_groups);
+
+static struct attribute *ahci_sdev_attrs[] = {
+ &dev_attr_sw_activity.attr,
+ &dev_attr_unload_heads.attr,
+ &dev_attr_ncq_prio_supported.attr,
+ &dev_attr_ncq_prio_enable.attr,
+ NULL
+};
+
+static const struct attribute_group ahci_sdev_attr_group = {
+ .attrs = ahci_sdev_attrs
+};
+
+const struct attribute_group *ahci_sdev_groups[] = {
+ &ahci_sdev_attr_group,
NULL
};
-EXPORT_SYMBOL_GPL(ahci_sdev_attrs);
+EXPORT_SYMBOL_GPL(ahci_sdev_groups);
struct ata_port_operations ahci_ops = {
.inherits = &sata_pmp_port_ops,
EXPORT_SYMBOL_GPL(ahci_port_resume);
#ifdef CONFIG_PM
+static void ahci_handle_s2idle(struct ata_port *ap)
+{
+ void __iomem *port_mmio = ahci_port_base(ap);
+ u32 devslp;
+
+ if (pm_suspend_via_firmware())
+ return;
+ devslp = readl(port_mmio + PORT_DEVSLP);
+ if ((devslp & PORT_DEVSLP_ADSE))
+ ata_msleep(ap, devslp_idle_timeout);
+}
+
static int ahci_port_suspend(struct ata_port *ap, pm_message_t mesg)
{
const char *emsg = NULL;
ata_port_freeze(ap);
}
+ if (acpi_storage_d3(ap->host->dev))
+ ahci_handle_s2idle(ap);
+
ahci_rpm_put_port(ap);
return rc;
}
dev->horkage |= ATA_HORKAGE_NO_DMA_LOG;
goto retry;
}
- ata_dev_err(dev, "Read log page 0x%02x failed, Emask 0x%x\n",
- (unsigned int)page, err_mask);
+ ata_dev_err(dev,
+ "Read log 0x%02x page 0x%02x failed, Emask 0x%x\n",
+ (unsigned int)log, (unsigned int)page, err_mask);
}
return err_mask;
struct ata_port *ap = dev->link->ap;
unsigned int err, i;
+ if (dev->horkage & ATA_HORKAGE_NO_ID_DEV_LOG)
+ return false;
+
if (!ata_log_supported(dev, ATA_LOG_IDENTIFY_DEVICE)) {
- ata_dev_warn(dev, "ATA Identify Device Log not supported\n");
+ /*
+ * IDENTIFY DEVICE data log is defined as mandatory starting
+ * with ACS-3 (ATA version 10). Warn about the missing log
+ * for drives which implement this ATA level or above.
+ */
+ if (ata_id_major_version(dev->id) >= 10)
+ ata_dev_warn(dev,
+ "ATA Identify Device Log not supported\n");
+ dev->horkage |= ATA_HORKAGE_NO_ID_DEV_LOG;
return false;
}
struct ata_port *ap = dev->link->ap;
unsigned int err_mask;
+ if (!ata_identify_page_supported(dev, ATA_LOG_SATA_SETTINGS))
+ return;
+
err_mask = ata_read_log_page(dev,
ATA_LOG_IDENTIFY_DEVICE,
ATA_LOG_SATA_SETTINGS,
* Check device sleep capability. Get DevSlp timing variables
* from SATA Settings page of Identify Device Data Log.
*/
- if (!ata_id_has_devslp(dev->id))
+ if (!ata_id_has_devslp(dev->id) ||
+ !ata_identify_page_supported(dev, ATA_LOG_SATA_SETTINGS))
return;
err_mask = ata_read_log_page(dev,
ULONG_MAX,
};
+static const unsigned long ata_eh_revalidate_timeouts[] = {
+ 15000, /* Some drives are slow to read log pages when waking-up */
+ 15000, /* combined time till here is enough even for media access */
+ ULONG_MAX,
+};
+
static const unsigned long ata_eh_flush_timeouts[] = {
15000, /* be generous with flush */
15000, /* ditto */
ata_eh_cmd_timeout_table[ATA_EH_CMD_TIMEOUT_TABLE_SIZE] = {
{ .commands = CMDS(ATA_CMD_ID_ATA, ATA_CMD_ID_ATAPI),
.timeouts = ata_eh_identify_timeouts, },
+ { .commands = CMDS(ATA_CMD_READ_LOG_EXT, ATA_CMD_READ_LOG_DMA_EXT),
+ .timeouts = ata_eh_revalidate_timeouts, },
{ .commands = CMDS(ATA_CMD_READ_NATIVE_MAX, ATA_CMD_READ_NATIVE_MAX_EXT),
.timeouts = ata_eh_other_timeouts, },
{ .commands = CMDS(ATA_CMD_SET_MAX, ATA_CMD_SET_MAX_EXT),
ata_ncq_prio_enable_show, ata_ncq_prio_enable_store);
EXPORT_SYMBOL_GPL(dev_attr_ncq_prio_enable);
-struct device_attribute *ata_ncq_sdev_attrs[] = {
- &dev_attr_unload_heads,
- &dev_attr_ncq_prio_enable,
- &dev_attr_ncq_prio_supported,
+static struct attribute *ata_ncq_sdev_attrs[] = {
+ &dev_attr_unload_heads.attr,
+ &dev_attr_ncq_prio_enable.attr,
+ &dev_attr_ncq_prio_supported.attr,
NULL
};
-EXPORT_SYMBOL_GPL(ata_ncq_sdev_attrs);
+
+static const struct attribute_group ata_ncq_sdev_attr_group = {
+ .attrs = ata_ncq_sdev_attrs
+};
+
+const struct attribute_group *ata_ncq_sdev_groups[] = {
+ &ata_ncq_sdev_attr_group,
+ NULL
+};
+EXPORT_SYMBOL_GPL(ata_ncq_sdev_groups);
static ssize_t
ata_scsi_em_message_store(struct device *dev, struct device_attribute *attr,
rc = __ata_scsi_queuecmd(cmd, ap->link.device);
else {
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
return rc;
}
field, 0xff, 0);
}
-struct device_attribute *ata_common_sdev_attrs[] = {
- &dev_attr_unload_heads,
+static struct attribute *ata_common_sdev_attrs[] = {
+ &dev_attr_unload_heads.attr,
NULL
};
-EXPORT_SYMBOL_GPL(ata_common_sdev_attrs);
+
+static const struct attribute_group ata_common_sdev_attr_group = {
+ .attrs = ata_common_sdev_attrs
+};
+
+const struct attribute_group *ata_common_sdev_groups[] = {
+ &ata_common_sdev_attr_group,
+ NULL
+};
+EXPORT_SYMBOL_GPL(ata_common_sdev_groups);
/**
* ata_std_bios_param - generic bios head/sector/cylinder calculator used by sd.
qc = ata_qc_new_init(dev, scsi_cmd_to_rq(cmd)->tag);
if (qc) {
qc->scsicmd = cmd;
- qc->scsidone = cmd->scsi_done;
+ qc->scsidone = scsi_done;
qc->sg = scsi_sglist(cmd);
qc->n_elem = scsi_sg_count(cmd);
qc->flags |= ATA_QCFLAG_QUIET;
} else {
cmd->result = (DID_OK << 16) | SAM_STAT_TASK_SET_FULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
return qc;
early_finish:
ata_qc_free(qc);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
DPRINTK("EXIT - early finish (good or error)\n");
return 0;
err_did:
ata_qc_free(qc);
cmd->result = (DID_ERROR << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
err_mem:
DPRINTK("EXIT - internal\n");
return 0;
DPRINTK("bad CDB len=%u, scsi_op=0x%02x, max=%u\n",
scmd->cmd_len, scsi_op, dev->cdb_len);
scmd->result = DID_ERROR << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
rc = __ata_scsi_queuecmd(cmd, dev);
else {
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
spin_unlock_irqrestore(ap->lock, irq_flags);
break;
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
int ata_scsi_add_hosts(struct ata_host *host, struct scsi_host_template *sht)
*/
.max_segment_size = MAX_DBDMA_SEG,
.slave_configure = pata_macio_slave_config,
- .sdev_attrs = ata_common_sdev_attrs,
+ .sdev_groups = ata_common_sdev_groups,
.can_queue = ATA_DEF_QUEUE,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
};
}
irq = platform_get_irq(pdev, 0);
- if (irq < 0) {
- dev_err(dev, "no irq\n");
+ if (irq < 0)
return irq;
- }
if (!irq)
return -EINVAL;
.can_queue = MV_MAX_Q_DEPTH - 1,
.sg_tablesize = MV_MAX_SG_CT / 2,
.dma_boundary = MV_DMA_BOUNDARY,
- .sdev_attrs = ata_ncq_sdev_attrs,
+ .sdev_groups = ata_ncq_sdev_groups,
.change_queue_depth = ata_scsi_change_queue_depth,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
.slave_configure = ata_scsi_slave_config
.sg_tablesize = NV_ADMA_SGTBL_TOTAL_LEN,
.dma_boundary = NV_ADMA_DMA_BOUNDARY,
.slave_configure = nv_adma_slave_config,
- .sdev_attrs = ata_ncq_sdev_attrs,
+ .sdev_groups = ata_ncq_sdev_groups,
.change_queue_depth = ata_scsi_change_queue_depth,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
};
.sg_tablesize = LIBATA_MAX_PRD,
.dma_boundary = ATA_DMA_BOUNDARY,
.slave_configure = nv_swncq_slave_config,
- .sdev_attrs = ata_ncq_sdev_attrs,
+ .sdev_groups = ata_ncq_sdev_groups,
.change_queue_depth = ata_scsi_change_queue_depth,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
};
.sg_tablesize = SIL24_MAX_SGE,
.dma_boundary = ATA_DMA_BOUNDARY,
.tag_alloc_policy = BLK_TAG_ALLOC_FIFO,
- .sdev_attrs = ata_ncq_sdev_attrs,
+ .sdev_groups = ata_ncq_sdev_groups,
.change_queue_depth = ata_scsi_change_queue_depth,
.slave_configure = ata_scsi_slave_config
};
This is some character LCD core interface that multiple drivers can
use.
+config LINEDISP
+ tristate "Character line display core support" if COMPILE_TEST
+ help
+ This is the core support for single-line character displays, to be
+ selected by drivers that use it.
+
config HD44780_COMMON
tristate "Common functions for HD44780 (and compatibles) LCD displays" if COMPILE_TEST
select CHARLCD
depends on HAS_IOMEM
default y if MIPS_MALTA
select MFD_SYSCON
+ select LINEDISP
help
Enable this to support the simple ASCII LCD displays found on
development boards such as the MIPS Boston, MIPS Malta & MIPS SEAD3
config HT16K33
tristate "Holtek Ht16K33 LED controller with keyscan"
- depends on FB && OF && I2C && INPUT
+ depends on FB && I2C && INPUT
select FB_SYS_FOPS
select FB_SYS_FILLRECT
select FB_SYS_COPYAREA
select FB_SYS_IMAGEBLIT
select INPUT_MATRIXKMAP
select FB_BACKLIGHT
+ select NEW_LEDS
+ select LEDS_CLASS
+ select LINEDISP
help
Say yes here to add support for Holtek HT16K33, RAM mapping 16*8
LED controller driver with keyscan.
obj-$(CONFIG_HT16K33) += ht16k33.o
obj-$(CONFIG_PARPORT_PANEL) += panel.o
obj-$(CONFIG_LCD2S) += lcd2s.o
+obj-$(CONFIG_LINEDISP) += line-display.o
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
-#include <linux/delay.h>
#include <linux/errno.h>
#include <linux/fb.h>
#include <linux/mm.h>
#include <linux/platform_device.h>
-#include <linux/string.h>
-#include <linux/uaccess.h>
#include <linux/cfag12864b.h>
#define CFAG12864BFB_NAME "cfag12864bfb"
.yres_virtual = CFAG12864B_HEIGHT,
.bits_per_pixel = 1,
.red = { 0, 1, 0 },
- .green = { 0, 1, 0 },
- .blue = { 0, 1, 0 },
+ .green = { 0, 1, 0 },
+ .blue = { 0, 1, 0 },
.left_margin = 0,
.right_margin = 0,
.upper_margin = 0,
static int cfag12864bfb_probe(struct platform_device *device)
{
int ret = -EINVAL;
- struct fb_info *info = framebuffer_alloc(0, &device->dev);
+ struct fb_info *info = framebuffer_alloc(0, &device->dev);
if (!info)
goto none;
* Author: Robin van der Gracht <robin@protonic.nl>
*
* Copyright: (C) 2016 Protonic Holland.
+ * Copyright (C) 2021 Glider bv
*/
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/interrupt.h>
#include <linux/i2c.h>
-#include <linux/of.h>
+#include <linux/property.h>
#include <linux/fb.h>
-#include <linux/slab.h>
#include <linux/backlight.h>
#include <linux/input.h>
#include <linux/input/matrix_keypad.h>
+#include <linux/leds.h>
#include <linux/workqueue.h>
#include <linux/mm.h>
+#include <linux/map_to_7segment.h>
+#include <linux/map_to_14segment.h>
+
+#include <asm/unaligned.h>
+
+#include "line-display.h"
+
/* Registers */
#define REG_SYSTEM_SETUP 0x20
#define REG_SYSTEM_SETUP_OSC_ON BIT(0)
#define REG_DISPLAY_SETUP 0x80
#define REG_DISPLAY_SETUP_ON BIT(0)
+#define REG_DISPLAY_SETUP_BLINK_OFF (0 << 1)
+#define REG_DISPLAY_SETUP_BLINK_2HZ (1 << 1)
+#define REG_DISPLAY_SETUP_BLINK_1HZ (2 << 1)
+#define REG_DISPLAY_SETUP_BLINK_0HZ5 (3 << 1)
#define REG_ROWINT_SET 0xA0
#define REG_ROWINT_SET_INT_EN BIT(0)
#define BYTES_PER_ROW (HT16K33_MATRIX_LED_MAX_ROWS / 8)
#define HT16K33_FB_SIZE (HT16K33_MATRIX_LED_MAX_COLS * BYTES_PER_ROW)
+enum display_type {
+ DISP_MATRIX = 0,
+ DISP_QUAD_7SEG,
+ DISP_QUAD_14SEG,
+};
+
struct ht16k33_keypad {
struct i2c_client *client;
struct input_dev *dev;
uint32_t refresh_rate;
uint8_t *buffer;
uint8_t *cache;
- struct delayed_work work;
+};
+
+struct ht16k33_seg {
+ struct linedisp linedisp;
+ union {
+ struct seg7_conversion_map seg7;
+ struct seg14_conversion_map seg14;
+ } map;
+ unsigned int map_size;
+ char curr[4];
};
struct ht16k33_priv {
struct i2c_client *client;
+ struct delayed_work work;
+ struct led_classdev led;
struct ht16k33_keypad keypad;
- struct ht16k33_fbdev fbdev;
+ union {
+ struct ht16k33_fbdev fbdev;
+ struct ht16k33_seg seg;
+ };
+ enum display_type type;
+ uint8_t blink;
};
static const struct fb_fix_screeninfo ht16k33_fb_fix = {
.vmode = FB_VMODE_NONINTERLACED,
};
+static const SEG7_DEFAULT_MAP(initial_map_seg7);
+static const SEG14_DEFAULT_MAP(initial_map_seg14);
+
+static ssize_t map_seg_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct ht16k33_priv *priv = dev_get_drvdata(dev);
+
+ memcpy(buf, &priv->seg.map, priv->seg.map_size);
+ return priv->seg.map_size;
+}
+
+static ssize_t map_seg_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t cnt)
+{
+ struct ht16k33_priv *priv = dev_get_drvdata(dev);
+
+ if (cnt != priv->seg.map_size)
+ return -EINVAL;
+
+ memcpy(&priv->seg.map, buf, cnt);
+ return cnt;
+}
+
+static DEVICE_ATTR(map_seg7, 0644, map_seg_show, map_seg_store);
+static DEVICE_ATTR(map_seg14, 0644, map_seg_show, map_seg_store);
+
static int ht16k33_display_on(struct ht16k33_priv *priv)
{
- uint8_t data = REG_DISPLAY_SETUP | REG_DISPLAY_SETUP_ON;
+ uint8_t data = REG_DISPLAY_SETUP | REG_DISPLAY_SETUP_ON | priv->blink;
return i2c_smbus_write_byte(priv->client, data);
}
return i2c_smbus_write_byte(priv->client, REG_DISPLAY_SETUP);
}
+static int ht16k33_brightness_set(struct ht16k33_priv *priv,
+ unsigned int brightness)
+{
+ int err;
+
+ if (brightness == 0) {
+ priv->blink = REG_DISPLAY_SETUP_BLINK_OFF;
+ return ht16k33_display_off(priv);
+ }
+
+ err = ht16k33_display_on(priv);
+ if (err)
+ return err;
+
+ return i2c_smbus_write_byte(priv->client,
+ REG_BRIGHTNESS | (brightness - 1));
+}
+
+static int ht16k33_brightness_set_blocking(struct led_classdev *led_cdev,
+ enum led_brightness brightness)
+{
+ struct ht16k33_priv *priv = container_of(led_cdev, struct ht16k33_priv,
+ led);
+
+ return ht16k33_brightness_set(priv, brightness);
+}
+
+static int ht16k33_blink_set(struct led_classdev *led_cdev,
+ unsigned long *delay_on, unsigned long *delay_off)
+{
+ struct ht16k33_priv *priv = container_of(led_cdev, struct ht16k33_priv,
+ led);
+ unsigned int delay;
+ uint8_t blink;
+ int err;
+
+ if (!*delay_on && !*delay_off) {
+ blink = REG_DISPLAY_SETUP_BLINK_1HZ;
+ delay = 1000;
+ } else if (*delay_on <= 750) {
+ blink = REG_DISPLAY_SETUP_BLINK_2HZ;
+ delay = 500;
+ } else if (*delay_on <= 1500) {
+ blink = REG_DISPLAY_SETUP_BLINK_1HZ;
+ delay = 1000;
+ } else {
+ blink = REG_DISPLAY_SETUP_BLINK_0HZ5;
+ delay = 2000;
+ }
+
+ err = i2c_smbus_write_byte(priv->client,
+ REG_DISPLAY_SETUP | REG_DISPLAY_SETUP_ON |
+ blink);
+ if (err)
+ return err;
+
+ priv->blink = blink;
+ *delay_on = *delay_off = delay;
+ return 0;
+}
+
static void ht16k33_fb_queue(struct ht16k33_priv *priv)
{
struct ht16k33_fbdev *fbdev = &priv->fbdev;
- schedule_delayed_work(&fbdev->work, HZ / fbdev->refresh_rate);
+ schedule_delayed_work(&priv->work, HZ / fbdev->refresh_rate);
}
/*
*/
static void ht16k33_fb_update(struct work_struct *work)
{
- struct ht16k33_fbdev *fbdev =
- container_of(work, struct ht16k33_fbdev, work.work);
- struct ht16k33_priv *priv =
- container_of(fbdev, struct ht16k33_priv, fbdev);
+ struct ht16k33_priv *priv = container_of(work, struct ht16k33_priv,
+ work.work);
+ struct ht16k33_fbdev *fbdev = &priv->fbdev;
uint8_t *p1, *p2;
int len, pos = 0, first = -1;
static int ht16k33_initialize(struct ht16k33_priv *priv)
{
+ uint8_t data[HT16K33_FB_SIZE];
uint8_t byte;
int err;
- uint8_t data[HT16K33_MATRIX_LED_MAX_COLS * 2];
/* Clear RAM (8 * 16 bits) */
memset(data, 0, sizeof(data));
if (bl->props.power != FB_BLANK_UNBLANK ||
bl->props.fb_blank != FB_BLANK_UNBLANK ||
- bl->props.state & BL_CORE_FBBLANK || brightness == 0) {
- return ht16k33_display_off(priv);
- }
+ bl->props.state & BL_CORE_FBBLANK)
+ brightness = 0;
- ht16k33_display_on(priv);
- return i2c_smbus_write_byte(priv->client,
- REG_BRIGHTNESS | (brightness - 1));
+ return ht16k33_brightness_set(priv, brightness);
}
static int ht16k33_bl_check_fb(struct backlight_device *bl, struct fb_info *fi)
.check_fb = ht16k33_bl_check_fb,
};
+/*
+ * Blank events will be passed to the actual device handling the backlight when
+ * we return zero here.
+ */
+static int ht16k33_blank(int blank, struct fb_info *info)
+{
+ return 0;
+}
+
static int ht16k33_mmap(struct fb_info *info, struct vm_area_struct *vma)
{
struct ht16k33_priv *priv = info->par;
.owner = THIS_MODULE,
.fb_read = fb_sys_read,
.fb_write = fb_sys_write,
+ .fb_blank = ht16k33_blank,
.fb_fillrect = sys_fillrect,
.fb_copyarea = sys_copyarea,
.fb_imageblit = sys_imageblit,
disable_irq(keypad->client->irq);
}
+static void ht16k33_linedisp_update(struct linedisp *linedisp)
+{
+ struct ht16k33_priv *priv = container_of(linedisp, struct ht16k33_priv,
+ seg.linedisp);
+
+ schedule_delayed_work(&priv->work, 0);
+}
+
+static void ht16k33_seg7_update(struct work_struct *work)
+{
+ struct ht16k33_priv *priv = container_of(work, struct ht16k33_priv,
+ work.work);
+ struct ht16k33_seg *seg = &priv->seg;
+ char *s = seg->curr;
+ uint8_t buf[9];
+
+ buf[0] = map_to_seg7(&seg->map.seg7, *s++);
+ buf[1] = 0;
+ buf[2] = map_to_seg7(&seg->map.seg7, *s++);
+ buf[3] = 0;
+ buf[4] = 0;
+ buf[5] = 0;
+ buf[6] = map_to_seg7(&seg->map.seg7, *s++);
+ buf[7] = 0;
+ buf[8] = map_to_seg7(&seg->map.seg7, *s++);
+
+ i2c_smbus_write_i2c_block_data(priv->client, 0, ARRAY_SIZE(buf), buf);
+}
+
+static void ht16k33_seg14_update(struct work_struct *work)
+{
+ struct ht16k33_priv *priv = container_of(work, struct ht16k33_priv,
+ work.work);
+ struct ht16k33_seg *seg = &priv->seg;
+ char *s = seg->curr;
+ uint8_t buf[8];
+
+ put_unaligned_le16(map_to_seg14(&seg->map.seg14, *s++), buf);
+ put_unaligned_le16(map_to_seg14(&seg->map.seg14, *s++), buf + 2);
+ put_unaligned_le16(map_to_seg14(&seg->map.seg14, *s++), buf + 4);
+ put_unaligned_le16(map_to_seg14(&seg->map.seg14, *s++), buf + 6);
+
+ i2c_smbus_write_i2c_block_data(priv->client, 0, ARRAY_SIZE(buf), buf);
+}
+
+static int ht16k33_led_probe(struct device *dev, struct led_classdev *led,
+ unsigned int brightness)
+{
+ struct led_init_data init_data = {};
+ int err;
+
+ /* The LED is optional */
+ init_data.fwnode = device_get_named_child_node(dev, "led");
+ if (!init_data.fwnode)
+ return 0;
+
+ init_data.devicename = "auxdisplay";
+ init_data.devname_mandatory = true;
+
+ led->brightness_set_blocking = ht16k33_brightness_set_blocking;
+ led->blink_set = ht16k33_blink_set;
+ led->flags = LED_CORE_SUSPENDRESUME;
+ led->brightness = brightness;
+ led->max_brightness = MAX_BRIGHTNESS;
+
+ err = devm_led_classdev_register_ext(dev, led, &init_data);
+ if (err)
+ dev_err(dev, "Failed to register LED\n");
+
+ return err;
+}
+
static int ht16k33_keypad_probe(struct i2c_client *client,
struct ht16k33_keypad *keypad)
{
- struct device_node *node = client->dev.of_node;
+ struct device *dev = &client->dev;
u32 rows = HT16K33_MATRIX_KEYPAD_MAX_ROWS;
u32 cols = HT16K33_MATRIX_KEYPAD_MAX_COLS;
int err;
keypad->client = client;
init_waitqueue_head(&keypad->wait);
- keypad->dev = devm_input_allocate_device(&client->dev);
+ keypad->dev = devm_input_allocate_device(dev);
if (!keypad->dev)
return -ENOMEM;
keypad->dev->open = ht16k33_keypad_start;
keypad->dev->close = ht16k33_keypad_stop;
- if (!of_get_property(node, "linux,no-autorepeat", NULL))
+ if (!device_property_read_bool(dev, "linux,no-autorepeat"))
__set_bit(EV_REP, keypad->dev->evbit);
- err = of_property_read_u32(node, "debounce-delay-ms",
- &keypad->debounce_ms);
+ err = device_property_read_u32(dev, "debounce-delay-ms",
+ &keypad->debounce_ms);
if (err) {
- dev_err(&client->dev, "key debounce delay not specified\n");
+ dev_err(dev, "key debounce delay not specified\n");
return err;
}
- err = matrix_keypad_parse_of_params(&client->dev, &rows, &cols);
+ err = matrix_keypad_parse_properties(dev, &rows, &cols);
if (err)
return err;
if (rows > HT16K33_MATRIX_KEYPAD_MAX_ROWS ||
cols > HT16K33_MATRIX_KEYPAD_MAX_COLS) {
- dev_err(&client->dev, "%u rows or %u cols out of range in DT\n",
- rows, cols);
+ dev_err(dev, "%u rows or %u cols out of range in DT\n", rows,
+ cols);
return -ERANGE;
}
err = matrix_keypad_build_keymap(NULL, NULL, rows, cols, NULL,
keypad->dev);
if (err) {
- dev_err(&client->dev, "failed to build keymap\n");
+ dev_err(dev, "failed to build keymap\n");
return err;
}
- err = devm_request_threaded_irq(&client->dev, client->irq,
- NULL, ht16k33_keypad_irq_thread,
+ err = devm_request_threaded_irq(dev, client->irq, NULL,
+ ht16k33_keypad_irq_thread,
IRQF_TRIGGER_HIGH | IRQF_ONESHOT,
DRIVER_NAME, keypad);
if (err) {
- dev_err(&client->dev, "irq request failed %d, error %d\n",
- client->irq, err);
+ dev_err(dev, "irq request failed %d, error %d\n", client->irq,
+ err);
return err;
}
ht16k33_keypad_stop(keypad->dev);
- err = input_register_device(keypad->dev);
- if (err)
- return err;
-
- return 0;
+ return input_register_device(keypad->dev);
}
-static int ht16k33_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int ht16k33_fbdev_probe(struct device *dev, struct ht16k33_priv *priv,
+ uint32_t brightness)
{
+ struct ht16k33_fbdev *fbdev = &priv->fbdev;
+ struct backlight_device *bl = NULL;
int err;
- uint32_t dft_brightness;
- struct backlight_device *bl;
- struct backlight_properties bl_props;
- struct ht16k33_priv *priv;
- struct ht16k33_fbdev *fbdev;
- struct device_node *node = client->dev.of_node;
- if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
- dev_err(&client->dev, "i2c_check_functionality error\n");
- return -EIO;
- }
-
- priv = devm_kzalloc(&client->dev, sizeof(*priv), GFP_KERNEL);
- if (!priv)
- return -ENOMEM;
-
- priv->client = client;
- i2c_set_clientdata(client, priv);
- fbdev = &priv->fbdev;
+ if (priv->led.dev) {
+ err = ht16k33_brightness_set(priv, brightness);
+ if (err)
+ return err;
+ } else {
+ /* backwards compatibility with DT lacking an led subnode */
+ struct backlight_properties bl_props;
+
+ memset(&bl_props, 0, sizeof(struct backlight_properties));
+ bl_props.type = BACKLIGHT_RAW;
+ bl_props.max_brightness = MAX_BRIGHTNESS;
+
+ bl = devm_backlight_device_register(dev, DRIVER_NAME"-bl", dev,
+ priv, &ht16k33_bl_ops,
+ &bl_props);
+ if (IS_ERR(bl)) {
+ dev_err(dev, "failed to register backlight\n");
+ return PTR_ERR(bl);
+ }
- err = ht16k33_initialize(priv);
- if (err)
- return err;
+ bl->props.brightness = brightness;
+ ht16k33_bl_update_status(bl);
+ }
/* Framebuffer (2 bytes per column) */
BUILD_BUG_ON(PAGE_SIZE < HT16K33_FB_SIZE);
if (!fbdev->buffer)
return -ENOMEM;
- fbdev->cache = devm_kmalloc(&client->dev, HT16K33_FB_SIZE, GFP_KERNEL);
+ fbdev->cache = devm_kmalloc(dev, HT16K33_FB_SIZE, GFP_KERNEL);
if (!fbdev->cache) {
err = -ENOMEM;
goto err_fbdev_buffer;
}
- fbdev->info = framebuffer_alloc(0, &client->dev);
+ fbdev->info = framebuffer_alloc(0, dev);
if (!fbdev->info) {
err = -ENOMEM;
goto err_fbdev_buffer;
}
- err = of_property_read_u32(node, "refresh-rate-hz",
- &fbdev->refresh_rate);
+ err = device_property_read_u32(dev, "refresh-rate-hz",
+ &fbdev->refresh_rate);
if (err) {
- dev_err(&client->dev, "refresh rate not specified\n");
+ dev_err(dev, "refresh rate not specified\n");
goto err_fbdev_info;
}
fb_bl_default_curve(fbdev->info, 0, MIN_BRIGHTNESS, MAX_BRIGHTNESS);
- INIT_DELAYED_WORK(&fbdev->work, ht16k33_fb_update);
+ INIT_DELAYED_WORK(&priv->work, ht16k33_fb_update);
fbdev->info->fbops = &ht16k33_fb_ops;
fbdev->info->screen_base = (char __iomem *) fbdev->buffer;
fbdev->info->screen_size = HT16K33_FB_SIZE;
fbdev->info->fix = ht16k33_fb_fix;
fbdev->info->var = ht16k33_fb_var;
+ fbdev->info->bl_dev = bl;
fbdev->info->pseudo_palette = NULL;
fbdev->info->flags = FBINFO_FLAG_DEFAULT;
fbdev->info->par = priv;
if (err)
goto err_fbdev_info;
- /* Keypad */
- if (client->irq > 0) {
- err = ht16k33_keypad_probe(client, &priv->keypad);
- if (err)
- goto err_fbdev_unregister;
+ ht16k33_fb_queue(priv);
+ return 0;
+
+err_fbdev_info:
+ framebuffer_release(fbdev->info);
+err_fbdev_buffer:
+ free_page((unsigned long) fbdev->buffer);
+
+ return err;
+}
+
+static int ht16k33_seg_probe(struct device *dev, struct ht16k33_priv *priv,
+ uint32_t brightness)
+{
+ struct ht16k33_seg *seg = &priv->seg;
+ int err;
+
+ err = ht16k33_brightness_set(priv, brightness);
+ if (err)
+ return err;
+
+ switch (priv->type) {
+ case DISP_MATRIX:
+ /* not handled here */
+ err = -EINVAL;
+ break;
+
+ case DISP_QUAD_7SEG:
+ INIT_DELAYED_WORK(&priv->work, ht16k33_seg7_update);
+ seg->map.seg7 = initial_map_seg7;
+ seg->map_size = sizeof(seg->map.seg7);
+ err = device_create_file(dev, &dev_attr_map_seg7);
+ break;
+
+ case DISP_QUAD_14SEG:
+ INIT_DELAYED_WORK(&priv->work, ht16k33_seg14_update);
+ seg->map.seg14 = initial_map_seg14;
+ seg->map_size = sizeof(seg->map.seg14);
+ err = device_create_file(dev, &dev_attr_map_seg14);
+ break;
}
+ if (err)
+ return err;
+
+ err = linedisp_register(&seg->linedisp, dev, 4, seg->curr,
+ ht16k33_linedisp_update);
+ if (err)
+ goto err_remove_map_file;
+
+ return 0;
+
+err_remove_map_file:
+ device_remove_file(dev, &dev_attr_map_seg7);
+ device_remove_file(dev, &dev_attr_map_seg14);
+ return err;
+}
+
+static int ht16k33_probe(struct i2c_client *client)
+{
+ struct device *dev = &client->dev;
+ const struct of_device_id *id;
+ struct ht16k33_priv *priv;
+ uint32_t dft_brightness;
+ int err;
- /* Backlight */
- memset(&bl_props, 0, sizeof(struct backlight_properties));
- bl_props.type = BACKLIGHT_RAW;
- bl_props.max_brightness = MAX_BRIGHTNESS;
-
- bl = devm_backlight_device_register(&client->dev, DRIVER_NAME"-bl",
- &client->dev, priv,
- &ht16k33_bl_ops, &bl_props);
- if (IS_ERR(bl)) {
- dev_err(&client->dev, "failed to register backlight\n");
- err = PTR_ERR(bl);
- goto err_fbdev_unregister;
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ dev_err(dev, "i2c_check_functionality error\n");
+ return -EIO;
}
- err = of_property_read_u32(node, "default-brightness-level",
- &dft_brightness);
+ priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->client = client;
+ id = i2c_of_match_device(dev->driver->of_match_table, client);
+ if (id)
+ priv->type = (uintptr_t)id->data;
+ i2c_set_clientdata(client, priv);
+
+ err = ht16k33_initialize(priv);
+ if (err)
+ return err;
+
+ err = device_property_read_u32(dev, "default-brightness-level",
+ &dft_brightness);
if (err) {
dft_brightness = MAX_BRIGHTNESS;
} else if (dft_brightness > MAX_BRIGHTNESS) {
- dev_warn(&client->dev,
+ dev_warn(dev,
"invalid default brightness level: %u, using %u\n",
dft_brightness, MAX_BRIGHTNESS);
dft_brightness = MAX_BRIGHTNESS;
}
- bl->props.brightness = dft_brightness;
- ht16k33_bl_update_status(bl);
-
- ht16k33_fb_queue(priv);
- return 0;
+ /* LED */
+ err = ht16k33_led_probe(dev, &priv->led, dft_brightness);
+ if (err)
+ return err;
-err_fbdev_unregister:
- unregister_framebuffer(fbdev->info);
-err_fbdev_info:
- framebuffer_release(fbdev->info);
-err_fbdev_buffer:
- free_page((unsigned long) fbdev->buffer);
+ /* Keypad */
+ if (client->irq > 0) {
+ err = ht16k33_keypad_probe(client, &priv->keypad);
+ if (err)
+ return err;
+ }
+ switch (priv->type) {
+ case DISP_MATRIX:
+ /* Frame Buffer Display */
+ err = ht16k33_fbdev_probe(dev, priv, dft_brightness);
+ break;
+
+ case DISP_QUAD_7SEG:
+ case DISP_QUAD_14SEG:
+ /* Segment Display */
+ err = ht16k33_seg_probe(dev, priv, dft_brightness);
+ break;
+ }
return err;
}
struct ht16k33_priv *priv = i2c_get_clientdata(client);
struct ht16k33_fbdev *fbdev = &priv->fbdev;
- cancel_delayed_work_sync(&fbdev->work);
- unregister_framebuffer(fbdev->info);
- framebuffer_release(fbdev->info);
- free_page((unsigned long) fbdev->buffer);
+ cancel_delayed_work_sync(&priv->work);
+
+ switch (priv->type) {
+ case DISP_MATRIX:
+ unregister_framebuffer(fbdev->info);
+ framebuffer_release(fbdev->info);
+ free_page((unsigned long)fbdev->buffer);
+ break;
+
+ case DISP_QUAD_7SEG:
+ case DISP_QUAD_14SEG:
+ linedisp_unregister(&priv->seg.linedisp);
+ device_remove_file(&client->dev, &dev_attr_map_seg7);
+ device_remove_file(&client->dev, &dev_attr_map_seg14);
+ break;
+ }
return 0;
}
MODULE_DEVICE_TABLE(i2c, ht16k33_i2c_match);
static const struct of_device_id ht16k33_of_match[] = {
- { .compatible = "holtek,ht16k33", },
+ {
+ /* 0.56" 4-Digit 7-Segment FeatherWing Display (Red) */
+ .compatible = "adafruit,3108", .data = (void *)DISP_QUAD_7SEG,
+ }, {
+ /* 0.54" Quad Alphanumeric FeatherWing Display (Red) */
+ .compatible = "adafruit,3130", .data = (void *)DISP_QUAD_14SEG,
+ }, {
+ /* Generic, assumed Dot-Matrix Display */
+ .compatible = "holtek,ht16k33", .data = (void *)DISP_MATRIX,
+ },
{ }
};
MODULE_DEVICE_TABLE(of, ht16k33_of_match);
static struct i2c_driver ht16k33_driver = {
- .probe = ht16k33_probe,
+ .probe_new = ht16k33_probe,
.remove = ht16k33_remove,
.driver = {
.name = DRIVER_NAME,
- .of_match_table = of_match_ptr(ht16k33_of_match),
+ .of_match_table = ht16k33_of_match,
},
.id_table = ht16k33_i2c_match,
};
* Author: Paul Burton <paul.burton@mips.com>
*/
-#include <generated/utsrelease.h>
#include <linux/kernel.h>
#include <linux/io.h>
#include <linux/mfd/syscon.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
#include <linux/slab.h>
-#include <linux/sysfs.h>
+
+#include "line-display.h"
struct img_ascii_lcd_ctx;
struct img_ascii_lcd_config {
unsigned int num_chars;
bool external_regmap;
- void (*update)(struct img_ascii_lcd_ctx *ctx);
+ void (*update)(struct linedisp *linedisp);
};
/**
* struct img_ascii_lcd_ctx - Private data structure
- * @pdev: the ASCII LCD platform device
* @base: the base address of the LCD registers
* @regmap: the regmap through which LCD registers are accessed
* @offset: the offset within regmap to the start of the LCD registers
* @cfg: pointer to the LCD model configuration
- * @message: the full message to display or scroll on the LCD
- * @message_len: the length of the @message string
- * @scroll_pos: index of the first character of @message currently displayed
- * @scroll_rate: scroll interval in jiffies
- * @timer: timer used to implement scrolling
+ * @linedisp: line display structure
* @curr: the string currently displayed on the LCD
*/
struct img_ascii_lcd_ctx {
- struct platform_device *pdev;
union {
void __iomem *base;
struct regmap *regmap;
};
u32 offset;
const struct img_ascii_lcd_config *cfg;
- char *message;
- unsigned int message_len;
- unsigned int scroll_pos;
- unsigned int scroll_rate;
- struct timer_list timer;
+ struct linedisp linedisp;
char curr[] __aligned(8);
};
* MIPS Boston development board
*/
-static void boston_update(struct img_ascii_lcd_ctx *ctx)
+static void boston_update(struct linedisp *linedisp)
{
+ struct img_ascii_lcd_ctx *ctx =
+ container_of(linedisp, struct img_ascii_lcd_ctx, linedisp);
ulong val;
#if BITS_PER_LONG == 64
* MIPS Malta development board
*/
-static void malta_update(struct img_ascii_lcd_ctx *ctx)
+static void malta_update(struct linedisp *linedisp)
{
+ struct img_ascii_lcd_ctx *ctx =
+ container_of(linedisp, struct img_ascii_lcd_ctx, linedisp);
unsigned int i;
int err = 0;
- for (i = 0; i < ctx->cfg->num_chars; i++) {
+ for (i = 0; i < linedisp->num_chars; i++) {
err = regmap_write(ctx->regmap,
ctx->offset + (i * 8), ctx->curr[i]);
if (err)
return 0;
}
-static void sead3_update(struct img_ascii_lcd_ctx *ctx)
+static void sead3_update(struct linedisp *linedisp)
{
+ struct img_ascii_lcd_ctx *ctx =
+ container_of(linedisp, struct img_ascii_lcd_ctx, linedisp);
unsigned int i;
int err = 0;
- for (i = 0; i < ctx->cfg->num_chars; i++) {
+ for (i = 0; i < linedisp->num_chars; i++) {
err = sead3_wait_lcd_idle(ctx);
if (err)
break;
MODULE_DEVICE_TABLE(of, img_ascii_lcd_matches);
/**
- * img_ascii_lcd_scroll() - scroll the display by a character
- * @t: really a pointer to the private data structure
- *
- * Scroll the current message along the LCD by one character, rearming the
- * timer if required.
- */
-static void img_ascii_lcd_scroll(struct timer_list *t)
-{
- struct img_ascii_lcd_ctx *ctx = from_timer(ctx, t, timer);
- unsigned int i, ch = ctx->scroll_pos;
- unsigned int num_chars = ctx->cfg->num_chars;
-
- /* update the current message string */
- for (i = 0; i < num_chars;) {
- /* copy as many characters from the string as possible */
- for (; i < num_chars && ch < ctx->message_len; i++, ch++)
- ctx->curr[i] = ctx->message[ch];
-
- /* wrap around to the start of the string */
- ch = 0;
- }
-
- /* update the LCD */
- ctx->cfg->update(ctx);
-
- /* move on to the next character */
- ctx->scroll_pos++;
- ctx->scroll_pos %= ctx->message_len;
-
- /* rearm the timer */
- if (ctx->message_len > ctx->cfg->num_chars)
- mod_timer(&ctx->timer, jiffies + ctx->scroll_rate);
-}
-
-/**
- * img_ascii_lcd_display() - set the message to be displayed
- * @ctx: pointer to the private data structure
- * @msg: the message to display
- * @count: length of msg, or -1
- *
- * Display a new message @msg on the LCD. @msg can be longer than the number of
- * characters the LCD can display, in which case it will begin scrolling across
- * the LCD display.
- *
- * Return: 0 on success, -ENOMEM on memory allocation failure
- */
-static int img_ascii_lcd_display(struct img_ascii_lcd_ctx *ctx,
- const char *msg, ssize_t count)
-{
- char *new_msg;
-
- /* stop the scroll timer */
- del_timer_sync(&ctx->timer);
-
- if (count == -1)
- count = strlen(msg);
-
- /* if the string ends with a newline, trim it */
- if (msg[count - 1] == '\n')
- count--;
-
- new_msg = devm_kmalloc(&ctx->pdev->dev, count + 1, GFP_KERNEL);
- if (!new_msg)
- return -ENOMEM;
-
- memcpy(new_msg, msg, count);
- new_msg[count] = 0;
-
- if (ctx->message)
- devm_kfree(&ctx->pdev->dev, ctx->message);
-
- ctx->message = new_msg;
- ctx->message_len = count;
- ctx->scroll_pos = 0;
-
- /* update the LCD */
- img_ascii_lcd_scroll(&ctx->timer);
-
- return 0;
-}
-
-/**
- * message_show() - read message via sysfs
- * @dev: the LCD device
- * @attr: the LCD message attribute
- * @buf: the buffer to read the message into
- *
- * Read the current message being displayed or scrolled across the LCD display
- * into @buf, for reads from sysfs.
- *
- * Return: the number of characters written to @buf
- */
-static ssize_t message_show(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- struct img_ascii_lcd_ctx *ctx = dev_get_drvdata(dev);
-
- return sprintf(buf, "%s\n", ctx->message);
-}
-
-/**
- * message_store() - write a new message via sysfs
- * @dev: the LCD device
- * @attr: the LCD message attribute
- * @buf: the buffer containing the new message
- * @count: the size of the message in @buf
- *
- * Write a new message to display or scroll across the LCD display from sysfs.
- *
- * Return: the size of the message on success, else -ERRNO
- */
-static ssize_t message_store(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
-{
- struct img_ascii_lcd_ctx *ctx = dev_get_drvdata(dev);
- int err;
-
- err = img_ascii_lcd_display(ctx, buf, count);
- return err ?: count;
-}
-
-static DEVICE_ATTR_RW(message);
-
-/**
* img_ascii_lcd_probe() - probe an LCD display device
* @pdev: the LCD platform device
*
{
const struct of_device_id *match;
const struct img_ascii_lcd_config *cfg;
+ struct device *dev = &pdev->dev;
struct img_ascii_lcd_ctx *ctx;
int err;
- match = of_match_device(img_ascii_lcd_matches, &pdev->dev);
+ match = of_match_device(img_ascii_lcd_matches, dev);
if (!match)
return -ENODEV;
cfg = match->data;
- ctx = devm_kzalloc(&pdev->dev, sizeof(*ctx) + cfg->num_chars,
- GFP_KERNEL);
+ ctx = devm_kzalloc(dev, sizeof(*ctx) + cfg->num_chars, GFP_KERNEL);
if (!ctx)
return -ENOMEM;
if (cfg->external_regmap) {
- ctx->regmap = syscon_node_to_regmap(pdev->dev.parent->of_node);
+ ctx->regmap = syscon_node_to_regmap(dev->parent->of_node);
if (IS_ERR(ctx->regmap))
return PTR_ERR(ctx->regmap);
- if (of_property_read_u32(pdev->dev.of_node, "offset",
- &ctx->offset))
+ if (of_property_read_u32(dev->of_node, "offset", &ctx->offset))
return -EINVAL;
} else {
ctx->base = devm_platform_ioremap_resource(pdev, 0);
return PTR_ERR(ctx->base);
}
- ctx->pdev = pdev;
- ctx->cfg = cfg;
- ctx->message = NULL;
- ctx->scroll_pos = 0;
- ctx->scroll_rate = HZ / 2;
-
- /* initialise a timer for scrolling the message */
- timer_setup(&ctx->timer, img_ascii_lcd_scroll, 0);
-
- platform_set_drvdata(pdev, ctx);
-
- /* display a default message */
- err = img_ascii_lcd_display(ctx, "Linux " UTS_RELEASE " ", -1);
+ err = linedisp_register(&ctx->linedisp, dev, cfg->num_chars, ctx->curr,
+ cfg->update);
if (err)
- goto out_del_timer;
+ return err;
- err = device_create_file(&pdev->dev, &dev_attr_message);
+ /* for backwards compatibility */
+ err = compat_only_sysfs_link_entry_to_kobj(&dev->kobj,
+ &ctx->linedisp.dev.kobj,
+ "message", NULL);
if (err)
- goto out_del_timer;
+ goto err_unregister;
+ platform_set_drvdata(pdev, ctx);
return 0;
-out_del_timer:
- del_timer_sync(&ctx->timer);
+
+err_unregister:
+ linedisp_unregister(&ctx->linedisp);
return err;
}
{
struct img_ascii_lcd_ctx *ctx = platform_get_drvdata(pdev);
- device_remove_file(&pdev->dev, &dev_attr_message);
- del_timer_sync(&ctx->timer);
+ sysfs_remove_link(&pdev->dev.kobj, "message");
+ linedisp_unregister(&ctx->linedisp);
return 0;
}
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/delay.h>
-#include <linux/fs.h>
-#include <linux/io.h>
#include <linux/parport.h>
-#include <linux/uaccess.h>
#include <linux/ks0108.h>
#define KS0108_NAME "ks0108"
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Character line display core support
+ *
+ * Copyright (C) 2016 Imagination Technologies
+ * Author: Paul Burton <paul.burton@mips.com>
+ *
+ * Copyright (C) 2021 Glider bv
+ */
+
+#include <generated/utsrelease.h>
+
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
+#include <linux/timer.h>
+
+#include "line-display.h"
+
+#define DEFAULT_SCROLL_RATE (HZ / 2)
+
+/**
+ * linedisp_scroll() - scroll the display by a character
+ * @t: really a pointer to the private data structure
+ *
+ * Scroll the current message along the display by one character, rearming the
+ * timer if required.
+ */
+static void linedisp_scroll(struct timer_list *t)
+{
+ struct linedisp *linedisp = from_timer(linedisp, t, timer);
+ unsigned int i, ch = linedisp->scroll_pos;
+ unsigned int num_chars = linedisp->num_chars;
+
+ /* update the current message string */
+ for (i = 0; i < num_chars;) {
+ /* copy as many characters from the string as possible */
+ for (; i < num_chars && ch < linedisp->message_len; i++, ch++)
+ linedisp->buf[i] = linedisp->message[ch];
+
+ /* wrap around to the start of the string */
+ ch = 0;
+ }
+
+ /* update the display */
+ linedisp->update(linedisp);
+
+ /* move on to the next character */
+ linedisp->scroll_pos++;
+ linedisp->scroll_pos %= linedisp->message_len;
+
+ /* rearm the timer */
+ if (linedisp->message_len > num_chars && linedisp->scroll_rate)
+ mod_timer(&linedisp->timer, jiffies + linedisp->scroll_rate);
+}
+
+/**
+ * linedisp_display() - set the message to be displayed
+ * @linedisp: pointer to the private data structure
+ * @msg: the message to display
+ * @count: length of msg, or -1
+ *
+ * Display a new message @msg on the display. @msg can be longer than the
+ * number of characters the display can display, in which case it will begin
+ * scrolling across the display.
+ *
+ * Return: 0 on success, -ENOMEM on memory allocation failure
+ */
+static int linedisp_display(struct linedisp *linedisp, const char *msg,
+ ssize_t count)
+{
+ char *new_msg;
+
+ /* stop the scroll timer */
+ del_timer_sync(&linedisp->timer);
+
+ if (count == -1)
+ count = strlen(msg);
+
+ /* if the string ends with a newline, trim it */
+ if (msg[count - 1] == '\n')
+ count--;
+
+ if (!count) {
+ /* Clear the display */
+ kfree(linedisp->message);
+ linedisp->message = NULL;
+ linedisp->message_len = 0;
+ memset(linedisp->buf, ' ', linedisp->num_chars);
+ linedisp->update(linedisp);
+ return 0;
+ }
+
+ new_msg = kmemdup_nul(msg, count, GFP_KERNEL);
+ if (!new_msg)
+ return -ENOMEM;
+
+ kfree(linedisp->message);
+
+ linedisp->message = new_msg;
+ linedisp->message_len = count;
+ linedisp->scroll_pos = 0;
+
+ /* update the display */
+ linedisp_scroll(&linedisp->timer);
+
+ return 0;
+}
+
+/**
+ * message_show() - read message via sysfs
+ * @dev: the display device
+ * @attr: the display message attribute
+ * @buf: the buffer to read the message into
+ *
+ * Read the current message being displayed or scrolled across the display into
+ * @buf, for reads from sysfs.
+ *
+ * Return: the number of characters written to @buf
+ */
+static ssize_t message_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct linedisp *linedisp = container_of(dev, struct linedisp, dev);
+
+ return sysfs_emit(buf, "%s\n", linedisp->message);
+}
+
+/**
+ * message_store() - write a new message via sysfs
+ * @dev: the display device
+ * @attr: the display message attribute
+ * @buf: the buffer containing the new message
+ * @count: the size of the message in @buf
+ *
+ * Write a new message to display or scroll across the display from sysfs.
+ *
+ * Return: the size of the message on success, else -ERRNO
+ */
+static ssize_t message_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct linedisp *linedisp = container_of(dev, struct linedisp, dev);
+ int err;
+
+ err = linedisp_display(linedisp, buf, count);
+ return err ?: count;
+}
+
+static DEVICE_ATTR_RW(message);
+
+static ssize_t scroll_step_ms_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct linedisp *linedisp = container_of(dev, struct linedisp, dev);
+
+ return sysfs_emit(buf, "%u\n", jiffies_to_msecs(linedisp->scroll_rate));
+}
+
+static ssize_t scroll_step_ms_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct linedisp *linedisp = container_of(dev, struct linedisp, dev);
+ unsigned int ms;
+
+ if (kstrtouint(buf, 10, &ms) != 0)
+ return -EINVAL;
+
+ linedisp->scroll_rate = msecs_to_jiffies(ms);
+ if (linedisp->message && linedisp->message_len > linedisp->num_chars) {
+ del_timer_sync(&linedisp->timer);
+ if (linedisp->scroll_rate)
+ linedisp_scroll(&linedisp->timer);
+ }
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(scroll_step_ms);
+
+static struct attribute *linedisp_attrs[] = {
+ &dev_attr_message.attr,
+ &dev_attr_scroll_step_ms.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(linedisp);
+
+static const struct device_type linedisp_type = {
+ .groups = linedisp_groups,
+};
+
+/**
+ * linedisp_register - register a character line display
+ * @linedisp: pointer to character line display structure
+ * @parent: parent device
+ * @num_chars: the number of characters that can be displayed
+ * @buf: pointer to a buffer that can hold @num_chars characters
+ * @update: Function called to update the display. This must not sleep!
+ *
+ * Return: zero on success, else a negative error code.
+ */
+int linedisp_register(struct linedisp *linedisp, struct device *parent,
+ unsigned int num_chars, char *buf,
+ void (*update)(struct linedisp *linedisp))
+{
+ static atomic_t linedisp_id = ATOMIC_INIT(-1);
+ int err;
+
+ memset(linedisp, 0, sizeof(*linedisp));
+ linedisp->dev.parent = parent;
+ linedisp->dev.type = &linedisp_type;
+ linedisp->update = update;
+ linedisp->buf = buf;
+ linedisp->num_chars = num_chars;
+ linedisp->scroll_rate = DEFAULT_SCROLL_RATE;
+
+ device_initialize(&linedisp->dev);
+ dev_set_name(&linedisp->dev, "linedisp.%lu",
+ (unsigned long)atomic_inc_return(&linedisp_id));
+
+ /* initialise a timer for scrolling the message */
+ timer_setup(&linedisp->timer, linedisp_scroll, 0);
+
+ err = device_add(&linedisp->dev);
+ if (err)
+ goto out_del_timer;
+
+ /* display a default message */
+ err = linedisp_display(linedisp, "Linux " UTS_RELEASE " ", -1);
+ if (err)
+ goto out_del_dev;
+
+ return 0;
+
+out_del_dev:
+ device_del(&linedisp->dev);
+out_del_timer:
+ del_timer_sync(&linedisp->timer);
+ put_device(&linedisp->dev);
+ return err;
+}
+EXPORT_SYMBOL_GPL(linedisp_register);
+
+/**
+ * linedisp_unregister - unregister a character line display
+ * @linedisp: pointer to character line display structure registered previously
+ * with linedisp_register()
+ */
+void linedisp_unregister(struct linedisp *linedisp)
+{
+ device_del(&linedisp->dev);
+ del_timer_sync(&linedisp->timer);
+ kfree(linedisp->message);
+ put_device(&linedisp->dev);
+}
+EXPORT_SYMBOL_GPL(linedisp_unregister);
+
+MODULE_LICENSE("GPL");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Character line display core support
+ *
+ * Copyright (C) 2016 Imagination Technologies
+ * Author: Paul Burton <paul.burton@mips.com>
+ *
+ * Copyright (C) 2021 Glider bv
+ */
+
+#ifndef _LINEDISP_H
+#define _LINEDISP_H
+
+/**
+ * struct linedisp - character line display private data structure
+ * @dev: the line display device
+ * @timer: timer used to implement scrolling
+ * @update: function called to update the display
+ * @buf: pointer to the buffer for the string currently displayed
+ * @message: the full message to display or scroll on the display
+ * @num_chars: the number of characters that can be displayed
+ * @message_len: the length of the @message string
+ * @scroll_pos: index of the first character of @message currently displayed
+ * @scroll_rate: scroll interval in jiffies
+ */
+struct linedisp {
+ struct device dev;
+ struct timer_list timer;
+ void (*update)(struct linedisp *linedisp);
+ char *buf;
+ char *message;
+ unsigned int num_chars;
+ unsigned int message_len;
+ unsigned int scroll_pos;
+ unsigned int scroll_rate;
+};
+
+int linedisp_register(struct linedisp *linedisp, struct device *parent,
+ unsigned int num_chars, char *buf,
+ void (*update)(struct linedisp *linedisp));
+void linedisp_unregister(struct linedisp *linedisp);
+
+#endif /* LINEDISP_H */
obj-$(CONFIG_ISA_BUS_API) += isa.o
obj-y += firmware_loader/
obj-$(CONFIG_NUMA) += node.o
-obj-$(CONFIG_MEMORY_HOTPLUG_SPARSE) += memory.o
+obj-$(CONFIG_MEMORY_HOTPLUG) += memory.o
ifeq ($(CONFIG_SYSFS),y)
obj-$(CONFIG_MODULES) += module.o
endif
#include <linux/of.h>
#include <asm/sections.h>
+#include <asm/pgalloc.h>
struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
EXPORT_SYMBOL(node_data);
static void __init pcpu_fc_free(void *ptr, size_t size)
{
- memblock_free_early(__pa(ptr), size);
+ memblock_free(ptr, size);
}
+#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK
+static void __init pcpu_populate_pte(unsigned long addr)
+{
+ pgd_t *pgd = pgd_offset_k(addr);
+ p4d_t *p4d;
+ pud_t *pud;
+ pmd_t *pmd;
+
+ p4d = p4d_offset(pgd, addr);
+ if (p4d_none(*p4d)) {
+ pud_t *new;
+
+ new = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!new)
+ goto err_alloc;
+ p4d_populate(&init_mm, p4d, new);
+ }
+
+ pud = pud_offset(p4d, addr);
+ if (pud_none(*pud)) {
+ pmd_t *new;
+
+ new = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!new)
+ goto err_alloc;
+ pud_populate(&init_mm, pud, new);
+ }
+
+ pmd = pmd_offset(pud, addr);
+ if (!pmd_present(*pmd)) {
+ pte_t *new;
+
+ new = memblock_alloc(PAGE_SIZE, PAGE_SIZE);
+ if (!new)
+ goto err_alloc;
+ pmd_populate_kernel(&init_mm, pmd, new);
+ }
+
+ return;
+
+err_alloc:
+ panic("%s: Failed to allocate %lu bytes align=%lx from=%lx\n",
+ __func__, PAGE_SIZE, PAGE_SIZE, PAGE_SIZE);
+}
+#endif
+
void __init setup_per_cpu_areas(void)
{
unsigned long delta;
unsigned int cpu;
- int rc;
+ int rc = -EINVAL;
+
+ if (pcpu_chosen_fc != PCPU_FC_PAGE) {
+ /*
+ * Always reserve area for module percpu variables. That's
+ * what the legacy allocator did.
+ */
+ rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE,
+ PERCPU_DYNAMIC_RESERVE, PAGE_SIZE,
+ pcpu_cpu_distance,
+ pcpu_fc_alloc, pcpu_fc_free);
+#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK
+ if (rc < 0)
+ pr_warn("PERCPU: %s allocator failed (%d), falling back to page size\n",
+ pcpu_fc_names[pcpu_chosen_fc], rc);
+#endif
+ }
- /*
- * Always reserve area for module percpu variables. That's
- * what the legacy allocator did.
- */
- rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE,
- PERCPU_DYNAMIC_RESERVE, PAGE_SIZE,
- pcpu_cpu_distance,
- pcpu_fc_alloc, pcpu_fc_free);
+#ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK
+ if (rc < 0)
+ rc = pcpu_page_first_chunk(PERCPU_MODULE_RESERVE,
+ pcpu_fc_alloc,
+ pcpu_fc_free,
+ pcpu_populate_pte);
+#endif
if (rc < 0)
- panic("Failed to initialize percpu areas.");
+ panic("Failed to initialize percpu areas (err=%d).", rc);
delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
for_each_possible_cpu(cpu)
size = numa_distance_cnt * numa_distance_cnt *
sizeof(numa_distance[0]);
- memblock_free_ptr(numa_distance, size);
+ memblock_free(numa_distance, size);
numa_distance_cnt = 0;
numa_distance = NULL;
}
static int __init numa_alloc_distance(void)
{
size_t size;
- u64 phys;
int i, j;
size = nr_node_ids * nr_node_ids * sizeof(numa_distance[0]);
- phys = memblock_phys_alloc_range(size, PAGE_SIZE, 0, PFN_PHYS(max_pfn));
- if (WARN_ON(!phys))
+ numa_distance = memblock_alloc(size, PAGE_SIZE);
+ if (WARN_ON(!numa_distance))
return -ENOMEM;
- numa_distance = __va(phys);
numa_distance_cnt = nr_node_ids;
/* fill with the default distances */
cpumask_clear_cpu(cpu, topology_core_cpumask(sibling));
for_each_cpu(sibling, topology_sibling_cpumask(cpu))
cpumask_clear_cpu(cpu, topology_sibling_cpumask(sibling));
+ for_each_cpu(sibling, topology_cluster_cpumask(cpu))
+ cpumask_clear_cpu(cpu, topology_cluster_cpumask(sibling));
for_each_cpu(sibling, topology_llc_cpumask(cpu))
cpumask_clear_cpu(cpu, topology_llc_cpumask(sibling));
{
struct node *node = to_node(dev);
-#if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_HUGETLBFS)
+#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS)
/*
* We schedule the work only when a memory section is
* onlined/offlined on this node. When we come here,
return 0;
}
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifdef CONFIG_MEMORY_HOTPLUG
static int __ref get_nid_for_pfn(unsigned long pfn)
{
#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
return NOTIFY_OK;
}
#endif /* CONFIG_HUGETLBFS */
-#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
+#endif /* CONFIG_MEMORY_HOTPLUG */
-#if !defined(CONFIG_MEMORY_HOTPLUG_SPARSE) || \
- !defined(CONFIG_HUGETLBFS)
+#if !defined(CONFIG_MEMORY_HOTPLUG) || !defined(CONFIG_HUGETLBFS)
static inline int node_memory_callback(struct notifier_block *self,
unsigned long action, void *arg)
{
dev = to_device(dpm_noirq_list.next);
get_device(dev);
list_move_tail(&dev->power.entry, &dpm_late_early_list);
+
mutex_unlock(&dpm_list_mtx);
if (!is_async(dev)) {
}
}
- mutex_lock(&dpm_list_mtx);
put_device(dev);
+
+ mutex_lock(&dpm_list_mtx);
}
mutex_unlock(&dpm_list_mtx);
async_synchronize_full();
dev = to_device(dpm_late_early_list.next);
get_device(dev);
list_move_tail(&dev->power.entry, &dpm_suspended_list);
+
mutex_unlock(&dpm_list_mtx);
if (!is_async(dev)) {
pm_dev_err(dev, state, " early", error);
}
}
- mutex_lock(&dpm_list_mtx);
+
put_device(dev);
+
+ mutex_lock(&dpm_list_mtx);
}
mutex_unlock(&dpm_list_mtx);
async_synchronize_full();
}
if (!list_empty(&dev->power.entry))
list_move_tail(&dev->power.entry, &dpm_prepared_list);
+
+ mutex_unlock(&dpm_list_mtx);
+
put_device(dev);
+
+ mutex_lock(&dpm_list_mtx);
}
mutex_unlock(&dpm_list_mtx);
async_synchronize_full();
get_device(dev);
dev->power.is_prepared = false;
list_move(&dev->power.entry, &list);
+
mutex_unlock(&dpm_list_mtx);
trace_device_pm_callback_start(dev, "", state.event);
device_complete(dev, state);
trace_device_pm_callback_end(dev, 0);
- mutex_lock(&dpm_list_mtx);
put_device(dev);
+
+ mutex_lock(&dpm_list_mtx);
}
list_splice(&list, &dpm_list);
mutex_unlock(&dpm_list_mtx);
error = device_suspend_noirq(dev);
mutex_lock(&dpm_list_mtx);
+
if (error) {
pm_dev_err(dev, state, " noirq", error);
dpm_save_failed_dev(dev_name(dev));
- put_device(dev);
- break;
- }
- if (!list_empty(&dev->power.entry))
+ } else if (!list_empty(&dev->power.entry)) {
list_move(&dev->power.entry, &dpm_noirq_list);
+ }
+
+ mutex_unlock(&dpm_list_mtx);
+
put_device(dev);
- if (async_error)
+ mutex_lock(&dpm_list_mtx);
+
+ if (error || async_error)
break;
}
mutex_unlock(&dpm_list_mtx);
int error = 0;
trace_suspend_resume(TPS("dpm_suspend_late"), state.event, true);
+ wake_up_all_idle_cpus();
mutex_lock(&dpm_list_mtx);
pm_transition = state;
async_error = 0;
struct device *dev = to_device(dpm_suspended_list.prev);
get_device(dev);
+
mutex_unlock(&dpm_list_mtx);
error = device_suspend_late(dev);
mutex_lock(&dpm_list_mtx);
+
if (!list_empty(&dev->power.entry))
list_move(&dev->power.entry, &dpm_late_early_list);
if (error) {
pm_dev_err(dev, state, " late", error);
dpm_save_failed_dev(dev_name(dev));
- put_device(dev);
- break;
}
+
+ mutex_unlock(&dpm_list_mtx);
+
put_device(dev);
- if (async_error)
+ mutex_lock(&dpm_list_mtx);
+
+ if (error || async_error)
break;
}
mutex_unlock(&dpm_list_mtx);
struct device *dev = to_device(dpm_prepared_list.prev);
get_device(dev);
+
mutex_unlock(&dpm_list_mtx);
error = device_suspend(dev);
mutex_lock(&dpm_list_mtx);
+
if (error) {
pm_dev_err(dev, state, "", error);
dpm_save_failed_dev(dev_name(dev));
- put_device(dev);
- break;
- }
- if (!list_empty(&dev->power.entry))
+ } else if (!list_empty(&dev->power.entry)) {
list_move(&dev->power.entry, &dpm_suspended_list);
+ }
+
+ mutex_unlock(&dpm_list_mtx);
+
put_device(dev);
- if (async_error)
+
+ mutex_lock(&dpm_list_mtx);
+
+ if (error || async_error)
break;
}
mutex_unlock(&dpm_list_mtx);
struct device *dev = to_device(dpm_list.next);
get_device(dev);
+
mutex_unlock(&dpm_list_mtx);
trace_device_pm_callback_start(dev, "", state.event);
trace_device_pm_callback_end(dev, error);
mutex_lock(&dpm_list_mtx);
- if (error) {
- if (error == -EAGAIN) {
- put_device(dev);
- error = 0;
- continue;
- }
+
+ if (!error) {
+ dev->power.is_prepared = true;
+ if (!list_empty(&dev->power.entry))
+ list_move_tail(&dev->power.entry, &dpm_prepared_list);
+ } else if (error == -EAGAIN) {
+ error = 0;
+ } else {
dev_info(dev, "not prepared for power transition: code %d\n",
error);
- put_device(dev);
- break;
}
- dev->power.is_prepared = true;
- if (!list_empty(&dev->power.entry))
- list_move_tail(&dev->power.entry, &dpm_prepared_list);
+
+ mutex_unlock(&dpm_list_mtx);
+
put_device(dev);
+
+ mutex_lock(&dpm_list_mtx);
}
mutex_unlock(&dpm_list_mtx);
trace_suspend_resume(TPS("dpm_prepare"), state.event, false);
{
struct bcma_bus *bus;
int err = -ENOMEM;
- const char *name;
u32 val;
/* Alloc */
if (err)
goto err_kfree_bus;
- name = dev_name(&dev->dev);
- if (dev->driver && dev->driver->name)
- name = dev->driver->name;
- err = pci_request_regions(dev, name);
+ err = pci_request_regions(dev, "bcma-pci-bridge");
if (err)
goto err_pci_disable;
pci_set_master(dev);
return 0;
}
-static DEFINE_MUTEX(ataflop_probe_lock);
-
static void ataflop_probe(dev_t dev)
{
int drive = MINOR(dev) & 3;
if (drive >= FD_MAX_UNITS || type >= NUM_DISK_MINORS)
return;
- mutex_lock(&ataflop_probe_lock);
- if (!unit[drive].disk[type]) {
- if (ataflop_alloc_disk(drive, type) == 0) {
- add_disk(unit[drive].disk[type]);
- unit[drive].registered[type] = true;
+ if (unit[drive].disk[type])
+ return;
+ if (ataflop_alloc_disk(drive, type))
+ return;
+ if (add_disk(unit[drive].disk[type]))
+ goto cleanup_disk;
+ unit[drive].registered[type] = true;
+ return;
+
+cleanup_disk:
+ blk_cleanup_disk(unit[drive].disk[type]);
+ unit[drive].disk[type] = NULL;
+}
+
+static void atari_floppy_cleanup(void)
+{
+ int i;
+ int type;
+
+ for (i = 0; i < FD_MAX_UNITS; i++) {
+ for (type = 0; type < NUM_DISK_MINORS; type++) {
+ if (!unit[i].disk[type])
+ continue;
+ del_gendisk(unit[i].disk[type]);
+ blk_cleanup_queue(unit[i].disk[type]->queue);
+ put_disk(unit[i].disk[type]);
}
+ blk_mq_free_tag_set(&unit[i].tag_set);
}
- mutex_unlock(&ataflop_probe_lock);
+
+ del_timer_sync(&fd_timer);
+ atari_stram_free(DMABuffer);
}
static void atari_cleanup_floppy_disk(struct atari_floppy_struct *fs)
/* Amiga, Mac, ... don't have Atari-compatible floppy :-) */
return -ENODEV;
- mutex_lock(&ataflop_probe_lock);
- ret = __register_blkdev(FLOPPY_MAJOR, "fd", ataflop_probe);
- if (ret)
- goto out_unlock;
-
for (i = 0; i < FD_MAX_UNITS; i++) {
memset(&unit[i].tag_set, 0, sizeof(unit[i].tag_set));
unit[i].tag_set.ops = &ataflop_mq_ops;
UseTrackbuffer ? "" : "no ");
config_types();
- return 0;
+ ret = __register_blkdev(FLOPPY_MAJOR, "fd", ataflop_probe);
+ if (ret) {
+ printk(KERN_ERR "atari_floppy_init: cannot register block device\n");
+ atari_floppy_cleanup();
+ }
+ return ret;
err_out_dma:
atari_stram_free(DMABuffer);
while (--i >= 0)
atari_cleanup_floppy_disk(&unit[i]);
- unregister_blkdev(FLOPPY_MAJOR, "fd");
-out_unlock:
- mutex_unlock(&ataflop_probe_lock);
return ret;
}
static void __exit atari_floppy_exit(void)
{
- int i;
-
- for (i = 0; i < FD_MAX_UNITS; i++)
- atari_cleanup_floppy_disk(&unit[i]);
unregister_blkdev(FLOPPY_MAJOR, "fd");
-
- del_timer_sync(&fd_timer);
- atari_stram_free( DMABuffer );
+ atari_floppy_cleanup();
}
module_init(atari_floppy_init)
struct brd_device *brd;
struct gendisk *disk;
char buf[DISK_NAME_LEN];
+ int err = -ENOMEM;
mutex_lock(&brd_devices_mutex);
list_for_each_entry(brd, &brd_devices, brd_list) {
/* Tell the block layer that this is not a rotational device */
blk_queue_flag_set(QUEUE_FLAG_NONROT, disk->queue);
blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM, disk->queue);
- add_disk(disk);
+ err = add_disk(disk);
+ if (err)
+ goto out_cleanup_disk;
return 0;
+out_cleanup_disk:
+ blk_cleanup_disk(disk);
out_free_dev:
mutex_lock(&brd_devices_mutex);
list_del(&brd->brd_list);
mutex_unlock(&brd_devices_mutex);
kfree(brd);
- return -ENOMEM;
+ return err;
}
static void brd_probe(dev_t dev)
err = add_disk(disk);
if (err)
- goto out_cleanup_disk;
+ goto out_idr_remove_vol;
/* inherit the connection state */
device->state.conn = first_connection(resource)->cstate;
drbd_debugfs_device_add(device);
return NO_ERROR;
-out_cleanup_disk:
- blk_cleanup_disk(disk);
out_idr_remove_vol:
idr_remove(&connection->peer_devices, vnr);
out_idr_remove_from_resource:
return;
mutex_lock(&floppy_probe_lock);
- if (!disks[drive][type]) {
- if (floppy_alloc_disk(drive, type) == 0)
- add_disk(disks[drive][type]);
- }
+ if (disks[drive][type])
+ goto out;
+ if (floppy_alloc_disk(drive, type))
+ goto out;
+ if (add_disk(disks[drive][type]))
+ goto cleanup_disk;
+out:
+ mutex_unlock(&floppy_probe_lock);
+ return;
+
+cleanup_disk:
+ blk_cleanup_disk(disks[drive][type]);
+ disks[drive][type] = NULL;
mutex_unlock(&floppy_probe_lock);
}
goto out_free_dev;
i = err;
- err = -ENOMEM;
lo->tag_set.ops = &loop_mq_ops;
lo->tag_set.nr_hw_queues = 1;
lo->tag_set.queue_depth = 128;
mutex_lock(&nbd_index_mutex);
idr_remove(&nbd_index_idr, nbd->index);
mutex_unlock(&nbd_index_mutex);
-
+ destroy_workqueue(nbd->recv_workq);
kfree(nbd);
}
if (cmd->index != index) {
dev_err(disk_to_dev(nbd->disk), "Unexpected reply %d from different sock %d (expected %d)",
tag, index, cmd->index);
+ ret = -ENOENT;
+ goto out;
}
if (cmd->cmd_cookie != nbd_handle_to_cookie(handle)) {
dev_err(disk_to_dev(nbd->disk), "Double reply on req %p, cmd_cookie %u, handle cookie %u\n",
kfree(nbd->config);
nbd->config = NULL;
- if (nbd->recv_workq)
- destroy_workqueue(nbd->recv_workq);
- nbd->recv_workq = NULL;
-
nbd->tag_set.timeout = 0;
nbd->disk->queue->limits.discard_granularity = 0;
nbd->disk->queue->limits.discard_alignment = 0;
return -EINVAL;
}
- nbd->recv_workq = alloc_workqueue("knbd%d-recv",
- WQ_MEM_RECLAIM | WQ_HIGHPRI |
- WQ_UNBOUND, 0, nbd->index);
- if (!nbd->recv_workq) {
- dev_err(disk_to_dev(nbd->disk), "Could not allocate knbd recv work queue.\n");
- return -ENOMEM;
- }
-
blk_mq_update_nr_hw_queues(&nbd->tag_set, config->num_connections);
nbd->pid = task_pid_nr(current);
}
nbd->disk = disk;
+ nbd->recv_workq = alloc_workqueue("nbd%d-recv",
+ WQ_MEM_RECLAIM | WQ_HIGHPRI |
+ WQ_UNBOUND, 0, nbd->index);
+ if (!nbd->recv_workq) {
+ dev_err(disk_to_dev(nbd->disk), "Could not allocate knbd recv work queue.\n");
+ err = -ENOMEM;
+ goto out_err_disk;
+ }
+
/*
* Tell the block layer that we are not a rotational device
*/
disk->major = NBD_MAJOR;
/* Too big first_minor can cause duplicate creation of
- * sysfs files/links, since first_minor will be truncated to
- * byte in __device_add_disk().
+ * sysfs files/links, since index << part_shift might overflow, or
+ * MKDEV() expect that the max bits of first_minor is 20.
*/
disk->first_minor = index << part_shift;
- if (disk->first_minor > 0xff) {
+ if (disk->first_minor < index || disk->first_minor > MINORMASK) {
err = -EINVAL;
- goto out_free_idr;
+ goto out_free_work;
}
disk->minors = 1 << part_shift;
sprintf(disk->disk_name, "nbd%d", index);
err = add_disk(disk);
if (err)
- goto out_err_disk;
+ goto out_free_work;
/*
* Now publish the device.
nbd_total_devices++;
return nbd;
+out_free_work:
+ destroy_workqueue(nbd->recv_workq);
out_err_disk:
blk_cleanup_disk(disk);
out_free_idr:
nbd_disconnect(nbd);
sock_shutdown(nbd);
/*
- * Make sure recv thread has finished, so it does not drop the last
- * config ref and try to destroy the workqueue from inside the work
- * queue. And this also ensure that we can safely call nbd_clear_que()
+ * Make sure recv thread has finished, we can safely call nbd_clear_que()
* to cancel the inflight I/Os.
*/
- if (nbd->recv_workq)
- flush_workqueue(nbd->recv_workq);
+ flush_workqueue(nbd->recv_workq);
nbd_clear_que(nbd);
nbd->task_setup = NULL;
mutex_unlock(&nbd->config_lock);
gendisk->disk_name, priv->model, priv->raw_capacity >> 11,
get_capacity(gendisk) >> 11);
- device_add_disk(&dev->sbd.core, gendisk, NULL);
- return 0;
+ error = device_add_disk(&dev->sbd.core, gendisk, NULL);
+ if (error)
+ goto fail_cleanup_disk;
+ return 0;
+fail_cleanup_disk:
+ blk_cleanup_disk(gendisk);
fail_free_tag_set:
blk_mq_free_tag_set(&priv->tag_set);
fail_teardown:
dev_info(&dev->core, "%s: Using %llu MiB of GPU memory\n",
gendisk->disk_name, get_capacity(gendisk) >> 11);
- device_add_disk(&dev->core, gendisk, NULL);
+ error = device_add_disk(&dev->core, gendisk, NULL);
+ if (error)
+ goto out_cleanup_disk;
+
return 0;
+out_cleanup_disk:
+ blk_cleanup_disk(gendisk);
out_cache_cleanup:
remove_proc_entry(DEVICE_NAME, NULL);
ps3vram_cache_cleanup(dev);
if (IS_ERR(g)) {
printk(KERN_ERR PFX "%s: Could not allocate gendisk.\n",
port->vio.name);
- blk_mq_free_tag_set(&port->tag_set);
- return PTR_ERR(g);
+ err = PTR_ERR(g);
+ goto out_free_tag;
}
port->disk = g;
port->vdisk_size, (port->vdisk_size >> (20 - 9)),
port->vio.ver.major, port->vio.ver.minor);
- device_add_disk(&port->vio.vdev->dev, g, NULL);
+ err = device_add_disk(&port->vio.vdev->dev, g, NULL);
+ if (err)
+ goto out_cleanup_disk;
return 0;
+
+out_cleanup_disk:
+ blk_cleanup_disk(g);
+out_free_tag:
+ blk_mq_free_tag_set(&port->tag_set);
+ return err;
}
static struct ldc_channel_config vdc_ldc_cfg = {
static int z2ram_register_disk(int minor)
{
struct gendisk *disk;
+ int err;
disk = blk_mq_alloc_disk(&tag_set, NULL);
if (IS_ERR(disk))
sprintf(disk->disk_name, "z2ram");
z2ram_gendisk[minor] = disk;
- add_disk(disk);
- return 0;
+ err = add_disk(disk);
+ if (err)
+ blk_cleanup_disk(disk);
+ return err;
}
static int __init z2_init(void)
return len;
}
-static ssize_t idle_store(struct device *dev,
- struct device_attribute *attr, const char *buf, size_t len)
+/*
+ * Mark all pages which are older than or equal to cutoff as IDLE.
+ * Callers should hold the zram init lock in read mode
+ */
+static void mark_idle(struct zram *zram, ktime_t cutoff)
{
- struct zram *zram = dev_to_zram(dev);
+ int is_idle = 1;
unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
int index;
- if (!sysfs_streq(buf, "all"))
- return -EINVAL;
-
- down_read(&zram->init_lock);
- if (!init_done(zram)) {
- up_read(&zram->init_lock);
- return -EINVAL;
- }
-
for (index = 0; index < nr_pages; index++) {
/*
* Do not mark ZRAM_UNDER_WB slot as ZRAM_IDLE to close race.
*/
zram_slot_lock(zram, index);
if (zram_allocated(zram, index) &&
- !zram_test_flag(zram, index, ZRAM_UNDER_WB))
- zram_set_flag(zram, index, ZRAM_IDLE);
+ !zram_test_flag(zram, index, ZRAM_UNDER_WB)) {
+#ifdef CONFIG_ZRAM_MEMORY_TRACKING
+ is_idle = !cutoff || ktime_after(cutoff, zram->table[index].ac_time);
+#endif
+ if (is_idle)
+ zram_set_flag(zram, index, ZRAM_IDLE);
+ }
zram_slot_unlock(zram, index);
}
+}
- up_read(&zram->init_lock);
+static ssize_t idle_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct zram *zram = dev_to_zram(dev);
+ ktime_t cutoff_time = 0;
+ ssize_t rv = -EINVAL;
- return len;
+ if (!sysfs_streq(buf, "all")) {
+ /*
+ * If it did not parse as 'all' try to treat it as an integer when
+ * we have memory tracking enabled.
+ */
+ u64 age_sec;
+
+ if (IS_ENABLED(CONFIG_ZRAM_MEMORY_TRACKING) && !kstrtoull(buf, 0, &age_sec))
+ cutoff_time = ktime_sub(ktime_get_boottime(),
+ ns_to_ktime(age_sec * NSEC_PER_SEC));
+ else
+ goto out;
+ }
+
+ down_read(&zram->init_lock);
+ if (!init_done(zram))
+ goto out_unlock;
+
+ /* A cutoff_time of 0 marks everything as idle, this is the "all" behavior */
+ mark_idle(zram, cutoff_time);
+ rv = len;
+
+out_unlock:
+ up_read(&zram->init_lock);
+out:
+ return rv;
}
#ifdef CONFIG_ZRAM_WRITEBACK
{
struct bio *bio;
- bio = bio_alloc(GFP_ATOMIC, 1);
+ bio = bio_alloc(GFP_NOIO, 1);
if (!bio)
return -ENOMEM;
zram_test_flag(zram, index, ZRAM_HUGE) ? 'h' : '.',
zram_test_flag(zram, index, ZRAM_IDLE) ? 'i' : '.');
- if (count < copied) {
+ if (count <= copied) {
zram_slot_unlock(zram, index);
break;
}
set_capacity_and_notify(zram->disk, 0);
part_stat_set_all(zram->disk->part0, 0);
- up_write(&zram->init_lock);
/* I/O operation under all of CPU are done so let's free */
zram_meta_free(zram, disksize);
memset(&zram->stats, 0, sizeof(zram->stats));
zcomp_destroy(comp);
reset_bdev(zram);
+
+ up_write(&zram->init_lock);
}
static ssize_t disksize_store(struct device *dev,
mutex_unlock(&bdev->bd_disk->open_mutex);
/* Make sure all the pending I/O are finished */
- fsync_bdev(bdev);
+ sync_blockdev(bdev);
zram_reset_device(zram);
mutex_lock(&bdev->bd_disk->open_mutex);
blk_queue_max_write_zeroes_sectors(zram->disk->queue, UINT_MAX);
blk_queue_flag_set(QUEUE_FLAG_STABLE_WRITES, zram->disk->queue);
- device_add_disk(NULL, zram->disk, zram_disk_attr_groups);
+ ret = device_add_disk(NULL, zram->disk, zram_disk_attr_groups);
+ if (ret)
+ goto out_cleanup_disk;
strlcpy(zram->compressor, default_compressor, sizeof(zram->compressor));
pr_info("Added device: %s\n", zram->disk->disk_name);
return device_id;
+out_cleanup_disk:
+ blk_cleanup_disk(zram->disk);
out_free_idr:
idr_remove(&zram_index_idr, device_id);
out_free_dev:
static int zram_remove(struct zram *zram)
{
struct block_device *bdev = zram->disk->part0;
+ bool claimed;
mutex_lock(&bdev->bd_disk->open_mutex);
- if (bdev->bd_openers || zram->claim) {
+ if (bdev->bd_openers) {
mutex_unlock(&bdev->bd_disk->open_mutex);
return -EBUSY;
}
- zram->claim = true;
+ claimed = zram->claim;
+ if (!claimed)
+ zram->claim = true;
mutex_unlock(&bdev->bd_disk->open_mutex);
zram_debugfs_unregister(zram);
- /* Make sure all the pending I/O are finished */
- fsync_bdev(bdev);
- zram_reset_device(zram);
+ if (claimed) {
+ /*
+ * If we were claimed by reset_store(), del_gendisk() will
+ * wait until reset_store() is done, so nothing need to do.
+ */
+ ;
+ } else {
+ /* Make sure all the pending I/O are finished */
+ sync_blockdev(bdev);
+ zram_reset_device(zram);
+ }
pr_info("Removed device: %s\n", zram->disk->disk_name);
del_gendisk(zram->disk);
+
+ /* del_gendisk drains pending reset_store */
+ WARN_ON_ONCE(claimed && zram->claim);
+
+ /*
+ * disksize_store() may be called in between zram_reset_device()
+ * and del_gendisk(), so run the last reset to avoid leaking
+ * anything allocated with disksize_store()
+ */
+ zram_reset_device(zram);
+
blk_cleanup_disk(zram->disk);
kfree(zram);
return 0;
static int zram_remove_cb(int id, void *ptr, void *data)
{
- zram_remove(ptr);
+ WARN_ON_ONCE(zram_remove(ptr));
return 0;
}
list_add_tail(&gdev->next, &brcmstb_gisb_arb_device_list);
#ifdef CONFIG_MIPS
- board_be_handler = brcmstb_bus_error_handler;
+ mips_set_be_handler(brcmstb_bus_error_handler);
#endif
if (list_is_singular(&brcmstb_gisb_arb_device_list)) {
#include <linux/clk-provider.h>
#include <linux/regmap.h>
-#include <linux/slab.h>
#include "owl-factor.h"
static struct clk_hw_onecell_data *aspeed_g6_clk_data;
static void __iomem *scu_g6_base;
+/* AST2600 revision: A0, A1, A2, etc */
+static u8 soc_rev;
/*
* Clocks marked with CLK_IS_CRITICAL:
static struct clk_hw *ast2600_calc_apll(const char *name, u32 val)
{
unsigned int mult, div;
- u32 chip_id = readl(scu_g6_base + ASPEED_G6_SILICON_REV);
- if (((chip_id & CHIP_REVISION_ID) >> 16) >= 2) {
+ if (soc_rev >= 2) {
if (val & BIT(24)) {
/* Pass through mode */
mult = div = 1;
static void __init aspeed_g6_cc(struct regmap *map)
{
struct clk_hw *hw;
- u32 val, div, divbits, chip_id, axi_div, ahb_div;
+ u32 val, div, divbits, axi_div, ahb_div;
clk_hw_register_fixed_rate(NULL, "clkin", NULL, 0, 25000000);
axi_div = 2;
divbits = (val >> 11) & 0x3;
- regmap_read(map, ASPEED_G6_SILICON_REV, &chip_id);
- if (chip_id & BIT(16)) {
+ if (soc_rev >= 1) {
if (!divbits) {
ahb_div = ast2600_a1_axi_ahb200_tbl[(val >> 8) & 0x3];
if (val & BIT(16))
if (!scu_g6_base)
return;
+ soc_rev = (readl(scu_g6_base + ASPEED_G6_SILICON_REV) & CHIP_REVISION_ID) >> 16;
+
aspeed_g6_clk_data = kzalloc(struct_size(aspeed_g6_clk_data, hws,
ASPEED_G6_NUM_CLKS), GFP_KERNEL);
if (!aspeed_g6_clk_data)
return ret;
req->rate = tmp_req.rate;
+ req->best_parent_hw = tmp_req.best_parent_hw;
req->best_parent_rate = tmp_req.best_parent_rate;
return 0;
// SPDX-License-Identifier: GPL-2.0-or-later
/*
- * clk-si5351.c: Silicon Laboratories Si5351A/B/C I2C Clock Generator
+ * clk-si5351.c: Skyworks / Silicon Labs Si5351A/B/C I2C Clock Generator
*
* Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com>
* Rabeeh Khoury <rabeeh@solid-run.com>
*
* References:
* [1] "Si5351A/B/C Data Sheet"
- * https://www.silabs.com/Support%20Documents/TechnicalDocs/Si5351.pdf
- * [2] "Manually Generating an Si5351 Register Map"
- * https://www.silabs.com/Support%20Documents/TechnicalDocs/AN619.pdf
+ * https://www.skyworksinc.com/-/media/Skyworks/SL/documents/public/data-sheets/Si5351-B.pdf
+ * [2] "AN619: Manually Generating an Si5351 Register Map"
+ * https://www.skyworksinc.com/-/media/Skyworks/SL/documents/public/application-notes/AN619.pdf
*/
#include <linux/module.h>
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
- * clk-si5351.h: Silicon Laboratories Si5351A/B/C I2C Clock Generator
+ * clk-si5351.h: Skyworks / Silicon Labs Si5351A/B/C I2C Clock Generator
*
* Sebastian Hesselbarth <sebastian.hesselbarth@gmail.com>
* Rabeeh Khoury <rabeeh@solid-run.com>
static const struct of_device_id clk_vc5_of_match[];
-static int vc5_probe(struct i2c_client *client, const struct i2c_device_id *id)
+static int vc5_probe(struct i2c_client *client)
{
unsigned int oe, sd, src_mask = 0, src_val = 0;
struct vc5_driver_data *vc5;
.pm = &vc5_pm_ops,
.of_match_table = clk_vc5_of_match,
},
- .probe = vc5_probe,
+ .probe_new = vc5_probe,
.remove = vc5_remove,
.id_table = vc5_id,
};
#define imx8m_clk_hw_composite(name, parent_names, reg) \
_imx8m_clk_hw_composite(name, parent_names, reg, \
- IMX_COMPOSITE_CORE, IMX_COMPOSITE_CLK_FLAGS_DEFAULT)
+ 0, IMX_COMPOSITE_CLK_FLAGS_DEFAULT)
#define imx8m_clk_hw_composite_critical(name, parent_names, reg) \
_imx8m_clk_hw_composite(name, parent_names, reg, \
- IMX_COMPOSITE_CORE, IMX_COMPOSITE_CLK_FLAGS_CRITICAL)
+ 0, IMX_COMPOSITE_CLK_FLAGS_CRITICAL)
#define imx8m_clk_hw_composite_bus(name, parent_names, reg) \
_imx8m_clk_hw_composite(name, parent_names, reg, \
}
/* Impose hardware constraints */
- div = min_t(unsigned, div, 1 << clk_info->div.bits);
- div = max_t(unsigned, div, 1);
+ div = clamp_t(unsigned int, div, clk_info->div.div,
+ clk_info->div.div << clk_info->div.bits);
/*
* If the divider value itself must be divided before being written to
* the divider register, we must ensure we don't have any bits set that
* would be lost as a result of doing so.
*/
- div /= clk_info->div.div;
+ div = DIV_ROUND_UP(div, clk_info->div.div);
div *= clk_info->div.div;
return div;
#include <linux/delay.h>
#include <linux/of.h>
-#include <dt-bindings/clock/jz4725b-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4725b-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/io.h>
#include <linux/of.h>
-#include <dt-bindings/clock/jz4740-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4740-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/clk.h>
-#include <dt-bindings/clock/jz4760-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4760-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/io.h>
#include <linux/of.h>
-#include <dt-bindings/clock/jz4770-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4770-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/iopoll.h>
#include <linux/of.h>
-#include <dt-bindings/clock/jz4780-cgu.h>
+#include <dt-bindings/clock/ingenic,jz4780-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/io.h>
#include <linux/of.h>
-#include <dt-bindings/clock/x1000-cgu.h>
+#include <dt-bindings/clock/ingenic,x1000-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/io.h>
#include <linux/of.h>
-#include <dt-bindings/clock/x1830-cgu.h>
+#include <dt-bindings/clock/ingenic,x1830-cgu.h>
#include "cgu.h"
#include "pm.h"
#include <linux/clk-provider.h>
#include <linux/platform_device.h>
-#include <dt-bindings/clock/mt8195-clk.h>
-
static const struct mtk_gate_regs imp_iic_wrap_cg_regs = {
.set_ofs = 0xe08,
.clr_ofs = 0xe04,
},
};
-static struct clk_branch gcc_aggre1_pnoc_ahb_clk = {
- .halt_reg = 0x82014,
- .clkr = {
- .enable_reg = 0x82014,
- .enable_mask = BIT(0),
- .hw.init = &(struct clk_init_data){
- .name = "gcc_aggre1_pnoc_ahb_clk",
- .parent_names = (const char *[]){ "periph_noc_clk_src" },
- .num_parents = 1,
- .ops = &clk_branch2_ops,
- },
- },
-};
-
static struct clk_branch gcc_aggre2_ufs_axi_clk = {
.halt_reg = 0x83014,
.clkr = {
[GCC_AGGRE0_CNOC_AHB_CLK] = &gcc_aggre0_cnoc_ahb_clk.clkr,
[GCC_SMMU_AGGRE0_AXI_CLK] = &gcc_smmu_aggre0_axi_clk.clkr,
[GCC_SMMU_AGGRE0_AHB_CLK] = &gcc_smmu_aggre0_ahb_clk.clkr,
- [GCC_AGGRE1_PNOC_AHB_CLK] = &gcc_aggre1_pnoc_ahb_clk.clkr,
[GCC_AGGRE2_UFS_AXI_CLK] = &gcc_aggre2_ufs_axi_clk.clkr,
[GCC_AGGRE2_USB3_AXI_CLK] = &gcc_aggre2_usb3_axi_clk.clkr,
[GCC_QSPI_AHB_CLK] = &gcc_qspi_ahb_clk.clkr,
Build the driver for RK3368 Clock Driver.
config CLK_RK3399
- tristate "Rockchip RK3399 clock controller support"
+ bool "Rockchip RK3399 clock controller support"
depends on ARM64 || COMPILE_TEST
default y
help
Build the driver for RK3399 Clock Driver.
config CLK_RK3568
- tristate "Rockchip RK3568 clock controller support"
+ bool "Rockchip RK3568 clock controller support"
depends on ARM64 || COMPILE_TEST
default y
help
},
{ }
};
-MODULE_DEVICE_TABLE(of, clk_rk3399_match_table);
static int __init clk_rk3399_probe(struct platform_device *pdev)
{
.suppress_bind_attrs = true,
},
};
-module_platform_driver_probe(clk_rk3399_driver, clk_rk3399_probe);
-
-MODULE_DESCRIPTION("Rockchip RK3399 Clock Driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver_probe(clk_rk3399_driver, clk_rk3399_probe);
},
{ }
};
-MODULE_DEVICE_TABLE(of, clk_rk3568_match_table);
static int __init clk_rk3568_probe(struct platform_device *pdev)
{
.suppress_bind_attrs = true,
},
};
-module_platform_driver_probe(clk_rk3568_driver, clk_rk3568_probe);
-
-MODULE_DESCRIPTION("Rockchip RK3568 Clock Driver");
-MODULE_LICENSE("GPL");
+builtin_platform_driver_probe(clk_rk3568_driver, clk_rk3568_probe);
{ AM4_L3S_VPFE0_CLKCTRL, NULL, CLKF_SW_SUP, "l3_gclk" },
{ AM4_L3S_VPFE1_CLKCTRL, NULL, CLKF_SW_SUP, "l3_gclk" },
{ AM4_L3S_GPMC_CLKCTRL, NULL, CLKF_SW_SUP, "l3s_gclk" },
+ { AM4_L3S_ADC1_CLKCTRL, NULL, CLKF_SW_SUP, "l3s_gclk" },
{ AM4_L3S_MCASP0_CLKCTRL, NULL, CLKF_SW_SUP, "mcasp0_fck" },
{ AM4_L3S_MCASP1_CLKCTRL, NULL, CLKF_SW_SUP, "mcasp1_fck" },
{ AM4_L3S_MMC3_CLKCTRL, NULL, CLKF_SW_SUP, "mmc_clk" },
.compatible = "socionext,uniphier-pxs3-clock",
.data = uniphier_pxs3_sys_clk_data,
},
+ {
+ .compatible = "socionext,uniphier-nx1-clock",
+ .data = uniphier_nx1_sys_clk_data,
+ },
/* Media I/O clock, SD clock */
{
.compatible = "socionext,uniphier-ld4-mio-clock",
.compatible = "socionext,uniphier-pxs3-sd-clock",
.data = uniphier_pro5_sd_clk_data,
},
+ {
+ .compatible = "socionext,uniphier-nx1-sd-clock",
+ .data = uniphier_pro5_sd_clk_data,
+ },
/* Peripheral clock */
{
.compatible = "socionext,uniphier-ld4-peri-clock",
.compatible = "socionext,uniphier-pxs3-peri-clock",
.data = uniphier_pro4_peri_clk_data,
},
+ {
+ .compatible = "socionext,uniphier-nx1-peri-clock",
+ .data = uniphier_pro4_peri_clk_data,
+ },
+ /* SoC-glue clock */
+ {
+ .compatible = "socionext,uniphier-pro4-sg-clock",
+ .data = uniphier_pro4_sg_clk_data,
+ },
{ /* sentinel */ }
};
UNIPHIER_CLK_FACTOR("sd-200m", -1, "spll", 1, 10), \
UNIPHIER_CLK_FACTOR("sd-133m", -1, "spll", 1, 15)
+#define UNIPHIER_NX1_SYS_CLK_SD \
+ UNIPHIER_CLK_FACTOR("sd-200m", -1, "spll", 1, 4), \
+ UNIPHIER_CLK_FACTOR("sd-133m", -1, "spll", 1, 6)
+
#define UNIPHIER_LD4_SYS_CLK_NAND(idx) \
UNIPHIER_CLK_FACTOR("nand-50m", -1, "spll", 1, 32), \
UNIPHIER_CLK_GATE("nand", (idx), "nand-50m", 0x2104, 2)
UNIPHIER_CLK_GATE("sata0", 28, NULL, 0x210c, 7),
UNIPHIER_CLK_GATE("sata1", 29, NULL, 0x210c, 8),
UNIPHIER_CLK_GATE("sata-phy", 30, NULL, 0x210c, 21),
+ UNIPHIER_LD11_SYS_CLK_AIO(40),
+ UNIPHIER_LD11_SYS_CLK_EXIV(42),
/* CPU gears */
UNIPHIER_CLK_DIV4("cpll", 2, 3, 4, 8),
UNIPHIER_CLK_DIV4("spll", 2, 3, 4, 8),
"spll/4", "spll/8", "s2pll/4", "s2pll/8"),
{ /* sentinel */ }
};
+
+const struct uniphier_clk_data uniphier_nx1_sys_clk_data[] = {
+ UNIPHIER_CLK_FACTOR("cpll", -1, "ref", 100, 1), /* ARM: 2500 MHz */
+ UNIPHIER_CLK_FACTOR("spll", -1, "ref", 32, 1), /* 800 MHz */
+ UNIPHIER_CLK_FACTOR("uart", 0, "spll", 1, 6),
+ UNIPHIER_CLK_FACTOR("i2c", 1, "spll", 1, 16),
+ UNIPHIER_NX1_SYS_CLK_SD,
+ UNIPHIER_CLK_GATE("emmc", 4, NULL, 0x2108, 8),
+ UNIPHIER_CLK_GATE("ether", 6, NULL, 0x210c, 0),
+ UNIPHIER_CLK_GATE("usb30-0", 12, NULL, 0x210c, 16), /* =GIO */
+ UNIPHIER_CLK_GATE("usb30-1", 13, NULL, 0x210c, 20), /* =GIO1P */
+ UNIPHIER_CLK_GATE("usb30-hsphy0", 16, NULL, 0x210c, 24),
+ UNIPHIER_CLK_GATE("usb30-ssphy0", 17, NULL, 0x210c, 25),
+ UNIPHIER_CLK_GATE("usb30-ssphy1", 18, NULL, 0x210c, 26),
+ UNIPHIER_CLK_GATE("pcie", 24, NULL, 0x210c, 8),
+ UNIPHIER_CLK_GATE("voc", 52, NULL, 0x2110, 0),
+ UNIPHIER_CLK_GATE("hdmitx", 58, NULL, 0x2110, 8),
+ /* CPU gears */
+ UNIPHIER_CLK_DIV5("cpll", 2, 4, 8, 16, 32),
+ UNIPHIER_CLK_CPUGEAR("cpu-ca53", 33, 0x8080, 0xf, 5,
+ "cpll/2", "cpll/4", "cpll/8", "cpll/16",
+ "cpll/32"),
+ { /* sentinel */ }
+};
+
+const struct uniphier_clk_data uniphier_pro4_sg_clk_data[] = {
+ UNIPHIER_CLK_DIV("gpll", 4),
+ {
+ .name = "sata-ref",
+ .type = UNIPHIER_CLK_TYPE_MUX,
+ .idx = 0,
+ .data.mux = {
+ .parent_names = { "gpll/4", "ref", },
+ .num_parents = 2,
+ .reg = 0x1a28,
+ .masks = { 0x1, 0x1, },
+ .vals = { 0x0, 0x1, },
+ },
+ },
+ { /* sentinel */ }
+};
UNIPHIER_CLK_DIV2(parent, div0, div1), \
UNIPHIER_CLK_DIV2(parent, div2, div3)
+#define UNIPHIER_CLK_DIV5(parent, div0, div1, div2, div3, div4) \
+ UNIPHIER_CLK_DIV4(parent, div0, div1, div2, div3), \
+ UNIPHIER_CLK_DIV(parent, div4)
+
struct clk_hw *uniphier_clk_register_cpugear(struct device *dev,
struct regmap *regmap,
const char *name,
extern const struct uniphier_clk_data uniphier_ld11_sys_clk_data[];
extern const struct uniphier_clk_data uniphier_ld20_sys_clk_data[];
extern const struct uniphier_clk_data uniphier_pxs3_sys_clk_data[];
+extern const struct uniphier_clk_data uniphier_nx1_sys_clk_data[];
extern const struct uniphier_clk_data uniphier_ld4_mio_clk_data[];
extern const struct uniphier_clk_data uniphier_pro5_sd_clk_data[];
extern const struct uniphier_clk_data uniphier_ld4_peri_clk_data[];
extern const struct uniphier_clk_data uniphier_pro4_peri_clk_data[];
+extern const struct uniphier_clk_data uniphier_pro4_sg_clk_data[];
#endif /* __CLK_UNIPHIER_H__ */
struct device_node *parent;
struct regmap *map;
struct clk_icst_desc icst_desc;
- const char *name = np->name;
+ const char *name;
const char *parent_name;
struct clk *regclk;
enum icst_control_type ctype;
icst_desc.params = &icst525_apcp_cm_params;
ctype = ICST_INTEGRATOR_CP_CM_MEM;
} else {
- pr_err("unknown ICST clock %s\n", name);
+ pr_err("unknown ICST clock %pOF\n", np);
return;
}
/* Parent clock name is not the same as node parent */
parent_name = of_clk_get_parent_name(np, 0);
+ name = kasprintf(GFP_KERNEL, "%pOFP", np);
regclk = icst_clk_setup(NULL, &icst_desc, name, parent_name, map, ctype);
if (IS_ERR(regclk)) {
+ kfree(name);
pr_err("error setting up syscon ICST clock %s\n", name);
return;
}
*/
value &= ~GENMASK_ULL(31, 24);
value |= HWP_ENERGY_PERF_PREFERENCE(cpu->epp_cached);
- WRITE_ONCE(cpu->hwp_req_cached, value);
}
+ /*
+ * Clear the desired perf field in the cached HWP request value to
+ * prevent nonzero desired values from being leaked into the active
+ * mode.
+ */
+ value &= ~HWP_DESIRED_PERF(~0L);
+ WRITE_ONCE(cpu->hwp_req_cached, value);
+
value &= ~GENMASK_ULL(31, 0);
min_perf = HWP_LOWEST_PERF(READ_ONCE(cpu->hwp_cap_cached));
{
unsigned long flags;
+ if (!boot_cpu_has(X86_FEATURE_HWP_NOTIFY))
+ return;
+
/* wrmsrl_on_cpu has to be outside spinlock as this can result in IPC */
wrmsrl_on_cpu(cpudata->cpu, MSR_HWP_INTERRUPT, 0x00);
/* wrmsrl_on_cpu has to be outside spinlock as this can result in IPC */
wrmsrl_on_cpu(cpudata->cpu, MSR_HWP_INTERRUPT, 0x01);
+ wrmsrl_on_cpu(cpudata->cpu, MSR_HWP_STATUS, 0);
}
}
return intel_pstate_cpu_exit(policy);
}
+static int intel_cpufreq_suspend(struct cpufreq_policy *policy)
+{
+ intel_pstate_suspend(policy);
+
+ if (hwp_active) {
+ struct cpudata *cpu = all_cpu_data[policy->cpu];
+ u64 value = READ_ONCE(cpu->hwp_req_cached);
+
+ /*
+ * Clear the desired perf field in MSR_HWP_REQUEST in case
+ * intel_cpufreq_adjust_perf() is in use and the last value
+ * written by it may not be suitable.
+ */
+ value &= ~HWP_DESIRED_PERF(~0L);
+ wrmsrl_on_cpu(cpu->cpu, MSR_HWP_REQUEST, value);
+ WRITE_ONCE(cpu->hwp_req_cached, value);
+ }
+
+ return 0;
+}
+
static struct cpufreq_driver intel_cpufreq = {
.flags = CPUFREQ_CONST_LOOPS,
.verify = intel_cpufreq_verify_policy,
.exit = intel_cpufreq_cpu_exit,
.offline = intel_cpufreq_cpu_offline,
.online = intel_pstate_cpu_online,
- .suspend = intel_pstate_suspend,
+ .suspend = intel_cpufreq_suspend,
.resume = intel_pstate_resume,
.update_limits = intel_pstate_update_limits,
.name = "intel_cpufreq",
};
int ret;
- ret = strscpy(interface.name, pdev->driver->name,
+ ret = strscpy(interface.name, dev_driver_string(&pdev->dev),
sizeof(interface.name));
if (ret < 0)
return -ENAMETOOLONG;
pci_set_master(pdev);
- if (adf_enable_aer(accel_dev)) {
- dev_err(&pdev->dev, "Failed to enable aer.\n");
- ret = -EFAULT;
- goto out_err;
- }
+ adf_enable_aer(accel_dev);
if (pci_save_state(pdev)) {
dev_err(&pdev->dev, "Failed to save pci state.\n");
.probe = adf_probe,
.remove = adf_remove,
.sriov_configure = adf_sriov_configure,
+ .err_handler = &adf_err_handler,
};
module_pci_driver(adf_driver);
.probe = adf_probe,
.remove = adf_remove,
.sriov_configure = adf_sriov_configure,
+ .err_handler = &adf_err_handler,
};
static void adf_cleanup_pci_dev(struct adf_accel_dev *accel_dev)
}
pci_set_master(pdev);
- if (adf_enable_aer(accel_dev)) {
- dev_err(&pdev->dev, "Failed to enable aer\n");
- ret = -EFAULT;
- goto out_err_free_reg;
- }
+ adf_enable_aer(accel_dev);
if (pci_save_state(pdev)) {
dev_err(&pdev->dev, "Failed to save pci state\n");
.probe = adf_probe,
.remove = adf_remove,
.sriov_configure = adf_sriov_configure,
+ .err_handler = &adf_err_handler,
};
static void adf_cleanup_pci_dev(struct adf_accel_dev *accel_dev)
}
pci_set_master(pdev);
- if (adf_enable_aer(accel_dev)) {
- dev_err(&pdev->dev, "Failed to enable aer\n");
- ret = -EFAULT;
- goto out_err_free_reg;
- }
+ adf_enable_aer(accel_dev);
if (pci_save_state(pdev)) {
dev_err(&pdev->dev, "Failed to save pci state\n");
dev_info(&pdev->dev, "Device is up and running\n");
}
-static const struct pci_error_handlers adf_err_handler = {
+const struct pci_error_handlers adf_err_handler = {
.error_detected = adf_error_detected,
.slot_reset = adf_slot_reset,
.resume = adf_resume,
};
+EXPORT_SYMBOL_GPL(adf_err_handler);
/**
* adf_enable_aer() - Enable Advance Error Reporting for acceleration device
* Function enables PCI Advance Error Reporting for the
* QAT acceleration device accel_dev.
* To be used by QAT device specific drivers.
- *
- * Return: 0 on success, error code otherwise.
*/
-int adf_enable_aer(struct adf_accel_dev *accel_dev)
+void adf_enable_aer(struct adf_accel_dev *accel_dev)
{
struct pci_dev *pdev = accel_to_pci_dev(accel_dev);
- struct pci_driver *pdrv = pdev->driver;
- pdrv->err_handler = &adf_err_handler;
pci_enable_pcie_error_reporting(pdev);
- return 0;
}
EXPORT_SYMBOL_GPL(adf_enable_aer);
int adf_ae_start(struct adf_accel_dev *accel_dev);
int adf_ae_stop(struct adf_accel_dev *accel_dev);
-int adf_enable_aer(struct adf_accel_dev *accel_dev);
+extern const struct pci_error_handlers adf_err_handler;
+void adf_enable_aer(struct adf_accel_dev *accel_dev);
void adf_disable_aer(struct adf_accel_dev *accel_dev);
void adf_reset_sbr(struct adf_accel_dev *accel_dev);
void adf_reset_flr(struct adf_accel_dev *accel_dev);
.probe = adf_probe,
.remove = adf_remove,
.sriov_configure = adf_sriov_configure,
+ .err_handler = &adf_err_handler,
};
static void adf_cleanup_pci_dev(struct adf_accel_dev *accel_dev)
}
pci_set_master(pdev);
- if (adf_enable_aer(accel_dev)) {
- dev_err(&pdev->dev, "Failed to enable aer\n");
- ret = -EFAULT;
- goto out_err_free_reg;
- }
+ adf_enable_aer(accel_dev);
if (pci_save_state(pdev)) {
dev_err(&pdev->dev, "Failed to save pci state\n");
return -EINVAL;
}
+ if (CFMWS_INTERLEAVE_WAYS(cfmws) > CXL_DECODER_MAX_INTERLEAVE) {
+ dev_err(dev, "CFMWS Interleave Ways (%d) too large\n",
+ CFMWS_INTERLEAVE_WAYS(cfmws));
+ return -EINVAL;
+ }
+
expected_len = struct_size((cfmws), interleave_targets,
CFMWS_INTERLEAVE_WAYS(cfmws));
static void cxl_add_cfmws_decoders(struct device *dev,
struct cxl_port *root_port)
{
+ int target_map[CXL_DECODER_MAX_INTERLEAVE];
struct acpi_cedt_cfmws *cfmws;
struct cxl_decoder *cxld;
acpi_size len, cur = 0;
void *cedt_subtable;
- unsigned long flags;
int rc;
len = acpi_cedt->length - sizeof(*acpi_cedt);
while (cur < len) {
struct acpi_cedt_header *c = cedt_subtable + cur;
+ int i;
if (c->type != ACPI_CEDT_TYPE_CFMWS) {
cur += c->length;
continue;
}
- flags = cfmws_to_decoder_flags(cfmws->restrictions);
- cxld = devm_cxl_add_decoder(dev, root_port,
- CFMWS_INTERLEAVE_WAYS(cfmws),
- cfmws->base_hpa, cfmws->window_size,
- CFMWS_INTERLEAVE_WAYS(cfmws),
- CFMWS_INTERLEAVE_GRANULARITY(cfmws),
- CXL_DECODER_EXPANDER,
- flags);
+ for (i = 0; i < CFMWS_INTERLEAVE_WAYS(cfmws); i++)
+ target_map[i] = cfmws->interleave_targets[i];
- if (IS_ERR(cxld)) {
+ cxld = cxl_decoder_alloc(root_port,
+ CFMWS_INTERLEAVE_WAYS(cfmws));
+ if (IS_ERR(cxld))
+ goto next;
+
+ cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
+ cxld->target_type = CXL_DECODER_EXPANDER;
+ cxld->range = (struct range) {
+ .start = cfmws->base_hpa,
+ .end = cfmws->base_hpa + cfmws->window_size - 1,
+ };
+ cxld->interleave_ways = CFMWS_INTERLEAVE_WAYS(cfmws);
+ cxld->interleave_granularity =
+ CFMWS_INTERLEAVE_GRANULARITY(cfmws);
+
+ rc = cxl_decoder_add(cxld, target_map);
+ if (rc)
+ put_device(&cxld->dev);
+ else
+ rc = cxl_decoder_autoremove(dev, cxld);
+ if (rc) {
dev_err(dev, "Failed to add decoder for %#llx-%#llx\n",
cfmws->base_hpa, cfmws->base_hpa +
cfmws->window_size - 1);
- } else {
- dev_dbg(dev, "add: %s range %#llx-%#llx\n",
- dev_name(&cxld->dev), cfmws->base_hpa,
- cfmws->base_hpa + cfmws->window_size - 1);
+ goto next;
}
+ dev_dbg(dev, "add: %s range %#llx-%#llx\n",
+ dev_name(&cxld->dev), cfmws->base_hpa,
+ cfmws->base_hpa + cfmws->window_size - 1);
+next:
cur += c->length;
}
}
return IS_ERR(chbs) ? CXL_RESOURCE_NONE : chbs->base;
}
-struct cxl_walk_context {
- struct device *dev;
- struct pci_bus *root;
- struct cxl_port *port;
- int error;
- int count;
-};
-
-static int match_add_root_ports(struct pci_dev *pdev, void *data)
+__mock int match_add_root_ports(struct pci_dev *pdev, void *data)
{
struct cxl_walk_context *ctx = data;
struct pci_bus *root_bus = ctx->root;
return NULL;
}
-static struct acpi_device *to_cxl_host_bridge(struct device *dev)
+__mock struct acpi_device *to_cxl_host_bridge(struct device *host,
+ struct device *dev)
{
struct acpi_device *adev = to_acpi_device(dev);
*/
static int add_host_bridge_uport(struct device *match, void *arg)
{
- struct acpi_device *bridge = to_cxl_host_bridge(match);
struct cxl_port *root_port = arg;
struct device *host = root_port->dev.parent;
+ struct acpi_device *bridge = to_cxl_host_bridge(host, match);
struct acpi_pci_root *pci_root;
struct cxl_walk_context ctx;
+ int single_port_map[1], rc;
struct cxl_decoder *cxld;
struct cxl_dport *dport;
struct cxl_port *port;
dport = find_dport_by_dev(root_port, match);
if (!dport) {
dev_dbg(host, "host bridge expected and not found\n");
- return -ENODEV;
+ return 0;
}
port = devm_cxl_add_port(host, match, dport->component_reg_phys,
return -ENODEV;
if (ctx.error)
return ctx.error;
+ if (ctx.count > 1)
+ return 0;
/* TODO: Scan CHBCR for HDM Decoder resources */
/*
- * In the single-port host-bridge case there are no HDM decoders
- * in the CHBCR and a 1:1 passthrough decode is implied.
+ * Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability
+ * Structure) single ported host-bridges need not publish a decoder
+ * capability when a passthrough decode can be assumed, i.e. all
+ * transactions that the uport sees are claimed and passed to the single
+ * dport. Disable the range until the first CXL region is enumerated /
+ * activated.
*/
- if (ctx.count == 1) {
- cxld = devm_cxl_add_passthrough_decoder(host, port);
- if (IS_ERR(cxld))
- return PTR_ERR(cxld);
+ cxld = cxl_decoder_alloc(port, 1);
+ if (IS_ERR(cxld))
+ return PTR_ERR(cxld);
+
+ cxld->interleave_ways = 1;
+ cxld->interleave_granularity = PAGE_SIZE;
+ cxld->target_type = CXL_DECODER_EXPANDER;
+ cxld->range = (struct range) {
+ .start = 0,
+ .end = -1,
+ };
- dev_dbg(host, "add: %s\n", dev_name(&cxld->dev));
- }
+ device_lock(&port->dev);
+ dport = list_first_entry(&port->dports, typeof(*dport), list);
+ device_unlock(&port->dev);
- return 0;
+ single_port_map[0] = dport->port_id;
+
+ rc = cxl_decoder_add(cxld, single_port_map);
+ if (rc)
+ put_device(&cxld->dev);
+ else
+ rc = cxl_decoder_autoremove(host, cxld);
+
+ if (rc == 0)
+ dev_dbg(host, "add: %s\n", dev_name(&cxld->dev));
+ return rc;
}
static int add_host_bridge_dport(struct device *match, void *arg)
struct acpi_cedt_chbs *chbs;
struct cxl_port *root_port = arg;
struct device *host = root_port->dev.parent;
- struct acpi_device *bridge = to_cxl_host_bridge(match);
+ struct acpi_device *bridge = to_cxl_host_bridge(host, match);
if (!bridge)
return 0;
}
chbs = cxl_acpi_match_chbs(host, uid);
- if (IS_ERR(chbs))
- dev_dbg(host, "No CHBS found for Host Bridge: %s\n",
- dev_name(match));
+ if (IS_ERR(chbs)) {
+ dev_warn(host, "No CHBS found for Host Bridge: %s\n",
+ dev_name(match));
+ return 0;
+ }
rc = cxl_add_dport(root_port, match, uid, get_chbcr(chbs));
if (rc) {
return 1;
}
+static u32 cedt_instance(struct platform_device *pdev)
+{
+ const bool *native_acpi0017 = acpi_device_get_match_data(&pdev->dev);
+
+ if (native_acpi0017 && *native_acpi0017)
+ return 0;
+
+ /* for cxl_test request a non-canonical instance */
+ return U32_MAX;
+}
+
static int cxl_acpi_probe(struct platform_device *pdev)
{
int rc;
return PTR_ERR(root_port);
dev_dbg(host, "add: %s\n", dev_name(&root_port->dev));
- status = acpi_get_table(ACPI_SIG_CEDT, 0, &acpi_cedt);
+ status = acpi_get_table(ACPI_SIG_CEDT, cedt_instance(pdev), &acpi_cedt);
if (ACPI_FAILURE(status))
return -ENXIO;
return 0;
}
+static bool native_acpi0017 = true;
+
static const struct acpi_device_id cxl_acpi_ids[] = {
- { "ACPI0017", 0 },
- { "", 0 },
+ { "ACPI0017", (unsigned long) &native_acpi0017 },
+ { },
};
MODULE_DEVICE_TABLE(acpi, cxl_acpi_ids);
cxl_core-y += pmem.o
cxl_core-y += regs.o
cxl_core-y += memdev.o
+cxl_core-y += mbox.o
}
EXPORT_SYMBOL_GPL(cxl_add_dport);
-static struct cxl_decoder *
-cxl_decoder_alloc(struct cxl_port *port, int nr_targets, resource_size_t base,
- resource_size_t len, int interleave_ways,
- int interleave_granularity, enum cxl_decoder_type type,
- unsigned long flags)
+static int decoder_populate_targets(struct cxl_decoder *cxld,
+ struct cxl_port *port, int *target_map)
{
- struct cxl_decoder *cxld;
- struct device *dev;
- int rc = 0;
+ int rc = 0, i;
- if (interleave_ways < 1)
- return ERR_PTR(-EINVAL);
+ if (!target_map)
+ return 0;
device_lock(&port->dev);
- if (list_empty(&port->dports))
+ if (list_empty(&port->dports)) {
rc = -EINVAL;
+ goto out_unlock;
+ }
+
+ for (i = 0; i < cxld->nr_targets; i++) {
+ struct cxl_dport *dport = find_dport(port, target_map[i]);
+
+ if (!dport) {
+ rc = -ENXIO;
+ goto out_unlock;
+ }
+ cxld->target[i] = dport;
+ }
+
+out_unlock:
device_unlock(&port->dev);
- if (rc)
- return ERR_PTR(rc);
+
+ return rc;
+}
+
+struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets)
+{
+ struct cxl_decoder *cxld, cxld_const_init = {
+ .nr_targets = nr_targets,
+ };
+ struct device *dev;
+ int rc = 0;
+
+ if (nr_targets > CXL_DECODER_MAX_INTERLEAVE || nr_targets < 1)
+ return ERR_PTR(-EINVAL);
cxld = kzalloc(struct_size(cxld, target, nr_targets), GFP_KERNEL);
if (!cxld)
return ERR_PTR(-ENOMEM);
+ memcpy(cxld, &cxld_const_init, sizeof(cxld_const_init));
rc = ida_alloc(&port->decoder_ida, GFP_KERNEL);
if (rc < 0)
goto err;
- *cxld = (struct cxl_decoder) {
- .id = rc,
- .range = {
- .start = base,
- .end = base + len - 1,
- },
- .flags = flags,
- .interleave_ways = interleave_ways,
- .interleave_granularity = interleave_granularity,
- .target_type = type,
- };
-
- /* handle implied target_list */
- if (interleave_ways == 1)
- cxld->target[0] =
- list_first_entry(&port->dports, struct cxl_dport, list);
+ cxld->id = rc;
dev = &cxld->dev;
device_initialize(dev);
device_set_pm_not_required(dev);
kfree(cxld);
return ERR_PTR(rc);
}
+EXPORT_SYMBOL_GPL(cxl_decoder_alloc);
-struct cxl_decoder *
-devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets,
- resource_size_t base, resource_size_t len,
- int interleave_ways, int interleave_granularity,
- enum cxl_decoder_type type, unsigned long flags)
+int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map)
{
- struct cxl_decoder *cxld;
+ struct cxl_port *port;
struct device *dev;
int rc;
- cxld = cxl_decoder_alloc(port, nr_targets, base, len, interleave_ways,
- interleave_granularity, type, flags);
- if (IS_ERR(cxld))
- return cxld;
+ if (WARN_ON_ONCE(!cxld))
+ return -EINVAL;
+
+ if (WARN_ON_ONCE(IS_ERR(cxld)))
+ return PTR_ERR(cxld);
+
+ if (cxld->interleave_ways < 1)
+ return -EINVAL;
+
+ port = to_cxl_port(cxld->dev.parent);
+ rc = decoder_populate_targets(cxld, port, target_map);
+ if (rc)
+ return rc;
dev = &cxld->dev;
rc = dev_set_name(dev, "decoder%d.%d", port->id, cxld->id);
if (rc)
- goto err;
+ return rc;
- rc = device_add(dev);
- if (rc)
- goto err;
+ return device_add(dev);
+}
+EXPORT_SYMBOL_GPL(cxl_decoder_add);
- rc = devm_add_action_or_reset(host, unregister_cxl_dev, dev);
- if (rc)
- return ERR_PTR(rc);
- return cxld;
+static void cxld_unregister(void *dev)
+{
+ device_unregister(dev);
+}
-err:
- put_device(dev);
- return ERR_PTR(rc);
+int cxl_decoder_autoremove(struct device *host, struct cxl_decoder *cxld)
+{
+ return devm_add_action_or_reset(host, cxld_unregister, &cxld->dev);
}
-EXPORT_SYMBOL_GPL(devm_cxl_add_decoder);
+EXPORT_SYMBOL_GPL(cxl_decoder_autoremove);
/**
* __cxl_driver_register - register a driver for the cxl bus
{
int rc;
+ cxl_mbox_init();
+
rc = cxl_memdev_init();
if (rc)
return rc;
err:
cxl_memdev_exit();
+ cxl_mbox_exit();
return rc;
}
{
bus_unregister(&cxl_bus_type);
cxl_memdev_exit();
+ cxl_mbox_exit();
}
module_init(cxl_core_init);
extern struct attribute_group cxl_base_attribute_group;
-static inline void unregister_cxl_dev(void *dev)
-{
- device_unregister(dev);
-}
+struct cxl_send_command;
+struct cxl_mem_query_commands;
+int cxl_query_cmd(struct cxl_memdev *cxlmd,
+ struct cxl_mem_query_commands __user *q);
+int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s);
int cxl_memdev_init(void);
void cxl_memdev_exit(void);
+void cxl_mbox_init(void);
+void cxl_mbox_exit(void);
#endif /* __CXL_CORE_H__ */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <linux/io-64-nonatomic-lo-hi.h>
+#include <linux/security.h>
+#include <linux/debugfs.h>
+#include <linux/mutex.h>
+#include <cxlmem.h>
+#include <cxl.h>
+
+#include "core.h"
+
+static bool cxl_raw_allow_all;
+
+/**
+ * DOC: cxl mbox
+ *
+ * Core implementation of the CXL 2.0 Type-3 Memory Device Mailbox. The
+ * implementation is used by the cxl_pci driver to initialize the device
+ * and implement the cxl_mem.h IOCTL UAPI. It also implements the
+ * backend of the cxl_pmem_ctl() transport for LIBNVDIMM.
+ */
+
+#define cxl_for_each_cmd(cmd) \
+ for ((cmd) = &cxl_mem_commands[0]; \
+ ((cmd) - cxl_mem_commands) < ARRAY_SIZE(cxl_mem_commands); (cmd)++)
+
+#define CXL_CMD(_id, sin, sout, _flags) \
+ [CXL_MEM_COMMAND_ID_##_id] = { \
+ .info = { \
+ .id = CXL_MEM_COMMAND_ID_##_id, \
+ .size_in = sin, \
+ .size_out = sout, \
+ }, \
+ .opcode = CXL_MBOX_OP_##_id, \
+ .flags = _flags, \
+ }
+
+/*
+ * This table defines the supported mailbox commands for the driver. This table
+ * is made up of a UAPI structure. Non-negative values as parameters in the
+ * table will be validated against the user's input. For example, if size_in is
+ * 0, and the user passed in 1, it is an error.
+ */
+static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
+ CXL_CMD(IDENTIFY, 0, 0x43, CXL_CMD_FLAG_FORCE_ENABLE),
+#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
+ CXL_CMD(RAW, ~0, ~0, 0),
+#endif
+ CXL_CMD(GET_SUPPORTED_LOGS, 0, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
+ CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
+ CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
+ CXL_CMD(GET_LSA, 0x8, ~0, 0),
+ CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
+ CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
+ CXL_CMD(SET_PARTITION_INFO, 0x0a, 0, 0),
+ CXL_CMD(SET_LSA, ~0, 0, 0),
+ CXL_CMD(GET_ALERT_CONFIG, 0, 0x10, 0),
+ CXL_CMD(SET_ALERT_CONFIG, 0xc, 0, 0),
+ CXL_CMD(GET_SHUTDOWN_STATE, 0, 0x1, 0),
+ CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
+ CXL_CMD(GET_POISON, 0x10, ~0, 0),
+ CXL_CMD(INJECT_POISON, 0x8, 0, 0),
+ CXL_CMD(CLEAR_POISON, 0x48, 0, 0),
+ CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
+ CXL_CMD(SCAN_MEDIA, 0x11, 0, 0),
+ CXL_CMD(GET_SCAN_MEDIA, 0, ~0, 0),
+};
+
+/*
+ * Commands that RAW doesn't permit. The rationale for each:
+ *
+ * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
+ * coordination of transaction timeout values at the root bridge level.
+ *
+ * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
+ * and needs to be coordinated with HDM updates.
+ *
+ * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
+ * driver and any writes from userspace invalidates those contents.
+ *
+ * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
+ * to the device after it is marked clean, userspace can not make that
+ * assertion.
+ *
+ * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
+ * is kept up to date with patrol notifications and error management.
+ */
+static u16 cxl_disabled_raw_commands[] = {
+ CXL_MBOX_OP_ACTIVATE_FW,
+ CXL_MBOX_OP_SET_PARTITION_INFO,
+ CXL_MBOX_OP_SET_LSA,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE,
+ CXL_MBOX_OP_SCAN_MEDIA,
+ CXL_MBOX_OP_GET_SCAN_MEDIA,
+};
+
+/*
+ * Command sets that RAW doesn't permit. All opcodes in this set are
+ * disabled because they pass plain text security payloads over the
+ * user/kernel boundary. This functionality is intended to be wrapped
+ * behind the keys ABI which allows for encrypted payloads in the UAPI
+ */
+static u8 security_command_sets[] = {
+ 0x44, /* Sanitize */
+ 0x45, /* Persistent Memory Data-at-rest Security */
+ 0x46, /* Security Passthrough */
+};
+
+static bool cxl_is_security_command(u16 opcode)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(security_command_sets); i++)
+ if (security_command_sets[i] == (opcode >> 8))
+ return true;
+ return false;
+}
+
+static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
+{
+ struct cxl_mem_command *c;
+
+ cxl_for_each_cmd(c)
+ if (c->opcode == opcode)
+ return c;
+
+ return NULL;
+}
+
+/**
+ * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
+ * @cxlm: The CXL memory device to communicate with.
+ * @opcode: Opcode for the mailbox command.
+ * @in: The input payload for the mailbox command.
+ * @in_size: The length of the input payload
+ * @out: Caller allocated buffer for the output.
+ * @out_size: Expected size of output.
+ *
+ * Context: Any context. Will acquire and release mbox_mutex.
+ * Return:
+ * * %>=0 - Number of bytes returned in @out.
+ * * %-E2BIG - Payload is too large for hardware.
+ * * %-EBUSY - Couldn't acquire exclusive mailbox access.
+ * * %-EFAULT - Hardware error occurred.
+ * * %-ENXIO - Command completed, but device reported an error.
+ * * %-EIO - Unexpected output size.
+ *
+ * Mailbox commands may execute successfully yet the device itself reported an
+ * error. While this distinction can be useful for commands from userspace, the
+ * kernel will only be able to use results when both are successful.
+ *
+ * See __cxl_mem_mbox_send_cmd()
+ */
+int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, u16 opcode, void *in,
+ size_t in_size, void *out, size_t out_size)
+{
+ const struct cxl_mem_command *cmd = cxl_mem_find_command(opcode);
+ struct cxl_mbox_cmd mbox_cmd = {
+ .opcode = opcode,
+ .payload_in = in,
+ .size_in = in_size,
+ .size_out = out_size,
+ .payload_out = out,
+ };
+ int rc;
+
+ if (out_size > cxlm->payload_size)
+ return -E2BIG;
+
+ rc = cxlm->mbox_send(cxlm, &mbox_cmd);
+ if (rc)
+ return rc;
+
+ /* TODO: Map return code to proper kernel style errno */
+ if (mbox_cmd.return_code != CXL_MBOX_SUCCESS)
+ return -ENXIO;
+
+ /*
+ * Variable sized commands can't be validated and so it's up to the
+ * caller to do that if they wish.
+ */
+ if (cmd->info.size_out >= 0 && mbox_cmd.size_out != out_size)
+ return -EIO;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_mem_mbox_send_cmd);
+
+static bool cxl_mem_raw_command_allowed(u16 opcode)
+{
+ int i;
+
+ if (!IS_ENABLED(CONFIG_CXL_MEM_RAW_COMMANDS))
+ return false;
+
+ if (security_locked_down(LOCKDOWN_PCI_ACCESS))
+ return false;
+
+ if (cxl_raw_allow_all)
+ return true;
+
+ if (cxl_is_security_command(opcode))
+ return false;
+
+ for (i = 0; i < ARRAY_SIZE(cxl_disabled_raw_commands); i++)
+ if (cxl_disabled_raw_commands[i] == opcode)
+ return false;
+
+ return true;
+}
+
+/**
+ * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
+ * @cxlm: &struct cxl_mem device whose mailbox will be used.
+ * @send_cmd: &struct cxl_send_command copied in from userspace.
+ * @out_cmd: Sanitized and populated &struct cxl_mem_command.
+ *
+ * Return:
+ * * %0 - @out_cmd is ready to send.
+ * * %-ENOTTY - Invalid command specified.
+ * * %-EINVAL - Reserved fields or invalid values were used.
+ * * %-ENOMEM - Input or output buffer wasn't sized properly.
+ * * %-EPERM - Attempted to use a protected command.
+ * * %-EBUSY - Kernel has claimed exclusive access to this opcode
+ *
+ * The result of this command is a fully validated command in @out_cmd that is
+ * safe to send to the hardware.
+ *
+ * See handle_mailbox_cmd_from_user()
+ */
+static int cxl_validate_cmd_from_user(struct cxl_mem *cxlm,
+ const struct cxl_send_command *send_cmd,
+ struct cxl_mem_command *out_cmd)
+{
+ const struct cxl_command_info *info;
+ struct cxl_mem_command *c;
+
+ if (send_cmd->id == 0 || send_cmd->id >= CXL_MEM_COMMAND_ID_MAX)
+ return -ENOTTY;
+
+ /*
+ * The user can never specify an input payload larger than what hardware
+ * supports, but output can be arbitrarily large (simply write out as
+ * much data as the hardware provides).
+ */
+ if (send_cmd->in.size > cxlm->payload_size)
+ return -EINVAL;
+
+ /*
+ * Checks are bypassed for raw commands but a WARN/taint will occur
+ * later in the callchain
+ */
+ if (send_cmd->id == CXL_MEM_COMMAND_ID_RAW) {
+ const struct cxl_mem_command temp = {
+ .info = {
+ .id = CXL_MEM_COMMAND_ID_RAW,
+ .flags = 0,
+ .size_in = send_cmd->in.size,
+ .size_out = send_cmd->out.size,
+ },
+ .opcode = send_cmd->raw.opcode
+ };
+
+ if (send_cmd->raw.rsvd)
+ return -EINVAL;
+
+ /*
+ * Unlike supported commands, the output size of RAW commands
+ * gets passed along without further checking, so it must be
+ * validated here.
+ */
+ if (send_cmd->out.size > cxlm->payload_size)
+ return -EINVAL;
+
+ if (!cxl_mem_raw_command_allowed(send_cmd->raw.opcode))
+ return -EPERM;
+
+ memcpy(out_cmd, &temp, sizeof(temp));
+
+ return 0;
+ }
+
+ if (send_cmd->flags & ~CXL_MEM_COMMAND_FLAG_MASK)
+ return -EINVAL;
+
+ if (send_cmd->rsvd)
+ return -EINVAL;
+
+ if (send_cmd->in.rsvd || send_cmd->out.rsvd)
+ return -EINVAL;
+
+ /* Convert user's command into the internal representation */
+ c = &cxl_mem_commands[send_cmd->id];
+ info = &c->info;
+
+ /* Check that the command is enabled for hardware */
+ if (!test_bit(info->id, cxlm->enabled_cmds))
+ return -ENOTTY;
+
+ /* Check that the command is not claimed for exclusive kernel use */
+ if (test_bit(info->id, cxlm->exclusive_cmds))
+ return -EBUSY;
+
+ /* Check the input buffer is the expected size */
+ if (info->size_in >= 0 && info->size_in != send_cmd->in.size)
+ return -ENOMEM;
+
+ /* Check the output buffer is at least large enough */
+ if (info->size_out >= 0 && send_cmd->out.size < info->size_out)
+ return -ENOMEM;
+
+ memcpy(out_cmd, c, sizeof(*c));
+ out_cmd->info.size_in = send_cmd->in.size;
+ /*
+ * XXX: out_cmd->info.size_out will be controlled by the driver, and the
+ * specified number of bytes @send_cmd->out.size will be copied back out
+ * to userspace.
+ */
+
+ return 0;
+}
+
+int cxl_query_cmd(struct cxl_memdev *cxlmd,
+ struct cxl_mem_query_commands __user *q)
+{
+ struct device *dev = &cxlmd->dev;
+ struct cxl_mem_command *cmd;
+ u32 n_commands;
+ int j = 0;
+
+ dev_dbg(dev, "Query IOCTL\n");
+
+ if (get_user(n_commands, &q->n_commands))
+ return -EFAULT;
+
+ /* returns the total number if 0 elements are requested. */
+ if (n_commands == 0)
+ return put_user(ARRAY_SIZE(cxl_mem_commands), &q->n_commands);
+
+ /*
+ * otherwise, return max(n_commands, total commands) cxl_command_info
+ * structures.
+ */
+ cxl_for_each_cmd(cmd) {
+ const struct cxl_command_info *info = &cmd->info;
+
+ if (copy_to_user(&q->commands[j++], info, sizeof(*info)))
+ return -EFAULT;
+
+ if (j == n_commands)
+ break;
+ }
+
+ return 0;
+}
+
+/**
+ * handle_mailbox_cmd_from_user() - Dispatch a mailbox command for userspace.
+ * @cxlm: The CXL memory device to communicate with.
+ * @cmd: The validated command.
+ * @in_payload: Pointer to userspace's input payload.
+ * @out_payload: Pointer to userspace's output payload.
+ * @size_out: (Input) Max payload size to copy out.
+ * (Output) Payload size hardware generated.
+ * @retval: Hardware generated return code from the operation.
+ *
+ * Return:
+ * * %0 - Mailbox transaction succeeded. This implies the mailbox
+ * protocol completed successfully not that the operation itself
+ * was successful.
+ * * %-ENOMEM - Couldn't allocate a bounce buffer.
+ * * %-EFAULT - Something happened with copy_to/from_user.
+ * * %-EINTR - Mailbox acquisition interrupted.
+ * * %-EXXX - Transaction level failures.
+ *
+ * Creates the appropriate mailbox command and dispatches it on behalf of a
+ * userspace request. The input and output payloads are copied between
+ * userspace.
+ *
+ * See cxl_send_cmd().
+ */
+static int handle_mailbox_cmd_from_user(struct cxl_mem *cxlm,
+ const struct cxl_mem_command *cmd,
+ u64 in_payload, u64 out_payload,
+ s32 *size_out, u32 *retval)
+{
+ struct device *dev = cxlm->dev;
+ struct cxl_mbox_cmd mbox_cmd = {
+ .opcode = cmd->opcode,
+ .size_in = cmd->info.size_in,
+ .size_out = cmd->info.size_out,
+ };
+ int rc;
+
+ if (cmd->info.size_out) {
+ mbox_cmd.payload_out = kvzalloc(cmd->info.size_out, GFP_KERNEL);
+ if (!mbox_cmd.payload_out)
+ return -ENOMEM;
+ }
+
+ if (cmd->info.size_in) {
+ mbox_cmd.payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
+ cmd->info.size_in);
+ if (IS_ERR(mbox_cmd.payload_in)) {
+ kvfree(mbox_cmd.payload_out);
+ return PTR_ERR(mbox_cmd.payload_in);
+ }
+ }
+
+ dev_dbg(dev,
+ "Submitting %s command for user\n"
+ "\topcode: %x\n"
+ "\tsize: %ub\n",
+ cxl_command_names[cmd->info.id].name, mbox_cmd.opcode,
+ cmd->info.size_in);
+
+ dev_WARN_ONCE(dev, cmd->info.id == CXL_MEM_COMMAND_ID_RAW,
+ "raw command path used\n");
+
+ rc = cxlm->mbox_send(cxlm, &mbox_cmd);
+ if (rc)
+ goto out;
+
+ /*
+ * @size_out contains the max size that's allowed to be written back out
+ * to userspace. While the payload may have written more output than
+ * this it will have to be ignored.
+ */
+ if (mbox_cmd.size_out) {
+ dev_WARN_ONCE(dev, mbox_cmd.size_out > *size_out,
+ "Invalid return size\n");
+ if (copy_to_user(u64_to_user_ptr(out_payload),
+ mbox_cmd.payload_out, mbox_cmd.size_out)) {
+ rc = -EFAULT;
+ goto out;
+ }
+ }
+
+ *size_out = mbox_cmd.size_out;
+ *retval = mbox_cmd.return_code;
+
+out:
+ kvfree(mbox_cmd.payload_in);
+ kvfree(mbox_cmd.payload_out);
+ return rc;
+}
+
+int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s)
+{
+ struct cxl_mem *cxlm = cxlmd->cxlm;
+ struct device *dev = &cxlmd->dev;
+ struct cxl_send_command send;
+ struct cxl_mem_command c;
+ int rc;
+
+ dev_dbg(dev, "Send IOCTL\n");
+
+ if (copy_from_user(&send, s, sizeof(send)))
+ return -EFAULT;
+
+ rc = cxl_validate_cmd_from_user(cxlmd->cxlm, &send, &c);
+ if (rc)
+ return rc;
+
+ /* Prepare to handle a full payload for variable sized output */
+ if (c.info.size_out < 0)
+ c.info.size_out = cxlm->payload_size;
+
+ rc = handle_mailbox_cmd_from_user(cxlm, &c, send.in.payload,
+ send.out.payload, &send.out.size,
+ &send.retval);
+ if (rc)
+ return rc;
+
+ if (copy_to_user(s, &send, sizeof(send)))
+ return -EFAULT;
+
+ return 0;
+}
+
+static int cxl_xfer_log(struct cxl_mem *cxlm, uuid_t *uuid, u32 size, u8 *out)
+{
+ u32 remaining = size;
+ u32 offset = 0;
+
+ while (remaining) {
+ u32 xfer_size = min_t(u32, remaining, cxlm->payload_size);
+ struct cxl_mbox_get_log log = {
+ .uuid = *uuid,
+ .offset = cpu_to_le32(offset),
+ .length = cpu_to_le32(xfer_size)
+ };
+ int rc;
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_LOG, &log,
+ sizeof(log), out, xfer_size);
+ if (rc < 0)
+ return rc;
+
+ out += xfer_size;
+ remaining -= xfer_size;
+ offset += xfer_size;
+ }
+
+ return 0;
+}
+
+/**
+ * cxl_walk_cel() - Walk through the Command Effects Log.
+ * @cxlm: Device.
+ * @size: Length of the Command Effects Log.
+ * @cel: CEL
+ *
+ * Iterate over each entry in the CEL and determine if the driver supports the
+ * command. If so, the command is enabled for the device and can be used later.
+ */
+static void cxl_walk_cel(struct cxl_mem *cxlm, size_t size, u8 *cel)
+{
+ struct cxl_cel_entry *cel_entry;
+ const int cel_entries = size / sizeof(*cel_entry);
+ int i;
+
+ cel_entry = (struct cxl_cel_entry *) cel;
+
+ for (i = 0; i < cel_entries; i++) {
+ u16 opcode = le16_to_cpu(cel_entry[i].opcode);
+ struct cxl_mem_command *cmd = cxl_mem_find_command(opcode);
+
+ if (!cmd) {
+ dev_dbg(cxlm->dev,
+ "Opcode 0x%04x unsupported by driver", opcode);
+ continue;
+ }
+
+ set_bit(cmd->info.id, cxlm->enabled_cmds);
+ }
+}
+
+static struct cxl_mbox_get_supported_logs *cxl_get_gsl(struct cxl_mem *cxlm)
+{
+ struct cxl_mbox_get_supported_logs *ret;
+ int rc;
+
+ ret = kvmalloc(cxlm->payload_size, GFP_KERNEL);
+ if (!ret)
+ return ERR_PTR(-ENOMEM);
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_SUPPORTED_LOGS, NULL,
+ 0, ret, cxlm->payload_size);
+ if (rc < 0) {
+ kvfree(ret);
+ return ERR_PTR(rc);
+ }
+
+ return ret;
+}
+
+enum {
+ CEL_UUID,
+ VENDOR_DEBUG_UUID,
+};
+
+/* See CXL 2.0 Table 170. Get Log Input Payload */
+static const uuid_t log_uuid[] = {
+ [CEL_UUID] = DEFINE_CXL_CEL_UUID,
+ [VENDOR_DEBUG_UUID] = DEFINE_CXL_VENDOR_DEBUG_UUID,
+};
+
+/**
+ * cxl_mem_enumerate_cmds() - Enumerate commands for a device.
+ * @cxlm: The device.
+ *
+ * Returns 0 if enumerate completed successfully.
+ *
+ * CXL devices have optional support for certain commands. This function will
+ * determine the set of supported commands for the hardware and update the
+ * enabled_cmds bitmap in the @cxlm.
+ */
+int cxl_mem_enumerate_cmds(struct cxl_mem *cxlm)
+{
+ struct cxl_mbox_get_supported_logs *gsl;
+ struct device *dev = cxlm->dev;
+ struct cxl_mem_command *cmd;
+ int i, rc;
+
+ gsl = cxl_get_gsl(cxlm);
+ if (IS_ERR(gsl))
+ return PTR_ERR(gsl);
+
+ rc = -ENOENT;
+ for (i = 0; i < le16_to_cpu(gsl->entries); i++) {
+ u32 size = le32_to_cpu(gsl->entry[i].size);
+ uuid_t uuid = gsl->entry[i].uuid;
+ u8 *log;
+
+ dev_dbg(dev, "Found LOG type %pU of size %d", &uuid, size);
+
+ if (!uuid_equal(&uuid, &log_uuid[CEL_UUID]))
+ continue;
+
+ log = kvmalloc(size, GFP_KERNEL);
+ if (!log) {
+ rc = -ENOMEM;
+ goto out;
+ }
+
+ rc = cxl_xfer_log(cxlm, &uuid, size, log);
+ if (rc) {
+ kvfree(log);
+ goto out;
+ }
+
+ cxl_walk_cel(cxlm, size, log);
+ kvfree(log);
+
+ /* In case CEL was bogus, enable some default commands. */
+ cxl_for_each_cmd(cmd)
+ if (cmd->flags & CXL_CMD_FLAG_FORCE_ENABLE)
+ set_bit(cmd->info.id, cxlm->enabled_cmds);
+
+ /* Found the required CEL */
+ rc = 0;
+ }
+
+out:
+ kvfree(gsl);
+ return rc;
+}
+EXPORT_SYMBOL_GPL(cxl_mem_enumerate_cmds);
+
+/**
+ * cxl_mem_get_partition_info - Get partition info
+ * @cxlm: cxl_mem instance to update partition info
+ *
+ * Retrieve the current partition info for the device specified. The active
+ * values are the current capacity in bytes. If not 0, the 'next' values are
+ * the pending values, in bytes, which take affect on next cold reset.
+ *
+ * Return: 0 if no error: or the result of the mailbox command.
+ *
+ * See CXL @8.2.9.5.2.1 Get Partition Info
+ */
+static int cxl_mem_get_partition_info(struct cxl_mem *cxlm)
+{
+ struct cxl_mbox_get_partition_info {
+ __le64 active_volatile_cap;
+ __le64 active_persistent_cap;
+ __le64 next_volatile_cap;
+ __le64 next_persistent_cap;
+ } __packed pi;
+ int rc;
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_PARTITION_INFO,
+ NULL, 0, &pi, sizeof(pi));
+
+ if (rc)
+ return rc;
+
+ cxlm->active_volatile_bytes =
+ le64_to_cpu(pi.active_volatile_cap) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->active_persistent_bytes =
+ le64_to_cpu(pi.active_persistent_cap) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->next_volatile_bytes =
+ le64_to_cpu(pi.next_volatile_cap) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->next_persistent_bytes =
+ le64_to_cpu(pi.next_volatile_cap) * CXL_CAPACITY_MULTIPLIER;
+
+ return 0;
+}
+
+/**
+ * cxl_mem_identify() - Send the IDENTIFY command to the device.
+ * @cxlm: The device to identify.
+ *
+ * Return: 0 if identify was executed successfully.
+ *
+ * This will dispatch the identify command to the device and on success populate
+ * structures to be exported to sysfs.
+ */
+int cxl_mem_identify(struct cxl_mem *cxlm)
+{
+ /* See CXL 2.0 Table 175 Identify Memory Device Output Payload */
+ struct cxl_mbox_identify id;
+ int rc;
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_IDENTIFY, NULL, 0, &id,
+ sizeof(id));
+ if (rc < 0)
+ return rc;
+
+ cxlm->total_bytes =
+ le64_to_cpu(id.total_capacity) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->volatile_only_bytes =
+ le64_to_cpu(id.volatile_capacity) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->persistent_only_bytes =
+ le64_to_cpu(id.persistent_capacity) * CXL_CAPACITY_MULTIPLIER;
+ cxlm->partition_align_bytes =
+ le64_to_cpu(id.partition_align) * CXL_CAPACITY_MULTIPLIER;
+
+ dev_dbg(cxlm->dev,
+ "Identify Memory Device\n"
+ " total_bytes = %#llx\n"
+ " volatile_only_bytes = %#llx\n"
+ " persistent_only_bytes = %#llx\n"
+ " partition_align_bytes = %#llx\n",
+ cxlm->total_bytes, cxlm->volatile_only_bytes,
+ cxlm->persistent_only_bytes, cxlm->partition_align_bytes);
+
+ cxlm->lsa_size = le32_to_cpu(id.lsa_size);
+ memcpy(cxlm->firmware_version, id.fw_revision, sizeof(id.fw_revision));
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_mem_identify);
+
+int cxl_mem_create_range_info(struct cxl_mem *cxlm)
+{
+ int rc;
+
+ if (cxlm->partition_align_bytes == 0) {
+ cxlm->ram_range.start = 0;
+ cxlm->ram_range.end = cxlm->volatile_only_bytes - 1;
+ cxlm->pmem_range.start = cxlm->volatile_only_bytes;
+ cxlm->pmem_range.end = cxlm->volatile_only_bytes +
+ cxlm->persistent_only_bytes - 1;
+ return 0;
+ }
+
+ rc = cxl_mem_get_partition_info(cxlm);
+ if (rc) {
+ dev_err(cxlm->dev, "Failed to query partition information\n");
+ return rc;
+ }
+
+ dev_dbg(cxlm->dev,
+ "Get Partition Info\n"
+ " active_volatile_bytes = %#llx\n"
+ " active_persistent_bytes = %#llx\n"
+ " next_volatile_bytes = %#llx\n"
+ " next_persistent_bytes = %#llx\n",
+ cxlm->active_volatile_bytes, cxlm->active_persistent_bytes,
+ cxlm->next_volatile_bytes, cxlm->next_persistent_bytes);
+
+ cxlm->ram_range.start = 0;
+ cxlm->ram_range.end = cxlm->active_volatile_bytes - 1;
+
+ cxlm->pmem_range.start = cxlm->active_volatile_bytes;
+ cxlm->pmem_range.end =
+ cxlm->active_volatile_bytes + cxlm->active_persistent_bytes - 1;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_mem_create_range_info);
+
+struct cxl_mem *cxl_mem_create(struct device *dev)
+{
+ struct cxl_mem *cxlm;
+
+ cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+ if (!cxlm) {
+ dev_err(dev, "No memory available\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ mutex_init(&cxlm->mbox_mutex);
+ cxlm->dev = dev;
+
+ return cxlm;
+}
+EXPORT_SYMBOL_GPL(cxl_mem_create);
+
+static struct dentry *cxl_debugfs;
+
+void __init cxl_mbox_init(void)
+{
+ struct dentry *mbox_debugfs;
+
+ cxl_debugfs = debugfs_create_dir("cxl", NULL);
+ mbox_debugfs = debugfs_create_dir("mbox", cxl_debugfs);
+ debugfs_create_bool("raw_allow_all", 0600, mbox_debugfs,
+ &cxl_raw_allow_all);
+}
+
+void cxl_mbox_exit(void)
+{
+ debugfs_remove_recursive(cxl_debugfs);
+}
#include <cxlmem.h>
#include "core.h"
+static DECLARE_RWSEM(cxl_memdev_rwsem);
+
/*
* An entire PCI topology full of devices should be enough for any
* config
.groups = cxl_memdev_attribute_groups,
};
+/**
+ * set_exclusive_cxl_commands() - atomically disable user cxl commands
+ * @cxlm: cxl_mem instance to modify
+ * @cmds: bitmap of commands to mark exclusive
+ *
+ * Grab the cxl_memdev_rwsem in write mode to flush in-flight
+ * invocations of the ioctl path and then disable future execution of
+ * commands with the command ids set in @cmds.
+ */
+void set_exclusive_cxl_commands(struct cxl_mem *cxlm, unsigned long *cmds)
+{
+ down_write(&cxl_memdev_rwsem);
+ bitmap_or(cxlm->exclusive_cmds, cxlm->exclusive_cmds, cmds,
+ CXL_MEM_COMMAND_ID_MAX);
+ up_write(&cxl_memdev_rwsem);
+}
+EXPORT_SYMBOL_GPL(set_exclusive_cxl_commands);
+
+/**
+ * clear_exclusive_cxl_commands() - atomically enable user cxl commands
+ * @cxlm: cxl_mem instance to modify
+ * @cmds: bitmap of commands to mark available for userspace
+ */
+void clear_exclusive_cxl_commands(struct cxl_mem *cxlm, unsigned long *cmds)
+{
+ down_write(&cxl_memdev_rwsem);
+ bitmap_andnot(cxlm->exclusive_cmds, cxlm->exclusive_cmds, cmds,
+ CXL_MEM_COMMAND_ID_MAX);
+ up_write(&cxl_memdev_rwsem);
+}
+EXPORT_SYMBOL_GPL(clear_exclusive_cxl_commands);
+
+static void cxl_memdev_shutdown(struct device *dev)
+{
+ struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+ down_write(&cxl_memdev_rwsem);
+ cxlmd->cxlm = NULL;
+ up_write(&cxl_memdev_rwsem);
+}
+
static void cxl_memdev_unregister(void *_cxlmd)
{
struct cxl_memdev *cxlmd = _cxlmd;
struct device *dev = &cxlmd->dev;
- struct cdev *cdev = &cxlmd->cdev;
- const struct cdevm_file_operations *cdevm_fops;
-
- cdevm_fops = container_of(cdev->ops, typeof(*cdevm_fops), fops);
- cdevm_fops->shutdown(dev);
+ cxl_memdev_shutdown(dev);
cdev_device_del(&cxlmd->cdev, dev);
put_device(dev);
}
static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm,
const struct file_operations *fops)
{
- struct pci_dev *pdev = cxlm->pdev;
struct cxl_memdev *cxlmd;
struct device *dev;
struct cdev *cdev;
dev = &cxlmd->dev;
device_initialize(dev);
- dev->parent = &pdev->dev;
+ dev->parent = cxlm->dev;
dev->bus = &cxl_bus_type;
dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
dev->type = &cxl_memdev_type;
return ERR_PTR(rc);
}
+static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
+ unsigned long arg)
+{
+ switch (cmd) {
+ case CXL_MEM_QUERY_COMMANDS:
+ return cxl_query_cmd(cxlmd, (void __user *)arg);
+ case CXL_MEM_SEND_COMMAND:
+ return cxl_send_cmd(cxlmd, (void __user *)arg);
+ default:
+ return -ENOTTY;
+ }
+}
+
+static long cxl_memdev_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ struct cxl_memdev *cxlmd = file->private_data;
+ int rc = -ENXIO;
+
+ down_read(&cxl_memdev_rwsem);
+ if (cxlmd->cxlm)
+ rc = __cxl_memdev_ioctl(cxlmd, cmd, arg);
+ up_read(&cxl_memdev_rwsem);
+
+ return rc;
+}
+
+static int cxl_memdev_open(struct inode *inode, struct file *file)
+{
+ struct cxl_memdev *cxlmd =
+ container_of(inode->i_cdev, typeof(*cxlmd), cdev);
+
+ get_device(&cxlmd->dev);
+ file->private_data = cxlmd;
+
+ return 0;
+}
+
+static int cxl_memdev_release_file(struct inode *inode, struct file *file)
+{
+ struct cxl_memdev *cxlmd =
+ container_of(inode->i_cdev, typeof(*cxlmd), cdev);
+
+ put_device(&cxlmd->dev);
+
+ return 0;
+}
+
+static const struct file_operations cxl_memdev_fops = {
+ .owner = THIS_MODULE,
+ .unlocked_ioctl = cxl_memdev_ioctl,
+ .open = cxl_memdev_open,
+ .release = cxl_memdev_release_file,
+ .compat_ioctl = compat_ptr_ioctl,
+ .llseek = noop_llseek,
+};
+
struct cxl_memdev *
-devm_cxl_add_memdev(struct device *host, struct cxl_mem *cxlm,
- const struct cdevm_file_operations *cdevm_fops)
+devm_cxl_add_memdev(struct cxl_mem *cxlm)
{
struct cxl_memdev *cxlmd;
struct device *dev;
struct cdev *cdev;
int rc;
- cxlmd = cxl_memdev_alloc(cxlm, &cdevm_fops->fops);
+ cxlmd = cxl_memdev_alloc(cxlm, &cxl_memdev_fops);
if (IS_ERR(cxlmd))
return cxlmd;
if (rc)
goto err;
- rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
+ rc = devm_add_action_or_reset(cxlm->dev, cxl_memdev_unregister, cxlmd);
if (rc)
return ERR_PTR(rc);
return cxlmd;
* The cdev was briefly live, shutdown any ioctl operations that
* saw that state.
*/
- cdevm_fops->shutdown(dev);
+ cxl_memdev_shutdown(dev);
put_device(dev);
return ERR_PTR(rc);
}
/* Copyright(c) 2020 Intel Corporation. */
#include <linux/device.h>
#include <linux/slab.h>
+#include <linux/idr.h>
#include <cxlmem.h>
#include <cxl.h>
#include "core.h"
* operations, for example, namespace label access commands.
*/
+static DEFINE_IDA(cxl_nvdimm_bridge_ida);
+
static void cxl_nvdimm_bridge_release(struct device *dev)
{
struct cxl_nvdimm_bridge *cxl_nvb = to_cxl_nvdimm_bridge(dev);
+ ida_free(&cxl_nvdimm_bridge_ida, cxl_nvb->id);
kfree(cxl_nvb);
}
}
EXPORT_SYMBOL_GPL(to_cxl_nvdimm_bridge);
+__mock int match_nvdimm_bridge(struct device *dev, const void *data)
+{
+ return dev->type == &cxl_nvdimm_bridge_type;
+}
+
+struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_nvdimm *cxl_nvd)
+{
+ struct device *dev;
+
+ dev = bus_find_device(&cxl_bus_type, NULL, cxl_nvd, match_nvdimm_bridge);
+ if (!dev)
+ return NULL;
+ return to_cxl_nvdimm_bridge(dev);
+}
+EXPORT_SYMBOL_GPL(cxl_find_nvdimm_bridge);
+
static struct cxl_nvdimm_bridge *
cxl_nvdimm_bridge_alloc(struct cxl_port *port)
{
struct cxl_nvdimm_bridge *cxl_nvb;
struct device *dev;
+ int rc;
cxl_nvb = kzalloc(sizeof(*cxl_nvb), GFP_KERNEL);
if (!cxl_nvb)
return ERR_PTR(-ENOMEM);
+ rc = ida_alloc(&cxl_nvdimm_bridge_ida, GFP_KERNEL);
+ if (rc < 0)
+ goto err;
+ cxl_nvb->id = rc;
+
dev = &cxl_nvb->dev;
cxl_nvb->port = port;
cxl_nvb->state = CXL_NVB_NEW;
dev->type = &cxl_nvdimm_bridge_type;
return cxl_nvb;
+
+err:
+ kfree(cxl_nvb);
+ return ERR_PTR(rc);
}
static void unregister_nvb(void *_cxl_nvb)
return cxl_nvb;
dev = &cxl_nvb->dev;
- rc = dev_set_name(dev, "nvdimm-bridge");
+ rc = dev_set_name(dev, "nvdimm-bridge%d", cxl_nvb->id);
if (rc)
goto err;
return cxl_nvd;
}
+static void cxl_nvd_unregister(void *dev)
+{
+ device_unregister(dev);
+}
+
/**
* devm_cxl_add_nvdimm() - add a bridge between a cxl_memdev and an nvdimm
* @host: same host as @cxlmd
dev_dbg(host, "%s: register %s\n", dev_name(dev->parent),
dev_name(dev));
- return devm_add_action_or_reset(host, unregister_cxl_dev, dev);
+ return devm_add_action_or_reset(host, cxl_nvd_unregister, dev);
err:
put_device(dev);
struct cxl_reg_map memdev;
};
+/**
+ * struct cxl_register_map - DVSEC harvested register block mapping parameters
+ * @base: virtual base of the register-block-BAR + @block_offset
+ * @block_offset: offset to start of register block in @barno
+ * @reg_type: see enum cxl_regloc_type
+ * @barno: PCI BAR number containing the register block
+ * @component_map: cxl_reg_map for component registers
+ * @device_map: cxl_reg_maps for device registers
+ */
struct cxl_register_map {
+ void __iomem *base;
u64 block_offset;
u8 reg_type;
u8 barno;
CXL_DECODER_EXPANDER = 3,
};
+/*
+ * Current specification goes up to 8, double that seems a reasonable
+ * software max for the foreseeable future
+ */
+#define CXL_DECODER_MAX_INTERLEAVE 16
+
/**
* struct cxl_decoder - CXL address range decode configuration
* @dev: this decoder's device
* @interleave_granularity: data stride per dport
* @target_type: accelerator vs expander (type2 vs type3) selector
* @flags: memory type capabilities and locking
+ * @nr_targets: number of elements in @target
* @target: active ordered target list in current decoder configuration
*/
struct cxl_decoder {
int interleave_granularity;
enum cxl_decoder_type target_type;
unsigned long flags;
+ const int nr_targets;
struct cxl_dport *target[];
};
};
struct cxl_nvdimm_bridge {
+ int id;
struct device dev;
struct cxl_port *port;
struct nvdimm_bus *nvdimm_bus;
struct nvdimm *nvdimm;
};
+struct cxl_walk_context {
+ struct device *dev;
+ struct pci_bus *root;
+ struct cxl_port *port;
+ int error;
+ int count;
+};
+
/**
* struct cxl_port - logical collection of upstream port devices and
* downstream port devices to construct a CXL memory
struct cxl_decoder *to_cxl_decoder(struct device *dev);
bool is_root_decoder(struct device *dev);
-struct cxl_decoder *
-devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets,
- resource_size_t base, resource_size_t len,
- int interleave_ways, int interleave_granularity,
- enum cxl_decoder_type type, unsigned long flags);
-
-/*
- * Per the CXL specification (8.2.5.12 CXL HDM Decoder Capability Structure)
- * single ported host-bridges need not publish a decoder capability when a
- * passthrough decode can be assumed, i.e. all transactions that the uport sees
- * are claimed and passed to the single dport. Default the range a 0-base
- * 0-length until the first CXL region is activated.
- */
-static inline struct cxl_decoder *
-devm_cxl_add_passthrough_decoder(struct device *host, struct cxl_port *port)
-{
- return devm_cxl_add_decoder(host, port, 1, 0, 0, 1, PAGE_SIZE,
- CXL_DECODER_EXPANDER, 0);
-}
+struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, int nr_targets);
+int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map);
+int cxl_decoder_autoremove(struct device *host, struct cxl_decoder *cxld);
extern struct bus_type cxl_bus_type;
struct cxl_nvdimm *to_cxl_nvdimm(struct device *dev);
bool is_cxl_nvdimm(struct device *dev);
int devm_cxl_add_nvdimm(struct device *host, struct cxl_memdev *cxlmd);
+struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_nvdimm *cxl_nvd);
+
+/*
+ * Unit test builds overrides this to __weak, find the 'strong' version
+ * of these symbols in tools/testing/cxl/.
+ */
+#ifndef __mock
+#define __mock static
+#endif
#endif /* __CXL_H__ */
/* Copyright(c) 2020-2021 Intel Corporation. */
#ifndef __CXL_MEM_H__
#define __CXL_MEM_H__
+#include <uapi/linux/cxl_mem.h>
#include <linux/cdev.h>
#include "cxl.h"
CXLMDEV_RESET_NEEDED_NOT)
/**
- * struct cdevm_file_operations - devm coordinated cdev file operations
- * @fops: file operations that are synchronized against @shutdown
- * @shutdown: disconnect driver data
- *
- * @shutdown is invoked in the devres release path to disconnect any
- * driver instance data from @dev. It assumes synchronization with any
- * fops operation that requires driver data. After @shutdown an
- * operation may only reference @device data.
- */
-struct cdevm_file_operations {
- struct file_operations fops;
- void (*shutdown)(struct device *dev);
-};
-
-/**
* struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
* @dev: driver core device object
* @cdev: char dev core object for ioctl operations
return container_of(dev, struct cxl_memdev, dev);
}
-struct cxl_memdev *
-devm_cxl_add_memdev(struct device *host, struct cxl_mem *cxlm,
- const struct cdevm_file_operations *cdevm_fops);
+struct cxl_memdev *devm_cxl_add_memdev(struct cxl_mem *cxlm);
+
+/**
+ * struct cxl_mbox_cmd - A command to be submitted to hardware.
+ * @opcode: (input) The command set and command submitted to hardware.
+ * @payload_in: (input) Pointer to the input payload.
+ * @payload_out: (output) Pointer to the output payload. Must be allocated by
+ * the caller.
+ * @size_in: (input) Number of bytes to load from @payload_in.
+ * @size_out: (input) Max number of bytes loaded into @payload_out.
+ * (output) Number of bytes generated by the device. For fixed size
+ * outputs commands this is always expected to be deterministic. For
+ * variable sized output commands, it tells the exact number of bytes
+ * written.
+ * @return_code: (output) Error code returned from hardware.
+ *
+ * This is the primary mechanism used to send commands to the hardware.
+ * All the fields except @payload_* correspond exactly to the fields described in
+ * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
+ * @payload_out are written to, and read from the Command Payload Registers
+ * defined in CXL 2.0 8.2.8.4.8.
+ */
+struct cxl_mbox_cmd {
+ u16 opcode;
+ void *payload_in;
+ void *payload_out;
+ size_t size_in;
+ size_t size_out;
+ u16 return_code;
+#define CXL_MBOX_SUCCESS 0
+};
+
+/*
+ * CXL 2.0 - Memory capacity multiplier
+ * See Section 8.2.9.5
+ *
+ * Volatile, Persistent, and Partition capacities are specified to be in
+ * multiples of 256MB - define a multiplier to convert to/from bytes.
+ */
+#define CXL_CAPACITY_MULTIPLIER SZ_256M
/**
* struct cxl_mem - A CXL memory device
- * @pdev: The PCI device associated with this CXL device.
+ * @dev: The device associated with this CXL device.
* @cxlmd: Logical memory device chardev / interface
* @regs: Parsed register blocks
* @payload_size: Size of space for payload
* @mbox_mutex: Mutex to synchronize mailbox access.
* @firmware_version: Firmware version for the memory device.
* @enabled_cmds: Hardware commands found enabled in CEL.
- * @pmem_range: Persistent memory capacity information.
- * @ram_range: Volatile memory capacity information.
+ * @exclusive_cmds: Commands that are kernel-internal only
+ * @pmem_range: Active Persistent memory capacity configuration
+ * @ram_range: Active Volatile memory capacity configuration
+ * @total_bytes: sum of all possible capacities
+ * @volatile_only_bytes: hard volatile capacity
+ * @persistent_only_bytes: hard persistent capacity
+ * @partition_align_bytes: alignment size for partition-able capacity
+ * @active_volatile_bytes: sum of hard + soft volatile
+ * @active_persistent_bytes: sum of hard + soft persistent
+ * @next_volatile_bytes: volatile capacity change pending device reset
+ * @next_persistent_bytes: persistent capacity change pending device reset
+ * @mbox_send: @dev specific transport for transmitting mailbox commands
+ *
+ * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
+ * details on capacity parameters.
*/
struct cxl_mem {
- struct pci_dev *pdev;
+ struct device *dev;
struct cxl_memdev *cxlmd;
struct cxl_regs regs;
size_t lsa_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
- unsigned long *enabled_cmds;
+ DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
+ DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
struct range pmem_range;
struct range ram_range;
u64 active_persistent_bytes;
u64 next_volatile_bytes;
u64 next_persistent_bytes;
+
+ int (*mbox_send)(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd);
+};
+
+enum cxl_opcode {
+ CXL_MBOX_OP_INVALID = 0x0000,
+ CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
+ CXL_MBOX_OP_GET_FW_INFO = 0x0200,
+ CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
+ CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
+ CXL_MBOX_OP_GET_LOG = 0x0401,
+ CXL_MBOX_OP_IDENTIFY = 0x4000,
+ CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
+ CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
+ CXL_MBOX_OP_GET_LSA = 0x4102,
+ CXL_MBOX_OP_SET_LSA = 0x4103,
+ CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
+ CXL_MBOX_OP_GET_ALERT_CONFIG = 0x4201,
+ CXL_MBOX_OP_SET_ALERT_CONFIG = 0x4202,
+ CXL_MBOX_OP_GET_SHUTDOWN_STATE = 0x4203,
+ CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
+ CXL_MBOX_OP_GET_POISON = 0x4300,
+ CXL_MBOX_OP_INJECT_POISON = 0x4301,
+ CXL_MBOX_OP_CLEAR_POISON = 0x4302,
+ CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS = 0x4303,
+ CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
+ CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
+ CXL_MBOX_OP_MAX = 0x10000
};
+
+#define DEFINE_CXL_CEL_UUID \
+ UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96, 0xb1, 0x62, \
+ 0x3b, 0x3f, 0x17)
+
+#define DEFINE_CXL_VENDOR_DEBUG_UUID \
+ UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f, 0xd6, 0x07, 0x19, \
+ 0x40, 0x3d, 0x86)
+
+struct cxl_mbox_get_supported_logs {
+ __le16 entries;
+ u8 rsvd[6];
+ struct cxl_gsl_entry {
+ uuid_t uuid;
+ __le32 size;
+ } __packed entry[];
+} __packed;
+
+struct cxl_cel_entry {
+ __le16 opcode;
+ __le16 effect;
+} __packed;
+
+struct cxl_mbox_get_log {
+ uuid_t uuid;
+ __le32 offset;
+ __le32 length;
+} __packed;
+
+/* See CXL 2.0 Table 175 Identify Memory Device Output Payload */
+struct cxl_mbox_identify {
+ char fw_revision[0x10];
+ __le64 total_capacity;
+ __le64 volatile_capacity;
+ __le64 persistent_capacity;
+ __le64 partition_align;
+ __le16 info_event_log_size;
+ __le16 warning_event_log_size;
+ __le16 failure_event_log_size;
+ __le16 fatal_event_log_size;
+ __le32 lsa_size;
+ u8 poison_list_max_mer[3];
+ __le16 inject_poison_limit;
+ u8 poison_caps;
+ u8 qos_telemetry_caps;
+} __packed;
+
+struct cxl_mbox_get_lsa {
+ u32 offset;
+ u32 length;
+} __packed;
+
+struct cxl_mbox_set_lsa {
+ u32 offset;
+ u32 reserved;
+ u8 data[];
+} __packed;
+
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ * @flags: Set of flags effecting driver behavior.
+ *
+ * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
+ * will be enabled by the driver regardless of what hardware may have
+ * advertised.
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware. The driver will use @info to validate the fields passed in by
+ * the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+ struct cxl_command_info info;
+ enum cxl_opcode opcode;
+ u32 flags;
+#define CXL_CMD_FLAG_NONE 0
+#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
+};
+
+int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, u16 opcode, void *in,
+ size_t in_size, void *out, size_t out_size);
+int cxl_mem_identify(struct cxl_mem *cxlm);
+int cxl_mem_enumerate_cmds(struct cxl_mem *cxlm);
+int cxl_mem_create_range_info(struct cxl_mem *cxlm);
+struct cxl_mem *cxl_mem_create(struct device *dev);
+void set_exclusive_cxl_commands(struct cxl_mem *cxlm, unsigned long *cmds);
+void clear_exclusive_cxl_commands(struct cxl_mem *cxlm, unsigned long *cmds);
#endif /* __CXL_MEM_H__ */
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
-#include <uapi/linux/cxl_mem.h>
-#include <linux/security.h>
-#include <linux/debugfs.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/module.h>
#include <linux/sizes.h>
#include <linux/mutex.h>
#include <linux/list.h>
-#include <linux/cdev.h>
-#include <linux/idr.h>
#include <linux/pci.h>
#include <linux/io.h>
-#include <linux/io-64-nonatomic-lo-hi.h>
#include "cxlmem.h"
#include "pci.h"
#include "cxl.h"
*
* This implements the PCI exclusive functionality for a CXL device as it is
* defined by the Compute Express Link specification. CXL devices may surface
- * certain functionality even if it isn't CXL enabled.
+ * certain functionality even if it isn't CXL enabled. While this driver is
+ * focused around the PCI specific aspects of a CXL device, it binds to the
+ * specific CXL memory device class code, and therefore the implementation of
+ * cxl_pci is focused around CXL memory devices.
*
* The driver has several responsibilities, mainly:
* - Create the memX device and register on the CXL bus.
* - Enumerate device's register interface and map them.
- * - Probe the device attributes to establish sysfs interface.
- * - Provide an IOCTL interface to userspace to communicate with the device for
- * things like firmware update.
+ * - Registers nvdimm bridge device with cxl_core.
+ * - Registers a CXL mailbox with cxl_core.
*/
#define cxl_doorbell_busy(cxlm) \
/* CXL 2.0 - 8.2.8.4 */
#define CXL_MAILBOX_TIMEOUT_MS (2 * HZ)
-enum opcode {
- CXL_MBOX_OP_INVALID = 0x0000,
- CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
- CXL_MBOX_OP_GET_FW_INFO = 0x0200,
- CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
- CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
- CXL_MBOX_OP_GET_LOG = 0x0401,
- CXL_MBOX_OP_IDENTIFY = 0x4000,
- CXL_MBOX_OP_GET_PARTITION_INFO = 0x4100,
- CXL_MBOX_OP_SET_PARTITION_INFO = 0x4101,
- CXL_MBOX_OP_GET_LSA = 0x4102,
- CXL_MBOX_OP_SET_LSA = 0x4103,
- CXL_MBOX_OP_GET_HEALTH_INFO = 0x4200,
- CXL_MBOX_OP_GET_ALERT_CONFIG = 0x4201,
- CXL_MBOX_OP_SET_ALERT_CONFIG = 0x4202,
- CXL_MBOX_OP_GET_SHUTDOWN_STATE = 0x4203,
- CXL_MBOX_OP_SET_SHUTDOWN_STATE = 0x4204,
- CXL_MBOX_OP_GET_POISON = 0x4300,
- CXL_MBOX_OP_INJECT_POISON = 0x4301,
- CXL_MBOX_OP_CLEAR_POISON = 0x4302,
- CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS = 0x4303,
- CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
- CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
- CXL_MBOX_OP_MAX = 0x10000
-};
-
-/*
- * CXL 2.0 - Memory capacity multiplier
- * See Section 8.2.9.5
- *
- * Volatile, Persistent, and Partition capacities are specified to be in
- * multiples of 256MB - define a multiplier to convert to/from bytes.
- */
-#define CXL_CAPACITY_MULTIPLIER SZ_256M
-
-/**
- * struct mbox_cmd - A command to be submitted to hardware.
- * @opcode: (input) The command set and command submitted to hardware.
- * @payload_in: (input) Pointer to the input payload.
- * @payload_out: (output) Pointer to the output payload. Must be allocated by
- * the caller.
- * @size_in: (input) Number of bytes to load from @payload_in.
- * @size_out: (input) Max number of bytes loaded into @payload_out.
- * (output) Number of bytes generated by the device. For fixed size
- * outputs commands this is always expected to be deterministic. For
- * variable sized output commands, it tells the exact number of bytes
- * written.
- * @return_code: (output) Error code returned from hardware.
- *
- * This is the primary mechanism used to send commands to the hardware.
- * All the fields except @payload_* correspond exactly to the fields described in
- * Command Register section of the CXL 2.0 8.2.8.4.5. @payload_in and
- * @payload_out are written to, and read from the Command Payload Registers
- * defined in CXL 2.0 8.2.8.4.8.
- */
-struct mbox_cmd {
- u16 opcode;
- void *payload_in;
- void *payload_out;
- size_t size_in;
- size_t size_out;
- u16 return_code;
-#define CXL_MBOX_SUCCESS 0
-};
-
-static DECLARE_RWSEM(cxl_memdev_rwsem);
-static struct dentry *cxl_debugfs;
-static bool cxl_raw_allow_all;
-
-enum {
- CEL_UUID,
- VENDOR_DEBUG_UUID,
-};
-
-/* See CXL 2.0 Table 170. Get Log Input Payload */
-static const uuid_t log_uuid[] = {
- [CEL_UUID] = UUID_INIT(0xda9c0b5, 0xbf41, 0x4b78, 0x8f, 0x79, 0x96,
- 0xb1, 0x62, 0x3b, 0x3f, 0x17),
- [VENDOR_DEBUG_UUID] = UUID_INIT(0xe1819d9, 0x11a9, 0x400c, 0x81, 0x1f,
- 0xd6, 0x07, 0x19, 0x40, 0x3d, 0x86),
-};
-
-/**
- * struct cxl_mem_command - Driver representation of a memory device command
- * @info: Command information as it exists for the UAPI
- * @opcode: The actual bits used for the mailbox protocol
- * @flags: Set of flags effecting driver behavior.
- *
- * * %CXL_CMD_FLAG_FORCE_ENABLE: In cases of error, commands with this flag
- * will be enabled by the driver regardless of what hardware may have
- * advertised.
- *
- * The cxl_mem_command is the driver's internal representation of commands that
- * are supported by the driver. Some of these commands may not be supported by
- * the hardware. The driver will use @info to validate the fields passed in by
- * the user then submit the @opcode to the hardware.
- *
- * See struct cxl_command_info.
- */
-struct cxl_mem_command {
- struct cxl_command_info info;
- enum opcode opcode;
- u32 flags;
-#define CXL_CMD_FLAG_NONE 0
-#define CXL_CMD_FLAG_FORCE_ENABLE BIT(0)
-};
-
-#define CXL_CMD(_id, sin, sout, _flags) \
- [CXL_MEM_COMMAND_ID_##_id] = { \
- .info = { \
- .id = CXL_MEM_COMMAND_ID_##_id, \
- .size_in = sin, \
- .size_out = sout, \
- }, \
- .opcode = CXL_MBOX_OP_##_id, \
- .flags = _flags, \
- }
-
-/*
- * This table defines the supported mailbox commands for the driver. This table
- * is made up of a UAPI structure. Non-negative values as parameters in the
- * table will be validated against the user's input. For example, if size_in is
- * 0, and the user passed in 1, it is an error.
- */
-static struct cxl_mem_command mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
- CXL_CMD(IDENTIFY, 0, 0x43, CXL_CMD_FLAG_FORCE_ENABLE),
-#ifdef CONFIG_CXL_MEM_RAW_COMMANDS
- CXL_CMD(RAW, ~0, ~0, 0),
-#endif
- CXL_CMD(GET_SUPPORTED_LOGS, 0, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
- CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
- CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
- CXL_CMD(GET_LSA, 0x8, ~0, 0),
- CXL_CMD(GET_HEALTH_INFO, 0, 0x12, 0),
- CXL_CMD(GET_LOG, 0x18, ~0, CXL_CMD_FLAG_FORCE_ENABLE),
- CXL_CMD(SET_PARTITION_INFO, 0x0a, 0, 0),
- CXL_CMD(SET_LSA, ~0, 0, 0),
- CXL_CMD(GET_ALERT_CONFIG, 0, 0x10, 0),
- CXL_CMD(SET_ALERT_CONFIG, 0xc, 0, 0),
- CXL_CMD(GET_SHUTDOWN_STATE, 0, 0x1, 0),
- CXL_CMD(SET_SHUTDOWN_STATE, 0x1, 0, 0),
- CXL_CMD(GET_POISON, 0x10, ~0, 0),
- CXL_CMD(INJECT_POISON, 0x8, 0, 0),
- CXL_CMD(CLEAR_POISON, 0x48, 0, 0),
- CXL_CMD(GET_SCAN_MEDIA_CAPS, 0x10, 0x4, 0),
- CXL_CMD(SCAN_MEDIA, 0x11, 0, 0),
- CXL_CMD(GET_SCAN_MEDIA, 0, ~0, 0),
-};
-
-/*
- * Commands that RAW doesn't permit. The rationale for each:
- *
- * CXL_MBOX_OP_ACTIVATE_FW: Firmware activation requires adjustment /
- * coordination of transaction timeout values at the root bridge level.
- *
- * CXL_MBOX_OP_SET_PARTITION_INFO: The device memory map may change live
- * and needs to be coordinated with HDM updates.
- *
- * CXL_MBOX_OP_SET_LSA: The label storage area may be cached by the
- * driver and any writes from userspace invalidates those contents.
- *
- * CXL_MBOX_OP_SET_SHUTDOWN_STATE: Set shutdown state assumes no writes
- * to the device after it is marked clean, userspace can not make that
- * assertion.
- *
- * CXL_MBOX_OP_[GET_]SCAN_MEDIA: The kernel provides a native error list that
- * is kept up to date with patrol notifications and error management.
- */
-static u16 cxl_disabled_raw_commands[] = {
- CXL_MBOX_OP_ACTIVATE_FW,
- CXL_MBOX_OP_SET_PARTITION_INFO,
- CXL_MBOX_OP_SET_LSA,
- CXL_MBOX_OP_SET_SHUTDOWN_STATE,
- CXL_MBOX_OP_SCAN_MEDIA,
- CXL_MBOX_OP_GET_SCAN_MEDIA,
-};
-
-/*
- * Command sets that RAW doesn't permit. All opcodes in this set are
- * disabled because they pass plain text security payloads over the
- * user/kernel boundary. This functionality is intended to be wrapped
- * behind the keys ABI which allows for encrypted payloads in the UAPI
- */
-static u8 security_command_sets[] = {
- 0x44, /* Sanitize */
- 0x45, /* Persistent Memory Data-at-rest Security */
- 0x46, /* Security Passthrough */
-};
-
-#define cxl_for_each_cmd(cmd) \
- for ((cmd) = &mem_commands[0]; \
- ((cmd) - mem_commands) < ARRAY_SIZE(mem_commands); (cmd)++)
-
-#define cxl_cmd_count ARRAY_SIZE(mem_commands)
-
-static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
+static int cxl_pci_mbox_wait_for_doorbell(struct cxl_mem *cxlm)
{
const unsigned long start = jiffies;
unsigned long end = start;
cpu_relax();
}
- dev_dbg(&cxlm->pdev->dev, "Doorbell wait took %dms",
+ dev_dbg(cxlm->dev, "Doorbell wait took %dms",
jiffies_to_msecs(end) - jiffies_to_msecs(start));
return 0;
}
-static bool cxl_is_security_command(u16 opcode)
+static void cxl_pci_mbox_timeout(struct cxl_mem *cxlm,
+ struct cxl_mbox_cmd *mbox_cmd)
{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(security_command_sets); i++)
- if (security_command_sets[i] == (opcode >> 8))
- return true;
- return false;
-}
-
-static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
- struct mbox_cmd *mbox_cmd)
-{
- struct device *dev = &cxlm->pdev->dev;
+ struct device *dev = cxlm->dev;
dev_dbg(dev, "Mailbox command (opcode: %#x size: %zub) timed out\n",
mbox_cmd->opcode, mbox_cmd->size_in);
}
/**
- * __cxl_mem_mbox_send_cmd() - Execute a mailbox command
+ * __cxl_pci_mbox_send_cmd() - Execute a mailbox command
* @cxlm: The CXL memory device to communicate with.
* @mbox_cmd: Command to send to the memory device.
*
* not need to coordinate with each other. The driver only uses the primary
* mailbox.
*/
-static int __cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
- struct mbox_cmd *mbox_cmd)
+static int __cxl_pci_mbox_send_cmd(struct cxl_mem *cxlm,
+ struct cxl_mbox_cmd *mbox_cmd)
{
void __iomem *payload = cxlm->regs.mbox + CXLDEV_MBOX_PAYLOAD_OFFSET;
+ struct device *dev = cxlm->dev;
u64 cmd_reg, status_reg;
size_t out_len;
int rc;
/* #1 */
if (cxl_doorbell_busy(cxlm)) {
- dev_err_ratelimited(&cxlm->pdev->dev,
- "Mailbox re-busy after acquiring\n");
+ dev_err_ratelimited(dev, "Mailbox re-busy after acquiring\n");
return -EBUSY;
}
writeq(cmd_reg, cxlm->regs.mbox + CXLDEV_MBOX_CMD_OFFSET);
/* #4 */
- dev_dbg(&cxlm->pdev->dev, "Sending command\n");
+ dev_dbg(dev, "Sending command\n");
writel(CXLDEV_MBOX_CTRL_DOORBELL,
cxlm->regs.mbox + CXLDEV_MBOX_CTRL_OFFSET);
/* #5 */
- rc = cxl_mem_wait_for_doorbell(cxlm);
+ rc = cxl_pci_mbox_wait_for_doorbell(cxlm);
if (rc == -ETIMEDOUT) {
- cxl_mem_mbox_timeout(cxlm, mbox_cmd);
+ cxl_pci_mbox_timeout(cxlm, mbox_cmd);
return rc;
}
FIELD_GET(CXLDEV_MBOX_STATUS_RET_CODE_MASK, status_reg);
if (mbox_cmd->return_code != 0) {
- dev_dbg(&cxlm->pdev->dev, "Mailbox operation had an error\n");
+ dev_dbg(dev, "Mailbox operation had an error\n");
return 0;
}
}
/**
- * cxl_mem_mbox_get() - Acquire exclusive access to the mailbox.
+ * cxl_pci_mbox_get() - Acquire exclusive access to the mailbox.
* @cxlm: The memory device to gain access to.
*
* Context: Any context. Takes the mbox_mutex.
* Return: 0 if exclusive access was acquired.
*/
-static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
+static int cxl_pci_mbox_get(struct cxl_mem *cxlm)
{
- struct device *dev = &cxlm->pdev->dev;
+ struct device *dev = cxlm->dev;
u64 md_status;
int rc;
* Mailbox Interface Ready bit. Therefore, waiting for the doorbell
* to be ready is sufficient.
*/
- rc = cxl_mem_wait_for_doorbell(cxlm);
+ rc = cxl_pci_mbox_wait_for_doorbell(cxlm);
if (rc) {
dev_warn(dev, "Mailbox interface not ready\n");
goto out;
}
/**
- * cxl_mem_mbox_put() - Release exclusive access to the mailbox.
+ * cxl_pci_mbox_put() - Release exclusive access to the mailbox.
* @cxlm: The CXL memory device to communicate with.
*
* Context: Any context. Expects mbox_mutex to be held.
*/
-static void cxl_mem_mbox_put(struct cxl_mem *cxlm)
+static void cxl_pci_mbox_put(struct cxl_mem *cxlm)
{
mutex_unlock(&cxlm->mbox_mutex);
}
-/**
- * handle_mailbox_cmd_from_user() - Dispatch a mailbox command for userspace.
- * @cxlm: The CXL memory device to communicate with.
- * @cmd: The validated command.
- * @in_payload: Pointer to userspace's input payload.
- * @out_payload: Pointer to userspace's output payload.
- * @size_out: (Input) Max payload size to copy out.
- * (Output) Payload size hardware generated.
- * @retval: Hardware generated return code from the operation.
- *
- * Return:
- * * %0 - Mailbox transaction succeeded. This implies the mailbox
- * protocol completed successfully not that the operation itself
- * was successful.
- * * %-ENOMEM - Couldn't allocate a bounce buffer.
- * * %-EFAULT - Something happened with copy_to/from_user.
- * * %-EINTR - Mailbox acquisition interrupted.
- * * %-EXXX - Transaction level failures.
- *
- * Creates the appropriate mailbox command and dispatches it on behalf of a
- * userspace request. The input and output payloads are copied between
- * userspace.
- *
- * See cxl_send_cmd().
- */
-static int handle_mailbox_cmd_from_user(struct cxl_mem *cxlm,
- const struct cxl_mem_command *cmd,
- u64 in_payload, u64 out_payload,
- s32 *size_out, u32 *retval)
-{
- struct device *dev = &cxlm->pdev->dev;
- struct mbox_cmd mbox_cmd = {
- .opcode = cmd->opcode,
- .size_in = cmd->info.size_in,
- .size_out = cmd->info.size_out,
- };
- int rc;
-
- if (cmd->info.size_out) {
- mbox_cmd.payload_out = kvzalloc(cmd->info.size_out, GFP_KERNEL);
- if (!mbox_cmd.payload_out)
- return -ENOMEM;
- }
-
- if (cmd->info.size_in) {
- mbox_cmd.payload_in = vmemdup_user(u64_to_user_ptr(in_payload),
- cmd->info.size_in);
- if (IS_ERR(mbox_cmd.payload_in)) {
- kvfree(mbox_cmd.payload_out);
- return PTR_ERR(mbox_cmd.payload_in);
- }
- }
-
- rc = cxl_mem_mbox_get(cxlm);
- if (rc)
- goto out;
-
- dev_dbg(dev,
- "Submitting %s command for user\n"
- "\topcode: %x\n"
- "\tsize: %ub\n",
- cxl_command_names[cmd->info.id].name, mbox_cmd.opcode,
- cmd->info.size_in);
-
- dev_WARN_ONCE(dev, cmd->info.id == CXL_MEM_COMMAND_ID_RAW,
- "raw command path used\n");
-
- rc = __cxl_mem_mbox_send_cmd(cxlm, &mbox_cmd);
- cxl_mem_mbox_put(cxlm);
- if (rc)
- goto out;
-
- /*
- * @size_out contains the max size that's allowed to be written back out
- * to userspace. While the payload may have written more output than
- * this it will have to be ignored.
- */
- if (mbox_cmd.size_out) {
- dev_WARN_ONCE(dev, mbox_cmd.size_out > *size_out,
- "Invalid return size\n");
- if (copy_to_user(u64_to_user_ptr(out_payload),
- mbox_cmd.payload_out, mbox_cmd.size_out)) {
- rc = -EFAULT;
- goto out;
- }
- }
-
- *size_out = mbox_cmd.size_out;
- *retval = mbox_cmd.return_code;
-
-out:
- kvfree(mbox_cmd.payload_in);
- kvfree(mbox_cmd.payload_out);
- return rc;
-}
-
-static bool cxl_mem_raw_command_allowed(u16 opcode)
-{
- int i;
-
- if (!IS_ENABLED(CONFIG_CXL_MEM_RAW_COMMANDS))
- return false;
-
- if (security_locked_down(LOCKDOWN_PCI_ACCESS))
- return false;
-
- if (cxl_raw_allow_all)
- return true;
-
- if (cxl_is_security_command(opcode))
- return false;
-
- for (i = 0; i < ARRAY_SIZE(cxl_disabled_raw_commands); i++)
- if (cxl_disabled_raw_commands[i] == opcode)
- return false;
-
- return true;
-}
-
-/**
- * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
- * @cxlm: &struct cxl_mem device whose mailbox will be used.
- * @send_cmd: &struct cxl_send_command copied in from userspace.
- * @out_cmd: Sanitized and populated &struct cxl_mem_command.
- *
- * Return:
- * * %0 - @out_cmd is ready to send.
- * * %-ENOTTY - Invalid command specified.
- * * %-EINVAL - Reserved fields or invalid values were used.
- * * %-ENOMEM - Input or output buffer wasn't sized properly.
- * * %-EPERM - Attempted to use a protected command.
- *
- * The result of this command is a fully validated command in @out_cmd that is
- * safe to send to the hardware.
- *
- * See handle_mailbox_cmd_from_user()
- */
-static int cxl_validate_cmd_from_user(struct cxl_mem *cxlm,
- const struct cxl_send_command *send_cmd,
- struct cxl_mem_command *out_cmd)
-{
- const struct cxl_command_info *info;
- struct cxl_mem_command *c;
-
- if (send_cmd->id == 0 || send_cmd->id >= CXL_MEM_COMMAND_ID_MAX)
- return -ENOTTY;
-
- /*
- * The user can never specify an input payload larger than what hardware
- * supports, but output can be arbitrarily large (simply write out as
- * much data as the hardware provides).
- */
- if (send_cmd->in.size > cxlm->payload_size)
- return -EINVAL;
-
- /*
- * Checks are bypassed for raw commands but a WARN/taint will occur
- * later in the callchain
- */
- if (send_cmd->id == CXL_MEM_COMMAND_ID_RAW) {
- const struct cxl_mem_command temp = {
- .info = {
- .id = CXL_MEM_COMMAND_ID_RAW,
- .flags = 0,
- .size_in = send_cmd->in.size,
- .size_out = send_cmd->out.size,
- },
- .opcode = send_cmd->raw.opcode
- };
-
- if (send_cmd->raw.rsvd)
- return -EINVAL;
-
- /*
- * Unlike supported commands, the output size of RAW commands
- * gets passed along without further checking, so it must be
- * validated here.
- */
- if (send_cmd->out.size > cxlm->payload_size)
- return -EINVAL;
-
- if (!cxl_mem_raw_command_allowed(send_cmd->raw.opcode))
- return -EPERM;
-
- memcpy(out_cmd, &temp, sizeof(temp));
-
- return 0;
- }
-
- if (send_cmd->flags & ~CXL_MEM_COMMAND_FLAG_MASK)
- return -EINVAL;
-
- if (send_cmd->rsvd)
- return -EINVAL;
-
- if (send_cmd->in.rsvd || send_cmd->out.rsvd)
- return -EINVAL;
-
- /* Convert user's command into the internal representation */
- c = &mem_commands[send_cmd->id];
- info = &c->info;
-
- /* Check that the command is enabled for hardware */
- if (!test_bit(info->id, cxlm->enabled_cmds))
- return -ENOTTY;
-
- /* Check the input buffer is the expected size */
- if (info->size_in >= 0 && info->size_in != send_cmd->in.size)
- return -ENOMEM;
-
- /* Check the output buffer is at least large enough */
- if (info->size_out >= 0 && send_cmd->out.size < info->size_out)
- return -ENOMEM;
-
- memcpy(out_cmd, c, sizeof(*c));
- out_cmd->info.size_in = send_cmd->in.size;
- /*
- * XXX: out_cmd->info.size_out will be controlled by the driver, and the
- * specified number of bytes @send_cmd->out.size will be copied back out
- * to userspace.
- */
-
- return 0;
-}
-
-static int cxl_query_cmd(struct cxl_memdev *cxlmd,
- struct cxl_mem_query_commands __user *q)
+static int cxl_pci_mbox_send(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
{
- struct device *dev = &cxlmd->dev;
- struct cxl_mem_command *cmd;
- u32 n_commands;
- int j = 0;
-
- dev_dbg(dev, "Query IOCTL\n");
-
- if (get_user(n_commands, &q->n_commands))
- return -EFAULT;
-
- /* returns the total number if 0 elements are requested. */
- if (n_commands == 0)
- return put_user(cxl_cmd_count, &q->n_commands);
-
- /*
- * otherwise, return max(n_commands, total commands) cxl_command_info
- * structures.
- */
- cxl_for_each_cmd(cmd) {
- const struct cxl_command_info *info = &cmd->info;
-
- if (copy_to_user(&q->commands[j++], info, sizeof(*info)))
- return -EFAULT;
-
- if (j == n_commands)
- break;
- }
-
- return 0;
-}
-
-static int cxl_send_cmd(struct cxl_memdev *cxlmd,
- struct cxl_send_command __user *s)
-{
- struct cxl_mem *cxlm = cxlmd->cxlm;
- struct device *dev = &cxlmd->dev;
- struct cxl_send_command send;
- struct cxl_mem_command c;
int rc;
- dev_dbg(dev, "Send IOCTL\n");
-
- if (copy_from_user(&send, s, sizeof(send)))
- return -EFAULT;
-
- rc = cxl_validate_cmd_from_user(cxlmd->cxlm, &send, &c);
- if (rc)
- return rc;
-
- /* Prepare to handle a full payload for variable sized output */
- if (c.info.size_out < 0)
- c.info.size_out = cxlm->payload_size;
-
- rc = handle_mailbox_cmd_from_user(cxlm, &c, send.in.payload,
- send.out.payload, &send.out.size,
- &send.retval);
+ rc = cxl_pci_mbox_get(cxlm);
if (rc)
return rc;
- if (copy_to_user(s, &send, sizeof(send)))
- return -EFAULT;
-
- return 0;
-}
-
-static long __cxl_memdev_ioctl(struct cxl_memdev *cxlmd, unsigned int cmd,
- unsigned long arg)
-{
- switch (cmd) {
- case CXL_MEM_QUERY_COMMANDS:
- return cxl_query_cmd(cxlmd, (void __user *)arg);
- case CXL_MEM_SEND_COMMAND:
- return cxl_send_cmd(cxlmd, (void __user *)arg);
- default:
- return -ENOTTY;
- }
-}
-
-static long cxl_memdev_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg)
-{
- struct cxl_memdev *cxlmd = file->private_data;
- int rc = -ENXIO;
-
- down_read(&cxl_memdev_rwsem);
- if (cxlmd->cxlm)
- rc = __cxl_memdev_ioctl(cxlmd, cmd, arg);
- up_read(&cxl_memdev_rwsem);
+ rc = __cxl_pci_mbox_send_cmd(cxlm, cmd);
+ cxl_pci_mbox_put(cxlm);
return rc;
}
-static int cxl_memdev_open(struct inode *inode, struct file *file)
-{
- struct cxl_memdev *cxlmd =
- container_of(inode->i_cdev, typeof(*cxlmd), cdev);
-
- get_device(&cxlmd->dev);
- file->private_data = cxlmd;
-
- return 0;
-}
-
-static int cxl_memdev_release_file(struct inode *inode, struct file *file)
-{
- struct cxl_memdev *cxlmd =
- container_of(inode->i_cdev, typeof(*cxlmd), cdev);
-
- put_device(&cxlmd->dev);
-
- return 0;
-}
-
-static void cxl_memdev_shutdown(struct device *dev)
-{
- struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
-
- down_write(&cxl_memdev_rwsem);
- cxlmd->cxlm = NULL;
- up_write(&cxl_memdev_rwsem);
-}
-
-static const struct cdevm_file_operations cxl_memdev_fops = {
- .fops = {
- .owner = THIS_MODULE,
- .unlocked_ioctl = cxl_memdev_ioctl,
- .open = cxl_memdev_open,
- .release = cxl_memdev_release_file,
- .compat_ioctl = compat_ptr_ioctl,
- .llseek = noop_llseek,
- },
- .shutdown = cxl_memdev_shutdown,
-};
-
-static inline struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
-{
- struct cxl_mem_command *c;
-
- cxl_for_each_cmd(c)
- if (c->opcode == opcode)
- return c;
-
- return NULL;
-}
-
-/**
- * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
- * @cxlm: The CXL memory device to communicate with.
- * @opcode: Opcode for the mailbox command.
- * @in: The input payload for the mailbox command.
- * @in_size: The length of the input payload
- * @out: Caller allocated buffer for the output.
- * @out_size: Expected size of output.
- *
- * Context: Any context. Will acquire and release mbox_mutex.
- * Return:
- * * %>=0 - Number of bytes returned in @out.
- * * %-E2BIG - Payload is too large for hardware.
- * * %-EBUSY - Couldn't acquire exclusive mailbox access.
- * * %-EFAULT - Hardware error occurred.
- * * %-ENXIO - Command completed, but device reported an error.
- * * %-EIO - Unexpected output size.
- *
- * Mailbox commands may execute successfully yet the device itself reported an
- * error. While this distinction can be useful for commands from userspace, the
- * kernel will only be able to use results when both are successful.
- *
- * See __cxl_mem_mbox_send_cmd()
- */
-static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, u16 opcode,
- void *in, size_t in_size,
- void *out, size_t out_size)
-{
- const struct cxl_mem_command *cmd = cxl_mem_find_command(opcode);
- struct mbox_cmd mbox_cmd = {
- .opcode = opcode,
- .payload_in = in,
- .size_in = in_size,
- .size_out = out_size,
- .payload_out = out,
- };
- int rc;
-
- if (out_size > cxlm->payload_size)
- return -E2BIG;
-
- rc = cxl_mem_mbox_get(cxlm);
- if (rc)
- return rc;
-
- rc = __cxl_mem_mbox_send_cmd(cxlm, &mbox_cmd);
- cxl_mem_mbox_put(cxlm);
- if (rc)
- return rc;
-
- /* TODO: Map return code to proper kernel style errno */
- if (mbox_cmd.return_code != CXL_MBOX_SUCCESS)
- return -ENXIO;
-
- /*
- * Variable sized commands can't be validated and so it's up to the
- * caller to do that if they wish.
- */
- if (cmd->info.size_out >= 0 && mbox_cmd.size_out != out_size)
- return -EIO;
-
- return 0;
-}
-
-static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
+static int cxl_pci_setup_mailbox(struct cxl_mem *cxlm)
{
const int cap = readl(cxlm->regs.mbox + CXLDEV_MBOX_CAPS_OFFSET);
+ cxlm->mbox_send = cxl_pci_mbox_send;
cxlm->payload_size =
1 << FIELD_GET(CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK, cap);
*/
cxlm->payload_size = min_t(size_t, cxlm->payload_size, SZ_1M);
if (cxlm->payload_size < 256) {
- dev_err(&cxlm->pdev->dev, "Mailbox is too small (%zub)",
+ dev_err(cxlm->dev, "Mailbox is too small (%zub)",
cxlm->payload_size);
return -ENXIO;
}
- dev_dbg(&cxlm->pdev->dev, "Mailbox payload sized %zu",
+ dev_dbg(cxlm->dev, "Mailbox payload sized %zu",
cxlm->payload_size);
return 0;
}
-static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev)
-{
- struct device *dev = &pdev->dev;
- struct cxl_mem *cxlm;
-
- cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
- if (!cxlm) {
- dev_err(dev, "No memory available\n");
- return ERR_PTR(-ENOMEM);
- }
-
- mutex_init(&cxlm->mbox_mutex);
- cxlm->pdev = pdev;
- cxlm->enabled_cmds =
- devm_kmalloc_array(dev, BITS_TO_LONGS(cxl_cmd_count),
- sizeof(unsigned long),
- GFP_KERNEL | __GFP_ZERO);
- if (!cxlm->enabled_cmds) {
- dev_err(dev, "No memory available for bitmap\n");
- return ERR_PTR(-ENOMEM);
- }
-
- return cxlm;
-}
-
-static void __iomem *cxl_mem_map_regblock(struct cxl_mem *cxlm,
- u8 bar, u64 offset)
+static int cxl_map_regblock(struct pci_dev *pdev, struct cxl_register_map *map)
{
- struct pci_dev *pdev = cxlm->pdev;
- struct device *dev = &pdev->dev;
void __iomem *addr;
+ int bar = map->barno;
+ struct device *dev = &pdev->dev;
+ resource_size_t offset = map->block_offset;
/* Basic sanity check that BAR is big enough */
if (pci_resource_len(pdev, bar) < offset) {
- dev_err(dev, "BAR%d: %pr: too small (offset: %#llx)\n", bar,
- &pdev->resource[bar], (unsigned long long)offset);
- return IOMEM_ERR_PTR(-ENXIO);
+ dev_err(dev, "BAR%d: %pr: too small (offset: %pa)\n", bar,
+ &pdev->resource[bar], &offset);
+ return -ENXIO;
}
addr = pci_iomap(pdev, bar, 0);
if (!addr) {
dev_err(dev, "failed to map registers\n");
- return addr;
+ return -ENOMEM;
}
- dev_dbg(dev, "Mapped CXL Memory Device resource bar %u @ %#llx\n",
- bar, offset);
+ dev_dbg(dev, "Mapped CXL Memory Device resource bar %u @ %pa\n",
+ bar, &offset);
- return addr;
+ map->base = addr + map->block_offset;
+ return 0;
}
-static void cxl_mem_unmap_regblock(struct cxl_mem *cxlm, void __iomem *base)
+static void cxl_unmap_regblock(struct pci_dev *pdev,
+ struct cxl_register_map *map)
{
- pci_iounmap(cxlm->pdev, base);
+ pci_iounmap(pdev, map->base - map->block_offset);
+ map->base = NULL;
}
-static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+static int cxl_probe_regs(struct pci_dev *pdev, struct cxl_register_map *map)
{
- int pos;
-
- pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
- if (!pos)
- return 0;
-
- while (pos) {
- u16 vendor, id;
-
- pci_read_config_word(pdev, pos + PCI_DVSEC_HEADER1, &vendor);
- pci_read_config_word(pdev, pos + PCI_DVSEC_HEADER2, &id);
- if (vendor == PCI_DVSEC_VENDOR_ID_CXL && dvsec == id)
- return pos;
-
- pos = pci_find_next_ext_capability(pdev, pos,
- PCI_EXT_CAP_ID_DVSEC);
- }
-
- return 0;
-}
-
-static int cxl_probe_regs(struct cxl_mem *cxlm, void __iomem *base,
- struct cxl_register_map *map)
-{
- struct pci_dev *pdev = cxlm->pdev;
- struct device *dev = &pdev->dev;
struct cxl_component_reg_map *comp_map;
struct cxl_device_reg_map *dev_map;
+ struct device *dev = &pdev->dev;
+ void __iomem *base = map->base;
switch (map->reg_type) {
case CXL_REGLOC_RBI_COMPONENT:
static int cxl_map_regs(struct cxl_mem *cxlm, struct cxl_register_map *map)
{
- struct pci_dev *pdev = cxlm->pdev;
- struct device *dev = &pdev->dev;
+ struct device *dev = cxlm->dev;
+ struct pci_dev *pdev = to_pci_dev(dev);
switch (map->reg_type) {
case CXL_REGLOC_RBI_COMPONENT:
return 0;
}
-static void cxl_decode_register_block(u32 reg_lo, u32 reg_hi,
- u8 *bar, u64 *offset, u8 *reg_type)
+static void cxl_decode_regblock(u32 reg_lo, u32 reg_hi,
+ struct cxl_register_map *map)
{
- *offset = ((u64)reg_hi << 32) | (reg_lo & CXL_REGLOC_ADDR_MASK);
- *bar = FIELD_GET(CXL_REGLOC_BIR_MASK, reg_lo);
- *reg_type = FIELD_GET(CXL_REGLOC_RBI_MASK, reg_lo);
+ map->block_offset =
+ ((u64)reg_hi << 32) | (reg_lo & CXL_REGLOC_ADDR_MASK);
+ map->barno = FIELD_GET(CXL_REGLOC_BIR_MASK, reg_lo);
+ map->reg_type = FIELD_GET(CXL_REGLOC_RBI_MASK, reg_lo);
}
/**
- * cxl_mem_setup_regs() - Setup necessary MMIO.
- * @cxlm: The CXL memory device to communicate with.
+ * cxl_find_regblock() - Locate register blocks by type
+ * @pdev: The CXL PCI device to enumerate.
+ * @type: Register Block Indicator id
+ * @map: Enumeration output, clobbered on error
*
- * Return: 0 if all necessary registers mapped.
+ * Return: 0 if register block enumerated, negative error code otherwise
*
- * A memory device is required by spec to implement a certain set of MMIO
- * regions. The purpose of this function is to enumerate and map those
- * registers.
+ * A CXL DVSEC may point to one or more register blocks, search for them
+ * by @type.
*/
-static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
+static int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
+ struct cxl_register_map *map)
{
- struct pci_dev *pdev = cxlm->pdev;
- struct device *dev = &pdev->dev;
u32 regloc_size, regblocks;
- void __iomem *base;
- int regloc, i, n_maps;
- struct cxl_register_map *map, maps[CXL_REGLOC_RBI_TYPES];
- int ret = 0;
-
- regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID);
- if (!regloc) {
- dev_err(dev, "register location dvsec not found\n");
- return -ENXIO;
- }
+ int regloc, i;
- if (pci_request_mem_regions(pdev, pci_name(pdev)))
- return -ENODEV;
+ regloc = pci_find_dvsec_capability(pdev, PCI_DVSEC_VENDOR_ID_CXL,
+ PCI_DVSEC_ID_CXL_REGLOC_DVSEC_ID);
+ if (!regloc)
+ return -ENXIO;
- /* Get the size of the Register Locator DVSEC */
pci_read_config_dword(pdev, regloc + PCI_DVSEC_HEADER1, ®loc_size);
regloc_size = FIELD_GET(PCI_DVSEC_HEADER1_LENGTH_MASK, regloc_size);
regloc += PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET;
regblocks = (regloc_size - PCI_DVSEC_ID_CXL_REGLOC_BLOCK1_OFFSET) / 8;
- for (i = 0, n_maps = 0; i < regblocks; i++, regloc += 8) {
+ for (i = 0; i < regblocks; i++, regloc += 8) {
u32 reg_lo, reg_hi;
- u8 reg_type;
- u64 offset;
- u8 bar;
pci_read_config_dword(pdev, regloc, ®_lo);
pci_read_config_dword(pdev, regloc + 4, ®_hi);
- cxl_decode_register_block(reg_lo, reg_hi, &bar, &offset,
- ®_type);
-
- dev_dbg(dev, "Found register block in bar %u @ 0x%llx of type %u\n",
- bar, offset, reg_type);
-
- /* Ignore unknown register block types */
- if (reg_type > CXL_REGLOC_RBI_MEMDEV)
- continue;
-
- base = cxl_mem_map_regblock(cxlm, bar, offset);
- if (!base)
- return -ENOMEM;
-
- map = &maps[n_maps];
- map->barno = bar;
- map->block_offset = offset;
- map->reg_type = reg_type;
-
- ret = cxl_probe_regs(cxlm, base + offset, map);
-
- /* Always unmap the regblock regardless of probe success */
- cxl_mem_unmap_regblock(cxlm, base);
-
- if (ret)
- return ret;
-
- n_maps++;
- }
-
- pci_release_mem_regions(pdev);
-
- for (i = 0; i < n_maps; i++) {
- ret = cxl_map_regs(cxlm, &maps[i]);
- if (ret)
- break;
- }
-
- return ret;
-}
-
-static int cxl_xfer_log(struct cxl_mem *cxlm, uuid_t *uuid, u32 size, u8 *out)
-{
- u32 remaining = size;
- u32 offset = 0;
-
- while (remaining) {
- u32 xfer_size = min_t(u32, remaining, cxlm->payload_size);
- struct cxl_mbox_get_log {
- uuid_t uuid;
- __le32 offset;
- __le32 length;
- } __packed log = {
- .uuid = *uuid,
- .offset = cpu_to_le32(offset),
- .length = cpu_to_le32(xfer_size)
- };
- int rc;
-
- rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_LOG, &log,
- sizeof(log), out, xfer_size);
- if (rc < 0)
- return rc;
-
- out += xfer_size;
- remaining -= xfer_size;
- offset += xfer_size;
- }
-
- return 0;
-}
-
-/**
- * cxl_walk_cel() - Walk through the Command Effects Log.
- * @cxlm: Device.
- * @size: Length of the Command Effects Log.
- * @cel: CEL
- *
- * Iterate over each entry in the CEL and determine if the driver supports the
- * command. If so, the command is enabled for the device and can be used later.
- */
-static void cxl_walk_cel(struct cxl_mem *cxlm, size_t size, u8 *cel)
-{
- struct cel_entry {
- __le16 opcode;
- __le16 effect;
- } __packed * cel_entry;
- const int cel_entries = size / sizeof(*cel_entry);
- int i;
-
- cel_entry = (struct cel_entry *)cel;
-
- for (i = 0; i < cel_entries; i++) {
- u16 opcode = le16_to_cpu(cel_entry[i].opcode);
- struct cxl_mem_command *cmd = cxl_mem_find_command(opcode);
-
- if (!cmd) {
- dev_dbg(&cxlm->pdev->dev,
- "Opcode 0x%04x unsupported by driver", opcode);
- continue;
- }
-
- set_bit(cmd->info.id, cxlm->enabled_cmds);
- }
-}
-
-struct cxl_mbox_get_supported_logs {
- __le16 entries;
- u8 rsvd[6];
- struct gsl_entry {
- uuid_t uuid;
- __le32 size;
- } __packed entry[];
-} __packed;
-
-static struct cxl_mbox_get_supported_logs *cxl_get_gsl(struct cxl_mem *cxlm)
-{
- struct cxl_mbox_get_supported_logs *ret;
- int rc;
+ cxl_decode_regblock(reg_lo, reg_hi, map);
- ret = kvmalloc(cxlm->payload_size, GFP_KERNEL);
- if (!ret)
- return ERR_PTR(-ENOMEM);
-
- rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_SUPPORTED_LOGS, NULL,
- 0, ret, cxlm->payload_size);
- if (rc < 0) {
- kvfree(ret);
- return ERR_PTR(rc);
+ if (map->reg_type == type)
+ return 0;
}
- return ret;
+ return -ENODEV;
}
-/**
- * cxl_mem_get_partition_info - Get partition info
- * @cxlm: The device to act on
- * @active_volatile_bytes: returned active volatile capacity
- * @active_persistent_bytes: returned active persistent capacity
- * @next_volatile_bytes: return next volatile capacity
- * @next_persistent_bytes: return next persistent capacity
- *
- * Retrieve the current partition info for the device specified. If not 0, the
- * 'next' values are pending and take affect on next cold reset.
- *
- * Return: 0 if no error: or the result of the mailbox command.
- *
- * See CXL @8.2.9.5.2.1 Get Partition Info
- */
-static int cxl_mem_get_partition_info(struct cxl_mem *cxlm,
- u64 *active_volatile_bytes,
- u64 *active_persistent_bytes,
- u64 *next_volatile_bytes,
- u64 *next_persistent_bytes)
+static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
+ struct cxl_register_map *map)
{
- struct cxl_mbox_get_partition_info {
- __le64 active_volatile_cap;
- __le64 active_persistent_cap;
- __le64 next_volatile_cap;
- __le64 next_persistent_cap;
- } __packed pi;
int rc;
- rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_PARTITION_INFO,
- NULL, 0, &pi, sizeof(pi));
+ rc = cxl_find_regblock(pdev, type, map);
if (rc)
return rc;
- *active_volatile_bytes = le64_to_cpu(pi.active_volatile_cap);
- *active_persistent_bytes = le64_to_cpu(pi.active_persistent_cap);
- *next_volatile_bytes = le64_to_cpu(pi.next_volatile_cap);
- *next_persistent_bytes = le64_to_cpu(pi.next_volatile_cap);
-
- *active_volatile_bytes *= CXL_CAPACITY_MULTIPLIER;
- *active_persistent_bytes *= CXL_CAPACITY_MULTIPLIER;
- *next_volatile_bytes *= CXL_CAPACITY_MULTIPLIER;
- *next_persistent_bytes *= CXL_CAPACITY_MULTIPLIER;
-
- return 0;
-}
-
-/**
- * cxl_mem_enumerate_cmds() - Enumerate commands for a device.
- * @cxlm: The device.
- *
- * Returns 0 if enumerate completed successfully.
- *
- * CXL devices have optional support for certain commands. This function will
- * determine the set of supported commands for the hardware and update the
- * enabled_cmds bitmap in the @cxlm.
- */
-static int cxl_mem_enumerate_cmds(struct cxl_mem *cxlm)
-{
- struct cxl_mbox_get_supported_logs *gsl;
- struct device *dev = &cxlm->pdev->dev;
- struct cxl_mem_command *cmd;
- int i, rc;
-
- gsl = cxl_get_gsl(cxlm);
- if (IS_ERR(gsl))
- return PTR_ERR(gsl);
-
- rc = -ENOENT;
- for (i = 0; i < le16_to_cpu(gsl->entries); i++) {
- u32 size = le32_to_cpu(gsl->entry[i].size);
- uuid_t uuid = gsl->entry[i].uuid;
- u8 *log;
-
- dev_dbg(dev, "Found LOG type %pU of size %d", &uuid, size);
-
- if (!uuid_equal(&uuid, &log_uuid[CEL_UUID]))
- continue;
-
- log = kvmalloc(size, GFP_KERNEL);
- if (!log) {
- rc = -ENOMEM;
- goto out;
- }
-
- rc = cxl_xfer_log(cxlm, &uuid, size, log);
- if (rc) {
- kvfree(log);
- goto out;
- }
-
- cxl_walk_cel(cxlm, size, log);
- kvfree(log);
-
- /* In case CEL was bogus, enable some default commands. */
- cxl_for_each_cmd(cmd)
- if (cmd->flags & CXL_CMD_FLAG_FORCE_ENABLE)
- set_bit(cmd->info.id, cxlm->enabled_cmds);
-
- /* Found the required CEL */
- rc = 0;
- }
-
-out:
- kvfree(gsl);
- return rc;
-}
-
-/**
- * cxl_mem_identify() - Send the IDENTIFY command to the device.
- * @cxlm: The device to identify.
- *
- * Return: 0 if identify was executed successfully.
- *
- * This will dispatch the identify command to the device and on success populate
- * structures to be exported to sysfs.
- */
-static int cxl_mem_identify(struct cxl_mem *cxlm)
-{
- /* See CXL 2.0 Table 175 Identify Memory Device Output Payload */
- struct cxl_mbox_identify {
- char fw_revision[0x10];
- __le64 total_capacity;
- __le64 volatile_capacity;
- __le64 persistent_capacity;
- __le64 partition_align;
- __le16 info_event_log_size;
- __le16 warning_event_log_size;
- __le16 failure_event_log_size;
- __le16 fatal_event_log_size;
- __le32 lsa_size;
- u8 poison_list_max_mer[3];
- __le16 inject_poison_limit;
- u8 poison_caps;
- u8 qos_telemetry_caps;
- } __packed id;
- int rc;
-
- rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_IDENTIFY, NULL, 0, &id,
- sizeof(id));
- if (rc < 0)
- return rc;
-
- cxlm->total_bytes = le64_to_cpu(id.total_capacity);
- cxlm->total_bytes *= CXL_CAPACITY_MULTIPLIER;
-
- cxlm->volatile_only_bytes = le64_to_cpu(id.volatile_capacity);
- cxlm->volatile_only_bytes *= CXL_CAPACITY_MULTIPLIER;
-
- cxlm->persistent_only_bytes = le64_to_cpu(id.persistent_capacity);
- cxlm->persistent_only_bytes *= CXL_CAPACITY_MULTIPLIER;
-
- cxlm->partition_align_bytes = le64_to_cpu(id.partition_align);
- cxlm->partition_align_bytes *= CXL_CAPACITY_MULTIPLIER;
-
- dev_dbg(&cxlm->pdev->dev, "Identify Memory Device\n"
- " total_bytes = %#llx\n"
- " volatile_only_bytes = %#llx\n"
- " persistent_only_bytes = %#llx\n"
- " partition_align_bytes = %#llx\n",
- cxlm->total_bytes,
- cxlm->volatile_only_bytes,
- cxlm->persistent_only_bytes,
- cxlm->partition_align_bytes);
-
- cxlm->lsa_size = le32_to_cpu(id.lsa_size);
- memcpy(cxlm->firmware_version, id.fw_revision, sizeof(id.fw_revision));
-
- return 0;
-}
-
-static int cxl_mem_create_range_info(struct cxl_mem *cxlm)
-{
- int rc;
-
- if (cxlm->partition_align_bytes == 0) {
- cxlm->ram_range.start = 0;
- cxlm->ram_range.end = cxlm->volatile_only_bytes - 1;
- cxlm->pmem_range.start = cxlm->volatile_only_bytes;
- cxlm->pmem_range.end = cxlm->volatile_only_bytes +
- cxlm->persistent_only_bytes - 1;
- return 0;
- }
-
- rc = cxl_mem_get_partition_info(cxlm,
- &cxlm->active_volatile_bytes,
- &cxlm->active_persistent_bytes,
- &cxlm->next_volatile_bytes,
- &cxlm->next_persistent_bytes);
- if (rc < 0) {
- dev_err(&cxlm->pdev->dev, "Failed to query partition information\n");
+ rc = cxl_map_regblock(pdev, map);
+ if (rc)
return rc;
- }
-
- dev_dbg(&cxlm->pdev->dev, "Get Partition Info\n"
- " active_volatile_bytes = %#llx\n"
- " active_persistent_bytes = %#llx\n"
- " next_volatile_bytes = %#llx\n"
- " next_persistent_bytes = %#llx\n",
- cxlm->active_volatile_bytes,
- cxlm->active_persistent_bytes,
- cxlm->next_volatile_bytes,
- cxlm->next_persistent_bytes);
- cxlm->ram_range.start = 0;
- cxlm->ram_range.end = cxlm->active_volatile_bytes - 1;
+ rc = cxl_probe_regs(pdev, map);
+ cxl_unmap_regblock(pdev, map);
- cxlm->pmem_range.start = cxlm->active_volatile_bytes;
- cxlm->pmem_range.end = cxlm->active_volatile_bytes +
- cxlm->active_persistent_bytes - 1;
-
- return 0;
+ return rc;
}
-static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
+ struct cxl_register_map map;
struct cxl_memdev *cxlmd;
struct cxl_mem *cxlm;
int rc;
+ /*
+ * Double check the anonymous union trickery in struct cxl_regs
+ * FIXME switch to struct_group()
+ */
+ BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
+ offsetof(struct cxl_regs, device_regs.memdev));
+
rc = pcim_enable_device(pdev);
if (rc)
return rc;
- cxlm = cxl_mem_create(pdev);
+ cxlm = cxl_mem_create(&pdev->dev);
if (IS_ERR(cxlm))
return PTR_ERR(cxlm);
- rc = cxl_mem_setup_regs(cxlm);
+ rc = cxl_setup_regs(pdev, CXL_REGLOC_RBI_MEMDEV, &map);
+ if (rc)
+ return rc;
+
+ rc = cxl_map_regs(cxlm, &map);
if (rc)
return rc;
- rc = cxl_mem_setup_mailbox(cxlm);
+ rc = cxl_pci_setup_mailbox(cxlm);
if (rc)
return rc;
if (rc)
return rc;
- cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm, &cxl_memdev_fops);
+ cxlmd = devm_cxl_add_memdev(cxlm);
if (IS_ERR(cxlmd))
return PTR_ERR(cxlmd);
};
MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
-static struct pci_driver cxl_mem_driver = {
+static struct pci_driver cxl_pci_driver = {
.name = KBUILD_MODNAME,
.id_table = cxl_mem_pci_tbl,
- .probe = cxl_mem_probe,
+ .probe = cxl_pci_probe,
.driver = {
.probe_type = PROBE_PREFER_ASYNCHRONOUS,
},
};
-static __init int cxl_mem_init(void)
-{
- struct dentry *mbox_debugfs;
- int rc;
-
- /* Double check the anonymous union trickery in struct cxl_regs */
- BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
- offsetof(struct cxl_regs, device_regs.memdev));
-
- rc = pci_register_driver(&cxl_mem_driver);
- if (rc)
- return rc;
-
- cxl_debugfs = debugfs_create_dir("cxl", NULL);
- mbox_debugfs = debugfs_create_dir("mbox", cxl_debugfs);
- debugfs_create_bool("raw_allow_all", 0600, mbox_debugfs,
- &cxl_raw_allow_all);
-
- return 0;
-}
-
-static __exit void cxl_mem_exit(void)
-{
- debugfs_remove_recursive(cxl_debugfs);
- pci_unregister_driver(&cxl_mem_driver);
-}
-
MODULE_LICENSE("GPL v2");
-module_init(cxl_mem_init);
-module_exit(cxl_mem_exit);
+module_pci_driver(cxl_pci_driver);
MODULE_IMPORT_NS(CXL);
#define CXL_REGLOC_BIR_MASK GENMASK(2, 0)
/* Register Block Identifier (RBI) */
-#define CXL_REGLOC_RBI_MASK GENMASK(15, 8)
-#define CXL_REGLOC_RBI_EMPTY 0
-#define CXL_REGLOC_RBI_COMPONENT 1
-#define CXL_REGLOC_RBI_VIRT 2
-#define CXL_REGLOC_RBI_MEMDEV 3
-#define CXL_REGLOC_RBI_TYPES CXL_REGLOC_RBI_MEMDEV + 1
+enum cxl_regloc_type {
+ CXL_REGLOC_RBI_EMPTY = 0,
+ CXL_REGLOC_RBI_COMPONENT,
+ CXL_REGLOC_RBI_VIRT,
+ CXL_REGLOC_RBI_MEMDEV,
+ CXL_REGLOC_RBI_TYPES
+};
+#define CXL_REGLOC_RBI_MASK GENMASK(15, 8)
#define CXL_REGLOC_ADDR_MASK GENMASK(31, 16)
#endif /* __CXL_PCI_H__ */
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
#include <linux/libnvdimm.h>
+#include <asm/unaligned.h>
#include <linux/device.h>
#include <linux/module.h>
#include <linux/ndctl.h>
*/
static struct workqueue_struct *cxl_pmem_wq;
-static void unregister_nvdimm(void *nvdimm)
-{
- nvdimm_delete(nvdimm);
-}
+static __read_mostly DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
-static int match_nvdimm_bridge(struct device *dev, const void *data)
+static void clear_exclusive(void *cxlm)
{
- return strcmp(dev_name(dev), "nvdimm-bridge") == 0;
+ clear_exclusive_cxl_commands(cxlm, exclusive_cmds);
}
-static struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(void)
+static void unregister_nvdimm(void *nvdimm)
{
- struct device *dev;
-
- dev = bus_find_device(&cxl_bus_type, NULL, NULL, match_nvdimm_bridge);
- if (!dev)
- return NULL;
- return to_cxl_nvdimm_bridge(dev);
+ nvdimm_delete(nvdimm);
}
static int cxl_nvdimm_probe(struct device *dev)
{
struct cxl_nvdimm *cxl_nvd = to_cxl_nvdimm(dev);
+ struct cxl_memdev *cxlmd = cxl_nvd->cxlmd;
+ unsigned long flags = 0, cmd_mask = 0;
+ struct cxl_mem *cxlm = cxlmd->cxlm;
struct cxl_nvdimm_bridge *cxl_nvb;
- unsigned long flags = 0;
struct nvdimm *nvdimm;
- int rc = -ENXIO;
+ int rc;
- cxl_nvb = cxl_find_nvdimm_bridge();
+ cxl_nvb = cxl_find_nvdimm_bridge(cxl_nvd);
if (!cxl_nvb)
return -ENXIO;
device_lock(&cxl_nvb->dev);
- if (!cxl_nvb->nvdimm_bus)
+ if (!cxl_nvb->nvdimm_bus) {
+ rc = -ENXIO;
+ goto out;
+ }
+
+ set_exclusive_cxl_commands(cxlm, exclusive_cmds);
+ rc = devm_add_action_or_reset(dev, clear_exclusive, cxlm);
+ if (rc)
goto out;
set_bit(NDD_LABELING, &flags);
- nvdimm = nvdimm_create(cxl_nvb->nvdimm_bus, cxl_nvd, NULL, flags, 0, 0,
- NULL);
- if (!nvdimm)
+ set_bit(ND_CMD_GET_CONFIG_SIZE, &cmd_mask);
+ set_bit(ND_CMD_GET_CONFIG_DATA, &cmd_mask);
+ set_bit(ND_CMD_SET_CONFIG_DATA, &cmd_mask);
+ nvdimm = nvdimm_create(cxl_nvb->nvdimm_bus, cxl_nvd, NULL, flags,
+ cmd_mask, 0, NULL);
+ if (!nvdimm) {
+ rc = -ENOMEM;
goto out;
+ }
+ dev_set_drvdata(dev, nvdimm);
rc = devm_add_action_or_reset(dev, unregister_nvdimm, nvdimm);
out:
device_unlock(&cxl_nvb->dev);
.id = CXL_DEVICE_NVDIMM,
};
+static int cxl_pmem_get_config_size(struct cxl_mem *cxlm,
+ struct nd_cmd_get_config_size *cmd,
+ unsigned int buf_len)
+{
+ if (sizeof(*cmd) > buf_len)
+ return -EINVAL;
+
+ *cmd = (struct nd_cmd_get_config_size) {
+ .config_size = cxlm->lsa_size,
+ .max_xfer = cxlm->payload_size,
+ };
+
+ return 0;
+}
+
+static int cxl_pmem_get_config_data(struct cxl_mem *cxlm,
+ struct nd_cmd_get_config_data_hdr *cmd,
+ unsigned int buf_len)
+{
+ struct cxl_mbox_get_lsa get_lsa;
+ int rc;
+
+ if (sizeof(*cmd) > buf_len)
+ return -EINVAL;
+ if (struct_size(cmd, out_buf, cmd->in_length) > buf_len)
+ return -EINVAL;
+
+ get_lsa = (struct cxl_mbox_get_lsa) {
+ .offset = cmd->in_offset,
+ .length = cmd->in_length,
+ };
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_GET_LSA, &get_lsa,
+ sizeof(get_lsa), cmd->out_buf,
+ cmd->in_length);
+ cmd->status = 0;
+
+ return rc;
+}
+
+static int cxl_pmem_set_config_data(struct cxl_mem *cxlm,
+ struct nd_cmd_set_config_hdr *cmd,
+ unsigned int buf_len)
+{
+ struct cxl_mbox_set_lsa *set_lsa;
+ int rc;
+
+ if (sizeof(*cmd) > buf_len)
+ return -EINVAL;
+
+ /* 4-byte status follows the input data in the payload */
+ if (struct_size(cmd, in_buf, cmd->in_length) + 4 > buf_len)
+ return -EINVAL;
+
+ set_lsa =
+ kvzalloc(struct_size(set_lsa, data, cmd->in_length), GFP_KERNEL);
+ if (!set_lsa)
+ return -ENOMEM;
+
+ *set_lsa = (struct cxl_mbox_set_lsa) {
+ .offset = cmd->in_offset,
+ };
+ memcpy(set_lsa->data, cmd->in_buf, cmd->in_length);
+
+ rc = cxl_mem_mbox_send_cmd(cxlm, CXL_MBOX_OP_SET_LSA, set_lsa,
+ struct_size(set_lsa, data, cmd->in_length),
+ NULL, 0);
+
+ /*
+ * Set "firmware" status (4-packed bytes at the end of the input
+ * payload.
+ */
+ put_unaligned(0, (u32 *) &cmd->in_buf[cmd->in_length]);
+ kvfree(set_lsa);
+
+ return rc;
+}
+
+static int cxl_pmem_nvdimm_ctl(struct nvdimm *nvdimm, unsigned int cmd,
+ void *buf, unsigned int buf_len)
+{
+ struct cxl_nvdimm *cxl_nvd = nvdimm_provider_data(nvdimm);
+ unsigned long cmd_mask = nvdimm_cmd_mask(nvdimm);
+ struct cxl_memdev *cxlmd = cxl_nvd->cxlmd;
+ struct cxl_mem *cxlm = cxlmd->cxlm;
+
+ if (!test_bit(cmd, &cmd_mask))
+ return -ENOTTY;
+
+ switch (cmd) {
+ case ND_CMD_GET_CONFIG_SIZE:
+ return cxl_pmem_get_config_size(cxlm, buf, buf_len);
+ case ND_CMD_GET_CONFIG_DATA:
+ return cxl_pmem_get_config_data(cxlm, buf, buf_len);
+ case ND_CMD_SET_CONFIG_DATA:
+ return cxl_pmem_set_config_data(cxlm, buf, buf_len);
+ default:
+ return -ENOTTY;
+ }
+}
+
static int cxl_pmem_ctl(struct nvdimm_bus_descriptor *nd_desc,
struct nvdimm *nvdimm, unsigned int cmd, void *buf,
unsigned int buf_len, int *cmd_rc)
{
- return -ENOTTY;
+ /*
+ * No firmware response to translate, let the transport error
+ * code take precedence.
+ */
+ *cmd_rc = 0;
+
+ if (!nvdimm)
+ return -ENOTTY;
+ return cxl_pmem_nvdimm_ctl(nvdimm, cmd, buf, buf_len);
}
static bool online_nvdimm_bus(struct cxl_nvdimm_bridge *cxl_nvb)
{
int rc;
+ set_bit(CXL_MEM_COMMAND_ID_SET_PARTITION_INFO, exclusive_cmds);
+ set_bit(CXL_MEM_COMMAND_ID_SET_SHUTDOWN_STATE, exclusive_cmds);
+ set_bit(CXL_MEM_COMMAND_ID_SET_LSA, exclusive_cmds);
+
cxl_pmem_wq = alloc_ordered_workqueue("cxl_pmem", 0);
if (!cxl_pmem_wq)
return -ENXIO;
return hashlen_hash(hashlen_string("DAX", host)) % DAX_HASH_SIZE;
}
+#ifdef CONFIG_BLOCK
+#include <linux/blkdev.h>
+
+int bdev_dax_pgoff(struct block_device *bdev, sector_t sector, size_t size,
+ pgoff_t *pgoff)
+{
+ sector_t start_sect = bdev ? get_start_sect(bdev) : 0;
+ phys_addr_t phys_off = (start_sect + sector) * 512;
+
+ if (pgoff)
+ *pgoff = PHYS_PFN(phys_off);
+ if (phys_off % PAGE_SIZE || size % PAGE_SIZE)
+ return -EINVAL;
+ return 0;
+}
+EXPORT_SYMBOL(bdev_dax_pgoff);
+
+#if IS_ENABLED(CONFIG_FS_DAX)
/**
* dax_get_by_host() - temporary lookup mechanism for filesystem-dax
* @host: alternate name for the device registered by a dax driver
return found;
}
-#ifdef CONFIG_BLOCK
-#include <linux/blkdev.h>
-
-int bdev_dax_pgoff(struct block_device *bdev, sector_t sector, size_t size,
- pgoff_t *pgoff)
-{
- sector_t start_sect = bdev ? get_start_sect(bdev) : 0;
- phys_addr_t phys_off = (start_sect + sector) * 512;
-
- if (pgoff)
- *pgoff = PHYS_PFN(phys_off);
- if (phys_off % PAGE_SIZE || size % PAGE_SIZE)
- return -EINVAL;
- return 0;
-}
-EXPORT_SYMBOL(bdev_dax_pgoff);
-
-#if IS_ENABLED(CONFIG_FS_DAX)
struct dax_device *fs_dax_get_by_bdev(struct block_device *bdev)
{
if (!blk_queue_dax(bdev->bd_disk->queue))
DAXDEV_SYNC,
};
-static ssize_t write_cache_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- struct dax_device *dax_dev = dax_get_by_host(dev_name(dev));
- ssize_t rc;
-
- WARN_ON_ONCE(!dax_dev);
- if (!dax_dev)
- return -ENXIO;
-
- rc = sprintf(buf, "%d\n", !!dax_write_cache_enabled(dax_dev));
- put_dax(dax_dev);
- return rc;
-}
-
-static ssize_t write_cache_store(struct device *dev,
- struct device_attribute *attr, const char *buf, size_t len)
-{
- bool write_cache;
- int rc = strtobool(buf, &write_cache);
- struct dax_device *dax_dev = dax_get_by_host(dev_name(dev));
-
- WARN_ON_ONCE(!dax_dev);
- if (!dax_dev)
- return -ENXIO;
-
- if (rc)
- len = rc;
- else
- dax_write_cache(dax_dev, write_cache);
-
- put_dax(dax_dev);
- return len;
-}
-static DEVICE_ATTR_RW(write_cache);
-
-static umode_t dax_visible(struct kobject *kobj, struct attribute *a, int n)
-{
- struct device *dev = container_of(kobj, typeof(*dev), kobj);
- struct dax_device *dax_dev = dax_get_by_host(dev_name(dev));
-
- WARN_ON_ONCE(!dax_dev);
- if (!dax_dev)
- return 0;
-
-#ifndef CONFIG_ARCH_HAS_PMEM_API
- if (a == &dev_attr_write_cache.attr)
- return 0;
-#endif
- return a->mode;
-}
-
-static struct attribute *dax_attributes[] = {
- &dev_attr_write_cache.attr,
- NULL,
-};
-
-struct attribute_group dax_attribute_group = {
- .name = "dax",
- .attrs = dax_attributes,
- .is_visible = dax_visible,
-};
-EXPORT_SYMBOL_GPL(dax_attribute_group);
-
/**
* dax_direct_access() - translate a device pgoff to an absolute pfn
* @dax_dev: a dax_device instance representing the logical memory range
}
EXPORT_SYMBOL(devfreq_add_governor);
+static void devm_devfreq_remove_governor(void *governor)
+{
+ WARN_ON(devfreq_remove_governor(governor));
+}
+
+/**
+ * devm_devfreq_add_governor() - Add devfreq governor
+ * @dev: device which adds devfreq governor
+ * @governor: the devfreq governor to be added
+ *
+ * This is a resource-managed variant of devfreq_add_governor().
+ */
+int devm_devfreq_add_governor(struct device *dev,
+ struct devfreq_governor *governor)
+{
+ int err;
+
+ err = devfreq_add_governor(governor);
+ if (err)
+ return err;
+
+ return devm_add_action_or_reset(dev, devm_devfreq_remove_governor,
+ governor);
+}
+EXPORT_SYMBOL(devm_devfreq_add_governor);
+
/**
* devfreq_remove_governor() - Remove devfreq feature from a device.
* @governor: the devfreq governor to be removed
int devfreq_add_governor(struct devfreq_governor *governor);
int devfreq_remove_governor(struct devfreq_governor *governor);
+int devm_devfreq_add_governor(struct device *dev,
+ struct devfreq_governor *governor);
+
int devfreq_update_status(struct devfreq *devfreq, unsigned long freq);
int devfreq_update_target(struct devfreq *devfreq, unsigned long freq);
struct tegra_devfreq {
struct devfreq *devfreq;
- struct opp_table *opp_table;
struct reset_control *reset;
struct clk *clock;
.event_handler = tegra_governor_event_handler,
};
+static void devm_tegra_devfreq_deinit_hw(void *data)
+{
+ struct tegra_devfreq *tegra = data;
+
+ reset_control_reset(tegra->reset);
+ clk_disable_unprepare(tegra->clock);
+}
+
+static int devm_tegra_devfreq_init_hw(struct device *dev,
+ struct tegra_devfreq *tegra)
+{
+ int err;
+
+ err = clk_prepare_enable(tegra->clock);
+ if (err) {
+ dev_err(dev, "Failed to prepare and enable ACTMON clock\n");
+ return err;
+ }
+
+ err = devm_add_action_or_reset(dev, devm_tegra_devfreq_deinit_hw,
+ tegra);
+ if (err)
+ return err;
+
+ err = reset_control_reset(tegra->reset);
+ if (err) {
+ dev_err(dev, "Failed to reset hardware: %d\n", err);
+ return err;
+ }
+
+ return err;
+}
+
static int tegra_devfreq_probe(struct platform_device *pdev)
{
u32 hw_version = BIT(tegra_sku_info.soc_speedo_id);
return err;
}
- tegra->opp_table = dev_pm_opp_set_supported_hw(&pdev->dev,
- &hw_version, 1);
- err = PTR_ERR_OR_ZERO(tegra->opp_table);
+ err = devm_pm_opp_set_supported_hw(&pdev->dev, &hw_version, 1);
if (err) {
dev_err(&pdev->dev, "Failed to set supported HW: %d\n", err);
return err;
}
- err = dev_pm_opp_of_add_table_noclk(&pdev->dev, 0);
+ err = devm_pm_opp_of_add_table_noclk(&pdev->dev, 0);
if (err) {
dev_err(&pdev->dev, "Failed to add OPP table: %d\n", err);
- goto put_hw;
- }
-
- err = clk_prepare_enable(tegra->clock);
- if (err) {
- dev_err(&pdev->dev,
- "Failed to prepare and enable ACTMON clock\n");
- goto remove_table;
+ return err;
}
- err = reset_control_reset(tegra->reset);
- if (err) {
- dev_err(&pdev->dev, "Failed to reset hardware: %d\n", err);
- goto disable_clk;
- }
+ err = devm_tegra_devfreq_init_hw(&pdev->dev, tegra);
+ if (err)
+ return err;
rate = clk_round_rate(tegra->emc_clock, ULONG_MAX);
- if (rate < 0) {
+ if (rate <= 0) {
dev_err(&pdev->dev, "Failed to round clock rate: %ld\n", rate);
- err = rate;
- goto disable_clk;
+ return rate ?: -EINVAL;
}
tegra->max_freq = rate / KHZ;
INIT_DELAYED_WORK(&tegra->cpufreq_update_work,
tegra_actmon_delayed_update);
- err = devfreq_add_governor(&tegra_devfreq_governor);
+ err = devm_devfreq_add_governor(&pdev->dev, &tegra_devfreq_governor);
if (err) {
dev_err(&pdev->dev, "Failed to add governor: %d\n", err);
- goto remove_opps;
+ return err;
}
tegra_devfreq_profile.initial_freq = clk_get_rate(tegra->emc_clock);
- devfreq = devfreq_add_device(&pdev->dev, &tegra_devfreq_profile,
- "tegra_actmon", NULL);
- if (IS_ERR(devfreq)) {
- err = PTR_ERR(devfreq);
- goto remove_governor;
- }
-
- return 0;
-
-remove_governor:
- devfreq_remove_governor(&tegra_devfreq_governor);
-
-remove_opps:
- dev_pm_opp_remove_all_dynamic(&pdev->dev);
-
- reset_control_reset(tegra->reset);
-disable_clk:
- clk_disable_unprepare(tegra->clock);
-remove_table:
- dev_pm_opp_of_remove_table(&pdev->dev);
-put_hw:
- dev_pm_opp_put_supported_hw(tegra->opp_table);
-
- return err;
-}
-
-static int tegra_devfreq_remove(struct platform_device *pdev)
-{
- struct tegra_devfreq *tegra = platform_get_drvdata(pdev);
-
- devfreq_remove_device(tegra->devfreq);
- devfreq_remove_governor(&tegra_devfreq_governor);
-
- reset_control_reset(tegra->reset);
- clk_disable_unprepare(tegra->clock);
-
- dev_pm_opp_of_remove_table(&pdev->dev);
- dev_pm_opp_put_supported_hw(tegra->opp_table);
+ devfreq = devm_devfreq_add_device(&pdev->dev, &tegra_devfreq_profile,
+ "tegra_actmon", NULL);
+ if (IS_ERR(devfreq))
+ return PTR_ERR(devfreq);
return 0;
}
static struct platform_driver tegra_devfreq_driver = {
.probe = tegra_devfreq_probe,
- .remove = tegra_devfreq_remove,
.driver = {
.name = "tegra-devfreq",
.of_match_table = tegra_devfreq_of_match,
BUG_ON(dmabuf->vmapping_counter);
/*
- * Any fences that a dma-buf poll can wait on should be signaled
- * before releasing dma-buf. This is the responsibility of each
- * driver that uses the reservation objects.
- *
- * If you hit this BUG() it means someone dropped their ref to the
- * dma-buf while still having pending operation to the buffer.
+ * If you hit this BUG() it could mean:
+ * * There's a file reference imbalance in dma_buf_poll / dma_buf_poll_cb or somewhere else
+ * * dmabuf->cb_in/out.active are non-0 despite no pending fence callback
*/
BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
static void dma_buf_poll_cb(struct dma_fence *fence, struct dma_fence_cb *cb)
{
struct dma_buf_poll_cb_t *dcb = (struct dma_buf_poll_cb_t *)cb;
+ struct dma_buf *dmabuf = container_of(dcb->poll, struct dma_buf, poll);
unsigned long flags;
spin_lock_irqsave(&dcb->poll->lock, flags);
dcb->active = 0;
spin_unlock_irqrestore(&dcb->poll->lock, flags);
dma_fence_put(fence);
+ /* Paired with get_file in dma_buf_poll */
+ fput(dmabuf->file);
}
-static bool dma_buf_poll_shared(struct dma_resv *resv,
+static bool dma_buf_poll_add_cb(struct dma_resv *resv, bool write,
struct dma_buf_poll_cb_t *dcb)
{
- struct dma_resv_list *fobj = dma_resv_shared_list(resv);
+ struct dma_resv_iter cursor;
struct dma_fence *fence;
- int i, r;
-
- if (!fobj)
- return false;
+ int r;
- for (i = 0; i < fobj->shared_count; ++i) {
- fence = rcu_dereference_protected(fobj->shared[i],
- dma_resv_held(resv));
+ dma_resv_for_each_fence(&cursor, resv, write, fence) {
dma_fence_get(fence);
r = dma_fence_add_callback(fence, &dcb->cb, dma_buf_poll_cb);
if (!r)
return false;
}
-static bool dma_buf_poll_excl(struct dma_resv *resv,
- struct dma_buf_poll_cb_t *dcb)
-{
- struct dma_fence *fence = dma_resv_excl_fence(resv);
- int r;
-
- if (!fence)
- return false;
-
- dma_fence_get(fence);
- r = dma_fence_add_callback(fence, &dcb->cb, dma_buf_poll_cb);
- if (!r)
- return true;
- dma_fence_put(fence);
-
- return false;
-}
-
static __poll_t dma_buf_poll(struct file *file, poll_table *poll)
{
struct dma_buf *dmabuf;
spin_unlock_irq(&dmabuf->poll.lock);
if (events & EPOLLOUT) {
- if (!dma_buf_poll_shared(resv, dcb) &&
- !dma_buf_poll_excl(resv, dcb))
+ /* Paired with fput in dma_buf_poll_cb */
+ get_file(dmabuf->file);
+
+ if (!dma_buf_poll_add_cb(resv, true, dcb))
/* No callback queued, wake up any other waiters */
dma_buf_poll_cb(NULL, &dcb->cb);
else
spin_unlock_irq(&dmabuf->poll.lock);
if (events & EPOLLIN) {
- if (!dma_buf_poll_excl(resv, dcb))
+ /* Paired with fput in dma_buf_poll_cb */
+ get_file(dmabuf->file);
+
+ if (!dma_buf_poll_add_cb(resv, false, dcb))
/* No callback queued, wake up any other waiters */
dma_buf_poll_cb(NULL, &dcb->cb);
else
{
struct dma_buf *buf_obj;
struct dma_buf_attachment *attach_obj;
- struct dma_resv *robj;
- struct dma_resv_list *fobj;
+ struct dma_resv_iter cursor;
struct dma_fence *fence;
- int count = 0, attach_count, shared_count, i;
+ int count = 0, attach_count;
size_t size = 0;
int ret;
if (ret)
goto error_unlock;
+
+ spin_lock(&buf_obj->name_lock);
seq_printf(s, "%08zu\t%08x\t%08x\t%08ld\t%s\t%08lu\t%s\n",
buf_obj->size,
buf_obj->file->f_flags, buf_obj->file->f_mode,
buf_obj->exp_name,
file_inode(buf_obj->file)->i_ino,
buf_obj->name ?: "");
+ spin_unlock(&buf_obj->name_lock);
- robj = buf_obj->resv;
- fence = dma_resv_excl_fence(robj);
- if (fence)
- seq_printf(s, "\tExclusive fence: %s %s %ssignalled\n",
- fence->ops->get_driver_name(fence),
- fence->ops->get_timeline_name(fence),
- dma_fence_is_signaled(fence) ? "" : "un");
-
- fobj = rcu_dereference_protected(robj->fence,
- dma_resv_held(robj));
- shared_count = fobj ? fobj->shared_count : 0;
- for (i = 0; i < shared_count; i++) {
- fence = rcu_dereference_protected(fobj->shared[i],
- dma_resv_held(robj));
- seq_printf(s, "\tShared fence: %s %s %ssignalled\n",
+ dma_resv_for_each_fence(&cursor, buf_obj->resv, true, fence) {
+ seq_printf(s, "\t%s fence: %s %s %ssignalled\n",
+ dma_resv_iter_is_exclusive(&cursor) ?
+ "Exclusive" : "Shared",
fence->ops->get_driver_name(fence),
fence->ops->get_timeline_name(fence),
dma_fence_is_signaled(fence) ? "" : "un");
{
cursor->seq = read_seqcount_begin(&cursor->obj->seq);
cursor->index = -1;
- if (cursor->all_fences)
+ cursor->shared_count = 0;
+ if (cursor->all_fences) {
cursor->fences = dma_resv_shared_list(cursor->obj);
- else
+ if (cursor->fences)
+ cursor->shared_count = cursor->fences->shared_count;
+ } else {
cursor->fences = NULL;
+ }
cursor->is_restarted = true;
}
continue;
} else if (!cursor->fences ||
- cursor->index >= cursor->fences->shared_count) {
+ cursor->index >= cursor->shared_count) {
cursor->fence = NULL;
break;
EXPORT_SYMBOL(dma_resv_iter_next_unlocked);
/**
+ * dma_resv_iter_first - first fence from a locked dma_resv object
+ * @cursor: cursor to record the current position
+ *
+ * Return the first fence in the dma_resv object while holding the
+ * &dma_resv.lock.
+ */
+struct dma_fence *dma_resv_iter_first(struct dma_resv_iter *cursor)
+{
+ struct dma_fence *fence;
+
+ dma_resv_assert_held(cursor->obj);
+
+ cursor->index = 0;
+ if (cursor->all_fences)
+ cursor->fences = dma_resv_shared_list(cursor->obj);
+ else
+ cursor->fences = NULL;
+
+ fence = dma_resv_excl_fence(cursor->obj);
+ if (!fence)
+ fence = dma_resv_iter_next(cursor);
+
+ cursor->is_restarted = true;
+ return fence;
+}
+EXPORT_SYMBOL_GPL(dma_resv_iter_first);
+
+/**
+ * dma_resv_iter_next - next fence from a locked dma_resv object
+ * @cursor: cursor to record the current position
+ *
+ * Return the next fences from the dma_resv object while holding the
+ * &dma_resv.lock.
+ */
+struct dma_fence *dma_resv_iter_next(struct dma_resv_iter *cursor)
+{
+ unsigned int idx;
+
+ dma_resv_assert_held(cursor->obj);
+
+ cursor->is_restarted = false;
+ if (!cursor->fences || cursor->index >= cursor->fences->shared_count)
+ return NULL;
+
+ idx = cursor->index++;
+ return rcu_dereference_protected(cursor->fences->shared[idx],
+ dma_resv_held(cursor->obj));
+}
+EXPORT_SYMBOL_GPL(dma_resv_iter_next);
+
+/**
* dma_resv_copy_fences - Copy all fences from src to dst.
* @dst: the destination reservation object
* @src: the source reservation object
dma_resv_list_free(list);
dma_fence_put(excl);
- if (cursor.fences) {
- unsigned int cnt = cursor.fences->shared_count;
-
- list = dma_resv_list_alloc(cnt);
+ if (cursor.shared_count) {
+ list = dma_resv_list_alloc(cursor.shared_count);
if (!list) {
dma_resv_iter_end(&cursor);
return -ENOMEM;
if (fence_excl)
dma_fence_put(*fence_excl);
- count = cursor.fences ? cursor.fences->shared_count : 0;
+ count = cursor.shared_count;
count += fence_excl ? 0 : 1;
/* Eventually re-allocate the array */
config XILINX_ZYNQMP_DMA
tristate "Xilinx ZynqMP DMA Engine"
- depends on (ARCH_ZYNQ || MICROBLAZE || ARM64)
+ depends on ARCH_ZYNQ || MICROBLAZE || ARM64 || COMPILE_TEST
select DMA_ENGINE
help
Enable support for Xilinx ZynqMP DMA controller.
struct msgdma_sw_desc *desc, *next;
list_for_each_entry_safe(desc, next, &mdev->done_list, node) {
- dma_async_tx_callback callback;
- void *callback_param;
+ struct dmaengine_desc_callback cb;
list_del(&desc->node);
- callback = desc->async_tx.callback;
- callback_param = desc->async_tx.callback_param;
- if (callback) {
+ dmaengine_desc_get_callback(&desc->async_tx, &cb);
+ if (dmaengine_desc_callback_valid(&cb)) {
spin_unlock(&mdev->lock);
- callback(callback_param);
+ dmaengine_desc_callback_invoke(&cb, NULL);
spin_lock(&mdev->lock);
}
#define AT_XDMAC_CC_WRIP (0x1 << 23) /* Write in Progress (read only) */
#define AT_XDMAC_CC_WRIP_DONE (0x0 << 23)
#define AT_XDMAC_CC_WRIP_IN_PROGRESS (0x1 << 23)
-#define AT_XDMAC_CC_PERID(i) (0x7f & (i) << 24) /* Channel Peripheral Identifier */
+#define AT_XDMAC_CC_PERID(i) ((0x7f & (i)) << 24) /* Channel Peripheral Identifier */
#define AT_XDMAC_CDS_MSP 0x2C /* Channel Data Stride Memory Set Pattern */
#define AT_XDMAC_CSUS 0x30 /* Channel Source Microblock Stride */
#define AT_XDMAC_CDUS 0x34 /* Channel Destination Microblock Stride */
return;
}
-#ifdef CONFIG_PM
-static int atmel_xdmac_prepare(struct device *dev)
+static void at_xdmac_axi_config(struct platform_device *pdev)
+{
+ struct at_xdmac *atxdmac = (struct at_xdmac *)platform_get_drvdata(pdev);
+ bool dev_m2m = false;
+ u32 dma_requests;
+
+ if (!atxdmac->layout->axi_config)
+ return; /* Not supported */
+
+ if (!of_property_read_u32(pdev->dev.of_node, "dma-requests",
+ &dma_requests)) {
+ dev_info(&pdev->dev, "controller in mem2mem mode.\n");
+ dev_m2m = true;
+ }
+
+ if (dev_m2m) {
+ at_xdmac_write(atxdmac, AT_XDMAC_GCFG, AT_XDMAC_GCFG_M2M);
+ at_xdmac_write(atxdmac, AT_XDMAC_GWAC, AT_XDMAC_GWAC_M2M);
+ } else {
+ at_xdmac_write(atxdmac, AT_XDMAC_GCFG, AT_XDMAC_GCFG_P2M);
+ at_xdmac_write(atxdmac, AT_XDMAC_GWAC, AT_XDMAC_GWAC_P2M);
+ }
+}
+
+static int __maybe_unused atmel_xdmac_prepare(struct device *dev)
{
struct at_xdmac *atxdmac = dev_get_drvdata(dev);
struct dma_chan *chan, *_chan;
}
return 0;
}
-#else
-# define atmel_xdmac_prepare NULL
-#endif
-#ifdef CONFIG_PM_SLEEP
-static int atmel_xdmac_suspend(struct device *dev)
+static int __maybe_unused atmel_xdmac_suspend(struct device *dev)
{
struct at_xdmac *atxdmac = dev_get_drvdata(dev);
struct dma_chan *chan, *_chan;
return 0;
}
-static int atmel_xdmac_resume(struct device *dev)
+static int __maybe_unused atmel_xdmac_resume(struct device *dev)
{
struct at_xdmac *atxdmac = dev_get_drvdata(dev);
struct at_xdmac_chan *atchan;
struct dma_chan *chan, *_chan;
+ struct platform_device *pdev = container_of(dev, struct platform_device, dev);
int i;
int ret;
if (ret)
return ret;
+ at_xdmac_axi_config(pdev);
+
/* Clear pending interrupts. */
for (i = 0; i < atxdmac->dma.chancnt; i++) {
atchan = &atxdmac->chan[i];
}
return 0;
}
-#endif /* CONFIG_PM_SLEEP */
-
-static void at_xdmac_axi_config(struct platform_device *pdev)
-{
- struct at_xdmac *atxdmac = (struct at_xdmac *)platform_get_drvdata(pdev);
- bool dev_m2m = false;
- u32 dma_requests;
-
- if (!atxdmac->layout->axi_config)
- return; /* Not supported */
-
- if (!of_property_read_u32(pdev->dev.of_node, "dma-requests",
- &dma_requests)) {
- dev_info(&pdev->dev, "controller in mem2mem mode.\n");
- dev_m2m = true;
- }
-
- if (dev_m2m) {
- at_xdmac_write(atxdmac, AT_XDMAC_GCFG, AT_XDMAC_GCFG_M2M);
- at_xdmac_write(atxdmac, AT_XDMAC_GWAC, AT_XDMAC_GWAC_M2M);
- } else {
- at_xdmac_write(atxdmac, AT_XDMAC_GCFG, AT_XDMAC_GCFG_P2M);
- at_xdmac_write(atxdmac, AT_XDMAC_GWAC, AT_XDMAC_GWAC_P2M);
- }
-}
static int at_xdmac_probe(struct platform_device *pdev)
{
return 0;
}
-static const struct dev_pm_ops atmel_xdmac_dev_pm_ops = {
+static const struct dev_pm_ops __maybe_unused atmel_xdmac_dev_pm_ops = {
.prepare = atmel_xdmac_prepare,
SET_LATE_SYSTEM_SLEEP_PM_OPS(atmel_xdmac_suspend, atmel_xdmac_resume)
};
.driver = {
.name = "at_xdmac",
.of_match_table = of_match_ptr(atmel_xdmac_dt_ids),
- .pm = &atmel_xdmac_dev_pm_ops,
+ .pm = pm_ptr(&atmel_xdmac_dev_pm_ops),
}
};
struct bcom_ata_var *var;
/* Reset all BD */
- memset(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
+ memset_io(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
tsk->index = 0;
tsk->outdex = 0;
tsk->bd = bcom_sram_alloc(bd_count * bd_size, 4, &tsk->bd_pa);
if (!tsk->bd)
goto error;
- memset(tsk->bd, 0x00, bd_count * bd_size);
+ memset_io(tsk->bd, 0x00, bd_count * bd_size);
tsk->num_bd = bd_count;
tsk->bd_size = bd_size;
inc = bcom_task_inc(task);
/* Clear & copy */
- memset(var, 0x00, BCOM_VAR_SIZE);
- memset(inc, 0x00, BCOM_INC_SIZE);
+ memset_io(var, 0x00, BCOM_VAR_SIZE);
+ memset_io(inc, 0x00, BCOM_INC_SIZE);
desc_src = (u32 *)(hdr + 1);
var_src = desc_src + hdr->desc_size;
inc_src = var_src + hdr->var_size;
- memcpy(desc, desc_src, hdr->desc_size * sizeof(u32));
- memcpy(var + hdr->first_var, var_src, hdr->var_size * sizeof(u32));
- memcpy(inc, inc_src, hdr->inc_size * sizeof(u32));
+ memcpy_toio(desc, desc_src, hdr->desc_size * sizeof(u32));
+ memcpy_toio(var + hdr->first_var, var_src, hdr->var_size * sizeof(u32));
+ memcpy_toio(inc, inc_src, hdr->inc_size * sizeof(u32));
return 0;
}
return -ENOMEM;
}
- memset(bcom_eng->tdt, 0x00, tdt_size);
- memset(bcom_eng->ctx, 0x00, ctx_size);
- memset(bcom_eng->var, 0x00, var_size);
- memset(bcom_eng->fdt, 0x00, fdt_size);
+ memset_io(bcom_eng->tdt, 0x00, tdt_size);
+ memset_io(bcom_eng->ctx, 0x00, ctx_size);
+ memset_io(bcom_eng->var, 0x00, var_size);
+ memset_io(bcom_eng->fdt, 0x00, fdt_size);
/* Copy the FDT for the EU#3 */
- memcpy(&bcom_eng->fdt[48], fdt_ops, sizeof(fdt_ops));
+ memcpy_toio(&bcom_eng->fdt[48], fdt_ops, sizeof(fdt_ops));
/* Initialize Task base structure */
for (task=0; task<BCOM_MAX_TASKS; task++)
tsk->index = 0;
tsk->outdex = 0;
- memset(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
+ memset_io(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
/* Configure some stuff */
bcom_set_task_pragma(tsk->tasknum, BCOM_FEC_RX_BD_PRAGMA);
tsk->index = 0;
tsk->outdex = 0;
- memset(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
+ memset_io(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
/* Configure some stuff */
bcom_set_task_pragma(tsk->tasknum, BCOM_FEC_TX_BD_PRAGMA);
tsk->index = 0;
tsk->outdex = 0;
- memset(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
+ memset_io(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
/* Configure some stuff */
bcom_set_task_pragma(tsk->tasknum, BCOM_GEN_RX_BD_PRAGMA);
tsk->index = 0;
tsk->outdex = 0;
- memset(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
+ memset_io(tsk->bd, 0x00, tsk->num_bd * tsk->bd_size);
/* Configure some stuff */
bcom_set_task_pragma(tsk->tasknum, BCOM_GEN_TX_BD_PRAGMA);
dd->dst_addr_widths = JZ_DMA_BUSWIDTHS;
dd->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
dd->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
+ dd->max_sg_burst = JZ_DMA_MAX_DESC;
/*
* Enable DMA controller, mark all channels as not programmable.
*/
struct dma_chan *dma_get_slave_channel(struct dma_chan *chan)
{
- int err = -EBUSY;
-
/* lock against __dma_request_channel */
mutex_lock(&dma_list_mutex);
if (chan->client_count == 0) {
struct dma_device *device = chan->device;
+ int err;
dma_cap_set(DMA_PRIVATE, device->cap_mask);
device->privatecnt++;
static inline bool
dmaengine_desc_callback_valid(struct dmaengine_desc_callback *cb)
{
- return (cb->callback) ? true : false;
+ return cb->callback || cb->callback_result;
}
struct dma_chan *dma_get_slave_channel(struct dma_chan *chan);
iowrite32(upper_32_bits(val), chan->chan_regs + reg + 4);
}
+static inline void axi_chan_config_write(struct axi_dma_chan *chan,
+ struct axi_dma_chan_config *config)
+{
+ u32 cfg_lo, cfg_hi;
+
+ cfg_lo = (config->dst_multblk_type << CH_CFG_L_DST_MULTBLK_TYPE_POS |
+ config->src_multblk_type << CH_CFG_L_SRC_MULTBLK_TYPE_POS);
+ if (chan->chip->dw->hdata->reg_map_8_channels) {
+ cfg_hi = config->tt_fc << CH_CFG_H_TT_FC_POS |
+ config->hs_sel_src << CH_CFG_H_HS_SEL_SRC_POS |
+ config->hs_sel_dst << CH_CFG_H_HS_SEL_DST_POS |
+ config->src_per << CH_CFG_H_SRC_PER_POS |
+ config->dst_per << CH_CFG_H_DST_PER_POS |
+ config->prior << CH_CFG_H_PRIORITY_POS;
+ } else {
+ cfg_lo |= config->src_per << CH_CFG2_L_SRC_PER_POS |
+ config->dst_per << CH_CFG2_L_DST_PER_POS;
+ cfg_hi = config->tt_fc << CH_CFG2_H_TT_FC_POS |
+ config->hs_sel_src << CH_CFG2_H_HS_SEL_SRC_POS |
+ config->hs_sel_dst << CH_CFG2_H_HS_SEL_DST_POS |
+ config->prior << CH_CFG2_H_PRIORITY_POS;
+ }
+ axi_chan_iowrite32(chan, CH_CFG_L, cfg_lo);
+ axi_chan_iowrite32(chan, CH_CFG_H, cfg_hi);
+}
+
static inline void axi_dma_disable(struct axi_dma_chip *chip)
{
u32 val;
val = axi_dma_ioread32(chan->chip, DMAC_CHEN);
val &= ~(BIT(chan->id) << DMAC_CHAN_EN_SHIFT);
- val |= BIT(chan->id) << DMAC_CHAN_EN_WE_SHIFT;
+ if (chan->chip->dw->hdata->reg_map_8_channels)
+ val |= BIT(chan->id) << DMAC_CHAN_EN_WE_SHIFT;
+ else
+ val |= BIT(chan->id) << DMAC_CHAN_EN2_WE_SHIFT;
axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
}
u32 val;
val = axi_dma_ioread32(chan->chip, DMAC_CHEN);
- val |= BIT(chan->id) << DMAC_CHAN_EN_SHIFT |
- BIT(chan->id) << DMAC_CHAN_EN_WE_SHIFT;
+ if (chan->chip->dw->hdata->reg_map_8_channels)
+ val |= BIT(chan->id) << DMAC_CHAN_EN_SHIFT |
+ BIT(chan->id) << DMAC_CHAN_EN_WE_SHIFT;
+ else
+ val |= BIT(chan->id) << DMAC_CHAN_EN_SHIFT |
+ BIT(chan->id) << DMAC_CHAN_EN2_WE_SHIFT;
axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
}
static void axi_dma_hw_init(struct axi_dma_chip *chip)
{
+ int ret;
u32 i;
for (i = 0; i < chip->dw->hdata->nr_channels; i++) {
axi_chan_irq_disable(&chip->dw->chan[i], DWAXIDMAC_IRQ_ALL);
axi_chan_disable(&chip->dw->chan[i]);
}
+ ret = dma_set_mask_and_coherent(chip->dev, DMA_BIT_MASK(64));
+ if (ret)
+ dev_warn(chip->dev, "Unable to set coherent mask\n");
}
static u32 axi_chan_get_xfer_width(struct axi_dma_chan *chan, dma_addr_t src,
struct axi_dma_desc *first)
{
u32 priority = chan->chip->dw->hdata->priority[chan->id];
- u32 reg, irq_mask;
+ struct axi_dma_chan_config config;
+ u32 irq_mask;
u8 lms = 0; /* Select AXI0 master for LLI fetching */
if (unlikely(axi_chan_is_hw_enable(chan))) {
axi_dma_enable(chan->chip);
- reg = (DWAXIDMAC_MBLK_TYPE_LL << CH_CFG_L_DST_MULTBLK_TYPE_POS |
- DWAXIDMAC_MBLK_TYPE_LL << CH_CFG_L_SRC_MULTBLK_TYPE_POS);
- axi_chan_iowrite32(chan, CH_CFG_L, reg);
-
- reg = (DWAXIDMAC_TT_FC_MEM_TO_MEM_DMAC << CH_CFG_H_TT_FC_POS |
- priority << CH_CFG_H_PRIORITY_POS |
- DWAXIDMAC_HS_SEL_HW << CH_CFG_H_HS_SEL_DST_POS |
- DWAXIDMAC_HS_SEL_HW << CH_CFG_H_HS_SEL_SRC_POS);
+ config.dst_multblk_type = DWAXIDMAC_MBLK_TYPE_LL;
+ config.src_multblk_type = DWAXIDMAC_MBLK_TYPE_LL;
+ config.tt_fc = DWAXIDMAC_TT_FC_MEM_TO_MEM_DMAC;
+ config.prior = priority;
+ config.hs_sel_dst = DWAXIDMAC_HS_SEL_HW;
+ config.hs_sel_dst = DWAXIDMAC_HS_SEL_HW;
switch (chan->direction) {
case DMA_MEM_TO_DEV:
dw_axi_dma_set_byte_halfword(chan, true);
- reg |= (chan->config.device_fc ?
- DWAXIDMAC_TT_FC_MEM_TO_PER_DST :
- DWAXIDMAC_TT_FC_MEM_TO_PER_DMAC)
- << CH_CFG_H_TT_FC_POS;
+ config.tt_fc = chan->config.device_fc ?
+ DWAXIDMAC_TT_FC_MEM_TO_PER_DST :
+ DWAXIDMAC_TT_FC_MEM_TO_PER_DMAC;
if (chan->chip->apb_regs)
- reg |= (chan->id << CH_CFG_H_DST_PER_POS);
+ config.dst_per = chan->id;
+ else
+ config.dst_per = chan->hw_handshake_num;
break;
case DMA_DEV_TO_MEM:
- reg |= (chan->config.device_fc ?
- DWAXIDMAC_TT_FC_PER_TO_MEM_SRC :
- DWAXIDMAC_TT_FC_PER_TO_MEM_DMAC)
- << CH_CFG_H_TT_FC_POS;
+ config.tt_fc = chan->config.device_fc ?
+ DWAXIDMAC_TT_FC_PER_TO_MEM_SRC :
+ DWAXIDMAC_TT_FC_PER_TO_MEM_DMAC;
if (chan->chip->apb_regs)
- reg |= (chan->id << CH_CFG_H_SRC_PER_POS);
+ config.src_per = chan->id;
+ else
+ config.src_per = chan->hw_handshake_num;
break;
default:
break;
}
- axi_chan_iowrite32(chan, CH_CFG_H, reg);
+ axi_chan_config_write(chan, &config);
write_chan_llp(chan, first->hw_desc[0].llp | lms);
spin_lock_irqsave(&chan->vc.lock, flags);
- val = axi_dma_ioread32(chan->chip, DMAC_CHEN);
- val |= BIT(chan->id) << DMAC_CHAN_SUSP_SHIFT |
- BIT(chan->id) << DMAC_CHAN_SUSP_WE_SHIFT;
- axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
+ if (chan->chip->dw->hdata->reg_map_8_channels) {
+ val = axi_dma_ioread32(chan->chip, DMAC_CHEN);
+ val |= BIT(chan->id) << DMAC_CHAN_SUSP_SHIFT |
+ BIT(chan->id) << DMAC_CHAN_SUSP_WE_SHIFT;
+ axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
+ } else {
+ val = BIT(chan->id) << DMAC_CHAN_SUSP2_SHIFT |
+ BIT(chan->id) << DMAC_CHAN_SUSP2_WE_SHIFT;
+ axi_dma_iowrite32(chan->chip, DMAC_CHSUSPREG, val);
+ }
do {
if (axi_chan_irq_read(chan) & DWAXIDMAC_IRQ_SUSPENDED)
u32 val;
val = axi_dma_ioread32(chan->chip, DMAC_CHEN);
- val &= ~(BIT(chan->id) << DMAC_CHAN_SUSP_SHIFT);
- val |= (BIT(chan->id) << DMAC_CHAN_SUSP_WE_SHIFT);
- axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
+ if (chan->chip->dw->hdata->reg_map_8_channels) {
+ val &= ~(BIT(chan->id) << DMAC_CHAN_SUSP_SHIFT);
+ val |= (BIT(chan->id) << DMAC_CHAN_SUSP_WE_SHIFT);
+ axi_dma_iowrite32(chan->chip, DMAC_CHEN, val);
+ } else {
+ val &= ~(BIT(chan->id) << DMAC_CHAN_SUSP2_SHIFT);
+ val |= (BIT(chan->id) << DMAC_CHAN_SUSP2_WE_SHIFT);
+ axi_dma_iowrite32(chan->chip, DMAC_CHSUSPREG, val);
+ }
chan->is_paused = false;
}
return -EINVAL;
chip->dw->hdata->nr_channels = tmp;
+ if (tmp <= DMA_REG_MAP_CH_REF)
+ chip->dw->hdata->reg_map_8_channels = true;
ret = device_property_read_u32(dev, "snps,dma-masters", &tmp);
if (ret)
#include "../virt-dma.h"
-#define DMAC_MAX_CHANNELS 8
+#define DMAC_MAX_CHANNELS 16
#define DMAC_MAX_MASTERS 2
#define DMAC_MAX_BLK_SIZE 0x200000
u32 priority[DMAC_MAX_CHANNELS];
/* maximum supported axi burst length */
u32 axi_rw_burst_len;
+ /* Register map for DMAX_NUM_CHANNELS <= 8 */
+ bool reg_map_8_channels;
bool restrict_axi_burst_len;
};
u32 period_len;
};
+struct axi_dma_chan_config {
+ u8 dst_multblk_type;
+ u8 src_multblk_type;
+ u8 dst_per;
+ u8 src_per;
+ u8 tt_fc;
+ u8 prior;
+ u8 hs_sel_dst;
+ u8 hs_sel_src;
+};
+
static inline struct device *dchan2dev(struct dma_chan *dchan)
{
return &dchan->dev->device;
#define DMAC_CHEN 0x018 /* R/W DMAC Channel Enable */
#define DMAC_CHEN_L 0x018 /* R/W DMAC Channel Enable 00-31 */
#define DMAC_CHEN_H 0x01C /* R/W DMAC Channel Enable 32-63 */
+#define DMAC_CHSUSPREG 0x020 /* R/W DMAC Channel Suspend */
+#define DMAC_CHABORTREG 0x028 /* R/W DMAC Channel Abort */
#define DMAC_INTSTATUS 0x030 /* R DMAC Interrupt Status */
#define DMAC_COMMON_INTCLEAR 0x038 /* W DMAC Interrupt Clear */
#define DMAC_COMMON_INTSTATUS_ENA 0x040 /* R DMAC Interrupt Status Enable */
#define DMA_APB_HS_SEL_BIT_SIZE 0x08 /* HW handshake bits per channel */
#define DMA_APB_HS_SEL_MASK 0xFF /* HW handshake select masks */
#define MAX_BLOCK_SIZE 0x1000 /* 1024 blocks * 4 bytes data width */
+#define DMA_REG_MAP_CH_REF 0x08 /* Channel count to choose register map */
/* DMAC_CFG */
#define DMAC_EN_POS 0
#define INT_EN_POS 1
#define INT_EN_MASK BIT(INT_EN_POS)
+/* DMAC_CHEN */
#define DMAC_CHAN_EN_SHIFT 0
#define DMAC_CHAN_EN_WE_SHIFT 8
#define DMAC_CHAN_SUSP_SHIFT 16
#define DMAC_CHAN_SUSP_WE_SHIFT 24
+/* DMAC_CHEN2 */
+#define DMAC_CHAN_EN2_WE_SHIFT 16
+
+/* DMAC_CHSUSP */
+#define DMAC_CHAN_SUSP2_SHIFT 0
+#define DMAC_CHAN_SUSP2_WE_SHIFT 16
+
/* CH_CTL_H */
#define CH_CTL_H_ARLEN_EN BIT(6)
#define CH_CTL_H_ARLEN_POS 7
DWAXIDMAC_MBLK_TYPE_LL
};
+/* CH_CFG2 */
+#define CH_CFG2_L_SRC_PER_POS 4
+#define CH_CFG2_L_DST_PER_POS 11
+
+#define CH_CFG2_H_TT_FC_POS 0
+#define CH_CFG2_H_HS_SEL_SRC_POS 3
+#define CH_CFG2_H_HS_SEL_DST_POS 4
+#define CH_CFG2_H_PRIORITY_POS 20
+
/**
* DW AXI DMA channel interrupts
*
{
struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
int err = 0;
- LIST_HEAD(head);
if (!chan->configured) {
/* Do nothing */
pci_set_master(pdev);
/* DMA configuration */
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
if (!err) {
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
- if (err) {
- pci_err(pdev, "consistent DMA mask 64 set failed\n");
- return err;
- }
+ pci_err(pdev, "DMA mask 64 set failed\n");
+ return err;
} else {
pci_err(pdev, "DMA mask 64 set failed\n");
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (err) {
pci_err(pdev, "DMA mask 32 set failed\n");
return err;
}
-
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
- if (err) {
- pci_err(pdev, "consistent DMA mask 32 set failed\n");
- return err;
- }
}
/* Data structure allocation */
pci_set_master(pdev);
pci_try_set_mwi(pdev);
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (ret)
- return ret;
-
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (ret)
return ret;
struct fsl_edma_engine *edma = fsl_chan->edma;
struct edma_regs *regs = &fsl_chan->edma->regs;
u32 ch = fsl_chan->vchan.chan.chan_id;
+ u16 csr = 0;
/*
* TCD parameters are stored in struct fsl_edma_hw_tcd in little
edma_writel(edma, (s32)tcd->dlast_sga,
®s->tcd[ch].dlast_sga);
+ if (fsl_chan->is_sw) {
+ csr = le16_to_cpu(tcd->csr);
+ csr |= EDMA_TCD_CSR_START;
+ tcd->csr = cpu_to_le16(csr);
+ }
+
edma_writew(edma, (s16)tcd->csr, ®s->tcd[ch].csr);
}
}
EXPORT_SYMBOL_GPL(fsl_edma_prep_slave_sg);
+struct dma_async_tx_descriptor *fsl_edma_prep_memcpy(struct dma_chan *chan,
+ dma_addr_t dma_dst, dma_addr_t dma_src,
+ size_t len, unsigned long flags)
+{
+ struct fsl_edma_chan *fsl_chan = to_fsl_edma_chan(chan);
+ struct fsl_edma_desc *fsl_desc;
+
+ fsl_desc = fsl_edma_alloc_desc(fsl_chan, 1);
+ if (!fsl_desc)
+ return NULL;
+ fsl_desc->iscyclic = false;
+
+ fsl_chan->is_sw = true;
+
+ /* To match with copy_align and max_seg_size so 1 tcd is enough */
+ fsl_edma_fill_tcd(fsl_desc->tcd[0].vtcd, dma_src, dma_dst,
+ EDMA_TCD_ATTR_SSIZE_32BYTE | EDMA_TCD_ATTR_DSIZE_32BYTE,
+ 32, len, 0, 1, 1, 32, 0, true, true, false);
+
+ return vchan_tx_prep(&fsl_chan->vchan, &fsl_desc->vdesc, flags);
+}
+EXPORT_SYMBOL_GPL(fsl_edma_prep_memcpy);
+
void fsl_edma_xfer_desc(struct fsl_edma_chan *fsl_chan)
{
struct virt_dma_desc *vdesc;
void fsl_edma_free_chan_resources(struct dma_chan *chan)
{
struct fsl_edma_chan *fsl_chan = to_fsl_edma_chan(chan);
+ struct fsl_edma_engine *edma = fsl_chan->edma;
unsigned long flags;
LIST_HEAD(head);
spin_lock_irqsave(&fsl_chan->vchan.lock, flags);
fsl_edma_disable_request(fsl_chan);
- fsl_edma_chan_mux(fsl_chan, 0, false);
+ if (edma->drvdata->dmamuxs)
+ fsl_edma_chan_mux(fsl_chan, 0, false);
fsl_chan->edesc = NULL;
vchan_get_all_descriptors(&fsl_chan->vchan, &head);
fsl_edma_unprep_slave_dma(fsl_chan);
vchan_dma_desc_free_list(&fsl_chan->vchan, &head);
dma_pool_destroy(fsl_chan->tcd_pool);
fsl_chan->tcd_pool = NULL;
+ fsl_chan->is_sw = false;
}
EXPORT_SYMBOL_GPL(fsl_edma_free_chan_resources);
struct fsl_edma_desc *edesc;
struct dma_slave_config cfg;
u32 attr;
+ bool is_sw;
struct dma_pool *tcd_pool;
dma_addr_t dma_dev_addr;
u32 dma_dev_size;
struct dma_chan *chan, struct scatterlist *sgl,
unsigned int sg_len, enum dma_transfer_direction direction,
unsigned long flags, void *context);
+struct dma_async_tx_descriptor *fsl_edma_prep_memcpy(
+ struct dma_chan *chan, dma_addr_t dma_dst, dma_addr_t dma_src,
+ size_t len, unsigned long flags);
void fsl_edma_xfer_desc(struct fsl_edma_chan *fsl_chan);
void fsl_edma_issue_pending(struct dma_chan *chan);
int fsl_edma_alloc_chan_resources(struct dma_chan *chan);
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/of_dma.h>
+#include <linux/dma-mapping.h>
#include "fsl-edma-common.h"
dma_cap_set(DMA_PRIVATE, fsl_edma->dma_dev.cap_mask);
dma_cap_set(DMA_SLAVE, fsl_edma->dma_dev.cap_mask);
dma_cap_set(DMA_CYCLIC, fsl_edma->dma_dev.cap_mask);
+ dma_cap_set(DMA_MEMCPY, fsl_edma->dma_dev.cap_mask);
fsl_edma->dma_dev.dev = &pdev->dev;
fsl_edma->dma_dev.device_alloc_chan_resources
fsl_edma->dma_dev.device_tx_status = fsl_edma_tx_status;
fsl_edma->dma_dev.device_prep_slave_sg = fsl_edma_prep_slave_sg;
fsl_edma->dma_dev.device_prep_dma_cyclic = fsl_edma_prep_dma_cyclic;
+ fsl_edma->dma_dev.device_prep_dma_memcpy = fsl_edma_prep_memcpy;
fsl_edma->dma_dev.device_config = fsl_edma_slave_config;
fsl_edma->dma_dev.device_pause = fsl_edma_pause;
fsl_edma->dma_dev.device_resume = fsl_edma_resume;
fsl_edma->dma_dev.dst_addr_widths = FSL_EDMA_BUSWIDTHS;
fsl_edma->dma_dev.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
+ fsl_edma->dma_dev.copy_align = DMAENGINE_ALIGN_32_BYTES;
+ /* Per worst case 'nbytes = 1' take CITER as the max_seg_size */
+ dma_set_max_seg_size(fsl_edma->dma_dev.dev, 0x3fff);
+
platform_set_drvdata(pdev, fsl_edma);
ret = dma_async_device_register(&fsl_edma->dma_dev);
return ret;
}
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
- if (ret)
- return ret;
-
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
if (ret)
return ret;
pci_set_master(pdev);
pci_try_set_mwi(pdev);
- ret = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (ret)
- return ret;
-
- ret = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (ret)
return ret;
struct idxd_device *idxd = wq->idxd;
struct device *dev = &idxd->pdev->dev;
int rc, num_descs, i;
- int align;
- u64 tmp;
if (wq->type != IDXD_WQT_KERNEL)
return 0;
if (rc < 0)
return rc;
- align = idxd->data->align;
- wq->compls_size = num_descs * idxd->data->compl_size + align;
- wq->compls_raw = dma_alloc_coherent(dev, wq->compls_size,
- &wq->compls_addr_raw, GFP_KERNEL);
- if (!wq->compls_raw) {
+ wq->compls_size = num_descs * idxd->data->compl_size;
+ wq->compls = dma_alloc_coherent(dev, wq->compls_size, &wq->compls_addr, GFP_KERNEL);
+ if (!wq->compls) {
rc = -ENOMEM;
goto fail_alloc_compls;
}
- /* Adjust alignment */
- wq->compls_addr = (wq->compls_addr_raw + (align - 1)) & ~(align - 1);
- tmp = (u64)wq->compls_raw;
- tmp = (tmp + (align - 1)) & ~(align - 1);
- wq->compls = (struct dsa_completion_record *)tmp;
-
rc = alloc_descs(wq, num_descs);
if (rc < 0)
goto fail_alloc_descs;
fail_sbitmap_init:
free_descs(wq);
fail_alloc_descs:
- dma_free_coherent(dev, wq->compls_size, wq->compls_raw,
- wq->compls_addr_raw);
+ dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr);
fail_alloc_compls:
free_hw_descs(wq);
return rc;
free_hw_descs(wq);
free_descs(wq);
- dma_free_coherent(dev, wq->compls_size, wq->compls_raw,
- wq->compls_addr_raw);
+ dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr);
sbitmap_queue_free(&wq->sbq);
}
{
percpu_ref_kill(&wq->wq_active);
wait_for_completion(&wq->wq_dead);
- percpu_ref_exit(&wq->wq_active);
}
/* Device control bits */
spin_lock(&idxd->dev_lock);
idxd_device_clear_state(idxd);
idxd->state = IDXD_DEV_DISABLED;
+ idxd_unmask_error_interrupts(idxd);
+ idxd_msix_perm_setup(idxd);
spin_unlock(&idxd->dev_lock);
}
struct device *dev = &idxd->pdev->dev;
/* Setup bandwidth token limit */
- if (idxd->token_limit) {
+ if (idxd->hw.gen_cap.config_en && idxd->token_limit) {
reg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET);
reg.token_limit = idxd->token_limit;
iowrite32(reg.bits, idxd->reg_base + IDXD_GENCFG_OFFSET);
wq->size = wq->wqcfg->wq_size;
wq->threshold = wq->wqcfg->wq_thresh;
- if (wq->wqcfg->priv)
- wq->type = IDXD_WQT_KERNEL;
/* The driver does not support shared WQ mode in read-only config yet */
if (wq->wqcfg->mode == 0 || wq->wqcfg->pasid_en)
err_dma:
idxd_wq_quiesce(wq);
+ percpu_ref_exit(&wq->wq_active);
err_ref:
idxd_wq_free_resources(wq);
err_res_alloc:
mutex_lock(&wq->wq_lock);
idxd_wq_quiesce(wq);
idxd_unregister_dma_channel(wq);
- __drv_disable_wq(wq);
idxd_wq_free_resources(wq);
- wq->type = IDXD_WQT_NONE;
+ __drv_disable_wq(wq);
+ percpu_ref_exit(&wq->wq_active);
mutex_unlock(&wq->wq_lock);
}
struct dsa_completion_record *compls;
struct iax_completion_record *iax_compls;
};
- void *compls_raw;
dma_addr_t compls_addr;
- dma_addr_t compls_addr_raw;
int compls_size;
struct idxd_desc **descs;
struct sbitmap_queue sbq;
int msixcnt = pci_msix_vec_count(pdev);
int i;
- dev_dbg(&pdev->dev, "%s called\n", __func__);
+ idxd_unregister_devices(idxd);
+ /*
+ * When ->release() is called for the idxd->conf_dev, it frees all the memory related
+ * to the idxd context. The driver still needs those bits in order to do the rest of
+ * the cleanup. However, we do need to unbound the idxd sub-driver. So take a ref
+ * on the device here to hold off the freeing while allowing the idxd sub-driver
+ * to unbind.
+ */
+ get_device(idxd_confdev(idxd));
+ device_unregister(idxd_confdev(idxd));
idxd_shutdown(pdev);
if (device_pasid_enabled(idxd))
idxd_disable_system_pasid(idxd);
- idxd_unregister_devices(idxd);
for (i = 0; i < msixcnt; i++) {
irq_entry = &idxd->irq_entries[i];
pci_disable_device(pdev);
destroy_workqueue(idxd->wq);
perfmon_pmu_remove(idxd);
- device_unregister(idxd_confdev(idxd));
+ put_device(idxd_confdev(idxd));
}
static struct pci_driver idxd_pci_driver = {
int i;
bool err = false;
+ if (cause & IDXD_INTC_HALT_STATE)
+ goto halt;
+
if (cause & IDXD_INTC_ERR) {
spin_lock(&idxd->dev_lock);
for (i = 0; i < 4; i++)
if (!err)
return 0;
+halt:
gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
if (gensts.state == IDXD_DEVICE_STATE_HALT) {
idxd->state = IDXD_DEV_HALTED;
queue_work(idxd->wq, &idxd->work);
} else {
spin_lock(&idxd->dev_lock);
+ idxd->state = IDXD_DEV_HALTED;
idxd_wqs_quiesce(idxd);
idxd_wqs_unmap_portal(idxd);
idxd_device_clear_state(idxd);
list_for_each_entry_safe(desc, n, &irq_entry->work_list, list) {
if (desc->completion->status) {
- list_del(&desc->list);
- list_add_tail(&desc->list, &flist);
+ list_move_tail(&desc->list, &flist);
}
}
u64 max_batch_shift:4;
u64 max_ims_mult:6;
u64 config_en:1;
- u64 max_descs_per_engine:8;
- u64 rsvd3:24;
+ u64 rsvd3:32;
};
u64 bits;
} __packed;
#define IDXD_INTC_CMD 0x02
#define IDXD_INTC_OCCUPY 0x04
#define IDXD_INTC_PERFMON_OVFL 0x08
+#define IDXD_INTC_HALT_STATE 0x10
#define IDXD_CMD_OFFSET 0xa0
union idxd_command_reg {
unsigned long flags;
buf_virt = dma_alloc_coherent(sdma->dev, size, &buf_phys, GFP_KERNEL);
- if (!buf_virt) {
+ if (!buf_virt)
return -ENOMEM;
- }
spin_lock_irqsave(&sdma->channel_0_lock, flags);
if (sdmac->peripheral_type == IMX_DMATYPE_ASRC_SP ||
sdmac->peripheral_type == IMX_DMATYPE_ASRC)
sdma_set_watermarklevel_for_p2p(sdmac);
- } else
+ } else {
__set_bit(sdmac->event_id0, sdmac->event_mask);
+ }
/* Address */
sdmac->shp_addr = sdmac->per_address;
}
static int sdma_set_channel_priority(struct sdma_channel *sdmac,
- unsigned int priority)
+ unsigned int priority)
{
struct sdma_engine *sdma = sdmac->sdma;
int channel = sdmac->channel;
int ret = -EBUSY;
sdma->bd0 = dma_alloc_coherent(sdma->dev, PAGE_SIZE, &sdma->bd0_phys,
- GFP_NOWAIT);
+ GFP_NOWAIT);
if (!sdma->bd0) {
ret = -ENOMEM;
goto out;
int ret = 0;
desc->bd = dma_alloc_coherent(desc->sdmac->sdma->dev, bd_size,
- &desc->bd_phys, GFP_NOWAIT);
+ &desc->bd_phys, GFP_NOWAIT);
if (!desc->bd) {
ret = -ENOMEM;
goto out;
#define SDMA_SCRIPT_ADDRS_ARRAY_SIZE_V4 46
static void sdma_add_scripts(struct sdma_engine *sdma,
- const struct sdma_script_start_addrs *addr)
+ const struct sdma_script_start_addrs *addr)
{
s32 *addr_arr = (u32 *)addr;
s32 *saddr_arr = (u32 *)sdma->script_addrs;
clk_enable(sdma->clk_ahb);
/* download the RAM image for SDMA */
sdma_load_script(sdma, ram_code,
- header->ram_code_size,
- addr->ram_code_start_addr);
+ header->ram_code_size,
+ addr->ram_code_start_addr);
clk_disable(sdma->clk_ipg);
clk_disable(sdma->clk_ahb);
sdma->fw_loaded = true;
dev_info(sdma->dev, "loaded firmware %d.%d\n",
- header->version_major,
- header->version_minor);
+ header->version_major,
+ header->version_minor);
err_firmware:
release_firmware(fw);
writel_relaxed(0, sdma->regs + SDMA_H_C0PTR);
sdma->channel_control = dma_alloc_coherent(sdma->dev,
- MAX_DMA_CHANNELS * sizeof (struct sdma_channel_control) +
+ MAX_DMA_CHANNELS * sizeof(struct sdma_channel_control) +
sizeof(struct sdma_context_data),
&ccb_phys, GFP_KERNEL);
}
sdma->context = (void *)sdma->channel_control +
- MAX_DMA_CHANNELS * sizeof (struct sdma_channel_control);
+ MAX_DMA_CHANNELS * sizeof(struct sdma_channel_control);
sdma->context_phys = ccb_phys +
- MAX_DMA_CHANNELS * sizeof (struct sdma_channel_control);
+ MAX_DMA_CHANNELS * sizeof(struct sdma_channel_control);
/* disable all channels */
for (i = 0; i < sdma->drvdata->num_events; i++)
if (!iomap)
return -ENOMEM;
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
if (err)
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (err)
- return err;
-
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
- if (err)
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (err)
return err;
if (!md)
return NULL;
- md->sgl = kzalloc(sizeof(*sgl) * sg_len, GFP_NOWAIT);
+ md->sgl = kcalloc(sg_len, sizeof(*sgl), GFP_NOWAIT);
if (!md->sgl) {
kfree(md);
return NULL;
mmp_pdma_dma_xlate, pdev);
if (ret < 0) {
dev_err(&op->dev, "of_dma_controller_register failed\n");
+ dma_async_device_unregister(&pdev->device);
return ret;
}
}
if (rc)
return rc;
- rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(48));
+ rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(48));
if (rc)
- rc = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
- if (rc)
- return rc;
-
- rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(48));
- if (rc)
- rc = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
+ rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
if (rc)
return rc;
/* execution environment ID, from DT */
u32 ee;
bool controlled_remotely;
+ bool powered_remotely;
+ u32 active_channels;
const struct reg_offset_data *layout;
}
/**
+ * bam_reset() - reset and initialize BAM registers
+ * @bdev: bam device
+ */
+static void bam_reset(struct bam_device *bdev)
+{
+ u32 val;
+
+ /* s/w reset bam */
+ /* after reset all pipes are disabled and idle */
+ val = readl_relaxed(bam_addr(bdev, 0, BAM_CTRL));
+ val |= BAM_SW_RST;
+ writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
+ val &= ~BAM_SW_RST;
+ writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
+
+ /* make sure previous stores are visible before enabling BAM */
+ wmb();
+
+ /* enable bam */
+ val |= BAM_EN;
+ writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
+
+ /* set descriptor threshhold, start with 4 bytes */
+ writel_relaxed(DEFAULT_CNT_THRSHLD,
+ bam_addr(bdev, 0, BAM_DESC_CNT_TRSHLD));
+
+ /* Enable default set of h/w workarounds, ie all except BAM_FULL_PIPE */
+ writel_relaxed(BAM_CNFG_BITS_DEFAULT, bam_addr(bdev, 0, BAM_CNFG_BITS));
+
+ /* enable irqs for errors */
+ writel_relaxed(BAM_ERROR_EN | BAM_HRESP_ERR_EN,
+ bam_addr(bdev, 0, BAM_IRQ_EN));
+
+ /* unmask global bam interrupt */
+ writel_relaxed(BAM_IRQ_MSK, bam_addr(bdev, 0, BAM_IRQ_SRCS_MSK_EE));
+}
+
+/**
* bam_reset_channel - Reset individual BAM DMA channel
* @bchan: bam channel
*
return -ENOMEM;
}
+ if (bdev->active_channels++ == 0 && bdev->powered_remotely)
+ bam_reset(bdev);
+
return 0;
}
/* disable irq */
writel_relaxed(0, bam_addr(bdev, bchan->id, BAM_P_IRQ_EN));
+ if (--bdev->active_channels == 0 && bdev->powered_remotely) {
+ /* s/w reset bam */
+ val = readl_relaxed(bam_addr(bdev, 0, BAM_CTRL));
+ val |= BAM_SW_RST;
+ writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
+ }
+
err:
pm_runtime_mark_last_busy(bdev->dev);
pm_runtime_put_autosuspend(bdev->dev);
bdev->num_channels = val & BAM_NUM_PIPES_MASK;
}
- if (bdev->controlled_remotely)
- return 0;
-
- /* s/w reset bam */
- /* after reset all pipes are disabled and idle */
- val = readl_relaxed(bam_addr(bdev, 0, BAM_CTRL));
- val |= BAM_SW_RST;
- writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
- val &= ~BAM_SW_RST;
- writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
-
- /* make sure previous stores are visible before enabling BAM */
- wmb();
-
- /* enable bam */
- val |= BAM_EN;
- writel_relaxed(val, bam_addr(bdev, 0, BAM_CTRL));
-
- /* set descriptor threshhold, start with 4 bytes */
- writel_relaxed(DEFAULT_CNT_THRSHLD,
- bam_addr(bdev, 0, BAM_DESC_CNT_TRSHLD));
-
- /* Enable default set of h/w workarounds, ie all except BAM_FULL_PIPE */
- writel_relaxed(BAM_CNFG_BITS_DEFAULT, bam_addr(bdev, 0, BAM_CNFG_BITS));
-
- /* enable irqs for errors */
- writel_relaxed(BAM_ERROR_EN | BAM_HRESP_ERR_EN,
- bam_addr(bdev, 0, BAM_IRQ_EN));
-
- /* unmask global bam interrupt */
- writel_relaxed(BAM_IRQ_MSK, bam_addr(bdev, 0, BAM_IRQ_SRCS_MSK_EE));
+ /* Reset BAM now if fully controlled locally */
+ if (!bdev->controlled_remotely && !bdev->powered_remotely)
+ bam_reset(bdev);
return 0;
}
bdev->controlled_remotely = of_property_read_bool(pdev->dev.of_node,
"qcom,controlled-remotely");
+ bdev->powered_remotely = of_property_read_bool(pdev->dev.of_node,
+ "qcom,powered-remotely");
- if (bdev->controlled_remotely) {
+ if (bdev->controlled_remotely || bdev->powered_remotely) {
ret = of_property_read_u32(pdev->dev.of_node, "num-channels",
&bdev->num_channels);
if (ret)
dev_err(bdev->dev, "num-ees unspecified in dt\n");
}
- if (bdev->controlled_remotely)
+ if (bdev->controlled_remotely || bdev->powered_remotely)
bdev->bamclk = devm_clk_get_optional(bdev->dev, "bam_clk");
else
bdev->bamclk = devm_clk_get(bdev->dev, "bam_clk");
return 0;
}
-static int sa11x0_dma_suspend(struct device *dev)
+static __maybe_unused int sa11x0_dma_suspend(struct device *dev)
{
struct sa11x0_dma_dev *d = dev_get_drvdata(dev);
unsigned pch;
return 0;
}
-static int sa11x0_dma_resume(struct device *dev)
+static __maybe_unused int sa11x0_dma_resume(struct device *dev)
{
struct sa11x0_dma_dev *d = dev_get_drvdata(dev);
unsigned pch;
}
static const struct dev_pm_ops sa11x0_dma_pm_ops = {
- .suspend_noirq = sa11x0_dma_suspend,
- .resume_noirq = sa11x0_dma_resume,
- .freeze_noirq = sa11x0_dma_suspend,
- .thaw_noirq = sa11x0_dma_resume,
- .poweroff_noirq = sa11x0_dma_suspend,
- .restore_noirq = sa11x0_dma_resume,
+ SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(sa11x0_dma_suspend, sa11x0_dma_resume)
};
static struct platform_driver sa11x0_dma_driver = {
ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0) {
dev_err(&pdev->dev, "runtime PM get sync failed (%d)\n", ret);
- return ret;
+ goto err_pm_disable;
}
ret = rcar_dmac_init(dmac);
if (ret) {
dev_err(&pdev->dev, "failed to reset device\n");
- goto error;
+ goto err_pm_disable;
}
/* Initialize engine */
for_each_rcar_dmac_chan(i, dmac, chan) {
ret = rcar_dmac_chan_probe(dmac, chan);
if (ret < 0)
- goto error;
+ goto err_pm_disable;
}
/* Register the DMAC as a DMA provider for DT. */
ret = of_dma_controller_register(pdev->dev.of_node, rcar_dmac_of_xlate,
NULL);
if (ret < 0)
- goto error;
+ goto err_pm_disable;
/*
* Register the DMA engine device.
*/
ret = dma_async_device_register(engine);
if (ret < 0)
- goto error;
+ goto err_dma_free;
return 0;
-error:
+err_dma_free:
of_dma_controller_free(pdev->dev.of_node);
+err_pm_disable:
pm_runtime_disable(&pdev->dev);
return ret;
}
#include <linux/of_dma.h>
#include <linux/of_platform.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
static u8 rz_dmac_ds_to_val_mapping(enum dma_slave_buswidth ds)
{
u8 i;
- const enum dma_slave_buswidth ds_lut[] = {
+ static const enum dma_slave_buswidth ds_lut[] = {
DMA_SLAVE_BUSWIDTH_1_BYTE,
DMA_SLAVE_BUSWIDTH_2_BYTES,
DMA_SLAVE_BUSWIDTH_4_BYTES,
/* Initialize the channels. */
INIT_LIST_HEAD(&dmac->engine.channels);
+ pm_runtime_enable(&pdev->dev);
+ ret = pm_runtime_resume_and_get(&pdev->dev);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "pm_runtime_resume_and_get failed\n");
+ goto err_pm_disable;
+ }
+
for (i = 0; i < dmac->n_channels; i++) {
ret = rz_dmac_chan_probe(dmac, &dmac->channels[i], i);
if (ret < 0)
channel->lmdesc.base_dma);
}
+ pm_runtime_put(&pdev->dev);
+err_pm_disable:
+ pm_runtime_disable(&pdev->dev);
+
return ret;
}
}
of_dma_controller_free(pdev->dev.of_node);
dma_async_device_unregister(&dmac->engine);
+ pm_runtime_put(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
return 0;
}
u32 threshold)
{
enum dma_slave_buswidth max_width;
- u64 addr = buf_addr;
if (threshold == STM32_DMA_FIFO_THRESHOLD_FULL)
max_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
max_width > DMA_SLAVE_BUSWIDTH_1_BYTE)
max_width = max_width >> 1;
- if (do_div(addr, max_width))
+ if (buf_addr & (max_width - 1))
max_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
return max_width;
spin_lock_irqsave(&chan->vchan.lock, flags);
if (chan->desc) {
+ dma_cookie_complete(&chan->desc->vdesc.tx);
vchan_terminate_vdesc(&chan->desc->vdesc);
if (chan->busy)
stm32_dma_stop(chan);
if (src_bus_width < 0)
return src_bus_width;
- /* Set memory burst size */
- src_maxburst = STM32_DMA_MAX_BURST;
+ /*
+ * Set memory burst size - burst not possible if address is not aligned on
+ * the address boundary equal to the size of the transfer
+ */
+ if (buf_addr & (buf_len - 1))
+ src_maxburst = 1;
+ else
+ src_maxburst = STM32_DMA_MAX_BURST;
src_best_burst = stm32_dma_get_best_burst(buf_len,
src_maxburst,
fifoth,
if (dst_bus_width < 0)
return dst_bus_width;
- /* Set memory burst size */
- dst_maxburst = STM32_DMA_MAX_BURST;
+ /*
+ * Set memory burst size - burst not possible if address is not aligned on
+ * the address boundary equal to the size of the transfer
+ */
+ if (buf_addr & (buf_len - 1))
+ dst_maxburst = 1;
+ else
+ dst_maxburst = STM32_DMA_MAX_BURST;
dst_best_burst = stm32_dma_get_best_burst(buf_len,
dst_maxburst,
fifoth,
if (count < 0)
count = 0;
- dmadev = devm_kzalloc(&pdev->dev, sizeof(*dmadev) + sizeof(u32) * count,
+ dmadev = devm_kzalloc(&pdev->dev,
+ struct_size(dmadev, ahb_addr_masks, count),
GFP_KERNEL);
if (!dmadev)
return -ENOMEM;
#define TEGRA186_ADMA_CH_CONFIG_OUTSTANDING_REQS(reqs) (reqs << 4)
#define ADMA_CH_FIFO_CTRL 0x2c
-#define TEGRA210_ADMA_CH_FIFO_CTRL_TXSIZE(val) (((val) & 0xf) << 8)
-#define TEGRA210_ADMA_CH_FIFO_CTRL_RXSIZE(val) ((val) & 0xf)
-#define TEGRA186_ADMA_CH_FIFO_CTRL_TXSIZE(val) (((val) & 0x1f) << 8)
-#define TEGRA186_ADMA_CH_FIFO_CTRL_RXSIZE(val) ((val) & 0x1f)
+#define ADMA_CH_TX_FIFO_SIZE_SHIFT 8
+#define ADMA_CH_RX_FIFO_SIZE_SHIFT 0
#define ADMA_CH_LOWER_SRC_ADDR 0x34
#define ADMA_CH_LOWER_TRG_ADDR 0x3c
#define TEGRA_ADMA_BURST_COMPLETE_TIME 20
-#define TEGRA210_FIFO_CTRL_DEFAULT (TEGRA210_ADMA_CH_FIFO_CTRL_TXSIZE(3) | \
- TEGRA210_ADMA_CH_FIFO_CTRL_RXSIZE(3))
-
-#define TEGRA186_FIFO_CTRL_DEFAULT (TEGRA186_ADMA_CH_FIFO_CTRL_TXSIZE(3) | \
- TEGRA186_ADMA_CH_FIFO_CTRL_RXSIZE(3))
-
#define ADMA_CH_REG_FIELD_VAL(val, mask, shift) (((val) & mask) << shift)
struct tegra_adma;
/*
* struct tegra_adma_chip_data - Tegra chip specific data
+ * @adma_get_burst_config: Function callback used to set DMA burst size.
* @global_reg_offset: Register offset of DMA global register.
* @global_int_clear: Register offset of DMA global interrupt clear.
* @ch_req_tx_shift: Register offset for AHUB transmit channel select.
* @ch_req_rx_shift: Register offset for AHUB receive channel select.
* @ch_base_offset: Register offset of DMA channel registers.
- * @has_outstanding_reqs: If DMA channel can have outstanding requests.
* @ch_fifo_ctrl: Default value for channel FIFO CTRL register.
* @ch_req_mask: Mask for Tx or Rx channel select.
* @ch_req_max: Maximum number of Tx or Rx channels available.
* @ch_reg_size: Size of DMA channel register space.
* @nr_channels: Number of DMA channels available.
+ * @ch_fifo_size_mask: Mask for FIFO size field.
+ * @sreq_index_offset: Slave channel index offset.
+ * @has_outstanding_reqs: If DMA channel can have outstanding requests.
*/
struct tegra_adma_chip_data {
unsigned int (*adma_get_burst_config)(unsigned int burst_size);
unsigned int ch_req_max;
unsigned int ch_reg_size;
unsigned int nr_channels;
+ unsigned int ch_fifo_size_mask;
+ unsigned int sreq_index_offset;
bool has_outstanding_reqs;
};
{
struct tegra_adma_chan_regs *ch_regs = &desc->ch_regs;
const struct tegra_adma_chip_data *cdata = tdc->tdma->cdata;
- unsigned int burst_size, adma_dir;
+ unsigned int burst_size, adma_dir, fifo_size_shift;
if (desc->num_periods > ADMA_CH_CONFIG_MAX_BUFS)
return -EINVAL;
switch (direction) {
case DMA_MEM_TO_DEV:
+ fifo_size_shift = ADMA_CH_TX_FIFO_SIZE_SHIFT;
adma_dir = ADMA_CH_CTRL_DIR_MEM2AHUB;
burst_size = tdc->sconfig.dst_maxburst;
ch_regs->config = ADMA_CH_CONFIG_SRC_BUF(desc->num_periods - 1);
break;
case DMA_DEV_TO_MEM:
+ fifo_size_shift = ADMA_CH_RX_FIFO_SIZE_SHIFT;
adma_dir = ADMA_CH_CTRL_DIR_AHUB2MEM;
burst_size = tdc->sconfig.src_maxburst;
ch_regs->config = ADMA_CH_CONFIG_TRG_BUF(desc->num_periods - 1);
ch_regs->config |= ADMA_CH_CONFIG_WEIGHT_FOR_WRR(1);
if (cdata->has_outstanding_reqs)
ch_regs->config |= TEGRA186_ADMA_CH_CONFIG_OUTSTANDING_REQS(8);
- ch_regs->fifo_ctrl = cdata->ch_fifo_ctrl;
+
+ /*
+ * 'sreq_index' represents the current ADMAIF channel number and as per
+ * HW recommendation its FIFO size should match with the corresponding
+ * ADMA channel.
+ *
+ * ADMA FIFO size is set as per below (based on default ADMAIF channel
+ * FIFO sizes):
+ * fifo_size = 0x2 (sreq_index > sreq_index_offset)
+ * fifo_size = 0x3 (sreq_index <= sreq_index_offset)
+ *
+ */
+ if (tdc->sreq_index > cdata->sreq_index_offset)
+ ch_regs->fifo_ctrl =
+ ADMA_CH_REG_FIELD_VAL(2, cdata->ch_fifo_size_mask,
+ fifo_size_shift);
+ else
+ ch_regs->fifo_ctrl =
+ ADMA_CH_REG_FIELD_VAL(3, cdata->ch_fifo_size_mask,
+ fifo_size_shift);
+
ch_regs->tc = desc->period_len & ADMA_CH_TC_COUNT_MASK;
return tegra_adma_request_alloc(tdc, direction);
.ch_req_tx_shift = 28,
.ch_req_rx_shift = 24,
.ch_base_offset = 0,
- .has_outstanding_reqs = false,
- .ch_fifo_ctrl = TEGRA210_FIFO_CTRL_DEFAULT,
.ch_req_mask = 0xf,
.ch_req_max = 10,
.ch_reg_size = 0x80,
.nr_channels = 22,
+ .ch_fifo_size_mask = 0xf,
+ .sreq_index_offset = 2,
+ .has_outstanding_reqs = false,
};
static const struct tegra_adma_chip_data tegra186_chip_data = {
.ch_req_tx_shift = 27,
.ch_req_rx_shift = 22,
.ch_base_offset = 0x10000,
- .has_outstanding_reqs = true,
- .ch_fifo_ctrl = TEGRA186_FIFO_CTRL_DEFAULT,
.ch_req_mask = 0x1f,
.ch_req_max = 20,
.ch_reg_size = 0x100,
.nr_channels = 32,
+ .ch_fifo_size_mask = 0x1f,
+ .sreq_index_offset = 4,
+ .has_outstanding_reqs = true,
};
static const struct of_device_id tegra_adma_of_match[] = {
pm_runtime_enable(&pdev->dev);
- ret = pm_runtime_get_sync(&pdev->dev);
+ ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0)
goto rpm_disable;
for (i = 0; i < tdma->nr_channels; ++i)
irq_dispose_mapping(tdma->channels[i].irq);
- pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
return 0;
{
struct udma_dev *ud = uc->ud;
enum udma_tp_level tpl;
+ int ret;
if (uc->bchan) {
dev_dbg(ud->dev, "chan%d: already have bchan%d allocated\n",
tpl = ud->bchan_tpl.levels - 1;
uc->bchan = __udma_reserve_bchan(ud, tpl, -1);
- if (IS_ERR(uc->bchan))
- return PTR_ERR(uc->bchan);
+ if (IS_ERR(uc->bchan)) {
+ ret = PTR_ERR(uc->bchan);
+ uc->bchan = NULL;
+ return ret;
+ }
uc->tchan = uc->bchan;
static int udma_get_tchan(struct udma_chan *uc)
{
struct udma_dev *ud = uc->ud;
+ int ret;
if (uc->tchan) {
dev_dbg(ud->dev, "chan%d: already have tchan%d allocated\n",
*/
uc->tchan = __udma_reserve_tchan(ud, uc->config.channel_tpl,
uc->config.mapped_channel_id);
- if (IS_ERR(uc->tchan))
- return PTR_ERR(uc->tchan);
+ if (IS_ERR(uc->tchan)) {
+ ret = PTR_ERR(uc->tchan);
+ uc->tchan = NULL;
+ return ret;
+ }
if (ud->tflow_cnt) {
int tflow_id;
static int udma_get_rchan(struct udma_chan *uc)
{
struct udma_dev *ud = uc->ud;
+ int ret;
if (uc->rchan) {
dev_dbg(ud->dev, "chan%d: already have rchan%d allocated\n",
*/
uc->rchan = __udma_reserve_rchan(ud, uc->config.channel_tpl,
uc->config.mapped_channel_id);
+ if (IS_ERR(uc->rchan)) {
+ ret = PTR_ERR(uc->rchan);
+ uc->rchan = NULL;
+ return ret;
+ }
- return PTR_ERR_OR_ZERO(uc->rchan);
+ return 0;
}
static int udma_get_chan_pair(struct udma_chan *uc)
static int udma_get_rflow(struct udma_chan *uc, int flow_id)
{
struct udma_dev *ud = uc->ud;
+ int ret;
if (!uc->rchan) {
dev_err(ud->dev, "chan%d: does not have rchan??\n", uc->id);
}
uc->rflow = __udma_get_rflow(ud, flow_id);
+ if (IS_ERR(uc->rflow)) {
+ ret = PTR_ERR(uc->rflow);
+ uc->rflow = NULL;
+ return ret;
+ }
- return PTR_ERR_OR_ZERO(uc->rflow);
+ return 0;
}
static void bcdma_put_bchan(struct udma_chan *uc)
}
/**
- * xilinx_dma_tx_descriptor - Allocate transaction descriptor
+ * xilinx_dma_alloc_tx_descriptor - Allocate transaction descriptor
* @chan: Driver specific DMA channel
*
* Return: The allocated descriptor on success and NULL on failure.
struct xilinx_dma_tx_descriptor *desc,
unsigned long *flags)
{
- dma_async_tx_callback callback;
- void *callback_param;
+ struct dmaengine_desc_callback cb;
- callback = desc->async_tx.callback;
- callback_param = desc->async_tx.callback_param;
- if (callback) {
+ dmaengine_desc_get_callback(&desc->async_tx, &cb);
+ if (dmaengine_desc_callback_valid(&cb)) {
spin_unlock_irqrestore(&chan->lock, *flags);
- callback(callback_param);
+ dmaengine_desc_callback_invoke(&cb, NULL);
spin_lock_irqsave(&chan->lock, *flags);
}
}
}
/**
- * xilinx_dma_channel_set_config - Configure VDMA channel
+ * xilinx_vdma_channel_set_config - Configure VDMA channel
* Run-time configuration for Axi VDMA, supports:
* . halt the channel
* . configure interrupt coalescing and inter-packet delay threshold
/* -----------------------------------------------------------------------------
* DebugFS
*/
-
-#ifdef CONFIG_DEBUG_FS
-
#define XILINX_DPDMA_DEBUGFS_READ_MAX_SIZE 32
#define XILINX_DPDMA_DEBUGFS_UINT16_MAX_STR "65535"
static void xilinx_dpdma_debugfs_desc_done_irq(struct xilinx_dpdma_chan *chan)
{
- if (chan->id == dpdma_debugfs.chan_id)
+ if (IS_ENABLED(CONFIG_DEBUG_FS) && chan->id == dpdma_debugfs.chan_id)
dpdma_debugfs.xilinx_dpdma_irq_done_count++;
}
dev_err(xdev->dev, "Failed to create debugfs testcase file\n");
}
-#else
-static void xilinx_dpdma_debugfs_init(struct xilinx_dpdma_device *xdev)
-{
-}
-
-static void xilinx_dpdma_debugfs_desc_done_irq(struct xilinx_dpdma_chan *chan)
-{
-}
-#endif /* CONFIG_DEBUG_FS */
-
/* -----------------------------------------------------------------------------
* I/O Accessors
*/
*/
#include <linux/bitops.h>
-#include <linux/dmapool.h>
-#include <linux/dma/xilinx_dma.h>
+#include <linux/dma-mapping.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/module.h>
-#include <linux/of_address.h>
#include <linux/of_dma.h>
-#include <linux/of_irq.h>
#include <linux/of_platform.h>
#include <linux/slab.h>
#include <linux/clk.h>
static void zynqmp_dma_chan_desc_cleanup(struct zynqmp_dma_chan *chan)
{
struct zynqmp_dma_desc_sw *desc, *next;
+ unsigned long irqflags;
+
+ spin_lock_irqsave(&chan->lock, irqflags);
list_for_each_entry_safe(desc, next, &chan->done_list, node) {
- dma_async_tx_callback callback;
- void *callback_param;
-
- callback = desc->async_tx.callback;
- callback_param = desc->async_tx.callback_param;
- if (callback) {
- spin_unlock(&chan->lock);
- callback(callback_param);
- spin_lock(&chan->lock);
+ struct dmaengine_desc_callback cb;
+
+ dmaengine_desc_get_callback(&desc->async_tx, &cb);
+ if (dmaengine_desc_callback_valid(&cb)) {
+ spin_unlock_irqrestore(&chan->lock, irqflags);
+ dmaengine_desc_callback_invoke(&cb, NULL);
+ spin_lock_irqsave(&chan->lock, irqflags);
}
/* Run any dependencies, then free the descriptor */
zynqmp_dma_free_descriptor(chan, desc);
}
+
+ spin_unlock_irqrestore(&chan->lock, irqflags);
}
/**
*/
static void zynqmp_dma_free_descriptors(struct zynqmp_dma_chan *chan)
{
+ unsigned long irqflags;
+
+ spin_lock_irqsave(&chan->lock, irqflags);
zynqmp_dma_free_desc_list(chan, &chan->active_list);
zynqmp_dma_free_desc_list(chan, &chan->pending_list);
zynqmp_dma_free_desc_list(chan, &chan->done_list);
+ spin_unlock_irqrestore(&chan->lock, irqflags);
}
/**
static void zynqmp_dma_free_chan_resources(struct dma_chan *dchan)
{
struct zynqmp_dma_chan *chan = to_chan(dchan);
- unsigned long irqflags;
- spin_lock_irqsave(&chan->lock, irqflags);
zynqmp_dma_free_descriptors(chan);
- spin_unlock_irqrestore(&chan->lock, irqflags);
dma_free_coherent(chan->dev,
(2 * ZYNQMP_DMA_DESC_SIZE(chan) * ZYNQMP_DMA_NUM_DESCS),
chan->desc_pool_v, chan->desc_pool_p);
*/
static void zynqmp_dma_reset(struct zynqmp_dma_chan *chan)
{
+ unsigned long irqflags;
+
writel(ZYNQMP_DMA_IDS_DEFAULT_MASK, chan->regs + ZYNQMP_DMA_IDS);
+ spin_lock_irqsave(&chan->lock, irqflags);
zynqmp_dma_complete_descriptor(chan);
+ spin_unlock_irqrestore(&chan->lock, irqflags);
zynqmp_dma_chan_desc_cleanup(chan);
zynqmp_dma_free_descriptors(chan);
+
zynqmp_dma_init(chan);
}
u32 count;
unsigned long irqflags;
- spin_lock_irqsave(&chan->lock, irqflags);
-
if (chan->err) {
zynqmp_dma_reset(chan);
chan->err = false;
- goto unlock;
+ return;
}
+ spin_lock_irqsave(&chan->lock, irqflags);
count = readl(chan->regs + ZYNQMP_DMA_IRQ_DST_ACCT);
-
while (count) {
zynqmp_dma_complete_descriptor(chan);
- zynqmp_dma_chan_desc_cleanup(chan);
count--;
}
+ spin_unlock_irqrestore(&chan->lock, irqflags);
- if (chan->idle)
- zynqmp_dma_start_transfer(chan);
+ zynqmp_dma_chan_desc_cleanup(chan);
-unlock:
- spin_unlock_irqrestore(&chan->lock, irqflags);
+ if (chan->idle) {
+ spin_lock_irqsave(&chan->lock, irqflags);
+ zynqmp_dma_start_transfer(chan);
+ spin_unlock_irqrestore(&chan->lock, irqflags);
+ }
}
/**
static int zynqmp_dma_device_terminate_all(struct dma_chan *dchan)
{
struct zynqmp_dma_chan *chan = to_chan(dchan);
- unsigned long irqflags;
- spin_lock_irqsave(&chan->lock, irqflags);
writel(ZYNQMP_DMA_IDS_DEFAULT_MASK, chan->regs + ZYNQMP_DMA_IDS);
zynqmp_dma_free_descriptors(chan);
- spin_unlock_irqrestore(&chan->lock, irqflags);
return 0;
}
p->dev = &pdev->dev;
zdev->clk_main = devm_clk_get(&pdev->dev, "clk_main");
- if (IS_ERR(zdev->clk_main)) {
- dev_err(&pdev->dev, "main clock not found.\n");
- return PTR_ERR(zdev->clk_main);
- }
+ if (IS_ERR(zdev->clk_main))
+ return dev_err_probe(&pdev->dev, PTR_ERR(zdev->clk_main),
+ "main clock not found.\n");
zdev->clk_apb = devm_clk_get(&pdev->dev, "clk_apb");
- if (IS_ERR(zdev->clk_apb)) {
- dev_err(&pdev->dev, "apb clock not found.\n");
- return PTR_ERR(zdev->clk_apb);
- }
+ if (IS_ERR(zdev->clk_apb))
+ return dev_err_probe(&pdev->dev, PTR_ERR(zdev->clk_apb),
+ "apb clock not found.\n");
platform_set_drvdata(pdev, zdev);
pm_runtime_set_autosuspend_delay(zdev->dev, ZDMA_PM_TIMEOUT);
ret = zynqmp_dma_chan_probe(zdev, pdev);
if (ret) {
- dev_err(&pdev->dev, "Probing channel failed\n");
+ dev_err_probe(&pdev->dev, ret, "Probing channel failed\n");
goto err_disable_pm;
}
ret = of_dma_controller_register(pdev->dev.of_node,
of_zynqmp_dma_xlate, zdev);
if (ret) {
- dev_err(&pdev->dev, "Unable to register DMA to DT\n");
+ dev_err_probe(&pdev->dev, ret, "Unable to register DMA to DT\n");
dma_async_device_unregister(&zdev->common);
goto free_chan_resources;
}
pm_runtime_mark_last_busy(zdev->dev);
pm_runtime_put_sync_autosuspend(zdev->dev);
- dev_info(&pdev->dev, "ZynqMP DMA driver Probe success\n");
-
return 0;
free_chan_resources:
sbp2_unmap_scatterlist(device->card->device, orb);
orb->cmd->result = result;
- orb->cmd->scsi_done(orb->cmd);
+ scsi_done(orb->cmd);
}
static int sbp2_map_scatterlist(struct sbp2_command_orb *orb,
static DEVICE_ATTR(ieee1394_id, S_IRUGO, sbp2_sysfs_ieee1394_id_show, NULL);
-static struct device_attribute *sbp2_scsi_sysfs_attrs[] = {
- &dev_attr_ieee1394_id,
+static struct attribute *sbp2_scsi_sysfs_attrs[] = {
+ &dev_attr_ieee1394_id.attr,
NULL
};
+ATTRIBUTE_GROUPS(sbp2_scsi_sysfs);
+
static struct scsi_host_template scsi_driver_template = {
.module = THIS_MODULE,
.name = "SBP-2 IEEE-1394",
.sg_tablesize = SG_ALL,
.max_segment_size = SBP2_MAX_SEG_SIZE,
.can_queue = 1,
- .sdev_attrs = sbp2_scsi_sysfs_attrs,
+ .sdev_groups = sbp2_scsi_sysfs_groups,
};
MODULE_AUTHOR("Kristian Hoegsberg <krh@bitplanet.net>");
if (slab_is_available())
memblock_free_late(phys, size);
else
- memblock_free(phys, size);
+ memblock_phys_free(phys, size);
} else if (flags & EFI_MEMMAP_SLAB) {
struct page *p = pfn_to_page(PHYS_PFN(phys));
unsigned int order = get_order(size);
* physical address of memory block reserved by secure monitor software at
* secure world.
*
- * svc_normal_to_secure_shm_thread() calls do_exit() directly since it is a
+ * svc_normal_to_secure_shm_thread() terminates directly since it is a
* standlone thread for which no one will call kthread_stop() or return when
* 'kthread_should_stop()' is true.
*/
}
complete(&sh_mem->sync_complete);
- do_exit(0);
+ return 0;
}
/**
/* Max HashMap Order for PM API feature check (1<<7 = 128) */
#define PM_API_FEATURE_CHECK_MAX_ORDER 7
+/* CRL registers and bitfields */
+#define CRL_APB_BASE 0xFF5E0000U
+/* BOOT_PIN_CTRL- Used to control the mode pins after boot */
+#define CRL_APB_BOOT_PIN_CTRL (CRL_APB_BASE + (0x250U))
+/* BOOT_PIN_CTRL_MASK- out_val[11:8], out_en[3:0] */
+#define CRL_APB_BOOTPIN_CTRL_MASK 0xF0FU
+
static bool feature_check_enabled;
static DEFINE_HASHTABLE(pm_api_features_map, PM_API_FEATURE_CHECK_MAX_ORDER);
EXPORT_SYMBOL_GPL(zynqmp_pm_pinctrl_set_config);
/**
+ * zynqmp_pm_bootmode_read() - PM Config API for read bootpin status
+ * @ps_mode: Returned output value of ps_mode
+ *
+ * This API function is to be used for notify the power management controller
+ * to read bootpin status.
+ *
+ * Return: status, either success or error+reason
+ */
+unsigned int zynqmp_pm_bootmode_read(u32 *ps_mode)
+{
+ unsigned int ret;
+ u32 ret_payload[PAYLOAD_ARG_CNT];
+
+ ret = zynqmp_pm_invoke_fn(PM_MMIO_READ, CRL_APB_BOOT_PIN_CTRL, 0,
+ 0, 0, ret_payload);
+
+ *ps_mode = ret_payload[1];
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_bootmode_read);
+
+/**
+ * zynqmp_pm_bootmode_write() - PM Config API for Configure bootpin
+ * @ps_mode: Value to be written to the bootpin ctrl register
+ *
+ * This API function is to be used for notify the power management controller
+ * to configure bootpin.
+ *
+ * Return: Returns status, either success or error+reason
+ */
+int zynqmp_pm_bootmode_write(u32 ps_mode)
+{
+ return zynqmp_pm_invoke_fn(PM_MMIO_WRITE, CRL_APB_BOOT_PIN_CTRL,
+ CRL_APB_BOOTPIN_CTRL_MASK, ps_mode, 0, NULL);
+}
+EXPORT_SYMBOL_GPL(zynqmp_pm_bootmode_write);
+
+/**
* zynqmp_pm_init_finalize() - PM call to inform firmware that the caller
* master has initialized its own power management
*
bool "GPIO Support"
help
This enables GPIO support through the generic GPIO library.
- You only need to enable this, if you also want to enable
+ You only need to enable this if you also want to enable
one or more of the GPIO drivers below.
If unsure, say N.
depends on ACPI
select GPIO_GENERIC
help
- driver for GPIO functionality on Promontory IOHub
- Require ACPI ASL code to enumerate as a platform device.
+ Driver for GPIO functionality on Promontory IOHub.
+ Requires ACPI ASL code to enumerate as a platform device.
config GPIO_ASPEED
tristate "Aspeed GPIO support"
help
Say Y or M here to build support for the HiSilicon GPIO controller
driver GPIO block.
- This GPIO controller support double-edge interrupt and multi-core
+ This GPIO controller supports double-edge interrupt and multi-core
concurrent access.
config GPIO_HLWD
help
Say yes here to support the GPIO functionality of a number of Intel
ICH-based chipsets. Currently supported devices: ICH6, ICH7, ICH8
- ICH9, ICH10, Series 5/3400 (eg Ibex Peak), Series 6/C200 (eg
+ ICH9, ICH10, Series 5/3400 (e.g. Ibex Peak), Series 6/C200 (e.g.
Cougar Point), NM10 (Tiger Point), and 3100 (Whitmore Lake).
If unsure, say N.
select GPIO_GENERIC
help
Say yes here to support the GPIO functionality of a number of Intel
- IOP32X or IOP33X.
+ IOP32X or IOP33X series of chips.
If unsure, say N.
bool "Loongson-2/3 GPIO support"
depends on CPU_LOONGSON2EF || CPU_LOONGSON64
help
- driver for GPIO functionality on Loongson-2F/3A/3B processors.
+ Driver for GPIO functionality on Loongson-2F/3A/3B processors.
config GPIO_LPC18XX
tristate "NXP LPC18XX/43XX GPIO support"
depends on MCB
select GPIO_GENERIC
help
- Say yes here to support the MEN 16Z127 GPIO Controller
+ Say yes here to support the MEN 16Z127 GPIO Controller.
config GPIO_MM_LANTIQ
bool "Lantiq Memory mapped GPIOs"
depends on LANTIQ && SOC_XWAY
help
This enables support for memory mapped GPIOs on the External Bus Unit
- (EBU) found on Lantiq SoCs. The gpios are output only as they are
- created by attaching a 16bit latch to the bus.
+ (EBU) found on Lantiq SoCs. The GPIOs are output only as they are
+ created by attaching a 16-bit latch to the bus.
config GPIO_MPC5200
def_bool y
select GPIO_GENERIC
select GPIOLIB_IRQCHIP
help
- Say yes here to support the Mediatek MT7621 SoC GPIO device
+ Say yes here to support the Mediatek MT7621 SoC GPIO device.
config GPIO_MVEBU
def_bool y
select IRQ_DOMAIN
select GPIOLIB_IRQCHIP
help
- Say yes here to support the PrimeCell PL061 GPIO device
+ Say yes here to support the PrimeCell PL061 GPIO device.
config GPIO_PMIC_EIC_SPRD
tristate "Spreadtrum PMIC EIC support"
bool "PXA GPIO support"
depends on ARCH_PXA || ARCH_MMP || COMPILE_TEST
help
- Say yes here to support the PXA GPIO device
+ Say yes here to support the PXA GPIO device.
config GPIO_RCAR
tristate "Renesas R-Car and RZ/G GPIO support"
config GPIO_ROCKCHIP
tristate "Rockchip GPIO support"
depends on ARCH_ROCKCHIP || COMPILE_TEST
+ select GENERIC_IRQ_CHIP
select GPIOLIB_IRQCHIP
default ARCH_ROCKCHIP
help
depends on PLAT_SPEAR
select GENERIC_IRQ_CHIP
help
- Say yes here to support ST SPEAr SPI Chip Select as GPIO device
+ Say yes here to support ST SPEAr SPI Chip Select as GPIO device.
config GPIO_SPRD
tristate "Spreadtrum GPIO support"
help
This enables support for the Serial To Parallel (STP) unit found on
XWAY SoC. The STP allows the SoC to drive a shift registers cascade,
- that can be up to 24 bit. This peripheral is aimed at driving leds.
- Some of the gpios/leds can be auto updated by the soc with dsl and
+ that can be up to 24 bits. This peripheral is aimed at driving LEDs.
+ Some of the GPIOs/LEDs can be auto updated by the SoC with DSL and
phy status.
config GPIO_SYSCON
Say yes here to support GPIO on Tohisba Visconti.
config GPIO_VR41XX
- tristate "NEC VR4100 series General-purpose I/O Uint support"
+ tristate "NEC VR4100 series General-purpose I/O Unit support"
depends on CPU_VR41XX
help
- Say yes here to support the NEC VR4100 series General-purpose I/O Uint
+ Say yes here to support the NEC VR4100 series General-purpose I/O Unit.
config GPIO_VX855
tristate "VIA VX855/VX875 GPIO"
select MFD_CORE
select MFD_VX855
help
- Support access to the VX855/VX875 GPIO lines through the gpio library.
+ Support access to the VX855/VX875 GPIO lines through the GPIO library.
- This driver provides common support for accessing the device,
- additional drivers must be enabled in order to use the
+ This driver provides common support for accessing the device.
+ Additional drivers must be enabled in order to use the
functionality of the device.
config GPIO_WCD934X
- tristate "Qualcomm Technologies Inc WCD9340/WCD9341 gpio controller driver"
+ tristate "Qualcomm Technologies Inc WCD9340/WCD9341 GPIO controller driver"
depends on MFD_WCD934X && OF_GPIO
help
This driver is to support GPIO block found on the Qualcomm Technologies
select GPIOLIB_IRQCHIP
depends on OF_GPIO
help
- Say yes here to support the Xilinx FPGA GPIO device
+ Say yes here to support the Xilinx FPGA GPIO device.
config GPIO_XLP
tristate "Netlogic XLP GPIO support"
depends on !SMP
help
Say yes here to support the Xtensa internal GPIO32 IMPWIRE (input)
- and EXPSTATE (output) ports
+ and EXPSTATE (output) ports.
config GPIO_ZEVIO
bool "LSI ZEVIO SoC memory mapped GPIOs"
help
Say yes here to support Xilinx Zynq GPIO controller.
+config GPIO_ZYNQMP_MODEPIN
+ tristate "ZynqMP ps-mode pin GPIO configuration driver"
+ depends on ZYNQMP_FIRMWARE
+ default ZYNQMP_FIRMWARE
+ help
+ Say yes here to support the ZynqMP ps-mode pin GPIO configuration
+ driver.
+
+ This ps-mode pin GPIO driver is based on GPIO framework. PS_MODE
+ is 4-bits boot mode pins. It sets and gets the status of
+ the ps-mode pin. Every pin can be configured as input/output.
+
config GPIO_LOONGSON1
tristate "Loongson1 GPIO support"
depends on MACH_LOONGSON32
config GPIO_AMD_FCH
tristate "GPIO support for AMD Fusion Controller Hub (G-series SOCs)"
help
- This option enables driver for GPIO on AMDs Fusion Controller Hub,
- as found on G-series SOCs (eg. GX-412TC)
+ This option enables driver for GPIO on AMD's Fusion Controller Hub,
+ as found on G-series SOCs (e.g. GX-412TC).
- Note: This driver doesn't registers itself automatically, as it
- needs to be provided with platform specific configuration.
- (See eg. CONFIG_PCENGINES_APU2.)
+ Note: This driver doesn't register itself automatically, as it
+ needs to be provided with platform-specific configuration.
+ (See e.g. CONFIG_PCENGINES_APU2.)
config GPIO_MSC313
bool "MStar MSC313 GPIO support"
select IRQ_DOMAIN_HIERARCHY
help
Say Y here to support the main GPIO block on MStar/SigmaStar
- ARMv7 based SoCs.
+ ARMv7-based SoCs.
config GPIO_IDT3243X
tristate "IDT 79RC3243X GPIO support"
select GPIOLIB_IRQCHIP
help
Select this option to enable GPIO driver for
- IDT 79RC3243X based devices like Mikrotik RB532.
+ IDT 79RC3243X-based devices like Mikrotik RB532.
To compile this driver as a module, choose M here: the module will
be called gpio-idt3243x.
well.
To compile this driver as a module, choose M here: the module will
- be called gpio_it87
+ be called gpio_it87.
config GPIO_SCH
tristate "Intel SCH/TunnelCreek/Centerton/Quark X1000 GPIO"
powered by the core power rail and are turned off during sleep
modes (S3 and higher). The remaining four GPIOs are powered by
the Intel SCH suspend power supply. These GPIOs remain
- active during S3. The suspend powered GPIOs can be used to wake the
+ active during S3. The suspend-powered GPIOs can be used to wake the
system from the Suspend-to-RAM state.
The Intel Tunnel Creek processor has 5 GPIOs powered by the
select GPIOLIB_IRQCHIP
help
Say yes here to enable the pca953x to be used as an interrupt
- controller. It requires the driver to be built in the kernel.
+ controller.
config GPIO_PCA9570
tristate "PCA9570 4-Bit I2C GPO expander"
help
Support for GPIO pins on Crystal Cove PMIC.
- Say Yes if you have a Intel SoC based tablet with Crystal Cove PMIC
+ Say Yes if you have a Intel SoC-based tablet with Crystal Cove PMIC
inside.
This driver can also be built as a module. If so, the module will be
Say yes here to enable the GPIO driver for the DA9055 chip.
The Dialog DA9055 PMIC chip has 3 GPIO pins that can be
- be controller by this driver.
+ be controlled by this driver.
If driver is built as a module it will be called gpio-da9055.
help
This driver supports the CPLD egpio chip present on
several HTC phones. It provides basic support for input
- pins, output pins, and irqs.
+ pins, output pins, and IRQs.
config GPIO_JANZ_TTL
tristate "Janz VMOD-TTL Digital IO Module"
help
GPIO driver for MAX77620 and MAX20024 PMIC from Maxim Semiconductor.
MAX77620 PMIC has 8 pins that can be configured as GPIOs. The
- driver also provides interrupt support for each of the gpios.
- Say yes here to enable the max77620 to be used as gpio controller.
+ driver also provides interrupt support for each of the GPIOs.
+ Say yes here to enable the max77620 to be used as GPIO controller.
config GPIO_MAX77650
tristate "Maxim MAX77650/77651 GPIO support"
help
Select this option to enable GPIO driver for the Ricoh RC5T583
chip family.
- This driver provides the support for driving/reading the gpio pins
- of RC5T583 device through standard gpio library.
+ This driver provides the support for driving/reading the GPIO pins
+ of RC5T583 device through standard GPIO library.
config GPIO_SL28CPLD
tristate "Kontron sl28cpld GPIO support"
tristate "TI TPS65912 GPIO"
depends on MFD_TPS65912
help
- This driver supports TPS65912 gpio chip
+ This driver supports TPS65912 GPIO chip.
config GPIO_TPS68470
bool "TPS68470 GPIO"
help
Select this option to enable GPIO driver for the TPS68470
chip family.
- There are 7 GPIOs and few sensor related GPIOs supported
+ There are 7 GPIOs and few sensor-related GPIOs supported
by the TPS68470. While the 7 GPIOs can be configured as
input or output as appropriate, the sensor related GPIOs
are "output only" GPIOs.
help
Support for GPIO pins on Whiskey Cove PMIC.
- Say Yes if you have a Intel SoC based tablet with Whiskey Cove PMIC
+ Say Yes if you have an Intel SoC-based tablet with Whiskey Cove PMIC
inside.
This driver can also be built as a module. If so, the module will be
depends on X86 || COMPILE_TEST
depends on HAS_IOPORT_MAP
help
- The AMD 8111 south bridge contains 32 GPIO pins which can be used.
+ The AMD 8111 southbridge contains 32 GPIO pins which can be used.
- Note, that usually system firmware/ACPI handles GPIO pins on their
- own and users might easily break their systems with uncarefull usage
+ Note that usually system firmware/ACPI handles GPIO pins on their
+ own and users might easily break their systems with uncareful usage
of this driver!
If unsure, say N
select GENERIC_IRQ_CHIP
help
ML7213 is companion chip for Intel Atom E6xx series.
- This driver can be used for OKI SEMICONDUCTOR ML7213 IOH(Input/Output
- Hub) which is for IVI(In-Vehicle Infotainment) use.
+ This driver can be used for OKI SEMICONDUCTOR ML7213 IOH (Input/Output
+ Hub) which is for IVI (In-Vehicle Infotainment) use.
This driver can access the IOH's GPIO device.
config GPIO_PCH
- tristate "Intel EG20T PCH/LAPIS Semiconductor IOH(ML7223/ML7831) GPIO"
+ tristate "Intel EG20T PCH/LAPIS Semiconductor IOH (ML7223/ML7831) GPIO"
depends on X86_32 || MIPS || COMPILE_TEST
select GENERIC_IRQ_CHIP
help
- This driver is for PCH(Platform controller Hub) GPIO of Intel Topcliff
- which is an IOH(Input/Output Hub) for x86 embedded processor.
+ This driver is for PCH (Platform Controller Hub) GPIO of Intel Topcliff,
+ which is an IOH (Input/Output Hub) for x86 embedded processor.
This driver can access PCH GPIO device.
- This driver also can be used for LAPIS Semiconductor IOH(Input/
+ This driver also can be used for LAPIS Semiconductor IOH (Input/
Output Hub), ML7223 and ML7831.
- ML7223 IOH is for MP(Media Phone) use.
+ ML7223 IOH is for MP (Media Phone) use.
ML7831 IOH is for general purpose use.
ML7223/ML7831 is companion chip for Intel Atom E6xx series.
ML7223/ML7831 is completely compatible for Intel EG20T PCH.
help
Driver for 74x164 compatible serial-in/parallel-out 8-outputs
shift registers. This driver can be used to provide access
- to more gpio outputs.
+ to more GPIO outputs.
config GPIO_MAX3191X
tristate "Maxim MAX3191x industrial serializer"
config GPIO_VIRTIO
tristate "VirtIO GPIO support"
depends on VIRTIO
+ select GPIOLIB_IRQCHIP
help
Say Y here to enable guest support for virtio-based GPIO controllers.
obj-$(CONFIG_GPIO_XTENSA) += gpio-xtensa.o
obj-$(CONFIG_GPIO_ZEVIO) += gpio-zevio.o
obj-$(CONFIG_GPIO_ZYNQ) += gpio-zynq.o
+obj-$(CONFIG_GPIO_ZYNQMP_MODEPIN) += gpio-zynqmp-modepin.o
unsigned long tmp[]; /* values and descs for multiple ops */
};
+#define fwd_tmp_values(fwd) &(fwd)->tmp[0]
+#define fwd_tmp_descs(fwd) (void *)&(fwd)->tmp[BITS_TO_LONGS((fwd)->chip.ngpio)]
+
+#define fwd_tmp_size(ngpios) (BITS_TO_LONGS((ngpios)) + (ngpios))
+
static int gpio_fwd_get_direction(struct gpio_chip *chip, unsigned int offset)
{
struct gpiochip_fwd *fwd = gpiochip_get_data(chip);
static int gpio_fwd_get_multiple(struct gpiochip_fwd *fwd, unsigned long *mask,
unsigned long *bits)
{
- struct gpio_desc **descs;
- unsigned long *values;
+ struct gpio_desc **descs = fwd_tmp_descs(fwd);
+ unsigned long *values = fwd_tmp_values(fwd);
unsigned int i, j = 0;
int error;
- /* Both values bitmap and desc pointers are stored in tmp[] */
- values = &fwd->tmp[0];
- descs = (void *)&fwd->tmp[BITS_TO_LONGS(fwd->chip.ngpio)];
-
bitmap_clear(values, 0, fwd->chip.ngpio);
for_each_set_bit(i, mask, fwd->chip.ngpio)
descs[j++] = fwd->descs[i];
static void gpio_fwd_set_multiple(struct gpiochip_fwd *fwd, unsigned long *mask,
unsigned long *bits)
{
- struct gpio_desc **descs;
- unsigned long *values;
+ struct gpio_desc **descs = fwd_tmp_descs(fwd);
+ unsigned long *values = fwd_tmp_values(fwd);
unsigned int i, j = 0;
- /* Both values bitmap and desc pointers are stored in tmp[] */
- values = &fwd->tmp[0];
- descs = (void *)&fwd->tmp[BITS_TO_LONGS(fwd->chip.ngpio)];
-
for_each_set_bit(i, mask, fwd->chip.ngpio) {
__assign_bit(j, values, test_bit(i, bits));
descs[j++] = fwd->descs[i];
unsigned int i;
int error;
- fwd = devm_kzalloc(dev, struct_size(fwd, tmp,
- BITS_TO_LONGS(ngpios) + ngpios), GFP_KERNEL);
+ fwd = devm_kzalloc(dev, struct_size(fwd, tmp, fwd_tmp_size(ngpios)),
+ GFP_KERNEL);
if (!fwd)
return ERR_PTR(-ENOMEM);
static int max7300_remove(struct i2c_client *client)
{
- return __max730x_remove(&client->dev);
+ __max730x_remove(&client->dev);
+
+ return 0;
}
static const struct i2c_device_id max7300_id[] = {
static int max7301_remove(struct spi_device *spi)
{
- return __max730x_remove(&spi->dev);
+ __max730x_remove(&spi->dev);
+
+ return 0;
}
static const struct spi_device_id max7301_id[] = {
}
EXPORT_SYMBOL_GPL(__max730x_probe);
-int __max730x_remove(struct device *dev)
+void __max730x_remove(struct device *dev)
{
struct max7301 *ts = dev_get_drvdata(dev);
- if (ts == NULL)
- return -ENODEV;
-
/* Power down the chip and disable IRQ output */
ts->write(dev, 0x04, 0x00);
gpiochip_remove(&ts->chip);
mutex_destroy(&ts->lock);
- return 0;
}
EXPORT_SYMBOL_GPL(__max730x_remove);
MODULE_DESCRIPTION("GPIO interface for MAX77620 and MAX20024 PMIC");
MODULE_AUTHOR("Laxman Dewangan <ldewangan@nvidia.com>");
MODULE_AUTHOR("Chaitanya Bandi <bandik@nvidia.com>");
-MODULE_ALIAS("platform:max77620-gpio");
MODULE_LICENSE("GPL v2");
struct mc33880 *mc;
mc = spi_get_drvdata(spi);
- if (!mc)
- return -ENODEV;
gpiochip_remove(&mc->chip);
mutex_destroy(&mc->lock);
// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2020-2021 NVIDIA CORPORATION & AFFILIATES
+ */
+
#include <linux/bitfield.h>
#include <linux/bitops.h>
#include <linux/device.h>
#include <linux/gpio/driver.h>
+#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/ioport.h>
#include <linux/kernel.h>
#define YU_GPIO_MODE0 0x0c
#define YU_GPIO_DATASET 0x14
#define YU_GPIO_DATACLEAR 0x18
+#define YU_GPIO_CAUSE_RISE_EN 0x44
+#define YU_GPIO_CAUSE_FALL_EN 0x48
#define YU_GPIO_MODE1_CLEAR 0x50
#define YU_GPIO_MODE0_SET 0x54
#define YU_GPIO_MODE0_CLEAR 0x58
+#define YU_GPIO_CAUSE_OR_CAUSE_EVTEN0 0x80
+#define YU_GPIO_CAUSE_OR_EVTEN0 0x94
+#define YU_GPIO_CAUSE_OR_CLRCAUSE 0x98
struct mlxbf2_gpio_context_save_regs {
u32 gpio_mode0;
/* BlueField-2 gpio block context structure. */
struct mlxbf2_gpio_context {
struct gpio_chip gc;
+ struct irq_chip irq_chip;
/* YU GPIO blocks address */
void __iomem *gpio_io;
return ret;
}
+static void mlxbf2_gpio_irq_enable(struct irq_data *irqd)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(irqd);
+ struct mlxbf2_gpio_context *gs = gpiochip_get_data(gc);
+ int offset = irqd_to_hwirq(irqd);
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&gs->gc.bgpio_lock, flags);
+ val = readl(gs->gpio_io + YU_GPIO_CAUSE_OR_CLRCAUSE);
+ val |= BIT(offset);
+ writel(val, gs->gpio_io + YU_GPIO_CAUSE_OR_CLRCAUSE);
+
+ val = readl(gs->gpio_io + YU_GPIO_CAUSE_OR_EVTEN0);
+ val |= BIT(offset);
+ writel(val, gs->gpio_io + YU_GPIO_CAUSE_OR_EVTEN0);
+ spin_unlock_irqrestore(&gs->gc.bgpio_lock, flags);
+}
+
+static void mlxbf2_gpio_irq_disable(struct irq_data *irqd)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(irqd);
+ struct mlxbf2_gpio_context *gs = gpiochip_get_data(gc);
+ int offset = irqd_to_hwirq(irqd);
+ unsigned long flags;
+ u32 val;
+
+ spin_lock_irqsave(&gs->gc.bgpio_lock, flags);
+ val = readl(gs->gpio_io + YU_GPIO_CAUSE_OR_EVTEN0);
+ val &= ~BIT(offset);
+ writel(val, gs->gpio_io + YU_GPIO_CAUSE_OR_EVTEN0);
+ spin_unlock_irqrestore(&gs->gc.bgpio_lock, flags);
+}
+
+static irqreturn_t mlxbf2_gpio_irq_handler(int irq, void *ptr)
+{
+ struct mlxbf2_gpio_context *gs = ptr;
+ struct gpio_chip *gc = &gs->gc;
+ unsigned long pending;
+ u32 level;
+
+ pending = readl(gs->gpio_io + YU_GPIO_CAUSE_OR_CAUSE_EVTEN0);
+ writel(pending, gs->gpio_io + YU_GPIO_CAUSE_OR_CLRCAUSE);
+
+ for_each_set_bit(level, &pending, gc->ngpio) {
+ int gpio_irq = irq_find_mapping(gc->irq.domain, level);
+ generic_handle_irq(gpio_irq);
+ }
+
+ return IRQ_RETVAL(pending);
+}
+
+static int
+mlxbf2_gpio_irq_set_type(struct irq_data *irqd, unsigned int type)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(irqd);
+ struct mlxbf2_gpio_context *gs = gpiochip_get_data(gc);
+ int offset = irqd_to_hwirq(irqd);
+ unsigned long flags;
+ bool fall = false;
+ bool rise = false;
+ u32 val;
+
+ switch (type & IRQ_TYPE_SENSE_MASK) {
+ case IRQ_TYPE_EDGE_BOTH:
+ fall = true;
+ rise = true;
+ break;
+ case IRQ_TYPE_EDGE_RISING:
+ rise = true;
+ break;
+ case IRQ_TYPE_EDGE_FALLING:
+ fall = true;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ spin_lock_irqsave(&gs->gc.bgpio_lock, flags);
+ if (fall) {
+ val = readl(gs->gpio_io + YU_GPIO_CAUSE_FALL_EN);
+ val |= BIT(offset);
+ writel(val, gs->gpio_io + YU_GPIO_CAUSE_FALL_EN);
+ }
+
+ if (rise) {
+ val = readl(gs->gpio_io + YU_GPIO_CAUSE_RISE_EN);
+ val |= BIT(offset);
+ writel(val, gs->gpio_io + YU_GPIO_CAUSE_RISE_EN);
+ }
+ spin_unlock_irqrestore(&gs->gc.bgpio_lock, flags);
+
+ return 0;
+}
+
/* BlueField-2 GPIO driver initialization routine. */
static int
mlxbf2_gpio_probe(struct platform_device *pdev)
{
struct mlxbf2_gpio_context *gs;
struct device *dev = &pdev->dev;
+ struct gpio_irq_chip *girq;
struct gpio_chip *gc;
unsigned int npins;
- int ret;
+ const char *name;
+ int ret, irq;
+
+ name = dev_name(dev);
gs = devm_kzalloc(dev, sizeof(*gs), GFP_KERNEL);
if (!gs)
gc->ngpio = npins;
gc->owner = THIS_MODULE;
+ irq = platform_get_irq(pdev, 0);
+ if (irq >= 0) {
+ gs->irq_chip.name = name;
+ gs->irq_chip.irq_set_type = mlxbf2_gpio_irq_set_type;
+ gs->irq_chip.irq_enable = mlxbf2_gpio_irq_enable;
+ gs->irq_chip.irq_disable = mlxbf2_gpio_irq_disable;
+
+ girq = &gs->gc.irq;
+ girq->chip = &gs->irq_chip;
+ girq->handler = handle_simple_irq;
+ girq->default_type = IRQ_TYPE_NONE;
+ /* This will let us handle the parent IRQ in the driver */
+ girq->num_parents = 0;
+ girq->parents = NULL;
+ girq->parent_handler = NULL;
+
+ /*
+ * Directly request the irq here instead of passing
+ * a flow-handler because the irq is shared.
+ */
+ ret = devm_request_irq(dev, irq, mlxbf2_gpio_irq_handler,
+ IRQF_SHARED, name, gs);
+ if (ret) {
+ dev_err(dev, "failed to request IRQ");
+ return ret;
+ }
+ }
+
platform_set_drvdata(pdev, gs);
ret = devm_gpiochip_add_data(dev, &gs->gc, gs);
module_platform_driver(mlxbf2_gpio_driver);
MODULE_DESCRIPTION("Mellanox BlueField-2 GPIO Driver");
-MODULE_AUTHOR("Mellanox Technologies");
+MODULE_AUTHOR("Asmaa Mnebhi <asmaa@nvidia.com>");
MODULE_LICENSE("GPL v2");
status = realtek_gpio_read_isr(ctrl, lines_done / 8);
port_pin_count = min(gc->ngpio - lines_done, 8U);
for_each_set_bit(offset, &status, port_pin_count)
- generic_handle_domain_irq(gc->irq.domain, offset);
+ generic_handle_domain_irq(gc->irq.domain, offset + lines_done);
}
chained_irq_exit(irq_chip, desc);
const char *name;
unsigned int instance;
+ unsigned int num_irqs_per_bank;
+
const struct tegra186_pin_range *pin_ranges;
unsigned int num_pin_ranges;
const char *pinmux;
unsigned int *irq;
const struct tegra_gpio_soc *soc;
+ unsigned int num_irqs_per_bank;
+ unsigned int num_banks;
void __iomem *secure;
void __iomem *base;
struct irq_domain *domain = gpio->gpio.irq.domain;
struct irq_chip *chip = irq_desc_get_chip(desc);
unsigned int parent = irq_desc_get_irq(desc);
- unsigned int i, offset = 0;
+ unsigned int i, j, offset = 0;
chained_irq_enter(chip, desc);
base = gpio->base + port->bank * 0x1000 + port->port * 0x200;
/* skip ports that are not associated with this bank */
- if (parent != gpio->irq[port->bank])
+ for (j = 0; j < gpio->num_irqs_per_bank; j++) {
+ if (parent == gpio->irq[port->bank * gpio->num_irqs_per_bank + j])
+ break;
+ }
+
+ if (j == gpio->num_irqs_per_bank)
goto skip;
value = readl(base + TEGRA186_GPIO_INTERRUPT_STATUS(1));
static void tegra186_gpio_init_route_mapping(struct tegra_gpio *gpio)
{
+ struct device *dev = gpio->gpio.parent;
unsigned int i, j;
u32 value;
*/
if ((value & TEGRA186_GPIO_CTL_SCR_SEC_REN) == 0 &&
(value & TEGRA186_GPIO_CTL_SCR_SEC_WEN) == 0) {
- for (j = 0; j < 8; j++) {
+ /*
+ * On Tegra194 and later, each pin can be routed to one or more
+ * interrupts.
+ */
+ for (j = 0; j < gpio->num_irqs_per_bank; j++) {
+ dev_dbg(dev, "programming default interrupt routing for port %s\n",
+ port->name);
+
offset = TEGRA186_GPIO_INT_ROUTE_MAPPING(p, j);
- value = readl(base + offset);
- value = BIT(port->pins) - 1;
- writel(value, base + offset);
+ /*
+ * By default we only want to route GPIO pins to IRQ 0. This works
+ * only under the assumption that we're running as the host kernel
+ * and hence all GPIO pins are owned by Linux.
+ *
+ * For cases where Linux is the guest OS, the hypervisor will have
+ * to configure the interrupt routing and pass only the valid
+ * interrupts via device tree.
+ */
+ if (j == 0) {
+ value = readl(base + offset);
+ value = BIT(port->pins) - 1;
+ writel(value, base + offset);
+ }
}
}
}
}
+static unsigned int tegra186_gpio_irqs_per_bank(struct tegra_gpio *gpio)
+{
+ struct device *dev = gpio->gpio.parent;
+
+ if (gpio->num_irq > gpio->num_banks) {
+ if (gpio->num_irq % gpio->num_banks != 0)
+ goto error;
+ }
+
+ if (gpio->num_irq < gpio->num_banks)
+ goto error;
+
+ gpio->num_irqs_per_bank = gpio->num_irq / gpio->num_banks;
+
+ if (gpio->num_irqs_per_bank > gpio->soc->num_irqs_per_bank)
+ goto error;
+
+ return 0;
+
+error:
+ dev_err(dev, "invalid number of interrupts (%u) for %u banks\n",
+ gpio->num_irq, gpio->num_banks);
+ return -EINVAL;
+}
+
static int tegra186_gpio_probe(struct platform_device *pdev)
{
unsigned int i, j, offset;
return -ENOMEM;
gpio->soc = device_get_match_data(&pdev->dev);
+ gpio->gpio.label = gpio->soc->name;
+ gpio->gpio.parent = &pdev->dev;
+
+ /* count the number of banks in the controller */
+ for (i = 0; i < gpio->soc->num_ports; i++)
+ if (gpio->soc->ports[i].bank > gpio->num_banks)
+ gpio->num_banks = gpio->soc->ports[i].bank;
+
+ gpio->num_banks++;
+ /* get register apertures */
gpio->secure = devm_platform_ioremap_resource_byname(pdev, "security");
if (IS_ERR(gpio->secure)) {
gpio->secure = devm_platform_ioremap_resource(pdev, 0);
gpio->num_irq = err;
+ err = tegra186_gpio_irqs_per_bank(gpio);
+ if (err < 0)
+ return err;
+
gpio->irq = devm_kcalloc(&pdev->dev, gpio->num_irq, sizeof(*gpio->irq),
GFP_KERNEL);
if (!gpio->irq)
gpio->irq[i] = err;
}
- gpio->gpio.label = gpio->soc->name;
- gpio->gpio.parent = &pdev->dev;
-
gpio->gpio.request = gpiochip_generic_request;
gpio->gpio.free = gpiochip_generic_free;
gpio->gpio.get_direction = tegra186_gpio_get_direction;
irq->parent_handler = tegra186_gpio_irq;
irq->parent_handler_data = gpio;
irq->num_parents = gpio->num_irq;
- irq->parents = gpio->irq;
+
+ /*
+ * To simplify things, use a single interrupt per bank for now. Some
+ * chips support up to 8 interrupts per bank, which can be useful to
+ * distribute the load and decrease the processing latency for GPIOs
+ * but it also requires a more complicated interrupt routing than we
+ * currently program.
+ */
+ if (gpio->num_irqs_per_bank > 1) {
+ irq->parents = devm_kcalloc(&pdev->dev, gpio->num_banks,
+ sizeof(*irq->parents), GFP_KERNEL);
+ if (!irq->parents)
+ return -ENOMEM;
+
+ for (i = 0; i < gpio->num_banks; i++)
+ irq->parents[i] = gpio->irq[i * gpio->num_irqs_per_bank];
+
+ irq->num_parents = gpio->num_banks;
+ } else {
+ irq->num_parents = gpio->num_irq;
+ irq->parents = gpio->irq;
+ }
+
+ if (gpio->soc->num_irqs_per_bank > 1)
+ tegra186_gpio_init_route_mapping(gpio);
np = of_find_matching_node(NULL, tegra186_pmc_of_match);
if (np) {
return -EPROBE_DEFER;
}
- tegra186_gpio_init_route_mapping(gpio);
-
irq->map = devm_kcalloc(&pdev->dev, gpio->gpio.ngpio,
sizeof(*irq->map), GFP_KERNEL);
if (!irq->map)
.ports = tegra186_main_ports,
.name = "tegra186-gpio",
.instance = 0,
+ .num_irqs_per_bank = 1,
};
#define TEGRA186_AON_GPIO_PORT(_name, _bank, _port, _pins) \
.ports = tegra186_aon_ports,
.name = "tegra186-gpio-aon",
.instance = 1,
+ .num_irqs_per_bank = 1,
};
#define TEGRA194_MAIN_GPIO_PORT(_name, _bank, _port, _pins) \
.ports = tegra194_main_ports,
.name = "tegra194-gpio",
.instance = 0,
+ .num_irqs_per_bank = 8,
.num_pin_ranges = ARRAY_SIZE(tegra194_main_pin_ranges),
.pin_ranges = tegra194_main_pin_ranges,
.pinmux = "nvidia,tegra194-pinmux",
.ports = tegra194_aon_ports,
.name = "tegra194-gpio-aon",
.instance = 1,
+ .num_irqs_per_bank = 8,
};
static const struct of_device_id tegra186_gpio_of_match[] = {
MODULE_AUTHOR("Nicolas Saenz Julienne <nicolassaenzj@gmail.com>");
MODULE_DESCRIPTION("GPO interface for TPS65218 PMICs");
MODULE_LICENSE("GPL v2");
-MODULE_ALIAS("platform:tps65218-gpio");
static void uniphier_gpio_irq_mask(struct irq_data *data)
{
- struct uniphier_gpio_priv *priv = data->chip_data;
- u32 mask = BIT(data->hwirq);
+ struct uniphier_gpio_priv *priv = irq_data_get_irq_chip_data(data);
+ u32 mask = BIT(irqd_to_hwirq(data));
uniphier_gpio_reg_update(priv, UNIPHIER_GPIO_IRQ_EN, mask, 0);
static void uniphier_gpio_irq_unmask(struct irq_data *data)
{
- struct uniphier_gpio_priv *priv = data->chip_data;
- u32 mask = BIT(data->hwirq);
+ struct uniphier_gpio_priv *priv = irq_data_get_irq_chip_data(data);
+ u32 mask = BIT(irqd_to_hwirq(data));
uniphier_gpio_reg_update(priv, UNIPHIER_GPIO_IRQ_EN, mask, mask);
static int uniphier_gpio_irq_set_type(struct irq_data *data, unsigned int type)
{
- struct uniphier_gpio_priv *priv = data->chip_data;
- u32 mask = BIT(data->hwirq);
+ struct uniphier_gpio_priv *priv = irq_data_get_irq_chip_data(data);
+ u32 mask = BIT(irqd_to_hwirq(data));
u32 val = 0;
if (type == IRQ_TYPE_EDGE_BOTH) {
struct uniphier_gpio_priv *priv = domain->host_data;
struct gpio_chip *chip = &priv->chip;
- return gpiochip_lock_as_irq(chip, data->hwirq + UNIPHIER_GPIO_IRQ_OFFSET);
+ return gpiochip_lock_as_irq(chip,
+ irqd_to_hwirq(data) + UNIPHIER_GPIO_IRQ_OFFSET);
}
static void uniphier_gpio_irq_domain_deactivate(struct irq_domain *domain,
struct uniphier_gpio_priv *priv = domain->host_data;
struct gpio_chip *chip = &priv->chip;
- gpiochip_unlock_as_irq(chip, data->hwirq + UNIPHIER_GPIO_IRQ_OFFSET);
+ gpiochip_unlock_as_irq(chip,
+ irqd_to_hwirq(data) + UNIPHIER_GPIO_IRQ_OFFSET);
}
static const struct irq_domain_ops uniphier_gpio_irq_domain_ops = {
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/mutex.h>
+#include <linux/spinlock.h>
#include <linux/virtio_config.h>
#include <uapi/linux/virtio_gpio.h>
#include <uapi/linux/virtio_ids.h>
unsigned int rxlen;
};
+struct vgpio_irq_line {
+ u8 type;
+ bool disabled;
+ bool masked;
+ bool queued;
+ bool update_pending;
+ bool queue_pending;
+
+ struct virtio_gpio_irq_request ireq ____cacheline_aligned;
+ struct virtio_gpio_irq_response ires ____cacheline_aligned;
+};
+
struct virtio_gpio {
struct virtio_device *vdev;
struct mutex lock; /* Protects virtqueue operation */
struct gpio_chip gc;
struct virtio_gpio_line *lines;
struct virtqueue *request_vq;
+
+ /* irq support */
+ struct virtqueue *event_vq;
+ struct mutex irq_lock; /* Protects irq operation */
+ raw_spinlock_t eventq_lock; /* Protects queuing of the buffer */
+ struct vgpio_irq_line *irq_lines;
};
static int _virtio_gpio_req(struct virtio_gpio *vgpio, u16 type, u16 gpio,
virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_SET_VALUE, gpio, value, NULL);
}
+/* Interrupt handling */
+static void virtio_gpio_irq_prepare(struct virtio_gpio *vgpio, u16 gpio)
+{
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[gpio];
+ struct virtio_gpio_irq_request *ireq = &irq_line->ireq;
+ struct virtio_gpio_irq_response *ires = &irq_line->ires;
+ struct scatterlist *sgs[2], req_sg, res_sg;
+ int ret;
+
+ if (WARN_ON(irq_line->queued || irq_line->masked || irq_line->disabled))
+ return;
+
+ ireq->gpio = cpu_to_le16(gpio);
+ sg_init_one(&req_sg, ireq, sizeof(*ireq));
+ sg_init_one(&res_sg, ires, sizeof(*ires));
+ sgs[0] = &req_sg;
+ sgs[1] = &res_sg;
+
+ ret = virtqueue_add_sgs(vgpio->event_vq, sgs, 1, 1, irq_line, GFP_ATOMIC);
+ if (ret) {
+ dev_err(&vgpio->vdev->dev, "failed to add request to eventq\n");
+ return;
+ }
+
+ irq_line->queued = true;
+ virtqueue_kick(vgpio->event_vq);
+}
+
+static void virtio_gpio_irq_enable(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+ raw_spin_lock(&vgpio->eventq_lock);
+ irq_line->disabled = false;
+ irq_line->masked = false;
+ irq_line->queue_pending = true;
+ raw_spin_unlock(&vgpio->eventq_lock);
+
+ irq_line->update_pending = true;
+}
+
+static void virtio_gpio_irq_disable(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+ raw_spin_lock(&vgpio->eventq_lock);
+ irq_line->disabled = true;
+ irq_line->masked = true;
+ irq_line->queue_pending = false;
+ raw_spin_unlock(&vgpio->eventq_lock);
+
+ irq_line->update_pending = true;
+}
+
+static void virtio_gpio_irq_mask(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+ raw_spin_lock(&vgpio->eventq_lock);
+ irq_line->masked = true;
+ raw_spin_unlock(&vgpio->eventq_lock);
+}
+
+static void virtio_gpio_irq_unmask(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+ raw_spin_lock(&vgpio->eventq_lock);
+ irq_line->masked = false;
+
+ /* Queue the buffer unconditionally on unmask */
+ virtio_gpio_irq_prepare(vgpio, d->hwirq);
+ raw_spin_unlock(&vgpio->eventq_lock);
+}
+
+static int virtio_gpio_irq_set_type(struct irq_data *d, unsigned int type)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+
+ switch (type) {
+ case IRQ_TYPE_EDGE_RISING:
+ type = VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING;
+ break;
+ case IRQ_TYPE_EDGE_FALLING:
+ type = VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING;
+ break;
+ case IRQ_TYPE_EDGE_BOTH:
+ type = VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH;
+ break;
+ case IRQ_TYPE_LEVEL_LOW:
+ type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW;
+ break;
+ case IRQ_TYPE_LEVEL_HIGH:
+ type = VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH;
+ break;
+ default:
+ dev_err(&vgpio->vdev->dev, "unsupported irq type: %u\n", type);
+ return -EINVAL;
+ }
+
+ irq_line->type = type;
+ irq_line->update_pending = true;
+
+ return 0;
+}
+
+static void virtio_gpio_irq_bus_lock(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+
+ mutex_lock(&vgpio->irq_lock);
+}
+
+static void virtio_gpio_irq_bus_sync_unlock(struct irq_data *d)
+{
+ struct gpio_chip *gc = irq_data_get_irq_chip_data(d);
+ struct virtio_gpio *vgpio = gpiochip_get_data(gc);
+ struct vgpio_irq_line *irq_line = &vgpio->irq_lines[d->hwirq];
+ u8 type = irq_line->disabled ? VIRTIO_GPIO_IRQ_TYPE_NONE : irq_line->type;
+ unsigned long flags;
+
+ if (irq_line->update_pending) {
+ irq_line->update_pending = false;
+ virtio_gpio_req(vgpio, VIRTIO_GPIO_MSG_IRQ_TYPE, d->hwirq, type,
+ NULL);
+
+ /* Queue the buffer only after interrupt is enabled */
+ raw_spin_lock_irqsave(&vgpio->eventq_lock, flags);
+ if (irq_line->queue_pending) {
+ irq_line->queue_pending = false;
+ virtio_gpio_irq_prepare(vgpio, d->hwirq);
+ }
+ raw_spin_unlock_irqrestore(&vgpio->eventq_lock, flags);
+ }
+
+ mutex_unlock(&vgpio->irq_lock);
+}
+
+static struct irq_chip vgpio_irq_chip = {
+ .name = "virtio-gpio",
+ .irq_enable = virtio_gpio_irq_enable,
+ .irq_disable = virtio_gpio_irq_disable,
+ .irq_mask = virtio_gpio_irq_mask,
+ .irq_unmask = virtio_gpio_irq_unmask,
+ .irq_set_type = virtio_gpio_irq_set_type,
+
+ /* These are required to implement irqchip for slow busses */
+ .irq_bus_lock = virtio_gpio_irq_bus_lock,
+ .irq_bus_sync_unlock = virtio_gpio_irq_bus_sync_unlock,
+};
+
+static bool ignore_irq(struct virtio_gpio *vgpio, int gpio,
+ struct vgpio_irq_line *irq_line)
+{
+ bool ignore = false;
+
+ raw_spin_lock(&vgpio->eventq_lock);
+ irq_line->queued = false;
+
+ /* Interrupt is disabled currently */
+ if (irq_line->masked || irq_line->disabled) {
+ ignore = true;
+ goto unlock;
+ }
+
+ /*
+ * Buffer is returned as the interrupt was disabled earlier, but is
+ * enabled again now. Requeue the buffers.
+ */
+ if (irq_line->ires.status == VIRTIO_GPIO_IRQ_STATUS_INVALID) {
+ virtio_gpio_irq_prepare(vgpio, gpio);
+ ignore = true;
+ goto unlock;
+ }
+
+ if (WARN_ON(irq_line->ires.status != VIRTIO_GPIO_IRQ_STATUS_VALID))
+ ignore = true;
+
+unlock:
+ raw_spin_unlock(&vgpio->eventq_lock);
+
+ return ignore;
+}
+
+static void virtio_gpio_event_vq(struct virtqueue *vq)
+{
+ struct virtio_gpio *vgpio = vq->vdev->priv;
+ struct device *dev = &vgpio->vdev->dev;
+ struct vgpio_irq_line *irq_line;
+ int gpio, ret;
+ unsigned int len;
+
+ while (true) {
+ irq_line = virtqueue_get_buf(vgpio->event_vq, &len);
+ if (!irq_line)
+ break;
+
+ if (len != sizeof(irq_line->ires)) {
+ dev_err(dev, "irq with incorrect length (%u : %u)\n",
+ len, (unsigned int)sizeof(irq_line->ires));
+ continue;
+ }
+
+ /*
+ * Find GPIO line number from the offset of irq_line within the
+ * irq_lines block. We can also get GPIO number from
+ * irq-request, but better not to rely on a buffer returned by
+ * remote.
+ */
+ gpio = irq_line - vgpio->irq_lines;
+ WARN_ON(gpio >= vgpio->gc.ngpio);
+
+ if (unlikely(ignore_irq(vgpio, gpio, irq_line)))
+ continue;
+
+ ret = generic_handle_domain_irq(vgpio->gc.irq.domain, gpio);
+ if (ret)
+ dev_err(dev, "failed to handle interrupt: %d\n", ret);
+ }
+}
+
static void virtio_gpio_request_vq(struct virtqueue *vq)
{
struct virtio_gpio_line *line;
static int virtio_gpio_alloc_vqs(struct virtio_gpio *vgpio,
struct virtio_device *vdev)
{
- const char * const names[] = { "requestq" };
+ const char * const names[] = { "requestq", "eventq" };
vq_callback_t *cbs[] = {
virtio_gpio_request_vq,
+ virtio_gpio_event_vq,
};
- struct virtqueue *vqs[1] = { NULL };
+ struct virtqueue *vqs[2] = { NULL, NULL };
int ret;
- ret = virtio_find_vqs(vdev, 1, vqs, cbs, names, NULL);
+ ret = virtio_find_vqs(vdev, vgpio->irq_lines ? 2 : 1, vqs, cbs, names, NULL);
if (ret) {
dev_err(&vdev->dev, "failed to find vqs: %d\n", ret);
return ret;
if (!vqs[0]) {
dev_err(&vdev->dev, "failed to find requestq vq\n");
- return -ENODEV;
+ goto out;
}
vgpio->request_vq = vqs[0];
+ if (vgpio->irq_lines && !vqs[1]) {
+ dev_err(&vdev->dev, "failed to find eventq vq\n");
+ goto out;
+ }
+ vgpio->event_vq = vqs[1];
+
return 0;
+
+out:
+ if (vqs[0] || vqs[1])
+ virtio_gpio_free_vqs(vdev);
+
+ return -ENODEV;
}
static const char **virtio_gpio_get_names(struct virtio_gpio *vgpio,
vgpio->gc.owner = THIS_MODULE;
vgpio->gc.can_sleep = true;
+ /* Interrupt support */
+ if (virtio_has_feature(vdev, VIRTIO_GPIO_F_IRQ)) {
+ vgpio->irq_lines = devm_kcalloc(dev, ngpio, sizeof(*vgpio->irq_lines), GFP_KERNEL);
+ if (!vgpio->irq_lines)
+ return -ENOMEM;
+
+ /* The event comes from the outside so no parent handler */
+ vgpio->gc.irq.parent_handler = NULL;
+ vgpio->gc.irq.num_parents = 0;
+ vgpio->gc.irq.parents = NULL;
+ vgpio->gc.irq.default_type = IRQ_TYPE_NONE;
+ vgpio->gc.irq.handler = handle_level_irq;
+ vgpio->gc.irq.chip = &vgpio_irq_chip;
+
+ for (i = 0; i < ngpio; i++) {
+ vgpio->irq_lines[i].type = VIRTIO_GPIO_IRQ_TYPE_NONE;
+ vgpio->irq_lines[i].disabled = true;
+ vgpio->irq_lines[i].masked = true;
+ }
+
+ mutex_init(&vgpio->irq_lock);
+ raw_spin_lock_init(&vgpio->eventq_lock);
+ }
+
ret = virtio_gpio_alloc_vqs(vgpio, vdev);
if (ret)
return ret;
};
MODULE_DEVICE_TABLE(virtio, id_table);
+static const unsigned int features[] = {
+ VIRTIO_GPIO_F_IRQ,
+};
+
static struct virtio_driver virtio_gpio_driver = {
+ .feature_table = features,
+ .feature_table_size = ARRAY_SIZE(features),
.id_table = id_table,
.probe = virtio_gpio_probe,
.remove = virtio_gpio_remove,
static int __maybe_unused xgpio_runtime_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct xgpio_instance *gpio = platform_get_drvdata(pdev);
+ struct xgpio_instance *gpio = dev_get_drvdata(dev);
clk_disable(gpio->clk);
static int __maybe_unused xgpio_runtime_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct xgpio_instance *gpio = platform_get_drvdata(pdev);
+ struct xgpio_instance *gpio = dev_get_drvdata(dev);
return clk_enable(gpio->clk);
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for the ps-mode pin configuration.
+ *
+ * Copyright (c) 2021 Xilinx, Inc.
+ */
+
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/gpio/driver.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/firmware/xlnx-zynqmp.h>
+
+/* 4-bit boot mode pins */
+#define MODE_PINS 4
+
+/**
+ * modepin_gpio_get_value - Get the state of the specified pin of GPIO device
+ * @chip: gpio_chip instance to be worked on
+ * @pin: gpio pin number within the device
+ *
+ * This function reads the state of the specified pin of the GPIO device.
+ *
+ * Return: 0 if the pin is low, 1 if pin is high, -EINVAL wrong pin configured
+ * or error value.
+ */
+static int modepin_gpio_get_value(struct gpio_chip *chip, unsigned int pin)
+{
+ u32 regval = 0;
+ int ret;
+
+ ret = zynqmp_pm_bootmode_read(®val);
+ if (ret)
+ return ret;
+
+ /* When [0:3] corresponding bit is set, then read output bit [8:11],
+ * if the bit is clear then read input bit [4:7] for status or value.
+ */
+ if (regval & BIT(pin))
+ return !!(regval & BIT(pin + 8));
+ else
+ return !!(regval & BIT(pin + 4));
+}
+
+/**
+ * modepin_gpio_set_value - Modify the state of the pin with specified value
+ * @chip: gpio_chip instance to be worked on
+ * @pin: gpio pin number within the device
+ * @state: value used to modify the state of the specified pin
+ *
+ * This function reads the state of the specified pin of the GPIO device, mask
+ * with the capture state of GPIO pin, and update pin of GPIO device.
+ *
+ * Return: None.
+ */
+static void modepin_gpio_set_value(struct gpio_chip *chip, unsigned int pin,
+ int state)
+{
+ u32 bootpin_val = 0;
+ int ret;
+
+ zynqmp_pm_bootmode_read(&bootpin_val);
+
+ /* Configure pin as an output by set bit [0:3] */
+ bootpin_val |= BIT(pin);
+
+ if (state)
+ bootpin_val |= BIT(pin + 8);
+ else
+ bootpin_val &= ~BIT(pin + 8);
+
+ /* Configure bootpin value */
+ ret = zynqmp_pm_bootmode_write(bootpin_val);
+ if (ret)
+ pr_err("modepin: set value error %d for pin %d\n", ret, pin);
+}
+
+/**
+ * modepin_gpio_dir_in - Set the direction of the specified GPIO pin as input
+ * @chip: gpio_chip instance to be worked on
+ * @pin: gpio pin number within the device
+ *
+ * Return: 0 always
+ */
+static int modepin_gpio_dir_in(struct gpio_chip *chip, unsigned int pin)
+{
+ return 0;
+}
+
+/**
+ * modepin_gpio_dir_out - Set the direction of the specified GPIO pin as output
+ * @chip: gpio_chip instance to be worked on
+ * @pin: gpio pin number within the device
+ * @state: value to be written to specified pin
+ *
+ * Return: 0 always
+ */
+static int modepin_gpio_dir_out(struct gpio_chip *chip, unsigned int pin,
+ int state)
+{
+ return 0;
+}
+
+/**
+ * modepin_gpio_probe - Initialization method for modepin_gpio
+ * @pdev: platform device instance
+ *
+ * Return: 0 on success, negative error otherwise.
+ */
+static int modepin_gpio_probe(struct platform_device *pdev)
+{
+ struct gpio_chip *chip;
+ int status;
+
+ chip = devm_kzalloc(&pdev->dev, sizeof(*chip), GFP_KERNEL);
+ if (!chip)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, chip);
+
+ /* configure the gpio chip */
+ chip->base = -1;
+ chip->ngpio = MODE_PINS;
+ chip->owner = THIS_MODULE;
+ chip->parent = &pdev->dev;
+ chip->get = modepin_gpio_get_value;
+ chip->set = modepin_gpio_set_value;
+ chip->direction_input = modepin_gpio_dir_in;
+ chip->direction_output = modepin_gpio_dir_out;
+ chip->label = dev_name(&pdev->dev);
+
+ /* modepin gpio registration */
+ status = devm_gpiochip_add_data(&pdev->dev, chip, chip);
+ if (status)
+ return dev_err_probe(&pdev->dev, status,
+ "Failed to add GPIO chip\n");
+
+ return status;
+}
+
+static const struct of_device_id modepin_platform_id[] = {
+ { .compatible = "xlnx,zynqmp-gpio-modepin", },
+ { }
+};
+
+static struct platform_driver modepin_platform_driver = {
+ .driver = {
+ .name = "modepin-gpio",
+ .of_match_table = modepin_platform_id,
+ },
+ .probe = modepin_gpio_probe,
+};
+
+module_platform_driver(modepin_platform_driver);
+
+MODULE_AUTHOR("Piyush Mehta <piyush.mehta@xilinx.com>");
+MODULE_DESCRIPTION("ZynqMP Boot PS_MODE Configuration");
+MODULE_LICENSE("GPL v2");
This has the potential to use a lot of memory and print some very
large kernel messages. If in doubt, say "N".
+config DRM_DEBUG_MODESET_LOCK
+ bool "Enable backtrace history for lock contention"
+ depends on STACKTRACE_SUPPORT
+ depends on DEBUG_KERNEL
+ depends on EXPERT
+ select STACKDEPOT
+ default y if DEBUG_WW_MUTEX_SLOWPATH
+ help
+ Enable debug tracing of failures to gracefully handle drm modeset lock
+ contention. A history of each drm modeset lock path hitting -EDEADLK
+ will be saved until gracefully handled, and the backtrace will be
+ printed when attempting to lock a contended lock.
+
+ If in doubt, say "N".
+
config DRM_FBDEV_EMULATION
bool "Enable legacy fbdev support for your modesetting driver"
- depends on DRM
- depends on FB=y || FB=DRM
- select DRM_KMS_HELPER
+ depends on DRM_KMS_HELPER
+ depends on FB=y || FB=DRM_KMS_HELPER
select FB_CFB_FILLRECT
select FB_CFB_COPYAREA
select FB_CFB_IMAGEBLIT
void amdgpu_amdkfd_gpuvm_init_mem_limits(void);
void amdgpu_amdkfd_gpuvm_destroy_cb(struct amdgpu_device *adev,
struct amdgpu_vm *vm);
-void amdgpu_amdkfd_unreserve_memory_limit(struct amdgpu_bo *bo);
+void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo);
void amdgpu_amdkfd_reserve_system_mem(uint64_t size);
#else
static inline
}
static inline
-void amdgpu_amdkfd_unreserve_memory_limit(struct amdgpu_bo *bo)
+void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo)
{
}
#endif
spin_unlock(&kfd_mem_limit.mem_limit_lock);
}
-void amdgpu_amdkfd_unreserve_memory_limit(struct amdgpu_bo *bo)
+void amdgpu_amdkfd_release_notify(struct amdgpu_bo *bo)
{
struct amdgpu_device *adev = amdgpu_ttm_adev(bo->tbo.bdev);
u32 domain = bo->preferred_domains;
}
unreserve_mem_limit(adev, amdgpu_bo_size(bo), domain, sg);
+
+ kfree(bo->kfd_bo);
}
}
/* Add BO to VM internal data structures */
+ ret = amdgpu_bo_reserve(bo[i], false);
+ if (ret) {
+ pr_debug("Unable to reserve BO during memory attach");
+ goto unwind;
+ }
attachment[i]->bo_va = amdgpu_vm_bo_add(adev, vm, bo[i]);
+ amdgpu_bo_unreserve(bo[i]);
if (unlikely(!attachment[i]->bo_va)) {
ret = -ENOMEM;
pr_err("Failed to add BO object to VM. ret == %d\n",
ret);
goto unwind;
}
-
attachment[i]->va = va;
attachment[i]->pte_flags = get_pte_flags(adev, mem);
attachment[i]->adev = adev;
if (!attachment[i])
continue;
if (attachment[i]->bo_va) {
+ amdgpu_bo_reserve(bo[i], true);
amdgpu_vm_bo_rmv(adev, attachment[i]->bo_va);
+ amdgpu_bo_unreserve(bo[i]);
list_del(&attachment[i]->list);
}
if (bo[i])
pr_debug("Release VA 0x%llx - 0x%llx\n", mem->va,
mem->va + bo_size * (1 + mem->aql_queue));
- ret = unreserve_bo_and_vms(&ctx, false, false);
-
/* Remove from VM internal data structures */
list_for_each_entry_safe(entry, tmp, &mem->attachments, list)
kfd_mem_detach(entry);
+ ret = unreserve_bo_and_vms(&ctx, false, false);
+
/* Free the sync object */
amdgpu_sync_free(&mem->sync);
drm_vma_node_revoke(&mem->bo->tbo.base.vma_node, drm_priv);
if (mem->dmabuf)
dma_buf_put(mem->dmabuf);
- drm_gem_object_put(&mem->bo->tbo.base);
mutex_destroy(&mem->lock);
- kfree(mem);
+
+ /* If this releases the last reference, it will end up calling
+ * amdgpu_amdkfd_release_notify and kfree the mem struct. That's why
+ * this needs to be the last call here.
+ */
+ drm_gem_object_put(&mem->bo->tbo.base);
return ret;
}
amdgpu_connector_get_edid(connector);
ret = amdgpu_connector_ddc_get_modes(connector);
+ amdgpu_get_native_mode(connector);
return ret;
}
if (!adev->gmc.xgmi.pending_reset)
amdgpu_amdkfd_device_init(adev);
- r = amdgpu_amdkfd_resume_iommu(adev);
- if (r)
- goto init_failed;
-
amdgpu_fru_get_product_info(adev);
init_failed:
{
switch (asic_type) {
#if defined(CONFIG_DRM_AMD_DC)
-#if defined(CONFIG_DRM_AMD_DC_SI)
case CHIP_TAHITI:
case CHIP_PITCAIRN:
case CHIP_VERDE:
case CHIP_OLAND:
+ /*
+ * We have systems in the wild with these ASICs that require
+ * LVDS and VGA support which is not supported with DC.
+ *
+ * Fallback to the non-DC driver here by default so as not to
+ * cause regressions.
+ */
+#if defined(CONFIG_DRM_AMD_DC_SI)
+ return amdgpu_dc > 0;
+#else
+ return false;
#endif
case CHIP_BONAIRE:
case CHIP_KAVERI:
adev->rmmio_size = pci_resource_len(adev->pdev, 2);
}
+ for (i = 0; i < AMD_IP_BLOCK_TYPE_NUM; i++)
+ atomic_set(&adev->pm.pwr_state[i], POWER_STATE_UNKNOWN);
+
adev->rmmio = ioremap(adev->rmmio_base, adev->rmmio_size);
if (adev->rmmio == NULL) {
return -ENOMEM;
if (r)
return r;
- amdgpu_amdkfd_pre_reset(adev);
-
/* Resume IP prior to SMC */
r = amdgpu_device_ip_reinit_early_sriov(adev);
if (r)
/* clear job's guilty and depend the folowing step to decide the real one */
drm_sched_reset_karma(s_job);
+ /* for the real bad job, it will be resubmitted twice, adding a dma_fence_get
+ * to make sure fence is balanced */
+ dma_fence_get(s_job->s_fence->parent);
drm_sched_resubmit_jobs_ext(&ring->sched, 1);
ret = dma_fence_wait_timeout(s_job->s_fence->parent, false, ring->sched.timeout);
/* got the hw fence, signal finished fence */
atomic_dec(ring->sched.score);
+ dma_fence_put(s_job->s_fence->parent);
dma_fence_get(&s_job->s_fence->finished);
dma_fence_signal(&s_job->s_fence->finished);
dma_fence_put(&s_job->s_fence->finished);
cancel_delayed_work_sync(&tmp_adev->delayed_init_work);
- if (!amdgpu_sriov_vf(tmp_adev))
- amdgpu_amdkfd_pre_reset(tmp_adev);
+ amdgpu_amdkfd_pre_reset(tmp_adev);
/*
* Mark these ASICs to be reseted as untracked first
amdgpu_device_ip_block_add(adev, &nv_common_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add common ip block(GC_HWIP:0x%x)\n",
+ adev->ip_versions[GC_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &gmc_v10_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add gmc ip block(GC_HWIP:0x%x)\n",
+ adev->ip_versions[GC_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &navi10_ih_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add ih ip block(OSSSYS_HWIP:0x%x)\n",
+ adev->ip_versions[OSSSYS_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &psp_v13_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add psp ip block(MP0_HWIP:0x%x)\n",
+ adev->ip_versions[MP0_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &smu_v13_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add smu ip block(MP1_HWIP:0x%x)\n",
+ adev->ip_versions[MP1_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &dm_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add dm ip block(DCE_HWIP:0x%x)\n",
+ adev->ip_versions[DCE_HWIP][0]);
return -EINVAL;
}
} else if (adev->ip_versions[DCI_HWIP][0]) {
amdgpu_device_ip_block_add(adev, &dm_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add dm ip block(DCI_HWIP:0x%x)\n",
+ adev->ip_versions[DCI_HWIP][0]);
return -EINVAL;
}
#endif
amdgpu_device_ip_block_add(adev, &gfx_v10_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add gfx ip block(GC_HWIP:0x%x)\n",
+ adev->ip_versions[GC_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &sdma_v5_2_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add sdma ip block(SDMA0_HWIP:0x%x)\n",
+ adev->ip_versions[SDMA0_HWIP][0]);
return -EINVAL;
}
return 0;
amdgpu_device_ip_block_add(adev, &uvd_v7_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add uvd v7 ip block(UVD_HWIP:0x%x)\n",
+ adev->ip_versions[UVD_HWIP][0]);
return -EINVAL;
}
switch (adev->ip_versions[VCE_HWIP][0]) {
amdgpu_device_ip_block_add(adev, &vce_v4_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add VCE v4 ip block(VCE_HWIP:0x%x)\n",
+ adev->ip_versions[VCE_HWIP][0]);
return -EINVAL;
}
} else {
case IP_VERSION(2, 0, 2):
case IP_VERSION(2, 2, 0):
amdgpu_device_ip_block_add(adev, &vcn_v2_0_ip_block);
- amdgpu_device_ip_block_add(adev, &jpeg_v2_0_ip_block);
+ if (!amdgpu_sriov_vf(adev))
+ amdgpu_device_ip_block_add(adev, &jpeg_v2_0_ip_block);
break;
case IP_VERSION(2, 0, 3):
break;
break;
case IP_VERSION(3, 0, 0):
case IP_VERSION(3, 0, 16):
+ case IP_VERSION(3, 0, 64):
case IP_VERSION(3, 1, 1):
case IP_VERSION(3, 0, 2):
amdgpu_device_ip_block_add(adev, &vcn_v3_0_ip_block);
amdgpu_device_ip_block_add(adev, &vcn_v3_0_ip_block);
break;
default:
+ dev_err(adev->dev,
+ "Failed to add vcn/jpeg ip block(UVD_HWIP:0x%x)\n",
+ adev->ip_versions[UVD_HWIP][0]);
return -EINVAL;
}
}
goto unlock;
}
- ret = ttm_bo_vm_fault_reserved(vmf, vmf->vma->vm_page_prot,
- TTM_BO_VM_NUM_PREFAULT, 1);
- drm_dev_exit(idx);
+ ret = ttm_bo_vm_fault_reserved(vmf, vmf->vma->vm_page_prot,
+ TTM_BO_VM_NUM_PREFAULT);
+
+ drm_dev_exit(idx);
} else {
ret = ttm_bo_vm_dummy_page(vmf, vmf->vma->vm_page_prot);
}
struct drm_amdgpu_info_firmware fw_info;
struct drm_amdgpu_query_fw query_fw;
struct atom_context *ctx = adev->mode_info.atom_context;
+ uint8_t smu_minor, smu_debug;
+ uint16_t smu_major;
int ret, i;
static const char *ta_fw_name[TA_FW_TYPE_MAX_INDEX] = {
ret = amdgpu_firmware_info(&fw_info, &query_fw, adev);
if (ret)
return ret;
- seq_printf(m, "SMC feature version: %u, firmware version: 0x%08x\n",
- fw_info.feature, fw_info.ver);
+ smu_major = (fw_info.ver >> 16) & 0xffff;
+ smu_minor = (fw_info.ver >> 8) & 0xff;
+ smu_debug = (fw_info.ver >> 0) & 0xff;
+ seq_printf(m, "SMC feature version: %u, firmware version: 0x%08x (%d.%d.%d)\n",
+ fw_info.feature, fw_info.ver, smu_major, smu_minor, smu_debug);
/* SDMA */
query_fw.fw_type = AMDGPU_INFO_FW_SDMA;
abo = ttm_to_amdgpu_bo(bo);
if (abo->kfd_bo)
- amdgpu_amdkfd_unreserve_memory_limit(abo);
+ amdgpu_amdkfd_release_notify(abo);
/* We only remove the fence if the resv has individualized. */
WARN_ON_ONCE(bo->type == ttm_bo_type_kernel
adev->vcn.indirect_sram = true;
break;
case IP_VERSION(3, 0, 0):
+ case IP_VERSION(3, 0, 64):
if (adev->ip_versions[GC_HWIP][0] == IP_VERSION(10, 3, 0))
fw_name = FIRMWARE_SIENNA_CICHLID;
else
"%s", "xgmi_hive_info");
if (ret) {
dev_err(adev->dev, "XGMI: failed initializing kobject for xgmi hive\n");
+ kobject_put(&hive->kobj);
kfree(hive);
hive = NULL;
goto pro_end;
for (i = 0; i < ARRAY_SIZE(xgmi23_pcs_err_status_reg_aldebaran); i++)
pcs_clear_status(adev,
xgmi23_pcs_err_status_reg_aldebaran[i]);
- for (i = 0; i < ARRAY_SIZE(xgmi23_pcs_err_status_reg_aldebaran); i++)
+ for (i = 0; i < ARRAY_SIZE(xgmi3x16_pcs_err_status_reg_aldebaran); i++)
pcs_clear_status(adev,
- xgmi23_pcs_err_status_reg_aldebaran[i]);
+ xgmi3x16_pcs_err_status_reg_aldebaran[i]);
for (i = 0; i < ARRAY_SIZE(walf_pcs_err_status_reg_aldebaran); i++)
pcs_clear_status(adev,
walf_pcs_err_status_reg_aldebaran[i]);
static void gfx_v10_0_update_spm_vmid(struct amdgpu_device *adev, unsigned vmid)
{
u32 reg, data;
+
+ amdgpu_gfx_off_ctrl(adev, false);
+
/* not for *_SOC15 */
reg = SOC15_REG_OFFSET(GC, 0, mmRLC_SPM_MC_CNTL);
if (amdgpu_sriov_is_pp_one_vf(adev))
WREG32_SOC15_NO_KIQ(GC, 0, mmRLC_SPM_MC_CNTL, data);
else
WREG32_SOC15(GC, 0, mmRLC_SPM_MC_CNTL, data);
+
+ amdgpu_gfx_off_ctrl(adev, true);
}
static bool gfx_v10_0_check_rlcg_range(struct amdgpu_device *adev,
if (enable && (adev->pg_flags & AMD_PG_SUPPORT_GFX_PG)) {
switch (adev->ip_versions[GC_HWIP][0]) {
case IP_VERSION(10, 3, 1):
- data = 0x4E20 & RLC_PG_DELAY_3__CGCG_ACTIVE_BEFORE_CGPG_MASK_Vangogh;
- WREG32_SOC15(GC, 0, mmRLC_PG_DELAY_3, data);
- break;
case IP_VERSION(10, 3, 3):
- data = 0x1388 & RLC_PG_DELAY_3__CGCG_ACTIVE_BEFORE_CGPG_MASK_Vangogh;
+ data = 0x4E20 & RLC_PG_DELAY_3__CGCG_ACTIVE_BEFORE_CGPG_MASK_Vangogh;
WREG32_SOC15(GC, 0, mmRLC_PG_DELAY_3, data);
break;
default:
{
u32 data;
+ amdgpu_gfx_off_ctrl(adev, false);
+
data = RREG32(mmRLC_SPM_VMID);
data &= ~RLC_SPM_VMID__RLC_SPM_VMID_MASK;
data |= (vmid & RLC_SPM_VMID__RLC_SPM_VMID_MASK) << RLC_SPM_VMID__RLC_SPM_VMID__SHIFT;
WREG32(mmRLC_SPM_VMID, data);
+
+ amdgpu_gfx_off_ctrl(adev, true);
}
static void gfx_v7_0_enable_cgcg(struct amdgpu_device *adev, bool enable)
{
u32 data;
+ amdgpu_gfx_off_ctrl(adev, false);
+
if (amdgpu_sriov_is_pp_one_vf(adev))
data = RREG32_NO_KIQ(mmRLC_SPM_VMID);
else
WREG32_NO_KIQ(mmRLC_SPM_VMID, data);
else
WREG32(mmRLC_SPM_VMID, data);
+
+ amdgpu_gfx_off_ctrl(adev, true);
}
static const struct amdgpu_rlc_funcs iceland_rlc_funcs = {
amdgpu_gfx_kiq_fini(adev);
gfx_v9_0_mec_fini(adev);
- amdgpu_bo_unref(&adev->gfx.rlc.clear_state_obj);
+ amdgpu_bo_free_kernel(&adev->gfx.rlc.clear_state_obj,
+ &adev->gfx.rlc.clear_state_gpu_addr,
+ (void **)&adev->gfx.rlc.cs_ptr);
if (adev->flags & AMD_IS_APU) {
amdgpu_bo_free_kernel(&adev->gfx.rlc.cp_table_obj,
&adev->gfx.rlc.cp_table_gpu_addr,
{
u32 reg, data;
+ amdgpu_gfx_off_ctrl(adev, false);
+
reg = SOC15_REG_OFFSET(GC, 0, mmRLC_SPM_MC_CNTL);
if (amdgpu_sriov_is_pp_one_vf(adev))
data = RREG32_NO_KIQ(reg);
WREG32_SOC15_NO_KIQ(GC, 0, mmRLC_SPM_MC_CNTL, data);
else
WREG32_SOC15(GC, 0, mmRLC_SPM_MC_CNTL, data);
+
+ amdgpu_gfx_off_ctrl(adev, true);
}
static bool gfx_v9_0_check_rlcg_range(struct amdgpu_device *adev,
WREG32_SOC15_OFFSET(GC, 0, mmVM_CONTEXT0_CNTL,
i * hub->ctx_distance, 0);
+ if (amdgpu_sriov_vf(adev))
+ /* Avoid write to GMC registers */
+ return;
+
/* Setup TLB control */
tmp = RREG32_SOC15(GC, 0, mmMC_VM_MX_L1_TLB_CNTL);
tmp = REG_SET_FIELD(tmp, MC_VM_MX_L1_TLB_CNTL, ENABLE_L1_TLB, 0);
seg_size = REG_GET_FIELD(
RREG32_SOC15(GC, 0, mmMC_VM_XGMI_LFB_SIZE_ALDE),
MC_VM_XGMI_LFB_SIZE, PF_LFB_SIZE) << 24;
+ max_region =
+ REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL_ALDE, PF_MAX_REGION);
} else {
xgmi_lfb_cntl = RREG32_SOC15(GC, 0, mmMC_VM_XGMI_LFB_CNTL);
seg_size = REG_GET_FIELD(
RREG32_SOC15(GC, 0, mmMC_VM_XGMI_LFB_SIZE),
MC_VM_XGMI_LFB_SIZE, PF_LFB_SIZE) << 24;
+ max_region =
+ REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL, PF_MAX_REGION);
}
- max_region =
- REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL, PF_MAX_REGION);
switch (adev->asic_type) {
if (adev->gmc.xgmi.num_physical_nodes > max_num_physical_nodes)
return -EINVAL;
- adev->gmc.xgmi.physical_node_id =
- REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL,
- PF_LFB_REGION);
+ if (adev->asic_type == CHIP_ALDEBARAN) {
+ adev->gmc.xgmi.physical_node_id =
+ REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL_ALDE,
+ PF_LFB_REGION);
+ } else {
+ adev->gmc.xgmi.physical_node_id =
+ REG_GET_FIELD(xgmi_lfb_cntl, MC_VM_XGMI_LFB_CNTL,
+ PF_LFB_REGION);
+ }
if (adev->gmc.xgmi.physical_node_id > max_physical_node_id)
return -EINVAL;
{
switch (adev->ip_versions[UVD_HWIP][0]) {
case IP_VERSION(3, 0, 0):
+ case IP_VERSION(3, 0, 64):
if (amdgpu_sriov_vf(adev)) {
if (encode)
*codecs = &sriov_sc_video_codecs_encode;
{
struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+ cancel_delayed_work_sync(&adev->uvd.idle_work);
+
+ if (RREG32(mmUVD_STATUS) != 0)
+ uvd_v6_0_stop(adev);
+
+ return 0;
+}
+
+static int uvd_v6_0_suspend(void *handle)
+{
+ int r;
+ struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
/*
* Proper cleanups before halting the HW engine:
* - cancel the delayed idle work
AMD_CG_STATE_GATE);
}
- if (RREG32(mmUVD_STATUS) != 0)
- uvd_v6_0_stop(adev);
-
- return 0;
-}
-
-static int uvd_v6_0_suspend(void *handle)
-{
- int r;
- struct amdgpu_device *adev = (struct amdgpu_device *)handle;
-
r = uvd_v6_0_hw_fini(adev);
if (r)
return r;
static const struct kfd_device_info renoir_device_info = {
.asic_family = CHIP_RENOIR,
.asic_name = "renoir",
- .gfx_target_version = 90002,
+ .gfx_target_version = 90012,
.max_pasid_bits = 16,
.max_no_of_hqd = 24,
.doorbell_size = 8,
bool hanging;
dqm_lock(dqm);
+ if (!dqm->sched_running) {
+ dqm_unlock(dqm);
+ return 0;
+ }
+
if (!dqm->is_hws_hang)
unmap_queues_cpsch(dqm, KFD_UNMAP_QUEUES_FILTER_ALL_QUEUES, 0);
hanging = dqm->is_hws_hang || dqm->is_resetting;
if (!dqm->sched_running)
return 0;
- if (dqm->is_hws_hang)
+ if (dqm->is_hws_hang || dqm->is_resetting)
return -EIO;
if (!dqm->active_runlist)
return retval;
* 16MB are reserved for kernel use (CWSR trap handler and kernel IB
* for now).
*/
-#define SVM_USER_BASE 0x1000000ull
+#define SVM_USER_BASE (u64)(KFD_CWSR_TBA_TMA_SIZE + 2*PAGE_SIZE)
#define SVM_CWSR_BASE (SVM_USER_BASE - KFD_CWSR_TBA_TMA_SIZE)
#define SVM_IB_BASE (SVM_CWSR_BASE - PAGE_SIZE)
return cpages;
}
+static unsigned long svm_migrate_unsuccessful_pages(struct migrate_vma *migrate)
+{
+ unsigned long upages = 0;
+ unsigned long i;
+
+ for (i = 0; i < migrate->npages; i++) {
+ if (migrate->src[i] & MIGRATE_PFN_VALID &&
+ !(migrate->src[i] & MIGRATE_PFN_MIGRATE))
+ upages++;
+ }
+ return upages;
+}
+
static int
svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
struct migrate_vma *migrate, struct dma_fence **mfence,
migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]);
svm_migrate_get_vram_page(prange, migrate->dst[i]);
migrate->dst[i] = migrate_pfn(migrate->dst[i]);
- migrate->dst[i] |= MIGRATE_PFN_LOCKED;
src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE,
DMA_TO_DEVICE);
r = dma_mapping_error(dev, src[i]);
dst[i] >> PAGE_SHIFT, page_to_pfn(dpage));
migrate->dst[i] = migrate_pfn(page_to_pfn(dpage));
- migrate->dst[i] |= MIGRATE_PFN_LOCKED;
j++;
}
struct vm_area_struct *vma, uint64_t start, uint64_t end)
{
uint64_t npages = (end - start) >> PAGE_SHIFT;
+ unsigned long upages = npages;
+ unsigned long cpages = 0;
struct kfd_process_device *pdd;
struct dma_fence *mfence = NULL;
struct migrate_vma migrate;
- unsigned long cpages = 0;
dma_addr_t *scratch;
size_t size;
void *buf;
if (!cpages) {
pr_debug("failed collect migrate device pages [0x%lx 0x%lx]\n",
prange->start, prange->last);
+ upages = svm_migrate_unsuccessful_pages(&migrate);
goto out_free;
}
if (cpages != npages)
scratch, npages);
migrate_vma_pages(&migrate);
- pr_debug("successful/cpages/npages 0x%lx/0x%lx/0x%lx\n",
- svm_migrate_successful_pages(&migrate), cpages, migrate.npages);
+ upages = svm_migrate_unsuccessful_pages(&migrate);
+ pr_debug("unsuccessful/cpages/npages 0x%lx/0x%lx/0x%lx\n",
+ upages, cpages, migrate.npages);
svm_migrate_copy_done(adev, mfence);
migrate_vma_finalize(&migrate);
if (pdd)
WRITE_ONCE(pdd->page_out, pdd->page_out + cpages);
- return cpages;
+ return upages;
}
- return r;
+ return r ? r : upages;
}
/**
unsigned long addr;
unsigned long start;
unsigned long end;
- unsigned long cpages = 0;
+ unsigned long upages = 0;
long r = 0;
if (!prange->actual_loc) {
pr_debug("failed %ld to migrate\n", r);
break;
} else {
- cpages += r;
+ upages += r;
}
addr = next;
}
- if (cpages) {
+ if (!upages) {
svm_range_vram_node_free(prange);
prange->actual_loc = 0;
}
svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc,
struct mm_struct *mm)
{
- int r;
+ int r, retries = 3;
/*
* TODO: for both devices with PCIe large bar or on same xgmi hive, skip
pr_debug("from gpu 0x%x to gpu 0x%x\n", prange->actual_loc, best_loc);
- r = svm_migrate_vram_to_ram(prange, mm);
- if (r)
- return r;
+ do {
+ r = svm_migrate_vram_to_ram(prange, mm);
+ if (r)
+ return r;
+ } while (prange->actual_loc && --retries);
+
+ if (prange->actual_loc)
+ return -EDEADLK;
return svm_migrate_ram_to_vram(prange, best_loc, mm);
}
pr_debug("failed find process at fault address 0x%lx\n", addr);
return VM_FAULT_SIGBUS;
}
+ if (READ_ONCE(p->svms.faulting_task) == current) {
+ pr_debug("skipping ram migration\n");
+ kfd_unref_process(p);
+ return 0;
+ }
addr >>= PAGE_SHIFT;
pr_debug("CPU page fault svms 0x%p address 0x%lx\n", &p->svms, addr);
struct list_head deferred_range_list;
spinlock_t deferred_list_lock;
atomic_t evicted_ranges;
+ bool drain_pagefaults;
struct delayed_work restore_work;
DECLARE_BITMAP(bitmap_supported, MAX_GPU_INSTANCE);
+ struct task_struct *faulting_task;
};
/* Process data */
r = pdd->dev->dqm->ops.evict_process_queues(pdd->dev->dqm,
&pdd->qpd);
- if (r) {
+ /* evict return -EIO if HWS is hang or asic is resetting, in this case
+ * we would like to set all the queues to be in evicted state to prevent
+ * them been add back since they actually not be saved right now.
+ */
+ if (r && r != -EIO) {
pr_err("Failed to evict process queues\n");
goto fail;
}
next = min(vma->vm_end, end);
npages = (next - addr) >> PAGE_SHIFT;
+ WRITE_ONCE(p->svms.faulting_task, current);
r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL,
addr, npages, &hmm_range,
readonly, true, owner);
+ WRITE_ONCE(p->svms.faulting_task, NULL);
if (r) {
pr_debug("failed %d to get svm range pages\n", r);
goto unreserve_out;
pr_debug("prange 0x%p [0x%lx 0x%lx] op %d\n", prange,
prange->start, prange->last, prange->work_item.op);
- /* Make sure no stale retry fault coming after range is freed */
- if (prange->work_item.op == SVM_OP_UNMAP_RANGE)
- svm_range_drain_retry_fault(prange->svms);
-
mm = prange->work_item.mm;
+retry:
mmap_write_lock(mm);
mutex_lock(&svms->lock);
- /* Remove from deferred_list must be inside mmap write lock,
+ /* Checking for the need to drain retry faults must be in
+ * mmap write lock to serialize with munmap notifiers.
+ *
+ * Remove from deferred_list must be inside mmap write lock,
* otherwise, svm_range_list_lock_and_flush_work may hold mmap
* write lock, and continue because deferred_list is empty, then
* deferred_list handle is blocked by mmap write lock.
*/
spin_lock(&svms->deferred_list_lock);
+ if (unlikely(svms->drain_pagefaults)) {
+ svms->drain_pagefaults = false;
+ spin_unlock(&svms->deferred_list_lock);
+ mutex_unlock(&svms->lock);
+ mmap_write_unlock(mm);
+ svm_range_drain_retry_fault(svms);
+ goto retry;
+ }
list_del_init(&prange->deferred_list);
spin_unlock(&svms->deferred_list_lock);
struct mm_struct *mm, enum svm_work_list_ops op)
{
spin_lock(&svms->deferred_list_lock);
+ /* Make sure pending page faults are drained in the deferred worker
+ * before the range is freed to avoid straggler interrupts on
+ * unmapped memory causing "phantom faults".
+ */
+ if (op == SVM_OP_UNMAP_RANGE)
+ svms->drain_pagefaults = true;
/* if prange is on the deferred list */
if (!list_empty(&prange->deferred_list)) {
pr_debug("update exist prange 0x%p work op %d\n", prange, op);
* migration if actual loc is not best location, then update GPU page table
* mapping to the best location.
*
- * If vm fault gpu is range preferred loc, the best_loc is preferred loc.
+ * If the preferred loc is accessible by faulting GPU, use preferred loc.
* If vm fault gpu idx is on range ACCESSIBLE bitmap, best_loc is vm fault gpu
* If vm fault gpu idx is on range ACCESSIBLE_IN_PLACE bitmap, then
* if range actual loc is cpu, best_loc is cpu
struct amdgpu_device *adev,
int32_t *gpuidx)
{
- struct amdgpu_device *bo_adev;
+ struct amdgpu_device *bo_adev, *preferred_adev;
struct kfd_process *p;
uint32_t gpuid;
int r;
return -1;
}
- if (prange->preferred_loc == gpuid)
+ if (prange->preferred_loc == gpuid ||
+ prange->preferred_loc == KFD_IOCTL_SVM_LOCATION_SYSMEM) {
return prange->preferred_loc;
+ } else if (prange->preferred_loc != KFD_IOCTL_SVM_LOCATION_UNDEFINED) {
+ preferred_adev = svm_range_get_adev_by_id(prange,
+ prange->preferred_loc);
+ if (amdgpu_xgmi_same_hive(adev, preferred_adev))
+ return prange->preferred_loc;
+ /* fall through */
+ }
if (test_bit(*gpuidx, prange->bitmap_access))
return gpuid;
static int
svm_range_get_range_boundaries(struct kfd_process *p, int64_t addr,
- unsigned long *start, unsigned long *last)
+ unsigned long *start, unsigned long *last,
+ bool *is_heap_stack)
{
struct vm_area_struct *vma;
struct interval_tree_node *node;
pr_debug("VMA does not exist in address [0x%llx]\n", addr);
return -EFAULT;
}
+
+ *is_heap_stack = (vma->vm_start <= vma->vm_mm->brk &&
+ vma->vm_end >= vma->vm_mm->start_brk) ||
+ (vma->vm_start <= vma->vm_mm->start_stack &&
+ vma->vm_end >= vma->vm_mm->start_stack);
+
start_limit = max(vma->vm_start >> PAGE_SHIFT,
(unsigned long)ALIGN_DOWN(addr, 2UL << 8));
end_limit = min(vma->vm_end >> PAGE_SHIFT,
*start = start_limit;
*last = end_limit - 1;
- pr_debug("vma start: 0x%lx start: 0x%lx vma end: 0x%lx last: 0x%lx\n",
- vma->vm_start >> PAGE_SHIFT, *start,
- vma->vm_end >> PAGE_SHIFT, *last);
+ pr_debug("vma [0x%lx 0x%lx] range [0x%lx 0x%lx] is_heap_stack %d\n",
+ vma->vm_start >> PAGE_SHIFT, vma->vm_end >> PAGE_SHIFT,
+ *start, *last, *is_heap_stack);
return 0;
}
struct svm_range *prange = NULL;
unsigned long start, last;
uint32_t gpuid, gpuidx;
+ bool is_heap_stack;
uint64_t bo_s = 0;
uint64_t bo_l = 0;
int r;
- if (svm_range_get_range_boundaries(p, addr, &start, &last))
+ if (svm_range_get_range_boundaries(p, addr, &start, &last,
+ &is_heap_stack))
return NULL;
r = svm_range_check_vm(p, start, last, &bo_s, &bo_l);
return NULL;
}
+ if (is_heap_stack)
+ prange->preferred_loc = KFD_IOCTL_SVM_LOCATION_SYSMEM;
+
svm_range_add_to_svms(prange);
svm_range_add_notifier_locked(mm, prange);
struct svm_range *prange =
list_first_entry(&svm_bo->range_list,
struct svm_range, svm_bo_list);
+ int retries = 3;
+
list_del_init(&prange->svm_bo_list);
spin_unlock(&svm_bo->list_lock);
prange->start, prange->last);
mutex_lock(&prange->migrate_mutex);
- svm_migrate_vram_to_ram(prange, svm_bo->eviction_fence->mm);
+ do {
+ svm_migrate_vram_to_ram(prange,
+ svm_bo->eviction_fence->mm);
+ } while (prange->actual_loc && --retries);
+ WARN(prange->actual_loc, "Migration failed during eviction");
mutex_lock(&prange->lock);
prange->svm_bo = NULL;
amd_get_format_info(const struct drm_mode_fb_cmd2 *cmd);
static void handle_hpd_irq_helper(struct amdgpu_dm_connector *aconnector);
+static void handle_hpd_rx_irq(void *param);
static bool
is_timing_unchanged_for_freesync(struct drm_crtc_state *old_crtc_state,
amdgpu_dm_crtc_handle_crc_window_irq(&acrtc->base);
}
-#endif
+#endif /* CONFIG_DRM_AMD_SECURE_DISPLAY */
/**
* dmub_aux_setconfig_reply_callback - Callback for AUX or SET_CONFIG command.
return;
}
- drm_modeset_lock(&dev->mode_config.connection_mutex, NULL);
-
link_index = notify->link_index;
-
link = adev->dm.dc->links[link_index];
drm_connector_list_iter_begin(dev, &iter);
}
}
drm_connector_list_iter_end(&iter);
- drm_modeset_unlock(&dev->mode_config.connection_mutex);
- if (hpd_aconnector)
- handle_hpd_irq_helper(hpd_aconnector);
+ if (hpd_aconnector) {
+ if (notify->type == DMUB_NOTIFICATION_HPD)
+ handle_hpd_irq_helper(hpd_aconnector);
+ else if (notify->type == DMUB_NOTIFICATION_HPD_IRQ)
+ handle_hpd_rx_irq(hpd_aconnector);
+ }
}
/**
DRM_ERROR("DM: notify type %d invalid!", notify.type);
continue;
}
+ if (!dm->dmub_callback[notify.type]) {
+ DRM_DEBUG_DRIVER("DMUB notification skipped, no handler: type=%d\n", notify.type);
+ continue;
+ }
if (dm->dmub_thread_offload[notify.type] == true) {
dmub_hpd_wrk = kzalloc(sizeof(*dmub_hpd_wrk), GFP_ATOMIC);
if (!dmub_hpd_wrk) {
if (count > DMUB_TRACE_MAX_READ)
DRM_DEBUG_DRIVER("Warning : count > DMUB_TRACE_MAX_READ");
}
-#endif
+#endif /* CONFIG_DRM_AMD_DC_DCN */
static int dm_set_clockgating_state(void *handle,
enum amd_clockgating_state state)
switch (adev->ip_versions[DCE_HWIP][0]) {
case IP_VERSION(2, 1, 0):
init_data.flags.gpu_vm_support = true;
- init_data.flags.disable_dmcu = true;
+ switch (adev->dm.dmcub_fw_version) {
+ case 0: /* development */
+ case 0x1: /* linux-firmware.git hash 6d9f399 */
+ case 0x01000000: /* linux-firmware.git hash 9a0b0f4 */
+ init_data.flags.disable_dmcu = false;
+ break;
+ default:
+ init_data.flags.disable_dmcu = true;
+ }
break;
case IP_VERSION(1, 0, 0):
case IP_VERSION(1, 0, 1):
DRM_ERROR("amdgpu: fail to register dmub hpd callback");
goto error;
}
-#endif
+ if (!register_dmub_notify_callback(adev, DMUB_NOTIFICATION_HPD_IRQ, dmub_hpd_callback, true)) {
+ DRM_ERROR("amdgpu: fail to register dmub hpd callback");
+ goto error;
+ }
+#endif /* CONFIG_DRM_AMD_DC_DCN */
}
if (amdgpu_dm_initialize_drm_device(adev)) {
} else if (dc_link_detect(link, DETECT_REASON_BOOT)) {
amdgpu_dm_update_connector_after_detect(aconnector);
register_backlight_device(dm, link);
-
+ if (dm->num_of_edps)
+ update_connector_ext_caps(aconnector);
if (psr_feature_enabled)
amdgpu_dm_set_psr_caps(link);
}
}
-static int fill_dc_scaling_info(const struct drm_plane_state *state,
+static int fill_dc_scaling_info(struct amdgpu_device *adev,
+ const struct drm_plane_state *state,
struct dc_scaling_info *scaling_info)
{
int scale_w, scale_h, min_downscale, max_upscale;
/*
* For reasons we don't (yet) fully understand a non-zero
* src_y coordinate into an NV12 buffer can cause a
- * system hang. To avoid hangs (and maybe be overly cautious)
+ * system hang on DCN1x.
+ * To avoid hangs (and maybe be overly cautious)
* let's reject both non-zero src_x and src_y.
*
* We currently know of only one use-case to reproduce a
* is to gesture the YouTube Android app into full screen
* on ChromeOS.
*/
- if (state->fb &&
- state->fb->format->format == DRM_FORMAT_NV12 &&
- (scaling_info->src_rect.x != 0 ||
- scaling_info->src_rect.y != 0))
+ if (((adev->ip_versions[DCE_HWIP][0] == IP_VERSION(1, 0, 0)) ||
+ (adev->ip_versions[DCE_HWIP][0] == IP_VERSION(1, 0, 1))) &&
+ (state->fb && state->fb->format->format == DRM_FORMAT_NV12 &&
+ (scaling_info->src_rect.x != 0 || scaling_info->src_rect.y != 0)))
return -EINVAL;
scaling_info->src_rect.width = state->src_w >> 16;
int ret;
bool force_disable_dcc = false;
- ret = fill_dc_scaling_info(plane_state, &scaling_info);
+ ret = fill_dc_scaling_info(adev, plane_state, &scaling_info);
if (ret)
return ret;
if (stream->timing.flags.DSC && aconnector->dsc_settings.dsc_bits_per_pixel)
stream->timing.dsc_cfg.bits_per_pixel = aconnector->dsc_settings.dsc_bits_per_pixel;
}
-#endif
+#endif /* CONFIG_DRM_AMD_DC_DCN */
/**
* DOC: FreeSync Video
struct drm_connector_state *new_con_state;
struct amdgpu_dm_connector *aconnector;
struct dm_connector_state *dm_conn_state;
- int i, j, clock;
- int vcpi, pbn_div, pbn = 0;
+ int i, j;
+ int vcpi, pbn_div, pbn, slot_num = 0;
for_each_new_connector_in_state(state, connector, new_con_state, i) {
if (!stream)
continue;
- if (stream->timing.flags.DSC != 1) {
- drm_dp_mst_atomic_enable_dsc(state,
- aconnector->port,
- dm_conn_state->pbn,
- 0,
- false);
- continue;
- }
-
pbn_div = dm_mst_get_pbn_divider(stream->link);
- clock = stream->timing.pix_clk_100hz / 10;
/* pbn is calculated by compute_mst_dsc_configs_for_state*/
for (j = 0; j < dc_state->stream_count; j++) {
if (vars[j].aconnector == aconnector) {
}
}
+ if (j == dc_state->stream_count)
+ continue;
+
+ slot_num = DIV_ROUND_UP(pbn, pbn_div);
+
+ if (stream->timing.flags.DSC != 1) {
+ dm_conn_state->pbn = pbn;
+ dm_conn_state->vcpi_slots = slot_num;
+
+ drm_dp_mst_atomic_enable_dsc(state,
+ aconnector->port,
+ dm_conn_state->pbn,
+ 0,
+ false);
+ continue;
+ }
+
vcpi = drm_dp_mst_atomic_enable_dsc(state,
aconnector->port,
pbn, pbn_div,
if (ret)
return ret;
- ret = fill_dc_scaling_info(new_plane_state, &scaling_info);
+ ret = fill_dc_scaling_info(adev, new_plane_state, &scaling_info);
if (ret)
return ret;
bundle->surface_updates[planes_count].gamut_remap_matrix = &dc_plane->gamut_remap_matrix;
}
- fill_dc_scaling_info(new_plane_state,
+ fill_dc_scaling_info(dm->adev, new_plane_state,
&bundle->scaling_infos[planes_count]);
bundle->surface_updates[planes_count].scaling_info =
ret = drm_atomic_add_affected_connectors(state, crtc);
if (ret)
- return ret;
+ goto fail;
ret = drm_atomic_add_affected_planes(state, crtc);
if (ret)
wr_buf_ptr = wr_buf;
- r = copy_from_user(wr_buf_ptr, buf, wr_buf_size);
-
- /* r is bytes not be copied */
- if (r >= wr_buf_size) {
- DRM_DEBUG_DRIVER("user data not be read\n");
- return -EINVAL;
+ /* r is bytes not be copied */
+ if (copy_from_user(wr_buf_ptr, buf, wr_buf_size)) {
+ DRM_DEBUG_DRIVER("user data could not be read successfully\n");
+ return -EFAULT;
}
/* check number of parameters. isspace could not differ space and \n */
static void set_dsc_configs_from_fairness_vars(struct dsc_mst_fairness_params *params,
struct dsc_mst_fairness_vars *vars,
- int count)
+ int count,
+ int k)
{
int i;
for (i = 0; i < count; i++) {
memset(¶ms[i].timing->dsc_cfg, 0, sizeof(params[i].timing->dsc_cfg));
- if (vars[i].dsc_enabled && dc_dsc_compute_config(
+ if (vars[i + k].dsc_enabled && dc_dsc_compute_config(
params[i].sink->ctx->dc->res_pool->dscs[0],
¶ms[i].sink->dsc_caps.dsc_dec_caps,
params[i].sink->ctx->dc->debug.dsc_min_slice_height_override,
if (params[i].bpp_overwrite)
params[i].timing->dsc_cfg.bits_per_pixel = params[i].bpp_overwrite;
else
- params[i].timing->dsc_cfg.bits_per_pixel = vars[i].bpp_x16;
+ params[i].timing->dsc_cfg.bits_per_pixel = vars[i + k].bpp_x16;
if (params[i].num_slices_h)
params[i].timing->dsc_cfg.num_slices_h = params[i].num_slices_h;
struct dc_link *dc_link,
struct dsc_mst_fairness_params *params,
struct dsc_mst_fairness_vars *vars,
- int count)
+ int count,
+ int k)
{
int i;
bool bpp_increased[MAX_PIPES];
pbn_per_timeslot = dm_mst_get_pbn_divider(dc_link);
for (i = 0; i < count; i++) {
- if (vars[i].dsc_enabled) {
- initial_slack[i] = kbps_to_peak_pbn(params[i].bw_range.max_kbps) - vars[i].pbn;
+ if (vars[i + k].dsc_enabled) {
+ initial_slack[i] =
+ kbps_to_peak_pbn(params[i].bw_range.max_kbps) - vars[i + k].pbn;
bpp_increased[i] = false;
remaining_to_increase += 1;
} else {
link_timeslots_used = 0;
for (i = 0; i < count; i++)
- link_timeslots_used += DIV_ROUND_UP(vars[i].pbn, pbn_per_timeslot);
+ link_timeslots_used += DIV_ROUND_UP(vars[i + k].pbn, pbn_per_timeslot);
fair_pbn_alloc = (63 - link_timeslots_used) / remaining_to_increase * pbn_per_timeslot;
struct dc_link *dc_link,
struct dsc_mst_fairness_params *params,
struct dsc_mst_fairness_vars *vars,
- int count)
+ int count,
+ int k)
{
int i;
bool tried[MAX_PIPES];
int remaining_to_try = 0;
for (i = 0; i < count; i++) {
- if (vars[i].dsc_enabled
- && vars[i].bpp_x16 == params[i].bw_range.max_target_bpp_x16
+ if (vars[i + k].dsc_enabled
+ && vars[i + k].bpp_x16 == params[i].bw_range.max_target_bpp_x16
&& params[i].clock_force_enable == DSC_CLK_FORCE_DEFAULT) {
kbps_increase[i] = params[i].bw_range.stream_kbps - params[i].bw_range.max_kbps;
tried[i] = false;
static bool compute_mst_dsc_configs_for_link(struct drm_atomic_state *state,
struct dc_state *dc_state,
struct dc_link *dc_link,
- struct dsc_mst_fairness_vars *vars)
+ struct dsc_mst_fairness_vars *vars,
+ int *link_vars_start_index)
{
- int i;
+ int i, k;
struct dc_stream_state *stream;
struct dsc_mst_fairness_params params[MAX_PIPES];
struct amdgpu_dm_connector *aconnector;
if (stream->link != dc_link)
continue;
+ aconnector = (struct amdgpu_dm_connector *)stream->dm_stream_context;
+ if (!aconnector)
+ continue;
+
+ if (!aconnector->port)
+ continue;
+
stream->timing.flags.DSC = 0;
params[count].timing = &stream->timing;
params[count].sink = stream->sink;
- aconnector = (struct amdgpu_dm_connector *)stream->dm_stream_context;
params[count].aconnector = aconnector;
params[count].port = aconnector->port;
params[count].clock_force_enable = aconnector->dsc_settings.dsc_force_enable;
count++;
}
+
+ if (count == 0) {
+ ASSERT(0);
+ return true;
+ }
+
+ /* k is start index of vars for current phy link used by mst hub */
+ k = *link_vars_start_index;
+ /* set vars start index for next mst hub phy link */
+ *link_vars_start_index += count;
+
/* Try no compression */
for (i = 0; i < count; i++) {
- vars[i].aconnector = params[i].aconnector;
- vars[i].pbn = kbps_to_peak_pbn(params[i].bw_range.stream_kbps);
- vars[i].dsc_enabled = false;
- vars[i].bpp_x16 = 0;
+ vars[i + k].aconnector = params[i].aconnector;
+ vars[i + k].pbn = kbps_to_peak_pbn(params[i].bw_range.stream_kbps);
+ vars[i + k].dsc_enabled = false;
+ vars[i + k].bpp_x16 = 0;
if (drm_dp_atomic_find_vcpi_slots(state,
params[i].port->mgr,
params[i].port,
- vars[i].pbn,
+ vars[i + k].pbn,
dm_mst_get_pbn_divider(dc_link)) < 0)
return false;
}
if (!drm_dp_mst_atomic_check(state) && !debugfs_overwrite) {
- set_dsc_configs_from_fairness_vars(params, vars, count);
+ set_dsc_configs_from_fairness_vars(params, vars, count, k);
return true;
}
/* Try max compression */
for (i = 0; i < count; i++) {
if (params[i].compression_possible && params[i].clock_force_enable != DSC_CLK_FORCE_DISABLE) {
- vars[i].pbn = kbps_to_peak_pbn(params[i].bw_range.min_kbps);
- vars[i].dsc_enabled = true;
- vars[i].bpp_x16 = params[i].bw_range.min_target_bpp_x16;
+ vars[i + k].pbn = kbps_to_peak_pbn(params[i].bw_range.min_kbps);
+ vars[i + k].dsc_enabled = true;
+ vars[i + k].bpp_x16 = params[i].bw_range.min_target_bpp_x16;
if (drm_dp_atomic_find_vcpi_slots(state,
params[i].port->mgr,
params[i].port,
- vars[i].pbn,
+ vars[i + k].pbn,
dm_mst_get_pbn_divider(dc_link)) < 0)
return false;
} else {
- vars[i].pbn = kbps_to_peak_pbn(params[i].bw_range.stream_kbps);
- vars[i].dsc_enabled = false;
- vars[i].bpp_x16 = 0;
+ vars[i + k].pbn = kbps_to_peak_pbn(params[i].bw_range.stream_kbps);
+ vars[i + k].dsc_enabled = false;
+ vars[i + k].bpp_x16 = 0;
if (drm_dp_atomic_find_vcpi_slots(state,
params[i].port->mgr,
params[i].port,
- vars[i].pbn,
+ vars[i + k].pbn,
dm_mst_get_pbn_divider(dc_link)) < 0)
return false;
}
return false;
/* Optimize degree of compression */
- increase_dsc_bpp(state, dc_link, params, vars, count);
+ increase_dsc_bpp(state, dc_link, params, vars, count, k);
- try_disable_dsc(state, dc_link, params, vars, count);
+ try_disable_dsc(state, dc_link, params, vars, count, k);
- set_dsc_configs_from_fairness_vars(params, vars, count);
+ set_dsc_configs_from_fairness_vars(params, vars, count, k);
return true;
}
+static bool is_dsc_need_re_compute(
+ struct drm_atomic_state *state,
+ struct dc_state *dc_state,
+ struct dc_link *dc_link)
+{
+ int i;
+ bool is_dsc_need_re_compute = false;
+
+ /* only check phy used by mst branch */
+ if (dc_link->type != dc_connection_mst_branch)
+ return false;
+
+ /* check if there is mode change in new request */
+ for (i = 0; i < dc_state->stream_count; i++) {
+ struct amdgpu_dm_connector *aconnector;
+ struct dc_stream_state *stream;
+ struct drm_crtc_state *new_crtc_state;
+ struct drm_connector_state *new_conn_state;
+
+ stream = dc_state->streams[i];
+
+ if (!stream)
+ continue;
+
+ /* check if stream using the same link for mst */
+ if (stream->link != dc_link)
+ continue;
+
+ aconnector = (struct amdgpu_dm_connector *) stream->dm_stream_context;
+ if (!aconnector)
+ continue;
+
+ new_conn_state = drm_atomic_get_new_connector_state(state, &aconnector->base);
+
+ if (!new_conn_state)
+ continue;
+
+ if (IS_ERR(new_conn_state))
+ continue;
+
+ if (!new_conn_state->crtc)
+ continue;
+
+ new_crtc_state = drm_atomic_get_new_crtc_state(state, new_conn_state->crtc);
+
+ if (!new_crtc_state)
+ continue;
+
+ if (IS_ERR(new_crtc_state))
+ continue;
+
+ if (new_crtc_state->enable && new_crtc_state->active) {
+ if (new_crtc_state->mode_changed || new_crtc_state->active_changed ||
+ new_crtc_state->connectors_changed)
+ is_dsc_need_re_compute = true;
+ }
+ }
+
+ return is_dsc_need_re_compute;
+}
+
bool compute_mst_dsc_configs_for_state(struct drm_atomic_state *state,
struct dc_state *dc_state,
struct dsc_mst_fairness_vars *vars)
struct dc_stream_state *stream;
bool computed_streams[MAX_PIPES];
struct amdgpu_dm_connector *aconnector;
+ int link_vars_start_index = 0;
for (i = 0; i < dc_state->stream_count; i++)
computed_streams[i] = false;
if (dcn20_remove_stream_from_ctx(stream->ctx->dc, dc_state, stream) != DC_OK)
return false;
+ if (!is_dsc_need_re_compute(state, dc_state, stream->link))
+ continue;
+
mutex_lock(&aconnector->mst_mgr.lock);
- if (!compute_mst_dsc_configs_for_link(state, dc_state, stream->link, vars)) {
+ if (!compute_mst_dsc_configs_for_link(state, dc_state, stream->link,
+ vars, &link_vars_start_index)) {
mutex_unlock(&aconnector->mst_mgr.lock);
return false;
}
struct dc_stream_state *old_stream =
dc->current_state->res_ctx.pipe_ctx[i].stream;
bool should_disable = true;
+ bool pipe_split_change =
+ context->res_ctx.pipe_ctx[i].top_pipe != dc->current_state->res_ctx.pipe_ctx[i].top_pipe;
for (j = 0; j < context->stream_count; j++) {
if (old_stream == context->streams[j]) {
break;
}
}
+ if (!should_disable && pipe_split_change)
+ should_disable = true;
+
if (should_disable && old_stream) {
dc_rem_all_planes_for_stream(dc, old_stream, dangling_context);
disable_all_writeback_pipes_for_stream(dc, old_stream, dangling_context);
return false;
}
+#ifdef CONFIG_DRM_AMD_DC_DCN
/* Perform updates here which need to be deferred until next vupdate
*
* i.e. blnd lut, 3dlut, and shaper lut bypass regs are double buffered
*/
static void process_deferred_updates(struct dc *dc)
{
-#ifdef CONFIG_DRM_AMD_DC_DCN
int i = 0;
if (dc->debug.enable_mem_low_power.bits.cm) {
if (dc->res_pool->dpps[i]->funcs->dpp_deferred_update)
dc->res_pool->dpps[i]->funcs->dpp_deferred_update(dc->res_pool->dpps[i]);
}
-#endif
}
+#endif /* CONFIG_DRM_AMD_DC_DCN */
void dc_post_update_surfaces_to_stream(struct dc *dc)
{
dc->hwss.disable_plane(dc, &context->res_ctx.pipe_ctx[i]);
}
+#ifdef CONFIG_DRM_AMD_DC_DCN
process_deferred_updates(dc);
+#endif
dc->hwss.optimize_bandwidth(dc, context);
#if defined(CONFIG_DRM_AMD_DC_DCN)
/* YELLOW_CARP B0 USB4 DPIA needs dmub notifications for interrupts */
if (dc->ctx->asic_id.chip_family == FAMILY_YELLOW_CARP &&
- dc->ctx->asic_id.hw_internal_rev == YELLOW_CARP_B0)
+ dc->ctx->asic_id.hw_internal_rev == YELLOW_CARP_B0 &&
+ !dc->debug.dpia_debug.bits.disable_dpia)
return true;
#endif
/* dmub aux needs dmub notifications to be enabled */
*/
if (status != DC_FAIL_DP_LINK_TRAINING ||
pipe_ctx->stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) {
+ if (false == stream->link->link_status.link_active)
+ disable_link(stream->link, pipe_ctx->stream->signal);
BREAK_TO_DEBUGGER();
return;
}
timing->dsc_cfg.bits_per_pixel,
timing->dsc_cfg.num_slices_h,
timing->dsc_cfg.is_dp);
-#endif
+#endif /* CONFIG_DRM_AMD_DC_DCN */
switch (timing->display_color_depth) {
case COLOR_DEPTH_666:
return false;
if (link->dpcd_caps.dpcd_rev.raw >= DPCD_REV_12) {
+#if defined(CONFIG_DRM_AMD_DC_DCN)
+ if (test_pattern == DP_TEST_PATTERN_SQUARE_PULSE)
+ core_link_write_dpcd(link,
+ DP_LINK_SQUARE_PATTERN,
+ p_custom_pattern,
+ 1);
+
+#endif
/* tell receiver that we are sending qualification
* pattern DP 1.2 or later - DP receiver's link quality
* pattern is set using DPCD LINK_QUAL_LANEx_SET
return link_enc;
}
+/* Clear all link encoder assignments. */
+static void clear_enc_assignments(struct dc_state *state)
+{
+ int i;
+ enum engine_id eng_id;
+ struct dc_stream_state *stream;
+
+ for (i = 0; i < MAX_PIPES; i++) {
+ state->res_ctx.link_enc_cfg_ctx.link_enc_assignments[i].valid = false;
+ eng_id = state->res_ctx.link_enc_cfg_ctx.link_enc_assignments[i].eng_id;
+ stream = state->res_ctx.link_enc_cfg_ctx.link_enc_assignments[i].stream;
+ if (eng_id != ENGINE_ID_UNKNOWN)
+ state->res_ctx.link_enc_cfg_ctx.link_enc_avail[eng_id - ENGINE_ID_DIGA] = eng_id;
+ if (stream)
+ stream->link_enc = NULL;
+ }
+}
void link_enc_cfg_init(
struct dc *dc,
state->res_ctx.link_enc_cfg_ctx.link_enc_avail[i] = ENGINE_ID_UNKNOWN;
}
+ clear_enc_assignments(state);
+
state->res_ctx.link_enc_cfg_ctx.mode = LINK_ENC_CFG_STEADY;
}
ASSERT(state->stream_count == stream_count);
+ if (stream_count == 0)
+ clear_enc_assignments(state);
+
/* Release DIG link encoder resources before running assignment algorithm. */
for (i = 0; i < stream_count; i++)
dc->res_pool->funcs->link_enc_unassign(state, streams[i]);
struct set_config_cmd_payload;
struct dmub_notification;
-#define DC_VER "3.2.159"
+#define DC_VER "3.2.160"
#define MAX_SURFACES 3
#define MAX_PLANES 6
#endif
union mem_low_power_enable_options enable_mem_low_power;
union root_clock_optimization_options root_clock_optimization;
+ bool hpo_optimization;
bool force_vblank_alignment;
/* Enable dmub aux for legacy ddc */
#ifndef DP_DFP_CAPABILITY_EXTENSION_SUPPORT
#define DP_DFP_CAPABILITY_EXTENSION_SUPPORT 0x0A3
#endif
+#ifndef DP_LINK_SQUARE_PATTERN
+#define DP_LINK_SQUARE_PATTERN 0x10F
+#endif
#ifndef DP_DSC_CONFIGURATION
#define DP_DSC_CONFIGURATION 0x161
#endif
uint32_t MC_VM_FB_LOCATION_BASE;
uint32_t MC_VM_FB_LOCATION_TOP;
uint32_t MC_VM_FB_OFFSET;
+ uint32_t HPO_TOP_HW_CONTROL;
};
/* set field name */
#define HWS_SF(blk_name, reg_name, field_name, post_fix)\
type DOMAIN_PGFSM_PWR_STATUS;\
type HPO_HDMISTREAMCLK_G_GATE_DIS;\
type DISABLE_HOSTVM_FORCE_ALLOW_PSTATE;\
- type I2C_LIGHT_SLEEP_FORCE;
+ type I2C_LIGHT_SLEEP_FORCE;\
+ type HPO_IO_EN;
struct dce_hwseq_shift {
HWSEQ_REG_FIELD_LIST(uint8_t)
#endif
if (dc_is_dp_signal(pipe_ctx->stream->signal))
dp_source_sequence_trace(link, DPCD_SOURCE_SEQ_AFTER_DISCONNECT_DIG_FE_BE);
+
+#if defined(CONFIG_DRM_AMD_DC_DCN)
+ if (dc->hwseq->funcs.setup_hpo_hw_control && is_dp_128b_132b_signal(pipe_ctx))
+ dc->hwseq->funcs.setup_hpo_hw_control(dc->hwseq, false);
+#endif
+
}
void dce110_unblank_stream(struct pipe_ctx *pipe_ctx,
if (!s->blank_en)
DTN_INFO("[%2d]: %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh"
- "% 8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh"
+ " %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh"
" %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh %8xh\n",
pool->hubps[i]->inst, dlg_regs->refcyc_h_blank_end, dlg_regs->dlg_vblank_end, dlg_regs->min_dst_y_next_start,
dlg_regs->refcyc_per_htotal, dlg_regs->refcyc_x_after_scaler, dlg_regs->dst_y_after_scaler,
* BY this, it is logic clean to separate stream and link
*/
if (is_dp_128b_132b_signal(pipe_ctx)) {
+ if (pipe_ctx->stream->ctx->dc->hwseq->funcs.setup_hpo_hw_control)
+ pipe_ctx->stream->ctx->dc->hwseq->funcs.setup_hpo_hw_control(
+ pipe_ctx->stream->ctx->dc->hwseq, true);
setup_dp_hpo_stream(pipe_ctx, true);
pipe_ctx->stream_res.hpo_dp_stream_enc->funcs->enable_stream(
pipe_ctx->stream_res.hpo_dp_stream_enc);
}
-static void mpc3_mpc_init(struct mpc *mpc)
+static void mpc3_set_mpc_mem_lp_mode(struct mpc *mpc)
{
struct dcn30_mpc *mpc30 = TO_DCN30_MPC(mpc);
int mpcc_id;
- mpc1_mpc_init(mpc);
-
if (mpc->ctx->dc->debug.enable_mem_low_power.bits.mpc) {
if (mpc30->mpc_mask->MPC_RMU0_MEM_LOW_PWR_MODE && mpc30->mpc_mask->MPC_RMU1_MEM_LOW_PWR_MODE) {
REG_UPDATE(MPC_RMU_MEM_PWR_CTRL, MPC_RMU0_MEM_LOW_PWR_MODE, 3);
.read_mpcc_state = mpc1_read_mpcc_state,
.insert_plane = mpc1_insert_plane,
.remove_mpcc = mpc1_remove_mpcc,
- .mpc_init = mpc3_mpc_init,
+ .mpc_init = mpc1_mpc_init,
.mpc_init_single_inst = mpc1_mpc_init_single_inst,
.update_blending = mpc2_update_blending,
.cursor_lock = mpc1_cursor_lock,
.power_on_mpc_mem_pwr = mpc3_power_on_ogam_lut,
.get_mpc_out_mux = mpc1_get_mpc_out_mux,
.set_bg_color = mpc1_set_bg_color,
+ .set_mpc_mem_lp_mode = mpc3_set_mpc_mem_lp_mode,
};
void dcn30_mpc_construct(struct dcn30_mpc *mpc30,
int pipe_cnt,
int vlevel)
{
+ int maxMpcComb = context->bw_ctx.dml.vba.maxMpcComb;
int i, pipe_idx;
- double dcfclk = context->bw_ctx.dml.vba.DCFCLKState[vlevel][context->bw_ctx.dml.vba.maxMpcComb];
- bool pstate_en = context->bw_ctx.dml.vba.DRAMClockChangeSupport[vlevel][context->bw_ctx.dml.vba.maxMpcComb] !=
- dm_dram_clock_change_unsupported;
+ double dcfclk = context->bw_ctx.dml.vba.DCFCLKState[vlevel][maxMpcComb];
+ bool pstate_en = context->bw_ctx.dml.vba.DRAMClockChangeSupport[vlevel][maxMpcComb] != dm_dram_clock_change_unsupported;
if (context->bw_ctx.dml.soc.min_dcfclk > dcfclk)
dcfclk = context->bw_ctx.dml.soc.min_dcfclk;
context->bw_ctx.dml.soc.sr_enter_plus_exit_time_us = dc->clk_mgr->bw_params->wm_table.nv_entries[WM_C].dml_input.sr_enter_plus_exit_time_us;
context->bw_ctx.dml.soc.sr_exit_time_us = dc->clk_mgr->bw_params->wm_table.nv_entries[WM_C].dml_input.sr_exit_time_us;
}
+
context->bw_ctx.bw.dcn.watermarks.c.urgent_ns = get_wm_urgent(&context->bw_ctx.dml, pipes, pipe_cnt) * 1000;
context->bw_ctx.bw.dcn.watermarks.c.cstate_pstate.cstate_enter_plus_exit_ns = get_wm_stutter_enter_exit(&context->bw_ctx.dml, pipes, pipe_cnt) * 1000;
context->bw_ctx.bw.dcn.watermarks.c.cstate_pstate.cstate_exit_ns = get_wm_stutter_exit(&context->bw_ctx.dml, pipes, pipe_cnt) * 1000;
#define FN(reg_name, field_name) \
hws->shifts->field_name, hws->masks->field_name
+static void enable_memory_low_power(struct dc *dc)
+{
+ struct dce_hwseq *hws = dc->hwseq;
+ int i;
+
+ if (dc->debug.enable_mem_low_power.bits.dmcu) {
+ // Force ERAM to shutdown if DMCU is not enabled
+ if (dc->debug.disable_dmcu || dc->config.disable_dmcu) {
+ REG_UPDATE(DMU_MEM_PWR_CNTL, DMCU_ERAM_MEM_PWR_FORCE, 3);
+ }
+ }
+
+ // Set default OPTC memory power states
+ if (dc->debug.enable_mem_low_power.bits.optc) {
+ // Shutdown when unassigned and light sleep in VBLANK
+ REG_SET_2(ODM_MEM_PWR_CTRL3, 0, ODM_MEM_UNASSIGNED_PWR_MODE, 3, ODM_MEM_VBLANK_PWR_MODE, 1);
+ }
+
+ if (dc->debug.enable_mem_low_power.bits.vga) {
+ // Power down VGA memory
+ REG_UPDATE(MMHUBBUB_MEM_PWR_CNTL, VGA_MEM_PWR_FORCE, 1);
+ }
+
+ if (dc->debug.enable_mem_low_power.bits.mpc)
+ dc->res_pool->mpc->funcs->set_mpc_mem_lp_mode(dc->res_pool->mpc);
+
+
+ if (dc->debug.enable_mem_low_power.bits.vpg && dc->res_pool->stream_enc[0]->vpg->funcs->vpg_powerdown) {
+ // Power down VPGs
+ for (i = 0; i < dc->res_pool->stream_enc_count; i++)
+ dc->res_pool->stream_enc[i]->vpg->funcs->vpg_powerdown(dc->res_pool->stream_enc[i]->vpg);
+#if defined(CONFIG_DRM_AMD_DC_DCN)
+ for (i = 0; i < dc->res_pool->hpo_dp_stream_enc_count; i++)
+ dc->res_pool->hpo_dp_stream_enc[i]->vpg->funcs->vpg_powerdown(dc->res_pool->hpo_dp_stream_enc[i]->vpg);
+#endif
+ }
+
+}
+
void dcn31_init_hw(struct dc *dc)
{
struct abm **abms = dc->res_pool->multiple_abms;
if (res_pool->dccg->funcs->dccg_init)
res_pool->dccg->funcs->dccg_init(res_pool->dccg);
- if (dc->debug.enable_mem_low_power.bits.dmcu) {
- // Force ERAM to shutdown if DMCU is not enabled
- if (dc->debug.disable_dmcu || dc->config.disable_dmcu) {
- REG_UPDATE(DMU_MEM_PWR_CNTL, DMCU_ERAM_MEM_PWR_FORCE, 3);
- }
- }
-
- // Set default OPTC memory power states
- if (dc->debug.enable_mem_low_power.bits.optc) {
- // Shutdown when unassigned and light sleep in VBLANK
- REG_SET_2(ODM_MEM_PWR_CTRL3, 0, ODM_MEM_UNASSIGNED_PWR_MODE, 3, ODM_MEM_VBLANK_PWR_MODE, 1);
- }
-
- if (dc->debug.enable_mem_low_power.bits.vga) {
- // Power down VGA memory
- REG_UPDATE(MMHUBBUB_MEM_PWR_CNTL, VGA_MEM_PWR_FORCE, 1);
- }
-
-#if defined(CONFIG_DRM_AMD_DC_DCN)
- if (dc->debug.enable_mem_low_power.bits.vpg && dc->res_pool->stream_enc[0]->vpg->funcs->vpg_powerdown) {
- // Power down VPGs
- for (i = 0; i < dc->res_pool->stream_enc_count; i++)
- dc->res_pool->stream_enc[i]->vpg->funcs->vpg_powerdown(dc->res_pool->stream_enc[i]->vpg);
-#if defined(CONFIG_DRM_AMD_DC_DP2_0)
- for (i = 0; i < dc->res_pool->hpo_dp_stream_enc_count; i++)
- dc->res_pool->hpo_dp_stream_enc[i]->vpg->funcs->vpg_powerdown(dc->res_pool->hpo_dp_stream_enc[i]->vpg);
-#endif
- }
-#endif
+ enable_memory_low_power(dc);
if (dc->ctx->dc_bios->fw_info_valid) {
res_pool->ref_clocks.xtalin_clock_inKhz =
if (dc->debug.enable_mem_low_power.bits.i2c)
REG_UPDATE(DIO_MEM_PWR_CTRL, I2C_LIGHT_SLEEP_FORCE, 1);
+ if (hws->funcs.setup_hpo_hw_control)
+ hws->funcs.setup_hpo_hw_control(hws, false);
+
if (!dc->debug.disable_clock_gate) {
/* enable all DCN clock gating */
REG_WRITE(DCCG_GATE_DISABLE_CNTL, 0);
/* New dc_state in the process of being applied to hardware. */
dc->current_state->res_ctx.link_enc_cfg_ctx.mode = LINK_ENC_CFG_TRANSIENT;
}
+
+void dcn31_setup_hpo_hw_control(const struct dce_hwseq *hws, bool enable)
+{
+ if (hws->ctx->dc->debug.hpo_optimization)
+ REG_UPDATE(HPO_TOP_HW_CONTROL, HPO_IO_EN, !!enable);
+}
bool dcn31_is_abm_supported(struct dc *dc,
struct dc_state *context, struct dc_stream_state *stream);
void dcn31_init_pipes(struct dc *dc, struct dc_state *context);
+void dcn31_setup_hpo_hw_control(const struct dce_hwseq *hws, bool enable);
#endif /* __DC_HWSS_DCN31_H__ */
.dccg_init = dcn20_dccg_init,
.set_blend_lut = dcn30_set_blend_lut,
.set_shaper_3dlut = dcn20_set_shaper_3dlut,
+ .setup_hpo_hw_control = dcn31_setup_hpo_hw_control,
};
void dcn31_hw_sequencer_construct(struct dc *dc)
SR(D6VGA_CONTROL), \
SR(DC_IP_REQUEST_CNTL), \
SR(AZALIA_AUDIO_DTO), \
- SR(AZALIA_CONTROLLER_CLOCK_GATING)
+ SR(AZALIA_CONTROLLER_CLOCK_GATING), \
+ SR(HPO_TOP_HW_CONTROL)
static const struct dce_hwseq_registers hwseq_reg = {
HWSEQ_DCN31_REG_LIST()
HWS_SF(, ODM_MEM_PWR_CTRL3, ODM_MEM_UNASSIGNED_PWR_MODE, mask_sh), \
HWS_SF(, ODM_MEM_PWR_CTRL3, ODM_MEM_VBLANK_PWR_MODE, mask_sh), \
HWS_SF(, MMHUBBUB_MEM_PWR_CNTL, VGA_MEM_PWR_FORCE, mask_sh), \
- HWS_SF(, DIO_MEM_PWR_CTRL, I2C_LIGHT_SLEEP_FORCE, mask_sh)
+ HWS_SF(, DIO_MEM_PWR_CTRL, I2C_LIGHT_SLEEP_FORCE, mask_sh), \
+ HWS_SF(, HPO_TOP_HW_CONTROL, HPO_IO_EN, mask_sh)
static const struct dce_hwseq_shift hwseq_shift = {
HWSEQ_DCN31_MASK_SH_LIST(__SHIFT)
MinDSCBPP = 8;
MaxDSCBPP = 3 * DSCInputBitPerComponent - 1.0 / 16;
} else {
- if (Output == dm_hdmi) {
- NonDSCBPP0 = 24;
- NonDSCBPP1 = 24;
- NonDSCBPP2 = 24;
- }
- else {
- NonDSCBPP0 = 16;
- NonDSCBPP1 = 20;
- NonDSCBPP2 = 24;
- }
+ NonDSCBPP0 = 16;
+ NonDSCBPP1 = 20;
+ NonDSCBPP2 = 24;
if (Format == dm_n422) {
MinDSCBPP = 7;
MinDSCBPP = 8;
MaxDSCBPP = 3 * DSCInputBitPerComponent - 1.0 / 16;
} else {
- if (Output == dm_hdmi) {
- NonDSCBPP0 = 24;
- NonDSCBPP1 = 24;
- NonDSCBPP2 = 24;
- } else {
- NonDSCBPP0 = 16;
- NonDSCBPP1 = 20;
- NonDSCBPP2 = 24;
- }
+
+ NonDSCBPP0 = 16;
+ NonDSCBPP1 = 20;
+ NonDSCBPP2 = 24;
+
if (Format == dm_n422) {
MinDSCBPP = 7;
MaxDSCBPP = 2 * DSCInputBitPerComponent - 1.0 / 16.0;
void (*set_bg_color)(struct mpc *mpc,
struct tg_color *bg_color,
int mpcc_id);
+ void (*set_mpc_mem_lp_mode)(struct mpc *mpc);
};
#endif
const struct dc_plane_state *plane_state);
void (*PLAT_58856_wa)(struct dc_state *context,
struct pipe_ctx *pipe_ctx);
+ void (*setup_hpo_hw_control)(const struct dce_hwseq *hws, bool enable);
};
struct dce_hwseq {
bool load_inst_const;
bool skip_panel_power_sequence;
bool disable_z10;
+ bool power_optimization;
bool dpia_supported;
bool disable_dpia;
};
/* Firmware versioning. */
#ifdef DMUB_EXPOSE_VERSION
-#define DMUB_FW_VERSION_GIT_HASH 0x9525efb5
+#define DMUB_FW_VERSION_GIT_HASH 0x1d82d23e
#define DMUB_FW_VERSION_MAJOR 0
#define DMUB_FW_VERSION_MINOR 0
-#define DMUB_FW_VERSION_REVISION 90
+#define DMUB_FW_VERSION_REVISION 91
#define DMUB_FW_VERSION_TEST 0
#define DMUB_FW_VERSION_VBIOS 0
#define DMUB_FW_VERSION_HOTFIX 0
boot_options.bits.z10_disable = params->disable_z10;
boot_options.bits.dpia_supported = params->dpia_supported;
boot_options.bits.enable_dpia = params->disable_dpia ? 0 : 1;
+ boot_options.bits.power_optimization = params->power_optimization;
boot_options.bits.sel_mux_phy_c_d_phy_f_g = (dmub->asic == DMUB_ASIC_DCN31B) ? 1 : 0;
AMD_IP_BLOCK_TYPE_ACP,
AMD_IP_BLOCK_TYPE_VCN,
AMD_IP_BLOCK_TYPE_MES,
- AMD_IP_BLOCK_TYPE_JPEG
+ AMD_IP_BLOCK_TYPE_JPEG,
+ AMD_IP_BLOCK_TYPE_NUM,
};
enum amd_clockgating_state {
{
int ret = 0;
const struct amd_pm_funcs *pp_funcs = adev->powerplay.pp_funcs;
+ enum ip_power_state pwr_state = gate ? POWER_STATE_OFF : POWER_STATE_ON;
+
+ if (atomic_read(&adev->pm.pwr_state[block_type]) == pwr_state) {
+ dev_dbg(adev->dev, "IP block%d already in the target %s state!",
+ block_type, gate ? "gate" : "ungate");
+ return 0;
+ }
switch (block_type) {
case AMD_IP_BLOCK_TYPE_UVD:
break;
}
+ if (!ret)
+ atomic_set(&adev->pm.pwr_state[block_type], pwr_state);
+
return ret;
}
} else if (DEVICE_ATTR_IS(pp_dpm_dclk)) {
if (!(asic_type == CHIP_VANGOGH || asic_type == CHIP_SIENNA_CICHLID))
*states = ATTR_STATE_UNSUPPORTED;
+ } else if (DEVICE_ATTR_IS(pp_power_profile_mode)) {
+ if (!adev->powerplay.pp_funcs->get_power_profile_mode ||
+ amdgpu_dpm_get_power_profile_mode(adev, NULL) == -EOPNOTSUPP)
+ *states = ATTR_STATE_UNSUPPORTED;
}
switch (asic_type) {
enum amd_dpm_forced_level forced_level;
};
+enum ip_power_state {
+ POWER_STATE_UNKNOWN,
+ POWER_STATE_ON,
+ POWER_STATE_OFF,
+};
+
struct amdgpu_pm {
struct mutex mutex;
u32 current_sclk;
struct i2c_adapter smu_i2c;
struct mutex smu_i2c_mutex;
struct list_head pm_attr_list;
+
+ atomic_t pwr_state[AMD_IP_BLOCK_TYPE_NUM];
};
#define R600_SSTU_DFLT 0
#define PPSMC_MSG_PowerUpVcn 0x07 ///< Power up VCN; VCN is power gated by default
#define PPSMC_MSG_SetHardMinVcn 0x08 ///< For wireless display
#define PPSMC_MSG_SetSoftMinGfxclk 0x09 ///< Set SoftMin for GFXCLK, argument is frequency in MHz
-#define PPSMC_MSG_ActiveProcessNotify 0x0A ///< Set active work load type
+#define PPSMC_MSG_ActiveProcessNotify 0x0A ///< Deprecated (Not to be used)
#define PPSMC_MSG_ForcePowerDownGfx 0x0B ///< Force power down GFX, i.e. enter GFXOFF
#define PPSMC_MSG_PrepareMp1ForUnload 0x0C ///< Prepare PMFW for GFX driver unload
#define PPSMC_MSG_SetDriverDramAddrHigh 0x0D ///< Set high 32 bits of DRAM address for Driver table transfer
#define PPSMC_MSG_SetHardMinSocclkByFreq 0x13 ///< Set hard min for SOC CLK
#define PPSMC_MSG_SetSoftMinFclk 0x14 ///< Set hard min for FCLK
#define PPSMC_MSG_SetSoftMinVcn 0x15 ///< Set soft min for VCN clocks (VCLK and DCLK)
-#define PPSMC_MSG_SPARE0 0x16 ///< Spared
+#define PPSMC_MSG_SPARE 0x16 ///< Spare
#define PPSMC_MSG_GetGfxclkFrequency 0x17 ///< Get GFX clock frequency
#define PPSMC_MSG_GetFclkFrequency 0x18 ///< Get FCLK frequency
#define PPSMC_MSG_AllowGfxOff 0x19 ///< Inform PMFW of allowing GFXOFF entry
static int pp_get_power_profile_mode(void *handle, char *buf)
{
struct pp_hwmgr *hwmgr = handle;
+ int ret;
- if (!hwmgr || !hwmgr->pm_en || !buf)
+ if (!hwmgr || !hwmgr->pm_en || !hwmgr->hwmgr_func->get_power_profile_mode)
+ return -EOPNOTSUPP;
+ if (!buf)
return -EINVAL;
- if (hwmgr->hwmgr_func->get_power_profile_mode == NULL) {
- pr_info_ratelimited("%s was not implemented.\n", __func__);
- return snprintf(buf, PAGE_SIZE, "\n");
- }
-
- return hwmgr->hwmgr_func->get_power_profile_mode(hwmgr, buf);
+ mutex_lock(&hwmgr->smu_lock);
+ ret = hwmgr->hwmgr_func->get_power_profile_mode(hwmgr, buf);
+ mutex_unlock(&hwmgr->smu_lock);
+ return ret;
}
static int pp_set_power_profile_mode(void *handle, long *input, uint32_t size)
{
struct pp_hwmgr *hwmgr = handle;
- int ret = -EINVAL;
+ int ret = -EOPNOTSUPP;
- if (!hwmgr || !hwmgr->pm_en)
- return ret;
-
- if (hwmgr->hwmgr_func->set_power_profile_mode == NULL) {
- pr_info_ratelimited("%s was not implemented.\n", __func__);
+ if (!hwmgr || !hwmgr->pm_en || !hwmgr->hwmgr_func->set_power_profile_mode)
return ret;
- }
if (hwmgr->dpm_level != AMD_DPM_FORCED_LEVEL_MANUAL) {
pr_debug("power profile setting is for manual dpm mode only.\n");
- return ret;
+ return -EINVAL;
}
mutex_lock(&hwmgr->smu_lock);
uint32_t min_freq, max_freq = 0;
uint32_t ret = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
smum_send_msg_to_smc(hwmgr, PPSMC_MSG_GetGfxclkFrequency, &now);
if (ret)
return ret;
- size = sysfs_emit(buf, "%s:\n", "OD_SCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_SCLK");
size += sysfs_emit_at(buf, size, "0: %10uMhz\n",
(data->gfx_actual_soft_min_freq > 0) ? data->gfx_actual_soft_min_freq : min_freq);
size += sysfs_emit_at(buf, size, "1: %10uMhz\n",
if (ret)
return ret;
- size = sysfs_emit(buf, "%s:\n", "OD_RANGE");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_RANGE");
size += sysfs_emit_at(buf, size, "SCLK: %7uMHz %10uMHz\n",
min_freq, max_freq);
}
if (!buf)
return -EINVAL;
+ phm_get_sysfs_buf(&buf, &size);
+
size += sysfs_emit_at(buf, size, "%s %16s %s %s %s %s\n",title[0],
title[1], title[2], title[3], title[4], title[5]);
int size = 0;
uint32_t i, now, clock, pcie_speed;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
smum_send_msg_to_smc(hwmgr, PPSMC_MSG_API_GetSclkFrequency, &clock);
break;
case OD_SCLK:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_SCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_SCLK");
for (i = 0; i < odn_sclk_table->num_of_pl; i++)
size += sysfs_emit_at(buf, size, "%d: %10uMHz %10umV\n",
i, odn_sclk_table->entries[i].clock/100,
break;
case OD_MCLK:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_MCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_MCLK");
for (i = 0; i < odn_mclk_table->num_of_pl; i++)
size += sysfs_emit_at(buf, size, "%d: %10uMHz %10umV\n",
i, odn_mclk_table->entries[i].clock/100,
break;
case OD_RANGE:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_RANGE");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_RANGE");
size += sysfs_emit_at(buf, size, "SCLK: %7uMHz %10uMHz\n",
data->golden_dpm_table.sclk_table.dpm_levels[0].value/100,
hwmgr->platform_descriptor.overdriveLimit.engineClock/100);
if (!buf)
return -EINVAL;
+ phm_get_sysfs_buf(&buf, &size);
+
size += sysfs_emit_at(buf, size, "%s %16s %16s %16s %16s %16s %16s %16s\n",
title[0], title[1], title[2], title[3],
title[4], title[5], title[6], title[7]);
uint32_t i, now;
int size = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
now = PHM_GET_FIELD(cgs_read_ind_register(hwmgr->device,
struct amdgpu_irq_src *source,
struct amdgpu_iv_entry *entry);
+/*
+ * Helper function to make sysfs_emit_at() happy. Align buf to
+ * the current page boundary and record the offset.
+ */
+static inline void phm_get_sysfs_buf(char **buf, int *offset)
+{
+ if (!*buf || !offset)
+ return;
+
+ *offset = offset_in_page(*buf);
+ *buf -= *offset;
+}
+
int smu9_register_irq_handlers(struct pp_hwmgr *hwmgr);
void *smu_atom_get_data_table(void *dev, uint32_t table, uint16_t *size,
int ret = 0;
int size = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
ret = vega10_get_enabled_smc_features(hwmgr, &features_enabled);
PP_ASSERT_WITH_CODE(!ret,
"[EnableAllSmuFeatures] Failed to get enabled smc features!",
int i, now, size = 0, count = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
if (data->registry_data.sclk_dpm_key_disabled)
case OD_SCLK:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_SCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_SCLK");
podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_sclk;
for (i = 0; i < podn_vdd_dep->count; i++)
size += sysfs_emit_at(buf, size, "%d: %10uMhz %10umV\n",
break;
case OD_MCLK:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_MCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_MCLK");
podn_vdd_dep = &data->odn_dpm_table.vdd_dep_on_mclk;
for (i = 0; i < podn_vdd_dep->count; i++)
size += sysfs_emit_at(buf, size, "%d: %10uMhz %10umV\n",
break;
case OD_RANGE:
if (hwmgr->od_enabled) {
- size = sysfs_emit(buf, "%s:\n", "OD_RANGE");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_RANGE");
size += sysfs_emit_at(buf, size, "SCLK: %7uMHz %10uMHz\n",
data->golden_dpm_table.gfx_table.dpm_levels[0].value/100,
hwmgr->platform_descriptor.overdriveLimit.engineClock/100);
if (!buf)
return -EINVAL;
+ phm_get_sysfs_buf(&buf, &size);
+
size += sysfs_emit_at(buf, size, "%s %16s %s %s %s %s\n",title[0],
title[1], title[2], title[3], title[4], title[5]);
int ret = 0;
int size = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
ret = vega12_get_enabled_smc_features(hwmgr, &features_enabled);
PP_ASSERT_WITH_CODE(!ret,
"[EnableAllSmuFeatures] Failed to get enabled smc features!",
int i, now, size = 0;
struct pp_clock_levels_with_latency clocks;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
PP_ASSERT_WITH_CODE(
int ret = 0;
int size = 0;
+ phm_get_sysfs_buf(&buf, &size);
+
ret = vega20_get_enabled_smc_features(hwmgr, &features_enabled);
PP_ASSERT_WITH_CODE(!ret,
"[EnableAllSmuFeatures] Failed to get enabled smc features!",
int ret = 0;
uint32_t gen_speed, lane_width, current_gen_speed, current_lane_width;
+ phm_get_sysfs_buf(&buf, &size);
+
switch (type) {
case PP_SCLK:
ret = vega20_get_current_clk_freq(hwmgr, PPCLK_GFXCLK, &now);
case OD_SCLK:
if (od8_settings[OD8_SETTING_GFXCLK_FMIN].feature_id &&
od8_settings[OD8_SETTING_GFXCLK_FMAX].feature_id) {
- size = sysfs_emit(buf, "%s:\n", "OD_SCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_SCLK");
size += sysfs_emit_at(buf, size, "0: %10uMhz\n",
od_table->GfxclkFmin);
size += sysfs_emit_at(buf, size, "1: %10uMhz\n",
case OD_MCLK:
if (od8_settings[OD8_SETTING_UCLK_FMAX].feature_id) {
- size = sysfs_emit(buf, "%s:\n", "OD_MCLK");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_MCLK");
size += sysfs_emit_at(buf, size, "1: %10uMhz\n",
od_table->UclkFmax);
}
od8_settings[OD8_SETTING_GFXCLK_VOLTAGE1].feature_id &&
od8_settings[OD8_SETTING_GFXCLK_VOLTAGE2].feature_id &&
od8_settings[OD8_SETTING_GFXCLK_VOLTAGE3].feature_id) {
- size = sysfs_emit(buf, "%s:\n", "OD_VDDC_CURVE");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_VDDC_CURVE");
size += sysfs_emit_at(buf, size, "0: %10uMhz %10dmV\n",
od_table->GfxclkFreq1,
od_table->GfxclkVolt1 / VOLTAGE_SCALE);
break;
case OD_RANGE:
- size = sysfs_emit(buf, "%s:\n", "OD_RANGE");
+ size += sysfs_emit_at(buf, size, "%s:\n", "OD_RANGE");
if (od8_settings[OD8_SETTING_GFXCLK_FMIN].feature_id &&
od8_settings[OD8_SETTING_GFXCLK_FMAX].feature_id) {
if (!buf)
return -EINVAL;
+ phm_get_sysfs_buf(&buf, &size);
+
size += sysfs_emit_at(buf, size, "%16s %s %s %s %s %s %s %s %s %s %s\n",
title[0], title[1], title[2], title[3], title[4], title[5],
title[6], title[7], title[8], title[9], title[10]);
dev_err(adev->dev, "Failed to disable smu features.\n");
}
- if (adev->ip_versions[MP1_HWIP][0] >= IP_VERSION(11, 0, 0) &&
+ if (adev->ip_versions[GC_HWIP][0] >= IP_VERSION(10, 0, 0) &&
adev->gfx.rlc.funcs->stop)
adev->gfx.rlc.funcs->stop(adev);
struct smu_context *smu = handle;
int ret = 0;
- if (!smu->pm_enabled || !smu->adev->pm.dpm_enabled)
+ if (!smu->pm_enabled || !smu->adev->pm.dpm_enabled ||
+ !smu->ppt_funcs->get_power_profile_mode)
return -EOPNOTSUPP;
+ if (!buf)
+ return -EINVAL;
mutex_lock(&smu->mutex);
- if (smu->ppt_funcs->get_power_profile_mode)
- ret = smu->ppt_funcs->get_power_profile_mode(smu, buf);
+ ret = smu->ppt_funcs->get_power_profile_mode(smu, buf);
mutex_unlock(&smu->mutex);
struct smu_context *smu = handle;
int ret = 0;
- if (!smu->pm_enabled || !smu->adev->pm.dpm_enabled)
+ if (!smu->pm_enabled || !smu->adev->pm.dpm_enabled ||
+ !smu->ppt_funcs->set_power_profile_mode)
return -EOPNOTSUPP;
mutex_lock(&smu->mutex);
{
int ret = 0, size = 0;
uint32_t cur_value = 0;
+ int i;
smu_cmn_get_sysfs_buf(&buf, &size);
size += sysfs_emit_at(buf, size, "VDDC: %7umV %10umV\n",
CYAN_SKILLFISH_VDDC_MIN, CYAN_SKILLFISH_VDDC_MAX);
break;
- case SMU_GFXCLK:
- case SMU_SCLK:
case SMU_FCLK:
case SMU_MCLK:
case SMU_SOCCLK:
return ret;
size += sysfs_emit_at(buf, size, "0: %uMhz *\n", cur_value);
break;
+ case SMU_SCLK:
+ case SMU_GFXCLK:
+ ret = cyan_skillfish_get_current_clk_freq(smu, clk_type, &cur_value);
+ if (ret)
+ return ret;
+ if (cur_value == CYAN_SKILLFISH_SCLK_MAX)
+ i = 2;
+ else if (cur_value == CYAN_SKILLFISH_SCLK_MIN)
+ i = 0;
+ else
+ i = 1;
+ size += sysfs_emit_at(buf, size, "0: %uMhz %s\n", CYAN_SKILLFISH_SCLK_MIN,
+ i == 0 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "1: %uMhz %s\n",
+ i == 1 ? cur_value : cyan_skillfish_sclk_default,
+ i == 1 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "2: %uMhz %s\n", CYAN_SKILLFISH_SCLK_MAX,
+ i == 2 ? "*" : "");
+ break;
default:
dev_warn(smu->adev->dev, "Unsupported clock type\n");
return ret;
enum smu_clk_type clk_type, char *buf)
{
uint16_t *curve_settings;
- int i, size = 0, ret = 0;
+ int i, levels, size = 0, ret = 0;
uint32_t cur_value = 0, value = 0, count = 0;
uint32_t freq_values[3] = {0};
uint32_t mark_index = 0;
freq_values[1] = cur_value;
mark_index = cur_value == freq_values[0] ? 0 :
cur_value == freq_values[2] ? 2 : 1;
- if (mark_index != 1)
- freq_values[1] = (freq_values[0] + freq_values[2]) / 2;
- for (i = 0; i < 3; i++) {
+ levels = 3;
+ if (mark_index != 1) {
+ levels = 2;
+ freq_values[1] = freq_values[2];
+ }
+
+ for (i = 0; i < levels; i++) {
size += sysfs_emit_at(buf, size, "%d: %uMhz %s\n", i, freq_values[i],
i == mark_index ? "*" : "");
}
-
}
break;
case SMU_PCIE:
int i, size = 0, ret = 0;
uint32_t cur_value = 0, value = 0, count = 0;
bool cur_value_match_level = false;
+ uint32_t min, max;
memset(&metrics, 0, sizeof(metrics));
if (ret)
return ret;
break;
+ case SMU_GFXCLK:
+ case SMU_SCLK:
+ ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_GetGfxclkFrequency, 0, &cur_value);
+ if (ret) {
+ return ret;
+ }
+ break;
default:
break;
}
if (!cur_value_match_level)
size += sysfs_emit_at(buf, size, " %uMhz *\n", cur_value);
break;
+ case SMU_GFXCLK:
+ case SMU_SCLK:
+ min = (smu->gfx_actual_hard_min_freq > 0) ? smu->gfx_actual_hard_min_freq : smu->gfx_default_hard_min_freq;
+ max = (smu->gfx_actual_soft_max_freq > 0) ? smu->gfx_actual_soft_max_freq : smu->gfx_default_soft_max_freq;
+ if (cur_value == max)
+ i = 2;
+ else if (cur_value == min)
+ i = 0;
+ else
+ i = 1;
+ size += sysfs_emit_at(buf, size, "0: %uMhz %s\n", min,
+ i == 0 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "1: %uMhz %s\n",
+ i == 1 ? cur_value : VANGOGH_UMD_PSTATE_STANDARD_GFXCLK,
+ i == 1 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "2: %uMhz %s\n", max,
+ i == 2 ? "*" : "");
+ break;
default:
break;
}
MSG_MAP(PowerDownVcn, PPSMC_MSG_PowerDownVcn, 1),
MSG_MAP(PowerUpVcn, PPSMC_MSG_PowerUpVcn, 1),
MSG_MAP(SetHardMinVcn, PPSMC_MSG_SetHardMinVcn, 1),
- MSG_MAP(ActiveProcessNotify, PPSMC_MSG_ActiveProcessNotify, 1),
MSG_MAP(PrepareMp1ForUnload, PPSMC_MSG_PrepareMp1ForUnload, 1),
MSG_MAP(SetDriverDramAddrHigh, PPSMC_MSG_SetDriverDramAddrHigh, 1),
MSG_MAP(SetDriverDramAddrLow, PPSMC_MSG_SetDriverDramAddrLow, 1),
TAB_MAP_VALID(CUSTOM_DPM),
TAB_MAP_VALID(DPMCLOCKS),
};
-
-static struct cmn2asic_mapping yellow_carp_workload_map[PP_SMC_POWER_PROFILE_COUNT] = {
- WORKLOAD_MAP(PP_SMC_POWER_PROFILE_FULLSCREEN3D, WORKLOAD_PPLIB_FULL_SCREEN_3D_BIT),
- WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VIDEO, WORKLOAD_PPLIB_VIDEO_BIT),
- WORKLOAD_MAP(PP_SMC_POWER_PROFILE_VR, WORKLOAD_PPLIB_VR_BIT),
- WORKLOAD_MAP(PP_SMC_POWER_PROFILE_COMPUTE, WORKLOAD_PPLIB_COMPUTE_BIT),
- WORKLOAD_MAP(PP_SMC_POWER_PROFILE_CUSTOM, WORKLOAD_PPLIB_CUSTOM_BIT),
-};
static int yellow_carp_init_smc_tables(struct smu_context *smu)
{
return 0;
}
-static int yellow_carp_get_power_profile_mode(struct smu_context *smu,
- char *buf)
-{
- static const char *profile_name[] = {
- "BOOTUP_DEFAULT",
- "3D_FULL_SCREEN",
- "POWER_SAVING",
- "VIDEO",
- "VR",
- "COMPUTE",
- "CUSTOM"};
- uint32_t i, size = 0;
- int16_t workload_type = 0;
-
- if (!buf)
- return -EINVAL;
-
- for (i = 0; i <= PP_SMC_POWER_PROFILE_CUSTOM; i++) {
- /*
- * Conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT.
- * Not all profile modes are supported on yellow carp.
- */
- workload_type = smu_cmn_to_asic_specific_index(smu,
- CMN2ASIC_MAPPING_WORKLOAD,
- i);
-
- if (workload_type < 0)
- continue;
-
- size += sysfs_emit_at(buf, size, "%2d %14s%s\n",
- i, profile_name[i], (i == smu->power_profile_mode) ? "*" : " ");
- }
-
- return size;
-}
-
-static int yellow_carp_set_power_profile_mode(struct smu_context *smu,
- long *input, uint32_t size)
-{
- int workload_type, ret;
- uint32_t profile_mode = input[size];
-
- if (profile_mode > PP_SMC_POWER_PROFILE_CUSTOM) {
- dev_err(smu->adev->dev, "Invalid power profile mode %d\n", profile_mode);
- return -EINVAL;
- }
-
- if (profile_mode == PP_SMC_POWER_PROFILE_BOOTUP_DEFAULT ||
- profile_mode == PP_SMC_POWER_PROFILE_POWERSAVING)
- return 0;
-
- /* conv PP_SMC_POWER_PROFILE* to WORKLOAD_PPLIB_*_BIT */
- workload_type = smu_cmn_to_asic_specific_index(smu,
- CMN2ASIC_MAPPING_WORKLOAD,
- profile_mode);
- if (workload_type < 0) {
- dev_dbg(smu->adev->dev, "Unsupported power profile mode %d on YELLOWCARP\n",
- profile_mode);
- return -EINVAL;
- }
-
- ret = smu_cmn_send_smc_msg_with_param(smu, SMU_MSG_ActiveProcessNotify,
- 1 << workload_type,
- NULL);
- if (ret) {
- dev_err_once(smu->adev->dev, "Fail to set workload type %d\n",
- workload_type);
- return ret;
- }
-
- smu->power_profile_mode = profile_mode;
-
- return 0;
-}
-
static ssize_t yellow_carp_get_gpu_metrics(struct smu_context *smu,
void **table)
{
case SMU_FCLK:
return smu_cmn_send_smc_msg_with_param(smu,
SMU_MSG_GetFclkFrequency, 0, value);
+ case SMU_GFXCLK:
+ case SMU_SCLK:
+ return smu_cmn_send_smc_msg_with_param(smu,
+ SMU_MSG_GetGfxclkFrequency, 0, value);
+ break;
default:
return -EINVAL;
}
{
int i, size = 0, ret = 0;
uint32_t cur_value = 0, value = 0, count = 0;
+ uint32_t min, max;
smu_cmn_get_sysfs_buf(&buf, &size);
cur_value == value ? "*" : "");
}
break;
+ case SMU_GFXCLK:
+ case SMU_SCLK:
+ ret = yellow_carp_get_current_clk_freq(smu, clk_type, &cur_value);
+ if (ret)
+ goto print_clk_out;
+ min = (smu->gfx_actual_hard_min_freq > 0) ? smu->gfx_actual_hard_min_freq : smu->gfx_default_hard_min_freq;
+ max = (smu->gfx_actual_soft_max_freq > 0) ? smu->gfx_actual_soft_max_freq : smu->gfx_default_soft_max_freq;
+ if (cur_value == max)
+ i = 2;
+ else if (cur_value == min)
+ i = 0;
+ else
+ i = 1;
+ size += sysfs_emit_at(buf, size, "0: %uMhz %s\n", min,
+ i == 0 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "1: %uMhz %s\n",
+ i == 1 ? cur_value : YELLOW_CARP_UMD_PSTATE_GFXCLK,
+ i == 1 ? "*" : "");
+ size += sysfs_emit_at(buf, size, "2: %uMhz %s\n", max,
+ i == 2 ? "*" : "");
+ break;
default:
break;
}
.read_sensor = yellow_carp_read_sensor,
.is_dpm_running = yellow_carp_is_dpm_running,
.set_watermarks_table = yellow_carp_set_watermarks_table,
- .get_power_profile_mode = yellow_carp_get_power_profile_mode,
- .set_power_profile_mode = yellow_carp_set_power_profile_mode,
.get_gpu_metrics = yellow_carp_get_gpu_metrics,
.get_enabled_mask = smu_cmn_get_enabled_32_bits_mask,
.get_pp_feature_mask = smu_cmn_get_pp_feature_mask,
smu->message_map = yellow_carp_message_map;
smu->feature_map = yellow_carp_feature_mask_map;
smu->table_map = yellow_carp_table_map;
- smu->workload_map = yellow_carp_workload_map;
smu->is_apu = true;
}
#define __YELLOW_CARP_PPT_H__
extern void yellow_carp_set_ppt_funcs(struct smu_context *smu);
+#define YELLOW_CARP_UMD_PSTATE_GFXCLK 1100
#endif
const char *message = smu_get_message_name(smu, msg);
switch (reg_c2pmsg_90) {
- case SMU_RESP_NONE:
+ case SMU_RESP_NONE: {
+ u32 msg_idx = RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_66);
+ u32 prm = RREG32_SOC15(MP1, 0, mmMP1_SMN_C2PMSG_82);
dev_err_ratelimited(adev->dev,
- "SMU: I'm not done with your previous command!");
+ "SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x%08X SMN_C2PMSG_82:0x%08X",
+ msg_idx, prm);
+ }
break;
case SMU_RESP_OK:
/* The SMU executed the command. It completed with a
struct lt9611uxc *lt9611uxc = container_of(work, struct lt9611uxc, work);
bool connected;
- if (lt9611uxc->connector.dev)
- drm_kms_helper_hotplug_event(lt9611uxc->connector.dev);
- else {
+ if (lt9611uxc->connector.dev) {
+ if (lt9611uxc->connector.dev->mode_config.funcs)
+ drm_kms_helper_hotplug_event(lt9611uxc->connector.dev);
+ } else {
mutex_lock(<9611uxc->ocm_lock);
connected = lt9611uxc->hdmi_connected;
return -ENODEV;
}
+ lt9611uxc->connector.polled = DRM_CONNECTOR_POLL_HPD;
+
drm_connector_helper_add(<9611uxc->connector,
<9611uxc_bridge_connector_helper_funcs);
ret = drm_connector_init(bridge->dev, <9611uxc->connector,
#include <linux/platform_device.h>
#include <linux/regulator/consumer.h>
+#include <drm/drm_atomic_helper.h>
#include <drm/drm_bridge.h>
#include <drm/drm_panel.h>
struct regulator *vcc;
struct gpio_desc *powerdown_gpio;
u32 connector_type;
+ unsigned int bus_format;
};
static inline struct lvds_codec *to_lvds_codec(struct drm_bridge *bridge)
.disable = lvds_codec_disable,
};
+#define MAX_INPUT_SEL_FORMATS 1
+static u32 *
+lvds_codec_atomic_get_input_bus_fmts(struct drm_bridge *bridge,
+ struct drm_bridge_state *bridge_state,
+ struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state,
+ u32 output_fmt,
+ unsigned int *num_input_fmts)
+{
+ struct lvds_codec *lvds_codec = to_lvds_codec(bridge);
+ u32 *input_fmts;
+
+ *num_input_fmts = 0;
+
+ input_fmts = kcalloc(MAX_INPUT_SEL_FORMATS, sizeof(*input_fmts),
+ GFP_KERNEL);
+ if (!input_fmts)
+ return NULL;
+
+ input_fmts[0] = lvds_codec->bus_format;
+ *num_input_fmts = MAX_INPUT_SEL_FORMATS;
+
+ return input_fmts;
+}
+
+static const struct drm_bridge_funcs funcs_decoder = {
+ .attach = lvds_codec_attach,
+ .enable = lvds_codec_enable,
+ .disable = lvds_codec_disable,
+ .atomic_duplicate_state = drm_atomic_helper_bridge_duplicate_state,
+ .atomic_destroy_state = drm_atomic_helper_bridge_destroy_state,
+ .atomic_reset = drm_atomic_helper_bridge_reset,
+ .atomic_get_input_bus_fmts = lvds_codec_atomic_get_input_bus_fmts,
+};
+
static int lvds_codec_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct device_node *panel_node;
+ struct device_node *bus_node;
struct drm_panel *panel;
struct lvds_codec *lvds_codec;
+ const char *mapping;
+ int ret;
lvds_codec = devm_kzalloc(dev, sizeof(*lvds_codec), GFP_KERNEL);
if (!lvds_codec)
if (IS_ERR(lvds_codec->panel_bridge))
return PTR_ERR(lvds_codec->panel_bridge);
+ lvds_codec->bridge.funcs = &funcs;
+
+ /*
+ * Decoder input LVDS format is a property of the decoder chip or even
+ * its strapping. Handle data-mapping the same way lvds-panel does. In
+ * case data-mapping is not present, do nothing, since there are still
+ * legacy bindings which do not specify this property.
+ */
+ if (lvds_codec->connector_type != DRM_MODE_CONNECTOR_LVDS) {
+ bus_node = of_graph_get_endpoint_by_regs(dev->of_node, 0, 0);
+ if (!bus_node) {
+ dev_dbg(dev, "bus DT node not found\n");
+ return -ENXIO;
+ }
+
+ ret = of_property_read_string(bus_node, "data-mapping",
+ &mapping);
+ of_node_put(bus_node);
+ if (ret < 0) {
+ dev_warn(dev, "missing 'data-mapping' DT property\n");
+ } else {
+ if (!strcmp(mapping, "jeida-18")) {
+ lvds_codec->bus_format = MEDIA_BUS_FMT_RGB666_1X7X3_SPWG;
+ } else if (!strcmp(mapping, "jeida-24")) {
+ lvds_codec->bus_format = MEDIA_BUS_FMT_RGB888_1X7X4_JEIDA;
+ } else if (!strcmp(mapping, "vesa-24")) {
+ lvds_codec->bus_format = MEDIA_BUS_FMT_RGB888_1X7X4_SPWG;
+ } else {
+ dev_err(dev, "invalid 'data-mapping' DT property\n");
+ return -EINVAL;
+ }
+ lvds_codec->bridge.funcs = &funcs_decoder;
+ }
+ }
+
/*
* The panel_bridge bridge is attached to the panel's of_node,
* but we need a bridge attached to our of_node for our user
* to look up.
*/
lvds_codec->bridge.of_node = dev->of_node;
- lvds_codec->bridge.funcs = &funcs;
drm_bridge_add(&lvds_codec->bridge);
platform_set_drvdata(pdev, lvds_codec);
drm_of_panel_bridge_remove(dsi->dev->of_node, 1, 0);
}
+static u32 *nwl_bridge_atomic_get_input_bus_fmts(struct drm_bridge *bridge,
+ struct drm_bridge_state *bridge_state,
+ struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state,
+ u32 output_fmt,
+ unsigned int *num_input_fmts)
+{
+ u32 *input_fmts, input_fmt;
+
+ *num_input_fmts = 0;
+
+ switch (output_fmt) {
+ /* If MEDIA_BUS_FMT_FIXED is tested, return default bus format */
+ case MEDIA_BUS_FMT_FIXED:
+ input_fmt = MEDIA_BUS_FMT_RGB888_1X24;
+ break;
+ case MEDIA_BUS_FMT_RGB888_1X24:
+ case MEDIA_BUS_FMT_RGB666_1X18:
+ case MEDIA_BUS_FMT_RGB565_1X16:
+ input_fmt = output_fmt;
+ break;
+ default:
+ return NULL;
+ }
+
+ input_fmts = kcalloc(1, sizeof(*input_fmts), GFP_KERNEL);
+ if (!input_fmts)
+ return NULL;
+ input_fmts[0] = input_fmt;
+ *num_input_fmts = 1;
+
+ return input_fmts;
+}
+
static const struct drm_bridge_funcs nwl_dsi_bridge_funcs = {
.atomic_duplicate_state = drm_atomic_helper_bridge_duplicate_state,
.atomic_destroy_state = drm_atomic_helper_bridge_destroy_state,
.atomic_check = nwl_dsi_bridge_atomic_check,
.atomic_enable = nwl_dsi_bridge_atomic_enable,
.atomic_disable = nwl_dsi_bridge_atomic_disable,
+ .atomic_get_input_bus_fmts = nwl_bridge_atomic_get_input_bus_fmts,
.mode_set = nwl_dsi_bridge_mode_set,
.mode_valid = nwl_dsi_bridge_mode_valid,
.attach = nwl_dsi_bridge_attach,
return ret;
}
+static void sn65dsi83_detach(struct drm_bridge *bridge)
+{
+ struct sn65dsi83 *ctx = bridge_to_sn65dsi83(bridge);
+
+ if (!ctx->dsi)
+ return;
+
+ mipi_dsi_detach(ctx->dsi);
+ mipi_dsi_device_unregister(ctx->dsi);
+ drm_bridge_remove(&ctx->bridge);
+ ctx->dsi = NULL;
+}
+
static void sn65dsi83_atomic_pre_enable(struct drm_bridge *bridge,
struct drm_bridge_state *old_bridge_state)
{
static const struct drm_bridge_funcs sn65dsi83_funcs = {
.attach = sn65dsi83_attach,
+ .detach = sn65dsi83_detach,
.atomic_pre_enable = sn65dsi83_atomic_pre_enable,
.atomic_enable = sn65dsi83_atomic_enable,
.atomic_disable = sn65dsi83_atomic_disable,
{
struct sn65dsi83 *ctx = i2c_get_clientdata(client);
- mipi_dsi_detach(ctx->dsi);
- mipi_dsi_device_unregister(ctx->dsi);
- drm_bridge_remove(&ctx->bridge);
of_node_put(ctx->host_node);
return 0;
*
* In contrast to the other drm_get_*_name functions this one here returns a
* const pointer and hence is threadsafe.
+ *
+ * Returns: connector status string
*/
const char *drm_get_connector_status_name(enum drm_connector_status status)
{
* drm_connector_list_iter_next - return next connector
* @iter: connector_list iterator
*
- * Returns the next connector for @iter, or NULL when the list walk has
+ * Returns: the next connector for @iter, or NULL when the list walk has
* completed.
*/
struct drm_connector *
*
* Note you could abuse this and return something out of bounds, but that
* would be a caller error. No unscrubbed user data should make it here.
+ *
+ * Returns: string describing an enumerated subpixel property
*/
const char *drm_get_subpixel_order_name(enum subpixel_order order)
{
* Store the supported bus formats in display info structure.
* See MEDIA_BUS_FMT_* definitions in include/uapi/linux/media-bus-format.h for
* a full list of available formats.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
*/
int drm_display_info_set_bus_formats(struct drm_display_info *info,
const u32 *formats,
* @dev: DRM device
*
* Called by a driver the first time a DVI-I connector is made.
+ *
+ * Returns: %0
*/
int drm_mode_create_dvi_i_properties(struct drm_device *dev)
{
* Game:
* Content type is game
*
+ * The meaning of each content type is defined in CTA-861-G table 15.
+ *
* Drivers can set up this property by calling
* drm_connector_attach_content_type_property(). Decoding to
* infoframe values is done through drm_hdmi_avi_infoframe_content_type().
* @connector: connector to attach content type property on.
*
* Called by a driver the first time a HDMI connector is made.
+ *
+ * Returns: %0
*/
int drm_connector_attach_content_type_property(struct drm_connector *connector)
{
* creates the TV margin properties for a given device. No need to call this
* function for an SDTV connector, it's already called from
* drm_mode_create_tv_properties().
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
*/
int drm_mode_create_tv_margin_properties(struct drm_device *dev)
{
* the TV specific connector properties for a given device. Caller is
* responsible for allocating a list of format names and passing them to
* this routine.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
*/
int drm_mode_create_tv_properties(struct drm_device *dev,
unsigned int num_modes,
* Atomic drivers should use drm_connector_attach_scaling_mode_property()
* instead to correctly assign &drm_connector_state.scaling_mode
* in the atomic state.
+ *
+ * Returns: %0
*/
int drm_mode_create_scaling_mode_property(struct drm_device *dev)
{
* @dev: DRM device
*
* Create the suggested x/y offset property for connectors.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
*/
int drm_mode_create_suggested_offset_properties(struct drm_device *dev)
{
EXPORT_SYMBOL(drm_connector_set_panel_orientation);
/**
- * drm_connector_set_panel_orientation_with_quirk -
- * set the connector's panel_orientation after checking for quirks
+ * drm_connector_set_panel_orientation_with_quirk - set the
+ * connector's panel_orientation after checking for quirks
* @connector: connector for which to init the panel-orientation property.
* @panel_orientation: drm_panel_orientation value to set
* @width: width in pixels of the panel, used for panel quirk detection
/**
* drm_connector_oob_hotplug_event - Report out-of-band hotplug event to connector
- * @connector: connector to report the event on
+ * @connector_fwnode: fwnode_handle to report the event on
*
* On some hardware a hotplug event notification may come from outside the display
* driver / device. An example of this is some USB Type-C setups where the hardware
for (i = 0; i < history->len; i++) {
const struct drm_dp_mst_topology_ref_entry *entry =
&history->entries[i];
- ulong *entries;
- uint nr_entries;
u64 ts_nsec = entry->ts_nsec;
u32 rem_nsec = do_div(ts_nsec, 1000000000);
- nr_entries = stack_depot_fetch(entry->backtrace, &entries);
- stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 4);
+ stack_depot_snprint(entry->backtrace, buf, PAGE_SIZE, 4);
drm_printf(&p, " %d %ss (last at %5llu.%06u):\n%s",
entry->count,
struct drm_gem_object *obj,
bool write)
{
- int ret;
- struct dma_fence **fences;
- unsigned int i, fence_count;
-
- if (!write) {
- struct dma_fence *fence =
- dma_resv_get_excl_unlocked(obj->resv);
-
- return drm_gem_fence_array_add(fence_array, fence);
- }
+ struct dma_resv_iter cursor;
+ struct dma_fence *fence;
+ int ret = 0;
- ret = dma_resv_get_fences(obj->resv, NULL,
- &fence_count, &fences);
- if (ret || !fence_count)
- return ret;
-
- for (i = 0; i < fence_count; i++) {
- ret = drm_gem_fence_array_add(fence_array, fences[i]);
+ dma_resv_for_each_fence(&cursor, obj->resv, write, fence) {
+ ret = drm_gem_fence_array_add(fence_array, fence);
if (ret)
break;
}
-
- for (; i < fence_count; i++)
- dma_fence_put(fences[i]);
- kfree(fences);
return ret;
}
EXPORT_SYMBOL(drm_gem_fence_array_add_implicit);
dma_buf_vunmap(gem_obj->import_attach->dmabuf, &map);
drm_prime_gem_destroy(gem_obj, cma_obj->sgt);
} else if (cma_obj->vaddr) {
- dma_free_wc(gem_obj->dev->dev, cma_obj->base.size,
- cma_obj->vaddr, cma_obj->paddr);
+ if (cma_obj->map_noncoherent)
+ dma_free_noncoherent(gem_obj->dev->dev, cma_obj->base.size,
+ cma_obj->vaddr, cma_obj->paddr,
+ DMA_TO_DEVICE);
+ else
+ dma_free_wc(gem_obj->dev->dev, cma_obj->base.size,
+ cma_obj->vaddr, cma_obj->paddr);
}
drm_gem_object_release(gem_obj);
static void show_leaks(struct drm_mm *mm)
{
struct drm_mm_node *node;
- unsigned long *entries;
- unsigned int nr_entries;
char *buf;
buf = kmalloc(BUFSZ, GFP_KERNEL);
continue;
}
- nr_entries = stack_depot_fetch(node->stack, &entries);
- stack_trace_snprint(buf, BUFSZ, entries, nr_entries, 0);
+ stack_depot_snprint(node->stack, buf, BUFSZ, 0);
DRM_ERROR("node [%08llx + %08llx]: inserted at\n%s",
node->start, node->size, buf);
}
#include <drm/drm_crtc.h>
#include <drm/drm_device.h>
#include <drm/drm_modeset_lock.h>
+#include <drm/drm_print.h>
/**
* DOC: kms locking
static DEFINE_WW_CLASS(crtc_ww_class);
+#if IS_ENABLED(CONFIG_DRM_DEBUG_MODESET_LOCK)
+static noinline depot_stack_handle_t __drm_stack_depot_save(void)
+{
+ unsigned long entries[8];
+ unsigned int n;
+
+ n = stack_trace_save(entries, ARRAY_SIZE(entries), 1);
+
+ return stack_depot_save(entries, n, GFP_NOWAIT | __GFP_NOWARN);
+}
+
+static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
+{
+ struct drm_printer p = drm_debug_printer("drm_modeset_lock");
+ unsigned long *entries;
+ unsigned int nr_entries;
+ char *buf;
+
+ buf = kmalloc(PAGE_SIZE, GFP_NOWAIT | __GFP_NOWARN);
+ if (!buf)
+ return;
+
+ nr_entries = stack_depot_fetch(stack_depot, &entries);
+ stack_trace_snprint(buf, PAGE_SIZE, entries, nr_entries, 2);
+
+ drm_printf(&p, "attempting to lock a contended lock without backoff:\n%s", buf);
+
+ kfree(buf);
+}
+#else /* CONFIG_DRM_DEBUG_MODESET_LOCK */
+static depot_stack_handle_t __drm_stack_depot_save(void)
+{
+ return 0;
+}
+static void __drm_stack_depot_print(depot_stack_handle_t stack_depot)
+{
+}
+#endif /* CONFIG_DRM_DEBUG_MODESET_LOCK */
+
/**
* drm_modeset_lock_all - take all modeset locks
* @dev: DRM device
*/
void drm_modeset_drop_locks(struct drm_modeset_acquire_ctx *ctx)
{
- WARN_ON(ctx->contended);
+ if (WARN_ON(ctx->contended))
+ __drm_stack_depot_print(ctx->stack_depot);
+
while (!list_empty(&ctx->locked)) {
struct drm_modeset_lock *lock;
{
int ret;
- WARN_ON(ctx->contended);
+ if (WARN_ON(ctx->contended))
+ __drm_stack_depot_print(ctx->stack_depot);
if (ctx->trylock_only) {
lockdep_assert_held(&ctx->ww_ctx);
ret = 0;
} else if (ret == -EDEADLK) {
ctx->contended = lock;
+ ctx->stack_depot = __drm_stack_depot_save();
}
return ret;
struct drm_modeset_lock *contended = ctx->contended;
ctx->contended = NULL;
+ ctx->stack_depot = 0;
if (WARN_ON(!contended))
return 0;
.crtc_w = drm_rect_width(dst),
.crtc_h = drm_rect_height(dst),
.rotation = rotation,
- .visible = *visible,
};
struct drm_crtc_state crtc_state = {
.crtc = crtc,
if (obj->funcs && obj->funcs->mmap) {
vma->vm_ops = obj->funcs->vm_ops;
+ drm_gem_object_get(obj);
ret = obj->funcs->mmap(obj, vma);
- if (ret)
+ if (ret) {
+ drm_gem_object_put(obj);
return ret;
+ }
vma->vm_private_data = obj;
- drm_gem_object_get(obj);
return 0;
}
else
intel_encoder->enable = g4x_enable_hdmi;
}
+ intel_encoder->shutdown = intel_hdmi_encoder_shutdown;
intel_encoder->type = INTEL_OUTPUT_HDMI;
intel_encoder->power_domain = intel_port_to_power_domain(port);
intel_de_write(dev_priv, ICL_DPCLKA_CFGCR0, val);
for_each_dsi_phy(phy, intel_dsi->phys) {
- if (DISPLAY_VER(dev_priv) >= 12)
- val |= ICL_DPCLKA_CFGCR0_DDI_CLK_OFF(phy);
- else
- val &= ~ICL_DPCLKA_CFGCR0_DDI_CLK_OFF(phy);
+ val &= ~ICL_DPCLKA_CFGCR0_DDI_CLK_OFF(phy);
}
intel_de_write(dev_priv, ICL_DPCLKA_CFGCR0, val);
gen11_dsi_enable_port_and_phy(struct intel_encoder *encoder,
const struct intel_crtc_state *crtc_state)
{
- struct drm_i915_private *dev_priv = to_i915(encoder->base.dev);
-
/* step 4a: power up all lanes of the DDI used by DSI */
gen11_dsi_power_up_lanes(encoder);
gen11_dsi_configure_transcoder(encoder, crtc_state);
/* Step 4l: Gate DDI clocks */
- if (DISPLAY_VER(dev_priv) == 11)
- gen11_dsi_gate_clocks(encoder);
+ gen11_dsi_gate_clocks(encoder);
}
static void gen11_dsi_powerup_panel(struct intel_encoder *encoder)
if (DISPLAY_VER(i915) == 13) {
for_each_dsi_port(port, intel_dsi->ports)
intel_de_rmw(i915, TGL_DSI_CHKN_REG(port),
- TGL_DSI_CHKN_LSHS_GB, 0x4);
+ TGL_DSI_CHKN_LSHS_GB_MASK,
+ TGL_DSI_CHKN_LSHS_GB(4));
}
}
child->aux_channel = 0;
}
+static u8 dvo_port_type(u8 dvo_port)
+{
+ switch (dvo_port) {
+ case DVO_PORT_HDMIA:
+ case DVO_PORT_HDMIB:
+ case DVO_PORT_HDMIC:
+ case DVO_PORT_HDMID:
+ case DVO_PORT_HDMIE:
+ case DVO_PORT_HDMIF:
+ case DVO_PORT_HDMIG:
+ case DVO_PORT_HDMIH:
+ case DVO_PORT_HDMII:
+ return DVO_PORT_HDMIA;
+ case DVO_PORT_DPA:
+ case DVO_PORT_DPB:
+ case DVO_PORT_DPC:
+ case DVO_PORT_DPD:
+ case DVO_PORT_DPE:
+ case DVO_PORT_DPF:
+ case DVO_PORT_DPG:
+ case DVO_PORT_DPH:
+ case DVO_PORT_DPI:
+ return DVO_PORT_DPA;
+ case DVO_PORT_MIPIA:
+ case DVO_PORT_MIPIB:
+ case DVO_PORT_MIPIC:
+ case DVO_PORT_MIPID:
+ return DVO_PORT_MIPIA;
+ default:
+ return dvo_port;
+ }
+}
+
static enum port __dvo_port_to_port(int n_ports, int n_dvo,
const int port_mapping[][3], u8 dvo_port)
{
}
}
-static enum port get_edp_port(struct drm_i915_private *i915)
-{
- const struct intel_bios_encoder_data *devdata;
- enum port port;
-
- for_each_port(port) {
- devdata = i915->vbt.ports[port];
-
- if (devdata && intel_bios_encoder_supports_edp(devdata))
- return port;
- }
-
- return PORT_NONE;
-}
-
-/*
- * FIXME: The power sequencer and backlight code currently do not support more
- * than one set registers, at least not on anything other than VLV/CHV. It will
- * clobber the registers. As a temporary workaround, gracefully prevent more
- * than one eDP from being registered.
- */
-static void sanitize_dual_edp(struct intel_bios_encoder_data *devdata,
- enum port port)
-{
- struct drm_i915_private *i915 = devdata->i915;
- struct child_device_config *child = &devdata->child;
- enum port p;
-
- /* CHV might not clobber PPS registers. */
- if (IS_CHERRYVIEW(i915))
- return;
-
- p = get_edp_port(i915);
- if (p == PORT_NONE)
- return;
-
- drm_dbg_kms(&i915->drm, "both ports %c and %c configured as eDP, "
- "disabling port %c eDP\n", port_name(p), port_name(port),
- port_name(port));
-
- child->device_type &= ~DEVICE_TYPE_DISPLAYPORT_OUTPUT;
- child->device_type &= ~DEVICE_TYPE_INTERNAL_CONNECTOR;
-}
-
static bool is_port_valid(struct drm_i915_private *i915, enum port port)
{
/*
supports_typec_usb, supports_tbt,
devdata->dsc != NULL);
- if (is_edp)
- sanitize_dual_edp(devdata, port);
-
if (is_dvi)
sanitize_ddc_pin(devdata, port);
return false;
}
-static bool child_dev_is_dp_dual_mode(const struct child_device_config *child,
- enum port port)
+static bool child_dev_is_dp_dual_mode(const struct child_device_config *child)
{
- static const struct {
- u16 dp, hdmi;
- } port_mapping[] = {
- /*
- * Buggy VBTs may declare DP ports as having
- * HDMI type dvo_port :( So let's check both.
- */
- [PORT_B] = { DVO_PORT_DPB, DVO_PORT_HDMIB, },
- [PORT_C] = { DVO_PORT_DPC, DVO_PORT_HDMIC, },
- [PORT_D] = { DVO_PORT_DPD, DVO_PORT_HDMID, },
- [PORT_E] = { DVO_PORT_DPE, DVO_PORT_HDMIE, },
- [PORT_F] = { DVO_PORT_DPF, DVO_PORT_HDMIF, },
- };
-
- if (port == PORT_A || port >= ARRAY_SIZE(port_mapping))
- return false;
-
if ((child->device_type & DEVICE_TYPE_DP_DUAL_MODE_BITS) !=
(DEVICE_TYPE_DP_DUAL_MODE & DEVICE_TYPE_DP_DUAL_MODE_BITS))
return false;
- if (child->dvo_port == port_mapping[port].dp)
+ if (dvo_port_type(child->dvo_port) == DVO_PORT_DPA)
return true;
/* Only accept a HDMI dvo_port as DP++ if it has an AUX channel */
- if (child->dvo_port == port_mapping[port].hdmi &&
+ if (dvo_port_type(child->dvo_port) == DVO_PORT_HDMIA &&
child->aux_channel != 0)
return true;
bool intel_bios_is_port_dp_dual_mode(struct drm_i915_private *i915,
enum port port)
{
+ static const struct {
+ u16 dp, hdmi;
+ } port_mapping[] = {
+ /*
+ * Buggy VBTs may declare DP ports as having
+ * HDMI type dvo_port :( So let's check both.
+ */
+ [PORT_B] = { DVO_PORT_DPB, DVO_PORT_HDMIB, },
+ [PORT_C] = { DVO_PORT_DPC, DVO_PORT_HDMIC, },
+ [PORT_D] = { DVO_PORT_DPD, DVO_PORT_HDMID, },
+ [PORT_E] = { DVO_PORT_DPE, DVO_PORT_HDMIE, },
+ [PORT_F] = { DVO_PORT_DPF, DVO_PORT_HDMIF, },
+ };
const struct intel_bios_encoder_data *devdata;
+ if (HAS_DDI(i915)) {
+ const struct intel_bios_encoder_data *devdata;
+
+ devdata = intel_bios_encoder_data_lookup(i915, port);
+
+ return devdata && child_dev_is_dp_dual_mode(&devdata->child);
+ }
+
+ if (port == PORT_A || port >= ARRAY_SIZE(port_mapping))
+ return false;
+
list_for_each_entry(devdata, &i915->vbt.display_devices, node) {
- if (child_dev_is_dp_dual_mode(&devdata->child, port))
+ if ((devdata->child.dvo_port == port_mapping[port].dp ||
+ devdata->child.dvo_port == port_mapping[port].hdmi) &&
+ child_dev_is_dp_dual_mode(&devdata->child))
return true;
}
return freq;
}
-static struct intel_cdclk_funcs tgl_cdclk_funcs = {
+static const struct intel_cdclk_funcs tgl_cdclk_funcs = {
.get_cdclk = bxt_get_cdclk,
.set_cdclk = bxt_set_cdclk,
.bw_calc_min_cdclk = skl_bw_calc_min_cdclk,
.calc_voltage_level = tgl_calc_voltage_level,
};
-static struct intel_cdclk_funcs ehl_cdclk_funcs = {
+static const struct intel_cdclk_funcs ehl_cdclk_funcs = {
.get_cdclk = bxt_get_cdclk,
.set_cdclk = bxt_set_cdclk,
.bw_calc_min_cdclk = skl_bw_calc_min_cdclk,
.calc_voltage_level = ehl_calc_voltage_level,
};
-static struct intel_cdclk_funcs icl_cdclk_funcs = {
+static const struct intel_cdclk_funcs icl_cdclk_funcs = {
.get_cdclk = bxt_get_cdclk,
.set_cdclk = bxt_set_cdclk,
.bw_calc_min_cdclk = skl_bw_calc_min_cdclk,
.calc_voltage_level = icl_calc_voltage_level,
};
-static struct intel_cdclk_funcs bxt_cdclk_funcs = {
+static const struct intel_cdclk_funcs bxt_cdclk_funcs = {
.get_cdclk = bxt_get_cdclk,
.set_cdclk = bxt_set_cdclk,
.bw_calc_min_cdclk = skl_bw_calc_min_cdclk,
.calc_voltage_level = bxt_calc_voltage_level,
};
-static struct intel_cdclk_funcs skl_cdclk_funcs = {
+static const struct intel_cdclk_funcs skl_cdclk_funcs = {
.get_cdclk = skl_get_cdclk,
.set_cdclk = skl_set_cdclk,
.bw_calc_min_cdclk = skl_bw_calc_min_cdclk,
.modeset_calc_cdclk = skl_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs bdw_cdclk_funcs = {
+static const struct intel_cdclk_funcs bdw_cdclk_funcs = {
.get_cdclk = bdw_get_cdclk,
.set_cdclk = bdw_set_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = bdw_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs chv_cdclk_funcs = {
+static const struct intel_cdclk_funcs chv_cdclk_funcs = {
.get_cdclk = vlv_get_cdclk,
.set_cdclk = chv_set_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = vlv_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs vlv_cdclk_funcs = {
+static const struct intel_cdclk_funcs vlv_cdclk_funcs = {
.get_cdclk = vlv_get_cdclk,
.set_cdclk = vlv_set_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = vlv_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs hsw_cdclk_funcs = {
+static const struct intel_cdclk_funcs hsw_cdclk_funcs = {
.get_cdclk = hsw_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
/* SNB, IVB, 965G, 945G */
-static struct intel_cdclk_funcs fixed_400mhz_cdclk_funcs = {
+static const struct intel_cdclk_funcs fixed_400mhz_cdclk_funcs = {
.get_cdclk = fixed_400mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs ilk_cdclk_funcs = {
+static const struct intel_cdclk_funcs ilk_cdclk_funcs = {
.get_cdclk = fixed_450mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs gm45_cdclk_funcs = {
+static const struct intel_cdclk_funcs gm45_cdclk_funcs = {
.get_cdclk = gm45_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
/* G45 uses G33 */
-static struct intel_cdclk_funcs i965gm_cdclk_funcs = {
+static const struct intel_cdclk_funcs i965gm_cdclk_funcs = {
.get_cdclk = i965gm_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
/* i965G uses fixed 400 */
-static struct intel_cdclk_funcs pnv_cdclk_funcs = {
+static const struct intel_cdclk_funcs pnv_cdclk_funcs = {
.get_cdclk = pnv_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs g33_cdclk_funcs = {
+static const struct intel_cdclk_funcs g33_cdclk_funcs = {
.get_cdclk = g33_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i945gm_cdclk_funcs = {
+static const struct intel_cdclk_funcs i945gm_cdclk_funcs = {
.get_cdclk = i945gm_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
/* i945G uses fixed 400 */
-static struct intel_cdclk_funcs i915gm_cdclk_funcs = {
+static const struct intel_cdclk_funcs i915gm_cdclk_funcs = {
.get_cdclk = i915gm_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i915g_cdclk_funcs = {
+static const struct intel_cdclk_funcs i915g_cdclk_funcs = {
.get_cdclk = fixed_333mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i865g_cdclk_funcs = {
+static const struct intel_cdclk_funcs i865g_cdclk_funcs = {
.get_cdclk = fixed_266mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i85x_cdclk_funcs = {
+static const struct intel_cdclk_funcs i85x_cdclk_funcs = {
.get_cdclk = i85x_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i845g_cdclk_funcs = {
+static const struct intel_cdclk_funcs i845g_cdclk_funcs = {
.get_cdclk = fixed_200mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
};
-static struct intel_cdclk_funcs i830_cdclk_funcs = {
+static const struct intel_cdclk_funcs i830_cdclk_funcs = {
.get_cdclk = fixed_133mhz_get_cdclk,
.bw_calc_min_cdclk = intel_bw_calc_min_cdclk,
.modeset_calc_cdclk = fixed_modeset_calc_cdclk,
enum phy phy = intel_port_to_phy(i915, encoder->port);
intel_dp_encoder_shutdown(encoder);
+ intel_hdmi_encoder_shutdown(encoder);
if (!intel_phy_is_tc(i915, phy))
return;
int i;
for (i = 0 ; i < ARRAY_SIZE(rem_info->plane); i++) {
+ unsigned int plane_size;
+
+ plane_size = rem_info->plane[i].dst_stride * rem_info->plane[i].height;
+ if (plane_size == 0)
+ continue;
+
if (rem_info->plane_alignment)
size = ALIGN(size, rem_info->plane_alignment);
- size += rem_info->plane[i].dst_stride * rem_info->plane[i].height;
+
+ size += plane_size;
}
return size;
return crtc_state->port_clock >= 1000000;
}
+static void intel_dp_set_default_sink_rates(struct intel_dp *intel_dp)
+{
+ intel_dp->sink_rates[0] = 162000;
+ intel_dp->num_sink_rates = 1;
+}
+
/* update sink rates from dpcd */
static void intel_dp_set_sink_rates(struct intel_dp *intel_dp)
{
*/
int max_link_rate_kbps = max_link_rate * 10;
- max_link_rate_kbps = DIV_ROUND_CLOSEST_ULL(max_link_rate_kbps * 9671, 10000);
+ max_link_rate_kbps = DIV_ROUND_CLOSEST_ULL(mul_u32_u32(max_link_rate_kbps, 9671), 10000);
max_link_rate = max_link_rate_kbps / 8;
}
intel_dp->lane_count = lane_count;
}
+static void intel_dp_reset_max_link_params(struct intel_dp *intel_dp)
+{
+ intel_dp->max_link_lane_count = intel_dp_max_common_lane_count(intel_dp);
+ intel_dp->max_link_rate = intel_dp_max_common_rate(intel_dp);
+}
+
/* Enable backlight PWM and backlight PP control. */
void intel_edp_backlight_on(const struct intel_crtc_state *crtc_state,
const struct drm_connector_state *conn_state)
if (intel_dp->dpcd[DP_DPCD_REV] == 0)
intel_dp_get_dpcd(intel_dp);
- intel_dp->max_link_lane_count = intel_dp_max_common_lane_count(intel_dp);
- intel_dp->max_link_rate = intel_dp_max_common_rate(intel_dp);
+ intel_dp_reset_max_link_params(intel_dp);
}
bool intel_dp_initial_fastset_check(struct intel_encoder *encoder,
*/
intel_psr_init_dpcd(intel_dp);
+ /* Clear the default sink rates */
+ intel_dp->num_sink_rates = 0;
+
/* Read the eDP 1.4+ supported link rates. */
if (intel_dp->edp_dpcd[0] >= DP_EDP_14) {
__le16 sink_rates[DP_MAX_SUPPORTED_RATES];
intel_dp_set_sink_rates(intel_dp);
intel_dp_set_common_rates(intel_dp);
+ intel_dp_reset_max_link_params(intel_dp);
/* Read the eDP DSC DPCD registers */
if (DISPLAY_VER(dev_priv) >= 10)
* supports link training fallback params.
*/
if (intel_dp->reset_link_params || intel_dp->is_mst) {
- /* Initial max link lane count */
- intel_dp->max_link_lane_count = intel_dp_max_common_lane_count(intel_dp);
-
- /* Initial max link rate */
- intel_dp->max_link_rate = intel_dp_max_common_rate(intel_dp);
-
+ intel_dp_reset_max_link_params(intel_dp);
intel_dp->reset_link_params = false;
}
}
intel_dp_set_source_rates(intel_dp);
+ intel_dp_set_default_sink_rates(intel_dp);
+ intel_dp_set_common_rates(intel_dp);
+ intel_dp_reset_max_link_params(intel_dp);
if (IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
intel_dp->pps.active_pipe = vlv_active_pipe(intel_dp);
intel_fb_plane_get_subsampling(&main_hsub, &main_vsub, &fb->base, main_plane);
intel_fb_plane_get_subsampling(&hsub, &vsub, &fb->base, color_plane);
- *w = main_width / main_hsub / hsub;
- *h = main_height / main_vsub / vsub;
+ *w = DIV_ROUND_UP(main_width, main_hsub * hsub);
+ *h = DIV_ROUND_UP(main_height, main_vsub * vsub);
}
static u32 intel_adjust_tile_offset(int *x, int *y,
void intel_dp_dual_mode_set_tmds_output(struct intel_hdmi *hdmi, bool enable)
{
struct drm_i915_private *dev_priv = intel_hdmi_to_i915(hdmi);
- struct i2c_adapter *adapter =
- intel_gmbus_get_adapter(dev_priv, hdmi->ddc_bus);
+ struct i2c_adapter *adapter;
if (hdmi->dp_dual_mode.type < DRM_DP_DUAL_MODE_TYPE2_DVI)
return;
+ adapter = intel_gmbus_get_adapter(dev_priv, hdmi->ddc_bus);
+
drm_dbg_kms(&dev_priv->drm, "%s DP dual mode adaptor TMDS output\n",
enable ? "Enabling" : "Disabling");
return 0;
}
+void intel_hdmi_encoder_shutdown(struct intel_encoder *encoder)
+{
+ struct intel_hdmi *intel_hdmi = enc_to_intel_hdmi(encoder);
+
+ /*
+ * Give a hand to buggy BIOSen which forget to turn
+ * the TMDS output buffers back on after a reboot.
+ */
+ intel_dp_dual_mode_set_tmds_output(intel_hdmi, true);
+}
+
static void
intel_hdmi_unset_edid(struct drm_connector *connector)
{
int intel_hdmi_compute_config(struct intel_encoder *encoder,
struct intel_crtc_state *pipe_config,
struct drm_connector_state *conn_state);
+void intel_hdmi_encoder_shutdown(struct intel_encoder *encoder);
bool intel_hdmi_handle_sink_scrambling(struct intel_encoder *encoder,
struct drm_connector *connector,
bool high_tmds_clock_ratio,
#include <linux/dma-resv.h>
#include <linux/module.h>
+#include <asm/smp.h>
+
#include "i915_drv.h"
#include "i915_gem_object.h"
#include "i915_scatterlist.h"
{
unsigned int row;
+ if (!width || !height)
+ return sg;
+
if (alignment_pad) {
st->nents++;
unsigned long flags;
bool disabled;
+ lockdep_assert_held(&guc->submission_state.lock);
GEM_BUG_ON(!intel_gt_pm_is_awake(gt));
GEM_BUG_ON(!lrc_desc_registered(guc, ce->guc_id.id));
GEM_BUG_ON(ce != __get_context(guc, ce->guc_id.id));
}
spin_unlock_irqrestore(&ce->guc_state.lock, flags);
if (unlikely(disabled)) {
- release_guc_id(guc, ce);
+ __release_guc_id(guc, ce);
__guc_context_destroy(ce);
return;
}
ce = intel_engine_create_virtual(siblings, num_siblings,
FORCE_VIRTUAL);
- if (!ce) {
- err = ERR_PTR(-ENOMEM);
+ if (IS_ERR(ce)) {
+ err = ERR_CAST(ce);
goto unwind;
}
#define TGL_DSI_CHKN_REG(port) _MMIO_PORT(port, \
_TGL_DSI_CHKN_REG_0, \
_TGL_DSI_CHKN_REG_1)
-#define TGL_DSI_CHKN_LSHS_GB REG_GENMASK(15, 12)
+#define TGL_DSI_CHKN_LSHS_GB_MASK REG_GENMASK(15, 12)
+#define TGL_DSI_CHKN_LSHS_GB(byte_clocks) REG_FIELD_PREP(TGL_DSI_CHKN_LSHS_GB_MASK, \
+ (byte_clocks))
/* Display Stream Splitter Control */
#define DSS_CTL1 _MMIO(0x67400)
struct drm_i915_gem_object *obj,
bool write)
{
- struct dma_fence *excl;
+ struct dma_resv_iter cursor;
+ struct dma_fence *fence;
int ret = 0;
- if (write) {
- struct dma_fence **shared;
- unsigned int count, i;
-
- ret = dma_resv_get_fences(obj->base.resv, &excl, &count,
- &shared);
+ dma_resv_for_each_fence(&cursor, obj->base.resv, write, fence) {
+ ret = i915_request_await_dma_fence(to, fence);
if (ret)
- return ret;
-
- for (i = 0; i < count; i++) {
- ret = i915_request_await_dma_fence(to, shared[i]);
- if (ret)
- break;
-
- dma_fence_put(shared[i]);
- }
-
- for (; i < count; i++)
- dma_fence_put(shared[i]);
- kfree(shared);
- } else {
- excl = dma_resv_get_excl_unlocked(obj->base.resv);
- }
-
- if (excl) {
- if (ret == 0)
- ret = i915_request_await_dma_fence(to, excl);
-
- dma_fence_put(excl);
+ break;
}
return ret;
static void vma_print_allocator(struct i915_vma *vma, const char *reason)
{
- unsigned long *entries;
- unsigned int nr_entries;
char buf[512];
if (!vma->node.stack) {
return;
}
- nr_entries = stack_depot_fetch(vma->node.stack, &entries);
- stack_trace_snprint(buf, sizeof(buf), entries, nr_entries, 0);
+ stack_depot_snprint(vma->node.stack, buf, sizeof(buf), 0);
DRM_DEBUG_DRIVER("vma.node [%08llx + %08llx] %s: inserted at %s\n",
vma->node.start, vma->node.size, reason, buf);
}
return stack_depot_save(entries, n, GFP_NOWAIT | __GFP_NOWARN);
}
-static void __print_depot_stack(depot_stack_handle_t stack,
- char *buf, int sz, int indent)
-{
- unsigned long *entries;
- unsigned int nr_entries;
-
- nr_entries = stack_depot_fetch(stack, &entries);
- stack_trace_snprint(buf, sz, entries, nr_entries, indent);
-}
-
static void init_intel_runtime_pm_wakeref(struct intel_runtime_pm *rpm)
{
spin_lock_init(&rpm->debug.lock);
if (!buf)
return;
- __print_depot_stack(stack, buf, PAGE_SIZE, 2);
+ stack_depot_snprint(stack, buf, PAGE_SIZE, 2);
DRM_DEBUG_DRIVER("wakeref %x from\n%s", stack, buf);
stack = READ_ONCE(rpm->debug.last_release);
if (stack) {
- __print_depot_stack(stack, buf, PAGE_SIZE, 2);
+ stack_depot_snprint(stack, buf, PAGE_SIZE, 2);
DRM_DEBUG_DRIVER("wakeref last released at\n%s", buf);
}
return;
if (dbg->last_acquire) {
- __print_depot_stack(dbg->last_acquire, buf, PAGE_SIZE, 2);
+ stack_depot_snprint(dbg->last_acquire, buf, PAGE_SIZE, 2);
drm_printf(p, "Wakeref last acquired:\n%s", buf);
}
if (dbg->last_release) {
- __print_depot_stack(dbg->last_release, buf, PAGE_SIZE, 2);
+ stack_depot_snprint(dbg->last_release, buf, PAGE_SIZE, 2);
drm_printf(p, "Wakeref last released:\n%s", buf);
}
rep = 1;
while (i + 1 < dbg->count && dbg->owners[i + 1] == stack)
rep++, i++;
- __print_depot_stack(stack, buf, PAGE_SIZE, 2);
+ stack_depot_snprint(stack, buf, PAGE_SIZE, 2);
drm_printf(p, "Wakeref x%lu taken at:\n%s", rep, buf);
}
struct drm_plane_state *old_plane_state, *new_plane_state;
bool plane_disabling = false;
int i;
- bool fence_cookie = dma_fence_begin_signalling();
drm_atomic_helper_commit_modeset_disables(dev, state);
}
drm_atomic_helper_commit_hw_done(state);
- dma_fence_end_signalling(fence_cookie);
}
static const struct drm_mode_config_helper_funcs imx_drm_mode_config_helpers = {
ctrl |= CTRL_BUS_WIDTH_24;
break;
default:
- dev_err(drm->dev, "Unknown media bus format %d\n", bus_format);
+ dev_err(drm->dev, "Unknown media bus format 0x%x\n", bus_format);
break;
}
drm_atomic_get_new_bridge_state(state,
mxsfb->bridge);
bus_format = bridge_state->input_bus_cfg.format;
+ if (bus_format == MEDIA_BUS_FMT_FIXED) {
+ dev_warn_once(drm->dev,
+ "Bridge does not provide bus format, assuming MEDIA_BUS_FMT_RGB888_1X24.\n"
+ "Please fix bridge driver by handling atomic_get_input_bus_fmts.\n");
+ bus_format = MEDIA_BUS_FMT_RGB888_1X24;
+ }
}
/* If there is no bridge, use bus format from connector */
{
struct ttm_tt *ttm_dma = (void *)ttm;
struct nouveau_drm *drm;
- struct device *dev;
bool slave = !!(ttm->page_flags & TTM_TT_FLAG_EXTERNAL);
if (ttm_tt_is_populated(ttm))
}
drm = nouveau_bdev(bdev);
- dev = drm->dev->dev;
return ttm_pool_alloc(&drm->ttm.bdev.pool, ttm, ctx);
}
struct ttm_tt *ttm)
{
struct nouveau_drm *drm;
- struct device *dev;
bool slave = !!(ttm->page_flags & TTM_TT_FLAG_EXTERNAL);
if (slave)
nouveau_ttm_tt_unbind(bdev, ttm);
drm = nouveau_bdev(bdev);
- dev = drm->dev->dev;
return ttm_pool_free(&drm->ttm.bdev.pool, ttm);
}
goto error_dma_unmap;
mutex_unlock(&svmm->mutex);
- args->dst[0] = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED;
+ args->dst[0] = migrate_pfn(page_to_pfn(dpage));
return 0;
error_dma_unmap:
((paddr >> PAGE_SHIFT) << NVIF_VMM_PFNMAP_V0_ADDR_SHIFT);
if (src & MIGRATE_PFN_WRITE)
*pfn |= NVIF_VMM_PFNMAP_V0_W;
- return migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED;
+ return migrate_pfn(page_to_pfn(dpage));
out_dma_unmap:
dma_unmap_page(dev, *dma_addr, PAGE_SIZE, DMA_BIDIRECTIONAL);
nvkm_dbgopt(nouveau_debug, "DRM");
INIT_LIST_HEAD(&drm->clients);
+ mutex_init(&drm->clients_lock);
spin_lock_init(&drm->tile.lock);
/* workaround an odd issue on nvc1 by disabling the device's
static void
nouveau_drm_device_fini(struct drm_device *dev)
{
+ struct nouveau_cli *cli, *temp_cli;
struct nouveau_drm *drm = nouveau_drm(dev);
if (nouveau_pmops_runtime()) {
nouveau_ttm_fini(drm);
nouveau_vga_fini(drm);
+ /*
+ * There may be existing clients from as-yet unclosed files. For now,
+ * clean them up here rather than deferring until the file is closed,
+ * but this likely not correct if we want to support hot-unplugging
+ * properly.
+ */
+ mutex_lock(&drm->clients_lock);
+ list_for_each_entry_safe(cli, temp_cli, &drm->clients, head) {
+ list_del(&cli->head);
+ mutex_lock(&cli->mutex);
+ if (cli->abi16)
+ nouveau_abi16_fini(cli->abi16);
+ mutex_unlock(&cli->mutex);
+ nouveau_cli_fini(cli);
+ kfree(cli);
+ }
+ mutex_unlock(&drm->clients_lock);
+
nouveau_cli_fini(&drm->client);
nouveau_cli_fini(&drm->master);
nvif_parent_dtor(&drm->parent);
+ mutex_destroy(&drm->clients_lock);
kfree(drm);
}
struct nvkm_client *client;
struct nvkm_device *device;
- drm_dev_unregister(dev);
+ drm_dev_unplug(dev);
client = nvxx_client(&drm->client.base);
device = nvkm_device_find(client->device);
fpriv->driver_priv = cli;
- mutex_lock(&drm->client.mutex);
+ mutex_lock(&drm->clients_lock);
list_add(&cli->head, &drm->clients);
- mutex_unlock(&drm->client.mutex);
+ mutex_unlock(&drm->clients_lock);
done:
if (ret && cli) {
{
struct nouveau_cli *cli = nouveau_cli(fpriv);
struct nouveau_drm *drm = nouveau_drm(dev);
+ int dev_index;
+
+ /*
+ * The device is gone, and as it currently stands all clients are
+ * cleaned up in the removal codepath. In the future this may change
+ * so that we can support hot-unplugging, but for now we immediately
+ * return to avoid a double-free situation.
+ */
+ if (!drm_dev_enter(dev, &dev_index))
+ return;
pm_runtime_get_sync(dev->dev);
nouveau_abi16_fini(cli->abi16);
mutex_unlock(&cli->mutex);
- mutex_lock(&drm->client.mutex);
+ mutex_lock(&drm->clients_lock);
list_del(&cli->head);
- mutex_unlock(&drm->client.mutex);
+ mutex_unlock(&drm->clients_lock);
nouveau_cli_fini(cli);
kfree(cli);
pm_runtime_mark_last_busy(dev->dev);
pm_runtime_put_autosuspend(dev->dev);
+ drm_dev_exit(dev_index);
}
static const struct drm_ioctl_desc
struct list_head clients;
+ /**
+ * @clients_lock: Protects access to the @clients list of &struct nouveau_cli.
+ */
+ struct mutex clients_lock;
+
u8 old_pm_cap;
struct {
nouveau_bo_del_io_reserve_lru(bo);
prot = vm_get_page_prot(vma->vm_flags);
- ret = ttm_bo_vm_fault_reserved(vmf, prot, TTM_BO_VM_NUM_PREFAULT, 1);
+ ret = ttm_bo_vm_fault_reserved(vmf, prot, TTM_BO_VM_NUM_PREFAULT);
nouveau_bo_add_io_reserve_lru(bo);
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
return ret;
struct ttm_buffer_object *bo = &nvbo->bo;
uint32_t domains = valid_domains & nvbo->valid_domains &
(write_domains ? write_domains : read_domains);
- uint32_t pref_domains = 0;;
+ uint32_t pref_domains = 0;
if (!domains)
return -EINVAL;
*/
mm = get_task_mm(current);
+ if (!mm) {
+ return -EINVAL;
+ }
mmap_read_lock(mm);
if (!cli->svm.svmm) {
mmap_read_unlock(mm);
+ mmput(mm);
return -EINVAL;
}
gt215_ce_new(struct nvkm_device *device, enum nvkm_subdev_type type, int inst,
struct nvkm_engine **pengine)
{
- return nvkm_falcon_new_(>215_ce, device, type, inst,
+ return nvkm_falcon_new_(>215_ce, device, type, -1,
(device->chipset != 0xaf), 0x104000, pengine);
}
WARN_ON(device->chip->ptr.inst & ~((1 << ARRAY_SIZE(device->ptr)) - 1)); \
for (j = 0; device->chip->ptr.inst && j < ARRAY_SIZE(device->ptr); j++) { \
if ((device->chip->ptr.inst & BIT(j)) && (subdev_mask & BIT_ULL(type))) { \
- int inst = (device->chip->ptr.inst == 1) ? -1 : (j); \
- ret = device->chip->ptr.ctor(device, (type), inst, &device->ptr[j]); \
+ ret = device->chip->ptr.ctor(device, (type), (j), &device->ptr[j]); \
subdev = nvkm_device_subdev(device, (type), (j)); \
if (ret) { \
nvkm_subdev_del(&subdev); \
nvkm_wr32(device, 0x6f0108 + hdmi, vendor_infoframe.header);
nvkm_wr32(device, 0x6f010c + hdmi, vendor_infoframe.subpack0_low);
nvkm_wr32(device, 0x6f0110 + hdmi, vendor_infoframe.subpack0_high);
- nvkm_wr32(device, 0x6f0110 + hdmi, 0x00000000);
nvkm_wr32(device, 0x6f0114 + hdmi, 0x00000000);
nvkm_wr32(device, 0x6f0118 + hdmi, 0x00000000);
nvkm_wr32(device, 0x6f011c + hdmi, 0x00000000);
*/
#include "priv.h"
-#include "priv.h"
#include <core/firmware.h>
static void *
page = uvmm->vmm->func->page;
for (nr = 0; page[nr].shift; nr++);
- if (!(ret = nvif_unpack(ret, &argv, &argc, args->v0, 0, 0, false))) {
+ if (!(nvif_unpack(ret, &argv, &argc, args->v0, 0, 0, false))) {
if ((index = args->v0.index) >= nr)
return -EINVAL;
type = page[index].type;
struct gp100_vmm_fault_cancel_v0 v0;
} *args = argv;
int ret = -ENOSYS;
- u32 inst, aper;
+ u32 aper;
if ((ret = nvif_unpack(ret, &argv, &argc, args->v0, 0, 0, false)))
return ret;
args->v0.inst |= 0x80000000;
if (!WARN_ON(nvkm_gr_ctxsw_pause(device))) {
- if ((inst = nvkm_gr_ctxsw_inst(device)) == args->v0.inst) {
+ if (nvkm_gr_ctxsw_inst(device) == args->v0.inst) {
gf100_vmm_invalidate(vmm, 0x0000001b
/* CANCEL_TARGETED. */ |
(args->v0.hub << 20) |
Say Y here if you want to enable support for Sharp LS043T1LE01 qHD
(540x960) DSI panel as found on the Qualcomm APQ8074 Dragonboard
+config DRM_PANEL_SHARP_LS060T1SX01
+ tristate "Sharp LS060T1SX01 FullHD video mode panel"
+ depends on OF
+ depends on DRM_MIPI_DSI
+ depends on BACKLIGHT_CLASS_DEVICE
+ help
+ Say Y here if you want to enable support for Sharp LS060T1SX01 6.0"
+ FullHD (1080x1920) DSI panel as found in Dragonboard Display Adapter
+ Bundle.
+
config DRM_PANEL_SITRONIX_ST7701
tristate "Sitronix ST7701 panel driver"
depends on OF
obj-$(CONFIG_DRM_PANEL_SHARP_LQ101R1SX01) += panel-sharp-lq101r1sx01.o
obj-$(CONFIG_DRM_PANEL_SHARP_LS037V7DW01) += panel-sharp-ls037v7dw01.o
obj-$(CONFIG_DRM_PANEL_SHARP_LS043T1LE01) += panel-sharp-ls043t1le01.o
+obj-$(CONFIG_DRM_PANEL_SHARP_LS060T1SX01) += panel-sharp-ls060t1sx01.o
obj-$(CONFIG_DRM_PANEL_SITRONIX_ST7701) += panel-sitronix-st7701.o
obj-$(CONFIG_DRM_PANEL_SITRONIX_ST7703) += panel-sitronix-st7703.o
obj-$(CONFIG_DRM_PANEL_SITRONIX_ST7789V) += panel-sitronix-st7789v.o
#include <linux/backlight.h>
#include <linux/delay.h>
#include <linux/gpio/consumer.h>
+#include <linux/media-bus-format.h>
#include <linux/module.h>
#include <linux/of_device.h>
#include <linux/regulator/consumer.h>
.height_mm = 130,
};
+static const u32 mantix_bus_formats[] = {
+ MEDIA_BUS_FMT_RGB888_1X24,
+};
+
static int mantix_get_modes(struct drm_panel *panel,
struct drm_connector *connector)
{
connector->display_info.height_mm = mode->height_mm;
drm_mode_probed_add(connector, mode);
+ drm_display_info_set_bus_formats(&connector->display_info,
+ mantix_bus_formats,
+ ARRAY_SIZE(mantix_bus_formats));
+
return 1;
}
static int s6e63m0_dsi_remove(struct mipi_dsi_device *dsi)
{
mipi_dsi_detach(dsi);
- return s6e63m0_remove(&dsi->dev);
+ s6e63m0_remove(&dsi->dev);
+ return 0;
}
static const struct of_device_id s6e63m0_dsi_of_match[] = {
static int s6e63m0_spi_remove(struct spi_device *spi)
{
- return s6e63m0_remove(&spi->dev);
+ s6e63m0_remove(&spi->dev);
+ return 0;
}
static const struct of_device_id s6e63m0_spi_of_match[] = {
}
EXPORT_SYMBOL_GPL(s6e63m0_probe);
-int s6e63m0_remove(struct device *dev)
+void s6e63m0_remove(struct device *dev)
{
struct s6e63m0 *ctx = dev_get_drvdata(dev);
drm_panel_remove(&ctx->panel);
-
- return 0;
}
EXPORT_SYMBOL_GPL(s6e63m0_remove);
const u8 *data,
size_t len),
bool dsi_mode);
-int s6e63m0_remove(struct device *dev);
+void s6e63m0_remove(struct device *dev);
#endif /* _PANEL_SAMSUNG_S6E63M0_H */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2021 Linaro Ltd.
+ * Generated with linux-mdss-dsi-panel-driver-generator from vendor device tree:
+ * Copyright (c) 2013-2014, The Linux Foundation. All rights reserved.
+ */
+
+#include <linux/delay.h>
+#include <linux/gpio/consumer.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/regulator/consumer.h>
+
+#include <video/mipi_display.h>
+
+#include <drm/drm_mipi_dsi.h>
+#include <drm/drm_modes.h>
+#include <drm/drm_panel.h>
+
+struct sharp_ls060 {
+ struct drm_panel panel;
+ struct mipi_dsi_device *dsi;
+ struct regulator *vddi_supply;
+ struct regulator *vddh_supply;
+ struct regulator *avdd_supply;
+ struct regulator *avee_supply;
+ struct gpio_desc *reset_gpio;
+ bool prepared;
+};
+
+static inline struct sharp_ls060 *to_sharp_ls060(struct drm_panel *panel)
+{
+ return container_of(panel, struct sharp_ls060, panel);
+}
+
+#define dsi_dcs_write_seq(dsi, seq...) ({ \
+ static const u8 d[] = { seq }; \
+ \
+ mipi_dsi_dcs_write_buffer(dsi, d, ARRAY_SIZE(d)); \
+ })
+
+static void sharp_ls060_reset(struct sharp_ls060 *ctx)
+{
+ gpiod_set_value_cansleep(ctx->reset_gpio, 0);
+ usleep_range(10000, 11000);
+ gpiod_set_value_cansleep(ctx->reset_gpio, 1);
+ usleep_range(10000, 11000);
+ gpiod_set_value_cansleep(ctx->reset_gpio, 0);
+ usleep_range(10000, 11000);
+}
+
+static int sharp_ls060_on(struct sharp_ls060 *ctx)
+{
+ struct mipi_dsi_device *dsi = ctx->dsi;
+ struct device *dev = &dsi->dev;
+ int ret;
+
+ dsi->mode_flags |= MIPI_DSI_MODE_LPM;
+
+ ret = dsi_dcs_write_seq(dsi, 0xbb, 0x13);
+ if (ret < 0) {
+ dev_err(dev, "Failed to send command: %d\n", ret);
+ return ret;
+ }
+
+ ret = dsi_dcs_write_seq(dsi, MIPI_DCS_WRITE_MEMORY_START);
+ if (ret < 0) {
+ dev_err(dev, "Failed to send command: %d\n", ret);
+ return ret;
+ }
+
+ ret = mipi_dsi_dcs_exit_sleep_mode(dsi);
+ if (ret < 0) {
+ dev_err(dev, "Failed to exit sleep mode: %d\n", ret);
+ return ret;
+ }
+ msleep(120);
+
+ ret = mipi_dsi_dcs_set_display_on(dsi);
+ if (ret < 0) {
+ dev_err(dev, "Failed to set display on: %d\n", ret);
+ return ret;
+ }
+ msleep(50);
+
+ return 0;
+}
+
+static int sharp_ls060_off(struct sharp_ls060 *ctx)
+{
+ struct mipi_dsi_device *dsi = ctx->dsi;
+ struct device *dev = &dsi->dev;
+ int ret;
+
+ dsi->mode_flags &= ~MIPI_DSI_MODE_LPM;
+
+ ret = mipi_dsi_dcs_set_display_off(dsi);
+ if (ret < 0) {
+ dev_err(dev, "Failed to set display off: %d\n", ret);
+ return ret;
+ }
+ usleep_range(2000, 3000);
+
+ ret = mipi_dsi_dcs_enter_sleep_mode(dsi);
+ if (ret < 0) {
+ dev_err(dev, "Failed to enter sleep mode: %d\n", ret);
+ return ret;
+ }
+ msleep(121);
+
+ return 0;
+}
+
+static int sharp_ls060_prepare(struct drm_panel *panel)
+{
+ struct sharp_ls060 *ctx = to_sharp_ls060(panel);
+ struct device *dev = &ctx->dsi->dev;
+ int ret;
+
+ if (ctx->prepared)
+ return 0;
+
+ ret = regulator_enable(ctx->vddi_supply);
+ if (ret < 0)
+ return ret;
+
+ ret = regulator_enable(ctx->avdd_supply);
+ if (ret < 0)
+ goto err_avdd;
+
+ usleep_range(1000, 2000);
+
+ ret = regulator_enable(ctx->avee_supply);
+ if (ret < 0)
+ goto err_avee;
+
+ usleep_range(10000, 11000);
+
+ ret = regulator_enable(ctx->vddh_supply);
+ if (ret < 0)
+ goto err_vddh;
+
+ usleep_range(10000, 11000);
+
+ sharp_ls060_reset(ctx);
+
+ ret = sharp_ls060_on(ctx);
+ if (ret < 0) {
+ dev_err(dev, "Failed to initialize panel: %d\n", ret);
+ goto err_on;
+ }
+
+ ctx->prepared = true;
+
+ return 0;
+
+err_on:
+ regulator_disable(ctx->vddh_supply);
+
+ usleep_range(10000, 11000);
+
+err_vddh:
+ regulator_disable(ctx->avee_supply);
+
+err_avee:
+ regulator_disable(ctx->avdd_supply);
+
+ gpiod_set_value_cansleep(ctx->reset_gpio, 1);
+
+err_avdd:
+ regulator_disable(ctx->vddi_supply);
+
+ return ret;
+}
+
+static int sharp_ls060_unprepare(struct drm_panel *panel)
+{
+ struct sharp_ls060 *ctx = to_sharp_ls060(panel);
+ struct device *dev = &ctx->dsi->dev;
+ int ret;
+
+ if (!ctx->prepared)
+ return 0;
+
+ ret = sharp_ls060_off(ctx);
+ if (ret < 0)
+ dev_err(dev, "Failed to un-initialize panel: %d\n", ret);
+
+ regulator_disable(ctx->vddh_supply);
+
+ usleep_range(10000, 11000);
+
+ regulator_disable(ctx->avee_supply);
+ regulator_disable(ctx->avdd_supply);
+
+ gpiod_set_value_cansleep(ctx->reset_gpio, 1);
+
+ regulator_disable(ctx->vddi_supply);
+
+ ctx->prepared = false;
+ return 0;
+}
+
+static const struct drm_display_mode sharp_ls060_mode = {
+ .clock = (1080 + 96 + 16 + 64) * (1920 + 4 + 1 + 16) * 60 / 1000,
+ .hdisplay = 1080,
+ .hsync_start = 1080 + 96,
+ .hsync_end = 1080 + 96 + 16,
+ .htotal = 1080 + 96 + 16 + 64,
+ .vdisplay = 1920,
+ .vsync_start = 1920 + 4,
+ .vsync_end = 1920 + 4 + 1,
+ .vtotal = 1920 + 4 + 1 + 16,
+ .width_mm = 75,
+ .height_mm = 132,
+};
+
+static int sharp_ls060_get_modes(struct drm_panel *panel,
+ struct drm_connector *connector)
+{
+ struct drm_display_mode *mode;
+
+ mode = drm_mode_duplicate(connector->dev, &sharp_ls060_mode);
+ if (!mode)
+ return -ENOMEM;
+
+ drm_mode_set_name(mode);
+
+ mode->type = DRM_MODE_TYPE_DRIVER | DRM_MODE_TYPE_PREFERRED;
+ connector->display_info.width_mm = mode->width_mm;
+ connector->display_info.height_mm = mode->height_mm;
+ drm_mode_probed_add(connector, mode);
+
+ return 1;
+}
+
+static const struct drm_panel_funcs sharp_ls060_panel_funcs = {
+ .prepare = sharp_ls060_prepare,
+ .unprepare = sharp_ls060_unprepare,
+ .get_modes = sharp_ls060_get_modes,
+};
+
+static int sharp_ls060_probe(struct mipi_dsi_device *dsi)
+{
+ struct device *dev = &dsi->dev;
+ struct sharp_ls060 *ctx;
+ int ret;
+
+ ctx = devm_kzalloc(dev, sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return -ENOMEM;
+
+ ctx->vddi_supply = devm_regulator_get(dev, "vddi");
+ if (IS_ERR(ctx->vddi_supply))
+ return PTR_ERR(ctx->vddi_supply);
+
+ ctx->vddh_supply = devm_regulator_get(dev, "vddh");
+ if (IS_ERR(ctx->vddh_supply))
+ return PTR_ERR(ctx->vddh_supply);
+
+ ctx->avdd_supply = devm_regulator_get(dev, "avdd");
+ if (IS_ERR(ctx->avdd_supply))
+ return PTR_ERR(ctx->avdd_supply);
+
+ ctx->avee_supply = devm_regulator_get(dev, "avee");
+ if (IS_ERR(ctx->avee_supply))
+ return PTR_ERR(ctx->avee_supply);
+
+ ctx->reset_gpio = devm_gpiod_get(dev, "reset", GPIOD_OUT_HIGH);
+ if (IS_ERR(ctx->reset_gpio))
+ return dev_err_probe(dev, PTR_ERR(ctx->reset_gpio),
+ "Failed to get reset-gpios\n");
+
+ ctx->dsi = dsi;
+ mipi_dsi_set_drvdata(dsi, ctx);
+
+ dsi->lanes = 4;
+ dsi->format = MIPI_DSI_FMT_RGB888;
+ dsi->mode_flags = MIPI_DSI_MODE_VIDEO | MIPI_DSI_MODE_VIDEO_BURST |
+ MIPI_DSI_MODE_NO_EOT_PACKET |
+ MIPI_DSI_CLOCK_NON_CONTINUOUS;
+
+ drm_panel_init(&ctx->panel, dev, &sharp_ls060_panel_funcs,
+ DRM_MODE_CONNECTOR_DSI);
+
+ ret = drm_panel_of_backlight(&ctx->panel);
+ if (ret)
+ return dev_err_probe(dev, ret, "Failed to get backlight\n");
+
+ drm_panel_add(&ctx->panel);
+
+ ret = mipi_dsi_attach(dsi);
+ if (ret < 0) {
+ dev_err(dev, "Failed to attach to DSI host: %d\n", ret);
+ drm_panel_remove(&ctx->panel);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int sharp_ls060_remove(struct mipi_dsi_device *dsi)
+{
+ struct sharp_ls060 *ctx = mipi_dsi_get_drvdata(dsi);
+ int ret;
+
+ ret = mipi_dsi_detach(dsi);
+ if (ret < 0)
+ dev_err(&dsi->dev, "Failed to detach from DSI host: %d\n", ret);
+
+ drm_panel_remove(&ctx->panel);
+
+ return 0;
+}
+
+static const struct of_device_id sharp_ls060t1sx01_of_match[] = {
+ { .compatible = "sharp,ls060t1sx01" },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, sharp_ls060t1sx01_of_match);
+
+static struct mipi_dsi_driver sharp_ls060_driver = {
+ .probe = sharp_ls060_probe,
+ .remove = sharp_ls060_remove,
+ .driver = {
+ .name = "panel-sharp-ls060t1sx01",
+ .of_match_table = sharp_ls060t1sx01_of_match,
+ },
+};
+module_mipi_dsi_driver(sharp_ls060_driver);
+
+MODULE_AUTHOR("Dmitry Baryshkov <dmitry.baryshkov@linaro.org>");
+MODULE_DESCRIPTION("DRM driver for Sharp LS060T1SX01 1080p video mode dsi panel");
+MODULE_LICENSE("GPL v2");
.connector_type = DRM_MODE_CONNECTOR_LVDS,
};
+static const struct drm_display_mode logictechno_lttd800480070_l2rt_mode = {
+ .clock = 33000,
+ .hdisplay = 800,
+ .hsync_start = 800 + 112,
+ .hsync_end = 800 + 112 + 3,
+ .htotal = 800 + 112 + 3 + 85,
+ .vdisplay = 480,
+ .vsync_start = 480 + 38,
+ .vsync_end = 480 + 38 + 3,
+ .vtotal = 480 + 38 + 3 + 29,
+ .flags = DRM_MODE_FLAG_NVSYNC | DRM_MODE_FLAG_NHSYNC,
+};
+
+static const struct panel_desc logictechno_lttd800480070_l2rt = {
+ .modes = &logictechno_lttd800480070_l2rt_mode,
+ .num_modes = 1,
+ .bpc = 8,
+ .size = {
+ .width = 154,
+ .height = 86,
+ },
+ .delay = {
+ .prepare = 45,
+ .enable = 100,
+ .disable = 100,
+ .unprepare = 45
+ },
+ .bus_format = MEDIA_BUS_FMT_RGB888_1X24,
+ .bus_flags = DRM_BUS_FLAG_PIXDATA_SAMPLE_NEGEDGE,
+ .connector_type = DRM_MODE_CONNECTOR_DPI,
+};
+
static const struct drm_display_mode logictechno_lttd800480070_l6wh_rt_mode = {
.clock = 33000,
.hdisplay = 800,
.compatible = "logictechno,lt170410-2whc",
.data = &logictechno_lt170410_2whc,
}, {
+ .compatible = "logictechno,lttd800480070-l2rt",
+ .data = &logictechno_lttd800480070_l2rt,
+ }, {
.compatible = "logictechno,lttd800480070-l6wh-rt",
.data = &logictechno_lttd800480070_l6wh_rt,
}, {
return ret;
}
+static const u32 mantix_bus_formats[] = {
+ MEDIA_BUS_FMT_RGB888_1X24,
+};
+
static int st7703_get_modes(struct drm_panel *panel,
struct drm_connector *connector)
{
connector->display_info.height_mm = mode->height_mm;
drm_mode_probed_add(connector, mode);
+ drm_display_info_set_bus_formats(&connector->display_info,
+ mantix_bus_formats,
+ ARRAY_SIZE(mantix_bus_formats));
+
return 1;
}
goto unlock_resv;
ret = ttm_bo_vm_fault_reserved(vmf, vmf->vma->vm_page_prot,
- TTM_BO_VM_NUM_PREFAULT, 1);
+ TTM_BO_VM_NUM_PREFAULT);
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
goto unlock_mclk;
struct drm_gem_object *obj,
bool write)
{
+ struct dma_resv_iter cursor;
+ struct dma_fence *fence;
int ret;
- struct dma_fence **fences;
- unsigned int i, fence_count;
-
- if (!write) {
- struct dma_fence *fence = dma_resv_get_excl_unlocked(obj->resv);
-
- return drm_sched_job_add_dependency(job, fence);
- }
-
- ret = dma_resv_get_fences(obj->resv, NULL, &fence_count, &fences);
- if (ret || !fence_count)
- return ret;
- for (i = 0; i < fence_count; i++) {
- ret = drm_sched_job_add_dependency(job, fences[i]);
- if (ret)
- break;
+ dma_resv_for_each_fence(&cursor, obj->resv, write, fence) {
+ /* Make sure to grab an additional ref on the added fence */
+ dma_fence_get(fence);
+ ret = drm_sched_job_add_dependency(job, fence);
+ if (ret) {
+ dma_fence_put(fence);
+ return ret;
+ }
}
-
- for (; i < fence_count; i++)
- dma_fence_put(fences[i]);
- kfree(fences);
- return ret;
+ return 0;
}
EXPORT_SYMBOL(drm_sched_job_add_implicit_dependencies);
default MACH_SUN8I
select CRC_CCITT
select DRM_MIPI_DSI
+ select RESET_CONTROLLER
select PHY_SUN6I_MIPI_DPHY
help
Choose this option if you want have an Allwinner SoC with
static void ttm_bo_flush_all_fences(struct ttm_buffer_object *bo)
{
struct dma_resv *resv = &bo->base._resv;
- struct dma_resv_list *fobj;
+ struct dma_resv_iter cursor;
struct dma_fence *fence;
- int i;
-
- rcu_read_lock();
- fobj = dma_resv_shared_list(resv);
- fence = dma_resv_excl_fence(resv);
- if (fence && !fence->ops->signaled)
- dma_fence_enable_sw_signaling(fence);
-
- for (i = 0; fobj && i < fobj->shared_count; ++i) {
- fence = rcu_dereference(fobj->shared[i]);
+ dma_resv_iter_begin(&cursor, resv, true);
+ dma_resv_for_each_fence_unlocked(&cursor, fence) {
if (!fence->ops->signaled)
dma_fence_enable_sw_signaling(fence);
}
- rcu_read_unlock();
+ dma_resv_iter_end(&cursor);
}
/**
*busy = !ret;
}
- if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
+ if (ret && place && (bo->resource->mem_type != place->mem_type ||
+ !bo->bdev->funcs->eviction_valuable(bo, place))) {
ret = false;
if (*locked) {
dma_resv_unlock(bo->base.resv);
}
EXPORT_SYMBOL(ttm_bo_vm_reserve);
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-/**
- * ttm_bo_vm_insert_huge - Insert a pfn for PUD or PMD faults
- * @vmf: Fault data
- * @bo: The buffer object
- * @page_offset: Page offset from bo start
- * @fault_page_size: The size of the fault in pages.
- * @pgprot: The page protections.
- * Does additional checking whether it's possible to insert a PUD or PMD
- * pfn and performs the insertion.
- *
- * Return: VM_FAULT_NOPAGE on successful insertion, VM_FAULT_FALLBACK if
- * a huge fault was not possible, or on insertion error.
- */
-static vm_fault_t ttm_bo_vm_insert_huge(struct vm_fault *vmf,
- struct ttm_buffer_object *bo,
- pgoff_t page_offset,
- pgoff_t fault_page_size,
- pgprot_t pgprot)
-{
- pgoff_t i;
- vm_fault_t ret;
- unsigned long pfn;
- pfn_t pfnt;
- struct ttm_tt *ttm = bo->ttm;
- bool write = vmf->flags & FAULT_FLAG_WRITE;
-
- /* Fault should not cross bo boundary. */
- page_offset &= ~(fault_page_size - 1);
- if (page_offset + fault_page_size > bo->resource->num_pages)
- goto out_fallback;
-
- if (bo->resource->bus.is_iomem)
- pfn = ttm_bo_io_mem_pfn(bo, page_offset);
- else
- pfn = page_to_pfn(ttm->pages[page_offset]);
-
- /* pfn must be fault_page_size aligned. */
- if ((pfn & (fault_page_size - 1)) != 0)
- goto out_fallback;
-
- /* Check that memory is contiguous. */
- if (!bo->resource->bus.is_iomem) {
- for (i = 1; i < fault_page_size; ++i) {
- if (page_to_pfn(ttm->pages[page_offset + i]) != pfn + i)
- goto out_fallback;
- }
- } else if (bo->bdev->funcs->io_mem_pfn) {
- for (i = 1; i < fault_page_size; ++i) {
- if (ttm_bo_io_mem_pfn(bo, page_offset + i) != pfn + i)
- goto out_fallback;
- }
- }
-
- pfnt = __pfn_to_pfn_t(pfn, PFN_DEV);
- if (fault_page_size == (HPAGE_PMD_SIZE >> PAGE_SHIFT))
- ret = vmf_insert_pfn_pmd_prot(vmf, pfnt, pgprot, write);
-#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
- else if (fault_page_size == (HPAGE_PUD_SIZE >> PAGE_SHIFT))
- ret = vmf_insert_pfn_pud_prot(vmf, pfnt, pgprot, write);
-#endif
- else
- WARN_ON_ONCE(ret = VM_FAULT_FALLBACK);
-
- if (ret != VM_FAULT_NOPAGE)
- goto out_fallback;
-
- return VM_FAULT_NOPAGE;
-out_fallback:
- count_vm_event(THP_FAULT_FALLBACK);
- return VM_FAULT_FALLBACK;
-}
-#else
-static vm_fault_t ttm_bo_vm_insert_huge(struct vm_fault *vmf,
- struct ttm_buffer_object *bo,
- pgoff_t page_offset,
- pgoff_t fault_page_size,
- pgprot_t pgprot)
-{
- return VM_FAULT_FALLBACK;
-}
-#endif
-
/**
* ttm_bo_vm_fault_reserved - TTM fault helper
* @vmf: The struct vm_fault given as argument to the fault callback
* @num_prefault: Maximum number of prefault pages. The caller may want to
* specify this based on madvice settings and the size of the GPU object
* backed by the memory.
- * @fault_page_size: The size of the fault in pages.
*
* This function inserts one or more page table entries pointing to the
* memory backing the buffer object, and then returns a return code
*/
vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
pgprot_t prot,
- pgoff_t num_prefault,
- pgoff_t fault_page_size)
+ pgoff_t num_prefault)
{
struct vm_area_struct *vma = vmf->vma;
struct ttm_buffer_object *bo = vma->vm_private_data;
prot = pgprot_decrypted(prot);
}
- /* We don't prefault on huge faults. Yet. */
- if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && fault_page_size != 1)
- return ttm_bo_vm_insert_huge(vmf, bo, page_offset,
- fault_page_size, prot);
-
/*
* Speculatively prefault a number of pages. Only error on
* first page.
prot = vma->vm_page_prot;
if (drm_dev_enter(ddev, &idx)) {
- ret = ttm_bo_vm_fault_reserved(vmf, prot, TTM_BO_VM_NUM_PREFAULT, 1);
+ ret = ttm_bo_vm_fault_reserved(vmf, prot, TTM_BO_VM_NUM_PREFAULT);
drm_dev_exit(idx);
} else {
ret = ttm_bo_vm_dummy_page(vmf, prot);
int bval = (i + block * EDID_LENGTH) << 8;
ret = usb_control_msg(udev, usb_rcvctrlpipe(udev, 0),
0x02, (0x80 | (0x02 << 5)), bval,
- 0xA1, read_buff, 2, HZ);
+ 0xA1, read_buff, 2, 1000);
if (ret < 1) {
DRM_ERROR("Read EDID byte %d failed err %x\n", i, ret);
kfree(read_buff);
for (i = 0; i < se->in_sync_count; i++) {
struct drm_v3d_sem in;
- ret = copy_from_user(&in, handle++, sizeof(in));
- if (ret) {
+ if (copy_from_user(&in, handle++, sizeof(in))) {
+ ret = -EFAULT;
DRM_DEBUG("Failed to copy wait dep handle.\n");
goto fail_deps;
}
for (i = 0; i < count; i++) {
struct drm_v3d_sem out;
- ret = copy_from_user(&out, post_deps++, sizeof(out));
- if (ret) {
+ if (copy_from_user(&out, post_deps++, sizeof(out))) {
+ ret = -EFAULT;
DRM_DEBUG("Failed to copy post dep handles\n");
goto fail;
}
struct v3d_submit_ext *se = data;
int ret;
- ret = copy_from_user(&multisync, ext, sizeof(multisync));
- if (ret)
- return ret;
+ if (copy_from_user(&multisync, ext, sizeof(multisync)))
+ return -EFAULT;
if (multisync.pad)
return -EINVAL;
return ERR_PTR(-EINVAL);
virtio_gpu_fb = kzalloc(sizeof(*virtio_gpu_fb), GFP_KERNEL);
- if (virtio_gpu_fb == NULL)
+ if (virtio_gpu_fb == NULL) {
+ drm_gem_object_put(obj);
return ERR_PTR(-ENOMEM);
+ }
ret = virtio_gpu_framebuffer_init(dev, virtio_gpu_fb, mode_cmd, obj);
if (ret) {
struct drm_file *drm_file = filp->private_data;
struct virtio_gpu_fpriv *vfpriv = drm_file->driver_priv;
struct drm_device *dev = drm_file->minor->dev;
+ struct virtio_gpu_device *vgdev = dev->dev_private;
struct drm_pending_event *e = NULL;
__poll_t mask = 0;
- if (!vfpriv->ring_idx_mask)
+ if (!vgdev->has_virgl_3d || !vfpriv || !vfpriv->ring_idx_mask)
return drm_poll(filp, wait);
poll_wait(filp, &drm_file->event_wait, wait);
pgoff_t start, pgoff_t end);
vm_fault_t vmw_bo_vm_fault(struct vm_fault *vmf);
vm_fault_t vmw_bo_vm_mkwrite(struct vm_fault *vmf);
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-vm_fault_t vmw_bo_vm_huge_fault(struct vm_fault *vmf,
- enum page_entry_size pe_size);
-#endif
/* Transparent hugepage support - vmwgfx_thp.c */
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
else
prot = vm_get_page_prot(vma->vm_flags);
- ret = ttm_bo_vm_fault_reserved(vmf, prot, num_prefault, 1);
+ ret = ttm_bo_vm_fault_reserved(vmf, prot, num_prefault);
if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
return ret;
return ret;
}
-
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-vm_fault_t vmw_bo_vm_huge_fault(struct vm_fault *vmf,
- enum page_entry_size pe_size)
-{
- struct vm_area_struct *vma = vmf->vma;
- struct ttm_buffer_object *bo = (struct ttm_buffer_object *)
- vma->vm_private_data;
- struct vmw_buffer_object *vbo =
- container_of(bo, struct vmw_buffer_object, base);
- pgprot_t prot;
- vm_fault_t ret;
- pgoff_t fault_page_size;
- bool write = vmf->flags & FAULT_FLAG_WRITE;
-
- switch (pe_size) {
- case PE_SIZE_PMD:
- fault_page_size = HPAGE_PMD_SIZE >> PAGE_SHIFT;
- break;
-#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
- case PE_SIZE_PUD:
- fault_page_size = HPAGE_PUD_SIZE >> PAGE_SHIFT;
- break;
-#endif
- default:
- WARN_ON_ONCE(1);
- return VM_FAULT_FALLBACK;
- }
-
- /* Always do write dirty-tracking and COW on PTE level. */
- if (write && (READ_ONCE(vbo->dirty) || is_cow_mapping(vma->vm_flags)))
- return VM_FAULT_FALLBACK;
-
- ret = ttm_bo_vm_reserve(bo, vmf);
- if (ret)
- return ret;
-
- if (vbo->dirty) {
- pgoff_t allowed_prefault;
- unsigned long page_offset;
-
- page_offset = vmf->pgoff -
- drm_vma_node_start(&bo->base.vma_node);
- if (page_offset >= bo->resource->num_pages ||
- vmw_resources_clean(vbo, page_offset,
- page_offset + PAGE_SIZE,
- &allowed_prefault)) {
- ret = VM_FAULT_SIGBUS;
- goto out_unlock;
- }
-
- /*
- * Write protect, so we get a new fault on write, and can
- * split.
- */
- prot = vm_get_page_prot(vma->vm_flags & ~VM_SHARED);
- } else {
- prot = vm_get_page_prot(vma->vm_flags);
- }
-
- ret = ttm_bo_vm_fault_reserved(vmf, prot, 1, fault_page_size);
- if (ret == VM_FAULT_RETRY && !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT))
- return ret;
-
-out_unlock:
- dma_resv_unlock(bo->base.resv);
-
- return ret;
-}
-#endif
.fault = vmw_bo_vm_fault,
.open = ttm_bo_vm_open,
.close = ttm_bo_vm_close,
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- .huge_fault = vmw_bo_vm_huge_fault,
-#endif
};
struct drm_file *file_priv = filp->private_data;
struct vmw_private *dev_priv = vmw_priv(file_priv->minor->dev);
config HID_CHICONY
tristate "Chicony devices"
- depends on HID
+ depends on USB_HID
default !EXPERT
help
Support for Chicony Tactical pad and special keys on Chicony keyboards.
config HID_CORSAIR
tristate "Corsair devices"
- depends on HID && USB && LEDS_CLASS
+ depends on USB_HID && LEDS_CLASS
help
Support for Corsair devices that are not fully compliant with the
HID standard.
config HID_PRODIKEYS
tristate "Prodikeys PC-MIDI Keyboard support"
- depends on HID && SND
+ depends on USB_HID && SND
select SND_RAWMIDI
help
Support for Prodikeys PC-MIDI Keyboard device support.
config HID_LOGITECH
tristate "Logitech devices"
- depends on HID
+ depends on USB_HID
depends on LEDS_CLASS
default !EXPERT
help
config HID_SAMSUNG
tristate "Samsung InfraRed remote control or keyboards"
- depends on HID
+ depends on USB_HID
help
Support for Samsung InfraRed remote control or keyboards.
switch (usage->hid & HID_USAGE) {
case 0x10: asus_map_key_clear(KEY_BRIGHTNESSDOWN); break;
case 0x20: asus_map_key_clear(KEY_BRIGHTNESSUP); break;
- case 0x35: asus_map_key_clear(KEY_SCREENLOCK); break;
+ case 0x35: asus_map_key_clear(KEY_DISPLAY_OFF); break;
case 0x6c: asus_map_key_clear(KEY_SLEEP); break;
case 0x7c: asus_map_key_clear(KEY_MICMUTE); break;
case 0x82: asus_map_key_clear(KEY_CAMERA); break;
if (drvdata->quirks & QUIRK_IS_MULTITOUCH)
drvdata->tp = &asus_i2c_tp;
- if ((drvdata->quirks & QUIRK_T100_KEYBOARD) &&
- hid_is_using_ll_driver(hdev, &usb_hid_driver)) {
+ if ((drvdata->quirks & QUIRK_T100_KEYBOARD) && hid_is_usb(hdev)) {
struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
if (intf->altsetting->desc.bInterfaceNumber == T100_TPAD_INTF) {
drvdata->tp = &asus_t100chi_tp;
}
- if ((drvdata->quirks & QUIRK_MEDION_E1239T) &&
- hid_is_using_ll_driver(hdev, &usb_hid_driver)) {
+ if ((drvdata->quirks & QUIRK_MEDION_E1239T) && hid_is_usb(hdev)) {
struct usb_host_interface *alt =
to_usb_interface(hdev->dev.parent)->altsetting;
struct bigben_device, worker);
struct hid_field *report_field = bigben->report->field[0];
- if (bigben->removed)
+ if (bigben->removed || !report_field)
return;
if (bigben->work_led) {
{
int ret;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
hdev->quirks |= HID_QUIRK_INPUT_PER_APP;
ret = hid_parse(hdev);
if (ret) {
}
EXPORT_SYMBOL_GPL(hid_hw_close);
+/**
+ * hid_hw_request - send report request to device
+ *
+ * @hdev: hid device
+ * @report: report to send
+ * @reqtype: hid request type
+ */
+void hid_hw_request(struct hid_device *hdev,
+ struct hid_report *report, int reqtype)
+{
+ if (hdev->ll_driver->request)
+ return hdev->ll_driver->request(hdev, report, reqtype);
+
+ __hid_request(hdev, report, reqtype);
+}
+EXPORT_SYMBOL_GPL(hid_hw_request);
+
+/**
+ * hid_hw_raw_request - send report request to device
+ *
+ * @hdev: hid device
+ * @reportnum: report ID
+ * @buf: in/out data to transfer
+ * @len: length of buf
+ * @rtype: HID report type
+ * @reqtype: HID_REQ_GET_REPORT or HID_REQ_SET_REPORT
+ *
+ * Return: count of data transferred, negative if error
+ *
+ * Same behavior as hid_hw_request, but with raw buffers instead.
+ */
+int hid_hw_raw_request(struct hid_device *hdev,
+ unsigned char reportnum, __u8 *buf,
+ size_t len, unsigned char rtype, int reqtype)
+{
+ if (len < 1 || len > HID_MAX_BUFFER_SIZE || !buf)
+ return -EINVAL;
+
+ return hdev->ll_driver->raw_request(hdev, reportnum, buf, len,
+ rtype, reqtype);
+}
+EXPORT_SYMBOL_GPL(hid_hw_raw_request);
+
+/**
+ * hid_hw_output_report - send output report to device
+ *
+ * @hdev: hid device
+ * @buf: raw data to transfer
+ * @len: length of buf
+ *
+ * Return: count of data transferred, negative if error
+ */
+int hid_hw_output_report(struct hid_device *hdev, __u8 *buf, size_t len)
+{
+ if (len < 1 || len > HID_MAX_BUFFER_SIZE || !buf)
+ return -EINVAL;
+
+ if (hdev->ll_driver->output_report)
+ return hdev->ll_driver->output_report(hdev, buf, len);
+
+ return -ENOSYS;
+}
+EXPORT_SYMBOL_GPL(hid_hw_output_report);
+
+#ifdef CONFIG_PM
+int hid_driver_suspend(struct hid_device *hdev, pm_message_t state)
+{
+ if (hdev->driver && hdev->driver->suspend)
+ return hdev->driver->suspend(hdev, state);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(hid_driver_suspend);
+
+int hid_driver_reset_resume(struct hid_device *hdev)
+{
+ if (hdev->driver && hdev->driver->reset_resume)
+ return hdev->driver->reset_resume(hdev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(hid_driver_reset_resume);
+
+int hid_driver_resume(struct hid_device *hdev)
+{
+ if (hdev->driver && hdev->driver->resume)
+ return hdev->driver->resume(hdev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(hid_driver_resume);
+#endif /* CONFIG_PM */
+
struct hid_dynid {
struct list_head list;
struct hid_device_id id;
int ret;
unsigned long quirks = id->driver_data;
struct corsair_drvdata *drvdata;
- struct usb_interface *usbif = to_usb_interface(dev->dev.parent);
+ struct usb_interface *usbif;
+
+ if (!hid_is_usb(dev))
+ return -EINVAL;
+
+ usbif = to_usb_interface(dev->dev.parent);
drvdata = devm_kzalloc(&dev->dev, sizeof(struct corsair_drvdata),
GFP_KERNEL);
{0, 0x33, "Touch"},
{0, 0x34, "UnTouch"},
{0, 0x35, "Tap"},
+ {0, 0x38, "Transducer Index"},
{0, 0x39, "TabletFunctionKey"},
{0, 0x3a, "ProgramChangeKey"},
+ {0, 0x3B, "Battery Strength"},
{0, 0x3c, "Invert"},
{0, 0x42, "TipSwitch"},
{0, 0x43, "SecondaryTipSwitch"},
{0, 0x59, "ButtonType"},
{0, 0x5A, "SecondaryBarrelSwitch"},
{0, 0x5B, "TransducerSerialNumber"},
+ {0, 0x5C, "Preferred Color"},
+ {0, 0x5D, "Preferred Color is Locked"},
+ {0, 0x5E, "Preferred Line Width"},
+ {0, 0x5F, "Preferred Line Width is Locked"},
{0, 0x6e, "TransducerSerialNumber2"},
+ {0, 0x70, "Preferred Line Style"},
+ {0, 0x71, "Preferred Line Style is Locked"},
+ {0, 0x72, "Ink"},
+ {0, 0x73, "Pencil"},
+ {0, 0x74, "Highlighter"},
+ {0, 0x75, "Chisel Marker"},
+ {0, 0x76, "Brush"},
+ {0, 0x77, "No Preference"},
+ {0, 0x80, "Digitizer Diagnostic"},
+ {0, 0x81, "Digitizer Error"},
+ {0, 0x82, "Err Normal Status"},
+ {0, 0x83, "Err Transducers Exceeded"},
+ {0, 0x84, "Err Full Trans Features Unavailable"},
+ {0, 0x85, "Err Charge Low"},
+ {0, 0x90, "Transducer Software Info"},
+ {0, 0x91, "Transducer Vendor Id"},
+ {0, 0x92, "Transducer Product Id"},
+ {0, 0x93, "Device Supported Protocols"},
+ {0, 0x94, "Transducer Supported Protocols"},
+ {0, 0x95, "No Protocol"},
+ {0, 0x96, "Wacom AES Protocol"},
+ {0, 0x97, "USI Protocol"},
+ {0, 0x98, "Microsoft Pen Protocol"},
+ {0, 0xA0, "Supported Report Rates"},
+ {0, 0xA1, "Report Rate"},
+ {0, 0xA2, "Transducer Connected"},
+ {0, 0xA3, "Switch Disabled"},
+ {0, 0xA4, "Switch Unimplemented"},
+ {0, 0xA5, "Transducer Switches"},
{ 15, 0, "PhysicalInterfaceDevice" },
{0, 0x00, "Undefined"},
{0, 0x01, "Physical_Interface_Device"},
static int is_not_elan_touchpad(struct hid_device *hdev)
{
- if (hdev->bus == BUS_USB) {
+ if (hid_is_usb(hdev)) {
struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
return (intf->altsetting->desc.bInterfaceNumber !=
int ret;
struct usb_device *udev;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
priv = kzalloc(sizeof(*priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
struct ft260_get_chip_version_report version;
int ret;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
dev = devm_kzalloc(&hdev->dev, sizeof(*dev), GFP_KERNEL);
if (!dev)
return -ENOMEM;
mutex_init(&dev->lock);
init_completion(&dev->wait);
+ ret = ft260_xfer_status(dev);
+ if (ret)
+ ft260_i2c_reset(hdev);
+
+ i2c_set_adapdata(&dev->adap, dev);
ret = i2c_add_adapter(&dev->adap);
if (ret) {
hid_err(hdev, "failed to add i2c adapter\n");
goto err_hid_close;
}
- i2c_set_adapdata(&dev->adap, dev);
-
ret = sysfs_create_group(&hdev->dev.kobj, &ft260_attr_group);
if (ret < 0) {
hid_err(hdev, "failed to create sysfs attrs\n");
goto err_i2c_free;
}
- ret = ft260_xfer_status(dev);
- if (ret)
- ft260_i2c_reset(hdev);
-
return 0;
err_i2c_free:
{ HID_DEVICE(BUS_USB, HID_GROUP_GENERIC,
USB_VENDOR_ID_GOOGLE, USB_DEVICE_ID_GOOGLE_DON) },
{ HID_DEVICE(BUS_USB, HID_GROUP_GENERIC,
+ USB_VENDOR_ID_GOOGLE, USB_DEVICE_ID_GOOGLE_EEL) },
+ { HID_DEVICE(BUS_USB, HID_GROUP_GENERIC,
USB_VENDOR_ID_GOOGLE, USB_DEVICE_ID_GOOGLE_HAMMER) },
{ HID_DEVICE(BUS_USB, HID_GROUP_GENERIC,
USB_VENDOR_ID_GOOGLE, USB_DEVICE_ID_GOOGLE_MAGNEMITE) },
static int holtek_kbd_probe(struct hid_device *hdev,
const struct hid_device_id *id)
{
- struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
- int ret = hid_parse(hdev);
+ struct usb_interface *intf;
+ int ret;
+
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+ ret = hid_parse(hdev);
if (!ret)
ret = hid_hw_start(hdev, HID_CONNECT_DEFAULT);
+ intf = to_usb_interface(hdev->dev.parent);
if (!ret && intf->cur_altsetting->desc.bInterfaceNumber == 1) {
struct hid_input *hidinput;
list_for_each_entry(hidinput, &hdev->inputs, list) {
return rdesc;
}
+static int holtek_mouse_probe(struct hid_device *hdev,
+ const struct hid_device_id *id)
+{
+ int ret;
+
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
+ ret = hid_parse(hdev);
+ if (ret) {
+ hid_err(hdev, "hid parse failed: %d\n", ret);
+ return ret;
+ }
+
+ ret = hid_hw_start(hdev, HID_CONNECT_DEFAULT);
+ if (ret) {
+ hid_err(hdev, "hw start failed: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
static const struct hid_device_id holtek_mouse_devices[] = {
{ HID_USB_DEVICE(USB_VENDOR_ID_HOLTEK_ALT,
USB_DEVICE_ID_HOLTEK_ALT_MOUSE_A067) },
.name = "holtek_mouse",
.id_table = holtek_mouse_devices,
.report_fixup = holtek_mouse_report_fixup,
+ .probe = holtek_mouse_probe,
};
module_hid_driver(holtek_mouse_driver);
#define USB_DEVICE_ID_TOSHIBA_CLICK_L9W 0x0401
#define USB_DEVICE_ID_HP_X2 0x074d
#define USB_DEVICE_ID_HP_X2_10_COVER 0x0755
+#define I2C_DEVICE_ID_HP_ENVY_X360_15 0x2d05
#define I2C_DEVICE_ID_HP_SPECTRE_X360_15 0x2817
+#define USB_DEVICE_ID_ASUS_UX550VE_TOUCHSCREEN 0x2544
#define USB_DEVICE_ID_ASUS_UX550_TOUCHSCREEN 0x2706
#define I2C_DEVICE_ID_SURFACE_GO_TOUCHSCREEN 0x261A
#define USB_DEVICE_ID_GOOGLE_MAGNEMITE 0x503d
#define USB_DEVICE_ID_GOOGLE_MOONBALL 0x5044
#define USB_DEVICE_ID_GOOGLE_DON 0x5050
+#define USB_DEVICE_ID_GOOGLE_EEL 0x5057
#define USB_VENDOR_ID_GOTOP 0x08f2
#define USB_DEVICE_ID_SUPER_Q2 0x007f
#define USB_DEVICE_ID_MS_TOUCH_COVER_2 0x07a7
#define USB_DEVICE_ID_MS_TYPE_COVER_2 0x07a9
#define USB_DEVICE_ID_MS_POWER_COVER 0x07da
+#define USB_DEVICE_ID_MS_SURFACE3_COVER 0x07de
#define USB_DEVICE_ID_MS_XBOX_ONE_S_CONTROLLER 0x02fd
#define USB_DEVICE_ID_MS_PIXART_MOUSE 0x00cb
#define USB_DEVICE_ID_8BITDO_SN30_PRO_PLUS 0x02e0
#define map_rel(c) hid_map_usage(hidinput, usage, &bit, &max, EV_REL, (c))
#define map_key(c) hid_map_usage(hidinput, usage, &bit, &max, EV_KEY, (c))
#define map_led(c) hid_map_usage(hidinput, usage, &bit, &max, EV_LED, (c))
+#define map_msc(c) hid_map_usage(hidinput, usage, &bit, &max, EV_MSC, (c))
#define map_abs_clear(c) hid_map_usage_clear(hidinput, usage, &bit, \
&max, EV_ABS, (c))
if (usage) {
*old_keycode = usage->type == EV_KEY ?
usage->code : KEY_RESERVED;
+ usage->type = EV_KEY;
usage->code = ke->keycode;
clear_bit(*old_keycode, dev->keybit);
HID_BATTERY_QUIRK_IGNORE },
{ HID_USB_DEVICE(USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ASUS_UX550_TOUCHSCREEN),
HID_BATTERY_QUIRK_IGNORE },
+ { HID_USB_DEVICE(USB_VENDOR_ID_ELAN, USB_DEVICE_ID_ASUS_UX550VE_TOUCHSCREEN),
+ HID_BATTERY_QUIRK_IGNORE },
+ { HID_I2C_DEVICE(USB_VENDOR_ID_ELAN, I2C_DEVICE_ID_HP_ENVY_X360_15),
+ HID_BATTERY_QUIRK_IGNORE },
{ HID_I2C_DEVICE(USB_VENDOR_ID_ELAN, I2C_DEVICE_ID_HP_SPECTRE_X360_15),
HID_BATTERY_QUIRK_IGNORE },
{ HID_I2C_DEVICE(USB_VENDOR_ID_ELAN, I2C_DEVICE_ID_SURFACE_GO_TOUCHSCREEN),
code += KEY_MACRO1;
else
code += BTN_TRIGGER_HAPPY - 0x1e;
- } else {
- goto ignore;
+ break;
}
- break;
+ fallthrough;
default:
switch (field->physical) {
case HID_GD_MOUSE:
case 0x5b: /* TransducerSerialNumber */
case 0x6e: /* TransducerSerialNumber2 */
- usage->type = EV_MSC;
- usage->code = MSC_SERIAL;
- bit = input->mscbit;
- max = MSC_MAX;
+ map_msc(MSC_SERIAL);
break;
default: goto unknown;
input = field->hidinput->input;
+ if (usage->type == EV_ABS &&
+ (((*quirks & HID_QUIRK_X_INVERT) && usage->code == ABS_X) ||
+ ((*quirks & HID_QUIRK_Y_INVERT) && usage->code == ABS_Y))) {
+ value = field->logical_maximum - value;
+ }
+
if (usage->hat_min < usage->hat_max || usage->hat_dir) {
int hat_dir = usage->hat_dir;
if (!hat_dir)
}
EXPORT_SYMBOL_GPL(hidinput_report_event);
-int hidinput_find_field(struct hid_device *hid, unsigned int type, unsigned int code, struct hid_field **field)
+static int hidinput_find_field(struct hid_device *hid, unsigned int type,
+ unsigned int code, struct hid_field **field)
{
struct hid_report *report;
int i, j;
}
return -1;
}
-EXPORT_SYMBOL_GPL(hidinput_find_field);
struct hid_field *hidinput_get_led_field(struct hid_device *hid)
{
case HID_GD_MOUSE:
suffix = "Mouse";
break;
+ case HID_DG_PEN:
+ /*
+ * yes, there is an issue here:
+ * DG_PEN -> "Stylus"
+ * DG_STYLUS -> "Pen"
+ * But changing this now means users with config snippets
+ * will have to change it and the test suite will not be happy.
+ */
+ suffix = "Stylus";
+ break;
case HID_DG_STYLUS:
suffix = "Pen";
break;
static int lg_probe(struct hid_device *hdev, const struct hid_device_id *id)
{
- struct usb_interface *iface = to_usb_interface(hdev->dev.parent);
- __u8 iface_num = iface->cur_altsetting->desc.bInterfaceNumber;
+ struct usb_interface *iface;
+ __u8 iface_num;
unsigned int connect_mask = HID_CONNECT_DEFAULT;
struct lg_drv_data *drv_data;
int ret;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
+ iface = to_usb_interface(hdev->dev.parent);
+ iface_num = iface->cur_altsetting->desc.bInterfaceNumber;
+
/* G29 only work with the 1st interface */
if ((hdev->product == USB_DEVICE_ID_LOGITECH_G29_WHEEL) &&
(iface_num != 0)) {
case recvr_type_bluetooth: no_dj_interfaces = 2; break;
case recvr_type_dinovo: no_dj_interfaces = 2; break;
}
- if (hid_is_using_ll_driver(hdev, &usb_hid_driver)) {
+ if (hid_is_usb(hdev)) {
intf = to_usb_interface(hdev->dev.parent);
if (intf && intf->altsetting->desc.bInterfaceNumber >=
no_dj_interfaces) {
unsigned long now = jiffies;
int step_x = msc->touches[id].scroll_x - x;
int step_y = msc->touches[id].scroll_y - y;
- int step_hr = ((64 - (int)scroll_speed) * msc->scroll_accel) /
- SCROLL_HR_STEPS;
+ int step_hr =
+ max_t(int,
+ ((64 - (int)scroll_speed) * msc->scroll_accel) /
+ SCROLL_HR_STEPS,
+ 1);
int step_x_hr = msc->touches[id].scroll_x_hr - x;
int step_y_hr = msc->touches[id].scroll_y_hr - y;
case HID_DG_STYLUS:
/* force BTN_STYLUS to allow tablet matching in udev */
__set_bit(BTN_STYLUS, hi->input->keybit);
- fallthrough;
- case HID_DG_PEN:
- suffix = "Stylus";
break;
default:
suffix = "UNKNOWN";
MT_USB_DEVICE(USB_VENDOR_ID_CVTOUCH,
USB_DEVICE_ID_CVTOUCH_SCREEN) },
+ /* eGalax devices (SAW) */
+ { .driver_data = MT_CLS_EXPORT_ALL_INPUTS,
+ MT_USB_DEVICE(USB_VENDOR_ID_DWAV,
+ USB_DEVICE_ID_EGALAX_TOUCHCONTROLLER) },
+
/* eGalax devices (resistive) */
{ .driver_data = MT_CLS_EGALAX,
MT_USB_DEVICE(USB_VENDOR_ID_DWAV,
u16 amp;
};
+#if IS_ENABLED(CONFIG_NINTENDO_FF)
/*
* These tables are from
* https://github.com/dekuNukem/Nintendo_Switch_Reverse_Engineering/blob/master/rumble_data_table.md
{ 0xc2, 0x8070, 940 }, { 0xc4, 0x0071, 960 }, { 0xc6, 0x8071, 981 },
{ 0xc8, 0x0072, joycon_max_rumble_amp }
};
+static const u16 JC_RUMBLE_DFLT_LOW_FREQ = 160;
+static const u16 JC_RUMBLE_DFLT_HIGH_FREQ = 320;
+#endif /* IS_ENABLED(CONFIG_NINTENDO_FF) */
+static const u16 JC_RUMBLE_PERIOD_MS = 50;
/* States for controller state machine */
enum joycon_ctlr_state {
#define JC_RUMBLE_DATA_SIZE 8
#define JC_RUMBLE_QUEUE_SIZE 8
-static const u16 JC_RUMBLE_DFLT_LOW_FREQ = 160;
-static const u16 JC_RUMBLE_DFLT_HIGH_FREQ = 320;
-static const u16 JC_RUMBLE_PERIOD_MS = 50;
static const unsigned short JC_RUMBLE_ZERO_AMP_PKT_CNT = 5;
static const char * const joycon_player_led_names[] = {
d_name,
"green",
joycon_player_led_names[i]);
- if (!name)
+ if (!name) {
+ mutex_unlock(&joycon_input_num_mutex);
return -ENOMEM;
+ }
led = &ctlr->leds[i];
led->name = name;
ret = devm_led_classdev_register(&hdev->dev, led);
if (ret) {
hid_err(hdev, "Failed registering %s LED\n", led->name);
+ mutex_unlock(&joycon_input_num_mutex);
return ret;
}
}
static int pk_probe(struct hid_device *hdev, const struct hid_device_id *id)
{
int ret;
- struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
- unsigned short ifnum = intf->cur_altsetting->desc.bInterfaceNumber;
+ struct usb_interface *intf;
+ unsigned short ifnum;
unsigned long quirks = id->driver_data;
struct pk_device *pk;
struct pcmidi_snd *pm = NULL;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
+ intf = to_usb_interface(hdev->dev.parent);
+ ifnum = intf->cur_altsetting->desc.bInterfaceNumber;
+
pk = kzalloc(sizeof(*pk), GFP_KERNEL);
if (pk == NULL) {
hid_err(hdev, "can't alloc descriptor\n");
{ HID_USB_DEVICE(USB_VENDOR_ID_MCS, USB_DEVICE_ID_MCS_GAMEPADBLOCK), HID_QUIRK_MULTI_INPUT },
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_PIXART_MOUSE), HID_QUIRK_ALWAYS_POLL },
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_POWER_COVER), HID_QUIRK_NO_INIT_REPORTS },
+ { HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_SURFACE3_COVER), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_SURFACE_PRO_2), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_TOUCH_COVER_2), HID_QUIRK_NO_INIT_REPORTS },
{ HID_USB_DEVICE(USB_VENDOR_ID_MICROSOFT, USB_DEVICE_ID_MS_TYPE_COVER_2), HID_QUIRK_NO_INIT_REPORTS },
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
{
int retval;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
retval = hid_parse(hdev);
if (retval) {
hid_err(hdev, "parse failed\n");
int ret;
unsigned int cmask = HID_CONNECT_DEFAULT;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
ret = hid_parse(hdev);
if (ret) {
hid_err(hdev, "parse failed\n");
sc->quirks = quirks;
hid_set_drvdata(hdev, sc);
sc->hdev = hdev;
- usbdev = to_usb_device(sc->hdev->dev.parent->parent);
ret = hid_parse(hdev);
if (ret) {
*/
if (!(hdev->claimed & HID_CLAIMED_INPUT)) {
hid_err(hdev, "failed to claim input\n");
- hid_hw_stop(hdev);
- return -ENODEV;
+ ret = -ENODEV;
+ goto err;
}
if (sc->quirks & (GHL_GUITAR_PS3WIIU | GHL_GUITAR_PS4)) {
+ if (!hid_is_usb(hdev)) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ usbdev = to_usb_device(sc->hdev->dev.parent->parent);
+
sc->ghl_urb = usb_alloc_urb(0, GFP_ATOMIC);
- if (!sc->ghl_urb)
- return -ENOMEM;
+ if (!sc->ghl_urb) {
+ ret = -ENOMEM;
+ goto err;
+ }
if (sc->quirks & GHL_GUITAR_PS3WIIU)
ret = ghl_init_urb(sc, usbdev, ghl_ps3wiiu_magic_data,
ARRAY_SIZE(ghl_ps4_magic_data));
if (ret) {
hid_err(hdev, "error preparing URB\n");
- return ret;
+ goto err;
}
timer_setup(&sc->ghl_poke_timer, ghl_magic_poke, 0);
}
return ret;
+
+err:
+ hid_hw_stop(hdev);
+ return ret;
}
static void sony_remove(struct hid_device *hdev)
struct tm_wheel *tm_wheel = hid_get_drvdata(hdev);
uint16_t model = 0;
int i, ret;
- const struct tm_wheel_info *twi = 0;
+ const struct tm_wheel_info *twi = NULL;
if (urb->status) {
hid_err(hdev, "URB to get model id failed with error %d\n", urb->status);
tm_wheel->usb_dev,
usb_sndctrlpipe(tm_wheel->usb_dev, 0),
(char *)tm_wheel->change_request,
- 0, 0, // We do not expect any response from the wheel
+ NULL, 0, // We do not expect any response from the wheel
thrustmaster_change_handler,
hdev
);
static int thrustmaster_probe(struct hid_device *hdev, const struct hid_device_id *id)
{
int ret = 0;
- struct tm_wheel *tm_wheel = 0;
+ struct tm_wheel *tm_wheel = NULL;
+
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
ret = hid_parse(hdev);
if (ret) {
unsigned int minor;
int ret;
- if (!hid_is_using_ll_driver(hdev, &usb_hid_driver))
+ if (!hid_is_usb(hdev))
return -EINVAL;
dev = devm_kzalloc(&hdev->dev, sizeof(*dev), GFP_KERNEL);
struct uclogic_drvdata *drvdata = NULL;
bool params_initialized = false;
+ if (!hid_is_usb(hdev))
+ return -EINVAL;
+
/*
* libinput requires the pad interface to be on a different node
* than the pen, so use QUIRK_MULTI_INPUT for all tablets.
__u8 idx, size_t len)
{
int rc;
- struct usb_device *udev = hid_to_usb_dev(hdev);
+ struct usb_device *udev;
__u8 *buf = NULL;
/* Check arguments */
goto cleanup;
}
+ udev = hid_to_usb_dev(hdev);
+
buf = kmalloc(len, GFP_KERNEL);
if (buf == NULL) {
rc = -ENOMEM;
{
int rc;
bool found = false;
- struct usb_device *usb_dev = hid_to_usb_dev(hdev);
+ struct usb_device *usb_dev;
char *str_buf = NULL;
const size_t str_len = 16;
goto cleanup;
}
+ usb_dev = hid_to_usb_dev(hdev);
+
/*
* Enable generic button mode
*/
struct hid_device *hdev)
{
int rc;
- struct usb_device *udev = hid_to_usb_dev(hdev);
- struct usb_interface *iface = to_usb_interface(hdev->dev.parent);
- __u8 bInterfaceNumber = iface->cur_altsetting->desc.bInterfaceNumber;
+ struct usb_device *udev;
+ struct usb_interface *iface;
+ __u8 bInterfaceNumber;
bool found;
/* The resulting parameters (noop) */
struct uclogic_params p = {0, };
goto cleanup;
}
+ udev = hid_to_usb_dev(hdev);
+ iface = to_usb_interface(hdev->dev.parent);
+ bInterfaceNumber = iface->cur_altsetting->desc.bInterfaceNumber;
+
/* If it's not a pen interface */
if (bInterfaceNumber != 0) {
/* TODO: Consider marking the interface invalid */
struct hid_device *hdev)
{
int rc;
- struct usb_device *udev = hid_to_usb_dev(hdev);
- __u8 bNumInterfaces = udev->config->desc.bNumInterfaces;
- struct usb_interface *iface = to_usb_interface(hdev->dev.parent);
- __u8 bInterfaceNumber = iface->cur_altsetting->desc.bInterfaceNumber;
+ struct usb_device *udev;
+ __u8 bNumInterfaces;
+ struct usb_interface *iface;
+ __u8 bInterfaceNumber;
bool found;
/* The resulting parameters (noop) */
struct uclogic_params p = {0, };
/* Check arguments */
- if (params == NULL || hdev == NULL ||
- !hid_is_using_ll_driver(hdev, &usb_hid_driver)) {
+ if (params == NULL || hdev == NULL || !hid_is_usb(hdev)) {
rc = -EINVAL;
goto cleanup;
}
+ udev = hid_to_usb_dev(hdev);
+ bNumInterfaces = udev->config->desc.bNumInterfaces;
+ iface = to_usb_interface(hdev->dev.parent);
+ bInterfaceNumber = iface->cur_altsetting->desc.bInterfaceNumber;
+
/*
* Set replacement report descriptor if the original matches the
* specified size. Otherwise keep interface unchanged.
int ret;
drvdata = devm_kzalloc(&hdev->dev, sizeof(*drvdata), GFP_KERNEL);
+ if (!drvdata)
+ return -ENOMEM;
+
hid_set_drvdata(hdev, drvdata);
ret = hid_parse(hdev);
}
return i2c_hid_core_probe(client, &ihid_acpi->ops,
- hid_descriptor_address);
+ hid_descriptor_address, 0);
}
static const struct acpi_device_id i2c_hid_acpi_match[] = {
}
int i2c_hid_core_probe(struct i2c_client *client, struct i2chid_ops *ops,
- u16 hid_descriptor_address)
+ u16 hid_descriptor_address, u32 quirks)
{
int ret;
struct i2c_hid *ihid;
goto err_mem_free;
}
+ hid->quirks |= quirks;
+
return 0;
err_mem_free:
int ret;
int wake_status;
- if (hid->driver && hid->driver->suspend) {
- ret = hid->driver->suspend(hid, PMSG_SUSPEND);
- if (ret < 0)
- return ret;
- }
+ ret = hid_driver_suspend(hid, PMSG_SUSPEND);
+ if (ret < 0)
+ return ret;
/* Save some power */
i2c_hid_set_power(client, I2C_HID_PWR_SLEEP);
if (ret)
return ret;
- if (hid->driver && hid->driver->reset_resume) {
- ret = hid->driver->reset_resume(hid);
- return ret;
- }
-
- return 0;
+ return hid_driver_reset_resume(hid);
}
#endif
goodix_i2c_hid_deassert_reset(ihid_goodix, true);
mutex_unlock(&ihid_goodix->regulator_mutex);
- return i2c_hid_core_probe(client, &ihid_goodix->ops, 0x0001);
+ return i2c_hid_core_probe(client, &ihid_goodix->ops, 0x0001, 0);
}
static const struct goodix_i2c_hid_timing_data goodix_gt7375p_timing_data = {
#include <linux/delay.h>
#include <linux/device.h>
+#include <linux/hid.h>
#include <linux/i2c.h>
#include <linux/kernel.h>
#include <linux/module.h>
struct device *dev = &client->dev;
struct i2c_hid_of *ihid_of;
u16 hid_descriptor_address;
+ u32 quirks = 0;
int ret;
u32 val;
if (ret)
return ret;
+ if (device_property_read_bool(dev, "touchscreen-inverted-x"))
+ quirks |= HID_QUIRK_X_INVERT;
+
+ if (device_property_read_bool(dev, "touchscreen-inverted-y"))
+ quirks |= HID_QUIRK_Y_INVERT;
+
return i2c_hid_core_probe(client, &ihid_of->ops,
- hid_descriptor_address);
+ hid_descriptor_address, quirks);
}
static const struct of_device_id i2c_hid_of_match[] = {
};
int i2c_hid_core_probe(struct i2c_client *client, struct i2chid_ops *ops,
- u16 hid_descriptor_address);
+ u16 hid_descriptor_address, u32 quirks);
int i2c_hid_core_remove(struct i2c_client *client);
void i2c_hid_core_shutdown(struct i2c_client *client);
*/
static bool _dma_no_cache_snooping(struct ishtp_device *dev)
{
- return dev->pdev->device == EHL_Ax_DEVICE_ID;
+ return (dev->pdev->device == EHL_Ax_DEVICE_ID ||
+ dev->pdev->device == TGL_LP_DEVICE_ID ||
+ dev->pdev->device == TGL_H_DEVICE_ID ||
+ dev->pdev->device == ADL_S_DEVICE_ID ||
+ dev->pdev->device == ADL_P_DEVICE_ID);
}
static const struct ishtp_hw_ops ish_hw_ops = {
if (ish_should_leave_d0i3(pdev) && !dev->suspend_flag
&& IPC_IS_ISH_ILUP(fwsts)) {
- disable_irq_wake(pdev->irq);
+ if (device_may_wakeup(&pdev->dev))
+ disable_irq_wake(pdev->irq);
ish_set_host_ready(dev);
*/
pci_save_state(pdev);
- enable_irq_wake(pdev->irq);
+ if (device_may_wakeup(&pdev->dev))
+ enable_irq_wake(pdev->irq);
}
} else {
/*
#define LOADER_XFER_MODE_ISHTP BIT(1)
/* ISH Transport Loader client unique GUID */
-static const guid_t loader_ishtp_guid =
- GUID_INIT(0xc804d06a, 0x55bd, 0x4ea7,
- 0xad, 0xed, 0x1e, 0x31, 0x22, 0x8c, 0x76, 0xdc);
+static const struct ishtp_device_id loader_ishtp_id_table[] = {
+ { .guid = GUID_INIT(0xc804d06a, 0x55bd, 0x4ea7,
+ 0xad, 0xed, 0x1e, 0x31, 0x22, 0x8c, 0x76, 0xdc) },
+ { }
+};
+MODULE_DEVICE_TABLE(ishtp, loader_ishtp_id_table);
#define FILENAME_SIZE 256
}
/**
- * loader_cl_send() Send message from host to firmware
+ * loader_cl_send() - Send message from host to firmware
+ *
* @client_data: Client data instance
* @out_msg: Message buffer to be sent to firmware
* @out_size: Size of out going message
fw_client =
ishtp_fw_cl_get_client(ishtp_get_ishtp_device(loader_ishtp_cl),
- &loader_ishtp_guid);
+ &loader_ishtp_id_table[0].guid);
if (!fw_client) {
dev_err(cl_data_to_dev(client_data),
"ISH client uuid not found\n");
static struct ishtp_cl_driver loader_ishtp_cl_driver = {
.name = "ish-loader",
- .guid = &loader_ishtp_guid,
+ .id = loader_ishtp_id_table,
.probe = loader_ishtp_cl_probe,
.remove = loader_ishtp_cl_remove,
.reset = loader_ishtp_cl_reset,
MODULE_AUTHOR("Rushikesh S Kadam <rushikesh.s.kadam@intel.com>");
MODULE_LICENSE("GPL v2");
-MODULE_ALIAS("ishtp:*");
#include "ishtp-hid.h"
/* ISH Transport protocol (ISHTP in short) GUID */
-static const guid_t hid_ishtp_guid =
- GUID_INIT(0x33AECD58, 0xB679, 0x4E54,
- 0x9B, 0xD9, 0xA0, 0x4D, 0x34, 0xF0, 0xC2, 0x26);
+static const struct ishtp_device_id hid_ishtp_id_table[] = {
+ { .guid = GUID_INIT(0x33AECD58, 0xB679, 0x4E54,
+ 0x9B, 0xD9, 0xA0, 0x4D, 0x34, 0xF0, 0xC2, 0x26), },
+ { }
+};
+MODULE_DEVICE_TABLE(ishtp, hid_ishtp_id_table);
/* Rx ring buffer pool size */
#define HID_CL_RX_RING_SIZE 32
ishtp_set_tx_ring_size(hid_ishtp_cl, HID_CL_TX_RING_SIZE);
ishtp_set_rx_ring_size(hid_ishtp_cl, HID_CL_RX_RING_SIZE);
- fw_client = ishtp_fw_cl_get_client(dev, &hid_ishtp_guid);
+ fw_client = ishtp_fw_cl_get_client(dev, &hid_ishtp_id_table[0].guid);
if (!fw_client) {
dev_err(cl_data_to_dev(client_data),
"ish client uuid not found\n");
static struct ishtp_cl_driver hid_ishtp_cl_driver = {
.name = "ish-hid",
- .guid = &hid_ishtp_guid,
+ .id = hid_ishtp_id_table,
.probe = hid_ishtp_cl_probe,
.remove = hid_ishtp_cl_remove,
.reset = hid_ishtp_cl_reset,
MODULE_AUTHOR("Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>");
MODULE_LICENSE("GPL");
-MODULE_ALIAS("ishtp:*");
struct ishtp_cl_device *device = to_ishtp_cl_device(dev);
struct ishtp_cl_driver *driver = to_ishtp_cl_driver(drv);
- return guid_equal(driver->guid,
+ return guid_equal(&driver->id[0].guid,
&device->fw_client->props.protocol_name);
}
{
int len;
- len = snprintf(buf, PAGE_SIZE, "ishtp:%s\n", dev_name(dev));
+ len = snprintf(buf, PAGE_SIZE, ISHTP_MODULE_PREFIX "%s\n", dev_name(dev));
return (len >= PAGE_SIZE) ? (PAGE_SIZE - 1) : len;
}
static DEVICE_ATTR_RO(modalias);
static int ishtp_cl_uevent(struct device *dev, struct kobj_uevent_env *env)
{
- if (add_uevent_var(env, "MODALIAS=ishtp:%s", dev_name(dev)))
+ if (add_uevent_var(env, "MODALIAS=" ISHTP_MODULE_PREFIX "%s", dev_name(dev)))
return -ENOMEM;
return 0;
}
{
struct surface_hid_device *d = dev_get_drvdata(dev);
- if (d->hid->driver && d->hid->driver->suspend)
- return d->hid->driver->suspend(d->hid, PMSG_SUSPEND);
-
- return 0;
+ return hid_driver_suspend(d->hid, PMSG_SUSPEND);
}
static int surface_hid_resume(struct device *dev)
{
struct surface_hid_device *d = dev_get_drvdata(dev);
- if (d->hid->driver && d->hid->driver->resume)
- return d->hid->driver->resume(d->hid);
-
- return 0;
+ return hid_driver_resume(d->hid);
}
static int surface_hid_freeze(struct device *dev)
{
struct surface_hid_device *d = dev_get_drvdata(dev);
- if (d->hid->driver && d->hid->driver->suspend)
- return d->hid->driver->suspend(d->hid, PMSG_FREEZE);
-
- return 0;
+ return hid_driver_suspend(d->hid, PMSG_FREEZE);
}
static int surface_hid_poweroff(struct device *dev)
{
struct surface_hid_device *d = dev_get_drvdata(dev);
- if (d->hid->driver && d->hid->driver->suspend)
- return d->hid->driver->suspend(d->hid, PMSG_HIBERNATE);
-
- return 0;
+ return hid_driver_suspend(d->hid, PMSG_HIBERNATE);
}
static int surface_hid_restore(struct device *dev)
{
struct surface_hid_device *d = dev_get_drvdata(dev);
- if (d->hid->driver && d->hid->driver->reset_resume)
- return d->hid->driver->reset_resume(d->hid);
-
- return 0;
+ return hid_driver_reset_resume(d->hid);
}
const struct dev_pm_ops surface_hid_pm_ops = {
int status = 0;
hid_restart_io(hid);
- if (driver_suspended && hid->driver && hid->driver->resume)
- status = hid->driver->resume(hid);
+ if (driver_suspended)
+ status = hid_driver_resume(hid);
return status;
}
{
set_bit(HID_SUSPENDED, &usbhid->iofl);
spin_unlock_irq(&usbhid->lock);
- if (hid->driver && hid->driver->suspend) {
- status = hid->driver->suspend(hid, message);
- if (status < 0)
- goto failed;
- }
+ status = hid_driver_suspend(hid, message);
+ if (status < 0)
+ goto failed;
driver_suspended = true;
} else {
usbhid_mark_busy(usbhid);
} else {
/* TODO: resume() might need to handle suspend failure */
- if (hid->driver && hid->driver->suspend)
- status = hid->driver->suspend(hid, message);
+ status = hid_driver_suspend(hid, message);
driver_suspended = true;
spin_lock_irq(&usbhid->lock);
set_bit(HID_SUSPENDED, &usbhid->iofl);
int status;
status = hid_post_reset(intf);
- if (status >= 0 && hid->driver && hid->driver->reset_resume) {
- int ret = hid->driver->reset_resume(hid);
+ if (status >= 0) {
+ int ret = hid_driver_reset_resume(hid);
if (ret < 0)
status = ret;
}
* Skip the query for this type and modify defaults based on
* interface number.
*/
- if (features->type == WIRELESS) {
+ if (features->type == WIRELESS && intf) {
if (intf->cur_altsetting->desc.bInterfaceNumber == 0)
features->device_type = WACOM_DEVICETYPE_WL_MONITOR;
else
if ((features->type == HID_GENERIC) && !strcmp("Wacom HID", features->name)) {
char *product_name = wacom->hdev->name;
- if (hid_is_using_ll_driver(wacom->hdev, &usb_hid_driver)) {
+ if (hid_is_usb(wacom->hdev)) {
struct usb_interface *intf = to_usb_interface(wacom->hdev->dev.parent);
struct usb_device *dev = interface_to_usbdev(intf);
product_name = dev->product;
wacom_destroy_battery(wacom);
+ if (!usbdev)
+ return;
+
/* Stylus interface */
hdev1 = usb_get_intfdata(usbdev->config->interface[1]);
wacom1 = hid_get_drvdata(hdev1);
static int wacom_probe(struct hid_device *hdev,
const struct hid_device_id *id)
{
- struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
- struct usb_device *dev = interface_to_usbdev(intf);
struct wacom *wacom;
struct wacom_wac *wacom_wac;
struct wacom_features *features;
wacom_wac->hid_data.inputmode = -1;
wacom_wac->mode_report = -1;
- wacom->usbdev = dev;
- wacom->intf = intf;
+ if (hid_is_usb(hdev)) {
+ struct usb_interface *intf = to_usb_interface(hdev->dev.parent);
+ struct usb_device *dev = interface_to_usbdev(intf);
+
+ wacom->usbdev = dev;
+ wacom->intf = intf;
+ }
+
mutex_init(&wacom->lock);
INIT_DELAYED_WORK(&wacom->init_work, wacom_init_work);
INIT_WORK(&wacom->wireless_work, wacom_wireless_work);
return;
switch (equivalent_usage) {
+ case HID_DG_CONFIDENCE:
+ wacom_wac->hid_data.confidence = value;
+ break;
case HID_GD_X:
wacom_wac->hid_data.x = value;
break;
}
if (usage->usage_index + 1 == field->report_count) {
- if (equivalent_usage == wacom_wac->hid_data.last_slot_field)
+ if (equivalent_usage == wacom_wac->hid_data.last_slot_field &&
+ wacom_wac->hid_data.confidence)
wacom_wac_finger_slot(wacom_wac, wacom_wac->touch_input);
}
}
wacom_wac->is_invalid_bt_frame = false;
+ hid_data->confidence = true;
+
for (i = 0; i < report->maxfield; i++) {
struct hid_field *field = report->field[i];
int j;
bool barrelswitch;
bool barrelswitch2;
bool serialhi;
+ bool confidence;
int x;
int y;
int pressure;
MODULE_PARM_DESC(pressure_report_delay, "Delay in secs in reporting pressure");
static atomic_t trans_id = ATOMIC_INIT(0);
-static int dm_ring_size = 20 * 1024;
+static int dm_ring_size = VMBUS_RING_SIZE(16 * 1024);
/*
* Driver specific state.
#include <linux/device.h>
#include <linux/errno.h>
+#include <linux/slab.h>
#include <linux/fsi-occ.h>
#include <linux/mm.h>
#include <linux/module.h>
depends on ARCH_EXYNOS || COMPILE_TEST
default y if ARCH_EXYNOS
help
- High-speed I2C controller on Exynos5 and newer Samsung SoCs.
+ High-speed I2C controller on Samsung Exynos5 and newer Samsung SoCs:
+ Exynos5250, Exynos5260, Exynos5410, Exynos542x, Exynos5800,
+ Exynos5433 and Exynos7.
+ Choose Y here only if you build for such Samsung SoC.
config I2C_GPIO
tristate "GPIO-based bitbanging I2C"
help
Supports the PA Semi PWRficient on-chip SMBus interfaces.
+config I2C_APPLE
+ tristate "Apple SMBus platform driver"
+ depends on ARCH_APPLE || COMPILE_TEST
+ default ARCH_APPLE
+ help
+ Say Y here if you want to use the I2C controller present on Apple
+ Silicon chips such as the M1.
+
+ This driver can also be built as a module. If so, the module
+ will be called i2c-apple.
+
config I2C_PCA_PLATFORM
tristate "PCA9564/PCA9665 as platform device"
select I2C_ALGOPCA
obj-$(CONFIG_I2C_OCORES) += i2c-ocores.o
obj-$(CONFIG_I2C_OMAP) += i2c-omap.o
obj-$(CONFIG_I2C_OWL) += i2c-owl.o
+i2c-pasemi-objs := i2c-pasemi-core.o i2c-pasemi-pci.o
obj-$(CONFIG_I2C_PASEMI) += i2c-pasemi.o
+i2c-apple-objs := i2c-pasemi-core.o i2c-pasemi-platform.o
+obj-$(CONFIG_I2C_APPLE) += i2c-apple.o
obj-$(CONFIG_I2C_PCA_PLATFORM) += i2c-pca-platform.o
obj-$(CONFIG_I2C_PNX) += i2c-pnx.o
obj-$(CONFIG_I2C_PXA) += i2c-pxa.o
pci_set_master(pci_dev);
- rc = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(64));
+ rc = dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(64));
if (rc) {
- rc = pci_set_dma_mask(pci_dev, DMA_BIT_MASK(32));
+ rc = dma_set_mask(&pci_dev->dev, DMA_BIT_MASK(32));
if (rc)
goto err_dma_mask;
}
{
int ret;
struct amd_i2c_dev *i2c_dev;
- acpi_handle handle = ACPI_HANDLE(&pdev->dev);
- struct acpi_device *adev;
+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev);
struct amd_mp2_dev *mp2_dev;
const char *uid;
- if (acpi_bus_get_device(handle, &adev))
+ if (!adev)
return -ENODEV;
/* The ACPI namespace doesn't contain information about which MP2 PCI
/* Map hardware registers */
dev->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(dev->base))
- return -ENOMEM;
+ return PTR_ERR(dev->base);
/* Get and enable external clock */
dev->external_clk = devm_clk_get(dev->device, NULL);
* Cannon Lake-LP (PCH) 0x9da3 32 hard yes yes yes
* Cedar Fork (PCH) 0x18df 32 hard yes yes yes
* Ice Lake-LP (PCH) 0x34a3 32 hard yes yes yes
+ * Ice Lake-N (PCH) 0x38a3 32 hard yes yes yes
* Comet Lake (PCH) 0x02a3 32 hard yes yes yes
* Comet Lake-H (PCH) 0x06a3 32 hard yes yes yes
* Elkhart Lake (PCH) 0x4b23 32 hard yes yes yes
#define PCI_DEVICE_ID_INTEL_COLETOCREEK_SMBUS 0x23b0
#define PCI_DEVICE_ID_INTEL_GEMINILAKE_SMBUS 0x31d4
#define PCI_DEVICE_ID_INTEL_ICELAKE_LP_SMBUS 0x34a3
+#define PCI_DEVICE_ID_INTEL_ICELAKE_N_SMBUS 0x38a3
#define PCI_DEVICE_ID_INTEL_5_3400_SERIES_SMBUS 0x3b30
#define PCI_DEVICE_ID_INTEL_TIGERLAKE_H_SMBUS 0x43a3
#define PCI_DEVICE_ID_INTEL_ELKHART_LAKE_SMBUS 0x4b23
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CANNONLAKE_H_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CANNONLAKE_LP_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICELAKE_LP_SMBUS) },
+ { PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICELAKE_N_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_COMETLAKE_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_COMETLAKE_H_SMBUS) },
{ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_COMETLAKE_V_SMBUS) },
kfree(info);
- *((bool *)return_value) = true;
+ *return_value = NULL;
return AE_CTRL_TERMINATE;
smo88xx_not_found:
static bool is_dell_system_with_lis3lv02d(void)
{
- bool found;
- const char *vendor;
+ void *err = ERR_PTR(-ENOENT);
- vendor = dmi_get_system_info(DMI_SYS_VENDOR);
- if (!vendor || strcmp(vendor, "Dell Inc."))
+ if (!dmi_match(DMI_SYS_VENDOR, "Dell Inc."))
return false;
/*
* accelerometer but unfortunately ACPI does not provide any other
* information (like I2C address).
*/
- found = false;
- acpi_get_devices(NULL, check_acpi_smo88xx_device, NULL,
- (void **)&found);
+ acpi_get_devices(NULL, check_acpi_smo88xx_device, NULL, &err);
- return found;
+ return !IS_ERR(err);
}
/*
};
/* Setup multiplexing if needed */
-static int i801_add_mux(struct i801_priv *priv)
+static void i801_add_mux(struct i801_priv *priv)
{
struct device *dev = &priv->adapter.dev;
const struct i801_mux_config *mux_config;
int i;
if (!priv->mux_drvdata)
- return 0;
+ return;
mux_config = priv->mux_drvdata;
/* Prepare the platform data */
struct_size(lookup, table, mux_config->n_gpios + 1),
GFP_KERNEL);
if (!lookup)
- return -ENOMEM;
+ return;
lookup->dev_id = "i2c-mux-gpio";
- for (i = 0; i < mux_config->n_gpios; i++) {
- lookup->table[i] = (struct gpiod_lookup)
- GPIO_LOOKUP(mux_config->gpio_chip,
- mux_config->gpios[i], "mux", 0);
- }
+ for (i = 0; i < mux_config->n_gpios; i++)
+ lookup->table[i] = GPIO_LOOKUP(mux_config->gpio_chip,
+ mux_config->gpios[i], "mux", 0);
gpiod_add_lookup_table(lookup);
priv->lookup = lookup;
gpiod_remove_lookup_table(lookup);
dev_err(dev, "Failed to register i2c-mux-gpio device\n");
}
-
- return PTR_ERR_OR_ZERO(priv->mux_pdev);
}
static void i801_del_mux(struct i801_priv *priv)
return class;
}
#else
-static inline int i801_add_mux(struct i801_priv *priv) { return 0; }
+static inline void i801_add_mux(struct i801_priv *priv) { }
static inline void i801_del_mux(struct i801_priv *priv) { }
static inline unsigned int i801_get_adapter_class(struct i801_priv *priv)
i801_add_tco_spt(struct i801_priv *priv, struct pci_dev *pci_dev,
struct resource *tco_res)
{
- static DEFINE_MUTEX(p2sb_mutex);
struct resource *res;
unsigned int devfn;
u64 base64_addr;
* enumerated by the PCI subsystem, so we need to unhide/hide it
* to lookup the P2SB BAR.
*/
- mutex_lock(&p2sb_mutex);
+ pci_lock_rescan_remove();
devfn = PCI_DEVFN(PCI_SLOT(pci_dev->devfn), 1);
/* Hide the P2SB device, if it was hidden before */
if (hidden)
pci_bus_write_config_byte(pci_dev->bus, devfn, 0xe1, hidden);
- mutex_unlock(&p2sb_mutex);
+ pci_unlock_rescan_remove();
res = &tco_res[1];
if (pci_dev->device == PCI_DEVICE_ID_INTEL_DNV_SMBUS)
* BIOS is accessing the host controller so prevent it from
* suspending automatically from now on.
*/
- pm_runtime_set_autosuspend_delay(&pdev->dev, -1);
+ pm_runtime_get_sync(&pdev->dev);
}
if ((function & ACPI_IO_MASK) == ACPI_READ)
static int i801_acpi_probe(struct i801_priv *priv)
{
- struct acpi_device *adev;
+ acpi_handle ah = ACPI_HANDLE(&priv->pci_dev->dev);
acpi_status status;
- adev = ACPI_COMPANION(&priv->pci_dev->dev);
- if (adev) {
- status = acpi_install_address_space_handler(adev->handle,
- ACPI_ADR_SPACE_SYSTEM_IO, i801_acpi_io_handler,
- NULL, priv);
- if (ACPI_SUCCESS(status))
- return 0;
- }
+ status = acpi_install_address_space_handler(ah, ACPI_ADR_SPACE_SYSTEM_IO,
+ i801_acpi_io_handler, NULL, priv);
+ if (ACPI_SUCCESS(status))
+ return 0;
return acpi_check_resource_conflict(&priv->pci_dev->resource[SMBBAR]);
}
static void i801_acpi_remove(struct i801_priv *priv)
{
- struct acpi_device *adev;
+ acpi_handle ah = ACPI_HANDLE(&priv->pci_dev->dev);
- adev = ACPI_COMPANION(&priv->pci_dev->dev);
- if (!adev)
- return;
-
- acpi_remove_address_space_handler(adev->handle,
- ACPI_ADR_SPACE_SYSTEM_IO, i801_acpi_io_handler);
+ acpi_remove_address_space_handler(ah, ACPI_ADR_SPACE_SYSTEM_IO, i801_acpi_io_handler);
}
#else
static inline int i801_acpi_probe(struct i801_priv *priv) { return 0; }
unsigned char hstcfg = priv->original_hstcfg;
hstcfg &= ~SMBHSTCFG_I2C_EN; /* SMBus timing */
- hstcfg &= ~SMBHSTCNT_PEC_EN; /* Disable software PEC */
hstcfg |= SMBHSTCFG_HST_EN;
pci_write_config_byte(priv->pci_dev, SMBHSTCFG, hstcfg);
}
case PCI_DEVICE_ID_INTEL_CANNONLAKE_LP_SMBUS:
case PCI_DEVICE_ID_INTEL_CDF_SMBUS:
case PCI_DEVICE_ID_INTEL_ICELAKE_LP_SMBUS:
+ case PCI_DEVICE_ID_INTEL_ICELAKE_N_SMBUS:
case PCI_DEVICE_ID_INTEL_COMETLAKE_SMBUS:
case PCI_DEVICE_ID_INTEL_COMETLAKE_H_SMBUS:
case PCI_DEVICE_ID_INTEL_ELKHART_LAKE_SMBUS:
priv->features &= ~FEATURE_IRQ;
if (priv->features & FEATURE_IRQ) {
- u16 pcictl, pcists;
+ u16 pcists;
/* Complain if an interrupt is already pending */
pci_read_config_word(priv->pci_dev, PCI_STATUS, &pcists);
if (pcists & PCI_STATUS_INTERRUPT)
dev_warn(&dev->dev, "An interrupt is pending!\n");
-
- /* Check if interrupts have been disabled */
- pci_read_config_word(priv->pci_dev, PCI_COMMAND, &pcictl);
- if (pcictl & PCI_COMMAND_INTX_DISABLE) {
- dev_info(&dev->dev, "Interrupts are disabled\n");
- priv->features &= ~FEATURE_IRQ;
- }
}
if (priv->features & FEATURE_IRQ) {
{
struct i801_priv *priv = pci_get_drvdata(dev);
- pm_runtime_forbid(&dev->dev);
- pm_runtime_get_noresume(&dev->dev);
-
i801_disable_host_notify(priv);
i801_del_mux(priv);
i2c_del_adapter(&priv->adapter);
platform_device_unregister(priv->tco_pdev);
+ /* if acpi_reserved is set then usage_count is incremented already */
+ if (!priv->acpi_reserved)
+ pm_runtime_get_noresume(&dev->dev);
+
/*
* do not call pci_disable_device(dev) since it can cause hard hangs on
* some systems during power-off (eg. Fujitsu-Siemens Lifebook E8010)
return -ENODEV;
}
- if ((pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) != 0) ||
- (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)) != 0)) {
- if ((pci_set_dma_mask(pdev, DMA_BIT_MASK(32)) != 0) ||
- (pci_set_consistent_dma_mask(pdev,
- DMA_BIT_MASK(32)) != 0)) {
- dev_err(&pdev->dev, "pci_set_dma_mask fail %p\n",
- pdev);
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (err) {
+ err = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (err) {
+ dev_err(&pdev->dev, "dma_set_mask fail\n");
return -ENODEV;
}
}
static const struct i2c_adapter kempld_i2c_adapter = {
.owner = THIS_MODULE,
.name = "i2c-kempld",
- .class = I2C_CLASS_HWMON | I2C_CLASS_SPD,
+ .class = I2C_CLASS_HWMON | I2C_CLASS_SPD |
+ I2C_CLASS_DEPRECATED,
.algo = &kempld_i2c_algorithm,
};
#define MLXCPLD_I2C_MAX_ADDR_LEN 4
#define MLXCPLD_I2C_RETR_NUM 2
#define MLXCPLD_I2C_XFER_TO 500000 /* usec */
-#define MLXCPLD_I2C_POLL_TIME 400 /* usec */
+#define MLXCPLD_I2C_POLL_TIME 200 /* usec */
/* LPC I2C registers */
#define MLXCPLD_LPCI2C_CPBLTY_REG 0x0
struct mlxcpld_i2c_curr_xfer xfer;
struct device *dev;
bool smbus_block;
+ int polling_time;
};
static void mlxcpld_i2c_lpc_write_buf(u8 *data, u8 len, u32 addr)
do {
if (!mlxcpld_i2c_check_busy(priv))
break;
- usleep_range(MLXCPLD_I2C_POLL_TIME / 2, MLXCPLD_I2C_POLL_TIME);
- timeout += MLXCPLD_I2C_POLL_TIME;
+ usleep_range(priv->polling_time / 2, priv->polling_time);
+ timeout += priv->polling_time;
} while (timeout <= MLXCPLD_I2C_XFER_TO);
if (timeout > MLXCPLD_I2C_XFER_TO)
u8 datalen, val;
do {
- usleep_range(MLXCPLD_I2C_POLL_TIME / 2, MLXCPLD_I2C_POLL_TIME);
+ usleep_range(priv->polling_time / 2, priv->polling_time);
if (!mlxcpld_i2c_check_status(priv, &status))
break;
- timeout += MLXCPLD_I2C_POLL_TIME;
+ timeout += priv->polling_time;
} while (status == 0 && timeout < MLXCPLD_I2C_XFER_TO);
switch (status) {
switch ((regval & data->mask) >> data->bit) {
case MLXCPLD_I2C_FREQ_1000KHZ:
freq = MLXCPLD_I2C_FREQ_1000KHZ_SET;
+ priv->polling_time /= 4;
break;
case MLXCPLD_I2C_FREQ_400KHZ:
freq = MLXCPLD_I2C_FREQ_400KHZ_SET;
+ priv->polling_time /= 4;
break;
default:
return 0;
priv->dev = &pdev->dev;
priv->base_addr = MLXPLAT_CPLD_LPC_I2C_BASE_ADDR;
+ priv->polling_time = MLXCPLD_I2C_POLL_TIME;
/* Set I2C bus frequency if platform data provides this info. */
pdata = dev_get_platdata(&pdev->dev);
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/io.h>
+#include <linux/iopoll.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/module.h>
#define I2C_RD_TRANAC_VALUE 0x0001
#define I2C_SCL_MIS_COMP_VALUE 0x0000
#define I2C_CHN_CLR_FLAG 0x0000
+#define I2C_RELIABILITY 0x0010
+#define I2C_DMAACK_ENABLE 0x0008
#define I2C_DMA_CON_TX 0x0000
#define I2C_DMA_CON_RX 0x0001
OFFSET_HS,
OFFSET_SOFTRESET,
OFFSET_DCM_EN,
+ OFFSET_MULTI_DMA,
OFFSET_PATH_DIR,
OFFSET_DEBUGSTAT,
OFFSET_DEBUGCTRL,
[OFFSET_TRANSFER_LEN_AUX] = 0x44,
[OFFSET_CLOCK_DIV] = 0x48,
[OFFSET_SOFTRESET] = 0x50,
+ [OFFSET_MULTI_DMA] = 0x8c,
[OFFSET_SCL_MIS_COMP_POINT] = 0x90,
- [OFFSET_DEBUGSTAT] = 0xe0,
+ [OFFSET_DEBUGSTAT] = 0xe4,
[OFFSET_DEBUGCTRL] = 0xe8,
[OFFSET_FIFO_STAT] = 0xf4,
[OFFSET_FIFO_THRESH] = 0xf8,
return 0;
}
+static void i2c_dump_register(struct mtk_i2c *i2c)
+{
+ dev_dbg(i2c->dev, "SLAVE_ADDR: 0x%x, INTR_MASK: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_SLAVE_ADDR),
+ mtk_i2c_readw(i2c, OFFSET_INTR_MASK));
+ dev_dbg(i2c->dev, "INTR_STAT: 0x%x, CONTROL: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_INTR_STAT),
+ mtk_i2c_readw(i2c, OFFSET_CONTROL));
+ dev_dbg(i2c->dev, "TRANSFER_LEN: 0x%x, TRANSAC_LEN: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_TRANSFER_LEN),
+ mtk_i2c_readw(i2c, OFFSET_TRANSAC_LEN));
+ dev_dbg(i2c->dev, "DELAY_LEN: 0x%x, HTIMING: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_DELAY_LEN),
+ mtk_i2c_readw(i2c, OFFSET_TIMING));
+ dev_dbg(i2c->dev, "START: 0x%x, EXT_CONF: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_START),
+ mtk_i2c_readw(i2c, OFFSET_EXT_CONF));
+ dev_dbg(i2c->dev, "HS: 0x%x, IO_CONFIG: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_HS),
+ mtk_i2c_readw(i2c, OFFSET_IO_CONFIG));
+ dev_dbg(i2c->dev, "DCM_EN: 0x%x, TRANSFER_LEN_AUX: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_DCM_EN),
+ mtk_i2c_readw(i2c, OFFSET_TRANSFER_LEN_AUX));
+ dev_dbg(i2c->dev, "CLOCK_DIV: 0x%x, FIFO_STAT: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_CLOCK_DIV),
+ mtk_i2c_readw(i2c, OFFSET_FIFO_STAT));
+ dev_dbg(i2c->dev, "DEBUGCTRL : 0x%x, DEBUGSTAT: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_DEBUGCTRL),
+ mtk_i2c_readw(i2c, OFFSET_DEBUGSTAT));
+ if (i2c->dev_comp->regs == mt_i2c_regs_v2) {
+ dev_dbg(i2c->dev, "LTIMING: 0x%x, MULTI_DMA: 0x%x\n",
+ mtk_i2c_readw(i2c, OFFSET_LTIMING),
+ mtk_i2c_readw(i2c, OFFSET_MULTI_DMA));
+ }
+ dev_dbg(i2c->dev, "\nDMA_INT_FLAG: 0x%x, DMA_INT_EN: 0x%x\n",
+ readl(i2c->pdmabase + OFFSET_INT_FLAG),
+ readl(i2c->pdmabase + OFFSET_INT_EN));
+ dev_dbg(i2c->dev, "DMA_EN: 0x%x, DMA_CON: 0x%x\n",
+ readl(i2c->pdmabase + OFFSET_EN),
+ readl(i2c->pdmabase + OFFSET_CON));
+ dev_dbg(i2c->dev, "DMA_TX_MEM_ADDR: 0x%x, DMA_RX_MEM_ADDR: 0x%x\n",
+ readl(i2c->pdmabase + OFFSET_TX_MEM_ADDR),
+ readl(i2c->pdmabase + OFFSET_RX_MEM_ADDR));
+ dev_dbg(i2c->dev, "DMA_TX_LEN: 0x%x, DMA_RX_LEN: 0x%x\n",
+ readl(i2c->pdmabase + OFFSET_TX_LEN),
+ readl(i2c->pdmabase + OFFSET_RX_LEN));
+ dev_dbg(i2c->dev, "DMA_TX_4G_MODE: 0x%x, DMA_RX_4G_MODE: 0x%x",
+ readl(i2c->pdmabase + OFFSET_TX_4G_MODE),
+ readl(i2c->pdmabase + OFFSET_RX_4G_MODE));
+}
+
static int mtk_i2c_do_transfer(struct mtk_i2c *i2c, struct i2c_msg *msgs,
int num, int left_num)
{
u16 restart_flag = 0;
u16 dma_sync = 0;
u32 reg_4g_mode;
+ u32 reg_dma_reset;
u8 *dma_rd_buf = NULL;
u8 *dma_wr_buf = NULL;
dma_addr_t rpaddr = 0;
reinit_completion(&i2c->msg_complete);
+ if (i2c->dev_comp->apdma_sync &&
+ i2c->op != I2C_MASTER_WRRD && num > 1) {
+ mtk_i2c_writew(i2c, 0x00, OFFSET_DEBUGCTRL);
+ writel(I2C_DMA_HANDSHAKE_RST | I2C_DMA_WARM_RST,
+ i2c->pdmabase + OFFSET_RST);
+
+ ret = readw_poll_timeout(i2c->pdmabase + OFFSET_RST,
+ reg_dma_reset,
+ !(reg_dma_reset & I2C_DMA_WARM_RST),
+ 0, 100);
+ if (ret) {
+ dev_err(i2c->dev, "DMA warm reset timeout\n");
+ return -ETIMEDOUT;
+ }
+
+ writel(I2C_DMA_CLR_FLAG, i2c->pdmabase + OFFSET_RST);
+ mtk_i2c_writew(i2c, I2C_HANDSHAKE_RST, OFFSET_SOFTRESET);
+ mtk_i2c_writew(i2c, I2C_CHN_CLR_FLAG, OFFSET_SOFTRESET);
+ mtk_i2c_writew(i2c, I2C_RELIABILITY | I2C_DMAACK_ENABLE,
+ OFFSET_DEBUGCTRL);
+ }
+
control_reg = mtk_i2c_readw(i2c, OFFSET_CONTROL) &
~(I2C_CONTROL_DIR_CHANGE | I2C_CONTROL_RS);
if ((i2c->speed_hz > I2C_MAX_FAST_MODE_PLUS_FREQ) || (left_num >= 1))
if (ret == 0) {
dev_dbg(i2c->dev, "addr: %x, transfer timeout\n", msgs->addr);
+ i2c_dump_register(i2c);
mtk_i2c_init_hw(i2c);
return -ETIMEDOUT;
}
#include <linux/slab.h>
#include <linux/io.h>
-static struct pci_driver pasemi_smb_driver;
-
-struct pasemi_smbus {
- struct pci_dev *dev;
- struct i2c_adapter adapter;
- unsigned long base;
- int size;
-};
+#include "i2c-pasemi-core.h"
/* Register offsets */
#define REG_MTXFIFO 0x00
#define REG_MRXFIFO 0x04
#define REG_SMSTA 0x14
#define REG_CTL 0x1c
+#define REG_REV 0x28
/* Register defs */
#define MTXFIFO_READ 0x00000400
#define CTL_MRR 0x00000400
#define CTL_MTR 0x00000200
+#define CTL_EN 0x00000800
#define CTL_CLK_M 0x000000ff
-#define CLK_100K_DIV 84
-#define CLK_400K_DIV 21
-
static inline void reg_write(struct pasemi_smbus *smbus, int reg, int val)
{
- dev_dbg(&smbus->dev->dev, "smbus write reg %lx val %08x\n",
- smbus->base + reg, val);
- outl(val, smbus->base + reg);
+ dev_dbg(smbus->dev, "smbus write reg %x val %08x\n", reg, val);
+ iowrite32(val, smbus->ioaddr + reg);
}
static inline int reg_read(struct pasemi_smbus *smbus, int reg)
{
int ret;
- ret = inl(smbus->base + reg);
- dev_dbg(&smbus->dev->dev, "smbus read reg %lx val %08x\n",
- smbus->base + reg, ret);
+ ret = ioread32(smbus->ioaddr + reg);
+ dev_dbg(smbus->dev, "smbus read reg %x val %08x\n", reg, ret);
return ret;
}
#define TXFIFO_WR(smbus, reg) reg_write((smbus), REG_MTXFIFO, (reg))
#define RXFIFO_RD(smbus) reg_read((smbus), REG_MRXFIFO)
+static void pasemi_reset(struct pasemi_smbus *smbus)
+{
+ u32 val = (CTL_MTR | CTL_MRR | (smbus->clk_div & CTL_CLK_M));
+
+ if (smbus->hw_rev >= 6)
+ val |= CTL_EN;
+
+ reg_write(smbus, REG_CTL, val);
+}
+
static void pasemi_smb_clear(struct pasemi_smbus *smbus)
{
unsigned int status;
return -ENXIO;
if (timeout < 0) {
- dev_warn(&smbus->dev->dev, "Timeout, status 0x%08x\n", status);
+ dev_warn(smbus->dev, "Timeout, status 0x%08x\n", status);
reg_write(smbus, REG_SMSTA, status);
return -ETIME;
}
return 0;
reset_out:
- reg_write(smbus, REG_CTL, (CTL_MTR | CTL_MRR |
- (CLK_100K_DIV & CTL_CLK_M)));
+ pasemi_reset(smbus);
return err;
}
return 0;
reset_out:
- reg_write(smbus, REG_CTL, (CTL_MTR | CTL_MRR |
- (CLK_100K_DIV & CTL_CLK_M)));
+ pasemi_reset(smbus);
return err;
}
.functionality = pasemi_smb_func,
};
-static int pasemi_smb_probe(struct pci_dev *dev,
- const struct pci_device_id *id)
+int pasemi_i2c_common_probe(struct pasemi_smbus *smbus)
{
- struct pasemi_smbus *smbus;
int error;
- if (!(pci_resource_flags(dev, 0) & IORESOURCE_IO))
- return -ENODEV;
-
- smbus = kzalloc(sizeof(struct pasemi_smbus), GFP_KERNEL);
- if (!smbus)
- return -ENOMEM;
-
- smbus->dev = dev;
- smbus->base = pci_resource_start(dev, 0);
- smbus->size = pci_resource_len(dev, 0);
-
- if (!request_region(smbus->base, smbus->size,
- pasemi_smb_driver.name)) {
- error = -EBUSY;
- goto out_kfree;
- }
-
smbus->adapter.owner = THIS_MODULE;
snprintf(smbus->adapter.name, sizeof(smbus->adapter.name),
- "PA Semi SMBus adapter at 0x%lx", smbus->base);
+ "PA Semi SMBus adapter (%s)", dev_name(smbus->dev));
smbus->adapter.class = I2C_CLASS_HWMON | I2C_CLASS_SPD;
smbus->adapter.algo = &smbus_algorithm;
smbus->adapter.algo_data = smbus;
/* set up the sysfs linkage to our parent device */
- smbus->adapter.dev.parent = &dev->dev;
+ smbus->adapter.dev.parent = smbus->dev;
- reg_write(smbus, REG_CTL, (CTL_MTR | CTL_MRR |
- (CLK_100K_DIV & CTL_CLK_M)));
+ if (smbus->hw_rev != PASEMI_HW_REV_PCI)
+ smbus->hw_rev = reg_read(smbus, REG_REV);
- error = i2c_add_adapter(&smbus->adapter);
- if (error)
- goto out_release_region;
+ pasemi_reset(smbus);
- pci_set_drvdata(dev, smbus);
+ error = devm_i2c_add_adapter(smbus->dev, &smbus->adapter);
+ if (error)
+ return error;
return 0;
-
- out_release_region:
- release_region(smbus->base, smbus->size);
- out_kfree:
- kfree(smbus);
- return error;
}
-
-static void pasemi_smb_remove(struct pci_dev *dev)
-{
- struct pasemi_smbus *smbus = pci_get_drvdata(dev);
-
- i2c_del_adapter(&smbus->adapter);
- release_region(smbus->base, smbus->size);
- kfree(smbus);
-}
-
-static const struct pci_device_id pasemi_smb_ids[] = {
- { PCI_DEVICE(0x1959, 0xa003) },
- { 0, }
-};
-
-MODULE_DEVICE_TABLE(pci, pasemi_smb_ids);
-
-static struct pci_driver pasemi_smb_driver = {
- .name = "i2c-pasemi",
- .id_table = pasemi_smb_ids,
- .probe = pasemi_smb_probe,
- .remove = pasemi_smb_remove,
-};
-
-module_pci_driver(pasemi_smb_driver);
-
-MODULE_LICENSE("GPL");
-MODULE_AUTHOR ("Olof Johansson <olof@lixom.net>");
-MODULE_DESCRIPTION("PA Semi PWRficient SMBus driver");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/atomic.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/i2c-smbus.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+
+#define PASEMI_HW_REV_PCI -1
+
+struct pasemi_smbus {
+ struct device *dev;
+ struct i2c_adapter adapter;
+ void __iomem *ioaddr;
+ unsigned int clk_div;
+ int hw_rev;
+};
+
+int pasemi_i2c_common_probe(struct pasemi_smbus *smbus);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2006-2007 PA Semi, Inc
+ *
+ * SMBus host driver for PA Semi PWRficient
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/kernel.h>
+#include <linux/stddef.h>
+#include <linux/sched.h>
+#include <linux/i2c.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include <linux/io.h>
+
+#include "i2c-pasemi-core.h"
+
+#define CLK_100K_DIV 84
+#define CLK_400K_DIV 21
+
+static struct pci_driver pasemi_smb_pci_driver;
+
+static int pasemi_smb_pci_probe(struct pci_dev *dev,
+ const struct pci_device_id *id)
+{
+ struct pasemi_smbus *smbus;
+ unsigned long base;
+ int size;
+ int error;
+
+ if (!(pci_resource_flags(dev, 0) & IORESOURCE_IO))
+ return -ENODEV;
+
+ smbus = devm_kzalloc(&dev->dev, sizeof(*smbus), GFP_KERNEL);
+ if (!smbus)
+ return -ENOMEM;
+
+ smbus->dev = &dev->dev;
+ base = pci_resource_start(dev, 0);
+ size = pci_resource_len(dev, 0);
+ smbus->clk_div = CLK_100K_DIV;
+
+ /*
+ * The original PASemi PCI controllers don't have a register for
+ * their HW revision.
+ */
+ smbus->hw_rev = PASEMI_HW_REV_PCI;
+
+ if (!devm_request_region(&dev->dev, base, size,
+ pasemi_smb_pci_driver.name))
+ return -EBUSY;
+
+ smbus->ioaddr = pcim_iomap(dev, 0, 0);
+ if (!smbus->ioaddr)
+ return -EBUSY;
+
+ error = pasemi_i2c_common_probe(smbus);
+ if (error)
+ return error;
+
+ pci_set_drvdata(dev, smbus);
+
+ return 0;
+}
+
+static const struct pci_device_id pasemi_smb_pci_ids[] = {
+ { PCI_DEVICE(0x1959, 0xa003) },
+ { 0, }
+};
+
+MODULE_DEVICE_TABLE(pci, pasemi_smb_pci_ids);
+
+static struct pci_driver pasemi_smb_pci_driver = {
+ .name = "i2c-pasemi",
+ .id_table = pasemi_smb_pci_ids,
+ .probe = pasemi_smb_pci_probe,
+};
+
+module_pci_driver(pasemi_smb_pci_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Olof Johansson <olof@lixom.net>");
+MODULE_DESCRIPTION("PA Semi PWRficient SMBus driver");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021 The Asahi Linux Contributors
+ *
+ * PA Semi PWRficient SMBus host driver for Apple SoCs
+ */
+
+#include <linux/clk.h>
+#include <linux/i2c.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/types.h>
+
+#include "i2c-pasemi-core.h"
+
+struct pasemi_platform_i2c_data {
+ struct pasemi_smbus smbus;
+ struct clk *clk_ref;
+};
+
+static int
+pasemi_platform_i2c_calc_clk_div(struct pasemi_platform_i2c_data *data,
+ u32 frequency)
+{
+ unsigned long clk_rate = clk_get_rate(data->clk_ref);
+
+ if (!clk_rate)
+ return -EINVAL;
+
+ data->smbus.clk_div = DIV_ROUND_UP(clk_rate, 16 * frequency);
+ if (data->smbus.clk_div < 4)
+ return dev_err_probe(data->smbus.dev, -EINVAL,
+ "Bus frequency %d is too fast.\n",
+ frequency);
+ if (data->smbus.clk_div > 0xff)
+ return dev_err_probe(data->smbus.dev, -EINVAL,
+ "Bus frequency %d is too slow.\n",
+ frequency);
+
+ return 0;
+}
+
+static int pasemi_platform_i2c_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct pasemi_platform_i2c_data *data;
+ struct pasemi_smbus *smbus;
+ u32 frequency;
+ int error;
+
+ data = devm_kzalloc(dev, sizeof(struct pasemi_platform_i2c_data),
+ GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ smbus = &data->smbus;
+ smbus->dev = dev;
+
+ smbus->ioaddr = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(smbus->ioaddr))
+ return PTR_ERR(smbus->ioaddr);
+
+ if (of_property_read_u32(dev->of_node, "clock-frequency", &frequency))
+ frequency = I2C_MAX_STANDARD_MODE_FREQ;
+
+ data->clk_ref = devm_clk_get(dev, NULL);
+ if (IS_ERR(data->clk_ref))
+ return PTR_ERR(data->clk_ref);
+
+ error = clk_prepare_enable(data->clk_ref);
+ if (error)
+ return error;
+
+ error = pasemi_platform_i2c_calc_clk_div(data, frequency);
+ if (error)
+ goto out_clk_disable;
+
+ smbus->adapter.dev.of_node = pdev->dev.of_node;
+ error = pasemi_i2c_common_probe(smbus);
+ if (error)
+ goto out_clk_disable;
+
+ platform_set_drvdata(pdev, data);
+
+ return 0;
+
+out_clk_disable:
+ clk_disable_unprepare(data->clk_ref);
+
+ return error;
+}
+
+static int pasemi_platform_i2c_remove(struct platform_device *pdev)
+{
+ struct pasemi_platform_i2c_data *data = platform_get_drvdata(pdev);
+
+ clk_disable_unprepare(data->clk_ref);
+ return 0;
+}
+
+static const struct of_device_id pasemi_platform_i2c_of_match[] = {
+ { .compatible = "apple,t8103-i2c" },
+ { .compatible = "apple,i2c" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, pasemi_platform_i2c_of_match);
+
+static struct platform_driver pasemi_platform_i2c_driver = {
+ .driver = {
+ .name = "i2c-apple",
+ .of_match_table = pasemi_platform_i2c_of_match,
+ },
+ .probe = pasemi_platform_i2c_probe,
+ .remove = pasemi_platform_i2c_remove,
+};
+module_platform_driver(pasemi_platform_i2c_driver);
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Sven Peter <sven@svenpeter.dev>");
+MODULE_DESCRIPTION("Apple/PASemi SMBus platform driver");
}
MODULE_LICENSE("GPL");
-MODULE_ALIAS("platform:pxa2xx-i2c");
subsys_initcall(i2c_adap_pxa_init);
module_exit(i2c_adap_pxa_exit);
* 1. Check if tx_tags_sent is false i.e. the start of QUP block so write the
* tags to TX FIFO and set tx_tags_sent to true.
* 2. Check if send_last_word is true. It will be set when last few data bytes
- * (less than 4 bytes) are reamining to be written in FIFO because of no FIFO
+ * (less than 4 bytes) are remaining to be written in FIFO because of no FIFO
* space. All this data bytes are available in tx_fifo_data so write this
* in FIFO.
* 3. Write the data to TX FIFO and check for cur_blk_len. If it is non zero
goto fail;
ret = devm_request_irq(qup->dev, qup->irq, qup_i2c_interrupt,
- IRQF_TRIGGER_HIGH, "i2c_qup", qup);
+ IRQF_TRIGGER_HIGH | IRQF_NO_AUTOEN,
+ "i2c_qup", qup);
if (ret) {
dev_err(qup->dev, "Request %d IRQ failed\n", qup->irq);
goto fail;
}
- disable_irq(qup->irq);
hw_ver = readl(qup->base + QUP_HW_VERSION);
dev_dbg(qup->dev, "Revision %x\n", hw_ver);
priv->flags |= ID_LAST_MSG;
rcar_i2c_write(priv, ICMAR, i2c_8bit_addr_from_msg(priv->msg));
+ if (!priv->atomic_xfer)
+ rcar_i2c_write(priv, ICMIER, read ? RCAR_IRQ_RECV : RCAR_IRQ_SEND);
+
/*
* We don't have a test case but the HW engineers say that the write order
* of ICMSR and ICMCR depends on whether we issue START or REP_START. Since
rcar_i2c_write(priv, ICMCR, RCAR_BUS_PHASE_START);
rcar_i2c_write(priv, ICMSR, 0);
}
-
- if (!priv->atomic_xfer)
- rcar_i2c_write(priv, ICMIER, read ? RCAR_IRQ_RECV : RCAR_IRQ_SEND);
}
static void rcar_i2c_next_msg(struct rcar_i2c_priv *priv)
else
ret = tegra_i2c_init(i2c_dev);
- pm_runtime_put(i2c_dev->dev);
+ pm_runtime_put_sync(i2c_dev->dev);
return ret;
}
struct tegra_i2c_dev *i2c_dev = platform_get_drvdata(pdev);
i2c_del_adapter(&i2c_dev->adapter);
- pm_runtime_disable(i2c_dev->dev);
+ pm_runtime_force_suspend(i2c_dev->dev);
tegra_i2c_release_dma(i2c_dev);
tegra_i2c_release_clocks(i2c_dev);
pcc_chan = pcc_mbox_request_channel(cl, ctx->mbox_idx);
if (IS_ERR(pcc_chan)) {
dev_err(&pdev->dev, "PCC mailbox channel request failed\n");
- return PTR_ERR(ctx->pcc_chan);
+ return PTR_ERR(pcc_chan);
}
ctx->pcc_chan = pcc_chan;
#include <linux/platform_device.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
-#include <linux/wait.h>
+#include <linux/completion.h>
#include <linux/platform_data/i2c-xiic.h>
#include <linux/io.h>
#include <linux/slab.h>
* struct xiic_i2c - Internal representation of the XIIC I2C bus
* @dev: Pointer to device structure
* @base: Memory base of the HW registers
- * @wait: Wait queue for callers
+ * @completion: Completion for callers
* @adap: Kernel adapter representation
* @tx_msg: Messages from above to be sent
* @lock: Mutual exclusion
struct xiic_i2c {
struct device *dev;
void __iomem *base;
- wait_queue_head_t wait;
+ struct completion completion;
struct i2c_adapter adap;
struct i2c_msg *tx_msg;
struct mutex lock;
#define XIIC_PM_TIMEOUT 1000 /* ms */
/* timeout waiting for the controller to respond */
#define XIIC_I2C_TIMEOUT (msecs_to_jiffies(1000))
+/* timeout waiting for the controller finish transfers */
+#define XIIC_XFER_TIMEOUT (msecs_to_jiffies(10000))
+
/*
* The following constant is used for the device global interrupt enable
* register, to enable all interrupts for the device, this is the only bit
#define xiic_tx_space(i2c) ((i2c)->tx_msg->len - (i2c)->tx_pos)
#define xiic_rx_space(i2c) ((i2c)->rx_msg->len - (i2c)->rx_pos)
-static int xiic_start_xfer(struct xiic_i2c *i2c);
+static int xiic_start_xfer(struct xiic_i2c *i2c, struct i2c_msg *msgs, int num);
static void __xiic_start_xfer(struct xiic_i2c *i2c);
/*
i2c->rx_msg = NULL;
i2c->nmsgs = 0;
i2c->state = code;
- wake_up(&i2c->wait);
+ complete(&i2c->completion);
}
static irqreturn_t xiic_process(int irq, void *dev_id)
struct xiic_i2c *i2c = dev_id;
u32 pend, isr, ier;
u32 clr = 0;
+ int xfer_more = 0;
+ int wakeup_req = 0;
+ int wakeup_code = 0;
/* Get the interrupt Status from the IPIF. There is no clearing of
* interrupts in the IPIF. Interrupts must be cleared at the source.
*/
xiic_reinit(i2c);
- if (i2c->rx_msg)
- xiic_wakeup(i2c, STATE_ERROR);
- if (i2c->tx_msg)
- xiic_wakeup(i2c, STATE_ERROR);
+ if (i2c->rx_msg) {
+ wakeup_req = 1;
+ wakeup_code = STATE_ERROR;
+ }
+ if (i2c->tx_msg) {
+ wakeup_req = 1;
+ wakeup_code = STATE_ERROR;
+ }
}
if (pend & XIIC_INTR_RX_FULL_MASK) {
/* Receive register/FIFO is full */
i2c->tx_msg++;
dev_dbg(i2c->adap.dev.parent,
"%s will start next...\n", __func__);
-
- __xiic_start_xfer(i2c);
+ xfer_more = 1;
}
}
}
if (!i2c->tx_msg)
goto out;
- if ((i2c->nmsgs == 1) && !i2c->rx_msg &&
- xiic_tx_space(i2c) == 0)
- xiic_wakeup(i2c, STATE_DONE);
+ wakeup_req = 1;
+
+ if (i2c->nmsgs == 1 && !i2c->rx_msg &&
+ xiic_tx_space(i2c) == 0)
+ wakeup_code = STATE_DONE;
else
- xiic_wakeup(i2c, STATE_ERROR);
+ wakeup_code = STATE_ERROR;
}
if (pend & (XIIC_INTR_TX_EMPTY_MASK | XIIC_INTR_TX_HALF_MASK)) {
/* Transmit register/FIFO is empty or ½ empty */
if (i2c->nmsgs > 1) {
i2c->nmsgs--;
i2c->tx_msg++;
- __xiic_start_xfer(i2c);
+ xfer_more = 1;
} else {
xiic_irq_dis(i2c, XIIC_INTR_TX_HALF_MASK);
dev_dbg(i2c->adap.dev.parent, "%s clr: 0x%x\n", __func__, clr);
xiic_setreg32(i2c, XIIC_IISR_OFFSET, clr);
+ if (xfer_more)
+ __xiic_start_xfer(i2c);
+ if (wakeup_req)
+ xiic_wakeup(i2c, wakeup_code);
+
+ WARN_ON(xfer_more && wakeup_req);
+
mutex_unlock(&i2c->lock);
return IRQ_HANDLED;
}
int tries = 3;
int err;
- if (i2c->tx_msg)
+ if (i2c->tx_msg || i2c->rx_msg)
return -EBUSY;
/* In single master mode bus can only be busy, when in use by this
{
u8 rx_watermark;
struct i2c_msg *msg = i2c->rx_msg = i2c->tx_msg;
- unsigned long flags;
/* Clear and enable Rx full interrupt. */
xiic_irq_clr_en(i2c, XIIC_INTR_RX_FULL_MASK | XIIC_INTR_TX_ERROR_MASK);
rx_watermark = IIC_RX_FIFO_DEPTH;
xiic_setreg8(i2c, XIIC_RFD_REG_OFFSET, rx_watermark - 1);
- local_irq_save(flags);
if (!(msg->flags & I2C_M_NOSTART))
/* write the address */
xiic_setreg16(i2c, XIIC_DTR_REG_OFFSET,
xiic_setreg16(i2c, XIIC_DTR_REG_OFFSET,
msg->len | ((i2c->nmsgs == 1) ? XIIC_TX_DYN_STOP_MASK : 0));
- local_irq_restore(flags);
if (i2c->nmsgs == 1)
/* very last, enable bus not busy as well */
{
struct i2c_msg *msg = i2c->tx_msg;
- xiic_irq_clr(i2c, XIIC_INTR_TX_ERROR_MASK);
-
dev_dbg(i2c->adap.dev.parent, "%s entry, msg: %p, len: %d",
__func__, msg, msg->len);
dev_dbg(i2c->adap.dev.parent, "%s entry, ISR: 0x%x, CR: 0x%x\n",
xiic_setreg16(i2c, XIIC_DTR_REG_OFFSET, data);
}
- xiic_fill_tx_fifo(i2c);
-
/* Clear any pending Tx empty, Tx Error and then enable them. */
xiic_irq_clr_en(i2c, XIIC_INTR_TX_EMPTY_MASK | XIIC_INTR_TX_ERROR_MASK |
- XIIC_INTR_BNB_MASK);
-}
-
-static irqreturn_t xiic_isr(int irq, void *dev_id)
-{
- struct xiic_i2c *i2c = dev_id;
- u32 pend, isr, ier;
- irqreturn_t ret = IRQ_NONE;
- /* Do not processes a devices interrupts if the device has no
- * interrupts pending
- */
-
- dev_dbg(i2c->adap.dev.parent, "%s entry\n", __func__);
-
- isr = xiic_getreg32(i2c, XIIC_IISR_OFFSET);
- ier = xiic_getreg32(i2c, XIIC_IIER_OFFSET);
- pend = isr & ier;
- if (pend)
- ret = IRQ_WAKE_THREAD;
+ XIIC_INTR_BNB_MASK |
+ ((i2c->nmsgs > 1 || xiic_tx_space(i2c)) ?
+ XIIC_INTR_TX_HALF_MASK : 0));
- return ret;
+ xiic_fill_tx_fifo(i2c);
}
static void __xiic_start_xfer(struct xiic_i2c *i2c)
{
- int first = 1;
int fifo_space = xiic_tx_fifo_space(i2c);
dev_dbg(i2c->adap.dev.parent, "%s entry, msg: %p, fifos space: %d\n",
__func__, i2c->tx_msg, fifo_space);
i2c->rx_pos = 0;
i2c->tx_pos = 0;
i2c->state = STATE_START;
- while ((fifo_space >= 2) && (first || (i2c->nmsgs > 1))) {
- if (!first) {
- i2c->nmsgs--;
- i2c->tx_msg++;
- i2c->tx_pos = 0;
- } else
- first = 0;
-
- if (i2c->tx_msg->flags & I2C_M_RD) {
- /* we dont date putting several reads in the FIFO */
- xiic_start_recv(i2c);
- return;
- } else {
- xiic_start_send(i2c);
- if (xiic_tx_space(i2c) != 0) {
- /* the message could not be completely sent */
- break;
- }
- }
-
- fifo_space = xiic_tx_fifo_space(i2c);
+ if (i2c->tx_msg->flags & I2C_M_RD) {
+ /* we dont date putting several reads in the FIFO */
+ xiic_start_recv(i2c);
+ } else {
+ xiic_start_send(i2c);
}
-
- /* there are more messages or the current one could not be completely
- * put into the FIFO, also enable the half empty interrupt
- */
- if (i2c->nmsgs > 1 || xiic_tx_space(i2c))
- xiic_irq_clr_en(i2c, XIIC_INTR_TX_HALF_MASK);
-
}
-static int xiic_start_xfer(struct xiic_i2c *i2c)
+static int xiic_start_xfer(struct xiic_i2c *i2c, struct i2c_msg *msgs, int num)
{
int ret;
+
mutex_lock(&i2c->lock);
+ ret = xiic_busy(i2c);
+ if (ret)
+ goto out;
+
+ i2c->tx_msg = msgs;
+ i2c->rx_msg = NULL;
+ i2c->nmsgs = num;
+ init_completion(&i2c->completion);
+
ret = xiic_reinit(i2c);
if (!ret)
__xiic_start_xfer(i2c);
+out:
mutex_unlock(&i2c->lock);
return ret;
if (err < 0)
return err;
- err = xiic_busy(i2c);
- if (err)
- goto out;
-
- i2c->tx_msg = msgs;
- i2c->nmsgs = num;
-
- err = xiic_start_xfer(i2c);
+ err = xiic_start_xfer(i2c, msgs, num);
if (err < 0) {
dev_err(adap->dev.parent, "Error xiic_start_xfer\n");
- goto out;
+ return err;
}
- if (wait_event_timeout(i2c->wait, (i2c->state == STATE_ERROR) ||
- (i2c->state == STATE_DONE), HZ)) {
- err = (i2c->state == STATE_DONE) ? num : -EIO;
- goto out;
- } else {
+ err = wait_for_completion_timeout(&i2c->completion, XIIC_XFER_TIMEOUT);
+ mutex_lock(&i2c->lock);
+ if (err == 0) { /* Timeout */
i2c->tx_msg = NULL;
i2c->rx_msg = NULL;
i2c->nmsgs = 0;
err = -ETIMEDOUT;
- goto out;
+ } else if (err < 0) { /* Completion error */
+ i2c->tx_msg = NULL;
+ i2c->rx_msg = NULL;
+ i2c->nmsgs = 0;
+ } else {
+ err = (i2c->state == STATE_DONE) ? num : -EIO;
}
-out:
+ mutex_unlock(&i2c->lock);
pm_runtime_mark_last_busy(i2c->dev);
pm_runtime_put_autosuspend(i2c->dev);
return err;
i2c->adap.dev.of_node = pdev->dev.of_node;
mutex_init(&i2c->lock);
- init_waitqueue_head(&i2c->wait);
i2c->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(i2c->clk))
pm_runtime_use_autosuspend(i2c->dev);
pm_runtime_set_active(i2c->dev);
pm_runtime_enable(i2c->dev);
- ret = devm_request_threaded_irq(&pdev->dev, irq, xiic_isr,
+ ret = devm_request_threaded_irq(&pdev->dev, irq, NULL,
xiic_process, IRQF_ONESHOT,
pdev->name, i2c);
i2c_set_adapdata(&priv->adap, priv);
ret = i2c_add_numbered_adapter(&priv->adap);
if (ret < 0)
- return ret;
+ goto err_unprepare_clk;
platform_set_drvdata(pdev, priv);
dev_info(&priv->adap.dev, "Added I2C Bus.\n");
return 0;
+
+err_unprepare_clk:
+ clk_unprepare(clk);
+ return ret;
}
static int xlr_i2c_remove(struct platform_device *pdev)
}
EXPORT_SYMBOL_GPL(i2c_acpi_new_device);
+bool i2c_acpi_waive_d0_probe(struct device *dev)
+{
+ struct i2c_driver *driver = to_i2c_driver(dev->driver);
+ struct acpi_device *adev = ACPI_COMPANION(dev);
+
+ return driver->flags & I2C_DRV_ACPI_WAIVE_D0_PROBE &&
+ adev && adev->power.state_for_enumeration >= adev->power.state;
+}
+EXPORT_SYMBOL_GPL(i2c_acpi_waive_d0_probe);
+
#ifdef CONFIG_ACPI_I2C_OPREGION
static int acpi_gsb_i2c_read_bytes(struct i2c_client *client,
u8 cmd, u8 *data, u8 data_len)
if (status < 0)
goto err_clear_wakeup_irq;
- status = dev_pm_domain_attach(&client->dev, true);
+ status = dev_pm_domain_attach(&client->dev,
+ !i2c_acpi_waive_d0_probe(dev));
if (status)
goto err_clear_wakeup_irq;
err_release_driver_resources:
devres_release_group(&client->dev, client->devres_group_id);
err_detach_pm_domain:
- dev_pm_domain_detach(&client->dev, true);
+ dev_pm_domain_detach(&client->dev, !i2c_acpi_waive_d0_probe(dev));
err_clear_wakeup_irq:
dev_pm_clear_wake_irq(&client->dev);
device_init_wakeup(&client->dev, false);
devres_release_group(&client->dev, client->devres_group_id);
- dev_pm_domain_detach(&client->dev, true);
+ dev_pm_domain_detach(&client->dev, !i2c_acpi_waive_d0_probe(dev));
if (!pm_runtime_status_suspended(&client->dev) && adap->bus_regulator)
regulator_disable(adap->bus_regulator);
+// SPDX-License-Identifier: GPL-2.0-only
/*
* TI ADC MFD driver
*
* Copyright (C) 2012 Texas Instruments Incorporated - https://www.ti.com/
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed "as is" WITHOUT ANY WARRANTY of any
- * kind, whether express or implied; without even the implied warranty
- * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/kernel.h>
#include <linux/of_device.h>
#include <linux/iio/machine.h>
#include <linux/iio/driver.h>
+#include <linux/iopoll.h>
#include <linux/mfd/ti_am335x_tscadc.h>
#include <linux/iio/buffer.h>
}
static void tiadc_writel(struct tiadc_device *adc, unsigned int reg,
- unsigned int val)
+ unsigned int val)
{
writel(val, adc->mfd_tscadc->tscadc_base + reg);
}
}
static u32 get_adc_chan_step_mask(struct tiadc_device *adc_dev,
- struct iio_chan_spec const *chan)
+ struct iio_chan_spec const *chan)
{
int i;
return 1 << adc_dev->channel_step[chan];
}
+static int tiadc_wait_idle(struct tiadc_device *adc_dev)
+{
+ u32 val;
+
+ return readl_poll_timeout(adc_dev->mfd_tscadc->tscadc_base + REG_ADCFSM,
+ val, !(val & SEQ_STATUS), 10,
+ IDLE_TIMEOUT_MS * 1000 * adc_dev->channels);
+}
+
static void tiadc_step_config(struct iio_dev *indio_dev)
{
struct tiadc_device *adc_dev = iio_priv(indio_dev);
- struct device *dev = adc_dev->mfd_tscadc->dev;
unsigned int stepconfig;
int i, steps = 0;
* Channel would represent which analog input
* needs to be given to ADC to digitalize data.
*/
-
-
for (i = 0; i < adc_dev->channels; i++) {
int chan;
chan = adc_dev->channel_line[i];
- if (adc_dev->step_avg[i] > STEPCONFIG_AVG_16) {
- dev_warn(dev, "chan %d step_avg truncating to %d\n",
- chan, STEPCONFIG_AVG_16);
- adc_dev->step_avg[i] = STEPCONFIG_AVG_16;
- }
-
if (adc_dev->step_avg[i])
- stepconfig =
- STEPCONFIG_AVG(ffs(adc_dev->step_avg[i]) - 1) |
- STEPCONFIG_FIFO1;
+ stepconfig = STEPCONFIG_AVG(ffs(adc_dev->step_avg[i]) - 1) |
+ STEPCONFIG_FIFO1;
else
stepconfig = STEPCONFIG_FIFO1;
stepconfig |= STEPCONFIG_MODE_SWCNT;
tiadc_writel(adc_dev, REG_STEPCONFIG(steps),
- stepconfig | STEPCONFIG_INP(chan) |
- STEPCONFIG_INM_ADCREFM |
- STEPCONFIG_RFP_VREFP |
- STEPCONFIG_RFM_VREFN);
-
- if (adc_dev->open_delay[i] > STEPDELAY_OPEN_MASK) {
- dev_warn(dev, "chan %d open delay truncating to 0x3FFFF\n",
- chan);
- adc_dev->open_delay[i] = STEPDELAY_OPEN_MASK;
- }
-
- if (adc_dev->sample_delay[i] > 0xFF) {
- dev_warn(dev, "chan %d sample delay truncating to 0xFF\n",
- chan);
- adc_dev->sample_delay[i] = 0xFF;
- }
+ stepconfig | STEPCONFIG_INP(chan) |
+ STEPCONFIG_INM_ADCREFM | STEPCONFIG_RFP_VREFP |
+ STEPCONFIG_RFM_VREFN);
tiadc_writel(adc_dev, REG_STEPDELAY(steps),
- STEPDELAY_OPEN(adc_dev->open_delay[i]) |
- STEPDELAY_SAMPLE(adc_dev->sample_delay[i]));
+ STEPDELAY_OPEN(adc_dev->open_delay[i]) |
+ STEPDELAY_SAMPLE(adc_dev->sample_delay[i]));
adc_dev->channel_step[i] = steps;
steps++;
if (status & IRQENB_FIFO1OVRRUN) {
/* FIFO Overrun. Clear flag. Disable/Enable ADC to recover */
config = tiadc_readl(adc_dev, REG_CTRL);
- config &= ~(CNTRLREG_TSCSSENB);
+ config &= ~(CNTRLREG_SSENB);
tiadc_writel(adc_dev, REG_CTRL, config);
- tiadc_writel(adc_dev, REG_IRQSTATUS, IRQENB_FIFO1OVRRUN
- | IRQENB_FIFO1UNDRFLW | IRQENB_FIFO1THRES);
+ tiadc_writel(adc_dev, REG_IRQSTATUS,
+ IRQENB_FIFO1OVRRUN | IRQENB_FIFO1UNDRFLW |
+ IRQENB_FIFO1THRES);
- /* wait for idle state.
+ /*
+ * Wait for the idle state.
* ADC needs to finish the current conversion
* before disabling the module
*/
adc_fsm = tiadc_readl(adc_dev, REG_ADCFSM);
} while (adc_fsm != 0x10 && count++ < 100);
- tiadc_writel(adc_dev, REG_CTRL, (config | CNTRLREG_TSCSSENB));
+ tiadc_writel(adc_dev, REG_CTRL, (config | CNTRLREG_SSENB));
return IRQ_HANDLED;
} else if (status & IRQENB_FIFO1THRES) {
/* Disable irq and wake worker thread */
fifo1count = tiadc_readl(adc_dev, REG_FIFO1CNT);
for (k = 0; k < fifo1count; k = k + i) {
- for (i = 0; i < (indio_dev->scan_bytes)/2; i++) {
+ for (i = 0; i < indio_dev->scan_bytes / 2; i++) {
read = tiadc_readl(adc_dev, REG_FIFO1);
data[i] = read & FIFOREAD_DATA_MASK;
}
- iio_push_to_buffers(indio_dev, (u8 *) data);
+ iio_push_to_buffers(indio_dev, (u8 *)data);
}
tiadc_writel(adc_dev, REG_IRQSTATUS, IRQENB_FIFO1THRES);
struct dma_async_tx_descriptor *desc;
dma->current_period = 0; /* We start to fill period 0 */
+
/*
* Make the fifo thresh as the multiple of total number of
* channels enabled, so make sure that cyclic DMA period
*/
dma->fifo_thresh = rounddown(FIFO1_THRESHOLD + 1,
adc_dev->total_ch_enabled) - 1;
+
/* Make sure that period length is multiple of fifo thresh level */
dma->period_size = rounddown(DMA_BUFFER_SIZE / 2,
- (dma->fifo_thresh + 1) * sizeof(u16));
+ (dma->fifo_thresh + 1) * sizeof(u16));
dma->conf.src_maxburst = dma->fifo_thresh + 1;
dmaengine_slave_config(dma->chan, &dma->conf);
{
struct tiadc_device *adc_dev = iio_priv(indio_dev);
int i, fifo1count;
+ int ret;
+
+ ret = tiadc_wait_idle(adc_dev);
+ if (ret)
+ return ret;
- tiadc_writel(adc_dev, REG_IRQCLR, (IRQENB_FIFO1THRES |
- IRQENB_FIFO1OVRRUN |
- IRQENB_FIFO1UNDRFLW));
+ tiadc_writel(adc_dev, REG_IRQCLR,
+ IRQENB_FIFO1THRES | IRQENB_FIFO1OVRRUN |
+ IRQENB_FIFO1UNDRFLW);
/* Flush FIFO. Needed in corner cases in simultaneous tsc/adc use */
fifo1count = tiadc_readl(adc_dev, REG_FIFO1CNT);
am335x_tsc_se_set_cache(adc_dev->mfd_tscadc, enb);
- tiadc_writel(adc_dev, REG_IRQSTATUS, IRQENB_FIFO1THRES
- | IRQENB_FIFO1OVRRUN | IRQENB_FIFO1UNDRFLW);
+ tiadc_writel(adc_dev, REG_IRQSTATUS,
+ IRQENB_FIFO1THRES | IRQENB_FIFO1OVRRUN |
+ IRQENB_FIFO1UNDRFLW);
irq_enable = IRQENB_FIFO1OVRRUN;
if (!dma->chan)
struct tiadc_dma *dma = &adc_dev->dma;
int fifo1count, i;
- tiadc_writel(adc_dev, REG_IRQCLR, (IRQENB_FIFO1THRES |
- IRQENB_FIFO1OVRRUN | IRQENB_FIFO1UNDRFLW));
+ tiadc_writel(adc_dev, REG_IRQCLR,
+ IRQENB_FIFO1THRES | IRQENB_FIFO1OVRRUN |
+ IRQENB_FIFO1UNDRFLW);
am335x_tsc_se_clr(adc_dev->mfd_tscadc, adc_dev->buffer_en_ch_steps);
adc_dev->buffer_en_ch_steps = 0;
adc_dev->total_ch_enabled = 0;
};
static int tiadc_iio_buffered_hardware_setup(struct device *dev,
- struct iio_dev *indio_dev,
- irqreturn_t (*pollfunc_bh)(int irq, void *p),
- irqreturn_t (*pollfunc_th)(int irq, void *p),
- int irq,
- unsigned long flags,
- const struct iio_buffer_setup_ops *setup_ops)
+ struct iio_dev *indio_dev,
+ irqreturn_t (*pollfunc_bh)(int irq, void *p),
+ irqreturn_t (*pollfunc_th)(int irq, void *p),
+ int irq, unsigned long flags,
+ const struct iio_buffer_setup_ops *setup_ops)
{
int ret;
return ret;
return devm_request_threaded_irq(dev, irq, pollfunc_th, pollfunc_bh,
- flags, indio_dev->name, indio_dev);
+ flags, indio_dev->name, indio_dev);
}
static const char * const chan_name_ain[] = {
indio_dev->num_channels = channels;
chan_array = devm_kcalloc(dev, channels, sizeof(*chan_array),
GFP_KERNEL);
- if (chan_array == NULL)
+ if (!chan_array)
return -ENOMEM;
chan = chan_array;
for (i = 0; i < channels; i++, chan++) {
-
chan->type = IIO_VOLTAGE;
chan->indexed = 1;
chan->channel = adc_dev->channel_line[i];
chan->info_mask_separate = BIT(IIO_CHAN_INFO_RAW);
+ chan->info_mask_shared_by_type = BIT(IIO_CHAN_INFO_SCALE);
chan->datasheet_name = chan_name_ain[chan->channel];
chan->scan_index = i;
chan->scan_type.sign = 'u';
}
static int tiadc_read_raw(struct iio_dev *indio_dev,
- struct iio_chan_spec const *chan,
- int *val, int *val2, long mask)
+ struct iio_chan_spec const *chan, int *val, int *val2,
+ long mask)
{
struct tiadc_device *adc_dev = iio_priv(indio_dev);
- int ret = IIO_VAL_INT;
int i, map_val;
unsigned int fifo1count, read, stepid;
bool found = false;
u32 step_en;
unsigned long timeout;
+ int ret;
+
+ switch (mask) {
+ case IIO_CHAN_INFO_RAW:
+ break;
+ case IIO_CHAN_INFO_SCALE:
+ switch (chan->type) {
+ case IIO_VOLTAGE:
+ *val = 1800;
+ *val2 = chan->scan_type.realbits;
+ return IIO_VAL_FRACTIONAL_LOG2;
+ default:
+ return -EINVAL;
+ }
+ break;
+ default:
+ return -EINVAL;
+ }
if (iio_buffer_enabled(indio_dev))
return -EBUSY;
return -EINVAL;
mutex_lock(&adc_dev->fifo1_lock);
+
+ ret = tiadc_wait_idle(adc_dev);
+ if (ret)
+ goto err_unlock;
+
fifo1count = tiadc_readl(adc_dev, REG_FIFO1CNT);
while (fifo1count--)
tiadc_readl(adc_dev, REG_FIFO1);
am335x_tsc_se_set_once(adc_dev->mfd_tscadc, step_en);
- timeout = jiffies + msecs_to_jiffies
- (IDLE_TIMEOUT * adc_dev->channels);
/* Wait for Fifo threshold interrupt */
+ timeout = jiffies + msecs_to_jiffies(IDLE_TIMEOUT_MS * adc_dev->channels);
while (1) {
fifo1count = tiadc_readl(adc_dev, REG_FIFO1CNT);
if (fifo1count)
goto err_unlock;
}
}
+
map_val = adc_dev->channel_step[chan->scan_index];
/*
if (stepid == map_val) {
read = read & FIFOREAD_DATA_MASK;
found = true;
- *val = (u16) read;
+ *val = (u16)read;
}
}
+
am335x_tsc_se_adc_done(adc_dev->mfd_tscadc);
if (!found)
- ret = -EBUSY;
+ ret = -EBUSY;
err_unlock:
mutex_unlock(&adc_dev->fifo1_lock);
- return ret;
+ return ret ? ret : IIO_VAL_INT;
}
static const struct iio_info tiadc_info = {
goto err;
return 0;
+
err:
dma_release_channel(dma->chan);
return -ENOMEM;
const __be32 *cur;
int channels = 0;
u32 val;
+ int i;
of_property_for_each_u32(node, "ti,adc-channels", prop, cur, val) {
adc_dev->channel_line[channels] = val;
channels++;
}
+ adc_dev->channels = channels;
+
of_property_read_u32_array(node, "ti,chan-step-avg",
adc_dev->step_avg, channels);
of_property_read_u32_array(node, "ti,chan-step-opendelay",
of_property_read_u32_array(node, "ti,chan-step-sampledelay",
adc_dev->sample_delay, channels);
- adc_dev->channels = channels;
+ for (i = 0; i < adc_dev->channels; i++) {
+ int chan;
+
+ chan = adc_dev->channel_line[i];
+
+ if (adc_dev->step_avg[i] > STEPCONFIG_AVG_16) {
+ dev_warn(&pdev->dev,
+ "chan %d: wrong step avg, truncated to %ld\n",
+ chan, STEPCONFIG_AVG_16);
+ adc_dev->step_avg[i] = STEPCONFIG_AVG_16;
+ }
+
+ if (adc_dev->open_delay[i] > STEPCONFIG_MAX_OPENDLY) {
+ dev_warn(&pdev->dev,
+ "chan %d: wrong open delay, truncated to 0x%lX\n",
+ chan, STEPCONFIG_MAX_OPENDLY);
+ adc_dev->open_delay[i] = STEPCONFIG_MAX_OPENDLY;
+ }
+
+ if (adc_dev->sample_delay[i] > STEPCONFIG_MAX_SAMPLE) {
+ dev_warn(&pdev->dev,
+ "chan %d: wrong sample delay, truncated to 0x%lX\n",
+ chan, STEPCONFIG_MAX_SAMPLE);
+ adc_dev->sample_delay[i] = STEPCONFIG_MAX_SAMPLE;
+ }
+ }
+
return 0;
}
}
indio_dev = devm_iio_device_alloc(&pdev->dev, sizeof(*adc_dev));
- if (indio_dev == NULL) {
+ if (!indio_dev) {
dev_err(&pdev->dev, "failed to allocate iio device\n");
return -ENOMEM;
}
return err;
err = tiadc_iio_buffered_hardware_setup(&pdev->dev, indio_dev,
- &tiadc_worker_h,
- &tiadc_irq_h,
- adc_dev->mfd_tscadc->irq,
- IRQF_SHARED,
- &tiadc_buffer_setup_ops);
-
+ &tiadc_worker_h,
+ &tiadc_irq_h,
+ adc_dev->mfd_tscadc->irq,
+ IRQF_SHARED,
+ &tiadc_buffer_setup_ops);
if (err)
- goto err_free_channels;
+ return err;
err = iio_device_register(indio_dev);
if (err)
- goto err_buffer_unregister;
+ return err;
platform_set_drvdata(pdev, indio_dev);
err_dma:
iio_device_unregister(indio_dev);
-err_buffer_unregister:
-err_free_channels:
+
return err;
}
unsigned int idle;
idle = tiadc_readl(adc_dev, REG_CTRL);
- idle &= ~(CNTRLREG_TSCSSENB);
- tiadc_writel(adc_dev, REG_CTRL, (idle |
- CNTRLREG_POWERDOWN));
+ idle &= ~(CNTRLREG_SSENB);
+ tiadc_writel(adc_dev, REG_CTRL, idle | CNTRLREG_POWERDOWN);
return 0;
}
/* Make sure ADC is powered up */
restore = tiadc_readl(adc_dev, REG_CTRL);
- restore &= ~(CNTRLREG_POWERDOWN);
+ restore &= ~CNTRLREG_POWERDOWN;
tiadc_writel(adc_dev, REG_CTRL, restore);
tiadc_step_config(indio_dev);
am335x_tsc_se_set_cache(adc_dev->mfd_tscadc,
- adc_dev->buffer_en_ch_steps);
+ adc_dev->buffer_en_ch_steps);
return 0;
}
static const struct of_device_id ti_adc_dt_ids[] = {
{ .compatible = "ti,am3359-adc", },
+ { .compatible = "ti,am4372-adc", },
{ }
};
MODULE_DEVICE_TABLE(of, ti_adc_dt_ids);
static int __maybe_unused cros_ec_sensors_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct iio_dev *indio_dev = platform_get_drvdata(pdev);
+ struct iio_dev *indio_dev = dev_get_drvdata(dev);
struct cros_ec_sensors_core_state *st = iio_priv(indio_dev);
int ret = 0;
int ret;
/* Currently only counter for QP is supported */
- if (nla_get_u32(tb[RDMA_NLDEV_ATTR_STAT_RES]) != RDMA_NLDEV_ATTR_RES_QP)
+ if (!tb[RDMA_NLDEV_ATTR_STAT_RES] ||
+ nla_get_u32(tb[RDMA_NLDEV_ATTR_STAT_RES]) != RDMA_NLDEV_ATTR_RES_QP)
return -EINVAL;
mode = nla_get_u32(tb[RDMA_NLDEV_ATTR_STAT_MODE]);
INIT_LIST_HEAD(&qp->rdma_mrs);
INIT_LIST_HEAD(&qp->sig_mrs);
+ qp->send_cq = attr->send_cq;
+ qp->recv_cq = attr->recv_cq;
+
rdma_restrack_new(&qp->res, RDMA_RESTRACK_QP);
WARN_ONCE(!udata && !caller, "Missing kernel QP owner");
rdma_restrack_set_name(&qp->res, udata ? NULL : caller);
n++;
names_out =
- kmalloc((n + num_extra_names) * sizeof(struct rdma_stat_desc) +
- names_len,
+ kzalloc((n + num_extra_names) * sizeof(*q) + names_len,
GFP_KERNEL);
if (!names_out) {
*num_cntrs = 0;
return -ENOMEM;
}
- p = names_out + (n + num_extra_names) * sizeof(struct rdma_stat_desc);
+ p = names_out + (n + num_extra_names) * sizeof(*q);
memcpy(p, names_in, names_len);
q = (struct rdma_stat_desc *)names_out;
.get_hw_stats = mlx4_ib_get_hw_stats,
};
+static const struct ib_device_ops mlx4_ib_hw_stats_ops1 = {
+ .alloc_hw_device_stats = mlx4_ib_alloc_hw_device_stats,
+ .get_hw_stats = mlx4_ib_get_hw_stats,
+};
+
static int mlx4_ib_alloc_diag_counters(struct mlx4_ib_dev *ibdev)
{
struct mlx4_ib_diag_counters *diag = ibdev->diag_counters;
return 0;
for (i = 0; i < MLX4_DIAG_COUNTERS_TYPES; i++) {
- /* i == 1 means we are building port counters */
- if (i && !per_port)
- continue;
+ /*
+ * i == 1 means we are building port counters, set a different
+ * stats ops without port stats callback.
+ */
+ if (i && !per_port) {
+ ib_set_device_ops(&ibdev->ib_dev,
+ &mlx4_ib_hw_stats_ops1);
+
+ return 0;
+ }
ret = __mlx4_ib_alloc_diag_counters(ibdev, &diag[i].descs,
&diag[i].offset,
*/
static void srp_del_scsi_host_attr(struct Scsi_Host *shost)
{
- struct device_attribute **attr;
+ const struct attribute_group **g;
+ struct attribute **attr;
- for (attr = shost->hostt->shost_attrs; attr && *attr; ++attr)
- device_remove_file(&shost->shost_dev, *attr);
+ for (g = shost->hostt->shost_groups; *g; ++g) {
+ for (attr = (*g)->attrs; *attr; ++attr) {
+ struct device_attribute *dev_attr =
+ container_of(*attr, typeof(*dev_attr), attr);
+
+ device_remove_file(&shost->shost_dev, dev_attr);
+ }
+ }
}
static void srp_remove_target(struct srp_target_port *target)
if (scmnd) {
srp_free_req(ch, req, scmnd, 0);
scmnd->result = result;
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
}
}
srp_free_req(ch, req, scmnd,
be32_to_cpu(rsp->req_lim_delta));
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
}
}
err:
if (scmnd->result) {
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
ret = 0;
} else {
ret = SCSI_MLQUEUE_HOST_BUSY;
if (ret == SUCCESS) {
srp_free_req(ch, req, scmnd, 0);
scmnd->result = DID_ABORT << 16;
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
}
return ret;
static DEVICE_ATTR_RO(allow_ext_sg);
-static struct device_attribute *srp_host_attrs[] = {
- &dev_attr_id_ext,
- &dev_attr_ioc_guid,
- &dev_attr_service_id,
- &dev_attr_pkey,
- &dev_attr_sgid,
- &dev_attr_dgid,
- &dev_attr_orig_dgid,
- &dev_attr_req_lim,
- &dev_attr_zero_req_lim,
- &dev_attr_local_ib_port,
- &dev_attr_local_ib_device,
- &dev_attr_ch_count,
- &dev_attr_comp_vector,
- &dev_attr_tl_retry_count,
- &dev_attr_cmd_sg_entries,
- &dev_attr_allow_ext_sg,
+static struct attribute *srp_host_attrs[] = {
+ &dev_attr_id_ext.attr,
+ &dev_attr_ioc_guid.attr,
+ &dev_attr_service_id.attr,
+ &dev_attr_pkey.attr,
+ &dev_attr_sgid.attr,
+ &dev_attr_dgid.attr,
+ &dev_attr_orig_dgid.attr,
+ &dev_attr_req_lim.attr,
+ &dev_attr_zero_req_lim.attr,
+ &dev_attr_local_ib_port.attr,
+ &dev_attr_local_ib_device.attr,
+ &dev_attr_ch_count.attr,
+ &dev_attr_comp_vector.attr,
+ &dev_attr_tl_retry_count.attr,
+ &dev_attr_cmd_sg_entries.attr,
+ &dev_attr_allow_ext_sg.attr,
NULL
};
+ATTRIBUTE_GROUPS(srp_host);
+
static struct scsi_host_template srp_template = {
.module = THIS_MODULE,
.name = "InfiniBand SRP initiator",
.can_queue = SRP_DEFAULT_CMD_SQ_SIZE,
.this_id = -1,
.cmd_per_lun = SRP_DEFAULT_CMD_SQ_SIZE,
- .shost_attrs = srp_host_attrs,
+ .shost_groups = srp_host_groups,
.track_queue_depth = 1,
.cmd_size = sizeof(struct srp_request),
};
NULL,
};
-static ssize_t srpt_tpg_enable_show(struct config_item *item, char *page)
+static int srpt_enable_tpg(struct se_portal_group *se_tpg, bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct srpt_port *sport = srpt_tpg_to_sport(se_tpg);
- return sysfs_emit(page, "%d\n", sport->enabled);
-}
-
-static ssize_t srpt_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct srpt_port *sport = srpt_tpg_to_sport(se_tpg);
- unsigned long tmp;
- int ret;
-
- ret = kstrtoul(page, 0, &tmp);
- if (ret < 0) {
- pr_err("Unable to extract srpt_tpg_store_enable\n");
- return -EINVAL;
- }
-
- if ((tmp != 0) && (tmp != 1)) {
- pr_err("Illegal value for srpt_tpg_store_enable: %lu\n", tmp);
- return -EINVAL;
- }
-
mutex_lock(&sport->mutex);
- srpt_set_enabled(sport, tmp);
+ srpt_set_enabled(sport, enable);
mutex_unlock(&sport->mutex);
- return count;
+ return 0;
}
-CONFIGFS_ATTR(srpt_tpg_, enable);
-
-static struct configfs_attribute *srpt_tpg_attrs[] = {
- &srpt_tpg_attr_enable,
- NULL,
-};
-
/**
* srpt_make_tpg - configfs callback invoked for mkdir /sys/kernel/config/target/$driver/$port/$tpg
* @wwn: Corresponds to $driver/$port.
.fabric_make_wwn = srpt_make_tport,
.fabric_drop_wwn = srpt_drop_tport,
.fabric_make_tpg = srpt_make_tpg,
+ .fabric_enable_tpg = srpt_enable_tpg,
.fabric_drop_tpg = srpt_drop_tpg,
.fabric_init_nodeacl = srpt_init_nodeacl,
.tfc_discovery_attrs = srpt_da_attrs,
.tfc_wwn_attrs = srpt_wwn_attrs,
- .tfc_tpg_base_attrs = srpt_tpg_attrs,
.tfc_tpg_attrib_attrs = srpt_tpg_attrib_attrs,
};
#include <linux/input.h>
#include <linux/gameport.h>
#include <linux/jiffies.h>
+#include <linux/seq_buf.h>
#include <linux/timex.h>
#include <linux/timekeeping.h>
static void analog_name(struct analog *analog)
{
- snprintf(analog->name, sizeof(analog->name), "Analog %d-axis %d-button",
+ struct seq_buf s;
+
+ seq_buf_init(&s, analog->name, sizeof(analog->name));
+ seq_buf_printf(&s, "Analog %d-axis %d-button",
hweight8(analog->mask & ANALOG_AXES_STD),
hweight8(analog->mask & ANALOG_BTNS_STD) + !!(analog->mask & ANALOG_BTNS_CHF) * 2 +
hweight16(analog->mask & ANALOG_BTNS_GAMEPAD) + !!(analog->mask & ANALOG_HBTN_CHF) * 4);
if (analog->mask & ANALOG_HATS_ALL)
- snprintf(analog->name, sizeof(analog->name), "%s %d-hat",
- analog->name, hweight16(analog->mask & ANALOG_HATS_ALL));
+ seq_buf_printf(&s, " %d-hat",
+ hweight16(analog->mask & ANALOG_HATS_ALL));
if (analog->mask & ANALOG_HAT_FCS)
- strlcat(analog->name, " FCS", sizeof(analog->name));
+ seq_buf_printf(&s, " FCS");
if (analog->mask & ANALOG_ANY_CHF)
- strlcat(analog->name, (analog->mask & ANALOG_SAITEK) ? " Saitek" : " CHF",
- sizeof(analog->name));
+ seq_buf_printf(&s, (analog->mask & ANALOG_SAITEK) ? " Saitek" : " CHF");
- strlcat(analog->name, (analog->mask & ANALOG_GAMEPAD) ? " gamepad": " joystick",
- sizeof(analog->name));
+ seq_buf_printf(&s, (analog->mask & ANALOG_GAMEPAD) ? " gamepad" : " joystick");
}
/*
id,
USB_TYPE_VENDOR | USB_DIR_IN |
USB_RECIP_INTERFACE,
- 0, 0, buf, IFORCE_MAX_LENGTH, HZ);
+ 0, 0, buf, IFORCE_MAX_LENGTH, 1000);
if (status < 0) {
dev_err(&iforce_usb->intf->dev,
"usb_submit_urb failed: %d\n", status);
const signed char *axes;
const short *buttons;
} tmdc_models[] = {
- { 1, "ThrustMaster Millenium 3D Inceptor", 6, 2, { 4, 2 }, { 4, 6 }, tmdc_abs, tmdc_btn_joy },
+ { 1, "ThrustMaster Millennium 3D Inceptor", 6, 2, { 4, 2 }, { 4, 6 }, tmdc_abs, tmdc_btn_joy },
{ 3, "ThrustMaster Rage 3D Gamepad", 2, 0, { 8, 2 }, { 0, 0 }, tmdc_abs, tmdc_btn_pad },
{ 4, "ThrustMaster Attack Throttle", 5, 2, { 4, 6 }, { 4, 2 }, tmdc_abs_at, tmdc_btn_at },
{ 8, "ThrustMaster FragMaster", 4, 0, { 8, 2 }, { 0, 0 }, tmdc_abs_fm, tmdc_btn_fm },
To compile this driver as a module, choose M here: the
module will be called pmic-keys.
+config KEYBOARD_CYPRESS_SF
+ tristate "Cypress StreetFighter touchkey support"
+ depends on I2C
+ help
+ Say Y here if you want to enable support for Cypress StreetFighter
+ touchkeys.
+
+ To compile this driver as a module, choose M here: the
+ module will be called cypress-sf.
+
endif
obj-$(CONFIG_KEYBOARD_CAP11XX) += cap11xx.o
obj-$(CONFIG_KEYBOARD_CLPS711X) += clps711x-keypad.o
obj-$(CONFIG_KEYBOARD_CROS_EC) += cros_ec_keyb.o
+obj-$(CONFIG_KEYBOARD_CYPRESS_SF) += cypress-sf.o
obj-$(CONFIG_KEYBOARD_DAVINCI) += davinci_keyscan.o
obj-$(CONFIG_KEYBOARD_DLINK_DIR685) += dlink-dir685-touchkeys.o
obj-$(CONFIG_KEYBOARD_EP93XX) += ep93xx_keypad.o
u8 product_id;
unsigned int num_channels;
unsigned int num_leds;
+ bool no_gain;
};
enum {
CAP1106,
CAP1126,
CAP1188,
+ CAP1206,
};
static const struct cap11xx_hw_model cap11xx_devices[] = {
- [CAP1106] = { .product_id = 0x55, .num_channels = 6, .num_leds = 0 },
- [CAP1126] = { .product_id = 0x53, .num_channels = 6, .num_leds = 2 },
- [CAP1188] = { .product_id = 0x50, .num_channels = 8, .num_leds = 8 },
+ [CAP1106] = { .product_id = 0x55, .num_channels = 6, .num_leds = 0, .no_gain = false },
+ [CAP1126] = { .product_id = 0x53, .num_channels = 6, .num_leds = 2, .no_gain = false },
+ [CAP1188] = { .product_id = 0x50, .num_channels = 8, .num_leds = 8, .no_gain = false },
+ [CAP1206] = { .product_id = 0x67, .num_channels = 6, .num_leds = 0, .no_gain = true },
};
static const struct reg_default cap11xx_reg_defaults[] = {
node = dev->of_node;
if (!of_property_read_u32(node, "microchip,sensor-gain", &gain32)) {
- if (is_power_of_2(gain32) && gain32 <= 8)
+ if (cap->no_gain)
+ dev_warn(dev,
+ "This version doesn't support sensor gain\n");
+ else if (is_power_of_2(gain32) && gain32 <= 8)
gain = ilog2(gain32);
else
dev_err(dev, "Invalid sensor-gain value %d\n", gain32);
}
- if (of_property_read_bool(node, "microchip,irq-active-high")) {
- error = regmap_update_bits(priv->regmap, CAP11XX_REG_CONFIG2,
- CAP11XX_REG_CONFIG2_ALT_POL, 0);
- if (error)
- return error;
+ if (id->driver_data != CAP1206) {
+ if (of_property_read_bool(node, "microchip,irq-active-high")) {
+ error = regmap_update_bits(priv->regmap,
+ CAP11XX_REG_CONFIG2,
+ CAP11XX_REG_CONFIG2_ALT_POL,
+ 0);
+ if (error)
+ return error;
+ }
}
/* Provide some useful defaults */
of_property_read_u32_array(node, "linux,keycodes",
priv->keycodes, cap->num_channels);
- error = regmap_update_bits(priv->regmap, CAP11XX_REG_MAIN_CONTROL,
- CAP11XX_REG_MAIN_CONTROL_GAIN_MASK,
- gain << CAP11XX_REG_MAIN_CONTROL_GAIN_SHIFT);
- if (error)
- return error;
+ if (!cap->no_gain) {
+ error = regmap_update_bits(priv->regmap,
+ CAP11XX_REG_MAIN_CONTROL,
+ CAP11XX_REG_MAIN_CONTROL_GAIN_MASK,
+ gain << CAP11XX_REG_MAIN_CONTROL_GAIN_SHIFT);
+ if (error)
+ return error;
+ }
/* Disable autorepeat. The Linux input system has its own handling. */
error = regmap_write(priv->regmap, CAP11XX_REG_REPEAT_RATE, 0);
{ .compatible = "microchip,cap1106", },
{ .compatible = "microchip,cap1126", },
{ .compatible = "microchip,cap1188", },
+ { .compatible = "microchip,cap1206", },
{}
};
MODULE_DEVICE_TABLE(of, cap11xx_dt_ids);
{ "cap1106", CAP1106 },
{ "cap1126", CAP1126 },
{ "cap1188", CAP1188 },
+ { "cap1206", CAP1206 },
{}
};
MODULE_DEVICE_TABLE(i2c, cap11xx_i2c_ids);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Cypress StreetFighter Touchkey Driver
+ *
+ * Copyright (c) 2021 Yassine Oudjana <y.oudjana@protonmail.com>
+ */
+
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/input.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pm.h>
+#include <linux/regulator/consumer.h>
+
+#define CYPRESS_SF_DEV_NAME "cypress-sf"
+
+#define CYPRESS_SF_REG_BUTTON_STATUS 0x4a
+
+struct cypress_sf_data {
+ struct i2c_client *client;
+ struct input_dev *input_dev;
+ struct regulator_bulk_data regulators[2];
+ u32 *keycodes;
+ unsigned long keystates;
+ int num_keys;
+};
+
+static irqreturn_t cypress_sf_irq_handler(int irq, void *devid)
+{
+ struct cypress_sf_data *touchkey = devid;
+ unsigned long keystates, changed;
+ bool new_state;
+ int val, key;
+
+ val = i2c_smbus_read_byte_data(touchkey->client,
+ CYPRESS_SF_REG_BUTTON_STATUS);
+ if (val < 0) {
+ dev_err(&touchkey->client->dev,
+ "Failed to read button status: %d", val);
+ return IRQ_NONE;
+ }
+ keystates = val;
+
+ bitmap_xor(&changed, &keystates, &touchkey->keystates,
+ touchkey->num_keys);
+
+ for_each_set_bit(key, &changed, touchkey->num_keys) {
+ new_state = keystates & BIT(key);
+ dev_dbg(&touchkey->client->dev,
+ "Key %d changed to %d", key, new_state);
+ input_report_key(touchkey->input_dev,
+ touchkey->keycodes[key], new_state);
+ }
+
+ input_sync(touchkey->input_dev);
+ touchkey->keystates = keystates;
+
+ return IRQ_HANDLED;
+}
+
+static int cypress_sf_probe(struct i2c_client *client)
+{
+ struct cypress_sf_data *touchkey;
+ int key, error;
+
+ touchkey = devm_kzalloc(&client->dev, sizeof(*touchkey), GFP_KERNEL);
+ if (!touchkey)
+ return -ENOMEM;
+
+ touchkey->client = client;
+ i2c_set_clientdata(client, touchkey);
+
+ touchkey->regulators[0].supply = "vdd";
+ touchkey->regulators[1].supply = "avdd";
+
+ error = devm_regulator_bulk_get(&client->dev,
+ ARRAY_SIZE(touchkey->regulators),
+ touchkey->regulators);
+ if (error) {
+ dev_err(&client->dev, "Failed to get regulators: %d\n", error);
+ return error;
+ }
+
+ touchkey->num_keys = device_property_read_u32_array(&client->dev,
+ "linux,keycodes",
+ NULL, 0);
+ if (touchkey->num_keys < 0) {
+ /* Default key count */
+ touchkey->num_keys = 2;
+ }
+
+ touchkey->keycodes = devm_kcalloc(&client->dev,
+ touchkey->num_keys,
+ sizeof(*touchkey->keycodes),
+ GFP_KERNEL);
+ if (!touchkey->keycodes)
+ return -ENOMEM;
+
+ error = device_property_read_u32_array(&client->dev, "linux,keycodes",
+ touchkey->keycodes,
+ touchkey->num_keys);
+
+ if (error) {
+ dev_warn(&client->dev,
+ "Failed to read keycodes: %d, using defaults\n",
+ error);
+
+ /* Default keycodes */
+ touchkey->keycodes[0] = KEY_BACK;
+ touchkey->keycodes[1] = KEY_MENU;
+ }
+
+ error = regulator_bulk_enable(ARRAY_SIZE(touchkey->regulators),
+ touchkey->regulators);
+ if (error) {
+ dev_err(&client->dev,
+ "Failed to enable regulators: %d\n", error);
+ return error;
+ }
+
+ touchkey->input_dev = devm_input_allocate_device(&client->dev);
+ if (!touchkey->input_dev) {
+ dev_err(&client->dev, "Failed to allocate input device\n");
+ return -ENOMEM;
+ }
+
+ touchkey->input_dev->name = CYPRESS_SF_DEV_NAME;
+ touchkey->input_dev->id.bustype = BUS_I2C;
+
+ for (key = 0; key < touchkey->num_keys; ++key)
+ input_set_capability(touchkey->input_dev,
+ EV_KEY, touchkey->keycodes[key]);
+
+ error = input_register_device(touchkey->input_dev);
+ if (error) {
+ dev_err(&client->dev,
+ "Failed to register input device: %d\n", error);
+ return error;
+ }
+
+ error = devm_request_threaded_irq(&client->dev, client->irq,
+ NULL, cypress_sf_irq_handler,
+ IRQF_ONESHOT,
+ CYPRESS_SF_DEV_NAME, touchkey);
+ if (error) {
+ dev_err(&client->dev,
+ "Failed to register threaded irq: %d", error);
+ return error;
+ }
+
+ return 0;
+};
+
+static int __maybe_unused cypress_sf_suspend(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct cypress_sf_data *touchkey = i2c_get_clientdata(client);
+ int error;
+
+ disable_irq(client->irq);
+
+ error = regulator_bulk_disable(ARRAY_SIZE(touchkey->regulators),
+ touchkey->regulators);
+ if (error) {
+ dev_err(dev, "Failed to disable regulators: %d", error);
+ enable_irq(client->irq);
+ return error;
+ }
+
+ return 0;
+}
+
+static int __maybe_unused cypress_sf_resume(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct cypress_sf_data *touchkey = i2c_get_clientdata(client);
+ int error;
+
+ error = regulator_bulk_enable(ARRAY_SIZE(touchkey->regulators),
+ touchkey->regulators);
+ if (error) {
+ dev_err(dev, "Failed to enable regulators: %d", error);
+ return error;
+ }
+
+ enable_irq(client->irq);
+
+ return 0;
+}
+
+static SIMPLE_DEV_PM_OPS(cypress_sf_pm_ops,
+ cypress_sf_suspend, cypress_sf_resume);
+
+static struct i2c_device_id cypress_sf_id_table[] = {
+ { CYPRESS_SF_DEV_NAME, 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, cypress_sf_id_table);
+
+#ifdef CONFIG_OF
+static const struct of_device_id cypress_sf_of_match[] = {
+ { .compatible = "cypress,sf3155", },
+ { },
+};
+MODULE_DEVICE_TABLE(of, cypress_sf_of_match);
+#endif
+
+static struct i2c_driver cypress_sf_driver = {
+ .driver = {
+ .name = CYPRESS_SF_DEV_NAME,
+ .pm = &cypress_sf_pm_ops,
+ .of_match_table = of_match_ptr(cypress_sf_of_match),
+ },
+ .id_table = cypress_sf_id_table,
+ .probe_new = cypress_sf_probe,
+};
+module_i2c_driver(cypress_sf_driver);
+
+MODULE_AUTHOR("Yassine Oudjana <y.oudjana@protonmail.com>");
+MODULE_DESCRIPTION("Cypress StreetFighter Touchkey Driver");
+MODULE_LICENSE("GPL v2");
* flag.
*/
+#include <linux/bits.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/interrupt.h>
#include <linux/slab.h>
#include <linux/soc/cirrus/ep93xx.h>
#include <linux/platform_data/keypad-ep93xx.h>
+#include <linux/pm_wakeirq.h>
/*
* Keypad Interface Register offsets
#define KEY_REG 0x08 /* Key Value Capture register */
/* Key Scan Initialization Register bit defines */
-#define KEY_INIT_DBNC_MASK (0x00ff0000)
-#define KEY_INIT_DBNC_SHIFT (16)
-#define KEY_INIT_DIS3KY (1<<15)
-#define KEY_INIT_DIAG (1<<14)
-#define KEY_INIT_BACK (1<<13)
-#define KEY_INIT_T2 (1<<12)
-#define KEY_INIT_PRSCL_MASK (0x000003ff)
-#define KEY_INIT_PRSCL_SHIFT (0)
+#define KEY_INIT_DBNC_MASK GENMASK(23, 16)
+#define KEY_INIT_DBNC_SHIFT 16
+#define KEY_INIT_DIS3KY BIT(15)
+#define KEY_INIT_DIAG BIT(14)
+#define KEY_INIT_BACK BIT(13)
+#define KEY_INIT_T2 BIT(12)
+#define KEY_INIT_PRSCL_MASK GENMASK(9, 0)
+#define KEY_INIT_PRSCL_SHIFT 0
/* Key Scan Diagnostic Register bit defines */
-#define KEY_DIAG_MASK (0x0000003f)
-#define KEY_DIAG_SHIFT (0)
+#define KEY_DIAG_MASK GENMASK(5, 0)
+#define KEY_DIAG_SHIFT 0
/* Key Value Capture Register bit defines */
-#define KEY_REG_K (1<<15)
-#define KEY_REG_INT (1<<14)
-#define KEY_REG_2KEYS (1<<13)
-#define KEY_REG_1KEY (1<<12)
-#define KEY_REG_KEY2_MASK (0x00000fc0)
-#define KEY_REG_KEY2_SHIFT (6)
-#define KEY_REG_KEY1_MASK (0x0000003f)
-#define KEY_REG_KEY1_SHIFT (0)
+#define KEY_REG_K BIT(15)
+#define KEY_REG_INT BIT(14)
+#define KEY_REG_2KEYS BIT(13)
+#define KEY_REG_1KEY BIT(12)
+#define KEY_REG_KEY2_MASK GENMASK(11, 6)
+#define KEY_REG_KEY2_SHIFT 6
+#define KEY_REG_KEY1_MASK GENMASK(5, 0)
+#define KEY_REG_KEY1_SHIFT 0
#define EP93XX_MATRIX_SIZE (EP93XX_MATRIX_ROWS * EP93XX_MATRIX_COLS)
}
-#ifdef CONFIG_PM_SLEEP
-static int ep93xx_keypad_suspend(struct device *dev)
+static int __maybe_unused ep93xx_keypad_suspend(struct device *dev)
{
struct platform_device *pdev = to_platform_device(dev);
struct ep93xx_keypad *keypad = platform_get_drvdata(pdev);
mutex_unlock(&input_dev->mutex);
- if (device_may_wakeup(&pdev->dev))
- enable_irq_wake(keypad->irq);
-
return 0;
}
-static int ep93xx_keypad_resume(struct device *dev)
+static int __maybe_unused ep93xx_keypad_resume(struct device *dev)
{
struct platform_device *pdev = to_platform_device(dev);
struct ep93xx_keypad *keypad = platform_get_drvdata(pdev);
struct input_dev *input_dev = keypad->input_dev;
- if (device_may_wakeup(&pdev->dev))
- disable_irq_wake(keypad->irq);
-
mutex_lock(&input_dev->mutex);
if (input_device_enabled(input_dev)) {
return 0;
}
-#endif
static SIMPLE_DEV_PM_OPS(ep93xx_keypad_pm_ops,
ep93xx_keypad_suspend, ep93xx_keypad_resume);
+static void ep93xx_keypad_release_gpio_action(void *_pdev)
+{
+ struct platform_device *pdev = _pdev;
+
+ ep93xx_keypad_release_gpio(pdev);
+}
+
static int ep93xx_keypad_probe(struct platform_device *pdev)
{
struct ep93xx_keypad *keypad;
struct resource *res;
int err;
- keypad = kzalloc(sizeof(struct ep93xx_keypad), GFP_KERNEL);
+ keypad = devm_kzalloc(&pdev->dev, sizeof(*keypad), GFP_KERNEL);
if (!keypad)
return -ENOMEM;
keypad->pdata = dev_get_platdata(&pdev->dev);
- if (!keypad->pdata) {
- err = -EINVAL;
- goto failed_free;
- }
+ if (!keypad->pdata)
+ return -EINVAL;
keymap_data = keypad->pdata->keymap_data;
- if (!keymap_data) {
- err = -EINVAL;
- goto failed_free;
- }
+ if (!keymap_data)
+ return -EINVAL;
keypad->irq = platform_get_irq(pdev, 0);
- if (keypad->irq < 0) {
- err = keypad->irq;
- goto failed_free;
- }
+ if (keypad->irq < 0)
+ return keypad->irq;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- err = -ENXIO;
- goto failed_free;
- }
+ if (!res)
+ return -ENXIO;
- res = request_mem_region(res->start, resource_size(res), pdev->name);
- if (!res) {
- err = -EBUSY;
- goto failed_free;
- }
-
- keypad->mmio_base = ioremap(res->start, resource_size(res));
- if (keypad->mmio_base == NULL) {
- err = -ENXIO;
- goto failed_free_mem;
- }
+ keypad->mmio_base = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(keypad->mmio_base))
+ return PTR_ERR(keypad->mmio_base);
err = ep93xx_keypad_acquire_gpio(pdev);
if (err)
- goto failed_free_io;
+ return err;
- keypad->clk = clk_get(&pdev->dev, NULL);
- if (IS_ERR(keypad->clk)) {
- err = PTR_ERR(keypad->clk);
- goto failed_free_gpio;
- }
+ err = devm_add_action_or_reset(&pdev->dev,
+ ep93xx_keypad_release_gpio_action, pdev);
+ if (err)
+ return err;
- input_dev = input_allocate_device();
- if (!input_dev) {
- err = -ENOMEM;
- goto failed_put_clk;
- }
+ keypad->clk = devm_clk_get(&pdev->dev, NULL);
+ if (IS_ERR(keypad->clk))
+ return PTR_ERR(keypad->clk);
+
+ input_dev = devm_input_allocate_device(&pdev->dev);
+ if (!input_dev)
+ return -ENOMEM;
keypad->input_dev = input_dev;
input_dev->id.bustype = BUS_HOST;
input_dev->open = ep93xx_keypad_open;
input_dev->close = ep93xx_keypad_close;
- input_dev->dev.parent = &pdev->dev;
err = matrix_keypad_build_keymap(keymap_data, NULL,
EP93XX_MATRIX_ROWS, EP93XX_MATRIX_COLS,
keypad->keycodes, input_dev);
if (err)
- goto failed_free_dev;
+ return err;
if (keypad->pdata->flags & EP93XX_KEYPAD_AUTOREPEAT)
__set_bit(EV_REP, input_dev->evbit);
input_set_drvdata(input_dev, keypad);
- err = request_irq(keypad->irq, ep93xx_keypad_irq_handler,
- 0, pdev->name, keypad);
+ err = devm_request_irq(&pdev->dev, keypad->irq,
+ ep93xx_keypad_irq_handler,
+ 0, pdev->name, keypad);
if (err)
- goto failed_free_dev;
+ return err;
err = input_register_device(input_dev);
if (err)
- goto failed_free_irq;
+ return err;
platform_set_drvdata(pdev, keypad);
+
device_init_wakeup(&pdev->dev, 1);
+ err = dev_pm_set_wake_irq(&pdev->dev, keypad->irq);
+ if (err)
+ dev_warn(&pdev->dev, "failed to set up wakeup irq: %d\n", err);
return 0;
-
-failed_free_irq:
- free_irq(keypad->irq, keypad);
-failed_free_dev:
- input_free_device(input_dev);
-failed_put_clk:
- clk_put(keypad->clk);
-failed_free_gpio:
- ep93xx_keypad_release_gpio(pdev);
-failed_free_io:
- iounmap(keypad->mmio_base);
-failed_free_mem:
- release_mem_region(res->start, resource_size(res));
-failed_free:
- kfree(keypad);
- return err;
}
static int ep93xx_keypad_remove(struct platform_device *pdev)
{
- struct ep93xx_keypad *keypad = platform_get_drvdata(pdev);
- struct resource *res;
-
- free_irq(keypad->irq, keypad);
-
- if (keypad->enabled)
- clk_disable(keypad->clk);
- clk_put(keypad->clk);
-
- input_unregister_device(keypad->input_dev);
-
- ep93xx_keypad_release_gpio(pdev);
-
- iounmap(keypad->mmio_base);
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- release_mem_region(res->start, resource_size(res));
-
- kfree(keypad);
+ dev_pm_clear_wake_irq(&pdev->dev);
return 0;
}
return ERR_PTR(err);
}
- err = devm_add_action(dev, mpr121_vdd_supply_disable, vdd_supply);
+ err = devm_add_action_or_reset(dev, mpr121_vdd_supply_disable,
+ vdd_supply);
if (err) {
- regulator_disable(vdd_supply);
dev_err(dev, "failed to add disable regulator action: %d\n",
err);
return ERR_PTR(err);
row_shift = get_count_order(pdata->cols);
keycodemax = pdata->rows << row_shift;
- omap_kp = kzalloc(sizeof(struct omap_kp) +
- keycodemax * sizeof(unsigned short), GFP_KERNEL);
+ omap_kp = kzalloc(struct_size(omap_kp, keymap, keycodemax), GFP_KERNEL);
input_dev = input_allocate_device();
if (!omap_kp || !input_dev) {
kfree(omap_kp);
goto out;
}
+ input_event(touchkey->input_dev, EV_MSC, MSC_SCAN, index);
+
if (data & TM2_TOUCHKEY_BIT_PRESS_EV) {
for (i = 0; i < touchkey->num_keycodes; i++)
input_report_key(touchkey->input_dev,
touchkey->input_dev->name = TM2_TOUCHKEY_DEV_NAME;
touchkey->input_dev->id.bustype = BUS_I2C;
+ touchkey->input_dev->keycode = touchkey->keycodes;
+ touchkey->input_dev->keycodemax = touchkey->num_keycodes;
+ touchkey->input_dev->keycodesize = sizeof(touchkey->keycodes[0]);
+
+ input_set_capability(touchkey->input_dev, EV_MSC, MSC_SCAN);
for (i = 0; i < touchkey->num_keycodes; i++)
input_set_capability(touchkey->input_dev, EV_KEY,
touchkey->keycodes[i]);
{
struct adxl34x *ac = i2c_get_clientdata(client);
- return adxl34x_remove(ac);
+ adxl34x_remove(ac);
+
+ return 0;
}
static int __maybe_unused adxl34x_i2c_suspend(struct device *dev)
{
struct adxl34x *ac = spi_get_drvdata(spi);
- return adxl34x_remove(ac);
+ adxl34x_remove(ac);
+
+ return 0;
}
static int __maybe_unused adxl34x_spi_suspend(struct device *dev)
static void adxl34x_get_triple(struct adxl34x *ac, struct axis_triple *axis)
{
- short buf[3];
+ __le16 buf[3];
ac->bops->read_block(ac->dev, DATAX0, DATAZ1 - DATAX0 + 1, buf);
}
EXPORT_SYMBOL_GPL(adxl34x_probe);
-int adxl34x_remove(struct adxl34x *ac)
+void adxl34x_remove(struct adxl34x *ac)
{
sysfs_remove_group(&ac->dev->kobj, &adxl34x_attr_group);
free_irq(ac->irq, ac);
input_unregister_device(ac->input);
dev_dbg(ac->dev, "unregistered accelerometer\n");
kfree(ac);
-
- return 0;
}
EXPORT_SYMBOL_GPL(adxl34x_remove);
struct adxl34x *adxl34x_probe(struct device *dev, int irq,
bool fifo_delay_default,
const struct adxl34x_bus_ops *bops);
-int adxl34x_remove(struct adxl34x *ac);
+void adxl34x_remove(struct adxl34x *ac);
#endif
};
MODULE_DEVICE_TABLE(of, ariel_pwrbutton_of_match);
+static const struct spi_device_id ariel_pwrbutton_spi_ids[] = {
+ { .name = "wyse-ariel-ec-input" },
+ { }
+};
+MODULE_DEVICE_TABLE(spi, ariel_pwrbutton_spi_ids);
+
static struct spi_driver ariel_pwrbutton_driver = {
.driver = {
.name = "dell-wyse-ariel-ec-input",
.of_match_table = ariel_pwrbutton_of_match,
},
.probe = ariel_pwrbutton_probe,
+ .id_table = ariel_pwrbutton_spi_ids,
};
module_spi_driver(ariel_pwrbutton_driver);
static int cpcap_power_button_probe(struct platform_device *pdev)
{
struct cpcap_power_button *button;
- int irq = platform_get_irq(pdev, 0);
+ int irq;
int err;
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
button = devm_kmalloc(&pdev->dev, sizeof(*button), GFP_KERNEL);
if (!button)
return -ENOMEM;
button->idev->name = "cpcap-pwrbutton";
button->idev->phys = "cpcap-pwrbutton/input0";
- button->idev->dev.parent = button->dev;
input_set_capability(button->idev, EV_KEY, KEY_POWER);
err = devm_request_threaded_irq(&pdev->dev, irq, NULL,
MODULE_AUTHOR("Jaewon Kim <jaewon02.kim@samsung.com>");
MODULE_AUTHOR("Krzysztof Kozlowski <krzk@kernel.org>");
MODULE_DESCRIPTION("MAXIM 77693/77843 Haptic driver");
-MODULE_ALIAS("platform:max77693-haptic");
MODULE_LICENSE("GPL");
-/**
+/*
* MAX8925 ONKEY driver
*
* Copyright (C) 2009 Marvell International Ltd.
INIT_DELAYED_WORK(&pwron->input_work, palmas_power_button_work);
pwron->irq = platform_get_irq(pdev, 0);
+ if (pwron->irq < 0) {
+ error = pwron->irq;
+ goto err_free_input;
+ }
+
error = request_threaded_irq(pwron->irq, NULL, pwron_irq,
IRQF_TRIGGER_HIGH |
IRQF_TRIGGER_LOW |
#define PON_PS_HOLD_RST_CTL2 0x5b
#define PON_PS_HOLD_ENABLE BIT(7)
#define PON_PS_HOLD_TYPE_MASK 0x0f
+#define PON_PS_HOLD_TYPE_WARM_RESET 1
#define PON_PS_HOLD_TYPE_SHUTDOWN 4
#define PON_PS_HOLD_TYPE_HARD_RESET 7
break;
case SYS_RESTART:
default:
- reset_type = PON_PS_HOLD_TYPE_HARD_RESET;
+ if (reboot_mode == REBOOT_WARM)
+ reset_type = PON_PS_HOLD_TYPE_WARM_RESET;
+ else
+ reset_type = PON_PS_HOLD_TYPE_HARD_RESET;
break;
}
case 0x16008020U:
case 0x26800010U:
case 0x36808000U:
+
+ /*
+ * This firmware misreport coordinates for trackpoint
+ * occasionally. Discard packets outside of [-127, 127] range
+ * to prevent cursor jumps.
+ */
+ if (packet[4] == 0x80 || packet[5] == 0x80 ||
+ packet[1] >> 7 == packet[4] >> 7 ||
+ packet[2] >> 7 == packet[5] >> 7) {
+ elantech_debug("discarding packet [%6ph]\n", packet);
+ break;
+
+ }
x = packet[4] - (int)((packet[1]^0x80) << 1);
y = (int)((packet[2]^0x80) << 1) - packet[5];
rmi_dev->dev.bus = &rmi_bus_type;
rmi_dev->dev.type = &rmi_device_type;
+ rmi_dev->dev.parent = xport->dev;
xport->rmi_dev = rmi_dev;
},
},
{
+ /* Fujitsu Lifebook T725 laptop */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK T725"),
+ },
+ },
+ {
/* Fujitsu Lifebook U745 */
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU"),
},
},
{
+ /* Fujitsu Lifebook T725 laptop */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK T725"),
+ },
+ },
+ {
/* Fujitsu U574 laptop */
/* https://bugzilla.kernel.org/show_bug.cgi?id=69731 */
.matches = {
config TOUCHSCREEN_ILI210X
tristate "Ilitek ILI210X based touchscreen"
depends on I2C
+ select CRC_CCITT
help
Say Y here if you have a ILI210X based touchscreen
controller. This driver supports models ILI2102,
# Each configuration option enables a list of files.
wm97xx-ts-y := wm97xx-core.o
+goodix_ts-y := goodix.o goodix_fwupload.o
obj-$(CONFIG_TOUCHSCREEN_88PM860X) += 88pm860x-ts.o
obj-$(CONFIG_TOUCHSCREEN_AD7877) += ad7877.o
obj-$(CONFIG_TOUCHSCREEN_EGALAX_SERIAL) += egalax_ts_serial.o
obj-$(CONFIG_TOUCHSCREEN_EXC3000) += exc3000.o
obj-$(CONFIG_TOUCHSCREEN_FUJITSU) += fujitsu_ts.o
-obj-$(CONFIG_TOUCHSCREEN_GOODIX) += goodix.o
+obj-$(CONFIG_TOUCHSCREEN_GOODIX) += goodix_ts.o
obj-$(CONFIG_TOUCHSCREEN_HIDEEP) += hideep.o
obj-$(CONFIG_TOUCHSCREEN_ILI210X) += ili210x.o
obj-$(CONFIG_TOUCHSCREEN_ILITEK) += ilitek_ts_i2c.o
struct spi_device *spi;
struct regulator *reg;
-#if IS_ENABLED(CONFIG_HWMON)
- struct device *hwmon;
-#endif
-
u16 model;
u16 vref_mv;
u16 vref_delay_usecs;
int (*filter)(void *data, int data_idx, int *val);
void *filter_data;
- void (*filter_cleanup)(void *data);
int (*get_pendown_state)(void);
int gpio_pendown;
void (*wait_for_sync)(void);
};
+enum ads7846_filter {
+ ADS7846_FILTER_OK,
+ ADS7846_FILTER_REPEAT,
+ ADS7846_FILTER_IGNORE,
+};
+
/* leave chip selected when we're done, for quicker re-select? */
#if 0
#define CS_CHANGE(xfer) ((xfer).cs_change = 1)
static int ads784x_hwmon_register(struct spi_device *spi, struct ads7846 *ts)
{
+ struct device *hwmon;
+
/* hwmon sensors need a reference voltage */
switch (ts->model) {
case 7846:
break;
}
- ts->hwmon = hwmon_device_register_with_groups(&spi->dev, spi->modalias,
- ts, ads7846_attr_groups);
+ hwmon = devm_hwmon_device_register_with_groups(&spi->dev,
+ spi->modalias, ts,
+ ads7846_attr_groups);
- return PTR_ERR_OR_ZERO(ts->hwmon);
-}
-
-static void ads784x_hwmon_unregister(struct spi_device *spi,
- struct ads7846 *ts)
-{
- if (ts->hwmon)
- hwmon_device_unregister(ts->hwmon);
+ return PTR_ERR_OR_ZERO(hwmon);
}
#else
{
return 0;
}
-
-static inline void ads784x_hwmon_unregister(struct spi_device *spi,
- struct ads7846 *ts)
-{
-}
#endif
static ssize_t ads7846_pen_down_show(struct device *dev,
ts->get_pendown_state = pdata->get_pendown_state;
} else if (gpio_is_valid(pdata->gpio_pendown)) {
- err = gpio_request_one(pdata->gpio_pendown, GPIOF_IN,
- "ads7846_pendown");
+ err = devm_gpio_request_one(&spi->dev, pdata->gpio_pendown,
+ GPIOF_IN, "ads7846_pendown");
if (err) {
dev_err(&spi->dev,
"failed to request/setup pendown GPIO%d: %d\n",
}
#endif
+static void ads7846_regulator_disable(void *regulator)
+{
+ regulator_disable(regulator);
+}
+
static int ads7846_probe(struct spi_device *spi)
{
const struct ads7846_platform_data *pdata;
struct ads7846 *ts;
+ struct device *dev = &spi->dev;
struct ads7846_packet *packet;
struct input_dev *input_dev;
unsigned long irq_flags;
int err;
if (!spi->irq) {
- dev_dbg(&spi->dev, "no IRQ?\n");
+ dev_dbg(dev, "no IRQ?\n");
return -EINVAL;
}
/* don't exceed max specified sample rate */
if (spi->max_speed_hz > (125000 * SAMPLE_BITS)) {
- dev_err(&spi->dev, "f(sample) %d KHz?\n",
- (spi->max_speed_hz/SAMPLE_BITS)/1000);
+ dev_err(dev, "f(sample) %d KHz?\n",
+ (spi->max_speed_hz/SAMPLE_BITS)/1000);
return -EINVAL;
}
if (err < 0)
return err;
- ts = kzalloc(sizeof(struct ads7846), GFP_KERNEL);
- packet = kzalloc(sizeof(struct ads7846_packet), GFP_KERNEL);
- input_dev = input_allocate_device();
- if (!ts || !packet || !input_dev) {
- err = -ENOMEM;
- goto err_free_mem;
- }
+ ts = devm_kzalloc(dev, sizeof(struct ads7846), GFP_KERNEL);
+ if (!ts)
+ return -ENOMEM;
+
+ packet = devm_kzalloc(dev, sizeof(struct ads7846_packet), GFP_KERNEL);
+ if (!packet)
+ return -ENOMEM;
+
+ input_dev = devm_input_allocate_device(dev);
+ if (!input_dev)
+ return -ENOMEM;
spi_set_drvdata(spi, ts);
mutex_init(&ts->lock);
init_waitqueue_head(&ts->wait);
- pdata = dev_get_platdata(&spi->dev);
+ pdata = dev_get_platdata(dev);
if (!pdata) {
- pdata = ads7846_probe_dt(&spi->dev);
- if (IS_ERR(pdata)) {
- err = PTR_ERR(pdata);
- goto err_free_mem;
- }
+ pdata = ads7846_probe_dt(dev);
+ if (IS_ERR(pdata))
+ return PTR_ERR(pdata);
}
ts->model = pdata->model ? : 7846;
ts->x_plate_ohms = pdata->x_plate_ohms ? : 400;
ts->vref_mv = pdata->vref_mv;
- if (pdata->filter != NULL) {
- if (pdata->filter_init != NULL) {
- err = pdata->filter_init(pdata, &ts->filter_data);
- if (err < 0)
- goto err_free_mem;
- }
- ts->filter = pdata->filter;
- ts->filter_cleanup = pdata->filter_cleanup;
- } else if (pdata->debounce_max) {
+ if (pdata->debounce_max) {
ts->debounce_max = pdata->debounce_max;
if (ts->debounce_max < 2)
ts->debounce_max = 2;
err = ads7846_setup_pendown(spi, ts, pdata);
if (err)
- goto err_cleanup_filter;
+ return err;
if (pdata->penirq_recheck_delay_usecs)
ts->penirq_recheck_delay_usecs =
ts->wait_for_sync = pdata->wait_for_sync ? : null_wait_for_sync;
- snprintf(ts->phys, sizeof(ts->phys), "%s/input0", dev_name(&spi->dev));
+ snprintf(ts->phys, sizeof(ts->phys), "%s/input0", dev_name(dev));
snprintf(ts->name, sizeof(ts->name), "ADS%d Touchscreen", ts->model);
input_dev->name = ts->name;
input_dev->phys = ts->phys;
- input_dev->dev.parent = &spi->dev;
- input_dev->evbit[0] = BIT_MASK(EV_KEY) | BIT_MASK(EV_ABS);
- input_dev->keybit[BIT_WORD(BTN_TOUCH)] = BIT_MASK(BTN_TOUCH);
+ input_dev->id.bustype = BUS_SPI;
+ input_dev->id.product = pdata->model;
+
+ input_set_capability(input_dev, EV_KEY, BTN_TOUCH);
input_set_abs_params(input_dev, ABS_X,
pdata->x_min ? : 0,
pdata->x_max ? : MAX_12BIT,
ads7846_setup_spi_msg(ts, pdata);
- ts->reg = regulator_get(&spi->dev, "vcc");
+ ts->reg = devm_regulator_get(dev, "vcc");
if (IS_ERR(ts->reg)) {
err = PTR_ERR(ts->reg);
- dev_err(&spi->dev, "unable to get regulator: %d\n", err);
- goto err_free_gpio;
+ dev_err(dev, "unable to get regulator: %d\n", err);
+ return err;
}
err = regulator_enable(ts->reg);
if (err) {
- dev_err(&spi->dev, "unable to enable regulator: %d\n", err);
- goto err_put_regulator;
+ dev_err(dev, "unable to enable regulator: %d\n", err);
+ return err;
}
+ err = devm_add_action_or_reset(dev, ads7846_regulator_disable, ts->reg);
+ if (err)
+ return err;
+
irq_flags = pdata->irq_flags ? : IRQF_TRIGGER_FALLING;
irq_flags |= IRQF_ONESHOT;
- err = request_threaded_irq(spi->irq, ads7846_hard_irq, ads7846_irq,
- irq_flags, spi->dev.driver->name, ts);
- if (err && !pdata->irq_flags) {
- dev_info(&spi->dev,
+ err = devm_request_threaded_irq(dev, spi->irq,
+ ads7846_hard_irq, ads7846_irq,
+ irq_flags, dev->driver->name, ts);
+ if (err && err != -EPROBE_DEFER && !pdata->irq_flags) {
+ dev_info(dev,
"trying pin change workaround on irq %d\n", spi->irq);
irq_flags |= IRQF_TRIGGER_RISING;
- err = request_threaded_irq(spi->irq,
- ads7846_hard_irq, ads7846_irq,
- irq_flags, spi->dev.driver->name, ts);
+ err = devm_request_threaded_irq(dev, spi->irq,
+ ads7846_hard_irq, ads7846_irq,
+ irq_flags, dev->driver->name,
+ ts);
}
if (err) {
- dev_dbg(&spi->dev, "irq %d busy?\n", spi->irq);
- goto err_disable_regulator;
+ dev_dbg(dev, "irq %d busy?\n", spi->irq);
+ return err;
}
err = ads784x_hwmon_register(spi, ts);
if (err)
- goto err_free_irq;
+ return err;
- dev_info(&spi->dev, "touchscreen, irq %d\n", spi->irq);
+ dev_info(dev, "touchscreen, irq %d\n", spi->irq);
/*
* Take a first sample, leaving nPENIRQ active and vREF off; avoid
* the touchscreen, in case it's not connected.
*/
if (ts->model == 7845)
- ads7845_read12_ser(&spi->dev, PWRDOWN);
+ ads7845_read12_ser(dev, PWRDOWN);
else
- (void) ads7846_read12_ser(&spi->dev, READ_12BIT_SER(vaux));
+ (void) ads7846_read12_ser(dev, READ_12BIT_SER(vaux));
- err = sysfs_create_group(&spi->dev.kobj, &ads784x_attr_group);
+ err = devm_device_add_group(dev, &ads784x_attr_group);
if (err)
- goto err_remove_hwmon;
+ return err;
err = input_register_device(input_dev);
if (err)
- goto err_remove_attr_group;
+ return err;
- device_init_wakeup(&spi->dev, pdata->wakeup);
+ device_init_wakeup(dev, pdata->wakeup);
/*
* If device does not carry platform data we must have allocated it
* when parsing DT data.
*/
- if (!dev_get_platdata(&spi->dev))
- devm_kfree(&spi->dev, (void *)pdata);
+ if (!dev_get_platdata(dev))
+ devm_kfree(dev, (void *)pdata);
return 0;
-
- err_remove_attr_group:
- sysfs_remove_group(&spi->dev.kobj, &ads784x_attr_group);
- err_remove_hwmon:
- ads784x_hwmon_unregister(spi, ts);
- err_free_irq:
- free_irq(spi->irq, ts);
- err_disable_regulator:
- regulator_disable(ts->reg);
- err_put_regulator:
- regulator_put(ts->reg);
- err_free_gpio:
- if (!ts->get_pendown_state)
- gpio_free(ts->gpio_pendown);
- err_cleanup_filter:
- if (ts->filter_cleanup)
- ts->filter_cleanup(ts->filter_data);
- err_free_mem:
- input_free_device(input_dev);
- kfree(packet);
- kfree(ts);
- return err;
}
static int ads7846_remove(struct spi_device *spi)
{
struct ads7846 *ts = spi_get_drvdata(spi);
- sysfs_remove_group(&spi->dev.kobj, &ads784x_attr_group);
-
- ads7846_disable(ts);
- free_irq(ts->spi->irq, ts);
-
- input_unregister_device(ts->input);
-
- ads784x_hwmon_unregister(spi, ts);
-
- regulator_put(ts->reg);
-
- if (!ts->get_pendown_state) {
- /*
- * If we are not using specialized pendown method we must
- * have been relying on gpio we set up ourselves.
- */
- gpio_free(ts->gpio_pendown);
- }
-
- if (ts->filter_cleanup)
- ts->filter_cleanup(ts->filter_data);
-
- kfree(ts->packet);
- kfree(ts);
-
- dev_dbg(&spi->dev, "unregistered touchscreen\n");
+ ads7846_stop(ts);
return 0;
}
if (error)
return error;
- error = devm_add_action(&client->dev, elants_i2c_power_off, ts);
+ error = devm_add_action_or_reset(&client->dev,
+ elants_i2c_power_off, ts);
if (error) {
dev_err(&client->dev,
"failed to install power off action: %d\n", error);
- elants_i2c_power_off(ts);
return error;
}
#include <linux/kernel.h>
#include <linux/dmi.h>
#include <linux/firmware.h>
-#include <linux/gpio/consumer.h>
-#include <linux/i2c.h>
-#include <linux/input.h>
-#include <linux/input/mt.h>
-#include <linux/input/touchscreen.h>
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/irq.h>
#include <linux/interrupt.h>
-#include <linux/regulator/consumer.h>
#include <linux/slab.h>
#include <linux/acpi.h>
#include <linux/of.h>
#include <asm/unaligned.h>
+#include "goodix.h"
#define GOODIX_GPIO_INT_NAME "irq"
#define GOODIX_GPIO_RST_NAME "reset"
#define GOODIX_CONTACT_SIZE 8
#define GOODIX_MAX_CONTACT_SIZE 9
#define GOODIX_MAX_CONTACTS 10
-#define GOODIX_MAX_KEYS 7
#define GOODIX_CONFIG_MIN_LENGTH 186
#define GOODIX_CONFIG_911_LENGTH 186
#define GOODIX_CONFIG_967_LENGTH 228
#define GOODIX_CONFIG_GT9X_LENGTH 240
-#define GOODIX_CONFIG_MAX_LENGTH 240
-
-/* Register defines */
-#define GOODIX_REG_COMMAND 0x8040
-#define GOODIX_CMD_SCREEN_OFF 0x05
-
-#define GOODIX_READ_COOR_ADDR 0x814E
-#define GOODIX_GT1X_REG_CONFIG_DATA 0x8050
-#define GOODIX_GT9X_REG_CONFIG_DATA 0x8047
-#define GOODIX_REG_ID 0x8140
#define GOODIX_BUFFER_STATUS_READY BIT(7)
#define GOODIX_HAVE_KEY BIT(4)
#define ACPI_GPIO_SUPPORT
#endif
-struct goodix_ts_data;
-
-enum goodix_irq_pin_access_method {
- IRQ_PIN_ACCESS_NONE,
- IRQ_PIN_ACCESS_GPIO,
- IRQ_PIN_ACCESS_ACPI_GPIO,
- IRQ_PIN_ACCESS_ACPI_METHOD,
-};
-
-struct goodix_chip_data {
- u16 config_addr;
- int config_len;
- int (*check_config)(struct goodix_ts_data *ts, const u8 *cfg, int len);
- void (*calc_config_checksum)(struct goodix_ts_data *ts);
-};
-
struct goodix_chip_id {
const char *id;
const struct goodix_chip_data *data;
};
-#define GOODIX_ID_MAX_LEN 4
-
-struct goodix_ts_data {
- struct i2c_client *client;
- struct input_dev *input_dev;
- const struct goodix_chip_data *chip;
- struct touchscreen_properties prop;
- unsigned int max_touch_num;
- unsigned int int_trigger_type;
- struct regulator *avdd28;
- struct regulator *vddio;
- struct gpio_desc *gpiod_int;
- struct gpio_desc *gpiod_rst;
- int gpio_count;
- int gpio_int_idx;
- char id[GOODIX_ID_MAX_LEN + 1];
- u16 version;
- const char *cfg_name;
- bool reset_controller_at_probe;
- bool load_cfg_from_disk;
- struct completion firmware_loading_complete;
- unsigned long irq_flags;
- enum goodix_irq_pin_access_method irq_pin_access_method;
- unsigned int contact_size;
- u8 config[GOODIX_CONFIG_MAX_LENGTH];
- unsigned short keymap[GOODIX_MAX_KEYS];
-};
-
static int goodix_check_cfg_8(struct goodix_ts_data *ts,
const u8 *cfg, int len);
static int goodix_check_cfg_16(struct goodix_ts_data *ts,
* @buf: raw write data buffer.
* @len: length of the buffer to write
*/
-static int goodix_i2c_read(struct i2c_client *client,
- u16 reg, u8 *buf, int len)
+int goodix_i2c_read(struct i2c_client *client, u16 reg, u8 *buf, int len)
{
struct i2c_msg msgs[2];
__be16 wbuf = cpu_to_be16(reg);
msgs[1].buf = buf;
ret = i2c_transfer(client->adapter, msgs, 2);
- return ret < 0 ? ret : (ret != ARRAY_SIZE(msgs) ? -EIO : 0);
+ if (ret >= 0)
+ ret = (ret == ARRAY_SIZE(msgs) ? 0 : -EIO);
+
+ if (ret)
+ dev_err(&client->dev, "Error reading %d bytes from 0x%04x: %d\n",
+ len, reg, ret);
+ return ret;
}
/**
* @buf: raw data buffer to write.
* @len: length of the buffer to write
*/
-static int goodix_i2c_write(struct i2c_client *client, u16 reg, const u8 *buf,
- unsigned len)
+int goodix_i2c_write(struct i2c_client *client, u16 reg, const u8 *buf, int len)
{
u8 *addr_buf;
struct i2c_msg msg;
msg.len = len + 2;
ret = i2c_transfer(client->adapter, &msg, 1);
+ if (ret >= 0)
+ ret = (ret == 1 ? 0 : -EIO);
+
kfree(addr_buf);
- return ret < 0 ? ret : (ret != 1 ? -EIO : 0);
+
+ if (ret)
+ dev_err(&client->dev, "Error writing %d bytes to 0x%04x: %d\n",
+ len, reg, ret);
+ return ret;
}
-static int goodix_i2c_write_u8(struct i2c_client *client, u16 reg, u8 value)
+int goodix_i2c_write_u8(struct i2c_client *client, u16 reg, u8 value)
{
return goodix_i2c_write(client, reg, &value, sizeof(value));
}
do {
error = goodix_i2c_read(ts->client, addr, data,
header_contact_keycode_size);
- if (error) {
- dev_err(&ts->client->dev, "I2C transfer error: %d\n",
- error);
+ if (error)
return error;
- }
if (data[0] & GOODIX_BUFFER_STATUS_READY) {
touch_num = data[0] & 0x0f;
return touch_num;
}
+ if (data[0] == 0 && ts->firmware_name) {
+ if (goodix_handle_fw_request(ts))
+ return 0;
+ }
+
usleep_range(1000, 2000); /* Poll every 1 - 2 ms */
} while (time_before(jiffies, max_timeout));
struct goodix_ts_data *ts = dev_id;
goodix_process_events(ts);
-
- if (goodix_i2c_write_u8(ts->client, GOODIX_READ_COOR_ADDR, 0) < 0)
- dev_err(&ts->client->dev, "I2C write end_cmd error\n");
+ goodix_i2c_write_u8(ts->client, GOODIX_READ_COOR_ADDR, 0);
return IRQ_HANDLED;
}
* @cfg: config firmware to write to device
* @len: config data length
*/
-static int goodix_send_cfg(struct goodix_ts_data *ts, const u8 *cfg, int len)
+int goodix_send_cfg(struct goodix_ts_data *ts, const u8 *cfg, int len)
{
int error;
return error;
error = goodix_i2c_write(ts->client, ts->chip->config_addr, cfg, len);
- if (error) {
- dev_err(&ts->client->dev, "Failed to write config data: %d",
- error);
+ if (error)
return error;
- }
+
dev_dbg(&ts->client->dev, "Config sent successfully.");
/* Let the firmware reconfigure itself, so sleep for 10ms */
return -EINVAL; /* Never reached */
}
-static int goodix_int_sync(struct goodix_ts_data *ts)
+int goodix_int_sync(struct goodix_ts_data *ts)
{
int error;
error = goodix_irq_direction_output(ts, 0);
if (error)
- return error;
+ goto error;
msleep(50); /* T5: 50ms */
error = goodix_irq_direction_input(ts);
if (error)
- return error;
+ goto error;
return 0;
+
+error:
+ dev_err(&ts->client->dev, "Controller irq sync failed.\n");
+ return error;
}
/**
- * goodix_reset - Reset device during power on
+ * goodix_reset_no_int_sync - Reset device, leaving interrupt line in output mode
*
* @ts: goodix_ts_data pointer
*/
-static int goodix_reset(struct goodix_ts_data *ts)
+int goodix_reset_no_int_sync(struct goodix_ts_data *ts)
{
int error;
/* begin select I2C slave addr */
error = gpiod_direction_output(ts->gpiod_rst, 0);
if (error)
- return error;
+ goto error;
msleep(20); /* T2: > 10ms */
/* HIGH: 0x28/0x29, LOW: 0xBA/0xBB */
error = goodix_irq_direction_output(ts, ts->client->addr == 0x14);
if (error)
- return error;
+ goto error;
usleep_range(100, 2000); /* T3: > 100us */
error = gpiod_direction_output(ts->gpiod_rst, 1);
if (error)
- return error;
+ goto error;
usleep_range(6000, 10000); /* T4: > 5ms */
/* end select I2C slave addr */
error = gpiod_direction_input(ts->gpiod_rst);
if (error)
- return error;
+ goto error;
- error = goodix_int_sync(ts);
+ return 0;
+
+error:
+ dev_err(&ts->client->dev, "Controller reset failed.\n");
+ return error;
+}
+
+/**
+ * goodix_reset - Reset device during power on
+ *
+ * @ts: goodix_ts_data pointer
+ */
+static int goodix_reset(struct goodix_ts_data *ts)
+{
+ int error;
+
+ error = goodix_reset_no_int_sync(ts);
if (error)
return error;
- return 0;
+ return goodix_int_sync(ts);
}
#ifdef ACPI_GPIO_SUPPORT
int x_max, y_max;
int error;
- error = goodix_i2c_read(ts->client, ts->chip->config_addr,
- ts->config, ts->chip->config_len);
- if (error) {
- dev_warn(&ts->client->dev, "Error reading config: %d\n",
- error);
- ts->int_trigger_type = GOODIX_INT_TRIGGER;
- ts->max_touch_num = GOODIX_MAX_CONTACTS;
- return;
+ /*
+ * On controllers where we need to upload the firmware
+ * (controllers without flash) ts->config already has the config
+ * at this point and the controller itself does not have it yet!
+ */
+ if (!ts->firmware_name) {
+ error = goodix_i2c_read(ts->client, ts->chip->config_addr,
+ ts->config, ts->chip->config_len);
+ if (error) {
+ ts->int_trigger_type = GOODIX_INT_TRIGGER;
+ ts->max_touch_num = GOODIX_MAX_CONTACTS;
+ return;
+ }
}
ts->int_trigger_type = ts->config[TRIGGER_LOC] & 0x03;
char id_str[GOODIX_ID_MAX_LEN + 1];
error = goodix_i2c_read(ts->client, GOODIX_REG_ID, buf, sizeof(buf));
- if (error) {
- dev_err(&ts->client->dev, "read version failed: %d\n", error);
+ if (error)
return error;
- }
memcpy(id_str, buf, GOODIX_ID_MAX_LEN);
id_str[GOODIX_ID_MAX_LEN] = 0;
u8 test;
while (retry++ < 2) {
- error = goodix_i2c_read(client, GOODIX_REG_ID,
- &test, 1);
+ error = goodix_i2c_read(client, GOODIX_REG_ID, &test, 1);
if (!error)
return 0;
- dev_err(&client->dev, "i2c test failed attempt %d: %d\n",
- retry, error);
msleep(20);
}
struct goodix_ts_data *ts = ctx;
int error;
- if (cfg) {
+ if (ts->firmware_name) {
+ if (!cfg)
+ goto err_release_cfg;
+
+ error = goodix_check_cfg(ts, cfg->data, cfg->size);
+ if (error)
+ goto err_release_cfg;
+
+ memcpy(ts->config, cfg->data, cfg->size);
+ } else if (cfg) {
/* send device configuration to the firmware */
error = goodix_send_cfg(ts, cfg->data, cfg->size);
if (error)
const struct i2c_device_id *id)
{
struct goodix_ts_data *ts;
+ const char *cfg_name;
int error;
dev_dbg(&client->dev, "I2C Address: 0x%02x\n", client->addr);
if (ts->reset_controller_at_probe) {
/* reset the controller */
error = goodix_reset(ts);
- if (error) {
- dev_err(&client->dev, "Controller reset failed.\n");
+ if (error)
return error;
- }
}
error = goodix_i2c_test(client);
return error;
}
+ error = goodix_firmware_check(ts);
+ if (error)
+ return error;
+
error = goodix_read_version(ts);
- if (error) {
- dev_err(&client->dev, "Read version failed.\n");
+ if (error)
return error;
- }
ts->chip = goodix_get_chip_data(ts->id);
if (ts->load_cfg_from_disk) {
/* update device config */
- ts->cfg_name = devm_kasprintf(&client->dev, GFP_KERNEL,
- "goodix_%s_cfg.bin", ts->id);
- if (!ts->cfg_name)
- return -ENOMEM;
+ error = device_property_read_string(&client->dev,
+ "goodix,config-name",
+ &cfg_name);
+ if (!error)
+ snprintf(ts->cfg_name, sizeof(ts->cfg_name),
+ "goodix/%s", cfg_name);
+ else
+ snprintf(ts->cfg_name, sizeof(ts->cfg_name),
+ "goodix_%s_cfg.bin", ts->id);
error = request_firmware_nowait(THIS_MODULE, true, ts->cfg_name,
&client->dev, GFP_KERNEL, ts,
/* Free IRQ as IRQ pin is used as output in the suspend sequence */
goodix_free_irq(ts);
+ /* Save reference (calibration) info if necessary */
+ goodix_save_bak_ref(ts);
+
/* Output LOW on the INT pin for 5 ms */
error = goodix_irq_direction_output(ts, 0);
if (error) {
error = goodix_i2c_write_u8(ts->client, GOODIX_REG_COMMAND,
GOODIX_CMD_SCREEN_OFF);
if (error) {
- dev_err(&ts->client->dev, "Screen off command failed\n");
goodix_irq_direction_input(ts);
goodix_request_irq(ts);
return -EAGAIN;
error = goodix_i2c_read(ts->client, ts->chip->config_addr,
&config_ver, 1);
- if (error)
- dev_warn(dev, "Error reading config version: %d, resetting controller\n",
- error);
- else if (config_ver != ts->config[0])
+ if (!error && config_ver != ts->config[0])
dev_info(dev, "Config version mismatch %d != %d, resetting controller\n",
config_ver, ts->config[0]);
if (error != 0 || config_ver != ts->config[0]) {
error = goodix_reset(ts);
- if (error) {
- dev_err(dev, "Controller reset failed.\n");
+ if (error)
return error;
- }
error = goodix_send_cfg(ts, ts->config, ts->chip->config_len);
if (error)
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __GOODIX_H__
+#define __GOODIX_H__
+
+#include <linux/gpio/consumer.h>
+#include <linux/i2c.h>
+#include <linux/input.h>
+#include <linux/input/mt.h>
+#include <linux/input/touchscreen.h>
+#include <linux/regulator/consumer.h>
+
+/* Register defines */
+#define GOODIX_REG_MISCTL_DSP_CTL 0x4010
+#define GOODIX_REG_MISCTL_SRAM_BANK 0x4048
+#define GOODIX_REG_MISCTL_MEM_CD_EN 0x4049
+#define GOODIX_REG_MISCTL_CACHE_EN 0x404B
+#define GOODIX_REG_MISCTL_TMR0_EN 0x40B0
+#define GOODIX_REG_MISCTL_SWRST 0x4180
+#define GOODIX_REG_MISCTL_CPU_SWRST_PULSE 0x4184
+#define GOODIX_REG_MISCTL_BOOTCTL 0x4190
+#define GOODIX_REG_MISCTL_BOOT_OPT 0x4218
+#define GOODIX_REG_MISCTL_BOOT_CTL 0x5094
+
+#define GOODIX_REG_FW_SIG 0x8000
+#define GOODIX_FW_SIG_LEN 10
+
+#define GOODIX_REG_MAIN_CLK 0x8020
+#define GOODIX_MAIN_CLK_LEN 6
+
+#define GOODIX_REG_COMMAND 0x8040
+#define GOODIX_CMD_SCREEN_OFF 0x05
+
+#define GOODIX_REG_SW_WDT 0x8041
+
+#define GOODIX_REG_REQUEST 0x8043
+#define GOODIX_RQST_RESPONDED 0x00
+#define GOODIX_RQST_CONFIG 0x01
+#define GOODIX_RQST_BAK_REF 0x02
+#define GOODIX_RQST_RESET 0x03
+#define GOODIX_RQST_MAIN_CLOCK 0x04
+/*
+ * Unknown request which gets send by the controller aprox.
+ * every 34 seconds once it is up and running.
+ */
+#define GOODIX_RQST_UNKNOWN 0x06
+#define GOODIX_RQST_IDLE 0xFF
+
+#define GOODIX_REG_STATUS 0x8044
+
+#define GOODIX_GT1X_REG_CONFIG_DATA 0x8050
+#define GOODIX_GT9X_REG_CONFIG_DATA 0x8047
+#define GOODIX_REG_ID 0x8140
+#define GOODIX_READ_COOR_ADDR 0x814E
+#define GOODIX_REG_BAK_REF 0x99D0
+
+#define GOODIX_ID_MAX_LEN 4
+#define GOODIX_CONFIG_MAX_LENGTH 240
+#define GOODIX_MAX_KEYS 7
+
+enum goodix_irq_pin_access_method {
+ IRQ_PIN_ACCESS_NONE,
+ IRQ_PIN_ACCESS_GPIO,
+ IRQ_PIN_ACCESS_ACPI_GPIO,
+ IRQ_PIN_ACCESS_ACPI_METHOD,
+};
+
+struct goodix_ts_data;
+
+struct goodix_chip_data {
+ u16 config_addr;
+ int config_len;
+ int (*check_config)(struct goodix_ts_data *ts, const u8 *cfg, int len);
+ void (*calc_config_checksum)(struct goodix_ts_data *ts);
+};
+
+struct goodix_ts_data {
+ struct i2c_client *client;
+ struct input_dev *input_dev;
+ const struct goodix_chip_data *chip;
+ const char *firmware_name;
+ struct touchscreen_properties prop;
+ unsigned int max_touch_num;
+ unsigned int int_trigger_type;
+ struct regulator *avdd28;
+ struct regulator *vddio;
+ struct gpio_desc *gpiod_int;
+ struct gpio_desc *gpiod_rst;
+ int gpio_count;
+ int gpio_int_idx;
+ char id[GOODIX_ID_MAX_LEN + 1];
+ char cfg_name[64];
+ u16 version;
+ bool reset_controller_at_probe;
+ bool load_cfg_from_disk;
+ struct completion firmware_loading_complete;
+ unsigned long irq_flags;
+ enum goodix_irq_pin_access_method irq_pin_access_method;
+ unsigned int contact_size;
+ u8 config[GOODIX_CONFIG_MAX_LENGTH];
+ unsigned short keymap[GOODIX_MAX_KEYS];
+ u8 main_clk[GOODIX_MAIN_CLK_LEN];
+ int bak_ref_len;
+ u8 *bak_ref;
+};
+
+int goodix_i2c_read(struct i2c_client *client, u16 reg, u8 *buf, int len);
+int goodix_i2c_write(struct i2c_client *client, u16 reg, const u8 *buf, int len);
+int goodix_i2c_write_u8(struct i2c_client *client, u16 reg, u8 value);
+int goodix_send_cfg(struct goodix_ts_data *ts, const u8 *cfg, int len);
+int goodix_int_sync(struct goodix_ts_data *ts);
+int goodix_reset_no_int_sync(struct goodix_ts_data *ts);
+
+int goodix_firmware_check(struct goodix_ts_data *ts);
+bool goodix_handle_fw_request(struct goodix_ts_data *ts);
+void goodix_save_bak_ref(struct goodix_ts_data *ts);
+
+#endif
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Goodix Touchscreen firmware upload support
+ *
+ * Copyright (c) 2021 Hans de Goede <hdegoede@redhat.com>
+ *
+ * This is a rewrite of gt9xx_update.c from the Allwinner H3 BSP which is:
+ * Copyright (c) 2010 - 2012 Goodix Technology.
+ * Author: andrew@goodix.com
+ */
+
+#include <linux/device.h>
+#include <linux/firmware.h>
+#include <linux/i2c.h>
+#include "goodix.h"
+
+#define GOODIX_FW_HEADER_LENGTH sizeof(struct goodix_fw_header)
+#define GOODIX_FW_SECTION_LENGTH 0x2000
+#define GOODIX_FW_DSP_LENGTH 0x1000
+#define GOODIX_FW_UPLOAD_ADDRESS 0xc000
+
+#define GOODIX_CFG_LOC_HAVE_KEY 7
+#define GOODIX_CFG_LOC_DRVA_NUM 27
+#define GOODIX_CFG_LOC_DRVB_NUM 28
+#define GOODIX_CFG_LOC_SENS_NUM 29
+
+struct goodix_fw_header {
+ u8 hw_info[4];
+ u8 pid[8];
+ u8 vid[2];
+} __packed;
+
+static u16 goodix_firmware_checksum(const u8 *data, int size)
+{
+ u16 checksum = 0;
+ int i;
+
+ for (i = 0; i < size; i += 2)
+ checksum += (data[i] << 8) + data[i + 1];
+
+ return checksum;
+}
+
+static int goodix_firmware_verify(struct device *dev, const struct firmware *fw)
+{
+ const struct goodix_fw_header *fw_header;
+ size_t expected_size;
+ const u8 *data;
+ u16 checksum;
+ char buf[9];
+
+ expected_size = GOODIX_FW_HEADER_LENGTH + 4 * GOODIX_FW_SECTION_LENGTH +
+ GOODIX_FW_DSP_LENGTH;
+ if (fw->size != expected_size) {
+ dev_err(dev, "Firmware has wrong size, expected %zu got %zu\n",
+ expected_size, fw->size);
+ return -EINVAL;
+ }
+
+ data = fw->data + GOODIX_FW_HEADER_LENGTH;
+ checksum = goodix_firmware_checksum(data, 4 * GOODIX_FW_SECTION_LENGTH);
+ if (checksum) {
+ dev_err(dev, "Main firmware checksum error\n");
+ return -EINVAL;
+ }
+
+ data += 4 * GOODIX_FW_SECTION_LENGTH;
+ checksum = goodix_firmware_checksum(data, GOODIX_FW_DSP_LENGTH);
+ if (checksum) {
+ dev_err(dev, "DSP firmware checksum error\n");
+ return -EINVAL;
+ }
+
+ fw_header = (const struct goodix_fw_header *)fw->data;
+ dev_info(dev, "Firmware hardware info %02x%02x%02x%02x\n",
+ fw_header->hw_info[0], fw_header->hw_info[1],
+ fw_header->hw_info[2], fw_header->hw_info[3]);
+ /* pid is a 8 byte buffer containing a string, weird I know */
+ memcpy(buf, fw_header->pid, 8);
+ buf[8] = 0;
+ dev_info(dev, "Firmware PID: %s VID: %02x%02x\n", buf,
+ fw_header->vid[0], fw_header->vid[1]);
+ return 0;
+}
+
+static int goodix_enter_upload_mode(struct i2c_client *client)
+{
+ int tries, error;
+ u8 val;
+
+ tries = 200;
+ do {
+ error = goodix_i2c_write_u8(client,
+ GOODIX_REG_MISCTL_SWRST, 0x0c);
+ if (error)
+ return error;
+
+ error = goodix_i2c_read(client,
+ GOODIX_REG_MISCTL_SWRST, &val, 1);
+ if (error)
+ return error;
+
+ if (val == 0x0c)
+ break;
+ } while (--tries);
+
+ if (!tries) {
+ dev_err(&client->dev, "Error could not hold ss51 & dsp\n");
+ return -EIO;
+ }
+
+ /* DSP_CK and DSP_ALU_CK PowerOn */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_DSP_CTL, 0x00);
+ if (error)
+ return error;
+
+ /* Disable watchdog */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_TMR0_EN, 0x00);
+ if (error)
+ return error;
+
+ /* Clear cache enable */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_CACHE_EN, 0x00);
+ if (error)
+ return error;
+
+ /* Set boot from SRAM */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_BOOTCTL, 0x02);
+ if (error)
+ return error;
+
+ /* Software reboot */
+ error = goodix_i2c_write_u8(client,
+ GOODIX_REG_MISCTL_CPU_SWRST_PULSE, 0x01);
+ if (error)
+ return error;
+
+ /* Clear control flag */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_BOOTCTL, 0x00);
+ if (error)
+ return error;
+
+ /* Set scramble */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_BOOT_OPT, 0x00);
+ if (error)
+ return error;
+
+ /* Enable accessing code */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_MEM_CD_EN, 0x01);
+ if (error)
+ return error;
+
+ return 0;
+}
+
+static int goodix_start_firmware(struct i2c_client *client)
+{
+ int error;
+ u8 val;
+
+ /* Init software watchdog */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_SW_WDT, 0xaa);
+ if (error)
+ return error;
+
+ /* Release SS51 & DSP */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_MISCTL_SWRST, 0x00);
+ if (error)
+ return error;
+
+ error = goodix_i2c_read(client, GOODIX_REG_SW_WDT, &val, 1);
+ if (error)
+ return error;
+
+ /* The value we've written to SW_WDT should have been cleared now */
+ if (val == 0xaa) {
+ dev_err(&client->dev, "Error SW_WDT reg not cleared on fw startup\n");
+ return -EIO;
+ }
+
+ /* Re-init software watchdog */
+ error = goodix_i2c_write_u8(client, GOODIX_REG_SW_WDT, 0xaa);
+ if (error)
+ return error;
+
+ return 0;
+}
+
+static int goodix_firmware_upload(struct goodix_ts_data *ts)
+{
+ const struct firmware *fw;
+ char fw_name[64];
+ const u8 *data;
+ int error;
+
+ snprintf(fw_name, sizeof(fw_name), "goodix/%s", ts->firmware_name);
+
+ error = request_firmware(&fw, fw_name, &ts->client->dev);
+ if (error) {
+ dev_err(&ts->client->dev, "Firmware request error %d\n", error);
+ return error;
+ }
+
+ error = goodix_firmware_verify(&ts->client->dev, fw);
+ if (error)
+ goto release;
+
+ error = goodix_reset_no_int_sync(ts);
+ if (error)
+ return error;
+
+ error = goodix_enter_upload_mode(ts->client);
+ if (error)
+ goto release;
+
+ /* Select SRAM bank 0 and upload section 1 & 2 */
+ error = goodix_i2c_write_u8(ts->client,
+ GOODIX_REG_MISCTL_SRAM_BANK, 0x00);
+ if (error)
+ goto release;
+
+ data = fw->data + GOODIX_FW_HEADER_LENGTH;
+ error = goodix_i2c_write(ts->client, GOODIX_FW_UPLOAD_ADDRESS,
+ data, 2 * GOODIX_FW_SECTION_LENGTH);
+ if (error)
+ goto release;
+
+ /* Select SRAM bank 1 and upload section 3 & 4 */
+ error = goodix_i2c_write_u8(ts->client,
+ GOODIX_REG_MISCTL_SRAM_BANK, 0x01);
+ if (error)
+ goto release;
+
+ data += 2 * GOODIX_FW_SECTION_LENGTH;
+ error = goodix_i2c_write(ts->client, GOODIX_FW_UPLOAD_ADDRESS,
+ data, 2 * GOODIX_FW_SECTION_LENGTH);
+ if (error)
+ goto release;
+
+ /* Select SRAM bank 2 and upload the DSP firmware */
+ error = goodix_i2c_write_u8(ts->client,
+ GOODIX_REG_MISCTL_SRAM_BANK, 0x02);
+ if (error)
+ goto release;
+
+ data += 2 * GOODIX_FW_SECTION_LENGTH;
+ error = goodix_i2c_write(ts->client, GOODIX_FW_UPLOAD_ADDRESS,
+ data, GOODIX_FW_DSP_LENGTH);
+ if (error)
+ goto release;
+
+ error = goodix_start_firmware(ts->client);
+ if (error)
+ goto release;
+
+ error = goodix_int_sync(ts);
+release:
+ release_firmware(fw);
+ return error;
+}
+
+static int goodix_prepare_bak_ref(struct goodix_ts_data *ts)
+{
+ u8 have_key, driver_num, sensor_num;
+
+ if (ts->bak_ref)
+ return 0; /* Already done */
+
+ have_key = (ts->config[GOODIX_CFG_LOC_HAVE_KEY] & 0x01);
+
+ driver_num = (ts->config[GOODIX_CFG_LOC_DRVA_NUM] & 0x1f) +
+ (ts->config[GOODIX_CFG_LOC_DRVB_NUM] & 0x1f);
+ if (have_key)
+ driver_num--;
+
+ sensor_num = (ts->config[GOODIX_CFG_LOC_SENS_NUM] & 0x0f) +
+ ((ts->config[GOODIX_CFG_LOC_SENS_NUM] >> 4) & 0x0f);
+
+ dev_dbg(&ts->client->dev, "Drv %d Sen %d Key %d\n",
+ driver_num, sensor_num, have_key);
+
+ ts->bak_ref_len = (driver_num * (sensor_num - 2) + 2) * 2;
+
+ ts->bak_ref = devm_kzalloc(&ts->client->dev,
+ ts->bak_ref_len, GFP_KERNEL);
+ if (!ts->bak_ref)
+ return -ENOMEM;
+
+ /*
+ * The bak_ref array contains the backup of an array of (self/auto)
+ * calibration related values which the Android version of the driver
+ * stores on the filesystem so that it can be restored after reboot.
+ * The mainline kernel never writes directly to the filesystem like
+ * this, we always start will all the values which give a correction
+ * factor in approx. the -20 - +20 range (in 2s complement) set to 0.
+ *
+ * Note the touchscreen works fine without restoring the reference
+ * values after a reboot / power-cycle.
+ *
+ * The last 2 bytes are a 16 bits unsigned checksum which is expected
+ * to make the addition al all 16 bit unsigned values in the array add
+ * up to 1 (rather then the usual 0), so we must set the last byte to 1.
+ */
+ ts->bak_ref[ts->bak_ref_len - 1] = 1;
+
+ return 0;
+}
+
+static int goodix_send_main_clock(struct goodix_ts_data *ts)
+{
+ u32 main_clk = 54; /* Default main clock */
+ u8 checksum = 0;
+ int i;
+
+ device_property_read_u32(&ts->client->dev,
+ "goodix,main-clk", &main_clk);
+
+ for (i = 0; i < (GOODIX_MAIN_CLK_LEN - 1); i++) {
+ ts->main_clk[i] = main_clk;
+ checksum += main_clk;
+ }
+
+ /* The value of all bytes combines must be 0 */
+ ts->main_clk[GOODIX_MAIN_CLK_LEN - 1] = 256 - checksum;
+
+ return goodix_i2c_write(ts->client, GOODIX_REG_MAIN_CLK,
+ ts->main_clk, GOODIX_MAIN_CLK_LEN);
+}
+
+int goodix_firmware_check(struct goodix_ts_data *ts)
+{
+ device_property_read_string(&ts->client->dev,
+ "firmware-name", &ts->firmware_name);
+ if (!ts->firmware_name)
+ return 0;
+
+ if (ts->irq_pin_access_method == IRQ_PIN_ACCESS_NONE) {
+ dev_err(&ts->client->dev, "Error no IRQ-pin access method, cannot upload fw.\n");
+ return -EINVAL;
+ }
+
+ dev_info(&ts->client->dev, "Touchscreen controller needs fw-upload\n");
+ ts->load_cfg_from_disk = true;
+
+ return goodix_firmware_upload(ts);
+}
+
+bool goodix_handle_fw_request(struct goodix_ts_data *ts)
+{
+ int error;
+ u8 val;
+
+ error = goodix_i2c_read(ts->client, GOODIX_REG_REQUEST, &val, 1);
+ if (error)
+ return false;
+
+ switch (val) {
+ case GOODIX_RQST_RESPONDED:
+ /*
+ * If we read back our own last ack the IRQ was not for
+ * a request.
+ */
+ return false;
+ case GOODIX_RQST_CONFIG:
+ error = goodix_send_cfg(ts, ts->config, ts->chip->config_len);
+ if (error)
+ return false;
+
+ break;
+ case GOODIX_RQST_BAK_REF:
+ error = goodix_prepare_bak_ref(ts);
+ if (error)
+ return false;
+
+ error = goodix_i2c_write(ts->client, GOODIX_REG_BAK_REF,
+ ts->bak_ref, ts->bak_ref_len);
+ if (error)
+ return false;
+
+ break;
+ case GOODIX_RQST_RESET:
+ error = goodix_firmware_upload(ts);
+ if (error)
+ return false;
+
+ break;
+ case GOODIX_RQST_MAIN_CLOCK:
+ error = goodix_send_main_clock(ts);
+ if (error)
+ return false;
+
+ break;
+ case GOODIX_RQST_UNKNOWN:
+ case GOODIX_RQST_IDLE:
+ break;
+ default:
+ dev_err_ratelimited(&ts->client->dev, "Unknown Request: 0x%02x\n", val);
+ }
+
+ /* Ack the request */
+ goodix_i2c_write_u8(ts->client,
+ GOODIX_REG_REQUEST, GOODIX_RQST_RESPONDED);
+ return true;
+}
+
+void goodix_save_bak_ref(struct goodix_ts_data *ts)
+{
+ int error;
+ u8 val;
+
+ if (!ts->firmware_name)
+ return;
+
+ error = goodix_i2c_read(ts->client, GOODIX_REG_STATUS, &val, 1);
+ if (error)
+ return;
+
+ if (!(val & 0x80))
+ return;
+
+ error = goodix_i2c_read(ts->client, GOODIX_REG_BAK_REF,
+ ts->bak_ref, ts->bak_ref_len);
+ if (error) {
+ memset(ts->bak_ref, 0, ts->bak_ref_len);
+ ts->bak_ref[ts->bak_ref_len - 1] = 1;
+ }
+}
// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/crc-ccitt.h>
#include <linux/delay.h>
#include <linux/gpio/consumer.h>
#include <linux/i2c.h>
+#include <linux/ihex.h>
#include <linux/input.h>
#include <linux/input/mt.h>
#include <linux/input/touchscreen.h>
#include <linux/slab.h>
#include <asm/unaligned.h>
-#define ILI2XXX_POLL_PERIOD 20
+#define ILI2XXX_POLL_PERIOD 15
#define ILI210X_DATA_SIZE 64
#define ILI211X_DATA_SIZE 43
/* Touchscreen commands */
#define REG_TOUCHDATA 0x10
#define REG_PANEL_INFO 0x20
+#define REG_FIRMWARE_VERSION 0x40
+#define REG_PROTOCOL_VERSION 0x42
+#define REG_KERNEL_VERSION 0x61
+#define REG_IC_BUSY 0x80
+#define REG_IC_BUSY_NOT_BUSY 0x50
+#define REG_GET_MODE 0xc0
+#define REG_GET_MODE_AP 0x5a
+#define REG_GET_MODE_BL 0x55
+#define REG_SET_MODE_AP 0xc1
+#define REG_SET_MODE_BL 0xc2
+#define REG_WRITE_DATA 0xc3
+#define REG_WRITE_ENABLE 0xc4
+#define REG_READ_DATA_CRC 0xc7
#define REG_CALIBRATE 0xcc
+#define ILI251X_FW_FILENAME "ilitek/ili251x.bin"
+
struct ili2xxx_chip {
int (*read_reg)(struct i2c_client *client, u8 reg,
void *buf, size_t len);
unsigned int max_touches;
unsigned int resolution;
bool has_calibrate_reg;
+ bool has_firmware_proto;
bool has_pressure_reg;
};
struct gpio_desc *reset_gpio;
struct touchscreen_properties prop;
const struct ili2xxx_chip *chip;
+ u8 version_firmware[8];
+ u8 version_kernel[5];
+ u8 version_proto[2];
+ u8 ic_mode[2];
bool stop;
};
.has_calibrate_reg = true,
};
-static int ili251x_read_reg(struct i2c_client *client,
- u8 reg, void *buf, size_t len)
+static int ili251x_read_reg_common(struct i2c_client *client,
+ u8 reg, void *buf, size_t len,
+ unsigned int delay)
{
int error;
int ret;
ret = i2c_master_send(client, ®, 1);
if (ret == 1) {
- usleep_range(5000, 5500);
+ if (delay)
+ usleep_range(delay, delay + 500);
ret = i2c_master_recv(client, buf, len);
if (ret == len)
return ret;
}
+static int ili251x_read_reg(struct i2c_client *client,
+ u8 reg, void *buf, size_t len)
+{
+ return ili251x_read_reg_common(client, reg, buf, len, 5000);
+}
+
static int ili251x_read_touch_data(struct i2c_client *client, u8 *data)
{
int error;
- error = ili251x_read_reg(client, REG_TOUCHDATA,
- data, ILI251X_DATA_SIZE1);
+ error = ili251x_read_reg_common(client, REG_TOUCHDATA,
+ data, ILI251X_DATA_SIZE1, 0);
if (!error && data[0] == 2) {
error = i2c_master_recv(client, data + ILI251X_DATA_SIZE1,
ILI251X_DATA_SIZE2);
.continue_polling = ili251x_check_continue_polling,
.max_touches = 10,
.has_calibrate_reg = true,
+ .has_firmware_proto = true,
.has_pressure_reg = true,
};
const struct ili2xxx_chip *chip = priv->chip;
u8 touchdata[ILI210X_DATA_SIZE] = { 0 };
bool keep_polling;
+ ktime_t time_next;
+ s64 time_delta;
bool touch;
int error;
do {
+ time_next = ktime_add_ms(ktime_get(), ILI2XXX_POLL_PERIOD);
error = chip->get_touch_data(client, touchdata);
if (error) {
dev_err(&client->dev,
touch = ili210x_report_events(priv, touchdata);
keep_polling = chip->continue_polling(touchdata, touch);
- if (keep_polling)
- msleep(ILI2XXX_POLL_PERIOD);
+ if (keep_polling) {
+ time_delta = ktime_us_delta(time_next, ktime_get());
+ if (time_delta > 0)
+ usleep_range(time_delta, time_delta + 1000);
+ }
} while (!priv->stop && keep_polling);
return IRQ_HANDLED;
}
+static int ili251x_firmware_update_resolution(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u16 resx, resy;
+ u8 rs[10];
+ int error;
+
+ /* The firmware update blob might have changed the resolution. */
+ error = priv->chip->read_reg(client, REG_PANEL_INFO, &rs, sizeof(rs));
+ if (error)
+ return error;
+
+ resx = le16_to_cpup((__le16 *)rs);
+ resy = le16_to_cpup((__le16 *)(rs + 2));
+
+ /* The value reported by the firmware is invalid. */
+ if (!resx || resx == 0xffff || !resy || resy == 0xffff)
+ return -EINVAL;
+
+ input_abs_set_max(priv->input, ABS_X, resx - 1);
+ input_abs_set_max(priv->input, ABS_Y, resy - 1);
+ input_abs_set_max(priv->input, ABS_MT_POSITION_X, resx - 1);
+ input_abs_set_max(priv->input, ABS_MT_POSITION_Y, resy - 1);
+
+ return 0;
+}
+
+static ssize_t ili251x_firmware_update_firmware_version(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error;
+ u8 fw[8];
+
+ /* Get firmware version */
+ error = priv->chip->read_reg(client, REG_FIRMWARE_VERSION,
+ &fw, sizeof(fw));
+ if (!error)
+ memcpy(priv->version_firmware, fw, sizeof(fw));
+
+ return error;
+}
+
+static ssize_t ili251x_firmware_update_kernel_version(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error;
+ u8 kv[5];
+
+ /* Get kernel version */
+ error = priv->chip->read_reg(client, REG_KERNEL_VERSION,
+ &kv, sizeof(kv));
+ if (!error)
+ memcpy(priv->version_kernel, kv, sizeof(kv));
+
+ return error;
+}
+
+static ssize_t ili251x_firmware_update_protocol_version(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error;
+ u8 pv[2];
+
+ /* Get protocol version */
+ error = priv->chip->read_reg(client, REG_PROTOCOL_VERSION,
+ &pv, sizeof(pv));
+ if (!error)
+ memcpy(priv->version_proto, pv, sizeof(pv));
+
+ return error;
+}
+
+static ssize_t ili251x_firmware_update_ic_mode(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error;
+ u8 md[2];
+
+ /* Get chip boot mode */
+ error = priv->chip->read_reg(client, REG_GET_MODE, &md, sizeof(md));
+ if (!error)
+ memcpy(priv->ic_mode, md, sizeof(md));
+
+ return error;
+}
+
+static int ili251x_firmware_update_cached_state(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error;
+
+ if (!priv->chip->has_firmware_proto)
+ return 0;
+
+ /* Wait for firmware to boot and stabilize itself. */
+ msleep(200);
+
+ /* Firmware does report valid information. */
+ error = ili251x_firmware_update_resolution(dev);
+ if (error)
+ return error;
+
+ error = ili251x_firmware_update_firmware_version(dev);
+ if (error)
+ return error;
+
+ error = ili251x_firmware_update_kernel_version(dev);
+ if (error)
+ return error;
+
+ error = ili251x_firmware_update_protocol_version(dev);
+ if (error)
+ return error;
+
+ error = ili251x_firmware_update_ic_mode(dev);
+ if (error)
+ return error;
+
+ return 0;
+}
+
+static ssize_t ili251x_firmware_version_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 *fw = priv->version_firmware;
+
+ return sysfs_emit(buf, "%02x%02x.%02x%02x.%02x%02x.%02x%02x\n",
+ fw[0], fw[1], fw[2], fw[3],
+ fw[4], fw[5], fw[6], fw[7]);
+}
+static DEVICE_ATTR(firmware_version, 0444, ili251x_firmware_version_show, NULL);
+
+static ssize_t ili251x_kernel_version_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 *kv = priv->version_kernel;
+
+ return sysfs_emit(buf, "%02x.%02x.%02x.%02x.%02x\n",
+ kv[0], kv[1], kv[2], kv[3], kv[4]);
+}
+static DEVICE_ATTR(kernel_version, 0444, ili251x_kernel_version_show, NULL);
+
+static ssize_t ili251x_protocol_version_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 *pv = priv->version_proto;
+
+ return sysfs_emit(buf, "%02x.%02x\n", pv[0], pv[1]);
+}
+static DEVICE_ATTR(protocol_version, 0444, ili251x_protocol_version_show, NULL);
+
+static ssize_t ili251x_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 *md = priv->ic_mode;
+ char *mode = "AP";
+
+ if (md[0] == REG_GET_MODE_AP) /* Application Mode */
+ mode = "AP";
+ else if (md[0] == REG_GET_MODE_BL) /* BootLoader Mode */
+ mode = "BL";
+ else /* Unknown Mode */
+ mode = "??";
+
+ return sysfs_emit(buf, "%02x.%02x:%s\n", md[0], md[1], mode);
+}
+static DEVICE_ATTR(mode, 0444, ili251x_mode_show, NULL);
+
static ssize_t ili210x_calibrate(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
}
static DEVICE_ATTR(calibrate, S_IWUSR, NULL, ili210x_calibrate);
+static int ili251x_firmware_to_buffer(const struct firmware *fw,
+ u8 **buf, u16 *ac_end, u16 *df_end)
+{
+ const struct ihex_binrec *rec;
+ u32 fw_addr, fw_last_addr = 0;
+ u16 fw_len;
+ u8 *fw_buf;
+ int error;
+
+ /*
+ * The firmware ihex blob can never be bigger than 64 kiB, so make this
+ * simple -- allocate a 64 kiB buffer, iterate over the ihex blob records
+ * once, copy them all into this buffer at the right locations, and then
+ * do all operations on this linear buffer.
+ */
+ fw_buf = kzalloc(SZ_64K, GFP_KERNEL);
+ if (!fw_buf)
+ return -ENOMEM;
+
+ rec = (const struct ihex_binrec *)fw->data;
+ while (rec) {
+ fw_addr = be32_to_cpu(rec->addr);
+ fw_len = be16_to_cpu(rec->len);
+
+ /* The last 32 Byte firmware block can be 0xffe0 */
+ if (fw_addr + fw_len > SZ_64K || fw_addr > SZ_64K - 32) {
+ error = -EFBIG;
+ goto err_big;
+ }
+
+ /* Find the last address before DF start address, that is AC end */
+ if (fw_addr == 0xf000)
+ *ac_end = fw_last_addr;
+ fw_last_addr = fw_addr + fw_len;
+
+ memcpy(fw_buf + fw_addr, rec->data, fw_len);
+ rec = ihex_next_binrec(rec);
+ }
+
+ /* DF end address is the last address in the firmware blob */
+ *df_end = fw_addr + fw_len;
+ *buf = fw_buf;
+ return 0;
+
+err_big:
+ kfree(fw_buf);
+ return error;
+}
+
+/* Switch mode between Application and BootLoader */
+static int ili251x_switch_ic_mode(struct i2c_client *client, u8 cmd_mode)
+{
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 cmd_wren[3] = { REG_WRITE_ENABLE, 0x5a, 0xa5 };
+ u8 md[2];
+ int error;
+
+ error = priv->chip->read_reg(client, REG_GET_MODE, md, sizeof(md));
+ if (error)
+ return error;
+ /* Mode already set */
+ if ((cmd_mode == REG_SET_MODE_AP && md[0] == REG_GET_MODE_AP) ||
+ (cmd_mode == REG_SET_MODE_BL && md[0] == REG_GET_MODE_BL))
+ return 0;
+
+ /* Unlock writes */
+ error = i2c_master_send(client, cmd_wren, sizeof(cmd_wren));
+ if (error != sizeof(cmd_wren))
+ return -EINVAL;
+
+ mdelay(20);
+
+ /* Select mode (BootLoader or Application) */
+ error = i2c_master_send(client, &cmd_mode, 1);
+ if (error != 1)
+ return -EINVAL;
+
+ mdelay(200); /* Reboot into bootloader takes a lot of time ... */
+
+ /* Read back mode */
+ error = priv->chip->read_reg(client, REG_GET_MODE, md, sizeof(md));
+ if (error)
+ return error;
+ /* Check if mode is correct now. */
+ if ((cmd_mode == REG_SET_MODE_AP && md[0] == REG_GET_MODE_AP) ||
+ (cmd_mode == REG_SET_MODE_BL && md[0] == REG_GET_MODE_BL))
+ return 0;
+
+ return -EINVAL;
+}
+
+static int ili251x_firmware_busy(struct i2c_client *client)
+{
+ struct ili210x *priv = i2c_get_clientdata(client);
+ int error, i = 0;
+ u8 data;
+
+ do {
+ /* The read_reg already contains suitable delay */
+ error = priv->chip->read_reg(client, REG_IC_BUSY, &data, 1);
+ if (error)
+ return error;
+ if (i++ == 100000)
+ return -ETIMEDOUT;
+ } while (data != REG_IC_BUSY_NOT_BUSY);
+
+ return 0;
+}
+
+static int ili251x_firmware_write_to_ic(struct device *dev, u8 *fwbuf,
+ u16 start, u16 end, u8 dataflash)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+ u8 cmd_crc = REG_READ_DATA_CRC;
+ u8 crcrb[4] = { 0 };
+ u8 fw_data[33];
+ u16 fw_addr;
+ int error;
+
+ /*
+ * The DF (dataflash) needs 2 bytes offset for unknown reasons,
+ * the AC (application) has 2 bytes CRC16-CCITT at the end.
+ */
+ u16 crc = crc_ccitt(0, fwbuf + start + (dataflash ? 2 : 0),
+ end - start - 2);
+
+ /* Unlock write to either AC (application) or DF (dataflash) area */
+ u8 cmd_wr[10] = {
+ REG_WRITE_ENABLE, 0x5a, 0xa5, dataflash,
+ (end >> 16) & 0xff, (end >> 8) & 0xff, end & 0xff,
+ (crc >> 16) & 0xff, (crc >> 8) & 0xff, crc & 0xff
+ };
+
+ error = i2c_master_send(client, cmd_wr, sizeof(cmd_wr));
+ if (error != sizeof(cmd_wr))
+ return -EINVAL;
+
+ error = ili251x_firmware_busy(client);
+ if (error)
+ return error;
+
+ for (fw_addr = start; fw_addr < end; fw_addr += 32) {
+ fw_data[0] = REG_WRITE_DATA;
+ memcpy(&(fw_data[1]), fwbuf + fw_addr, 32);
+ error = i2c_master_send(client, fw_data, 33);
+ if (error != sizeof(fw_data))
+ return error;
+ error = ili251x_firmware_busy(client);
+ if (error)
+ return error;
+ }
+
+ error = i2c_master_send(client, &cmd_crc, 1);
+ if (error != 1)
+ return -EINVAL;
+
+ error = ili251x_firmware_busy(client);
+ if (error)
+ return error;
+
+ error = priv->chip->read_reg(client, REG_READ_DATA_CRC,
+ &crcrb, sizeof(crcrb));
+ if (error)
+ return error;
+
+ /* Check CRC readback */
+ if ((crcrb[0] != (crc & 0xff)) || crcrb[1] != ((crc >> 8) & 0xff))
+ return -EINVAL;
+
+ return 0;
+}
+
+static int ili251x_firmware_reset(struct i2c_client *client)
+{
+ u8 cmd_reset[2] = { 0xf2, 0x01 };
+ int error;
+
+ error = i2c_master_send(client, cmd_reset, sizeof(cmd_reset));
+ if (error != sizeof(cmd_reset))
+ return -EINVAL;
+
+ return ili251x_firmware_busy(client);
+}
+
+static void ili251x_hardware_reset(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct ili210x *priv = i2c_get_clientdata(client);
+
+ /* Reset the controller */
+ gpiod_set_value_cansleep(priv->reset_gpio, 1);
+ usleep_range(10000, 15000);
+ gpiod_set_value_cansleep(priv->reset_gpio, 0);
+ msleep(300);
+}
+
+static ssize_t ili210x_firmware_update_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ const char *fwname = ILI251X_FW_FILENAME;
+ const struct firmware *fw;
+ u16 ac_end, df_end;
+ u8 *fwbuf;
+ int error;
+ int i;
+
+ error = request_ihex_firmware(&fw, fwname, dev);
+ if (error) {
+ dev_err(dev, "Failed to request firmware %s, error=%d\n",
+ fwname, error);
+ return error;
+ }
+
+ error = ili251x_firmware_to_buffer(fw, &fwbuf, &ac_end, &df_end);
+ release_firmware(fw);
+ if (error)
+ return error;
+
+ /*
+ * Disable touchscreen IRQ, so that we would not get spurious touch
+ * interrupt during firmware update, and so that the IRQ handler won't
+ * trigger and interfere with the firmware update. There is no bit in
+ * the touch controller to disable the IRQs during update, so we have
+ * to do it this way here.
+ */
+ disable_irq(client->irq);
+
+ dev_dbg(dev, "Firmware update started, firmware=%s\n", fwname);
+
+ ili251x_hardware_reset(dev);
+
+ error = ili251x_firmware_reset(client);
+ if (error)
+ goto exit;
+
+ /* This may not succeed on first try, so re-try a few times. */
+ for (i = 0; i < 5; i++) {
+ error = ili251x_switch_ic_mode(client, REG_SET_MODE_BL);
+ if (!error)
+ break;
+ }
+
+ if (error)
+ goto exit;
+
+ dev_dbg(dev, "IC is now in BootLoader mode\n");
+
+ msleep(200); /* The bootloader seems to need some time too. */
+
+ error = ili251x_firmware_write_to_ic(dev, fwbuf, 0xf000, df_end, 1);
+ if (error) {
+ dev_err(dev, "DF firmware update failed, error=%d\n", error);
+ goto exit;
+ }
+
+ dev_dbg(dev, "DataFlash firmware written\n");
+
+ error = ili251x_firmware_write_to_ic(dev, fwbuf, 0x2000, ac_end, 0);
+ if (error) {
+ dev_err(dev, "AC firmware update failed, error=%d\n", error);
+ goto exit;
+ }
+
+ dev_dbg(dev, "Application firmware written\n");
+
+ /* This may not succeed on first try, so re-try a few times. */
+ for (i = 0; i < 5; i++) {
+ error = ili251x_switch_ic_mode(client, REG_SET_MODE_AP);
+ if (!error)
+ break;
+ }
+
+ if (error)
+ goto exit;
+
+ dev_dbg(dev, "IC is now in Application mode\n");
+
+ error = ili251x_firmware_update_cached_state(dev);
+ if (error)
+ goto exit;
+
+ error = count;
+
+exit:
+ ili251x_hardware_reset(dev);
+ dev_dbg(dev, "Firmware update ended, error=%i\n", error);
+ enable_irq(client->irq);
+ kfree(fwbuf);
+ return error;
+}
+
+static DEVICE_ATTR(firmware_update, 0200, NULL, ili210x_firmware_update_store);
+
static struct attribute *ili210x_attributes[] = {
&dev_attr_calibrate.attr,
+ &dev_attr_firmware_update.attr,
+ &dev_attr_firmware_version.attr,
+ &dev_attr_kernel_version.attr,
+ &dev_attr_protocol_version.attr,
+ &dev_attr_mode.attr,
NULL,
};
-static umode_t ili210x_calibrate_visible(struct kobject *kobj,
+static umode_t ili210x_attributes_visible(struct kobject *kobj,
struct attribute *attr, int index)
{
struct device *dev = kobj_to_dev(kobj);
struct i2c_client *client = to_i2c_client(dev);
struct ili210x *priv = i2c_get_clientdata(client);
- return priv->chip->has_calibrate_reg ? attr->mode : 0;
+ /* Calibrate is present on all ILI2xxx which have calibrate register */
+ if (attr == &dev_attr_calibrate.attr)
+ return priv->chip->has_calibrate_reg ? attr->mode : 0;
+
+ /* Firmware/Kernel/Protocol/BootMode is implememted only for ILI251x */
+ if (!priv->chip->has_firmware_proto)
+ return 0;
+
+ return attr->mode;
}
static const struct attribute_group ili210x_attr_group = {
.attrs = ili210x_attributes,
- .is_visible = ili210x_calibrate_visible,
+ .is_visible = ili210x_attributes_visible,
};
static void ili210x_power_down(void *data)
input_set_abs_params(input, ABS_MT_POSITION_Y, 0, max_xy, 0, 0);
if (priv->chip->has_pressure_reg)
input_set_abs_params(input, ABS_MT_PRESSURE, 0, 0xa, 0, 0);
+ error = ili251x_firmware_update_cached_state(dev);
+ if (error) {
+ dev_err(dev, "Unable to cache firmware information, err: %d\n",
+ error);
+ return error;
+ }
touchscreen_parse_properties(input, true, &priv->prop);
error = input_mt_init_slots(input, priv->chip->max_touches,
#define RM_CMD_BOOT_READ 0x44 /* send wait bl data ready*/
#define RM_BOOT_RDY 0xFF /* bl data ready */
+#define RM_BOOT_CMD_READHWID 0x0E /* read hwid */
/* I2C main commands */
#define RM_CMD_QUERY_BANK 0x2B
return 0;
}
+static int raydium_i2c_query_ts_bootloader_info(struct raydium_data *ts)
+{
+ struct i2c_client *client = ts->client;
+ static const u8 get_hwid[] = { RM_BOOT_CMD_READHWID,
+ 0x10, 0xc0, 0x01, 0x00, 0x04, 0x00 };
+ u8 rbuf[5] = { 0 };
+ u32 hw_ver;
+ int error;
+
+ error = raydium_i2c_send(client, RM_CMD_BOOT_WRT,
+ get_hwid, sizeof(get_hwid));
+ if (error) {
+ dev_err(&client->dev, "WRT HWID command failed: %d\n", error);
+ return error;
+ }
+
+ error = raydium_i2c_send(client, RM_CMD_BOOT_ACK, rbuf, 1);
+ if (error) {
+ dev_err(&client->dev, "Ack HWID command failed: %d\n", error);
+ return error;
+ }
+
+ error = raydium_i2c_read(client, RM_CMD_BOOT_CHK, rbuf, sizeof(rbuf));
+ if (error) {
+ dev_err(&client->dev, "Read HWID command failed: %d (%4ph)\n",
+ error, rbuf + 1);
+ hw_ver = 0xffffffffUL;
+ } else {
+ hw_ver = get_unaligned_be32(rbuf + 1);
+ }
+
+ ts->info.hw_ver = cpu_to_le32(hw_ver);
+ ts->info.main_ver = 0xff;
+ ts->info.sub_ver = 0xff;
+
+ return error;
+}
+
static int raydium_i2c_query_ts_info(struct raydium_data *ts)
{
struct i2c_client *client = ts->client;
if (error)
ts->boot_mode = RAYDIUM_TS_BLDR;
- if (ts->boot_mode == RAYDIUM_TS_BLDR) {
- ts->info.hw_ver = cpu_to_le32(0xffffffffUL);
- ts->info.main_ver = 0xff;
- ts->info.sub_ver = 0xff;
- } else {
+ if (ts->boot_mode == RAYDIUM_TS_BLDR)
+ raydium_i2c_query_ts_bootloader_info(ts);
+ else
raydium_i2c_query_ts_info(ts);
- }
return error;
}
if (error)
return error;
- error = devm_add_action(&client->dev, raydium_i2c_power_off, ts);
+ error = devm_add_action_or_reset(&client->dev,
+ raydium_i2c_power_off, ts);
if (error) {
dev_err(&client->dev,
"failed to install power off action: %d\n", error);
- raydium_i2c_power_off(ts);
return error;
}
raydium_i2c_suspend, raydium_i2c_resume);
static const struct i2c_device_id raydium_i2c_id[] = {
- { "raydium_i2c" , 0 },
+ { "raydium_i2c", 0 },
{ "rm32380", 0 },
{ /* sentinel */ }
};
unsigned int retries;
int error;
- for (retries = 10; retries; retries--) {
+ for (retries = 100; retries; retries--) {
error = st1232_ts_read_data(ts, REG_STATUS, 1);
if (!error) {
switch (ts->read_buf[0]) {
.driver = {
.name = ST1232_TS_NAME,
.of_match_table = st1232_ts_dt_ids,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
.pm = &st1232_ts_pm_ops,
},
};
static int tsc2004_remove(struct i2c_client *i2c)
{
- return tsc200x_remove(&i2c->dev);
+ tsc200x_remove(&i2c->dev);
+
+ return 0;
}
static const struct i2c_device_id tsc2004_idtable[] = {
static int tsc2005_remove(struct spi_device *spi)
{
- return tsc200x_remove(&spi->dev);
+ tsc200x_remove(&spi->dev);
+
+ return 0;
}
#ifdef CONFIG_OF
}
EXPORT_SYMBOL_GPL(tsc200x_probe);
-int tsc200x_remove(struct device *dev)
+void tsc200x_remove(struct device *dev)
{
struct tsc200x *ts = dev_get_drvdata(dev);
sysfs_remove_group(&dev->kobj, &tsc200x_attr_group);
regulator_disable(ts->vio);
-
- return 0;
}
EXPORT_SYMBOL_GPL(tsc200x_remove);
int tsc200x_probe(struct device *dev, int irq, const struct input_id *tsc_id,
struct regmap *regmap,
int (*tsc200x_cmd)(struct device *dev, u8 cmd));
-int tsc200x_remove(struct device *dev);
+void tsc200x_remove(struct device *dev);
#endif
* <tobita.tatsunosuke@wacom.co.jp>
*/
+#include <linux/bits.h>
#include <linux/module.h>
#include <linux/input.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
#include <asm/unaligned.h>
+/* Bitmasks (for data[3]) */
+#define WACOM_TIP_SWITCH BIT(0)
+#define WACOM_BARREL_SWITCH BIT(1)
+#define WACOM_ERASER BIT(2)
+#define WACOM_INVERT BIT(3)
+#define WACOM_BARREL_SWITCH_2 BIT(4)
+#define WACOM_IN_PROXIMITY BIT(5)
+
+/* Registers */
#define WACOM_CMD_QUERY0 0x04
#define WACOM_CMD_QUERY1 0x00
#define WACOM_CMD_QUERY2 0x33
if (error < 0)
goto out;
- tsw = data[3] & 0x01;
- ers = data[3] & 0x04;
- f1 = data[3] & 0x02;
- f2 = data[3] & 0x10;
+ tsw = data[3] & WACOM_TIP_SWITCH;
+ ers = data[3] & WACOM_ERASER;
+ f1 = data[3] & WACOM_BARREL_SWITCH;
+ f2 = data[3] & WACOM_BARREL_SWITCH_2;
x = le16_to_cpup((__le16 *)&data[4]);
y = le16_to_cpup((__le16 *)&data[6]);
pressure = le16_to_cpup((__le16 *)&data[8]);
if (!wac_i2c->prox)
- wac_i2c->tool = (data[3] & 0x0c) ?
+ wac_i2c->tool = (data[3] & (WACOM_ERASER | WACOM_INVERT)) ?
BTN_TOOL_RUBBER : BTN_TOOL_PEN;
- wac_i2c->prox = data[3] & 0x20;
+ wac_i2c->prox = data[3] & WACOM_IN_PROXIMITY;
input_report_key(input, BTN_TOUCH, tsw || ers);
input_report_key(input, wac_i2c->tool, wac_i2c->prox);
#include <linux/bitfield.h>
#include <linux/clk.h>
#include <linux/dev_printk.h>
+#include <linux/dma-iommu.h>
#include <linux/dma-mapping.h>
#include <linux/err.h>
#include <linux/interrupt.h>
return 0;
}
+#ifndef CONFIG_PCIE_APPLE_MSI_DOORBELL_ADDR
+/* Keep things compiling when CONFIG_PCI_APPLE isn't selected */
+#define CONFIG_PCIE_APPLE_MSI_DOORBELL_ADDR 0
+#endif
+#define DOORBELL_ADDR (CONFIG_PCIE_APPLE_MSI_DOORBELL_ADDR & PAGE_MASK)
+
+static void apple_dart_get_resv_regions(struct device *dev,
+ struct list_head *head)
+{
+ if (IS_ENABLED(CONFIG_PCIE_APPLE) && dev_is_pci(dev)) {
+ struct iommu_resv_region *region;
+ int prot = IOMMU_WRITE | IOMMU_NOEXEC | IOMMU_MMIO;
+
+ region = iommu_alloc_resv_region(DOORBELL_ADDR,
+ PAGE_SIZE, prot,
+ IOMMU_RESV_MSI);
+ if (!region)
+ return;
+
+ list_add_tail(®ion->list, head);
+ }
+
+ iommu_dma_get_resv_regions(dev, head);
+}
+
static const struct iommu_ops apple_dart_iommu_ops = {
.domain_alloc = apple_dart_domain_alloc,
.domain_free = apple_dart_domain_free,
.device_group = apple_dart_device_group,
.of_xlate = apple_dart_of_xlate,
.def_domain_type = apple_dart_def_domain_type,
+ .get_resv_regions = apple_dart_get_resv_regions,
+ .put_resv_regions = generic_iommu_put_resv_regions,
.pgsize_bitmap = -1UL, /* Restricted during dart probe */
};
readl_relaxed(reg_base + INTCL_RDYIR));
}
-static void csky_mpintc_enable(struct irq_data *d)
+static void csky_mpintc_unmask(struct irq_data *d)
{
void __iomem *reg_base = this_cpu_read(intcl_reg);
writel_relaxed(d->hwirq, reg_base + INTCL_SENR);
}
-static void csky_mpintc_disable(struct irq_data *d)
+static void csky_mpintc_mask(struct irq_data *d)
{
void __iomem *reg_base = this_cpu_read(intcl_reg);
static struct irq_chip csky_irq_chip = {
.name = "C-SKY SMP Intc",
.irq_eoi = csky_mpintc_eoi,
- .irq_enable = csky_mpintc_enable,
- .irq_disable = csky_mpintc_disable,
+ .irq_unmask = csky_mpintc_unmask,
+ .irq_mask = csky_mpintc_mask,
.irq_set_type = csky_mpintc_set_type,
#ifdef CONFIG_SMP
.irq_set_affinity = csky_irq_set_affinity,
{
struct plic_handler *handler = this_cpu_ptr(&plic_handlers);
- writel(d->hwirq, handler->hart_base + CONTEXT_CLAIM);
+ if (irqd_irq_masked(d)) {
+ plic_irq_unmask(d);
+ writel(d->hwirq, handler->hart_base + CONTEXT_CLAIM);
+ plic_irq_mask(d);
+ } else {
+ writel(d->hwirq, handler->hart_base + CONTEXT_CLAIM);
+ }
}
static struct irq_chip plic_chip = {
fail_db_node:
of_node_put(smu->db_node);
fail_bootmem:
- memblock_free_ptr(smu, sizeof(struct smu_device));
+ memblock_free(smu, sizeof(struct smu_device));
smu = NULL;
fail_np:
of_node_put(np);
select CRYPTO
select CRYPTO_SKCIPHER
select ASYNC_XOR
+ select DM_AUDIT if AUDIT
help
This device-mapper target emulates a block device that has
additional per-sector tags that can be used for storing
If unsure, say N.
+config DM_AUDIT
+ bool "DM audit events"
+ depends on AUDIT
+ help
+ Generate audit events for device-mapper.
+
+ Enables audit logging of several security relevant events in the
+ particular device-mapper targets, especially the integrity target.
+
endif # MD
ifeq ($(CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG),y)
dm-verity-objs += dm-verity-verify-sig.o
endif
+
+ifeq ($(CONFIG_DM_AUDIT),y)
+dm-mod-objs += dm-audit.o
+endif
struct bvec_iter_all iter_all;
bio_for_each_segment_all(bv, b->bio, iter_all) {
- memcpy(bvec_virt(bv), addr, PAGE_SIZE);
+ memcpy(page_address(bv->bv_page), addr, PAGE_SIZE);
addr += PAGE_SIZE;
}
bcache_device_detach(d);
if (disk) {
- blk_cleanup_disk(disk);
ida_simple_remove(&bcache_device_idx,
first_minor_to_idx(disk->first_minor));
+ blk_cleanup_disk(disk);
}
bioset_exit(&d->bio_split);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Creating audit records for mapped devices.
+ *
+ * Copyright (C) 2021 Fraunhofer AISEC. All rights reserved.
+ *
+ * Authors: Michael Weiß <michael.weiss@aisec.fraunhofer.de>
+ */
+
+#include <linux/audit.h>
+#include <linux/module.h>
+#include <linux/device-mapper.h>
+#include <linux/bio.h>
+#include <linux/blkdev.h>
+
+#include "dm-audit.h"
+#include "dm-core.h"
+
+static struct audit_buffer *dm_audit_log_start(int audit_type,
+ const char *dm_msg_prefix,
+ const char *op)
+{
+ struct audit_buffer *ab;
+
+ if (audit_enabled == AUDIT_OFF)
+ return NULL;
+
+ ab = audit_log_start(audit_context(), GFP_KERNEL, audit_type);
+ if (unlikely(!ab))
+ return NULL;
+
+ audit_log_format(ab, "module=%s op=%s", dm_msg_prefix, op);
+ return ab;
+}
+
+void dm_audit_log_ti(int audit_type, const char *dm_msg_prefix, const char *op,
+ struct dm_target *ti, int result)
+{
+ struct audit_buffer *ab = NULL;
+ struct mapped_device *md = dm_table_get_md(ti->table);
+ int dev_major = dm_disk(md)->major;
+ int dev_minor = dm_disk(md)->first_minor;
+
+ switch (audit_type) {
+ case AUDIT_DM_CTRL:
+ ab = dm_audit_log_start(audit_type, dm_msg_prefix, op);
+ if (unlikely(!ab))
+ return;
+ audit_log_task_info(ab);
+ audit_log_format(ab, " dev=%d:%d error_msg='%s'", dev_major,
+ dev_minor, !result ? ti->error : "success");
+ break;
+ case AUDIT_DM_EVENT:
+ ab = dm_audit_log_start(audit_type, dm_msg_prefix, op);
+ if (unlikely(!ab))
+ return;
+ audit_log_format(ab, " dev=%d:%d sector=?", dev_major,
+ dev_minor);
+ break;
+ default: /* unintended use */
+ return;
+ }
+
+ audit_log_format(ab, " res=%d", result);
+ audit_log_end(ab);
+}
+EXPORT_SYMBOL_GPL(dm_audit_log_ti);
+
+void dm_audit_log_bio(const char *dm_msg_prefix, const char *op,
+ struct bio *bio, sector_t sector, int result)
+{
+ struct audit_buffer *ab;
+ int dev_major = MAJOR(bio->bi_bdev->bd_dev);
+ int dev_minor = MINOR(bio->bi_bdev->bd_dev);
+
+ ab = dm_audit_log_start(AUDIT_DM_EVENT, dm_msg_prefix, op);
+ if (unlikely(!ab))
+ return;
+
+ audit_log_format(ab, " dev=%d:%d sector=%llu res=%d",
+ dev_major, dev_minor, sector, result);
+ audit_log_end(ab);
+}
+EXPORT_SYMBOL_GPL(dm_audit_log_bio);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Creating audit records for mapped devices.
+ *
+ * Copyright (C) 2021 Fraunhofer AISEC. All rights reserved.
+ *
+ * Authors: Michael Weiß <michael.weiss@aisec.fraunhofer.de>
+ */
+
+#ifndef DM_AUDIT_H
+#define DM_AUDIT_H
+
+#include <linux/device-mapper.h>
+#include <linux/audit.h>
+
+#ifdef CONFIG_DM_AUDIT
+void dm_audit_log_bio(const char *dm_msg_prefix, const char *op,
+ struct bio *bio, sector_t sector, int result);
+
+/*
+ * dm_audit_log_ti() is not intended to be used directly in dm modules,
+ * the wrapper functions below should be called by dm modules instead.
+ */
+void dm_audit_log_ti(int audit_type, const char *dm_msg_prefix, const char *op,
+ struct dm_target *ti, int result);
+
+static inline void dm_audit_log_ctr(const char *dm_msg_prefix,
+ struct dm_target *ti, int result)
+{
+ dm_audit_log_ti(AUDIT_DM_CTRL, dm_msg_prefix, "ctr", ti, result);
+}
+
+static inline void dm_audit_log_dtr(const char *dm_msg_prefix,
+ struct dm_target *ti, int result)
+{
+ dm_audit_log_ti(AUDIT_DM_CTRL, dm_msg_prefix, "dtr", ti, result);
+}
+
+static inline void dm_audit_log_target(const char *dm_msg_prefix, const char *op,
+ struct dm_target *ti, int result)
+{
+ dm_audit_log_ti(AUDIT_DM_EVENT, dm_msg_prefix, op, ti, result);
+}
+#else
+static inline void dm_audit_log_bio(const char *dm_msg_prefix, const char *op,
+ struct bio *bio, sector_t sector,
+ int result)
+{
+}
+static inline void dm_audit_log_target(const char *dm_msg_prefix,
+ const char *op, struct dm_target *ti,
+ int result)
+{
+}
+static inline void dm_audit_log_ctr(const char *dm_msg_prefix,
+ struct dm_target *ti, int result)
+{
+}
+
+static inline void dm_audit_log_dtr(const char *dm_msg_prefix,
+ struct dm_target *ti, int result)
+{
+}
+#endif
+
+#endif
int bug = 0;
cancel_delayed_work_sync(&dm_bufio_cleanup_old_work);
- flush_workqueue(dm_bufio_wq);
destroy_workqueue(dm_bufio_wq);
if (dm_bufio_client_count) {
#include <linux/device-mapper.h>
+#include "dm-audit.h"
+
#define DM_MSG_PREFIX "crypt"
/*
if (r == -EBADMSG) {
char b[BDEVNAME_SIZE];
- DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu", bio_devname(ctx->bio_in, b),
- (unsigned long long)le64_to_cpu(*sector));
+ sector_t s = le64_to_cpu(*sector);
+
+ DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu",
+ bio_devname(ctx->bio_in, b), s);
+ dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",
+ ctx->bio_in, s, 0);
}
if (!r && cc->iv_gen_ops && cc->iv_gen_ops->post)
if (error == -EBADMSG) {
char b[BDEVNAME_SIZE];
- DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu", bio_devname(ctx->bio_in, b),
- (unsigned long long)le64_to_cpu(*org_sector_of_dmreq(cc, dmreq)));
+ sector_t s = le64_to_cpu(*org_sector_of_dmreq(cc, dmreq));
+
+ DMERR_LIMIT("%s: INTEGRITY AEAD ERROR, sector %llu",
+ bio_devname(ctx->bio_in, b), s);
+ dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead",
+ ctx->bio_in, s, 0);
io->error = BLK_STS_PROTECTION;
} else if (error < 0)
io->error = BLK_STS_IOERR;
dm_crypt_clients_n--;
crypt_calculate_pages_per_client();
spin_unlock(&dm_crypt_clients_lock);
+
+ dm_audit_log_dtr(DM_MSG_PREFIX, ti, 1);
}
static int crypt_ctr_ivmode(struct dm_target *ti, const char *ivmode)
spin_lock_init(&cc->write_thread_lock);
cc->write_tree = RB_ROOT;
- cc->write_thread = kthread_create(dmcrypt_write, cc, "dmcrypt_write/%s", devname);
+ cc->write_thread = kthread_run(dmcrypt_write, cc, "dmcrypt_write/%s", devname);
if (IS_ERR(cc->write_thread)) {
ret = PTR_ERR(cc->write_thread);
cc->write_thread = NULL;
ti->error = "Couldn't spawn write thread";
goto bad;
}
- wake_up_process(cc->write_thread);
ti->num_flush_bios = 1;
ti->limit_swap_bios = true;
+ dm_audit_log_ctr(DM_MSG_PREFIX, ti, 1);
return 0;
bad:
+ dm_audit_log_ctr(DM_MSG_PREFIX, ti, 0);
crypt_dtr(ti);
return ret;
}
#include <linux/async_tx.h>
#include <linux/dm-bufio.h>
+#include "dm-audit.h"
+
#define DM_MSG_PREFIX "integrity"
#define DEFAULT_INTERLEAVE_SECTORS 32768
}
if (memcmp((__u8 *)ic->sb + (1 << SECTOR_SHIFT) - size, result, size)) {
dm_integrity_io_error(ic, "superblock mac", -EILSEQ);
+ dm_audit_log_target(DM_MSG_PREFIX, "mac-superblock", ic->ti, 0);
return -EILSEQ;
}
}
if (likely(wr))
memcpy(&js->mac, result + (j * JOURNAL_MAC_PER_SECTOR), JOURNAL_MAC_PER_SECTOR);
else {
- if (memcmp(&js->mac, result + (j * JOURNAL_MAC_PER_SECTOR), JOURNAL_MAC_PER_SECTOR))
+ if (memcmp(&js->mac, result + (j * JOURNAL_MAC_PER_SECTOR), JOURNAL_MAC_PER_SECTOR)) {
dm_integrity_io_error(ic, "journal mac", -EILSEQ);
+ dm_audit_log_target(DM_MSG_PREFIX, "mac-journal", ic->ti, 0);
+ }
}
}
}
char *mem, *checksums_ptr;
again:
- mem = (char *)kmap_atomic(bv.bv_page) + bv.bv_offset;
+ mem = bvec_kmap_local(&bv);
pos = 0;
checksums_ptr = checksums;
do {
pos += ic->sectors_per_block << SECTOR_SHIFT;
sector += ic->sectors_per_block;
} while (pos < bv.bv_len && sectors_to_process && checksums != checksums_onstack);
- kunmap_atomic(mem);
+ kunmap_local(mem);
r = dm_integrity_rw_tag(ic, checksums, &dio->metadata_block, &dio->metadata_offset,
checksums_ptr - checksums, dio->op == REQ_OP_READ ? TAG_CMP : TAG_WRITE);
if (unlikely(r)) {
if (r > 0) {
char b[BDEVNAME_SIZE];
- DMERR_LIMIT("%s: Checksum failed at sector 0x%llx", bio_devname(bio, b),
- (sector - ((r + ic->tag_size - 1) / ic->tag_size)));
+ sector_t s;
+
+ s = sector - ((r + ic->tag_size - 1) / ic->tag_size);
+ DMERR_LIMIT("%s: Checksum failed at sector 0x%llx",
+ bio_devname(bio, b), s);
r = -EILSEQ;
atomic64_inc(&ic->number_of_mismatches);
+ dm_audit_log_bio(DM_MSG_PREFIX, "integrity-checksum",
+ bio, s, 0);
}
if (likely(checksums != checksums_onstack))
kfree(checksums);
n_sectors -= bv.bv_len >> SECTOR_SHIFT;
bio_advance_iter(bio, &bio->bi_iter, bv.bv_len);
retry_kmap:
- mem = kmap_atomic(bv.bv_page);
+ mem = bvec_kmap_local(&bv);
if (likely(dio->op == REQ_OP_WRITE))
flush_dcache_page(bv.bv_page);
if (unlikely(journal_entry_is_inprogress(je))) {
flush_dcache_page(bv.bv_page);
- kunmap_atomic(mem);
+ kunmap_local(mem);
__io_wait_event(ic->copy_to_journal_wait, !journal_entry_is_inprogress(je));
goto retry_kmap;
if (unlikely(memcmp(checksums_onstack, journal_entry_tag(ic, je), ic->tag_size))) {
DMERR_LIMIT("Checksum failed when reading from journal, at sector 0x%llx",
logical_sector);
+ dm_audit_log_bio(DM_MSG_PREFIX, "journal-checksum",
+ bio, logical_sector, 0);
}
}
#endif
if (unlikely(dio->op == REQ_OP_READ))
flush_dcache_page(bv.bv_page);
- kunmap_atomic(mem);
+ kunmap_local(mem);
} while (n_sectors);
if (likely(dio->op == REQ_OP_WRITE)) {
integrity_sector_checksum(ic, sec + ((l - j) << ic->sb->log2_sectors_per_block),
(char *)access_journal_data(ic, i, l), test_tag);
- if (unlikely(memcmp(test_tag, journal_entry_tag(ic, je2), ic->tag_size)))
+ if (unlikely(memcmp(test_tag, journal_entry_tag(ic, je2), ic->tag_size))) {
dm_integrity_io_error(ic, "tag mismatch when replaying journal", -EILSEQ);
+ dm_audit_log_target(DM_MSG_PREFIX, "integrity-replay-journal", ic->ti, 0);
+ }
}
journal_entry_set_unused(je2);
if (ic->discard)
ti->num_discard_bios = 1;
+ dm_audit_log_ctr(DM_MSG_PREFIX, ti, 1);
return 0;
bad:
+ dm_audit_log_ctr(DM_MSG_PREFIX, ti, 0);
dm_integrity_dtr(ti);
return r;
}
free_alg(&ic->journal_mac_alg);
kfree(ic);
+ dm_audit_log_dtr(DM_MSG_PREFIX, ti, 1);
}
static struct target_type integrity_target = {
*/
bio_for_each_segment(bv, bio, iter) {
struct page *page;
- void *src, *dst;
+ void *dst;
page = alloc_page(GFP_NOIO);
if (!page) {
return DM_MAPIO_KILL;
}
- src = kmap_atomic(bv.bv_page);
dst = kmap_atomic(page);
- memcpy(dst, src + bv.bv_offset, bv.bv_len);
+ memcpy_from_bvec(dst, &bv);
kunmap_atomic(dst);
- kunmap_atomic(src);
block->vecs[i].bv_page = page;
block->vecs[i].bv_len = bv.bv_len;
block->vec_cnt++;
r = dm_split_args(&argc, &argv, params);
if (r) {
- tgt->error = "couldn't split parameters (insufficient memory)";
+ tgt->error = "couldn't split parameters";
goto bad;
}
return 0;
bad:
- DMERR("%s: %s: %s", dm_device_name(t->md), type, tgt->error);
+ DMERR("%s: %s: %s (%pe)", dm_device_name(t->md), type, tgt->error, ERR_PTR(r));
dm_put_target_type(tgt->type);
return r;
}
unsigned len;
struct bio_vec bv = bio_iter_iovec(bio, *iter);
- page = kmap_atomic(bv.bv_page);
+ page = bvec_kmap_local(&bv);
len = bv.bv_len;
if (likely(len >= todo))
len = todo;
- r = process(v, io, page + bv.bv_offset, len);
- kunmap_atomic(page);
+ r = process(v, io, page, len);
+ kunmap_local(page);
if (r < 0)
return r;
raw_spin_lock_init(&wc->endio_list_lock);
INIT_LIST_HEAD(&wc->endio_list);
- wc->endio_thread = kthread_create(writecache_endio_thread, wc, "writecache_endio");
+ wc->endio_thread = kthread_run(writecache_endio_thread, wc, "writecache_endio");
if (IS_ERR(wc->endio_thread)) {
r = PTR_ERR(wc->endio_thread);
wc->endio_thread = NULL;
ti->error = "Couldn't spawn endio thread";
goto bad;
}
- wake_up_process(wc->endio_thread);
/*
* Parse the mode (pmem or ssd)
wc->memory_map_size -= (uint64_t)wc->start_sector << SECTOR_SHIFT;
bio_list_init(&wc->flush_list);
- wc->flush_thread = kthread_create(writecache_flush_thread, wc, "dm_writecache_flush");
+ wc->flush_thread = kthread_run(writecache_flush_thread, wc, "dm_writecache_flush");
if (IS_ERR(wc->flush_thread)) {
r = PTR_ERR(wc->flush_thread);
wc->flush_thread = NULL;
ti->error = "Couldn't spawn flush thread";
goto bad;
}
- wake_up_process(wc->flush_thread);
r = calculate_memory_size(wc->memory_map_size, wc->block_size,
&n_blocks, &n_metadata_blocks);
struct dmz_target *dmz = ti->private;
int i;
- flush_workqueue(dmz->chunk_wq);
destroy_workqueue(dmz->chunk_wq);
for (i = 0; i < dmz->nr_ddevs; i++)
format_dev_t(md->name, MKDEV(_major, minor));
- md->wq = alloc_workqueue("kdmflush", WQ_MEM_RECLAIM, 0);
+ md->wq = alloc_workqueue("kdmflush/%s", WQ_MEM_RECLAIM, 0, md->name);
if (!md->wq)
goto bad;
dm_table_event_callback(t, event_callback, md);
- /*
- * The queue hasn't been stopped yet, if the old table type wasn't
- * for request-based during suspension. So stop it to prevent
- * I/O mapping before resume.
- * This must be done before setting the queue restrictions,
- * because request-based dm may be run just after the setting.
- */
- if (request_based)
- dm_stop_queue(q);
-
if (request_based) {
/*
* Leverage the fact that request-based DM targets are
{
unsigned long backlog;
unsigned long old_mwb = mddev->bitmap_info.max_write_behind;
+ struct md_rdev *rdev;
+ bool has_write_mostly = false;
int rv = kstrtoul(buf, 10, &backlog);
if (rv)
return rv;
if (backlog > COUNTER_MAX)
return -EINVAL;
+
+ /*
+ * Without write mostly device, it doesn't make sense to set
+ * backlog for max_write_behind.
+ */
+ rdev_for_each(rdev, mddev) {
+ if (test_bit(WriteMostly, &rdev->flags)) {
+ has_write_mostly = true;
+ break;
+ }
+ }
+ if (!has_write_mostly) {
+ pr_warn_ratelimited("%s: can't set backlog, no write mostly device available\n",
+ mdname(mddev));
+ return -EINVAL;
+ }
+
mddev->bitmap_info.max_write_behind = backlog;
if (!backlog && mddev->serial_info_pool) {
/* serial_info_pool is not needed if backlog is zero */
struct ppl_conf *ppl_conf = log->ppl_conf;
struct md_rdev *rdev = log->rdev;
struct mddev *mddev = rdev->mddev;
- struct page *page, *page2, *tmp;
+ struct page *page, *page2;
struct ppl_header *pplhdr = NULL, *prev_pplhdr = NULL;
u32 crc, crc_stored;
u32 signature;
prev_pplhdr_offset = pplhdr_offset;
prev_pplhdr = pplhdr;
- tmp = page;
- page = page2;
- page2 = tmp;
+ swap(page, page2);
/* calculate next potential ppl offset */
for (i = 0; i < le32_to_cpu(pplhdr->entries_count); i++)
if (abort)
dst->rx_status |= CEC_RX_STATUS_FEATURE_ABORT;
msg->flags = dst->flags;
+ msg->sequence = dst->sequence;
/* Remove it from the wait_queue */
list_del_init(&data->list);
buf->offset = vaddr & ~PAGE_MASK;
buf->size = size;
buf->dma_sgt = &buf->sg_table;
+ buf->vb = vb;
vec = vb2_create_framevec(vaddr, size);
if (IS_ERR(vec))
goto userptr_fail_pfnvec;
buf->dma_dir = vb->vb2_queue->dma_dir;
buf->size = size;
buf->db_attach = dba;
+ buf->vb = vb;
return buf;
}
#include <linux/types.h>
#include <linux/errno.h>
#include <linux/delay.h>
+#include <linux/bits.h>
#include <linux/string.h>
int cxd2880_convert2s_complement(u32 value, u32 bitlen);
#include <linux/gpio/consumer.h>
#include <linux/i2c.h>
#include <linux/module.h>
-#include <linux/of_graph.h>
#include <linux/pm_runtime.h>
#include <linux/pm.h>
+#include <linux/property.h>
#include <linux/regulator/consumer.h>
#include <media/v4l2-ctrls.h>
#include <media/v4l2-device.h>
.driver = {
.name = "hi846",
.pm = &hi846_pm_ops,
- .of_match_table = of_match_ptr(hi846_of_match),
+ .of_match_table = hi846_of_match,
},
.probe_new = hi846_probe,
.remove = hi846_remove,
/* Streaming on/off */
bool streaming;
+ /* True if the device has been identified */
+ bool identified;
};
static const struct imx319_reg imx319_global_regs[] = {
return 0;
}
+/* Verify chip ID */
+static int imx319_identify_module(struct imx319 *imx319)
+{
+ struct i2c_client *client = v4l2_get_subdevdata(&imx319->sd);
+ int ret;
+ u32 val;
+
+ if (imx319->identified)
+ return 0;
+
+ ret = imx319_read_reg(imx319, IMX319_REG_CHIP_ID, 2, &val);
+ if (ret)
+ return ret;
+
+ if (val != IMX319_CHIP_ID) {
+ dev_err(&client->dev, "chip id mismatch: %x!=%x",
+ IMX319_CHIP_ID, val);
+ return -EIO;
+ }
+
+ imx319->identified = true;
+
+ return 0;
+}
+
/* Start streaming */
static int imx319_start_streaming(struct imx319 *imx319)
{
const struct imx319_reg_list *reg_list;
int ret;
+ ret = imx319_identify_module(imx319);
+ if (ret)
+ return ret;
+
/* Global Setting */
reg_list = &imx319_global_setting;
ret = imx319_write_regs(imx319, reg_list->regs, reg_list->num_of_regs);
return ret;
}
-/* Verify chip ID */
-static int imx319_identify_module(struct imx319 *imx319)
-{
- struct i2c_client *client = v4l2_get_subdevdata(&imx319->sd);
- int ret;
- u32 val;
-
- ret = imx319_read_reg(imx319, IMX319_REG_CHIP_ID, 2, &val);
- if (ret)
- return ret;
-
- if (val != IMX319_CHIP_ID) {
- dev_err(&client->dev, "chip id mismatch: %x!=%x",
- IMX319_CHIP_ID, val);
- return -EIO;
- }
-
- return 0;
-}
-
static const struct v4l2_subdev_core_ops imx319_subdev_core_ops = {
.subscribe_event = v4l2_ctrl_subdev_subscribe_event,
.unsubscribe_event = v4l2_event_subdev_unsubscribe,
static int imx319_probe(struct i2c_client *client)
{
struct imx319 *imx319;
+ bool full_power;
int ret;
u32 i;
/* Initialize subdev */
v4l2_i2c_subdev_init(&imx319->sd, client, &imx319_subdev_ops);
- /* Check module identity */
- ret = imx319_identify_module(imx319);
- if (ret) {
- dev_err(&client->dev, "failed to find sensor: %d", ret);
- goto error_probe;
+ full_power = acpi_dev_state_d0(&client->dev);
+ if (full_power) {
+ /* Check module identity */
+ ret = imx319_identify_module(imx319);
+ if (ret) {
+ dev_err(&client->dev, "failed to find sensor: %d", ret);
+ goto error_probe;
+ }
}
imx319->hwcfg = imx319_get_hwcfg(&client->dev);
if (ret < 0)
goto error_media_entity;
- /*
- * Device is already turned on by i2c-core with ACPI domain PM.
- * Enable runtime PM and turn off the device.
- */
- pm_runtime_set_active(&client->dev);
+ /* Set the device's state to active if it's in D0 state. */
+ if (full_power)
+ pm_runtime_set_active(&client->dev);
pm_runtime_enable(&client->dev);
pm_runtime_idle(&client->dev);
},
.probe_new = imx319_probe,
.remove = imx319_remove,
+ .flags = I2C_DRV_ACPI_WAIVE_D0_PROBE,
};
module_i2c_driver(imx319_i2c_driver);
/*
* x86 is the only compat architecture with different struct alignment
* between 32-bit and 64-bit tasks.
- *
- * On all other architectures, v4l2_event32 and v4l2_event32_time32 are
- * the same as v4l2_event and v4l2_event_time32, so we can use the native
- * handlers, converting v4l2_event to v4l2_event_time32 if necessary.
*/
struct v4l2_event32 {
__u32 type;
__u32 reserved[8];
};
-#ifdef CONFIG_COMPAT_32BIT_TIME
-struct v4l2_event32_time32 {
- __u32 type;
- union {
- compat_s64 value64;
- __u8 data[64];
- } u;
- __u32 pending;
- __u32 sequence;
- struct old_timespec32 timestamp;
- __u32 id;
- __u32 reserved[8];
-};
-#endif
-
static int put_v4l2_event32(struct v4l2_event *p64,
struct v4l2_event32 __user *p32)
{
return 0;
}
+#endif
+
#ifdef CONFIG_COMPAT_32BIT_TIME
+struct v4l2_event32_time32 {
+ __u32 type;
+ union {
+ compat_s64 value64;
+ __u8 data[64];
+ } u;
+ __u32 pending;
+ __u32 sequence;
+ struct old_timespec32 timestamp;
+ __u32 id;
+ __u32 reserved[8];
+};
+
static int put_v4l2_event32_time32(struct v4l2_event *p64,
struct v4l2_event32_time32 __user *p32)
{
return 0;
}
#endif
-#endif
struct v4l2_edid32 {
__u32 pad;
#define VIDIOC_QUERYBUF32_TIME32 _IOWR('V', 9, struct v4l2_buffer32_time32)
#define VIDIOC_QBUF32_TIME32 _IOWR('V', 15, struct v4l2_buffer32_time32)
#define VIDIOC_DQBUF32_TIME32 _IOWR('V', 17, struct v4l2_buffer32_time32)
-#ifdef CONFIG_X86_64
#define VIDIOC_DQEVENT32_TIME32 _IOR ('V', 89, struct v4l2_event32_time32)
-#endif
#define VIDIOC_PREPARE_BUF32_TIME32 _IOWR('V', 93, struct v4l2_buffer32_time32)
#endif
#ifdef CONFIG_X86_64
case VIDIOC_DQEVENT32:
return VIDIOC_DQEVENT;
+#endif
#ifdef CONFIG_COMPAT_32BIT_TIME
case VIDIOC_DQEVENT32_TIME32:
return VIDIOC_DQEVENT;
#endif
-#endif
}
return cmd;
}
#ifdef CONFIG_X86_64
case VIDIOC_DQEVENT32:
return put_v4l2_event32(parg, arg);
+#endif
#ifdef CONFIG_COMPAT_32BIT_TIME
case VIDIOC_DQEVENT32_TIME32:
return put_v4l2_event32_time32(parg, arg);
#endif
-#endif
}
return 0;
}
mpt_device_driver_register(struct mpt_pci_driver * dd_cbfunc, u8 cb_idx)
{
MPT_ADAPTER *ioc;
- const struct pci_device_id *id;
if (!cb_idx || cb_idx >= MPT_MAX_PROTOCOL_DRIVERS)
return -EINVAL;
/* call per pci device probe entry point */
list_for_each_entry(ioc, &ioc_list, list) {
- id = ioc->pcidev->driver ?
- ioc->pcidev->driver->id_table : NULL;
if (dd_cbfunc->probe)
- dd_cbfunc->probe(ioc->pcidev, id);
+ dd_cbfunc->probe(ioc->pcidev);
}
return 0;
for(cb_idx = 0; cb_idx < MPT_MAX_PROTOCOL_DRIVERS; cb_idx++) {
if(MptDeviceDriverHandlers[cb_idx] &&
MptDeviceDriverHandlers[cb_idx]->probe) {
- MptDeviceDriverHandlers[cb_idx]->probe(pdev,id);
+ MptDeviceDriverHandlers[cb_idx]->probe(pdev);
}
}
} MPT_DRIVER_CLASS;
struct mpt_pci_driver{
- int (*probe) (struct pci_dev *dev, const struct pci_device_id *id);
+ int (*probe) (struct pci_dev *dev);
void (*remove) (struct pci_dev *dev);
};
static int mptctl_hp_hostinfo(MPT_ADAPTER *iocp, unsigned long arg, unsigned int cmd);
static int mptctl_hp_targetinfo(MPT_ADAPTER *iocp, unsigned long arg);
-static int mptctl_probe(struct pci_dev *, const struct pci_device_id *);
+static int mptctl_probe(struct pci_dev *);
static void mptctl_remove(struct pci_dev *);
#ifdef CONFIG_COMPAT
*/
static int
-mptctl_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+mptctl_probe(struct pci_dev *pdev)
{
MPT_ADAPTER *ioc = pci_get_drvdata(pdev);
.sg_tablesize = MPT_SCSI_SG_DEPTH,
.max_sectors = 8192,
.cmd_per_lun = 7,
- .shost_attrs = mptscsih_host_attrs,
+ .shost_groups = mptscsih_host_attr_groups,
};
/****************************************************************************
if (!vdevice || !vdevice->vtarget) {
SCpnt->result = DID_NO_CONNECT << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
err = fc_remote_port_chkready(rport);
if (unlikely(err)) {
SCpnt->result = err;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
ri = *((struct mptfc_rport_info **)rport->dd_data);
if (unlikely(!ri)) {
SCpnt->result = DID_IMM_RETRY << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
}
static int
-mptlan_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+mptlan_probe(struct pci_dev *pdev)
{
MPT_ADAPTER *ioc = pci_get_drvdata(pdev);
struct net_device *dev;
if (!vdevice || !vdevice->vtarget || vdevice->vtarget->deleted) {
SCpnt->result = DID_NO_CONNECT << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
.sg_tablesize = MPT_SCSI_SG_DEPTH,
.max_sectors = 8192,
.cmd_per_lun = 7,
- .shost_attrs = mptscsih_host_attrs,
+ .shost_groups = mptscsih_host_attr_groups,
.no_write_same = 1,
};
/* Unmap the DMA buffers, if any. */
scsi_dma_unmap(sc);
- sc->scsi_done(sc); /* Issue the command callback */
+ scsi_done(sc); /* Issue the command callback */
/* Free Chain buffers */
mptscsih_freeChainBuffers(ioc, req_idx);
dtmprintk(ioc, sdev_printk(KERN_INFO, sc->device, MYIOC_s_FMT
"completing cmds: fw_channel %d, fw_id %d, sc=%p, mf = %p, "
"idx=%x\n", ioc->name, channel, id, sc, mf, ii));
- sc->scsi_done(sc);
+ scsi_done(sc);
}
}
EXPORT_SYMBOL(mptscsih_flush_running_cmds);
"fw_id %d, sc=%p, mf = %p, idx=%x\n", ioc->name,
vdevice->vtarget->channel, vdevice->vtarget->id,
sc, mf, ii));
- sc->scsi_done(sc);
+ scsi_done(sc);
spin_lock_irqsave(&ioc->scsi_lookup_lock, flags);
}
}
*/
if ((hd = shost_priv(SCpnt->device->host)) == NULL) {
SCpnt->result = DID_RESET << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
printk(KERN_ERR MYNAM ": task abort: "
"can't locate host! (sc=%p)\n", SCpnt);
return FAILED;
"task abort: device has been deleted (sc=%p)\n",
ioc->name, SCpnt));
SCpnt->result = DID_NO_CONNECT << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
retval = SUCCESS;
goto out;
}
static DEVICE_ATTR(debug_level, S_IRUGO | S_IWUSR,
mptscsih_debug_level_show, mptscsih_debug_level_store);
-struct device_attribute *mptscsih_host_attrs[] = {
- &dev_attr_version_fw,
- &dev_attr_version_bios,
- &dev_attr_version_mpi,
- &dev_attr_version_product,
- &dev_attr_version_nvdata_persistent,
- &dev_attr_version_nvdata_default,
- &dev_attr_board_name,
- &dev_attr_board_assembly,
- &dev_attr_board_tracer,
- &dev_attr_io_delay,
- &dev_attr_device_delay,
- &dev_attr_debug_level,
+static struct attribute *mptscsih_host_attrs[] = {
+ &dev_attr_version_fw.attr,
+ &dev_attr_version_bios.attr,
+ &dev_attr_version_mpi.attr,
+ &dev_attr_version_product.attr,
+ &dev_attr_version_nvdata_persistent.attr,
+ &dev_attr_version_nvdata_default.attr,
+ &dev_attr_board_name.attr,
+ &dev_attr_board_assembly.attr,
+ &dev_attr_board_tracer.attr,
+ &dev_attr_io_delay.attr,
+ &dev_attr_device_delay.attr,
+ &dev_attr_debug_level.attr,
NULL,
};
-EXPORT_SYMBOL(mptscsih_host_attrs);
+static const struct attribute_group mptscsih_host_attr_group = {
+ .attrs = mptscsih_host_attrs
+};
+
+const struct attribute_group *mptscsih_host_attr_groups[] = {
+ &mptscsih_host_attr_group,
+ NULL
+};
+EXPORT_SYMBOL(mptscsih_host_attr_groups);
EXPORT_SYMBOL(mptscsih_remove);
EXPORT_SYMBOL(mptscsih_shutdown);
extern int mptscsih_change_queue_depth(struct scsi_device *sdev, int qdepth);
extern u8 mptscsih_raid_id_to_num(MPT_ADAPTER *ioc, u8 channel, u8 id);
extern int mptscsih_is_phys_disk(MPT_ADAPTER *ioc, u8 channel, u8 id);
-extern struct device_attribute *mptscsih_host_attrs[];
+extern const struct attribute_group *mptscsih_host_attr_groups[];
extern struct scsi_cmnd *mptscsih_get_scsi_lookup(MPT_ADAPTER *ioc, int i);
extern void mptscsih_taskmgmt_response_code(MPT_ADAPTER *ioc, u8 response_code);
extern void mptscsih_flush_running_cmds(MPT_SCSI_HOST *hd);
if (!vdevice || !vdevice->vtarget) {
SCpnt->result = DID_NO_CONNECT << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
if (SCpnt->device->channel == 1 &&
mptscsih_is_phys_disk(ioc, 0, SCpnt->device->id) == 0) {
SCpnt->result = DID_NO_CONNECT << 16;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
.sg_tablesize = MPT_SCSI_SG_DEPTH,
.max_sectors = 8192,
.cmd_per_lun = 7,
- .shost_attrs = mptscsih_host_attrs,
+ .shost_groups = mptscsih_host_attr_groups,
};
static int mptspi_write_spi_device_pg1(struct scsi_target *starget,
bool "Analog Devices ADP5520/01 MFD PMIC Core Support"
depends on I2C=y
help
- Say yes here to add support for Analog Devices AD5520 and ADP5501,
+ Say yes here to add support for Analog Devices ADP5520 and ADP5501,
Multifunction Power Management IC. This includes
the I2C driver and the core APIs _only_, you have to select
individual components like LCD backlight, LEDs, GPIOs and Kepad
select REGMAP_MMIO
help
Select this option to enable support for Samsung Exynos Low Power
- Audio Subsystem.
+ Audio Subsystem present on some of Samsung Exynos
+ SoCs (e.g. Exynos5433).
+ Choose Y here only if you build for such Samsung SoC.
config MFD_GATEWORKS_GSC
tristate "Gateworks System Controller"
config MFD_INTEL_PMT
tristate "Intel Platform Monitoring Technology (PMT) support"
- depends on PCI
+ depends on X86 && PCI
select MFD_CORE
help
The Intel Platform Monitoring Technology (PMT) is an interface that
config MFD_SIMPLE_MFD_I2C
tristate
depends on I2C
+ select MFD_CORE
select REGMAP_I2C
help
This driver creates a single register map with the intention for it
If you say yes here you get support for the TPS65912 series of
PM chips with SPI interface.
-config MFD_TPS80031
- bool "TI TPS80031/TPS80032 Power Management chips"
- depends on I2C=y
- select MFD_CORE
- select REGMAP_I2C
- select REGMAP_IRQ
- help
- If you say yes here you get support for the Texas Instruments
- TPS80031/ TPS80032 Fully Integrated Power Management with Power
- Path and Battery Charger. The device provides five configurable
- step-down converters, 11 general purpose LDOs, USB OTG Module,
- ADC, RTC, 2 PWM, System Voltage Regulator/Battery Charger with
- Power Path from USB, 32K clock generator.
-
config TWL4030_CORE
bool "TI TWL4030/TWL5030/TWL6030/TPS659x0 Support"
depends on I2C=y
obj-$(CONFIG_MFD_TPS65912) += tps65912-core.o
obj-$(CONFIG_MFD_TPS65912_I2C) += tps65912-i2c.o
obj-$(CONFIG_MFD_TPS65912_SPI) += tps65912-spi.o
-obj-$(CONFIG_MFD_TPS80031) += tps80031.o
obj-$(CONFIG_MENELAUS) += menelaus.o
obj-$(CONFIG_TWL4030_CORE) += twl-core.o twl4030-irq.o twl6030-irq.o
#include <linux/mfd/altera-a10sr.h>
#include <linux/mfd/core.h>
#include <linux/init.h>
+#include <linux/module.h>
#include <linux/of.h>
#include <linux/spi/spi.h>
{ .compatible = "altr,a10sr" },
{ },
};
+MODULE_DEVICE_TABLE(of, altr_a10sr_spi_of_match);
+
+static const struct spi_device_id altr_a10sr_spi_ids[] = {
+ { .name = "a10sr" },
+ { },
+};
+MODULE_DEVICE_TABLE(spi, altr_a10sr_spi_ids);
static struct spi_driver altr_a10sr_spi_driver = {
.probe = altr_a10sr_spi_probe,
.name = "altr_a10sr",
.of_match_table = of_match_ptr(altr_a10sr_spi_of_match),
},
+ .id_table = altr_a10sr_spi_ids,
};
builtin_driver(altr_a10sr_spi_driver, spi_register_driver)
if (!base)
return -ENOMEM;
- sysmgr_config.max_register = resource_size(res) - 3;
+ sysmgr_config.max_register = resource_size(res) - 4;
regmap = devm_regmap_init_mmio(dev, base, &sysmgr_config);
}
return 0;
}
-
-const struct of_device_id arizona_of_match[] = {
- { .compatible = "wlf,wm5102", .data = (void *)WM5102 },
- { .compatible = "wlf,wm5110", .data = (void *)WM5110 },
- { .compatible = "wlf,wm8280", .data = (void *)WM8280 },
- { .compatible = "wlf,wm8997", .data = (void *)WM8997 },
- { .compatible = "wlf,wm8998", .data = (void *)WM8998 },
- { .compatible = "wlf,wm1814", .data = (void *)WM1814 },
- { .compatible = "wlf,wm1831", .data = (void *)WM1831 },
- { .compatible = "cirrus,cs47l24", .data = (void *)CS47L24 },
- {},
-};
-EXPORT_SYMBOL_GPL(arizona_of_match);
#else
static inline int arizona_of_get_core_pdata(struct arizona *arizona)
{
};
MODULE_DEVICE_TABLE(i2c, arizona_i2c_id);
+#ifdef CONFIG_OF
+const struct of_device_id arizona_i2c_of_match[] = {
+ { .compatible = "wlf,wm5102", .data = (void *)WM5102 },
+ { .compatible = "wlf,wm5110", .data = (void *)WM5110 },
+ { .compatible = "wlf,wm8280", .data = (void *)WM8280 },
+ { .compatible = "wlf,wm8997", .data = (void *)WM8997 },
+ { .compatible = "wlf,wm8998", .data = (void *)WM8998 },
+ { .compatible = "wlf,wm1814", .data = (void *)WM1814 },
+ {},
+};
+#endif
+
static struct i2c_driver arizona_i2c_driver = {
.driver = {
.name = "arizona",
.pm = &arizona_pm_ops,
- .of_match_table = of_match_ptr(arizona_of_match),
+ .of_match_table = of_match_ptr(arizona_i2c_of_match),
},
.probe = arizona_i2c_probe,
.remove = arizona_i2c_remove,
};
MODULE_DEVICE_TABLE(spi, arizona_spi_ids);
+#ifdef CONFIG_OF
+const struct of_device_id arizona_spi_of_match[] = {
+ { .compatible = "wlf,wm5102", .data = (void *)WM5102 },
+ { .compatible = "wlf,wm5110", .data = (void *)WM5110 },
+ { .compatible = "wlf,wm8280", .data = (void *)WM8280 },
+ { .compatible = "wlf,wm1831", .data = (void *)WM1831 },
+ { .compatible = "cirrus,cs47l24", .data = (void *)CS47L24 },
+ {},
+};
+#endif
+
static struct spi_driver arizona_spi_driver = {
.driver = {
.name = "arizona",
.pm = &arizona_pm_ops,
- .of_match_table = of_match_ptr(arizona_of_match),
+ .of_match_table = of_match_ptr(arizona_spi_of_match),
.acpi_match_table = ACPI_PTR(arizona_acpi_match),
},
.probe = arizona_spi_probe,
extern const struct dev_pm_ops arizona_pm_ops;
-extern const struct of_device_id arizona_of_match[];
-
extern const struct regmap_irq_chip wm5102_aod;
extern const struct regmap_irq_chip wm5102_irq;
ec->ec_dev = dev_get_drvdata(dev->parent);
ec->dev = dev;
ec->cmd_offset = ec_platform->cmd_offset;
- ec->features[0] = -1U; /* Not cached yet */
- ec->features[1] = -1U; /* Not cached yet */
+ ec->features.flags[0] = -1U; /* Not cached yet */
+ ec->features.flags[1] = -1U; /* Not cached yet */
device_initialize(&ec->class_dev);
for (i = 0; i < ARRAY_SIZE(cros_mcu_devices); i++) {
module_init(cros_ec_dev_init);
module_exit(cros_ec_dev_exit);
-MODULE_ALIAS("platform:" DRV_NAME);
MODULE_AUTHOR("Bill Richardson <wfrichar@chromium.org>");
MODULE_DESCRIPTION("Userspace interface to the Chrome OS Embedded Controller");
MODULE_VERSION("1.0");
&da9063_bb_da_volatile_table;
break;
case PMIC_DA9063_DA:
+ case PMIC_DA9063_EA:
da9063_regmap_config.rd_table =
&da9063_da_readable_table;
da9063_regmap_config.wr_table =
&da9063l_bb_da_volatile_table;
break;
case PMIC_DA9063_DA:
+ case PMIC_DA9063_EA:
da9063_regmap_config.rd_table =
&da9063l_da_readable_table;
da9063_regmap_config.wr_table =
#include <linux/mfd/abx500/ab8500.h>
#include <linux/regulator/db8500-prcmu.h>
#include <linux/regulator/machine.h>
-#include <linux/platform_data/ux500_wdt.h>
#include "db8500-prcmu-regs.h"
/* Index of different voltages to be used when accessing AVSData */
},
};
-static struct ux500_wdt_data db8500_wdt_pdata = {
- .timeout = 600, /* 10 minutes */
- .has_28_bits_resolution = true,
-};
-
static const struct mfd_cell common_prcmu_devs[] = {
- {
- .name = "ux500_wdt",
- .platform_data = &db8500_wdt_pdata,
- .pdata_size = sizeof(db8500_wdt_pdata),
- .id = -1,
- },
+ MFD_CELL_NAME("db8500_wdt"),
MFD_CELL_NAME("db8500-cpuidle"),
};
DLN2_HANDLE_GPIO,
DLN2_HANDLE_I2C,
DLN2_HANDLE_SPI,
+ DLN2_HANDLE_ADC,
DLN2_HANDLES
};
DLN2_ACPI_MATCH_GPIO = 0,
DLN2_ACPI_MATCH_I2C = 1,
DLN2_ACPI_MATCH_SPI = 2,
+ DLN2_ACPI_MATCH_ADC = 3,
};
static struct dln2_platform_data dln2_pdata_gpio = {
.adr = DLN2_ACPI_MATCH_SPI,
};
+/* Only one ADC port supported */
+static struct dln2_platform_data dln2_pdata_adc = {
+ .handle = DLN2_HANDLE_ADC,
+ .port = 0,
+};
+
+static struct mfd_cell_acpi_match dln2_acpi_match_adc = {
+ .adr = DLN2_ACPI_MATCH_ADC,
+};
+
static const struct mfd_cell dln2_devs[] = {
{
.name = "dln2-gpio",
.platform_data = &dln2_pdata_spi,
.pdata_size = sizeof(struct dln2_platform_data),
},
+ {
+ .name = "dln2-adc",
+ .acpi_match = &dln2_acpi_match_adc,
+ .platform_data = &dln2_pdata_adc,
+ .pdata_size = sizeof(struct dln2_platform_data),
+ },
};
static void dln2_stop(struct dln2_dev *dln2)
*/
#include <linux/mfd/core.h>
-#include <linux/mfd/hi6421-spmi-pmic.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
static int hi6421_spmi_pmic_probe(struct spmi_device *sdev)
{
struct device *dev = &sdev->dev;
+ struct regmap *regmap;
int ret;
- struct hi6421_spmi_pmic *ddata;
- ddata = devm_kzalloc(dev, sizeof(*ddata), GFP_KERNEL);
- if (!ddata)
- return -ENOMEM;
- ddata->regmap = devm_regmap_init_spmi_ext(sdev, ®map_config);
- if (IS_ERR(ddata->regmap))
- return PTR_ERR(ddata->regmap);
+ regmap = devm_regmap_init_spmi_ext(sdev, ®map_config);
+ if (IS_ERR(regmap))
+ return PTR_ERR(regmap);
- ddata->dev = dev;
-
- dev_set_drvdata(&sdev->dev, ddata);
+ dev_set_drvdata(&sdev->dev, regmap);
ret = devm_mfd_add_devices(&sdev->dev, PLATFORM_DEVID_NONE,
hi6421v600_devs, ARRAY_SIZE(hi6421v600_devs),
{ PCI_VDEVICE(INTEL, 0x34ea), (kernel_ulong_t)&bxt_i2c_info },
{ PCI_VDEVICE(INTEL, 0x34eb), (kernel_ulong_t)&bxt_i2c_info },
{ PCI_VDEVICE(INTEL, 0x34fb), (kernel_ulong_t)&spt_info },
+ /* ICL-N */
+ { PCI_VDEVICE(INTEL, 0x38a8), (kernel_ulong_t)&bxt_uart_info },
/* TGL-H */
{ PCI_VDEVICE(INTEL, 0x43a7), (kernel_ulong_t)&bxt_uart_info },
{ PCI_VDEVICE(INTEL, 0x43a8), (kernel_ulong_t)&bxt_uart_info },
{
struct cmodio_device *priv = dev_get_drvdata(dev);
- return snprintf(buf, PAGE_SIZE, "%x\n", priv->hex);
+ return sysfs_emit(buf, "%x\n", priv->hex);
}
static DEVICE_ATTR_RO(modulbus_number);
}
ret = regmap_add_irq_chip(max14577->regmap_pmic, max14577->irq,
- IRQF_TRIGGER_FALLING | IRQF_ONESHOT | IRQF_SHARED,
+ IRQF_ONESHOT | IRQF_SHARED,
0, &max77836_pmic_irq_chip,
&max14577->irq_data_pmic);
if (ret != 0) {
irq_chip = &max77836_muic_irq_chip;
mfd_devs = max77836_devs;
mfd_devs_size = ARRAY_SIZE(max77836_devs);
- irq_flags = IRQF_TRIGGER_FALLING | IRQF_ONESHOT | IRQF_SHARED;
+ irq_flags = IRQF_ONESHOT | IRQF_SHARED;
break;
case MAXIM_DEVICE_TYPE_MAX14577:
default:
irq_chip = &max14577_irq_chip;
mfd_devs = max14577_devs;
mfd_devs_size = ARRAY_SIZE(max14577_devs);
- irq_flags = IRQF_TRIGGER_FALLING | IRQF_ONESHOT;
+ irq_flags = IRQF_ONESHOT;
break;
}
ret = devm_regmap_add_irq_chip(&i2c->dev, max77686->regmap,
max77686->irq,
- IRQF_TRIGGER_FALLING | IRQF_ONESHOT |
- IRQF_SHARED, 0, irq_chip,
+ IRQF_ONESHOT | IRQF_SHARED, 0, irq_chip,
&max77686->irq_data);
if (ret < 0) {
dev_err(&i2c->dev, "failed to add PMIC irq chip: %d\n", ret);
}
ret = regmap_add_irq_chip(max77693->regmap, max77693->irq,
- IRQF_ONESHOT | IRQF_SHARED |
- IRQF_TRIGGER_FALLING, 0,
+ IRQF_ONESHOT | IRQF_SHARED, 0,
&max77693_led_irq_chip,
&max77693->irq_data_led);
if (ret) {
}
ret = regmap_add_irq_chip(max77693->regmap, max77693->irq,
- IRQF_ONESHOT | IRQF_SHARED |
- IRQF_TRIGGER_FALLING, 0,
+ IRQF_ONESHOT | IRQF_SHARED, 0,
&max77693_topsys_irq_chip,
&max77693->irq_data_topsys);
if (ret) {
}
ret = regmap_add_irq_chip(max77693->regmap, max77693->irq,
- IRQF_ONESHOT | IRQF_SHARED |
- IRQF_TRIGGER_FALLING, 0,
+ IRQF_ONESHOT | IRQF_SHARED, 0,
&max77693_charger_irq_chip,
&max77693->irq_data_chg);
if (ret) {
}
ret = regmap_add_irq_chip(max77693->regmap_muic, max77693->irq,
- IRQF_ONESHOT | IRQF_SHARED |
- IRQF_TRIGGER_FALLING, 0,
+ IRQF_ONESHOT | IRQF_SHARED, 0,
&max77693_muic_irq_chip,
&max77693->irq_data_muic);
if (ret) {
}
EXPORT_SYMBOL_GPL(mc13xxx_common_init);
-int mc13xxx_common_exit(struct device *dev)
+void mc13xxx_common_exit(struct device *dev)
{
struct mc13xxx *mc13xxx = dev_get_drvdata(dev);
mfd_remove_devices(dev);
regmap_del_irq_chip(mc13xxx->irq, mc13xxx->irq_data);
mutex_destroy(&mc13xxx->lock);
-
- return 0;
}
EXPORT_SYMBOL_GPL(mc13xxx_common_exit);
static int mc13xxx_i2c_remove(struct i2c_client *client)
{
- return mc13xxx_common_exit(&client->dev);
+ mc13xxx_common_exit(&client->dev);
+ return 0;
}
static struct i2c_driver mc13xxx_i2c_driver = {
static int mc13xxx_spi_remove(struct spi_device *spi)
{
- return mc13xxx_common_exit(&spi->dev);
+ mc13xxx_common_exit(&spi->dev);
+ return 0;
}
static struct spi_driver mc13xxx_spi_driver = {
};
int mc13xxx_common_init(struct device *dev);
-int mc13xxx_common_exit(struct device *dev);
+void mc13xxx_common_exit(struct device *dev);
#endif /* __DRIVERS_MFD_MC13XXX_H */
if (of_device_is_compatible(np, cell->of_compatible)) {
/* Ignore 'disabled' devices error free */
if (!of_device_is_available(np)) {
+ of_node_put(np);
ret = 0;
goto fail_alias;
}
ret = mfd_match_of_node_to_dev(pdev, np, cell);
if (ret == -EAGAIN)
continue;
+ of_node_put(np);
if (ret)
goto fail_alias;
};
MODULE_DEVICE_TABLE(of, cpcap_of_match);
+static const struct spi_device_id cpcap_spi_ids[] = {
+ { .name = "cpcap", },
+ { .name = "6556002", },
+ {},
+};
+MODULE_DEVICE_TABLE(spi, cpcap_spi_ids);
+
static const struct regmap_config cpcap_regmap_config = {
.reg_bits = 16,
.reg_stride = 4,
.pm = &cpcap_pm,
},
.probe = cpcap_probe,
+ .id_table = cpcap_spi_ids,
};
module_spi_driver(cpcap_driver);
struct pm_irq_data {
int num_irqs;
struct irq_chip *irq_chip;
- void (*irq_handler)(struct irq_desc *desc);
+ irq_handler_t irq_handler;
};
struct pm_irq_chip {
return ret;
}
-static void pm8xxx_irq_handler(struct irq_desc *desc)
+static irqreturn_t pm8xxx_irq_handler(int irq, void *data)
{
- struct pm_irq_chip *chip = irq_desc_get_handler_data(desc);
- struct irq_chip *irq_chip = irq_desc_get_chip(desc);
+ struct pm_irq_chip *chip = data;
unsigned int root;
int i, ret, masters = 0;
- chained_irq_enter(irq_chip, desc);
-
ret = regmap_read(chip->regmap, SSBI_REG_ADDR_IRQ_ROOT, &root);
if (ret) {
pr_err("Can't read root status ret=%d\n", ret);
- return;
+ return IRQ_NONE;
}
/* on pm8xxx series masters start from bit 1 of the root */
if (masters & (1 << i))
pm8xxx_irq_master_handler(chip, i);
- chained_irq_exit(irq_chip, desc);
+ return IRQ_HANDLED;
}
static void pm8821_irq_block_handler(struct pm_irq_chip *chip,
pm8821_irq_block_handler(chip, master, block);
}
-static void pm8821_irq_handler(struct irq_desc *desc)
+static irqreturn_t pm8821_irq_handler(int irq, void *data)
{
- struct pm_irq_chip *chip = irq_desc_get_handler_data(desc);
- struct irq_chip *irq_chip = irq_desc_get_chip(desc);
+ struct pm_irq_chip *chip = data;
unsigned int master;
int ret;
- chained_irq_enter(irq_chip, desc);
ret = regmap_read(chip->regmap,
PM8821_SSBI_REG_ADDR_IRQ_MASTER0, &master);
if (ret) {
pr_err("Failed to read master 0 ret=%d\n", ret);
- goto done;
+ return IRQ_NONE;
}
/* bits 1 through 7 marks the first 7 blocks in master 0 */
/* bit 0 marks if master 1 contains any bits */
if (!(master & BIT(0)))
- goto done;
+ return IRQ_NONE;
ret = regmap_read(chip->regmap,
PM8821_SSBI_REG_ADDR_IRQ_MASTER1, &master);
if (ret) {
pr_err("Failed to read master 1 ret=%d\n", ret);
- goto done;
+ return IRQ_NONE;
}
pm8821_irq_master_handler(chip, 1, master);
-done:
- chained_irq_exit(irq_chip, desc);
+ return IRQ_HANDLED;
}
static void pm8xxx_irq_mask_ack(struct irq_data *d)
if (!chip->irqdomain)
return -ENODEV;
- irq_set_chained_handler_and_data(irq, data->irq_handler, chip);
+ rc = devm_request_irq(&pdev->dev, irq, data->irq_handler, 0, dev_name(&pdev->dev), chip);
+ if (rc)
+ return rc;
+
irq_set_irq_wake(irq, 1);
rc = of_platform_populate(pdev->dev.of_node, NULL, NULL, &pdev->dev);
- if (rc) {
- irq_set_chained_handler_and_data(irq, NULL, NULL);
+ if (rc)
irq_domain_remove(chip->irqdomain);
- }
return rc;
}
static int pm8xxx_remove(struct platform_device *pdev)
{
- int irq = platform_get_irq(pdev, 0);
struct pm_irq_chip *chip = platform_get_drvdata(pdev);
device_for_each_child(&pdev->dev, NULL, pm8xxx_remove_child);
- irq_set_chained_handler_and_data(irq, NULL, NULL);
irq_domain_remove(chip->irqdomain);
return 0;
#define PM8916_SUBTYPE 0x0b
#define PM8004_SUBTYPE 0x0c
#define PM8909_SUBTYPE 0x0d
+#define PM8028_SUBTYPE 0x0e
+#define PM8901_SUBTYPE 0x0f
#define PM8950_SUBTYPE 0x10
#define PMI8950_SUBTYPE 0x11
#define PM8998_SUBTYPE 0x14
#define PM8005_SUBTYPE 0x18
#define PM660L_SUBTYPE 0x1A
#define PM660_SUBTYPE 0x1B
+#define PM8150_SUBTYPE 0x1E
+#define PM8150L_SUBTYPE 0x1f
+#define PM8150B_SUBTYPE 0x20
+#define PMK8002_SUBTYPE 0x21
+#define PM8009_SUBTYPE 0x24
+#define PM8150C_SUBTYPE 0x26
+#define SMB2351_SUBTYPE 0x29
static const struct of_device_id pmic_spmi_id_table[] = {
- { .compatible = "qcom,spmi-pmic", .data = (void *)COMMON_SUBTYPE },
- { .compatible = "qcom,pm8941", .data = (void *)PM8941_SUBTYPE },
- { .compatible = "qcom,pm8841", .data = (void *)PM8841_SUBTYPE },
+ { .compatible = "qcom,pm660", .data = (void *)PM660_SUBTYPE },
+ { .compatible = "qcom,pm660l", .data = (void *)PM660L_SUBTYPE },
+ { .compatible = "qcom,pm8004", .data = (void *)PM8004_SUBTYPE },
+ { .compatible = "qcom,pm8005", .data = (void *)PM8005_SUBTYPE },
{ .compatible = "qcom,pm8019", .data = (void *)PM8019_SUBTYPE },
- { .compatible = "qcom,pm8226", .data = (void *)PM8226_SUBTYPE },
+ { .compatible = "qcom,pm8028", .data = (void *)PM8028_SUBTYPE },
{ .compatible = "qcom,pm8110", .data = (void *)PM8110_SUBTYPE },
- { .compatible = "qcom,pma8084", .data = (void *)PMA8084_SUBTYPE },
- { .compatible = "qcom,pmi8962", .data = (void *)PMI8962_SUBTYPE },
- { .compatible = "qcom,pmd9635", .data = (void *)PMD9635_SUBTYPE },
- { .compatible = "qcom,pm8994", .data = (void *)PM8994_SUBTYPE },
- { .compatible = "qcom,pmi8994", .data = (void *)PMI8994_SUBTYPE },
- { .compatible = "qcom,pm8916", .data = (void *)PM8916_SUBTYPE },
- { .compatible = "qcom,pm8004", .data = (void *)PM8004_SUBTYPE },
+ { .compatible = "qcom,pm8150", .data = (void *)PM8150_SUBTYPE },
+ { .compatible = "qcom,pm8150b", .data = (void *)PM8150B_SUBTYPE },
+ { .compatible = "qcom,pm8150c", .data = (void *)PM8150C_SUBTYPE },
+ { .compatible = "qcom,pm8150l", .data = (void *)PM8150L_SUBTYPE },
+ { .compatible = "qcom,pm8226", .data = (void *)PM8226_SUBTYPE },
+ { .compatible = "qcom,pm8841", .data = (void *)PM8841_SUBTYPE },
+ { .compatible = "qcom,pm8901", .data = (void *)PM8901_SUBTYPE },
{ .compatible = "qcom,pm8909", .data = (void *)PM8909_SUBTYPE },
+ { .compatible = "qcom,pm8916", .data = (void *)PM8916_SUBTYPE },
+ { .compatible = "qcom,pm8941", .data = (void *)PM8941_SUBTYPE },
{ .compatible = "qcom,pm8950", .data = (void *)PM8950_SUBTYPE },
- { .compatible = "qcom,pmi8950", .data = (void *)PMI8950_SUBTYPE },
+ { .compatible = "qcom,pm8994", .data = (void *)PM8994_SUBTYPE },
{ .compatible = "qcom,pm8998", .data = (void *)PM8998_SUBTYPE },
+ { .compatible = "qcom,pma8084", .data = (void *)PMA8084_SUBTYPE },
+ { .compatible = "qcom,pmd9635", .data = (void *)PMD9635_SUBTYPE },
+ { .compatible = "qcom,pmi8950", .data = (void *)PMI8950_SUBTYPE },
+ { .compatible = "qcom,pmi8962", .data = (void *)PMI8962_SUBTYPE },
+ { .compatible = "qcom,pmi8994", .data = (void *)PMI8994_SUBTYPE },
{ .compatible = "qcom,pmi8998", .data = (void *)PMI8998_SUBTYPE },
- { .compatible = "qcom,pm8005", .data = (void *)PM8005_SUBTYPE },
- { .compatible = "qcom,pm660l", .data = (void *)PM660L_SUBTYPE },
- { .compatible = "qcom,pm660", .data = (void *)PM660_SUBTYPE },
+ { .compatible = "qcom,pmk8002", .data = (void *)PMK8002_SUBTYPE },
+ { .compatible = "qcom,smb2351", .data = (void *)SMB2351_SUBTYPE },
+ { .compatible = "qcom,spmi-pmic", .data = (void *)COMMON_SUBTYPE },
{ }
};
reg = RK808_DEVCTRL_REG,
bit = DEV_OFF_RST;
break;
+ case RK817_ID:
+ reg = RK817_SYS_CFG(3);
+ bit = DEV_OFF;
+ break;
case RK818_ID:
reg = RK818_DEVCTRL_REG;
bit = DEV_OFF;
}
ret = devm_regmap_add_irq_chip(sec_pmic->dev, sec_pmic->regmap_pmic,
- sec_pmic->irq,
- IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
+ sec_pmic->irq, IRQF_ONESHOT,
0, sec_irq_chip, &sec_pmic->irq_data);
if (ret != 0) {
dev_err(sec_pmic->dev, "Failed to register IRQ chip: %d\n", ret);
#define SPRD_PMIC_INT_RAW_STATUS 0x4
#define SPRD_PMIC_INT_EN 0x8
+#define SPRD_SC2730_IRQ_BASE 0x80
+#define SPRD_SC2730_IRQ_NUMS 10
+#define SPRD_SC2730_CHG_DET 0x1b9c
#define SPRD_SC2731_IRQ_BASE 0x140
#define SPRD_SC2731_IRQ_NUMS 16
#define SPRD_SC2731_CHG_DET 0xedc
* base address and irq number, we should save irq number and irq base
* in the device data structure.
*/
+static const struct sprd_pmic_data sc2730_data = {
+ .irq_base = SPRD_SC2730_IRQ_BASE,
+ .num_irqs = SPRD_SC2730_IRQ_NUMS,
+ .charger_det = SPRD_SC2730_CHG_DET,
+};
+
static const struct sprd_pmic_data sc2731_data = {
.irq_base = SPRD_SC2731_IRQ_BASE,
.num_irqs = SPRD_SC2731_IRQ_NUMS,
static const struct of_device_id sprd_pmic_match[] = {
{ .compatible = "sprd,sc2731", .data = &sc2731_data },
+ { .compatible = "sprd,sc2730", .data = &sc2730_data },
{},
};
MODULE_DEVICE_TABLE(of, sprd_pmic_match);
+static const struct spi_device_id sprd_pmic_spi_ids[] = {
+ { .name = "sc2731", .driver_data = (unsigned long)&sc2731_data },
+ {},
+};
+MODULE_DEVICE_TABLE(spi, sprd_pmic_spi_ids);
+
static struct spi_driver sprd_pmic_driver = {
.driver = {
.name = "sc27xx-pmic",
.pm = &sprd_pmic_pm_ops,
},
.probe = sprd_pmic_probe,
+ .id_table = sprd_pmic_spi_ids,
};
static int __init sprd_pmic_init(void)
{
struct stmpe *stmpe = dev_get_drvdata(&i2c->dev);
- return stmpe_remove(stmpe);
+ stmpe_remove(stmpe);
+
+ return 0;
}
static const struct i2c_device_id stmpe_i2c_id[] = {
{
struct stmpe *stmpe = spi_get_drvdata(spi);
- return stmpe_remove(stmpe);
+ stmpe_remove(stmpe);
+
+ return 0;
}
static const struct of_device_id stmpe_spi_of_match[] = {
return ret;
}
-int stmpe_remove(struct stmpe *stmpe)
+void stmpe_remove(struct stmpe *stmpe)
{
if (!IS_ERR(stmpe->vio))
regulator_disable(stmpe->vio);
__stmpe_disable(stmpe, STMPE_BLOCK_ADC);
mfd_remove_devices(stmpe->dev);
-
- return 0;
}
#ifdef CONFIG_PM
};
int stmpe_probe(struct stmpe_client_info *ci, enum stmpe_partnum partnum);
-int stmpe_remove(struct stmpe *stmpe);
+void stmpe_remove(struct stmpe *stmpe);
#define STMPE_ICR_LSB_HIGH (1 << 2)
#define STMPE_ICR_LSB_EDGE (1 << 1)
+// SPDX-License-Identifier: GPL-2.0-only
/*
* TI Touch Screen / ADC MFD driver
*
* Copyright (C) 2012 Texas Instruments Incorporated - https://www.ti.com/
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed "as is" WITHOUT ANY WARRANTY of any
- * kind, whether express or implied; without even the implied warranty
- * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
#include <linux/module.h>
{
unsigned int idleconfig;
- idleconfig = STEPCONFIG_YNN | STEPCONFIG_INM_ADCREFM |
- STEPCONFIG_INP_ADCREFM | STEPCONFIG_YPN;
+ idleconfig = STEPCONFIG_INM_ADCREFM | STEPCONFIG_INP_ADCREFM;
+ if (ti_adc_with_touchscreen(tscadc))
+ idleconfig |= STEPCONFIG_YNN | STEPCONFIG_YPN;
regmap_write(tscadc->regmap, REG_IDLECONFIG, idleconfig);
}
static int ti_tscadc_probe(struct platform_device *pdev)
{
- struct ti_tscadc_dev *tscadc;
- struct resource *res;
- struct clk *clk;
- struct device_node *node;
- struct mfd_cell *cell;
- struct property *prop;
- const __be32 *cur;
- u32 val;
- int err, ctrl;
- int clock_rate;
- int tsc_wires = 0, adc_channels = 0, total_channels;
- int readouts = 0;
+ struct ti_tscadc_dev *tscadc;
+ struct resource *res;
+ struct clk *clk;
+ struct device_node *node;
+ struct mfd_cell *cell;
+ struct property *prop;
+ const __be32 *cur;
+ bool use_tsc = false, use_mag = false;
+ u32 val;
+ int err;
+ int tscmag_wires = 0, adc_channels = 0, cell_idx = 0, total_channels;
+ int readouts = 0, mag_tracks = 0;
+
+ /* Allocate memory for device */
+ tscadc = devm_kzalloc(&pdev->dev, sizeof(*tscadc), GFP_KERNEL);
+ if (!tscadc)
+ return -ENOMEM;
+
+ tscadc->dev = &pdev->dev;
if (!pdev->dev.of_node) {
dev_err(&pdev->dev, "Could not find valid DT data.\n");
return -EINVAL;
}
- node = of_get_child_by_name(pdev->dev.of_node, "tsc");
- of_property_read_u32(node, "ti,wires", &tsc_wires);
- of_property_read_u32(node, "ti,coordiante-readouts", &readouts);
+ tscadc->data = of_device_get_match_data(&pdev->dev);
+
+ if (ti_adc_with_touchscreen(tscadc)) {
+ node = of_get_child_by_name(pdev->dev.of_node, "tsc");
+ of_property_read_u32(node, "ti,wires", &tscmag_wires);
+ err = of_property_read_u32(node, "ti,coordinate-readouts",
+ &readouts);
+ if (err < 0)
+ of_property_read_u32(node, "ti,coordiante-readouts",
+ &readouts);
+
+ of_node_put(node);
+
+ if (tscmag_wires)
+ use_tsc = true;
+ } else {
+ /*
+ * When adding support for the magnetic stripe reader, here is
+ * the place to look for the number of tracks used from device
+ * tree. Let's default to 0 for now.
+ */
+ mag_tracks = 0;
+ tscmag_wires = mag_tracks * 2;
+ if (tscmag_wires)
+ use_mag = true;
+ }
node = of_get_child_by_name(pdev->dev.of_node, "adc");
of_property_for_each_u32(node, "ti,adc-channels", prop, cur, val) {
adc_channels++;
if (val > 7) {
dev_err(&pdev->dev, " PIN numbers are 0..7 (not %d)\n",
- val);
+ val);
+ of_node_put(node);
return -EINVAL;
}
}
- total_channels = tsc_wires + adc_channels;
+
+ of_node_put(node);
+
+ total_channels = tscmag_wires + adc_channels;
if (total_channels > 8) {
dev_err(&pdev->dev, "Number of i/p channels more than 8\n");
return -EINVAL;
}
+
if (total_channels == 0) {
- dev_err(&pdev->dev, "Need atleast one channel.\n");
+ dev_err(&pdev->dev, "Need at least one channel.\n");
return -EINVAL;
}
- if (readouts * 2 + 2 + adc_channels > 16) {
+ if (use_tsc && (readouts * 2 + 2 + adc_channels > 16)) {
dev_err(&pdev->dev, "Too many step configurations requested\n");
return -EINVAL;
}
- /* Allocate memory for device */
- tscadc = devm_kzalloc(&pdev->dev, sizeof(*tscadc), GFP_KERNEL);
- if (!tscadc)
- return -ENOMEM;
-
- tscadc->dev = &pdev->dev;
-
err = platform_get_irq(pdev, 0);
if (err < 0)
- goto ret;
+ return err;
else
tscadc->irq = err;
tscadc->tscadc_phys_base = res->start;
tscadc->regmap = devm_regmap_init_mmio(&pdev->dev,
- tscadc->tscadc_base, &tscadc_regmap_config);
+ tscadc->tscadc_base,
+ &tscadc_regmap_config);
if (IS_ERR(tscadc->regmap)) {
dev_err(&pdev->dev, "regmap init failed\n");
- err = PTR_ERR(tscadc->regmap);
- goto ret;
+ return PTR_ERR(tscadc->regmap);
}
spin_lock_init(&tscadc->reg_lock);
pm_runtime_get_sync(&pdev->dev);
/*
- * The TSC_ADC_Subsystem has 2 clock domains
- * OCP_CLK and ADC_CLK.
- * The ADC clock is expected to run at target of 3MHz,
- * and expected to capture 12-bit data at a rate of 200 KSPS.
- * The TSC_ADC_SS controller design assumes the OCP clock is
- * at least 6x faster than the ADC clock.
+ * The TSC_ADC_Subsystem has 2 clock domains: OCP_CLK and ADC_CLK.
+ * ADCs produce a 12-bit sample every 15 ADC_CLK cycles.
+ * am33xx ADCs expect to capture 200ksps.
+ * am47xx ADCs expect to capture 867ksps.
+ * We need ADC clocks respectively running at 3MHz and 13MHz.
+ * These frequencies are valid since TSC_ADC_SS controller design
+ * assumes the OCP clock is at least 6x faster than the ADC clock.
*/
- clk = devm_clk_get(&pdev->dev, "adc_tsc_fck");
+ clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(clk)) {
- dev_err(&pdev->dev, "failed to get TSC fck\n");
+ dev_err(&pdev->dev, "failed to get fck\n");
err = PTR_ERR(clk);
goto err_disable_clk;
}
- clock_rate = clk_get_rate(clk);
- tscadc->clk_div = clock_rate / ADC_CLK;
- /* TSCADC_CLKDIV needs to be configured to the value minus 1 */
- tscadc->clk_div--;
+ tscadc->clk_div = (clk_get_rate(clk) / tscadc->data->target_clk_rate) - 1;
regmap_write(tscadc->regmap, REG_CLKDIV, tscadc->clk_div);
- /* Set the control register bits */
- ctrl = CNTRLREG_STEPCONFIGWRT | CNTRLREG_STEPID;
- regmap_write(tscadc->regmap, REG_CTRL, ctrl);
-
- /* Set register bits for Idle Config Mode */
- if (tsc_wires > 0) {
- tscadc->tsc_wires = tsc_wires;
- if (tsc_wires == 5)
- ctrl |= CNTRLREG_5WIRE | CNTRLREG_TSCENB;
- else
- ctrl |= CNTRLREG_4WIRE | CNTRLREG_TSCENB;
- tscadc_idle_config(tscadc);
+ /*
+ * Set the control register bits. tscadc->ctrl stores the configuration
+ * of the CTRL register but not the subsystem enable bit which must be
+ * added manually when timely.
+ */
+ tscadc->ctrl = CNTRLREG_STEPID;
+ if (ti_adc_with_touchscreen(tscadc)) {
+ tscadc->ctrl |= CNTRLREG_TSC_STEPCONFIGWRT;
+ if (use_tsc) {
+ tscadc->ctrl |= CNTRLREG_TSC_ENB;
+ if (tscmag_wires == 5)
+ tscadc->ctrl |= CNTRLREG_TSC_5WIRE;
+ else
+ tscadc->ctrl |= CNTRLREG_TSC_4WIRE;
+ }
+ } else {
+ tscadc->ctrl |= CNTRLREG_MAG_PREAMP_PWRDOWN |
+ CNTRLREG_MAG_PREAMP_BYPASS;
}
+ regmap_write(tscadc->regmap, REG_CTRL, tscadc->ctrl);
+
+ tscadc_idle_config(tscadc);
/* Enable the TSC module enable bit */
- ctrl |= CNTRLREG_TSCSSENB;
- regmap_write(tscadc->regmap, REG_CTRL, ctrl);
-
- tscadc->used_cells = 0;
- tscadc->tsc_cell = -1;
- tscadc->adc_cell = -1;
-
- /* TSC Cell */
- if (tsc_wires > 0) {
- tscadc->tsc_cell = tscadc->used_cells;
- cell = &tscadc->cells[tscadc->used_cells++];
- cell->name = "TI-am335x-tsc";
- cell->of_compatible = "ti,am3359-tsc";
+ regmap_write(tscadc->regmap, REG_CTRL, tscadc->ctrl | CNTRLREG_SSENB);
+
+ /* TSC or MAG Cell */
+ if (use_tsc || use_mag) {
+ cell = &tscadc->cells[cell_idx++];
+ cell->name = tscadc->data->secondary_feature_name;
+ cell->of_compatible = tscadc->data->secondary_feature_compatible;
cell->platform_data = &tscadc;
cell->pdata_size = sizeof(tscadc);
}
/* ADC Cell */
if (adc_channels > 0) {
- tscadc->adc_cell = tscadc->used_cells;
- cell = &tscadc->cells[tscadc->used_cells++];
- cell->name = "TI-am335x-adc";
- cell->of_compatible = "ti,am3359-adc";
+ cell = &tscadc->cells[cell_idx++];
+ cell->name = tscadc->data->adc_feature_name;
+ cell->of_compatible = tscadc->data->adc_feature_compatible;
cell->platform_data = &tscadc;
cell->pdata_size = sizeof(tscadc);
}
err = mfd_add_devices(&pdev->dev, PLATFORM_DEVID_AUTO,
- tscadc->cells, tscadc->used_cells, NULL,
- 0, NULL);
+ tscadc->cells, cell_idx, NULL, 0, NULL);
if (err < 0)
goto err_disable_clk;
err_disable_clk:
pm_runtime_put_sync(&pdev->dev);
pm_runtime_disable(&pdev->dev);
-ret:
+
return err;
}
static int ti_tscadc_remove(struct platform_device *pdev)
{
- struct ti_tscadc_dev *tscadc = platform_get_drvdata(pdev);
+ struct ti_tscadc_dev *tscadc = platform_get_drvdata(pdev);
regmap_write(tscadc->regmap, REG_SE, 0x00);
static int __maybe_unused tscadc_suspend(struct device *dev)
{
- struct ti_tscadc_dev *tscadc = dev_get_drvdata(dev);
+ struct ti_tscadc_dev *tscadc = dev_get_drvdata(dev);
regmap_write(tscadc->regmap, REG_SE, 0x00);
if (device_for_each_child(dev, NULL, ti_tscadc_can_wakeup)) {
regmap_read(tscadc->regmap, REG_CTRL, &ctrl);
ctrl &= ~(CNTRLREG_POWERDOWN);
- ctrl |= CNTRLREG_TSCSSENB;
+ ctrl |= CNTRLREG_SSENB;
regmap_write(tscadc->regmap, REG_CTRL, ctrl);
}
pm_runtime_put_sync(dev);
static int __maybe_unused tscadc_resume(struct device *dev)
{
- struct ti_tscadc_dev *tscadc = dev_get_drvdata(dev);
- u32 ctrl;
+ struct ti_tscadc_dev *tscadc = dev_get_drvdata(dev);
pm_runtime_get_sync(dev);
- /* context restore */
- ctrl = CNTRLREG_STEPCONFIGWRT | CNTRLREG_STEPID;
- regmap_write(tscadc->regmap, REG_CTRL, ctrl);
-
- if (tscadc->tsc_cell != -1) {
- if (tscadc->tsc_wires == 5)
- ctrl |= CNTRLREG_5WIRE | CNTRLREG_TSCENB;
- else
- ctrl |= CNTRLREG_4WIRE | CNTRLREG_TSCENB;
- tscadc_idle_config(tscadc);
- }
- ctrl |= CNTRLREG_TSCSSENB;
- regmap_write(tscadc->regmap, REG_CTRL, ctrl);
-
regmap_write(tscadc->regmap, REG_CLKDIV, tscadc->clk_div);
+ regmap_write(tscadc->regmap, REG_CTRL, tscadc->ctrl);
+ tscadc_idle_config(tscadc);
+ regmap_write(tscadc->regmap, REG_CTRL, tscadc->ctrl | CNTRLREG_SSENB);
return 0;
}
static SIMPLE_DEV_PM_OPS(tscadc_pm_ops, tscadc_suspend, tscadc_resume);
+static const struct ti_tscadc_data tscdata = {
+ .adc_feature_name = "TI-am335x-adc",
+ .adc_feature_compatible = "ti,am3359-adc",
+ .secondary_feature_name = "TI-am335x-tsc",
+ .secondary_feature_compatible = "ti,am3359-tsc",
+ .target_clk_rate = TSC_ADC_CLK,
+};
+
+static const struct ti_tscadc_data magdata = {
+ .adc_feature_name = "TI-am43xx-adc",
+ .adc_feature_compatible = "ti,am4372-adc",
+ .secondary_feature_name = "TI-am43xx-mag",
+ .secondary_feature_compatible = "ti,am4372-mag",
+ .target_clk_rate = MAG_ADC_CLK,
+};
+
static const struct of_device_id ti_tscadc_dt_ids[] = {
- { .compatible = "ti,am3359-tscadc", },
+ { .compatible = "ti,am3359-tscadc", .data = &tscdata },
+ { .compatible = "ti,am4372-magadc", .data = &magdata },
{ }
};
MODULE_DEVICE_TABLE(of, ti_tscadc_dt_ids);
module_platform_driver(ti_tscadc_driver);
-MODULE_DESCRIPTION("TI touchscreen / ADC MFD controller driver");
+MODULE_DESCRIPTION("TI touchscreen/magnetic stripe reader/ADC MFD controller driver");
MODULE_AUTHOR("Rachna Patil <rachna@ti.com>");
MODULE_LICENSE("GPL");
}
EXPORT_SYMBOL_GPL(tps65912_device_init);
-int tps65912_device_exit(struct tps65912 *tps)
+void tps65912_device_exit(struct tps65912 *tps)
{
regmap_del_irq_chip(tps->irq, tps->irq_data);
-
- return 0;
}
EXPORT_SYMBOL_GPL(tps65912_device_exit);
{
struct tps65912 *tps = i2c_get_clientdata(client);
- return tps65912_device_exit(tps);
+ tps65912_device_exit(tps);
+
+ return 0;
}
static const struct i2c_device_id tps65912_i2c_id_table[] = {
{
struct tps65912 *tps = spi_get_drvdata(spi);
- return tps65912_device_exit(tps);
+ tps65912_device_exit(tps);
+
+ return 0;
}
static const struct spi_device_id tps65912_spi_id_table[] = {
+++ /dev/null
-/*
- * tps80031.c -- TI TPS80031/TPS80032 mfd core driver.
- *
- * MFD core driver for TI TPS80031/TPS80032 Fully Integrated
- * Power Management with Power Path and Battery Charger
- *
- * Copyright (c) 2012, NVIDIA Corporation.
- *
- * Author: Laxman Dewangan <ldewangan@nvidia.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed "as is" WITHOUT ANY WARRANTY of any kind,
- * whether express or implied; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
- * 02111-1307, USA
- */
-
-#include <linux/err.h>
-#include <linux/i2c.h>
-#include <linux/init.h>
-#include <linux/interrupt.h>
-#include <linux/irq.h>
-#include <linux/mfd/core.h>
-#include <linux/mfd/tps80031.h>
-#include <linux/pm.h>
-#include <linux/regmap.h>
-#include <linux/slab.h>
-
-static const struct resource tps80031_rtc_resources[] = {
- DEFINE_RES_IRQ(TPS80031_INT_RTC_ALARM),
-};
-
-/* TPS80031 sub mfd devices */
-static const struct mfd_cell tps80031_cell[] = {
- {
- .name = "tps80031-pmic",
- },
- {
- .name = "tps80031-clock",
- },
- {
- .name = "tps80031-rtc",
- .num_resources = ARRAY_SIZE(tps80031_rtc_resources),
- .resources = tps80031_rtc_resources,
- },
- {
- .name = "tps80031-gpadc",
- },
- {
- .name = "tps80031-fuel-gauge",
- },
- {
- .name = "tps80031-charger",
- },
-};
-
-static int tps80031_slave_address[TPS80031_NUM_SLAVES] = {
- TPS80031_I2C_ID0_ADDR,
- TPS80031_I2C_ID1_ADDR,
- TPS80031_I2C_ID2_ADDR,
- TPS80031_I2C_ID3_ADDR,
-};
-
-struct tps80031_pupd_data {
- u8 reg;
- u8 pullup_bit;
- u8 pulldown_bit;
-};
-
-#define TPS80031_IRQ(_reg, _mask) \
- { \
- .reg_offset = (TPS80031_INT_MSK_LINE_##_reg) - \
- TPS80031_INT_MSK_LINE_A, \
- .mask = BIT(_mask), \
- }
-
-static const struct regmap_irq tps80031_main_irqs[] = {
- [TPS80031_INT_PWRON] = TPS80031_IRQ(A, 0),
- [TPS80031_INT_RPWRON] = TPS80031_IRQ(A, 1),
- [TPS80031_INT_SYS_VLOW] = TPS80031_IRQ(A, 2),
- [TPS80031_INT_RTC_ALARM] = TPS80031_IRQ(A, 3),
- [TPS80031_INT_RTC_PERIOD] = TPS80031_IRQ(A, 4),
- [TPS80031_INT_HOT_DIE] = TPS80031_IRQ(A, 5),
- [TPS80031_INT_VXX_SHORT] = TPS80031_IRQ(A, 6),
- [TPS80031_INT_SPDURATION] = TPS80031_IRQ(A, 7),
- [TPS80031_INT_WATCHDOG] = TPS80031_IRQ(B, 0),
- [TPS80031_INT_BAT] = TPS80031_IRQ(B, 1),
- [TPS80031_INT_SIM] = TPS80031_IRQ(B, 2),
- [TPS80031_INT_MMC] = TPS80031_IRQ(B, 3),
- [TPS80031_INT_RES] = TPS80031_IRQ(B, 4),
- [TPS80031_INT_GPADC_RT] = TPS80031_IRQ(B, 5),
- [TPS80031_INT_GPADC_SW2_EOC] = TPS80031_IRQ(B, 6),
- [TPS80031_INT_CC_AUTOCAL] = TPS80031_IRQ(B, 7),
- [TPS80031_INT_ID_WKUP] = TPS80031_IRQ(C, 0),
- [TPS80031_INT_VBUSS_WKUP] = TPS80031_IRQ(C, 1),
- [TPS80031_INT_ID] = TPS80031_IRQ(C, 2),
- [TPS80031_INT_VBUS] = TPS80031_IRQ(C, 3),
- [TPS80031_INT_CHRG_CTRL] = TPS80031_IRQ(C, 4),
- [TPS80031_INT_EXT_CHRG] = TPS80031_IRQ(C, 5),
- [TPS80031_INT_INT_CHRG] = TPS80031_IRQ(C, 6),
- [TPS80031_INT_RES2] = TPS80031_IRQ(C, 7),
-};
-
-static struct regmap_irq_chip tps80031_irq_chip = {
- .name = "tps80031",
- .irqs = tps80031_main_irqs,
- .num_irqs = ARRAY_SIZE(tps80031_main_irqs),
- .num_regs = 3,
- .status_base = TPS80031_INT_STS_A,
- .mask_base = TPS80031_INT_MSK_LINE_A,
-};
-
-#define PUPD_DATA(_reg, _pulldown_bit, _pullup_bit) \
- { \
- .reg = TPS80031_CFG_INPUT_PUPD##_reg, \
- .pulldown_bit = _pulldown_bit, \
- .pullup_bit = _pullup_bit, \
- }
-
-static const struct tps80031_pupd_data tps80031_pupds[] = {
- [TPS80031_PREQ1] = PUPD_DATA(1, BIT(0), BIT(1)),
- [TPS80031_PREQ2A] = PUPD_DATA(1, BIT(2), BIT(3)),
- [TPS80031_PREQ2B] = PUPD_DATA(1, BIT(4), BIT(5)),
- [TPS80031_PREQ2C] = PUPD_DATA(1, BIT(6), BIT(7)),
- [TPS80031_PREQ3] = PUPD_DATA(2, BIT(0), BIT(1)),
- [TPS80031_NRES_WARM] = PUPD_DATA(2, 0, BIT(2)),
- [TPS80031_PWM_FORCE] = PUPD_DATA(2, BIT(5), 0),
- [TPS80031_CHRG_EXT_CHRG_STATZ] = PUPD_DATA(2, 0, BIT(6)),
- [TPS80031_SIM] = PUPD_DATA(3, BIT(0), BIT(1)),
- [TPS80031_MMC] = PUPD_DATA(3, BIT(2), BIT(3)),
- [TPS80031_GPADC_START] = PUPD_DATA(3, BIT(4), 0),
- [TPS80031_DVSI2C_SCL] = PUPD_DATA(4, 0, BIT(0)),
- [TPS80031_DVSI2C_SDA] = PUPD_DATA(4, 0, BIT(1)),
- [TPS80031_CTLI2C_SCL] = PUPD_DATA(4, 0, BIT(2)),
- [TPS80031_CTLI2C_SDA] = PUPD_DATA(4, 0, BIT(3)),
-};
-static struct tps80031 *tps80031_power_off_dev;
-
-int tps80031_ext_power_req_config(struct device *dev,
- unsigned long ext_ctrl_flag, int preq_bit,
- int state_reg_add, int trans_reg_add)
-{
- u8 res_ass_reg = 0;
- int preq_mask_bit = 0;
- int ret;
-
- if (!(ext_ctrl_flag & TPS80031_EXT_PWR_REQ))
- return 0;
-
- if (ext_ctrl_flag & TPS80031_PWR_REQ_INPUT_PREQ1) {
- res_ass_reg = TPS80031_PREQ1_RES_ASS_A + (preq_bit >> 3);
- preq_mask_bit = 5;
- } else if (ext_ctrl_flag & TPS80031_PWR_REQ_INPUT_PREQ2) {
- res_ass_reg = TPS80031_PREQ2_RES_ASS_A + (preq_bit >> 3);
- preq_mask_bit = 6;
- } else if (ext_ctrl_flag & TPS80031_PWR_REQ_INPUT_PREQ3) {
- res_ass_reg = TPS80031_PREQ3_RES_ASS_A + (preq_bit >> 3);
- preq_mask_bit = 7;
- }
-
- /* Configure REQ_ASS registers */
- ret = tps80031_set_bits(dev, TPS80031_SLAVE_ID1, res_ass_reg,
- BIT(preq_bit & 0x7));
- if (ret < 0) {
- dev_err(dev, "reg 0x%02x setbit failed, err = %d\n",
- res_ass_reg, ret);
- return ret;
- }
-
- /* Unmask the PREQ */
- ret = tps80031_clr_bits(dev, TPS80031_SLAVE_ID1,
- TPS80031_PHOENIX_MSK_TRANSITION, BIT(preq_mask_bit));
- if (ret < 0) {
- dev_err(dev, "reg 0x%02x clrbit failed, err = %d\n",
- TPS80031_PHOENIX_MSK_TRANSITION, ret);
- return ret;
- }
-
- /* Switch regulator control to resource now */
- if (ext_ctrl_flag & (TPS80031_PWR_REQ_INPUT_PREQ2 |
- TPS80031_PWR_REQ_INPUT_PREQ3)) {
- ret = tps80031_update(dev, TPS80031_SLAVE_ID1, state_reg_add,
- 0x0, TPS80031_STATE_MASK);
- if (ret < 0)
- dev_err(dev, "reg 0x%02x update failed, err = %d\n",
- state_reg_add, ret);
- } else {
- ret = tps80031_update(dev, TPS80031_SLAVE_ID1, trans_reg_add,
- TPS80031_TRANS_SLEEP_OFF,
- TPS80031_TRANS_SLEEP_MASK);
- if (ret < 0)
- dev_err(dev, "reg 0x%02x update failed, err = %d\n",
- trans_reg_add, ret);
- }
- return ret;
-}
-EXPORT_SYMBOL_GPL(tps80031_ext_power_req_config);
-
-static void tps80031_power_off(void)
-{
- dev_info(tps80031_power_off_dev->dev, "switching off PMU\n");
- tps80031_write(tps80031_power_off_dev->dev, TPS80031_SLAVE_ID1,
- TPS80031_PHOENIX_DEV_ON, TPS80031_DEVOFF);
-}
-
-static void tps80031_pupd_init(struct tps80031 *tps80031,
- struct tps80031_platform_data *pdata)
-{
- struct tps80031_pupd_init_data *pupd_init_data = pdata->pupd_init_data;
- int data_size = pdata->pupd_init_data_size;
- int i;
-
- for (i = 0; i < data_size; ++i) {
- struct tps80031_pupd_init_data *pupd_init = &pupd_init_data[i];
- const struct tps80031_pupd_data *pupd =
- &tps80031_pupds[pupd_init->input_pin];
- u8 update_value = 0;
- u8 update_mask = pupd->pulldown_bit | pupd->pullup_bit;
-
- if (pupd_init->setting == TPS80031_PUPD_PULLDOWN)
- update_value = pupd->pulldown_bit;
- else if (pupd_init->setting == TPS80031_PUPD_PULLUP)
- update_value = pupd->pullup_bit;
-
- tps80031_update(tps80031->dev, TPS80031_SLAVE_ID1, pupd->reg,
- update_value, update_mask);
- }
-}
-
-static int tps80031_init_ext_control(struct tps80031 *tps80031,
- struct tps80031_platform_data *pdata)
-{
- struct device *dev = tps80031->dev;
- int ret;
- int i;
-
- /* Clear all external control for this rail */
- for (i = 0; i < 9; ++i) {
- ret = tps80031_write(dev, TPS80031_SLAVE_ID1,
- TPS80031_PREQ1_RES_ASS_A + i, 0);
- if (ret < 0) {
- dev_err(dev, "reg 0x%02x write failed, err = %d\n",
- TPS80031_PREQ1_RES_ASS_A + i, ret);
- return ret;
- }
- }
-
- /* Mask the PREQ */
- ret = tps80031_set_bits(dev, TPS80031_SLAVE_ID1,
- TPS80031_PHOENIX_MSK_TRANSITION, 0x7 << 5);
- if (ret < 0) {
- dev_err(dev, "reg 0x%02x set_bits failed, err = %d\n",
- TPS80031_PHOENIX_MSK_TRANSITION, ret);
- return ret;
- }
- return ret;
-}
-
-static int tps80031_irq_init(struct tps80031 *tps80031, int irq, int irq_base)
-{
- struct device *dev = tps80031->dev;
- int i, ret;
-
- /*
- * The MASK register used for updating status register when
- * interrupt occurs and LINE register used to pass the status
- * to actual interrupt line. As per datasheet:
- * When INT_MSK_LINE [i] is set to 1, the associated interrupt
- * number i is INT line masked, which means that no interrupt is
- * generated on the INT line.
- * When INT_MSK_LINE [i] is set to 0, the associated interrupt
- * number i is line enabled: An interrupt is generated on the
- * INT line.
- * In any case, the INT_STS [i] status bit may or may not be updated,
- * only linked to the INT_MSK_STS [i] configuration register bit.
- *
- * When INT_MSK_STS [i] is set to 1, the associated interrupt number
- * i is status masked, which means that no interrupt is stored in
- * the INT_STS[i] status bit. Note that no interrupt number i is
- * generated on the INT line, even if the INT_MSK_LINE [i] register
- * bit is set to 0.
- * When INT_MSK_STS [i] is set to 0, the associated interrupt number i
- * is status enabled: An interrupt status is updated in the INT_STS [i]
- * register. The interrupt may or may not be generated on the INT line,
- * depending on the INT_MSK_LINE [i] configuration register bit.
- */
- for (i = 0; i < 3; i++)
- tps80031_write(dev, TPS80031_SLAVE_ID2,
- TPS80031_INT_MSK_STS_A + i, 0x00);
-
- ret = regmap_add_irq_chip(tps80031->regmap[TPS80031_SLAVE_ID2], irq,
- IRQF_ONESHOT, irq_base,
- &tps80031_irq_chip, &tps80031->irq_data);
- if (ret < 0) {
- dev_err(dev, "add irq failed, err = %d\n", ret);
- return ret;
- }
- return ret;
-}
-
-static bool rd_wr_reg_id0(struct device *dev, unsigned int reg)
-{
- switch (reg) {
- case TPS80031_SMPS1_CFG_FORCE ... TPS80031_SMPS2_CFG_VOLTAGE:
- return true;
- default:
- return false;
- }
-}
-
-static bool rd_wr_reg_id1(struct device *dev, unsigned int reg)
-{
- switch (reg) {
- case TPS80031_SECONDS_REG ... TPS80031_RTC_RESET_STATUS_REG:
- case TPS80031_VALIDITY0 ... TPS80031_VALIDITY7:
- case TPS80031_PHOENIX_START_CONDITION ... TPS80031_KEY_PRESS_DUR_CFG:
- case TPS80031_SMPS4_CFG_TRANS ... TPS80031_SMPS3_CFG_VOLTAGE:
- case TPS80031_BROADCAST_ADDR_ALL ... TPS80031_BROADCAST_ADDR_CLK_RST:
- case TPS80031_VANA_CFG_TRANS ... TPS80031_LDO7_CFG_VOLTAGE:
- case TPS80031_REGEN1_CFG_TRANS ... TPS80031_TMP_CFG_STATE:
- case TPS80031_PREQ1_RES_ASS_A ... TPS80031_PREQ3_RES_ASS_C:
- case TPS80031_SMPS_OFFSET ... TPS80031_BATDEBOUNCING:
- case TPS80031_CFG_INPUT_PUPD1 ... TPS80031_CFG_SMPS_PD:
- case TPS80031_BACKUP_REG:
- return true;
- default:
- return false;
- }
-}
-
-static bool is_volatile_reg_id1(struct device *dev, unsigned int reg)
-{
- switch (reg) {
- case TPS80031_SMPS4_CFG_TRANS ... TPS80031_SMPS3_CFG_VOLTAGE:
- case TPS80031_VANA_CFG_TRANS ... TPS80031_LDO7_CFG_VOLTAGE:
- case TPS80031_REGEN1_CFG_TRANS ... TPS80031_TMP_CFG_STATE:
- case TPS80031_PREQ1_RES_ASS_A ... TPS80031_PREQ3_RES_ASS_C:
- case TPS80031_SMPS_OFFSET ... TPS80031_BATDEBOUNCING:
- case TPS80031_CFG_INPUT_PUPD1 ... TPS80031_CFG_SMPS_PD:
- return true;
- default:
- return false;
- }
-}
-
-static bool rd_wr_reg_id2(struct device *dev, unsigned int reg)
-{
- switch (reg) {
- case TPS80031_USB_VENDOR_ID_LSB ... TPS80031_USB_OTG_REVISION:
- case TPS80031_GPADC_CTRL ... TPS80031_CTRL_P1:
- case TPS80031_RTCH0_LSB ... TPS80031_GPCH0_MSB:
- case TPS80031_TOGGLE1 ... TPS80031_VIBMODE:
- case TPS80031_PWM1ON ... TPS80031_PWM2OFF:
- case TPS80031_FG_REG_00 ... TPS80031_FG_REG_11:
- case TPS80031_INT_STS_A ... TPS80031_INT_MSK_STS_C:
- case TPS80031_CONTROLLER_CTRL2 ... TPS80031_LED_PWM_CTRL2:
- return true;
- default:
- return false;
- }
-}
-
-static bool rd_wr_reg_id3(struct device *dev, unsigned int reg)
-{
- switch (reg) {
- case TPS80031_GPADC_TRIM0 ... TPS80031_GPADC_TRIM18:
- return true;
- default:
- return false;
- }
-}
-
-static const struct regmap_config tps80031_regmap_configs[] = {
- {
- .reg_bits = 8,
- .val_bits = 8,
- .writeable_reg = rd_wr_reg_id0,
- .readable_reg = rd_wr_reg_id0,
- .max_register = TPS80031_MAX_REGISTER,
- },
- {
- .reg_bits = 8,
- .val_bits = 8,
- .writeable_reg = rd_wr_reg_id1,
- .readable_reg = rd_wr_reg_id1,
- .volatile_reg = is_volatile_reg_id1,
- .max_register = TPS80031_MAX_REGISTER,
- },
- {
- .reg_bits = 8,
- .val_bits = 8,
- .writeable_reg = rd_wr_reg_id2,
- .readable_reg = rd_wr_reg_id2,
- .max_register = TPS80031_MAX_REGISTER,
- },
- {
- .reg_bits = 8,
- .val_bits = 8,
- .writeable_reg = rd_wr_reg_id3,
- .readable_reg = rd_wr_reg_id3,
- .max_register = TPS80031_MAX_REGISTER,
- },
-};
-
-static int tps80031_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
-{
- struct tps80031_platform_data *pdata = dev_get_platdata(&client->dev);
- struct tps80031 *tps80031;
- int ret;
- uint8_t es_version;
- uint8_t ep_ver;
- int i;
-
- if (!pdata) {
- dev_err(&client->dev, "tps80031 requires platform data\n");
- return -EINVAL;
- }
-
- tps80031 = devm_kzalloc(&client->dev, sizeof(*tps80031), GFP_KERNEL);
- if (!tps80031)
- return -ENOMEM;
-
- for (i = 0; i < TPS80031_NUM_SLAVES; i++) {
- if (tps80031_slave_address[i] == client->addr)
- tps80031->clients[i] = client;
- else
- tps80031->clients[i] = devm_i2c_new_dummy_device(&client->dev,
- client->adapter, tps80031_slave_address[i]);
- if (IS_ERR(tps80031->clients[i])) {
- dev_err(&client->dev, "can't attach client %d\n", i);
- return PTR_ERR(tps80031->clients[i]);
- }
-
- i2c_set_clientdata(tps80031->clients[i], tps80031);
- tps80031->regmap[i] = devm_regmap_init_i2c(tps80031->clients[i],
- &tps80031_regmap_configs[i]);
- if (IS_ERR(tps80031->regmap[i])) {
- ret = PTR_ERR(tps80031->regmap[i]);
- dev_err(&client->dev,
- "regmap %d init failed, err %d\n", i, ret);
- return ret;
- }
- }
-
- ret = tps80031_read(&client->dev, TPS80031_SLAVE_ID3,
- TPS80031_JTAGVERNUM, &es_version);
- if (ret < 0) {
- dev_err(&client->dev,
- "Silicon version number read failed: %d\n", ret);
- return ret;
- }
-
- ret = tps80031_read(&client->dev, TPS80031_SLAVE_ID3,
- TPS80031_EPROM_REV, &ep_ver);
- if (ret < 0) {
- dev_err(&client->dev,
- "Silicon eeprom version read failed: %d\n", ret);
- return ret;
- }
-
- dev_info(&client->dev, "ES version 0x%02x and EPROM version 0x%02x\n",
- es_version, ep_ver);
- tps80031->es_version = es_version;
- tps80031->dev = &client->dev;
- i2c_set_clientdata(client, tps80031);
- tps80031->chip_info = id->driver_data;
-
- ret = tps80031_irq_init(tps80031, client->irq, pdata->irq_base);
- if (ret) {
- dev_err(&client->dev, "IRQ init failed: %d\n", ret);
- return ret;
- }
-
- tps80031_pupd_init(tps80031, pdata);
-
- tps80031_init_ext_control(tps80031, pdata);
-
- ret = mfd_add_devices(tps80031->dev, -1,
- tps80031_cell, ARRAY_SIZE(tps80031_cell),
- NULL, 0,
- regmap_irq_get_domain(tps80031->irq_data));
- if (ret < 0) {
- dev_err(&client->dev, "mfd_add_devices failed: %d\n", ret);
- goto fail_mfd_add;
- }
-
- if (pdata->use_power_off && !pm_power_off) {
- tps80031_power_off_dev = tps80031;
- pm_power_off = tps80031_power_off;
- }
- return 0;
-
-fail_mfd_add:
- regmap_del_irq_chip(client->irq, tps80031->irq_data);
- return ret;
-}
-
-static const struct i2c_device_id tps80031_id_table[] = {
- { "tps80031", TPS80031 },
- { "tps80032", TPS80032 },
- { }
-};
-
-static struct i2c_driver tps80031_driver = {
- .driver = {
- .name = "tps80031",
- .suppress_bind_attrs = true,
- },
- .probe = tps80031_probe,
- .id_table = tps80031_id_table,
-};
-
-static int __init tps80031_init(void)
-{
- return i2c_add_driver(&tps80031_driver);
-}
-subsys_initcall(tps80031_init);
// Copyright (c) 2019, Linaro Limited
#include <linux/clk.h>
-#include <linux/gpio.h>
+#include <linux/gpio/consumer.h>
#include <linux/interrupt.h>
#include <linux/kernel.h>
#include <linux/mfd/core.h>
#include <linux/mfd/wcd934x/registers.h>
#include <linux/mfd/wcd934x/wcd934x.h>
#include <linux/module.h>
-#include <linux/of_gpio.h>
#include <linux/of.h>
#include <linux/of_irq.h>
#include <linux/platform_device.h>
struct device *dev = &sdev->dev;
struct device_node *np = dev->of_node;
struct wcd934x_ddata *ddata;
- int reset_gpio, ret;
+ struct gpio_desc *reset_gpio;
+ int ret;
ddata = devm_kzalloc(dev, sizeof(*ddata), GFP_KERNEL);
if (!ddata)
return dev_err_probe(ddata->dev, ddata->irq,
"Failed to get IRQ\n");
- reset_gpio = of_get_named_gpio(np, "reset-gpios", 0);
- if (reset_gpio < 0) {
- dev_err(dev, "Failed to get reset gpio: err = %d\n",
- reset_gpio);
- return reset_gpio;
- }
-
ddata->extclk = devm_clk_get(dev, "extclk");
if (IS_ERR(ddata->extclk)) {
dev_err(dev, "Failed to get extclk");
* SYS_RST_N shouldn't be pulled high during this time
*/
usleep_range(600, 650);
- gpio_direction_output(reset_gpio, 0);
+ reset_gpio = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
+ if (IS_ERR(reset_gpio)) {
+ return dev_err_probe(dev, PTR_ERR(reset_gpio),
+ "Failed to get reset gpio: err = %ld\n", PTR_ERR(reset_gpio));
+ }
msleep(20);
- gpio_set_value(reset_gpio, 1);
+ gpiod_set_value(reset_gpio, 1);
msleep(20);
ddata->dev = dev;
pci_channel_state_t state)
{
struct pci_dev *afu_dev;
+ struct pci_driver *afu_drv;
+ const struct pci_error_handlers *err_handler;
if (afu->phb == NULL)
return;
list_for_each_entry(afu_dev, &afu->phb->bus->devices, bus_list) {
- if (!afu_dev->driver)
+ afu_drv = to_pci_driver(afu_dev->dev.driver);
+ if (!afu_drv)
continue;
+ err_handler = afu_drv->err_handler;
switch (bus_error_event) {
case CXL_ERROR_DETECTED_EVENT:
afu_dev->error_state = state;
- if (afu_dev->driver->err_handler &&
- afu_dev->driver->err_handler->error_detected)
- afu_dev->driver->err_handler->error_detected(afu_dev, state);
- break;
+ if (err_handler &&
+ err_handler->error_detected)
+ err_handler->error_detected(afu_dev, state);
+ break;
case CXL_SLOT_RESET_EVENT:
afu_dev->error_state = state;
- if (afu_dev->driver->err_handler &&
- afu_dev->driver->err_handler->slot_reset)
- afu_dev->driver->err_handler->slot_reset(afu_dev);
- break;
+ if (err_handler &&
+ err_handler->slot_reset)
+ err_handler->slot_reset(afu_dev);
+ break;
case CXL_RESUME_EVENT:
- if (afu_dev->driver->err_handler &&
- afu_dev->driver->err_handler->resume)
- afu_dev->driver->err_handler->resume(afu_dev);
- break;
+ if (err_handler &&
+ err_handler->resume)
+ err_handler->resume(afu_dev);
+ break;
}
}
}
pci_channel_state_t state)
{
struct pci_dev *afu_dev;
+ struct pci_driver *afu_drv;
+ const struct pci_error_handlers *err_handler;
pci_ers_result_t result = PCI_ERS_RESULT_NEED_RESET;
pci_ers_result_t afu_result = PCI_ERS_RESULT_NEED_RESET;
return result;
list_for_each_entry(afu_dev, &afu->phb->bus->devices, bus_list) {
- if (!afu_dev->driver)
+ afu_drv = to_pci_driver(afu_dev->dev.driver);
+ if (!afu_drv)
continue;
afu_dev->error_state = state;
- if (afu_dev->driver->err_handler)
- afu_result = afu_dev->driver->err_handler->error_detected(afu_dev,
- state);
+ err_handler = afu_drv->err_handler;
+ if (err_handler)
+ afu_result = err_handler->error_detected(afu_dev,
+ state);
/* Disconnect trumps all, NONE trumps NEED_RESET */
if (afu_result == PCI_ERS_RESULT_DISCONNECT)
result = PCI_ERS_RESULT_DISCONNECT;
struct cxl_afu *afu;
struct cxl_context *ctx;
struct pci_dev *afu_dev;
+ struct pci_driver *afu_drv;
+ const struct pci_error_handlers *err_handler;
pci_ers_result_t afu_result = PCI_ERS_RESULT_RECOVERED;
pci_ers_result_t result = PCI_ERS_RESULT_RECOVERED;
int i;
* shouldn't start new work until we call
* their resume function.
*/
- if (!afu_dev->driver)
+ afu_drv = to_pci_driver(afu_dev->dev.driver);
+ if (!afu_drv)
continue;
- if (afu_dev->driver->err_handler &&
- afu_dev->driver->err_handler->slot_reset)
- afu_result = afu_dev->driver->err_handler->slot_reset(afu_dev);
+ err_handler = afu_drv->err_handler;
+ if (err_handler && err_handler->slot_reset)
+ afu_result = err_handler->slot_reset(afu_dev);
if (afu_result == PCI_ERS_RESULT_DISCONNECT)
result = PCI_ERS_RESULT_DISCONNECT;
struct cxl *adapter = pci_get_drvdata(pdev);
struct cxl_afu *afu;
struct pci_dev *afu_dev;
+ struct pci_driver *afu_drv;
+ const struct pci_error_handlers *err_handler;
int i;
/* Everything is back now. Drivers should restart work now.
continue;
list_for_each_entry(afu_dev, &afu->phb->bus->devices, bus_list) {
- if (afu_dev->driver && afu_dev->driver->err_handler &&
- afu_dev->driver->err_handler->resume)
- afu_dev->driver->err_handler->resume(afu_dev);
+ afu_drv = to_pci_driver(afu_dev->dev.driver);
+ if (!afu_drv)
+ continue;
+
+ err_handler = afu_drv->err_handler;
+ if (err_handler && err_handler->resume)
+ err_handler->resume(afu_dev);
}
}
spin_unlock(&adapter->afu_list_lock);
bool i2c_fn_i2c, i2c_fn_block;
unsigned int i, num_addresses;
struct at24_data *at24;
+ bool full_power;
struct regmap *regmap;
bool writable;
u8 test_byte;
i2c_set_clientdata(client, at24);
- err = regulator_enable(at24->vcc_reg);
- if (err) {
- dev_err(dev, "Failed to enable vcc regulator\n");
- return err;
- }
+ full_power = acpi_dev_state_d0(&client->dev);
+ if (full_power) {
+ err = regulator_enable(at24->vcc_reg);
+ if (err) {
+ dev_err(dev, "Failed to enable vcc regulator\n");
+ return err;
+ }
- /* enable runtime pm */
- pm_runtime_set_active(dev);
+ pm_runtime_set_active(dev);
+ }
pm_runtime_enable(dev);
at24->nvmem = devm_nvmem_register(dev, &nvmem_config);
}
/*
- * Perform a one-byte test read to verify that the
- * chip is functional.
+ * Perform a one-byte test read to verify that the chip is functional,
+ * unless powering on the device is to be avoided during probe (i.e.
+ * it's powered off right now).
*/
- err = at24_read(at24, 0, &test_byte, 1);
- if (err) {
- pm_runtime_disable(dev);
- if (!pm_runtime_status_suspended(dev))
- regulator_disable(at24->vcc_reg);
- return -ENODEV;
+ if (full_power) {
+ err = at24_read(at24, 0, &test_byte, 1);
+ if (err) {
+ pm_runtime_disable(dev);
+ if (!pm_runtime_status_suspended(dev))
+ regulator_disable(at24->vcc_reg);
+ return -ENODEV;
+ }
}
pm_runtime_idle(dev);
struct at24_data *at24 = i2c_get_clientdata(client);
pm_runtime_disable(&client->dev);
- if (!pm_runtime_status_suspended(&client->dev))
- regulator_disable(at24->vcc_reg);
- pm_runtime_set_suspended(&client->dev);
+ if (acpi_dev_state_d0(&client->dev)) {
+ if (!pm_runtime_status_suspended(&client->dev))
+ regulator_disable(at24->vcc_reg);
+ pm_runtime_set_suspended(&client->dev);
+ }
return 0;
}
.probe_new = at24_probe,
.remove = at24_remove,
.id_table = at24_ids,
+ .flags = I2C_DRV_ACPI_WAIVE_D0_PROBE,
};
static int __init at24_init(void)
#include <linux/bitops.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
-#include <linux/mfd/hi6421-spmi-pmic.h>
#include <linux/module.h>
#include <linux/of_gpio.h>
#include <linux/platform_device.h>
struct platform_device *pmic_pdev;
struct device *dev = &pdev->dev;
struct hi6421v600_irq *priv;
- struct hi6421_spmi_pmic *pmic;
+ struct regmap *regmap;
unsigned int virq;
int i, ret;
* which should first set drvdata. If this doesn't happen, hit
* a warn on and return.
*/
- pmic = dev_get_drvdata(pmic_dev);
- if (WARN_ON(!pmic))
+ regmap = dev_get_drvdata(pmic_dev);
+ if (WARN_ON(!regmap))
return -ENODEV;
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
return -ENOMEM;
priv->dev = dev;
- priv->regmap = pmic->regmap;
+ priv->regmap = regmap;
spin_lock_init(&priv->lock);
static int find_dvsec(struct pci_dev *dev, int dvsec_id)
{
- int vsec = 0;
- u16 vendor, id;
-
- while ((vsec = pci_find_next_ext_capability(dev, vsec,
- OCXL_EXT_CAP_ID_DVSEC))) {
- pci_read_config_word(dev, vsec + OCXL_DVSEC_VENDOR_OFFSET,
- &vendor);
- pci_read_config_word(dev, vsec + OCXL_DVSEC_ID_OFFSET, &id);
- if (vendor == PCI_VENDOR_ID_IBM && id == dvsec_id)
- return vsec;
- }
- return 0;
+ return pci_find_dvsec_capability(dev, PCI_VENDOR_ID_IBM, dvsec_id);
}
static int find_dvsec_afu_ctrl(struct pci_dev *dev, u8 afu_idx)
#include <linux/slab.h>
#include <linux/scatterlist.h>
-#include <linux/swap.h> /* For nr_free_buffer_pages() */
#include <linux/list.h>
#include <linux/debugfs.h>
LITTLE_ENDIAN_BYTE, if the bytes are reversed.
config MTD_CFI_NOSWAP
+ depends on !ARCH_IXP4XX || CPU_BIG_ENDIAN
bool "NO"
config MTD_CFI_BE_BYTE_SWAP
bool "BIG_ENDIAN_BYTE"
config MTD_CFI_LE_BYTE_SWAP
+ depends on !ARCH_IXP4XX
bool "LITTLE_ENDIAN_BYTE"
endchoice
#include <linux/slab.h>
#include <linux/major.h>
+/* Maximum number of comma-separated items in the 'block2mtd=' parameter */
+#define BLOCK2MTD_PARAM_MAX_COUNT 3
+
/* Info for the block device */
struct block2mtd_dev {
struct list_head list;
static struct block2mtd_dev *add_device(char *devname, int erase_size,
- int timeout)
+ char *label, int timeout)
{
#ifndef MODULE
int i;
/* Setup the MTD structure */
/* make the name contain the block device in */
- name = kasprintf(GFP_KERNEL, "block2mtd: %s", devname);
+ if (!label)
+ name = kasprintf(GFP_KERNEL, "block2mtd: %s", devname);
+ else
+ name = kstrdup(label, GFP_KERNEL);
if (!name)
goto err_destroy_mutex;
list_add(&dev->list, &blkmtd_device_list);
pr_info("mtd%d: [%s] erase_size = %dKiB [%d]\n",
dev->mtd.index,
- dev->mtd.name + strlen("block2mtd: "),
+ label ? label : dev->mtd.name + strlen("block2mtd: "),
dev->mtd.erasesize >> 10, dev->mtd.erasesize);
return dev;
/* 80 for device, 12 for erase size, 80 for name, 8 for timeout */
char buf[80 + 12 + 80 + 8];
char *str = buf;
- char *token[2];
+ char *token[BLOCK2MTD_PARAM_MAX_COUNT];
char *name;
+ char *label = NULL;
size_t erase_size = PAGE_SIZE;
unsigned long timeout = MTD_DEFAULT_TIMEOUT;
int i, ret;
strcpy(str, val);
kill_final_newline(str);
- for (i = 0; i < 2; i++)
+ for (i = 0; i < BLOCK2MTD_PARAM_MAX_COUNT; i++)
token[i] = strsep(&str, ",");
if (str) {
return 0;
}
- if (token[1]) {
+ /* Optional argument when custom label is used */
+ if (token[1] && strlen(token[1])) {
ret = parse_num(&erase_size, token[1]);
if (ret) {
pr_err("illegal erase size\n");
}
}
- add_device(name, erase_size, timeout);
+ if (token[2]) {
+ label = token[2];
+ pr_info("Using custom MTD label '%s' for dev %s\n", label, name);
+ }
+
+ add_device(name, erase_size, label, timeout);
return 0;
}
module_param_call(block2mtd, block2mtd_setup, NULL, NULL, 0200);
-MODULE_PARM_DESC(block2mtd, "Device to use. \"block2mtd=<dev>[,<erasesize>]\"");
+MODULE_PARM_DESC(block2mtd, "Device to use. \"block2mtd=<dev>[,[<erasesize>][,<label>]]\"");
static int __init block2mtd_init(void)
{
config MTD_IXP4XX
tristate "CFI Flash device mapped on Intel IXP4xx based systems"
- depends on MTD_CFI && MTD_COMPLEX_MAPPINGS && ARCH_IXP4XX
+ depends on MTD_CFI && MTD_COMPLEX_MAPPINGS && ARCH_IXP4XX && MTD_CFI_ADV_OPTIONS
help
This enables MTD access to flash devices on platforms based
on Intel's IXP4xx family of network processors such as the
mutex_lock(&mtd_table_mutex);
- debugfs_remove_recursive(mtd->dbg.dfs_dir);
-
if (idr_find(&mtd_idr, mtd->index) != mtd) {
ret = -ENODEV;
goto out_error;
mtd->index, mtd->name, mtd->usecount);
ret = -EBUSY;
} else {
+ debugfs_remove_recursive(mtd->dbg.dfs_dir);
+
/* Try to remove the NVMEM provider */
if (mtd->nvmem)
nvmem_unregister(mtd->nvmem);
if (proc_mtd)
remove_proc_entry("mtd", NULL);
class_unregister(&mtd_class);
+ bdi_unregister(mtd_bdi);
bdi_put(mtd_bdi);
idr_destroy(&mtd_idr);
}
return ret;
}
- eb = d->eb_data + *newblock / d->pages_per_eblk;
d->page_data[page] = *newblock;
d->revmap[oldblock] = PAGE_UNDEF;
eb = d->eb_data + oldblock / d->pages_per_eblk;
{
struct nand_ecc_sw_hamming_conf *engine_conf = nand->ecc.ctx.priv;
unsigned int step_size = nand->ecc.ctx.conf.step_size;
+ bool sm_order = engine_conf ? engine_conf->sm_order : false;
- return ecc_sw_hamming_calculate(buf, step_size, code,
- engine_conf->sm_order);
+ return ecc_sw_hamming_calculate(buf, step_size, code, sm_order);
}
EXPORT_SYMBOL(nand_ecc_sw_hamming_calculate);
{
struct nand_ecc_sw_hamming_conf *engine_conf = nand->ecc.ctx.priv;
unsigned int step_size = nand->ecc.ctx.conf.step_size;
+ bool sm_order = engine_conf ? engine_conf->sm_order : false;
return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc, step_size,
- engine_conf->sm_order);
+ sm_order);
}
EXPORT_SYMBOL(nand_ecc_sw_hamming_correct);
config MTD_ONENAND_SAMSUNG
tristate "OneNAND on Samsung SOC controller support"
- depends on ARCH_S3C64XX || ARCH_S5PV210 || ARCH_EXYNOS4 || COMPILE_TEST
+ depends on ARCH_S3C64XX || ARCH_S5PV210 || COMPILE_TEST
help
- Support for a OneNAND flash device connected to an Samsung SOC.
- S3C64XX uses command mapping method.
- S5PC110/S5PC210 use generic OneNAND method.
+ Support for a OneNAND flash device connected to Samsung S3C64XX
+ (using command mapping method) and S5PC110/S5PC210 (using generic
+ OneNAND method) SoCs.
+ Choose Y here only if you build for such Samsung SoC.
config MTD_ONENAND_OTP
bool "OneNAND OTP Support"
static int gpio_nand_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
/* Release write protection */
gpiod_set_value(priv->gpiod_nwp, 0);
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ this->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
/* Scan to find existence of the device */
err = nand_scan(this, 1);
if (err)
nvddr = nand_get_nvddr_timings(conf);
if (IS_ERR(nvddr))
return PTR_ERR(nvddr);
+
+ /*
+ * The controller only supports data payload requests which are
+ * a multiple of 4. In practice, most data accesses are 4-byte
+ * aligned and this is not an issue. However, rounding up will
+ * simply be refused by the controller if we reached the end of
+ * the device *and* we are using the NV-DDR interface(!). In
+ * this situation, unaligned data requests ending at the device
+ * boundary will confuse the controller and cannot be performed.
+ *
+ * This is something that happens in nand_read_subpage() when
+ * selecting software ECC support and must be avoided.
+ */
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT)
+ return -ENOTSUPP;
} else {
sdr = nand_get_sdr_timings(conf);
if (IS_ERR(sdr))
{
struct device *dev = &pdev->dev;
struct atmel_pmecc *pmecc;
- struct resource *res;
pmecc = devm_kzalloc(dev, sizeof(*pmecc), GFP_KERNEL);
if (!pmecc)
pmecc->dev = dev;
mutex_init(&pmecc->lock);
- res = platform_get_resource(pdev, IORESOURCE_MEM, pmecc_res_idx);
- pmecc->regs.base = devm_ioremap_resource(dev, res);
+ pmecc->regs.base = devm_platform_ioremap_resource(pdev, pmecc_res_idx);
if (IS_ERR(pmecc->regs.base))
return ERR_CAST(pmecc->regs.base);
- res = platform_get_resource(pdev, IORESOURCE_MEM, errloc_res_idx);
- pmecc->regs.errloc = devm_ioremap_resource(dev, res);
+ pmecc->regs.errloc = devm_platform_ioremap_resource(pdev, errloc_res_idx);
if (IS_ERR(pmecc->regs.errloc))
return ERR_CAST(pmecc->regs.errloc);
static int au1550nd_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
if (pd->devwidth)
this->options |= NAND_BUSWIDTH_16;
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ this->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
ret = nand_scan(this, 1);
if (ret) {
dev_err(&pdev->dev, "NAND scan failed with %d\n", ret);
struct device *dev = &pdev->dev;
struct bcm6368_nand_soc *priv;
struct brcmnand_soc *soc;
- struct resource *res;
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
if (!priv)
return -ENOMEM;
soc = &priv->soc;
- res = platform_get_resource_byname(pdev,
- IORESOURCE_MEM, "nand-int-base");
- priv->base = devm_ioremap_resource(dev, res);
+ priv->base = devm_platform_ioremap_resource_byname(pdev, "nand-int-base");
if (IS_ERR(priv->base))
return PTR_ERR(priv->base);
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/iopoll.h>
return 0;
}
-static int cs553x_ecc_correct(struct nand_chip *chip,
- unsigned char *buf,
- unsigned char *read_ecc,
- unsigned char *calc_ecc)
-{
- return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
-}
-
static struct cs553x_nand_controller *controllers[4];
static int cs553x_attach_chip(struct nand_chip *chip)
chip->ecc.bytes = 3;
chip->ecc.hwctl = cs_enable_hwecc;
chip->ecc.calculate = cs_calculate_ecc;
- chip->ecc.correct = cs553x_ecc_correct;
+ chip->ecc.correct = rawnand_sw_hamming_correct;
chip->ecc.strength = 1;
return 0;
static int denali_dt_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- struct resource *res;
struct denali_dt *dt;
const struct denali_dt_data *data;
struct denali_controller *denali;
if (denali->irq < 0)
return denali->irq;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "denali_reg");
- denali->reg = devm_ioremap_resource(dev, res);
+ denali->reg = devm_platform_ioremap_resource_byname(pdev, "denali_reg");
if (IS_ERR(denali->reg))
return PTR_ERR(denali->reg);
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "nand_data");
- denali->host = devm_ioremap_resource(dev, res);
+ denali->host = devm_platform_ioremap_resource_byname(pdev, "nand_data");
if (IS_ERR(denali->host))
return PTR_ERR(denali->host);
unsigned char *read_ecc,
unsigned char *calc_ecc)
{
+ bool sm_order = chip->ecc.options & NAND_ECC_SOFT_HAMMING_SM_ORDER;
+
return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
+ chip->ecc.size, sm_order);
}
/* Count the number of 0's in buff upto a max of max_bits */
static int gpio_nand_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
chip = &gpiomtd->nand_chip;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- gpiomtd->io = devm_ioremap_resource(dev, res);
+ gpiomtd->io = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(gpiomtd->io))
return PTR_ERR(gpiomtd->io);
if (gpiomtd->nwp && !IS_ERR(gpiomtd->nwp))
gpiod_direction_output(gpiomtd->nwp, 1);
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
ret = nand_scan(chip, 1);
if (ret)
goto err_wp;
{
struct platform_device *pdev = this->pdev;
struct resources *res = &this->resources;
- struct resource *r;
void __iomem *p;
- r = platform_get_resource_byname(pdev, IORESOURCE_MEM, res_name);
- p = devm_ioremap_resource(&pdev->dev, r);
+ p = devm_platform_ioremap_resource_byname(pdev, res_name);
if (IS_ERR(p))
return PTR_ERR(p);
struct hinfc_host *host;
struct nand_chip *chip;
struct mtd_info *mtd;
- struct resource *res;
struct device_node *np = dev->of_node;
host = devm_kzalloc(dev, sizeof(*host), GFP_KERNEL);
if (irq < 0)
return -ENXIO;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- host->iobase = devm_ioremap_resource(dev, res);
+ host->iobase = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(host->iobase))
return PTR_ERR(host->iobase);
- res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
- host->mmio = devm_ioremap_resource(dev, res);
+ host->mmio = devm_platform_ioremap_resource(pdev, 1);
if (IS_ERR(host->mmio))
return PTR_ERR(host->mmio);
dev_err(dev, "failed to get chip select: %d\n", ret);
return ret;
}
+ if (cs >= MAX_CS) {
+ dev_err(dev, "got invalid chip select: %d\n", cs);
+ return -EINVAL;
+ }
+
ebu_host->cs_num = cs;
resname = devm_kasprintf(dev, GFP_KERNEL, "nand_cs%d", cs);
#include <linux/of.h>
#include <linux/of_gpio.h>
#include <linux/mtd/lpc32xx_slc.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#define LPC32XX_MODNAME "lpc32xx-nand"
}
/*
- * Corrects the data
- */
-static int lpc32xx_nand_ecc_correct(struct nand_chip *chip,
- unsigned char *buf,
- unsigned char *read_ecc,
- unsigned char *calc_ecc)
-{
- return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
-}
-
-/*
* Read a single byte from NAND device
*/
static uint8_t lpc32xx_nand_read_byte(struct nand_chip *chip)
chip->ecc.write_oob = lpc32xx_nand_write_oob_syndrome;
chip->ecc.read_oob = lpc32xx_nand_read_oob_syndrome;
chip->ecc.calculate = lpc32xx_nand_ecc_calculate;
- chip->ecc.correct = lpc32xx_nand_ecc_correct;
+ chip->ecc.correct = rawnand_sw_hamming_correct;
chip->ecc.hwctl = lpc32xx_nand_ecc_enable;
/*
static int mpc5121_nfc_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
goto error;
}
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
/* Detect NAND chips */
retval = nand_scan(chip, be32_to_cpup(chips_no));
if (retval) {
{
struct device *dev = &pdev->dev;
struct mtk_ecc *ecc;
- struct resource *res;
u32 max_eccdata_size;
int irq, ret;
if (!ecc->eccdata)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ecc->regs = devm_ioremap_resource(dev, res);
+ ecc->regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(ecc->regs))
return PTR_ERR(ecc->regs);
struct device *dev = &pdev->dev;
struct device_node *np = dev->of_node;
struct mtk_nfc *nfc;
- struct resource *res;
int ret, irq;
nfc = devm_kzalloc(dev, sizeof(*nfc), GFP_KERNEL);
nfc->caps = of_device_get_match_data(dev);
nfc->dev = dev;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- nfc->regs = devm_ioremap_resource(dev, res);
+ nfc->regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(nfc->regs)) {
ret = PTR_ERR(nfc->regs);
goto release_ecc;
return nand_choose_best_sdr_timings(chip, iface, NULL);
}
+static int h27ucg8t2etrbc_init(struct nand_chip *chip)
+{
+ struct mtd_info *mtd = nand_to_mtd(chip);
+
+ chip->options |= NAND_NEED_SCRAMBLING;
+ mtd_set_pairing_scheme(mtd, &dist3_pairing_scheme);
+
+ return 0;
+}
+
static int hynix_nand_init(struct nand_chip *chip)
{
struct hynix_nand *hynix;
chip->ops.choose_interface_config =
h27ucg8t2atrbc_choose_interface_config;
+ if (!strncmp("H27UCG8T2ETR-BC", chip->parameters.model,
+ sizeof("H27UCG8T2ETR-BC") - 1))
+ h27ucg8t2etrbc_init(chip);
+
ret = hynix_nand_rr_init(chip);
if (ret)
hynix_nand_cleanup(chip);
{ .id = {0xad, 0xde, 0x94, 0xda, 0x74, 0xc4} },
SZ_8K, SZ_8K, SZ_2M, NAND_NEED_SCRAMBLING, 6, 640,
NAND_ECC_INFO(40, SZ_1K) },
+ {"H27UCG8T2ETR-BC 64G 3.3V 8-bit",
+ { .id = {0xad, 0xde, 0x14, 0xa7, 0x42, 0x4a} },
+ SZ_16K, SZ_8K, SZ_4M, NAND_NEED_SCRAMBLING, 6, 1664,
+ NAND_ECC_INFO(40, SZ_1K) },
{"TH58NVG2S3HBAI4 4G 3.3V 8-bit",
{ .id = {0x98, 0xdc, 0x91, 0x15, 0x76} },
SZ_2K, SZ_512, SZ_128K, 0, 5, 128, NAND_ECC_INFO(8, SZ_512) },
#include <linux/mtd/ndfc.h>
#include <linux/slab.h>
#include <linux/mtd/mtd.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/of_address.h>
#include <linux/of_platform.h>
#include <asm/io.h>
return 0;
}
-static int ndfc_correct_ecc(struct nand_chip *chip,
- unsigned char *buf,
- unsigned char *read_ecc,
- unsigned char *calc_ecc)
-{
- return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
-}
-
/*
* Speedups for buffer read/write/verify
*
chip->controller = &ndfc->ndfc_control;
chip->legacy.read_buf = ndfc_read_buf;
chip->legacy.write_buf = ndfc_write_buf;
- chip->ecc.correct = ndfc_correct_ecc;
+ chip->ecc.correct = rawnand_sw_hamming_correct;
chip->ecc.hwctl = ndfc_enable_hwecc;
chip->ecc.calculate = ndfc_calculate_ecc;
chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_ON_HOST;
static int elm_probe(struct platform_device *pdev)
{
int ret = 0;
- struct resource *res, *irq;
+ struct resource *irq;
struct elm_info *info;
info = devm_kzalloc(&pdev->dev, sizeof(*info), GFP_KERNEL);
return -ENODEV;
}
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- info->elm_base = devm_ioremap_resource(&pdev->dev, res);
+ info->elm_base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(info->elm_base))
return PTR_ERR(info->elm_base);
static int orion_nand_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
return ret;
}
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ nc->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
ret = nand_scan(nc, 1);
if (ret)
goto no_dev;
struct oxnas_nand_ctrl *oxnas;
struct nand_chip *chip;
struct mtd_info *mtd;
- struct resource *res;
int count = 0;
int err = 0;
int i;
nand_controller_init(&oxnas->base);
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- oxnas->io_base = devm_ioremap_resource(&pdev->dev, res);
+ oxnas->io_base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(oxnas->io_base))
return PTR_ERR(oxnas->io_base);
static int pasemi_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
/* Enable the following for a flash based bad block table */
chip->bbt_options = NAND_BBT_USE_FLASH;
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
/* Scan to find existence of the device */
err = nand_scan(chip, 1);
if (err)
static int plat_nand_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
struct platform_nand_data *pdata = dev_get_platdata(&pdev->dev);
struct plat_nand_data *data;
struct mtd_info *mtd;
- struct resource *res;
const char **part_types;
int err = 0;
nand_controller_init(&data->controller);
data->chip.controller = &data->controller;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- data->io_base = devm_ioremap_resource(&pdev->dev, res);
+ data->io_base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(data->io_base))
return PTR_ERR(data->io_base);
goto out;
}
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ data->chip.ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
/* Scan to find existence of the device */
err = nand_scan(&data->chip, pdata->chip.nr_chips);
if (err)
#include <linux/module.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/mtd/sharpsl.h>
return readb(sharpsl->io + ECCCNTR) != 0;
}
-static int sharpsl_nand_correct_ecc(struct nand_chip *chip,
- unsigned char *buf,
- unsigned char *read_ecc,
- unsigned char *calc_ecc)
-{
- return ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
-}
-
static int sharpsl_attach_chip(struct nand_chip *chip)
{
if (chip->ecc.engine_type != NAND_ECC_ENGINE_TYPE_ON_HOST)
chip->ecc.strength = 1;
chip->ecc.hwctl = sharpsl_nand_enable_hwecc;
chip->ecc.calculate = sharpsl_nand_calculate_ecc;
- chip->ecc.correct = sharpsl_nand_correct_ecc;
+ chip->ecc.correct = rawnand_sw_hamming_correct;
return 0;
}
static int socrates_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
/* TODO: I have no idea what real delay is. */
nand_chip->legacy.chip_delay = 20; /* 20us command delay time */
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ nand_chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
dev_set_drvdata(&ofdev->dev, host);
res = nand_scan(nand_chip, 1);
nfc->data_phys_addr[chip_cs] = res->start;
- res = platform_get_resource(pdev, IORESOURCE_MEM,
- mem_region + 1);
- nfc->cmd_base[chip_cs] = devm_ioremap_resource(dev, res);
+ nfc->cmd_base[chip_cs] = devm_platform_ioremap_resource(pdev, mem_region + 1);
if (IS_ERR(nfc->cmd_base[chip_cs]))
return PTR_ERR(nfc->cmd_base[chip_cs]);
- res = platform_get_resource(pdev, IORESOURCE_MEM,
- mem_region + 2);
- nfc->addr_base[chip_cs] = devm_ioremap_resource(dev, res);
+ nfc->addr_base[chip_cs] = devm_platform_ioremap_resource(pdev, mem_region + 2);
if (IS_ERR(nfc->addr_base[chip_cs]))
return PTR_ERR(nfc->addr_base[chip_cs]);
}
{
struct reset_control *rst;
struct tegra_nand_controller *ctrl;
- struct resource *res;
int err = 0;
ctrl = devm_kzalloc(&pdev->dev, sizeof(*ctrl), GFP_KERNEL);
nand_controller_init(&ctrl->controller);
ctrl->controller.ops = &tegra_nand_controller_ops;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ctrl->regs = devm_ioremap_resource(&pdev->dev, res);
+ ctrl->regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(ctrl->regs))
return PTR_ERR(ctrl->regs);
#include <linux/interrupt.h>
#include <linux/ioport.h>
#include <linux/mtd/mtd.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/slab.h>
int r0, r1;
/* assume ecc.size = 512 and ecc.bytes = 6 */
- r0 = ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
+ r0 = rawnand_sw_hamming_correct(chip, buf, read_ecc, calc_ecc);
if (r0 < 0)
return r0;
- r1 = ecc_sw_hamming_correct(buf + 256, read_ecc + 3, calc_ecc + 3,
- chip->ecc.size, false);
+ r1 = rawnand_sw_hamming_correct(chip, buf + 256, read_ecc + 3,
+ calc_ecc + 3);
if (r1 < 0)
return r1;
return r0 + r1;
#include <linux/platform_device.h>
#include <linux/delay.h>
#include <linux/mtd/mtd.h>
-#include <linux/mtd/nand-ecc-sw-hamming.h>
#include <linux/mtd/rawnand.h>
#include <linux/mtd/partitions.h>
#include <linux/io.h>
int stat;
for (eccsize = chip->ecc.size; eccsize > 0; eccsize -= 256) {
- stat = ecc_sw_hamming_correct(buf, read_ecc, calc_ecc,
- chip->ecc.size, false);
+ stat = rawnand_sw_hamming_correct(chip, buf, read_ecc,
+ calc_ecc);
if (stat < 0)
return stat;
corrected += stat;
int i;
struct txx9ndfmc_drvdata *drvdata;
unsigned long gbusclk = plat->gbus_clock;
- struct resource *res;
drvdata = devm_kzalloc(&dev->dev, sizeof(*drvdata), GFP_KERNEL);
if (!drvdata)
return -ENOMEM;
- res = platform_get_resource(dev, IORESOURCE_MEM, 0);
- drvdata->base = devm_ioremap_resource(&dev->dev, res);
+ drvdata->base = devm_platform_ioremap_resource(dev, 0);
if (IS_ERR(drvdata->base))
return PTR_ERR(drvdata->base);
static int vf610_nfc_probe(struct platform_device *pdev)
{
struct vf610_nfc *nfc;
- struct resource *res;
struct mtd_info *mtd;
struct nand_chip *chip;
struct device_node *child;
if (irq <= 0)
return -EINVAL;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- nfc->regs = devm_ioremap_resource(nfc->dev, res);
+ nfc->regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(nfc->regs))
return PTR_ERR(nfc->regs);
static int xway_attach_chip(struct nand_chip *chip)
{
- chip->ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
-
- if (chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
+ if (chip->ecc.engine_type == NAND_ECC_ENGINE_TYPE_SOFT &&
+ chip->ecc.algo == NAND_ECC_ALGO_UNKNOWN)
chip->ecc.algo = NAND_ECC_ALGO_HAMMING;
return 0;
{
struct xway_nand_data *data;
struct mtd_info *mtd;
- struct resource *res;
int err;
u32 cs;
u32 cs_flag = 0;
if (!data)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- data->nandaddr = devm_ioremap_resource(&pdev->dev, res);
+ data->nandaddr = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(data->nandaddr))
return PTR_ERR(data->nandaddr);
| NAND_CON_SE_P | NAND_CON_WP_P | NAND_CON_PRE_P
| cs_flag, EBU_NAND_CON);
+ /*
+ * This driver assumes that the default ECC engine should be TYPE_SOFT.
+ * Set ->engine_type before registering the NAND devices in order to
+ * provide a driver specific default value.
+ */
+ data->chip.ecc.engine_type = NAND_ECC_ENGINE_TYPE_SOFT;
+
/* Scan to find existence of the device */
err = nand_scan(&data->chip, 1);
if (err)
static int hisi_spi_nor_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- struct resource *res;
struct hifmc_host *host;
int ret;
platform_set_drvdata(pdev, host);
host->dev = dev;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "control");
- host->regbase = devm_ioremap_resource(dev, res);
+ host->regbase = devm_platform_ioremap_resource_byname(pdev, "control");
if (IS_ERR(host->regbase))
return PTR_ERR(host->regbase);
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "memory");
- host->iobase = devm_ioremap_resource(dev, res);
+ host->iobase = devm_platform_ioremap_resource_byname(pdev, "memory");
if (IS_ERR(host->iobase))
return PTR_ERR(host->iobase);
hisi_spi_nor_unregister_all(host);
mutex_destroy(&host->lock);
- clk_disable_unprepare(host->clk);
return 0;
}
{
struct device_node *flash_np;
struct nxp_spifi *spifi;
- struct resource *res;
int ret;
spifi = devm_kzalloc(&pdev->dev, sizeof(*spifi), GFP_KERNEL);
if (!spifi)
return -ENOMEM;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "spifi");
- spifi->io_base = devm_ioremap_resource(&pdev->dev, res);
+ spifi->io_base = devm_platform_ioremap_resource_byname(pdev, "spifi");
if (IS_ERR(spifi->io_base))
return PTR_ERR(spifi->io_base);
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "flash");
- spifi->flash_base = devm_ioremap_resource(&pdev->dev, res);
+ spifi->flash_base = devm_platform_ioremap_resource_byname(pdev, "flash");
if (IS_ERR(spifi->flash_base))
return PTR_ERR(spifi->flash_base);
SPI_NOR_HAS_LOCK | SPI_NOR_HAS_TB |
SPI_NOR_4BIT_BP | SPI_NOR_BP3_SR_BIT6) },
{ "n25q128a13", INFO(0x20ba18, 0, 64 * 1024, 256,
- SECT_4K | USE_FSR | SPI_NOR_QUAD_READ) },
+ SECT_4K | USE_FSR | SPI_NOR_QUAD_READ |
+ SPI_NOR_HAS_LOCK | SPI_NOR_HAS_TB |
+ SPI_NOR_4BIT_BP | SPI_NOR_BP3_SR_BIT6) },
{ "mt25ql256a", INFO6(0x20ba19, 0x104400, 64 * 1024, 512,
SECT_4K | USE_FSR | SPI_NOR_DUAL_READ |
SPI_NOR_QUAD_READ | SPI_NOR_4B_OPCODES) },
list_add_tail(&dev->list, &ubiblock_devices);
/* Must be the last step: anyone can call file ops from now on */
- add_disk(dev->gd);
+ ret = add_disk(dev->gd);
+ if (ret)
+ goto out_destroy_wq;
+
dev_info(disk_to_dev(dev->gd), "created from ubi%d:%d(%s)",
dev->ubi_num, dev->vol_id, vi->name);
mutex_unlock(&devices_mutex);
return 0;
+out_destroy_wq:
+ list_del(&dev->list);
+ destroy_workqueue(dev->wq);
out_remove_minor:
idr_remove(&ubiblock_minor_idr, gd->first_minor);
out_cleanup_disk:
config AMT
tristate "Automatic Multicast Tunneling (AMT)"
depends on INET && IP_MULTICAST
+ depends on IPV6 || !IPV6
select NET_UDP_TUNNEL
help
This allows one to create AMT(Automatic Multicast Tunneling)
#include <linux/igmp.h>
#include <linux/workqueue.h>
#include <net/net_namespace.h>
-#include <net/protocol.h>
#include <net/ip.h>
#include <net/udp.h>
#include <net/udp_tunnel.h>
#include <linux/security.h>
#include <net/gro_cells.h>
#include <net/ipv6.h>
-#include <net/protocol.h>
#include <net/if_inet6.h>
#include <net/ndisc.h>
#include <net/addrconf.h>
rcu_read_lock_bh();
amt = rcu_dereference_sk_user_data(sk);
if (!amt)
- goto drop;
+ goto out;
if (amt->mode != AMT_MODE_GATEWAY)
goto drop;
default:
goto drop;
}
+out:
rcu_read_unlock_bh();
return 0;
drop:
goto unregister_notifier;
amt_wq = alloc_workqueue("amt", WQ_UNBOUND, 1);
- if (!amt_wq)
+ if (!amt_wq) {
+ err = -ENOMEM;
goto rtnl_unregister;
+ }
spin_lock_init(&source_gc_lock);
spin_lock_bh(&source_gc_lock);
{
rtnl_link_unregister(&amt_link_ops);
unregister_netdevice_notifier(&amt_notifier_block);
- flush_delayed_work(&source_gc_wq);
+ cancel_delayed_work_sync(&source_gc_wq);
__amt_source_gc_work();
destroy_workqueue(amt_wq);
}
}
static SLAVE_ATTR_RO(ad_partner_oper_port_state);
-static const struct slave_attribute *slave_attrs[] = {
- &slave_attr_state,
- &slave_attr_mii_status,
- &slave_attr_link_failure_count,
- &slave_attr_perm_hwaddr,
- &slave_attr_queue_id,
- &slave_attr_ad_aggregator_id,
- &slave_attr_ad_actor_oper_port_state,
- &slave_attr_ad_partner_oper_port_state,
+static const struct attribute *slave_attrs[] = {
+ &slave_attr_state.attr,
+ &slave_attr_mii_status.attr,
+ &slave_attr_link_failure_count.attr,
+ &slave_attr_perm_hwaddr.attr,
+ &slave_attr_queue_id.attr,
+ &slave_attr_ad_aggregator_id.attr,
+ &slave_attr_ad_actor_oper_port_state.attr,
+ &slave_attr_ad_partner_oper_port_state.attr,
NULL
};
int bond_sysfs_slave_add(struct slave *slave)
{
- const struct slave_attribute **a;
- int err;
-
- for (a = slave_attrs; *a; ++a) {
- err = sysfs_create_file(&slave->kobj, &((*a)->attr));
- if (err) {
- kobject_put(&slave->kobj);
- return err;
- }
- }
-
- return 0;
+ return sysfs_create_files(&slave->kobj, slave_attrs);
}
void bond_sysfs_slave_del(struct slave *slave)
{
- const struct slave_attribute **a;
-
- for (a = slave_attrs; *a; ++a)
- sysfs_remove_file(&slave->kobj, &((*a)->attr));
+ sysfs_remove_files(&slave->kobj, slave_attrs);
}
err = mcp251xfd_chip_rx_int_enable(priv);
if (err)
- return err;
+ goto out_chip_stop;
err = mcp251xfd_chip_ecc_init(priv);
if (err)
* check will fail, too. So leave IRQ handler
* directly.
*/
- if (priv->can.state == CAN_STATE_BUS_OFF)
+ if (priv->can.state == CAN_STATE_BUS_OFF) {
+ can_rx_offload_threaded_irq_finish(&priv->offload);
return IRQ_HANDLED;
+ }
}
handled = IRQ_HANDLED;
struct can_device_stats *can_stats = &can->can_stats;
struct can_frame *cf = NULL;
struct sk_buff *skb;
- int ret;
+ int ret = 0;
if (!netif_running(netdev)) {
if (net_ratelimit())
can->state = CAN_STATE_BUS_OFF;
can_bus_off(netdev);
ret = can->do_set_mode(netdev, CAN_MODE_STOP);
- if (ret)
- return ret;
}
break;
ES58X_EVENT_BUSOFF, timestamp);
}
- return 0;
+ return ret;
}
/**
pdev->bec.rxerr = 0;
pdev->bec.txerr = 0;
- /* be notified on error counter changes (if requested by user) */
- if (dev->can.ctrlmode & CAN_CTRLMODE_BERR_REPORTING) {
- err = pcan_usb_set_err_frame(dev, PCAN_USB_BERR_MASK);
- if (err)
- netdev_warn(dev->netdev,
- "Asking for BERR reporting error %u\n",
- err);
- }
+ /* always ask the device for BERR reporting, to be able to switch from
+ * WARNING to PASSIVE state
+ */
+ err = pcan_usb_set_err_frame(dev, PCAN_USB_BERR_MASK);
+ if (err)
+ netdev_warn(dev->netdev,
+ "Asking for BERR reporting error %u\n",
+ err);
/* if revision greater than 3, can put silent mode on/off */
if (dev->device_rev > 3) {
return err;
}
+ dev_info(dev->netdev->dev.parent,
+ "PEAK-System %s adapter hwrev %u serial %08X (%u channel)\n",
+ pcan_usb.name, dev->device_rev, serial_number,
+ pcan_usb.ctrl_count);
+
/* Since rev 4.1, PCAN-USB is able to make single-shot as well as
* looped back frames.
*/
"Firmware update available. Please contact support@peak-system.com\n");
}
- dev_info(dev->netdev->dev.parent,
- "PEAK-System %s adapter hwrev %u serial %08X (%u channel)\n",
- pcan_usb.name, dev->device_rev, serial_number,
- pcan_usb.ctrl_count);
-
return 0;
}
.device_id = PCAN_USB_PRODUCT_ID,
.ctrl_count = 1,
.ctrlmode_supported = CAN_CTRLMODE_3_SAMPLES | CAN_CTRLMODE_LISTENONLY |
- CAN_CTRLMODE_BERR_REPORTING |
CAN_CTRLMODE_CC_LEN8_DLC,
.clock = {
.freq = PCAN_USB_CRYSTAL_HZ / 2,
unsigned long *mask,
struct phylink_link_state *state)
{
- if (port == 0 || port == 9 || port == 10) {
+ bool is_6191x =
+ chip->info->prod_num == MV88E6XXX_PORT_SWITCH_ID_PROD_6191X;
+
+ if (((port == 0 || port == 9) && !is_6191x) || port == 10) {
phylink_set(mask, 10000baseT_Full);
phylink_set(mask, 10000baseKR_Full);
phylink_set(mask, 10000baseCR_Full);
static bool felix_rxtstamp(struct dsa_switch *ds, int port,
struct sk_buff *skb, unsigned int type)
{
- u8 *extraction = skb->data - ETH_HLEN - OCELOT_TAG_LEN;
+ u32 tstamp_lo = OCELOT_SKB_CB(skb)->tstamp_lo;
struct skb_shared_hwtstamps *shhwtstamps;
struct ocelot *ocelot = ds->priv;
- u32 tstamp_lo, tstamp_hi;
struct timespec64 ts;
- u64 tstamp, val;
+ u32 tstamp_hi;
+ u64 tstamp;
/* If the "no XTR IRQ" workaround is in use, tell DSA to defer this skb
* for RX timestamping. Then free it, and poll for its copy through
ocelot_ptp_gettime64(&ocelot->ptp_info, &ts);
tstamp = ktime_set(ts.tv_sec, ts.tv_nsec);
- ocelot_xfh_get_rew_val(extraction, &val);
- tstamp_lo = (u32)val;
-
tstamp_hi = tstamp >> 32;
if ((tstamp & 0xffffffff) < tstamp_lo)
tstamp_hi--;
if (ret)
return ret;
+ /* Make sure MAC06 is disabled */
+ ret = qca8k_reg_clear(priv, QCA8K_REG_PORT0_PAD_CTRL,
+ QCA8K_PORT0_PAD_MAC06_EXCHANGE_EN);
+ if (ret) {
+ dev_err(priv->dev, "failed disabling MAC06 exchange");
+ return ret;
+ }
+
/* Enable CPU Port */
ret = qca8k_reg_set(priv, QCA8K_REG_GLOBAL_FW_CTRL0,
QCA8K_GLOBAL_FW_CTRL0_CPU_PORT_EN);
#define QCA8K_MASK_CTRL_DEVICE_ID_MASK GENMASK(15, 8)
#define QCA8K_MASK_CTRL_DEVICE_ID(x) ((x) >> 8)
#define QCA8K_REG_PORT0_PAD_CTRL 0x004
+#define QCA8K_PORT0_PAD_MAC06_EXCHANGE_EN BIT(31)
#define QCA8K_PORT0_PAD_SGMII_RXCLK_FALLING_EDGE BIT(19)
#define QCA8K_PORT0_PAD_SGMII_TXCLK_FALLING_EDGE BIT(18)
#define QCA8K_REG_PORT5_PAD_CTRL 0x008
goto err_exit;
if (fw.len == 0xFFFFU) {
+ if (sw.len > sizeof(self->rpc)) {
+ printk(KERN_INFO "Invalid sw len: %x\n", sw.len);
+ err = -EINVAL;
+ goto err_exit;
+ }
err = hw_atl_utils_fw_rpc_call(self, sw.len);
if (err < 0)
goto err_exit;
if (rpc) {
if (fw.len) {
+ if (fw.len > sizeof(self->rpc)) {
+ printk(KERN_INFO "Invalid fw len: %x\n", fw.len);
+ err = -EINVAL;
+ goto err_exit;
+ }
err =
hw_atl_utils_fw_downld_dwords(self,
self->rpc_addr,
.ndo_stop = ax88796c_close,
.ndo_start_xmit = ax88796c_start_xmit,
.ndo_get_stats64 = ax88796c_get_stats64,
- .ndo_do_ioctl = ax88796c_ioctl,
+ .ndo_eth_ioctl = ax88796c_ioctl,
.ndo_set_mac_address = eth_mac_addr,
.ndo_set_features = ax88796c_set_features,
};
return 0;
}
+#ifdef CONFIG_OF
static const struct of_device_id ax88796c_dt_ids[] = {
{ .compatible = "asix,ax88796c" },
{},
};
MODULE_DEVICE_TABLE(of, ax88796c_dt_ids);
+#endif
static const struct spi_device_id asix_id[] = {
{ "ax88796c", 0 },
#define AX_PRIV_FLAGS_MASK (AX_CAP_COMP)
unsigned long flags;
- #define EVENT_INTR BIT(0)
- #define EVENT_TX BIT(1)
- #define EVENT_SET_MULTI BIT(2)
+ #define EVENT_INTR 0
+ #define EVENT_TX 1
+ #define EVENT_SET_MULTI 2
};
{
int i, rc;
struct bnx2x_ilt *ilt = BP_ILT(bp);
- struct ilt_client_info *ilt_cli = &ilt->clients[cli_num];
+ struct ilt_client_info *ilt_cli;
if (!ilt || !ilt->lines)
return -1;
+ ilt_cli = &ilt->clients[cli_num];
+
if (ilt_cli->flags & (ILT_CLIENT_SKIP_INIT | ILT_CLIENT_SKIP_MEM))
return 0;
}
}
+/* Must hold rtnl_lock */
+static inline bool bnxt_sriov_cfg(struct bnxt *bp)
+{
+#if defined(CONFIG_BNXT_SRIOV)
+ return BNXT_PF(bp) && (bp->pf.active_vfs || bp->sriov_cfg);
+#else
+ return false;
+#endif
+}
+
extern const u16 bnxt_lhint_arr[];
int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
NL_SET_ERR_MSG_MOD(extack, "Live patch already applied");
break;
default:
- netdev_err(bp->dev, "Unexpected live patch error: %hhd\n", err);
+ netdev_err(bp->dev, "Unexpected live patch error: %d\n", err);
NL_SET_ERR_MSG_MOD(extack, "Failed to activate live patch");
break;
}
switch (action) {
case DEVLINK_RELOAD_ACTION_DRIVER_REINIT: {
- if (BNXT_PF(bp) && bp->pf.active_vfs) {
+ rtnl_lock();
+ if (bnxt_sriov_cfg(bp)) {
NL_SET_ERR_MSG_MOD(extack,
- "reload is unsupported when VFs are allocated\n");
+ "reload is unsupported while VFs are allocated or being configured");
+ rtnl_unlock();
return -EOPNOTSUPP;
}
- rtnl_lock();
if (bp->dev->reg_state == NETREG_UNREGISTERED) {
rtnl_unlock();
return -ENODEV;
struct flow_cls_offload *flower = type_data;
struct bnxt *bp = priv->bp;
- if (flower->common.chain_index)
+ if (!tc_cls_can_offload_and_chain0(bp->dev, type_data))
return -EOPNOTSUPP;
switch (type) {
int workaround, port_a;
serdes_cfg = 0;
- expected_sg_dig_ctrl = 0;
workaround = 0;
port_a = 1;
current_link_up = false;
void t3_link_fault(struct adapter *adapter, int port_id);
int t3_link_start(struct cphy *phy, struct cmac *mac, struct link_config *lc);
const struct adapter_info *t3_get_adapter_info(unsigned int board_id);
-int t3_seeprom_read(struct adapter *adapter, u32 addr, __le32 *data);
-int t3_seeprom_write(struct adapter *adapter, u32 addr, __le32 data);
int t3_seeprom_wp(struct adapter *adapter, int enable);
int t3_get_tp_version(struct adapter *adapter, u32 *vers);
int t3_check_tpsram_version(struct adapter *adapter);
{
struct port_info *pi = netdev_priv(dev);
struct adapter *adapter = pi->adapter;
- int i, err = 0;
-
- u8 *buf = kmalloc(EEPROMSIZE, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
+ int cnt;
e->magic = EEPROM_MAGIC;
- for (i = e->offset & ~3; !err && i < e->offset + e->len; i += 4)
- err = t3_seeprom_read(adapter, i, (__le32 *) & buf[i]);
+ cnt = pci_read_vpd(adapter->pdev, e->offset, e->len, data);
+ if (cnt < 0)
+ return cnt;
- if (!err)
- memcpy(data, buf + e->offset, e->len);
- kfree(buf);
- return err;
+ e->len = cnt;
+
+ return 0;
}
static int set_eeprom(struct net_device *dev, struct ethtool_eeprom *eeprom,
struct port_info *pi = netdev_priv(dev);
struct adapter *adapter = pi->adapter;
u32 aligned_offset, aligned_len;
- __le32 *p;
u8 *buf;
int err;
buf = kmalloc(aligned_len, GFP_KERNEL);
if (!buf)
return -ENOMEM;
- err = t3_seeprom_read(adapter, aligned_offset, (__le32 *) buf);
- if (!err && aligned_len > 4)
- err = t3_seeprom_read(adapter,
- aligned_offset + aligned_len - 4,
- (__le32 *) & buf[aligned_len - 4]);
- if (err)
+ err = pci_read_vpd(adapter->pdev, aligned_offset, aligned_len,
+ buf);
+ if (err < 0)
goto out;
memcpy(buf + (eeprom->offset & 3), data, eeprom->len);
} else
if (err)
goto out;
- for (p = (__le32 *) buf; !err && aligned_len; aligned_len -= 4, p++) {
- err = t3_seeprom_write(adapter, aligned_offset, *p);
- aligned_offset += 4;
- }
-
- if (!err)
+ err = pci_write_vpd(adapter->pdev, aligned_offset, aligned_len, buf);
+ if (err >= 0)
err = t3_seeprom_wp(adapter, 1);
out:
if (buf != data)
kfree(buf);
- return err;
+ return err < 0 ? err : 0;
}
static void get_wol(struct net_device *dev, struct ethtool_wolinfo *wol)
u32 pad; /* for multiple-of-4 sizing and alignment */
};
-#define EEPROM_MAX_POLL 40
#define EEPROM_STAT_ADDR 0x4000
#define VPD_BASE 0xc00
/**
- * t3_seeprom_read - read a VPD EEPROM location
- * @adapter: adapter to read
- * @addr: EEPROM address
- * @data: where to store the read data
- *
- * Read a 32-bit word from a location in VPD EEPROM using the card's PCI
- * VPD ROM capability. A zero is written to the flag bit when the
- * address is written to the control register. The hardware device will
- * set the flag to 1 when 4 bytes have been read into the data register.
- */
-int t3_seeprom_read(struct adapter *adapter, u32 addr, __le32 *data)
-{
- u16 val;
- int attempts = EEPROM_MAX_POLL;
- u32 v;
- unsigned int base = adapter->params.pci.vpd_cap_addr;
-
- if ((addr >= EEPROMSIZE && addr != EEPROM_STAT_ADDR) || (addr & 3))
- return -EINVAL;
-
- pci_write_config_word(adapter->pdev, base + PCI_VPD_ADDR, addr);
- do {
- udelay(10);
- pci_read_config_word(adapter->pdev, base + PCI_VPD_ADDR, &val);
- } while (!(val & PCI_VPD_ADDR_F) && --attempts);
-
- if (!(val & PCI_VPD_ADDR_F)) {
- CH_ERR(adapter, "reading EEPROM address 0x%x failed\n", addr);
- return -EIO;
- }
- pci_read_config_dword(adapter->pdev, base + PCI_VPD_DATA, &v);
- *data = cpu_to_le32(v);
- return 0;
-}
-
-/**
- * t3_seeprom_write - write a VPD EEPROM location
- * @adapter: adapter to write
- * @addr: EEPROM address
- * @data: value to write
- *
- * Write a 32-bit word to a location in VPD EEPROM using the card's PCI
- * VPD ROM capability.
- */
-int t3_seeprom_write(struct adapter *adapter, u32 addr, __le32 data)
-{
- u16 val;
- int attempts = EEPROM_MAX_POLL;
- unsigned int base = adapter->params.pci.vpd_cap_addr;
-
- if ((addr >= EEPROMSIZE && addr != EEPROM_STAT_ADDR) || (addr & 3))
- return -EINVAL;
-
- pci_write_config_dword(adapter->pdev, base + PCI_VPD_DATA,
- le32_to_cpu(data));
- pci_write_config_word(adapter->pdev,base + PCI_VPD_ADDR,
- addr | PCI_VPD_ADDR_F);
- do {
- msleep(1);
- pci_read_config_word(adapter->pdev, base + PCI_VPD_ADDR, &val);
- } while ((val & PCI_VPD_ADDR_F) && --attempts);
-
- if (val & PCI_VPD_ADDR_F) {
- CH_ERR(adapter, "write to EEPROM address 0x%x failed\n", addr);
- return -EIO;
- }
- return 0;
-}
-
-/**
* t3_seeprom_wp - enable/disable EEPROM write protection
* @adapter: the adapter
* @enable: 1 to enable write protection, 0 to disable it
*/
int t3_seeprom_wp(struct adapter *adapter, int enable)
{
- return t3_seeprom_write(adapter, EEPROM_STAT_ADDR, enable ? 0xc : 0);
+ u32 data = enable ? 0xc : 0;
+ int ret;
+
+ /* EEPROM_STAT_ADDR is outside VPD area, use pci_write_vpd_any() */
+ ret = pci_write_vpd_any(adapter->pdev, EEPROM_STAT_ADDR, sizeof(u32),
+ &data);
+
+ return ret < 0 ? ret : 0;
}
static int vpdstrtouint(char *s, u8 len, unsigned int base, unsigned int *val)
*/
static int get_vpd_params(struct adapter *adapter, struct vpd_params *p)
{
- int i, addr, ret;
struct t3_vpd vpd;
+ u8 base_val = 0;
+ int addr, ret;
/*
* Card information is normally at VPD_BASE but some early cards had
* it at 0.
*/
- ret = t3_seeprom_read(adapter, VPD_BASE, (__le32 *)&vpd);
- if (ret)
+ ret = pci_read_vpd(adapter->pdev, VPD_BASE, 1, &base_val);
+ if (ret < 0)
return ret;
- addr = vpd.id_tag == 0x82 ? VPD_BASE : 0;
+ addr = base_val == PCI_VPD_LRDT_ID_STRING ? VPD_BASE : 0;
- for (i = 0; i < sizeof(vpd); i += 4) {
- ret = t3_seeprom_read(adapter, addr + i,
- (__le32 *)((u8 *)&vpd + i));
- if (ret)
- return ret;
- }
+ ret = pci_read_vpd(adapter->pdev, addr, sizeof(vpd), &vpd);
+ if (ret < 0)
+ return ret;
ret = vpdstrtouint(vpd.cclk_data, vpd.cclk_len, 10, &p->cclk);
if (ret)
if (ret)
return ret;
- if (!sff8472_comp || (sff_diag_type & 4)) {
+ if (!sff8472_comp || (sff_diag_type & SFP_DIAG_ADDRMODE)) {
modinfo->type = ETH_MODULE_SFF_8079;
modinfo->eeprom_len = ETH_MODULE_SFF_8079_LEN;
} else {
modinfo->type = ETH_MODULE_SFF_8472;
- modinfo->eeprom_len = ETH_MODULE_SFF_8472_LEN;
+ if (sff_diag_type & SFP_DIAG_IMPLEMENTED)
+ modinfo->eeprom_len = ETH_MODULE_SFF_8472_LEN;
+ else
+ modinfo->eeprom_len = ETH_MODULE_SFF_8472_LEN / 2;
}
break;
#define I2C_PAGE_SIZE 0x100
#define SFP_DIAG_TYPE_ADDR 0x5c
#define SFP_DIAG_TYPE_LEN 0x1
+#define SFP_DIAG_ADDRMODE BIT(2)
+#define SFP_DIAG_IMPLEMENTED BIT(6)
#define SFF_8472_COMP_ADDR 0x5e
#define SFF_8472_COMP_LEN 0x1
#define SFF_REV_ADDR 0x1
lp->ibn = 3;
lp->active = *p++;
if (MOTO_SROM_BUG) lp->active = 0;
+ /* if (MOTO_SROM_BUG) statement indicates lp->active could
+ * be 8 (i.e. the size of array lp->phy) */
+ if (WARN_ON(lp->active >= ARRAY_SIZE(lp->phy)))
+ return -EINVAL;
lp->phy[lp->active].gep = (*p ? p : NULL); p += (2 * (*p) + 1);
lp->phy[lp->active].rst = (*p ? p : NULL); p += (2 * (*p) + 1);
lp->phy[lp->active].mc = get_unaligned_le16(p); p += 2;
}
if ((j == limit) && (i < DE4X5_MAX_MII)) {
for (k=0; k < DE4X5_MAX_PHY && lp->phy[k].id; k++);
- lp->phy[k].addr = i;
- lp->phy[k].id = id;
- lp->phy[k].spd.reg = GENERIC_REG; /* ANLPA register */
- lp->phy[k].spd.mask = GENERIC_MASK; /* 100Mb/s technologies */
- lp->phy[k].spd.value = GENERIC_VALUE; /* TX & T4, H/F Duplex */
- lp->mii_cnt++;
- lp->active++;
- printk("%s: Using generic MII device control. If the board doesn't operate,\nplease mail the following dump to the author:\n", dev->name);
- j = de4x5_debug;
- de4x5_debug |= DEBUG_MII;
- de4x5_dbg_mii(dev, k);
- de4x5_debug = j;
- printk("\n");
+ if (k < DE4X5_MAX_PHY) {
+ lp->phy[k].addr = i;
+ lp->phy[k].id = id;
+ lp->phy[k].spd.reg = GENERIC_REG; /* ANLPA register */
+ lp->phy[k].spd.mask = GENERIC_MASK; /* 100Mb/s technologies */
+ lp->phy[k].spd.value = GENERIC_VALUE; /* TX & T4, H/F Duplex */
+ lp->mii_cnt++;
+ lp->active++;
+ printk("%s: Using generic MII device control. If the board doesn't operate,\nplease mail the following dump to the author:\n", dev->name);
+ j = de4x5_debug;
+ de4x5_debug |= DEBUG_MII;
+ de4x5_dbg_mii(dev, k);
+ de4x5_debug = j;
+ printk("\n");
+ } else {
+ goto purgatory;
+ }
}
}
purgatory:
fsl_mc_portal_free(priv->mc_io);
- free_netdev(net_dev);
-
dev_dbg(net_dev->dev.parent, "Removed interface %s\n", net_dev->name);
+ free_netdev(net_dev);
+
return 0;
}
goto reset;
ntfy_idx = gve_tx_idx_to_ntfy(priv, txqueue);
- if (ntfy_idx > priv->num_ntfy_blks)
+ if (ntfy_idx >= priv->num_ntfy_blks)
goto reset;
block = &priv->ntfy_blocks[ntfy_idx];
rx->rx_copied_pkt++;
rx->rx_frag_copy_cnt++;
rx->rx_copybreak_pkt++;
- } u64_stats_update_end(&rx->statss);
+ u64_stats_update_end(&rx->statss);
+ }
} else {
if (rx->data.raw_addressing) {
int recycle = gve_rx_can_recycle_buffer(page_info);
return;
if (!HNS_DSAF_IS_DEBUG(dsaf_dev)) {
+ /* DSAF_MAX_PORT_NUM is 6, but DSAF_GE_NUM is 8.
+ We need check to prevent array overflow */
+ if (port >= DSAF_MAX_PORT_NUM)
+ return;
reg_val_1 = 0x1 << port;
port_rst_off = dsaf_dev->mac_cb[port]->port_rst_off;
/* there is difference between V1 and V2 in register.*/
}
out:
+ /* sync head pointer before exiting, since hardware will calculate
+ * FBD number with head pointer
+ */
+ if (unused_count > 0)
+ failure = failure ||
+ hns3_nic_alloc_rx_buffers(ring, unused_count);
+
return failure ? budget : recv_pkts;
}
}
/**
- * hns3_lp_run_test - run loopback test
+ * hns3_lp_run_test - run loopback test
* @ndev: net device
* @mode: loopback type
+ *
+ * Return: %0 for success or a NIC loopback test error code on failure
*/
static int hns3_lp_run_test(struct net_device *ndev, enum hnae3_loop mode)
{
}
/**
- * hns3_nic_self_test - self test
+ * hns3_self_test - self test
* @ndev: net device
* @eth_test: test cmd
* @data: test result
return;
}
- strncpy(drvinfo->driver, h->pdev->driver->name,
+ strncpy(drvinfo->driver, dev_driver_string(&h->pdev->dev),
sizeof(drvinfo->driver));
drvinfo->driver[sizeof(drvinfo->driver) - 1] = '\0';
if (hnae3_dev_phy_imp_supported(hdev))
hnae3_set_bit(compat, HCLGE_PHY_IMP_EN_B, 1);
hnae3_set_bit(compat, HCLGE_MAC_STATS_EXT_EN_B, 1);
+ hnae3_set_bit(compat, HCLGE_SYNC_RX_RING_HEAD_EN_B, 1);
req->compat = cpu_to_le32(compat);
}
#define HCLGE_NCSI_ERROR_REPORT_EN_B 1
#define HCLGE_PHY_IMP_EN_B 2
#define HCLGE_MAC_STATS_EXT_EN_B 3
+#define HCLGE_SYNC_RX_RING_HEAD_EN_B 4
struct hclge_firmware_compat_cmd {
__le32 compat;
u8 rsv[20];
u32 total_ets_bw = 0;
u8 i;
- for (i = 0; i < hdev->tc_max; i++) {
+ for (i = 0; i < HNAE3_MAX_TC; i++) {
switch (ets->tc_tsa[i]) {
case IEEE_8021QAZ_TSA_STRICT:
if (hdev->tm_info.tc_info[i].tc_sch_mode !=
static int hclge_ieee_getpfc(struct hnae3_handle *h, struct ieee_pfc *pfc)
{
- u64 requests[HNAE3_MAX_TC], indications[HNAE3_MAX_TC];
struct hclge_vport *vport = hclge_get_vport(h);
struct hclge_dev *hdev = vport->back;
int ret;
- u8 i;
memset(pfc, 0, sizeof(*pfc));
pfc->pfc_cap = hdev->pfc_max;
pfc->pfc_en = hdev->tm_info.pfc_en;
- ret = hclge_pfc_tx_stats_get(hdev, requests);
- if (ret)
+ ret = hclge_mac_update_stats(hdev);
+ if (ret) {
+ dev_err(&hdev->pdev->dev,
+ "failed to update MAC stats, ret = %d.\n", ret);
return ret;
+ }
- ret = hclge_pfc_rx_stats_get(hdev, indications);
- if (ret)
- return ret;
+ hclge_pfc_tx_stats_get(hdev, pfc->requests);
+ hclge_pfc_rx_stats_get(hdev, pfc->indications);
- for (i = 0; i < HCLGE_MAX_TC_NUM; i++) {
- pfc->requests[i] = requests[i];
- pfc->indications[i] = indications[i];
- }
return 0;
}
#include "hclge_devlink.h"
#define HCLGE_NAME "hclge"
-#define HCLGE_STATS_READ(p, offset) (*(u64 *)((u8 *)(p) + (offset)))
-#define HCLGE_MAC_STATS_FIELD_OFF(f) (offsetof(struct hclge_mac_stats, f))
#define HCLGE_BUF_SIZE_UNIT 256U
#define HCLGE_BUF_MUL_BY 2
struct hclge_desc desc;
int ret;
+ /* Driver needs total register number of both valid registers and
+ * reserved registers, but the old firmware only returns number
+ * of valid registers in device V2. To be compatible with these
+ * devices, driver uses a fixed value.
+ */
+ if (hdev->ae_dev->dev_version == HNAE3_DEVICE_VERSION_V2) {
+ *reg_num = HCLGE_MAC_STATS_MAX_NUM_V1;
+ return 0;
+ }
+
hclge_cmd_setup_basic_desc(&desc, HCLGE_OPC_QUERY_MAC_REG_NUM, true);
ret = hclge_cmd_send(&hdev->hw, &desc, 1);
if (ret) {
return 0;
}
-static int hclge_mac_update_stats(struct hclge_dev *hdev)
+int hclge_mac_update_stats(struct hclge_dev *hdev)
{
/* The firmware supports the new statistics acquisition method */
if (hdev->ae_dev->dev_specs.mac_stats_num)
if (hdev->num_msi < hdev->num_nic_msi + hdev->num_roce_msi)
return -EINVAL;
- roce->rinfo.base_vector = hdev->roce_base_vector;
+ roce->rinfo.base_vector = hdev->num_nic_msi;
roce->rinfo.netdev = nic->kinfo.netdev;
roce->rinfo.roce_io_base = hdev->hw.io_base;
hdev->num_msi = vectors;
hdev->num_msi_left = vectors;
- hdev->base_msi_vector = pdev->irq;
- hdev->roce_base_vector = hdev->base_msi_vector +
- hdev->num_nic_msi;
-
hdev->vector_status = devm_kcalloc(&pdev->dev, hdev->num_msi,
sizeof(u16), GFP_KERNEL);
if (!hdev->vector_status) {
err_no_space:
/* if already overflow, not to print each time */
- if (!(vport->overflow_promisc_flags & HNAE3_OVERFLOW_MPE))
+ if (!(vport->overflow_promisc_flags & HNAE3_OVERFLOW_MPE)) {
+ vport->overflow_promisc_flags |= HNAE3_OVERFLOW_MPE;
dev_err(&hdev->pdev->dev, "mc mac vlan table is full\n");
+ }
+
return -ENOSPC;
}
static void hclge_sync_vport_mac_list(struct hclge_vport *vport,
struct list_head *list,
- int (*sync)(struct hclge_vport *,
- const unsigned char *))
+ enum HCLGE_MAC_ADDR_TYPE mac_type)
{
+ int (*sync)(struct hclge_vport *vport, const unsigned char *addr);
struct hclge_mac_node *mac_node, *tmp;
int ret;
+ if (mac_type == HCLGE_MAC_ADDR_UC)
+ sync = hclge_add_uc_addr_common;
+ else
+ sync = hclge_add_mc_addr_common;
+
list_for_each_entry_safe(mac_node, tmp, list, node) {
ret = sync(vport, mac_node->mac_addr);
if (!ret) {
/* If one unicast mac address is existing in hardware,
* we need to try whether other unicast mac addresses
* are new addresses that can be added.
+ * Multicast mac address can be reusable, even though
+ * there is no space to add new multicast mac address,
+ * we should check whether other mac addresses are
+ * existing in hardware for reuse.
*/
- if (ret != -EEXIST)
+ if ((mac_type == HCLGE_MAC_ADDR_UC && ret != -EEXIST) ||
+ (mac_type == HCLGE_MAC_ADDR_MC && ret != -ENOSPC))
break;
}
}
static void hclge_unsync_vport_mac_list(struct hclge_vport *vport,
struct list_head *list,
- int (*unsync)(struct hclge_vport *,
- const unsigned char *))
+ enum HCLGE_MAC_ADDR_TYPE mac_type)
{
+ int (*unsync)(struct hclge_vport *vport, const unsigned char *addr);
struct hclge_mac_node *mac_node, *tmp;
int ret;
+ if (mac_type == HCLGE_MAC_ADDR_UC)
+ unsync = hclge_rm_uc_addr_common;
+ else
+ unsync = hclge_rm_mc_addr_common;
+
list_for_each_entry_safe(mac_node, tmp, list, node) {
ret = unsync(vport, mac_node->mac_addr);
if (!ret || ret == -ENOENT) {
spin_unlock_bh(&vport->mac_list_lock);
/* delete first, in order to get max mac table space for adding */
- if (mac_type == HCLGE_MAC_ADDR_UC) {
- hclge_unsync_vport_mac_list(vport, &tmp_del_list,
- hclge_rm_uc_addr_common);
- hclge_sync_vport_mac_list(vport, &tmp_add_list,
- hclge_add_uc_addr_common);
- } else {
- hclge_unsync_vport_mac_list(vport, &tmp_del_list,
- hclge_rm_mc_addr_common);
- hclge_sync_vport_mac_list(vport, &tmp_add_list,
- hclge_add_mc_addr_common);
- }
+ hclge_unsync_vport_mac_list(vport, &tmp_del_list, mac_type);
+ hclge_sync_vport_mac_list(vport, &tmp_add_list, mac_type);
/* if some mac addresses were added/deleted fail, move back to the
* mac_list, and retry at next time.
spin_unlock_bh(&vport->mac_list_lock);
- if (mac_type == HCLGE_MAC_ADDR_UC)
- hclge_unsync_vport_mac_list(vport, &tmp_del_list,
- hclge_rm_uc_addr_common);
- else
- hclge_unsync_vport_mac_list(vport, &tmp_del_list,
- hclge_rm_mc_addr_common);
+ hclge_unsync_vport_mac_list(vport, &tmp_del_list, mac_type);
if (!list_empty(&tmp_del_list))
dev_warn(&hdev->pdev->dev,
return return_status;
}
-static bool hclge_check_vf_mac_exist(struct hclge_vport *vport, int vf_idx,
- u8 *mac_addr)
-{
- struct hclge_mac_vlan_tbl_entry_cmd req;
- struct hclge_dev *hdev = vport->back;
- struct hclge_desc desc;
- u16 egress_port = 0;
- int i;
-
- if (is_zero_ether_addr(mac_addr))
- return false;
-
- memset(&req, 0, sizeof(req));
- hnae3_set_field(egress_port, HCLGE_MAC_EPORT_VFID_M,
- HCLGE_MAC_EPORT_VFID_S, vport->vport_id);
- req.egress_port = cpu_to_le16(egress_port);
- hclge_prepare_mac_addr(&req, mac_addr, false);
-
- if (hclge_lookup_mac_vlan_tbl(vport, &req, &desc, false) != -ENOENT)
- return true;
-
- vf_idx += HCLGE_VF_VPORT_START_NUM;
- for (i = HCLGE_VF_VPORT_START_NUM; i < hdev->num_alloc_vport; i++)
- if (i != vf_idx &&
- ether_addr_equal(mac_addr, hdev->vport[i].vf_info.mac))
- return true;
-
- return false;
-}
-
static int hclge_set_vf_mac(struct hnae3_handle *handle, int vf,
u8 *mac_addr)
{
return 0;
}
- if (hclge_check_vf_mac_exist(vport, vf, mac_addr)) {
- dev_err(&hdev->pdev->dev, "Specified MAC(=%pM) exists!\n",
- mac_addr);
- return -EEXIST;
- }
-
ether_addr_copy(vport->vf_info.mac, mac_addr);
if (test_bit(HCLGE_VPORT_STATE_ALIVE, &vport->state)) {
};
/* max number of mac statistics on each version */
-#define HCLGE_MAC_STATS_MAX_NUM_V1 84
+#define HCLGE_MAC_STATS_MAX_NUM_V1 87
#define HCLGE_MAC_STATS_MAX_NUM_V2 105
struct hclge_comm_stats_str {
(y) = (_k_ ^ ~_v_) & (_k_); \
} while (0)
+#define HCLGE_MAC_STATS_FIELD_OFF(f) (offsetof(struct hclge_mac_stats, f))
+#define HCLGE_STATS_READ(p, offset) (*(u64 *)((u8 *)(p) + (offset)))
+
#define HCLGE_MAC_TNL_LOG_SIZE 8
#define HCLGE_VPORT_NUM 256
struct hclge_dev {
u16 num_msi;
u16 num_msi_left;
u16 num_msi_used;
- u32 base_msi_vector;
u16 *vector_status;
int *vector_irq;
u16 num_nic_msi; /* Num of nic vectors for this PF */
u16 num_roce_msi; /* Num of roce vectors for this PF */
- int roce_base_vector;
unsigned long service_timer_period;
unsigned long service_timer_previous;
int hclge_dbg_dump_rst_info(struct hclge_dev *hdev, char *buf, int len);
int hclge_push_vf_link_status(struct hclge_vport *vport);
int hclge_enable_vport_vlan_filter(struct hclge_vport *vport, bool request_en);
+int hclge_mac_update_stats(struct hclge_dev *hdev);
#endif
return 0;
}
-static int hclge_pfc_stats_get(struct hclge_dev *hdev,
- enum hclge_opcode_type opcode, u64 *stats)
-{
- struct hclge_desc desc[HCLGE_TM_PFC_PKT_GET_CMD_NUM];
- int ret, i, j;
-
- if (!(opcode == HCLGE_OPC_QUERY_PFC_RX_PKT_CNT ||
- opcode == HCLGE_OPC_QUERY_PFC_TX_PKT_CNT))
- return -EINVAL;
-
- for (i = 0; i < HCLGE_TM_PFC_PKT_GET_CMD_NUM - 1; i++) {
- hclge_cmd_setup_basic_desc(&desc[i], opcode, true);
- desc[i].flag |= cpu_to_le16(HCLGE_CMD_FLAG_NEXT);
- }
-
- hclge_cmd_setup_basic_desc(&desc[i], opcode, true);
+static const u16 hclge_pfc_tx_stats_offset[] = {
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri0_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri1_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri2_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri3_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri4_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri5_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri6_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_tx_pfc_pri7_pkt_num)
+};
- ret = hclge_cmd_send(&hdev->hw, desc, HCLGE_TM_PFC_PKT_GET_CMD_NUM);
- if (ret)
- return ret;
+static const u16 hclge_pfc_rx_stats_offset[] = {
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri0_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri1_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri2_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri3_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri4_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri5_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri6_pkt_num),
+ HCLGE_MAC_STATS_FIELD_OFF(mac_rx_pfc_pri7_pkt_num)
+};
- for (i = 0; i < HCLGE_TM_PFC_PKT_GET_CMD_NUM; i++) {
- struct hclge_pfc_stats_cmd *pfc_stats =
- (struct hclge_pfc_stats_cmd *)desc[i].data;
+static void hclge_pfc_stats_get(struct hclge_dev *hdev, bool tx, u64 *stats)
+{
+ const u16 *offset;
+ int i;
- for (j = 0; j < HCLGE_TM_PFC_NUM_GET_PER_CMD; j++) {
- u32 index = i * HCLGE_TM_PFC_PKT_GET_CMD_NUM + j;
+ if (tx)
+ offset = hclge_pfc_tx_stats_offset;
+ else
+ offset = hclge_pfc_rx_stats_offset;
- if (index < HCLGE_MAX_TC_NUM)
- stats[index] =
- le64_to_cpu(pfc_stats->pkt_num[j]);
- }
- }
- return 0;
+ for (i = 0; i < HCLGE_MAX_TC_NUM; i++)
+ stats[i] = HCLGE_STATS_READ(&hdev->mac_stats, offset[i]);
}
-int hclge_pfc_rx_stats_get(struct hclge_dev *hdev, u64 *stats)
+void hclge_pfc_rx_stats_get(struct hclge_dev *hdev, u64 *stats)
{
- return hclge_pfc_stats_get(hdev, HCLGE_OPC_QUERY_PFC_RX_PKT_CNT, stats);
+ hclge_pfc_stats_get(hdev, false, stats);
}
-int hclge_pfc_tx_stats_get(struct hclge_dev *hdev, u64 *stats)
+void hclge_pfc_tx_stats_get(struct hclge_dev *hdev, u64 *stats)
{
- return hclge_pfc_stats_get(hdev, HCLGE_OPC_QUERY_PFC_TX_PKT_CNT, stats);
+ hclge_pfc_stats_get(hdev, true, stats);
}
int hclge_mac_pause_en_cfg(struct hclge_dev *hdev, bool tx, bool rx)
static int hclge_tm_ets_tc_dwrr_cfg(struct hclge_dev *hdev)
{
-#define DEFAULT_TC_WEIGHT 1
#define DEFAULT_TC_OFFSET 14
struct hclge_ets_tc_weight_cmd *ets_weight;
for (i = 0; i < HNAE3_MAX_TC; i++) {
struct hclge_pg_info *pg_info;
- ets_weight->tc_weight[i] = DEFAULT_TC_WEIGHT;
-
- if (!(hdev->hw_tc_map & BIT(i)))
- continue;
-
- pg_info =
- &hdev->tm_info.pg_info[hdev->tm_info.tc_info[i].pgid];
+ pg_info = &hdev->tm_info.pg_info[hdev->tm_info.tc_info[i].pgid];
ets_weight->tc_weight[i] = pg_info->tc_dwrr[i];
}
int hclge_tm_init_hw(struct hclge_dev *hdev, bool init);
int hclge_mac_pause_en_cfg(struct hclge_dev *hdev, bool tx, bool rx);
int hclge_pause_addr_cfg(struct hclge_dev *hdev, const u8 *mac_addr);
-int hclge_pfc_rx_stats_get(struct hclge_dev *hdev, u64 *stats);
-int hclge_pfc_tx_stats_get(struct hclge_dev *hdev, u64 *stats);
+void hclge_pfc_rx_stats_get(struct hclge_dev *hdev, u64 *stats);
+void hclge_pfc_tx_stats_get(struct hclge_dev *hdev, u64 *stats);
int hclge_tm_qs_shaper_cfg(struct hclge_vport *vport, int max_tx_rate);
int hclge_tm_get_qset_num(struct hclge_dev *hdev, u16 *qset_num);
int hclge_tm_get_pri_num(struct hclge_dev *hdev, u8 *pri_num);
return ret;
}
+static int hclgevf_firmware_compat_config(struct hclgevf_dev *hdev, bool en)
+{
+ struct hclgevf_firmware_compat_cmd *req;
+ struct hclgevf_desc desc;
+ u32 compat = 0;
+
+ hclgevf_cmd_setup_basic_desc(&desc, HCLGEVF_OPC_IMP_COMPAT_CFG, false);
+
+ if (en) {
+ req = (struct hclgevf_firmware_compat_cmd *)desc.data;
+
+ hnae3_set_bit(compat, HCLGEVF_SYNC_RX_RING_HEAD_EN_B, 1);
+
+ req->compat = cpu_to_le32(compat);
+ }
+
+ return hclgevf_cmd_send(&hdev->hw, &desc, 1);
+}
+
int hclgevf_cmd_init(struct hclgevf_dev *hdev)
{
+ struct hnae3_ae_dev *ae_dev = pci_get_drvdata(hdev->pdev);
int ret;
spin_lock_bh(&hdev->hw.cmq.csq.lock);
hnae3_get_field(hdev->fw_version, HNAE3_FW_VERSION_BYTE0_MASK,
HNAE3_FW_VERSION_BYTE0_SHIFT));
+ if (ae_dev->dev_version >= HNAE3_DEVICE_VERSION_V3) {
+ /* ask the firmware to enable some features, driver can work
+ * without it.
+ */
+ ret = hclgevf_firmware_compat_config(hdev, true);
+ if (ret)
+ dev_warn(&hdev->pdev->dev,
+ "Firmware compatible features not enabled(%d).\n",
+ ret);
+ }
+
return 0;
err_cmd_init:
void hclgevf_cmd_uninit(struct hclgevf_dev *hdev)
{
+ hclgevf_firmware_compat_config(hdev, false);
set_bit(HCLGEVF_STATE_CMD_DISABLE, &hdev->state);
/* wait to ensure that the firmware completes the possible left
* over commands.
struct hclgevf_hw;
struct hclgevf_dev;
+#define HCLGEVF_SYNC_RX_RING_HEAD_EN_B 4
+struct hclgevf_firmware_compat_cmd {
+ __le32 compat;
+ u8 rsv[20];
+};
+
struct hclgevf_desc {
__le16 opcode;
__le16 flag;
HCLGEVF_OPC_RSS_TC_MODE = 0x0D08,
/* Mailbox cmd */
HCLGEVF_OPC_MBX_VF_TO_PF = 0x2001,
+
+ /* IMP stats command */
+ HCLGEVF_OPC_IMP_COMPAT_CFG = 0x701A,
};
#define HCLGEVF_TQP_REG_OFFSET 0x80000
hdev->num_msi_left == 0)
return -EINVAL;
- roce->rinfo.base_vector = hdev->roce_base_vector;
+ roce->rinfo.base_vector = hdev->roce_base_msix_offset;
roce->rinfo.netdev = nic->kinfo.netdev;
roce->rinfo.roce_io_base = hdev->hw.io_base;
hdev->num_msi = vectors;
hdev->num_msi_left = vectors;
- hdev->base_msi_vector = pdev->irq;
- hdev->roce_base_vector = pdev->irq + hdev->roce_base_msix_offset;
-
hdev->vector_status = devm_kcalloc(&pdev->dev, hdev->num_msi,
sizeof(u16), GFP_KERNEL);
if (!hdev->vector_status) {
/* un-init roce, if it exists */
if (hdev->roce_client) {
+ while (test_bit(HCLGEVF_STATE_RST_HANDLING, &hdev->state))
+ msleep(HCLGEVF_WAIT_RESET_DONE);
clear_bit(HCLGEVF_STATE_ROCE_REGISTERED, &hdev->state);
+
hdev->roce_client->ops->uninit_instance(&hdev->roce, 0);
hdev->roce_client = NULL;
hdev->roce.client = NULL;
/* un-init nic/unic, if this was not called by roce client */
if (client->ops->uninit_instance && hdev->nic_client &&
client->type != HNAE3_CLIENT_ROCE) {
+ while (test_bit(HCLGEVF_STATE_RST_HANDLING, &hdev->state))
+ msleep(HCLGEVF_WAIT_RESET_DONE);
clear_bit(HCLGEVF_STATE_NIC_REGISTERED, &hdev->state);
client->ops->uninit_instance(&hdev->nic, 0);
#define HCLGEVF_VF_RST_ING 0x07008
#define HCLGEVF_VF_RST_ING_BIT BIT(16)
+#define HCLGEVF_WAIT_RESET_DONE 100
+
#define HCLGEVF_RSS_IND_TBL_SIZE 512
#define HCLGEVF_RSS_SET_BITMAP_MSK 0xffff
#define HCLGEVF_RSS_KEY_SIZE 40
u16 num_nic_msix; /* Num of nic vectors for this VF */
u16 num_roce_msix; /* Num of roce vectors for this VF */
u16 roce_base_msix_offset;
- int roce_base_vector;
- u32 base_msi_vector;
u16 *vector_status;
int *vector_irq;
struct net_device *netdev = pci_get_drvdata(pdev);
struct nic *nic = netdev_priv(netdev);
+ netif_device_detach(netdev);
+
if (netif_running(netdev))
e100_down(nic);
- netif_device_detach(netdev);
if ((nic->flags & wol_magic) | e100_asf(nic)) {
/* enable reverse auto-negotiation */
*enable_wake = false;
}
- pci_clear_master(pdev);
+ pci_disable_device(pdev);
}
static int __e100_power_off(struct pci_dev *pdev, bool wake)
__e100_shutdown(to_pci_dev(dev_d), &wake);
- device_wakeup_disable(dev_d);
-
return 0;
}
{
struct net_device *netdev = dev_get_drvdata(dev_d);
struct nic *nic = netdev_priv(netdev);
+ int err;
+
+ err = pci_enable_device(to_pci_dev(dev_d));
+ if (err) {
+ netdev_err(netdev, "Resume cannot enable PCI device, aborting\n");
+ return err;
+ }
+ pci_set_master(to_pci_dev(dev_d));
/* disable reverse auto-negotiation */
if (nic->phy == phy_82552_v) {
smartspeed & ~(E100_82552_REV_ANEG));
}
- netif_device_attach(netdev);
if (netif_running(netdev))
e100_up(nic);
+ netif_device_attach(netdev);
+
return 0;
}
__I40E_VSI_OVERFLOW_PROMISC,
__I40E_VSI_REINIT_REQUESTED,
__I40E_VSI_DOWN_REQUESTED,
+ __I40E_VSI_RELEASING,
/* This must be last as it determines the size of the BITMAP */
__I40E_VSI_STATE_SIZE__,
};
void i40e_ptp_init(struct i40e_pf *pf);
void i40e_ptp_stop(struct i40e_pf *pf);
int i40e_ptp_alloc_pins(struct i40e_pf *pf);
+int i40e_update_adq_vsi_queues(struct i40e_vsi *vsi, int vsi_offset);
int i40e_is_vsi_uplink_mode_veb(struct i40e_vsi *vsi);
i40e_status i40e_get_partition_bw_setting(struct i40e_pf *pf);
i40e_status i40e_set_partition_bw_setting(struct i40e_pf *pf);
bool is_add)
{
struct i40e_pf *pf = vsi->back;
+ u16 num_tc_qps = 0;
u16 sections = 0;
u8 netdev_tc = 0;
u16 numtc = 1;
u8 offset;
u16 qmap;
int i;
- u16 num_tc_qps = 0;
sections = I40E_AQ_VSI_PROP_QUEUE_MAP_VALID;
offset = 0;
+ /* zero out queue mapping, it will get updated on the end of the function */
+ memset(ctxt->info.queue_mapping, 0, sizeof(ctxt->info.queue_mapping));
+
+ if (vsi->type == I40E_VSI_MAIN) {
+ /* This code helps add more queue to the VSI if we have
+ * more cores than RSS can support, the higher cores will
+ * be served by ATR or other filters. Furthermore, the
+ * non-zero req_queue_pairs says that user requested a new
+ * queue count via ethtool's set_channels, so use this
+ * value for queues distribution across traffic classes
+ */
+ if (vsi->req_queue_pairs > 0)
+ vsi->num_queue_pairs = vsi->req_queue_pairs;
+ else if (pf->flags & I40E_FLAG_MSIX_ENABLED)
+ vsi->num_queue_pairs = pf->num_lan_msix;
+ }
/* Number of queues per enabled TC */
- num_tc_qps = vsi->alloc_queue_pairs;
+ if (vsi->type == I40E_VSI_MAIN ||
+ (vsi->type == I40E_VSI_SRIOV && vsi->num_queue_pairs != 0))
+ num_tc_qps = vsi->num_queue_pairs;
+ else
+ num_tc_qps = vsi->alloc_queue_pairs;
+
if (enabled_tc && (vsi->back->flags & I40E_FLAG_DCB_ENABLED)) {
/* Find numtc from enabled TC bitmap */
for (i = 0, numtc = 0; i < I40E_MAX_TRAFFIC_CLASS; i++) {
}
ctxt->info.tc_mapping[i] = cpu_to_le16(qmap);
}
-
- /* Set actual Tx/Rx queue pairs */
- vsi->num_queue_pairs = offset;
- if ((vsi->type == I40E_VSI_MAIN) && (numtc == 1)) {
- if (vsi->req_queue_pairs > 0)
- vsi->num_queue_pairs = vsi->req_queue_pairs;
- else if (pf->flags & I40E_FLAG_MSIX_ENABLED)
- vsi->num_queue_pairs = pf->num_lan_msix;
- }
+ /* Do not change previously set num_queue_pairs for PFs and VFs*/
+ if ((vsi->type == I40E_VSI_MAIN && numtc != 1) ||
+ (vsi->type == I40E_VSI_SRIOV && vsi->num_queue_pairs == 0) ||
+ (vsi->type != I40E_VSI_MAIN && vsi->type != I40E_VSI_SRIOV))
+ vsi->num_queue_pairs = offset;
/* Scheduler section valid can only be set for ADD VSI */
if (is_add) {
for (v = 0; v < pf->num_alloc_vsi; v++) {
if (pf->vsi[v] &&
- (pf->vsi[v]->flags & I40E_VSI_FLAG_FILTER_CHANGED)) {
+ (pf->vsi[v]->flags & I40E_VSI_FLAG_FILTER_CHANGED) &&
+ !test_bit(__I40E_VSI_RELEASING, pf->vsi[v]->state)) {
int ret = i40e_sync_vsi_filters(pf->vsi[v]);
if (ret) {
}
/**
+ * i40e_update_adq_vsi_queues - update queue mapping for ADq VSI
+ * @vsi: the VSI being reconfigured
+ * @vsi_offset: offset from main VF VSI
+ */
+int i40e_update_adq_vsi_queues(struct i40e_vsi *vsi, int vsi_offset)
+{
+ struct i40e_vsi_context ctxt = {};
+ struct i40e_pf *pf;
+ struct i40e_hw *hw;
+ int ret;
+
+ if (!vsi)
+ return I40E_ERR_PARAM;
+ pf = vsi->back;
+ hw = &pf->hw;
+
+ ctxt.seid = vsi->seid;
+ ctxt.pf_num = hw->pf_id;
+ ctxt.vf_num = vsi->vf_id + hw->func_caps.vf_base_id + vsi_offset;
+ ctxt.uplink_seid = vsi->uplink_seid;
+ ctxt.connection_type = I40E_AQ_VSI_CONN_TYPE_NORMAL;
+ ctxt.flags = I40E_AQ_VSI_TYPE_VF;
+ ctxt.info = vsi->info;
+
+ i40e_vsi_setup_queue_map(vsi, &ctxt, vsi->tc_config.enabled_tc,
+ false);
+ if (vsi->reconfig_rss) {
+ vsi->rss_size = min_t(int, pf->alloc_rss_size,
+ vsi->num_queue_pairs);
+ ret = i40e_vsi_config_rss(vsi);
+ if (ret) {
+ dev_info(&pf->pdev->dev, "Failed to reconfig rss for num_queues\n");
+ return ret;
+ }
+ vsi->reconfig_rss = false;
+ }
+
+ ret = i40e_aq_update_vsi_params(hw, &ctxt, NULL);
+ if (ret) {
+ dev_info(&pf->pdev->dev, "Update vsi config failed, err %s aq_err %s\n",
+ i40e_stat_str(hw, ret),
+ i40e_aq_str(hw, hw->aq.asq_last_status));
+ return ret;
+ }
+ /* update the local VSI info with updated queue map */
+ i40e_vsi_update_queue_map(vsi, &ctxt);
+ vsi->info.valid_sections = 0;
+
+ return ret;
+}
+
+/**
* i40e_vsi_config_tc - Configure VSI Tx Scheduler for given TC map
* @vsi: VSI to be configured
* @enabled_tc: TC bitmap
}
/**
- * i40e_is_any_channel - channel exist or not
- * @vsi: ptr to VSI to which channels are associated with
- *
- * Returns true or false if channel(s) exist for associated VSI or not
- **/
-static bool i40e_is_any_channel(struct i40e_vsi *vsi)
-{
- struct i40e_channel *ch, *ch_tmp;
-
- list_for_each_entry_safe(ch, ch_tmp, &vsi->ch_list, list) {
- if (ch->initialized)
- return true;
- }
-
- return false;
-}
-
-/**
* i40e_get_max_queues_for_channel
* @vsi: ptr to VSI to which channels are associated with
*
/* By default we are in VEPA mode, if this is the first VF/VMDq
* VSI to be added switch to VEB mode.
*/
- if ((!(pf->flags & I40E_FLAG_VEB_MODE_ENABLED)) ||
- (!i40e_is_any_channel(vsi))) {
- if (!is_power_of_2(vsi->tc_config.tc_info[0].qcount)) {
- dev_dbg(&pf->pdev->dev,
- "Failed to create channel. Override queues (%u) not power of 2\n",
- vsi->tc_config.tc_info[0].qcount);
- return -EINVAL;
- }
- if (!(pf->flags & I40E_FLAG_VEB_MODE_ENABLED)) {
- pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
+ if (!(pf->flags & I40E_FLAG_VEB_MODE_ENABLED)) {
+ pf->flags |= I40E_FLAG_VEB_MODE_ENABLED;
- if (vsi->type == I40E_VSI_MAIN) {
- if (pf->flags & I40E_FLAG_TC_MQPRIO)
- i40e_do_reset(pf, I40E_PF_RESET_FLAG,
- true);
- else
- i40e_do_reset_safe(pf,
- I40E_PF_RESET_FLAG);
- }
+ if (vsi->type == I40E_VSI_MAIN) {
+ if (pf->flags & I40E_FLAG_TC_MQPRIO)
+ i40e_do_reset(pf, I40E_PF_RESET_FLAG, true);
+ else
+ i40e_do_reset_safe(pf, I40E_PF_RESET_FLAG);
}
/* now onwards for main VSI, number of queues will be value
* of TC0's queue count
vsi->seid);
need_reset = true;
goto exit;
- } else {
- dev_info(&vsi->back->pdev->dev,
- "Setup channel (id:%u) utilizing num_queues %d\n",
- vsi->seid, vsi->tc_config.tc_info[0].qcount);
+ } else if (enabled_tc &&
+ (!is_power_of_2(vsi->tc_config.tc_info[0].qcount))) {
+ netdev_info(netdev,
+ "Failed to create channel. Override queues (%u) not power of 2\n",
+ vsi->tc_config.tc_info[0].qcount);
+ ret = -EINVAL;
+ need_reset = true;
+ goto exit;
}
+ dev_info(&vsi->back->pdev->dev,
+ "Setup channel (id:%u) utilizing num_queues %d\n",
+ vsi->seid, vsi->tc_config.tc_info[0].qcount);
+
if (pf->flags & I40E_FLAG_TC_MQPRIO) {
if (vsi->mqprio_qopt.max_rate[0]) {
u64 max_tx_rate = vsi->mqprio_qopt.max_rate[0];
err = i40e_add_del_cloud_filter(vsi, filter, true);
if (err) {
- dev_err(&pf->pdev->dev,
- "Failed to add cloud filter, err %s\n",
- i40e_stat_str(&pf->hw, err));
+ dev_err(&pf->pdev->dev, "Failed to add cloud filter, err %d\n",
+ err);
goto err;
}
dev_info(&pf->pdev->dev, "Can't remove PF VSI\n");
return -ENODEV;
}
-
+ set_bit(__I40E_VSI_RELEASING, vsi->state);
uplink_seid = vsi->uplink_seid;
if (vsi->type != I40E_VSI_SRIOV) {
if (vsi->netdev_registered) {
/***********************misc routines*****************************/
/**
- * i40e_vc_disable_vf
+ * i40e_vc_reset_vf
* @vf: pointer to the VF info
- *
- * Disable the VF through a SW reset.
+ * @notify_vf: notify vf about reset or not
+ * Reset VF handler.
**/
-static inline void i40e_vc_disable_vf(struct i40e_vf *vf)
+static void i40e_vc_reset_vf(struct i40e_vf *vf, bool notify_vf)
{
struct i40e_pf *pf = vf->pf;
int i;
- i40e_vc_notify_vf_reset(vf);
+ if (notify_vf)
+ i40e_vc_notify_vf_reset(vf);
/* We want to ensure that an actual reset occurs initiated after this
* function was called. However, we do not want to wait forever, so
usleep_range(10000, 20000);
}
- dev_warn(&vf->pf->pdev->dev,
- "Failed to initiate reset for VF %d after 200 milliseconds\n",
- vf->vf_id);
+ if (notify_vf)
+ dev_warn(&vf->pf->pdev->dev,
+ "Failed to initiate reset for VF %d after 200 milliseconds\n",
+ vf->vf_id);
+ else
+ dev_dbg(&vf->pf->pdev->dev,
+ "Failed to initiate reset for VF %d after 200 milliseconds\n",
+ vf->vf_id);
}
/**
u16 vsi_queue_id,
struct virtchnl_rxq_info *info)
{
+ u16 pf_queue_id = i40e_vc_get_pf_queue_id(vf, vsi_id, vsi_queue_id);
struct i40e_pf *pf = vf->pf;
+ struct i40e_vsi *vsi = pf->vsi[vf->lan_vsi_idx];
struct i40e_hw *hw = &pf->hw;
struct i40e_hmc_obj_rxq rx_ctx;
- u16 pf_queue_id;
int ret = 0;
- pf_queue_id = i40e_vc_get_pf_queue_id(vf, vsi_id, vsi_queue_id);
-
/* clear the context structure first */
memset(&rx_ctx, 0, sizeof(struct i40e_hmc_obj_rxq));
}
rx_ctx.rxmax = info->max_pkt_size;
+ /* if port VLAN is configured increase the max packet size */
+ if (vsi->info.pvid)
+ rx_ctx.rxmax += VLAN_HLEN;
+
/* enable 32bytes desc always */
rx_ctx.dsize = 1;
}
/**
- * i40e_vc_reset_vf_msg
- * @vf: pointer to the VF info
- *
- * called from the VF to reset itself,
- * unlike other virtchnl messages, PF driver
- * doesn't send the response back to the VF
- **/
-static void i40e_vc_reset_vf_msg(struct i40e_vf *vf)
-{
- if (test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states))
- i40e_reset_vf(vf, false);
-}
-
-/**
* i40e_vc_config_promiscuous_mode_msg
* @vf: pointer to the VF info
* @msg: pointer to the msg buffer
struct virtchnl_vsi_queue_config_info *qci =
(struct virtchnl_vsi_queue_config_info *)msg;
struct virtchnl_queue_pair_info *qpi;
- struct i40e_pf *pf = vf->pf;
u16 vsi_id, vsi_queue_id = 0;
- u16 num_qps_all = 0;
+ struct i40e_pf *pf = vf->pf;
i40e_status aq_ret = 0;
int i, j = 0, idx = 0;
+ struct i40e_vsi *vsi;
+ u16 num_qps_all = 0;
if (!test_bit(I40E_VF_STATE_ACTIVE, &vf->vf_states)) {
aq_ret = I40E_ERR_PARAM;
pf->vsi[vf->lan_vsi_idx]->num_queue_pairs =
qci->num_queue_pairs;
} else {
- for (i = 0; i < vf->num_tc; i++)
- pf->vsi[vf->ch[i].vsi_idx]->num_queue_pairs =
- vf->ch[i].num_qps;
+ for (i = 0; i < vf->num_tc; i++) {
+ vsi = pf->vsi[vf->ch[i].vsi_idx];
+ vsi->num_queue_pairs = vf->ch[i].num_qps;
+
+ if (i40e_update_adq_vsi_queues(vsi, i)) {
+ aq_ret = I40E_ERR_CONFIG;
+ goto error_param;
+ }
+ }
}
error_param:
} else {
/* successful request */
vf->num_req_queues = req_pairs;
- i40e_vc_notify_vf_reset(vf);
- i40e_reset_vf(vf, false);
+ i40e_vc_reset_vf(vf, true);
return 0;
}
vf->num_req_queues = 0;
/* reset the VF in order to allocate resources */
- i40e_vc_notify_vf_reset(vf);
- i40e_reset_vf(vf, false);
+ i40e_vc_reset_vf(vf, true);
return I40E_SUCCESS;
}
/* reset the VF in order to allocate resources */
- i40e_vc_notify_vf_reset(vf);
- i40e_reset_vf(vf, false);
+ i40e_vc_reset_vf(vf, true);
return I40E_SUCCESS;
i40e_vc_notify_vf_link_state(vf);
break;
case VIRTCHNL_OP_RESET_VF:
- i40e_vc_reset_vf_msg(vf);
+ i40e_vc_reset_vf(vf, false);
ret = 0;
break;
case VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE:
/* Force the VF interface down so it has to bring up with new MAC
* address
*/
- i40e_vc_disable_vf(vf);
+ i40e_vc_reset_vf(vf, true);
dev_info(&pf->pdev->dev, "Bring down and up the VF interface to make this change effective.\n");
error_param:
}
/**
- * i40e_vsi_has_vlans - True if VSI has configured VLANs
- * @vsi: pointer to the vsi
- *
- * Check if a VSI has configured any VLANs. False if we have a port VLAN or if
- * we have no configured VLANs. Do not call while holding the
- * mac_filter_hash_lock.
- */
-static bool i40e_vsi_has_vlans(struct i40e_vsi *vsi)
-{
- bool have_vlans;
-
- /* If we have a port VLAN, then the VSI cannot have any VLANs
- * configured, as all MAC/VLAN filters will be assigned to the PVID.
- */
- if (vsi->info.pvid)
- return false;
-
- /* Since we don't have a PVID, we know that if the device is in VLAN
- * mode it must be because of a VLAN filter configured on this VSI.
- */
- spin_lock_bh(&vsi->mac_filter_hash_lock);
- have_vlans = i40e_is_vsi_in_vlan(vsi);
- spin_unlock_bh(&vsi->mac_filter_hash_lock);
-
- return have_vlans;
-}
-
-/**
* i40e_ndo_set_vf_port_vlan
* @netdev: network interface device structure
* @vf_id: VF identifier
/* duplicate request, so just return success */
goto error_pvid;
- if (i40e_vsi_has_vlans(vsi)) {
- dev_err(&pf->pdev->dev,
- "VF %d has already configured VLAN filters and the administrator is requesting a port VLAN override.\nPlease unload and reload the VF driver for this change to take effect.\n",
- vf_id);
- /* Administrator Error - knock the VF offline until he does
- * the right thing by reconfiguring his network correctly
- * and then reloading the VF driver.
- */
- i40e_vc_disable_vf(vf);
- /* During reset the VF got a new VSI, so refresh the pointer. */
- vsi = pf->vsi[vf->lan_vsi_idx];
- }
-
+ i40e_vc_reset_vf(vf, true);
+ /* During reset the VF got a new VSI, so refresh a pointer. */
+ vsi = pf->vsi[vf->lan_vsi_idx];
/* Locked once because multiple functions below iterate list */
spin_lock_bh(&vsi->mac_filter_hash_lock);
goto out;
vf->trusted = setting;
- i40e_vc_disable_vf(vf);
+ i40e_vc_reset_vf(vf, true);
dev_info(&pf->pdev->dev, "VF %u is now %strusted\n",
vf_id, setting ? "" : "un");
#include "iavf_txrx.h"
#include "iavf_fdir.h"
#include "iavf_adv_rss.h"
+#include <linux/bitmap.h>
#define DEFAULT_DEBUG_LEVEL_SHIFT 3
#define PFX "iavf: "
{
struct iavf_adapter *adapter = netdev_priv(netdev);
u32 num_req = ch->combined_count;
+ int i;
if ((adapter->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_ADQ) &&
adapter->num_tc) {
/* All of these should have already been checked by ethtool before this
* even gets to us, but just to be sure.
*/
- if (num_req > adapter->vsi_res->num_queue_pairs)
+ if (num_req == 0 || num_req > adapter->vsi_res->num_queue_pairs)
return -EINVAL;
if (num_req == adapter->num_active_queues)
adapter->num_req_queues = num_req;
adapter->flags |= IAVF_FLAG_REINIT_ITR_NEEDED;
iavf_schedule_reset(adapter);
+
+ /* wait for the reset is done */
+ for (i = 0; i < IAVF_RESET_WAIT_COMPLETE_COUNT; i++) {
+ msleep(IAVF_RESET_WAIT_MS);
+ if (adapter->flags & IAVF_FLAG_RESET_PENDING)
+ continue;
+ break;
+ }
+ if (i == IAVF_RESET_WAIT_COMPLETE_COUNT) {
+ adapter->flags &= ~IAVF_FLAG_REINIT_ITR_NEEDED;
+ adapter->num_active_queues = num_req;
+ return -EOPNOTSUPP;
+ }
+
return 0;
}
if (hfunc)
*hfunc = ETH_RSS_HASH_TOP;
- if (!indir)
- return 0;
-
- memcpy(key, adapter->rss_key, adapter->rss_key_size);
+ if (key)
+ memcpy(key, adapter->rss_key, adapter->rss_key_size);
- /* Each 32 bits pointed by 'indir' is stored with a lut entry */
- for (i = 0; i < adapter->rss_lut_size; i++)
- indir[i] = (u32)adapter->rss_lut[i];
+ if (indir)
+ /* Each 32 bits pointed by 'indir' is stored with a lut entry */
+ for (i = 0; i < adapter->rss_lut_size; i++)
+ indir[i] = (u32)adapter->rss_lut[i];
return 0;
}
}
/**
+ * iavf_restore_filters
+ * @adapter: board private structure
+ *
+ * Restore existing non MAC filters when VF netdev comes back up
+ **/
+static void iavf_restore_filters(struct iavf_adapter *adapter)
+{
+ /* re-add all VLAN filters */
+ if (VLAN_ALLOWED(adapter)) {
+ u16 vid;
+
+ for_each_set_bit(vid, adapter->vsi.active_vlans, VLAN_N_VID)
+ iavf_add_vlan(adapter, vid);
+ }
+}
+
+/**
* iavf_vlan_rx_add_vid - Add a VLAN filter to a device
* @netdev: network device struct
* @proto: unused protocol data
if (!VLAN_ALLOWED(adapter))
return -EIO;
+
if (iavf_add_vlan(adapter, vid) == NULL)
return -ENOMEM;
+
+ set_bit(vid, adapter->vsi.active_vlans);
return 0;
}
{
struct iavf_adapter *adapter = netdev_priv(netdev);
- if (VLAN_ALLOWED(adapter)) {
- iavf_del_vlan(adapter, vid);
- return 0;
- }
- return -EIO;
+ if (!VLAN_ALLOWED(adapter))
+ return -EIO;
+
+ iavf_del_vlan(adapter, vid);
+ clear_bit(vid, adapter->vsi.active_vlans);
+
+ return 0;
}
/**
iavf_set_promiscuous(adapter, FLAG_VF_MULTICAST_PROMISC);
return 0;
}
-
- if ((adapter->aq_required & IAVF_FLAG_AQ_RELEASE_PROMISC) &&
+ if ((adapter->aq_required & IAVF_FLAG_AQ_RELEASE_PROMISC) ||
(adapter->aq_required & IAVF_FLAG_AQ_RELEASE_ALLMULTI)) {
iavf_set_promiscuous(adapter, 0);
return 0;
iavf_free_misc_irq(adapter);
iavf_reset_interrupt_capability(adapter);
- iavf_free_queues(adapter);
iavf_free_q_vectors(adapter);
+ iavf_free_queues(adapter);
memset(adapter->vf_res, 0, IAVF_VIRTCHNL_VF_RESOURCE_SIZE);
iavf_shutdown_adminq(&adapter->hw);
adapter->netdev->flags &= ~IFF_UP;
/* check for error indications */
val = rd32(hw, hw->aq.arq.len);
- if (val == 0xdeadbeef) /* indicates device in reset */
+ if (val == 0xdeadbeef || val == 0xffffffff) /* device in reset */
goto freedom;
oldval = val;
if (val & IAVF_VF_ARQLEN1_ARQVFE_MASK) {
return -ENOMEM;
while (!mutex_trylock(&adapter->crit_lock)) {
- if (--count == 0)
- goto err;
+ if (--count == 0) {
+ kfree(filter);
+ return err;
+ }
udelay(1);
}
/* start out with flow type and eth type IPv4 to begin with */
filter->f.flow_type = VIRTCHNL_TCP_V4_FLOW;
err = iavf_parse_cls_flower(adapter, cls_flower, filter);
- if (err < 0)
+ if (err)
goto err;
err = iavf_handle_tclass(adapter, tc, filter);
- if (err < 0)
+ if (err)
goto err;
/* add filter to the list */
spin_unlock_bh(&adapter->mac_vlan_list_lock);
+ /* Restore VLAN filters that were removed with IFF_DOWN */
+ iavf_restore_filters(adapter);
+
iavf_configure(adapter);
iavf_up_complete(adapter);
{
struct iavf_adapter *adapter = netdev_priv(netdev);
- if (!(adapter->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_VLAN))
+ if (adapter->vf_res &&
+ !(adapter->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_VLAN))
features &= ~(NETIF_F_HW_VLAN_CTAG_TX |
NETIF_F_HW_VLAN_CTAG_RX |
NETIF_F_HW_VLAN_CTAG_FILTER);
#define ice_for_each_chnl_tc(i) \
for ((i) = ICE_CHNL_START_TC; (i) < ICE_CHNL_MAX_TC; (i)++)
-#define ICE_UCAST_PROMISC_BITS (ICE_PROMISC_UCAST_TX | ICE_PROMISC_MCAST_TX | \
- ICE_PROMISC_UCAST_RX | ICE_PROMISC_MCAST_RX)
+#define ICE_UCAST_PROMISC_BITS (ICE_PROMISC_UCAST_TX | ICE_PROMISC_UCAST_RX)
#define ICE_UCAST_VLAN_PROMISC_BITS (ICE_PROMISC_UCAST_TX | \
- ICE_PROMISC_MCAST_TX | \
ICE_PROMISC_UCAST_RX | \
- ICE_PROMISC_MCAST_RX | \
ICE_PROMISC_VLAN_TX | \
ICE_PROMISC_VLAN_RX)
} else if (status == ICE_ERR_DOES_NOT_EXIST) {
dev_dbg(ice_pf_to_dev(vsi->back), "LAN Tx queues do not exist, nothing to disable\n");
} else if (status) {
- dev_err(ice_pf_to_dev(vsi->back), "Failed to disable LAN Tx queues, error: %s\n",
+ dev_dbg(ice_pf_to_dev(vsi->back), "Failed to disable LAN Tx queues, error: %s\n",
ice_stat_str(status));
return -ENODEV;
}
/* Avoid wait time by stopping all VFs at the same time */
ice_for_each_vf(pf, i)
- if (test_bit(ICE_VF_STATE_QS_ENA, pf->vf[i].vf_states))
- ice_dis_vf_qs(&pf->vf[i]);
+ ice_dis_vf_qs(&pf->vf[i]);
tmp = pf->num_alloc_vfs;
pf->num_qps_per_vf = 0;
set_bit(ICE_VF_STATE_DIS, pf->vf[i].vf_states);
ice_free_vf_res(&pf->vf[i]);
}
+
+ mutex_destroy(&pf->vf[i].cfg_lock);
}
if (ice_sriov_free_msix_res(pf))
vsi = ice_get_vf_vsi(vf);
- if (test_bit(ICE_VF_STATE_QS_ENA, vf->vf_states))
- ice_dis_vf_qs(vf);
+ ice_dis_vf_qs(vf);
/* Call Disable LAN Tx queue AQ whether or not queues are
* enabled. This is needed for successful completion of VFR.
ice_vf_fdir_init(vf);
ice_vc_set_dflt_vf_ops(&vf->vc_ops);
+
+ mutex_init(&vf->cfg_lock);
}
}
static int ice_vc_cfg_promiscuous_mode_msg(struct ice_vf *vf, u8 *msg)
{
enum virtchnl_status_code v_ret = VIRTCHNL_STATUS_SUCCESS;
+ enum ice_status mcast_status = 0, ucast_status = 0;
bool rm_promisc, alluni = false, allmulti = false;
struct virtchnl_promisc_info *info =
(struct virtchnl_promisc_info *)msg;
rm_promisc = !allmulti && !alluni;
if (vsi->num_vlan || vf->port_vlan_info) {
- struct ice_vsi *pf_vsi = ice_get_main_vsi(pf);
- struct net_device *pf_netdev;
-
- if (!pf_vsi) {
- v_ret = VIRTCHNL_STATUS_ERR_PARAM;
- goto error_param;
- }
-
- pf_netdev = pf_vsi->netdev;
-
- ret = ice_set_vf_spoofchk(pf_netdev, vf->vf_id, rm_promisc);
- if (ret) {
- dev_err(dev, "Failed to update spoofchk to %s for VF %d VSI %d when setting promiscuous mode\n",
- rm_promisc ? "ON" : "OFF", vf->vf_id,
- vsi->vsi_num);
- v_ret = VIRTCHNL_STATUS_ERR_PARAM;
- }
-
if (rm_promisc)
ret = ice_cfg_vlan_pruning(vsi, true);
else
goto error_param;
}
} else {
- enum ice_status status;
- u8 promisc_m;
-
- if (alluni) {
- if (vf->port_vlan_info || vsi->num_vlan)
- promisc_m = ICE_UCAST_VLAN_PROMISC_BITS;
- else
- promisc_m = ICE_UCAST_PROMISC_BITS;
- } else if (allmulti) {
- if (vf->port_vlan_info || vsi->num_vlan)
- promisc_m = ICE_MCAST_VLAN_PROMISC_BITS;
- else
- promisc_m = ICE_MCAST_PROMISC_BITS;
+ u8 mcast_m, ucast_m;
+
+ if (vf->port_vlan_info || vsi->num_vlan > 1) {
+ mcast_m = ICE_MCAST_VLAN_PROMISC_BITS;
+ ucast_m = ICE_UCAST_VLAN_PROMISC_BITS;
} else {
- if (vf->port_vlan_info || vsi->num_vlan)
- promisc_m = ICE_UCAST_VLAN_PROMISC_BITS;
- else
- promisc_m = ICE_UCAST_PROMISC_BITS;
+ mcast_m = ICE_MCAST_PROMISC_BITS;
+ ucast_m = ICE_UCAST_PROMISC_BITS;
}
- /* Configure multicast/unicast with or without VLAN promiscuous
- * mode
- */
- status = ice_vf_set_vsi_promisc(vf, vsi, promisc_m, rm_promisc);
- if (status) {
- dev_err(dev, "%sable Tx/Rx filter promiscuous mode on VF-%d failed, error: %s\n",
- rm_promisc ? "dis" : "en", vf->vf_id,
- ice_stat_str(status));
- v_ret = ice_err_to_virt_err(status);
- goto error_param;
- } else {
- dev_dbg(dev, "%sable Tx/Rx filter promiscuous mode on VF-%d succeeded\n",
- rm_promisc ? "dis" : "en", vf->vf_id);
+ ucast_status = ice_vf_set_vsi_promisc(vf, vsi, ucast_m,
+ !alluni);
+ if (ucast_status) {
+ dev_err(dev, "%sable Tx/Rx filter promiscuous mode on VF-%d failed\n",
+ alluni ? "en" : "dis", vf->vf_id);
+ v_ret = ice_err_to_virt_err(ucast_status);
+ }
+
+ mcast_status = ice_vf_set_vsi_promisc(vf, vsi, mcast_m,
+ !allmulti);
+ if (mcast_status) {
+ dev_err(dev, "%sable Tx/Rx filter promiscuous mode on VF-%d failed\n",
+ allmulti ? "en" : "dis", vf->vf_id);
+ v_ret = ice_err_to_virt_err(mcast_status);
}
}
- if (allmulti &&
- !test_and_set_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states))
- dev_info(dev, "VF %u successfully set multicast promiscuous mode\n", vf->vf_id);
- else if (!allmulti && test_and_clear_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states))
- dev_info(dev, "VF %u successfully unset multicast promiscuous mode\n", vf->vf_id);
+ if (!mcast_status) {
+ if (allmulti &&
+ !test_and_set_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states))
+ dev_info(dev, "VF %u successfully set multicast promiscuous mode\n",
+ vf->vf_id);
+ else if (!allmulti && test_and_clear_bit(ICE_VF_STATE_MC_PROMISC, vf->vf_states))
+ dev_info(dev, "VF %u successfully unset multicast promiscuous mode\n",
+ vf->vf_id);
+ }
- if (alluni && !test_and_set_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states))
- dev_info(dev, "VF %u successfully set unicast promiscuous mode\n", vf->vf_id);
- else if (!alluni && test_and_clear_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states))
- dev_info(dev, "VF %u successfully unset unicast promiscuous mode\n", vf->vf_id);
+ if (!ucast_status) {
+ if (alluni && !test_and_set_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states))
+ dev_info(dev, "VF %u successfully set unicast promiscuous mode\n",
+ vf->vf_id);
+ else if (!alluni && test_and_clear_bit(ICE_VF_STATE_UC_PROMISC, vf->vf_states))
+ dev_info(dev, "VF %u successfully unset unicast promiscuous mode\n",
+ vf->vf_id);
+ }
error_param:
return ice_vc_send_msg_to_vf(vf, VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE,
struct device *dev = ice_pf_to_dev(vf->pf);
u8 *mac_addr = vc_ether_addr->addr;
enum ice_status status;
+ int ret = 0;
/* device MAC already added */
if (ether_addr_equal(mac_addr, vf->dev_lan_addr.addr))
status = ice_fltr_add_mac(vsi, mac_addr, ICE_FWD_TO_VSI);
if (status == ICE_ERR_ALREADY_EXISTS) {
- dev_err(dev, "MAC %pM already exists for VF %d\n", mac_addr,
+ dev_dbg(dev, "MAC %pM already exists for VF %d\n", mac_addr,
vf->vf_id);
- return -EEXIST;
+ /* don't return since we might need to update
+ * the primary MAC in ice_vfhw_mac_add() below
+ */
+ ret = -EEXIST;
} else if (status) {
dev_err(dev, "Failed to add MAC %pM for VF %d\n, error %s\n",
mac_addr, vf->vf_id, ice_stat_str(status));
return -EIO;
+ } else {
+ vf->num_mac++;
}
ice_vfhw_mac_add(vf, vc_ether_addr);
- vf->num_mac++;
-
- return 0;
+ return ret;
}
/**
return 0;
}
+ mutex_lock(&vf->cfg_lock);
+
vf->port_vlan_info = vlanprio;
if (vf->port_vlan_info)
dev_info(dev, "Clearing port VLAN on VF %d\n", vf_id);
ice_vc_reset_vf(vf);
+ mutex_unlock(&vf->cfg_lock);
return 0;
}
return;
}
+ /* VF is being configured in another context that triggers a VFR, so no
+ * need to process this message
+ */
+ if (!mutex_trylock(&vf->cfg_lock)) {
+ dev_info(dev, "VF %u is being configured in another context that will trigger a VFR, so there is no need to handle this message\n",
+ vf->vf_id);
+ return;
+ }
+
switch (v_opcode) {
case VIRTCHNL_OP_VERSION:
err = ops->get_ver_msg(vf, msg);
dev_info(dev, "PF failed to honor VF %d, opcode %d, error %d\n",
vf_id, v_opcode, err);
}
+
+ mutex_unlock(&vf->cfg_lock);
}
/**
return -EINVAL;
}
+ mutex_lock(&vf->cfg_lock);
+
/* VF is notified of its new MAC via the PF's response to the
* VIRTCHNL_OP_GET_VF_RESOURCES message after the VF has been reset
*/
}
ice_vc_reset_vf(vf);
+ mutex_unlock(&vf->cfg_lock);
return 0;
}
if (trusted == vf->trusted)
return 0;
+ mutex_lock(&vf->cfg_lock);
+
vf->trusted = trusted;
ice_vc_reset_vf(vf);
dev_info(ice_pf_to_dev(pf), "VF %u is now %strusted\n",
vf_id, trusted ? "" : "un");
+ mutex_unlock(&vf->cfg_lock);
+
return 0;
}
struct ice_vf {
struct ice_pf *pf;
+ /* Used during virtchnl message handling and NDO ops against the VF
+ * that will trigger a VFR
+ */
+ struct mutex cfg_lock;
+
u16 vf_id; /* VF ID in the PF space */
u16 lan_vsi_idx; /* index into PF struct */
u16 ctrl_vsi_idx;
#include <linux/io.h>
#include <linux/dma-mapping.h>
#include <linux/module.h>
+#include <linux/property.h>
#include <asm/checksum.h>
{
struct ltq_etop_priv *priv = netdev_priv(dev);
int i;
+ int err;
ltq_pmu_enable(PMU_PPE);
/* enable crc generation */
ltq_etop_w32(PPE32_CGEN, LQ_PPE32_ENET_MAC_CFG);
- ltq_dma_init_port(DMA_PORT_ETOP, priv->tx_burst_len, rx_burst_len);
+ ltq_dma_init_port(DMA_PORT_ETOP, priv->tx_burst_len, priv->rx_burst_len);
for (i = 0; i < MAX_DMA_CHAN; i++) {
int irq = LTQ_DMA_CH0_INT + i;
if (IS_TX(i)) {
ltq_dma_alloc_tx(&ch->dma);
- request_irq(irq, ltq_etop_dma_irq, 0, "etop_tx", priv);
+ err = request_irq(irq, ltq_etop_dma_irq, 0, "etop_tx", priv);
+ if (err) {
+ netdev_err(dev,
+ "Unable to get Tx DMA IRQ %d\n",
+ irq);
+ return err;
+ }
} else if (IS_RX(i)) {
ltq_dma_alloc_rx(&ch->dma);
for (ch->dma.desc = 0; ch->dma.desc < LTQ_DESC_NUM;
if (ltq_etop_alloc_skb(ch))
return -ENOMEM;
ch->dma.desc = 0;
- request_irq(irq, ltq_etop_dma_irq, 0, "etop_rx", priv);
+ err = request_irq(irq, ltq_etop_dma_irq, 0, "etop_rx", priv);
+ if (err) {
+ netdev_err(dev,
+ "Unable to get Rx DMA IRQ %d\n",
+ irq);
+ return err;
+ }
}
ch->dma.irq = irq;
}
},
};
-int __init
+static int __init
init_ltq_etop(void)
{
int ret = platform_driver_probe(<q_mii_driver, ltq_etop_probe);
priv->dev = &pdev->dev;
irq = platform_get_irq(pdev, 0);
- if (irq < 0) {
- dev_err(&pdev->dev, "Failed to get IRQ %d\n", irq);
+ if (irq < 0)
return irq;
- }
netdev->irq = irq;
priv->base = devm_platform_ioremap_resource_byname(pdev, "mac");
struct net_device *netdev = platform_get_drvdata(pdev);
unregister_netdev(netdev);
- free_netdev(netdev);
return 0;
}
};
MODULE_DEVICE_TABLE(of, orion_mdio_match);
+#ifdef CONFIG_ACPI
static const struct acpi_device_id orion_mdio_acpi_match[] = {
{ "MRVL0100", BUS_TYPE_SMI },
{ "MRVL0101", BUS_TYPE_XSMI },
{ },
};
MODULE_DEVICE_TABLE(acpi, orion_mdio_acpi_match);
+#endif
static struct platform_driver orion_mdio_driver = {
.probe = orion_mdio_probe,
mvpp22_gop_fca_enable_periodic(port, true);
}
-static int mvpp22_gop_init(struct mvpp2_port *port)
+static int mvpp22_gop_init(struct mvpp2_port *port, phy_interface_t interface)
{
struct mvpp2 *priv = port->priv;
u32 val;
if (!priv->sysctrl_base)
return 0;
- switch (port->phy_interface) {
+ switch (interface) {
case PHY_INTERFACE_MODE_RGMII:
case PHY_INTERFACE_MODE_RGMII_ID:
case PHY_INTERFACE_MODE_RGMII_RXID:
* lanes by the physical layer. This is why configurations like
* "PPv2 (2500BaseX) - COMPHY (2500SGMII)" are valid.
*/
-static int mvpp22_comphy_init(struct mvpp2_port *port)
+static int mvpp22_comphy_init(struct mvpp2_port *port,
+ phy_interface_t interface)
{
int ret;
if (!port->comphy)
return 0;
- ret = phy_set_mode_ext(port->comphy, PHY_MODE_ETHERNET,
- port->phy_interface);
+ ret = phy_set_mode_ext(port->comphy, PHY_MODE_ETHERNET, interface);
if (ret)
return ret;
writel(val & ~MVPP22_XPCS_CFG0_RESET_DIS, xpcs + MVPP22_XPCS_CFG0);
}
-static void mvpp22_pcs_reset_deassert(struct mvpp2_port *port)
+static void mvpp22_pcs_reset_deassert(struct mvpp2_port *port,
+ phy_interface_t interface)
{
struct mvpp2 *priv = port->priv;
void __iomem *mpcs, *xpcs;
mpcs = priv->iface_base + MVPP22_MPCS_BASE(port->gop_id);
xpcs = priv->iface_base + MVPP22_XPCS_BASE(port->gop_id);
- switch (port->phy_interface) {
+ switch (interface) {
case PHY_INTERFACE_MODE_10GBASER:
val = readl(mpcs + MVPP22_MPCS_CLK_RESET);
val |= MAC_CLK_RESET_MAC | MAC_CLK_RESET_SD_RX |
return rx_done;
}
-static void mvpp22_mode_reconfigure(struct mvpp2_port *port)
+static void mvpp22_mode_reconfigure(struct mvpp2_port *port,
+ phy_interface_t interface)
{
u32 ctrl3;
mvpp22_pcs_reset_assert(port);
/* comphy reconfiguration */
- mvpp22_comphy_init(port);
+ mvpp22_comphy_init(port, interface);
/* gop reconfiguration */
- mvpp22_gop_init(port);
+ mvpp22_gop_init(port, interface);
- mvpp22_pcs_reset_deassert(port);
+ mvpp22_pcs_reset_deassert(port, interface);
if (mvpp2_port_supports_xlg(port)) {
ctrl3 = readl(port->base + MVPP22_XLG_CTRL3_REG);
ctrl3 &= ~MVPP22_XLG_CTRL3_MACMODESELECT_MASK;
- if (mvpp2_is_xlg(port->phy_interface))
+ if (mvpp2_is_xlg(interface))
ctrl3 |= MVPP22_XLG_CTRL3_MACMODESELECT_10G;
else
ctrl3 |= MVPP22_XLG_CTRL3_MACMODESELECT_GMAC;
writel(ctrl3, port->base + MVPP22_XLG_CTRL3_REG);
}
- if (mvpp2_port_supports_xlg(port) && mvpp2_is_xlg(port->phy_interface))
+ if (mvpp2_port_supports_xlg(port) && mvpp2_is_xlg(interface))
mvpp2_xlg_max_rx_size_set(port);
else
mvpp2_gmac_max_rx_size_set(port);
mvpp2_interrupts_enable(port);
if (port->priv->hw_version >= MVPP22)
- mvpp22_mode_reconfigure(port);
+ mvpp22_mode_reconfigure(port, port->phy_interface);
if (port->phylink) {
phylink_start(port->phylink);
mvpp22_gop_mask_irq(port);
phy_power_off(port->comphy);
+
+ /* Reconfigure the serdes lanes */
+ mvpp22_mode_reconfigure(port, interface);
}
}
port->phy_interface != interface) {
port->phy_interface = interface;
- /* Reconfigure the serdes lanes */
- mvpp22_mode_reconfigure(port);
-
/* Unmask interrupts */
mvpp22_gop_unmask_irq(port);
}
* driver does this, we can remove this code.
*/
if (port->comphy) {
- err = mvpp22_comphy_init(port);
+ err = mvpp22_comphy_init(port, port->phy_interface);
if (err == 0)
phy_power_off(port->comphy);
}
config OCTEONTX2_PF
tristate "Marvell OcteonTX2 NIC Physical Function driver"
select OCTEONTX2_MBOX
+ select NET_DEVLINK
depends on (64BIT && COMPILE_TEST) || ARM64
depends on PCI
depends on PTP_1588_CLOCK_OPTIONAL
u64 lmt_addr, val, tbl_base;
int pf, vf, num_vfs, hw_vfs;
void __iomem *lmt_map_base;
- int index = 0, off = 0;
- int bytes_not_copied;
int buf_size = 10240;
+ size_t off = 0;
+ int index = 0;
char *buf;
+ int ret;
/* don't allow partial reads */
if (*ppos != 0)
}
off += scnprintf(&buf[off], buf_size - 1 - off, "\n");
- bytes_not_copied = copy_to_user(buffer, buf, off);
+ ret = min(off, count);
+ if (copy_to_user(buffer, buf, ret))
+ ret = -EFAULT;
kfree(buf);
iounmap(lmt_map_base);
- if (bytes_not_copied)
- return -EFAULT;
+ if (ret < 0)
+ return ret;
- *ppos = off;
- return off;
+ *ppos = ret;
+ return ret;
}
RVU_DEBUG_FOPS(lmtst_map_table, lmtst_map_table_display, NULL);
bmap = mcam->bmap_reverse;
start = mcam->bmap_entries - start;
end = mcam->bmap_entries - end;
- index = start;
- start = end;
- end = index;
+ swap(start, end);
} else {
bmap = mcam->bmap;
}
.ndo_set_features = otx2vf_set_features,
.ndo_get_stats64 = otx2_get_stats64,
.ndo_tx_timeout = otx2_tx_timeout,
- .ndo_do_ioctl = otx2_ioctl,
+ .ndo_eth_ioctl = otx2_ioctl,
};
static int otx2_wq_init(struct otx2_nic *vf)
{
struct prestera_port_phy_state *state = &port->state_phy;
- if (prestera_hw_port_phy_mode_get(port, &state->mdix, NULL, NULL, NULL)) {
+ if (prestera_hw_port_phy_mode_get(port,
+ &state->mdix, NULL, NULL, NULL)) {
netdev_warn(port->dev, "MDIX params get failed");
state->mdix = ETH_TP_MDI_INVALID;
}
struct prestera_msg_ret ret;
};
-union prestera_msg_switch_param {
- u8 mac[ETH_ALEN];
- __le32 ageing_timeout_ms;
-} __packed;
-
struct prestera_msg_switch_attr_req {
struct prestera_msg_cmd cmd;
__le32 attr;
- union prestera_msg_switch_param param;
+ union {
+ __le32 ageing_timeout_ms;
+ struct {
+ u8 mac[ETH_ALEN];
+ u8 __pad[2];
+ };
+ } param;
};
struct prestera_msg_switch_init_resp {
struct prestera_msg_ret ret;
__le32 port_count;
__le32 mtu_max;
- u8 switch_id;
- u8 lag_max;
- u8 lag_member_max;
__le32 size_tbl_router_nexthop;
-} __packed __aligned(4);
+ u8 switch_id;
+ u8 lag_max;
+ u8 lag_member_max;
+};
struct prestera_msg_event_port_param {
union {
struct {
- u8 oper;
__le32 mode;
__le32 speed;
+ u8 oper;
u8 duplex;
u8 fc;
u8 fec;
- } __packed mac;
+ } mac;
struct {
- u8 mdix;
__le64 lmode_bmap;
+ u8 mdix;
u8 fc;
- } __packed phy;
- } __packed;
-} __packed __aligned(4);
+ u8 __pad[2];
+ } __packed phy; /* make sure always 12 bytes size */
+ };
+};
struct prestera_msg_port_cap_param {
__le64 link_mode;
- u8 type;
- u8 fec;
- u8 fc;
- u8 transceiver;
+ u8 type;
+ u8 fec;
+ u8 fc;
+ u8 transceiver;
};
struct prestera_msg_port_flood_param {
u8 type;
u8 enable;
+ u8 __pad[2];
};
union prestera_msg_port_param {
+ __le32 mtu;
+ __le32 speed;
+ __le32 link_mode;
u8 admin_state;
u8 oper_state;
- __le32 mtu;
u8 mac[ETH_ALEN];
u8 accept_frm_type;
- __le32 speed;
u8 learning;
u8 flood;
- __le32 link_mode;
u8 type;
u8 duplex;
u8 fec;
u8 fc;
-
union {
struct {
- u8 admin:1;
+ u8 admin;
u8 fc;
u8 ap_enable;
+ u8 __reserved[5];
union {
struct {
__le32 mode;
- u8 inband:1;
__le32 speed;
- u8 duplex;
- u8 fec;
- u8 fec_supp;
- } __packed reg_mode;
+ u8 inband;
+ u8 duplex;
+ u8 fec;
+ u8 fec_supp;
+ } reg_mode;
struct {
__le32 mode;
__le32 speed;
- u8 fec;
- u8 fec_supp;
- } __packed ap_modes[PRESTERA_AP_PORT_MAX];
- } __packed;
- } __packed mac;
+ u8 fec;
+ u8 fec_supp;
+ u8 __pad[2];
+ } ap_modes[PRESTERA_AP_PORT_MAX];
+ };
+ } mac;
struct {
- u8 admin:1;
- u8 adv_enable;
__le64 modes;
__le32 mode;
+ u8 admin;
+ u8 adv_enable;
u8 mdix;
- } __packed phy;
- } __packed link;
+ u8 __pad;
+ } phy;
+ } link;
struct prestera_msg_port_cap_param cap;
struct prestera_msg_port_flood_param flood_ext;
struct prestera_msg_event_port_param link_evt;
-} __packed;
+};
struct prestera_msg_port_attr_req {
struct prestera_msg_cmd cmd;
__le32 port;
__le32 dev;
union prestera_msg_port_param param;
-} __packed __aligned(4);
-
+};
struct prestera_msg_port_attr_resp {
struct prestera_msg_ret ret;
union prestera_msg_port_param param;
-} __packed __aligned(4);
-
+};
struct prestera_msg_port_stats_resp {
struct prestera_msg_ret ret;
__le32 hw_id;
__le32 dev_id;
__le16 fp_id;
+ u8 pad[2];
};
struct prestera_msg_vlan_req {
__le32 port;
__le32 dev;
__le16 vid;
- u8 is_member;
- u8 is_tagged;
+ u8 is_member;
+ u8 is_tagged;
};
struct prestera_msg_fdb_req {
struct prestera_msg_cmd cmd;
- u8 dest_type;
+ __le32 flush_mode;
union {
struct {
__le32 port;
};
__le16 lag_id;
} dest;
- u8 mac[ETH_ALEN];
__le16 vid;
- u8 dynamic;
- __le32 flush_mode;
-} __packed __aligned(4);
+ u8 dest_type;
+ u8 dynamic;
+ u8 mac[ETH_ALEN];
+ u8 __pad[2];
+};
struct prestera_msg_bridge_req {
struct prestera_msg_cmd cmd;
__le32 port;
__le32 dev;
__le16 bridge;
+ u8 pad[2];
};
struct prestera_msg_bridge_resp {
struct prestera_msg_ret ret;
__le16 bridge;
+ u8 pad[2];
};
struct prestera_msg_acl_action {
struct prestera_msg_acl_match {
__le32 type;
+ __le32 __reserved;
union {
struct {
u8 key;
u8 mask;
- } __packed u8;
+ } u8;
struct {
__le16 key;
__le16 mask;
struct {
u8 key[ETH_ALEN];
u8 mask[ETH_ALEN];
- } __packed mac;
+ } mac;
} keymask;
};
__le32 port;
__le32 dev;
__le16 ruleset_id;
+ u8 pad[2];
};
struct prestera_msg_acl_ruleset_req {
struct prestera_msg_cmd cmd;
__le16 id;
+ u8 pad[2];
};
struct prestera_msg_acl_ruleset_resp {
struct prestera_msg_ret ret;
__le16 id;
+ u8 pad[2];
};
struct prestera_msg_span_req {
__le32 port;
__le32 dev;
u8 id;
+ u8 pad[3];
};
struct prestera_msg_span_resp {
struct prestera_msg_ret ret;
u8 id;
+ u8 pad[3];
};
struct prestera_msg_stp_req {
__le32 port;
__le32 dev;
__le16 vid;
- u8 state;
+ u8 state;
+ u8 __pad;
};
struct prestera_msg_rxtx_req {
struct prestera_msg_cmd cmd;
u8 use_sdma;
+ u8 pad[3];
};
struct prestera_msg_rxtx_resp {
__le32 port;
__le32 dev;
__le16 lag_id;
+ u8 pad[2];
};
struct prestera_msg_cpu_code_counter_req {
struct prestera_msg_cmd cmd;
u8 counter_type;
u8 code;
+ u8 pad[2];
};
struct mvsw_msg_cpu_code_counter_ret {
struct prestera_msg_event_fdb {
struct prestera_msg_event id;
- u8 dest_type;
+ __le32 vid;
union {
__le32 port_id;
__le16 lag_id;
} dest;
- __le32 vid;
union prestera_msg_event_fdb_param param;
-} __packed __aligned(4);
+ u8 dest_type;
+};
-static inline void prestera_hw_build_tests(void)
+static void prestera_hw_build_tests(void)
{
/* check requests */
BUILD_BUG_ON(sizeof(struct prestera_msg_common_req) != 4);
BUILD_BUG_ON(sizeof(struct prestera_msg_switch_attr_req) != 16);
- BUILD_BUG_ON(sizeof(struct prestera_msg_port_attr_req) != 120);
+ BUILD_BUG_ON(sizeof(struct prestera_msg_port_attr_req) != 144);
BUILD_BUG_ON(sizeof(struct prestera_msg_port_info_req) != 8);
BUILD_BUG_ON(sizeof(struct prestera_msg_vlan_req) != 16);
BUILD_BUG_ON(sizeof(struct prestera_msg_fdb_req) != 28);
/* check responses */
BUILD_BUG_ON(sizeof(struct prestera_msg_common_resp) != 8);
BUILD_BUG_ON(sizeof(struct prestera_msg_switch_init_resp) != 24);
- BUILD_BUG_ON(sizeof(struct prestera_msg_port_attr_resp) != 112);
+ BUILD_BUG_ON(sizeof(struct prestera_msg_port_attr_resp) != 136);
BUILD_BUG_ON(sizeof(struct prestera_msg_port_stats_resp) != 248);
BUILD_BUG_ON(sizeof(struct prestera_msg_port_info_resp) != 20);
BUILD_BUG_ON(sizeof(struct prestera_msg_bridge_resp) != 12);
if (err)
return err;
- if (__le32_to_cpu(ret->cmd.type) != PRESTERA_CMD_TYPE_ACK)
+ if (ret->cmd.type != __cpu_to_le32(PRESTERA_CMD_TYPE_ACK))
return -EBADE;
- if (__le32_to_cpu(ret->status) != PRESTERA_CMD_ACK_OK)
+ if (ret->status != __cpu_to_le32(PRESTERA_CMD_ACK_OK))
return -EINVAL;
return 0;
int prestera_hw_port_autoneg_restart(struct prestera_port *port)
{
struct prestera_msg_port_attr_req req = {
- .attr = __cpu_to_le32(PRESTERA_CMD_PORT_ATTR_PHY_AUTONEG_RESTART),
+ .attr =
+ __cpu_to_le32(PRESTERA_CMD_PORT_ATTR_PHY_AUTONEG_RESTART),
.port = __cpu_to_le32(port->hw_id),
.dev = __cpu_to_le32(port->dev_id),
};
err = prestera_port_cfg_mac_write(port, &cfg_mac);
if (err) {
- dev_err(prestera_dev(sw), "Failed to set port(%u) mac mode\n", id);
+ dev_err(prestera_dev(sw),
+ "Failed to set port(%u) mac mode\n", id);
goto err_port_init;
}
false, 0, 0,
port->cfg_phy.mdix);
if (err) {
- dev_err(prestera_dev(sw), "Failed to set port(%u) phy mode\n", id);
+ dev_err(prestera_dev(sw),
+ "Failed to set port(%u) phy mode\n", id);
goto err_port_init;
}
}
goto cmd_exit;
}
- memcpy_fromio(out_msg, prestera_fw_cmdq_buf(fw, qid) + in_size, ret_size);
+ memcpy_fromio(out_msg,
+ prestera_fw_cmdq_buf(fw, qid) + in_size, ret_size);
cmd_exit:
prestera_fw_write(fw, PRESTERA_CMDQ_REQ_CTL_REG(qid),
static int prestera_pci_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
{
- const char *driver_name = pdev->driver->name;
+ const char *driver_name = dev_driver_string(&pdev->dev);
struct prestera_fw *fw;
int err;
case MLX5_CMD_OP_PAGE_FAULT_RESUME:
case MLX5_CMD_OP_QUERY_ESW_FUNCTIONS:
case MLX5_CMD_OP_DEALLOC_SF:
+ case MLX5_CMD_OP_DESTROY_UCTX:
+ case MLX5_CMD_OP_DESTROY_UMEM:
return MLX5_CMD_STAT_OK;
case MLX5_CMD_OP_QUERY_HCA_CAP:
case MLX5_CMD_OP_MODIFY_GENERAL_OBJECT:
case MLX5_CMD_OP_QUERY_GENERAL_OBJECT:
case MLX5_CMD_OP_CREATE_UCTX:
- case MLX5_CMD_OP_DESTROY_UCTX:
case MLX5_CMD_OP_CREATE_UMEM:
- case MLX5_CMD_OP_DESTROY_UMEM:
case MLX5_CMD_OP_ALLOC_MEMIC:
case MLX5_CMD_OP_MODIFY_XRQ:
case MLX5_CMD_OP_RELEASE_XRQ_ERROR:
MLX5_SET(destroy_cq_in, in, cqn, cq->cqn);
MLX5_SET(destroy_cq_in, in, uid, cq->uid);
err = mlx5_cmd_exec_in(dev, destroy_cq, in);
+ if (err)
+ return err;
synchronize_irq(cq->irqn);
-
mlx5_cq_put(cq);
wait_for_completion(&cq->free);
- return err;
+ return 0;
}
EXPORT_SYMBOL(mlx5_core_destroy_cq);
if (!mlx5_debugfs_root)
return;
- if (cq->dbg)
+ if (cq->dbg) {
rem_res_tree(cq->dbg);
+ cq->dbg = NULL;
+ }
}
int
mlx5_tc_ct_parse_action(struct mlx5_tc_ct_priv *priv,
struct mlx5_flow_attr *attr,
+ struct mlx5e_tc_mod_hdr_acts *mod_acts,
const struct flow_action_entry *act,
struct netlink_ext_ack *extack)
{
+ bool clear_action = act->ct.action & TCA_CT_ACT_CLEAR;
+ int err;
+
if (!priv) {
NL_SET_ERR_MSG_MOD(extack,
"offload of ct action isn't available");
attr->ct_attr.ct_action = act->ct.action;
attr->ct_attr.nf_ft = act->ct.flow_table;
+ if (!clear_action)
+ goto out;
+
+ err = mlx5_tc_ct_entry_set_registers(priv, mod_acts, 0, 0, 0, 0);
+ if (err) {
+ NL_SET_ERR_MSG_MOD(extack, "Failed to set registers for ct clear");
+ return err;
+ }
+ attr->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
+
+out:
return 0;
}
memcpy(pre_ct_attr, attr, attr_sz);
- err = mlx5_tc_ct_entry_set_registers(ct_priv, mod_acts, 0, 0, 0, 0);
- if (err) {
- ct_dbg("Failed to set register for ct clear");
- goto err_set_registers;
- }
-
mod_hdr = mlx5_modify_header_alloc(priv->mdev, ct_priv->ns_type,
mod_acts->num_actions,
mod_acts->actions);
if (IS_ERR(mod_hdr)) {
err = PTR_ERR(mod_hdr);
ct_dbg("Failed to add create ct clear mod hdr");
- goto err_set_registers;
+ goto err_mod_hdr;
}
pre_ct_attr->modify_hdr = mod_hdr;
- pre_ct_attr->action |= MLX5_FLOW_CONTEXT_ACTION_MOD_HDR;
rule = mlx5_tc_rule_insert(priv, orig_spec, pre_ct_attr);
if (IS_ERR(rule)) {
err_insert:
mlx5_modify_header_dealloc(priv->mdev, mod_hdr);
-err_set_registers:
+err_mod_hdr:
netdev_warn(priv->netdev,
"Failed to offload ct clear flow, err %d\n", err);
kfree(pre_ct_attr);
int
mlx5_tc_ct_parse_action(struct mlx5_tc_ct_priv *priv,
struct mlx5_flow_attr *attr,
+ struct mlx5e_tc_mod_hdr_acts *mod_acts,
const struct flow_action_entry *act,
struct netlink_ext_ack *extack);
static inline int
mlx5_tc_ct_parse_action(struct mlx5_tc_ct_priv *priv,
struct mlx5_flow_attr *attr,
+ struct mlx5e_tc_mod_hdr_acts *mod_acts,
const struct flow_action_entry *act,
struct netlink_ext_ack *extack)
{
refcount_t refcnt;
struct rcu_head rcu_head;
struct completion init_done;
+ struct completion del_hw_done;
int tunnel_id; /* the mapped tunnel id of this flow */
struct mlx5_flow_attr *attr;
};
struct list_head *flow_list,
int index)
{
- if (IS_ERR(mlx5e_flow_get(flow)))
+ if (IS_ERR(mlx5e_flow_get(flow))) {
+ /* Flow is being deleted concurrently. Wait for it to be
+ * unoffloaded from hardware, otherwise deleting encap will
+ * fail.
+ */
+ wait_for_completion(&flow->del_hw_done);
return;
+ }
wait_for_completion(&flow->init_done);
flow->tmp_entry_index = index;
DECLARE_BITMAP(flags, MLX5E_NUM_PRIV_RX_FLAGS);
/* resync */
+ spinlock_t lock; /* protects resync fields */
struct mlx5e_ktls_rx_resync_ctx resync;
struct list_head list;
};
struct mlx5e_icosq *sq;
bool trigger_poll;
- memcpy(info->rec_seq, &priv_rx->resync.sw_rcd_sn_be, sizeof(info->rec_seq));
-
sq = &c->async_icosq;
ktls_resync = sq->ktls_resync;
+ trigger_poll = false;
spin_lock_bh(&ktls_resync->lock);
- list_add_tail(&priv_rx->list, &ktls_resync->list);
- trigger_poll = !test_and_set_bit(MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC, &sq->state);
+ spin_lock_bh(&priv_rx->lock);
+ memcpy(info->rec_seq, &priv_rx->resync.sw_rcd_sn_be, sizeof(info->rec_seq));
+ if (list_empty(&priv_rx->list)) {
+ list_add_tail(&priv_rx->list, &ktls_resync->list);
+ trigger_poll = !test_and_set_bit(MLX5E_SQ_STATE_PENDING_TLS_RX_RESYNC, &sq->state);
+ }
+ spin_unlock_bh(&priv_rx->lock);
spin_unlock_bh(&ktls_resync->lock);
if (!trigger_poll)
if (err)
goto err_create_key;
+ INIT_LIST_HEAD(&priv_rx->list);
+ spin_lock_init(&priv_rx->lock);
priv_rx->crypto_info =
*(struct tls12_crypto_info_aes_gcm_128 *)crypto_info;
priv_rx = list_first_entry(&local_list,
struct mlx5e_ktls_offload_context_rx,
list);
+ spin_lock(&priv_rx->lock);
cseg = post_static_params(sq, priv_rx);
- if (IS_ERR(cseg))
+ if (IS_ERR(cseg)) {
+ spin_unlock(&priv_rx->lock);
break;
- list_del(&priv_rx->list);
+ }
+ list_del_init(&priv_rx->list);
+ spin_unlock(&priv_rx->lock);
db_cseg = cseg;
}
if (db_cseg)
else
mlx5e_tc_unoffload_fdb_rules(esw, flow, attr);
}
+ complete_all(&flow->del_hw_done);
if (mlx5_flow_has_geneve_opt(flow))
mlx5_geneve_tlv_option_del(priv->mdev->geneve);
attr->dest_chain = act->chain_index;
break;
case FLOW_ACTION_CT:
- err = mlx5_tc_ct_parse_action(get_ct_priv(priv), attr, act, extack);
+ err = mlx5_tc_ct_parse_action(get_ct_priv(priv), attr,
+ &parse_attr->mod_hdr_acts,
+ act, extack);
if (err)
return err;
NL_SET_ERR_MSG_MOD(extack, "Sample action with connection tracking is not supported");
return -EOPNOTSUPP;
}
- err = mlx5_tc_ct_parse_action(get_ct_priv(priv), attr, act, extack);
+ err = mlx5_tc_ct_parse_action(get_ct_priv(priv), attr,
+ &parse_attr->mod_hdr_acts,
+ act, extack);
if (err)
return err;
INIT_LIST_HEAD(&flow->l3_to_l2_reformat);
refcount_set(&flow->refcnt, 1);
init_completion(&flow->init_done);
+ init_completion(&flow->del_hw_done);
*__flow = flow;
*__parse_attr = parse_attr;
*/
int mlx5_eswitch_enable(struct mlx5_eswitch *esw, int num_vfs)
{
+ bool toggle_lag;
int ret;
if (!mlx5_esw_allowed(esw))
return 0;
- mlx5_lag_disable_change(esw->dev);
+ toggle_lag = esw->mode == MLX5_ESWITCH_NONE;
+
+ if (toggle_lag)
+ mlx5_lag_disable_change(esw->dev);
+
down_write(&esw->mode_lock);
if (esw->mode == MLX5_ESWITCH_NONE) {
ret = mlx5_eswitch_enable_locked(esw, MLX5_ESWITCH_LEGACY, num_vfs);
esw->esw_funcs.num_vfs = num_vfs;
}
up_write(&esw->mode_lock);
- mlx5_lag_enable_change(esw->dev);
+
+ if (toggle_lag)
+ mlx5_lag_enable_change(esw->dev);
+
return ret;
}
esw->enabled_vports = 0;
esw->mode = MLX5_ESWITCH_NONE;
esw->offloads.inline_mode = MLX5_INLINE_MODE_NONE;
+ if (MLX5_CAP_ESW_FLOWTABLE_FDB(dev, reformat) &&
+ MLX5_CAP_ESW_FLOWTABLE_FDB(dev, decap))
+ esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_BASIC;
+ else
+ esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
dev->priv.eswitch = esw;
BLOCKING_INIT_NOTIFIER_HEAD(&esw->n_head);
return err;
}
-u8 mlx5_eswitch_mode(struct mlx5_core_dev *dev)
+u8 mlx5_eswitch_mode(const struct mlx5_core_dev *dev)
{
struct mlx5_eswitch *esw = dev->priv.eswitch;
struct mlx5_eswitch *esw;
esw = dev->priv.eswitch;
- return mlx5_esw_allowed(esw) ? esw->offloads.encap :
+ return (mlx5_eswitch_mode(dev) == MLX5_ESWITCH_OFFLOADS) ? esw->offloads.encap :
DEVLINK_ESWITCH_ENCAP_MODE_NONE;
}
EXPORT_SYMBOL(mlx5_eswitch_get_encap_mode);
u64 mapping_id;
int err;
- if (MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, reformat) &&
- MLX5_CAP_ESW_FLOWTABLE_FDB(esw->dev, decap))
- esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_BASIC;
- else
- esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
-
mutex_init(&esw->offloads.termtbl_mutex);
mlx5_rdma_enable_roce(esw->dev);
esw_offloads_metadata_uninit(esw);
mlx5_rdma_disable_roce(esw->dev);
mutex_destroy(&esw->offloads.termtbl_mutex);
- esw->offloads.encap = DEVLINK_ESWITCH_ENCAP_MODE_NONE;
}
static int esw_mode_from_devlink(u16 mode, u16 *mlx5_mode)
*encap = esw->offloads.encap;
unlock:
up_write(&esw->mode_lock);
- return 0;
+ return err;
}
static bool
#define MLX5_FC_STATS_PERIOD msecs_to_jiffies(1000)
/* Max number of counters to query in bulk read is 32K */
#define MLX5_SW_MAX_COUNTERS_BULK BIT(15)
-#define MLX5_SF_NUM_COUNTERS_BULK 6
+#define MLX5_SF_NUM_COUNTERS_BULK 8
#define MLX5_FC_POOL_MAX_THRESHOLD BIT(18)
#define MLX5_FC_POOL_USED_BUFF_RATIO 10
bool is_bonded, is_in_lag, mode_supported;
int bond_status = 0;
int num_slaves = 0;
+ int changed = 0;
int idx;
if (!netif_is_lag_master(upper))
*/
is_in_lag = num_slaves == MLX5_MAX_PORTS && bond_status == 0x3;
- if (!mlx5_lag_is_ready(ldev) && is_in_lag) {
- NL_SET_ERR_MSG_MOD(info->info.extack,
- "Can't activate LAG offload, PF is configured with more than 64 VFs");
- return 0;
- }
-
/* Lag mode must be activebackup or hash. */
mode_supported = tracker->tx_type == NETDEV_LAG_TX_TYPE_ACTIVEBACKUP ||
tracker->tx_type == NETDEV_LAG_TX_TYPE_HASH;
- if (is_in_lag && !mode_supported)
- NL_SET_ERR_MSG_MOD(info->info.extack,
- "Can't activate LAG offload, TX type isn't supported");
-
is_bonded = is_in_lag && mode_supported;
if (tracker->is_bonded != is_bonded) {
tracker->is_bonded = is_bonded;
- return 1;
+ changed = 1;
}
- return 0;
+ if (!is_in_lag)
+ return changed;
+
+ if (!mlx5_lag_is_ready(ldev))
+ NL_SET_ERR_MSG_MOD(info->info.extack,
+ "Can't activate LAG offload, PF is configured with more than 64 VFs");
+ else if (!mode_supported)
+ NL_SET_ERR_MSG_MOD(info->info.extack,
+ "Can't activate LAG offload, TX type isn't supported");
+
+ return changed;
}
static int mlx5_handle_changelowerstate_event(struct mlx5_lag *ldev,
ldev = container_of(this, struct mlx5_lag, nb);
- if (!mlx5_lag_is_ready(ldev) && event == NETDEV_CHANGELOWERSTATE)
- return NOTIFY_DONE;
-
tracker = ldev->tracker;
switch (event) {
lag_definer = kzalloc(sizeof(*lag_definer), GFP_KERNEL);
if (!lag_definer)
- return ERR_PTR(ENOMEM);
+ return ERR_PTR(-ENOMEM);
match_definer_mask = kvzalloc(MLX5_FLD_SZ_BYTES(match_definer,
match_mask),
static int dr_domain_query_vport(struct mlx5dr_domain *dmn,
u16 vport_number,
+ bool other_vport,
struct mlx5dr_cmd_vport_cap *vport_caps)
{
- u16 cmd_vport = vport_number;
- bool other_vport = true;
int ret;
- if (vport_number == MLX5_VPORT_UPLINK) {
- dr_domain_fill_uplink_caps(dmn, vport_caps);
- return 0;
- }
-
- if (dmn->info.caps.is_ecpf && vport_number == MLX5_VPORT_ECPF) {
- other_vport = false;
- cmd_vport = 0;
- }
-
ret = mlx5dr_cmd_query_esw_vport_context(dmn->mdev,
other_vport,
- cmd_vport,
+ vport_number,
&vport_caps->icm_address_rx,
&vport_caps->icm_address_tx);
if (ret)
ret = mlx5dr_cmd_query_gvmi(dmn->mdev,
other_vport,
- cmd_vport,
+ vport_number,
&vport_caps->vport_gvmi);
if (ret)
return ret;
{
return dr_domain_query_vport(dmn,
dmn->info.caps.is_ecpf ? MLX5_VPORT_ECPF : 0,
+ false,
&dmn->info.caps.vports.esw_manager_caps);
}
+static void dr_domain_query_uplink(struct mlx5dr_domain *dmn)
+{
+ dr_domain_fill_uplink_caps(dmn, &dmn->info.caps.vports.uplink_caps);
+}
+
static struct mlx5dr_cmd_vport_cap *
dr_domain_add_vport_cap(struct mlx5dr_domain *dmn, u16 vport)
{
if (!vport_caps)
return NULL;
- ret = dr_domain_query_vport(dmn, vport, vport_caps);
+ ret = dr_domain_query_vport(dmn, vport, true, vport_caps);
if (ret) {
kvfree(vport_caps);
return NULL;
return vport_caps;
}
+static bool dr_domain_is_esw_mgr_vport(struct mlx5dr_domain *dmn, u16 vport)
+{
+ struct mlx5dr_cmd_caps *caps = &dmn->info.caps;
+
+ return (caps->is_ecpf && vport == MLX5_VPORT_ECPF) ||
+ (!caps->is_ecpf && vport == 0);
+}
+
struct mlx5dr_cmd_vport_cap *
mlx5dr_domain_get_vport_cap(struct mlx5dr_domain *dmn, u16 vport)
{
struct mlx5dr_cmd_caps *caps = &dmn->info.caps;
struct mlx5dr_cmd_vport_cap *vport_caps;
- if ((caps->is_ecpf && vport == MLX5_VPORT_ECPF) ||
- (!caps->is_ecpf && vport == 0))
+ if (dr_domain_is_esw_mgr_vport(dmn, vport))
return &caps->vports.esw_manager_caps;
+ if (vport == MLX5_VPORT_UPLINK)
+ return &caps->vports.uplink_caps;
+
vport_load:
vport_caps = xa_load(&caps->vports.vports_caps_xa, vport);
if (vport_caps)
}
}
-static int dr_domain_query_uplink(struct mlx5dr_domain *dmn)
-{
- struct mlx5dr_cmd_vport_cap *vport_caps;
-
- vport_caps = mlx5dr_domain_get_vport_cap(dmn, MLX5_VPORT_UPLINK);
- if (!vport_caps)
- return -EINVAL;
-
- return 0;
-}
-
static int dr_domain_query_fdb_caps(struct mlx5_core_dev *mdev,
struct mlx5dr_domain *dmn)
{
goto free_vports_caps_xa;
}
- ret = dr_domain_query_uplink(dmn);
- if (ret) {
- mlx5dr_err(dmn, "Failed to query uplink vport caps (err: %d)", ret);
- goto free_vports_caps_xa;
- }
+ dr_domain_query_uplink(dmn);
return 0;
/* Check that all mask data was consumed */
for (i = 0; i < consumed_mask.match_sz; i++) {
- if (consumed_mask.match_buf[i]) {
- mlx5dr_dbg(dmn, "Match param mask contains unsupported parameters\n");
- ret = -EOPNOTSUPP;
- goto free_consumed_mask;
- }
+ if (!((u8 *)consumed_mask.match_buf)[i])
+ continue;
+
+ mlx5dr_dbg(dmn, "Match param mask contains unsupported parameters\n");
+ ret = -EOPNOTSUPP;
+ goto free_consumed_mask;
}
ret = 0;
struct mlx5dr_vports {
struct mlx5dr_cmd_vport_cap esw_manager_caps;
+ struct mlx5dr_cmd_vport_cap uplink_caps;
struct xarray vports_caps_xa;
};
obj-$(CONFIG_MLXBF_GIGE) += mlxbf_gige.o
mlxbf_gige-y := mlxbf_gige_ethtool.o \
- mlxbf_gige_gpio.o \
mlxbf_gige_intr.o \
mlxbf_gige_main.o \
mlxbf_gige_mdio.o \
#define MLXBF_GIGE_ERROR_INTR_IDX 0
#define MLXBF_GIGE_RECEIVE_PKT_INTR_IDX 1
#define MLXBF_GIGE_LLU_PLU_INTR_IDX 2
-#define MLXBF_GIGE_PHY_INT_N 3
-
-#define MLXBF_GIGE_MDIO_DEFAULT_PHY_ADDR 0x3
-
-#define MLXBF_GIGE_DEFAULT_PHY_INT_GPIO 12
struct mlxbf_gige_stats {
u64 hw_access_errors;
struct platform_device *pdev;
void __iomem *mdio_io;
struct mii_bus *mdiobus;
- void __iomem *gpio_io;
- struct irq_domain *irqdomain;
- u32 phy_int_gpio_mask;
spinlock_t lock; /* for packet processing indices */
- spinlock_t gpio_lock; /* for GPIO bus access */
u16 rx_q_entries;
u16 tx_q_entries;
u64 *tx_wqe_base;
extern const struct ethtool_ops mlxbf_gige_ethtool_ops;
void mlxbf_gige_update_tx_wqe_next(struct mlxbf_gige *priv);
-int mlxbf_gige_gpio_init(struct platform_device *pdev, struct mlxbf_gige *priv);
-void mlxbf_gige_gpio_free(struct mlxbf_gige *priv);
-
#endif /* !defined(__MLXBF_GIGE_H__) */
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0-only OR BSD-3-Clause
-
-/* Initialize and handle GPIO interrupt triggered by INT_N PHY signal.
- * This GPIO interrupt triggers the PHY state machine to bring the link
- * up/down.
- *
- * Copyright (C) 2021 NVIDIA CORPORATION & AFFILIATES
- */
-
-#include <linux/acpi.h>
-#include <linux/bitfield.h>
-#include <linux/device.h>
-#include <linux/err.h>
-#include <linux/gpio/driver.h>
-#include <linux/interrupt.h>
-#include <linux/io.h>
-#include <linux/irq.h>
-#include <linux/irqdomain.h>
-#include <linux/irqreturn.h>
-#include <linux/platform_device.h>
-#include <linux/property.h>
-
-#include "mlxbf_gige.h"
-#include "mlxbf_gige_regs.h"
-
-#define MLXBF_GIGE_GPIO_CAUSE_FALL_EN 0x48
-#define MLXBF_GIGE_GPIO_CAUSE_OR_CAUSE_EVTEN0 0x80
-#define MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0 0x94
-#define MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE 0x98
-
-static void mlxbf_gige_gpio_enable(struct mlxbf_gige *priv)
-{
- unsigned long flags;
- u32 val;
-
- spin_lock_irqsave(&priv->gpio_lock, flags);
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
- val |= priv->phy_int_gpio_mask;
- writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
-
- /* The INT_N interrupt level is active low.
- * So enable cause fall bit to detect when GPIO
- * state goes low.
- */
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_FALL_EN);
- val |= priv->phy_int_gpio_mask;
- writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_FALL_EN);
-
- /* Enable PHY interrupt by setting the priority level */
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
- val |= priv->phy_int_gpio_mask;
- writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
- spin_unlock_irqrestore(&priv->gpio_lock, flags);
-}
-
-static void mlxbf_gige_gpio_disable(struct mlxbf_gige *priv)
-{
- unsigned long flags;
- u32 val;
-
- spin_lock_irqsave(&priv->gpio_lock, flags);
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
- val &= ~priv->phy_int_gpio_mask;
- writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_EVTEN0);
- spin_unlock_irqrestore(&priv->gpio_lock, flags);
-}
-
-static irqreturn_t mlxbf_gige_gpio_handler(int irq, void *ptr)
-{
- struct mlxbf_gige *priv;
- u32 val;
-
- priv = ptr;
-
- /* Check if this interrupt is from PHY device.
- * Return if it is not.
- */
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CAUSE_EVTEN0);
- if (!(val & priv->phy_int_gpio_mask))
- return IRQ_NONE;
-
- /* Clear interrupt when done, otherwise, no further interrupt
- * will be triggered.
- */
- val = readl(priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
- val |= priv->phy_int_gpio_mask;
- writel(val, priv->gpio_io + MLXBF_GIGE_GPIO_CAUSE_OR_CLRCAUSE);
-
- generic_handle_irq(priv->phy_irq);
-
- return IRQ_HANDLED;
-}
-
-static void mlxbf_gige_gpio_mask(struct irq_data *irqd)
-{
- struct mlxbf_gige *priv = irq_data_get_irq_chip_data(irqd);
-
- mlxbf_gige_gpio_disable(priv);
-}
-
-static void mlxbf_gige_gpio_unmask(struct irq_data *irqd)
-{
- struct mlxbf_gige *priv = irq_data_get_irq_chip_data(irqd);
-
- mlxbf_gige_gpio_enable(priv);
-}
-
-static struct irq_chip mlxbf_gige_gpio_chip = {
- .name = "mlxbf_gige_phy",
- .irq_mask = mlxbf_gige_gpio_mask,
- .irq_unmask = mlxbf_gige_gpio_unmask,
-};
-
-static int mlxbf_gige_gpio_domain_map(struct irq_domain *d,
- unsigned int irq,
- irq_hw_number_t hwirq)
-{
- irq_set_chip_data(irq, d->host_data);
- irq_set_chip_and_handler(irq, &mlxbf_gige_gpio_chip, handle_simple_irq);
- irq_set_noprobe(irq);
-
- return 0;
-}
-
-static const struct irq_domain_ops mlxbf_gige_gpio_domain_ops = {
- .map = mlxbf_gige_gpio_domain_map,
- .xlate = irq_domain_xlate_twocell,
-};
-
-#ifdef CONFIG_ACPI
-static int mlxbf_gige_gpio_resources(struct acpi_resource *ares,
- void *data)
-{
- struct acpi_resource_gpio *gpio;
- u32 *phy_int_gpio = data;
-
- if (ares->type == ACPI_RESOURCE_TYPE_GPIO) {
- gpio = &ares->data.gpio;
- *phy_int_gpio = gpio->pin_table[0];
- }
-
- return 1;
-}
-#endif
-
-void mlxbf_gige_gpio_free(struct mlxbf_gige *priv)
-{
- irq_dispose_mapping(priv->phy_irq);
- irq_domain_remove(priv->irqdomain);
-}
-
-int mlxbf_gige_gpio_init(struct platform_device *pdev,
- struct mlxbf_gige *priv)
-{
- struct device *dev = &pdev->dev;
- struct resource *res;
- u32 phy_int_gpio = 0;
- int ret;
-
- LIST_HEAD(resources);
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, MLXBF_GIGE_RES_GPIO0);
- if (!res)
- return -ENODEV;
-
- priv->gpio_io = devm_ioremap(dev, res->start, resource_size(res));
- if (!priv->gpio_io)
- return -ENOMEM;
-
-#ifdef CONFIG_ACPI
- ret = acpi_dev_get_resources(ACPI_COMPANION(dev),
- &resources, mlxbf_gige_gpio_resources,
- &phy_int_gpio);
- acpi_dev_free_resource_list(&resources);
- if (ret < 0 || !phy_int_gpio) {
- dev_err(dev, "Error retrieving the gpio phy pin");
- return -EINVAL;
- }
-#endif
-
- priv->phy_int_gpio_mask = BIT(phy_int_gpio);
-
- mlxbf_gige_gpio_disable(priv);
-
- priv->hw_phy_irq = platform_get_irq(pdev, MLXBF_GIGE_PHY_INT_N);
-
- priv->irqdomain = irq_domain_add_simple(NULL, 1, 0,
- &mlxbf_gige_gpio_domain_ops,
- priv);
- if (!priv->irqdomain) {
- dev_err(dev, "Failed to add IRQ domain\n");
- return -ENOMEM;
- }
-
- priv->phy_irq = irq_create_mapping(priv->irqdomain, 0);
- if (!priv->phy_irq) {
- irq_domain_remove(priv->irqdomain);
- priv->irqdomain = NULL;
- dev_err(dev, "Error mapping PHY IRQ\n");
- return -EINVAL;
- }
-
- ret = devm_request_irq(dev, priv->hw_phy_irq, mlxbf_gige_gpio_handler,
- IRQF_ONESHOT | IRQF_SHARED, "mlxbf_gige_phy", priv);
- if (ret) {
- dev_err(dev, "Failed to request PHY IRQ");
- mlxbf_gige_gpio_free(priv);
- return ret;
- }
-
- return ret;
-}
void __iomem *llu_base;
void __iomem *plu_base;
void __iomem *base;
+ int addr, phy_irq;
u64 control;
- int addr;
int err;
base = devm_platform_ioremap_resource(pdev, MLXBF_GIGE_RES_MAC);
priv->pdev = pdev;
spin_lock_init(&priv->lock);
- spin_lock_init(&priv->gpio_lock);
/* Attach MDIO device */
err = mlxbf_gige_mdio_probe(pdev, priv);
if (err)
return err;
- err = mlxbf_gige_gpio_init(pdev, priv);
- if (err) {
- dev_err(&pdev->dev, "PHY IRQ initialization failed\n");
- mlxbf_gige_mdio_remove(priv);
- return -ENODEV;
- }
-
priv->base = base;
priv->llu_base = llu_base;
priv->plu_base = plu_base;
priv->rx_irq = platform_get_irq(pdev, MLXBF_GIGE_RECEIVE_PKT_INTR_IDX);
priv->llu_plu_irq = platform_get_irq(pdev, MLXBF_GIGE_LLU_PLU_INTR_IDX);
+ phy_irq = acpi_dev_gpio_irq_get_by(ACPI_COMPANION(&pdev->dev), "phy-gpios", 0);
+ if (phy_irq < 0) {
+ dev_err(&pdev->dev, "Error getting PHY irq. Use polling instead");
+ phy_irq = PHY_POLL;
+ }
+
phydev = phy_find_first(priv->mdiobus);
if (!phydev) {
err = -ENODEV;
}
addr = phydev->mdio.addr;
- priv->mdiobus->irq[addr] = priv->phy_irq;
- phydev->irq = priv->phy_irq;
+ priv->mdiobus->irq[addr] = phy_irq;
+ phydev->irq = phy_irq;
err = phy_connect_direct(netdev, phydev,
mlxbf_gige_adjust_link,
return 0;
out:
- mlxbf_gige_gpio_free(priv);
mlxbf_gige_mdio_remove(priv);
return err;
}
unregister_netdev(priv->netdev);
phy_disconnect(priv->netdev->phydev);
- mlxbf_gige_gpio_free(priv);
mlxbf_gige_mdio_remove(priv);
platform_set_drvdata(pdev, NULL);
static int mlxsw_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
- const char *driver_name = pdev->driver->name;
+ const char *driver_name = dev_driver_string(&pdev->dev);
struct mlxsw_pci *mlxsw_pci;
int err;
{
struct gdma_context *gc = pci_get_drvdata(pdev);
- dev_info(&pdev->dev, "Shutdown was calledd\n");
+ dev_info(&pdev->dev, "Shutdown was called\n");
mana_remove(&gc->mana, true);
{
char nsp_version[ETHTOOL_FWVERS_LEN] = {};
- strlcpy(drvinfo->driver, pdev->driver->name, sizeof(drvinfo->driver));
+ strlcpy(drvinfo->driver, dev_driver_string(&pdev->dev),
+ sizeof(drvinfo->driver));
nfp_net_get_nspinfo(app, nsp_version);
snprintf(drvinfo->fw_version, sizeof(drvinfo->fw_version),
"%s %s %s %s", vnic_version, nsp_version,
efx->rxq_entries = rxq_entries;
efx->txq_entries = txq_entries;
for (i = 0; i < efx->n_channels; i++) {
- channel = efx->channel[i];
- efx->channel[i] = other_channel[i];
- other_channel[i] = channel;
+ swap(efx->channel[i], other_channel[i]);
}
/* Restart buffer table allocation */
efx->rxq_entries = old_rxq_entries;
efx->txq_entries = old_txq_entries;
for (i = 0; i < efx->n_channels; i++) {
- channel = efx->channel[i];
- efx->channel[i] = other_channel[i];
- other_channel[i] = channel;
+ swap(efx->channel[i], other_channel[i]);
}
goto out;
}
/* get MAC address from EEPROM */
for (i = 0; i < 3; i++)
addr[i] = read_eeprom(ioaddr, i + EEPROMMACAddr);
- eth_hw_addr_set(net_dev, (u8 *)addr);
+ eth_hw_addr_set(net_dev, (u8 *)addr);
rc = 1;
break;
}
#endif /* CONFIG_PM_SLEEP */
-static SIMPLE_DEV_PM_OPS(socfpga_dwmac_pm_ops, stmmac_suspend,
- socfpga_dwmac_resume);
+static int __maybe_unused socfpga_dwmac_runtime_suspend(struct device *dev)
+{
+ struct net_device *ndev = dev_get_drvdata(dev);
+ struct stmmac_priv *priv = netdev_priv(ndev);
+
+ stmmac_bus_clks_config(priv, false);
+
+ return 0;
+}
+
+static int __maybe_unused socfpga_dwmac_runtime_resume(struct device *dev)
+{
+ struct net_device *ndev = dev_get_drvdata(dev);
+ struct stmmac_priv *priv = netdev_priv(ndev);
+
+ return stmmac_bus_clks_config(priv, true);
+}
+
+static const struct dev_pm_ops socfpga_dwmac_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(stmmac_suspend, socfpga_dwmac_resume)
+ SET_RUNTIME_PM_OPS(socfpga_dwmac_runtime_suspend, socfpga_dwmac_runtime_resume, NULL)
+};
static const struct socfpga_dwmac_ops socfpga_gen5_ops = {
.set_phy_mode = socfpga_gen5_set_phy_mode,
return true;
}
+static inline u32 stmmac_cdc_adjust(struct stmmac_priv *priv)
+{
+ /* Correct the clk domain crossing(CDC) error */
+ if (priv->plat->has_gmac4 && priv->plat->clk_ptp_rate)
+ return (2 * NSEC_PER_SEC) / priv->plat->clk_ptp_rate;
+ return 0;
+}
+
/* stmmac_get_tx_hwtstamp - get HW TX timestamps
* @priv: driver private structure
* @p : descriptor pointer
{
struct skb_shared_hwtstamps shhwtstamp;
bool found = false;
- s64 adjust = 0;
u64 ns = 0;
if (!priv->hwts_tx_en)
}
if (found) {
- /* Correct the clk domain crossing(CDC) error */
- if (priv->plat->has_gmac4 && priv->plat->clk_ptp_rate) {
- adjust += -(2 * (NSEC_PER_SEC /
- priv->plat->clk_ptp_rate));
- ns += adjust;
- }
+ ns -= stmmac_cdc_adjust(priv);
memset(&shhwtstamp, 0, sizeof(struct skb_shared_hwtstamps));
shhwtstamp.hwtstamp = ns_to_ktime(ns);
{
struct skb_shared_hwtstamps *shhwtstamp = NULL;
struct dma_desc *desc = p;
- u64 adjust = 0;
u64 ns = 0;
if (!priv->hwts_rx_en)
if (stmmac_get_rx_timestamp_status(priv, p, np, priv->adv_ts)) {
stmmac_get_timestamp(priv, desc, priv->adv_ts, &ns);
- /* Correct the clk domain crossing(CDC) error */
- if (priv->plat->has_gmac4 && priv->plat->clk_ptp_rate) {
- adjust += 2 * (NSEC_PER_SEC / priv->plat->clk_ptp_rate);
- ns -= adjust;
- }
+ ns -= stmmac_cdc_adjust(priv);
netdev_dbg(priv->dev, "get valid RX hw timestamp %llu\n", ns);
shhwtstamp = skb_hwtstamps(skb);
goto disable;
if (qopt->num_entries >= dep)
return -EINVAL;
- if (!qopt->base_time)
- return -ERANGE;
if (!qopt->cycle_time)
return -ERANGE;
if (!ale)
return ERR_PTR(-ENOMEM);
- ale->p0_untag_vid_mask =
- devm_kmalloc_array(params->dev, BITS_TO_LONGS(VLAN_N_VID),
- sizeof(unsigned long),
- GFP_KERNEL);
+ ale->p0_untag_vid_mask = devm_bitmap_zalloc(params->dev, VLAN_N_VID,
+ GFP_KERNEL);
if (!ale->p0_untag_vid_mask)
return ERR_PTR(-ENOMEM);
u32 int_ctrl, num_interrupts = 0;
u32 prescale = 0, addnl_dvdr = 1, coal_intvl = 0;
- if (!coal->rx_coalesce_usecs)
- return -EINVAL;
+ if (!coal->rx_coalesce_usecs) {
+ priv->coal_intvl = 0;
+
+ switch (priv->version) {
+ case EMAC_VERSION_2:
+ emac_ctrl_write(EMAC_DM646X_CMINTCTRL, 0);
+ break;
+ default:
+ emac_ctrl_write(EMAC_CTRL_EWINTTCNT, 0);
+ break;
+ }
+
+ return 0;
+ }
coal_intvl = coal->rx_coalesce_usecs;
{
/* Finish setting up the DEVICE info. */
dev->netdev_ops = &sp_netdev_ops;
- dev->needs_free_netdev = true;
dev->mtu = SIXP_MTU;
dev->hard_header_len = AX25_MAX_HEADER_LEN;
dev->header_ops = &ax25_header_ops;
del_timer_sync(&sp->tx_t);
del_timer_sync(&sp->resync_t);
- /* Free all 6pack frame buffers. */
+ unregister_netdev(sp->dev);
+
+ /* Free all 6pack frame buffers after unreg. */
kfree(sp->rbuff);
kfree(sp->xbuff);
- unregister_netdev(sp->dev);
+ free_netdev(sp->dev);
}
/* Perform I/O control on an active 6pack channel. */
*/
netif_stop_queue(ax->dev);
- /* Free all AX25 frame buffers. */
- kfree(ax->rbuff);
- kfree(ax->xbuff);
-
ax->tty = NULL;
unregister_netdev(ax->dev);
+
+ /* Free all AX25 frame buffers after unreg. */
+ kfree(ax->rbuff);
+ kfree(ax->xbuff);
+
free_netdev(ax->dev);
}
u32 offset;
u32 val;
+ /* This should only be changed when HOL_BLOCK_EN is disabled */
offset = IPA_REG_ENDP_INIT_HOL_BLOCK_TIMER_N_OFFSET(endpoint_id);
val = hol_block_timer_val(ipa, microseconds);
iowrite32(val, ipa->reg_virt + offset);
val = enable ? HOL_BLOCK_EN_FMASK : 0;
offset = IPA_REG_ENDP_INIT_HOL_BLOCK_EN_N_OFFSET(endpoint_id);
iowrite32(val, endpoint->ipa->reg_virt + offset);
+ /* When enabling, the register must be written twice for IPA v4.5+ */
+ if (enable && endpoint->ipa->version >= IPA_VERSION_4_5)
+ iowrite32(val, endpoint->ipa->reg_virt + offset);
}
void ipa_endpoint_modem_hol_block_clear_all(struct ipa *ipa)
if (endpoint->toward_ipa || endpoint->ee_id != GSI_EE_MODEM)
continue;
+ ipa_endpoint_init_hol_block_enable(endpoint, false);
ipa_endpoint_init_hol_block_timer(endpoint, 0);
ipa_endpoint_init_hol_block_enable(endpoint, true);
}
return false;
}
- group_count = data->rsrc_group_src_count;
+ group_count = data->rsrc_group_dst_count;
if (!group_count || group_count > IPA_RESOURCE_GROUP_MAX)
return false;
#define LAN87XX_MASK_LINK_UP (0x0004)
#define LAN87XX_MASK_LINK_DOWN (0x0002)
+/* MISC Control 1 Register */
+#define LAN87XX_CTRL_1 (0x11)
+#define LAN87XX_MASK_RGMII_TXC_DLY_EN (0x4000)
+#define LAN87XX_MASK_RGMII_RXC_DLY_EN (0x2000)
+
/* phyaccess nested types */
#define PHYACC_ATTR_MODE_READ 0
#define PHYACC_ATTR_MODE_WRITE 1
return rc;
}
+static int lan87xx_config_rgmii_delay(struct phy_device *phydev)
+{
+ int rc;
+
+ if (!phy_interface_is_rgmii(phydev))
+ return 0;
+
+ rc = access_ereg(phydev, PHYACC_ATTR_MODE_READ,
+ PHYACC_ATTR_BANK_MISC, LAN87XX_CTRL_1, 0);
+ if (rc < 0)
+ return rc;
+
+ switch (phydev->interface) {
+ case PHY_INTERFACE_MODE_RGMII:
+ rc &= ~LAN87XX_MASK_RGMII_TXC_DLY_EN;
+ rc &= ~LAN87XX_MASK_RGMII_RXC_DLY_EN;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_ID:
+ rc |= LAN87XX_MASK_RGMII_TXC_DLY_EN;
+ rc |= LAN87XX_MASK_RGMII_RXC_DLY_EN;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_RXID:
+ rc &= ~LAN87XX_MASK_RGMII_TXC_DLY_EN;
+ rc |= LAN87XX_MASK_RGMII_RXC_DLY_EN;
+ break;
+ case PHY_INTERFACE_MODE_RGMII_TXID:
+ rc |= LAN87XX_MASK_RGMII_TXC_DLY_EN;
+ rc &= ~LAN87XX_MASK_RGMII_RXC_DLY_EN;
+ break;
+ default:
+ return 0;
+ }
+
+ return access_ereg(phydev, PHYACC_ATTR_MODE_WRITE,
+ PHYACC_ATTR_BANK_MISC, LAN87XX_CTRL_1, rc);
+}
+
static int lan87xx_phy_init(struct phy_device *phydev)
{
static const struct access_ereg_val init[] = {
return rc;
}
- return 0;
+ return lan87xx_config_rgmii_delay(phydev);
}
static int lan87xx_phy_config_intr(struct phy_device *phydev)
phydev->mdix_ctrl = cmd->base.eth_tp_mdix_ctrl;
/* Restart the PHY */
- _phy_start_aneg(phydev);
+ if (phy_is_started(phydev)) {
+ phydev->state = PHY_UP;
+ phy_trigger_machine(phydev);
+ } else {
+ _phy_start_aneg(phydev);
+ }
mutex_unlock(&phydev->lock);
return 0;
* though magic-aneg shouldn't prevent this case from occurring
*/
- return 0;
+ return 0;
}
static int generic_suspend(struct mii_phy* phy)
{
struct tun_struct *tun = netdev_priv(dev);
int txq = skb->queue_mapping;
+ struct netdev_queue *queue;
struct tun_file *tfile;
int len = skb->len;
if (ptr_ring_produce(&tfile->tx_ring, skb))
goto drop;
+ /* NETIF_F_LLTX requires to do our own update of trans_start */
+ queue = netdev_get_tx_queue(dev, txq);
+ queue->trans_start = jiffies;
+
/* Notify and wake up reader process */
if (tfile->flags & TUN_FASYNC)
kill_fasync(&tfile->fasync, SIGIO, POLL_IN);
netdev->hw_features &= ~NETIF_F_RXCSUM;
}
- if (le16_to_cpu(udev->descriptor.idVendor) == VENDOR_ID_LENOVO) {
- switch (le16_to_cpu(udev->descriptor.idProduct)) {
- case DEVICE_ID_THINKPAD_THUNDERBOLT3_DOCK_GEN2:
- case DEVICE_ID_THINKPAD_USB_C_DOCK_GEN2:
- tp->lenovo_macpassthru = 1;
- }
+ if (udev->parent &&
+ le16_to_cpu(udev->parent->descriptor.idVendor) == VENDOR_ID_LENOVO) {
+ tp->lenovo_macpassthru = 1;
}
if (le16_to_cpu(udev->descriptor.bcdDevice) == 0x3011 && udev->serial &&
int boot_check_timeout = BOOT_CHECK_DEFAULT_TIMEOUT;
enum ipc_mem_exec_stage exec_stage;
struct ipc_mem_channel *channel;
- enum ipc_phase curr_phase;
int status = 0;
u32 tail = 0;
channel = ipc_imem->ipc_devlink->devlink_sio.channel;
- curr_phase = ipc_imem->phase;
/* Increase the total wait time to boot_check_timeout */
do {
exec_stage = ipc_mmio_get_exec_stage(ipc_imem->mmio);
frag = pn533_alloc_skb(dev, frag_size);
if (!frag) {
skb_queue_purge(&dev->fragment_skb);
- break;
+ return -ENOMEM;
}
if (!dev->tgt_mode) {
/* jumbo frame ? */
if (skb->len > PN533_CMD_DATAEXCH_DATA_MAXLEN) {
rc = pn533_fill_fragment_skbs(dev, skb);
- if (rc <= 0)
+ if (rc < 0)
goto error;
skb = skb_dequeue(&dev->fragment_skb);
/* let's split in multiple chunks if size's too big */
if (skb->len > PN533_CMD_DATAEXCH_DATA_MAXLEN) {
rc = pn533_fill_fragment_skbs(dev, skb);
- if (rc <= 0)
+ if (rc < 0)
goto error;
/* get the first skb */
break; /* success */
case -ECONNRESET:
case -ENOENT:
- nfc_err(&dev->interface->dev,
+ nfc_dbg(&dev->interface->dev,
"The urb has been canceled (status %d)\n", urb->status);
goto sched_wq;
case -ESHUTDOWN:
break; /* success */
case -ECONNRESET:
case -ENOENT:
- nfc_err(&dev->interface->dev,
+ nfc_dbg(&dev->interface->dev,
"The urb has been stopped (status %d)\n", urb->status);
goto sched_wq;
case -ESHUTDOWN:
break; /* success */
case -ECONNRESET:
case -ENOENT:
- nfc_err(&dev->interface->dev,
+ nfc_dbg(&dev->interface->dev,
"The urb has been stopped (status %d)\n", urb->status);
break;
case -ESHUTDOWN:
resource_size_t available_disk_size;
struct gendisk *disk;
u64 internal_nlba;
+ int rc;
internal_nlba = div_u64(nsblk->size, nsblk_internal_lbasize(nsblk));
available_disk_size = internal_nlba * nsblk_sector_size(nsblk);
blk_queue_logical_block_size(disk->queue, nsblk_sector_size(nsblk));
blk_queue_flag_set(QUEUE_FLAG_NONROT, disk->queue);
- if (devm_add_action_or_reset(dev, nd_blk_release_disk, disk))
- return -ENOMEM;
-
if (nsblk_meta_size(nsblk)) {
- int rc = nd_integrity_init(disk, nsblk_meta_size(nsblk));
+ rc = nd_integrity_init(disk, nsblk_meta_size(nsblk));
if (rc)
- return rc;
+ goto out_before_devm_err;
}
set_capacity(disk, available_disk_size >> SECTOR_SHIFT);
- device_add_disk(dev, disk, NULL);
+ rc = device_add_disk(dev, disk, NULL);
+ if (rc)
+ goto out_before_devm_err;
+
+ /* nd_blk_release_disk() is called if this fails */
+ if (devm_add_action_or_reset(dev, nd_blk_release_disk, disk))
+ return -ENOMEM;
+
nvdimm_check_and_set_ro(disk);
return 0;
+
+out_before_devm_err:
+ blk_cleanup_disk(disk);
+ return rc;
}
static int nd_blk_probe(struct device *dev)
u64 sum;
struct btt_sb *super;
struct nd_btt *nd_btt = arena->nd_btt;
- const u8 *parent_uuid = nd_dev_to_uuid(&nd_btt->ndns->dev);
+ const uuid_t *parent_uuid = nd_dev_to_uuid(&nd_btt->ndns->dev);
ret = btt_map_init(arena);
if (ret)
return -ENOMEM;
strncpy(super->signature, BTT_SIG, BTT_SIG_LEN);
- memcpy(super->uuid, nd_btt->uuid, 16);
- memcpy(super->parent_uuid, parent_uuid, 16);
+ export_uuid(super->uuid, nd_btt->uuid);
+ export_uuid(super->parent_uuid, parent_uuid);
super->flags = cpu_to_le32(arena->flags);
super->version_major = cpu_to_le16(arena->version_major);
super->version_minor = cpu_to_le16(arena->version_minor);
{
struct nd_btt *nd_btt = btt->nd_btt;
struct nd_namespace_common *ndns = nd_btt->ndns;
+ int rc = -ENOMEM;
btt->btt_disk = blk_alloc_disk(NUMA_NO_NODE);
if (!btt->btt_disk)
blk_queue_flag_set(QUEUE_FLAG_NONROT, btt->btt_disk->queue);
if (btt_meta_size(btt)) {
- int rc = nd_integrity_init(btt->btt_disk, btt_meta_size(btt));
-
- if (rc) {
- del_gendisk(btt->btt_disk);
- blk_cleanup_disk(btt->btt_disk);
- return rc;
- }
+ rc = nd_integrity_init(btt->btt_disk, btt_meta_size(btt));
+ if (rc)
+ goto out_cleanup_disk;
}
+
set_capacity(btt->btt_disk, btt->nlba * btt->sector_size >> 9);
- device_add_disk(&btt->nd_btt->dev, btt->btt_disk, NULL);
+ rc = device_add_disk(&btt->nd_btt->dev, btt->btt_disk, NULL);
+ if (rc)
+ goto out_cleanup_disk;
+
btt->nd_btt->size = btt->nlba * (u64)btt->sector_size;
nvdimm_check_and_set_ro(btt->btt_disk);
return 0;
+
+out_cleanup_disk:
+ blk_cleanup_disk(btt->btt_disk);
+ return rc;
}
static void btt_blk_cleanup(struct btt *btt)
* Pointer to a new struct btt on success, NULL on failure.
*/
static struct btt *btt_init(struct nd_btt *nd_btt, unsigned long long rawsize,
- u32 lbasize, u8 *uuid, struct nd_region *nd_region)
+ u32 lbasize, uuid_t *uuid,
+ struct nd_region *nd_region)
{
int ret;
struct btt *btt;
}
nd_region = to_nd_region(nd_btt->dev.parent);
btt = btt_init(nd_btt, rawsize, nd_btt->lbasize, nd_btt->uuid,
- nd_region);
+ nd_region);
if (!btt)
return -ENOMEM;
nd_btt->btt = btt;
EXPORT_SYMBOL(is_nd_btt);
static struct device *__nd_btt_create(struct nd_region *nd_region,
- unsigned long lbasize, u8 *uuid,
- struct nd_namespace_common *ndns)
+ unsigned long lbasize, uuid_t *uuid,
+ struct nd_namespace_common *ndns)
{
struct nd_btt *nd_btt;
struct device *dev;
*/
bool nd_btt_arena_is_valid(struct nd_btt *nd_btt, struct btt_sb *super)
{
- const u8 *parent_uuid = nd_dev_to_uuid(&nd_btt->ndns->dev);
+ const uuid_t *ns_uuid = nd_dev_to_uuid(&nd_btt->ndns->dev);
+ uuid_t parent_uuid;
u64 checksum;
if (memcmp(super->signature, BTT_SIG, BTT_SIG_LEN) != 0)
return false;
- if (!guid_is_null((guid_t *)&super->parent_uuid))
- if (memcmp(super->parent_uuid, parent_uuid, 16) != 0)
+ import_uuid(&parent_uuid, super->parent_uuid);
+ if (!uuid_is_null(&parent_uuid))
+ if (!uuid_equal(&parent_uuid, ns_uuid))
return false;
checksum = le64_to_cpu(super->checksum);
return rc;
nd_btt->lbasize = le32_to_cpu(btt_sb->external_lbasize);
- nd_btt->uuid = kmemdup(btt_sb->uuid, 16, GFP_KERNEL);
+ nd_btt->uuid = kmemdup(&btt_sb->uuid, sizeof(uuid_t), GFP_KERNEL);
if (!nd_btt->uuid)
return -ENOMEM;
}
EXPORT_SYMBOL_GPL(to_nvdimm_bus_dev);
-static bool is_uuid_sep(char sep)
-{
- if (sep == '\n' || sep == '-' || sep == ':' || sep == '\0')
- return true;
- return false;
-}
-
-static int nd_uuid_parse(struct device *dev, u8 *uuid_out, const char *buf,
- size_t len)
-{
- const char *str = buf;
- u8 uuid[16];
- int i;
-
- for (i = 0; i < 16; i++) {
- if (!isxdigit(str[0]) || !isxdigit(str[1])) {
- dev_dbg(dev, "pos: %d buf[%zd]: %c buf[%zd]: %c\n",
- i, str - buf, str[0],
- str + 1 - buf, str[1]);
- return -EINVAL;
- }
-
- uuid[i] = (hex_to_bin(str[0]) << 4) | hex_to_bin(str[1]);
- str += 2;
- if (is_uuid_sep(*str))
- str++;
- }
-
- memcpy(uuid_out, uuid, sizeof(uuid));
- return 0;
-}
-
/**
* nd_uuid_store: common implementation for writing 'uuid' sysfs attributes
* @dev: container device for the uuid property
* (driver detached)
* LOCKING: expects nd_device_lock() is held on entry
*/
-int nd_uuid_store(struct device *dev, u8 **uuid_out, const char *buf,
+int nd_uuid_store(struct device *dev, uuid_t **uuid_out, const char *buf,
size_t len)
{
- u8 uuid[16];
+ uuid_t uuid;
int rc;
if (dev->driver)
return -EBUSY;
- rc = nd_uuid_parse(dev, uuid, buf, len);
+ rc = uuid_parse(buf, &uuid);
if (rc)
return rc;
kfree(*uuid_out);
- *uuid_out = kmemdup(uuid, sizeof(uuid), GFP_KERNEL);
+ *uuid_out = kmemdup(&uuid, sizeof(uuid), GFP_KERNEL);
if (!(*uuid_out))
return -ENOMEM;
static guid_t nvdimm_pfn_guid;
static guid_t nvdimm_dax_guid;
+static uuid_t nvdimm_btt_uuid;
+static uuid_t nvdimm_btt2_uuid;
+static uuid_t nvdimm_pfn_uuid;
+static uuid_t nvdimm_dax_uuid;
+
+static uuid_t cxl_region_uuid;
+static uuid_t cxl_namespace_uuid;
+
static const char NSINDEX_SIGNATURE[] = "NAMESPACE_INDEX\0";
static u32 best_seq(u32 a, u32 b)
return true;
}
-char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags)
+char *nd_label_gen_id(struct nd_label_id *label_id, const uuid_t *uuid,
+ u32 flags)
{
if (!label_id || !uuid)
return NULL;
{
u64 sum, sum_save;
- if (!namespace_label_has(ndd, checksum))
+ if (!ndd->cxl && !efi_namespace_label_has(ndd, checksum))
return true;
sum_save = nsl_get_checksum(ndd, nd_label);
{
u64 sum;
- if (!namespace_label_has(ndd, checksum))
+ if (!ndd->cxl && !efi_namespace_label_has(ndd, checksum))
return;
nsl_set_checksum(ndd, nd_label, 0);
sum = nd_fletcher64(nd_label, sizeof_namespace_label(ndd), 1);
struct nvdimm *nvdimm = to_nvdimm(ndd->dev);
struct nd_namespace_label *nd_label;
struct nd_region *nd_region = NULL;
- u8 label_uuid[NSLABEL_UUID_LEN];
struct nd_label_id label_id;
struct resource *res;
+ uuid_t label_uuid;
u32 flags;
nd_label = to_label(ndd, slot);
if (!slot_valid(ndd, nd_label, slot))
continue;
- memcpy(label_uuid, nd_label->uuid, NSLABEL_UUID_LEN);
+ nsl_get_uuid(ndd, nd_label, &label_uuid);
flags = nsl_get_flags(ndd, nd_label);
if (test_bit(NDD_NOBLK, &nvdimm->flags))
flags &= ~NSLABEL_FLAG_LOCAL;
- nd_label_gen_id(&label_id, label_uuid, flags);
+ nd_label_gen_id(&label_id, &label_uuid, flags);
res = nvdimm_allocate_dpa(ndd, &label_id,
nsl_get_dpa(ndd, nd_label),
nsl_get_rawsize(ndd, nd_label));
- (unsigned long) to_namespace_index(ndd, 0);
}
-static enum nvdimm_claim_class to_nvdimm_cclass(guid_t *guid)
+static enum nvdimm_claim_class guid_to_nvdimm_cclass(guid_t *guid)
{
if (guid_equal(guid, &nvdimm_btt_guid))
return NVDIMM_CCLASS_BTT;
return NVDIMM_CCLASS_UNKNOWN;
}
+/* CXL labels store UUIDs instead of GUIDs for the same data */
+static enum nvdimm_claim_class uuid_to_nvdimm_cclass(uuid_t *uuid)
+{
+ if (uuid_equal(uuid, &nvdimm_btt_uuid))
+ return NVDIMM_CCLASS_BTT;
+ else if (uuid_equal(uuid, &nvdimm_btt2_uuid))
+ return NVDIMM_CCLASS_BTT2;
+ else if (uuid_equal(uuid, &nvdimm_pfn_uuid))
+ return NVDIMM_CCLASS_PFN;
+ else if (uuid_equal(uuid, &nvdimm_dax_uuid))
+ return NVDIMM_CCLASS_DAX;
+ else if (uuid_equal(uuid, &uuid_null))
+ return NVDIMM_CCLASS_NONE;
+
+ return NVDIMM_CCLASS_UNKNOWN;
+}
+
static const guid_t *to_abstraction_guid(enum nvdimm_claim_class claim_class,
guid_t *target)
{
return &guid_null;
}
+/* CXL labels store UUIDs instead of GUIDs for the same data */
+static const uuid_t *to_abstraction_uuid(enum nvdimm_claim_class claim_class,
+ uuid_t *target)
+{
+ if (claim_class == NVDIMM_CCLASS_BTT)
+ return &nvdimm_btt_uuid;
+ else if (claim_class == NVDIMM_CCLASS_BTT2)
+ return &nvdimm_btt2_uuid;
+ else if (claim_class == NVDIMM_CCLASS_PFN)
+ return &nvdimm_pfn_uuid;
+ else if (claim_class == NVDIMM_CCLASS_DAX)
+ return &nvdimm_dax_uuid;
+ else if (claim_class == NVDIMM_CCLASS_UNKNOWN) {
+ /*
+ * If we're modifying a namespace for which we don't
+ * know the claim_class, don't touch the existing uuid.
+ */
+ return target;
+ } else
+ return &uuid_null;
+}
+
static void reap_victim(struct nd_mapping *nd_mapping,
struct nd_label_ent *victim)
{
static void nsl_set_type_guid(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, guid_t *guid)
{
- if (namespace_label_has(ndd, type_guid))
- guid_copy(&nd_label->type_guid, guid);
+ if (efi_namespace_label_has(ndd, type_guid))
+ guid_copy(&nd_label->efi.type_guid, guid);
}
bool nsl_validate_type_guid(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, guid_t *guid)
{
- if (!namespace_label_has(ndd, type_guid))
+ if (ndd->cxl || !efi_namespace_label_has(ndd, type_guid))
return true;
- if (!guid_equal(&nd_label->type_guid, guid)) {
+ if (!guid_equal(&nd_label->efi.type_guid, guid)) {
dev_dbg(ndd->dev, "expect type_guid %pUb got %pUb\n", guid,
- &nd_label->type_guid);
+ &nd_label->efi.type_guid);
return false;
}
return true;
struct nd_namespace_label *nd_label,
enum nvdimm_claim_class claim_class)
{
- if (!namespace_label_has(ndd, abstraction_guid))
+ if (ndd->cxl) {
+ uuid_t uuid;
+
+ import_uuid(&uuid, nd_label->cxl.abstraction_uuid);
+ export_uuid(nd_label->cxl.abstraction_uuid,
+ to_abstraction_uuid(claim_class, &uuid));
+ return;
+ }
+
+ if (!efi_namespace_label_has(ndd, abstraction_guid))
return;
- guid_copy(&nd_label->abstraction_guid,
+ guid_copy(&nd_label->efi.abstraction_guid,
to_abstraction_guid(claim_class,
- &nd_label->abstraction_guid));
+ &nd_label->efi.abstraction_guid));
}
enum nvdimm_claim_class nsl_get_claim_class(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- if (!namespace_label_has(ndd, abstraction_guid))
+ if (ndd->cxl) {
+ uuid_t uuid;
+
+ import_uuid(&uuid, nd_label->cxl.abstraction_uuid);
+ return uuid_to_nvdimm_cclass(&uuid);
+ }
+ if (!efi_namespace_label_has(ndd, abstraction_guid))
return NVDIMM_CCLASS_NONE;
- return to_nvdimm_cclass(&nd_label->abstraction_guid);
+ return guid_to_nvdimm_cclass(&nd_label->efi.abstraction_guid);
}
static int __pmem_label_update(struct nd_region *nd_region,
nd_label = to_label(ndd, slot);
memset(nd_label, 0, sizeof_namespace_label(ndd));
- memcpy(nd_label->uuid, nspm->uuid, NSLABEL_UUID_LEN);
+ nsl_set_uuid(ndd, nd_label, nspm->uuid);
nsl_set_name(ndd, nd_label, nspm->alt_name);
nsl_set_flags(ndd, nd_label, flags);
nsl_set_nlabel(ndd, nd_label, nd_region->ndr_mappings);
+ nsl_set_nrange(ndd, nd_label, 1);
nsl_set_position(ndd, nd_label, pos);
nsl_set_isetcookie(ndd, nd_label, cookie);
nsl_set_rawsize(ndd, nd_label, resource_size(res));
list_for_each_entry(label_ent, &nd_mapping->labels, list) {
if (!label_ent->label)
continue;
- if (test_and_clear_bit(ND_LABEL_REAP, &label_ent->flags)
- || memcmp(nspm->uuid, label_ent->label->uuid,
- NSLABEL_UUID_LEN) == 0)
+ if (test_and_clear_bit(ND_LABEL_REAP, &label_ent->flags) ||
+ nsl_uuid_equal(ndd, label_ent->label, nspm->uuid))
reap_victim(nd_mapping, label_ent);
}
struct nd_namespace_label *nd_label,
u64 isetcookie)
{
- if (namespace_label_has(ndd, type_guid)) {
+ if (efi_namespace_label_has(ndd, type_guid)) {
nsl_set_isetcookie(ndd, nd_label, isetcookie);
return;
}
struct nd_namespace_label *nd_label,
u64 isetcookie)
{
- if (!namespace_label_has(ndd, type_guid))
+ if (!efi_namespace_label_has(ndd, type_guid))
return true;
if (nsl_get_isetcookie(ndd, nd_label) != isetcookie) {
struct nd_namespace_label *nd_label, int nlabel,
bool first)
{
- if (!namespace_label_has(ndd, type_guid)) {
+ if (!efi_namespace_label_has(ndd, type_guid)) {
nsl_set_nlabel(ndd, nd_label, 0); /* N/A */
return;
}
struct nd_namespace_label *nd_label,
bool first)
{
- if (!namespace_label_has(ndd, type_guid)) {
+ if (!efi_namespace_label_has(ndd, type_guid)) {
nsl_set_position(ndd, nd_label, 0);
return;
}
unsigned long *free, *victim_map = NULL;
struct resource *res, **old_res_list;
struct nd_label_id label_id;
- u8 uuid[NSLABEL_UUID_LEN];
int min_dpa_idx = 0;
LIST_HEAD(list);
u32 nslot, slot;
/* mark unused labels for garbage collection */
for_each_clear_bit_le(slot, free, nslot) {
nd_label = to_label(ndd, slot);
- memcpy(uuid, nd_label->uuid, NSLABEL_UUID_LEN);
- if (memcmp(uuid, nsblk->uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, nsblk->uuid))
continue;
res = to_resource(ndd, nd_label);
if (res && is_old_resource(res, old_res_list,
nd_label = to_label(ndd, slot);
memset(nd_label, 0, sizeof_namespace_label(ndd));
- memcpy(nd_label->uuid, nsblk->uuid, NSLABEL_UUID_LEN);
+ nsl_set_uuid(ndd, nd_label, nsblk->uuid);
nsl_set_name(ndd, nd_label, nsblk->alt_name);
nsl_set_flags(ndd, nd_label, NSLABEL_FLAG_LOCAL);
if (!nd_label)
continue;
nlabel++;
- memcpy(uuid, nd_label->uuid, NSLABEL_UUID_LEN);
- if (memcmp(uuid, nsblk->uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, nsblk->uuid))
continue;
nlabel--;
list_move(&label_ent->list, &list);
}
for_each_clear_bit_le(slot, free, nslot) {
nd_label = to_label(ndd, slot);
- memcpy(uuid, nd_label->uuid, NSLABEL_UUID_LEN);
- if (memcmp(uuid, nsblk->uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, nsblk->uuid))
continue;
res = to_resource(ndd, nd_label);
res->flags &= ~DPA_RESOURCE_ADJUSTED;
return max(num_labels, old_num_labels);
}
-static int del_labels(struct nd_mapping *nd_mapping, u8 *uuid)
+static int del_labels(struct nd_mapping *nd_mapping, uuid_t *uuid)
{
struct nvdimm_drvdata *ndd = to_ndd(nd_mapping);
struct nd_label_ent *label_ent, *e;
struct nd_namespace_index *nsindex;
- u8 label_uuid[NSLABEL_UUID_LEN];
unsigned long *free;
LIST_HEAD(list);
u32 nslot, slot;
if (!nd_label)
continue;
active++;
- memcpy(label_uuid, nd_label->uuid, NSLABEL_UUID_LEN);
- if (memcmp(label_uuid, uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, uuid))
continue;
active--;
slot = to_slot(ndd, nd_label);
WARN_ON(guid_parse(NVDIMM_PFN_GUID, &nvdimm_pfn_guid));
WARN_ON(guid_parse(NVDIMM_DAX_GUID, &nvdimm_dax_guid));
+ WARN_ON(uuid_parse(NVDIMM_BTT_GUID, &nvdimm_btt_uuid));
+ WARN_ON(uuid_parse(NVDIMM_BTT2_GUID, &nvdimm_btt2_uuid));
+ WARN_ON(uuid_parse(NVDIMM_PFN_GUID, &nvdimm_pfn_uuid));
+ WARN_ON(uuid_parse(NVDIMM_DAX_GUID, &nvdimm_dax_uuid));
+
+ WARN_ON(uuid_parse(CXL_REGION_UUID, &cxl_region_uuid));
+ WARN_ON(uuid_parse(CXL_NAMESPACE_UUID, &cxl_namespace_uuid));
+
return 0;
}
* struct nd_namespace_index - label set superblock
* @sig: NAMESPACE_INDEX\0
* @flags: placeholder
+ * @labelsize: log2 size (v1 labels 128 bytes v2 labels 256 bytes)
* @seq: sequence number for this index
* @myoff: offset of this index in label area
* @mysize: size of this index struct
* @major: label area major version
* @minor: label area minor version
* @checksum: fletcher64 of all fields
- * @free[0]: bitmap, nlabel bits
+ * @free: bitmap, nlabel bits
*
* The size of free[] is rounded up so the total struct size is a
* multiple of NSINDEX_ALIGN bytes. Any bits this allocates beyond
};
/**
- * struct nd_namespace_label - namespace superblock
+ * struct cxl_region_label - CXL 2.0 Table 211
+ * @type: uuid identifying this label format (region)
+ * @uuid: uuid for the region this label describes
+ * @flags: NSLABEL_FLAG_UPDATING (all other flags reserved)
+ * @nlabel: 1 per interleave-way in the region
+ * @position: this label's position in the set
+ * @dpa: start address in device-local capacity for this label
+ * @rawsize: size of this label's contribution to region
+ * @hpa: mandatory system physical address to map this region
+ * @slot: slot id of this label in label area
+ * @ig: interleave granularity (1 << @ig) * 256 bytes
+ * @align: alignment in SZ_256M blocks
+ * @reserved: reserved
+ * @checksum: fletcher64 sum of this label
+ */
+struct cxl_region_label {
+ u8 type[NSLABEL_UUID_LEN];
+ u8 uuid[NSLABEL_UUID_LEN];
+ __le32 flags;
+ __le16 nlabel;
+ __le16 position;
+ __le64 dpa;
+ __le64 rawsize;
+ __le64 hpa;
+ __le32 slot;
+ __le32 ig;
+ __le32 align;
+ u8 reserved[0xac];
+ __le64 checksum;
+};
+
+/**
+ * struct nvdimm_efi_label - namespace superblock
* @uuid: UUID per RFC 4122
* @name: optional name (NULL-terminated)
* @flags: see NSLABEL_FLAG_*
* @dpa: DPA of NVM range on this DIMM
* @rawsize: size of namespace
* @slot: slot of this label in label area
- * @unused: must be zero
+ * @align: physical address alignment of the namespace
+ * @reserved: reserved
+ * @type_guid: copy of struct acpi_nfit_system_address.range_guid
+ * @abstraction_guid: personality id (btt, btt2, fsdax, devdax....)
+ * @reserved2: reserved
+ * @checksum: fletcher64 sum of this object
*/
-struct nd_namespace_label {
+struct nvdimm_efi_label {
u8 uuid[NSLABEL_UUID_LEN];
u8 name[NSLABEL_NAME_LEN];
__le32 flags;
__le32 slot;
/*
* Accessing fields past this point should be gated by a
- * namespace_label_has() check.
+ * efi_namespace_label_has() check.
*/
u8 align;
u8 reserved[3];
__le64 checksum;
};
+/**
+ * struct nvdimm_cxl_label - CXL 2.0 Table 212
+ * @type: uuid identifying this label format (namespace)
+ * @uuid: uuid for the namespace this label describes
+ * @name: friendly name for the namespace
+ * @flags: NSLABEL_FLAG_UPDATING (all other flags reserved)
+ * @nrange: discontiguous namespace support
+ * @position: this label's position in the set
+ * @dpa: start address in device-local capacity for this label
+ * @rawsize: size of this label's contribution to namespace
+ * @slot: slot id of this label in label area
+ * @align: alignment in SZ_256M blocks
+ * @region_uuid: host interleave set identifier
+ * @abstraction_uuid: personality driver for this namespace
+ * @lbasize: address geometry for disk-like personalities
+ * @reserved: reserved
+ * @checksum: fletcher64 sum of this label
+ */
+struct nvdimm_cxl_label {
+ u8 type[NSLABEL_UUID_LEN];
+ u8 uuid[NSLABEL_UUID_LEN];
+ u8 name[NSLABEL_NAME_LEN];
+ __le32 flags;
+ __le16 nrange;
+ __le16 position;
+ __le64 dpa;
+ __le64 rawsize;
+ __le32 slot;
+ __le32 align;
+ u8 region_uuid[16];
+ u8 abstraction_uuid[16];
+ __le16 lbasize;
+ u8 reserved[0x56];
+ __le64 checksum;
+};
+
+struct nd_namespace_label {
+ union {
+ struct nvdimm_cxl_label cxl;
+ struct nvdimm_efi_label efi;
+ };
+};
+
#define NVDIMM_BTT_GUID "8aed63a2-29a2-4c66-8b12-f05d15d3922a"
#define NVDIMM_BTT2_GUID "18633bfc-1735-4217-8ac9-17239282d3f8"
#define NVDIMM_PFN_GUID "266400ba-fb9f-4677-bcb0-968f11d0d225"
#define NVDIMM_DAX_GUID "97a86d9c-3cdd-4eda-986f-5068b4f80088"
+#define CXL_REGION_UUID "529d7c61-da07-47c4-a93f-ecdf2c06f444"
+#define CXL_NAMESPACE_UUID "68bb2c0a-5a77-4937-9f85-3caf41a0f93c"
+
/**
* struct nd_label_id - identifier string for dpa allocation
* @id: "{blk|pmem}-<namespace uuid>"
static int is_uuid_busy(struct device *dev, void *data)
{
- u8 *uuid1 = data, *uuid2 = NULL;
+ uuid_t *uuid1 = data, *uuid2 = NULL;
if (is_namespace_pmem(dev)) {
struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev);
uuid2 = nd_pfn->uuid;
}
- if (uuid2 && memcmp(uuid1, uuid2, NSLABEL_UUID_LEN) == 0)
+ if (uuid2 && uuid_equal(uuid1, uuid2))
return -EBUSY;
return 0;
* @dev: any device on a nvdimm_bus
* @uuid: uuid to check
*/
-bool nd_is_uuid_unique(struct device *dev, u8 *uuid)
+bool nd_is_uuid_unique(struct device *dev, uuid_t *uuid)
{
struct nvdimm_bus *nvdimm_bus = walk_to_nvdimm_bus(dev);
}
EXPORT_SYMBOL(nvdimm_namespace_disk_name);
-const u8 *nd_dev_to_uuid(struct device *dev)
+const uuid_t *nd_dev_to_uuid(struct device *dev)
{
- static const u8 null_uuid[16];
-
if (!dev)
- return null_uuid;
+ return &uuid_null;
if (is_namespace_pmem(dev)) {
struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev);
return nsblk->uuid;
} else
- return null_uuid;
+ return &uuid_null;
}
EXPORT_SYMBOL(nd_dev_to_uuid);
res->end = res->start + size - 1;
}
-static bool uuid_not_set(const u8 *uuid, struct device *dev, const char *where)
+static bool uuid_not_set(const uuid_t *uuid, struct device *dev,
+ const char *where)
{
if (!uuid) {
dev_dbg(dev, "%s: uuid not set\n", where);
struct nd_label_id label_id;
u32 flags = 0, remainder;
int rc, i, id = -1;
- u8 *uuid = NULL;
+ uuid_t *uuid = NULL;
if (dev->driver || ndns->claim)
return -EBUSY;
{
struct nd_region *nd_region = to_nd_region(dev->parent);
unsigned long long val;
- u8 **uuid = NULL;
+ uuid_t **uuid = NULL;
int rc;
rc = kstrtoull(buf, 0, &val);
}
static DEVICE_ATTR(size, 0444, size_show, size_store);
-static u8 *namespace_to_uuid(struct device *dev)
+static uuid_t *namespace_to_uuid(struct device *dev)
{
if (is_namespace_pmem(dev)) {
struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev);
return ERR_PTR(-ENXIO);
}
-static ssize_t uuid_show(struct device *dev,
- struct device_attribute *attr, char *buf)
+static ssize_t uuid_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
{
- u8 *uuid = namespace_to_uuid(dev);
+ uuid_t *uuid = namespace_to_uuid(dev);
if (IS_ERR(uuid))
return PTR_ERR(uuid);
* @old_uuid: reference to the uuid storage location in the namespace object
*/
static int namespace_update_uuid(struct nd_region *nd_region,
- struct device *dev, u8 *new_uuid, u8 **old_uuid)
+ struct device *dev, uuid_t *new_uuid,
+ uuid_t **old_uuid)
{
u32 flags = is_namespace_blk(dev) ? NSLABEL_FLAG_LOCAL : 0;
struct nd_label_id old_label_id;
list_for_each_entry(label_ent, &nd_mapping->labels, list) {
struct nd_namespace_label *nd_label = label_ent->label;
struct nd_label_id label_id;
+ uuid_t uuid;
if (!nd_label)
continue;
- nd_label_gen_id(&label_id, nd_label->uuid,
+ nsl_get_uuid(ndd, nd_label, &uuid);
+ nd_label_gen_id(&label_id, &uuid,
nsl_get_flags(ndd, nd_label));
if (strcmp(old_label_id.id, label_id.id) == 0)
set_bit(ND_LABEL_REAP, &label_ent->flags);
struct device_attribute *attr, const char *buf, size_t len)
{
struct nd_region *nd_region = to_nd_region(dev->parent);
- u8 *uuid = NULL;
+ uuid_t *uuid = NULL;
+ uuid_t **ns_uuid;
ssize_t rc = 0;
- u8 **ns_uuid;
if (is_namespace_pmem(dev)) {
struct nd_namespace_pmem *nspm = to_nd_namespace_pmem(dev);
{
struct nd_region *nd_region = to_nd_region(dev->parent);
struct nd_label_id label_id;
+ uuid_t *uuid = NULL;
int count = 0, i;
- u8 *uuid = NULL;
u32 flags = 0;
nvdimm_bus_lock(dev);
return devs;
}
-static bool has_uuid_at_pos(struct nd_region *nd_region, u8 *uuid,
- u64 cookie, u16 pos)
+static bool has_uuid_at_pos(struct nd_region *nd_region, const uuid_t *uuid,
+ u64 cookie, u16 pos)
{
struct nd_namespace_label *found = NULL;
int i;
list_for_each_entry(label_ent, &nd_mapping->labels, list) {
struct nd_namespace_label *nd_label = label_ent->label;
- u16 position, nlabel;
+ u16 position;
if (!nd_label)
continue;
position = nsl_get_position(ndd, nd_label);
- nlabel = nsl_get_nlabel(ndd, nd_label);
if (!nsl_validate_isetcookie(ndd, nd_label, cookie))
continue;
- if (memcmp(nd_label->uuid, uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, uuid))
continue;
if (!nsl_validate_type_guid(ndd, nd_label,
return false;
}
found_uuid = true;
- if (nlabel != nd_region->ndr_mappings)
+ if (!nsl_validate_nlabel(nd_region, ndd, nd_label))
continue;
if (position != pos)
continue;
return found != NULL;
}
-static int select_pmem_id(struct nd_region *nd_region, u8 *pmem_id)
+static int select_pmem_id(struct nd_region *nd_region, const uuid_t *pmem_id)
{
int i;
nd_label = label_ent->label;
if (!nd_label)
continue;
- if (memcmp(nd_label->uuid, pmem_id, NSLABEL_UUID_LEN) == 0)
+ if (nsl_uuid_equal(ndd, nd_label, pmem_id))
break;
nd_label = NULL;
}
/* pass */;
else {
dev_dbg(&nd_region->dev, "%s invalid label for %pUb\n",
- dev_name(ndd->dev), nd_label->uuid);
+ dev_name(ndd->dev),
+ nsl_uuid_raw(ndd, nd_label));
return -EINVAL;
}
resource_size_t size = 0;
struct resource *res;
struct device *dev;
+ uuid_t uuid;
int rc = 0;
u16 i;
if (!nsl_validate_isetcookie(ndd, nd_label, cookie)) {
dev_dbg(&nd_region->dev, "invalid cookie in label: %pUb\n",
- nd_label->uuid);
+ nsl_uuid_raw(ndd, nd_label));
if (!nsl_validate_isetcookie(ndd, nd_label, altcookie))
return ERR_PTR(-EAGAIN);
dev_dbg(&nd_region->dev, "valid altcookie in label: %pUb\n",
- nd_label->uuid);
+ nsl_uuid_raw(ndd, nd_label));
}
nspm = kzalloc(sizeof(*nspm), GFP_KERNEL);
res->flags = IORESOURCE_MEM;
for (i = 0; i < nd_region->ndr_mappings; i++) {
- if (has_uuid_at_pos(nd_region, nd_label->uuid, cookie, i))
+ uuid_t uuid;
+
+ nsl_get_uuid(ndd, nd_label, &uuid);
+ if (has_uuid_at_pos(nd_region, &uuid, cookie, i))
continue;
- if (has_uuid_at_pos(nd_region, nd_label->uuid, altcookie, i))
+ if (has_uuid_at_pos(nd_region, &uuid, altcookie, i))
continue;
break;
}
* find a dimm with two instances of the same uuid.
*/
dev_err(&nd_region->dev, "%s missing label for %pUb\n",
- nvdimm_name(nvdimm), nd_label->uuid);
+ nvdimm_name(nvdimm), nsl_uuid_raw(ndd, nd_label));
rc = -EINVAL;
goto err;
}
* the dimm being enabled (i.e. nd_label_reserve_dpa()
* succeeded).
*/
- rc = select_pmem_id(nd_region, nd_label->uuid);
+ nsl_get_uuid(ndd, nd_label, &uuid);
+ rc = select_pmem_id(nd_region, &uuid);
if (rc)
goto err;
WARN_ON(nspm->alt_name || nspm->uuid);
nspm->alt_name = kmemdup(nsl_ref_name(ndd, label0),
NSLABEL_NAME_LEN, GFP_KERNEL);
- nspm->uuid = kmemdup((void __force *) label0->uuid,
- NSLABEL_UUID_LEN, GFP_KERNEL);
+ nsl_get_uuid(ndd, label0, &uuid);
+ nspm->uuid = kmemdup(&uuid, sizeof(uuid_t), GFP_KERNEL);
nspm->lbasize = nsl_get_lbasize(ndd, label0);
nspm->nsio.common.claim_class =
nsl_get_claim_class(ndd, label0);
int i;
for (i = 0; i < count; i++) {
- u8 *uuid = namespace_to_uuid(devs[i]);
+ uuid_t *uuid = namespace_to_uuid(devs[i]);
struct resource *res;
- if (IS_ERR_OR_NULL(uuid)) {
+ if (IS_ERR(uuid)) {
WARN_ON(1);
continue;
}
- if (memcmp(uuid, nd_label->uuid, NSLABEL_UUID_LEN) != 0)
+ if (!nsl_uuid_equal(ndd, nd_label, uuid))
continue;
if (is_namespace_blk(devs[i])) {
res = nsblk_add_resource(nd_region, ndd,
nd_dbg_dpa(nd_region, ndd, res, "%d assign\n", count);
} else {
dev_err(&nd_region->dev,
- "error: conflicting extents for uuid: %pUb\n",
- nd_label->uuid);
+ "error: conflicting extents for uuid: %pUb\n",
+ uuid);
return -ENXIO;
}
break;
char name[NSLABEL_NAME_LEN];
struct device *dev = NULL;
struct resource *res;
+ uuid_t uuid;
if (!nsl_validate_type_guid(ndd, nd_label, &nd_set->type_guid))
return ERR_PTR(-EAGAIN);
dev->parent = &nd_region->dev;
nsblk->id = -1;
nsblk->lbasize = nsl_get_lbasize(ndd, nd_label);
- nsblk->uuid = kmemdup(nd_label->uuid, NSLABEL_UUID_LEN, GFP_KERNEL);
+ nsl_get_uuid(ndd, nd_label, &uuid);
+ nsblk->uuid = kmemdup(&uuid, sizeof(uuid_t), GFP_KERNEL);
nsblk->common.claim_class = nsl_get_claim_class(ndd, nd_label);
if (!nsblk->uuid)
goto blk_err;
void nd_synchronize(void);
void __nd_device_register(struct device *dev);
struct nd_label_id;
-char *nd_label_gen_id(struct nd_label_id *label_id, u8 *uuid, u32 flags);
-bool nd_is_uuid_unique(struct device *dev, u8 *uuid);
+char *nd_label_gen_id(struct nd_label_id *label_id, const uuid_t *uuid,
+ u32 flags);
+bool nd_is_uuid_unique(struct device *dev, uuid_t *uuid);
struct nd_region;
struct nvdimm_drvdata;
struct nd_mapping;
int nslabel_size;
struct nd_cmd_get_config_size nsarea;
void *data;
+ bool cxl;
int ns_current, ns_next;
struct resource dpa;
struct kref kref;
static inline const u8 *nsl_ref_name(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return nd_label->name;
+ if (ndd->cxl)
+ return nd_label->cxl.name;
+ return nd_label->efi.name;
}
static inline u8 *nsl_get_name(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, u8 *name)
{
- return memcpy(name, nd_label->name, NSLABEL_NAME_LEN);
+ if (ndd->cxl)
+ return memcpy(name, nd_label->cxl.name, NSLABEL_NAME_LEN);
+ return memcpy(name, nd_label->efi.name, NSLABEL_NAME_LEN);
}
static inline u8 *nsl_set_name(struct nvdimm_drvdata *ndd,
{
if (!name)
return NULL;
- return memcpy(nd_label->name, name, NSLABEL_NAME_LEN);
+ if (ndd->cxl)
+ return memcpy(nd_label->cxl.name, name, NSLABEL_NAME_LEN);
+ return memcpy(nd_label->efi.name, name, NSLABEL_NAME_LEN);
}
static inline u32 nsl_get_slot(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le32_to_cpu(nd_label->slot);
+ if (ndd->cxl)
+ return __le32_to_cpu(nd_label->cxl.slot);
+ return __le32_to_cpu(nd_label->efi.slot);
}
static inline void nsl_set_slot(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, u32 slot)
{
- nd_label->slot = __cpu_to_le32(slot);
+ if (ndd->cxl)
+ nd_label->cxl.slot = __cpu_to_le32(slot);
+ else
+ nd_label->efi.slot = __cpu_to_le32(slot);
}
static inline u64 nsl_get_checksum(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le64_to_cpu(nd_label->checksum);
+ if (ndd->cxl)
+ return __le64_to_cpu(nd_label->cxl.checksum);
+ return __le64_to_cpu(nd_label->efi.checksum);
}
static inline void nsl_set_checksum(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u64 checksum)
{
- nd_label->checksum = __cpu_to_le64(checksum);
+ if (ndd->cxl)
+ nd_label->cxl.checksum = __cpu_to_le64(checksum);
+ else
+ nd_label->efi.checksum = __cpu_to_le64(checksum);
}
static inline u32 nsl_get_flags(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le32_to_cpu(nd_label->flags);
+ if (ndd->cxl)
+ return __le32_to_cpu(nd_label->cxl.flags);
+ return __le32_to_cpu(nd_label->efi.flags);
}
static inline void nsl_set_flags(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, u32 flags)
{
- nd_label->flags = __cpu_to_le32(flags);
+ if (ndd->cxl)
+ nd_label->cxl.flags = __cpu_to_le32(flags);
+ else
+ nd_label->efi.flags = __cpu_to_le32(flags);
}
static inline u64 nsl_get_dpa(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le64_to_cpu(nd_label->dpa);
+ if (ndd->cxl)
+ return __le64_to_cpu(nd_label->cxl.dpa);
+ return __le64_to_cpu(nd_label->efi.dpa);
}
static inline void nsl_set_dpa(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label, u64 dpa)
{
- nd_label->dpa = __cpu_to_le64(dpa);
+ if (ndd->cxl)
+ nd_label->cxl.dpa = __cpu_to_le64(dpa);
+ else
+ nd_label->efi.dpa = __cpu_to_le64(dpa);
}
static inline u64 nsl_get_rawsize(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le64_to_cpu(nd_label->rawsize);
+ if (ndd->cxl)
+ return __le64_to_cpu(nd_label->cxl.rawsize);
+ return __le64_to_cpu(nd_label->efi.rawsize);
}
static inline void nsl_set_rawsize(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u64 rawsize)
{
- nd_label->rawsize = __cpu_to_le64(rawsize);
+ if (ndd->cxl)
+ nd_label->cxl.rawsize = __cpu_to_le64(rawsize);
+ else
+ nd_label->efi.rawsize = __cpu_to_le64(rawsize);
}
static inline u64 nsl_get_isetcookie(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le64_to_cpu(nd_label->isetcookie);
+ /* WARN future refactor attempts that break this assumption */
+ if (dev_WARN_ONCE(ndd->dev, ndd->cxl,
+ "CXL labels do not use the isetcookie concept\n"))
+ return 0;
+ return __le64_to_cpu(nd_label->efi.isetcookie);
}
static inline void nsl_set_isetcookie(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u64 isetcookie)
{
- nd_label->isetcookie = __cpu_to_le64(isetcookie);
+ if (!ndd->cxl)
+ nd_label->efi.isetcookie = __cpu_to_le64(isetcookie);
}
static inline bool nsl_validate_isetcookie(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u64 cookie)
{
- return cookie == __le64_to_cpu(nd_label->isetcookie);
+ /*
+ * Let the EFI and CXL validation comingle, where fields that
+ * don't matter to CXL always validate.
+ */
+ if (ndd->cxl)
+ return true;
+ return cookie == __le64_to_cpu(nd_label->efi.isetcookie);
}
static inline u16 nsl_get_position(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le16_to_cpu(nd_label->position);
+ if (ndd->cxl)
+ return __le16_to_cpu(nd_label->cxl.position);
+ return __le16_to_cpu(nd_label->efi.position);
}
static inline void nsl_set_position(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u16 position)
{
- nd_label->position = __cpu_to_le16(position);
+ if (ndd->cxl)
+ nd_label->cxl.position = __cpu_to_le16(position);
+ else
+ nd_label->efi.position = __cpu_to_le16(position);
}
-
static inline u16 nsl_get_nlabel(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le16_to_cpu(nd_label->nlabel);
+ if (ndd->cxl)
+ return 0;
+ return __le16_to_cpu(nd_label->efi.nlabel);
}
static inline void nsl_set_nlabel(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u16 nlabel)
{
- nd_label->nlabel = __cpu_to_le16(nlabel);
+ if (!ndd->cxl)
+ nd_label->efi.nlabel = __cpu_to_le16(nlabel);
+}
+
+static inline u16 nsl_get_nrange(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label)
+{
+ if (ndd->cxl)
+ return __le16_to_cpu(nd_label->cxl.nrange);
+ return 1;
+}
+
+static inline void nsl_set_nrange(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label,
+ u16 nrange)
+{
+ if (ndd->cxl)
+ nd_label->cxl.nrange = __cpu_to_le16(nrange);
}
static inline u64 nsl_get_lbasize(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label)
{
- return __le64_to_cpu(nd_label->lbasize);
+ /*
+ * Yes, for some reason the EFI labels convey a massive 64-bit
+ * lbasize, that got fixed for CXL.
+ */
+ if (ndd->cxl)
+ return __le16_to_cpu(nd_label->cxl.lbasize);
+ return __le64_to_cpu(nd_label->efi.lbasize);
}
static inline void nsl_set_lbasize(struct nvdimm_drvdata *ndd,
struct nd_namespace_label *nd_label,
u64 lbasize)
{
- nd_label->lbasize = __cpu_to_le64(lbasize);
+ if (ndd->cxl)
+ nd_label->cxl.lbasize = __cpu_to_le16(lbasize);
+ else
+ nd_label->efi.lbasize = __cpu_to_le64(lbasize);
+}
+
+static inline const uuid_t *nsl_get_uuid(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label,
+ uuid_t *uuid)
+{
+ if (ndd->cxl)
+ import_uuid(uuid, nd_label->cxl.uuid);
+ else
+ import_uuid(uuid, nd_label->efi.uuid);
+ return uuid;
+}
+
+static inline const uuid_t *nsl_set_uuid(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label,
+ const uuid_t *uuid)
+{
+ if (ndd->cxl)
+ export_uuid(nd_label->cxl.uuid, uuid);
+ else
+ export_uuid(nd_label->efi.uuid, uuid);
+ return uuid;
+}
+
+static inline bool nsl_uuid_equal(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label,
+ const uuid_t *uuid)
+{
+ uuid_t tmp;
+
+ if (ndd->cxl)
+ import_uuid(&tmp, nd_label->cxl.uuid);
+ else
+ import_uuid(&tmp, nd_label->efi.uuid);
+ return uuid_equal(&tmp, uuid);
+}
+
+static inline const u8 *nsl_uuid_raw(struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label)
+{
+ if (ndd->cxl)
+ return nd_label->cxl.uuid;
+ return nd_label->efi.uuid;
}
bool nsl_validate_blk_isetcookie(struct nvdimm_drvdata *ndd,
unsigned sizeof_namespace_label(struct nvdimm_drvdata *ndd);
-#define namespace_label_has(ndd, field) \
- (offsetof(struct nd_namespace_label, field) \
+#define efi_namespace_label_has(ndd, field) \
+ (!ndd->cxl && offsetof(struct nvdimm_efi_label, field) \
< sizeof_namespace_label(ndd))
#define nd_dbg_dpa(r, d, res, fmt, arg...) \
struct nd_mapping mapping[];
};
+static inline bool nsl_validate_nlabel(struct nd_region *nd_region,
+ struct nvdimm_drvdata *ndd,
+ struct nd_namespace_label *nd_label)
+{
+ if (ndd->cxl)
+ return true;
+ return nsl_get_nlabel(ndd, nd_label) == nd_region->ndr_mappings;
+}
+
struct nd_blk_region {
int (*enable)(struct nvdimm_bus *nvdimm_bus, struct device *dev);
int (*do_io)(struct nd_blk_region *ndbr, resource_size_t dpa,
struct btt *btt;
unsigned long lbasize;
u64 size;
- u8 *uuid;
+ uuid_t *uuid;
int id;
int initial_offset;
u16 version_major;
struct nd_pfn {
int id;
- u8 *uuid;
+ uuid_t *uuid;
struct device dev;
unsigned long align;
unsigned long npfns;
void nd_device_register(struct device *dev);
void nd_device_unregister(struct device *dev, enum nd_async_mode mode);
void nd_device_notify(struct device *dev, enum nvdimm_event event);
-int nd_uuid_store(struct device *dev, u8 **uuid_out, const char *buf,
+int nd_uuid_store(struct device *dev, uuid_t **uuid_out, const char *buf,
size_t len);
ssize_t nd_size_select_show(unsigned long current_size,
const unsigned long *supported, char *buf);
return false;
}
resource_size_t nd_namespace_blk_validate(struct nd_namespace_blk *nsblk);
-const u8 *nd_dev_to_uuid(struct device *dev);
+const uuid_t *nd_dev_to_uuid(struct device *dev);
bool pmem_should_map_pages(struct device *dev);
#endif /* __ND_H__ */
unsigned long align, start_pad;
struct nd_pfn_sb *pfn_sb = nd_pfn->pfn_sb;
struct nd_namespace_common *ndns = nd_pfn->ndns;
- const u8 *parent_uuid = nd_dev_to_uuid(&ndns->dev);
+ const uuid_t *parent_uuid = nd_dev_to_uuid(&ndns->dev);
if (!pfn_sb || !ndns)
return -ENODEV;
.zero_page_range = pmem_dax_zero_page_range,
};
+static ssize_t write_cache_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct pmem_device *pmem = dev_to_disk(dev)->private_data;
+
+ return sprintf(buf, "%d\n", !!dax_write_cache_enabled(pmem->dax_dev));
+}
+
+static ssize_t write_cache_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct pmem_device *pmem = dev_to_disk(dev)->private_data;
+ bool write_cache;
+ int rc;
+
+ rc = strtobool(buf, &write_cache);
+ if (rc)
+ return rc;
+ dax_write_cache(pmem->dax_dev, write_cache);
+ return len;
+}
+static DEVICE_ATTR_RW(write_cache);
+
+static umode_t dax_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+#ifndef CONFIG_ARCH_HAS_PMEM_API
+ if (a == &dev_attr_write_cache.attr)
+ return 0;
+#endif
+ return a->mode;
+}
+
+static struct attribute *dax_attributes[] = {
+ &dev_attr_write_cache.attr,
+ NULL,
+};
+
+static const struct attribute_group dax_attribute_group = {
+ .name = "dax",
+ .attrs = dax_attributes,
+ .is_visible = dax_visible,
+};
+
static const struct attribute_group *pmem_attribute_groups[] = {
&dax_attribute_group,
NULL,
bb_range.end = res->end;
}
- if (IS_ERR(addr))
- return PTR_ERR(addr);
+ if (IS_ERR(addr)) {
+ rc = PTR_ERR(addr);
+ goto out;
+ }
pmem->virt_addr = addr;
blk_queue_write_cache(q, true, fua);
flags = DAXDEV_F_SYNC;
dax_dev = alloc_dax(pmem, disk->disk_name, &pmem_dax_ops, flags);
if (IS_ERR(dax_dev)) {
- return PTR_ERR(dax_dev);
+ rc = PTR_ERR(dax_dev);
+ goto out;
}
dax_write_cache(dax_dev, nvdimm_has_cache(nd_region));
pmem->dax_dev = dax_dev;
- device_add_disk(dev, disk, pmem_attribute_groups);
+ rc = device_add_disk(dev, disk, pmem_attribute_groups);
+ if (rc)
+ goto out_cleanup_dax;
if (devm_add_action_or_reset(dev, pmem_release_disk, pmem))
return -ENOMEM;
"badblocks");
if (!pmem->bb_state)
dev_warn(dev, "'badblocks' notification disabled\n");
-
return 0;
+
+out_cleanup_dax:
+ kill_dax(pmem->dax_dev);
+ put_dax(pmem->dax_dev);
+out:
+ blk_cleanup_disk(pmem->disk);
+ return rc;
}
static int nd_pmem_probe(struct device *dev)
{
if (!test_and_set_bit(NVME_NS_STOPPED, &ns->flags))
blk_mq_quiesce_queue(ns->queue);
+ else
+ blk_mq_wait_quiesce_done(ns->queue);
}
/*
{
if (!test_and_set_bit(NVME_CTRL_ADMIN_Q_STOPPED, &ctrl->flags))
blk_mq_quiesce_queue(ctrl->admin_q);
+ else
+ blk_mq_wait_quiesce_done(ctrl->admin_q);
}
EXPORT_SYMBOL_GPL(nvme_stop_admin_queue);
/* Now start the actual "proper" walk of the interrupt tree */
while (ipar != NULL) {
- /* Now check if cursor is an interrupt-controller and if it is
- * then we are done
+ /*
+ * Now check if cursor is an interrupt-controller and
+ * if it is then we are done, unless there is an
+ * interrupt-map which takes precedence.
*/
- if (of_property_read_bool(ipar, "interrupt-controller")) {
+ bool intc = of_property_read_bool(ipar, "interrupt-controller");
+
+ imap = of_get_property(ipar, "interrupt-map", &imaplen);
+ if (imap == NULL && intc) {
pr_debug(" -> got it !\n");
return 0;
}
goto fail;
}
- /* Now look for an interrupt-map */
- imap = of_get_property(ipar, "interrupt-map", &imaplen);
/* No interrupt map, check for an interrupt parent */
if (imap == NULL) {
pr_debug(" -> no map, getting parent\n");
pr_debug(" -> imaplen=%d\n", imaplen);
}
- if (!match)
+ if (!match) {
+ if (intc) {
+ /*
+ * The PASEMI Nemo is a known offender, so
+ * let's only warn for anyone else.
+ */
+ WARN(!IS_ENABLED(CONFIG_PPC_PASEMI),
+ "%pOF interrupt-map failed, using interrupt-controller\n",
+ ipar);
+ return 0;
+ }
+
goto fail;
+ }
/*
* Successfully parsed an interrrupt-map translation; copy new
out_irq->args_count = intsize = newintsize;
addrsize = newaddrsize;
+ if (ipar == newpar) {
+ pr_debug("%pOF interrupt-map entry to self\n", ipar);
+ return 0;
+ }
+
skiplevel:
/* Iterate again with new parent */
out_irq->np = newpar;
#include <linux/of.h>
#include <linux/of_fdt.h>
#include <linux/random.h>
+#include <linux/slab.h>
#include <linux/types.h>
#define RNG_SEED_SIZE 128
if (ret)
return ret;
- return memblock_free(addr, size);
-
+ return memblock_phys_free(addr, size);
}
/**
if (nomap) {
err = memblock_mark_nomap(base, size);
if (err)
- memblock_free(base, size);
+ memblock_phys_free(base, size);
kmemleak_ignore_phys(base);
}
if (nomap)
memblock_clear_nomap(rmem->base, rmem->size);
else
- memblock_free(rmem->base, rmem->size);
+ memblock_phys_free(rmem->base,
+ rmem->size);
}
}
}
struct device_node *node = dev->of_node;
const __be32 *reg;
u64 addr;
+ u32 mask;
/* Construct the name, using parent nodes if necessary to ensure uniqueness */
while (node->parent) {
*/
reg = of_get_property(node, "reg", NULL);
if (reg && (addr = of_translate_address(node, reg)) != OF_BAD_ADDR) {
- dev_set_name(dev, dev_name(dev) ? "%llx.%pOFn:%s" : "%llx.%pOFn",
- addr, node, dev_name(dev));
+ if (!of_property_read_u32(node, "mask", &mask))
+ dev_set_name(dev, dev_name(dev) ? "%llx.%x.%pOFn:%s" : "%llx.%x.%pOFn",
+ addr, ffs(mask) - 1, node, dev_name(dev));
+
+ else
+ dev_set_name(dev, dev_name(dev) ? "%llx.%pOFn:%s" : "%llx.%pOFn",
+ addr, node, dev_name(dev));
return;
}
* "required-opps" are added in DT.
*/
struct opp_table *dev_pm_opp_attach_genpd(struct device *dev,
- const char **names, struct device ***virt_devs)
+ const char * const *names, struct device ***virt_devs)
{
struct opp_table *opp_table;
struct device *virt_dev;
int index = 0, ret = -EINVAL;
- const char **name = names;
+ const char * const *name = names;
opp_table = _add_opp_table(dev, false);
if (IS_ERR(opp_table))
*
* Return: 0 on success and errorno otherwise.
*/
-int devm_pm_opp_attach_genpd(struct device *dev, const char **names,
+int devm_pm_opp_attach_genpd(struct device *dev, const char * const *names,
struct device ***virt_devs)
{
struct opp_table *opp_table;
}
count = of_count_phandle_with_args(np, "required-opps", NULL);
- if (!count)
+ if (count <= 0)
goto put_np;
required_opp_tables = kcalloc(count, sizeof(*required_opp_tables),
free_opp:
_opp_free(new_opp);
- return ERR_PTR(ret);
+ return ret ? ERR_PTR(ret) : NULL;
}
/* Initializes OPP tables based on new bindings */
dev_pm_opp_of_remove_table(data);
}
+static int _devm_of_add_table_indexed(struct device *dev, int index, bool getclk)
+{
+ int ret;
+
+ ret = _of_add_table_indexed(dev, index, getclk);
+ if (ret)
+ return ret;
+
+ return devm_add_action_or_reset(dev, devm_pm_opp_of_table_release, dev);
+}
+
/**
* devm_pm_opp_of_add_table() - Initialize opp table from device tree
* @dev: device pointer used to lookup OPP table.
*/
int devm_pm_opp_of_add_table(struct device *dev)
{
- int ret;
-
- ret = dev_pm_opp_of_add_table(dev);
- if (ret)
- return ret;
-
- return devm_add_action_or_reset(dev, devm_pm_opp_of_table_release, dev);
+ return _devm_of_add_table_indexed(dev, 0, true);
}
EXPORT_SYMBOL_GPL(devm_pm_opp_of_add_table);
EXPORT_SYMBOL_GPL(dev_pm_opp_of_add_table_indexed);
/**
+ * devm_pm_opp_of_add_table_indexed() - Initialize indexed opp table from device tree
+ * @dev: device pointer used to lookup OPP table.
+ * @index: Index number.
+ *
+ * This is a resource-managed variant of dev_pm_opp_of_add_table_indexed().
+ */
+int devm_pm_opp_of_add_table_indexed(struct device *dev, int index)
+{
+ return _devm_of_add_table_indexed(dev, index, true);
+}
+EXPORT_SYMBOL_GPL(devm_pm_opp_of_add_table_indexed);
+
+/**
* dev_pm_opp_of_add_table_noclk() - Initialize indexed opp table from device
* tree without getting clk for device.
* @dev: device pointer used to lookup OPP table.
}
EXPORT_SYMBOL_GPL(dev_pm_opp_of_add_table_noclk);
+/**
+ * devm_pm_opp_of_add_table_noclk() - Initialize indexed opp table from device
+ * tree without getting clk for device.
+ * @dev: device pointer used to lookup OPP table.
+ * @index: Index number.
+ *
+ * This is a resource-managed variant of dev_pm_opp_of_add_table_noclk().
+ */
+int devm_pm_opp_of_add_table_noclk(struct device *dev, int index)
+{
+ return _devm_of_add_table_indexed(dev, index, false);
+}
+EXPORT_SYMBOL_GPL(devm_pm_opp_of_add_table_noclk);
+
/* CPU device specific helpers */
/**
MediaTek SoCs.
config VMD
- depends on PCI_MSI && X86_64 && SRCU
+ depends on PCI_MSI && X86_64 && SRCU && !UML
tristate "Intel Volume Management Device Driver"
help
Adds support for the Intel Volume Management Device (VMD). VMD is a
config PCIE_BRCMSTB
tristate "Broadcom Brcmstb PCIe host controller"
- depends on ARCH_BRCMSTB || ARCH_BCM2835 || ARCH_BCM4908 || COMPILE_TEST
+ depends on ARCH_BRCMSTB || ARCH_BCM2835 || ARCH_BCM4908 || \
+ BMIPS_GENERIC || COMPILE_TEST
depends on OF
depends on PCI_MSI_IRQ_DOMAIN
default ARCH_BRCMSTB
Say Y here if you want error handling support
for the PCIe controller's errors on HiSilicon HIP SoCs
+config PCIE_APPLE_MSI_DOORBELL_ADDR
+ hex
+ default 0xfffff000
+ depends on PCIE_APPLE
+
+config PCIE_APPLE
+ tristate "Apple PCIe controller"
+ depends on ARCH_APPLE || COMPILE_TEST
+ depends on OF
+ depends on PCI_MSI_IRQ_DOMAIN
+ select PCI_HOST_COMMON
+ help
+ Say Y here if you want to enable PCIe controller support on Apple
+ system-on-chips, like the Apple M1. This is required for the USB
+ type-A ports, Ethernet, Wi-Fi, and Bluetooth.
+
+ If unsure, say Y if you have an Apple Silicon system.
+
+config PCIE_MT7621
+ tristate "MediaTek MT7621 PCIe Controller"
+ depends on (RALINK && SOC_MT7621) || (MIPS && COMPILE_TEST)
+ select PHY_MT7621_PCI
+ default SOC_MT7621
+ help
+ This selects a driver for the MediaTek MT7621 PCIe Controller.
+
source "drivers/pci/controller/dwc/Kconfig"
source "drivers/pci/controller/mobiveil/Kconfig"
source "drivers/pci/controller/cadence/Kconfig"
obj-$(CONFIG_PCIE_BRCMSTB) += pcie-brcmstb.o
obj-$(CONFIG_PCI_LOONGSON) += pci-loongson.o
obj-$(CONFIG_PCIE_HISI_ERR) += pcie-hisi-error.o
+obj-$(CONFIG_PCIE_APPLE) += pcie-apple.o
+obj-$(CONFIG_PCIE_MT7621) += pcie-mt7621.o
+
# pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
obj-y += dwc/
obj-y += mobiveil/
ret = clk_prepare_enable(clk);
if (ret) {
dev_err(dev, "failed to enable pcie_refclk\n");
- goto err_get_sync;
+ goto err_pcie_setup;
}
pcie->refclk = clk;
goto err_init;
}
+ return 0;
+
err_init:
err_get_sync:
pm_runtime_put_sync(dev);
config PCIE_DW_HOST
bool
- depends on PCI_MSI_IRQ_DOMAIN
select PCIE_DW
config PCIE_DW_EP
bool
- depends on PCI_ENDPOINT
select PCIE_DW
config PCI_DRA7XX
- bool
+ tristate
config PCI_DRA7XX_HOST
- bool "TI DRA7xx PCIe controller Host Mode"
+ tristate "TI DRA7xx PCIe controller Host Mode"
depends on SOC_DRA7XX || COMPILE_TEST
- depends on PCI_MSI_IRQ_DOMAIN
depends on OF && HAS_IOMEM && TI_PIPE3
+ depends on PCI_MSI_IRQ_DOMAIN
select PCIE_DW_HOST
select PCI_DRA7XX
default y if SOC_DRA7XX
This uses the DesignWare core.
config PCI_DRA7XX_EP
- bool "TI DRA7xx PCIe controller Endpoint Mode"
+ tristate "TI DRA7xx PCIe controller Endpoint Mode"
depends on SOC_DRA7XX || COMPILE_TEST
- depends on PCI_ENDPOINT
depends on OF && HAS_IOMEM && TI_PIPE3
+ depends on PCI_ENDPOINT
select PCIE_DW_EP
select PCI_DRA7XX
help
config PCIE_DW_PLAT_HOST
bool "Platform bus based DesignWare PCIe Controller - Host mode"
- depends on PCI && PCI_MSI_IRQ_DOMAIN
+ depends on PCI_MSI_IRQ_DOMAIN
select PCIE_DW_HOST
select PCIE_DW_PLAT
help
bool "Freescale Layerscape PCIe controller - Host mode"
depends on OF && (ARM || ARCH_LAYERSCAPE || COMPILE_TEST)
depends on PCI_MSI_IRQ_DOMAIN
- select MFD_SYSCON
select PCIE_DW_HOST
+ select MFD_SYSCON
help
Say Y here if you want to enable PCIe controller support on Layerscape
SoCs to work in Host mode.
PCIe controller uses the DesignWare core plus Qualcomm-specific
hardware wrappers.
+config PCIE_QCOM_EP
+ tristate "Qualcomm PCIe controller - Endpoint mode"
+ depends on OF && (ARCH_QCOM || COMPILE_TEST)
+ depends on PCI_ENDPOINT
+ select PCIE_DW_EP
+ help
+ Say Y here to enable support for the PCIe controllers on Qualcomm SoCs
+ to work in endpoint mode. The PCIe controller uses the DesignWare core
+ plus Qualcomm-specific hardware wrappers.
+
config PCIE_ARMADA_8K
bool "Marvell Armada-8K PCIe controller"
depends on ARCH_MVEBU || COMPILE_TEST
config PCIE_KIRIN
depends on OF && (ARM64 || COMPILE_TEST)
- bool "HiSilicon Kirin series SoCs PCIe controllers"
+ tristate "HiSilicon Kirin series SoCs PCIe controllers"
depends on PCI_MSI_IRQ_DOMAIN
select PCIE_DW_HOST
help
config PCI_MESON
tristate "MESON PCIe controller"
- depends on PCI_MSI_IRQ_DOMAIN
default m if ARCH_MESON
+ depends on PCI_MSI_IRQ_DOMAIN
select PCIE_DW_HOST
help
Say Y here if you want to enable PCI controller support on Amlogic
obj-$(CONFIG_PCI_LAYERSCAPE) += pci-layerscape.o
obj-$(CONFIG_PCI_LAYERSCAPE_EP) += pci-layerscape-ep.o
obj-$(CONFIG_PCIE_QCOM) += pcie-qcom.o
+obj-$(CONFIG_PCIE_QCOM_EP) += pcie-qcom-ep.o
obj-$(CONFIG_PCIE_ARMADA_8K) += pcie-armada8k.o
obj-$(CONFIG_PCIE_ARTPEC6) += pcie-artpec6.o
obj-$(CONFIG_PCIE_ROCKCHIP_DW_HOST) += pcie-dw-rockchip.o
* Authors: Kishon Vijay Abraham I <kishon@ti.com>
*/
+#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/device.h>
#include <linux/err.h>
#include <linux/irq.h>
#include <linux/irqdomain.h>
#include <linux/kernel.h>
-#include <linux/init.h>
+#include <linux/module.h>
#include <linux/of_device.h>
#include <linux/of_gpio.h>
#include <linux/of_pci.h>
int phy_count; /* DT phy-names count */
struct phy **phy;
struct irq_domain *irq_domain;
+ struct clk *clk;
enum dw_pcie_device_mode mode;
};
},
{},
};
+MODULE_DEVICE_TABLE(of, of_dra7xx_pcie_match);
/*
* dra7xx_pcie_unaligned_memaccess: workaround for AM572x/AM571x Errata i870
if (!link)
return -ENOMEM;
+ dra7xx->clk = devm_clk_get_optional(dev, NULL);
+ if (IS_ERR(dra7xx->clk))
+ return dev_err_probe(dev, PTR_ERR(dra7xx->clk),
+ "clock request failed");
+
+ ret = clk_prepare_enable(dra7xx->clk);
+ if (ret)
+ return ret;
+
for (i = 0; i < phy_count; i++) {
snprintf(name, sizeof(name), "pcie-phy%d", i);
phy[i] = devm_phy_get(dev, name);
pm_runtime_disable(dev);
dra7xx_pcie_disable_phy(dra7xx);
+
+ clk_disable_unprepare(dra7xx->clk);
}
static const struct dev_pm_ops dra7xx_pcie_pm_ops = {
},
.shutdown = dra7xx_pcie_shutdown,
};
-builtin_platform_driver(dra7xx_pcie_driver);
+module_platform_driver(dra7xx_pcie_driver);
+
+MODULE_AUTHOR("Kishon Vijay Abraham I <kishon@ti.com>");
+MODULE_DESCRIPTION("PCIe controller driver for TI DRA7xx SoCs");
+MODULE_LICENSE("GPL v2");
/* Limit link speed */
pci->link_gen = 1;
- ret = of_property_read_u32(node, "fsl,max-link-speed", &pci->link_gen);
+ of_property_read_u32(node, "fsl,max-link-speed", &pci->link_gen);
imx6_pcie->vpcie = devm_regulator_get_optional(&pdev->dev, "vpcie");
if (IS_ERR(imx6_pcie->vpcie)) {
for (func_no = 0; func_no < funcs; func_no++)
__dw_pcie_ep_reset_bar(pci, func_no, bar, 0);
}
+EXPORT_SYMBOL_GPL(dw_pcie_ep_reset_bar);
static u8 __dw_pcie_ep_find_next_cap(struct dw_pcie_ep *ep, u8 func_no,
u8 cap_ptr, u8 cap)
return -EINVAL;
}
+EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_legacy_irq);
int dw_pcie_ep_raise_msi_irq(struct dw_pcie_ep *ep, u8 func_no,
u8 interrupt_num)
return 0;
}
+EXPORT_SYMBOL_GPL(dw_pcie_ep_raise_msi_irq);
int dw_pcie_ep_raise_msix_irq_doorbell(struct dw_pcie_ep *ep, u8 func_no,
u16 interrupt_num)
if (pci->link_gen < 1)
pci->link_gen = of_pci_get_max_link_speed(np);
+ /* Set default bus ops */
+ bridge->ops = &dw_pcie_ops;
+ bridge->child_ops = &dw_child_pcie_ops;
+
+ if (pp->ops->host_init) {
+ ret = pp->ops->host_init(pp);
+ if (ret)
+ return ret;
+ }
+
if (pci_msi_enabled()) {
pp->has_msi_ctrl = !(pp->ops->msi_host_init ||
of_property_read_bool(np, "msi-parent") ||
}
}
- /* Set default bus ops */
- bridge->ops = &dw_pcie_ops;
- bridge->child_ops = &dw_child_pcie_ops;
-
- if (pp->ops->host_init) {
- ret = pp->ops->host_init(pp);
- if (ret)
- goto err_free_msi;
- }
dw_pcie_iatu_detect(pci);
dw_pcie_setup_rc(pp);
return ((val & PCIE_PORT_DEBUG1_LINK_UP) &&
(!(val & PCIE_PORT_DEBUG1_LINK_IN_TRAINING)));
}
+EXPORT_SYMBOL_GPL(dw_pcie_link_up);
void dw_pcie_upconfig_setup(struct dw_pcie *pci)
{
* Author: Xiaowei Song <songxiaowei@huawei.com>
*/
-#include <linux/compiler.h>
#include <linux/clk.h>
+#include <linux/compiler.h>
#include <linux/delay.h>
#include <linux/err.h>
#include <linux/gpio.h>
#include <linux/interrupt.h>
#include <linux/mfd/syscon.h>
#include <linux/of_address.h>
+#include <linux/of_device.h>
#include <linux/of_gpio.h>
#include <linux/of_pci.h>
+#include <linux/phy/phy.h>
#include <linux/pci.h>
#include <linux/pci_regs.h>
#include <linux/platform_device.h>
#define to_kirin_pcie(x) dev_get_drvdata((x)->dev)
-#define REF_CLK_FREQ 100000000
-
/* PCIe ELBI registers */
#define SOC_PCIECTRL_CTRL0_ADDR 0x000
#define SOC_PCIECTRL_CTRL1_ADDR 0x004
-#define SOC_PCIEPHY_CTRL2_ADDR 0x008
-#define SOC_PCIEPHY_CTRL3_ADDR 0x00c
#define PCIE_ELBI_SLV_DBI_ENABLE (0x1 << 21)
/* info located in APB */
#define PCIE_APP_LTSSM_ENABLE 0x01c
-#define PCIE_APB_PHY_CTRL0 0x0
-#define PCIE_APB_PHY_CTRL1 0x4
#define PCIE_APB_PHY_STATUS0 0x400
#define PCIE_LINKUP_ENABLE (0x8020)
#define PCIE_LTSSM_ENABLE_BIT (0x1 << 11)
-#define PIPE_CLK_STABLE (0x1 << 19)
-#define PHY_REF_PAD_BIT (0x1 << 8)
-#define PHY_PWR_DOWN_BIT (0x1 << 22)
-#define PHY_RST_ACK_BIT (0x1 << 16)
/* info located in sysctrl */
#define SCTRL_PCIE_CMOS_OFFSET 0x60
#define PCIE_DEBOUNCE_PARAM 0xF0F400
#define PCIE_OE_BYPASS (0x3 << 28)
+/*
+ * Max number of connected PCI slots at an external PCI bridge
+ *
+ * This is used on HiKey 970, which has a PEX 8606 bridge with 4 connected
+ * lanes (lane 0 upstream, and the other three lanes, one connected to an
+ * in-board Ethernet adapter and the other two connected to M.2 and mini
+ * PCI slots.
+ *
+ * Each slot has a different clock source and uses a separate PERST# pin.
+ */
+#define MAX_PCI_SLOTS 3
+
+enum pcie_kirin_phy_type {
+ PCIE_KIRIN_INTERNAL_PHY,
+ PCIE_KIRIN_EXTERNAL_PHY
+};
+
+struct kirin_pcie {
+ enum pcie_kirin_phy_type type;
+
+ struct dw_pcie *pci;
+ struct regmap *apb;
+ struct phy *phy;
+ void *phy_priv; /* only for PCIE_KIRIN_INTERNAL_PHY */
+
+ /* DWC PERST# */
+ int gpio_id_dwc_perst;
+
+ /* Per-slot PERST# */
+ int num_slots;
+ int gpio_id_reset[MAX_PCI_SLOTS];
+ const char *reset_names[MAX_PCI_SLOTS];
+
+ /* Per-slot clkreq */
+ int n_gpio_clkreq;
+ int gpio_id_clkreq[MAX_PCI_SLOTS];
+ const char *clkreq_names[MAX_PCI_SLOTS];
+};
+
+/*
+ * Kirin 960 PHY. Can't be split into a PHY driver without changing the
+ * DT schema.
+ */
+
+#define REF_CLK_FREQ 100000000
+
+/* PHY info located in APB */
+#define PCIE_APB_PHY_CTRL0 0x0
+#define PCIE_APB_PHY_CTRL1 0x4
+#define PCIE_APB_PHY_STATUS0 0x400
+#define PIPE_CLK_STABLE BIT(19)
+#define PHY_REF_PAD_BIT BIT(8)
+#define PHY_PWR_DOWN_BIT BIT(22)
+#define PHY_RST_ACK_BIT BIT(16)
+
/* peri_crg ctrl */
#define CRGCTRL_PCIE_ASSERT_OFFSET 0x88
#define CRGCTRL_PCIE_ASSERT_BIT 0x8c000000
/* Time for delay */
-#define REF_2_PERST_MIN 20000
+#define REF_2_PERST_MIN 21000
#define REF_2_PERST_MAX 25000
#define PERST_2_ACCESS_MIN 10000
#define PERST_2_ACCESS_MAX 12000
-#define LINK_WAIT_MIN 900
-#define LINK_WAIT_MAX 1000
#define PIPE_CLK_WAIT_MIN 550
#define PIPE_CLK_WAIT_MAX 600
#define TIME_CMOS_MIN 100
#define TIME_PHY_PD_MIN 10
#define TIME_PHY_PD_MAX 11
-struct kirin_pcie {
- struct dw_pcie *pci;
- void __iomem *apb_base;
- void __iomem *phy_base;
+struct hi3660_pcie_phy {
+ struct device *dev;
+ void __iomem *base;
struct regmap *crgctrl;
struct regmap *sysctrl;
struct clk *apb_sys_clk;
struct clk *apb_phy_clk;
struct clk *phy_ref_clk;
- struct clk *pcie_aclk;
- struct clk *pcie_aux_clk;
- int gpio_id_reset;
+ struct clk *aclk;
+ struct clk *aux_clk;
};
-/* Registers in PCIeCTRL */
-static inline void kirin_apb_ctrl_writel(struct kirin_pcie *kirin_pcie,
- u32 val, u32 reg)
-{
- writel(val, kirin_pcie->apb_base + reg);
-}
-
-static inline u32 kirin_apb_ctrl_readl(struct kirin_pcie *kirin_pcie, u32 reg)
-{
- return readl(kirin_pcie->apb_base + reg);
-}
-
/* Registers in PCIePHY */
-static inline void kirin_apb_phy_writel(struct kirin_pcie *kirin_pcie,
+static inline void kirin_apb_phy_writel(struct hi3660_pcie_phy *hi3660_pcie_phy,
u32 val, u32 reg)
{
- writel(val, kirin_pcie->phy_base + reg);
+ writel(val, hi3660_pcie_phy->base + reg);
}
-static inline u32 kirin_apb_phy_readl(struct kirin_pcie *kirin_pcie, u32 reg)
+static inline u32 kirin_apb_phy_readl(struct hi3660_pcie_phy *hi3660_pcie_phy,
+ u32 reg)
{
- return readl(kirin_pcie->phy_base + reg);
+ return readl(hi3660_pcie_phy->base + reg);
}
-static long kirin_pcie_get_clk(struct kirin_pcie *kirin_pcie,
- struct platform_device *pdev)
+static int hi3660_pcie_phy_get_clk(struct hi3660_pcie_phy *phy)
{
- struct device *dev = &pdev->dev;
+ struct device *dev = phy->dev;
- kirin_pcie->phy_ref_clk = devm_clk_get(dev, "pcie_phy_ref");
- if (IS_ERR(kirin_pcie->phy_ref_clk))
- return PTR_ERR(kirin_pcie->phy_ref_clk);
+ phy->phy_ref_clk = devm_clk_get(dev, "pcie_phy_ref");
+ if (IS_ERR(phy->phy_ref_clk))
+ return PTR_ERR(phy->phy_ref_clk);
- kirin_pcie->pcie_aux_clk = devm_clk_get(dev, "pcie_aux");
- if (IS_ERR(kirin_pcie->pcie_aux_clk))
- return PTR_ERR(kirin_pcie->pcie_aux_clk);
+ phy->aux_clk = devm_clk_get(dev, "pcie_aux");
+ if (IS_ERR(phy->aux_clk))
+ return PTR_ERR(phy->aux_clk);
- kirin_pcie->apb_phy_clk = devm_clk_get(dev, "pcie_apb_phy");
- if (IS_ERR(kirin_pcie->apb_phy_clk))
- return PTR_ERR(kirin_pcie->apb_phy_clk);
+ phy->apb_phy_clk = devm_clk_get(dev, "pcie_apb_phy");
+ if (IS_ERR(phy->apb_phy_clk))
+ return PTR_ERR(phy->apb_phy_clk);
- kirin_pcie->apb_sys_clk = devm_clk_get(dev, "pcie_apb_sys");
- if (IS_ERR(kirin_pcie->apb_sys_clk))
- return PTR_ERR(kirin_pcie->apb_sys_clk);
+ phy->apb_sys_clk = devm_clk_get(dev, "pcie_apb_sys");
+ if (IS_ERR(phy->apb_sys_clk))
+ return PTR_ERR(phy->apb_sys_clk);
- kirin_pcie->pcie_aclk = devm_clk_get(dev, "pcie_aclk");
- if (IS_ERR(kirin_pcie->pcie_aclk))
- return PTR_ERR(kirin_pcie->pcie_aclk);
+ phy->aclk = devm_clk_get(dev, "pcie_aclk");
+ if (IS_ERR(phy->aclk))
+ return PTR_ERR(phy->aclk);
return 0;
}
-static long kirin_pcie_get_resource(struct kirin_pcie *kirin_pcie,
- struct platform_device *pdev)
+static int hi3660_pcie_phy_get_resource(struct hi3660_pcie_phy *phy)
{
- kirin_pcie->apb_base =
- devm_platform_ioremap_resource_byname(pdev, "apb");
- if (IS_ERR(kirin_pcie->apb_base))
- return PTR_ERR(kirin_pcie->apb_base);
-
- kirin_pcie->phy_base =
- devm_platform_ioremap_resource_byname(pdev, "phy");
- if (IS_ERR(kirin_pcie->phy_base))
- return PTR_ERR(kirin_pcie->phy_base);
-
- kirin_pcie->crgctrl =
- syscon_regmap_lookup_by_compatible("hisilicon,hi3660-crgctrl");
- if (IS_ERR(kirin_pcie->crgctrl))
- return PTR_ERR(kirin_pcie->crgctrl);
-
- kirin_pcie->sysctrl =
- syscon_regmap_lookup_by_compatible("hisilicon,hi3660-sctrl");
- if (IS_ERR(kirin_pcie->sysctrl))
- return PTR_ERR(kirin_pcie->sysctrl);
+ struct device *dev = phy->dev;
+ struct platform_device *pdev;
+
+ /* registers */
+ pdev = container_of(dev, struct platform_device, dev);
+
+ phy->base = devm_platform_ioremap_resource_byname(pdev, "phy");
+ if (IS_ERR(phy->base))
+ return PTR_ERR(phy->base);
+
+ phy->crgctrl = syscon_regmap_lookup_by_compatible("hisilicon,hi3660-crgctrl");
+ if (IS_ERR(phy->crgctrl))
+ return PTR_ERR(phy->crgctrl);
+
+ phy->sysctrl = syscon_regmap_lookup_by_compatible("hisilicon,hi3660-sctrl");
+ if (IS_ERR(phy->sysctrl))
+ return PTR_ERR(phy->sysctrl);
return 0;
}
-static int kirin_pcie_phy_init(struct kirin_pcie *kirin_pcie)
+static int hi3660_pcie_phy_start(struct hi3660_pcie_phy *phy)
{
- struct device *dev = kirin_pcie->pci->dev;
+ struct device *dev = phy->dev;
u32 reg_val;
- reg_val = kirin_apb_phy_readl(kirin_pcie, PCIE_APB_PHY_CTRL1);
+ reg_val = kirin_apb_phy_readl(phy, PCIE_APB_PHY_CTRL1);
reg_val &= ~PHY_REF_PAD_BIT;
- kirin_apb_phy_writel(kirin_pcie, reg_val, PCIE_APB_PHY_CTRL1);
+ kirin_apb_phy_writel(phy, reg_val, PCIE_APB_PHY_CTRL1);
- reg_val = kirin_apb_phy_readl(kirin_pcie, PCIE_APB_PHY_CTRL0);
+ reg_val = kirin_apb_phy_readl(phy, PCIE_APB_PHY_CTRL0);
reg_val &= ~PHY_PWR_DOWN_BIT;
- kirin_apb_phy_writel(kirin_pcie, reg_val, PCIE_APB_PHY_CTRL0);
+ kirin_apb_phy_writel(phy, reg_val, PCIE_APB_PHY_CTRL0);
usleep_range(TIME_PHY_PD_MIN, TIME_PHY_PD_MAX);
- reg_val = kirin_apb_phy_readl(kirin_pcie, PCIE_APB_PHY_CTRL1);
+ reg_val = kirin_apb_phy_readl(phy, PCIE_APB_PHY_CTRL1);
reg_val &= ~PHY_RST_ACK_BIT;
- kirin_apb_phy_writel(kirin_pcie, reg_val, PCIE_APB_PHY_CTRL1);
+ kirin_apb_phy_writel(phy, reg_val, PCIE_APB_PHY_CTRL1);
usleep_range(PIPE_CLK_WAIT_MIN, PIPE_CLK_WAIT_MAX);
- reg_val = kirin_apb_phy_readl(kirin_pcie, PCIE_APB_PHY_STATUS0);
+ reg_val = kirin_apb_phy_readl(phy, PCIE_APB_PHY_STATUS0);
if (reg_val & PIPE_CLK_STABLE) {
dev_err(dev, "PIPE clk is not stable\n");
return -EINVAL;
return 0;
}
-static void kirin_pcie_oe_enable(struct kirin_pcie *kirin_pcie)
+static void hi3660_pcie_phy_oe_enable(struct hi3660_pcie_phy *phy)
{
u32 val;
- regmap_read(kirin_pcie->sysctrl, SCTRL_PCIE_OE_OFFSET, &val);
+ regmap_read(phy->sysctrl, SCTRL_PCIE_OE_OFFSET, &val);
val |= PCIE_DEBOUNCE_PARAM;
val &= ~PCIE_OE_BYPASS;
- regmap_write(kirin_pcie->sysctrl, SCTRL_PCIE_OE_OFFSET, val);
+ regmap_write(phy->sysctrl, SCTRL_PCIE_OE_OFFSET, val);
}
-static int kirin_pcie_clk_ctrl(struct kirin_pcie *kirin_pcie, bool enable)
+static int hi3660_pcie_phy_clk_ctrl(struct hi3660_pcie_phy *phy, bool enable)
{
int ret = 0;
if (!enable)
goto close_clk;
- ret = clk_set_rate(kirin_pcie->phy_ref_clk, REF_CLK_FREQ);
+ ret = clk_set_rate(phy->phy_ref_clk, REF_CLK_FREQ);
if (ret)
return ret;
- ret = clk_prepare_enable(kirin_pcie->phy_ref_clk);
+ ret = clk_prepare_enable(phy->phy_ref_clk);
if (ret)
return ret;
- ret = clk_prepare_enable(kirin_pcie->apb_sys_clk);
+ ret = clk_prepare_enable(phy->apb_sys_clk);
if (ret)
goto apb_sys_fail;
- ret = clk_prepare_enable(kirin_pcie->apb_phy_clk);
+ ret = clk_prepare_enable(phy->apb_phy_clk);
if (ret)
goto apb_phy_fail;
- ret = clk_prepare_enable(kirin_pcie->pcie_aclk);
+ ret = clk_prepare_enable(phy->aclk);
if (ret)
goto aclk_fail;
- ret = clk_prepare_enable(kirin_pcie->pcie_aux_clk);
+ ret = clk_prepare_enable(phy->aux_clk);
if (ret)
goto aux_clk_fail;
return 0;
close_clk:
- clk_disable_unprepare(kirin_pcie->pcie_aux_clk);
+ clk_disable_unprepare(phy->aux_clk);
aux_clk_fail:
- clk_disable_unprepare(kirin_pcie->pcie_aclk);
+ clk_disable_unprepare(phy->aclk);
aclk_fail:
- clk_disable_unprepare(kirin_pcie->apb_phy_clk);
+ clk_disable_unprepare(phy->apb_phy_clk);
apb_phy_fail:
- clk_disable_unprepare(kirin_pcie->apb_sys_clk);
+ clk_disable_unprepare(phy->apb_sys_clk);
apb_sys_fail:
- clk_disable_unprepare(kirin_pcie->phy_ref_clk);
+ clk_disable_unprepare(phy->phy_ref_clk);
return ret;
}
-static int kirin_pcie_power_on(struct kirin_pcie *kirin_pcie)
+static int hi3660_pcie_phy_power_on(struct kirin_pcie *pcie)
{
+ struct hi3660_pcie_phy *phy = pcie->phy_priv;
int ret;
/* Power supply for Host */
- regmap_write(kirin_pcie->sysctrl,
+ regmap_write(phy->sysctrl,
SCTRL_PCIE_CMOS_OFFSET, SCTRL_PCIE_CMOS_BIT);
usleep_range(TIME_CMOS_MIN, TIME_CMOS_MAX);
- kirin_pcie_oe_enable(kirin_pcie);
- ret = kirin_pcie_clk_ctrl(kirin_pcie, true);
+ hi3660_pcie_phy_oe_enable(phy);
+
+ ret = hi3660_pcie_phy_clk_ctrl(phy, true);
if (ret)
return ret;
/* ISO disable, PCIeCtrl, PHY assert and clk gate clear */
- regmap_write(kirin_pcie->sysctrl,
+ regmap_write(phy->sysctrl,
SCTRL_PCIE_ISO_OFFSET, SCTRL_PCIE_ISO_BIT);
- regmap_write(kirin_pcie->crgctrl,
+ regmap_write(phy->crgctrl,
CRGCTRL_PCIE_ASSERT_OFFSET, CRGCTRL_PCIE_ASSERT_BIT);
- regmap_write(kirin_pcie->sysctrl,
+ regmap_write(phy->sysctrl,
SCTRL_PCIE_HPCLK_OFFSET, SCTRL_PCIE_HPCLK_BIT);
- ret = kirin_pcie_phy_init(kirin_pcie);
+ ret = hi3660_pcie_phy_start(phy);
if (ret)
- goto close_clk;
+ goto disable_clks;
- /* perst assert Endpoint */
- if (!gpio_request(kirin_pcie->gpio_id_reset, "pcie_perst")) {
- usleep_range(REF_2_PERST_MIN, REF_2_PERST_MAX);
- ret = gpio_direction_output(kirin_pcie->gpio_id_reset, 1);
- if (ret)
- goto close_clk;
- usleep_range(PERST_2_ACCESS_MIN, PERST_2_ACCESS_MAX);
+ return 0;
+
+disable_clks:
+ hi3660_pcie_phy_clk_ctrl(phy, false);
+ return ret;
+}
+
+static int hi3660_pcie_phy_init(struct platform_device *pdev,
+ struct kirin_pcie *pcie)
+{
+ struct device *dev = &pdev->dev;
+ struct hi3660_pcie_phy *phy;
+ int ret;
+ phy = devm_kzalloc(dev, sizeof(*phy), GFP_KERNEL);
+ if (!phy)
+ return -ENOMEM;
+
+ pcie->phy_priv = phy;
+ phy->dev = dev;
+
+ /* registers */
+ pdev = container_of(dev, struct platform_device, dev);
+
+ ret = hi3660_pcie_phy_get_clk(phy);
+ if (ret)
+ return ret;
+
+ return hi3660_pcie_phy_get_resource(phy);
+}
+
+static int hi3660_pcie_phy_power_off(struct kirin_pcie *pcie)
+{
+ struct hi3660_pcie_phy *phy = pcie->phy_priv;
+
+ /* Drop power supply for Host */
+ regmap_write(phy->sysctrl, SCTRL_PCIE_CMOS_OFFSET, 0x00);
+
+ hi3660_pcie_phy_clk_ctrl(phy, false);
+
+ return 0;
+}
+
+/*
+ * The non-PHY part starts here
+ */
+
+static const struct regmap_config pcie_kirin_regmap_conf = {
+ .name = "kirin_pcie_apb",
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+};
+
+static int kirin_pcie_get_gpio_enable(struct kirin_pcie *pcie,
+ struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *np = dev->of_node;
+ char name[32];
+ int ret, i;
+
+ /* This is an optional property */
+ ret = of_gpio_named_count(np, "hisilicon,clken-gpios");
+ if (ret < 0)
return 0;
+
+ if (ret > MAX_PCI_SLOTS) {
+ dev_err(dev, "Too many GPIO clock requests!\n");
+ return -EINVAL;
}
-close_clk:
- kirin_pcie_clk_ctrl(kirin_pcie, false);
+ pcie->n_gpio_clkreq = ret;
+
+ for (i = 0; i < pcie->n_gpio_clkreq; i++) {
+ pcie->gpio_id_clkreq[i] = of_get_named_gpio(dev->of_node,
+ "hisilicon,clken-gpios", i);
+ if (pcie->gpio_id_clkreq[i] < 0)
+ return pcie->gpio_id_clkreq[i];
+
+ sprintf(name, "pcie_clkreq_%d", i);
+ pcie->clkreq_names[i] = devm_kstrdup_const(dev, name,
+ GFP_KERNEL);
+ if (!pcie->clkreq_names[i])
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static int kirin_pcie_parse_port(struct kirin_pcie *pcie,
+ struct platform_device *pdev,
+ struct device_node *node)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *parent, *child;
+ int ret, slot, i;
+ char name[32];
+
+ for_each_available_child_of_node(node, parent) {
+ for_each_available_child_of_node(parent, child) {
+ i = pcie->num_slots;
+
+ pcie->gpio_id_reset[i] = of_get_named_gpio(child,
+ "reset-gpios", 0);
+ if (pcie->gpio_id_reset[i] < 0)
+ continue;
+
+ pcie->num_slots++;
+ if (pcie->num_slots > MAX_PCI_SLOTS) {
+ dev_err(dev, "Too many PCI slots!\n");
+ ret = -EINVAL;
+ goto put_node;
+ }
+
+ ret = of_pci_get_devfn(child);
+ if (ret < 0) {
+ dev_err(dev, "failed to parse devfn: %d\n", ret);
+ goto put_node;
+ }
+
+ slot = PCI_SLOT(ret);
+
+ sprintf(name, "pcie_perst_%d", slot);
+ pcie->reset_names[i] = devm_kstrdup_const(dev, name,
+ GFP_KERNEL);
+ if (!pcie->reset_names[i]) {
+ ret = -ENOMEM;
+ goto put_node;
+ }
+ }
+ }
+
+ return 0;
+
+put_node:
+ of_node_put(child);
+ of_node_put(parent);
+ return ret;
+}
+
+static long kirin_pcie_get_resource(struct kirin_pcie *kirin_pcie,
+ struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *child, *node = dev->of_node;
+ void __iomem *apb_base;
+ int ret;
+
+ apb_base = devm_platform_ioremap_resource_byname(pdev, "apb");
+ if (IS_ERR(apb_base))
+ return PTR_ERR(apb_base);
+
+ kirin_pcie->apb = devm_regmap_init_mmio(dev, apb_base,
+ &pcie_kirin_regmap_conf);
+ if (IS_ERR(kirin_pcie->apb))
+ return PTR_ERR(kirin_pcie->apb);
+
+ /* pcie internal PERST# gpio */
+ kirin_pcie->gpio_id_dwc_perst = of_get_named_gpio(dev->of_node,
+ "reset-gpios", 0);
+ if (kirin_pcie->gpio_id_dwc_perst == -EPROBE_DEFER) {
+ return -EPROBE_DEFER;
+ } else if (!gpio_is_valid(kirin_pcie->gpio_id_dwc_perst)) {
+ dev_err(dev, "unable to get a valid gpio pin\n");
+ return -ENODEV;
+ }
+
+ ret = kirin_pcie_get_gpio_enable(kirin_pcie, pdev);
+ if (ret)
+ return ret;
+
+ /* Parse OF children */
+ for_each_available_child_of_node(node, child) {
+ ret = kirin_pcie_parse_port(kirin_pcie, pdev, child);
+ if (ret)
+ goto put_node;
+ }
+
+ return 0;
+
+put_node:
+ of_node_put(child);
return ret;
}
{
u32 val;
- val = kirin_apb_ctrl_readl(kirin_pcie, SOC_PCIECTRL_CTRL0_ADDR);
+ regmap_read(kirin_pcie->apb, SOC_PCIECTRL_CTRL0_ADDR, &val);
if (on)
val = val | PCIE_ELBI_SLV_DBI_ENABLE;
else
val = val & ~PCIE_ELBI_SLV_DBI_ENABLE;
- kirin_apb_ctrl_writel(kirin_pcie, val, SOC_PCIECTRL_CTRL0_ADDR);
+ regmap_write(kirin_pcie->apb, SOC_PCIECTRL_CTRL0_ADDR, val);
}
static void kirin_pcie_sideband_dbi_r_mode(struct kirin_pcie *kirin_pcie,
{
u32 val;
- val = kirin_apb_ctrl_readl(kirin_pcie, SOC_PCIECTRL_CTRL1_ADDR);
+ regmap_read(kirin_pcie->apb, SOC_PCIECTRL_CTRL1_ADDR, &val);
if (on)
val = val | PCIE_ELBI_SLV_DBI_ENABLE;
else
val = val & ~PCIE_ELBI_SLV_DBI_ENABLE;
- kirin_apb_ctrl_writel(kirin_pcie, val, SOC_PCIECTRL_CTRL1_ADDR);
+ regmap_write(kirin_pcie->apb, SOC_PCIECTRL_CTRL1_ADDR, val);
}
static int kirin_pcie_rd_own_conf(struct pci_bus *bus, unsigned int devfn,
return PCIBIOS_SUCCESSFUL;
}
+static int kirin_pcie_add_bus(struct pci_bus *bus)
+{
+ struct dw_pcie *pci = to_dw_pcie_from_pp(bus->sysdata);
+ struct kirin_pcie *kirin_pcie = to_kirin_pcie(pci);
+ int i, ret;
+
+ if (!kirin_pcie->num_slots)
+ return 0;
+
+ /* Send PERST# to each slot */
+ for (i = 0; i < kirin_pcie->num_slots; i++) {
+ ret = gpio_direction_output(kirin_pcie->gpio_id_reset[i], 1);
+ if (ret) {
+ dev_err(pci->dev, "PERST# %s error: %d\n",
+ kirin_pcie->reset_names[i], ret);
+ }
+ }
+ usleep_range(PERST_2_ACCESS_MIN, PERST_2_ACCESS_MAX);
+
+ return 0;
+}
+
static struct pci_ops kirin_pci_ops = {
.read = kirin_pcie_rd_own_conf,
.write = kirin_pcie_wr_own_conf,
+ .add_bus = kirin_pcie_add_bus,
};
static u32 kirin_pcie_read_dbi(struct dw_pcie *pci, void __iomem *base,
static int kirin_pcie_link_up(struct dw_pcie *pci)
{
struct kirin_pcie *kirin_pcie = to_kirin_pcie(pci);
- u32 val = kirin_apb_ctrl_readl(kirin_pcie, PCIE_APB_PHY_STATUS0);
+ u32 val;
+ regmap_read(kirin_pcie->apb, PCIE_APB_PHY_STATUS0, &val);
if ((val & PCIE_LINKUP_ENABLE) == PCIE_LINKUP_ENABLE)
return 1;
struct kirin_pcie *kirin_pcie = to_kirin_pcie(pci);
/* assert LTSSM enable */
- kirin_apb_ctrl_writel(kirin_pcie, PCIE_LTSSM_ENABLE_BIT,
- PCIE_APP_LTSSM_ENABLE);
+ regmap_write(kirin_pcie->apb, PCIE_APP_LTSSM_ENABLE,
+ PCIE_LTSSM_ENABLE_BIT);
return 0;
}
return 0;
}
+static int kirin_pcie_gpio_request(struct kirin_pcie *kirin_pcie,
+ struct device *dev)
+{
+ int ret, i;
+
+ for (i = 0; i < kirin_pcie->num_slots; i++) {
+ if (!gpio_is_valid(kirin_pcie->gpio_id_reset[i])) {
+ dev_err(dev, "unable to get a valid %s gpio\n",
+ kirin_pcie->reset_names[i]);
+ return -ENODEV;
+ }
+
+ ret = devm_gpio_request(dev, kirin_pcie->gpio_id_reset[i],
+ kirin_pcie->reset_names[i]);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < kirin_pcie->n_gpio_clkreq; i++) {
+ if (!gpio_is_valid(kirin_pcie->gpio_id_clkreq[i])) {
+ dev_err(dev, "unable to get a valid %s gpio\n",
+ kirin_pcie->clkreq_names[i]);
+ return -ENODEV;
+ }
+
+ ret = devm_gpio_request(dev, kirin_pcie->gpio_id_clkreq[i],
+ kirin_pcie->clkreq_names[i]);
+ if (ret)
+ return ret;
+
+ ret = gpio_direction_output(kirin_pcie->gpio_id_clkreq[i], 0);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
static const struct dw_pcie_ops kirin_dw_pcie_ops = {
.read_dbi = kirin_pcie_read_dbi,
.write_dbi = kirin_pcie_write_dbi,
.host_init = kirin_pcie_host_init,
};
+static int kirin_pcie_power_off(struct kirin_pcie *kirin_pcie)
+{
+ int i;
+
+ if (kirin_pcie->type == PCIE_KIRIN_INTERNAL_PHY)
+ return hi3660_pcie_phy_power_off(kirin_pcie);
+
+ for (i = 0; i < kirin_pcie->n_gpio_clkreq; i++)
+ gpio_direction_output(kirin_pcie->gpio_id_clkreq[i], 1);
+
+ phy_power_off(kirin_pcie->phy);
+ phy_exit(kirin_pcie->phy);
+
+ return 0;
+}
+
+static int kirin_pcie_power_on(struct platform_device *pdev,
+ struct kirin_pcie *kirin_pcie)
+{
+ struct device *dev = &pdev->dev;
+ int ret;
+
+ if (kirin_pcie->type == PCIE_KIRIN_INTERNAL_PHY) {
+ ret = hi3660_pcie_phy_init(pdev, kirin_pcie);
+ if (ret)
+ return ret;
+
+ ret = hi3660_pcie_phy_power_on(kirin_pcie);
+ if (ret)
+ return ret;
+ } else {
+ kirin_pcie->phy = devm_of_phy_get(dev, dev->of_node, NULL);
+ if (IS_ERR(kirin_pcie->phy))
+ return PTR_ERR(kirin_pcie->phy);
+
+ ret = kirin_pcie_gpio_request(kirin_pcie, dev);
+ if (ret)
+ return ret;
+
+ ret = phy_init(kirin_pcie->phy);
+ if (ret)
+ goto err;
+
+ ret = phy_power_on(kirin_pcie->phy);
+ if (ret)
+ goto err;
+ }
+
+ /* perst assert Endpoint */
+ usleep_range(REF_2_PERST_MIN, REF_2_PERST_MAX);
+
+ if (!gpio_request(kirin_pcie->gpio_id_dwc_perst, "pcie_perst_bridge")) {
+ ret = gpio_direction_output(kirin_pcie->gpio_id_dwc_perst, 1);
+ if (ret)
+ goto err;
+ }
+
+ usleep_range(PERST_2_ACCESS_MIN, PERST_2_ACCESS_MAX);
+
+ return 0;
+err:
+ kirin_pcie_power_off(kirin_pcie);
+
+ return ret;
+}
+
+static int __exit kirin_pcie_remove(struct platform_device *pdev)
+{
+ struct kirin_pcie *kirin_pcie = platform_get_drvdata(pdev);
+
+ dw_pcie_host_deinit(&kirin_pcie->pci->pp);
+
+ kirin_pcie_power_off(kirin_pcie);
+
+ return 0;
+}
+
+static const struct of_device_id kirin_pcie_match[] = {
+ {
+ .compatible = "hisilicon,kirin960-pcie",
+ .data = (void *)PCIE_KIRIN_INTERNAL_PHY
+ },
+ {
+ .compatible = "hisilicon,kirin970-pcie",
+ .data = (void *)PCIE_KIRIN_EXTERNAL_PHY
+ },
+ {},
+};
+
static int kirin_pcie_probe(struct platform_device *pdev)
{
+ enum pcie_kirin_phy_type phy_type;
+ const struct of_device_id *of_id;
struct device *dev = &pdev->dev;
struct kirin_pcie *kirin_pcie;
struct dw_pcie *pci;
return -EINVAL;
}
+ of_id = of_match_device(kirin_pcie_match, dev);
+ if (!of_id) {
+ dev_err(dev, "OF data missing\n");
+ return -EINVAL;
+ }
+
+ phy_type = (long)of_id->data;
+
kirin_pcie = devm_kzalloc(dev, sizeof(struct kirin_pcie), GFP_KERNEL);
if (!kirin_pcie)
return -ENOMEM;
pci->ops = &kirin_dw_pcie_ops;
pci->pp.ops = &kirin_pcie_host_ops;
kirin_pcie->pci = pci;
-
- ret = kirin_pcie_get_clk(kirin_pcie, pdev);
- if (ret)
- return ret;
+ kirin_pcie->type = phy_type;
ret = kirin_pcie_get_resource(kirin_pcie, pdev);
if (ret)
return ret;
- kirin_pcie->gpio_id_reset = of_get_named_gpio(dev->of_node,
- "reset-gpios", 0);
- if (kirin_pcie->gpio_id_reset == -EPROBE_DEFER) {
- return -EPROBE_DEFER;
- } else if (!gpio_is_valid(kirin_pcie->gpio_id_reset)) {
- dev_err(dev, "unable to get a valid gpio pin\n");
- return -ENODEV;
- }
+ platform_set_drvdata(pdev, kirin_pcie);
- ret = kirin_pcie_power_on(kirin_pcie);
+ ret = kirin_pcie_power_on(pdev, kirin_pcie);
if (ret)
return ret;
- platform_set_drvdata(pdev, kirin_pcie);
-
return dw_pcie_host_init(&pci->pp);
}
-static const struct of_device_id kirin_pcie_match[] = {
- { .compatible = "hisilicon,kirin960-pcie" },
- {},
-};
-
static struct platform_driver kirin_pcie_driver = {
.probe = kirin_pcie_probe,
+ .remove = __exit_p(kirin_pcie_remove),
.driver = {
.name = "kirin-pcie",
- .of_match_table = kirin_pcie_match,
- .suppress_bind_attrs = true,
+ .of_match_table = kirin_pcie_match,
+ .suppress_bind_attrs = true,
},
};
-builtin_platform_driver(kirin_pcie_driver);
+module_platform_driver(kirin_pcie_driver);
+
+MODULE_DEVICE_TABLE(of, kirin_pcie_match);
+MODULE_DESCRIPTION("PCIe host controller driver for Kirin Phone SoCs");
+MODULE_AUTHOR("Xiaowei Song <songxiaowei@huawei.com>");
+MODULE_LICENSE("GPL v2");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Qualcomm PCIe Endpoint controller driver
+ *
+ * Copyright (c) 2020, The Linux Foundation. All rights reserved.
+ * Author: Siddartha Mohanadoss <smohanad@codeaurora.org
+ *
+ * Copyright (c) 2021, Linaro Ltd.
+ * Author: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org
+ */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/gpio/consumer.h>
+#include <linux/mfd/syscon.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm_domain.h>
+#include <linux/regmap.h>
+#include <linux/reset.h>
+
+#include "pcie-designware.h"
+
+/* PARF registers */
+#define PARF_SYS_CTRL 0x00
+#define PARF_DB_CTRL 0x10
+#define PARF_PM_CTRL 0x20
+#define PARF_MHI_BASE_ADDR_LOWER 0x178
+#define PARF_MHI_BASE_ADDR_UPPER 0x17c
+#define PARF_DEBUG_INT_EN 0x190
+#define PARF_AXI_MSTR_RD_HALT_NO_WRITES 0x1a4
+#define PARF_AXI_MSTR_WR_ADDR_HALT 0x1a8
+#define PARF_Q2A_FLUSH 0x1ac
+#define PARF_LTSSM 0x1b0
+#define PARF_CFG_BITS 0x210
+#define PARF_INT_ALL_STATUS 0x224
+#define PARF_INT_ALL_CLEAR 0x228
+#define PARF_INT_ALL_MASK 0x22c
+#define PARF_SLV_ADDR_MSB_CTRL 0x2c0
+#define PARF_DBI_BASE_ADDR 0x350
+#define PARF_DBI_BASE_ADDR_HI 0x354
+#define PARF_SLV_ADDR_SPACE_SIZE 0x358
+#define PARF_SLV_ADDR_SPACE_SIZE_HI 0x35c
+#define PARF_ATU_BASE_ADDR 0x634
+#define PARF_ATU_BASE_ADDR_HI 0x638
+#define PARF_SRIS_MODE 0x644
+#define PARF_DEVICE_TYPE 0x1000
+#define PARF_BDF_TO_SID_CFG 0x2c00
+
+/* PARF_INT_ALL_{STATUS/CLEAR/MASK} register fields */
+#define PARF_INT_ALL_LINK_DOWN BIT(1)
+#define PARF_INT_ALL_BME BIT(2)
+#define PARF_INT_ALL_PM_TURNOFF BIT(3)
+#define PARF_INT_ALL_DEBUG BIT(4)
+#define PARF_INT_ALL_LTR BIT(5)
+#define PARF_INT_ALL_MHI_Q6 BIT(6)
+#define PARF_INT_ALL_MHI_A7 BIT(7)
+#define PARF_INT_ALL_DSTATE_CHANGE BIT(8)
+#define PARF_INT_ALL_L1SUB_TIMEOUT BIT(9)
+#define PARF_INT_ALL_MMIO_WRITE BIT(10)
+#define PARF_INT_ALL_CFG_WRITE BIT(11)
+#define PARF_INT_ALL_BRIDGE_FLUSH_N BIT(12)
+#define PARF_INT_ALL_LINK_UP BIT(13)
+#define PARF_INT_ALL_AER_LEGACY BIT(14)
+#define PARF_INT_ALL_PLS_ERR BIT(15)
+#define PARF_INT_ALL_PME_LEGACY BIT(16)
+#define PARF_INT_ALL_PLS_PME BIT(17)
+
+/* PARF_BDF_TO_SID_CFG register fields */
+#define PARF_BDF_TO_SID_BYPASS BIT(0)
+
+/* PARF_DEBUG_INT_EN register fields */
+#define PARF_DEBUG_INT_PM_DSTATE_CHANGE BIT(1)
+#define PARF_DEBUG_INT_CFG_BUS_MASTER_EN BIT(2)
+#define PARF_DEBUG_INT_RADM_PM_TURNOFF BIT(3)
+
+/* PARF_DEVICE_TYPE register fields */
+#define PARF_DEVICE_TYPE_EP 0x0
+
+/* PARF_PM_CTRL register fields */
+#define PARF_PM_CTRL_REQ_EXIT_L1 BIT(1)
+#define PARF_PM_CTRL_READY_ENTR_L23 BIT(2)
+#define PARF_PM_CTRL_REQ_NOT_ENTR_L1 BIT(5)
+
+/* PARF_AXI_MSTR_RD_HALT_NO_WRITES register fields */
+#define PARF_AXI_MSTR_RD_HALT_NO_WRITE_EN BIT(0)
+
+/* PARF_AXI_MSTR_WR_ADDR_HALT register fields */
+#define PARF_AXI_MSTR_WR_ADDR_HALT_EN BIT(31)
+
+/* PARF_Q2A_FLUSH register fields */
+#define PARF_Q2A_FLUSH_EN BIT(16)
+
+/* PARF_SYS_CTRL register fields */
+#define PARF_SYS_CTRL_AUX_PWR_DET BIT(4)
+#define PARF_SYS_CTRL_CORE_CLK_CGC_DIS BIT(6)
+#define PARF_SYS_CTRL_SLV_DBI_WAKE_DISABLE BIT(11)
+
+/* PARF_DB_CTRL register fields */
+#define PARF_DB_CTRL_INSR_DBNCR_BLOCK BIT(0)
+#define PARF_DB_CTRL_RMVL_DBNCR_BLOCK BIT(1)
+#define PARF_DB_CTRL_DBI_WKP_BLOCK BIT(4)
+#define PARF_DB_CTRL_SLV_WKP_BLOCK BIT(5)
+#define PARF_DB_CTRL_MST_WKP_BLOCK BIT(6)
+
+/* PARF_CFG_BITS register fields */
+#define PARF_CFG_BITS_REQ_EXIT_L1SS_MSI_LTR_EN BIT(1)
+
+/* ELBI registers */
+#define ELBI_SYS_STTS 0x08
+
+/* DBI registers */
+#define DBI_CON_STATUS 0x44
+
+/* DBI register fields */
+#define DBI_CON_STATUS_POWER_STATE_MASK GENMASK(1, 0)
+
+#define XMLH_LINK_UP 0x400
+#define CORE_RESET_TIME_US_MIN 1000
+#define CORE_RESET_TIME_US_MAX 1005
+#define WAKE_DELAY_US 2000 /* 2 ms */
+
+#define to_pcie_ep(x) dev_get_drvdata((x)->dev)
+
+enum qcom_pcie_ep_link_status {
+ QCOM_PCIE_EP_LINK_DISABLED,
+ QCOM_PCIE_EP_LINK_ENABLED,
+ QCOM_PCIE_EP_LINK_UP,
+ QCOM_PCIE_EP_LINK_DOWN,
+};
+
+static struct clk_bulk_data qcom_pcie_ep_clks[] = {
+ { .id = "cfg" },
+ { .id = "aux" },
+ { .id = "bus_master" },
+ { .id = "bus_slave" },
+ { .id = "ref" },
+ { .id = "sleep" },
+ { .id = "slave_q2a" },
+};
+
+struct qcom_pcie_ep {
+ struct dw_pcie pci;
+
+ void __iomem *parf;
+ void __iomem *elbi;
+ struct regmap *perst_map;
+ struct resource *mmio_res;
+
+ struct reset_control *core_reset;
+ struct gpio_desc *reset;
+ struct gpio_desc *wake;
+ struct phy *phy;
+
+ u32 perst_en;
+ u32 perst_sep_en;
+
+ enum qcom_pcie_ep_link_status link_status;
+ int global_irq;
+ int perst_irq;
+};
+
+static int qcom_pcie_ep_core_reset(struct qcom_pcie_ep *pcie_ep)
+{
+ struct dw_pcie *pci = &pcie_ep->pci;
+ struct device *dev = pci->dev;
+ int ret;
+
+ ret = reset_control_assert(pcie_ep->core_reset);
+ if (ret) {
+ dev_err(dev, "Cannot assert core reset\n");
+ return ret;
+ }
+
+ usleep_range(CORE_RESET_TIME_US_MIN, CORE_RESET_TIME_US_MAX);
+
+ ret = reset_control_deassert(pcie_ep->core_reset);
+ if (ret) {
+ dev_err(dev, "Cannot de-assert core reset\n");
+ return ret;
+ }
+
+ usleep_range(CORE_RESET_TIME_US_MIN, CORE_RESET_TIME_US_MAX);
+
+ return 0;
+}
+
+/*
+ * Delatch PERST_EN and PERST_SEPARATION_ENABLE with TCSR to avoid
+ * device reset during host reboot and hibernation. The driver is
+ * expected to handle this situation.
+ */
+static void qcom_pcie_ep_configure_tcsr(struct qcom_pcie_ep *pcie_ep)
+{
+ regmap_write(pcie_ep->perst_map, pcie_ep->perst_en, 0);
+ regmap_write(pcie_ep->perst_map, pcie_ep->perst_sep_en, 0);
+}
+
+static int qcom_pcie_dw_link_up(struct dw_pcie *pci)
+{
+ struct qcom_pcie_ep *pcie_ep = to_pcie_ep(pci);
+ u32 reg;
+
+ reg = readl_relaxed(pcie_ep->elbi + ELBI_SYS_STTS);
+
+ return reg & XMLH_LINK_UP;
+}
+
+static int qcom_pcie_dw_start_link(struct dw_pcie *pci)
+{
+ struct qcom_pcie_ep *pcie_ep = to_pcie_ep(pci);
+
+ enable_irq(pcie_ep->perst_irq);
+
+ return 0;
+}
+
+static void qcom_pcie_dw_stop_link(struct dw_pcie *pci)
+{
+ struct qcom_pcie_ep *pcie_ep = to_pcie_ep(pci);
+
+ disable_irq(pcie_ep->perst_irq);
+}
+
+static int qcom_pcie_perst_deassert(struct dw_pcie *pci)
+{
+ struct qcom_pcie_ep *pcie_ep = to_pcie_ep(pci);
+ struct device *dev = pci->dev;
+ u32 val, offset;
+ int ret;
+
+ ret = clk_bulk_prepare_enable(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+ if (ret)
+ return ret;
+
+ ret = qcom_pcie_ep_core_reset(pcie_ep);
+ if (ret)
+ goto err_disable_clk;
+
+ ret = phy_init(pcie_ep->phy);
+ if (ret)
+ goto err_disable_clk;
+
+ ret = phy_power_on(pcie_ep->phy);
+ if (ret)
+ goto err_phy_exit;
+
+ /* Assert WAKE# to RC to indicate device is ready */
+ gpiod_set_value_cansleep(pcie_ep->wake, 1);
+ usleep_range(WAKE_DELAY_US, WAKE_DELAY_US + 500);
+ gpiod_set_value_cansleep(pcie_ep->wake, 0);
+
+ qcom_pcie_ep_configure_tcsr(pcie_ep);
+
+ /* Disable BDF to SID mapping */
+ val = readl_relaxed(pcie_ep->parf + PARF_BDF_TO_SID_CFG);
+ val |= PARF_BDF_TO_SID_BYPASS;
+ writel_relaxed(val, pcie_ep->parf + PARF_BDF_TO_SID_CFG);
+
+ /* Enable debug IRQ */
+ val = readl_relaxed(pcie_ep->parf + PARF_DEBUG_INT_EN);
+ val |= PARF_DEBUG_INT_RADM_PM_TURNOFF |
+ PARF_DEBUG_INT_CFG_BUS_MASTER_EN |
+ PARF_DEBUG_INT_PM_DSTATE_CHANGE;
+ writel_relaxed(val, pcie_ep->parf + PARF_DEBUG_INT_EN);
+
+ /* Configure PCIe to endpoint mode */
+ writel_relaxed(PARF_DEVICE_TYPE_EP, pcie_ep->parf + PARF_DEVICE_TYPE);
+
+ /* Allow entering L1 state */
+ val = readl_relaxed(pcie_ep->parf + PARF_PM_CTRL);
+ val &= ~PARF_PM_CTRL_REQ_NOT_ENTR_L1;
+ writel_relaxed(val, pcie_ep->parf + PARF_PM_CTRL);
+
+ /* Read halts write */
+ val = readl_relaxed(pcie_ep->parf + PARF_AXI_MSTR_RD_HALT_NO_WRITES);
+ val &= ~PARF_AXI_MSTR_RD_HALT_NO_WRITE_EN;
+ writel_relaxed(val, pcie_ep->parf + PARF_AXI_MSTR_RD_HALT_NO_WRITES);
+
+ /* Write after write halt */
+ val = readl_relaxed(pcie_ep->parf + PARF_AXI_MSTR_WR_ADDR_HALT);
+ val |= PARF_AXI_MSTR_WR_ADDR_HALT_EN;
+ writel_relaxed(val, pcie_ep->parf + PARF_AXI_MSTR_WR_ADDR_HALT);
+
+ /* Q2A flush disable */
+ val = readl_relaxed(pcie_ep->parf + PARF_Q2A_FLUSH);
+ val &= ~PARF_Q2A_FLUSH_EN;
+ writel_relaxed(val, pcie_ep->parf + PARF_Q2A_FLUSH);
+
+ /* Disable DBI Wakeup, core clock CGC and enable AUX power */
+ val = readl_relaxed(pcie_ep->parf + PARF_SYS_CTRL);
+ val |= PARF_SYS_CTRL_SLV_DBI_WAKE_DISABLE |
+ PARF_SYS_CTRL_CORE_CLK_CGC_DIS |
+ PARF_SYS_CTRL_AUX_PWR_DET;
+ writel_relaxed(val, pcie_ep->parf + PARF_SYS_CTRL);
+
+ /* Disable the debouncers */
+ val = readl_relaxed(pcie_ep->parf + PARF_DB_CTRL);
+ val |= PARF_DB_CTRL_INSR_DBNCR_BLOCK | PARF_DB_CTRL_RMVL_DBNCR_BLOCK |
+ PARF_DB_CTRL_DBI_WKP_BLOCK | PARF_DB_CTRL_SLV_WKP_BLOCK |
+ PARF_DB_CTRL_MST_WKP_BLOCK;
+ writel_relaxed(val, pcie_ep->parf + PARF_DB_CTRL);
+
+ /* Request to exit from L1SS for MSI and LTR MSG */
+ val = readl_relaxed(pcie_ep->parf + PARF_CFG_BITS);
+ val |= PARF_CFG_BITS_REQ_EXIT_L1SS_MSI_LTR_EN;
+ writel_relaxed(val, pcie_ep->parf + PARF_CFG_BITS);
+
+ dw_pcie_dbi_ro_wr_en(pci);
+
+ /* Set the L0s Exit Latency to 2us-4us = 0x6 */
+ offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
+ val = dw_pcie_readl_dbi(pci, offset + PCI_EXP_LNKCAP);
+ val &= ~PCI_EXP_LNKCAP_L0SEL;
+ val |= FIELD_PREP(PCI_EXP_LNKCAP_L0SEL, 0x6);
+ dw_pcie_writel_dbi(pci, offset + PCI_EXP_LNKCAP, val);
+
+ /* Set the L1 Exit Latency to be 32us-64 us = 0x6 */
+ offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP);
+ val = dw_pcie_readl_dbi(pci, offset + PCI_EXP_LNKCAP);
+ val &= ~PCI_EXP_LNKCAP_L1EL;
+ val |= FIELD_PREP(PCI_EXP_LNKCAP_L1EL, 0x6);
+ dw_pcie_writel_dbi(pci, offset + PCI_EXP_LNKCAP, val);
+
+ dw_pcie_dbi_ro_wr_dis(pci);
+
+ writel_relaxed(0, pcie_ep->parf + PARF_INT_ALL_MASK);
+ val = PARF_INT_ALL_LINK_DOWN | PARF_INT_ALL_BME |
+ PARF_INT_ALL_PM_TURNOFF | PARF_INT_ALL_DSTATE_CHANGE |
+ PARF_INT_ALL_LINK_UP;
+ writel_relaxed(val, pcie_ep->parf + PARF_INT_ALL_MASK);
+
+ ret = dw_pcie_ep_init_complete(&pcie_ep->pci.ep);
+ if (ret) {
+ dev_err(dev, "Failed to complete initialization: %d\n", ret);
+ goto err_phy_power_off;
+ }
+
+ /*
+ * The physical address of the MMIO region which is exposed as the BAR
+ * should be written to MHI BASE registers.
+ */
+ writel_relaxed(pcie_ep->mmio_res->start,
+ pcie_ep->parf + PARF_MHI_BASE_ADDR_LOWER);
+ writel_relaxed(0, pcie_ep->parf + PARF_MHI_BASE_ADDR_UPPER);
+
+ dw_pcie_ep_init_notify(&pcie_ep->pci.ep);
+
+ /* Enable LTSSM */
+ val = readl_relaxed(pcie_ep->parf + PARF_LTSSM);
+ val |= BIT(8);
+ writel_relaxed(val, pcie_ep->parf + PARF_LTSSM);
+
+ return 0;
+
+err_phy_power_off:
+ phy_power_off(pcie_ep->phy);
+err_phy_exit:
+ phy_exit(pcie_ep->phy);
+err_disable_clk:
+ clk_bulk_disable_unprepare(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+
+ return ret;
+}
+
+static void qcom_pcie_perst_assert(struct dw_pcie *pci)
+{
+ struct qcom_pcie_ep *pcie_ep = to_pcie_ep(pci);
+ struct device *dev = pci->dev;
+
+ if (pcie_ep->link_status == QCOM_PCIE_EP_LINK_DISABLED) {
+ dev_dbg(dev, "Link is already disabled\n");
+ return;
+ }
+
+ phy_power_off(pcie_ep->phy);
+ phy_exit(pcie_ep->phy);
+ clk_bulk_disable_unprepare(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+ pcie_ep->link_status = QCOM_PCIE_EP_LINK_DISABLED;
+}
+
+/* Common DWC controller ops */
+static const struct dw_pcie_ops pci_ops = {
+ .link_up = qcom_pcie_dw_link_up,
+ .start_link = qcom_pcie_dw_start_link,
+ .stop_link = qcom_pcie_dw_stop_link,
+};
+
+static int qcom_pcie_ep_get_io_resources(struct platform_device *pdev,
+ struct qcom_pcie_ep *pcie_ep)
+{
+ struct device *dev = &pdev->dev;
+ struct dw_pcie *pci = &pcie_ep->pci;
+ struct device_node *syscon;
+ struct resource *res;
+ int ret;
+
+ pcie_ep->parf = devm_platform_ioremap_resource_byname(pdev, "parf");
+ if (IS_ERR(pcie_ep->parf))
+ return PTR_ERR(pcie_ep->parf);
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "dbi");
+ pci->dbi_base = devm_pci_remap_cfg_resource(dev, res);
+ if (IS_ERR(pci->dbi_base))
+ return PTR_ERR(pci->dbi_base);
+ pci->dbi_base2 = pci->dbi_base;
+
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "elbi");
+ pcie_ep->elbi = devm_pci_remap_cfg_resource(dev, res);
+ if (IS_ERR(pcie_ep->elbi))
+ return PTR_ERR(pcie_ep->elbi);
+
+ pcie_ep->mmio_res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+ "mmio");
+
+ syscon = of_parse_phandle(dev->of_node, "qcom,perst-regs", 0);
+ if (!syscon) {
+ dev_err(dev, "Failed to parse qcom,perst-regs\n");
+ return -EINVAL;
+ }
+
+ pcie_ep->perst_map = syscon_node_to_regmap(syscon);
+ of_node_put(syscon);
+ if (IS_ERR(pcie_ep->perst_map))
+ return PTR_ERR(pcie_ep->perst_map);
+
+ ret = of_property_read_u32_index(dev->of_node, "qcom,perst-regs",
+ 1, &pcie_ep->perst_en);
+ if (ret < 0) {
+ dev_err(dev, "No Perst Enable offset in syscon\n");
+ return ret;
+ }
+
+ ret = of_property_read_u32_index(dev->of_node, "qcom,perst-regs",
+ 2, &pcie_ep->perst_sep_en);
+ if (ret < 0) {
+ dev_err(dev, "No Perst Separation Enable offset in syscon\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int qcom_pcie_ep_get_resources(struct platform_device *pdev,
+ struct qcom_pcie_ep *pcie_ep)
+{
+ struct device *dev = &pdev->dev;
+ int ret;
+
+ ret = qcom_pcie_ep_get_io_resources(pdev, pcie_ep);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to get io resources %d\n", ret);
+ return ret;
+ }
+
+ ret = devm_clk_bulk_get(dev, ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+ if (ret)
+ return ret;
+
+ pcie_ep->core_reset = devm_reset_control_get_exclusive(dev, "core");
+ if (IS_ERR(pcie_ep->core_reset))
+ return PTR_ERR(pcie_ep->core_reset);
+
+ pcie_ep->reset = devm_gpiod_get(dev, "reset", GPIOD_IN);
+ if (IS_ERR(pcie_ep->reset))
+ return PTR_ERR(pcie_ep->reset);
+
+ pcie_ep->wake = devm_gpiod_get_optional(dev, "wake", GPIOD_OUT_LOW);
+ if (IS_ERR(pcie_ep->wake))
+ return PTR_ERR(pcie_ep->wake);
+
+ pcie_ep->phy = devm_phy_optional_get(&pdev->dev, "pciephy");
+ if (IS_ERR(pcie_ep->phy))
+ ret = PTR_ERR(pcie_ep->phy);
+
+ return ret;
+}
+
+/* TODO: Notify clients about PCIe state change */
+static irqreturn_t qcom_pcie_ep_global_irq_thread(int irq, void *data)
+{
+ struct qcom_pcie_ep *pcie_ep = data;
+ struct dw_pcie *pci = &pcie_ep->pci;
+ struct device *dev = pci->dev;
+ u32 status = readl_relaxed(pcie_ep->parf + PARF_INT_ALL_STATUS);
+ u32 mask = readl_relaxed(pcie_ep->parf + PARF_INT_ALL_MASK);
+ u32 dstate, val;
+
+ writel_relaxed(status, pcie_ep->parf + PARF_INT_ALL_CLEAR);
+ status &= mask;
+
+ if (FIELD_GET(PARF_INT_ALL_LINK_DOWN, status)) {
+ dev_dbg(dev, "Received Linkdown event\n");
+ pcie_ep->link_status = QCOM_PCIE_EP_LINK_DOWN;
+ } else if (FIELD_GET(PARF_INT_ALL_BME, status)) {
+ dev_dbg(dev, "Received BME event. Link is enabled!\n");
+ pcie_ep->link_status = QCOM_PCIE_EP_LINK_ENABLED;
+ } else if (FIELD_GET(PARF_INT_ALL_PM_TURNOFF, status)) {
+ dev_dbg(dev, "Received PM Turn-off event! Entering L23\n");
+ val = readl_relaxed(pcie_ep->parf + PARF_PM_CTRL);
+ val |= PARF_PM_CTRL_READY_ENTR_L23;
+ writel_relaxed(val, pcie_ep->parf + PARF_PM_CTRL);
+ } else if (FIELD_GET(PARF_INT_ALL_DSTATE_CHANGE, status)) {
+ dstate = dw_pcie_readl_dbi(pci, DBI_CON_STATUS) &
+ DBI_CON_STATUS_POWER_STATE_MASK;
+ dev_dbg(dev, "Received D%d state event\n", dstate);
+ if (dstate == 3) {
+ val = readl_relaxed(pcie_ep->parf + PARF_PM_CTRL);
+ val |= PARF_PM_CTRL_REQ_EXIT_L1;
+ writel_relaxed(val, pcie_ep->parf + PARF_PM_CTRL);
+ }
+ } else if (FIELD_GET(PARF_INT_ALL_LINK_UP, status)) {
+ dev_dbg(dev, "Received Linkup event. Enumeration complete!\n");
+ dw_pcie_ep_linkup(&pci->ep);
+ pcie_ep->link_status = QCOM_PCIE_EP_LINK_UP;
+ } else {
+ dev_dbg(dev, "Received unknown event: %d\n", status);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t qcom_pcie_ep_perst_irq_thread(int irq, void *data)
+{
+ struct qcom_pcie_ep *pcie_ep = data;
+ struct dw_pcie *pci = &pcie_ep->pci;
+ struct device *dev = pci->dev;
+ u32 perst;
+
+ perst = gpiod_get_value(pcie_ep->reset);
+ if (perst) {
+ dev_dbg(dev, "PERST asserted by host. Shutting down the PCIe link!\n");
+ qcom_pcie_perst_assert(pci);
+ } else {
+ dev_dbg(dev, "PERST de-asserted by host. Starting link training!\n");
+ qcom_pcie_perst_deassert(pci);
+ }
+
+ irq_set_irq_type(gpiod_to_irq(pcie_ep->reset),
+ (perst ? IRQF_TRIGGER_HIGH : IRQF_TRIGGER_LOW));
+
+ return IRQ_HANDLED;
+}
+
+static int qcom_pcie_ep_enable_irq_resources(struct platform_device *pdev,
+ struct qcom_pcie_ep *pcie_ep)
+{
+ int irq, ret;
+
+ irq = platform_get_irq_byname(pdev, "global");
+ if (irq < 0) {
+ dev_err(&pdev->dev, "Failed to get Global IRQ\n");
+ return irq;
+ }
+
+ ret = devm_request_threaded_irq(&pdev->dev, irq, NULL,
+ qcom_pcie_ep_global_irq_thread,
+ IRQF_ONESHOT,
+ "global_irq", pcie_ep);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to request Global IRQ\n");
+ return ret;
+ }
+
+ pcie_ep->perst_irq = gpiod_to_irq(pcie_ep->reset);
+ irq_set_status_flags(pcie_ep->perst_irq, IRQ_NOAUTOEN);
+ ret = devm_request_threaded_irq(&pdev->dev, pcie_ep->perst_irq, NULL,
+ qcom_pcie_ep_perst_irq_thread,
+ IRQF_TRIGGER_HIGH | IRQF_ONESHOT,
+ "perst_irq", pcie_ep);
+ if (ret) {
+ dev_err(&pdev->dev, "Failed to request PERST IRQ\n");
+ disable_irq(irq);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int qcom_pcie_ep_raise_irq(struct dw_pcie_ep *ep, u8 func_no,
+ enum pci_epc_irq_type type, u16 interrupt_num)
+{
+ struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+
+ switch (type) {
+ case PCI_EPC_IRQ_LEGACY:
+ return dw_pcie_ep_raise_legacy_irq(ep, func_no);
+ case PCI_EPC_IRQ_MSI:
+ return dw_pcie_ep_raise_msi_irq(ep, func_no, interrupt_num);
+ default:
+ dev_err(pci->dev, "Unknown IRQ type\n");
+ return -EINVAL;
+ }
+}
+
+static const struct pci_epc_features qcom_pcie_epc_features = {
+ .linkup_notifier = true,
+ .core_init_notifier = true,
+ .msi_capable = true,
+ .msix_capable = false,
+};
+
+static const struct pci_epc_features *
+qcom_pcie_epc_get_features(struct dw_pcie_ep *pci_ep)
+{
+ return &qcom_pcie_epc_features;
+}
+
+static void qcom_pcie_ep_init(struct dw_pcie_ep *ep)
+{
+ struct dw_pcie *pci = to_dw_pcie_from_ep(ep);
+ enum pci_barno bar;
+
+ for (bar = BAR_0; bar <= BAR_5; bar++)
+ dw_pcie_ep_reset_bar(pci, bar);
+}
+
+static struct dw_pcie_ep_ops pci_ep_ops = {
+ .ep_init = qcom_pcie_ep_init,
+ .raise_irq = qcom_pcie_ep_raise_irq,
+ .get_features = qcom_pcie_epc_get_features,
+};
+
+static int qcom_pcie_ep_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct qcom_pcie_ep *pcie_ep;
+ int ret;
+
+ pcie_ep = devm_kzalloc(dev, sizeof(*pcie_ep), GFP_KERNEL);
+ if (!pcie_ep)
+ return -ENOMEM;
+
+ pcie_ep->pci.dev = dev;
+ pcie_ep->pci.ops = &pci_ops;
+ pcie_ep->pci.ep.ops = &pci_ep_ops;
+ platform_set_drvdata(pdev, pcie_ep);
+
+ ret = qcom_pcie_ep_get_resources(pdev, pcie_ep);
+ if (ret)
+ return ret;
+
+ ret = clk_bulk_prepare_enable(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+ if (ret)
+ return ret;
+
+ ret = qcom_pcie_ep_core_reset(pcie_ep);
+ if (ret)
+ goto err_disable_clk;
+
+ ret = phy_init(pcie_ep->phy);
+ if (ret)
+ goto err_disable_clk;
+
+ /* PHY needs to be powered on for dw_pcie_ep_init() */
+ ret = phy_power_on(pcie_ep->phy);
+ if (ret)
+ goto err_phy_exit;
+
+ ret = dw_pcie_ep_init(&pcie_ep->pci.ep);
+ if (ret) {
+ dev_err(dev, "Failed to initialize endpoint: %d\n", ret);
+ goto err_phy_power_off;
+ }
+
+ ret = qcom_pcie_ep_enable_irq_resources(pdev, pcie_ep);
+ if (ret)
+ goto err_phy_power_off;
+
+ return 0;
+
+err_phy_power_off:
+ phy_power_off(pcie_ep->phy);
+err_phy_exit:
+ phy_exit(pcie_ep->phy);
+err_disable_clk:
+ clk_bulk_disable_unprepare(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+
+ return ret;
+}
+
+static int qcom_pcie_ep_remove(struct platform_device *pdev)
+{
+ struct qcom_pcie_ep *pcie_ep = platform_get_drvdata(pdev);
+
+ if (pcie_ep->link_status == QCOM_PCIE_EP_LINK_DISABLED)
+ return 0;
+
+ phy_power_off(pcie_ep->phy);
+ phy_exit(pcie_ep->phy);
+ clk_bulk_disable_unprepare(ARRAY_SIZE(qcom_pcie_ep_clks),
+ qcom_pcie_ep_clks);
+
+ return 0;
+}
+
+static const struct of_device_id qcom_pcie_ep_match[] = {
+ { .compatible = "qcom,sdx55-pcie-ep", },
+ { }
+};
+
+static struct platform_driver qcom_pcie_ep_driver = {
+ .probe = qcom_pcie_ep_probe,
+ .remove = qcom_pcie_ep_remove,
+ .driver = {
+ .name = "qcom-pcie-ep",
+ .of_match_table = qcom_pcie_ep_match,
+ },
+};
+builtin_platform_driver(qcom_pcie_ep_driver);
+
+MODULE_AUTHOR("Siddartha Mohanadoss <smohanad@codeaurora.org>");
+MODULE_AUTHOR("Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>");
+MODULE_DESCRIPTION("Qualcomm PCIe Endpoint controller driver");
+MODULE_LICENSE("GPL v2");
struct regulator_bulk_data supplies[2];
struct reset_control *pci_reset;
struct clk *pipe_clk;
+ struct clk *pipe_clk_src;
+ struct clk *phy_pipe_clk;
+ struct clk *ref_clk_src;
};
union qcom_pcie_resources {
int (*config_sid)(struct qcom_pcie *pcie);
};
+struct qcom_pcie_cfg {
+ const struct qcom_pcie_ops *ops;
+ unsigned int pipe_clk_need_muxing:1;
+};
+
struct qcom_pcie {
struct dw_pcie *pci;
void __iomem *parf; /* DT parf */
struct phy *phy;
struct gpio_desc *reset;
const struct qcom_pcie_ops *ops;
+ unsigned int pipe_clk_need_muxing:1;
};
#define to_qcom_pcie(x) dev_get_drvdata((x)->dev)
if (ret < 0)
return ret;
+ if (pcie->pipe_clk_need_muxing) {
+ res->pipe_clk_src = devm_clk_get(dev, "pipe_mux");
+ if (IS_ERR(res->pipe_clk_src))
+ return PTR_ERR(res->pipe_clk_src);
+
+ res->phy_pipe_clk = devm_clk_get(dev, "phy_pipe");
+ if (IS_ERR(res->phy_pipe_clk))
+ return PTR_ERR(res->phy_pipe_clk);
+
+ res->ref_clk_src = devm_clk_get(dev, "ref");
+ if (IS_ERR(res->ref_clk_src))
+ return PTR_ERR(res->ref_clk_src);
+ }
+
res->pipe_clk = devm_clk_get(dev, "pipe");
return PTR_ERR_OR_ZERO(res->pipe_clk);
}
return ret;
}
+ /* Set TCXO as clock source for pcie_pipe_clk_src */
+ if (pcie->pipe_clk_need_muxing)
+ clk_set_parent(res->pipe_clk_src, res->ref_clk_src);
+
ret = clk_bulk_prepare_enable(res->num_clks, res->clks);
if (ret < 0)
goto err_disable_regulators;
{
struct qcom_pcie_resources_2_7_0 *res = &pcie->res.v2_7_0;
+ /* Set pipe clock as clock source for pcie_pipe_clk_src */
+ if (pcie->pipe_clk_need_muxing)
+ clk_set_parent(res->pipe_clk_src, res->phy_pipe_clk);
+
return clk_prepare_enable(res->pipe_clk);
}
.config_sid = qcom_pcie_config_sid_sm8250,
};
+static const struct qcom_pcie_cfg apq8084_cfg = {
+ .ops = &ops_1_0_0,
+};
+
+static const struct qcom_pcie_cfg ipq8064_cfg = {
+ .ops = &ops_2_1_0,
+};
+
+static const struct qcom_pcie_cfg msm8996_cfg = {
+ .ops = &ops_2_3_2,
+};
+
+static const struct qcom_pcie_cfg ipq8074_cfg = {
+ .ops = &ops_2_3_3,
+};
+
+static const struct qcom_pcie_cfg ipq4019_cfg = {
+ .ops = &ops_2_4_0,
+};
+
+static const struct qcom_pcie_cfg sdm845_cfg = {
+ .ops = &ops_2_7_0,
+};
+
+static const struct qcom_pcie_cfg sm8250_cfg = {
+ .ops = &ops_1_9_0,
+};
+
+static const struct qcom_pcie_cfg sc7280_cfg = {
+ .ops = &ops_1_9_0,
+ .pipe_clk_need_muxing = true,
+};
+
static const struct dw_pcie_ops dw_pcie_ops = {
.link_up = qcom_pcie_link_up,
.start_link = qcom_pcie_start_link,
struct pcie_port *pp;
struct dw_pcie *pci;
struct qcom_pcie *pcie;
+ const struct qcom_pcie_cfg *pcie_cfg;
int ret;
pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
pcie->pci = pci;
- pcie->ops = of_device_get_match_data(dev);
+ pcie_cfg = of_device_get_match_data(dev);
+ if (!pcie_cfg || !pcie_cfg->ops) {
+ dev_err(dev, "Invalid platform data\n");
+ return -EINVAL;
+ }
+
+ pcie->ops = pcie_cfg->ops;
+ pcie->pipe_clk_need_muxing = pcie_cfg->pipe_clk_need_muxing;
pcie->reset = devm_gpiod_get_optional(dev, "perst", GPIOD_OUT_HIGH);
if (IS_ERR(pcie->reset)) {
}
static const struct of_device_id qcom_pcie_match[] = {
- { .compatible = "qcom,pcie-apq8084", .data = &ops_1_0_0 },
- { .compatible = "qcom,pcie-ipq8064", .data = &ops_2_1_0 },
- { .compatible = "qcom,pcie-ipq8064-v2", .data = &ops_2_1_0 },
- { .compatible = "qcom,pcie-apq8064", .data = &ops_2_1_0 },
- { .compatible = "qcom,pcie-msm8996", .data = &ops_2_3_2 },
- { .compatible = "qcom,pcie-ipq8074", .data = &ops_2_3_3 },
- { .compatible = "qcom,pcie-ipq4019", .data = &ops_2_4_0 },
- { .compatible = "qcom,pcie-qcs404", .data = &ops_2_4_0 },
- { .compatible = "qcom,pcie-sdm845", .data = &ops_2_7_0 },
- { .compatible = "qcom,pcie-sm8250", .data = &ops_1_9_0 },
+ { .compatible = "qcom,pcie-apq8084", .data = &apq8084_cfg },
+ { .compatible = "qcom,pcie-ipq8064", .data = &ipq8064_cfg },
+ { .compatible = "qcom,pcie-ipq8064-v2", .data = &ipq8064_cfg },
+ { .compatible = "qcom,pcie-apq8064", .data = &ipq8064_cfg },
+ { .compatible = "qcom,pcie-msm8996", .data = &msm8996_cfg },
+ { .compatible = "qcom,pcie-ipq8074", .data = &ipq8074_cfg },
+ { .compatible = "qcom,pcie-ipq4019", .data = &ipq4019_cfg },
+ { .compatible = "qcom,pcie-qcs404", .data = &ipq4019_cfg },
+ { .compatible = "qcom,pcie-sdm845", .data = &sdm845_cfg },
+ { .compatible = "qcom,pcie-sm8250", .data = &sm8250_cfg },
+ { .compatible = "qcom,pcie-sc8180x", .data = &sm8250_cfg },
+ { .compatible = "qcom,pcie-sc7280", .data = &sc7280_cfg },
{ }
};
writel(PCL_RCV_INTX_ALL_ENABLE, priv->base + PCL_RCV_INTX);
}
-static void uniphier_pcie_irq_ack(struct irq_data *d)
-{
- struct pcie_port *pp = irq_data_get_irq_chip_data(d);
- struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
- struct uniphier_pcie_priv *priv = to_uniphier_pcie(pci);
- u32 val;
-
- val = readl(priv->base + PCL_RCV_INTX);
- val &= ~PCL_RCV_INTX_ALL_STATUS;
- val |= BIT(irqd_to_hwirq(d) + PCL_RCV_INTX_STATUS_SHIFT);
- writel(val, priv->base + PCL_RCV_INTX);
-}
-
static void uniphier_pcie_irq_mask(struct irq_data *d)
{
struct pcie_port *pp = irq_data_get_irq_chip_data(d);
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
struct uniphier_pcie_priv *priv = to_uniphier_pcie(pci);
+ unsigned long flags;
u32 val;
+ raw_spin_lock_irqsave(&pp->lock, flags);
+
val = readl(priv->base + PCL_RCV_INTX);
- val &= ~PCL_RCV_INTX_ALL_MASK;
val |= BIT(irqd_to_hwirq(d) + PCL_RCV_INTX_MASK_SHIFT);
writel(val, priv->base + PCL_RCV_INTX);
+
+ raw_spin_unlock_irqrestore(&pp->lock, flags);
}
static void uniphier_pcie_irq_unmask(struct irq_data *d)
struct pcie_port *pp = irq_data_get_irq_chip_data(d);
struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
struct uniphier_pcie_priv *priv = to_uniphier_pcie(pci);
+ unsigned long flags;
u32 val;
+ raw_spin_lock_irqsave(&pp->lock, flags);
+
val = readl(priv->base + PCL_RCV_INTX);
- val &= ~PCL_RCV_INTX_ALL_MASK;
val &= ~BIT(irqd_to_hwirq(d) + PCL_RCV_INTX_MASK_SHIFT);
writel(val, priv->base + PCL_RCV_INTX);
+
+ raw_spin_unlock_irqrestore(&pp->lock, flags);
}
static struct irq_chip uniphier_pcie_irq_chip = {
.name = "PCI",
- .irq_ack = uniphier_pcie_irq_ack,
.irq_mask = uniphier_pcie_irq_mask,
.irq_unmask = uniphier_pcie_irq_unmask,
};
{
struct dw_pcie *pci = &pcie->pci;
struct pcie_port *pp = &pci->pp;
- struct device *dev = &pdev->dev;
pp->irq = platform_get_irq_byname(pdev, "intr");
- if (pp->irq < 0) {
- dev_err(dev, "Interrupt intr is missing");
+ if (pp->irq < 0)
return pp->irq;
- }
pp->ops = &visconti_pcie_host_ops;
/* PCIe core registers */
#define PCIE_CORE_DEV_ID_REG 0x0
#define PCIE_CORE_CMD_STATUS_REG 0x4
-#define PCIE_CORE_CMD_IO_ACCESS_EN BIT(0)
-#define PCIE_CORE_CMD_MEM_ACCESS_EN BIT(1)
-#define PCIE_CORE_CMD_MEM_IO_REQ_EN BIT(2)
#define PCIE_CORE_DEV_REV_REG 0x8
+#define PCIE_CORE_EXP_ROM_BAR_REG 0x30
#define PCIE_CORE_PCIEXP_CAP 0xc0
#define PCIE_CORE_ERR_CAPCTL_REG 0x118
#define PCIE_CORE_ERR_CAPCTL_ECRC_CHK_TX BIT(5)
#define PCIE_CORE_CTRL2_MSI_ENABLE BIT(10)
#define PCIE_CORE_REF_CLK_REG (CONTROL_BASE_ADDR + 0x14)
#define PCIE_CORE_REF_CLK_TX_ENABLE BIT(1)
+#define PCIE_CORE_REF_CLK_RX_ENABLE BIT(2)
#define PCIE_MSG_LOG_REG (CONTROL_BASE_ADDR + 0x30)
#define PCIE_ISR0_REG (CONTROL_BASE_ADDR + 0x40)
#define PCIE_MSG_PM_PME_MASK BIT(7)
#define PCIE_ISR0_MSI_INT_PENDING BIT(24)
#define PCIE_ISR0_INTX_ASSERT(val) BIT(16 + (val))
#define PCIE_ISR0_INTX_DEASSERT(val) BIT(20 + (val))
-#define PCIE_ISR0_ALL_MASK GENMASK(26, 0)
+#define PCIE_ISR0_ALL_MASK GENMASK(31, 0)
#define PCIE_ISR1_REG (CONTROL_BASE_ADDR + 0x48)
#define PCIE_ISR1_MASK_REG (CONTROL_BASE_ADDR + 0x4C)
#define PCIE_ISR1_POWER_STATE_CHANGE BIT(4)
#define PCIE_ISR1_FLUSH BIT(5)
#define PCIE_ISR1_INTX_ASSERT(val) BIT(8 + (val))
-#define PCIE_ISR1_ALL_MASK GENMASK(11, 4)
+#define PCIE_ISR1_ALL_MASK GENMASK(31, 0)
#define PCIE_MSI_ADDR_LOW_REG (CONTROL_BASE_ADDR + 0x50)
#define PCIE_MSI_ADDR_HIGH_REG (CONTROL_BASE_ADDR + 0x54)
#define PCIE_MSI_STATUS_REG (CONTROL_BASE_ADDR + 0x58)
#define PCIE_MSI_MASK_REG (CONTROL_BASE_ADDR + 0x5C)
#define PCIE_MSI_PAYLOAD_REG (CONTROL_BASE_ADDR + 0x9C)
+#define PCIE_MSI_DATA_MASK GENMASK(15, 0)
/* PCIe window configuration */
#define OB_WIN_BASE_ADDR 0x4c00
#define CFG_REG (LMI_BASE_ADDR + 0x0)
#define LTSSM_SHIFT 24
#define LTSSM_MASK 0x3f
-#define LTSSM_L0 0x10
#define RC_BAR_CONFIG 0x300
+
+/* LTSSM values in CFG_REG */
+enum {
+ LTSSM_DETECT_QUIET = 0x0,
+ LTSSM_DETECT_ACTIVE = 0x1,
+ LTSSM_POLLING_ACTIVE = 0x2,
+ LTSSM_POLLING_COMPLIANCE = 0x3,
+ LTSSM_POLLING_CONFIGURATION = 0x4,
+ LTSSM_CONFIG_LINKWIDTH_START = 0x5,
+ LTSSM_CONFIG_LINKWIDTH_ACCEPT = 0x6,
+ LTSSM_CONFIG_LANENUM_ACCEPT = 0x7,
+ LTSSM_CONFIG_LANENUM_WAIT = 0x8,
+ LTSSM_CONFIG_COMPLETE = 0x9,
+ LTSSM_CONFIG_IDLE = 0xa,
+ LTSSM_RECOVERY_RCVR_LOCK = 0xb,
+ LTSSM_RECOVERY_SPEED = 0xc,
+ LTSSM_RECOVERY_RCVR_CFG = 0xd,
+ LTSSM_RECOVERY_IDLE = 0xe,
+ LTSSM_L0 = 0x10,
+ LTSSM_RX_L0S_ENTRY = 0x11,
+ LTSSM_RX_L0S_IDLE = 0x12,
+ LTSSM_RX_L0S_FTS = 0x13,
+ LTSSM_TX_L0S_ENTRY = 0x14,
+ LTSSM_TX_L0S_IDLE = 0x15,
+ LTSSM_TX_L0S_FTS = 0x16,
+ LTSSM_L1_ENTRY = 0x17,
+ LTSSM_L1_IDLE = 0x18,
+ LTSSM_L2_IDLE = 0x19,
+ LTSSM_L2_TRANSMIT_WAKE = 0x1a,
+ LTSSM_DISABLED = 0x20,
+ LTSSM_LOOPBACK_ENTRY_MASTER = 0x21,
+ LTSSM_LOOPBACK_ACTIVE_MASTER = 0x22,
+ LTSSM_LOOPBACK_EXIT_MASTER = 0x23,
+ LTSSM_LOOPBACK_ENTRY_SLAVE = 0x24,
+ LTSSM_LOOPBACK_ACTIVE_SLAVE = 0x25,
+ LTSSM_LOOPBACK_EXIT_SLAVE = 0x26,
+ LTSSM_HOT_RESET = 0x27,
+ LTSSM_RECOVERY_EQUALIZATION_PHASE0 = 0x28,
+ LTSSM_RECOVERY_EQUALIZATION_PHASE1 = 0x29,
+ LTSSM_RECOVERY_EQUALIZATION_PHASE2 = 0x2a,
+ LTSSM_RECOVERY_EQUALIZATION_PHASE3 = 0x2b,
+};
+
#define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44)
/* PCIe core controller registers */
#define PCIE_IRQ_MSI_INT2_DET BIT(21)
#define PCIE_IRQ_RC_DBELL_DET BIT(22)
#define PCIE_IRQ_EP_STATUS BIT(23)
-#define PCIE_IRQ_ALL_MASK 0xfff0fb
+#define PCIE_IRQ_ALL_MASK GENMASK(31, 0)
#define PCIE_IRQ_ENABLE_INTS_MASK PCIE_IRQ_CORE_INT
/* Transaction types */
return readl(pcie->base + reg);
}
-static inline u16 advk_read16(struct advk_pcie *pcie, u64 reg)
+static u8 advk_pcie_ltssm_state(struct advk_pcie *pcie)
{
- return advk_readl(pcie, (reg & ~0x3)) >> ((reg & 0x3) * 8);
+ u32 val;
+ u8 ltssm_state;
+
+ val = advk_readl(pcie, CFG_REG);
+ ltssm_state = (val >> LTSSM_SHIFT) & LTSSM_MASK;
+ return ltssm_state;
}
-static int advk_pcie_link_up(struct advk_pcie *pcie)
+static inline bool advk_pcie_link_up(struct advk_pcie *pcie)
{
- u32 val, ltssm_state;
+ /* check if LTSSM is in normal operation - some L* state */
+ u8 ltssm_state = advk_pcie_ltssm_state(pcie);
+ return ltssm_state >= LTSSM_L0 && ltssm_state < LTSSM_DISABLED;
+}
- val = advk_readl(pcie, CFG_REG);
- ltssm_state = (val >> LTSSM_SHIFT) & LTSSM_MASK;
- return ltssm_state >= LTSSM_L0;
+static inline bool advk_pcie_link_active(struct advk_pcie *pcie)
+{
+ /*
+ * According to PCIe Base specification 3.0, Table 4-14: Link
+ * Status Mapped to the LTSSM, and 4.2.6.3.6 Configuration.Idle
+ * is Link Up mapped to LTSSM Configuration.Idle, Recovery, L0,
+ * L0s, L1 and L2 states. And according to 3.2.1. Data Link
+ * Control and Management State Machine Rules is DL Up status
+ * reported in DL Active state.
+ */
+ u8 ltssm_state = advk_pcie_ltssm_state(pcie);
+ return ltssm_state >= LTSSM_CONFIG_IDLE && ltssm_state < LTSSM_DISABLED;
+}
+
+static inline bool advk_pcie_link_training(struct advk_pcie *pcie)
+{
+ /*
+ * According to PCIe Base specification 3.0, Table 4-14: Link
+ * Status Mapped to the LTSSM is Link Training mapped to LTSSM
+ * Configuration and Recovery states.
+ */
+ u8 ltssm_state = advk_pcie_ltssm_state(pcie);
+ return ((ltssm_state >= LTSSM_CONFIG_LINKWIDTH_START &&
+ ltssm_state < LTSSM_L0) ||
+ (ltssm_state >= LTSSM_RECOVERY_EQUALIZATION_PHASE0 &&
+ ltssm_state <= LTSSM_RECOVERY_EQUALIZATION_PHASE3));
}
static int advk_pcie_wait_for_link(struct advk_pcie *pcie)
size_t retries;
for (retries = 0; retries < RETRAIN_WAIT_MAX_RETRIES; ++retries) {
- if (!advk_pcie_link_up(pcie))
+ if (advk_pcie_link_training(pcie))
break;
udelay(RETRAIN_WAIT_USLEEP_US);
}
static void advk_pcie_issue_perst(struct advk_pcie *pcie)
{
- u32 reg;
-
if (!pcie->reset_gpio)
return;
- /*
- * As required by PCI Express spec (PCI Express Base Specification, REV.
- * 4.0 PCI Express, February 19 2014, 6.6.1 Conventional Reset) a delay
- * for at least 100ms after de-asserting PERST# signal is needed before
- * link training is enabled. So ensure that link training is disabled
- * prior de-asserting PERST# signal to fulfill that PCI Express spec
- * requirement.
- */
- reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
- reg &= ~LINK_TRAINING_EN;
- advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
-
/* 10ms delay is needed for some cards */
dev_info(&pcie->pdev->dev, "issuing PERST via reset GPIO for 10ms\n");
gpiod_set_value_cansleep(pcie->reset_gpio, 1);
gpiod_set_value_cansleep(pcie->reset_gpio, 0);
}
-static int advk_pcie_train_at_gen(struct advk_pcie *pcie, int gen)
+static void advk_pcie_train_link(struct advk_pcie *pcie)
{
- int ret, neg_gen;
+ struct device *dev = &pcie->pdev->dev;
u32 reg;
+ int ret;
- /* Setup link speed */
+ /*
+ * Setup PCIe rev / gen compliance based on device tree property
+ * 'max-link-speed' which also forces maximal link speed.
+ */
reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
reg &= ~PCIE_GEN_SEL_MSK;
- if (gen == 3)
+ if (pcie->link_gen == 3)
reg |= SPEED_GEN_3;
- else if (gen == 2)
+ else if (pcie->link_gen == 2)
reg |= SPEED_GEN_2;
else
reg |= SPEED_GEN_1;
advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
/*
- * Enable link training. This is not needed in every call to this
- * function, just once suffices, but it does not break anything either.
+ * Set maximal link speed value also into PCIe Link Control 2 register.
+ * Armada 3700 Functional Specification says that default value is based
+ * on SPEED_GEN but tests showed that default value is always 8.0 GT/s.
*/
+ reg = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + PCI_EXP_LNKCTL2);
+ reg &= ~PCI_EXP_LNKCTL2_TLS;
+ if (pcie->link_gen == 3)
+ reg |= PCI_EXP_LNKCTL2_TLS_8_0GT;
+ else if (pcie->link_gen == 2)
+ reg |= PCI_EXP_LNKCTL2_TLS_5_0GT;
+ else
+ reg |= PCI_EXP_LNKCTL2_TLS_2_5GT;
+ advk_writel(pcie, reg, PCIE_CORE_PCIEXP_CAP + PCI_EXP_LNKCTL2);
+
+ /* Enable link training after selecting PCIe generation */
reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG);
reg |= LINK_TRAINING_EN;
advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG);
/*
- * Start link training immediately after enabling it.
- * This solves problems for some buggy cards.
- */
- reg = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + PCI_EXP_LNKCTL);
- reg |= PCI_EXP_LNKCTL_RL;
- advk_writel(pcie, reg, PCIE_CORE_PCIEXP_CAP + PCI_EXP_LNKCTL);
-
- ret = advk_pcie_wait_for_link(pcie);
- if (ret)
- return ret;
-
- reg = advk_read16(pcie, PCIE_CORE_PCIEXP_CAP + PCI_EXP_LNKSTA);
- neg_gen = reg & PCI_EXP_LNKSTA_CLS;
-
- return neg_gen;
-}
-
-static void advk_pcie_train_link(struct advk_pcie *pcie)
-{
- struct device *dev = &pcie->pdev->dev;
- int neg_gen = -1, gen;
-
- /*
* Reset PCIe card via PERST# signal. Some cards are not detected
* during link training when they are in some non-initial state.
*/
* PERST# signal could have been asserted by pinctrl subsystem before
* probe() callback has been called or issued explicitly by reset gpio
* function advk_pcie_issue_perst(), making the endpoint going into
- * fundamental reset. As required by PCI Express spec a delay for at
- * least 100ms after such a reset before link training is needed.
+ * fundamental reset. As required by PCI Express spec (PCI Express
+ * Base Specification, REV. 4.0 PCI Express, February 19 2014, 6.6.1
+ * Conventional Reset) a delay for at least 100ms after such a reset
+ * before sending a Configuration Request to the device is needed.
+ * So wait until PCIe link is up. Function advk_pcie_wait_for_link()
+ * waits for link at least 900ms.
*/
- msleep(PCI_PM_D3COLD_WAIT);
-
- /*
- * Try link training at link gen specified by device tree property
- * 'max-link-speed'. If this fails, iteratively train at lower gen.
- */
- for (gen = pcie->link_gen; gen > 0; --gen) {
- neg_gen = advk_pcie_train_at_gen(pcie, gen);
- if (neg_gen > 0)
- break;
- }
-
- if (neg_gen < 0)
- goto err;
-
- /*
- * After successful training if negotiated gen is lower than requested,
- * train again on negotiated gen. This solves some stability issues for
- * some buggy gen1 cards.
- */
- if (neg_gen < gen) {
- gen = neg_gen;
- neg_gen = advk_pcie_train_at_gen(pcie, gen);
- }
-
- if (neg_gen == gen) {
- dev_info(dev, "link up at gen %i\n", gen);
- return;
- }
-
-err:
- dev_err(dev, "link never came up\n");
+ ret = advk_pcie_wait_for_link(pcie);
+ if (ret < 0)
+ dev_err(dev, "link never came up\n");
+ else
+ dev_info(dev, "link up\n");
}
/*
u32 reg;
int i;
- /* Enable TX */
+ /*
+ * Configure PCIe Reference clock. Direction is from the PCIe
+ * controller to the endpoint card, so enable transmitting of
+ * Reference clock differential signal off-chip and disable
+ * receiving off-chip differential signal.
+ */
reg = advk_readl(pcie, PCIE_CORE_REF_CLK_REG);
reg |= PCIE_CORE_REF_CLK_TX_ENABLE;
+ reg &= ~PCIE_CORE_REF_CLK_RX_ENABLE;
advk_writel(pcie, reg, PCIE_CORE_REF_CLK_REG);
/* Set to Direct mode */
reg = (PCI_VENDOR_ID_MARVELL << 16) | PCI_VENDOR_ID_MARVELL;
advk_writel(pcie, reg, VENDOR_ID_REG);
+ /*
+ * Change Class Code of PCI Bridge device to PCI Bridge (0x600400),
+ * because the default value is Mass storage controller (0x010400).
+ *
+ * Note that this Aardvark PCI Bridge does not have compliant Type 1
+ * Configuration Space and it even cannot be accessed via Aardvark's
+ * PCI config space access method. Something like config space is
+ * available in internal Aardvark registers starting at offset 0x0
+ * and is reported as Type 0. In range 0x10 - 0x34 it has totally
+ * different registers.
+ *
+ * Therefore driver uses emulation of PCI Bridge which emulates
+ * access to configuration space via internal Aardvark registers or
+ * emulated configuration buffer.
+ */
+ reg = advk_readl(pcie, PCIE_CORE_DEV_REV_REG);
+ reg &= ~0xffffff00;
+ reg |= (PCI_CLASS_BRIDGE_PCI << 8) << 8;
+ advk_writel(pcie, reg, PCIE_CORE_DEV_REV_REG);
+
+ /* Disable Root Bridge I/O space, memory space and bus mastering */
+ reg = advk_readl(pcie, PCIE_CORE_CMD_STATUS_REG);
+ reg &= ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER);
+ advk_writel(pcie, reg, PCIE_CORE_CMD_STATUS_REG);
+
/* Set Advanced Error Capabilities and Control PF0 register */
reg = PCIE_CORE_ERR_CAPCTL_ECRC_CHK_TX |
PCIE_CORE_ERR_CAPCTL_ECRC_CHK_TX_EN |
reg = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + PCI_EXP_DEVCTL);
reg &= ~PCI_EXP_DEVCTL_RELAX_EN;
reg &= ~PCI_EXP_DEVCTL_NOSNOOP_EN;
+ reg &= ~PCI_EXP_DEVCTL_PAYLOAD;
reg &= ~PCI_EXP_DEVCTL_READRQ;
- reg |= PCI_EXP_DEVCTL_PAYLOAD; /* Set max payload size */
+ reg |= PCI_EXP_DEVCTL_PAYLOAD_512B;
reg |= PCI_EXP_DEVCTL_READRQ_512B;
advk_writel(pcie, reg, PCIE_CORE_PCIEXP_CAP + PCI_EXP_DEVCTL);
advk_pcie_disable_ob_win(pcie, i);
advk_pcie_train_link(pcie);
-
- /*
- * FIXME: The following register update is suspicious. This register is
- * applicable only when the PCI controller is configured for Endpoint
- * mode, not as a Root Complex. But apparently when this code is
- * removed, some cards stop working. This should be investigated and
- * a comment explaining this should be put here.
- */
- reg = advk_readl(pcie, PCIE_CORE_CMD_STATUS_REG);
- reg |= PCIE_CORE_CMD_MEM_ACCESS_EN |
- PCIE_CORE_CMD_IO_ACCESS_EN |
- PCIE_CORE_CMD_MEM_IO_REQ_EN;
- advk_writel(pcie, reg, PCIE_CORE_CMD_STATUS_REG);
}
static int advk_pcie_check_pio_status(struct advk_pcie *pcie, bool allow_crs, u32 *val)
u32 reg;
unsigned int status;
char *strcomp_status, *str_posted;
+ int ret;
reg = advk_readl(pcie, PIO_STAT);
status = (reg & PIO_COMPLETION_STATUS_MASK) >>
case PIO_COMPLETION_STATUS_OK:
if (reg & PIO_ERR_STATUS) {
strcomp_status = "COMP_ERR";
+ ret = -EFAULT;
break;
}
/* Get the read result */
*val = advk_readl(pcie, PIO_RD_DATA);
/* No error */
strcomp_status = NULL;
+ ret = 0;
break;
case PIO_COMPLETION_STATUS_UR:
strcomp_status = "UR";
+ ret = -EOPNOTSUPP;
break;
case PIO_COMPLETION_STATUS_CRS:
if (allow_crs && val) {
*/
*val = CFG_RD_CRS_VAL;
strcomp_status = NULL;
+ ret = 0;
break;
}
/* PCIe r4.0, sec 2.3.2, says:
* Request and taking appropriate action, e.g., complete the
* Request to the host as a failed transaction.
*
- * To simplify implementation do not re-issue the Configuration
- * Request and complete the Request as a failed transaction.
+ * So return -EAGAIN and caller (pci-aardvark.c driver) will
+ * re-issue request again up to the PIO_RETRY_CNT retries.
*/
strcomp_status = "CRS";
+ ret = -EAGAIN;
break;
case PIO_COMPLETION_STATUS_CA:
strcomp_status = "CA";
+ ret = -ECANCELED;
break;
default:
strcomp_status = "Unknown";
+ ret = -EINVAL;
break;
}
if (!strcomp_status)
- return 0;
+ return ret;
if (reg & PIO_NON_POSTED_REQ)
str_posted = "Non-posted";
else
str_posted = "Posted";
- dev_err(dev, "%s PIO Response Status: %s, %#x @ %#x\n",
+ dev_dbg(dev, "%s PIO Response Status: %s, %#x @ %#x\n",
str_posted, strcomp_status, reg, advk_readl(pcie, PIO_ADDR_LS));
- return -EFAULT;
+ return ret;
}
static int advk_pcie_wait_pio(struct advk_pcie *pcie)
struct device *dev = &pcie->pdev->dev;
int i;
- for (i = 0; i < PIO_RETRY_CNT; i++) {
+ for (i = 1; i <= PIO_RETRY_CNT; i++) {
u32 start, isr;
start = advk_readl(pcie, PIO_START);
isr = advk_readl(pcie, PIO_ISR);
if (!start && isr)
- return 0;
+ return i;
udelay(PIO_RETRY_DELAY);
}
return -ETIMEDOUT;
}
+static pci_bridge_emul_read_status_t
+advk_pci_bridge_emul_base_conf_read(struct pci_bridge_emul *bridge,
+ int reg, u32 *value)
+{
+ struct advk_pcie *pcie = bridge->data;
+
+ switch (reg) {
+ case PCI_COMMAND:
+ *value = advk_readl(pcie, PCIE_CORE_CMD_STATUS_REG);
+ return PCI_BRIDGE_EMUL_HANDLED;
+
+ case PCI_ROM_ADDRESS1:
+ *value = advk_readl(pcie, PCIE_CORE_EXP_ROM_BAR_REG);
+ return PCI_BRIDGE_EMUL_HANDLED;
+
+ case PCI_INTERRUPT_LINE: {
+ /*
+ * From the whole 32bit register we support reading from HW only
+ * one bit: PCI_BRIDGE_CTL_BUS_RESET.
+ * Other bits are retrieved only from emulated config buffer.
+ */
+ __le32 *cfgspace = (__le32 *)&bridge->conf;
+ u32 val = le32_to_cpu(cfgspace[PCI_INTERRUPT_LINE / 4]);
+ if (advk_readl(pcie, PCIE_CORE_CTRL1_REG) & HOT_RESET_GEN)
+ val |= PCI_BRIDGE_CTL_BUS_RESET << 16;
+ else
+ val &= ~(PCI_BRIDGE_CTL_BUS_RESET << 16);
+ *value = val;
+ return PCI_BRIDGE_EMUL_HANDLED;
+ }
+
+ default:
+ return PCI_BRIDGE_EMUL_NOT_HANDLED;
+ }
+}
+
+static void
+advk_pci_bridge_emul_base_conf_write(struct pci_bridge_emul *bridge,
+ int reg, u32 old, u32 new, u32 mask)
+{
+ struct advk_pcie *pcie = bridge->data;
+
+ switch (reg) {
+ case PCI_COMMAND:
+ advk_writel(pcie, new, PCIE_CORE_CMD_STATUS_REG);
+ break;
+
+ case PCI_ROM_ADDRESS1:
+ advk_writel(pcie, new, PCIE_CORE_EXP_ROM_BAR_REG);
+ break;
+
+ case PCI_INTERRUPT_LINE:
+ if (mask & (PCI_BRIDGE_CTL_BUS_RESET << 16)) {
+ u32 val = advk_readl(pcie, PCIE_CORE_CTRL1_REG);
+ if (new & (PCI_BRIDGE_CTL_BUS_RESET << 16))
+ val |= HOT_RESET_GEN;
+ else
+ val &= ~HOT_RESET_GEN;
+ advk_writel(pcie, val, PCIE_CORE_CTRL1_REG);
+ }
+ break;
+
+ default:
+ break;
+ }
+}
static pci_bridge_emul_read_status_t
advk_pci_bridge_emul_pcie_conf_read(struct pci_bridge_emul *bridge,
case PCI_EXP_RTCTL: {
u32 val = advk_readl(pcie, PCIE_ISR0_MASK_REG);
*value = (val & PCIE_MSG_PM_PME_MASK) ? 0 : PCI_EXP_RTCTL_PMEIE;
+ *value |= le16_to_cpu(bridge->pcie_conf.rootctl) & PCI_EXP_RTCTL_CRSSVE;
*value |= PCI_EXP_RTCAP_CRSVIS << 16;
return PCI_BRIDGE_EMUL_HANDLED;
}
return PCI_BRIDGE_EMUL_HANDLED;
}
+ case PCI_EXP_LNKCAP: {
+ u32 val = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + reg);
+ /*
+ * PCI_EXP_LNKCAP_DLLLARC bit is hardwired in aardvark HW to 0.
+ * But support for PCI_EXP_LNKSTA_DLLLA is emulated via ltssm
+ * state so explicitly enable PCI_EXP_LNKCAP_DLLLARC flag.
+ */
+ val |= PCI_EXP_LNKCAP_DLLLARC;
+ *value = val;
+ return PCI_BRIDGE_EMUL_HANDLED;
+ }
+
case PCI_EXP_LNKCTL: {
/* u32 contains both PCI_EXP_LNKCTL and PCI_EXP_LNKSTA */
u32 val = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + reg) &
~(PCI_EXP_LNKSTA_LT << 16);
- if (!advk_pcie_link_up(pcie))
+ if (advk_pcie_link_training(pcie))
val |= (PCI_EXP_LNKSTA_LT << 16);
+ if (advk_pcie_link_active(pcie))
+ val |= (PCI_EXP_LNKSTA_DLLLA << 16);
*value = val;
return PCI_BRIDGE_EMUL_HANDLED;
}
case PCI_CAP_LIST_ID:
case PCI_EXP_DEVCAP:
case PCI_EXP_DEVCTL:
- case PCI_EXP_LNKCAP:
*value = advk_readl(pcie, PCIE_CORE_PCIEXP_CAP + reg);
return PCI_BRIDGE_EMUL_HANDLED;
default:
}
static struct pci_bridge_emul_ops advk_pci_bridge_emul_ops = {
+ .read_base = advk_pci_bridge_emul_base_conf_read,
+ .write_base = advk_pci_bridge_emul_base_conf_write,
.read_pcie = advk_pci_bridge_emul_pcie_conf_read,
.write_pcie = advk_pci_bridge_emul_pcie_conf_write,
};
static int advk_sw_pci_bridge_init(struct advk_pcie *pcie)
{
struct pci_bridge_emul *bridge = &pcie->bridge;
- int ret;
bridge->conf.vendor =
cpu_to_le16(advk_readl(pcie, PCIE_CORE_DEV_ID_REG) & 0xffff);
/* Support interrupt A for MSI feature */
bridge->conf.intpin = PCIE_CORE_INT_A_ASSERT_ENABLE;
+ /* Indicates supports for Completion Retry Status */
+ bridge->pcie_conf.rootcap = cpu_to_le16(PCI_EXP_RTCAP_CRSVIS);
+
bridge->has_pcie = true;
bridge->data = pcie;
bridge->ops = &advk_pci_bridge_emul_ops;
- /* PCIe config space can be initialized after pci_bridge_emul_init() */
- ret = pci_bridge_emul_init(bridge, 0);
- if (ret < 0)
- return ret;
-
- /* Indicates supports for Completion Retry Status */
- bridge->pcie_conf.rootcap = cpu_to_le16(PCI_EXP_RTCAP_CRSVIS);
-
- return 0;
+ return pci_bridge_emul_init(bridge, 0);
}
static bool advk_pcie_valid_device(struct advk_pcie *pcie, struct pci_bus *bus,
int where, int size, u32 *val)
{
struct advk_pcie *pcie = bus->sysdata;
+ int retry_count;
bool allow_crs;
u32 reg;
int ret;
(le16_to_cpu(pcie->bridge.pcie_conf.rootctl) &
PCI_EXP_RTCTL_CRSSVE);
- if (advk_pcie_pio_is_running(pcie)) {
- /*
- * If it is possible return Completion Retry Status so caller
- * tries to issue the request again instead of failing.
- */
- if (allow_crs) {
- *val = CFG_RD_CRS_VAL;
- return PCIBIOS_SUCCESSFUL;
- }
- *val = 0xffffffff;
- return PCIBIOS_SET_FAILED;
- }
+ if (advk_pcie_pio_is_running(pcie))
+ goto try_crs;
/* Program the control register */
reg = advk_readl(pcie, PIO_CTRL);
/* Program the data strobe */
advk_writel(pcie, 0xf, PIO_WR_DATA_STRB);
- /* Clear PIO DONE ISR and start the transfer */
- advk_writel(pcie, 1, PIO_ISR);
- advk_writel(pcie, 1, PIO_START);
+ retry_count = 0;
+ do {
+ /* Clear PIO DONE ISR and start the transfer */
+ advk_writel(pcie, 1, PIO_ISR);
+ advk_writel(pcie, 1, PIO_START);
- ret = advk_pcie_wait_pio(pcie);
- if (ret < 0) {
- /*
- * If it is possible return Completion Retry Status so caller
- * tries to issue the request again instead of failing.
- */
- if (allow_crs) {
- *val = CFG_RD_CRS_VAL;
- return PCIBIOS_SUCCESSFUL;
- }
- *val = 0xffffffff;
- return PCIBIOS_SET_FAILED;
- }
+ ret = advk_pcie_wait_pio(pcie);
+ if (ret < 0)
+ goto try_crs;
- /* Check PIO status and get the read result */
- ret = advk_pcie_check_pio_status(pcie, allow_crs, val);
- if (ret < 0) {
- *val = 0xffffffff;
- return PCIBIOS_SET_FAILED;
- }
+ retry_count += ret;
+
+ /* Check PIO status and get the read result */
+ ret = advk_pcie_check_pio_status(pcie, allow_crs, val);
+ } while (ret == -EAGAIN && retry_count < PIO_RETRY_CNT);
+
+ if (ret < 0)
+ goto fail;
if (size == 1)
*val = (*val >> (8 * (where & 3))) & 0xff;
*val = (*val >> (8 * (where & 3))) & 0xffff;
return PCIBIOS_SUCCESSFUL;
+
+try_crs:
+ /*
+ * If it is possible, return Completion Retry Status so that caller
+ * tries to issue the request again instead of failing.
+ */
+ if (allow_crs) {
+ *val = CFG_RD_CRS_VAL;
+ return PCIBIOS_SUCCESSFUL;
+ }
+
+fail:
+ *val = 0xffffffff;
+ return PCIBIOS_SET_FAILED;
}
static int advk_pcie_wr_conf(struct pci_bus *bus, u32 devfn,
struct advk_pcie *pcie = bus->sysdata;
u32 reg;
u32 data_strobe = 0x0;
+ int retry_count;
int offset;
int ret;
/* Program the data strobe */
advk_writel(pcie, data_strobe, PIO_WR_DATA_STRB);
- /* Clear PIO DONE ISR and start the transfer */
- advk_writel(pcie, 1, PIO_ISR);
- advk_writel(pcie, 1, PIO_START);
+ retry_count = 0;
+ do {
+ /* Clear PIO DONE ISR and start the transfer */
+ advk_writel(pcie, 1, PIO_ISR);
+ advk_writel(pcie, 1, PIO_START);
- ret = advk_pcie_wait_pio(pcie);
- if (ret < 0)
- return PCIBIOS_SET_FAILED;
+ ret = advk_pcie_wait_pio(pcie);
+ if (ret < 0)
+ return PCIBIOS_SET_FAILED;
- ret = advk_pcie_check_pio_status(pcie, false, NULL);
- if (ret < 0)
- return PCIBIOS_SET_FAILED;
+ retry_count += ret;
- return PCIBIOS_SUCCESSFUL;
+ ret = advk_pcie_check_pio_status(pcie, false, NULL);
+ } while (ret == -EAGAIN && retry_count < PIO_RETRY_CNT);
+
+ return ret < 0 ? PCIBIOS_SET_FAILED : PCIBIOS_SUCCESSFUL;
}
static struct pci_ops advk_pcie_ops = {
domain->host_data, handle_simple_irq,
NULL, NULL);
- return hwirq;
+ return 0;
}
static void advk_msi_irq_domain_free(struct irq_domain *domain,
if (!(BIT(msi_idx) & msi_status))
continue;
+ /*
+ * msi_idx contains bits [4:0] of the msi_data and msi_data
+ * contains 16bit MSI interrupt number
+ */
advk_writel(pcie, BIT(msi_idx), PCIE_MSI_STATUS_REG);
- msi_data = advk_readl(pcie, PCIE_MSI_PAYLOAD_REG) & 0xFF;
+ msi_data = advk_readl(pcie, PCIE_MSI_PAYLOAD_REG) & PCIE_MSI_DATA_MASK;
generic_handle_irq(msi_data);
}
isr1_mask = advk_readl(pcie, PCIE_ISR1_MASK_REG);
isr1_status = isr1_val & ((~isr1_mask) & PCIE_ISR1_ALL_MASK);
- if (!isr0_status && !isr1_status) {
- advk_writel(pcie, isr0_val, PCIE_ISR0_REG);
- advk_writel(pcie, isr1_val, PCIE_ISR1_REG);
- return;
- }
-
/* Process MSI interrupts */
if (isr0_status & PCIE_ISR0_MSI_INT_PENDING)
advk_pcie_handle_msi(pcie);
if (dom == HVPCI_DOM_INVALID) {
dev_err(&hdev->device,
- "Unable to use dom# 0x%hx or other numbers", dom_req);
+ "Unable to use dom# 0x%x or other numbers", dom_req);
ret = -EINVAL;
goto free_bus;
}
if (dom != dom_req)
dev_info(&hdev->device,
- "PCI dom# 0x%hx has collision, using 0x%hx",
+ "PCI dom# 0x%x has collision, using 0x%x",
dom_req, dom);
hbus->bridge->domain_nr = dom;
{
int shift = (where & 3) * 8;
- pr_debug("set_val %04x: %08x\n", (unsigned)(where & ~3), v);
+ pr_debug("set_val %04x: %08x\n", (unsigned int)(where & ~3), v);
v >>= shift;
if (size == 1)
v &= 0xff;
pr_debug("%04x:%04x - Fix pass#: %08x, where: %03x, devfn: %03x\n",
vendor_device & 0xffff, vendor_device >> 16, class_rev,
- (unsigned) where, devfn);
+ (unsigned int)where, devfn);
/* Check for non type-00 header */
if (cfg_type == 0) {
/*
* MSIINTn (n is 0..F) indicates if there is a pending MSI interrupt
- * If bit x of this register is set (x is 0..7), one or more interupts
+ * If bit x of this register is set (x is 0..7), one or more interrupts
* corresponding to MSInIRx is set.
*/
grp_select = xgene_msi_int_read(xgene_msi, msi_grp);
#define EN_COHERENCY 0xF0000000
#define EN_REG 0x00000001
#define OB_LO_IO 0x00000002
-#define XGENE_PCIE_VENDORID 0x10E8
#define XGENE_PCIE_DEVICEID 0xE004
#define SZ_1T (SZ_1G*1024ULL)
#define PIPE_PHY_RATE_RD(src) ((0xc000 & (u32)(src)) >> 0xe)
xgene_pcie_clear_config(port);
/* setup the vendor and device IDs correctly */
- val = (XGENE_PCIE_DEVICEID << 16) | XGENE_PCIE_VENDORID;
+ val = (XGENE_PCIE_DEVICEID << 16) | PCI_VENDOR_ID_AMCC;
xgene_pcie_writel(port, BRIDGE_CFG_0, val);
ret = xgene_pcie_map_ranges(port);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCIe host bridge driver for Apple system-on-chips.
+ *
+ * The HW is ECAM compliant, so once the controller is initialized,
+ * the driver mostly deals MSI mapping and handling of per-port
+ * interrupts (INTx, management and error signals).
+ *
+ * Initialization requires enabling power and clocks, along with a
+ * number of register pokes.
+ *
+ * Copyright (C) 2021 Alyssa Rosenzweig <alyssa@rosenzweig.io>
+ * Copyright (C) 2021 Google LLC
+ * Copyright (C) 2021 Corellium LLC
+ * Copyright (C) 2021 Mark Kettenis <kettenis@openbsd.org>
+ *
+ * Author: Alyssa Rosenzweig <alyssa@rosenzweig.io>
+ * Author: Marc Zyngier <maz@kernel.org>
+ */
+
+#include <linux/gpio/consumer.h>
+#include <linux/kernel.h>
+#include <linux/iopoll.h>
+#include <linux/irqchip/chained_irq.h>
+#include <linux/irqdomain.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/msi.h>
+#include <linux/notifier.h>
+#include <linux/of_irq.h>
+#include <linux/pci-ecam.h>
+
+#define CORE_RC_PHYIF_CTL 0x00024
+#define CORE_RC_PHYIF_CTL_RUN BIT(0)
+#define CORE_RC_PHYIF_STAT 0x00028
+#define CORE_RC_PHYIF_STAT_REFCLK BIT(4)
+#define CORE_RC_CTL 0x00050
+#define CORE_RC_CTL_RUN BIT(0)
+#define CORE_RC_STAT 0x00058
+#define CORE_RC_STAT_READY BIT(0)
+#define CORE_FABRIC_STAT 0x04000
+#define CORE_FABRIC_STAT_MASK 0x001F001F
+#define CORE_LANE_CFG(port) (0x84000 + 0x4000 * (port))
+#define CORE_LANE_CFG_REFCLK0REQ BIT(0)
+#define CORE_LANE_CFG_REFCLK1 BIT(1)
+#define CORE_LANE_CFG_REFCLK0ACK BIT(2)
+#define CORE_LANE_CFG_REFCLKEN (BIT(9) | BIT(10))
+#define CORE_LANE_CTL(port) (0x84004 + 0x4000 * (port))
+#define CORE_LANE_CTL_CFGACC BIT(15)
+
+#define PORT_LTSSMCTL 0x00080
+#define PORT_LTSSMCTL_START BIT(0)
+#define PORT_INTSTAT 0x00100
+#define PORT_INT_TUNNEL_ERR 31
+#define PORT_INT_CPL_TIMEOUT 23
+#define PORT_INT_RID2SID_MAPERR 22
+#define PORT_INT_CPL_ABORT 21
+#define PORT_INT_MSI_BAD_DATA 19
+#define PORT_INT_MSI_ERR 18
+#define PORT_INT_REQADDR_GT32 17
+#define PORT_INT_AF_TIMEOUT 15
+#define PORT_INT_LINK_DOWN 14
+#define PORT_INT_LINK_UP 12
+#define PORT_INT_LINK_BWMGMT 11
+#define PORT_INT_AER_MASK (15 << 4)
+#define PORT_INT_PORT_ERR 4
+#define PORT_INT_INTx(i) i
+#define PORT_INT_INTx_MASK 15
+#define PORT_INTMSK 0x00104
+#define PORT_INTMSKSET 0x00108
+#define PORT_INTMSKCLR 0x0010c
+#define PORT_MSICFG 0x00124
+#define PORT_MSICFG_EN BIT(0)
+#define PORT_MSICFG_L2MSINUM_SHIFT 4
+#define PORT_MSIBASE 0x00128
+#define PORT_MSIBASE_1_SHIFT 16
+#define PORT_MSIADDR 0x00168
+#define PORT_LINKSTS 0x00208
+#define PORT_LINKSTS_UP BIT(0)
+#define PORT_LINKSTS_BUSY BIT(2)
+#define PORT_LINKCMDSTS 0x00210
+#define PORT_OUTS_NPREQS 0x00284
+#define PORT_OUTS_NPREQS_REQ BIT(24)
+#define PORT_OUTS_NPREQS_CPL BIT(16)
+#define PORT_RXWR_FIFO 0x00288
+#define PORT_RXWR_FIFO_HDR GENMASK(15, 10)
+#define PORT_RXWR_FIFO_DATA GENMASK(9, 0)
+#define PORT_RXRD_FIFO 0x0028C
+#define PORT_RXRD_FIFO_REQ GENMASK(6, 0)
+#define PORT_OUTS_CPLS 0x00290
+#define PORT_OUTS_CPLS_SHRD GENMASK(14, 8)
+#define PORT_OUTS_CPLS_WAIT GENMASK(6, 0)
+#define PORT_APPCLK 0x00800
+#define PORT_APPCLK_EN BIT(0)
+#define PORT_APPCLK_CGDIS BIT(8)
+#define PORT_STATUS 0x00804
+#define PORT_STATUS_READY BIT(0)
+#define PORT_REFCLK 0x00810
+#define PORT_REFCLK_EN BIT(0)
+#define PORT_REFCLK_CGDIS BIT(8)
+#define PORT_PERST 0x00814
+#define PORT_PERST_OFF BIT(0)
+#define PORT_RID2SID(i16) (0x00828 + 4 * (i16))
+#define PORT_RID2SID_VALID BIT(31)
+#define PORT_RID2SID_SID_SHIFT 16
+#define PORT_RID2SID_BUS_SHIFT 8
+#define PORT_RID2SID_DEV_SHIFT 3
+#define PORT_RID2SID_FUNC_SHIFT 0
+#define PORT_OUTS_PREQS_HDR 0x00980
+#define PORT_OUTS_PREQS_HDR_MASK GENMASK(9, 0)
+#define PORT_OUTS_PREQS_DATA 0x00984
+#define PORT_OUTS_PREQS_DATA_MASK GENMASK(15, 0)
+#define PORT_TUNCTRL 0x00988
+#define PORT_TUNCTRL_PERST_ON BIT(0)
+#define PORT_TUNCTRL_PERST_ACK_REQ BIT(1)
+#define PORT_TUNSTAT 0x0098c
+#define PORT_TUNSTAT_PERST_ON BIT(0)
+#define PORT_TUNSTAT_PERST_ACK_PEND BIT(1)
+#define PORT_PREFMEM_ENABLE 0x00994
+
+#define MAX_RID2SID 64
+
+/*
+ * The doorbell address is set to 0xfffff000, which by convention
+ * matches what MacOS does, and it is possible to use any other
+ * address (in the bottom 4GB, as the base register is only 32bit).
+ * However, it has to be excluded from the IOVA range, and the DART
+ * driver has to know about it.
+ */
+#define DOORBELL_ADDR CONFIG_PCIE_APPLE_MSI_DOORBELL_ADDR
+
+struct apple_pcie {
+ struct mutex lock;
+ struct device *dev;
+ void __iomem *base;
+ struct irq_domain *domain;
+ unsigned long *bitmap;
+ struct list_head ports;
+ struct completion event;
+ struct irq_fwspec fwspec;
+ u32 nvecs;
+};
+
+struct apple_pcie_port {
+ struct apple_pcie *pcie;
+ struct device_node *np;
+ void __iomem *base;
+ struct irq_domain *domain;
+ struct list_head entry;
+ DECLARE_BITMAP(sid_map, MAX_RID2SID);
+ int sid_map_sz;
+ int idx;
+};
+
+static void rmw_set(u32 set, void __iomem *addr)
+{
+ writel_relaxed(readl_relaxed(addr) | set, addr);
+}
+
+static void rmw_clear(u32 clr, void __iomem *addr)
+{
+ writel_relaxed(readl_relaxed(addr) & ~clr, addr);
+}
+
+static void apple_msi_top_irq_mask(struct irq_data *d)
+{
+ pci_msi_mask_irq(d);
+ irq_chip_mask_parent(d);
+}
+
+static void apple_msi_top_irq_unmask(struct irq_data *d)
+{
+ pci_msi_unmask_irq(d);
+ irq_chip_unmask_parent(d);
+}
+
+static struct irq_chip apple_msi_top_chip = {
+ .name = "PCIe MSI",
+ .irq_mask = apple_msi_top_irq_mask,
+ .irq_unmask = apple_msi_top_irq_unmask,
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_affinity = irq_chip_set_affinity_parent,
+ .irq_set_type = irq_chip_set_type_parent,
+};
+
+static void apple_msi_compose_msg(struct irq_data *data, struct msi_msg *msg)
+{
+ msg->address_hi = upper_32_bits(DOORBELL_ADDR);
+ msg->address_lo = lower_32_bits(DOORBELL_ADDR);
+ msg->data = data->hwirq;
+}
+
+static struct irq_chip apple_msi_bottom_chip = {
+ .name = "MSI",
+ .irq_mask = irq_chip_mask_parent,
+ .irq_unmask = irq_chip_unmask_parent,
+ .irq_eoi = irq_chip_eoi_parent,
+ .irq_set_affinity = irq_chip_set_affinity_parent,
+ .irq_set_type = irq_chip_set_type_parent,
+ .irq_compose_msi_msg = apple_msi_compose_msg,
+};
+
+static int apple_msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs, void *args)
+{
+ struct apple_pcie *pcie = domain->host_data;
+ struct irq_fwspec fwspec = pcie->fwspec;
+ unsigned int i;
+ int ret, hwirq;
+
+ mutex_lock(&pcie->lock);
+
+ hwirq = bitmap_find_free_region(pcie->bitmap, pcie->nvecs,
+ order_base_2(nr_irqs));
+
+ mutex_unlock(&pcie->lock);
+
+ if (hwirq < 0)
+ return -ENOSPC;
+
+ fwspec.param[1] += hwirq;
+
+ ret = irq_domain_alloc_irqs_parent(domain, virq, nr_irqs, &fwspec);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < nr_irqs; i++) {
+ irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
+ &apple_msi_bottom_chip,
+ domain->host_data);
+ }
+
+ return 0;
+}
+
+static void apple_msi_domain_free(struct irq_domain *domain, unsigned int virq,
+ unsigned int nr_irqs)
+{
+ struct irq_data *d = irq_domain_get_irq_data(domain, virq);
+ struct apple_pcie *pcie = domain->host_data;
+
+ mutex_lock(&pcie->lock);
+
+ bitmap_release_region(pcie->bitmap, d->hwirq, order_base_2(nr_irqs));
+
+ mutex_unlock(&pcie->lock);
+}
+
+static const struct irq_domain_ops apple_msi_domain_ops = {
+ .alloc = apple_msi_domain_alloc,
+ .free = apple_msi_domain_free,
+};
+
+static struct msi_domain_info apple_msi_info = {
+ .flags = (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
+ MSI_FLAG_MULTI_PCI_MSI | MSI_FLAG_PCI_MSIX),
+ .chip = &apple_msi_top_chip,
+};
+
+static void apple_port_irq_mask(struct irq_data *data)
+{
+ struct apple_pcie_port *port = irq_data_get_irq_chip_data(data);
+
+ writel_relaxed(BIT(data->hwirq), port->base + PORT_INTMSKSET);
+}
+
+static void apple_port_irq_unmask(struct irq_data *data)
+{
+ struct apple_pcie_port *port = irq_data_get_irq_chip_data(data);
+
+ writel_relaxed(BIT(data->hwirq), port->base + PORT_INTMSKCLR);
+}
+
+static bool hwirq_is_intx(unsigned int hwirq)
+{
+ return BIT(hwirq) & PORT_INT_INTx_MASK;
+}
+
+static void apple_port_irq_ack(struct irq_data *data)
+{
+ struct apple_pcie_port *port = irq_data_get_irq_chip_data(data);
+
+ if (!hwirq_is_intx(data->hwirq))
+ writel_relaxed(BIT(data->hwirq), port->base + PORT_INTSTAT);
+}
+
+static int apple_port_irq_set_type(struct irq_data *data, unsigned int type)
+{
+ /*
+ * It doesn't seem that there is any way to configure the
+ * trigger, so assume INTx have to be level (as per the spec),
+ * and the rest is edge (which looks likely).
+ */
+ if (hwirq_is_intx(data->hwirq) ^ !!(type & IRQ_TYPE_LEVEL_MASK))
+ return -EINVAL;
+
+ irqd_set_trigger_type(data, type);
+ return 0;
+}
+
+static struct irq_chip apple_port_irqchip = {
+ .name = "PCIe",
+ .irq_ack = apple_port_irq_ack,
+ .irq_mask = apple_port_irq_mask,
+ .irq_unmask = apple_port_irq_unmask,
+ .irq_set_type = apple_port_irq_set_type,
+};
+
+static int apple_port_irq_domain_alloc(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs,
+ void *args)
+{
+ struct apple_pcie_port *port = domain->host_data;
+ struct irq_fwspec *fwspec = args;
+ int i;
+
+ for (i = 0; i < nr_irqs; i++) {
+ irq_flow_handler_t flow = handle_edge_irq;
+ unsigned int type = IRQ_TYPE_EDGE_RISING;
+
+ if (hwirq_is_intx(fwspec->param[0] + i)) {
+ flow = handle_level_irq;
+ type = IRQ_TYPE_LEVEL_HIGH;
+ }
+
+ irq_domain_set_info(domain, virq + i, fwspec->param[0] + i,
+ &apple_port_irqchip, port, flow,
+ NULL, NULL);
+
+ irq_set_irq_type(virq + i, type);
+ }
+
+ return 0;
+}
+
+static void apple_port_irq_domain_free(struct irq_domain *domain,
+ unsigned int virq, unsigned int nr_irqs)
+{
+ int i;
+
+ for (i = 0; i < nr_irqs; i++) {
+ struct irq_data *d = irq_domain_get_irq_data(domain, virq + i);
+
+ irq_set_handler(virq + i, NULL);
+ irq_domain_reset_irq_data(d);
+ }
+}
+
+static const struct irq_domain_ops apple_port_irq_domain_ops = {
+ .translate = irq_domain_translate_onecell,
+ .alloc = apple_port_irq_domain_alloc,
+ .free = apple_port_irq_domain_free,
+};
+
+static void apple_port_irq_handler(struct irq_desc *desc)
+{
+ struct apple_pcie_port *port = irq_desc_get_handler_data(desc);
+ struct irq_chip *chip = irq_desc_get_chip(desc);
+ unsigned long stat;
+ int i;
+
+ chained_irq_enter(chip, desc);
+
+ stat = readl_relaxed(port->base + PORT_INTSTAT);
+
+ for_each_set_bit(i, &stat, 32)
+ generic_handle_domain_irq(port->domain, i);
+
+ chained_irq_exit(chip, desc);
+}
+
+static int apple_pcie_port_setup_irq(struct apple_pcie_port *port)
+{
+ struct fwnode_handle *fwnode = &port->np->fwnode;
+ unsigned int irq;
+
+ /* FIXME: consider moving each interrupt under each port */
+ irq = irq_of_parse_and_map(to_of_node(dev_fwnode(port->pcie->dev)),
+ port->idx);
+ if (!irq)
+ return -ENXIO;
+
+ port->domain = irq_domain_create_linear(fwnode, 32,
+ &apple_port_irq_domain_ops,
+ port);
+ if (!port->domain)
+ return -ENOMEM;
+
+ /* Disable all interrupts */
+ writel_relaxed(~0, port->base + PORT_INTMSKSET);
+ writel_relaxed(~0, port->base + PORT_INTSTAT);
+
+ irq_set_chained_handler_and_data(irq, apple_port_irq_handler, port);
+
+ /* Configure MSI base address */
+ BUILD_BUG_ON(upper_32_bits(DOORBELL_ADDR));
+ writel_relaxed(lower_32_bits(DOORBELL_ADDR), port->base + PORT_MSIADDR);
+
+ /* Enable MSIs, shared between all ports */
+ writel_relaxed(0, port->base + PORT_MSIBASE);
+ writel_relaxed((ilog2(port->pcie->nvecs) << PORT_MSICFG_L2MSINUM_SHIFT) |
+ PORT_MSICFG_EN, port->base + PORT_MSICFG);
+
+ return 0;
+}
+
+static irqreturn_t apple_pcie_port_irq(int irq, void *data)
+{
+ struct apple_pcie_port *port = data;
+ unsigned int hwirq = irq_domain_get_irq_data(port->domain, irq)->hwirq;
+
+ switch (hwirq) {
+ case PORT_INT_LINK_UP:
+ dev_info_ratelimited(port->pcie->dev, "Link up on %pOF\n",
+ port->np);
+ complete_all(&port->pcie->event);
+ break;
+ case PORT_INT_LINK_DOWN:
+ dev_info_ratelimited(port->pcie->dev, "Link down on %pOF\n",
+ port->np);
+ break;
+ default:
+ return IRQ_NONE;
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int apple_pcie_port_register_irqs(struct apple_pcie_port *port)
+{
+ static struct {
+ unsigned int hwirq;
+ const char *name;
+ } port_irqs[] = {
+ { PORT_INT_LINK_UP, "Link up", },
+ { PORT_INT_LINK_DOWN, "Link down", },
+ };
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(port_irqs); i++) {
+ struct irq_fwspec fwspec = {
+ .fwnode = &port->np->fwnode,
+ .param_count = 1,
+ .param = {
+ [0] = port_irqs[i].hwirq,
+ },
+ };
+ unsigned int irq;
+ int ret;
+
+ irq = irq_domain_alloc_irqs(port->domain, 1, NUMA_NO_NODE,
+ &fwspec);
+ if (WARN_ON(!irq))
+ continue;
+
+ ret = request_irq(irq, apple_pcie_port_irq, 0,
+ port_irqs[i].name, port);
+ WARN_ON(ret);
+ }
+
+ return 0;
+}
+
+static int apple_pcie_setup_refclk(struct apple_pcie *pcie,
+ struct apple_pcie_port *port)
+{
+ u32 stat;
+ int res;
+
+ res = readl_relaxed_poll_timeout(pcie->base + CORE_RC_PHYIF_STAT, stat,
+ stat & CORE_RC_PHYIF_STAT_REFCLK,
+ 100, 50000);
+ if (res < 0)
+ return res;
+
+ rmw_set(CORE_LANE_CTL_CFGACC, pcie->base + CORE_LANE_CTL(port->idx));
+ rmw_set(CORE_LANE_CFG_REFCLK0REQ, pcie->base + CORE_LANE_CFG(port->idx));
+
+ res = readl_relaxed_poll_timeout(pcie->base + CORE_LANE_CFG(port->idx),
+ stat, stat & CORE_LANE_CFG_REFCLK0ACK,
+ 100, 50000);
+ if (res < 0)
+ return res;
+
+ rmw_set(CORE_LANE_CFG_REFCLK1, pcie->base + CORE_LANE_CFG(port->idx));
+ res = readl_relaxed_poll_timeout(pcie->base + CORE_LANE_CFG(port->idx),
+ stat, stat & CORE_LANE_CFG_REFCLK1,
+ 100, 50000);
+
+ if (res < 0)
+ return res;
+
+ rmw_clear(CORE_LANE_CTL_CFGACC, pcie->base + CORE_LANE_CTL(port->idx));
+
+ rmw_set(CORE_LANE_CFG_REFCLKEN, pcie->base + CORE_LANE_CFG(port->idx));
+ rmw_set(PORT_REFCLK_EN, port->base + PORT_REFCLK);
+
+ return 0;
+}
+
+static u32 apple_pcie_rid2sid_write(struct apple_pcie_port *port,
+ int idx, u32 val)
+{
+ writel_relaxed(val, port->base + PORT_RID2SID(idx));
+ /* Read back to ensure completion of the write */
+ return readl_relaxed(port->base + PORT_RID2SID(idx));
+}
+
+static int apple_pcie_setup_port(struct apple_pcie *pcie,
+ struct device_node *np)
+{
+ struct platform_device *platform = to_platform_device(pcie->dev);
+ struct apple_pcie_port *port;
+ struct gpio_desc *reset;
+ u32 stat, idx;
+ int ret, i;
+
+ reset = gpiod_get_from_of_node(np, "reset-gpios", 0,
+ GPIOD_OUT_LOW, "#PERST");
+ if (IS_ERR(reset))
+ return PTR_ERR(reset);
+
+ port = devm_kzalloc(pcie->dev, sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return -ENOMEM;
+
+ ret = of_property_read_u32_index(np, "reg", 0, &idx);
+ if (ret)
+ return ret;
+
+ /* Use the first reg entry to work out the port index */
+ port->idx = idx >> 11;
+ port->pcie = pcie;
+ port->np = np;
+
+ port->base = devm_platform_ioremap_resource(platform, port->idx + 2);
+ if (IS_ERR(port->base))
+ return PTR_ERR(port->base);
+
+ rmw_set(PORT_APPCLK_EN, port->base + PORT_APPCLK);
+
+ ret = apple_pcie_setup_refclk(pcie, port);
+ if (ret < 0)
+ return ret;
+
+ rmw_set(PORT_PERST_OFF, port->base + PORT_PERST);
+ gpiod_set_value(reset, 1);
+
+ ret = readl_relaxed_poll_timeout(port->base + PORT_STATUS, stat,
+ stat & PORT_STATUS_READY, 100, 250000);
+ if (ret < 0) {
+ dev_err(pcie->dev, "port %pOF ready wait timeout\n", np);
+ return ret;
+ }
+
+ ret = apple_pcie_port_setup_irq(port);
+ if (ret)
+ return ret;
+
+ /* Reset all RID/SID mappings, and check for RAZ/WI registers */
+ for (i = 0; i < MAX_RID2SID; i++) {
+ if (apple_pcie_rid2sid_write(port, i, 0xbad1d) != 0xbad1d)
+ break;
+ apple_pcie_rid2sid_write(port, i, 0);
+ }
+
+ dev_dbg(pcie->dev, "%pOF: %d RID/SID mapping entries\n", np, i);
+
+ port->sid_map_sz = i;
+
+ list_add_tail(&port->entry, &pcie->ports);
+ init_completion(&pcie->event);
+
+ ret = apple_pcie_port_register_irqs(port);
+ WARN_ON(ret);
+
+ writel_relaxed(PORT_LTSSMCTL_START, port->base + PORT_LTSSMCTL);
+
+ if (!wait_for_completion_timeout(&pcie->event, HZ / 10))
+ dev_warn(pcie->dev, "%pOF link didn't come up\n", np);
+
+ return 0;
+}
+
+static int apple_msi_init(struct apple_pcie *pcie)
+{
+ struct fwnode_handle *fwnode = dev_fwnode(pcie->dev);
+ struct of_phandle_args args = {};
+ struct irq_domain *parent;
+ int ret;
+
+ ret = of_parse_phandle_with_args(to_of_node(fwnode), "msi-ranges",
+ "#interrupt-cells", 0, &args);
+ if (ret)
+ return ret;
+
+ ret = of_property_read_u32_index(to_of_node(fwnode), "msi-ranges",
+ args.args_count + 1, &pcie->nvecs);
+ if (ret)
+ return ret;
+
+ of_phandle_args_to_fwspec(args.np, args.args, args.args_count,
+ &pcie->fwspec);
+
+ pcie->bitmap = devm_bitmap_zalloc(pcie->dev, pcie->nvecs, GFP_KERNEL);
+ if (!pcie->bitmap)
+ return -ENOMEM;
+
+ parent = irq_find_matching_fwspec(&pcie->fwspec, DOMAIN_BUS_WIRED);
+ if (!parent) {
+ dev_err(pcie->dev, "failed to find parent domain\n");
+ return -ENXIO;
+ }
+
+ parent = irq_domain_create_hierarchy(parent, 0, pcie->nvecs, fwnode,
+ &apple_msi_domain_ops, pcie);
+ if (!parent) {
+ dev_err(pcie->dev, "failed to create IRQ domain\n");
+ return -ENOMEM;
+ }
+ irq_domain_update_bus_token(parent, DOMAIN_BUS_NEXUS);
+
+ pcie->domain = pci_msi_create_irq_domain(fwnode, &apple_msi_info,
+ parent);
+ if (!pcie->domain) {
+ dev_err(pcie->dev, "failed to create MSI domain\n");
+ irq_domain_remove(parent);
+ return -ENOMEM;
+ }
+
+ return 0;
+}
+
+static struct apple_pcie_port *apple_pcie_get_port(struct pci_dev *pdev)
+{
+ struct pci_config_window *cfg = pdev->sysdata;
+ struct apple_pcie *pcie = cfg->priv;
+ struct pci_dev *port_pdev;
+ struct apple_pcie_port *port;
+
+ /* Find the root port this device is on */
+ port_pdev = pcie_find_root_port(pdev);
+
+ /* If finding the port itself, nothing to do */
+ if (WARN_ON(!port_pdev) || pdev == port_pdev)
+ return NULL;
+
+ list_for_each_entry(port, &pcie->ports, entry) {
+ if (port->idx == PCI_SLOT(port_pdev->devfn))
+ return port;
+ }
+
+ return NULL;
+}
+
+static int apple_pcie_add_device(struct apple_pcie_port *port,
+ struct pci_dev *pdev)
+{
+ u32 sid, rid = PCI_DEVID(pdev->bus->number, pdev->devfn);
+ int idx, err;
+
+ dev_dbg(&pdev->dev, "added to bus %s, index %d\n",
+ pci_name(pdev->bus->self), port->idx);
+
+ err = of_map_id(port->pcie->dev->of_node, rid, "iommu-map",
+ "iommu-map-mask", NULL, &sid);
+ if (err)
+ return err;
+
+ mutex_lock(&port->pcie->lock);
+
+ idx = bitmap_find_free_region(port->sid_map, port->sid_map_sz, 0);
+ if (idx >= 0) {
+ apple_pcie_rid2sid_write(port, idx,
+ PORT_RID2SID_VALID |
+ (sid << PORT_RID2SID_SID_SHIFT) | rid);
+
+ dev_dbg(&pdev->dev, "mapping RID%x to SID%x (index %d)\n",
+ rid, sid, idx);
+ }
+
+ mutex_unlock(&port->pcie->lock);
+
+ return idx >= 0 ? 0 : -ENOSPC;
+}
+
+static void apple_pcie_release_device(struct apple_pcie_port *port,
+ struct pci_dev *pdev)
+{
+ u32 rid = PCI_DEVID(pdev->bus->number, pdev->devfn);
+ int idx;
+
+ mutex_lock(&port->pcie->lock);
+
+ for_each_set_bit(idx, port->sid_map, port->sid_map_sz) {
+ u32 val;
+
+ val = readl_relaxed(port->base + PORT_RID2SID(idx));
+ if ((val & 0xffff) == rid) {
+ apple_pcie_rid2sid_write(port, idx, 0);
+ bitmap_release_region(port->sid_map, idx, 0);
+ dev_dbg(&pdev->dev, "Released %x (%d)\n", val, idx);
+ break;
+ }
+ }
+
+ mutex_unlock(&port->pcie->lock);
+}
+
+static int apple_pcie_bus_notifier(struct notifier_block *nb,
+ unsigned long action,
+ void *data)
+{
+ struct device *dev = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct apple_pcie_port *port;
+ int err;
+
+ /*
+ * This is a bit ugly. We assume that if we get notified for
+ * any PCI device, we must be in charge of it, and that there
+ * is no other PCI controller in the whole system. It probably
+ * holds for now, but who knows for how long?
+ */
+ port = apple_pcie_get_port(pdev);
+ if (!port)
+ return NOTIFY_DONE;
+
+ switch (action) {
+ case BUS_NOTIFY_ADD_DEVICE:
+ err = apple_pcie_add_device(port, pdev);
+ if (err)
+ return notifier_from_errno(err);
+ break;
+ case BUS_NOTIFY_DEL_DEVICE:
+ apple_pcie_release_device(port, pdev);
+ break;
+ default:
+ return NOTIFY_DONE;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block apple_pcie_nb = {
+ .notifier_call = apple_pcie_bus_notifier,
+};
+
+static int apple_pcie_init(struct pci_config_window *cfg)
+{
+ struct device *dev = cfg->parent;
+ struct platform_device *platform = to_platform_device(dev);
+ struct device_node *of_port;
+ struct apple_pcie *pcie;
+ int ret;
+
+ pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
+ if (!pcie)
+ return -ENOMEM;
+
+ pcie->dev = dev;
+
+ mutex_init(&pcie->lock);
+
+ pcie->base = devm_platform_ioremap_resource(platform, 1);
+ if (IS_ERR(pcie->base))
+ return PTR_ERR(pcie->base);
+
+ cfg->priv = pcie;
+ INIT_LIST_HEAD(&pcie->ports);
+
+ for_each_child_of_node(dev->of_node, of_port) {
+ ret = apple_pcie_setup_port(pcie, of_port);
+ if (ret) {
+ dev_err(pcie->dev, "Port %pOF setup fail: %d\n", of_port, ret);
+ of_node_put(of_port);
+ return ret;
+ }
+ }
+
+ return apple_msi_init(pcie);
+}
+
+static int apple_pcie_probe(struct platform_device *pdev)
+{
+ int ret;
+
+ ret = bus_register_notifier(&pci_bus_type, &apple_pcie_nb);
+ if (ret)
+ return ret;
+
+ ret = pci_host_common_probe(pdev);
+ if (ret)
+ bus_unregister_notifier(&pci_bus_type, &apple_pcie_nb);
+
+ return ret;
+}
+
+static const struct pci_ecam_ops apple_pcie_cfg_ecam_ops = {
+ .init = apple_pcie_init,
+ .pci_ops = {
+ .map_bus = pci_ecam_map_bus,
+ .read = pci_generic_config_read,
+ .write = pci_generic_config_write,
+ }
+};
+
+static const struct of_device_id apple_pcie_of_match[] = {
+ { .compatible = "apple,pcie", .data = &apple_pcie_cfg_ecam_ops },
+ { }
+};
+MODULE_DEVICE_TABLE(of, apple_pcie_of_match);
+
+static struct platform_driver apple_pcie_driver = {
+ .probe = apple_pcie_probe,
+ .driver = {
+ .name = "pcie-apple",
+ .of_match_table = apple_pcie_of_match,
+ .suppress_bind_attrs = true,
+ },
+};
+module_platform_driver(apple_pcie_driver);
+
+MODULE_LICENSE("GPL v2");
#define BRCM_INT_PCI_MSI_LEGACY_NR 8
#define BRCM_INT_PCI_MSI_SHIFT 0
-/* MSI target adresses */
+/* MSI target addresses */
#define BRCM_MSI_TARGET_ADDR_LT_4GB 0x0fffffffcULL
#define BRCM_MSI_TARGET_ADDR_GT_4GB 0xffffffffcULL
/*
* To hold the address of the register where the MSI writes are
- * programed. When ARM GICv3 ITS is used, this should be programmed
+ * programmed. When ARM GICv3 ITS is used, this should be programmed
* with the address of the GITS_TRANSLATER register.
*/
IPROC_PCIE_MSI_ADDR_LO,
#include <linux/reset.h>
#include <linux/sys_soc.h>
-/* MediaTek specific configuration registers */
+/* MediaTek-specific configuration registers */
#define PCIE_FTS_NUM 0x70c
#define PCIE_FTS_NUM_MASK GENMASK(15, 8)
#define PCIE_FTS_NUM_L0(x) (((x) & 0xff) << 8)
/* Host-PCI bridge registers */
#define RALINK_PCI_PCICFG_ADDR 0x0000
-#define RALINK_PCI_PCIMSK_ADDR 0x000C
+#define RALINK_PCI_PCIMSK_ADDR 0x000c
#define RALINK_PCI_CONFIG_ADDR 0x0020
#define RALINK_PCI_CONFIG_DATA 0x0024
#define RALINK_PCI_MEMBASE 0x0028
-#define RALINK_PCI_IOBASE 0x002C
+#define RALINK_PCI_IOBASE 0x002c
/* PCIe RC control registers */
#define RALINK_PCI_ID 0x0030
static inline u32 mt7621_pci_get_cfgaddr(unsigned int bus, unsigned int slot,
unsigned int func, unsigned int where)
{
- return (((where & 0xF00) >> 8) << 24) | (bus << 16) | (slot << 11) |
+ return (((where & 0xf00) >> 8) << 24) | (bus << 16) | (slot << 11) |
(func << 8) | (where & 0xfc) | 0x80000000;
}
entry = resource_list_first_type(&host->windows, IORESOURCE_MEM);
if (!entry) {
- dev_err(dev, "Cannot get memory resource\n");
+ dev_err(dev, "cannot get memory resource\n");
return -EINVAL;
}
port->gpio_rst = devm_gpiod_get_index_optional(dev, "reset", slot,
GPIOD_OUT_LOW);
if (IS_ERR(port->gpio_rst)) {
- dev_err(dev, "Failed to get GPIO for PCIe%d\n", slot);
+ dev_err(dev, "failed to get GPIO for PCIe%d\n", slot);
err = PTR_ERR(port->gpio_rst);
goto remove_reset;
}
err = mt7621_pcie_init_port(port);
if (err) {
- dev_err(dev, "Initiating port %d failed\n", slot);
+ dev_err(dev, "initializing port %d failed\n", slot);
list_del(&port->list);
}
}
entry = resource_list_first_type(&host->windows, IORESOURCE_IO);
if (!entry) {
- dev_err(dev, "Cannot get io resource\n");
+ dev_err(dev, "cannot get io resource\n");
return -EINVAL;
}
err = mt7621_pcie_parse_dt(pcie);
if (err) {
- dev_err(dev, "Parsing DT failed\n");
+ dev_err(dev, "parsing DT failed\n");
return err;
}
err = mt7621_pcie_init_ports(pcie);
if (err) {
- dev_err(dev, "Nothing connected in virtual bridges\n");
+ dev_err(dev, "nothing connected in virtual bridges\n");
return 0;
}
err = mt7621_pcie_enable_ports(bridge);
if (err) {
- dev_err(dev, "Error enabling pcie ports\n");
+ dev_err(dev, "error enabling pcie ports\n");
goto remove_resets;
}
err = setup_cm_memory_region(bridge);
if (err) {
- dev_err(dev, "Error setting up iocu mem regions\n");
+ dev_err(dev, "error setting up iocu mem regions\n");
goto remove_resets;
}
* Author: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
*/
-#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/of_address.h>
-#include <linux/of_irq.h>
-#include <linux/of_pci.h>
#include <linux/of_platform.h>
#include <linux/pci.h>
#include <linux/pci-epc.h>
-#include <linux/phy/phy.h>
#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
#include "pcie-rcar.h"
#include <linux/msi.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
-#include <linux/of_pci.h>
#include <linux/of_platform.h>
#include <linux/pci.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
-#include <linux/slab.h>
#include "pcie-rcar.h"
#include <linux/device.h>
#include <linux/interrupt.h>
+#include <linux/iommu.h>
#include <linux/irq.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/rcupdate.h>
#include <asm/irqdomain.h>
-#include <asm/device.h>
-#include <asm/msi.h>
#define VMD_CFGBAR 0
#define VMD_MEMBAR1 2
VMD_FEAT_CAN_BYPASS_MSI_REMAP = (1 << 4),
};
+static DEFINE_IDA(vmd_instance_ida);
+
/*
* Lock for manipulating VMD IRQ lists.
*/
struct pci_bus *bus;
u8 busn_start;
u8 first_vec;
+ char *name;
+ int instance;
};
static inline struct vmd_dev *vmd_from_bus(struct pci_bus *bus)
INIT_LIST_HEAD(&vmd->irqs[i].irq_list);
err = devm_request_irq(&dev->dev, pci_irq_vector(dev, i),
vmd_irq, IRQF_NO_THREAD,
- "vmd", &vmd->irqs[i]);
+ vmd->name, &vmd->irqs[i]);
if (err)
return err;
}
* acceptable because the guest is usually CPU-limited and MSI
* remapping doesn't become a performance bottleneck.
*/
- if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
+ if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
+ !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
offset[0] || offset[1]) {
ret = vmd_alloc_irqs(vmd);
if (ret)
return -ENOMEM;
vmd->dev = dev;
+ vmd->instance = ida_simple_get(&vmd_instance_ida, 0, 0, GFP_KERNEL);
+ if (vmd->instance < 0)
+ return vmd->instance;
+
+ vmd->name = kasprintf(GFP_KERNEL, "vmd%d", vmd->instance);
+ if (!vmd->name) {
+ err = -ENOMEM;
+ goto out_release_instance;
+ }
+
err = pcim_enable_device(dev);
if (err < 0)
- return err;
+ goto out_release_instance;
vmd->cfgbar = pcim_iomap(dev, VMD_CFGBAR, 0);
- if (!vmd->cfgbar)
- return -ENOMEM;
+ if (!vmd->cfgbar) {
+ err = -ENOMEM;
+ goto out_release_instance;
+ }
pci_set_master(dev);
if (dma_set_mask_and_coherent(&dev->dev, DMA_BIT_MASK(64)) &&
- dma_set_mask_and_coherent(&dev->dev, DMA_BIT_MASK(32)))
- return -ENODEV;
+ dma_set_mask_and_coherent(&dev->dev, DMA_BIT_MASK(32))) {
+ err = -ENODEV;
+ goto out_release_instance;
+ }
if (features & VMD_FEAT_OFFSET_FIRST_VECTOR)
vmd->first_vec = 1;
pci_set_drvdata(dev, vmd);
err = vmd_enable_domain(vmd, features);
if (err)
- return err;
+ goto out_release_instance;
dev_info(&vmd->dev->dev, "Bound to PCI domain %04x\n",
vmd->sysdata.domain);
return 0;
+
+ out_release_instance:
+ ida_simple_remove(&vmd_instance_ida, vmd->instance);
+ kfree(vmd->name);
+ return err;
}
static void vmd_cleanup_srcu(struct vmd_dev *vmd)
vmd_cleanup_srcu(vmd);
vmd_detach_resources(vmd);
vmd_remove_irq_domain(vmd);
+ ida_simple_remove(&vmd_instance_ida, vmd->instance);
+ kfree(vmd->name);
}
#ifdef CONFIG_PM_SLEEP
for (i = 0; i < vmd->msix_count; i++) {
err = devm_request_irq(dev, pci_irq_vector(pdev, i),
vmd_irq, IRQF_NO_THREAD,
- "vmd", &vmd->irqs[i]);
+ vmd->name, &vmd->irqs[i]);
if (err)
return err;
}
struct config_group *group = to_config_group(item); \
struct epf_ntb *ntb = to_epf_ntb(group); \
\
- return sprintf(page, "%d\n", ntb->_name); \
+ return sysfs_emit(page, "%d\n", ntb->_name); \
}
#define EPF_NTB_W(_name) \
struct config_group *group = to_config_group(item); \
struct epf_ntb *ntb = to_epf_ntb(group); \
u32 val; \
- int ret; \
\
- ret = kstrtou32(page, 0, &val); \
- if (ret) \
- return ret; \
+ if (kstrtou32(page, 0, &val) < 0) \
+ return -EINVAL; \
\
ntb->_name = val; \
\
\
sscanf(#_name, "mw%d", &win_no); \
\
- return sprintf(page, "%lld\n", ntb->mws_size[win_no - 1]); \
+ return sysfs_emit(page, "%lld\n", ntb->mws_size[win_no - 1]); \
}
#define EPF_NTB_MW_W(_name) \
struct device *dev = &ntb->epf->dev; \
int win_no; \
u64 val; \
- int ret; \
\
- ret = kstrtou64(page, 0, &val); \
- if (ret) \
- return ret; \
+ if (kstrtou64(page, 0, &val) < 0) \
+ return -EINVAL; \
\
if (sscanf(#_name, "mw%d", &win_no) != 1) \
return -EINVAL; \
struct config_group *group = to_config_group(item);
struct epf_ntb *ntb = to_epf_ntb(group);
u32 val;
- int ret;
- ret = kstrtou32(page, 0, &val);
- if (ret)
- return ret;
+ if (kstrtou32(page, 0, &val) < 0)
+ return -EINVAL;
if (val > MAX_MW)
return -EINVAL;
epc = epc_group->epc;
- ret = kstrtobool(page, &start);
- if (ret)
- return ret;
+ if (kstrtobool(page, &start) < 0)
+ return -EINVAL;
if (!start) {
pci_epc_stop(epc);
static ssize_t pci_epc_start_show(struct config_item *item, char *page)
{
- return sprintf(page, "%d\n",
- to_pci_epc_group(item)->start);
+ return sysfs_emit(page, "%d\n", to_pci_epc_group(item)->start);
}
CONFIGFS_ATTR(pci_epc_, start);
struct pci_epf *epf = to_pci_epf_group(item)->epf; \
if (WARN_ON_ONCE(!epf->header)) \
return -EINVAL; \
- return sprintf(page, "0x%04x\n", epf->header->_name); \
+ return sysfs_emit(page, "0x%04x\n", epf->header->_name); \
}
#define PCI_EPF_HEADER_W_u32(_name) \
const char *page, size_t len) \
{ \
u32 val; \
- int ret; \
struct pci_epf *epf = to_pci_epf_group(item)->epf; \
if (WARN_ON_ONCE(!epf->header)) \
return -EINVAL; \
- ret = kstrtou32(page, 0, &val); \
- if (ret) \
- return ret; \
+ if (kstrtou32(page, 0, &val) < 0) \
+ return -EINVAL; \
epf->header->_name = val; \
return len; \
}
const char *page, size_t len) \
{ \
u16 val; \
- int ret; \
struct pci_epf *epf = to_pci_epf_group(item)->epf; \
if (WARN_ON_ONCE(!epf->header)) \
return -EINVAL; \
- ret = kstrtou16(page, 0, &val); \
- if (ret) \
- return ret; \
+ if (kstrtou16(page, 0, &val) < 0) \
+ return -EINVAL; \
epf->header->_name = val; \
return len; \
}
const char *page, size_t len) \
{ \
u8 val; \
- int ret; \
struct pci_epf *epf = to_pci_epf_group(item)->epf; \
if (WARN_ON_ONCE(!epf->header)) \
return -EINVAL; \
- ret = kstrtou8(page, 0, &val); \
- if (ret) \
- return ret; \
+ if (kstrtou8(page, 0, &val) < 0) \
+ return -EINVAL; \
epf->header->_name = val; \
return len; \
}
const char *page, size_t len)
{
u8 val;
- int ret;
- ret = kstrtou8(page, 0, &val);
- if (ret)
- return ret;
+ if (kstrtou8(page, 0, &val) < 0)
+ return -EINVAL;
to_pci_epf_group(item)->epf->msi_interrupts = val;
static ssize_t pci_epf_msi_interrupts_show(struct config_item *item,
char *page)
{
- return sprintf(page, "%d\n",
- to_pci_epf_group(item)->epf->msi_interrupts);
+ return sysfs_emit(page, "%d\n",
+ to_pci_epf_group(item)->epf->msi_interrupts);
}
static ssize_t pci_epf_msix_interrupts_store(struct config_item *item,
const char *page, size_t len)
{
u16 val;
- int ret;
- ret = kstrtou16(page, 0, &val);
- if (ret)
- return ret;
+ if (kstrtou16(page, 0, &val) < 0)
+ return -EINVAL;
to_pci_epf_group(item)->epf->msix_interrupts = val;
static ssize_t pci_epf_msix_interrupts_show(struct config_item *item,
char *page)
{
- return sprintf(page, "%d\n",
- to_pci_epf_group(item)->epf->msix_interrupts);
+ return sysfs_emit(page, "%d\n",
+ to_pci_epf_group(item)->epf->msix_interrupts);
}
PCI_EPF_HEADER_R(vendorid)
/**
* pci_epc_init_notify() - Notify the EPF device that EPC device's core
* initialization is completed.
- * @epc: the EPC device whose core initialization is completeds
+ * @epc: the EPC device whose core initialization is completed
*
* Invoke to Notify the EPF device that the EPC device's initialization
* is completed.
* be removed
* @epf_vf: the virtual EP function to be removed
*
- * Invoke to remove a virtual endpoint function from the physcial endpoint
+ * Invoke to remove a virtual endpoint function from the physical endpoint
* function.
*/
void pci_epf_remove_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf)
/**
* pci_epf_create() - create a new PCI EPF device
* @name: the name of the PCI EPF device. This name will be used to bind the
- * the EPF device to a EPF driver
+ * EPF device to a EPF driver
*
* Invoke to create a new PCI EPF device by providing the name of the function
* device.
* when the bridge is scanned and it loses a refcount when the bridge
* is removed.
* - When a P2P bridge is present, we elevate the refcount on the subordinate
- * bus. It loses the refcount when the the driver unloads.
+ * bus. It loses the refcount when the driver unloads.
*/
#define pr_fmt(fmt) "acpiphp_glue: " fmt
#define _CPQPHP_H
#include <linux/interrupt.h>
-#include <asm/io.h> /* for read? and write? functions */
+#include <linux/io.h> /* for read? and write? functions */
#include <linux/delay.h> /* for delays */
#include <linux/mutex.h>
#include <linux/sched/signal.h> /* for signal_pending() */
* @head: list to search
* @size: size of node to find, must be a power of two.
*
- * Description: This function sorts the resource list by size and then returns
+ * Description: This function sorts the resource list by size and then
* returns the first node of "size" length that is not in the ISA aliasing
* window. If it finds a node larger than "size" it will split it up.
*/
mdelay(5);
- /* Reenable interrupts */
+ /* Re-enable interrupts */
writel(0, ctrl->hpc_reg + INT_MASK);
pci_write_config_byte(ctrl->pci_dev, 0x41, reg);
/* This should only be for x86 as it sets the Edge Level
* Control Register
*/
- outb((u8) (temp_word & 0xFF), 0x4d0); outb((u8) ((temp_word &
- 0xFF00) >> 8), 0x4d1); rc = 0; }
+ outb((u8)(temp_word & 0xFF), 0x4d0);
+ outb((u8)((temp_word & 0xFF00) >> 8), 0x4d1);
+ rc = 0;
+ }
return rc;
}
u32 len;
int type; /* MEM, IO, PFMEM */
u8 fromMem; /* this is to indicate that the range is from
- * from the Memory bucket rather than from PFMem */
+ * the Memory bucket rather than from PFMem */
struct resource_node *next;
struct resource_node *nextRange; /* for the other mem range on bus */
};
int ibmphp_init_devno(struct slot **); /* This function is called from EBDA, so we need it not be static */
int ibmphp_do_disable_slot(struct slot *slot_cur);
-int ibmphp_update_slot_info(struct slot *); /* This function is called from HPC, so we need it to not be be static */
+int ibmphp_update_slot_info(struct slot *); /* This function is called from HPC, so we need it to not be static */
int ibmphp_configure_card(struct pci_func *, u8);
int ibmphp_unconfigure_card(struct slot **, int);
extern const struct hotplug_slot_ops ibmphp_hotplug_slot_ops;
int pciehp_set_raw_indicator_status(struct hotplug_slot *h_slot, u8 status);
int pciehp_get_raw_indicator_status(struct hotplug_slot *h_slot, u8 *status);
+int pciehp_slot_reset(struct pcie_device *dev);
+
static inline const char *slot_name(struct controller *ctrl)
{
return hotplug_slot_name(&ctrl->hotplug_slot);
.runtime_suspend = pciehp_runtime_suspend,
.runtime_resume = pciehp_runtime_resume,
#endif /* PM */
+
+ .slot_reset = pciehp_slot_reset,
};
int __init pcie_hp_init(void)
pcie_write_cmd(ctrl, 0, mask);
}
+/**
+ * pciehp_slot_reset() - ignore link event caused by error-induced hot reset
+ * @dev: PCI Express port service device
+ *
+ * Called from pcie_portdrv_slot_reset() after AER or DPC initiated a reset
+ * further up in the hierarchy to recover from an error. The reset was
+ * propagated down to this hotplug port. Ignore the resulting link flap.
+ * If the link failed to retrain successfully, synthesize the ignored event.
+ * Surprise removal during reset is detected through Presence Detect Changed.
+ */
+int pciehp_slot_reset(struct pcie_device *dev)
+{
+ struct controller *ctrl = get_service_data(dev);
+
+ if (ctrl->state != ON_STATE)
+ return 0;
+
+ pcie_capability_write_word(dev->port, PCI_EXP_SLTSTA,
+ PCI_EXP_SLTSTA_DLLSC);
+
+ if (!pciehp_check_link_active(ctrl))
+ pciehp_request(ctrl, PCI_EXP_SLTSTA_DLLSC);
+
+ return 0;
+}
+
/*
* pciehp has a 1:1 bus:slot relationship so we ultimately want a secondary
* bus reset of the bridge, but at the same time we want to ensure that it is
return zpci_deconfigure_device(zdev);
}
+static int reset_slot(struct hotplug_slot *hotplug_slot, bool probe)
+{
+ struct zpci_dev *zdev = container_of(hotplug_slot, struct zpci_dev,
+ hotplug_slot);
+
+ if (zdev->state != ZPCI_FN_STATE_CONFIGURED)
+ return -EIO;
+ /*
+ * We can't take the zdev->lock as reset_slot may be called during
+ * probing and/or device removal which already happens under the
+ * zdev->lock. Instead the user should use the higher level
+ * pci_reset_function() or pci_bus_reset() which hold the PCI device
+ * lock preventing concurrent removal. If not using these functions
+ * holding the PCI device lock is required.
+ */
+
+ /* As long as the function is configured we can reset */
+ if (probe)
+ return 0;
+
+ return zpci_hot_reset_device(zdev);
+}
+
static int get_power_status(struct hotplug_slot *hotplug_slot, u8 *value)
{
struct zpci_dev *zdev = container_of(hotplug_slot, struct zpci_dev,
static const struct hotplug_slot_ops s390_hotplug_slot_ops = {
.enable_slot = enable_slot,
.disable_slot = disable_slot,
+ .reset_slot = reset_slot,
.get_power_status = get_power_status,
.get_adapter_status = get_adapter_status,
};
mutex_lock(&slot->ctrl->cmd_lock);
if (!shpc_poll_ctrl_busy(ctrl)) {
- /* After 1 sec and and the controller is still busy */
+ /* After 1 sec and the controller is still busy */
ctrl_err(ctrl, "Controller is still busy after 1 sec\n");
retval = -EBUSY;
goto out;
{
struct pci_dev *vf_dev = to_pci_dev(dev);
struct pci_dev *pdev = pci_physfn(vf_dev);
- int val, ret;
+ int val, ret = 0;
- ret = kstrtoint(buf, 0, &val);
- if (ret)
- return ret;
+ if (kstrtoint(buf, 0, &val) < 0)
+ return -EINVAL;
if (val < 0)
return -EINVAL;
const char *buf, size_t count)
{
struct pci_dev *pdev = to_pci_dev(dev);
- int ret;
+ int ret = 0;
u16 num_vfs;
- ret = kstrtou16(buf, 0, &num_vfs);
- if (ret < 0)
- return ret;
+ if (kstrtou16(buf, 0, &num_vfs) < 0)
+ return -EINVAL;
if (num_vfs > pci_sriov_get_totalvfs(pdev))
return -ERANGE;
raw_spinlock_t *lock = &desc->dev->msi_lock;
unsigned long flags;
+ if (!desc->msi_attrib.can_mask)
+ return;
+
raw_spin_lock_irqsave(lock, flags);
desc->msi_mask &= ~clear;
desc->msi_mask |= set;
{
void __iomem *desc_addr = pci_msix_desc_addr(desc);
- writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
+ if (desc->msi_attrib.can_mask)
+ writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
}
static inline void pci_msix_mask(struct msi_desc *desc)
static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
{
- if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
- return;
-
if (desc->msi_attrib.is_msix)
pci_msix_mask(desc);
- else if (desc->msi_attrib.maskbit)
+ else
pci_msi_mask(desc, mask);
}
static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
{
- if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
- return;
-
if (desc->msi_attrib.is_msix)
pci_msix_unmask(desc);
- else if (desc->msi_attrib.maskbit)
+ else
pci_msi_unmask(desc, mask);
}
for (i = 0; i < entry->nvec_used; i++)
BUG_ON(irq_has_action(entry->irq + i));
+ if (dev->msi_irq_groups) {
+ msi_destroy_sysfs(&dev->dev, dev->msi_irq_groups);
+ dev->msi_irq_groups = NULL;
+ }
+
pci_msi_teardown_msi_irqs(dev);
list_for_each_entry_safe(entry, tmp, msi_list, list) {
list_del(&entry->list);
free_msi_entry(entry);
}
-
- if (dev->msi_irq_groups) {
- msi_destroy_sysfs(&dev->dev, dev->msi_irq_groups);
- dev->msi_irq_groups = NULL;
- }
}
static void pci_intx_for_msi(struct pci_dev *dev, int enable)
goto out;
pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
+ /* Lies, damned lies, and MSIs */
+ if (dev->dev_flags & PCI_DEV_FLAGS_HAS_MSI_MASKING)
+ control |= PCI_MSI_FLAGS_MASKBIT;
entry->msi_attrib.is_msix = 0;
entry->msi_attrib.is_64 = !!(control & PCI_MSI_FLAGS_64BIT);
entry->msi_attrib.is_virtual = 0;
entry->msi_attrib.entry_nr = 0;
- entry->msi_attrib.maskbit = !!(control & PCI_MSI_FLAGS_MASKBIT);
+ entry->msi_attrib.can_mask = !pci_msi_ignore_mask &&
+ !!(control & PCI_MSI_FLAGS_MASKBIT);
entry->msi_attrib.default_irq = dev->irq; /* Save IOAPIC IRQ */
entry->msi_attrib.multi_cap = (control & PCI_MSI_FLAGS_QMASK) >> 1;
entry->msi_attrib.multiple = ilog2(__roundup_pow_of_two(nvec));
entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
/* Save the initial mask status */
- if (entry->msi_attrib.maskbit)
+ if (entry->msi_attrib.can_mask)
pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
out:
return ret;
}
-static void __iomem *msix_map_region(struct pci_dev *dev, unsigned nr_entries)
+static void __iomem *msix_map_region(struct pci_dev *dev,
+ unsigned int nr_entries)
{
resource_size_t phys_addr;
u32 table_offset;
entry->msi_attrib.is_virtual =
entry->msi_attrib.entry_nr >= vec_count;
+ entry->msi_attrib.can_mask = !pci_msi_ignore_mask &&
+ !entry->msi_attrib.is_virtual;
+
entry->msi_attrib.default_irq = dev->irq;
entry->mask_base = base;
- if (!entry->msi_attrib.is_virtual) {
+ if (entry->msi_attrib.can_mask) {
addr = pci_msix_desc_addr(entry);
entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
}
*/
static int of_irq_parse_pci(const struct pci_dev *pdev, struct of_phandle_args *out_irq)
{
- struct device_node *dn, *ppnode;
+ struct device_node *dn, *ppnode = NULL;
struct pci_dev *ppdev;
__be32 laddr[3];
u8 pin;
if (pin == 0)
return -ENODEV;
+ /* Local interrupt-map in the device node? Use it! */
+ if (of_get_property(dn, "interrupt-map", NULL)) {
+ pin = pci_swizzle_interrupt_pin(pdev, pin);
+ ppnode = dn;
+ }
+
/* Now we walk up the PCI tree */
- for (;;) {
+ while (!ppnode) {
/* Get the pci_dev of our parent */
ppdev = pdev->bus->self;
int i;
for_each_sg(sg, s, nents, i) {
- s->dma_address = sg_phys(s) - p2p_pgmap->bus_offset;
+ s->dma_address = sg_phys(s) + p2p_pgmap->bus_offset;
sg_dma_len(s) = s->length;
}
*
* Parses an attribute value to decide whether to enable p2pdma.
* The value can select a PCI device (using its full BDF device
- * name) or a boolean (in any format strtobool() accepts). A false
+ * name) or a boolean (in any format kstrtobool() accepts). A false
* value disables p2pdma, a true value expects the caller
* to automatically find a compatible device and specifying a PCI device
* expects the caller to use the specific provider.
} else if ((page[0] == '0' || page[0] == '1') && !iscntrl(page[1])) {
/*
* If the user enters a PCI device that doesn't exist
- * like "0000:01:00.1", we don't want strtobool to think
+ * like "0000:01:00.1", we don't want kstrtobool to think
* it's a '0' when it's clearly not what the user wanted.
* So we require 0's and 1's to be exactly one character.
*/
- } else if (!strtobool(page, use_p2pdma)) {
+ } else if (!kstrtobool(page, use_p2pdma)) {
return 0;
}
/* Clear the W1C bits */
new &= ~((value << shift) & (behavior[reg / 4].w1c & mask));
+ /* Save the new value with the cleared W1C bits into the cfgspace */
cfgspace[reg / 4] = cpu_to_le32(new);
+ /*
+ * Clear the W1C bits not specified by the write mask, so that the
+ * write_op() does not clear them.
+ */
+ new &= ~(behavior[reg / 4].w1c & ~mask);
+
+ /*
+ * Set the W1C bits specified by the write mask, so that write_op()
+ * knows about that they are to be cleared.
+ */
+ new |= (value << shift) & (behavior[reg / 4].w1c & mask);
+
if (write_op)
write_op(bridge, reg, old, new, mask);
const struct pci_device_id *id;
int error = 0;
- if (!pci_dev->driver && drv->probe) {
+ if (drv->probe) {
error = -ENODEV;
id = pci_match_device(drv, pci_dev);
struct pci_dev *pci_dev = to_pci_dev(dev);
struct pci_driver *drv = pci_dev->driver;
- if (drv) {
- if (drv->remove) {
- pm_runtime_get_sync(dev);
- drv->remove(pci_dev);
- pm_runtime_put_noidle(dev);
- }
- pcibios_free_irq(pci_dev);
- pci_dev->driver = NULL;
- pci_iov_remove(pci_dev);
+ if (drv->remove) {
+ pm_runtime_get_sync(dev);
+ drv->remove(pci_dev);
+ pm_runtime_put_noidle(dev);
}
+ pcibios_free_irq(pci_dev);
+ pci_dev->driver = NULL;
+ pci_iov_remove(pci_dev);
/* Undo the runtime PM settings in local_pci_probe() */
pm_runtime_put_sync(dev);
{
int retval;
- /* if the device was enabled before suspend, reenable */
+ /* if the device was enabled before suspend, re-enable */
retval = pci_reenable_device(pci_dev);
/*
* if the device was busmaster before the suspend, make it busmaster
return 0;
}
-#if defined(CONFIG_PCIEPORTBUS) || defined(CONFIG_EEH)
+#if defined(CONFIG_PCIEAER) || defined(CONFIG_EEH)
/**
* pci_uevent_ers - emit a uevent during recovery path of PCI device
* @pdev: PCI device undergoing error recovery
#include <linux/slab.h>
#include <linux/vgaarb.h>
#include <linux/pm_runtime.h>
+#include <linux/msi.h>
#include <linux/of.h>
#include "pci.h"
pci_config_attr(subsystem_device, "0x%04x\n");
pci_config_attr(revision, "0x%02x\n");
pci_config_attr(class, "0x%06x\n");
-pci_config_attr(irq, "%u\n");
+
+static ssize_t irq_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+#ifdef CONFIG_PCI_MSI
+ /*
+ * For MSI, show the first MSI IRQ; for all other cases including
+ * MSI-X, show the legacy INTx IRQ.
+ */
+ if (pdev->msi_enabled) {
+ struct msi_desc *desc = first_pci_msi_entry(pdev);
+
+ return sysfs_emit(buf, "%u\n", desc->irq);
+ }
+#endif
+
+ return sysfs_emit(buf, "%u\n", pdev->irq);
+}
+static DEVICE_ATTR_RO(irq);
static ssize_t broken_parity_status_show(struct device *dev,
struct device_attribute *attr,
{
struct pci_dev *pdev = to_pci_dev(dev);
unsigned long val;
- ssize_t result = kstrtoul(buf, 0, &val);
-
- if (result < 0)
- return result;
+ ssize_t result = 0;
/* this can crash the machine when done on the "wrong" device */
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
+ if (kstrtoul(buf, 0, &val) < 0)
+ return -EINVAL;
+
device_lock(dev);
if (dev->driver)
result = -EBUSY;
size_t count)
{
struct pci_dev *pdev = to_pci_dev(dev);
- int node, ret;
+ int node;
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- ret = kstrtoint(buf, 0, &node);
- if (ret)
- return ret;
+ if (kstrtoint(buf, 0, &node) < 0)
+ return -EINVAL;
if ((node < 0 && node != NUMA_NO_NODE) || node >= MAX_NUMNODES)
return -EINVAL;
struct pci_bus *subordinate = pdev->subordinate;
unsigned long val;
- if (kstrtoul(buf, 0, &val) < 0)
- return -EINVAL;
-
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
+ if (kstrtoul(buf, 0, &val) < 0)
+ return -EINVAL;
+
/*
* "no_msi" and "bus_flags" only affect what happens when a driver
* requests MSI or MSI-X. They don't affect any drivers that have
{
struct pci_dev *pdev = to_pci_dev(dev);
unsigned long val;
- ssize_t result = kstrtoul(buf, 0, &val);
+ ssize_t result;
- if (result < 0)
- return result;
+ if (kstrtoul(buf, 0, &val) < 0)
+ return -EINVAL;
if (val != 1)
return -EINVAL;
const char **endptr)
{
int ret;
- int seg, bus, slot, func;
+ unsigned int seg, bus, slot, func;
char *wpath, *p;
char end;
EXPORT_SYMBOL_GPL(pci_find_vsec_capability);
/**
+ * pci_find_dvsec_capability - Find DVSEC for vendor
+ * @dev: PCI device to query
+ * @vendor: Vendor ID to match for the DVSEC
+ * @dvsec: Designated Vendor-specific capability ID
+ *
+ * If DVSEC has Vendor ID @vendor and DVSEC ID @dvsec return the capability
+ * offset in config space; otherwise return 0.
+ */
+u16 pci_find_dvsec_capability(struct pci_dev *dev, u16 vendor, u16 dvsec)
+{
+ int pos;
+
+ pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_DVSEC);
+ if (!pos)
+ return 0;
+
+ while (pos) {
+ u16 v, id;
+
+ pci_read_config_word(dev, pos + PCI_DVSEC_HEADER1, &v);
+ pci_read_config_word(dev, pos + PCI_DVSEC_HEADER2, &id);
+ if (vendor == v && dvsec == id)
+ return pos;
+
+ pos = pci_find_next_ext_capability(dev, pos, PCI_EXT_CAP_ID_DVSEC);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pci_find_dvsec_capability);
+
+/**
* pci_find_parent_resource - return resource region of parent bus of given
* region
* @dev: PCI device structure contains resources to be searched
return 0;
}
+void pci_bridge_reconfigure_ltr(struct pci_dev *dev)
+{
+#ifdef CONFIG_PCIEASPM
+ struct pci_dev *bridge;
+ u32 ctl;
+
+ bridge = pci_upstream_bridge(dev);
+ if (bridge && bridge->ltr_path) {
+ pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl);
+ if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) {
+ pci_dbg(bridge, "re-enabling LTR\n");
+ pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2,
+ PCI_EXP_DEVCTL2_LTR_EN);
+ }
+ }
+#endif
+}
+
static void pci_restore_pcie_state(struct pci_dev *dev)
{
int i = 0;
if (!save_state)
return;
+ /*
+ * Downstream ports reset the LTR enable bit when link goes down.
+ * Check and re-configure the bit here before restoring device.
+ * PCIe r5.0, sec 7.5.3.16.
+ */
+ pci_bridge_reconfigure_ltr(dev);
+
cap = (u16 *)&save_state->cap.data[0];
pcie_capability_write_word(dev, PCI_EXP_DEVCTL, cap[i++]);
pcie_capability_write_word(dev, PCI_EXP_LNKCTL, cap[i++]);
EXPORT_SYMBOL(pcim_pin_device);
/*
- * pcibios_add_device - provide arch specific hooks when adding device dev
+ * pcibios_device_add - provide arch specific hooks when adding device dev
* @dev: the PCI device being added
*
* Permits the platform to provide architecture specific functionality when
* devices are added. This is the default implementation. Architecture
* implementations can override this.
*/
-int __weak pcibios_add_device(struct pci_dev *dev)
+int __weak pcibios_device_add(struct pci_dev *dev)
{
return 0;
}
}
EXPORT_SYMBOL_GPL(pci_set_pcie_reset_state);
+#ifdef CONFIG_PCIEAER
void pcie_clear_device_status(struct pci_dev *dev)
{
u16 sta;
pcie_capability_read_word(dev, PCI_EXP_DEVSTA, &sta);
pcie_capability_write_word(dev, PCI_EXP_DEVSTA, sta);
}
+#endif
/**
* pcie_clear_root_pme_status - Clear root port PME interrupt status.
struct pci_dev *bridge;
u32 cap, ctl2;
+ /*
+ * Per PCIe r5.0, sec 9.3.5.10, the AtomicOp Requester Enable bit
+ * in Device Control 2 is reserved in VFs and the PF value applies
+ * to all associated VFs.
+ */
+ if (dev->is_virtfn)
+ return -EINVAL;
+
if (!pci_is_pcie(dev))
return -EINVAL;
return pci_parent_bus_reset(dev, probe);
}
-static void pci_dev_lock(struct pci_dev *dev)
+void pci_dev_lock(struct pci_dev *dev)
{
pci_cfg_access_lock(dev);
/* block PM suspend, driver probe, etc. */
device_lock(&dev->dev);
}
+EXPORT_SYMBOL_GPL(pci_dev_lock);
/* Return 1 on successful lock, 0 on contention */
int pci_dev_trylock(struct pci_dev *dev)
*/
int __pci_reset_function_locked(struct pci_dev *dev)
{
- int i, m, rc = -ENOTTY;
+ int i, m, rc;
might_sleep();
* cannot be left as a userspace activity). DMA aliases should therefore
* be configured via quirks, such as the PCI fixup header quirk.
*/
-void pci_add_dma_alias(struct pci_dev *dev, u8 devfn_from, unsigned nr_devfns)
+void pci_add_dma_alias(struct pci_dev *dev, u8 devfn_from,
+ unsigned int nr_devfns)
{
int devfn_to;
- nr_devfns = min(nr_devfns, (unsigned) MAX_NR_DEVFNS - devfn_from);
+ nr_devfns = min(nr_devfns, (unsigned int)MAX_NR_DEVFNS - devfn_from);
devfn_to = devfn_from + nr_devfns - 1;
if (!dev->dma_alias_mask)
bool pci_bridge_d3_possible(struct pci_dev *dev);
void pci_bridge_d3_update(struct pci_dev *dev);
void pci_bridge_wait_for_secondary_bus(struct pci_dev *dev);
+void pci_bridge_reconfigure_ltr(struct pci_dev *dev);
static inline void pci_wakeup_event(struct pci_dev *dev)
{
#
# Makefile for PCI Express features and port driver
-pcieportdrv-y := portdrv_core.o portdrv_pci.o err.o rcec.o
+pcieportdrv-y := portdrv_core.o portdrv_pci.o rcec.o
obj-$(CONFIG_PCIEPORTBUS) += pcieportdrv.o
obj-$(CONFIG_PCIEASPM) += aspm.o
-obj-$(CONFIG_PCIEAER) += aer.o
+obj-$(CONFIG_PCIEAER) += aer.o err.o
obj-$(CONFIG_PCIEAER_INJECT) += aer_inject.o
obj-$(CONFIG_PCIE_PME) += pme.o
obj-$(CONFIG_PCIE_DPC) += dpc.o
* "as seen by this device". Note that this may mean that if an
* end point is causing problems, the AER counters may increment
* at its link partner (e.g. root port) because the errors will be
- * "seen" by the link partner and not the the problematic end point
+ * "seen" by the link partner and not the problematic end point
* itself (which may report all counters as 0 as it never saw any
* problems).
*/
struct pcie_link_state *link = pcie_aspm_get_link(pdev);
bool state_enable;
- if (strtobool(buf, &state_enable) < 0)
+ if (kstrtobool(buf, &state_enable) < 0)
return -EINVAL;
down_read(&pci_bus_sem);
struct pcie_link_state *link = pcie_aspm_get_link(pdev);
bool state_enable;
- if (strtobool(buf, &state_enable) < 0)
+ if (kstrtobool(buf, &state_enable) < 0)
return -EINVAL;
down_read(&pci_bus_sem);
pci_channel_state_t state,
enum pci_ers_result *result)
{
+ struct pci_driver *pdrv;
pci_ers_result_t vote;
const struct pci_error_handlers *err_handler;
device_lock(&dev->dev);
+ pdrv = dev->driver;
if (!pci_dev_set_io_state(dev, state) ||
- !dev->driver ||
- !dev->driver->err_handler ||
- !dev->driver->err_handler->error_detected) {
+ !pdrv ||
+ !pdrv->err_handler ||
+ !pdrv->err_handler->error_detected) {
/*
* If any device in the subtree does not have an error_detected
* callback, PCI_ERS_RESULT_NO_AER_DRIVER prevents subsequent
vote = PCI_ERS_RESULT_NONE;
}
} else {
- err_handler = dev->driver->err_handler;
+ err_handler = pdrv->err_handler;
vote = err_handler->error_detected(dev, state);
}
pci_uevent_ers(dev, vote);
static int report_mmio_enabled(struct pci_dev *dev, void *data)
{
+ struct pci_driver *pdrv;
pci_ers_result_t vote, *result = data;
const struct pci_error_handlers *err_handler;
device_lock(&dev->dev);
- if (!dev->driver ||
- !dev->driver->err_handler ||
- !dev->driver->err_handler->mmio_enabled)
+ pdrv = dev->driver;
+ if (!pdrv ||
+ !pdrv->err_handler ||
+ !pdrv->err_handler->mmio_enabled)
goto out;
- err_handler = dev->driver->err_handler;
+ err_handler = pdrv->err_handler;
vote = err_handler->mmio_enabled(dev);
*result = merge_result(*result, vote);
out:
static int report_slot_reset(struct pci_dev *dev, void *data)
{
+ struct pci_driver *pdrv;
pci_ers_result_t vote, *result = data;
const struct pci_error_handlers *err_handler;
device_lock(&dev->dev);
- if (!dev->driver ||
- !dev->driver->err_handler ||
- !dev->driver->err_handler->slot_reset)
+ pdrv = dev->driver;
+ if (!pdrv ||
+ !pdrv->err_handler ||
+ !pdrv->err_handler->slot_reset)
goto out;
- err_handler = dev->driver->err_handler;
+ err_handler = pdrv->err_handler;
vote = err_handler->slot_reset(dev);
*result = merge_result(*result, vote);
out:
static int report_resume(struct pci_dev *dev, void *data)
{
+ struct pci_driver *pdrv;
const struct pci_error_handlers *err_handler;
device_lock(&dev->dev);
+ pdrv = dev->driver;
if (!pci_dev_set_io_state(dev, pci_channel_io_normal) ||
- !dev->driver ||
- !dev->driver->err_handler ||
- !dev->driver->err_handler->resume)
+ !pdrv ||
+ !pdrv->err_handler ||
+ !pdrv->err_handler->resume)
goto out;
- err_handler = dev->driver->err_handler;
+ err_handler = pdrv->err_handler;
err_handler->resume(dev);
out:
pci_uevent_ers(dev, PCI_ERS_RESULT_RECOVERED);
int (*runtime_suspend)(struct pcie_device *dev);
int (*runtime_resume)(struct pcie_device *dev);
- /* Device driver may resume normal operations */
- void (*error_resume)(struct pci_dev *dev);
+ int (*slot_reset)(struct pcie_device *dev);
int port_type; /* Type of the port this driver can handle */
u32 service; /* Port service this device represents */
extern struct bus_type pcie_port_bus_type;
int pcie_port_device_register(struct pci_dev *dev);
+int pcie_port_device_iter(struct device *dev, void *data);
#ifdef CONFIG_PM
int pcie_port_device_suspend(struct device *dev);
int pcie_port_device_resume_noirq(struct device *dev);
int pcie_port_device_runtime_resume(struct device *dev);
#endif
void pcie_port_device_remove(struct pci_dev *dev);
-int __must_check pcie_port_bus_register(void);
-void pcie_port_bus_unregister(void);
struct pci_dev;
{
int ret, i;
- for (i = 0; i < PCIE_PORT_DEVICE_MAXSERVICES; i++)
- irqs[i] = -1;
-
/*
* If we support PME but can't use MSI/MSI-X for it, we have to
* fall back to INTx or other interrupts, e.g., a system shared
*/
int pcie_port_device_register(struct pci_dev *dev)
{
- int status, capabilities, i, nr_service;
- int irqs[PCIE_PORT_DEVICE_MAXSERVICES];
+ int status, capabilities, irq_services, i, nr_service;
+ int irqs[PCIE_PORT_DEVICE_MAXSERVICES] = {
+ [0 ... PCIE_PORT_DEVICE_MAXSERVICES-1] = -1
+ };
/* Enable PCI Express port device */
status = pci_enable_device(dev);
return 0;
pci_set_master(dev);
- /*
- * Initialize service irqs. Don't use service devices that
- * require interrupts if there is no way to generate them.
- * However, some drivers may have a polling mode (e.g. pciehp_poll_mode)
- * that can be used in the absence of irqs. Allow them to determine
- * if that is to be used.
- */
- status = pcie_init_service_irqs(dev, irqs, capabilities);
- if (status) {
- capabilities &= PCIE_PORT_SERVICE_HP;
- if (!capabilities)
- goto error_disable;
+
+ irq_services = 0;
+ if (IS_ENABLED(CONFIG_PCIE_PME))
+ irq_services |= PCIE_PORT_SERVICE_PME;
+ if (IS_ENABLED(CONFIG_PCIEAER))
+ irq_services |= PCIE_PORT_SERVICE_AER;
+ if (IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE))
+ irq_services |= PCIE_PORT_SERVICE_HP;
+ if (IS_ENABLED(CONFIG_PCIE_DPC))
+ irq_services |= PCIE_PORT_SERVICE_DPC;
+ irq_services &= capabilities;
+
+ if (irq_services) {
+ /*
+ * Initialize service IRQs. Don't use service devices that
+ * require interrupts if there is no way to generate them.
+ * However, some drivers may have a polling mode (e.g.
+ * pciehp_poll_mode) that can be used in the absence of IRQs.
+ * Allow them to determine if that is to be used.
+ */
+ status = pcie_init_service_irqs(dev, irqs, irq_services);
+ if (status) {
+ irq_services &= PCIE_PORT_SERVICE_HP;
+ if (!irq_services)
+ goto error_disable;
+ }
}
/* Allocate child services if any */
return status;
}
-#ifdef CONFIG_PM
-typedef int (*pcie_pm_callback_t)(struct pcie_device *);
+typedef int (*pcie_callback_t)(struct pcie_device *);
-static int pm_iter(struct device *dev, void *data)
+int pcie_port_device_iter(struct device *dev, void *data)
{
struct pcie_port_service_driver *service_driver;
size_t offset = *(size_t *)data;
- pcie_pm_callback_t cb;
+ pcie_callback_t cb;
if ((dev->bus == &pcie_port_bus_type) && dev->driver) {
service_driver = to_service_driver(dev->driver);
- cb = *(pcie_pm_callback_t *)((void *)service_driver + offset);
+ cb = *(pcie_callback_t *)((void *)service_driver + offset);
if (cb)
return cb(to_pcie_device(dev));
}
return 0;
}
+#ifdef CONFIG_PM
/**
* pcie_port_device_suspend - suspend port services associated with a PCIe port
* @dev: PCI Express port to handle
int pcie_port_device_suspend(struct device *dev)
{
size_t off = offsetof(struct pcie_port_service_driver, suspend);
- return device_for_each_child(dev, &off, pm_iter);
+ return device_for_each_child(dev, &off, pcie_port_device_iter);
}
int pcie_port_device_resume_noirq(struct device *dev)
{
size_t off = offsetof(struct pcie_port_service_driver, resume_noirq);
- return device_for_each_child(dev, &off, pm_iter);
+ return device_for_each_child(dev, &off, pcie_port_device_iter);
}
/**
int pcie_port_device_resume(struct device *dev)
{
size_t off = offsetof(struct pcie_port_service_driver, resume);
- return device_for_each_child(dev, &off, pm_iter);
+ return device_for_each_child(dev, &off, pcie_port_device_iter);
}
/**
int pcie_port_device_runtime_suspend(struct device *dev)
{
size_t off = offsetof(struct pcie_port_service_driver, runtime_suspend);
- return device_for_each_child(dev, &off, pm_iter);
+ return device_for_each_child(dev, &off, pcie_port_device_iter);
}
/**
int pcie_port_device_runtime_resume(struct device *dev)
{
size_t off = offsetof(struct pcie_port_service_driver, runtime_resume);
- return device_for_each_child(dev, &off, pm_iter);
+ return device_for_each_child(dev, &off, pcie_port_device_iter);
}
#endif /* PM */
static pci_ers_result_t pcie_portdrv_slot_reset(struct pci_dev *dev)
{
+ size_t off = offsetof(struct pcie_port_service_driver, slot_reset);
+ device_for_each_child(&dev->dev, &off, pcie_port_device_iter);
+
pci_restore_state(dev);
pci_save_state(dev);
return PCI_ERS_RESULT_RECOVERED;
return PCI_ERS_RESULT_RECOVERED;
}
-static int resume_iter(struct device *device, void *data)
-{
- struct pcie_device *pcie_device;
- struct pcie_port_service_driver *driver;
-
- if (device->bus == &pcie_port_bus_type && device->driver) {
- driver = to_service_driver(device->driver);
- if (driver && driver->error_resume) {
- pcie_device = to_pcie_device(device);
-
- /* Forward error message to service drivers */
- driver->error_resume(pcie_device->port);
- }
- }
-
- return 0;
-}
-
-static void pcie_portdrv_err_resume(struct pci_dev *dev)
-{
- device_for_each_child(&dev->dev, NULL, resume_iter);
-}
-
/*
* LINUX Device Driver Model
*/
.error_detected = pcie_portdrv_error_detected,
.slot_reset = pcie_portdrv_slot_reset,
.mmio_enabled = pcie_portdrv_mmio_enabled,
- .resume = pcie_portdrv_err_resume,
};
static struct pci_driver pcie_portdriver = {
static int pci_register_host_bridge(struct pci_host_bridge *bridge)
{
struct device *parent = bridge->dev.parent;
- struct resource_entry *window, *n;
+ struct resource_entry *window, *next, *n;
struct pci_bus *bus, *b;
- resource_size_t offset;
+ resource_size_t offset, next_offset;
LIST_HEAD(resources);
- struct resource *res;
+ struct resource *res, *next_res;
char addr[64], *fmt;
const char *name;
int err;
if (nr_node_ids > 1 && pcibus_to_node(bus) == NUMA_NO_NODE)
dev_warn(&bus->dev, "Unknown NUMA node; performance will be reduced\n");
+ /* Coalesce contiguous windows */
+ resource_list_for_each_entry_safe(window, n, &resources) {
+ if (list_is_last(&window->node, &resources))
+ break;
+
+ next = list_next_entry(window, node);
+ offset = window->offset;
+ res = window->res;
+ next_offset = next->offset;
+ next_res = next->res;
+
+ if (res->flags != next_res->flags || offset != next_offset)
+ continue;
+
+ if (res->end + 1 == next_res->start) {
+ next_res->start = res->start;
+ res->flags = res->start = res->end = 0;
+ }
+ }
+
/* Add initial resources to the bus */
resource_list_for_each_entry_safe(window, n, &resources) {
- list_move_tail(&window->node, &bridge->windows);
offset = window->offset;
res = window->res;
+ if (!res->end)
+ continue;
+
+ list_move_tail(&window->node, &bridge->windows);
if (res->flags & IORESOURCE_BUS)
pci_bus_insert_busn_res(bus, bus->number, res->end);
* Complex and all intermediate Switches indicate support for LTR.
* PCIe r4.0, sec 6.18.
*/
- if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
- ((bridge = pci_upstream_bridge(dev)) &&
- bridge->ltr_path)) {
+ if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) {
+ pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
+ PCI_EXP_DEVCTL2_LTR_EN);
+ dev->ltr_path = 1;
+ return;
+ }
+
+ /*
+ * If we're configuring a hot-added device, LTR was likely
+ * disabled in the upstream bridge, so re-enable it before enabling
+ * it in the new device.
+ */
+ bridge = pci_upstream_bridge(dev);
+ if (bridge && bridge->ltr_path) {
+ pci_bridge_reconfigure_ltr(dev);
pcie_capability_set_word(dev, PCI_EXP_DEVCTL2,
PCI_EXP_DEVCTL2_LTR_EN);
dev->ltr_path = 1;
struct irq_domain *d;
/*
- * If a domain has been set through the pcibios_add_device()
+ * If a domain has been set through the pcibios_device_add()
* callback, then this is the one (platform code knows best).
*/
d = dev_get_msi_domain(&dev->dev);
list_add_tail(&dev->bus_list, &bus->devices);
up_write(&pci_bus_sem);
- ret = pcibios_add_device(dev);
+ ret = pcibios_device_add(dev);
WARN_ON(ret < 0);
/* Set up MSI IRQ domain */
}
EXPORT_SYMBOL(pci_scan_single_device);
-static unsigned next_fn(struct pci_bus *bus, struct pci_dev *dev, unsigned fn)
+static unsigned int next_fn(struct pci_bus *bus, struct pci_dev *dev,
+ unsigned int fn)
{
int pos;
u16 cap = 0;
- unsigned next_fn;
+ unsigned int next_fn;
if (pci_ari_enabled(bus)) {
if (!dev)
*/
int pci_scan_slot(struct pci_bus *bus, int devfn)
{
- unsigned fn, nr = 0;
+ unsigned int fn, nr = 0;
struct pci_dev *dev;
if (only_one_child(bus) && (devfn > 0))
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_S3, PCI_DEVICE_ID_S3_868, quirk_s3_64M);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_S3, PCI_DEVICE_ID_S3_968, quirk_s3_64M);
-static void quirk_io(struct pci_dev *dev, int pos, unsigned size,
+static void quirk_io(struct pci_dev *dev, int pos, unsigned int size,
const char *name)
{
u32 region;
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_CS5536_ISA, quirk_cs5536_vsa);
static void quirk_io_region(struct pci_dev *dev, int port,
- unsigned size, int nr, const char *name)
+ unsigned int size, int nr, const char *name)
{
u16 region;
struct pci_bus_region bus_region;
base = devres & 0xffff;
size = 16;
for (;;) {
- unsigned bit = size >> 1;
+ unsigned int bit = size >> 1;
if ((bit & mask) == bit)
break;
size = bit;
mask = (devres & 0x3f) << 16;
size = 128 << 16;
for (;;) {
- unsigned bit = size >> 1;
+ unsigned int bit = size >> 1;
if ((bit & mask) == bit)
break;
size = bit;
"ICH6 GPIO");
}
-static void ich6_lpc_generic_decode(struct pci_dev *dev, unsigned reg,
+static void ich6_lpc_generic_decode(struct pci_dev *dev, unsigned int reg,
const char *name, int dynsize)
{
u32 val;
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH6_0, quirk_ich6_lpc);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH6_1, quirk_ich6_lpc);
-static void ich7_lpc_generic_decode(struct pci_dev *dev, unsigned reg,
+static void ich7_lpc_generic_decode(struct pci_dev *dev, unsigned int reg,
const char *name)
{
u32 val;
* then the device can't use INTx interrupts. Tegra's PCIe root ports don't
* generate MSI interrupts for PME and AER events instead only INTx interrupts
* are generated. Though Tegra's PCIe root ports can generate MSI interrupts
- * for other events, since PCIe specificiation doesn't support using a mix of
+ * for other events, since PCIe specification doesn't support using a mix of
* INTx and MSI/MSI-X, it is required to disable MSI interrupts to avoid port
* service drivers registering their respective ISRs for MSIs.
*/
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c, quirk_no_bus_reset);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033, quirk_no_bus_reset);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034, quirk_no_bus_reset);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003e, quirk_no_bus_reset);
/*
* Root port on some Cavium CN8xxx chips do not successfully complete a bus
}
DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
+
+/*
+ * Pericom PI7C9X2G404/PI7C9X2G304/PI7C9X2G303 switch erratum E5 -
+ * ACS P2P Request Redirect is not functional
+ *
+ * When ACS P2P Request Redirect is enabled and bandwidth is not balanced
+ * between upstream and downstream ports, packets are queued in an internal
+ * buffer until CPLD packet. The workaround is to use the switch in store and
+ * forward mode.
+ */
+#define PI7C9X2Gxxx_MODE_REG 0x74
+#define PI7C9X2Gxxx_STORE_FORWARD_MODE BIT(0)
+static void pci_fixup_pericom_acs_store_forward(struct pci_dev *pdev)
+{
+ struct pci_dev *upstream;
+ u16 val;
+
+ /* Downstream ports only */
+ if (pci_pcie_type(pdev) != PCI_EXP_TYPE_DOWNSTREAM)
+ return;
+
+ /* Check for ACS P2P Request Redirect use */
+ if (!pdev->acs_cap)
+ return;
+ pci_read_config_word(pdev, pdev->acs_cap + PCI_ACS_CTRL, &val);
+ if (!(val & PCI_ACS_RR))
+ return;
+
+ upstream = pci_upstream_bridge(pdev);
+ if (!upstream)
+ return;
+
+ pci_read_config_word(upstream, PI7C9X2Gxxx_MODE_REG, &val);
+ if (!(val & PI7C9X2Gxxx_STORE_FORWARD_MODE)) {
+ pci_info(upstream, "Setting PI7C9X2Gxxx store-forward mode to avoid ACS erratum\n");
+ pci_write_config_word(upstream, PI7C9X2Gxxx_MODE_REG, val |
+ PI7C9X2Gxxx_STORE_FORWARD_MODE);
+ }
+}
+/*
+ * Apply fixup on enable and on resume, in order to apply the fix up whenever
+ * ACS configuration changes or switch mode is reset
+ */
+DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2404,
+ pci_fixup_pericom_acs_store_forward);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2404,
+ pci_fixup_pericom_acs_store_forward);
+DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2304,
+ pci_fixup_pericom_acs_store_forward);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2304,
+ pci_fixup_pericom_acs_store_forward);
+DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2303,
+ pci_fixup_pericom_acs_store_forward);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2303,
+ pci_fixup_pericom_acs_store_forward);
+
+static void nvidia_ion_ahci_fixup(struct pci_dev *pdev)
+{
+ pdev->dev_flags |= PCI_DEV_FLAGS_HAS_MSI_MASKING;
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x0ab8, nvidia_ion_ahci_fixup);
{
void __iomem *image;
int last_image;
- unsigned length;
+ unsigned int length;
image = rom;
do {
{
struct pci_dev *dev = bus->self;
struct resource *r;
- unsigned old_flags = 0;
+ unsigned int old_flags = 0;
struct resource *b_res;
int idx = 1;
* David Miller (davem@redhat.com)
*/
-
#include <linux/kernel.h>
#include <linux/pci.h>
#include <linux/errno.h>
return;
}
- /* If this device is not on the primary bus, we need to figure out
- which interrupt pin it will come in on. We know which slot it
- will come in on 'cos that slot is where the bridge is. Each
- time the interrupt line passes through a PCI-PCI bridge we must
- apply the swizzle function. */
-
+ /*
+ * If this device is not on the primary bus, we need to figure out
+ * which interrupt pin it will come in on. We know which slot it
+ * will come in on because that slot is where the bridge is. Each
+ * time the interrupt line passes through a PCI-PCI bridge we must
+ * apply the swizzle function.
+ */
pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &pin);
/* Cope with illegal. */
if (pin > 4)
pin = 1;
if (pin) {
- /* Follow the chain of bridges, swizzling as we go. */
+ /* Follow the chain of bridges, swizzling as we go. */
if (hbrg->swizzle_irq)
slot = (*(hbrg->swizzle_irq))(dev, &pin);
/*
- * If a swizzling function is not used map_irq must
- * ignore slot
+ * If a swizzling function is not used, map_irq() must
+ * ignore slot.
*/
irq = (*(hbrg->map_irq))(dev, slot, pin);
if (irq == -1)
pci_dbg(dev, "assign IRQ: got %d\n", dev->irq);
- /* Always tell the device, so the driver knows what is
- the real IRQ to use; the device does not use it. */
+ /*
+ * Always tell the device, so the driver knows what is the real IRQ
+ * to use; the device does not use it.
+ */
pci_write_config_byte(dev, PCI_INTERRUPT_LINE, irq);
}
MRPC_QUEUED,
MRPC_RUNNING,
MRPC_DONE,
+ MRPC_IO_ERROR,
};
struct switchtec_user {
int event_cnt;
};
+/*
+ * The MMIO reads to the device_id register should always return the device ID
+ * of the device, otherwise the firmware is probably stuck or unreachable
+ * due to a firmware reset which clears PCI state including the BARs and Memory
+ * Space Enable bits.
+ */
+static int is_firmware_running(struct switchtec_dev *stdev)
+{
+ u32 device = ioread32(&stdev->mmio_sys_info->device_id);
+
+ return stdev->pdev->device == device;
+}
+
static struct switchtec_user *stuser_create(struct switchtec_dev *stdev)
{
struct switchtec_user *stuser;
[MRPC_QUEUED] = "QUEUED",
[MRPC_RUNNING] = "RUNNING",
[MRPC_DONE] = "DONE",
+ [MRPC_IO_ERROR] = "IO_ERROR",
};
stuser->state = state;
return 0;
}
+static void mrpc_cleanup_cmd(struct switchtec_dev *stdev)
+{
+ /* requires the mrpc_mutex to already be held when called */
+
+ struct switchtec_user *stuser = list_entry(stdev->mrpc_queue.next,
+ struct switchtec_user, list);
+
+ stuser->cmd_done = true;
+ wake_up_interruptible(&stuser->cmd_comp);
+ list_del_init(&stuser->list);
+ stuser_put(stuser);
+ stdev->mrpc_busy = 0;
+
+ mrpc_cmd_submit(stdev);
+}
+
static void mrpc_complete_cmd(struct switchtec_dev *stdev)
{
/* requires the mrpc_mutex to already be held when called */
+
struct switchtec_user *stuser;
if (list_empty(&stdev->mrpc_queue))
stuser_set_state(stuser, MRPC_DONE);
stuser->return_code = 0;
- if (stuser->status != SWITCHTEC_MRPC_STATUS_DONE)
+ if (stuser->status != SWITCHTEC_MRPC_STATUS_DONE &&
+ stuser->status != SWITCHTEC_MRPC_STATUS_ERROR)
goto out;
if (stdev->dma_mrpc)
memcpy_fromio(stuser->data, &stdev->mmio_mrpc->output_data,
stuser->read_len);
out:
- stuser->cmd_done = true;
- wake_up_interruptible(&stuser->cmd_comp);
- list_del_init(&stuser->list);
- stuser_put(stuser);
- stdev->mrpc_busy = 0;
-
- mrpc_cmd_submit(stdev);
+ mrpc_cleanup_cmd(stdev);
}
static void mrpc_event_work(struct work_struct *work)
mutex_unlock(&stdev->mrpc_mutex);
}
+static void mrpc_error_complete_cmd(struct switchtec_dev *stdev)
+{
+ /* requires the mrpc_mutex to already be held when called */
+
+ struct switchtec_user *stuser;
+
+ if (list_empty(&stdev->mrpc_queue))
+ return;
+
+ stuser = list_entry(stdev->mrpc_queue.next,
+ struct switchtec_user, list);
+
+ stuser_set_state(stuser, MRPC_IO_ERROR);
+
+ mrpc_cleanup_cmd(stdev);
+}
+
static void mrpc_timeout_work(struct work_struct *work)
{
struct switchtec_dev *stdev;
mutex_lock(&stdev->mrpc_mutex);
+ if (!is_firmware_running(stdev)) {
+ mrpc_error_complete_cmd(stdev);
+ goto out;
+ }
+
if (stdev->dma_mrpc)
status = stdev->dma_mrpc->status;
else
return io_string_show(buf, &si->gen4.field, \
sizeof(si->gen4.field)); \
else \
- return -ENOTSUPP; \
+ return -EOPNOTSUPP; \
} \
\
static DEVICE_ATTR_RO(field)
if (rc)
return rc;
+ if (stuser->state == MRPC_IO_ERROR) {
+ mutex_unlock(&stdev->mrpc_mutex);
+ return -EIO;
+ }
+
if (stuser->state != MRPC_DONE) {
mutex_unlock(&stdev->mrpc_mutex);
return -EBADE;
out:
mutex_unlock(&stdev->mrpc_mutex);
- if (stuser->status == SWITCHTEC_MRPC_STATUS_DONE)
+ if (stuser->status == SWITCHTEC_MRPC_STATUS_DONE ||
+ stuser->status == SWITCHTEC_MRPC_STATUS_ERROR)
return size;
else if (stuser->status == SWITCHTEC_MRPC_STATUS_INTERRUPTED)
return -ENXIO;
info.flash_length = ioread32(&fi->gen4.flash_length);
info.num_partitions = SWITCHTEC_NUM_PARTITIONS_GEN4;
} else {
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
if (copy_to_user(uinfo, &info, sizeof(info)))
if (ret)
return ret;
} else {
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
}
if (copy_to_user(uinfo, &info, sizeof(info)))
return PTR_ERR(reg);
hdr = ioread32(reg);
+ if (hdr & SWITCHTEC_EVENT_NOT_SUPP)
+ return -EOPNOTSUPP;
+
for (i = 0; i < ARRAY_SIZE(ctl->data); i++)
ctl->data[i] = ioread32(®[i + 1]);
for (ctl.index = 0; ctl.index < nr_idxs; ctl.index++) {
ctl.flags = event_flags;
ret = event_ctl(stdev, &ctl);
- if (ret < 0)
+ if (ret < 0 && ret != -EOPNOTSUPP)
return ret;
}
} else {
break;
}
- reg = ioread32(&pcfg->vep_pff_inst_id);
+ reg = ioread32(&pcfg->vep_pff_inst_id) & 0xFF;
if (reg == p.pff) {
p.port = SWITCHTEC_IOCTL_PFF_VEP;
break;
p.pff = ioread32(&pcfg->usp_pff_inst_id);
break;
case SWITCHTEC_IOCTL_PFF_VEP:
- p.pff = ioread32(&pcfg->vep_pff_inst_id);
+ p.pff = ioread32(&pcfg->vep_pff_inst_id) & 0xFF;
break;
default:
if (p.port > ARRAY_SIZE(pcfg->dsp_pff_inst_id))
hdr_reg = event_regs[eid].map_reg(stdev, off, idx);
hdr = ioread32(hdr_reg);
+ if (hdr & SWITCHTEC_EVENT_NOT_SUPP)
+ return 0;
+
if (!(hdr & SWITCHTEC_EVENT_OCCURRED && hdr & SWITCHTEC_EVENT_EN_IRQ))
return 0;
if (reg < stdev->pff_csr_count)
stdev->pff_local[reg] = 1;
- reg = ioread32(&pcfg->vep_pff_inst_id);
+ reg = ioread32(&pcfg->vep_pff_inst_id) & 0xFF;
if (reg < stdev->pff_csr_count)
stdev->pff_local[reg] = 1;
else if (stdev->gen == SWITCHTEC_GEN4)
part_id = &stdev->mmio_sys_info->gen4.partition_id;
else
- return -ENOTSUPP;
+ return -EOPNOTSUPP;
stdev->partition = ioread8(part_id);
stdev->partition_count = ioread8(&stdev->mmio_ntb->partition_count);
size_t off = 0, size;
unsigned char tag, header[1+2]; /* 1 byte tag, 2 bytes length */
- /* Otherwise the following reads would fail. */
- dev->vpd.len = PCI_VPD_MAX_SIZE;
-
- while (pci_read_vpd(dev, off, 1, header) == 1) {
+ while (pci_read_vpd_any(dev, off, 1, header) == 1) {
size = 0;
if (off == 0 && (header[0] == 0x00 || header[0] == 0xff))
if (header[0] & PCI_VPD_LRDT) {
/* Large Resource Data Type Tag */
- if (pci_read_vpd(dev, off + 1, 2, &header[1]) != 2) {
+ if (pci_read_vpd_any(dev, off + 1, 2, &header[1]) != 2) {
pci_warn(dev, "failed VPD read at offset %zu\n",
off + 1);
return off ?: PCI_VPD_SZ_INVALID;
return off ?: PCI_VPD_SZ_INVALID;
}
-static bool pci_vpd_available(struct pci_dev *dev)
+static bool pci_vpd_available(struct pci_dev *dev, bool check_size)
{
struct pci_vpd *vpd = &dev->vpd;
if (!vpd->cap)
return false;
- if (vpd->len == 0) {
+ if (vpd->len == 0 && check_size) {
vpd->len = pci_vpd_size(dev);
if (vpd->len == PCI_VPD_SZ_INVALID) {
vpd->cap = 0;
}
static ssize_t pci_vpd_read(struct pci_dev *dev, loff_t pos, size_t count,
- void *arg)
+ void *arg, bool check_size)
{
struct pci_vpd *vpd = &dev->vpd;
+ unsigned int max_len;
int ret = 0;
loff_t end = pos + count;
u8 *buf = arg;
- if (!pci_vpd_available(dev))
+ if (!pci_vpd_available(dev, check_size))
return -ENODEV;
if (pos < 0)
return -EINVAL;
- if (pos > vpd->len)
+ max_len = check_size ? vpd->len : PCI_VPD_MAX_SIZE;
+
+ if (pos >= max_len)
return 0;
- if (end > vpd->len) {
- end = vpd->len;
+ if (end > max_len) {
+ end = max_len;
count = end - pos;
}
}
static ssize_t pci_vpd_write(struct pci_dev *dev, loff_t pos, size_t count,
- const void *arg)
+ const void *arg, bool check_size)
{
struct pci_vpd *vpd = &dev->vpd;
+ unsigned int max_len;
const u8 *buf = arg;
loff_t end = pos + count;
int ret = 0;
- if (!pci_vpd_available(dev))
+ if (!pci_vpd_available(dev, check_size))
return -ENODEV;
if (pos < 0 || (pos & 3) || (count & 3))
return -EINVAL;
- if (end > vpd->len)
+ max_len = check_size ? vpd->len : PCI_VPD_MAX_SIZE;
+
+ if (end > max_len)
return -EINVAL;
if (mutex_lock_killable(&vpd->lock))
void *buf;
int cnt;
- if (!pci_vpd_available(dev))
+ if (!pci_vpd_available(dev, true))
return ERR_PTR(-ENODEV);
len = dev->vpd.len;
return -ENOENT;
}
+static ssize_t __pci_read_vpd(struct pci_dev *dev, loff_t pos, size_t count, void *buf,
+ bool check_size)
+{
+ ssize_t ret;
+
+ if (dev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0) {
+ dev = pci_get_func0_dev(dev);
+ if (!dev)
+ return -ENODEV;
+
+ ret = pci_vpd_read(dev, pos, count, buf, check_size);
+ pci_dev_put(dev);
+ return ret;
+ }
+
+ return pci_vpd_read(dev, pos, count, buf, check_size);
+}
+
/**
* pci_read_vpd - Read one entry from Vital Product Data
* @dev: PCI device struct
*/
ssize_t pci_read_vpd(struct pci_dev *dev, loff_t pos, size_t count, void *buf)
{
+ return __pci_read_vpd(dev, pos, count, buf, true);
+}
+EXPORT_SYMBOL(pci_read_vpd);
+
+/* Same, but allow to access any address */
+ssize_t pci_read_vpd_any(struct pci_dev *dev, loff_t pos, size_t count, void *buf)
+{
+ return __pci_read_vpd(dev, pos, count, buf, false);
+}
+EXPORT_SYMBOL(pci_read_vpd_any);
+
+static ssize_t __pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count,
+ const void *buf, bool check_size)
+{
ssize_t ret;
if (dev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0) {
if (!dev)
return -ENODEV;
- ret = pci_vpd_read(dev, pos, count, buf);
+ ret = pci_vpd_write(dev, pos, count, buf, check_size);
pci_dev_put(dev);
return ret;
}
- return pci_vpd_read(dev, pos, count, buf);
+ return pci_vpd_write(dev, pos, count, buf, check_size);
}
-EXPORT_SYMBOL(pci_read_vpd);
/**
* pci_write_vpd - Write entry to Vital Product Data
*/
ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void *buf)
{
- ssize_t ret;
-
- if (dev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0) {
- dev = pci_get_func0_dev(dev);
- if (!dev)
- return -ENODEV;
-
- ret = pci_vpd_write(dev, pos, count, buf);
- pci_dev_put(dev);
- return ret;
- }
-
- return pci_vpd_write(dev, pos, count, buf);
+ return __pci_write_vpd(dev, pos, count, buf, true);
}
EXPORT_SYMBOL(pci_write_vpd);
+/* Same, but allow to access any address */
+ssize_t pci_write_vpd_any(struct pci_dev *dev, loff_t pos, size_t count, const void *buf)
+{
+ return __pci_write_vpd(dev, pos, count, buf, false);
+}
+EXPORT_SYMBOL(pci_write_vpd_any);
+
int pci_vpd_find_ro_info_keyword(const void *buf, unsigned int len,
const char *kw, unsigned int *size)
{
struct pcifront_device *pdev,
pci_channel_state_t state)
{
- pci_ers_result_t result;
struct pci_driver *pdrv;
int bus = pdev->sh_info->aer_op.bus;
int devfn = pdev->sh_info->aer_op.devfn;
int domain = pdev->sh_info->aer_op.domain;
struct pci_dev *pcidev;
- int flag = 0;
dev_dbg(&pdev->xdev->dev,
"pcifront AER process: cmd %x (bus:%x, devfn%x)",
cmd, bus, devfn);
- result = PCI_ERS_RESULT_NONE;
pcidev = pci_get_domain_bus_and_slot(domain, bus, devfn);
- if (!pcidev || !pcidev->driver) {
+ if (!pcidev || !pcidev->dev.driver) {
dev_err(&pdev->xdev->dev, "device or AER driver is NULL\n");
pci_dev_put(pcidev);
- return result;
+ return PCI_ERS_RESULT_NONE;
}
- pdrv = pcidev->driver;
-
- if (pdrv) {
- if (pdrv->err_handler && pdrv->err_handler->error_detected) {
- pci_dbg(pcidev, "trying to call AER service\n");
- if (pcidev) {
- flag = 1;
- switch (cmd) {
- case XEN_PCI_OP_aer_detected:
- result = pdrv->err_handler->
- error_detected(pcidev, state);
- break;
- case XEN_PCI_OP_aer_mmio:
- result = pdrv->err_handler->
- mmio_enabled(pcidev);
- break;
- case XEN_PCI_OP_aer_slotreset:
- result = pdrv->err_handler->
- slot_reset(pcidev);
- break;
- case XEN_PCI_OP_aer_resume:
- pdrv->err_handler->resume(pcidev);
- break;
- default:
- dev_err(&pdev->xdev->dev,
- "bad request in aer recovery "
- "operation!\n");
-
- }
- }
+ pdrv = to_pci_driver(pcidev->dev.driver);
+
+ if (pdrv->err_handler && pdrv->err_handler->error_detected) {
+ pci_dbg(pcidev, "trying to call AER service\n");
+ switch (cmd) {
+ case XEN_PCI_OP_aer_detected:
+ return pdrv->err_handler->error_detected(pcidev, state);
+ case XEN_PCI_OP_aer_mmio:
+ return pdrv->err_handler->mmio_enabled(pcidev);
+ case XEN_PCI_OP_aer_slotreset:
+ return pdrv->err_handler->slot_reset(pcidev);
+ case XEN_PCI_OP_aer_resume:
+ pdrv->err_handler->resume(pcidev);
+ return PCI_ERS_RESULT_NONE;
+ default:
+ dev_err(&pdev->xdev->dev,
+ "bad request in aer recovery operation!\n");
}
}
- if (!flag)
- result = PCI_ERS_RESULT_NONE;
- return result;
+ return PCI_ERS_RESULT_NONE;
}
#define PIN_IRQ_PENDING (BIT(INTERRUPT_STS_OFF) | BIT(WAKE_STS_OFF))
-static irqreturn_t amd_gpio_irq_handler(int irq, void *dev_id)
+static bool do_amd_gpio_irq_handler(int irq, void *dev_id)
{
struct amd_gpio *gpio_dev = dev_id;
struct gpio_chip *gc = &gpio_dev->gc;
- irqreturn_t ret = IRQ_NONE;
unsigned int i, irqnr;
unsigned long flags;
u32 __iomem *regs;
+ bool ret = false;
u32 regval;
u64 status, mask;
/* Each status bit covers four pins */
for (i = 0; i < 4; i++) {
regval = readl(regs + i);
+ /* caused wake on resume context for shared IRQ */
+ if (irq < 0 && (regval & BIT(WAKE_STS_OFF))) {
+ dev_dbg(&gpio_dev->pdev->dev,
+ "Waking due to GPIO %d: 0x%x",
+ irqnr + i, regval);
+ return true;
+ }
+
if (!(regval & PIN_IRQ_PENDING) ||
!(regval & BIT(INTERRUPT_MASK_OFF)))
continue;
}
writel(regval, regs + i);
raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
- ret = IRQ_HANDLED;
+ ret = true;
}
}
+ /* did not cause wake on resume context for shared IRQ */
+ if (irq < 0)
+ return false;
/* Signal EOI to the GPIO unit */
raw_spin_lock_irqsave(&gpio_dev->lock, flags);
return ret;
}
+static irqreturn_t amd_gpio_irq_handler(int irq, void *dev_id)
+{
+ return IRQ_RETVAL(do_amd_gpio_irq_handler(irq, dev_id));
+}
+
+static bool __maybe_unused amd_gpio_check_wake(void *dev_id)
+{
+ return do_amd_gpio_irq_handler(-1, dev_id);
+}
+
static int amd_get_groups_count(struct pinctrl_dev *pctldev)
{
struct amd_gpio *gpio_dev = pinctrl_dev_get_drvdata(pctldev);
goto out2;
platform_set_drvdata(pdev, gpio_dev);
+ acpi_register_wakeup_handler(gpio_dev->irq, amd_gpio_check_wake, gpio_dev);
dev_dbg(&pdev->dev, "amd gpio driver loaded\n");
return ret;
gpio_dev = platform_get_drvdata(pdev);
gpiochip_remove(&gpio_dev->gc);
+ acpi_unregister_wakeup_handler(amd_gpio_check_wake, gpio_dev);
return 0;
}
pctl->base + REG_IRQ(irqgrp, data->hwirq));
}
-static int apple_gpio_irq_type(unsigned int type)
+static unsigned int apple_gpio_irq_type(unsigned int type)
{
switch (type & IRQ_TYPE_SENSE_MASK) {
case IRQ_TYPE_EDGE_RISING:
case IRQ_TYPE_LEVEL_LOW:
return REG_GPIOx_IN_IRQ_LO;
default:
- return -EINVAL;
+ return REG_GPIOx_IN_IRQ_OFF;
}
}
{
struct apple_gpio_pinctrl *pctl =
gpiochip_get_data(irq_data_get_irq_chip_data(data));
- int irqtype = apple_gpio_irq_type(irqd_get_trigger_type(data));
+ unsigned int irqtype = apple_gpio_irq_type(irqd_get_trigger_type(data));
apple_gpio_set_reg(pctl, data->hwirq, REG_GPIOx_MODE,
FIELD_PREP(REG_GPIOx_MODE, irqtype));
{
struct apple_gpio_pinctrl *pctl =
gpiochip_get_data(irq_data_get_irq_chip_data(data));
- int irqtype = apple_gpio_irq_type(type);
+ unsigned int irqtype = apple_gpio_irq_type(type);
- if (irqtype < 0)
- return irqtype;
+ if (irqtype == REG_GPIOx_IN_IRQ_OFF)
+ return -EINVAL;
apple_gpio_set_reg(pctl, data->hwirq, REG_GPIOx_MODE,
FIELD_PREP(REG_GPIOx_MODE, irqtype));
select PINMUX
select PINCONF
select GENERIC_PINCONF
+ select GPIOLIB
select GPIOLIB_IRQCHIP
select IRQ_DOMAIN_HIERARCHY
help
select PINMUX
select PINCONF
select GENERIC_PINCONF
+ select GPIOLIB
select GPIOLIB_IRQCHIP
select IRQ_DOMAIN_HIERARCHY
help
.ngpios = 151,
.wakeirq_map = sdm845_pdc_map,
.nwakeirq_map = ARRAY_SIZE(sdm845_pdc_map),
+ .wakeirq_dual_edge_errata = true,
};
static const struct msm_pinctrl_soc_data sdm845_acpi_pinctrl = {
[200] = PINGROUP(200, qdss_gpio, _, _, _, _, _, _, _, _),
[201] = PINGROUP(201, _, _, _, _, _, _, _, _, _),
[202] = PINGROUP(202, _, _, _, _, _, _, _, _, _),
- [203] = UFS_RESET(ufs_reset, 0x1d8000),
- [204] = SDC_PINGROUP(sdc2_clk, 0x1cf000, 14, 6),
- [205] = SDC_PINGROUP(sdc2_cmd, 0x1cf000, 11, 3),
- [206] = SDC_PINGROUP(sdc2_data, 0x1cf000, 9, 0),
+ [203] = UFS_RESET(ufs_reset, 0xd8000),
+ [204] = SDC_PINGROUP(sdc2_clk, 0xcf000, 14, 6),
+ [205] = SDC_PINGROUP(sdc2_cmd, 0xcf000, 11, 3),
+ [206] = SDC_PINGROUP(sdc2_data, 0xcf000, 9, 0),
};
static const struct msm_gpio_wakeirq_map sm8350_pdc_map[] = {
// SPDX-License-Identifier: GPL-2.0-only
+#include <asm/mach-ralink/ralink_regs.h>
#include <asm/mach-ralink/mt7620.h>
#include <linux/module.h>
#include <linux/platform_device.h>
return 0;
}
-static struct tegra_pingroup *tegra_pinctrl_get_group(struct pinctrl_dev *pctldev,
+static const struct tegra_pingroup *tegra_pinctrl_get_group(struct pinctrl_dev *pctldev,
unsigned int offset)
{
struct tegra_pmx *pmx = pinctrl_dev_get_drvdata(pctldev);
continue;
for (j = 0; j < num_pins; j++) {
if (offset == pins[j])
- return (struct tegra_pingroup *)&pmx->soc->groups[group];
+ return &pmx->soc->groups[group];
}
}
.schmitt_bit = schmitt_b, \
.drvtype_bit = 13, \
.lpdr_bit = e_lpdr, \
- .drv_reg = -1, \
#define drive_touch_clk_pcc4 DRV_PINGROUP_ENTRY_Y(0x2004, 12, 5, 20, 5, -1, -1, -1, -1, 1)
#define drive_uart3_rx_pcc6 DRV_PINGROUP_ENTRY_Y(0x200c, 12, 5, 20, 5, -1, -1, -1, -1, 1)
#define ISHTP_SEND_TIMEOUT (3 * HZ)
/* ISH Transport CrOS EC ISH client unique GUID */
-static const guid_t cros_ish_guid =
- GUID_INIT(0x7b7154d0, 0x56f4, 0x4bdc,
- 0xb0, 0xd8, 0x9e, 0x7c, 0xda, 0xe0, 0xd6, 0xa0);
+static const struct ishtp_device_id cros_ec_ishtp_id_table[] = {
+ { .guid = GUID_INIT(0x7b7154d0, 0x56f4, 0x4bdc,
+ 0xb0, 0xd8, 0x9e, 0x7c, 0xda, 0xe0, 0xd6, 0xa0), },
+ { }
+};
+MODULE_DEVICE_TABLE(ishtp, cros_ec_ishtp_id_table);
struct header {
u8 channel;
ishtp_set_tx_ring_size(cros_ish_cl, CROS_ISH_CL_TX_RING_SIZE);
ishtp_set_rx_ring_size(cros_ish_cl, CROS_ISH_CL_RX_RING_SIZE);
- fw_client = ishtp_fw_cl_get_client(dev, &cros_ish_guid);
+ fw_client = ishtp_fw_cl_get_client(dev, &cros_ec_ishtp_id_table[0].guid);
if (!fw_client) {
dev_err(cl_data_to_dev(client_data),
"ish client uuid not found\n");
static struct ishtp_cl_driver cros_ec_ishtp_driver = {
.name = "cros_ec_ishtp",
- .guid = &cros_ish_guid,
+ .id = cros_ec_ishtp_id_table,
.probe = cros_ec_ishtp_probe,
.remove = cros_ec_ishtp_remove,
.reset = cros_ec_ishtp_reset,
MODULE_AUTHOR("Rushikesh S Kadam <rushikesh.s.kadam@intel.com>");
MODULE_LICENSE("GPL v2");
-MODULE_ALIAS("ishtp:*");
cros_ec_lpc_ops.write(EC_LPC_ADDR_HOST_CMD, 1, &sum);
if (ec_response_timed_out()) {
- dev_warn(ec->dev, "EC responsed timed out\n");
+ dev_warn(ec->dev, "EC response timed out\n");
ret = -EIO;
goto done;
}
cros_ec_lpc_ops.write(EC_LPC_ADDR_HOST_CMD, 1, &sum);
if (ec_response_timed_out()) {
- dev_warn(ec->dev, "EC responsed timed out\n");
+ dev_warn(ec->dev, "EC response timed out\n");
ret = -EIO;
goto done;
}
*
* Call this function to test whether the ChromeOS EC supports a feature.
*
- * Return: 1 if supported, 0 if not
+ * Return: true if supported, false if not (or if an error was encountered).
*/
-int cros_ec_check_features(struct cros_ec_dev *ec, int feature)
+bool cros_ec_check_features(struct cros_ec_dev *ec, int feature)
{
- struct cros_ec_command *msg;
+ struct ec_response_get_features *features = &ec->features;
int ret;
- if (ec->features[0] == -1U && ec->features[1] == -1U) {
+ if (features->flags[0] == -1U && features->flags[1] == -1U) {
/* features bitmap not read yet */
- msg = kzalloc(sizeof(*msg) + sizeof(ec->features), GFP_KERNEL);
- if (!msg)
- return -ENOMEM;
-
- msg->command = EC_CMD_GET_FEATURES + ec->cmd_offset;
- msg->insize = sizeof(ec->features);
-
- ret = cros_ec_cmd_xfer_status(ec->ec_dev, msg);
+ ret = cros_ec_command(ec->ec_dev, 0, EC_CMD_GET_FEATURES + ec->cmd_offset,
+ NULL, 0, features, sizeof(*features));
if (ret < 0) {
- dev_warn(ec->dev, "cannot get EC features: %d/%d\n",
- ret, msg->result);
- memset(ec->features, 0, sizeof(ec->features));
- } else {
- memcpy(ec->features, msg->data, sizeof(ec->features));
+ dev_warn(ec->dev, "cannot get EC features: %d\n", ret);
+ memset(features, 0, sizeof(*features));
}
dev_dbg(ec->dev, "EC features %08x %08x\n",
- ec->features[0], ec->features[1]);
-
- kfree(msg);
+ features->flags[0], features->flags[1]);
}
- return ec->features[feature / 32] & EC_FEATURE_MASK_0(feature);
+ return !!(features->flags[feature / 32] & EC_FEATURE_MASK_0(feature));
}
EXPORT_SYMBOL_GPL(cros_ec_check_features);
return sensor_count;
}
EXPORT_SYMBOL_GPL(cros_ec_get_sensor_count);
+
+/**
+ * cros_ec_command - Send a command to the EC.
+ *
+ * @ec_dev: EC device
+ * @version: EC command version
+ * @command: EC command
+ * @outdata: EC command output data
+ * @outsize: Size of outdata
+ * @indata: EC command input data
+ * @insize: Size of indata
+ *
+ * Return: >= 0 on success, negative error number on failure.
+ */
+int cros_ec_command(struct cros_ec_device *ec_dev,
+ unsigned int version,
+ int command,
+ void *outdata,
+ int outsize,
+ void *indata,
+ int insize)
+{
+ struct cros_ec_command *msg;
+ int ret;
+
+ msg = kzalloc(sizeof(*msg) + max(insize, outsize), GFP_KERNEL);
+ if (!msg)
+ return -ENOMEM;
+
+ msg->version = version;
+ msg->command = command;
+ msg->outsize = outsize;
+ msg->insize = insize;
+
+ if (outsize)
+ memcpy(msg->data, outdata, outsize);
+
+ ret = cros_ec_cmd_xfer_status(ec_dev, msg);
+ if (ret < 0)
+ goto error;
+
+ if (insize)
+ memcpy(indata, msg->data, insize);
+error:
+ kfree(msg);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(cros_ec_command);
*/
static int cros_ec_sensorhub_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct cros_ec_sensorhub *sensorhub = platform_get_drvdata(pdev);
+ struct cros_ec_sensorhub *sensorhub = dev_get_drvdata(dev);
struct cros_ec_dev *ec = sensorhub->ec;
if (cros_ec_check_features(ec, EC_FEATURE_MOTION_SENSE_FIFO))
static int cros_ec_sensorhub_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct cros_ec_sensorhub *sensorhub = platform_get_drvdata(pdev);
+ struct cros_ec_sensorhub *sensorhub = dev_get_drvdata(dev);
struct cros_ec_dev *ec = sensorhub->ec;
if (cros_ec_check_features(ec, EC_FEATURE_MOTION_SENSE_FIFO))
return ret;
}
-static int cros_typec_ec_command(struct cros_typec_data *typec,
- unsigned int version,
- unsigned int command,
- void *outdata,
- unsigned int outsize,
- void *indata,
- unsigned int insize)
-{
- struct cros_ec_command *msg;
- int ret;
-
- msg = kzalloc(sizeof(*msg) + max(outsize, insize), GFP_KERNEL);
- if (!msg)
- return -ENOMEM;
-
- msg->version = version;
- msg->command = command;
- msg->outsize = outsize;
- msg->insize = insize;
-
- if (outsize)
- memcpy(msg->data, outdata, outsize);
-
- ret = cros_ec_cmd_xfer_status(typec->ec, msg);
- if (ret >= 0 && insize)
- memcpy(indata, msg->data, insize);
-
- kfree(msg);
- return ret;
-}
-
static int cros_typec_usb_safe_state(struct cros_typec_port *port)
{
port->state.mode = TYPEC_STATE_SAFE;
/* Sending Acknowledgment to EC */
mux_ack.port = port_num;
- if (cros_typec_ec_command(typec, 0, EC_CMD_USB_PD_MUX_ACK, &mux_ack,
- sizeof(mux_ack), NULL, 0) < 0)
+ if (cros_ec_command(typec->ec, 0, EC_CMD_USB_PD_MUX_ACK, &mux_ack,
+ sizeof(mux_ack), NULL, 0) < 0)
dev_warn(typec->dev,
"Failed to send Mux ACK to EC for port: %d\n",
port_num);
.port = port_num,
};
- return cros_typec_ec_command(typec, 0, EC_CMD_USB_PD_MUX_INFO, &req,
- sizeof(req), resp, sizeof(*resp));
+ return cros_ec_command(typec->ec, 0, EC_CMD_USB_PD_MUX_INFO, &req,
+ sizeof(req), resp, sizeof(*resp));
}
/*
int ret = 0;
memset(disc, 0, EC_PROTO2_MAX_RESPONSE_SIZE);
- ret = cros_typec_ec_command(typec, 0, EC_CMD_TYPEC_DISCOVERY, &req, sizeof(req),
- disc, EC_PROTO2_MAX_RESPONSE_SIZE);
+ ret = cros_ec_command(typec->ec, 0, EC_CMD_TYPEC_DISCOVERY, &req, sizeof(req),
+ disc, EC_PROTO2_MAX_RESPONSE_SIZE);
if (ret < 0) {
dev_err(typec->dev, "Failed to get SOP' discovery data for port: %d\n", port_num);
goto sop_prime_disc_exit;
typec_partner_set_pd_revision(port->partner, pd_revision);
memset(sop_disc, 0, EC_PROTO2_MAX_RESPONSE_SIZE);
- ret = cros_typec_ec_command(typec, 0, EC_CMD_TYPEC_DISCOVERY, &req, sizeof(req),
- sop_disc, EC_PROTO2_MAX_RESPONSE_SIZE);
+ ret = cros_ec_command(typec->ec, 0, EC_CMD_TYPEC_DISCOVERY, &req, sizeof(req),
+ sop_disc, EC_PROTO2_MAX_RESPONSE_SIZE);
if (ret < 0) {
dev_err(typec->dev, "Failed to get SOP discovery data for port: %d\n", port_num);
goto disc_exit;
.clear_events_mask = events_mask,
};
- return cros_typec_ec_command(typec, 0, EC_CMD_TYPEC_CONTROL, &req,
- sizeof(req), NULL, 0);
+ return cros_ec_command(typec->ec, 0, EC_CMD_TYPEC_CONTROL, &req,
+ sizeof(req), NULL, 0);
}
static void cros_typec_handle_status(struct cros_typec_data *typec, int port_num)
};
int ret;
- ret = cros_typec_ec_command(typec, 0, EC_CMD_TYPEC_STATUS, &req, sizeof(req),
- &resp, sizeof(resp));
+ ret = cros_ec_command(typec->ec, 0, EC_CMD_TYPEC_STATUS, &req, sizeof(req),
+ &resp, sizeof(resp));
if (ret < 0) {
dev_warn(typec->dev, "EC_CMD_TYPEC_STATUS failed for port: %d\n", port_num);
return;
req.mux = USB_PD_CTRL_MUX_NO_CHANGE;
req.swap = USB_PD_CTRL_SWAP_NONE;
- ret = cros_typec_ec_command(typec, typec->pd_ctrl_ver,
- EC_CMD_USB_PD_CONTROL, &req, sizeof(req),
- &resp, sizeof(resp));
+ ret = cros_ec_command(typec->ec, typec->pd_ctrl_ver,
+ EC_CMD_USB_PD_CONTROL, &req, sizeof(req),
+ &resp, sizeof(resp));
if (ret < 0)
return ret;
/* We're interested in the PD control command version. */
req_v1.cmd = EC_CMD_USB_PD_CONTROL;
- ret = cros_typec_ec_command(typec, 1, EC_CMD_GET_CMD_VERSIONS,
- &req_v1, sizeof(req_v1), &resp,
+ ret = cros_ec_command(typec->ec, 1, EC_CMD_GET_CMD_VERSIONS,
+ &req_v1, sizeof(req_v1), &resp,
sizeof(resp));
if (ret < 0)
return ret;
}
ec_dev = dev_get_drvdata(&typec->ec->ec->dev);
- typec->typec_cmd_supported = !!cros_ec_check_features(ec_dev, EC_FEATURE_TYPEC_CMD);
- typec->needs_mux_ack = !!cros_ec_check_features(ec_dev,
- EC_FEATURE_TYPEC_MUX_REQUIRE_AP_ACK);
+ typec->typec_cmd_supported = cros_ec_check_features(ec_dev, EC_FEATURE_TYPEC_CMD);
+ typec->needs_mux_ack = cros_ec_check_features(ec_dev, EC_FEATURE_TYPEC_MUX_REQUIRE_AP_ACK);
- ret = cros_typec_ec_command(typec, 0, EC_CMD_USB_PD_PORTS, NULL, 0,
- &resp, sizeof(resp));
+ ret = cros_ec_command(typec->ec, 0, EC_CMD_USB_PD_PORTS, NULL, 0,
+ &resp, sizeof(resp));
if (ret < 0)
return ret;
}
EXPORT_SYMBOL_GPL(cros_usbpd_unregister_notify);
-/**
- * cros_ec_pd_command - Send a command to the EC.
- *
- * @ec_dev: EC device
- * @command: EC command
- * @outdata: EC command output data
- * @outsize: Size of outdata
- * @indata: EC command input data
- * @insize: Size of indata
- *
- * Return: >= 0 on success, negative error number on failure.
- */
-static int cros_ec_pd_command(struct cros_ec_device *ec_dev,
- int command,
- uint8_t *outdata,
- int outsize,
- uint8_t *indata,
- int insize)
-{
- struct cros_ec_command *msg;
- int ret;
-
- msg = kzalloc(sizeof(*msg) + max(insize, outsize), GFP_KERNEL);
- if (!msg)
- return -ENOMEM;
-
- msg->command = command;
- msg->outsize = outsize;
- msg->insize = insize;
-
- if (outsize)
- memcpy(msg->data, outdata, outsize);
-
- ret = cros_ec_cmd_xfer_status(ec_dev, msg);
- if (ret < 0)
- goto error;
-
- if (insize)
- memcpy(indata, msg->data, insize);
-error:
- kfree(msg);
- return ret;
-}
-
static void cros_usbpd_get_event_and_notify(struct device *dev,
struct cros_ec_device *ec_dev)
{
}
/* Check for PD host events on EC. */
- ret = cros_ec_pd_command(ec_dev, EC_CMD_PD_HOST_EVENT_STATUS,
- NULL, 0,
- (uint8_t *)&host_event_status,
- sizeof(host_event_status));
+ ret = cros_ec_command(ec_dev, 0, EC_CMD_PD_HOST_EVENT_STATUS,
+ NULL, 0, &host_event_status, sizeof(host_event_status));
if (ret < 0) {
dev_warn(dev, "Can't get host event status (err: %d)\n", ret);
goto send_notify;
int size)
{
struct mlxreg_hotplug_device *dev = devs;
- int i;
+ int i, ret;
/* Create static I2C device feeding by auxiliary or main power. */
for (i = 0; i < size; i++, dev++) {
dev->brdinfo->type, dev->nr, dev->brdinfo->addr);
dev->adapter = NULL;
+ ret = PTR_ERR(dev->client);
goto fail_create_static_devices;
}
}
i2c_unregister_device(dev->client);
dev->client = NULL;
}
- return IS_ERR(dev->client);
+ return ret;
}
static void
config AMD_PMC
tristate "AMD SoC PMC driver"
- depends on ACPI && PCI
+ depends on ACPI && PCI && RTC_CLASS
help
The driver provides support for AMD Power Management Controller
primarily responsible for S2Idle transactions that are driven from
config DELL_WMI_DESCRIPTOR
tristate
- default m
+ default n
depends on ACPI_WMI
config DELL_WMI_LED
INIT_WORK(&hpled_led.work, delayed_set_status_worker);
ret = led_classdev_register(NULL, &hpled_led.led_classdev);
if (ret) {
+ i8042_remove_filter(hp_accel_i8042_filter);
lis3lv02d_joystick_disable(&lis3_dev);
lis3lv02d_poweroff(&lis3_dev);
flush_work(&hpled_led.work);
+ lis3lv02d_remove_fs(&lis3_dev);
return ret;
}
};
/* eclite ishtp client UUID: 6a19cc4b-d760-4de3-b14d-f25ebd0fbcd9 */
-static const guid_t ecl_ishtp_guid =
- GUID_INIT(0x6a19cc4b, 0xd760, 0x4de3,
- 0xb1, 0x4d, 0xf2, 0x5e, 0xbd, 0xf, 0xbc, 0xd9);
+static const struct ishtp_device_id ecl_ishtp_id_table[] = {
+ { .guid = GUID_INIT(0x6a19cc4b, 0xd760, 0x4de3,
+ 0xb1, 0x4d, 0xf2, 0x5e, 0xbd, 0xf, 0xbc, 0xd9), },
+ { }
+};
+MODULE_DEVICE_TABLE(ishtp, ecl_ishtp_id_table);
/* ACPI DSM UUID: 91d936a7-1f01-49c6-a6b4-72f00ad8d8a5 */
static const guid_t ecl_acpi_guid =
ishtp_set_tx_ring_size(ecl_ishtp_cl, ECL_CL_TX_RING_SIZE);
ishtp_set_rx_ring_size(ecl_ishtp_cl, ECL_CL_RX_RING_SIZE);
- fw_client = ishtp_fw_cl_get_client(dev, &ecl_ishtp_guid);
+ fw_client = ishtp_fw_cl_get_client(dev, &ecl_ishtp_id_table[0].guid);
if (!fw_client) {
dev_err(cl_data_to_dev(opr_dev), "fw client not found\n");
return -ENOENT;
static struct ishtp_cl_driver ecl_ishtp_cl_driver = {
.name = "ishtp-eclite",
- .guid = &ecl_ishtp_guid,
+ .id = ecl_ishtp_id_table,
.probe = ecl_ishtp_cl_probe,
.remove = ecl_ishtp_cl_remove,
.reset = ecl_ishtp_cl_reset,
MODULE_AUTHOR("K Naduvalath, Sumesh <sumesh.k.naduvalath@intel.com>");
MODULE_LICENSE("GPL v2");
-MODULE_ALIAS("ishtp:*");
static const struct sabi_config sabi_configs[] = {
{
- /* I don't know if it is really 2, but it it is
+ /* I don't know if it is really 2, but it is
* less than 3 anyway */
.sabi_version = 2,
break;
if (!item)
break;
- if (!*item)
+ if (!*item) {
+ kfree(item);
continue;
+ }
/* It is not allowed to have '/' for file name. Convert it into '\'. */
strreplace(item, '/', '\\');
setting = kzalloc(sizeof(*setting), GFP_KERNEL);
if (!setting) {
ret = -ENOMEM;
+ kfree(item);
goto fail_clear_attr;
}
setting->index = i;
}
kobject_init(&setting->kobj, &tlmi_attr_setting_ktype);
tlmi_priv.setting[i] = setting;
- tlmi_priv.settings_count++;
kfree(item);
}
static int tlmi_probe(struct wmi_device *wdev, const void *context)
{
- tlmi_analyze();
+ int ret;
+
+ ret = tlmi_analyze();
+ if (ret)
+ return ret;
+
return tlmi_sysfs_init();
}
struct think_lmi {
struct wmi_device *wmi_device;
- int settings_count;
bool can_set_bios_settings;
bool can_get_bios_selections;
bool can_set_bios_password;
return status;
}
-/* Query FW and update rfkill sw state for all rfkill switches */
-static void tpacpi_rfk_update_swstate_all(void)
-{
- unsigned int i;
-
- for (i = 0; i < TPACPI_RFK_SW_MAX; i++)
- tpacpi_rfk_update_swstate(tpacpi_rfkill_switches[i]);
-}
-
/*
* Sync the HW-blocking state of all rfkill switches,
* do notice it causes the rfkill core to schedule uevents
if (wlsw == TPACPI_RFK_RADIO_OFF)
tpacpi_rfk_update_hwblock_state(true);
- /* Sync sw blocking state */
- tpacpi_rfk_update_swstate_all();
-
/* Sync hw blocking state last if it is hw-unblocked */
if (wlsw == TPACPI_RFK_RADIO_ON)
tpacpi_rfk_update_hwblock_state(false);
TPACPI_Q_LNV3('N', '2', 'E', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (1st gen) */
TPACPI_Q_LNV3('N', '2', 'O', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (2nd gen) */
TPACPI_Q_LNV3('N', '2', 'V', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (3nd gen) */
+ TPACPI_Q_LNV3('N', '4', '0', TPACPI_FAN_2CTL), /* P1 / X1 Extreme (4nd gen) */
TPACPI_Q_LNV3('N', '3', '0', TPACPI_FAN_2CTL), /* P15 (1st gen) / P15v (1st gen) */
TPACPI_Q_LNV3('N', '3', '2', TPACPI_FAN_2CTL), /* X1 Carbon (9th gen) */
};
static int cpuhp_dtpm_cpu_offline(unsigned int cpu)
{
- struct em_perf_domain *pd;
struct dtpm_cpu *dtpm_cpu;
- pd = em_cpu_get(cpu);
- if (!pd)
- return -EINVAL;
-
dtpm_cpu = per_cpu(dtpm_per_cpu, cpu);
+ if (dtpm_cpu)
+ dtpm_update_power(&dtpm_cpu->dtpm);
- return dtpm_update_power(&dtpm_cpu->dtpm);
+ return 0;
}
static int cpuhp_dtpm_cpu_online(unsigned int cpu)
/* PTP Hardware Clock interface */
-/**
+/*
* Maximum absolute value for write phase offset in picoseconds
*
- * @channel: channel
- * @delta_ns: delta in nanoseconds
- *
* Destination signed register is 32-bit register in resolution of 50ps
*
* 0x7fffffff * 50 = 2147483647 * 50 = 107374182350
if (!ext)
return -ENOMEM;
- err = -EINVAL;
ext->mem = ptp_ocp_get_mem(bp, r);
- if (!ext->mem)
+ if (IS_ERR(ext->mem)) {
+ err = PTR_ERR(ext->mem);
goto out;
+ }
ext->bp = bp;
ext->info = r->extra;
void __iomem *mem;
mem = ptp_ocp_get_mem(bp, r);
- if (!mem)
- return -EINVAL;
+ if (IS_ERR(mem))
+ return PTR_ERR(mem);
bp_assign_entry(bp, r, mem);
depends on PLAT_SAMSUNG || ARCH_S5PV210 || ARCH_EXYNOS || COMPILE_TEST
depends on HAS_IOMEM
help
- Generic PWM framework driver for Samsung.
+ Generic PWM framework driver for Samsung S3C24xx, S3C64xx, S5Pv210
+ and Exynos SoCs.
+ Choose Y here only if you build for such Samsung SoC.
To compile this driver as a module, choose M here: the module
will be called pwm-samsung.
struct pwm_chip *chip;
int err;
+ /*
+ * Some lowlevel driver's implementations of .apply() make use of
+ * mutexes, also with some drivers only returning when the new
+ * configuration is active calling pwm_apply_state() from atomic context
+ * is a bad idea. So make it explicit that calling this function might
+ * sleep.
+ */
+ might_sleep();
+
if (!pwm || !state || !state->period ||
state->duty_cycle > state->period)
return -EINVAL;
#include <linux/err.h>
#include <linux/io.h>
#include <linux/module.h>
-#include <linux/mutex.h>
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/platform_device.h>
return (channel == 0) ? 0 : (channel + 1);
}
+static void __pwm_samsung_manual_update(struct samsung_pwm_chip *chip,
+ struct pwm_device *pwm)
+{
+ unsigned int tcon_chan = to_tcon_channel(pwm->hwpwm);
+ u32 tcon;
+
+ tcon = readl(chip->base + REG_TCON);
+ tcon |= TCON_MANUALUPDATE(tcon_chan);
+ writel(tcon, chip->base + REG_TCON);
+
+ tcon &= ~TCON_MANUALUPDATE(tcon_chan);
+ writel(tcon, chip->base + REG_TCON);
+}
+
static void pwm_samsung_set_divisor(struct samsung_pwm_chip *pwm,
unsigned int channel, u8 divisor)
{
tcon &= ~TCON_AUTORELOAD(tcon_chan);
writel(tcon, our_chip->base + REG_TCON);
+ /*
+ * In case the PWM is at 100% duty cycle, force a manual
+ * update to prevent the signal from staying high.
+ */
+ if (readl(our_chip->base + REG_TCMPB(pwm->hwpwm)) == (u32)-1U)
+ __pwm_samsung_manual_update(our_chip, pwm);
+
our_chip->disabled_mask |= BIT(pwm->hwpwm);
spin_unlock_irqrestore(&samsung_pwm_lock, flags);
static void pwm_samsung_manual_update(struct samsung_pwm_chip *chip,
struct pwm_device *pwm)
{
- unsigned int tcon_chan = to_tcon_channel(pwm->hwpwm);
- u32 tcon;
unsigned long flags;
spin_lock_irqsave(&samsung_pwm_lock, flags);
- tcon = readl(chip->base + REG_TCON);
- tcon |= TCON_MANUALUPDATE(tcon_chan);
- writel(tcon, chip->base + REG_TCON);
-
- tcon &= ~TCON_MANUALUPDATE(tcon_chan);
- writel(tcon, chip->base + REG_TCON);
+ __pwm_samsung_manual_update(chip, pwm);
spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
if (IS_ERR(priv->base))
return PTR_ERR(priv->base);
- platform_set_drvdata(pdev, priv);
-
priv->chip.dev = dev;
priv->chip.ops = &visconti_pwm_ops;
priv->chip.npwm = 4;
- ret = pwmchip_add(&priv->chip);
+ ret = devm_pwmchip_add(&pdev->dev, &priv->chip);
if (ret < 0)
return dev_err_probe(&pdev->dev, ret, "Cannot register visconti PWM\n");
return 0;
}
-static int visconti_pwm_remove(struct platform_device *pdev)
-{
- struct visconti_pwm_chip *priv = platform_get_drvdata(pdev);
-
- pwmchip_remove(&priv->chip);
-
- return 0;
-}
-
static const struct of_device_id visconti_pwm_of_match[] = {
{ .compatible = "toshiba,visconti-pwm", },
{ }
.of_match_table = visconti_pwm_of_match,
},
.probe = visconti_pwm_probe,
- .remove = visconti_pwm_remove,
};
module_platform_driver(visconti_pwm_driver);
#define to_vt8500_chip(chip) container_of(chip, struct vt8500_chip, chip)
#define msecs_to_loops(t) (loops_per_jiffy / 1000 * HZ * t)
-static inline void pwm_busy_wait(struct vt8500_chip *vt8500, int nr, u8 bitmask)
+static inline void vt8500_pwm_busy_wait(struct vt8500_chip *vt8500, int nr, u8 bitmask)
{
int loops = msecs_to_loops(10);
u32 mask = bitmask << (nr << 8);
dc = c;
writel(prescale, vt8500->base + REG_SCALAR(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_SCALAR_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_SCALAR_UPDATE);
writel(pv, vt8500->base + REG_PERIOD(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_PERIOD_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_PERIOD_UPDATE);
writel(dc, vt8500->base + REG_DUTY(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_DUTY_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_DUTY_UPDATE);
val = readl(vt8500->base + REG_CTRL(pwm->hwpwm));
val |= CTRL_AUTOLOAD;
writel(val, vt8500->base + REG_CTRL(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
clk_disable(vt8500->clk);
return 0;
val = readl(vt8500->base + REG_CTRL(pwm->hwpwm));
val |= CTRL_ENABLE;
writel(val, vt8500->base + REG_CTRL(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
return 0;
}
val = readl(vt8500->base + REG_CTRL(pwm->hwpwm));
val &= ~CTRL_ENABLE;
writel(val, vt8500->base + REG_CTRL(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
clk_disable(vt8500->clk);
}
val &= ~CTRL_INVERT;
writel(val, vt8500->base + REG_CTRL(pwm->hwpwm));
- pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
+ vt8500_pwm_busy_wait(vt8500, pwm->hwpwm, STATUS_CTRL_UPDATE);
return 0;
}
struct rio_transfer_io *transfer;
enum dma_data_direction dir;
int i, ret = 0;
+ size_t size;
if (unlikely(copy_from_user(&transaction, arg, sizeof(transaction))))
return -EFAULT;
priv->md->properties.transfer_mode) == 0)
return -ENODEV;
- transfer = vmalloc(array_size(sizeof(*transfer), transaction.count));
+ size = array_size(sizeof(*transfer), transaction.count);
+ transfer = vmalloc(size);
if (!transfer)
return -ENOMEM;
if (unlikely(copy_from_user(transfer,
(void __user *)(uintptr_t)transaction.block,
- array_size(sizeof(*transfer), transaction.count)))) {
+ size))) {
ret = -EFAULT;
goto out_free;
}
transaction.sync, dir, &transfer[i]);
if (unlikely(copy_to_user((void __user *)(uintptr_t)transaction.block,
- transfer,
- array_size(sizeof(*transfer), transaction.count))))
+ transfer, size)))
ret = -EFAULT;
out_free:
// Guodong Xu <guodong.xu@linaro.org>
#include <linux/delay.h>
-#include <linux/mfd/hi6421-spmi-pmic.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
#include <linux/regulator/driver.h>
struct hi6421_spmi_reg_priv *priv;
struct hi6421_spmi_reg_info *info;
struct device *dev = &pdev->dev;
- struct hi6421_spmi_pmic *pmic;
+ struct regmap *regmap;
struct regulator_dev *rdev;
int i;
* which should first set drvdata. If this doesn't happen, hit
* a warn on and return.
*/
- pmic = dev_get_drvdata(pmic_dev);
- if (WARN_ON(!pmic))
+ regmap = dev_get_drvdata(pmic_dev);
+ if (WARN_ON(!regmap))
return -ENODEV;
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
config.dev = pdev->dev.parent;
config.driver_data = priv;
- config.regmap = pmic->regmap;
+ config.regmap = regmap;
rdev = devm_regulator_register(dev, &info->desc, &config);
if (IS_ERR(rdev)) {
It's safe to say N here.
+config IMX_DSP_REMOTEPROC
+ tristate "i.MX DSP remoteproc support"
+ depends on ARCH_MXC
+ depends on HAVE_ARM_SMCCC
+ select MAILBOX
+ help
+ Say y here to support iMX's DSP remote processors via the remote
+ processor framework.
+
+ It's safe to say N here.
+
config INGENIC_VPU_RPROC
tristate "Ingenic JZ47xx VPU remoteproc support"
depends on MIPS || COMPILE_TEST
It's safe to say N here if you're not interested in the Keystone
DSPs or just want to use a bare minimum kernel.
+config MESON_MX_AO_ARC_REMOTEPROC
+ tristate "Amlogic Meson6/8/8b/8m2 AO ARC remote processor support"
+ depends on HAS_IOMEM
+ depends on (ARM && ARCH_MESON) || COMPILE_TEST
+ select GENERIC_ALLOCATOR
+ help
+ Say m or y here to have support for the AO ARC remote processor
+ on Amlogic Meson6/Meson8/Meson8b/Meson8m2 SoCs. This is
+ typically used for system suspend.
+ If unsure say N.
+
config PRU_REMOTEPROC
tristate "TI PRU remoteproc support"
depends on TI_PRUSS
tristate "Qualcomm Technology Inc ADSP Peripheral Image Loader"
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
- depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
depends on RPMSG_QCOM_GLINK_SMEM || RPMSG_QCOM_GLINK_SMEM=n
depends on QCOM_SYSMON || QCOM_SYSMON=n
depends on RPMSG_QCOM_GLINK || RPMSG_QCOM_GLINK=n
tristate "Qualcomm Hexagon V5 self-authenticating modem subsystem support"
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
- depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
depends on RPMSG_QCOM_GLINK_SMEM || RPMSG_QCOM_GLINK_SMEM=n
depends on QCOM_SYSMON || QCOM_SYSMON=n
depends on RPMSG_QCOM_GLINK || RPMSG_QCOM_GLINK=n
tristate "Qualcomm Hexagon v5 Peripheral Authentication Service support"
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
- depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
depends on RPMSG_QCOM_GLINK_SMEM || RPMSG_QCOM_GLINK_SMEM=n
depends on QCOM_SYSMON || QCOM_SYSMON=n
depends on RPMSG_QCOM_GLINK || RPMSG_QCOM_GLINK=n
tristate "Qualcomm Hexagon based WCSS Peripheral Image Loader"
depends on OF && ARCH_QCOM
depends on QCOM_SMEM
- depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
depends on RPMSG_QCOM_GLINK_SMEM || RPMSG_QCOM_GLINK_SMEM=n
depends on QCOM_SYSMON || QCOM_SYSMON=n
depends on RPMSG_QCOM_GLINK || RPMSG_QCOM_GLINK=n
config QCOM_WCNSS_PIL
tristate "Qualcomm WCNSS Peripheral Image Loader"
depends on OF && ARCH_QCOM
- depends on RPMSG_QCOM_SMD || (COMPILE_TEST && RPMSG_QCOM_SMD=n)
+ depends on RPMSG_QCOM_SMD || RPMSG_QCOM_SMD=n
depends on RPMSG_QCOM_GLINK_SMEM || RPMSG_QCOM_GLINK_SMEM=n
depends on QCOM_SMEM
depends on QCOM_SYSMON || QCOM_SYSMON=n
remoteproc-y += remoteproc_elf_loader.o
obj-$(CONFIG_REMOTEPROC_CDEV) += remoteproc_cdev.o
obj-$(CONFIG_IMX_REMOTEPROC) += imx_rproc.o
+obj-$(CONFIG_IMX_DSP_REMOTEPROC) += imx_dsp_rproc.o
obj-$(CONFIG_INGENIC_VPU_RPROC) += ingenic_rproc.o
obj-$(CONFIG_MTK_SCP) += mtk_scp.o mtk_scp_ipi.o
obj-$(CONFIG_OMAP_REMOTEPROC) += omap_remoteproc.o
obj-$(CONFIG_WKUP_M3_RPROC) += wkup_m3_rproc.o
obj-$(CONFIG_DA8XX_REMOTEPROC) += da8xx_remoteproc.o
obj-$(CONFIG_KEYSTONE_REMOTEPROC) += keystone_remoteproc.o
+obj-$(CONFIG_MESON_MX_AO_ARC_REMOTEPROC)+= meson_mx_ao_arc.o
obj-$(CONFIG_PRU_REMOTEPROC) += pru_rproc.o
obj-$(CONFIG_QCOM_PIL_INFO) += qcom_pil_info.o
obj-$(CONFIG_QCOM_RPROC_COMMON) += qcom_common.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright 2021 NXP */
+
+#include <dt-bindings/firmware/imx/rsrc.h>
+#include <linux/arm-smccc.h>
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/firmware.h>
+#include <linux/firmware/imx/sci.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/mailbox_client.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/of_address.h>
+#include <linux/of_device.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/platform_device.h>
+#include <linux/pm_domain.h>
+#include <linux/pm_runtime.h>
+#include <linux/regmap.h>
+#include <linux/remoteproc.h>
+#include <linux/slab.h>
+
+#include "imx_rproc.h"
+#include "remoteproc_elf_helpers.h"
+#include "remoteproc_internal.h"
+
+#define DSP_RPROC_CLK_MAX 5
+
+#define REMOTE_IS_READY BIT(0)
+#define REMOTE_READY_WAIT_MAX_RETRIES 500
+
+/* att flags */
+/* DSP own area */
+#define ATT_OWN BIT(31)
+/* DSP instruction area */
+#define ATT_IRAM BIT(30)
+
+/* Definitions for i.MX8MP */
+/* DAP registers */
+#define IMX8M_DAP_DEBUG 0x28800000
+#define IMX8M_DAP_DEBUG_SIZE (64 * 1024)
+#define IMX8M_DAP_PWRCTL (0x4000 + 0x3020)
+#define IMX8M_PWRCTL_CORERESET BIT(16)
+
+/* DSP audio mix registers */
+#define IMX8M_AudioDSP_REG0 0x100
+#define IMX8M_AudioDSP_REG1 0x104
+#define IMX8M_AudioDSP_REG2 0x108
+#define IMX8M_AudioDSP_REG3 0x10c
+
+#define IMX8M_AudioDSP_REG2_RUNSTALL BIT(5)
+#define IMX8M_AudioDSP_REG2_PWAITMODE BIT(1)
+
+/* Definitions for i.MX8ULP */
+#define IMX8ULP_SIM_LPAV_REG_SYSCTRL0 0x8
+#define IMX8ULP_SYSCTRL0_DSP_DBG_RST BIT(25)
+#define IMX8ULP_SYSCTRL0_DSP_PLAT_CLK_EN BIT(19)
+#define IMX8ULP_SYSCTRL0_DSP_PBCLK_EN BIT(18)
+#define IMX8ULP_SYSCTRL0_DSP_CLK_EN BIT(17)
+#define IMX8ULP_SYSCTRL0_DSP_RST BIT(16)
+#define IMX8ULP_SYSCTRL0_DSP_OCD_HALT BIT(14)
+#define IMX8ULP_SYSCTRL0_DSP_STALL BIT(13)
+
+#define IMX8ULP_SIP_HIFI_XRDC 0xc200000e
+
+/*
+ * enum - Predefined Mailbox Messages
+ *
+ * @RP_MBOX_SUSPEND_SYSTEM: system suspend request for the remote processor
+ *
+ * @RP_MBOX_SUSPEND_ACK: successful response from remote processor for a
+ * suspend request
+ *
+ * @RP_MBOX_RESUME_SYSTEM: system resume request for the remote processor
+ *
+ * @RP_MBOX_RESUME_ACK: successful response from remote processor for a
+ * resume request
+ */
+enum imx_dsp_rp_mbox_messages {
+ RP_MBOX_SUSPEND_SYSTEM = 0xFF11,
+ RP_MBOX_SUSPEND_ACK = 0xFF12,
+ RP_MBOX_RESUME_SYSTEM = 0xFF13,
+ RP_MBOX_RESUME_ACK = 0xFF14,
+};
+
+/**
+ * struct imx_dsp_rproc - DSP remote processor state
+ * @regmap: regmap handler
+ * @rproc: rproc handler
+ * @dsp_dcfg: device configuration pointer
+ * @clks: clocks needed by this device
+ * @cl: mailbox client to request the mailbox channel
+ * @cl_rxdb: mailbox client to request the mailbox channel for doorbell
+ * @tx_ch: mailbox tx channel handle
+ * @rx_ch: mailbox rx channel handle
+ * @rxdb_ch: mailbox rx doorbell channel handle
+ * @pd_dev: power domain device
+ * @pd_dev_link: power domain device link
+ * @ipc_handle: System Control Unit ipc handle
+ * @rproc_work: work for processing virtio interrupts
+ * @pm_comp: completion primitive to sync for suspend response
+ * @num_domains: power domain number
+ * @flags: control flags
+ */
+struct imx_dsp_rproc {
+ struct regmap *regmap;
+ struct rproc *rproc;
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg;
+ struct clk_bulk_data clks[DSP_RPROC_CLK_MAX];
+ struct mbox_client cl;
+ struct mbox_client cl_rxdb;
+ struct mbox_chan *tx_ch;
+ struct mbox_chan *rx_ch;
+ struct mbox_chan *rxdb_ch;
+ struct device **pd_dev;
+ struct device_link **pd_dev_link;
+ struct imx_sc_ipc *ipc_handle;
+ struct work_struct rproc_work;
+ struct completion pm_comp;
+ int num_domains;
+ u32 flags;
+};
+
+/**
+ * struct imx_dsp_rproc_dcfg - DSP remote processor configuration
+ * @dcfg: imx_rproc_dcfg handler
+ * @reset: reset callback function
+ */
+struct imx_dsp_rproc_dcfg {
+ const struct imx_rproc_dcfg *dcfg;
+ int (*reset)(struct imx_dsp_rproc *priv);
+};
+
+static const struct imx_rproc_att imx_dsp_rproc_att_imx8qm[] = {
+ /* dev addr , sys addr , size , flags */
+ { 0x596e8000, 0x556e8000, 0x00008000, ATT_OWN },
+ { 0x596f0000, 0x556f0000, 0x00008000, ATT_OWN },
+ { 0x596f8000, 0x556f8000, 0x00000800, ATT_OWN | ATT_IRAM},
+ { 0x55700000, 0x55700000, 0x00070000, ATT_OWN },
+ /* DDR (Data) */
+ { 0x80000000, 0x80000000, 0x60000000, 0},
+};
+
+static const struct imx_rproc_att imx_dsp_rproc_att_imx8qxp[] = {
+ /* dev addr , sys addr , size , flags */
+ { 0x596e8000, 0x596e8000, 0x00008000, ATT_OWN },
+ { 0x596f0000, 0x596f0000, 0x00008000, ATT_OWN },
+ { 0x596f8000, 0x596f8000, 0x00000800, ATT_OWN | ATT_IRAM},
+ { 0x59700000, 0x59700000, 0x00070000, ATT_OWN },
+ /* DDR (Data) */
+ { 0x80000000, 0x80000000, 0x60000000, 0},
+};
+
+static const struct imx_rproc_att imx_dsp_rproc_att_imx8mp[] = {
+ /* dev addr , sys addr , size , flags */
+ { 0x3b6e8000, 0x3b6e8000, 0x00008000, ATT_OWN },
+ { 0x3b6f0000, 0x3b6f0000, 0x00008000, ATT_OWN },
+ { 0x3b6f8000, 0x3b6f8000, 0x00000800, ATT_OWN | ATT_IRAM},
+ { 0x3b700000, 0x3b700000, 0x00040000, ATT_OWN },
+ /* DDR (Data) */
+ { 0x40000000, 0x40000000, 0x80000000, 0},
+};
+
+static const struct imx_rproc_att imx_dsp_rproc_att_imx8ulp[] = {
+ /* dev addr , sys addr , size , flags */
+ { 0x21170000, 0x21170000, 0x00010000, ATT_OWN | ATT_IRAM},
+ { 0x21180000, 0x21180000, 0x00010000, ATT_OWN },
+ /* DDR (Data) */
+ { 0x0c000000, 0x80000000, 0x10000000, 0},
+ { 0x30000000, 0x90000000, 0x10000000, 0},
+};
+
+/* Reset function for DSP on i.MX8MP */
+static int imx8mp_dsp_reset(struct imx_dsp_rproc *priv)
+{
+ void __iomem *dap = ioremap_wc(IMX8M_DAP_DEBUG, IMX8M_DAP_DEBUG_SIZE);
+ int pwrctl;
+
+ /* Put DSP into reset and stall */
+ pwrctl = readl(dap + IMX8M_DAP_PWRCTL);
+ pwrctl |= IMX8M_PWRCTL_CORERESET;
+ writel(pwrctl, dap + IMX8M_DAP_PWRCTL);
+
+ /* Keep reset asserted for 10 cycles */
+ usleep_range(1, 2);
+
+ regmap_update_bits(priv->regmap, IMX8M_AudioDSP_REG2,
+ IMX8M_AudioDSP_REG2_RUNSTALL,
+ IMX8M_AudioDSP_REG2_RUNSTALL);
+
+ /* Take the DSP out of reset and keep stalled for FW loading */
+ pwrctl = readl(dap + IMX8M_DAP_PWRCTL);
+ pwrctl &= ~IMX8M_PWRCTL_CORERESET;
+ writel(pwrctl, dap + IMX8M_DAP_PWRCTL);
+
+ iounmap(dap);
+ return 0;
+}
+
+/* Reset function for DSP on i.MX8ULP */
+static int imx8ulp_dsp_reset(struct imx_dsp_rproc *priv)
+{
+ struct arm_smccc_res res;
+
+ /* Put DSP into reset and stall */
+ regmap_update_bits(priv->regmap, IMX8ULP_SIM_LPAV_REG_SYSCTRL0,
+ IMX8ULP_SYSCTRL0_DSP_RST, IMX8ULP_SYSCTRL0_DSP_RST);
+ regmap_update_bits(priv->regmap, IMX8ULP_SIM_LPAV_REG_SYSCTRL0,
+ IMX8ULP_SYSCTRL0_DSP_STALL,
+ IMX8ULP_SYSCTRL0_DSP_STALL);
+
+ /* Configure resources of DSP through TFA */
+ arm_smccc_smc(IMX8ULP_SIP_HIFI_XRDC, 0, 0, 0, 0, 0, 0, 0, &res);
+
+ /* Take the DSP out of reset and keep stalled for FW loading */
+ regmap_update_bits(priv->regmap, IMX8ULP_SIM_LPAV_REG_SYSCTRL0,
+ IMX8ULP_SYSCTRL0_DSP_RST, 0);
+ regmap_update_bits(priv->regmap, IMX8ULP_SIM_LPAV_REG_SYSCTRL0,
+ IMX8ULP_SYSCTRL0_DSP_DBG_RST, 0);
+
+ return 0;
+}
+
+/* Specific configuration for i.MX8MP */
+static const struct imx_rproc_dcfg dsp_rproc_cfg_imx8mp = {
+ .src_reg = IMX8M_AudioDSP_REG2,
+ .src_mask = IMX8M_AudioDSP_REG2_RUNSTALL,
+ .src_start = 0,
+ .src_stop = IMX8M_AudioDSP_REG2_RUNSTALL,
+ .att = imx_dsp_rproc_att_imx8mp,
+ .att_size = ARRAY_SIZE(imx_dsp_rproc_att_imx8mp),
+ .method = IMX_RPROC_MMIO,
+};
+
+static const struct imx_dsp_rproc_dcfg imx_dsp_rproc_cfg_imx8mp = {
+ .dcfg = &dsp_rproc_cfg_imx8mp,
+ .reset = imx8mp_dsp_reset,
+};
+
+/* Specific configuration for i.MX8ULP */
+static const struct imx_rproc_dcfg dsp_rproc_cfg_imx8ulp = {
+ .src_reg = IMX8ULP_SIM_LPAV_REG_SYSCTRL0,
+ .src_mask = IMX8ULP_SYSCTRL0_DSP_STALL,
+ .src_start = 0,
+ .src_stop = IMX8ULP_SYSCTRL0_DSP_STALL,
+ .att = imx_dsp_rproc_att_imx8ulp,
+ .att_size = ARRAY_SIZE(imx_dsp_rproc_att_imx8ulp),
+ .method = IMX_RPROC_MMIO,
+};
+
+static const struct imx_dsp_rproc_dcfg imx_dsp_rproc_cfg_imx8ulp = {
+ .dcfg = &dsp_rproc_cfg_imx8ulp,
+ .reset = imx8ulp_dsp_reset,
+};
+
+/* Specific configuration for i.MX8QXP */
+static const struct imx_rproc_dcfg dsp_rproc_cfg_imx8qxp = {
+ .att = imx_dsp_rproc_att_imx8qxp,
+ .att_size = ARRAY_SIZE(imx_dsp_rproc_att_imx8qxp),
+ .method = IMX_RPROC_SCU_API,
+};
+
+static const struct imx_dsp_rproc_dcfg imx_dsp_rproc_cfg_imx8qxp = {
+ .dcfg = &dsp_rproc_cfg_imx8qxp,
+};
+
+/* Specific configuration for i.MX8QM */
+static const struct imx_rproc_dcfg dsp_rproc_cfg_imx8qm = {
+ .att = imx_dsp_rproc_att_imx8qm,
+ .att_size = ARRAY_SIZE(imx_dsp_rproc_att_imx8qm),
+ .method = IMX_RPROC_SCU_API,
+};
+
+static const struct imx_dsp_rproc_dcfg imx_dsp_rproc_cfg_imx8qm = {
+ .dcfg = &dsp_rproc_cfg_imx8qm,
+};
+
+static int imx_dsp_rproc_ready(struct rproc *rproc)
+{
+ struct imx_dsp_rproc *priv = rproc->priv;
+ int i;
+
+ if (!priv->rxdb_ch)
+ return 0;
+
+ for (i = 0; i < REMOTE_READY_WAIT_MAX_RETRIES; i++) {
+ if (priv->flags & REMOTE_IS_READY)
+ return 0;
+ usleep_range(100, 200);
+ }
+
+ return -ETIMEDOUT;
+}
+
+/*
+ * Start function for rproc_ops
+ *
+ * There is a handshake for start procedure: when DSP starts, it
+ * will send a doorbell message to this driver, then the
+ * REMOTE_IS_READY flags is set, then driver will kick
+ * a message to DSP.
+ */
+static int imx_dsp_rproc_start(struct rproc *rproc)
+{
+ struct imx_dsp_rproc *priv = rproc->priv;
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ const struct imx_rproc_dcfg *dcfg = dsp_dcfg->dcfg;
+ struct device *dev = rproc->dev.parent;
+ int ret;
+
+ switch (dcfg->method) {
+ case IMX_RPROC_MMIO:
+ ret = regmap_update_bits(priv->regmap,
+ dcfg->src_reg,
+ dcfg->src_mask,
+ dcfg->src_start);
+ break;
+ case IMX_RPROC_SCU_API:
+ ret = imx_sc_pm_cpu_start(priv->ipc_handle,
+ IMX_SC_R_DSP,
+ true,
+ rproc->bootaddr);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (ret)
+ dev_err(dev, "Failed to enable remote core!\n");
+ else
+ ret = imx_dsp_rproc_ready(rproc);
+
+ return ret;
+}
+
+/*
+ * Stop function for rproc_ops
+ * It clears the REMOTE_IS_READY flags
+ */
+static int imx_dsp_rproc_stop(struct rproc *rproc)
+{
+ struct imx_dsp_rproc *priv = rproc->priv;
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ const struct imx_rproc_dcfg *dcfg = dsp_dcfg->dcfg;
+ struct device *dev = rproc->dev.parent;
+ int ret = 0;
+
+ /* Make sure work is finished */
+ flush_work(&priv->rproc_work);
+
+ if (rproc->state == RPROC_CRASHED) {
+ priv->flags &= ~REMOTE_IS_READY;
+ return 0;
+ }
+
+ switch (dcfg->method) {
+ case IMX_RPROC_MMIO:
+ ret = regmap_update_bits(priv->regmap, dcfg->src_reg, dcfg->src_mask,
+ dcfg->src_stop);
+ break;
+ case IMX_RPROC_SCU_API:
+ ret = imx_sc_pm_cpu_start(priv->ipc_handle,
+ IMX_SC_R_DSP,
+ false,
+ rproc->bootaddr);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ if (ret)
+ dev_err(dev, "Failed to stop remote core\n");
+ else
+ priv->flags &= ~REMOTE_IS_READY;
+
+ return ret;
+}
+
+/**
+ * imx_dsp_rproc_sys_to_da() - internal memory translation helper
+ * @priv: private data pointer
+ * @sys: system address (DDR address)
+ * @len: length of the memory buffer
+ * @da: device address to translate
+ *
+ * Convert system address (DDR address) to device address (DSP)
+ * for there may be memory remap for device.
+ */
+static int imx_dsp_rproc_sys_to_da(struct imx_dsp_rproc *priv, u64 sys,
+ size_t len, u64 *da)
+{
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ const struct imx_rproc_dcfg *dcfg = dsp_dcfg->dcfg;
+ int i;
+
+ /* Parse address translation table */
+ for (i = 0; i < dcfg->att_size; i++) {
+ const struct imx_rproc_att *att = &dcfg->att[i];
+
+ if (sys >= att->sa && sys + len <= att->sa + att->size) {
+ unsigned int offset = sys - att->sa;
+
+ *da = att->da + offset;
+ return 0;
+ }
+ }
+
+ return -ENOENT;
+}
+
+/* Main virtqueue message work function
+ *
+ * This function is executed upon scheduling of the i.MX DSP remoteproc
+ * driver's workqueue. The workqueue is scheduled by the mailbox rx
+ * handler.
+ *
+ * This work function processes both the Tx and Rx virtqueue indices on
+ * every invocation. The rproc_vq_interrupt function can detect if there
+ * are new unprocessed messages or not (returns IRQ_NONE vs IRQ_HANDLED),
+ * but there is no need to check for these return values. The index 0
+ * triggering will process all pending Rx buffers, and the index 1 triggering
+ * will process all newly available Tx buffers and will wakeup any potentially
+ * blocked senders.
+ *
+ * NOTE:
+ * The current logic is based on an inherent design assumption of supporting
+ * only 2 vrings, but this can be changed if needed.
+ */
+static void imx_dsp_rproc_vq_work(struct work_struct *work)
+{
+ struct imx_dsp_rproc *priv = container_of(work, struct imx_dsp_rproc,
+ rproc_work);
+
+ rproc_vq_interrupt(priv->rproc, 0);
+ rproc_vq_interrupt(priv->rproc, 1);
+}
+
+/**
+ * imx_dsp_rproc_rx_tx_callback() - inbound mailbox message handler
+ * @cl: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * This handler is invoked by mailbox driver whenever a mailbox
+ * message is received. Usually, the SUSPEND and RESUME related messages
+ * are handled in this function, other messages are handled by remoteproc core
+ */
+static void imx_dsp_rproc_rx_tx_callback(struct mbox_client *cl, void *data)
+{
+ struct rproc *rproc = dev_get_drvdata(cl->dev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+ struct device *dev = rproc->dev.parent;
+ u32 message = (u32)(*(u32 *)data);
+
+ dev_dbg(dev, "mbox msg: 0x%x\n", message);
+
+ switch (message) {
+ case RP_MBOX_SUSPEND_ACK:
+ complete(&priv->pm_comp);
+ break;
+ case RP_MBOX_RESUME_ACK:
+ complete(&priv->pm_comp);
+ break;
+ default:
+ schedule_work(&priv->rproc_work);
+ break;
+ }
+}
+
+/**
+ * imx_dsp_rproc_rxdb_callback() - inbound mailbox message handler
+ * @cl: mailbox client pointer used for requesting the mailbox channel
+ * @data: mailbox payload
+ *
+ * For doorbell, there is no message specified, just set REMOTE_IS_READY
+ * flag.
+ */
+static void imx_dsp_rproc_rxdb_callback(struct mbox_client *cl, void *data)
+{
+ struct rproc *rproc = dev_get_drvdata(cl->dev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+
+ /* Remote is ready after firmware is loaded and running */
+ priv->flags |= REMOTE_IS_READY;
+}
+
+/**
+ * imx_dsp_rproc_mbox_init() - request mailbox channels
+ * @priv: private data pointer
+ *
+ * Request three mailbox channels (tx, rx, rxdb).
+ */
+static int imx_dsp_rproc_mbox_init(struct imx_dsp_rproc *priv)
+{
+ struct device *dev = priv->rproc->dev.parent;
+ struct mbox_client *cl;
+ int ret;
+
+ if (!of_get_property(dev->of_node, "mbox-names", NULL))
+ return 0;
+
+ cl = &priv->cl;
+ cl->dev = dev;
+ cl->tx_block = true;
+ cl->tx_tout = 100;
+ cl->knows_txdone = false;
+ cl->rx_callback = imx_dsp_rproc_rx_tx_callback;
+
+ /* Channel for sending message */
+ priv->tx_ch = mbox_request_channel_byname(cl, "tx");
+ if (IS_ERR(priv->tx_ch)) {
+ ret = PTR_ERR(priv->tx_ch);
+ dev_dbg(cl->dev, "failed to request tx mailbox channel: %d\n",
+ ret);
+ goto err_out;
+ }
+
+ /* Channel for receiving message */
+ priv->rx_ch = mbox_request_channel_byname(cl, "rx");
+ if (IS_ERR(priv->rx_ch)) {
+ ret = PTR_ERR(priv->rx_ch);
+ dev_dbg(cl->dev, "failed to request rx mailbox channel: %d\n",
+ ret);
+ goto err_out;
+ }
+
+ cl = &priv->cl_rxdb;
+ cl->dev = dev;
+ cl->rx_callback = imx_dsp_rproc_rxdb_callback;
+
+ /*
+ * RX door bell is used to receive the ready signal from remote
+ * after firmware loaded.
+ */
+ priv->rxdb_ch = mbox_request_channel_byname(cl, "rxdb");
+ if (IS_ERR(priv->rxdb_ch)) {
+ ret = PTR_ERR(priv->rxdb_ch);
+ dev_dbg(cl->dev, "failed to request mbox chan rxdb, ret %d\n",
+ ret);
+ goto err_out;
+ }
+
+ return 0;
+
+err_out:
+ if (!IS_ERR(priv->tx_ch))
+ mbox_free_channel(priv->tx_ch);
+ if (!IS_ERR(priv->rx_ch))
+ mbox_free_channel(priv->rx_ch);
+ if (!IS_ERR(priv->rxdb_ch))
+ mbox_free_channel(priv->rxdb_ch);
+
+ return ret;
+}
+
+static void imx_dsp_rproc_free_mbox(struct imx_dsp_rproc *priv)
+{
+ mbox_free_channel(priv->tx_ch);
+ mbox_free_channel(priv->rx_ch);
+ mbox_free_channel(priv->rxdb_ch);
+}
+
+/**
+ * imx_dsp_rproc_add_carveout() - request mailbox channels
+ * @priv: private data pointer
+ *
+ * This function registers specified memory entry in @rproc carveouts list
+ * The carveouts can help to mapping the memory address for DSP.
+ */
+static int imx_dsp_rproc_add_carveout(struct imx_dsp_rproc *priv)
+{
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ const struct imx_rproc_dcfg *dcfg = dsp_dcfg->dcfg;
+ struct rproc *rproc = priv->rproc;
+ struct device *dev = rproc->dev.parent;
+ struct device_node *np = dev->of_node;
+ struct of_phandle_iterator it;
+ struct rproc_mem_entry *mem;
+ struct reserved_mem *rmem;
+ void __iomem *cpu_addr;
+ int a;
+ u64 da;
+
+ /* Remap required addresses */
+ for (a = 0; a < dcfg->att_size; a++) {
+ const struct imx_rproc_att *att = &dcfg->att[a];
+
+ if (!(att->flags & ATT_OWN))
+ continue;
+
+ if (imx_dsp_rproc_sys_to_da(priv, att->sa, att->size, &da))
+ return -EINVAL;
+
+ cpu_addr = devm_ioremap_wc(dev, att->sa, att->size);
+ if (!cpu_addr) {
+ dev_err(dev, "failed to map memory %p\n", &att->sa);
+ return -ENOMEM;
+ }
+
+ /* Register memory region */
+ mem = rproc_mem_entry_init(dev, cpu_addr, (dma_addr_t)att->sa,
+ att->size, da, NULL, NULL, "dsp_mem");
+
+ if (mem)
+ rproc_coredump_add_segment(rproc, da, att->size);
+ else
+ return -ENOMEM;
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ of_phandle_iterator_init(&it, np, "memory-region", NULL, 0);
+ while (of_phandle_iterator_next(&it) == 0) {
+ /*
+ * Ignore the first memory region which will be used vdev buffer.
+ * No need to do extra handlings, rproc_add_virtio_dev will handle it.
+ */
+ if (!strcmp(it.node->name, "vdev0buffer"))
+ continue;
+
+ rmem = of_reserved_mem_lookup(it.node);
+ if (!rmem) {
+ dev_err(dev, "unable to acquire memory-region\n");
+ return -EINVAL;
+ }
+
+ if (imx_dsp_rproc_sys_to_da(priv, rmem->base, rmem->size, &da))
+ return -EINVAL;
+
+ cpu_addr = devm_ioremap_wc(dev, rmem->base, rmem->size);
+ if (!cpu_addr) {
+ dev_err(dev, "failed to map memory %p\n", &rmem->base);
+ return -ENOMEM;
+ }
+
+ /* Register memory region */
+ mem = rproc_mem_entry_init(dev, cpu_addr, (dma_addr_t)rmem->base,
+ rmem->size, da, NULL, NULL, it.node->name);
+
+ if (mem)
+ rproc_coredump_add_segment(rproc, da, rmem->size);
+ else
+ return -ENOMEM;
+
+ rproc_add_carveout(rproc, mem);
+ }
+
+ return 0;
+}
+
+/**
+ * imx_dsp_rproc_elf_load_segments() - load firmware segments to memory
+ * @rproc: remote processor which will be booted using these fw segments
+ * @fw: the ELF firmware image
+ *
+ * This function specially checks if memsz is zero or not, otherwise it
+ * is mostly same as rproc_elf_load_segments().
+ */
+static int imx_dsp_rproc_elf_load_segments(struct rproc *rproc,
+ const struct firmware *fw)
+{
+ struct device *dev = &rproc->dev;
+ u8 class = fw_elf_get_class(fw);
+ u32 elf_phdr_get_size = elf_size_of_phdr(class);
+ const u8 *elf_data = fw->data;
+ const void *ehdr, *phdr;
+ int i, ret = 0;
+ u16 phnum;
+
+ ehdr = elf_data;
+ phnum = elf_hdr_get_e_phnum(class, ehdr);
+ phdr = elf_data + elf_hdr_get_e_phoff(class, ehdr);
+
+ /* go through the available ELF segments */
+ for (i = 0; i < phnum; i++, phdr += elf_phdr_get_size) {
+ u64 da = elf_phdr_get_p_paddr(class, phdr);
+ u64 memsz = elf_phdr_get_p_memsz(class, phdr);
+ u64 filesz = elf_phdr_get_p_filesz(class, phdr);
+ u64 offset = elf_phdr_get_p_offset(class, phdr);
+ u32 type = elf_phdr_get_p_type(class, phdr);
+ void *ptr;
+
+ /*
+ * There is a case that with PT_LOAD type, the
+ * filesz = memsz = 0. If memsz = 0, rproc_da_to_va
+ * should return NULL ptr, then error is returned.
+ * So this case should be skipped from the loop.
+ * Add !memsz checking here.
+ */
+ if (type != PT_LOAD || !memsz)
+ continue;
+
+ dev_dbg(dev, "phdr: type %d da 0x%llx memsz 0x%llx filesz 0x%llx\n",
+ type, da, memsz, filesz);
+
+ if (filesz > memsz) {
+ dev_err(dev, "bad phdr filesz 0x%llx memsz 0x%llx\n",
+ filesz, memsz);
+ ret = -EINVAL;
+ break;
+ }
+
+ if (offset + filesz > fw->size) {
+ dev_err(dev, "truncated fw: need 0x%llx avail 0x%zx\n",
+ offset + filesz, fw->size);
+ ret = -EINVAL;
+ break;
+ }
+
+ if (!rproc_u64_fit_in_size_t(memsz)) {
+ dev_err(dev, "size (%llx) does not fit in size_t type\n",
+ memsz);
+ ret = -EOVERFLOW;
+ break;
+ }
+
+ /* grab the kernel address for this device address */
+ ptr = rproc_da_to_va(rproc, da, memsz, NULL);
+ if (!ptr) {
+ dev_err(dev, "bad phdr da 0x%llx mem 0x%llx\n", da,
+ memsz);
+ ret = -EINVAL;
+ break;
+ }
+
+ /* put the segment where the remote processor expects it */
+ if (filesz)
+ memcpy(ptr, elf_data + offset, filesz);
+
+ /*
+ * Zero out remaining memory for this segment.
+ *
+ * This isn't strictly required since dma_alloc_coherent already
+ * did this for us. albeit harmless, we may consider removing
+ * this.
+ */
+ if (memsz > filesz)
+ memset(ptr + filesz, 0, memsz - filesz);
+ }
+
+ return ret;
+}
+
+/* Prepare function for rproc_ops */
+static int imx_dsp_rproc_prepare(struct rproc *rproc)
+{
+ struct imx_dsp_rproc *priv = rproc->priv;
+ struct device *dev = rproc->dev.parent;
+ struct rproc_mem_entry *carveout;
+ int ret;
+
+ ret = imx_dsp_rproc_add_carveout(priv);
+ if (ret) {
+ dev_err(dev, "failed on imx_dsp_rproc_add_carveout\n");
+ return ret;
+ }
+
+ pm_runtime_get_sync(dev);
+
+ /*
+ * Clear buffers after pm rumtime for internal ocram is not
+ * accessible if power and clock are not enabled.
+ */
+ list_for_each_entry(carveout, &rproc->carveouts, node) {
+ if (carveout->va)
+ memset(carveout->va, 0, carveout->len);
+ }
+
+ return 0;
+}
+
+/* Unprepare function for rproc_ops */
+static int imx_dsp_rproc_unprepare(struct rproc *rproc)
+{
+ pm_runtime_put_sync(rproc->dev.parent);
+
+ return 0;
+}
+
+/* Kick function for rproc_ops */
+static void imx_dsp_rproc_kick(struct rproc *rproc, int vqid)
+{
+ struct imx_dsp_rproc *priv = rproc->priv;
+ struct device *dev = rproc->dev.parent;
+ int err;
+ __u32 mmsg;
+
+ if (!priv->tx_ch) {
+ dev_err(dev, "No initialized mbox tx channel\n");
+ return;
+ }
+
+ /*
+ * Send the index of the triggered virtqueue as the mu payload.
+ * Let remote processor know which virtqueue is used.
+ */
+ mmsg = vqid;
+
+ err = mbox_send_message(priv->tx_ch, (void *)&mmsg);
+ if (err < 0)
+ dev_err(dev, "%s: failed (%d, err:%d)\n", __func__, vqid, err);
+}
+
+static const struct rproc_ops imx_dsp_rproc_ops = {
+ .prepare = imx_dsp_rproc_prepare,
+ .unprepare = imx_dsp_rproc_unprepare,
+ .start = imx_dsp_rproc_start,
+ .stop = imx_dsp_rproc_stop,
+ .kick = imx_dsp_rproc_kick,
+ .load = imx_dsp_rproc_elf_load_segments,
+ .parse_fw = rproc_elf_load_rsc_table,
+ .sanity_check = rproc_elf_sanity_check,
+ .get_boot_addr = rproc_elf_get_boot_addr,
+};
+
+/**
+ * imx_dsp_attach_pm_domains() - attach the power domains
+ * @priv: private data pointer
+ *
+ * On i.MX8QM and i.MX8QXP there is multiple power domains
+ * required, so need to link them.
+ */
+static int imx_dsp_attach_pm_domains(struct imx_dsp_rproc *priv)
+{
+ struct device *dev = priv->rproc->dev.parent;
+ int ret, i;
+
+ priv->num_domains = of_count_phandle_with_args(dev->of_node,
+ "power-domains",
+ "#power-domain-cells");
+
+ /* If only one domain, then no need to link the device */
+ if (priv->num_domains <= 1)
+ return 0;
+
+ priv->pd_dev = devm_kmalloc_array(dev, priv->num_domains,
+ sizeof(*priv->pd_dev),
+ GFP_KERNEL);
+ if (!priv->pd_dev)
+ return -ENOMEM;
+
+ priv->pd_dev_link = devm_kmalloc_array(dev, priv->num_domains,
+ sizeof(*priv->pd_dev_link),
+ GFP_KERNEL);
+ if (!priv->pd_dev_link)
+ return -ENOMEM;
+
+ for (i = 0; i < priv->num_domains; i++) {
+ priv->pd_dev[i] = dev_pm_domain_attach_by_id(dev, i);
+ if (IS_ERR(priv->pd_dev[i])) {
+ ret = PTR_ERR(priv->pd_dev[i]);
+ goto detach_pm;
+ }
+
+ /*
+ * device_link_add will check priv->pd_dev[i], if it is
+ * NULL, then will break.
+ */
+ priv->pd_dev_link[i] = device_link_add(dev,
+ priv->pd_dev[i],
+ DL_FLAG_STATELESS |
+ DL_FLAG_PM_RUNTIME);
+ if (!priv->pd_dev_link[i]) {
+ dev_pm_domain_detach(priv->pd_dev[i], false);
+ ret = -EINVAL;
+ goto detach_pm;
+ }
+ }
+
+ return 0;
+
+detach_pm:
+ while (--i >= 0) {
+ device_link_del(priv->pd_dev_link[i]);
+ dev_pm_domain_detach(priv->pd_dev[i], false);
+ }
+
+ return ret;
+}
+
+static int imx_dsp_detach_pm_domains(struct imx_dsp_rproc *priv)
+{
+ int i;
+
+ if (priv->num_domains <= 1)
+ return 0;
+
+ for (i = 0; i < priv->num_domains; i++) {
+ device_link_del(priv->pd_dev_link[i]);
+ dev_pm_domain_detach(priv->pd_dev[i], false);
+ }
+
+ return 0;
+}
+
+/**
+ * imx_dsp_rproc_detect_mode() - detect DSP control mode
+ * @priv: private data pointer
+ *
+ * Different platform has different control method for DSP, which depends
+ * on how the DSP is integrated in platform.
+ *
+ * For i.MX8QXP and i.MX8QM, DSP should be started and stopped by System
+ * Control Unit.
+ * For i.MX8MP and i.MX8ULP, DSP should be started and stopped by system
+ * integration module.
+ */
+static int imx_dsp_rproc_detect_mode(struct imx_dsp_rproc *priv)
+{
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ struct device *dev = priv->rproc->dev.parent;
+ struct regmap *regmap;
+ int ret = 0;
+
+ switch (dsp_dcfg->dcfg->method) {
+ case IMX_RPROC_SCU_API:
+ ret = imx_scu_get_handle(&priv->ipc_handle);
+ if (ret)
+ return ret;
+ break;
+ case IMX_RPROC_MMIO:
+ regmap = syscon_regmap_lookup_by_phandle(dev->of_node, "fsl,dsp-ctrl");
+ if (IS_ERR(regmap)) {
+ dev_err(dev, "failed to find syscon\n");
+ return PTR_ERR(regmap);
+ }
+
+ priv->regmap = regmap;
+ break;
+ default:
+ ret = -EOPNOTSUPP;
+ break;
+ }
+
+ return ret;
+}
+
+static const char *imx_dsp_clks_names[DSP_RPROC_CLK_MAX] = {
+ /* DSP clocks */
+ "core", "ocram", "debug", "ipg", "mu",
+};
+
+static int imx_dsp_rproc_clk_get(struct imx_dsp_rproc *priv)
+{
+ struct device *dev = priv->rproc->dev.parent;
+ struct clk_bulk_data *clks = priv->clks;
+ int i;
+
+ for (i = 0; i < DSP_RPROC_CLK_MAX; i++)
+ clks[i].id = imx_dsp_clks_names[i];
+
+ return devm_clk_bulk_get_optional(dev, DSP_RPROC_CLK_MAX, clks);
+}
+
+static int imx_dsp_rproc_probe(struct platform_device *pdev)
+{
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg;
+ struct device *dev = &pdev->dev;
+ struct imx_dsp_rproc *priv;
+ struct rproc *rproc;
+ const char *fw_name;
+ int ret;
+
+ dsp_dcfg = of_device_get_match_data(dev);
+ if (!dsp_dcfg)
+ return -ENODEV;
+
+ ret = rproc_of_parse_firmware(dev, 0, &fw_name);
+ if (ret) {
+ dev_err(dev, "failed to parse firmware-name property, ret = %d\n",
+ ret);
+ return ret;
+ }
+
+ rproc = rproc_alloc(dev, "imx-dsp-rproc", &imx_dsp_rproc_ops, fw_name,
+ sizeof(*priv));
+ if (!rproc)
+ return -ENOMEM;
+
+ priv = rproc->priv;
+ priv->rproc = rproc;
+ priv->dsp_dcfg = dsp_dcfg;
+
+ dev_set_drvdata(dev, rproc);
+
+ INIT_WORK(&priv->rproc_work, imx_dsp_rproc_vq_work);
+
+ ret = imx_dsp_rproc_detect_mode(priv);
+ if (ret) {
+ dev_err(dev, "failed on imx_dsp_rproc_detect_mode\n");
+ goto err_put_rproc;
+ }
+
+ /* There are multiple power domains required by DSP on some platform */
+ ret = imx_dsp_attach_pm_domains(priv);
+ if (ret) {
+ dev_err(dev, "failed on imx_dsp_attach_pm_domains\n");
+ goto err_put_rproc;
+ }
+ /* Get clocks */
+ ret = imx_dsp_rproc_clk_get(priv);
+ if (ret) {
+ dev_err(dev, "failed on imx_dsp_rproc_clk_get\n");
+ goto err_detach_domains;
+ }
+
+ init_completion(&priv->pm_comp);
+ rproc->auto_boot = false;
+ ret = rproc_add(rproc);
+ if (ret) {
+ dev_err(dev, "rproc_add failed\n");
+ goto err_detach_domains;
+ }
+
+ pm_runtime_enable(dev);
+
+ return 0;
+
+err_detach_domains:
+ imx_dsp_detach_pm_domains(priv);
+err_put_rproc:
+ rproc_free(rproc);
+
+ return ret;
+}
+
+static int imx_dsp_rproc_remove(struct platform_device *pdev)
+{
+ struct rproc *rproc = platform_get_drvdata(pdev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+
+ pm_runtime_disable(&pdev->dev);
+ rproc_del(rproc);
+ imx_dsp_detach_pm_domains(priv);
+ rproc_free(rproc);
+
+ return 0;
+}
+
+/* pm runtime functions */
+static int imx_dsp_runtime_resume(struct device *dev)
+{
+ struct rproc *rproc = dev_get_drvdata(dev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+ const struct imx_dsp_rproc_dcfg *dsp_dcfg = priv->dsp_dcfg;
+ int ret;
+
+ /*
+ * There is power domain attached with mailbox, if setup mailbox
+ * in probe(), then the power of mailbox is always enabled,
+ * the power can't be saved.
+ * So move setup of mailbox to runtime resume.
+ */
+ ret = imx_dsp_rproc_mbox_init(priv);
+ if (ret) {
+ dev_err(dev, "failed on imx_dsp_rproc_mbox_init\n");
+ return ret;
+ }
+
+ ret = clk_bulk_prepare_enable(DSP_RPROC_CLK_MAX, priv->clks);
+ if (ret) {
+ dev_err(dev, "failed on clk_bulk_prepare_enable\n");
+ return ret;
+ }
+
+ /* Reset DSP if needed */
+ if (dsp_dcfg->reset)
+ dsp_dcfg->reset(priv);
+
+ return 0;
+}
+
+static int imx_dsp_runtime_suspend(struct device *dev)
+{
+ struct rproc *rproc = dev_get_drvdata(dev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+
+ clk_bulk_disable_unprepare(DSP_RPROC_CLK_MAX, priv->clks);
+
+ imx_dsp_rproc_free_mbox(priv);
+
+ return 0;
+}
+
+static void imx_dsp_load_firmware(const struct firmware *fw, void *context)
+{
+ struct rproc *rproc = context;
+ int ret;
+
+ /*
+ * Same flow as start procedure.
+ * Load the ELF segments to memory firstly.
+ */
+ ret = rproc_load_segments(rproc, fw);
+ if (ret)
+ goto out;
+
+ /* Start the remote processor */
+ ret = rproc->ops->start(rproc);
+ if (ret)
+ goto out;
+
+ rproc->ops->kick(rproc, 0);
+
+out:
+ release_firmware(fw);
+}
+
+static __maybe_unused int imx_dsp_suspend(struct device *dev)
+{
+ struct rproc *rproc = dev_get_drvdata(dev);
+ struct imx_dsp_rproc *priv = rproc->priv;
+ __u32 mmsg = RP_MBOX_SUSPEND_SYSTEM;
+ int ret;
+
+ if (rproc->state != RPROC_RUNNING)
+ goto out;
+
+ reinit_completion(&priv->pm_comp);
+
+ /* Tell DSP that suspend is happening */
+ ret = mbox_send_message(priv->tx_ch, (void *)&mmsg);
+ if (ret < 0) {
+ dev_err(dev, "PM mbox_send_message failed: %d\n", ret);
+ return ret;
+ }
+
+ /*
+ * DSP need to save the context at suspend.
+ * Here waiting the response for DSP, then power can be disabled.
+ */
+ if (!wait_for_completion_timeout(&priv->pm_comp, msecs_to_jiffies(100)))
+ return -EBUSY;
+
+out:
+ /*
+ * The power of DSP is disabled in suspend, so force pm runtime
+ * to be suspend, then we can reenable the power and clocks at
+ * resume stage.
+ */
+ return pm_runtime_force_suspend(dev);
+}
+
+static __maybe_unused int imx_dsp_resume(struct device *dev)
+{
+ struct rproc *rproc = dev_get_drvdata(dev);
+ int ret = 0;
+
+ ret = pm_runtime_force_resume(dev);
+ if (ret)
+ return ret;
+
+ if (rproc->state != RPROC_RUNNING)
+ return 0;
+
+ /*
+ * The power of DSP is disabled at suspend, the memory of dsp
+ * is reset, the image segments are lost. So need to reload
+ * firmware and restart the DSP if it is in running state.
+ */
+ ret = request_firmware_nowait(THIS_MODULE, FW_ACTION_UEVENT,
+ rproc->firmware, dev, GFP_KERNEL,
+ rproc, imx_dsp_load_firmware);
+ if (ret < 0) {
+ dev_err(dev, "load firmware failed: %d\n", ret);
+ goto err;
+ }
+
+ return 0;
+
+err:
+ pm_runtime_force_suspend(dev);
+
+ return ret;
+}
+
+static const struct dev_pm_ops imx_dsp_rproc_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(imx_dsp_suspend, imx_dsp_resume)
+ SET_RUNTIME_PM_OPS(imx_dsp_runtime_suspend,
+ imx_dsp_runtime_resume, NULL)
+};
+
+static const struct of_device_id imx_dsp_rproc_of_match[] = {
+ { .compatible = "fsl,imx8qxp-hifi4", .data = &imx_dsp_rproc_cfg_imx8qxp },
+ { .compatible = "fsl,imx8qm-hifi4", .data = &imx_dsp_rproc_cfg_imx8qm },
+ { .compatible = "fsl,imx8mp-hifi4", .data = &imx_dsp_rproc_cfg_imx8mp },
+ { .compatible = "fsl,imx8ulp-hifi4", .data = &imx_dsp_rproc_cfg_imx8ulp },
+ {},
+};
+MODULE_DEVICE_TABLE(of, imx_dsp_rproc_of_match);
+
+static struct platform_driver imx_dsp_rproc_driver = {
+ .probe = imx_dsp_rproc_probe,
+ .remove = imx_dsp_rproc_remove,
+ .driver = {
+ .name = "imx-dsp-rproc",
+ .of_match_table = imx_dsp_rproc_of_match,
+ .pm = &imx_dsp_rproc_pm_ops,
+ },
+};
+module_platform_driver(imx_dsp_rproc_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("i.MX HiFi Core Remote Processor Control Driver");
+MODULE_AUTHOR("Shengjiu Wang <shengjiu.wang@nxp.com>");
#include <linux/remoteproc.h>
#include <linux/workqueue.h>
+#include "imx_rproc.h"
#include "remoteproc_internal.h"
#define IMX7D_SRC_SCR 0x0C
/* att flags */
/* M4 own area. Can be mapped at probe */
#define ATT_OWN BIT(1)
-
-/* address translation table */
-struct imx_rproc_att {
- u32 da; /* device address (From Cortex M4 view)*/
- u32 sa; /* system bus address */
- u32 size; /* size of reg range */
- int flags;
-};
-
-/* Remote core start/stop method */
-enum imx_rproc_method {
- IMX_RPROC_NONE,
- /* Through syscon regmap */
- IMX_RPROC_MMIO,
- /* Through ARM SMCCC */
- IMX_RPROC_SMC,
-};
-
-struct imx_rproc_dcfg {
- u32 src_reg;
- u32 src_mask;
- u32 src_start;
- u32 src_stop;
- const struct imx_rproc_att *att;
- size_t att_size;
- enum imx_rproc_method method;
-};
+#define ATT_IOMEM BIT(2)
struct imx_rproc {
struct device *dev;
static const struct imx_rproc_att imx_rproc_att_imx8mn[] = {
/* dev addr , sys addr , size , flags */
/* ITCM */
- { 0x00000000, 0x007E0000, 0x00020000, ATT_OWN },
+ { 0x00000000, 0x007E0000, 0x00020000, ATT_OWN | ATT_IOMEM },
/* OCRAM_S */
{ 0x00180000, 0x00180000, 0x00009000, 0 },
/* OCRAM */
/* DDR (Code) - alias */
{ 0x10000000, 0x40000000, 0x0FFE0000, 0 },
/* DTCM */
- { 0x20000000, 0x00800000, 0x00020000, ATT_OWN },
+ { 0x20000000, 0x00800000, 0x00020000, ATT_OWN | ATT_IOMEM },
/* OCRAM_S - alias */
{ 0x20180000, 0x00180000, 0x00008000, ATT_OWN },
/* OCRAM */
static const struct imx_rproc_att imx_rproc_att_imx8mq[] = {
/* dev addr , sys addr , size , flags */
/* TCML - alias */
- { 0x00000000, 0x007e0000, 0x00020000, 0 },
+ { 0x00000000, 0x007e0000, 0x00020000, ATT_IOMEM},
/* OCRAM_S */
{ 0x00180000, 0x00180000, 0x00008000, 0 },
/* OCRAM */
/* DDR (Code) - alias */
{ 0x10000000, 0x80000000, 0x0FFE0000, 0 },
/* TCML */
- { 0x1FFE0000, 0x007E0000, 0x00020000, ATT_OWN },
+ { 0x1FFE0000, 0x007E0000, 0x00020000, ATT_OWN | ATT_IOMEM},
/* TCMU */
- { 0x20000000, 0x00800000, 0x00020000, ATT_OWN },
+ { 0x20000000, 0x00800000, 0x00020000, ATT_OWN | ATT_IOMEM},
/* OCRAM_S */
{ 0x20180000, 0x00180000, 0x00008000, ATT_OWN },
/* OCRAM */
/* OCRAM_PXP (Code) - alias */
{ 0x00940000, 0x00940000, 0x00008000, 0 },
/* TCML (Code) */
- { 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN },
+ { 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN | ATT_IOMEM },
/* DDR (Code) - alias, first part of DDR (Data) */
{ 0x10000000, 0x80000000, 0x0FFF0000, 0 },
/* TCMU (Data) */
- { 0x20000000, 0x00800000, 0x00008000, ATT_OWN },
+ { 0x20000000, 0x00800000, 0x00008000, ATT_OWN | ATT_IOMEM },
/* OCRAM (Data) */
{ 0x20200000, 0x00900000, 0x00020000, 0 },
/* OCRAM_EPDC (Data) */
static const struct imx_rproc_att imx_rproc_att_imx6sx[] = {
/* dev addr , sys addr , size , flags */
/* TCML (M4 Boot Code) - alias */
- { 0x00000000, 0x007F8000, 0x00008000, 0 },
+ { 0x00000000, 0x007F8000, 0x00008000, ATT_IOMEM },
/* OCRAM_S (Code) */
{ 0x00180000, 0x008F8000, 0x00004000, 0 },
/* OCRAM_S (Code) - alias */
{ 0x00180000, 0x008FC000, 0x00004000, 0 },
/* TCML (Code) */
- { 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN },
+ { 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN | ATT_IOMEM },
/* DDR (Code) - alias, first part of DDR (Data) */
{ 0x10000000, 0x80000000, 0x0FFF8000, 0 },
/* TCMU (Data) */
- { 0x20000000, 0x00800000, 0x00008000, ATT_OWN },
+ { 0x20000000, 0x00800000, 0x00008000, ATT_OWN | ATT_IOMEM },
/* OCRAM_S (Data) - alias? */
{ 0x208F8000, 0x008F8000, 0x00004000, 0 },
/* DDR (Data) */
}
static int imx_rproc_da_to_sys(struct imx_rproc *priv, u64 da,
- size_t len, u64 *sys)
+ size_t len, u64 *sys, bool *is_iomem)
{
const struct imx_rproc_dcfg *dcfg = priv->dcfg;
int i;
unsigned int offset = da - att->da;
*sys = att->sa + offset;
+ if (is_iomem)
+ *is_iomem = att->flags & ATT_IOMEM;
return 0;
}
}
* On device side we have many aliases, so we need to convert device
* address (M4) to system bus address first.
*/
- if (imx_rproc_da_to_sys(priv, da, len, &sys))
+ if (imx_rproc_da_to_sys(priv, da, len, &sys, is_iomem))
return NULL;
for (i = 0; i < IMX_RPROC_MEM_MAX; i++) {
if (b >= IMX_RPROC_MEM_MAX)
break;
- priv->mem[b].cpu_addr = devm_ioremap(&pdev->dev,
- att->sa, att->size);
+ if (att->flags & ATT_IOMEM)
+ priv->mem[b].cpu_addr = devm_ioremap(&pdev->dev,
+ att->sa, att->size);
+ else
+ priv->mem[b].cpu_addr = devm_ioremap_wc(&pdev->dev,
+ att->sa, att->size);
if (!priv->mem[b].cpu_addr) {
dev_err(dev, "failed to remap %#x bytes from %#x\n", att->size, att->sa);
return -ENOMEM;
struct resource res;
node = of_parse_phandle(np, "memory-region", a);
- /* Not map vdev region */
- if (!strcmp(node->name, "vdev"))
+ /* Not map vdevbuffer, vdevring region */
+ if (!strncmp(node->name, "vdev", strlen("vdev")))
continue;
err = of_address_to_resource(node, 0, &res);
if (err) {
break;
/* Not use resource version, because we might share region */
- priv->mem[b].cpu_addr = devm_ioremap(&pdev->dev, res.start, resource_size(&res));
+ priv->mem[b].cpu_addr = devm_ioremap_wc(&pdev->dev, res.start, resource_size(&res));
if (!priv->mem[b].cpu_addr) {
dev_err(dev, "failed to remap %pr\n", &res);
return -ENOMEM;
}
priv->mem[b].sys_addr = res.start;
priv->mem[b].size = resource_size(&res);
- if (!strcmp(node->name, "rsc_table"))
+ if (!strcmp(node->name, "rsc-table"))
priv->rsc_table = priv->mem[b].cpu_addr;
b++;
}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (c) 2017 Pengutronix, Oleksij Rempel <kernel@pengutronix.de>
+ * Copyright 2021 NXP
+ */
+
+#ifndef _IMX_RPROC_H
+#define _IMX_RPROC_H
+
+/* address translation table */
+struct imx_rproc_att {
+ u32 da; /* device address (From Cortex M4 view)*/
+ u32 sa; /* system bus address */
+ u32 size; /* size of reg range */
+ int flags;
+};
+
+/* Remote core start/stop method */
+enum imx_rproc_method {
+ IMX_RPROC_NONE,
+ /* Through syscon regmap */
+ IMX_RPROC_MMIO,
+ /* Through ARM SMCCC */
+ IMX_RPROC_SMC,
+ /* Through System Control Unit API */
+ IMX_RPROC_SCU_API,
+};
+
+struct imx_rproc_dcfg {
+ u32 src_reg;
+ u32 src_mask;
+ u32 src_start;
+ u32 src_stop;
+ const struct imx_rproc_att *att;
+ size_t att_size;
+ enum imx_rproc_method method;
+};
+
+#endif /* _IMX_RPROC_H */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2020 Martin Blumenstingl <martin.blumenstingl@googlemail.com>
+ */
+
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/genalloc.h>
+#include <linux/io.h>
+#include <linux/mfd/syscon.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/property.h>
+#include <linux/regmap.h>
+#include <linux/remoteproc.h>
+#include <linux/reset.h>
+#include <linux/sizes.h>
+
+#include "remoteproc_internal.h"
+
+#define AO_REMAP_REG0 0x0
+#define AO_REMAP_REG0_REMAP_AHB_SRAM_BITS_17_14_FOR_ARM_CPU GENMASK(3, 0)
+
+#define AO_REMAP_REG1 0x4
+#define AO_REMAP_REG1_MOVE_AHB_SRAM_TO_0X0_INSTEAD_OF_DDR BIT(4)
+#define AO_REMAP_REG1_REMAP_AHB_SRAM_BITS_17_14_FOR_MEDIA_CPU GENMASK(3, 0)
+
+#define AO_CPU_CNTL 0x0
+#define AO_CPU_CNTL_AHB_SRAM_BITS_31_20 GENMASK(28, 16)
+#define AO_CPU_CNTL_HALT BIT(9)
+#define AO_CPU_CNTL_UNKNONWN BIT(8)
+#define AO_CPU_CNTL_RUN BIT(0)
+
+#define AO_CPU_STAT 0x4
+
+#define AO_SECURE_REG0 0x0
+#define AO_SECURE_REG0_AHB_SRAM_BITS_19_12 GENMASK(15, 8)
+
+/* Only bits [31:20] and [17:14] are usable, all other bits must be zero */
+#define MESON_AO_RPROC_SRAM_USABLE_BITS 0xfff3c000ULL
+
+#define MESON_AO_RPROC_MEMORY_OFFSET 0x10000000
+
+struct meson_mx_ao_arc_rproc_priv {
+ void __iomem *remap_base;
+ void __iomem *cpu_base;
+ unsigned long sram_va;
+ phys_addr_t sram_pa;
+ size_t sram_size;
+ struct gen_pool *sram_pool;
+ struct reset_control *arc_reset;
+ struct clk *arc_pclk;
+ struct regmap *secbus2_regmap;
+};
+
+static int meson_mx_ao_arc_rproc_start(struct rproc *rproc)
+{
+ struct meson_mx_ao_arc_rproc_priv *priv = rproc->priv;
+ phys_addr_t translated_sram_addr;
+ u32 tmp;
+ int ret;
+
+ ret = clk_prepare_enable(priv->arc_pclk);
+ if (ret)
+ return ret;
+
+ tmp = FIELD_PREP(AO_REMAP_REG0_REMAP_AHB_SRAM_BITS_17_14_FOR_ARM_CPU,
+ priv->sram_pa >> 14);
+ writel(tmp, priv->remap_base + AO_REMAP_REG0);
+
+ /*
+ * The SRAM content as seen by the ARC core always starts at 0x0
+ * regardless of the value given here (this was discovered by trial and
+ * error). For SoCs older than Meson6 we probably have to set
+ * AO_REMAP_REG1_MOVE_AHB_SRAM_TO_0X0_INSTEAD_OF_DDR to achieve the
+ * same. (At least) For Meson8 and newer that bit must not be set.
+ */
+ writel(0x0, priv->remap_base + AO_REMAP_REG1);
+
+ regmap_update_bits(priv->secbus2_regmap, AO_SECURE_REG0,
+ AO_SECURE_REG0_AHB_SRAM_BITS_19_12,
+ FIELD_PREP(AO_SECURE_REG0_AHB_SRAM_BITS_19_12,
+ priv->sram_pa >> 12));
+
+ ret = reset_control_reset(priv->arc_reset);
+ if (ret) {
+ clk_disable_unprepare(priv->arc_pclk);
+ return ret;
+ }
+
+ usleep_range(10, 100);
+
+ /*
+ * Convert from 0xd9000000 to 0xc9000000 as the vendor driver does.
+ * This only seems to be relevant for the AO_CPU_CNTL register. It is
+ * unknown why this is needed.
+ */
+ translated_sram_addr = priv->sram_pa - MESON_AO_RPROC_MEMORY_OFFSET;
+
+ tmp = FIELD_PREP(AO_CPU_CNTL_AHB_SRAM_BITS_31_20,
+ translated_sram_addr >> 20);
+ tmp |= AO_CPU_CNTL_UNKNONWN | AO_CPU_CNTL_RUN;
+ writel(tmp, priv->cpu_base + AO_CPU_CNTL);
+
+ usleep_range(20, 200);
+
+ return 0;
+}
+
+static int meson_mx_ao_arc_rproc_stop(struct rproc *rproc)
+{
+ struct meson_mx_ao_arc_rproc_priv *priv = rproc->priv;
+
+ writel(AO_CPU_CNTL_HALT, priv->cpu_base + AO_CPU_CNTL);
+
+ clk_disable_unprepare(priv->arc_pclk);
+
+ return 0;
+}
+
+static void *meson_mx_ao_arc_rproc_da_to_va(struct rproc *rproc, u64 da,
+ size_t len, bool *is_iomem)
+{
+ struct meson_mx_ao_arc_rproc_priv *priv = rproc->priv;
+
+ /* The memory from the ARC core's perspective always starts at 0x0. */
+ if ((da + len) > priv->sram_size)
+ return NULL;
+
+ return (void *)priv->sram_va + da;
+}
+
+static struct rproc_ops meson_mx_ao_arc_rproc_ops = {
+ .start = meson_mx_ao_arc_rproc_start,
+ .stop = meson_mx_ao_arc_rproc_stop,
+ .da_to_va = meson_mx_ao_arc_rproc_da_to_va,
+ .get_boot_addr = rproc_elf_get_boot_addr,
+ .load = rproc_elf_load_segments,
+ .sanity_check = rproc_elf_sanity_check,
+};
+
+static int meson_mx_ao_arc_rproc_probe(struct platform_device *pdev)
+{
+ struct meson_mx_ao_arc_rproc_priv *priv;
+ struct device *dev = &pdev->dev;
+ const char *fw_name = NULL;
+ struct rproc *rproc;
+ int ret;
+
+ device_property_read_string(dev, "firmware-name", &fw_name);
+
+ rproc = devm_rproc_alloc(dev, "meson-mx-ao-arc",
+ &meson_mx_ao_arc_rproc_ops, fw_name,
+ sizeof(*priv));
+ if (!rproc)
+ return -ENOMEM;
+
+ rproc->has_iommu = false;
+ priv = rproc->priv;
+
+ priv->sram_pool = of_gen_pool_get(dev->of_node, "sram", 0);
+ if (!priv->sram_pool) {
+ dev_err(dev, "Could not get SRAM pool\n");
+ return -ENODEV;
+ }
+
+ priv->sram_size = gen_pool_avail(priv->sram_pool);
+
+ priv->sram_va = gen_pool_alloc(priv->sram_pool, priv->sram_size);
+ if (!priv->sram_va) {
+ dev_err(dev, "Could not alloc memory in SRAM pool\n");
+ return -ENOMEM;
+ }
+
+ priv->sram_pa = gen_pool_virt_to_phys(priv->sram_pool, priv->sram_va);
+ if (priv->sram_pa & ~MESON_AO_RPROC_SRAM_USABLE_BITS) {
+ dev_err(dev, "SRAM address contains unusable bits\n");
+ ret = -EINVAL;
+ goto err_free_genpool;
+ }
+
+ priv->secbus2_regmap = syscon_regmap_lookup_by_phandle(dev->of_node,
+ "amlogic,secbus2");
+ if (IS_ERR(priv->secbus2_regmap)) {
+ dev_err(dev, "Failed to find SECBUS2 regmap\n");
+ ret = PTR_ERR(priv->secbus2_regmap);
+ goto err_free_genpool;
+ }
+
+ priv->remap_base = devm_platform_ioremap_resource_byname(pdev, "remap");
+ if (IS_ERR(priv->remap_base)) {
+ ret = PTR_ERR(priv->remap_base);
+ goto err_free_genpool;
+ }
+
+ priv->cpu_base = devm_platform_ioremap_resource_byname(pdev, "cpu");
+ if (IS_ERR(priv->cpu_base)) {
+ ret = PTR_ERR(priv->cpu_base);
+ goto err_free_genpool;
+ }
+
+ priv->arc_reset = devm_reset_control_get_exclusive(dev, NULL);
+ if (IS_ERR(priv->arc_reset)) {
+ dev_err(dev, "Failed to get ARC reset\n");
+ ret = PTR_ERR(priv->arc_reset);
+ goto err_free_genpool;
+ }
+
+ priv->arc_pclk = devm_clk_get(dev, NULL);
+ if (IS_ERR(priv->arc_pclk)) {
+ dev_err(dev, "Failed to get the ARC PCLK\n");
+ ret = PTR_ERR(priv->arc_pclk);
+ goto err_free_genpool;
+ }
+
+ platform_set_drvdata(pdev, rproc);
+
+ ret = rproc_add(rproc);
+ if (ret)
+ goto err_free_genpool;
+
+ return 0;
+
+err_free_genpool:
+ gen_pool_free(priv->sram_pool, priv->sram_va, priv->sram_size);
+ return ret;
+}
+
+static int meson_mx_ao_arc_rproc_remove(struct platform_device *pdev)
+{
+ struct rproc *rproc = platform_get_drvdata(pdev);
+ struct meson_mx_ao_arc_rproc_priv *priv = rproc->priv;
+
+ rproc_del(rproc);
+ gen_pool_free(priv->sram_pool, priv->sram_va, priv->sram_size);
+
+ return 0;
+}
+
+static const struct of_device_id meson_mx_ao_arc_rproc_match[] = {
+ { .compatible = "amlogic,meson8-ao-arc" },
+ { .compatible = "amlogic,meson8b-ao-arc" },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, meson_mx_ao_arc_rproc_match);
+
+static struct platform_driver meson_mx_ao_arc_rproc_driver = {
+ .probe = meson_mx_ao_arc_rproc_probe,
+ .remove = meson_mx_ao_arc_rproc_remove,
+ .driver = {
+ .name = "meson-mx-ao-arc-rproc",
+ .of_match_table = meson_mx_ao_arc_rproc_match,
+ },
+};
+module_platform_driver(meson_mx_ao_arc_rproc_driver);
+
+MODULE_DESCRIPTION("Amlogic Meson6/8/8b/8m2 AO ARC remote processor driver");
+MODULE_AUTHOR("Martin Blumenstingl <martin.blumenstingl@googlemail.com>");
+MODULE_LICENSE("GPL v2");
struct mtk_scp;
struct mtk_scp_of_data {
+ int (*scp_clk_get)(struct mtk_scp *scp);
int (*scp_before_load)(struct mtk_scp *scp);
void (*scp_irq_handler)(struct mtk_scp *scp);
void (*scp_reset_assert)(struct mtk_scp *scp);
return -ENOENT;
}
+static int mt8183_scp_clk_get(struct mtk_scp *scp)
+{
+ struct device *dev = scp->dev;
+ int ret = 0;
+
+ scp->clk = devm_clk_get(dev, "main");
+ if (IS_ERR(scp->clk)) {
+ dev_err(dev, "Failed to get clock\n");
+ ret = PTR_ERR(scp->clk);
+ }
+
+ return ret;
+}
+
+static int mt8192_scp_clk_get(struct mtk_scp *scp)
+{
+ return mt8183_scp_clk_get(scp);
+}
+
+static int mt8195_scp_clk_get(struct mtk_scp *scp)
+{
+ scp->clk = NULL;
+
+ return 0;
+}
+
static int mt8183_scp_before_load(struct mtk_scp *scp)
{
/* Clear SCP to host interrupt */
if (ret)
goto destroy_mutex;
- scp->clk = devm_clk_get(dev, "main");
- if (IS_ERR(scp->clk)) {
- dev_err(dev, "Failed to get clock\n");
- ret = PTR_ERR(scp->clk);
+ ret = scp->data->scp_clk_get(scp);
+ if (ret)
goto release_dev_mem;
- }
/* register SCP initialization IPI */
ret = scp_ipi_register(scp, SCP_IPI_INIT, scp_init_ipi_handler, scp);
}
static const struct mtk_scp_of_data mt8183_of_data = {
+ .scp_clk_get = mt8183_scp_clk_get,
.scp_before_load = mt8183_scp_before_load,
.scp_irq_handler = mt8183_scp_irq_handler,
.scp_reset_assert = mt8183_scp_reset_assert,
};
static const struct mtk_scp_of_data mt8192_of_data = {
+ .scp_clk_get = mt8192_scp_clk_get,
+ .scp_before_load = mt8192_scp_before_load,
+ .scp_irq_handler = mt8192_scp_irq_handler,
+ .scp_reset_assert = mt8192_scp_reset_assert,
+ .scp_reset_deassert = mt8192_scp_reset_deassert,
+ .scp_stop = mt8192_scp_stop,
+ .scp_da_to_va = mt8192_scp_da_to_va,
+ .host_to_scp_reg = MT8192_GIPC_IN_SET,
+ .host_to_scp_int_bit = MT8192_HOST_IPC_INT_BIT,
+};
+
+static const struct mtk_scp_of_data mt8195_of_data = {
+ .scp_clk_get = mt8195_scp_clk_get,
.scp_before_load = mt8192_scp_before_load,
.scp_irq_handler = mt8192_scp_irq_handler,
.scp_reset_assert = mt8192_scp_reset_assert,
static const struct of_device_id mtk_scp_of_match[] = {
{ .compatible = "mediatek,mt8183-scp", .data = &mt8183_of_data },
{ .compatible = "mediatek,mt8192-scp", .data = &mt8192_of_data },
+ { .compatible = "mediatek,mt8195-scp", .data = &mt8195_of_data },
{},
};
MODULE_DEVICE_TABLE(of, mtk_scp_of_match);
static int __maybe_unused omap_rproc_suspend(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct rproc *rproc = platform_get_drvdata(pdev);
+ struct rproc *rproc = dev_get_drvdata(dev);
struct omap_rproc *oproc = rproc->priv;
int ret = 0;
static int __maybe_unused omap_rproc_resume(struct device *dev)
{
- struct platform_device *pdev = to_platform_device(dev);
- struct rproc *rproc = platform_get_drvdata(pdev);
+ struct rproc *rproc = dev_get_drvdata(dev);
struct omap_rproc *oproc = rproc->priv;
int ret = 0;
#include "qcom_common.h"
#include "qcom_q6v5.h"
+#define Q6V5_LOAD_STATE_MSG_LEN 64
#define Q6V5_PANIC_DELAY_MS 200
+static int q6v5_load_state_toggle(struct qcom_q6v5 *q6v5, bool enable)
+{
+ char buf[Q6V5_LOAD_STATE_MSG_LEN];
+ int ret;
+
+ if (!q6v5->qmp)
+ return 0;
+
+ ret = snprintf(buf, sizeof(buf),
+ "{class: image, res: load_state, name: %s, val: %s}",
+ q6v5->load_state, enable ? "on" : "off");
+
+ WARN_ON(ret >= Q6V5_LOAD_STATE_MSG_LEN);
+
+ ret = qmp_send(q6v5->qmp, buf, sizeof(buf));
+ if (ret)
+ dev_err(q6v5->dev, "failed to toggle load state\n");
+
+ return ret;
+}
+
/**
* qcom_q6v5_prepare() - reinitialize the qcom_q6v5 context before start
* @q6v5: reference to qcom_q6v5 context to be reinitialized
*/
int qcom_q6v5_prepare(struct qcom_q6v5 *q6v5)
{
+ int ret;
+
+ ret = q6v5_load_state_toggle(q6v5, true);
+ if (ret)
+ return ret;
+
reinit_completion(&q6v5->start_done);
reinit_completion(&q6v5->stop_done);
int qcom_q6v5_unprepare(struct qcom_q6v5 *q6v5)
{
disable_irq(q6v5->handover_irq);
+ q6v5_load_state_toggle(q6v5, false);
return !q6v5->handover_issued;
}
* @pdev: platform_device reference for acquiring resources
* @rproc: associated remoteproc instance
* @crash_reason: SMEM id for crash reason string, or 0 if none
+ * @load_state: load state resource string
* @handover: function to be called when proxy resources should be released
*
* Return: 0 on success, negative errno on failure
*/
int qcom_q6v5_init(struct qcom_q6v5 *q6v5, struct platform_device *pdev,
- struct rproc *rproc, int crash_reason,
+ struct rproc *rproc, int crash_reason, const char *load_state,
void (*handover)(struct qcom_q6v5 *q6v5))
{
int ret;
return PTR_ERR(q6v5->state);
}
+ q6v5->load_state = devm_kstrdup_const(&pdev->dev, load_state, GFP_KERNEL);
+ q6v5->qmp = qmp_get(&pdev->dev);
+ if (IS_ERR(q6v5->qmp)) {
+ if (PTR_ERR(q6v5->qmp) != -ENODEV)
+ return dev_err_probe(&pdev->dev, PTR_ERR(q6v5->qmp),
+ "failed to acquire load state\n");
+ q6v5->qmp = NULL;
+ } else if (!q6v5->load_state) {
+ if (!load_state)
+ dev_err(&pdev->dev, "load state resource string empty\n");
+
+ qmp_put(q6v5->qmp);
+ return load_state ? -ENOMEM : -EINVAL;
+ }
+
return 0;
}
EXPORT_SYMBOL_GPL(qcom_q6v5_init);
+/**
+ * qcom_q6v5_deinit() - deinitialize the q6v5 common struct
+ * @q6v5: reference to qcom_q6v5 context to be deinitialized
+ */
+void qcom_q6v5_deinit(struct qcom_q6v5 *q6v5)
+{
+ qmp_put(q6v5->qmp);
+}
+EXPORT_SYMBOL_GPL(qcom_q6v5_deinit);
+
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("Qualcomm Peripheral Image Loader for Q6V5");
#include <linux/kernel.h>
#include <linux/completion.h>
+#include <linux/soc/qcom/qcom_aoss.h>
struct rproc;
struct qcom_smem_state;
struct rproc *rproc;
struct qcom_smem_state *state;
+ struct qmp *qmp;
+
unsigned stop_bit;
int wdog_irq;
bool running;
+ const char *load_state;
void (*handover)(struct qcom_q6v5 *q6v5);
};
int qcom_q6v5_init(struct qcom_q6v5 *q6v5, struct platform_device *pdev,
- struct rproc *rproc, int crash_reason,
+ struct rproc *rproc, int crash_reason, const char *load_state,
void (*handover)(struct qcom_q6v5 *q6v5));
+void qcom_q6v5_deinit(struct qcom_q6v5 *q6v5);
int qcom_q6v5_prepare(struct qcom_q6v5 *q6v5);
int qcom_q6v5_unprepare(struct qcom_q6v5 *q6v5);
int ret;
unsigned int val;
- qcom_q6v5_prepare(&adsp->q6v5);
+ ret = qcom_q6v5_prepare(&adsp->q6v5);
+ if (ret)
+ return ret;
ret = clk_prepare_enable(adsp->xo);
if (ret)
if (ret)
goto disable_pm;
- ret = qcom_q6v5_init(&adsp->q6v5, pdev, rproc, desc->crash_reason_smem,
+ ret = qcom_q6v5_init(&adsp->q6v5, pdev, rproc, desc->crash_reason_smem, NULL,
qcom_adsp_pil_handover);
if (ret)
goto disable_pm;
rproc_del(adsp->rproc);
+ qcom_q6v5_deinit(&adsp->q6v5);
qcom_remove_glink_subdev(adsp->rproc, &adsp->glink_subdev);
qcom_remove_sysmon_subdev(adsp->sysmon);
qcom_remove_ssr_subdev(adsp->rproc, &adsp->ssr_subdev);
#define HALT_ACK_TIMEOUT_US 100000
+/* QACCEPT Register Offsets */
+#define QACCEPT_ACCEPT_REG 0x0
+#define QACCEPT_ACTIVE_REG 0x4
+#define QACCEPT_DENY_REG 0x8
+#define QACCEPT_REQ_REG 0xC
+
+#define QACCEPT_TIMEOUT_US 50
+
/* QDSP6SS_RESET */
#define Q6SS_STOP_CORE BIT(0)
#define Q6SS_CORE_ARES BIT(1)
char **proxy_clk_names;
char **reset_clk_names;
char **active_clk_names;
- char **active_pd_names;
char **proxy_pd_names;
int version;
bool need_mem_protection;
bool has_alt_reset;
bool has_mba_logs;
bool has_spare_reg;
+ bool has_qaccept_regs;
+ bool has_ext_cntl_regs;
+ bool has_vq6;
};
struct q6v5 {
u32 halt_q6;
u32 halt_modem;
u32 halt_nc;
+ u32 halt_vq6;
u32 conn_box;
+ u32 qaccept_mdm;
+ u32 qaccept_cx;
+ u32 qaccept_axi;
+
+ u32 axim1_clk_off;
+ u32 crypto_clk_off;
+ u32 force_clk_on;
+ u32 rscc_disable;
+
struct reset_control *mss_restart;
struct reset_control *pdc_reset;
struct clk *active_clks[8];
struct clk *reset_clks[4];
struct clk *proxy_clks[4];
- struct device *active_pds[1];
struct device *proxy_pds[3];
int active_clk_count;
int reset_clk_count;
int proxy_clk_count;
- int active_pd_count;
int proxy_pd_count;
struct reg_info active_regs[1];
bool has_alt_reset;
bool has_mba_logs;
bool has_spare_reg;
+ bool has_qaccept_regs;
+ bool has_ext_cntl_regs;
+ bool has_vq6;
int mpss_perm;
int mba_perm;
const char *hexagon_mdt_image;
MSS_MSM8996,
MSS_MSM8998,
MSS_SC7180,
+ MSS_SC7280,
MSS_SDM845,
};
regmap_update_bits(qproc->conn_map, qproc->conn_box,
AXI_GATING_VALID_OVERRIDE, 0);
ret = reset_control_deassert(qproc->mss_restart);
+ } else if (qproc->has_ext_cntl_regs) {
+ regmap_write(qproc->conn_map, qproc->rscc_disable, 0);
+ reset_control_assert(qproc->pdc_reset);
+ reset_control_assert(qproc->mss_restart);
+ reset_control_deassert(qproc->pdc_reset);
+ ret = reset_control_deassert(qproc->mss_restart);
} else {
ret = reset_control_assert(qproc->mss_restart);
}
ret = reset_control_reset(qproc->mss_restart);
writel(0, qproc->rmb_base + RMB_MBA_ALT_RESET);
reset_control_deassert(qproc->pdc_reset);
- } else if (qproc->has_spare_reg) {
+ } else if (qproc->has_spare_reg || qproc->has_ext_cntl_regs) {
ret = reset_control_reset(qproc->mss_restart);
} else {
ret = reset_control_deassert(qproc->mss_restart);
}
goto pbl_wait;
- } else if (qproc->version == MSS_SC7180) {
+ } else if (qproc->version == MSS_SC7180 || qproc->version == MSS_SC7280) {
val = readl(qproc->reg_base + QDSP6SS_SLEEP);
val |= Q6SS_CBCR_CLKEN;
writel(val, qproc->reg_base + QDSP6SS_SLEEP);
return ret;
}
+static int q6v5proc_enable_qchannel(struct q6v5 *qproc, struct regmap *map, u32 offset)
+{
+ unsigned int val;
+ int ret;
+
+ if (!qproc->has_qaccept_regs)
+ return 0;
+
+ if (qproc->has_ext_cntl_regs) {
+ regmap_write(qproc->conn_map, qproc->rscc_disable, 0);
+ regmap_write(qproc->conn_map, qproc->force_clk_on, 1);
+
+ ret = regmap_read_poll_timeout(qproc->halt_map, qproc->axim1_clk_off, val,
+ !val, 1, Q6SS_CBCR_TIMEOUT_US);
+ if (ret) {
+ dev_err(qproc->dev, "failed to enable axim1 clock\n");
+ return -ETIMEDOUT;
+ }
+ }
+
+ regmap_write(map, offset + QACCEPT_REQ_REG, 1);
+
+ /* Wait for accept */
+ ret = regmap_read_poll_timeout(map, offset + QACCEPT_ACCEPT_REG, val, val, 5,
+ QACCEPT_TIMEOUT_US);
+ if (ret) {
+ dev_err(qproc->dev, "qchannel enable failed\n");
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static void q6v5proc_disable_qchannel(struct q6v5 *qproc, struct regmap *map, u32 offset)
+{
+ int ret;
+ unsigned int val, retry;
+ unsigned int nretry = 10;
+ bool takedown_complete = false;
+
+ if (!qproc->has_qaccept_regs)
+ return;
+
+ while (!takedown_complete && nretry) {
+ nretry--;
+
+ /* Wait for active transactions to complete */
+ regmap_read_poll_timeout(map, offset + QACCEPT_ACTIVE_REG, val, !val, 5,
+ QACCEPT_TIMEOUT_US);
+
+ /* Request Q-channel transaction takedown */
+ regmap_write(map, offset + QACCEPT_REQ_REG, 0);
+
+ /*
+ * If the request is denied, reset the Q-channel takedown request,
+ * wait for active transactions to complete and retry takedown.
+ */
+ retry = 10;
+ while (retry) {
+ usleep_range(5, 10);
+ retry--;
+ ret = regmap_read(map, offset + QACCEPT_DENY_REG, &val);
+ if (!ret && val) {
+ regmap_write(map, offset + QACCEPT_REQ_REG, 1);
+ break;
+ }
+
+ ret = regmap_read(map, offset + QACCEPT_ACCEPT_REG, &val);
+ if (!ret && !val) {
+ takedown_complete = true;
+ break;
+ }
+ }
+
+ if (!retry)
+ break;
+ }
+
+ /* Rely on mss_restart to clear out pending transactions on takedown failure */
+ if (!takedown_complete)
+ dev_err(qproc->dev, "qchannel takedown failed\n");
+}
+
static void q6v5proc_halt_axi_port(struct q6v5 *qproc,
struct regmap *halt_map,
u32 offset)
int xfermemop_ret;
bool mba_load_err = false;
- qcom_q6v5_prepare(&qproc->q6v5);
-
- ret = q6v5_pds_enable(qproc, qproc->active_pds, qproc->active_pd_count);
- if (ret < 0) {
- dev_err(qproc->dev, "failed to enable active power domains\n");
- goto disable_irqs;
- }
+ ret = qcom_q6v5_prepare(&qproc->q6v5);
+ if (ret)
+ return ret;
ret = q6v5_pds_enable(qproc, qproc->proxy_pds, qproc->proxy_pd_count);
if (ret < 0) {
dev_err(qproc->dev, "failed to enable proxy power domains\n");
- goto disable_active_pds;
+ goto disable_irqs;
}
ret = q6v5_regulator_enable(qproc, qproc->fallback_proxy_regs,
goto assert_reset;
}
+ ret = q6v5proc_enable_qchannel(qproc, qproc->halt_map, qproc->qaccept_axi);
+ if (ret) {
+ dev_err(qproc->dev, "failed to enable axi bridge\n");
+ goto disable_active_clks;
+ }
+
/*
* Some versions of the MBA firmware will upon boot wipe the MPSS region as well, so provide
* the Q6 access to this region.
halt_axi_ports:
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_q6);
+ if (qproc->has_vq6)
+ q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_vq6);
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_modem);
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_nc);
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_mdm);
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_cx);
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_axi);
mba_load_err = true;
reclaim_mba:
xfermemop_ret = q6v5_xfer_mem_ownership(qproc, &qproc->mba_perm, true,
qproc->fallback_proxy_reg_count);
disable_proxy_pds:
q6v5_pds_disable(qproc, qproc->proxy_pds, qproc->proxy_pd_count);
-disable_active_pds:
- q6v5_pds_disable(qproc, qproc->active_pds, qproc->active_pd_count);
disable_irqs:
qcom_q6v5_unprepare(&qproc->q6v5);
qproc->dp_size = 0;
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_q6);
+ if (qproc->has_vq6)
+ q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_vq6);
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_modem);
q6v5proc_halt_axi_port(qproc, qproc->halt_map, qproc->halt_nc);
if (qproc->version == MSS_MSM8996) {
writel(val, qproc->reg_base + QDSP6SS_PWR_CTL_REG);
}
+ if (qproc->has_ext_cntl_regs) {
+ regmap_write(qproc->conn_map, qproc->rscc_disable, 1);
+
+ ret = regmap_read_poll_timeout(qproc->halt_map, qproc->axim1_clk_off, val,
+ !val, 1, Q6SS_CBCR_TIMEOUT_US);
+ if (ret)
+ dev_err(qproc->dev, "failed to enable axim1 clock\n");
+
+ ret = regmap_read_poll_timeout(qproc->halt_map, qproc->crypto_clk_off, val,
+ !val, 1, Q6SS_CBCR_TIMEOUT_US);
+ if (ret)
+ dev_err(qproc->dev, "failed to enable crypto clock\n");
+ }
+
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_mdm);
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_cx);
+ q6v5proc_disable_qchannel(qproc, qproc->halt_map, qproc->qaccept_axi);
+
q6v5_reset_assert(qproc);
q6v5_clk_disable(qproc->dev, qproc->reset_clks,
qproc->active_clk_count);
q6v5_regulator_disable(qproc, qproc->active_regs,
qproc->active_reg_count);
- q6v5_pds_disable(qproc, qproc->active_pds, qproc->active_pd_count);
/* In case of failure or coredump scenario where reclaiming MBA memory
* could not happen reclaim it here.
static int q6v5_init_mem(struct q6v5 *qproc, struct platform_device *pdev)
{
struct of_phandle_args args;
- struct resource *res;
+ int halt_cell_cnt = 3;
int ret;
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "qdsp6");
- qproc->reg_base = devm_ioremap_resource(&pdev->dev, res);
+ qproc->reg_base = devm_platform_ioremap_resource_byname(pdev, "qdsp6");
if (IS_ERR(qproc->reg_base))
return PTR_ERR(qproc->reg_base);
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "rmb");
- qproc->rmb_base = devm_ioremap_resource(&pdev->dev, res);
+ qproc->rmb_base = devm_platform_ioremap_resource_byname(pdev, "rmb");
if (IS_ERR(qproc->rmb_base))
return PTR_ERR(qproc->rmb_base);
+ if (qproc->has_vq6)
+ halt_cell_cnt++;
+
ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
- "qcom,halt-regs", 3, 0, &args);
+ "qcom,halt-regs", halt_cell_cnt, 0, &args);
if (ret < 0) {
dev_err(&pdev->dev, "failed to parse qcom,halt-regs\n");
return -EINVAL;
qproc->halt_modem = args.args[1];
qproc->halt_nc = args.args[2];
+ if (qproc->has_vq6)
+ qproc->halt_vq6 = args.args[3];
+
+ if (qproc->has_qaccept_regs) {
+ ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
+ "qcom,qaccept-regs",
+ 3, 0, &args);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "failed to parse qaccept-regs\n");
+ return -EINVAL;
+ }
+
+ qproc->qaccept_mdm = args.args[0];
+ qproc->qaccept_cx = args.args[1];
+ qproc->qaccept_axi = args.args[2];
+ }
+
+ if (qproc->has_ext_cntl_regs) {
+ ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
+ "qcom,ext-regs",
+ 2, 0, &args);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "failed to parse ext-regs index 0\n");
+ return -EINVAL;
+ }
+
+ qproc->conn_map = syscon_node_to_regmap(args.np);
+ of_node_put(args.np);
+ if (IS_ERR(qproc->conn_map))
+ return PTR_ERR(qproc->conn_map);
+
+ qproc->force_clk_on = args.args[0];
+ qproc->rscc_disable = args.args[1];
+
+ ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
+ "qcom,ext-regs",
+ 2, 1, &args);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "failed to parse ext-regs index 1\n");
+ return -EINVAL;
+ }
+
+ qproc->axim1_clk_off = args.args[0];
+ qproc->crypto_clk_off = args.args[1];
+ }
+
if (qproc->has_spare_reg) {
ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
"qcom,spare-regs",
return PTR_ERR(qproc->mss_restart);
}
- if (qproc->has_alt_reset || qproc->has_spare_reg) {
+ if (qproc->has_alt_reset || qproc->has_spare_reg || qproc->has_ext_cntl_regs) {
qproc->pdc_reset = devm_reset_control_get_exclusive(qproc->dev,
"pdc_reset");
if (IS_ERR(qproc->pdc_reset)) {
platform_set_drvdata(pdev, qproc);
+ qproc->has_qaccept_regs = desc->has_qaccept_regs;
+ qproc->has_ext_cntl_regs = desc->has_ext_cntl_regs;
+ qproc->has_vq6 = desc->has_vq6;
qproc->has_spare_reg = desc->has_spare_reg;
ret = q6v5_init_mem(qproc, pdev);
if (ret)
}
qproc->active_reg_count = ret;
- ret = q6v5_pds_attach(&pdev->dev, qproc->active_pds,
- desc->active_pd_names);
- if (ret < 0) {
- dev_err(&pdev->dev, "Failed to attach active power domains\n");
- goto free_rproc;
- }
- qproc->active_pd_count = ret;
-
ret = q6v5_pds_attach(&pdev->dev, qproc->proxy_pds,
desc->proxy_pd_names);
/* Fallback to regulators for old device trees */
desc->fallback_proxy_supply);
if (ret < 0) {
dev_err(&pdev->dev, "Failed to get fallback proxy regulators.\n");
- goto detach_active_pds;
+ goto free_rproc;
}
qproc->fallback_proxy_reg_count = ret;
} else if (ret < 0) {
dev_err(&pdev->dev, "Failed to init power domains\n");
- goto detach_active_pds;
+ goto free_rproc;
} else {
qproc->proxy_pd_count = ret;
}
qproc->need_mem_protection = desc->need_mem_protection;
qproc->has_mba_logs = desc->has_mba_logs;
- ret = qcom_q6v5_init(&qproc->q6v5, pdev, rproc, MPSS_CRASH_REASON_SMEM,
+ ret = qcom_q6v5_init(&qproc->q6v5, pdev, rproc, MPSS_CRASH_REASON_SMEM, "modem",
qcom_msa_handover);
if (ret)
goto detach_proxy_pds;
qcom_remove_glink_subdev(rproc, &qproc->glink_subdev);
detach_proxy_pds:
q6v5_pds_detach(qproc, qproc->proxy_pds, qproc->proxy_pd_count);
-detach_active_pds:
- q6v5_pds_detach(qproc, qproc->active_pds, qproc->active_pd_count);
free_rproc:
rproc_free(rproc);
rproc_del(rproc);
+ qcom_q6v5_deinit(&qproc->q6v5);
qcom_remove_sysmon_subdev(qproc->sysmon);
qcom_remove_ssr_subdev(rproc, &qproc->ssr_subdev);
qcom_remove_smd_subdev(rproc, &qproc->smd_subdev);
qcom_remove_glink_subdev(rproc, &qproc->glink_subdev);
q6v5_pds_detach(qproc, qproc->proxy_pds, qproc->proxy_pd_count);
- q6v5_pds_detach(qproc, qproc->active_pds, qproc->active_pd_count);
rproc_free(rproc);
"nav",
NULL
},
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
"mx",
.has_alt_reset = false,
.has_mba_logs = true,
.has_spare_reg = true,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_SC7180,
};
+static const struct rproc_hexagon_res sc7280_mss = {
+ .hexagon_mba_image = "mba.mbn",
+ .proxy_clk_names = (char*[]){
+ "xo",
+ "pka",
+ NULL
+ },
+ .active_clk_names = (char*[]){
+ "iface",
+ "offline",
+ "snoc_axi",
+ NULL
+ },
+ .proxy_pd_names = (char*[]){
+ "cx",
+ "mss",
+ NULL
+ },
+ .need_mem_protection = true,
+ .has_alt_reset = false,
+ .has_mba_logs = true,
+ .has_spare_reg = false,
+ .has_qaccept_regs = true,
+ .has_ext_cntl_regs = true,
+ .has_vq6 = true,
+ .version = MSS_SC7280,
+};
+
static const struct rproc_hexagon_res sdm845_mss = {
.hexagon_mba_image = "mba.mbn",
.proxy_clk_names = (char*[]){
"mnoc_axi",
NULL
},
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
"mx",
.has_alt_reset = true,
.has_mba_logs = false,
.has_spare_reg = false,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_SDM845,
};
.has_alt_reset = false,
.has_mba_logs = false,
.has_spare_reg = false,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_MSM8998,
};
.has_alt_reset = false,
.has_mba_logs = false,
.has_spare_reg = false,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_MSM8996,
};
.has_alt_reset = false,
.has_mba_logs = false,
.has_spare_reg = false,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_MSM8916,
};
.has_alt_reset = false,
.has_mba_logs = false,
.has_spare_reg = false,
+ .has_qaccept_regs = false,
+ .has_ext_cntl_regs = false,
+ .has_vq6 = false,
.version = MSS_MSM8974,
};
{ .compatible = "qcom,msm8996-mss-pil", .data = &msm8996_mss},
{ .compatible = "qcom,msm8998-mss-pil", .data = &msm8998_mss},
{ .compatible = "qcom,sc7180-mss-pil", .data = &sc7180_mss},
+ { .compatible = "qcom,sc7280-mss-pil", .data = &sc7280_mss},
{ .compatible = "qcom,sdm845-mss-pil", .data = &sdm845_mss},
{ },
};
bool has_aggre2_clk;
bool auto_boot;
- char **active_pd_names;
char **proxy_pd_names;
+ const char *load_state;
const char *ssr_name;
const char *sysmon_name;
int ssctl_id;
struct regulator *cx_supply;
struct regulator *px_supply;
- struct device *active_pds[1];
struct device *proxy_pds[3];
- int active_pd_count;
int proxy_pd_count;
int pas_id;
struct qcom_adsp *adsp = (struct qcom_adsp *)rproc->priv;
int ret;
- qcom_q6v5_prepare(&adsp->q6v5);
-
- ret = adsp_pds_enable(adsp, adsp->active_pds, adsp->active_pd_count);
- if (ret < 0)
- goto disable_irqs;
+ ret = qcom_q6v5_prepare(&adsp->q6v5);
+ if (ret)
+ return ret;
ret = adsp_pds_enable(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
if (ret < 0)
- goto disable_active_pds;
+ goto disable_irqs;
ret = clk_prepare_enable(adsp->xo);
if (ret)
clk_disable_unprepare(adsp->xo);
disable_proxy_pds:
adsp_pds_disable(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
-disable_active_pds:
- adsp_pds_disable(adsp, adsp->active_pds, adsp->active_pd_count);
disable_irqs:
qcom_q6v5_unprepare(&adsp->q6v5);
if (ret)
dev_err(adsp->dev, "failed to shutdown: %d\n", ret);
- adsp_pds_disable(adsp, adsp->active_pds, adsp->active_pd_count);
handover = qcom_q6v5_unprepare(&adsp->q6v5);
if (handover)
qcom_pas_handover(&adsp->q6v5);
if (ret)
goto free_rproc;
- ret = adsp_pds_attach(&pdev->dev, adsp->active_pds,
- desc->active_pd_names);
- if (ret < 0)
- goto free_rproc;
- adsp->active_pd_count = ret;
-
ret = adsp_pds_attach(&pdev->dev, adsp->proxy_pds,
desc->proxy_pd_names);
if (ret < 0)
- goto detach_active_pds;
+ goto free_rproc;
adsp->proxy_pd_count = ret;
- ret = qcom_q6v5_init(&adsp->q6v5, pdev, rproc, desc->crash_reason_smem,
+ ret = qcom_q6v5_init(&adsp->q6v5, pdev, rproc, desc->crash_reason_smem, desc->load_state,
qcom_pas_handover);
if (ret)
goto detach_proxy_pds;
detach_proxy_pds:
adsp_pds_detach(adsp, adsp->proxy_pds, adsp->proxy_pd_count);
-detach_active_pds:
- adsp_pds_detach(adsp, adsp->active_pds, adsp->active_pd_count);
free_rproc:
rproc_free(rproc);
rproc_del(adsp->rproc);
+ qcom_q6v5_deinit(&adsp->q6v5);
qcom_remove_glink_subdev(adsp->rproc, &adsp->glink_subdev);
qcom_remove_sysmon_subdev(adsp->sysmon);
qcom_remove_smd_subdev(adsp->rproc, &adsp->smd_subdev);
.ssctl_id = 0x14,
};
+static const struct adsp_data sdm845_adsp_resource_init = {
+ .crash_reason_smem = 423,
+ .firmware_name = "adsp.mdt",
+ .pas_id = 1,
+ .has_aggre2_clk = false,
+ .auto_boot = true,
+ .load_state = "adsp",
+ .ssr_name = "lpass",
+ .sysmon_name = "adsp",
+ .ssctl_id = 0x14,
+};
+
static const struct adsp_data sm8150_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
NULL
},
+ .load_state = "adsp",
.ssr_name = "lpass",
.sysmon_name = "adsp",
.ssctl_id = 0x14,
.pas_id = 1,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"lcx",
"lmx",
NULL
},
+ .load_state = "adsp",
.ssr_name = "lpass",
.sysmon_name = "adsp",
.ssctl_id = 0x14,
.pas_id = 1,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"lcx",
"lmx",
NULL
},
+ .load_state = "adsp",
.ssr_name = "lpass",
.sysmon_name = "adsp",
.ssctl_id = 0x14,
};
-static const struct adsp_data msm8998_adsp_resource = {
+static const struct adsp_data msm8996_adsp_resource = {
.crash_reason_smem = 423,
.firmware_name = "adsp.mdt",
.pas_id = 1,
.ssctl_id = 0x17,
};
+static const struct adsp_data sdm845_cdsp_resource_init = {
+ .crash_reason_smem = 601,
+ .firmware_name = "cdsp.mdt",
+ .pas_id = 18,
+ .has_aggre2_clk = false,
+ .auto_boot = true,
+ .load_state = "cdsp",
+ .ssr_name = "cdsp",
+ .sysmon_name = "cdsp",
+ .ssctl_id = 0x17,
+};
+
static const struct adsp_data sm8150_cdsp_resource = {
.crash_reason_smem = 601,
.firmware_name = "cdsp.mdt",
.pas_id = 18,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
NULL
},
+ .load_state = "cdsp",
.ssr_name = "cdsp",
.sysmon_name = "cdsp",
.ssctl_id = 0x17,
.pas_id = 18,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
NULL
},
+ .load_state = "cdsp",
.ssr_name = "cdsp",
.sysmon_name = "cdsp",
.ssctl_id = 0x17,
.pas_id = 18,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
NULL
},
+ .load_state = "cdsp",
.ssr_name = "cdsp",
.sysmon_name = "cdsp",
.ssctl_id = 0x17,
.minidump_id = 3,
.has_aggre2_clk = false,
.auto_boot = false,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
"mss",
NULL
},
+ .load_state = "modem",
.ssr_name = "mpss",
.sysmon_name = "modem",
.ssctl_id = 0x12,
.pas_id = 4,
.has_aggre2_clk = false,
.auto_boot = false,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"cx",
NULL
},
+ .load_state = "modem",
.ssr_name = "mpss",
.sysmon_name = "modem",
.ssctl_id = 0x12,
.pas_id = 12,
.has_aggre2_clk = true,
.auto_boot = true,
+ .proxy_pd_names = (char*[]){
+ "ssc_cx",
+ NULL
+ },
.ssr_name = "dsps",
.sysmon_name = "slpi",
.ssctl_id = 0x16,
.pas_id = 12,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"lcx",
"lmx",
NULL
},
+ .load_state = "slpi",
.ssr_name = "dsps",
.sysmon_name = "slpi",
.ssctl_id = 0x16,
.pas_id = 12,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"lcx",
"lmx",
NULL
},
+ .load_state = "slpi",
.ssr_name = "dsps",
.sysmon_name = "slpi",
.ssctl_id = 0x16,
.pas_id = 12,
.has_aggre2_clk = false,
.auto_boot = true,
- .active_pd_names = (char*[]){
- "load_state",
- NULL
- },
.proxy_pd_names = (char*[]){
"lcx",
"lmx",
NULL
},
+ .load_state = "slpi",
.ssr_name = "dsps",
.sysmon_name = "slpi",
.ssctl_id = 0x16,
};
-static const struct adsp_data msm8998_slpi_resource = {
- .crash_reason_smem = 424,
- .firmware_name = "slpi.mdt",
- .pas_id = 12,
- .has_aggre2_clk = true,
- .auto_boot = true,
- .proxy_pd_names = (char*[]){
- "ssc_cx",
- NULL
- },
- .ssr_name = "dsps",
- .sysmon_name = "slpi",
- .ssctl_id = 0x16,
-};
-
static const struct adsp_data wcss_resource_init = {
.crash_reason_smem = 421,
.firmware_name = "wcnss.mdt",
static const struct of_device_id adsp_of_match[] = {
{ .compatible = "qcom,msm8974-adsp-pil", .data = &adsp_resource_init},
- { .compatible = "qcom,msm8996-adsp-pil", .data = &adsp_resource_init},
+ { .compatible = "qcom,msm8996-adsp-pil", .data = &msm8996_adsp_resource},
{ .compatible = "qcom,msm8996-slpi-pil", .data = &slpi_resource_init},
- { .compatible = "qcom,msm8998-adsp-pas", .data = &msm8998_adsp_resource},
- { .compatible = "qcom,msm8998-slpi-pas", .data = &msm8998_slpi_resource},
+ { .compatible = "qcom,msm8998-adsp-pas", .data = &msm8996_adsp_resource},
+ { .compatible = "qcom,msm8998-slpi-pas", .data = &slpi_resource_init},
{ .compatible = "qcom,qcs404-adsp-pas", .data = &adsp_resource_init },
{ .compatible = "qcom,qcs404-cdsp-pas", .data = &cdsp_resource_init },
{ .compatible = "qcom,qcs404-wcss-pas", .data = &wcss_resource_init },
{ .compatible = "qcom,sc7180-mpss-pas", .data = &mpss_resource_init},
+ { .compatible = "qcom,sc7280-mpss-pas", .data = &mpss_resource_init},
{ .compatible = "qcom,sc8180x-adsp-pas", .data = &sm8150_adsp_resource},
{ .compatible = "qcom,sc8180x-cdsp-pas", .data = &sm8150_cdsp_resource},
{ .compatible = "qcom,sc8180x-mpss-pas", .data = &sc8180x_mpss_resource},
{ .compatible = "qcom,sdm660-adsp-pas", .data = &adsp_resource_init},
- { .compatible = "qcom,sdm845-adsp-pas", .data = &adsp_resource_init},
- { .compatible = "qcom,sdm845-cdsp-pas", .data = &cdsp_resource_init},
+ { .compatible = "qcom,sdm845-adsp-pas", .data = &sdm845_adsp_resource_init},
+ { .compatible = "qcom,sdm845-cdsp-pas", .data = &sdm845_cdsp_resource_init},
{ .compatible = "qcom,sdx55-mpss-pas", .data = &sdx55_mpss_resource},
{ .compatible = "qcom,sm8150-adsp-pas", .data = &sm8150_adsp_resource},
{ .compatible = "qcom,sm8150-cdsp-pas", .data = &sm8150_cdsp_resource},
if (ret)
goto free_rproc;
- ret = qcom_q6v5_init(&wcss->q6v5, pdev, rproc, desc->crash_reason_smem,
- NULL);
+ ret = qcom_q6v5_init(&wcss->q6v5, pdev, rproc, desc->crash_reason_smem, NULL, NULL);
if (ret)
goto free_rproc;
static int q6v5_wcss_remove(struct platform_device *pdev)
{
struct rproc *rproc = platform_get_drvdata(pdev);
+ struct q6v5_wcss *wcss = rproc->priv;
+ qcom_q6v5_deinit(&wcss->q6v5);
rproc_del(rproc);
rproc_free(rproc);
#include <linux/soc/qcom/mdt_loader.h>
#include <linux/soc/qcom/smem.h>
#include <linux/soc/qcom/smem_state.h>
-#include <linux/rpmsg/qcom_smd.h>
#include "qcom_common.h"
#include "remoteproc_internal.h"
/* Initialise vdev subdevice */
snprintf(name, sizeof(name), "vdev%dbuffer", rvdev->index);
rvdev->dev.parent = &rproc->dev;
- ret = copy_dma_range_map(&rvdev->dev, rproc->dev.parent);
- if (ret)
- return ret;
rvdev->dev.release = rproc_rvdev_release;
dev_set_name(&rvdev->dev, "%s#%s", dev_name(rvdev->dev.parent), name);
dev_set_drvdata(&rvdev->dev, rvdev);
put_device(&rvdev->dev);
return ret;
}
+
+ ret = copy_dma_range_map(&rvdev->dev, rproc->dev.parent);
+ if (ret)
+ goto free_rvdev;
+
/* Make device dma capable by inheriting from parent's capabilities */
set_dma_ops(&rvdev->dev, get_dma_ops(rproc->dev.parent));
struct rproc_dump_segment *segment,
size_t offset, size_t size)
{
+ bool is_iomem = false;
void *ptr;
- bool is_iomem;
if (segment->dump) {
segment->dump(rproc, segment, dest, offset, size);
u64 filesz = elf_phdr_get_p_filesz(class, phdr);
u64 offset = elf_phdr_get_p_offset(class, phdr);
u32 type = elf_phdr_get_p_type(class, phdr);
+ bool is_iomem = false;
void *ptr;
- bool is_iomem;
if (type != PT_LOAD)
continue;
/* put the segment where the remote processor expects it */
if (filesz) {
if (is_iomem)
- memcpy_fromio(ptr, (void __iomem *)(elf_data + offset), filesz);
+ memcpy_toio((void __iomem *)ptr, elf_data + offset, filesz);
else
memcpy(ptr, elf_data + offset, filesz);
}
#include "remoteproc_internal.h"
+static struct rproc_vdev *vdev_to_rvdev(struct virtio_device *vdev)
+{
+ return container_of(vdev->dev.parent, struct rproc_vdev, dev);
+}
+
+static struct rproc *vdev_to_rproc(struct virtio_device *vdev)
+{
+ struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
+
+ return rvdev->rproc;
+}
+
/* kick the remote processor, and let it know which virtqueue to poke at */
static bool rproc_virtio_notify(struct virtqueue *vq)
{
return -EINVAL;
}
if (num_rmems < 2) {
- dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+ dev_err(dev, "device needs at least two memory regions to be defined, num = %d\n",
num_rmems);
return -EINVAL;
}
return -EINVAL;
}
if (num_rmems < 2) {
- dev_err(dev, "device needs atleast two memory regions to be defined, num = %d\n",
+ dev_err(dev, "device needs at least two memory regions to be defined, num = %d\n",
num_rmems);
return -EINVAL;
}
int ret;
for_each_available_child_of_node(node, child) {
- ret = of_property_read_string(child, "mtk,rpmsg-name", &name);
+ ret = of_property_read_string(child, "mediatek,rpmsg-name", &name);
if (ret)
continue;
* @rcids: idr of all channels with a known remote channel id
* @features: remote features
* @intentless: flag to indicate that there is no intent
+ * @tx_avail_notify: Waitqueue for pending tx tasks
+ * @sent_read_notify: flag to check cmd sent or not
*/
struct qcom_glink {
struct device *dev;
unsigned long features;
bool intentless;
+ wait_queue_head_t tx_avail_notify;
+ bool sent_read_notify;
};
enum {
glink->tx_pipe->write(glink->tx_pipe, hdr, hlen, data, dlen);
}
+static void qcom_glink_send_read_notify(struct qcom_glink *glink)
+{
+ struct glink_msg msg;
+
+ msg.cmd = cpu_to_le16(RPM_CMD_READ_NOTIF);
+ msg.param1 = 0;
+ msg.param2 = 0;
+
+ qcom_glink_tx_write(glink, &msg, sizeof(msg), NULL, 0);
+
+ mbox_send_message(glink->mbox_chan, NULL);
+ mbox_client_txdone(glink->mbox_chan, 0);
+}
+
static int qcom_glink_tx(struct qcom_glink *glink,
const void *hdr, size_t hlen,
const void *data, size_t dlen, bool wait)
goto out;
}
+ if (!glink->sent_read_notify) {
+ glink->sent_read_notify = true;
+ qcom_glink_send_read_notify(glink);
+ }
+
/* Wait without holding the tx_lock */
spin_unlock_irqrestore(&glink->tx_lock, flags);
- usleep_range(10000, 15000);
+ wait_event_timeout(glink->tx_avail_notify,
+ qcom_glink_tx_avail(glink) >= tlen, 10 * HZ);
spin_lock_irqsave(&glink->tx_lock, flags);
+
+ if (qcom_glink_tx_avail(glink) >= tlen)
+ glink->sent_read_notify = false;
}
qcom_glink_tx_write(glink, hdr, hlen, data, dlen);
unsigned int cmd;
int ret = 0;
+ /* To wakeup any blocking writers */
+ wake_up_all(&glink->tx_avail_notify);
+
for (;;) {
avail = qcom_glink_rx_avail(glink);
if (avail < sizeof(msg))
} __packed req;
int ret;
unsigned long flags;
+ int chunk_size = len;
+ int left_size = 0;
if (!glink->intentless) {
while (!intent) {
iid = intent->id;
}
+ if (wait && chunk_size > SZ_8K) {
+ chunk_size = SZ_8K;
+ left_size = len - chunk_size;
+ }
req.msg.cmd = cpu_to_le16(RPM_CMD_TX_DATA);
req.msg.param1 = cpu_to_le16(channel->lcid);
req.msg.param2 = cpu_to_le32(iid);
- req.chunk_size = cpu_to_le32(len);
- req.left_size = cpu_to_le32(0);
+ req.chunk_size = cpu_to_le32(chunk_size);
+ req.left_size = cpu_to_le32(left_size);
- ret = qcom_glink_tx(glink, &req, sizeof(req), data, len, wait);
+ ret = qcom_glink_tx(glink, &req, sizeof(req), data, chunk_size, wait);
/* Mark intent available if we failed */
- if (ret && intent)
+ if (ret && intent) {
intent->in_use = false;
+ return ret;
+ }
+ while (left_size > 0) {
+ data = (void *)((char *)data + chunk_size);
+ chunk_size = left_size;
+ if (chunk_size > SZ_8K)
+ chunk_size = SZ_8K;
+ left_size -= chunk_size;
+
+ req.msg.cmd = cpu_to_le16(RPM_CMD_TX_DATA_CONT);
+ req.msg.param1 = cpu_to_le16(channel->lcid);
+ req.msg.param2 = cpu_to_le32(iid);
+ req.chunk_size = cpu_to_le32(chunk_size);
+ req.left_size = cpu_to_le32(left_size);
+
+ ret = qcom_glink_tx(glink, &req, sizeof(req), data,
+ chunk_size, wait);
+
+ /* Mark intent available if we failed */
+ if (ret && intent) {
+ intent->in_use = false;
+ break;
+ }
+ }
return ret;
}
static void qcom_glink_rpdev_release(struct device *dev)
{
struct rpmsg_device *rpdev = to_rpmsg_device(dev);
- struct glink_channel *channel = to_glink_channel(rpdev->ept);
- channel->rpdev = NULL;
kfree(rpdev);
}
}
rpdev->ept = &channel->ept;
- strncpy(rpdev->id.name, name, RPMSG_NAME_SIZE);
+ strscpy_pad(rpdev->id.name, name, RPMSG_NAME_SIZE);
rpdev->src = RPMSG_ADDR_ANY;
rpdev->dst = RPMSG_ADDR_ANY;
rpdev->ops = &glink_device_ops;
rpmsg_unregister_device(glink->dev, &chinfo);
}
+ channel->rpdev = NULL;
qcom_glink_send_close_ack(glink, channel->rcid);
static void qcom_glink_rx_close_ack(struct qcom_glink *glink, unsigned int lcid)
{
+ struct rpmsg_channel_info chinfo;
struct glink_channel *channel;
unsigned long flags;
+ /* To wakeup any blocking writers */
+ wake_up_all(&glink->tx_avail_notify);
+
spin_lock_irqsave(&glink->idr_lock, flags);
channel = idr_find(&glink->lcids, lcid);
if (WARN(!channel, "close ack on unknown channel\n")) {
channel->lcid = 0;
spin_unlock_irqrestore(&glink->idr_lock, flags);
+ /* Decouple the potential rpdev from the channel */
+ if (channel->rpdev) {
+ strscpy(chinfo.name, channel->name, sizeof(chinfo.name));
+ chinfo.src = RPMSG_ADDR_ANY;
+ chinfo.dst = RPMSG_ADDR_ANY;
+
+ rpmsg_unregister_device(glink->dev, &chinfo);
+ }
+ channel->rpdev = NULL;
+
kref_put(&channel->refcount, qcom_glink_channel_release);
}
spin_lock_init(&glink->rx_lock);
INIT_LIST_HEAD(&glink->rx_queue);
INIT_WORK(&glink->rx_work, qcom_glink_work);
+ init_waitqueue_head(&glink->tx_avail_notify);
spin_lock_init(&glink->idr_lock);
idr_init(&glink->lcids);
#include <linux/uaccess.h>
#include <uapi/linux/rpmsg.h>
-#include "rpmsg_internal.h"
-
#define RPMSG_DEV_MAX (MINORMASK + 1)
static dev_t rpmsg_major;
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/mutex.h>
-#include <linux/of_device.h>
#include <linux/rpmsg.h>
#include <linux/rpmsg/byteorder.h>
#include <linux/rpmsg/ns.h>
/* farewell, ept, we don't need you anymore */
kref_put(&ept->refcount, __ept_release);
} else
- dev_warn(dev, "msg received with no recipient\n");
+ dev_warn_ratelimited(dev, "msg received with no recipient\n");
/* publish the real size of the buffer */
rpmsg_sg_init(&sg, msg, vrp->buf_size);
config RTC_DRV_PCF8523
tristate "NXP PCF8523"
+ select REGMAP_I2C
help
If you say yes here you get support for the NXP PCF8523 RTC
chips.
This driver can also be built as a module. If so, the module
will be called rtc-tps65910.
-config RTC_DRV_TPS80031
- tristate "TI TPS80031/TPS80032 RTC driver"
- depends on MFD_TPS80031
- help
- TI Power Management IC TPS80031 supports RTC functionality
- along with alarm. This driver supports the RTC driver for
- the TPS80031 RTC module.
-
config RTC_DRV_RC5T583
tristate "RICOH 5T583 RTC driver"
depends on MFD_RC5T583
This can also be built as a module. If so, the module will
be named "rtc_wilco_ec".
+config RTC_DRV_MSC313
+ tristate "MStar MSC313 RTC"
+ depends on ARCH_MSTARV7 || COMPILE_TEST
+ help
+ If you say yes here you get support for the Mstar MSC313e On-Chip
+ Real Time Clock.
+
+ This driver can also be built as a module, if so, the module
+ will be called "rtc-msc313".
+
endif # RTC_CLASS
obj-$(CONFIG_RTC_DRV_MESON) += rtc-meson.o
obj-$(CONFIG_RTC_DRV_MOXART) += rtc-moxart.o
obj-$(CONFIG_RTC_DRV_MPC5121) += rtc-mpc5121.o
+obj-$(CONFIG_RTC_DRV_MSC313) += rtc-msc313.o
obj-$(CONFIG_RTC_DRV_MSM6242) += rtc-msm6242.o
obj-$(CONFIG_RTC_DRV_MT2712) += rtc-mt2712.o
obj-$(CONFIG_RTC_DRV_MT6397) += rtc-mt6397.o
obj-$(CONFIG_RTC_DRV_TEST) += rtc-test.o
obj-$(CONFIG_RTC_DRV_TPS6586X) += rtc-tps6586x.o
obj-$(CONFIG_RTC_DRV_TPS65910) += rtc-tps65910.o
-obj-$(CONFIG_RTC_DRV_TPS80031) += rtc-tps80031.o
obj-$(CONFIG_RTC_DRV_TWL4030) += rtc-twl.o
obj-$(CONFIG_RTC_DRV_V3020) += rtc-v3020.o
obj-$(CONFIG_RTC_DRV_VR41XX) += rtc-vr41xx.o
rtc->pie_enabled = 0;
set_bit(RTC_FEATURE_ALARM, rtc->features);
+ set_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->features);
return rtc;
}
* letting any rtc_class_open() users access it again
*/
rtc_proc_del_device(rtc);
- cdev_device_del(&rtc->char_dev, &rtc->dev);
+ if (!test_bit(RTC_NO_CDEV, &rtc->flags))
+ cdev_device_del(&rtc->char_dev, &rtc->dev);
rtc->ops = NULL;
mutex_unlock(&rtc->ops_lock);
}
rtc->id = id;
rtc->dev.parent = dev;
- dev_set_name(&rtc->dev, "rtc%d", id);
+ err = dev_set_name(&rtc->dev, "rtc%d", id);
+ if (err)
+ return ERR_PTR(err);
err = devm_add_action_or_reset(dev, devm_rtc_release_device, rtc);
if (err)
if (!rtc->ops->set_alarm)
clear_bit(RTC_FEATURE_ALARM, rtc->features);
+ if (rtc->uie_unsupported)
+ clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->features);
+
+ if (rtc->ops->set_offset)
+ set_bit(RTC_FEATURE_CORRECTION, rtc->features);
+
rtc->owner = owner;
rtc_device_get_offset(rtc);
rtc_dev_prepare(rtc);
err = cdev_device_add(&rtc->char_dev, &rtc->dev);
- if (err)
+ if (err) {
+ set_bit(RTC_NO_CDEV, &rtc->flags);
dev_warn(rtc->dev.parent, "failed to add char device %d:%d\n",
MAJOR(rtc->dev.devt), rtc->id);
- else
+ } else {
dev_dbg(rtc->dev.parent, "char device (%d:%d)\n",
MAJOR(rtc->dev.devt), rtc->id);
+ }
rtc_proc_add_device(rtc);
const struct rtc_class_ops *ops = rtc->ops;
struct rtc_time tm;
struct rtc_wkalrm alarm;
+ struct rtc_param param;
void __user *uarg = (void __user *)arg;
err = mutex_lock_interruptible(&rtc->ops_lock);
switch (cmd) {
case RTC_EPOCH_SET:
case RTC_SET_TIME:
+ case RTC_PARAM_SET:
if (!capable(CAP_SYS_TIME))
err = -EACCES;
break;
err = -EFAULT;
return err;
+ case RTC_PARAM_GET:
+ if (copy_from_user(¶m, uarg, sizeof(param))) {
+ mutex_unlock(&rtc->ops_lock);
+ return -EFAULT;
+ }
+
+ switch(param.param) {
+ long offset;
+ case RTC_PARAM_FEATURES:
+ if (param.index != 0)
+ err = -EINVAL;
+ param.uvalue = rtc->features[0];
+ break;
+
+ case RTC_PARAM_CORRECTION:
+ mutex_unlock(&rtc->ops_lock);
+ if (param.index != 0)
+ return -EINVAL;
+ err = rtc_read_offset(rtc, &offset);
+ mutex_lock(&rtc->ops_lock);
+ if (err == 0)
+ param.svalue = offset;
+ break;
+
+ default:
+ if (rtc->ops->param_get)
+ err = rtc->ops->param_get(rtc->dev.parent, ¶m);
+ else
+ err = -EINVAL;
+ }
+
+ if (!err)
+ if (copy_to_user(uarg, ¶m, sizeof(param)))
+ err = -EFAULT;
+
+ break;
+
+ case RTC_PARAM_SET:
+ if (copy_from_user(¶m, uarg, sizeof(param))) {
+ mutex_unlock(&rtc->ops_lock);
+ return -EFAULT;
+ }
+
+ switch(param.param) {
+ case RTC_PARAM_FEATURES:
+ err = -EINVAL;
+ break;
+
+ case RTC_PARAM_CORRECTION:
+ mutex_unlock(&rtc->ops_lock);
+ if (param.index != 0)
+ return -EINVAL;
+ return rtc_set_offset(rtc, param.svalue);
+
+ default:
+ if (rtc->ops->param_set)
+ err = rtc->ops->param_set(rtc->dev.parent, ¶m);
+ else
+ err = -EINVAL;
+ }
+
+ break;
+
default:
/* Finally try the driver's ioctl interface */
if (ops->ioctl) {
if (err)
return err;
now = rtc_tm_to_time64(&tm);
+
if (scheduled <= now)
return -ETIME;
/*
int rtc_set_alarm(struct rtc_device *rtc, struct rtc_wkalrm *alarm)
{
+ ktime_t alarm_time;
int err;
if (!rtc->ops)
if (rtc->aie_timer.enabled)
rtc_timer_remove(rtc, &rtc->aie_timer);
- rtc->aie_timer.node.expires = rtc_tm_to_ktime(alarm->time);
+ alarm_time = rtc_tm_to_ktime(alarm->time);
+ /*
+ * Round down so we never miss a deadline, checking for past deadline is
+ * done in __rtc_set_alarm
+ */
+ if (test_bit(RTC_FEATURE_ALARM_RES_MINUTE, rtc->features))
+ alarm_time = ktime_sub_ns(alarm_time, (u64)alarm->time.tm_sec * NSEC_PER_SEC);
+
+ rtc->aie_timer.node.expires = alarm_time;
rtc->aie_timer.period = 0;
if (alarm->enabled)
err = rtc_timer_enqueue(rtc, &rtc->aie_timer);
if (rtc->uie_rtctimer.enabled == enabled)
goto out;
- if (rtc->uie_unsupported || !test_bit(RTC_FEATURE_ALARM, rtc->features)) {
+ if (!test_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->features) ||
+ !test_bit(RTC_FEATURE_ALARM, rtc->features)) {
mutex_unlock(&rtc->ops_lock);
#ifdef CONFIG_RTC_INTF_DEV_UIE_EMUL
return rtc_dev_update_irq_enable_emul(rtc, enabled);
data->rtc->ops = &rtc_ops;
data->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
data->rtc->range_max = RTC_TIMESTAMP_END_2099;
- data->rtc->uie_unsupported = 1;
clear_bit(RTC_FEATURE_ALARM, data->rtc->features);
if (client->irq > 0) {
dev_err(dev, "failed to request alarm irq\n");
return ret;
}
+ } else {
+ clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, data->rtc->features);
}
if (client->irq > 0 || device_property_read_bool(dev, "wakeup-source")) {
{
int retval, i;
unsigned char buf[ARRAY_SIZE(ab8500_rtc_alarm_regs)];
- unsigned long mins, secs = 0, cursec = 0;
- struct rtc_time curtm;
+ unsigned long mins;
- /* Get the number of seconds since 1970 */
- secs = rtc_tm_to_time64(&alarm->time);
-
- /*
- * Check whether alarm is set less than 1min.
- * Since our RTC doesn't support alarm resolution less than 1min,
- * return -EINVAL, so UIE EMUL can take it up, incase of UIE_ON
- */
- ab8500_rtc_read_time(dev, &curtm); /* Read current time */
- cursec = rtc_tm_to_time64(&curtm);
- if ((secs - cursec) < 59) {
- dev_dbg(dev, "Alarm less than 1 minute not supported\r\n");
- return -EINVAL;
- }
-
- mins = secs / 60;
+ mins = (unsigned long)rtc_tm_to_time64(&alarm->time) / 60;
buf[2] = mins & 0xFF;
buf[1] = (mins >> 8) & 0xFF;
dev_pm_set_wake_irq(&pdev->dev, irq);
platform_set_drvdata(pdev, rtc);
- rtc->uie_unsupported = 1;
+ set_bit(RTC_FEATURE_ALARM_RES_MINUTE, rtc->features);
+ clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->features);
rtc->range_max = (1ULL << 24) * 60 - 1; // 24-bit minutes + 59 secs
rtc->start_secs = RTC_TIMESTAMP_BEGIN_2000;
MODULE_DEVICE_TABLE(of, ds1302_dt_ids);
#endif
+static const struct spi_device_id ds1302_spi_ids[] = {
+ { .name = "ds1302", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(spi, ds1302_spi_ids);
+
static struct spi_driver ds1302_driver = {
.driver.name = "rtc-ds1302",
.driver.of_match_table = of_match_ptr(ds1302_dt_ids),
.probe = ds1302_probe,
.remove = ds1302_remove,
+ .id_table = ds1302_spi_ids,
};
module_spi_driver(ds1302_driver);
};
MODULE_DEVICE_TABLE(of, ds1390_of_match);
+static const struct spi_device_id ds1390_spi_ids[] = {
+ { .name = "ds1390" },
+ {}
+};
+MODULE_DEVICE_TABLE(spi, ds1390_spi_ids);
+
static struct spi_driver ds1390_driver = {
.driver = {
.name = "rtc-ds1390",
.of_match_table = of_match_ptr(ds1390_of_match),
},
.probe = ds1390_probe,
+ .id_table = ds1390_spi_ids,
};
module_spi_driver(ds1390_driver);
* registered automatically when being referenced.
*/
of_node_put(fixed_clock);
- return 0;
+ return NULL;
}
/* First disable the clock */
MODULE_DEVICE_TABLE(of, mcp795_of_match);
#endif
+static const struct spi_device_id mcp795_spi_ids[] = {
+ { .name = "mcp795" },
+ { }
+};
+MODULE_DEVICE_TABLE(spi, mcp795_spi_ids);
+
static struct spi_driver mcp795_driver = {
.driver = {
.name = "rtc-mcp795",
.of_match_table = of_match_ptr(mcp795_of_match),
},
.probe = mcp795_probe,
+ .id_table = mcp795_spi_ids,
};
module_spi_driver(mcp795_driver);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Real time clocks driver for MStar/SigmaStar ARMv7 SoCs.
+ * Based on "Real Time Clock driver for msb252x." that was contained
+ * in various MStar kernels.
+ *
+ * (C) 2019 Daniel Palmer
+ * (C) 2021 Romain Perier
+ */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/rtc.h>
+
+/* Registers */
+#define REG_RTC_CTRL 0x00
+#define REG_RTC_FREQ_CW_L 0x04
+#define REG_RTC_FREQ_CW_H 0x08
+#define REG_RTC_LOAD_VAL_L 0x0C
+#define REG_RTC_LOAD_VAL_H 0x10
+#define REG_RTC_MATCH_VAL_L 0x14
+#define REG_RTC_MATCH_VAL_H 0x18
+#define REG_RTC_STATUS_INT 0x1C
+#define REG_RTC_CNT_VAL_L 0x20
+#define REG_RTC_CNT_VAL_H 0x24
+
+/* Control bits for REG_RTC_CTRL */
+#define SOFT_RSTZ_BIT BIT(0)
+#define CNT_EN_BIT BIT(1)
+#define WRAP_EN_BIT BIT(2)
+#define LOAD_EN_BIT BIT(3)
+#define READ_EN_BIT BIT(4)
+#define INT_MASK_BIT BIT(5)
+#define INT_FORCE_BIT BIT(6)
+#define INT_CLEAR_BIT BIT(7)
+
+/* Control bits for REG_RTC_STATUS_INT */
+#define RAW_INT_BIT BIT(0)
+#define ALM_INT_BIT BIT(1)
+
+struct msc313_rtc {
+ struct rtc_device *rtc_dev;
+ void __iomem *rtc_base;
+};
+
+static int msc313_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alarm)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev);
+ unsigned long seconds;
+
+ seconds = readw(priv->rtc_base + REG_RTC_MATCH_VAL_L)
+ | ((unsigned long)readw(priv->rtc_base + REG_RTC_MATCH_VAL_H) << 16);
+
+ rtc_time64_to_tm(seconds, &alarm->time);
+
+ if (!(readw(priv->rtc_base + REG_RTC_CTRL) & INT_MASK_BIT))
+ alarm->enabled = 1;
+
+ return 0;
+}
+
+static int msc313_rtc_alarm_irq_enable(struct device *dev, unsigned int enabled)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev);
+ u16 reg;
+
+ reg = readw(priv->rtc_base + REG_RTC_CTRL);
+ if (enabled)
+ reg &= ~INT_MASK_BIT;
+ else
+ reg |= INT_MASK_BIT;
+ writew(reg, priv->rtc_base + REG_RTC_CTRL);
+ return 0;
+}
+
+static int msc313_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alarm)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev);
+ unsigned long seconds;
+
+ seconds = rtc_tm_to_time64(&alarm->time);
+ writew((seconds & 0xFFFF), priv->rtc_base + REG_RTC_MATCH_VAL_L);
+ writew((seconds >> 16) & 0xFFFF, priv->rtc_base + REG_RTC_MATCH_VAL_H);
+
+ msc313_rtc_alarm_irq_enable(dev, alarm->enabled);
+
+ return 0;
+}
+
+static bool msc313_rtc_get_enabled(struct msc313_rtc *priv)
+{
+ return readw(priv->rtc_base + REG_RTC_CTRL) & CNT_EN_BIT;
+}
+
+static void msc313_rtc_set_enabled(struct msc313_rtc *priv)
+{
+ u16 reg;
+
+ reg = readw(priv->rtc_base + REG_RTC_CTRL);
+ reg |= CNT_EN_BIT;
+ writew(reg, priv->rtc_base + REG_RTC_CTRL);
+}
+
+static int msc313_rtc_read_time(struct device *dev, struct rtc_time *tm)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev);
+ u32 seconds;
+ u16 reg;
+
+ if (!msc313_rtc_get_enabled(priv))
+ return -EINVAL;
+
+ reg = readw(priv->rtc_base + REG_RTC_CTRL);
+ writew(reg | READ_EN_BIT, priv->rtc_base + REG_RTC_CTRL);
+
+ /* Wait for HW latch done */
+ while (readw(priv->rtc_base + REG_RTC_CTRL) & READ_EN_BIT)
+ udelay(1);
+
+ seconds = readw(priv->rtc_base + REG_RTC_CNT_VAL_L)
+ | ((unsigned long)readw(priv->rtc_base + REG_RTC_CNT_VAL_H) << 16);
+
+ rtc_time64_to_tm(seconds, tm);
+
+ return 0;
+}
+
+static int msc313_rtc_set_time(struct device *dev, struct rtc_time *tm)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev);
+ unsigned long seconds;
+ u16 reg;
+
+ seconds = rtc_tm_to_time64(tm);
+ writew(seconds & 0xFFFF, priv->rtc_base + REG_RTC_LOAD_VAL_L);
+ writew((seconds >> 16) & 0xFFFF, priv->rtc_base + REG_RTC_LOAD_VAL_H);
+
+ /* Enable load for loading value into internal RTC counter */
+ reg = readw(priv->rtc_base + REG_RTC_CTRL);
+ writew(reg | LOAD_EN_BIT, priv->rtc_base + REG_RTC_CTRL);
+
+ /* Wait for HW latch done */
+ while (readw(priv->rtc_base + REG_RTC_CTRL) & LOAD_EN_BIT)
+ udelay(1);
+ msc313_rtc_set_enabled(priv);
+ return 0;
+}
+
+static const struct rtc_class_ops msc313_rtc_ops = {
+ .read_time = msc313_rtc_read_time,
+ .set_time = msc313_rtc_set_time,
+ .read_alarm = msc313_rtc_read_alarm,
+ .set_alarm = msc313_rtc_set_alarm,
+ .alarm_irq_enable = msc313_rtc_alarm_irq_enable,
+};
+
+static irqreturn_t msc313_rtc_interrupt(s32 irq, void *dev_id)
+{
+ struct msc313_rtc *priv = dev_get_drvdata(dev_id);
+ u16 reg;
+
+ reg = readw(priv->rtc_base + REG_RTC_STATUS_INT);
+ if (!(reg & ALM_INT_BIT))
+ return IRQ_NONE;
+
+ reg = readw(priv->rtc_base + REG_RTC_CTRL);
+ reg |= INT_CLEAR_BIT;
+ reg &= ~INT_FORCE_BIT;
+ writew(reg, priv->rtc_base + REG_RTC_CTRL);
+
+ rtc_update_irq(priv->rtc_dev, 1, RTC_IRQF | RTC_AF);
+
+ return IRQ_HANDLED;
+}
+
+static int msc313_rtc_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct msc313_rtc *priv;
+ unsigned long rate;
+ struct clk *clk;
+ int ret;
+ int irq;
+
+ priv = devm_kzalloc(&pdev->dev, sizeof(struct msc313_rtc), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ priv->rtc_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(priv->rtc_base))
+ return PTR_ERR(priv->rtc_base);
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return -EINVAL;
+
+ priv->rtc_dev = devm_rtc_allocate_device(dev);
+ if (IS_ERR(priv->rtc_dev))
+ return PTR_ERR(priv->rtc_dev);
+
+ priv->rtc_dev->ops = &msc313_rtc_ops;
+ priv->rtc_dev->range_max = U32_MAX;
+
+ ret = devm_request_irq(dev, irq, msc313_rtc_interrupt, IRQF_SHARED,
+ dev_name(&pdev->dev), &pdev->dev);
+ if (ret) {
+ dev_err(dev, "Could not request IRQ\n");
+ return ret;
+ }
+
+ clk = devm_clk_get(dev, NULL);
+ if (IS_ERR(clk)) {
+ dev_err(dev, "No input reference clock\n");
+ return PTR_ERR(clk);
+ }
+
+ ret = clk_prepare_enable(clk);
+ if (ret) {
+ dev_err(dev, "Failed to enable the reference clock, %d\n", ret);
+ return ret;
+ }
+
+ ret = devm_add_action_or_reset(dev, (void (*) (void *))clk_disable_unprepare, clk);
+ if (ret)
+ return ret;
+
+ rate = clk_get_rate(clk);
+ writew(rate & 0xFFFF, priv->rtc_base + REG_RTC_FREQ_CW_L);
+ writew((rate >> 16) & 0xFFFF, priv->rtc_base + REG_RTC_FREQ_CW_H);
+
+ platform_set_drvdata(pdev, priv);
+
+ return devm_rtc_register_device(priv->rtc_dev);
+}
+
+static const struct of_device_id msc313_rtc_of_match_table[] = {
+ { .compatible = "mstar,msc313-rtc" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, msc313_rtc_of_match_table);
+
+static struct platform_driver msc313_rtc_driver = {
+ .probe = msc313_rtc_probe,
+ .driver = {
+ .name = "msc313-rtc",
+ .of_match_table = msc313_rtc_of_match_table,
+ },
+};
+
+module_platform_driver(msc313_rtc_driver);
+
+MODULE_AUTHOR("Daniel Palmer <daniel@thingy.jp>");
+MODULE_AUTHOR("Romain Perier <romain.perier@gmail.com>");
+MODULE_DESCRIPTION("MStar RTC Driver");
+MODULE_LICENSE("GPL v2");
module_platform_driver(omap_rtc_driver);
-MODULE_ALIAS("platform:omap_rtc");
MODULE_AUTHOR("George G. Davis (and others)");
MODULE_LICENSE("GPL");
MODULE_DEVICE_TABLE(of, pcf2123_dt_ids);
#endif
+static const struct spi_device_id pcf2123_spi_ids[] = {
+ { .name = "pcf2123", },
+ { .name = "rv2123", },
+ { .name = "rtc-pcf2123", },
+ { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(spi, pcf2123_spi_ids);
+
static struct spi_driver pcf2123_driver = {
.driver = {
.name = "rtc-pcf2123",
.of_match_table = of_match_ptr(pcf2123_dt_ids),
},
.probe = pcf2123_probe,
+ .id_table = pcf2123_spi_ids,
};
module_spi_driver(pcf2123_driver);
#define PCF85063_REG_CTRL1 0x00 /* status */
#define PCF85063_REG_CTRL1_CAP_SEL BIT(0)
#define PCF85063_REG_CTRL1_STOP BIT(5)
+#define PCF85063_REG_CTRL1_EXT_TEST BIT(7)
#define PCF85063_REG_CTRL2 0x01
#define PCF85063_CTRL2_AF BIT(6)
* reset state until all time/date registers are written
*/
rc = regmap_update_bits(pcf85063->regmap, PCF85063_REG_CTRL1,
+ PCF85063_REG_CTRL1_EXT_TEST |
PCF85063_REG_CTRL1_STOP,
PCF85063_REG_CTRL1_STOP);
if (rc)
if (ret < 0)
return ret;
- status = status & PCF85063_REG_SC_OS ? RTC_VL_DATA_INVALID : 0;
+ status = (status & PCF85063_REG_SC_OS) ? RTC_VL_DATA_INVALID : 0;
return put_user(status, (unsigned int __user *)arg);
struct clk *clk;
struct clk_init_data init;
struct device_node *node = pcf85063->rtc->dev.parent->of_node;
+ struct device_node *fixed_clock;
+
+ fixed_clock = of_get_child_by_name(node, "clock");
+ if (fixed_clock) {
+ /*
+ * skip registering square wave clock when a fixed
+ * clock has been registered. The fixed clock is
+ * registered automatically when being referenced.
+ */
+ of_node_put(fixed_clock);
+ return NULL;
+ }
init.name = "pcf85063-clkout";
init.ops = &pcf85063_clkout_ops;
*/
#include <linux/bcd.h>
+#include <linux/bitfield.h>
#include <linux/i2c.h>
#include <linux/module.h>
+#include <linux/regmap.h>
#include <linux/rtc.h>
#include <linux/of.h>
#include <linux/pm_wakeirq.h>
#define PCF8523_CONTROL2_AF BIT(3)
#define PCF8523_REG_CONTROL3 0x02
-#define PCF8523_CONTROL3_PM_BLD BIT(7) /* battery low detection disabled */
-#define PCF8523_CONTROL3_PM_VDD BIT(6) /* switch-over disabled */
-#define PCF8523_CONTROL3_PM_DSM BIT(5) /* direct switching mode */
-#define PCF8523_CONTROL3_PM_MASK 0xe0
+#define PCF8523_CONTROL3_PM GENMASK(7,5)
+#define PCF8523_PM_STANDBY 0x7
#define PCF8523_CONTROL3_BLF BIT(2) /* battery low bit, read-only */
+#define PCF8523_CONTROL3_BSF BIT(3)
#define PCF8523_REG_SECONDS 0x03
#define PCF8523_SECONDS_OS BIT(7)
struct pcf8523 {
struct rtc_device *rtc;
- struct i2c_client *client;
+ struct regmap *regmap;
};
-static int pcf8523_read(struct i2c_client *client, u8 reg, u8 *valuep)
+static int pcf8523_load_capacitance(struct pcf8523 *pcf8523, struct device_node *node)
{
- struct i2c_msg msgs[2];
- u8 value = 0;
- int err;
-
- msgs[0].addr = client->addr;
- msgs[0].flags = 0;
- msgs[0].len = sizeof(reg);
- msgs[0].buf = ®
-
- msgs[1].addr = client->addr;
- msgs[1].flags = I2C_M_RD;
- msgs[1].len = sizeof(value);
- msgs[1].buf = &value;
-
- err = i2c_transfer(client->adapter, msgs, ARRAY_SIZE(msgs));
- if (err < 0)
- return err;
-
- *valuep = value;
-
- return 0;
-}
-
-static int pcf8523_write(struct i2c_client *client, u8 reg, u8 value)
-{
- u8 buffer[2] = { reg, value };
- struct i2c_msg msg;
- int err;
-
- msg.addr = client->addr;
- msg.flags = 0;
- msg.len = sizeof(buffer);
- msg.buf = buffer;
-
- err = i2c_transfer(client->adapter, &msg, 1);
- if (err < 0)
- return err;
-
- return 0;
-}
-
-static int pcf8523_voltage_low(struct i2c_client *client)
-{
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL3, &value);
- if (err < 0)
- return err;
-
- return !!(value & PCF8523_CONTROL3_BLF);
-}
-
-static int pcf8523_load_capacitance(struct i2c_client *client)
-{
- u32 load;
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL1, &value);
- if (err < 0)
- return err;
+ u32 load, value = 0;
load = 12500;
- of_property_read_u32(client->dev.of_node, "quartz-load-femtofarads",
- &load);
+ of_property_read_u32(node, "quartz-load-femtofarads", &load);
switch (load) {
default:
- dev_warn(&client->dev, "Unknown quartz-load-femtofarads value: %d. Assuming 12500",
+ dev_warn(&pcf8523->rtc->dev, "Unknown quartz-load-femtofarads value: %d. Assuming 12500",
load);
fallthrough;
case 12500:
value |= PCF8523_CONTROL1_CAP_SEL;
break;
case 7000:
- value &= ~PCF8523_CONTROL1_CAP_SEL;
break;
}
- err = pcf8523_write(client, PCF8523_REG_CONTROL1, value);
-
- return err;
-}
-
-static int pcf8523_set_pm(struct i2c_client *client, u8 pm)
-{
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL3, &value);
- if (err < 0)
- return err;
-
- value = (value & ~PCF8523_CONTROL3_PM_MASK) | pm;
-
- err = pcf8523_write(client, PCF8523_REG_CONTROL3, value);
- if (err < 0)
- return err;
-
- return 0;
+ return regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL1,
+ PCF8523_CONTROL1_CAP_SEL, value);
}
static irqreturn_t pcf8523_irq(int irq, void *dev_id)
{
- struct pcf8523 *pcf8523 = i2c_get_clientdata(dev_id);
- u8 value;
+ struct pcf8523 *pcf8523 = dev_id;
+ u32 value;
int err;
- err = pcf8523_read(pcf8523->client, PCF8523_REG_CONTROL2, &value);
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL2, &value);
if (err < 0)
return IRQ_HANDLED;
if (value & PCF8523_CONTROL2_AF) {
value &= ~PCF8523_CONTROL2_AF;
- pcf8523_write(pcf8523->client, PCF8523_REG_CONTROL2, value);
+ regmap_write(pcf8523->regmap, PCF8523_REG_CONTROL2, value);
rtc_update_irq(pcf8523->rtc, 1, RTC_IRQF | RTC_AF);
return IRQ_HANDLED;
return IRQ_NONE;
}
-static int pcf8523_stop_rtc(struct i2c_client *client)
-{
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL1, &value);
- if (err < 0)
- return err;
-
- value |= PCF8523_CONTROL1_STOP;
-
- err = pcf8523_write(client, PCF8523_REG_CONTROL1, value);
- if (err < 0)
- return err;
-
- return 0;
-}
-
-static int pcf8523_start_rtc(struct i2c_client *client)
-{
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL1, &value);
- if (err < 0)
- return err;
-
- value &= ~PCF8523_CONTROL1_STOP;
-
- err = pcf8523_write(client, PCF8523_REG_CONTROL1, value);
- if (err < 0)
- return err;
-
- return 0;
-}
-
static int pcf8523_rtc_read_time(struct device *dev, struct rtc_time *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
- u8 start = PCF8523_REG_SECONDS, regs[7];
- struct i2c_msg msgs[2];
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
+ u8 regs[7];
int err;
- err = pcf8523_voltage_low(client);
- if (err < 0) {
- return err;
- } else if (err > 0) {
- dev_err(dev, "low voltage detected, time is unreliable\n");
- return -EINVAL;
- }
-
- msgs[0].addr = client->addr;
- msgs[0].flags = 0;
- msgs[0].len = 1;
- msgs[0].buf = &start;
-
- msgs[1].addr = client->addr;
- msgs[1].flags = I2C_M_RD;
- msgs[1].len = sizeof(regs);
- msgs[1].buf = regs;
-
- err = i2c_transfer(client->adapter, msgs, ARRAY_SIZE(msgs));
+ err = regmap_bulk_read(pcf8523->regmap, PCF8523_REG_SECONDS, regs,
+ sizeof(regs));
if (err < 0)
return err;
static int pcf8523_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
- struct i2c_msg msg;
- u8 regs[8];
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
+ u8 regs[7];
int err;
- err = pcf8523_stop_rtc(client);
+ err = regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL1,
+ PCF8523_CONTROL1_STOP, PCF8523_CONTROL1_STOP);
if (err < 0)
return err;
- regs[0] = PCF8523_REG_SECONDS;
/* This will purposely overwrite PCF8523_SECONDS_OS */
- regs[1] = bin2bcd(tm->tm_sec);
- regs[2] = bin2bcd(tm->tm_min);
- regs[3] = bin2bcd(tm->tm_hour);
- regs[4] = bin2bcd(tm->tm_mday);
- regs[5] = tm->tm_wday;
- regs[6] = bin2bcd(tm->tm_mon + 1);
- regs[7] = bin2bcd(tm->tm_year - 100);
-
- msg.addr = client->addr;
- msg.flags = 0;
- msg.len = sizeof(regs);
- msg.buf = regs;
-
- err = i2c_transfer(client->adapter, &msg, 1);
+ regs[0] = bin2bcd(tm->tm_sec);
+ regs[1] = bin2bcd(tm->tm_min);
+ regs[2] = bin2bcd(tm->tm_hour);
+ regs[3] = bin2bcd(tm->tm_mday);
+ regs[4] = tm->tm_wday;
+ regs[5] = bin2bcd(tm->tm_mon + 1);
+ regs[6] = bin2bcd(tm->tm_year - 100);
+
+ err = regmap_bulk_write(pcf8523->regmap, PCF8523_REG_SECONDS, regs,
+ sizeof(regs));
if (err < 0) {
/*
* If the time cannot be set, restart the RTC anyway. Note
* that errors are ignored if the RTC cannot be started so
* that we have a chance to propagate the original error.
*/
- pcf8523_start_rtc(client);
+ regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL1,
+ PCF8523_CONTROL1_STOP, 0);
return err;
}
- return pcf8523_start_rtc(client);
+ return regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL1,
+ PCF8523_CONTROL1_STOP, 0);
}
static int pcf8523_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
- u8 start = PCF8523_REG_MINUTE_ALARM, regs[4];
- struct i2c_msg msgs[2];
- u8 value;
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
+ u8 regs[4];
+ u32 value;
int err;
- msgs[0].addr = client->addr;
- msgs[0].flags = 0;
- msgs[0].len = 1;
- msgs[0].buf = &start;
-
- msgs[1].addr = client->addr;
- msgs[1].flags = I2C_M_RD;
- msgs[1].len = sizeof(regs);
- msgs[1].buf = regs;
-
- err = i2c_transfer(client->adapter, msgs, ARRAY_SIZE(msgs));
+ err = regmap_bulk_read(pcf8523->regmap, PCF8523_REG_MINUTE_ALARM, regs,
+ sizeof(regs));
if (err < 0)
return err;
tm->time.tm_mday = bcd2bin(regs[2] & 0x3F);
tm->time.tm_wday = bcd2bin(regs[3] & 0x7);
- err = pcf8523_read(client, PCF8523_REG_CONTROL1, &value);
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL1, &value);
if (err < 0)
return err;
tm->enabled = !!(value & PCF8523_CONTROL1_AIE);
- err = pcf8523_read(client, PCF8523_REG_CONTROL2, &value);
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL2, &value);
if (err < 0)
return err;
tm->pending = !!(value & PCF8523_CONTROL2_AF);
static int pcf8523_irq_enable(struct device *dev, unsigned int enabled)
{
- struct i2c_client *client = to_i2c_client(dev);
- u8 value;
- int err;
-
- err = pcf8523_read(client, PCF8523_REG_CONTROL1, &value);
- if (err < 0)
- return err;
-
- value &= PCF8523_CONTROL1_AIE;
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
- if (enabled)
- value |= PCF8523_CONTROL1_AIE;
-
- err = pcf8523_write(client, PCF8523_REG_CONTROL1, value);
- if (err < 0)
- return err;
-
- return 0;
+ return regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL1,
+ PCF8523_CONTROL1_AIE, enabled ?
+ PCF8523_CONTROL1_AIE : 0);
}
static int pcf8523_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *tm)
{
- struct i2c_client *client = to_i2c_client(dev);
- struct i2c_msg msg;
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
u8 regs[5];
int err;
if (err)
return err;
- err = pcf8523_write(client, PCF8523_REG_CONTROL2, 0);
+ err = regmap_write(pcf8523->regmap, PCF8523_REG_CONTROL2, 0);
if (err < 0)
return err;
rtc_time64_to_tm(alarm_time, &tm->time);
}
- regs[0] = PCF8523_REG_MINUTE_ALARM;
- regs[1] = bin2bcd(tm->time.tm_min);
- regs[2] = bin2bcd(tm->time.tm_hour);
- regs[3] = bin2bcd(tm->time.tm_mday);
- regs[4] = ALARM_DIS;
- msg.addr = client->addr;
- msg.flags = 0;
- msg.len = sizeof(regs);
- msg.buf = regs;
- err = i2c_transfer(client->adapter, &msg, 1);
+ regs[0] = bin2bcd(tm->time.tm_min);
+ regs[1] = bin2bcd(tm->time.tm_hour);
+ regs[2] = bin2bcd(tm->time.tm_mday);
+ regs[3] = ALARM_DIS;
+
+ err = regmap_bulk_write(pcf8523->regmap, PCF8523_REG_MINUTE_ALARM, regs,
+ sizeof(regs));
if (err < 0)
return err;
return 0;
}
-#ifdef CONFIG_RTC_INTF_DEV
+static int pcf8523_param_get(struct device *dev, struct rtc_param *param)
+{
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
+ int ret;
+
+ switch(param->param) {
+ u32 value;
+
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ ret = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL3, &value);
+ if (ret < 0)
+ return ret;
+
+ value = FIELD_GET(PCF8523_CONTROL3_PM, value);
+
+ switch(value) {
+ case 0x0:
+ case 0x4:
+ param->uvalue = RTC_BSM_LEVEL;
+ break;
+ case 0x1:
+ case 0x5:
+ param->uvalue = RTC_BSM_DIRECT;
+ break;
+ case PCF8523_PM_STANDBY:
+ param->uvalue = RTC_BSM_STANDBY;
+ break;
+ default:
+ param->uvalue = RTC_BSM_DISABLED;
+ }
+
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int pcf8523_param_set(struct device *dev, struct rtc_param *param)
+{
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
+
+ switch(param->param) {
+ u8 mode;
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ switch (param->uvalue) {
+ case RTC_BSM_DISABLED:
+ mode = 0x2;
+ break;
+ case RTC_BSM_DIRECT:
+ mode = 0x1;
+ break;
+ case RTC_BSM_LEVEL:
+ mode = 0x0;
+ break;
+ case RTC_BSM_STANDBY:
+ mode = PCF8523_PM_STANDBY;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return regmap_update_bits(pcf8523->regmap, PCF8523_REG_CONTROL3,
+ PCF8523_CONTROL3_PM,
+ FIELD_PREP(PCF8523_CONTROL3_PM, mode));
+
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static int pcf8523_rtc_ioctl(struct device *dev, unsigned int cmd,
unsigned long arg)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
unsigned int flags = 0;
- u8 value;
+ u32 value;
int ret;
switch (cmd) {
case RTC_VL_READ:
- ret = pcf8523_voltage_low(client);
+ ret = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL3, &value);
if (ret < 0)
return ret;
- if (ret)
+
+ if (value & PCF8523_CONTROL3_BLF)
flags |= RTC_VL_BACKUP_LOW;
- ret = pcf8523_read(client, PCF8523_REG_SECONDS, &value);
+ ret = regmap_read(pcf8523->regmap, PCF8523_REG_SECONDS, &value);
if (ret < 0)
return ret;
return -ENOIOCTLCMD;
}
}
-#else
-#define pcf8523_rtc_ioctl NULL
-#endif
static int pcf8523_rtc_read_offset(struct device *dev, long *offset)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
int err;
- u8 value;
+ u32 value;
s8 val;
- err = pcf8523_read(client, PCF8523_REG_OFFSET, &value);
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_OFFSET, &value);
if (err < 0)
return err;
static int pcf8523_rtc_set_offset(struct device *dev, long offset)
{
- struct i2c_client *client = to_i2c_client(dev);
+ struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
long reg_m0, reg_m1;
- u8 value;
+ u32 value;
reg_m0 = clamp(DIV_ROUND_CLOSEST(offset, 4340), -64L, 63L);
reg_m1 = clamp(DIV_ROUND_CLOSEST(offset, 4069), -64L, 63L);
else
value = (reg_m1 & 0x7f) | PCF8523_OFFSET_MODE;
- return pcf8523_write(client, PCF8523_REG_OFFSET, value);
+ return regmap_write(pcf8523->regmap, PCF8523_REG_OFFSET, value);
}
static const struct rtc_class_ops pcf8523_rtc_ops = {
.ioctl = pcf8523_rtc_ioctl,
.read_offset = pcf8523_rtc_read_offset,
.set_offset = pcf8523_rtc_set_offset,
+ .param_get = pcf8523_param_get,
+ .param_set = pcf8523_param_set,
+};
+
+static const struct regmap_config regmap_config = {
+ .reg_bits = 8,
+ .val_bits = 8,
+ .max_register = 0x13,
};
static int pcf8523_probe(struct i2c_client *client,
struct pcf8523 *pcf8523;
struct rtc_device *rtc;
bool wakeup_source = false;
+ u32 value;
int err;
if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C))
if (!pcf8523)
return -ENOMEM;
+ pcf8523->regmap = devm_regmap_init_i2c(client, ®map_config);
+ if (IS_ERR(pcf8523->regmap))
+ return PTR_ERR(pcf8523->regmap);
+
i2c_set_clientdata(client, pcf8523);
- pcf8523->client = client;
- err = pcf8523_load_capacitance(client);
+ rtc = devm_rtc_allocate_device(&client->dev);
+ if (IS_ERR(rtc))
+ return PTR_ERR(rtc);
+ pcf8523->rtc = rtc;
+
+ err = pcf8523_load_capacitance(pcf8523, client->dev.of_node);
if (err < 0)
dev_warn(&client->dev, "failed to set xtal load capacitance: %d",
err);
- err = pcf8523_set_pm(client, 0);
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_SECONDS, &value);
if (err < 0)
return err;
- rtc = devm_rtc_allocate_device(&client->dev);
- if (IS_ERR(rtc))
- return PTR_ERR(rtc);
+ if (value & PCF8523_SECONDS_OS) {
+ err = regmap_read(pcf8523->regmap, PCF8523_REG_CONTROL3, &value);
+ if (err < 0)
+ return err;
+
+ if (FIELD_GET(PCF8523_CONTROL3_PM, value) == PCF8523_PM_STANDBY) {
+ err = regmap_write(pcf8523->regmap, PCF8523_REG_CONTROL3,
+ value & ~PCF8523_CONTROL3_PM);
+ if (err < 0)
+ return err;
+ }
+ }
- pcf8523->rtc = rtc;
rtc->ops = &pcf8523_rtc_ops;
rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
rtc->range_max = RTC_TIMESTAMP_END_2099;
rtc->uie_unsupported = 1;
if (client->irq > 0) {
- err = pcf8523_write(client, PCF8523_TMR_CLKOUT_CTRL, 0x38);
+ err = regmap_write(pcf8523->regmap, PCF8523_TMR_CLKOUT_CTRL, 0x38);
if (err < 0)
return err;
err = devm_request_threaded_irq(&client->dev, client->irq,
NULL, pcf8523_irq,
IRQF_SHARED | IRQF_ONESHOT | IRQF_TRIGGER_LOW,
- dev_name(&rtc->dev), client);
+ dev_name(&rtc->dev), pcf8523);
if (err)
return err;
dev_pm_set_wake_irq(&client->dev, client->irq);
}
-#ifdef CONFIG_OF
wakeup_source = of_property_read_bool(client->dev.of_node, "wakeup-source");
-#endif
if (client->irq > 0 || wakeup_source)
device_init_wakeup(&client->dev, true);
};
MODULE_DEVICE_TABLE(i2c, pcf8523_id);
-#ifdef CONFIG_OF
static const struct of_device_id pcf8523_of_match[] = {
{ .compatible = "nxp,pcf8523" },
{ .compatible = "microcrystal,rv8523" },
{ }
};
MODULE_DEVICE_TABLE(of, pcf8523_of_match);
-#endif
static struct i2c_driver pcf8523_driver = {
.driver = {
.name = "rtc-pcf8523",
- .of_match_table = of_match_ptr(pcf8523_of_match),
+ .of_match_table = pcf8523_of_match,
},
.probe = pcf8523_probe,
.id_table = pcf8523_id,
#include <linux/clk-provider.h>
#include <linux/bcd.h>
+#include <linux/bitfield.h>
#include <linux/bitops.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
#define RV3028_BACKUP_TCE BIT(5)
#define RV3028_BACKUP_TCR_MASK GENMASK(1,0)
+#define RV3028_BACKUP_BSM GENMASK(3,2)
+
+#define RV3028_BACKUP_BSM_DSM 0x1
+#define RV3028_BACKUP_BSM_LSM 0x3
#define OFFSET_STEP_PPT 953674
}
+static int rv3028_param_get(struct device *dev, struct rtc_param *param)
+{
+ struct rv3028_data *rv3028 = dev_get_drvdata(dev);
+ int ret;
+
+ switch(param->param) {
+ u32 value;
+
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ ret = regmap_read(rv3028->regmap, RV3028_BACKUP, &value);
+ if (ret < 0)
+ return ret;
+
+ value = FIELD_GET(RV3028_BACKUP_BSM, value);
+
+ switch(value) {
+ case RV3028_BACKUP_BSM_DSM:
+ param->uvalue = RTC_BSM_DIRECT;
+ break;
+ case RV3028_BACKUP_BSM_LSM:
+ param->uvalue = RTC_BSM_LEVEL;
+ break;
+ default:
+ param->uvalue = RTC_BSM_DISABLED;
+ }
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int rv3028_param_set(struct device *dev, struct rtc_param *param)
+{
+ struct rv3028_data *rv3028 = dev_get_drvdata(dev);
+
+ switch(param->param) {
+ u8 mode;
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ switch (param->uvalue) {
+ case RTC_BSM_DISABLED:
+ mode = 0;
+ break;
+ case RTC_BSM_DIRECT:
+ mode = RV3028_BACKUP_BSM_DSM;
+ break;
+ case RTC_BSM_LEVEL:
+ mode = RV3028_BACKUP_BSM_LSM;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return rv3028_update_cfg(rv3028, RV3028_BACKUP, RV3028_BACKUP_BSM,
+ FIELD_PREP(RV3028_BACKUP_BSM, mode));
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static int rv3028_ioctl(struct device *dev, unsigned int cmd, unsigned long arg)
{
struct rv3028_data *rv3028 = dev_get_drvdata(dev);
.read_offset = rv3028_read_offset,
.set_offset = rv3028_set_offset,
.ioctl = rv3028_ioctl,
+ .param_get = rv3028_param_get,
+ .param_set = rv3028_param_set,
};
static const struct regmap_config regmap_config = {
if (ret)
return ret;
+ set_bit(RTC_FEATURE_BACKUP_SWITCH_MODE, rv3028->rtc->features);
+
rv3028->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
rv3028->rtc->range_max = RTC_TIMESTAMP_END_2099;
rv3028->rtc->ops = &rv3028_rtc_ops;
struct rv3032_data {
struct regmap *regmap;
struct rtc_device *rtc;
+ bool trickle_charger_set;
#ifdef CONFIG_COMMON_CLK
struct clk_hw clkout_hw;
#endif
u8 ctrl = 0;
int ret;
- /* The alarm has no seconds, round up to nearest minute */
- if (alrm->time.tm_sec) {
- time64_t alarm_time = rtc_tm_to_time64(&alrm->time);
-
- alarm_time += 60 - alrm->time.tm_sec;
- rtc_time64_to_tm(alarm_time, &alrm->time);
- }
-
ret = regmap_update_bits(rv3032->regmap, RV3032_CTRL2,
RV3032_CTRL2_AIE | RV3032_CTRL2_UIE, 0);
if (ret)
FIELD_PREP(RV3032_OFFSET_MSK, offset));
}
+static int rv3032_param_get(struct device *dev, struct rtc_param *param)
+{
+ struct rv3032_data *rv3032 = dev_get_drvdata(dev);
+ int ret;
+
+ switch(param->param) {
+ u32 value;
+
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ ret = regmap_read(rv3032->regmap, RV3032_PMU, &value);
+ if (ret < 0)
+ return ret;
+
+ value = FIELD_GET(RV3032_PMU_BSM, value);
+
+ switch(value) {
+ case RV3032_PMU_BSM_DSM:
+ param->uvalue = RTC_BSM_DIRECT;
+ break;
+ case RV3032_PMU_BSM_LSM:
+ param->uvalue = RTC_BSM_LEVEL;
+ break;
+ default:
+ param->uvalue = RTC_BSM_DISABLED;
+ }
+
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int rv3032_param_set(struct device *dev, struct rtc_param *param)
+{
+ struct rv3032_data *rv3032 = dev_get_drvdata(dev);
+
+ switch(param->param) {
+ u8 mode;
+ case RTC_PARAM_BACKUP_SWITCH_MODE:
+ if (rv3032->trickle_charger_set)
+ return -EINVAL;
+
+ switch (param->uvalue) {
+ case RTC_BSM_DISABLED:
+ mode = 0;
+ break;
+ case RTC_BSM_DIRECT:
+ mode = RV3032_PMU_BSM_DSM;
+ break;
+ case RTC_BSM_LEVEL:
+ mode = RV3032_PMU_BSM_LSM;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return rv3032_update_cfg(rv3032, RV3032_PMU, RV3032_PMU_BSM,
+ FIELD_PREP(RV3032_PMU_BSM, mode));
+
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static int rv3032_ioctl(struct device *dev, unsigned int cmd, unsigned long arg)
{
struct rv3032_data *rv3032 = dev_get_drvdata(dev);
return 0;
}
+ rv3032->trickle_charger_set = true;
+
return rv3032_update_cfg(rv3032, RV3032_PMU,
RV3032_PMU_TCR | RV3032_PMU_TCM | RV3032_PMU_BSM,
val | FIELD_PREP(RV3032_PMU_TCR, i));
ret = rv3032_enter_eerd(rv3032, &eerd);
if (ret)
- goto exit_eerd;
+ return ret;
ret = regmap_write(rv3032->regmap, RV3032_CLKOUT1, hfd & 0xff);
if (ret)
- return ret;
+ goto exit_eerd;
ret = regmap_write(rv3032->regmap, RV3032_CLKOUT2, RV3032_CLKOUT2_OS |
FIELD_PREP(RV3032_CLKOUT2_HFD_MSK, hfd >> 8));
.read_alarm = rv3032_get_alarm,
.set_alarm = rv3032_set_alarm,
.alarm_irq_enable = rv3032_alarm_irq_enable,
+ .param_get = rv3032_param_get,
+ .param_set = rv3032_param_set,
};
static const struct regmap_config regmap_config = {
rv3032_trickle_charger_setup(&client->dev, rv3032);
+ set_bit(RTC_FEATURE_BACKUP_SWITCH_MODE, rv3032->rtc->features);
+ set_bit(RTC_FEATURE_ALARM_RES_MINUTE, rv3032->rtc->features);
+
rv3032->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
rv3032->rtc->range_max = RTC_TIMESTAMP_END_2099;
rv3032->rtc->ops = &rv3032_rtc_ops;
}
}
- ctrl[1] &= ~RV8803_FLAG_AF;
- err = rv8803_write_reg(rv8803->client, RV8803_FLAG, ctrl[1]);
+ ctrl[0] &= ~RV8803_FLAG_AF;
+ err = rv8803_write_reg(rv8803->client, RV8803_FLAG, ctrl[0]);
mutex_unlock(&rv8803->flags_lock);
if (err)
return err;
static int rx6110_i2c_probe(struct i2c_client *client,
const struct i2c_device_id *id)
{
- struct i2c_adapter *adapter = to_i2c_adapter(client->dev.parent);
+ struct i2c_adapter *adapter = client->adapter;
struct rx6110_data *rx6110;
if (!i2c_check_functionality(adapter, I2C_FUNC_SMBUS_BYTE_DATA
u8 date[7];
int ret;
- if ((dt->tm_year < 100) || (dt->tm_year > 199))
- return -EINVAL;
-
/*
* Here the read-only bits are written as "0". I'm not sure if that
* is sound.
u8 ald[2];
int ctrl2, err;
- if (client->irq <= 0)
- return -EINVAL;
-
err = rx8025_read_regs(client, RX8025_REG_ALDMIN, 2, ald);
if (err)
return err;
u8 ald[2];
int err;
- if (client->irq <= 0)
- return -EINVAL;
-
- /*
- * Hardware alarm precision is 1 minute!
- * round up to nearest minute
- */
- if (t->time.tm_sec) {
- time64_t alarm_time = rtc_tm_to_time64(&t->time);
-
- alarm_time += 60 - t->time.tm_sec;
- rtc_time64_to_tm(alarm_time, &t->time);
- }
-
ald[0] = bin2bcd(t->time.tm_min);
if (rx8025->ctrl1 & RX8025_BIT_CTRL1_1224)
ald[1] = bin2bcd(t->time.tm_hour);
return 0;
}
-static const struct rtc_class_ops rx8025_rtc_ops = {
- .read_time = rx8025_get_time,
- .set_time = rx8025_set_time,
- .read_alarm = rx8025_read_alarm,
- .set_alarm = rx8025_set_alarm,
- .alarm_irq_enable = rx8025_alarm_irq_enable,
-};
-
/*
- * Clock precision adjustment support
- *
* According to the RX8025 SA/NB application manual the frequency and
* temperature characteristics can be approximated using the following
* equation:
* a : Coefficient = (-35 +-5) * 10**-9
* ut: Ultimate temperature in degree = +25 +-5 degree
* t : Any temperature in degree
- *
- * Note that the clock adjustment in ppb must be entered (which is
- * the negative value of the deviation).
*/
-static int rx8025_get_clock_adjust(struct device *dev, int *adj)
+static int rx8025_read_offset(struct device *dev, long *offset)
{
struct i2c_client *client = to_i2c_client(dev);
int digoff;
if (digoff < 0)
return digoff;
- *adj = digoff >= 64 ? digoff - 128 : digoff;
- if (*adj > 0)
- (*adj)--;
- *adj *= -RX8025_ADJ_RESOLUTION;
+ *offset = digoff >= 64 ? digoff - 128 : digoff;
+ if (*offset > 0)
+ (*offset)--;
+ *offset *= RX8025_ADJ_RESOLUTION;
return 0;
}
-static int rx8025_set_clock_adjust(struct device *dev, int adj)
+static int rx8025_set_offset(struct device *dev, long offset)
{
struct i2c_client *client = to_i2c_client(dev);
u8 digoff;
int err;
- adj /= -RX8025_ADJ_RESOLUTION;
- if (adj > RX8025_ADJ_DATA_MAX)
- adj = RX8025_ADJ_DATA_MAX;
- else if (adj < RX8025_ADJ_DATA_MIN)
- adj = RX8025_ADJ_DATA_MIN;
- else if (adj > 0)
- adj++;
- else if (adj < 0)
- adj += 128;
- digoff = adj;
+ offset /= RX8025_ADJ_RESOLUTION;
+ if (offset > RX8025_ADJ_DATA_MAX)
+ offset = RX8025_ADJ_DATA_MAX;
+ else if (offset < RX8025_ADJ_DATA_MIN)
+ offset = RX8025_ADJ_DATA_MIN;
+ else if (offset > 0)
+ offset++;
+ else if (offset < 0)
+ offset += 128;
+ digoff = offset;
err = rx8025_write_reg(client, RX8025_REG_DIGOFF, digoff);
if (err)
return err;
- dev_dbg(dev, "%s: write 0x%02x\n", __func__, digoff);
-
return 0;
}
+static const struct rtc_class_ops rx8025_rtc_ops = {
+ .read_time = rx8025_get_time,
+ .set_time = rx8025_set_time,
+ .read_alarm = rx8025_read_alarm,
+ .set_alarm = rx8025_set_alarm,
+ .alarm_irq_enable = rx8025_alarm_irq_enable,
+ .read_offset = rx8025_read_offset,
+ .set_offset = rx8025_set_offset,
+};
+
static ssize_t rx8025_sysfs_show_clock_adjust(struct device *dev,
struct device_attribute *attr,
char *buf)
{
- int err, adj;
+ long adj;
+ int err;
- err = rx8025_get_clock_adjust(dev, &adj);
+ dev_warn_once(dev, "clock_adjust_ppb is deprecated, use offset\n");
+ err = rx8025_read_offset(dev, &adj);
if (err)
return err;
- return sprintf(buf, "%d\n", adj);
+ return sprintf(buf, "%ld\n", -adj);
}
static ssize_t rx8025_sysfs_store_clock_adjust(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
- int adj, err;
+ long adj;
+ int err;
- if (sscanf(buf, "%i", &adj) != 1)
+ dev_warn_once(dev, "clock_adjust_ppb is deprecated, use offset\n");
+ if (kstrtol(buf, 10, &adj) != 0)
return -EINVAL;
- err = rx8025_set_clock_adjust(dev, adj);
+ err = rx8025_set_offset(dev, -adj);
return err ? err : count;
}
rx8025_sysfs_show_clock_adjust,
rx8025_sysfs_store_clock_adjust);
-static int rx8025_sysfs_register(struct device *dev)
-{
- return device_create_file(dev, &dev_attr_clock_adjust_ppb);
-}
+static struct attribute *rx8025_attrs[] = {
+ &dev_attr_clock_adjust_ppb.attr,
+ NULL
+};
-static void rx8025_sysfs_unregister(struct device *dev)
-{
- device_remove_file(dev, &dev_attr_clock_adjust_ppb);
-}
+static const struct attribute_group rx8025_attr_group = {
+ .attrs = rx8025_attrs,
+};
static int rx8025_probe(struct i2c_client *client,
const struct i2c_device_id *id)
if (err)
return err;
- rx8025->rtc = devm_rtc_device_register(&client->dev, client->name,
- &rx8025_rtc_ops, THIS_MODULE);
- if (IS_ERR(rx8025->rtc)) {
- dev_err(&client->dev, "unable to register the class device\n");
+ rx8025->rtc = devm_rtc_allocate_device(&client->dev);
+ if (IS_ERR(rx8025->rtc))
return PTR_ERR(rx8025->rtc);
- }
+
+ rx8025->rtc->ops = &rx8025_rtc_ops;
+ rx8025->rtc->range_min = RTC_TIMESTAMP_BEGIN_1900;
+ rx8025->rtc->range_max = RTC_TIMESTAMP_END_2099;
if (client->irq > 0) {
dev_info(&client->dev, "IRQ %d supplied\n", client->irq);
rx8025_handle_irq,
IRQF_ONESHOT,
"rx8025", client);
- if (err) {
- dev_err(&client->dev, "unable to request IRQ, alarms disabled\n");
- client->irq = 0;
- }
+ if (err)
+ clear_bit(RTC_FEATURE_ALARM, rx8025->rtc->features);
}
rx8025->rtc->max_user_freq = 1;
- /* the rx8025 alarm only supports a minute accuracy */
- rx8025->rtc->uie_unsupported = 1;
+ set_bit(RTC_FEATURE_ALARM_RES_MINUTE, rx8025->rtc->features);
+ clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rx8025->rtc->features);
- err = rx8025_sysfs_register(&client->dev);
- return err;
-}
+ err = rtc_add_group(rx8025->rtc, &rx8025_attr_group);
+ if (err)
+ return err;
-static int rx8025_remove(struct i2c_client *client)
-{
- rx8025_sysfs_unregister(&client->dev);
- return 0;
+ return devm_rtc_register_device(rx8025->rtc);
}
static struct i2c_driver rx8025_driver = {
.name = "rtc-rx8025",
},
.probe = rx8025_probe,
- .remove = rx8025_remove,
.id_table = rx8025_id,
};
alm->time.tm_min, alm->time.tm_hour, alm->time.tm_mday,
alm->time.tm_mon, alm->time.tm_year, alm->time.tm_wday);
- if (alm->time.tm_sec != 0)
- dev_warn(&client->dev, "Alarms are only supported on a per minute basis!\n");
-
/* disable interrupt (which deasserts the irq line) */
err = s35390a_set_reg(s35390a, S35390A_CMD_STATUS2, &sts, sizeof(sts));
if (err < 0)
s35390a->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
s35390a->rtc->range_max = RTC_TIMESTAMP_END_2099;
- /* supports per-minute alarms only, therefore set uie_unsupported */
- s35390a->rtc->uie_unsupported = 1;
+ set_bit(RTC_FEATURE_ALARM_RES_MINUTE, s35390a->rtc->features);
+ clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, s35390a->rtc->features );
if (status1 & S35390A_FLAG_INT2)
rtc_update_irq(s35390a->rtc, 1, RTC_AF);
return ret;
}
-/* Time read/write */
-static int s3c_rtc_gettime(struct device *dev, struct rtc_time *rtc_tm)
+/* Read time from RTC and convert it from BCD */
+static int s3c_rtc_read_time(struct s3c_rtc *info, struct rtc_time *tm)
{
- struct s3c_rtc *info = dev_get_drvdata(dev);
unsigned int have_retried = 0;
int ret;
return ret;
retry_get_time:
- rtc_tm->tm_min = readb(info->base + S3C2410_RTCMIN);
- rtc_tm->tm_hour = readb(info->base + S3C2410_RTCHOUR);
- rtc_tm->tm_mday = readb(info->base + S3C2410_RTCDATE);
- rtc_tm->tm_mon = readb(info->base + S3C2410_RTCMON);
- rtc_tm->tm_year = readb(info->base + S3C2410_RTCYEAR);
- rtc_tm->tm_sec = readb(info->base + S3C2410_RTCSEC);
-
- /* the only way to work out whether the system was mid-update
+ tm->tm_min = readb(info->base + S3C2410_RTCMIN);
+ tm->tm_hour = readb(info->base + S3C2410_RTCHOUR);
+ tm->tm_mday = readb(info->base + S3C2410_RTCDATE);
+ tm->tm_mon = readb(info->base + S3C2410_RTCMON);
+ tm->tm_year = readb(info->base + S3C2410_RTCYEAR);
+ tm->tm_sec = readb(info->base + S3C2410_RTCSEC);
+
+ /*
+ * The only way to work out whether the system was mid-update
* when we read it is to check the second counter, and if it
* is zero, then we re-try the entire read
*/
-
- if (rtc_tm->tm_sec == 0 && !have_retried) {
+ if (tm->tm_sec == 0 && !have_retried) {
have_retried = 1;
goto retry_get_time;
}
- rtc_tm->tm_sec = bcd2bin(rtc_tm->tm_sec);
- rtc_tm->tm_min = bcd2bin(rtc_tm->tm_min);
- rtc_tm->tm_hour = bcd2bin(rtc_tm->tm_hour);
- rtc_tm->tm_mday = bcd2bin(rtc_tm->tm_mday);
- rtc_tm->tm_mon = bcd2bin(rtc_tm->tm_mon);
- rtc_tm->tm_year = bcd2bin(rtc_tm->tm_year);
-
s3c_rtc_disable_clk(info);
- rtc_tm->tm_year += 100;
- rtc_tm->tm_mon -= 1;
+ tm->tm_sec = bcd2bin(tm->tm_sec);
+ tm->tm_min = bcd2bin(tm->tm_min);
+ tm->tm_hour = bcd2bin(tm->tm_hour);
+ tm->tm_mday = bcd2bin(tm->tm_mday);
+ tm->tm_mon = bcd2bin(tm->tm_mon);
+ tm->tm_year = bcd2bin(tm->tm_year);
- dev_dbg(dev, "read time %ptR\n", rtc_tm);
return 0;
}
-static int s3c_rtc_settime(struct device *dev, struct rtc_time *tm)
+/* Convert time to BCD and write it to RTC */
+static int s3c_rtc_write_time(struct s3c_rtc *info, const struct rtc_time *tm)
{
- struct s3c_rtc *info = dev_get_drvdata(dev);
- int year = tm->tm_year - 100;
int ret;
- dev_dbg(dev, "set time %ptR\n", tm);
-
- /* we get around y2k by simply not supporting it */
-
- if (year < 0 || year >= 100) {
- dev_err(dev, "rtc only supports 100 years\n");
- return -EINVAL;
- }
-
ret = s3c_rtc_enable_clk(info);
if (ret)
return ret;
writeb(bin2bcd(tm->tm_min), info->base + S3C2410_RTCMIN);
writeb(bin2bcd(tm->tm_hour), info->base + S3C2410_RTCHOUR);
writeb(bin2bcd(tm->tm_mday), info->base + S3C2410_RTCDATE);
- writeb(bin2bcd(tm->tm_mon + 1), info->base + S3C2410_RTCMON);
- writeb(bin2bcd(year), info->base + S3C2410_RTCYEAR);
+ writeb(bin2bcd(tm->tm_mon), info->base + S3C2410_RTCMON);
+ writeb(bin2bcd(tm->tm_year), info->base + S3C2410_RTCYEAR);
s3c_rtc_disable_clk(info);
return 0;
}
+static int s3c_rtc_gettime(struct device *dev, struct rtc_time *tm)
+{
+ struct s3c_rtc *info = dev_get_drvdata(dev);
+ int ret;
+
+ ret = s3c_rtc_read_time(info, tm);
+ if (ret)
+ return ret;
+
+ /* Convert internal representation to actual date/time */
+ tm->tm_year += 100;
+ tm->tm_mon -= 1;
+
+ dev_dbg(dev, "read time %ptR\n", tm);
+ return 0;
+}
+
+static int s3c_rtc_settime(struct device *dev, struct rtc_time *tm)
+{
+ struct s3c_rtc *info = dev_get_drvdata(dev);
+ struct rtc_time rtc_tm = *tm;
+
+ dev_dbg(dev, "set time %ptR\n", tm);
+
+ /*
+ * Convert actual date/time to internal representation.
+ * We get around Y2K by simply not supporting it.
+ */
+ rtc_tm.tm_year -= 100;
+ rtc_tm.tm_mon += 1;
+
+ return s3c_rtc_write_time(info, &rtc_tm);
+}
+
static int s3c_rtc_getalarm(struct device *dev, struct rtc_wkalrm *alrm)
{
struct s3c_rtc *info = dev_get_drvdata(dev);
device_init_wakeup(&pdev->dev, 1);
- /* register RTC and exit */
- info->rtc = devm_rtc_device_register(&pdev->dev, "s3c", &s3c_rtcops,
- THIS_MODULE);
+ info->rtc = devm_rtc_allocate_device(&pdev->dev);
if (IS_ERR(info->rtc)) {
- dev_err(&pdev->dev, "cannot attach rtc\n");
ret = PTR_ERR(info->rtc);
goto err_nortc;
}
+ info->rtc->ops = &s3c_rtcops;
+ info->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
+ info->rtc->range_max = RTC_TIMESTAMP_END_2099;
+
+ ret = devm_rtc_register_device(info->rtc);
+ if (ret)
+ goto err_nortc;
+
ret = devm_request_irq(&pdev->dev, info->irq_alarm, s3c_rtc_alarmirq,
0, "s3c2410-rtc alarm", info);
if (ret) {
MODULE_AUTHOR("Sangbeom Kim <sbkim73@samsung.com>");
MODULE_DESCRIPTION("Samsung S5M/S2MPS14 RTC driver");
MODULE_LICENSE("GPL");
-MODULE_ALIAS("platform:s5m-rtc");
struct sun6i_rtc_dev *chip = sun6i_rtc;
int ret;
- if (!chip)
- return -ENODEV;
+ if (!chip) {
+ chip = devm_kzalloc(&pdev->dev, sizeof(*chip), GFP_KERNEL);
+ if (!chip)
+ return -ENOMEM;
+
+ spin_lock_init(&chip->lock);
+
+ chip->base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(chip->base))
+ return PTR_ERR(chip->base);
+ }
platform_set_drvdata(pdev, chip);
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * rtc-tps80031.c -- TI TPS80031/TPS80032 RTC driver
- *
- * RTC driver for TI TPS80031/TPS80032 Fully Integrated
- * Power Management with Power Path and Battery Charger
- *
- * Copyright (c) 2012, NVIDIA Corporation.
- *
- * Author: Laxman Dewangan <ldewangan@nvidia.com>
- */
-
-#include <linux/bcd.h>
-#include <linux/device.h>
-#include <linux/err.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/mfd/tps80031.h>
-#include <linux/platform_device.h>
-#include <linux/pm.h>
-#include <linux/rtc.h>
-#include <linux/slab.h>
-
-#define ENABLE_ALARM_INT 0x08
-#define ALARM_INT_STATUS 0x40
-
-/**
- * Setting bit to 1 in STOP_RTC will run the RTC and
- * setting this bit to 0 will freeze RTC.
- */
-#define STOP_RTC 0x1
-
-/* Power on reset Values of RTC registers */
-#define TPS80031_RTC_POR_YEAR 0
-#define TPS80031_RTC_POR_MONTH 1
-#define TPS80031_RTC_POR_DAY 1
-
-/* Numbers of registers for time and alarms */
-#define TPS80031_RTC_TIME_NUM_REGS 7
-#define TPS80031_RTC_ALARM_NUM_REGS 6
-
-/**
- * PMU RTC have only 2 nibbles to store year information, so using an
- * offset of 100 to set the base year as 2000 for our driver.
- */
-#define RTC_YEAR_OFFSET 100
-
-struct tps80031_rtc {
- struct rtc_device *rtc;
- int irq;
-};
-
-static int tps80031_rtc_read_time(struct device *dev, struct rtc_time *tm)
-{
- u8 buff[TPS80031_RTC_TIME_NUM_REGS];
- int ret;
-
- ret = tps80031_reads(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_SECONDS_REG, TPS80031_RTC_TIME_NUM_REGS, buff);
- if (ret < 0) {
- dev_err(dev, "reading RTC_SECONDS_REG failed, err = %d\n", ret);
- return ret;
- }
-
- tm->tm_sec = bcd2bin(buff[0]);
- tm->tm_min = bcd2bin(buff[1]);
- tm->tm_hour = bcd2bin(buff[2]);
- tm->tm_mday = bcd2bin(buff[3]);
- tm->tm_mon = bcd2bin(buff[4]) - 1;
- tm->tm_year = bcd2bin(buff[5]) + RTC_YEAR_OFFSET;
- tm->tm_wday = bcd2bin(buff[6]);
- return 0;
-}
-
-static int tps80031_rtc_set_time(struct device *dev, struct rtc_time *tm)
-{
- u8 buff[7];
- int ret;
-
- buff[0] = bin2bcd(tm->tm_sec);
- buff[1] = bin2bcd(tm->tm_min);
- buff[2] = bin2bcd(tm->tm_hour);
- buff[3] = bin2bcd(tm->tm_mday);
- buff[4] = bin2bcd(tm->tm_mon + 1);
- buff[5] = bin2bcd(tm->tm_year % RTC_YEAR_OFFSET);
- buff[6] = bin2bcd(tm->tm_wday);
-
- /* Stop RTC while updating the RTC time registers */
- ret = tps80031_clr_bits(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_CTRL_REG, STOP_RTC);
- if (ret < 0) {
- dev_err(dev->parent, "Stop RTC failed, err = %d\n", ret);
- return ret;
- }
-
- ret = tps80031_writes(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_SECONDS_REG,
- TPS80031_RTC_TIME_NUM_REGS, buff);
- if (ret < 0) {
- dev_err(dev, "writing RTC_SECONDS_REG failed, err %d\n", ret);
- return ret;
- }
-
- ret = tps80031_set_bits(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_CTRL_REG, STOP_RTC);
- if (ret < 0)
- dev_err(dev->parent, "Start RTC failed, err = %d\n", ret);
- return ret;
-}
-
-static int tps80031_rtc_alarm_irq_enable(struct device *dev,
- unsigned int enable)
-{
- int ret;
-
- if (enable)
- ret = tps80031_set_bits(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_INTERRUPTS_REG, ENABLE_ALARM_INT);
- else
- ret = tps80031_clr_bits(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_INTERRUPTS_REG, ENABLE_ALARM_INT);
- if (ret < 0) {
- dev_err(dev, "Update on RTC_INT failed, err = %d\n", ret);
- return ret;
- }
- return 0;
-}
-
-static int tps80031_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
-{
- u8 buff[TPS80031_RTC_ALARM_NUM_REGS];
- int ret;
-
- buff[0] = bin2bcd(alrm->time.tm_sec);
- buff[1] = bin2bcd(alrm->time.tm_min);
- buff[2] = bin2bcd(alrm->time.tm_hour);
- buff[3] = bin2bcd(alrm->time.tm_mday);
- buff[4] = bin2bcd(alrm->time.tm_mon + 1);
- buff[5] = bin2bcd(alrm->time.tm_year % RTC_YEAR_OFFSET);
- ret = tps80031_writes(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_ALARM_SECONDS_REG,
- TPS80031_RTC_ALARM_NUM_REGS, buff);
- if (ret < 0) {
- dev_err(dev, "Writing RTC_ALARM failed, err %d\n", ret);
- return ret;
- }
- return tps80031_rtc_alarm_irq_enable(dev, alrm->enabled);
-}
-
-static int tps80031_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
-{
- u8 buff[6];
- int ret;
-
- ret = tps80031_reads(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_ALARM_SECONDS_REG,
- TPS80031_RTC_ALARM_NUM_REGS, buff);
- if (ret < 0) {
- dev_err(dev->parent,
- "reading RTC_ALARM failed, err = %d\n", ret);
- return ret;
- }
-
- alrm->time.tm_sec = bcd2bin(buff[0]);
- alrm->time.tm_min = bcd2bin(buff[1]);
- alrm->time.tm_hour = bcd2bin(buff[2]);
- alrm->time.tm_mday = bcd2bin(buff[3]);
- alrm->time.tm_mon = bcd2bin(buff[4]) - 1;
- alrm->time.tm_year = bcd2bin(buff[5]) + RTC_YEAR_OFFSET;
- return 0;
-}
-
-static int clear_alarm_int_status(struct device *dev, struct tps80031_rtc *rtc)
-{
- int ret;
- u8 buf;
-
- /**
- * As per datasheet, A dummy read of this RTC_STATUS_REG register
- * is necessary before each I2C read in order to update the status
- * register value.
- */
- ret = tps80031_read(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_STATUS_REG, &buf);
- if (ret < 0) {
- dev_err(dev, "reading RTC_STATUS failed. err = %d\n", ret);
- return ret;
- }
-
- /* clear Alarm status bits.*/
- ret = tps80031_set_bits(dev->parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_STATUS_REG, ALARM_INT_STATUS);
- if (ret < 0) {
- dev_err(dev, "clear Alarm INT failed, err = %d\n", ret);
- return ret;
- }
- return 0;
-}
-
-static irqreturn_t tps80031_rtc_irq(int irq, void *data)
-{
- struct device *dev = data;
- struct tps80031_rtc *rtc = dev_get_drvdata(dev);
- int ret;
-
- ret = clear_alarm_int_status(dev, rtc);
- if (ret < 0)
- return ret;
-
- rtc_update_irq(rtc->rtc, 1, RTC_IRQF | RTC_AF);
- return IRQ_HANDLED;
-}
-
-static const struct rtc_class_ops tps80031_rtc_ops = {
- .read_time = tps80031_rtc_read_time,
- .set_time = tps80031_rtc_set_time,
- .set_alarm = tps80031_rtc_set_alarm,
- .read_alarm = tps80031_rtc_read_alarm,
- .alarm_irq_enable = tps80031_rtc_alarm_irq_enable,
-};
-
-static int tps80031_rtc_probe(struct platform_device *pdev)
-{
- struct tps80031_rtc *rtc;
- struct rtc_time tm;
- int ret;
-
- rtc = devm_kzalloc(&pdev->dev, sizeof(*rtc), GFP_KERNEL);
- if (!rtc)
- return -ENOMEM;
-
- rtc->irq = platform_get_irq(pdev, 0);
- platform_set_drvdata(pdev, rtc);
-
- /* Start RTC */
- ret = tps80031_set_bits(pdev->dev.parent, TPS80031_SLAVE_ID1,
- TPS80031_RTC_CTRL_REG, STOP_RTC);
- if (ret < 0) {
- dev_err(&pdev->dev, "failed to start RTC. err = %d\n", ret);
- return ret;
- }
-
- /* If RTC have POR values, set time 01:01:2000 */
- tps80031_rtc_read_time(&pdev->dev, &tm);
- if ((tm.tm_year == RTC_YEAR_OFFSET + TPS80031_RTC_POR_YEAR) &&
- (tm.tm_mon == (TPS80031_RTC_POR_MONTH - 1)) &&
- (tm.tm_mday == TPS80031_RTC_POR_DAY)) {
- tm.tm_year = 2000;
- tm.tm_mday = 1;
- tm.tm_mon = 1;
- ret = tps80031_rtc_set_time(&pdev->dev, &tm);
- if (ret < 0) {
- dev_err(&pdev->dev,
- "RTC set time failed, err = %d\n", ret);
- return ret;
- }
- }
-
- /* Clear alarm intretupt status if it is there */
- ret = clear_alarm_int_status(&pdev->dev, rtc);
- if (ret < 0) {
- dev_err(&pdev->dev, "Clear alarm int failed, err = %d\n", ret);
- return ret;
- }
-
- rtc->rtc = devm_rtc_device_register(&pdev->dev, pdev->name,
- &tps80031_rtc_ops, THIS_MODULE);
- if (IS_ERR(rtc->rtc)) {
- ret = PTR_ERR(rtc->rtc);
- dev_err(&pdev->dev, "RTC registration failed, err %d\n", ret);
- return ret;
- }
-
- ret = devm_request_threaded_irq(&pdev->dev, rtc->irq, NULL,
- tps80031_rtc_irq,
- IRQF_ONESHOT,
- dev_name(&pdev->dev), rtc);
- if (ret < 0) {
- dev_err(&pdev->dev, "request IRQ:%d failed, err = %d\n",
- rtc->irq, ret);
- return ret;
- }
- device_set_wakeup_capable(&pdev->dev, 1);
- return 0;
-}
-
-#ifdef CONFIG_PM_SLEEP
-static int tps80031_rtc_suspend(struct device *dev)
-{
- struct tps80031_rtc *rtc = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- enable_irq_wake(rtc->irq);
- return 0;
-}
-
-static int tps80031_rtc_resume(struct device *dev)
-{
- struct tps80031_rtc *rtc = dev_get_drvdata(dev);
-
- if (device_may_wakeup(dev))
- disable_irq_wake(rtc->irq);
- return 0;
-};
-#endif
-
-static SIMPLE_DEV_PM_OPS(tps80031_pm_ops, tps80031_rtc_suspend,
- tps80031_rtc_resume);
-
-static struct platform_driver tps80031_rtc_driver = {
- .driver = {
- .name = "tps80031-rtc",
- .pm = &tps80031_pm_ops,
- },
- .probe = tps80031_rtc_probe,
-};
-
-module_platform_driver(tps80031_rtc_driver);
-
-MODULE_ALIAS("platform:tps80031-rtc");
-MODULE_DESCRIPTION("TI TPS80031/TPS80032 RTC driver");
-MODULE_AUTHOR("Laxman Dewangan <ldewangan@nvidia.com>");
-MODULE_LICENSE("GPL v2");
ff_flag = (devmap->features & DASD_FEATURE_FAILFAST) != 0;
else
ff_flag = (DASD_FEATURE_DEFAULT & DASD_FEATURE_FAILFAST) != 0;
- return snprintf(buf, PAGE_SIZE, ff_flag ? "1\n" : "0\n");
+ return sysfs_emit(buf, ff_flag ? "1\n" : "0\n");
}
static ssize_t dasd_ff_store(struct device *dev, struct device_attribute *attr,
spin_unlock(&dasd_devmap_lock);
out:
- return snprintf(buf, PAGE_SIZE, ro_flag ? "1\n" : "0\n");
+ return sysfs_emit(buf, ro_flag ? "1\n" : "0\n");
}
static ssize_t
erplog = (devmap->features & DASD_FEATURE_ERPLOG) != 0;
else
erplog = (DASD_FEATURE_DEFAULT & DASD_FEATURE_ERPLOG) != 0;
- return snprintf(buf, PAGE_SIZE, erplog ? "1\n" : "0\n");
+ return sysfs_emit(buf, erplog ? "1\n" : "0\n");
}
static ssize_t
dasd_put_device(device);
goto out;
} else {
- len = snprintf(buf, PAGE_SIZE, "%s\n",
- device->discipline->name);
+ len = sysfs_emit(buf, "%s\n",
+ device->discipline->name);
dasd_put_device(device);
return len;
}
out:
- len = snprintf(buf, PAGE_SIZE, "none\n");
+ len = sysfs_emit(buf, "none\n");
return len;
}
if (!IS_ERR(device)) {
switch (device->state) {
case DASD_STATE_NEW:
- len = snprintf(buf, PAGE_SIZE, "new\n");
+ len = sysfs_emit(buf, "new\n");
break;
case DASD_STATE_KNOWN:
- len = snprintf(buf, PAGE_SIZE, "detected\n");
+ len = sysfs_emit(buf, "detected\n");
break;
case DASD_STATE_BASIC:
- len = snprintf(buf, PAGE_SIZE, "basic\n");
+ len = sysfs_emit(buf, "basic\n");
break;
case DASD_STATE_UNFMT:
- len = snprintf(buf, PAGE_SIZE, "unformatted\n");
+ len = sysfs_emit(buf, "unformatted\n");
break;
case DASD_STATE_READY:
- len = snprintf(buf, PAGE_SIZE, "ready\n");
+ len = sysfs_emit(buf, "ready\n");
break;
case DASD_STATE_ONLINE:
- len = snprintf(buf, PAGE_SIZE, "online\n");
+ len = sysfs_emit(buf, "online\n");
break;
default:
- len = snprintf(buf, PAGE_SIZE, "no stat\n");
+ len = sysfs_emit(buf, "no stat\n");
break;
}
dasd_put_device(device);
} else
- len = snprintf(buf, PAGE_SIZE, "unknown\n");
+ len = sysfs_emit(buf, "unknown\n");
return len;
}
device = dasd_device_from_cdev(to_ccwdev(dev));
vendor = "";
if (IS_ERR(device))
- return snprintf(buf, PAGE_SIZE, "%s\n", vendor);
+ return sysfs_emit(buf, "%s\n", vendor);
if (device->discipline && device->discipline->get_uid &&
!device->discipline->get_uid(device, &uid))
dasd_put_device(device);
- return snprintf(buf, PAGE_SIZE, "%s\n", vendor);
+ return sysfs_emit(buf, "%s\n", vendor);
}
static DEVICE_ATTR(vendor, 0444, dasd_vendor_show, NULL);
device = dasd_device_from_cdev(to_ccwdev(dev));
uid_string[0] = 0;
if (IS_ERR(device))
- return snprintf(buf, PAGE_SIZE, "%s\n", uid_string);
+ return sysfs_emit(buf, "%s\n", uid_string);
if (device->discipline && device->discipline->get_uid &&
!device->discipline->get_uid(device, &uid)) {
}
dasd_put_device(device);
- return snprintf(buf, PAGE_SIZE, "%s\n", uid_string);
+ return sysfs_emit(buf, "%s\n", uid_string);
}
static DEVICE_ATTR(uid, 0444, dasd_uid_show, NULL);
eer_flag = dasd_eer_enabled(devmap->device);
else
eer_flag = 0;
- return snprintf(buf, PAGE_SIZE, eer_flag ? "1\n" : "0\n");
+ return sysfs_emit(buf, eer_flag ? "1\n" : "0\n");
}
static ssize_t
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
- len = snprintf(buf, PAGE_SIZE, "%lu\n", device->default_expires);
+ len = sysfs_emit(buf, "%lu\n", device->default_expires);
dasd_put_device(device);
return len;
}
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
- len = snprintf(buf, PAGE_SIZE, "%lu\n", device->default_retries);
+ len = sysfs_emit(buf, "%lu\n", device->default_retries);
dasd_put_device(device);
return len;
}
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
- len = snprintf(buf, PAGE_SIZE, "%lu\n", device->blk_timeout);
+ len = sysfs_emit(buf, "%lu\n", device->blk_timeout);
dasd_put_device(device);
return len;
}
return -ENODEV;
if (!device->discipline || !device->discipline->hpf_enabled) {
dasd_put_device(device);
- return snprintf(buf, PAGE_SIZE, "%d\n", dasd_nofcx);
+ return sysfs_emit(buf, "%d\n", dasd_nofcx);
}
hpf = device->discipline->hpf_enabled(device);
dasd_put_device(device);
- return snprintf(buf, PAGE_SIZE, "%d\n", hpf);
+ return sysfs_emit(buf, "%d\n", hpf);
}
static DEVICE_ATTR(hpf, 0444, dasd_hpf_show, NULL);
devmap = dasd_find_busid(dev_name(dev));
if (IS_ERR(devmap)) {
- rc = snprintf(buf, PAGE_SIZE, "ignore\n");
+ rc = sysfs_emit(buf, "ignore\n");
} else {
spin_lock(&dasd_devmap_lock);
if (devmap->features & DASD_FEATURE_FAILONSLCK)
- rc = snprintf(buf, PAGE_SIZE, "fail\n");
+ rc = sysfs_emit(buf, "fail\n");
else
- rc = snprintf(buf, PAGE_SIZE, "ignore\n");
+ rc = sysfs_emit(buf, "ignore\n");
spin_unlock(&dasd_devmap_lock);
}
return rc;
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
- return snprintf(buf, PAGE_SIZE, "none\n");
+ return sysfs_emit(buf, "none\n");
if (test_bit(DASD_FLAG_IS_RESERVED, &device->flags))
- rc = snprintf(buf, PAGE_SIZE, "reserved\n");
+ rc = sysfs_emit(buf, "reserved\n");
else if (test_bit(DASD_FLAG_LOCK_STOLEN, &device->flags))
- rc = snprintf(buf, PAGE_SIZE, "lost\n");
+ rc = sysfs_emit(buf, "lost\n");
else
- rc = snprintf(buf, PAGE_SIZE, "none\n");
+ rc = sysfs_emit(buf, "none\n");
dasd_put_device(device);
return rc;
}
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
- len = snprintf(buf, PAGE_SIZE, "%lu\n", device->path_thrhld);
+ len = sysfs_emit(buf, "%lu\n", device->path_thrhld);
dasd_put_device(device);
return len;
}
else
flag = (DASD_FEATURE_DEFAULT &
DASD_FEATURE_PATH_AUTODISABLE) != 0;
- return snprintf(buf, PAGE_SIZE, flag ? "1\n" : "0\n");
+ return sysfs_emit(buf, flag ? "1\n" : "0\n");
}
static ssize_t
device = dasd_device_from_cdev(to_ccwdev(dev));
if (IS_ERR(device))
return -ENODEV;
- len = snprintf(buf, PAGE_SIZE, "%lu\n", device->path_interval);
+ len = sysfs_emit(buf, "%lu\n", device->path_interval);
dasd_put_device(device);
return len;
}
return -ENODEV;
fc_sec = dasd_path_get_fcs_device(device);
if (fc_sec == -EINVAL)
- rc = snprintf(buf, PAGE_SIZE, "Inconsistent\n");
+ rc = sysfs_emit(buf, "Inconsistent\n");
else
- rc = snprintf(buf, PAGE_SIZE, "%s\n", dasd_path_get_fcs_str(fc_sec));
+ rc = sysfs_emit(buf, "%s\n", dasd_path_get_fcs_str(fc_sec));
dasd_put_device(device);
return rc;
struct dasd_path *path = to_dasd_path(kobj);
unsigned int fc_sec = path->fc_security;
- return snprintf(buf, PAGE_SIZE, "%s\n", dasd_path_get_fcs_str(fc_sec));
+ return sysfs_emit(buf, "%s\n", dasd_path_get_fcs_str(fc_sec));
}
static struct kobj_attribute path_fcs_attribute =
val = _func(device); \
dasd_put_device(device); \
\
- return snprintf(buf, PAGE_SIZE, "%d\n", val); \
+ return sysfs_emit(buf, "%d\n", val); \
} \
static DEVICE_ATTR(_name, 0444, dasd_##_name##_show, NULL); \
{
struct gendisk *gdp;
struct dasd_device *base;
- int len;
+ int len, rc;
/* Make sure the minor for this device exists. */
base = block->base;
dasd_add_link_to_gendisk(gdp, base);
block->gdp = gdp;
set_capacity(block->gdp, 0);
- device_add_disk(&base->cdev->dev, block->gdp, NULL);
+
+ rc = device_add_disk(&base->cdev->dev, block->gdp, NULL);
+ if (rc) {
+ dasd_gendisk_free(block);
+ return rc;
+ }
+
return 0;
}
}
get_device(&dev_info->dev);
- device_add_disk(&dev_info->dev, dev_info->gd, NULL);
+ rc = device_add_disk(&dev_info->dev, dev_info->gd, NULL);
+ if (rc)
+ goto out_dax;
switch (dev_info->segment_type) {
case SEG_TYPE_SR:
rc = count;
goto out;
+out_dax:
+ put_device(&dev_info->dev);
+ kill_dax(dev_info->dax_dev);
+ put_dax(dev_info->dax_dev);
put_dev:
list_del(&dev_info->lh);
blk_cleanup_disk(dev_info->gd);
/* 512 byte sectors */
set_capacity(bdev->gendisk, scmdev->size >> 9);
- device_add_disk(&scmdev->dev, bdev->gendisk, NULL);
+ ret = device_add_disk(&scmdev->dev, bdev->gendisk, NULL);
+ if (ret)
+ goto out_cleanup_disk;
+
return 0;
+out_cleanup_disk:
+ blk_cleanup_disk(bdev->gendisk);
out_tag:
blk_mq_free_tag_set(&bdev->tag_set);
out:
static ssize_t
raw3270_model_show(struct device *dev, struct device_attribute *attr, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%i\n",
- ((struct raw3270 *) dev_get_drvdata(dev))->model);
+ return sysfs_emit(buf, "%i\n",
+ ((struct raw3270 *)dev_get_drvdata(dev))->model);
}
static DEVICE_ATTR(model, 0444, raw3270_model_show, NULL);
static ssize_t
raw3270_rows_show(struct device *dev, struct device_attribute *attr, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%i\n",
- ((struct raw3270 *) dev_get_drvdata(dev))->rows);
+ return sysfs_emit(buf, "%i\n",
+ ((struct raw3270 *)dev_get_drvdata(dev))->rows);
}
static DEVICE_ATTR(rows, 0444, raw3270_rows_show, NULL);
static ssize_t
raw3270_columns_show(struct device *dev, struct device_attribute *attr, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%i\n",
- ((struct raw3270 *) dev_get_drvdata(dev))->cols);
+ return sysfs_emit(buf, "%i\n",
+ ((struct raw3270 *)dev_get_drvdata(dev))->cols);
}
static DEVICE_ATTR(columns, 0444, raw3270_columns_show, NULL);
summary.timeout = (u16)req->queue_timeout;
summary.start_count = (u16)req->start_count;
- sclp_trace(prio, id, (u32)(addr_t)sccb, summary.b, err);
+ sclp_trace(prio, id, __pa(sccb), summary.b, err);
}
static inline void sclp_trace_register(int prio, char *id, u32 a, u64 b,
}
/* RQAD: Request was added (a=sccb, b=caller) */
- sclp_trace(2, "RQAD", (u32)(addr_t)req->sccb, _RET_IP_, false);
+ sclp_trace(2, "RQAD", __pa(req->sccb), _RET_IP_, false);
req->status = SCLP_REQ_QUEUED;
req->start_count = 0;
list_for_each(l, &sclp_req_queue) {
req = list_entry(l, struct sclp_req, list);
- if (sccb == (u32) (addr_t) req->sccb)
- return req;
+ if (sccb == __pa(req->sccb))
+ return req;
}
return NULL;
}
static bool ok_response(u32 sccb_int, sclp_cmdw_t cmd)
{
- struct sccb_header *sccb = (struct sccb_header *)(addr_t)sccb_int;
+ struct sccb_header *sccb = (struct sccb_header *)__va(sccb_int);
struct evbuf_header *evbuf;
u16 response;
/* INT: Interrupt received (a=intparm, b=cmd) */
sclp_trace_sccb(0, "INT", param32, active_cmd, active_cmd,
- (struct sccb_header *)(addr_t)finished_sccb,
+ (struct sccb_header *)__va(finished_sccb),
!ok_response(finished_sccb, active_cmd));
if (finished_sccb) {
/* Is this the interrupt we are waiting for? */
if (finished_sccb == 0)
return;
- if (finished_sccb != (u32) (addr_t) sclp_init_sccb)
+ if (finished_sccb != __pa(sclp_init_sccb))
panic("sclp: unsolicited interrupt for buffer at 0x%x\n",
finished_sccb);
spin_lock(&sclp_lock);
"2:\n"
EX_TABLE(0b, 2b)
EX_TABLE(1b, 2b)
- : "+&d" (cc) : "d" (command), "a" ((unsigned long)sccb)
+ : "+&d" (cc) : "d" (command), "a" (__pa(sccb))
: "cc", "memory");
if (cc == 4)
return -EINVAL;
}
sclp_fill_core_info(info, sccb);
out:
- memblock_free_early((unsigned long)sccb, length);
+ memblock_phys_free((unsigned long)sccb, length);
return rc;
}
sclp.has_linemode = 1;
}
+void __init sclp_early_adjust_va(void)
+{
+ sclp_early_sccb = __va((unsigned long)sclp_early_sccb);
+}
+
void __init sclp_early_detect(void)
{
void *sccb = sclp_early_sccb;
/**
* sclp_ftp_txcb() - Diagnostic Test FTP services SCLP command callback
+ * @req: sclp request
+ * @data: pointer to struct completion
*/
static void sclp_ftp_txcb(struct sclp_req *req, void *data)
{
/**
* sclp_ftp_rxcb() - Diagnostic Test FTP services receiver event callback
+ * @evbuf: pointer to Diagnostic Test (ET7) event buffer
*/
static void sclp_ftp_rxcb(struct evbuf_header *evbuf)
{
/**
* sclp_sd_listener_init() - Initialize a Store Data response listener
+ * @listener: Response listener to initialize
* @id: Event ID to listen for
*
* Initialize a listener for asynchronous Store Data responses. This listener
struct sclp_sd_evbuf *evbuf;
int rc;
- sclp_sd_listener_init(&listener, (u32) (addr_t) sccb);
+ sclp_sd_listener_init(&listener, __pa(sccb));
sclp_sd_listener_add(&listener);
/* Prepare SCCB */
/**
* sclp_sd_file_update_async() - Wrapper for asynchronous update call
* @data: Object to update
+ * @cookie: Unused
*/
static void sclp_sd_file_update_async(void *data, async_cookie_t cookie)
{
/**
* reload_store() - Store function for "reload" sysfs attribute
* @kobj: Kobject of sclp_sd_file object
+ * @attr: Reload attribute
+ * @buf: Data written to sysfs attribute
+ * @count: Count of bytes written
*
* Initiate a reload of the data associated with an sclp_sd_file object.
*/
};
/**
- * data_read() - Read function for "read" sysfs attribute
+ * data_read() - Read function for "data" sysfs attribute
+ * @file: Open file pointer
* @kobj: Kobject of sclp_sd_file object
+ * @attr: Data attribute
* @buffer: Target buffer
* @off: Requested file offset
* @size: Requested number of bytes
}
__initcall(sclp_vt220_tty_init);
+#ifdef CONFIG_SCLP_VT220_CONSOLE
+
static void __sclp_vt220_flush_buffer(void)
{
unsigned long flags;
spin_unlock_irqrestore(&sclp_vt220_lock, flags);
}
-#ifdef CONFIG_SCLP_VT220_CONSOLE
-
static void
sclp_vt220_con_write(struct console *con, const char *buf, unsigned int count)
{
tape_std_assign(struct tape_device *device)
{
int rc;
- struct timer_list timeout;
struct tape_request *request;
request = tape_alloc_request(2, 11);
* So we set up a timeout for this call.
*/
timer_setup(&request->timer, tape_std_assign_timeout, 0);
- mod_timer(&timeout, jiffies + 2 * HZ);
+ mod_timer(&request->timer, jiffies + msecs_to_jiffies(2000));
rc = tape_do_io_interruptible(device, request);
if (status < 0)
return status;
- return snprintf(buf, PAGE_SIZE, "%d\n", status);
+ return sysfs_emit(buf, "%d\n", status);
}
static int cfg_wait_idle(void);
struct subchannel *sch = to_subchannel(dev);
struct pmcw *pmcw = &sch->schib.pmcw;
- if ((pmcw->st == SUBCHANNEL_TYPE_IO ||
- pmcw->st == SUBCHANNEL_TYPE_MSG) && pmcw->dnv)
+ if ((pmcw->st == SUBCHANNEL_TYPE_IO && pmcw->dnv) ||
+ (pmcw->st == SUBCHANNEL_TYPE_MSG && pmcw->w))
return sysfs_emit(buf, "0.%x.%04x\n", sch->schid.ssid,
pmcw->dev);
else
{
struct idset *set = data;
struct subchannel *sch = to_subchannel(dev);
- struct ccw_device *cdev = sch_get_cdev(sch);
+ struct ccw_device *cdev;
- if (cdev && cdev->online)
- idset_sch_del(set, sch->schid);
+ if (sch->st == SUBCHANNEL_TYPE_IO) {
+ cdev = sch_get_cdev(sch);
+ if (cdev && cdev->online)
+ idset_sch_del(set, sch->schid);
+ }
return 0;
}
{
struct ccw_device *cdev = to_ccwdev(dev);
struct ccw_dev_id *id = &cdev->private->dev_id;
+ struct subchannel *sch = to_subchannel(cdev->dev.parent);
spin_lock_irq(cdev->ccwlock);
if (is_blacklisted(id->ssid, id->devno) &&
CIO_MSG_EVENT(3, "ccw: purging 0.%x.%04x\n", id->ssid,
id->devno);
ccw_device_sched_todo(cdev, CDEV_TODO_UNREG);
+ css_sched_sch_todo(sch, SCH_TODO_UNREG);
atomic_set(&cdev->private->onoff, 0);
}
spin_unlock_irq(cdev->ccwlock);
*/
void *ccw_device_dma_zalloc(struct ccw_device *cdev, size_t size)
{
- return cio_gp_dma_zalloc(cdev->private->dma_pool, &cdev->dev, size);
+ void *addr;
+
+ if (!get_device(&cdev->dev))
+ return NULL;
+ addr = cio_gp_dma_zalloc(cdev->private->dma_pool, &cdev->dev, size);
+ if (IS_ERR_OR_NULL(addr))
+ put_device(&cdev->dev);
+ return addr;
}
EXPORT_SYMBOL(ccw_device_dma_zalloc);
void ccw_device_dma_free(struct ccw_device *cdev, void *cpu_addr, size_t size)
{
+ if (!cpu_addr)
+ return;
cio_gp_dma_free(cdev->private->dma_pool, cpu_addr, size);
+ put_device(&cdev->dev);
}
EXPORT_SYMBOL(ccw_device_dma_free);
module_param_named(aqmask, aqm_str, charp, 0440);
MODULE_PARM_DESC(aqmask, "AP bus domain mask.");
+static int ap_useirq = 1;
+module_param_named(useirq, ap_useirq, int, 0440);
+MODULE_PARM_DESC(useirq, "Use interrupt if available, default is 1 (on).");
+
atomic_t ap_max_msg_size = ATOMIC_INIT(AP_DEFAULT_MAX_MSG_SIZE);
EXPORT_SYMBOL(ap_max_msg_size);
if (bound == apqns) {
if (!completion_done(&ap_init_apqn_bindings_complete)) {
complete_all(&ap_init_apqn_bindings_complete);
- AP_DBF(DBF_INFO, "%s complete\n", __func__);
+ AP_DBF_INFO("%s complete\n", __func__);
}
ap_send_bindings_complete_uevent();
}
drvres = to_ap_drv(dev->driver)->flags
& AP_DRIVER_FLAG_DEFAULT;
if (!!devres != !!drvres) {
- AP_DBF_DBG("reprobing queue=%02x.%04x\n",
- card, queue);
+ AP_DBF_DBG("%s reprobing queue=%02x.%04x\n",
+ __func__, card, queue);
rc = device_reprobe(dev);
+ if (rc)
+ AP_DBF_WARN("%s reprobing queue=%02x.%04x failed\n",
+ __func__, card, queue);
}
}
ap_domain_index = domain;
spin_unlock_bh(&ap_domain_lock);
- AP_DBF_INFO("stored new default domain=%d\n", domain);
+ AP_DBF_INFO("%s stored new default domain=%d\n",
+ __func__, domain);
return count;
}
/* < CEX2A is not supported */
if (rawtype < AP_DEVICE_TYPE_CEX2A) {
- AP_DBF_WARN("get_comp_type queue=%02x.%04x unsupported type %d\n",
- AP_QID_CARD(qid), AP_QID_QUEUE(qid), rawtype);
+ AP_DBF_WARN("%s queue=%02x.%04x unsupported type %d\n",
+ __func__, AP_QID_CARD(qid),
+ AP_QID_QUEUE(qid), rawtype);
return 0;
}
/* up to CEX7 known and fully supported */
comp_type = apinfo.cat;
}
if (!comp_type)
- AP_DBF_WARN("get_comp_type queue=%02x.%04x unable to map type %d\n",
- AP_QID_CARD(qid), AP_QID_QUEUE(qid), rawtype);
+ AP_DBF_WARN("%s queue=%02x.%04x unable to map type %d\n",
+ __func__, AP_QID_CARD(qid),
+ AP_QID_QUEUE(qid), rawtype);
else if (comp_type != rawtype)
- AP_DBF_INFO("get_comp_type queue=%02x.%04x map type %d to %d\n",
- AP_QID_CARD(qid), AP_QID_QUEUE(qid),
+ AP_DBF_INFO("%s queue=%02x.%04x map type %d to %d\n",
+ __func__, AP_QID_CARD(qid), AP_QID_QUEUE(qid),
rawtype, comp_type);
return comp_type;
}
aq = dev ? to_ap_queue(dev) : NULL;
if (!ap_test_config_usage_domain(dom)) {
if (dev) {
- AP_DBF_INFO("%s(%d,%d) not in config any more, rm queue device\n",
+ AP_DBF_INFO("%s(%d,%d) not in config anymore, rm queue dev\n",
__func__, ac->id, dom);
device_unregister(dev);
put_device(dev);
/* domain is valid, get info from this APQN */
if (!ap_queue_info(qid, &type, &func, &depth, &ml, &decfg)) {
if (aq) {
- AP_DBF_INFO(
- "%s(%d,%d) ap_queue_info() not successful, rm queue device\n",
- __func__, ac->id, dom);
+ AP_DBF_INFO("%s(%d,%d) queue_info() failed, rm queue dev\n",
+ __func__, ac->id, dom);
device_unregister(dev);
put_device(dev);
}
/* get it and thus adjust reference counter */
get_device(dev);
if (decfg)
- AP_DBF_INFO("%s(%d,%d) new (decfg) queue device created\n",
+ AP_DBF_INFO("%s(%d,%d) new (decfg) queue dev created\n",
__func__, ac->id, dom);
else
- AP_DBF_INFO("%s(%d,%d) new queue device created\n",
+ AP_DBF_INFO("%s(%d,%d) new queue dev created\n",
__func__, ac->id, dom);
goto put_dev_and_continue;
}
aq->last_err_rc = AP_RESPONSE_DECONFIGURED;
}
spin_unlock_bh(&aq->lock);
- AP_DBF_INFO("%s(%d,%d) queue device config off\n",
+ AP_DBF_INFO("%s(%d,%d) queue dev config off\n",
__func__, ac->id, dom);
ap_send_config_uevent(&aq->ap_dev, aq->config);
/* 'receive' pending messages with -EAGAIN */
aq->sm_state = AP_SM_STATE_RESET_START;
}
spin_unlock_bh(&aq->lock);
- AP_DBF_INFO("%s(%d,%d) queue device config on\n",
+ AP_DBF_INFO("%s(%d,%d) queue dev config on\n",
__func__, ac->id, dom);
ap_send_config_uevent(&aq->ap_dev, aq->config);
goto put_dev_and_continue;
ap_flush_queue(aq);
/* re-init (with reset) the queue device */
ap_queue_init_state(aq);
- AP_DBF_INFO("%s(%d,%d) queue device reinit enforced\n",
+ AP_DBF_INFO("%s(%d,%d) queue dev reinit enforced\n",
__func__, ac->id, dom);
goto put_dev_and_continue;
}
/* Adapter not in configuration ? */
if (!ap_test_config_card_id(ap)) {
if (ac) {
- AP_DBF_INFO("%s(%d) ap not in config any more, rm card and queue devices\n",
+ AP_DBF_INFO("%s(%d) ap not in config any more, rm card and queue devs\n",
__func__, ap);
ap_scan_rm_card_dev_and_queue_devs(ac);
put_device(dev);
if (dom > ap_max_domain_id) {
/* Could not find a valid APQN for this adapter */
if (ac) {
- AP_DBF_INFO(
- "%s(%d) no type info (no APQN found), rm card and queue devices\n",
- __func__, ap);
+ AP_DBF_INFO("%s(%d) no type info (no APQN found), rm card and queue devs\n",
+ __func__, ap);
ap_scan_rm_card_dev_and_queue_devs(ac);
put_device(dev);
} else {
if (!type) {
/* No apdater type info available, an unusable adapter */
if (ac) {
- AP_DBF_INFO("%s(%d) no valid type (0) info, rm card and queue devices\n",
+ AP_DBF_INFO("%s(%d) no valid type (0) info, rm card and queue devs\n",
__func__, ap);
ap_scan_rm_card_dev_and_queue_devs(ac);
put_device(dev);
if (ac) {
/* Check APQN against existing card device for changes */
if (ac->raw_hwtype != type) {
- AP_DBF_INFO("%s(%d) hwtype %d changed, rm card and queue devices\n",
+ AP_DBF_INFO("%s(%d) hwtype %d changed, rm card and queue devs\n",
__func__, ap, type);
ap_scan_rm_card_dev_and_queue_devs(ac);
put_device(dev);
ac = NULL;
} else if (ac->functions != func) {
- AP_DBF_INFO("%s(%d) functions 0x%08x changed, rm card and queue devices\n",
+ AP_DBF_INFO("%s(%d) functions 0x%08x changed, rm card and queue devs\n",
__func__, ap, type);
ap_scan_rm_card_dev_and_queue_devs(ac);
put_device(dev);
} else {
if (decfg && ac->config) {
ac->config = false;
- AP_DBF_INFO("%s(%d) card device config off\n",
+ AP_DBF_INFO("%s(%d) card dev config off\n",
__func__, ap);
ap_send_config_uevent(&ac->ap_dev, ac->config);
}
if (!decfg && !ac->config) {
ac->config = true;
- AP_DBF_INFO("%s(%d) card device config on\n",
+ AP_DBF_INFO("%s(%d) card dev config on\n",
__func__, ap);
ap_send_config_uevent(&ac->ap_dev, ac->config);
}
if (ac->maxmsgsize > atomic_read(&ap_max_msg_size)) {
atomic_set(&ap_max_msg_size, ac->maxmsgsize);
AP_DBF_INFO("%s(%d) ap_max_msg_size update to %d byte\n",
- __func__, ap, atomic_read(&ap_max_msg_size));
+ __func__, ap,
+ atomic_read(&ap_max_msg_size));
}
/* Register the new card device with AP bus */
rc = device_register(dev);
/* get it and thus adjust reference counter */
get_device(dev);
if (decfg)
- AP_DBF_INFO("%s(%d) new (decfg) card device type=%d func=0x%08x created\n",
+ AP_DBF_INFO("%s(%d) new (decfg) card dev type=%d func=0x%08x created\n",
__func__, ap, type, func);
else
- AP_DBF_INFO("%s(%d) new card device type=%d func=0x%08x created\n",
+ AP_DBF_INFO("%s(%d) new card dev type=%d func=0x%08x created\n",
__func__, ap, type, func);
}
if (dev)
put_device(dev);
else
- AP_DBF_INFO("no queue device with default domain %d available\n",
- ap_domain_index);
+ AP_DBF_INFO("%s no queue device with default domain %d available\n",
+ __func__, ap_domain_index);
}
if (atomic64_inc_return(&ap_scan_bus_count) == 1) {
- AP_DBF(DBF_DEBUG, "%s init scan complete\n", __func__);
+ AP_DBF_DBG("%s init scan complete\n", __func__);
ap_send_init_scan_done_uevent();
ap_check_bindings_complete();
}
static int __init ap_debug_init(void)
{
- ap_dbf_info = debug_register("ap", 1, 1,
+ ap_dbf_info = debug_register("ap", 2, 1,
DBF_MAX_SPRINTF_ARGS * sizeof(long));
debug_register_view(ap_dbf_info, &debug_sprintf_view);
debug_set_level(ap_dbf_info, DBF_ERR);
}
/* enable interrupts if available */
- if (ap_interrupts_available()) {
+ if (ap_interrupts_available() && ap_useirq) {
rc = register_adapter_interrupt(&ap_airq);
ap_irq_flag = (rc == 0);
}
#define RC2ERR(rc) ((rc) ? DBF_ERR : DBF_INFO)
#define RC2WARN(rc) ((rc) ? DBF_WARN : DBF_INFO)
-#define DBF_MAX_SPRINTF_ARGS 5
+#define DBF_MAX_SPRINTF_ARGS 6
#define AP_DBF(...) \
debug_sprintf_event(ap_dbf_info, ##__VA_ARGS__)
switch (status.response_code) {
case AP_RESPONSE_NORMAL:
aq->queue_count = max_t(int, 0, aq->queue_count - 1);
+ if (!status.queue_empty && !aq->queue_count)
+ aq->queue_count++;
if (aq->queue_count > 0)
mod_timer(&aq->timeout,
jiffies + aq->request_timeout);
if (aq->requestq_count <= 0)
return AP_SM_WAIT_NONE;
+
/* Start the next request on the queue. */
ap_msg = list_entry(aq->requestq.next, struct ap_message, list);
#ifdef CONFIG_ZCRYPT_DEBUG
aq->sm_state = AP_SM_STATE_RESET_WAIT;
return AP_SM_WAIT_TIMEOUT;
case AP_RESPONSE_INVALID_DOMAIN:
- AP_DBF(DBF_WARN, "AP_RESPONSE_INVALID_DOMAIN on NQAP\n");
+ AP_DBF_WARN("%s RESPONSE_INVALID_DOMAIN on NQAP\n", __func__);
fallthrough;
case AP_RESPONSE_MESSAGE_TOO_BIG:
case AP_RESPONSE_REQ_FAC_NOT_INST:
ap_wait(ap_sm_event(aq, AP_SM_EVENT_POLL));
spin_unlock_bh(&aq->lock);
- AP_DBF(DBF_INFO, "reset queue=%02x.%04x triggered by user\n",
- AP_QID_CARD(aq->qid), AP_QID_QUEUE(aq->qid));
+ AP_DBF_INFO("%s reset queue=%02x.%04x triggered by user\n",
+ __func__, AP_QID_CARD(aq->qid), AP_QID_QUEUE(aq->qid));
return count;
}
MODULE_DEVICE_TABLE(vfio_ap, ap_queue_ids);
/**
- * vfio_ap_queue_dev_probe:
+ * vfio_ap_queue_dev_probe: Allocate a vfio_ap_queue structure and associate it
+ * with the device as driver_data.
*
- * Allocate a vfio_ap_queue structure and associate it
- * with the device as driver_data.
+ * @apdev: the AP device being probed
+ *
+ * Return: returns 0 if the probe succeeded; otherwise, returns -ENOMEM if
+ * storage could not be allocated for a vfio_ap_queue object.
*/
static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
{
}
/**
- * vfio_ap_queue_dev_remove:
+ * vfio_ap_queue_dev_remove: Free the associated vfio_ap_queue structure.
+ *
+ * @apdev: the AP device being removed
*
- * Takes the matrix lock to avoid actions on this device while removing
- * Free the associated vfio_ap_queue structure
+ * Takes the matrix lock to avoid actions on this device while doing the remove.
*/
static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
{
* vfio_ap_irq_enable - Enable Interruption for a APQN
*
* @q: the vfio_ap_queue holding AQIC parameters
+ * @isc: the guest ISC to register with the GIB interface
+ * @nib: the notification indicator byte to pin.
*
* Pin the NIB saved in *q
* Register the guest ISC to GIB interface and retrieve the
* assign_domain_store - parses the APQI from @buf and sets the
* corresponding bit in the mediated matrix device's AQM
*
- *
* @dev: the matrix device
* @attr: the mediated matrix device's assign_domain attribute
* @buf: a buffer containing the AP queue index (APQI) of the domain to
* assign_control_domain_store - parses the domain ID from @buf and sets
* the corresponding bit in the mediated matrix device's ADM
*
- *
* @dev: the matrix device
* @attr: the mediated matrix device's assign_control_domain attribute
* @buf: a buffer containing the domain ID to be assigned
* by @matrix_mdev.
*
* @matrix_mdev: a matrix mediated device
+ * @kvm: the pointer to the kvm structure being unset.
*
* Note: The matrix_dev->lock must be taken prior to calling
* this function; however, the lock will be temporarily released while the
#define VFIO_AP_DRV_NAME "vfio_ap"
/**
- * ap_matrix_dev - the AP matrix device structure
+ * struct ap_matrix_dev - Contains the data for the matrix device.
+ *
* @device: generic device structure associated with the AP matrix device
* @available_instances: number of mediated matrix devices that can be created
* @info: the struct containing the output from the PQAP(QCI) instruction
- * mdev_list: the list of mediated matrix devices created
- * lock: mutex for locking the AP matrix device. This lock will be
+ * @mdev_list: the list of mediated matrix devices created
+ * @lock: mutex for locking the AP matrix device. This lock will be
* taken every time we fiddle with state managed by the vfio_ap
* driver, be it using @mdev_list or writing the state of a
* single ap_matrix_mdev device. It's quite coarse but we don't
* expect much contention.
+ * @vfio_ap_drv: the vfio_ap device driver
*/
struct ap_matrix_dev {
struct device device;
extern struct ap_matrix_dev *matrix_dev;
/**
- * The AP matrix is comprised of three bit masks identifying the adapters,
- * queues (domains) and control domains that belong to an AP matrix. The bits i
- * each mask, from least significant to most significant bit, correspond to IDs
- * 0 to 255. When a bit is set, the corresponding ID belongs to the matrix.
+ * struct ap_matrix - matrix of adapters, domains and control domains
*
* @apm_max: max adapter number in @apm
- * @apm identifies the AP adapters in the matrix
+ * @apm: identifies the AP adapters in the matrix
* @aqm_max: max domain number in @aqm
- * @aqm identifies the AP queues (domains) in the matrix
+ * @aqm: identifies the AP queues (domains) in the matrix
* @adm_max: max domain number in @adm
- * @adm identifies the AP control domains in the matrix
+ * @adm: identifies the AP control domains in the matrix
+ *
+ * The AP matrix is comprised of three bit masks identifying the adapters,
+ * queues (domains) and control domains that belong to an AP matrix. The bits in
+ * each mask, from left to right, correspond to IDs 0 to 255. When a bit is set
+ * the corresponding ID belongs to the matrix.
*/
struct ap_matrix {
unsigned long apm_max;
};
/**
- * struct ap_matrix_mdev - the mediated matrix device structure
- * @list: allows the ap_matrix_mdev struct to be added to a list
+ * struct ap_matrix_mdev - Contains the data associated with a matrix mediated
+ * device.
+ * @vdev: the vfio device
+ * @node: allows the ap_matrix_mdev struct to be added to a list
* @matrix: the adapters, usage domains and control domains assigned to the
* mediated matrix device.
* @group_notifier: notifier block used for specifying callback function for
* handling the VFIO_GROUP_NOTIFY_SET_KVM event
+ * @iommu_notifier: notifier block used for specifying callback function for
+ * handling the VFIO_IOMMU_NOTIFY_DMA_UNMAP even
* @kvm: the struct holding guest's state
+ * @pqap_hook: the function pointer to the interception handler for the
+ * PQAP(AQIC) instruction.
+ * @mdev: the mediated device
*/
struct ap_matrix_mdev {
struct vfio_device vdev;
struct mdev_device *mdev;
};
+/**
+ * struct vfio_ap_queue - contains the data associated with a queue bound to the
+ * vfio_ap device driver
+ * @matrix_mdev: the matrix mediated device
+ * @saved_pfn: the guest PFN pinned for the guest
+ * @apqn: the APQN of the AP queue device
+ * @saved_isc: the guest ISC registered with the GIB interface
+ */
struct vfio_ap_queue {
struct ap_matrix_mdev *matrix_mdev;
unsigned long saved_pfn;
atomic_set(&zcrypt_rescan_req, 0);
atomic_inc(&zcrypt_rescan_count);
ap_bus_force_rescan();
- ZCRYPT_DBF(DBF_INFO, "rescan count=%07d\n",
- atomic_inc_return(&zcrypt_rescan_count));
+ ZCRYPT_DBF_INFO("%s rescan count=%07d\n", __func__,
+ atomic_inc_return(&zcrypt_rescan_count));
return 1;
}
return 0;
{
struct zcdn_device *zcdndev = to_zcdn_dev(dev);
- ZCRYPT_DBF(DBF_INFO, "releasing zcdn device %d:%d\n",
- MAJOR(dev->devt), MINOR(dev->devt));
+ ZCRYPT_DBF_INFO("%s releasing zcdn device %d:%d\n",
+ __func__, MAJOR(dev->devt), MINOR(dev->devt));
kfree(zcdndev);
}
goto unlockout;
}
- ZCRYPT_DBF(DBF_INFO, "created zcdn device %d:%d\n",
- MAJOR(devt), MINOR(devt));
+ ZCRYPT_DBF_INFO("%s created zcdn device %d:%d\n",
+ __func__, MAJOR(devt), MINOR(devt));
unlockout:
mutex_unlock(&ap_perms_mutex);
}
if (rc)
- ZCRYPT_DBF(DBF_WARN,
- "ioctl check failed: ioctlnr=0x%04x rc=%d\n",
- ioctlnr, rc);
+ ZCRYPT_DBF_WARN("%s ioctl check failed: ioctlnr=0x%04x rc=%d\n",
+ __func__, ioctlnr, rc);
return rc;
}
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
if (rc) {
- ZCRYPT_DBF(DBF_DEBUG, "ioctl ICARSAMODEXPO rc=%d\n", rc);
+ ZCRYPT_DBF_DBG("ioctl ICARSAMODEXPO rc=%d\n", rc);
return rc;
}
return put_user(mex.outputdatalength, &umex->outputdatalength);
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
if (rc) {
- ZCRYPT_DBF(DBF_DEBUG, "ioctl ICARSACRT rc=%d\n", rc);
+ ZCRYPT_DBF_DBG("ioctl ICARSACRT rc=%d\n", rc);
return rc;
}
return put_user(crt.outputdatalength, &ucrt->outputdatalength);
return -EFAULT;
#ifdef CONFIG_ZCRYPT_DEBUG
- if (xcRB.status & (1U << 31)) {
+ if ((xcRB.status & 0x8000FFFF) == 0x80004649 /* 'FI' */) {
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
tr.fi.cmd = (u16)(xcRB.status >> 16);
}
- xcRB.status &= 0x0000FFFF;
+ xcRB.status = 0;
#endif
do {
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
if (rc)
- ZCRYPT_DBF(DBF_DEBUG, "ioctl ZSENDCPRB rc=%d status=0x%x\n",
- rc, xcRB.status);
+ ZCRYPT_DBF_DBG("ioctl ZSENDCPRB rc=%d status=0x%x\n",
+ rc, xcRB.status);
if (copy_to_user(uxcRB, &xcRB, sizeof(xcRB)))
return -EFAULT;
return rc;
if (rc == -EAGAIN && tr.again_counter >= TRACK_AGAIN_MAX)
rc = -EIO;
if (rc)
- ZCRYPT_DBF(DBF_DEBUG, "ioctl ZSENDEP11CPRB rc=%d\n", rc);
+ ZCRYPT_DBF_DBG("ioctl ZSENDEP11CPRB rc=%d\n", rc);
if (copy_to_user(uxcrb, &xcrb, sizeof(xcrb)))
return -EFAULT;
return rc;
}
/* unknown ioctl number */
default:
- ZCRYPT_DBF(DBF_DEBUG, "unknown ioctl 0x%08x\n", cmd);
+ ZCRYPT_DBF_DBG("unknown ioctl 0x%08x\n", cmd);
return -ENOIOCTLCMD;
}
}
break;
case -ETIME:
/* timeout */
- ZCRYPT_DBF(DBF_WARN,
- "%s ap_wait_init_apqn_bindings_complete() returned with ETIME\n",
- __func__);
+ ZCRYPT_DBF_WARN("%s ap_wait_init_apqn_bindings_complete()=ETIME\n",
+ __func__);
zcrypt_wait_api_state = -ETIME;
break;
default:
/* other failure */
- ZCRYPT_DBF(DBF_DEBUG,
- "%s ap_wait_init_apqn_bindings_complete() failure rc=%d\n",
- __func__, rc);
+ ZCRYPT_DBF_DBG("%s ap_wait_init_apqn_bindings_complete()=%d\n",
+ __func__, rc);
break;
}
break;
int __init zcrypt_debug_init(void)
{
- zcrypt_dbf_info = debug_register("zcrypt", 1, 1,
+ zcrypt_dbf_info = debug_register("zcrypt", 2, 1,
DBF_MAX_SPRINTF_ARGS * sizeof(long));
debug_register_view(zcrypt_dbf_info, &debug_sprintf_view);
debug_set_level(zcrypt_dbf_info, DBF_ERR);
zc->online = online;
id = zc->card->id;
- ZCRYPT_DBF(DBF_INFO, "card=%02x online=%d\n", id, online);
+ ZCRYPT_DBF_INFO("%s card=%02x online=%d\n", __func__, id, online);
ap_send_online_uevent(&ac->ap_dev, online);
zc->online = 1;
- ZCRYPT_DBF(DBF_INFO, "card=%02x register online=1\n", zc->card->id);
+ ZCRYPT_DBF_INFO("%s card=%02x register online=1\n",
+ __func__, zc->card->id);
rc = sysfs_create_group(&zc->card->ap_dev.device.kobj,
&zcrypt_card_attr_group);
*/
void zcrypt_card_unregister(struct zcrypt_card *zc)
{
- ZCRYPT_DBF(DBF_INFO, "card=%02x unregister\n", zc->card->id);
+ ZCRYPT_DBF_INFO("%s card=%02x unregister\n",
+ __func__, zc->card->id);
spin_lock(&zcrypt_list_lock);
list_del_init(&zc->list);
#define RC2ERR(rc) ((rc) ? DBF_ERR : DBF_INFO)
#define RC2WARN(rc) ((rc) ? DBF_WARN : DBF_INFO)
-#define DBF_MAX_SPRINTF_ARGS 5
+#define DBF_MAX_SPRINTF_ARGS 6
#define ZCRYPT_DBF(...) \
debug_sprintf_event(zcrypt_dbf_info, ##__VA_ARGS__)
case REP88_ERROR_MESSAGE_MALFORMD: /* 0x22 */
case REP88_ERROR_KEY_TYPE: /* 0x34 */
/* RY indicates malformed request */
- ZCRYPT_DBF(DBF_WARN,
- "dev=%02x.%04x RY=0x%02x => rc=EINVAL\n",
- card, queue, ehdr->reply_code);
+ ZCRYPT_DBF_WARN("%s dev=%02x.%04x RY=0x%02x => rc=EINVAL\n",
+ __func__, card, queue, ehdr->reply_code);
return -EINVAL;
case REP82_ERROR_MACHINE_FAILURE: /* 0x10 */
case REP82_ERROR_MESSAGE_TYPE: /* 0x20 */
} __packed * head = reply->msg;
unsigned int apfs = *((u32 *)head->fmt2.apfs);
- ZCRYPT_DBF(DBF_WARN,
- "dev=%02x.%04x RY=0x%02x apfs=0x%x => bus rescan, rc=EAGAIN\n",
- card, queue, ehdr->reply_code, apfs);
+ ZCRYPT_DBF_WARN(
+ "%s dev=%02x.%04x RY=0x%02x apfs=0x%x => bus rescan, rc=EAGAIN\n",
+ __func__, card, queue, ehdr->reply_code, apfs);
} else
- ZCRYPT_DBF(DBF_WARN,
- "dev=%02x.%04x RY=0x%02x => bus rescan, rc=EAGAIN\n",
- card, queue, ehdr->reply_code);
+ ZCRYPT_DBF_WARN("%s dev=%02x.%04x RY=0x%02x => bus rescan, rc=EAGAIN\n",
+ __func__, card, queue,
+ ehdr->reply_code);
return -EAGAIN;
default:
/* Assume request is valid and a retry will be worth it */
- ZCRYPT_DBF(DBF_WARN,
- "dev=%02x.%04x RY=0x%02x => rc=EAGAIN\n",
- card, queue, ehdr->reply_code);
+ ZCRYPT_DBF_WARN("%s dev=%02x.%04x RY=0x%02x => rc=EAGAIN\n",
+ __func__, card, queue, ehdr->reply_code);
return -EAGAIN;
}
}
zq->online = 0;
pr_err("Crypto dev=%02x.%04x code=0x%02x => online=0 rc=EAGAIN\n",
AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- t80h->code);
- ZCRYPT_DBF_ERR("dev=%02x.%04x code=0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- t80h->code);
+ AP_QID_QUEUE(zq->queue->qid), t80h->code);
+ ZCRYPT_DBF_ERR("%s dev=%02x.%04x code=0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), t80h->code);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) rtype);
- ZCRYPT_DBF_ERR("dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- (int) rtype);
+ ZCRYPT_DBF_ERR(
+ "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), (int) rtype);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
(service_rc == 8 && service_rs == 72) ||
(service_rc == 8 && service_rs == 770) ||
(service_rc == 12 && service_rs == 769)) {
- ZCRYPT_DBF_WARN("dev=%02x.%04x rc/rs=%d/%d => rc=EINVAL\n",
- AP_QID_CARD(zq->queue->qid),
+ ZCRYPT_DBF_WARN("%s dev=%02x.%04x rc/rs=%d/%d => rc=EINVAL\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) service_rc, (int) service_rs);
return -EINVAL;
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) service_rc, (int) service_rs);
- ZCRYPT_DBF_ERR("dev=%02x.%04x rc/rs=%d/%d => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
+ ZCRYPT_DBF_ERR("%s dev=%02x.%04x rc/rs=%d/%d => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) service_rc, (int) service_rs);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) msg->hdr.type);
- ZCRYPT_DBF_ERR("dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- (int) msg->hdr.type);
+ ZCRYPT_DBF_ERR(
+ "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), (int) msg->hdr.type);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) msg->hdr.type);
- ZCRYPT_DBF_ERR("dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- (int) msg->hdr.type);
+ ZCRYPT_DBF_ERR(
+ "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), (int) msg->hdr.type);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) msg->hdr.type);
- ZCRYPT_DBF_ERR("dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- (int) msg->hdr.type);
+ ZCRYPT_DBF_ERR(
+ "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), (int) msg->hdr.type);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
AP_QID_CARD(zq->queue->qid),
AP_QID_QUEUE(zq->queue->qid),
(int) msg->hdr.type);
- ZCRYPT_DBF_ERR("dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- (int) msg->hdr.type);
+ ZCRYPT_DBF_ERR(
+ "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), (int) msg->hdr.type);
ap_send_online_uevent(&zq->queue->ap_dev, zq->online);
return -EAGAIN;
}
return -EINVAL;
zq->online = online;
- ZCRYPT_DBF(DBF_INFO, "queue=%02x.%04x online=%d\n",
- AP_QID_CARD(zq->queue->qid),
- AP_QID_QUEUE(zq->queue->qid),
- online);
+ ZCRYPT_DBF_INFO("%s queue=%02x.%04x online=%d\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid), online);
ap_send_online_uevent(&aq->ap_dev, online);
zq->zcard = zc;
zq->online = 1; /* New devices are online by default. */
- ZCRYPT_DBF(DBF_INFO, "queue=%02x.%04x register online=1\n",
- AP_QID_CARD(zq->queue->qid), AP_QID_QUEUE(zq->queue->qid));
+ ZCRYPT_DBF_INFO("%s queue=%02x.%04x register online=1\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid));
list_add_tail(&zq->list, &zc->zqueues);
spin_unlock(&zcrypt_list_lock);
{
struct zcrypt_card *zc;
- ZCRYPT_DBF(DBF_INFO, "queue=%02x.%04x unregister\n",
- AP_QID_CARD(zq->queue->qid), AP_QID_QUEUE(zq->queue->qid));
+ ZCRYPT_DBF_INFO("%s queue=%02x.%04x unregister\n",
+ __func__, AP_QID_CARD(zq->queue->qid),
+ AP_QID_QUEUE(zq->queue->qid));
zc = zq->zcard;
spin_lock(&zcrypt_list_lock);
extern const struct attribute_group *zfcp_unit_attr_groups[];
extern const struct attribute_group *zfcp_port_attr_groups[];
extern struct mutex zfcp_sysfs_port_units_mutex;
-extern struct device_attribute *zfcp_sysfs_sdev_attrs[];
-extern struct device_attribute *zfcp_sysfs_shost_attrs[];
+extern const struct attribute_group *zfcp_sysfs_sdev_attr_groups[];
+extern const struct attribute_group *zfcp_sysfs_shost_attr_groups[];
bool zfcp_sysfs_port_is_removing(const struct zfcp_port *const port);
/* zfcp_unit.c */
zfcp_dbf_scsi_result(scpnt, req);
scpnt->host_scribble = NULL;
- (scpnt->scsi_done) (scpnt);
+ scsi_done(scpnt);
/*
* We must hold this lock until scsi_done has been called.
* Otherwise we may call scsi_done after abort regarding this
{
set_host_byte(scpnt, result);
zfcp_dbf_scsi_fail_send(scpnt);
- scpnt->scsi_done(scpnt);
+ scsi_done(scpnt);
}
static
if (unlikely(scsi_result)) {
scpnt->result = scsi_result;
zfcp_dbf_scsi_fail_send(scpnt);
- scpnt->scsi_done(scpnt);
+ scsi_done(scpnt);
return 0;
}
/* report size limit per scatter-gather segment */
.max_segment_size = ZFCP_QDIO_SBALE_LEN,
.dma_boundary = ZFCP_QDIO_SBALE_LEN - 1,
- .shost_attrs = zfcp_sysfs_shost_attrs,
- .sdev_attrs = zfcp_sysfs_sdev_attrs,
+ .shost_groups = zfcp_sysfs_shost_attr_groups,
+ .sdev_groups = zfcp_sysfs_sdev_attr_groups,
.track_queue_depth = 1,
.supported_mode = MODE_INITIATOR,
};
ZFCP_DEFINE_SCSI_ATTR(zfcp_status, "0x%08x\n",
atomic_read(&zfcp_sdev->status));
-struct device_attribute *zfcp_sysfs_sdev_attrs[] = {
- &dev_attr_fcp_lun,
- &dev_attr_wwpn,
- &dev_attr_hba_id,
- &dev_attr_read_latency,
- &dev_attr_write_latency,
- &dev_attr_cmd_latency,
- &dev_attr_zfcp_access_denied,
- &dev_attr_zfcp_failed,
- &dev_attr_zfcp_in_recovery,
- &dev_attr_zfcp_status,
+struct attribute *zfcp_sdev_attrs[] = {
+ &dev_attr_fcp_lun.attr,
+ &dev_attr_wwpn.attr,
+ &dev_attr_hba_id.attr,
+ &dev_attr_read_latency.attr,
+ &dev_attr_write_latency.attr,
+ &dev_attr_cmd_latency.attr,
+ &dev_attr_zfcp_access_denied.attr,
+ &dev_attr_zfcp_failed.attr,
+ &dev_attr_zfcp_in_recovery.attr,
+ &dev_attr_zfcp_status.attr,
+ NULL
+};
+
+static const struct attribute_group zfcp_sysfs_sdev_attr_group = {
+ .attrs = zfcp_sdev_attrs
+};
+
+const struct attribute_group *zfcp_sysfs_sdev_attr_groups[] = {
+ &zfcp_sysfs_sdev_attr_group,
NULL
};
}
static DEVICE_ATTR(queue_full, S_IRUGO, zfcp_sysfs_adapter_q_full_show, NULL);
-struct device_attribute *zfcp_sysfs_shost_attrs[] = {
- &dev_attr_utilization,
- &dev_attr_requests,
- &dev_attr_megabytes,
- &dev_attr_seconds_active,
- &dev_attr_queue_full,
+static struct attribute *zfcp_sysfs_shost_attrs[] = {
+ &dev_attr_utilization.attr,
+ &dev_attr_requests.attr,
+ &dev_attr_megabytes.attr,
+ &dev_attr_seconds_active.attr,
+ &dev_attr_queue_full.attr,
+ NULL
+};
+
+static const struct attribute_group zfcp_sysfs_shost_attr_group = {
+ .attrs = zfcp_sysfs_shost_attrs
+};
+
+const struct attribute_group *zfcp_sysfs_shost_attr_groups[] = {
+ &zfcp_sysfs_shost_attr_group,
NULL
};
};
/* Host attributes initializer */
-static struct device_attribute *twa_host_attrs[] = {
- &twa_host_stats_attr,
+static struct attribute *twa_host_attrs[] = {
+ &twa_host_stats_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(twa_host);
+
/* File operations struct for character device */
static const struct file_operations twa_fops = {
.owner = THIS_MODULE,
/* Now complete the io */
if (twa_command_mapped(cmd))
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
tw_dev->state[request_id] = TW_S_COMPLETED;
twa_free_request_id(tw_dev, request_id);
tw_dev->posted_request_count--;
cmd->result = (DID_RESET << 16);
if (twa_command_mapped(cmd))
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
}
} /* End twa_scsi_eh_reset() */
/* This is the main scsi queue function to handle scsi opcodes */
-static int twa_scsi_queue_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *))
+static int twa_scsi_queue_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
int request_id, retval;
TW_Device_Extension *tw_dev = (TW_Device_Extension *)SCpnt->device->host->hostdata;
goto out;
}
- /* Save done function into scsi_cmnd struct */
- SCpnt->scsi_done = done;
-
/* Get a free request id */
twa_get_request_id(tw_dev, &request_id);
.sg_tablesize = TW_APACHE_MAX_SGL_LENGTH,
.max_sectors = TW_MAX_SECTORS,
.cmd_per_lun = TW_MAX_CMDS_PER_LUN,
- .shost_attrs = twa_host_attrs,
+ .shost_groups = twa_host_groups,
.emulated = 1,
.no_write_same = 1,
};
};
/* Host attributes initializer */
-static struct device_attribute *twl_host_attrs[] = {
- &twl_host_stats_attr,
+static struct attribute *twl_host_attrs[] = {
+ &twl_host_stats_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(twl_host);
+
/* This function will look up an AEN severity string */
static char *twl_aen_severity_lookup(unsigned char severity_code)
{
/* Now complete the io */
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
tw_dev->state[request_id] = TW_S_COMPLETED;
twl_free_request_id(tw_dev, request_id);
tw_dev->posted_request_count--;
if (cmd) {
cmd->result = (DID_RESET << 16);
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
}
} /* End twl_scsi_eh_reset() */
/* This is the main scsi queue function to handle scsi opcodes */
-static int twl_scsi_queue_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *))
+static int twl_scsi_queue_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
int request_id, retval;
TW_Device_Extension *tw_dev = (TW_Device_Extension *)SCpnt->device->host->hostdata;
goto out;
}
- /* Save done function into scsi_cmnd struct */
- SCpnt->scsi_done = done;
-
/* Get a free request id */
twl_get_request_id(tw_dev, &request_id);
.sg_tablesize = TW_LIBERATOR_MAX_SGL_LENGTH,
.max_sectors = TW_MAX_SECTORS,
.cmd_per_lun = TW_MAX_CMDS_PER_LUN,
- .shost_attrs = twl_host_attrs,
+ .shost_groups = twl_host_groups,
.emulated = 1,
.no_write_same = 1,
};
};
/* Host attributes initializer */
-static struct device_attribute *tw_host_attrs[] = {
- &tw_host_stats_attr,
+static struct attribute *tw_host_attrs[] = {
+ &tw_host_stats_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(tw_host);
+
/* This function will read the aen queue from the isr */
static int tw_aen_read_queue(TW_Device_Extension *tw_dev, int request_id)
{
tw_dev->state[request_id] = TW_S_COMPLETED;
tw_state_request_finish(tw_dev, request_id);
tw_dev->srb[request_id]->result = (DID_OK << 16);
- tw_dev->srb[request_id]->scsi_done(tw_dev->srb[request_id]);
+ scsi_done(tw_dev->srb[request_id]);
}
command_packet->byte8.param.sgl[0].address = param_value;
command_packet->byte8.param.sgl[0].length = sizeof(TW_Sector);
if (srb != NULL) {
srb->result = (DID_RESET << 16);
scsi_dma_unmap(srb);
- srb->scsi_done(srb);
+ scsi_done(srb);
}
}
}
tw_dev->state[request_id] = TW_S_COMPLETED;
tw_state_request_finish(tw_dev, request_id);
tw_dev->srb[request_id]->result = (DID_OK << 16);
- tw_dev->srb[request_id]->scsi_done(tw_dev->srb[request_id]);
+ scsi_done(tw_dev->srb[request_id]);
return 0;
}
/* If we got a request_sense, we probably want a reset, return error */
tw_dev->srb[request_id]->result = (DID_ERROR << 16);
- tw_dev->srb[request_id]->scsi_done(tw_dev->srb[request_id]);
+ scsi_done(tw_dev->srb[request_id]);
return 0;
} /* End tw_scsiop_request_sense() */
} /* End tw_scsiop_test_unit_ready_complete() */
/* This is the main scsi queue function to handle scsi opcodes */
-static int tw_scsi_queue_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *))
+static int tw_scsi_queue_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
unsigned char *command = SCpnt->cmnd;
int request_id = 0;
int retval = 1;
if (test_bit(TW_IN_RESET, &tw_dev->flags))
return SCSI_MLQUEUE_HOST_BUSY;
- /* Save done function into struct scsi_cmnd */
- SCpnt->scsi_done = done;
-
/* Queue the command and get a request id */
tw_state_request_start(tw_dev, &request_id);
/* Now complete the io */
if ((error != TW_ISR_DONT_COMPLETE)) {
scsi_dma_unmap(tw_dev->srb[request_id]);
- tw_dev->srb[request_id]->scsi_done(tw_dev->srb[request_id]);
+ scsi_done(tw_dev->srb[request_id]);
tw_dev->state[request_id] = TW_S_COMPLETED;
tw_state_request_finish(tw_dev, request_id);
tw_dev->posted_request_count--;
.sg_tablesize = TW_MAX_SGL_LENGTH,
.max_sectors = TW_MAX_SECTORS,
.cmd_per_lun = TW_MAX_CMDS_PER_LUN,
- .shost_attrs = tw_host_attrs,
+ .shost_groups = tw_host_groups,
.emulated = 1,
.no_write_same = 1,
};
{
struct Scsi_Host *host = NULL;
TW_Device_Extension *tw_dev;
- int retval = -ENODEV;
+ int retval;
retval = pci_enable_device(pdev);
if (retval) {
STATIC void NCR_700_slave_destroy(struct scsi_device *SDpnt);
static int NCR_700_change_queue_depth(struct scsi_device *SDpnt, int depth);
-STATIC struct device_attribute *NCR_700_dev_attrs[];
+STATIC const struct attribute_group *NCR_700_dev_groups[];
STATIC struct scsi_transport_template *NCR_700_transport_template = NULL;
static int banner = 0;
int j;
- if(tpnt->sdev_attrs == NULL)
- tpnt->sdev_attrs = NCR_700_dev_attrs;
+ if (tpnt->sdev_groups == NULL)
+ tpnt->sdev_groups = NCR_700_dev_groups;
memory = dma_alloc_coherent(dev, TOTAL_MEM_SIZE, &pScript, GFP_KERNEL);
if (!memory) {
SCp->host_scribble = NULL;
SCp->result = result;
- SCp->scsi_done(SCp);
+ scsi_done(SCp);
} else {
printk(KERN_ERR "53c700: SCSI DONE HAS NULL SCp\n");
}
* deadlock on the
* hostdata->state_lock */
SCp->result = DID_RESET << 16;
- SCp->scsi_done(SCp);
+ scsi_done(SCp);
}
mdelay(25);
NCR_700_chip_setup(host);
return IRQ_RETVAL(handled);
}
-static int
-NCR_700_queuecommand_lck(struct scsi_cmnd *SCp, void (*done)(struct scsi_cmnd *))
+static int NCR_700_queuecommand_lck(struct scsi_cmnd *SCp)
{
struct NCR_700_Host_Parameters *hostdata =
(struct NCR_700_Host_Parameters *)SCp->device->host->hostdata[0];
slot->cmnd = SCp;
- SCp->scsi_done = done;
SCp->host_scribble = (unsigned char *)slot;
SCp->SCp.ptr = NULL;
SCp->SCp.buffer = NULL;
.show = NCR_700_show_active_tags,
};
-STATIC struct device_attribute *NCR_700_dev_attrs[] = {
- &NCR_700_active_tags_attr,
+STATIC struct attribute *NCR_700_dev_attrs[] = {
+ &NCR_700_active_tags_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(NCR_700_dev);
+
EXPORT_SYMBOL(NCR_700_detect);
EXPORT_SYMBOL(NCR_700_release);
EXPORT_SYMBOL(NCR_700_intr);
command->reset_chain;
command->reset_chain = NULL;
command->result = DID_RESET << 16;
- command->scsi_done(command);
+ scsi_done(command);
command = nxt_cmd;
}
#endif
blogic_dealloc_ccb(ccb, 1);
adapter->active_cmds[tgt_id]--;
command->result = DID_RESET << 16;
- command->scsi_done(command);
+ scsi_done(command);
}
adapter->bdr_pend[tgt_id] = NULL;
} else {
/*
Call the SCSI Command Completion Routine.
*/
- command->scsi_done(command);
+ scsi_done(command);
}
}
adapter->processing_ccbs = false;
Outgoing Mailbox for execution by the associated Host Adapter.
*/
-static int blogic_qcmd_lck(struct scsi_cmnd *command,
- void (*comp_cb) (struct scsi_cmnd *))
+static int blogic_qcmd_lck(struct scsi_cmnd *command)
{
+ void (*comp_cb)(struct scsi_cmnd *) = scsi_done;
struct blogic_adapter *adapter =
(struct blogic_adapter *) command->device->host->hostdata;
struct blogic_tgt_flags *tgt_flags =
return SCSI_MLQUEUE_HOST_BUSY;
}
ccb->sensedata = sense_buf;
- command->scsi_done = comp_cb;
if (blogic_multimaster_type(adapter)) {
/*
Place the CCB in an Outgoing Mailbox. The higher levels
blogic_warn("Still unable to write Outgoing Mailbox - Host Adapter Dead?\n", adapter);
blogic_dealloc_ccb(ccb, 1);
command->result = DID_ERROR << 16;
- command->scsi_done(command);
+ scsi_done(command);
}
}
} else {
hostdata->sensing = NULL;
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
/**
case WRITE_10:
shost_printk(KERN_DEBUG, instance, "WRITE attempted with NDEBUG_NO_WRITE set\n");
cmd->result = (DID_ERROR << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
#endif /* (NDEBUG & NDEBUG_NO_WRITE) */
* hostdata->connected will be set to cmd.
* SELECT interrupt will be disabled.
*
- * If failed (no target) : cmd->scsi_done() will be called, and the
+ * If failed (no target) : scsi_done() will be called, and the
* cmd->result host byte set to DID_BAD_TARGET.
*/
dsprintk(NDEBUG_ABORT, instance,
"abort: removed %p from issue queue\n", cmd);
cmd->result = DID_ABORT << 16;
- cmd->scsi_done(cmd); /* No tag or busy flag to worry about */
+ scsi_done(cmd); /* No tag or busy flag to worry about */
goto out;
}
list_for_each_entry(ncmd, &hostdata->autosense, list) {
struct scsi_cmnd *cmd = NCR5380_to_scmd(ncmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
INIT_LIST_HEAD(&hostdata->autosense);
struct scsi_cmnd *scmd = NCR5380_to_scmd(ncmd);
scmd->result = DID_RESET << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
}
INIT_LIST_HEAD(&hostdata->unissued);
* queue the command down to the controller
*/
-static int inia100_queue_lck(struct scsi_cmnd * cmd, void (*done) (struct scsi_cmnd *))
+static int inia100_queue_lck(struct scsi_cmnd *cmd)
{
struct orc_scb *scb;
struct orc_host *host; /* Point to Host adapter control block */
host = (struct orc_host *) cmd->device->host->hostdata;
- cmd->scsi_done = done;
/* Get free SCSI control block */
if ((scb = orc_alloc_scb(host)) == NULL)
return SCSI_MLQUEUE_HOST_BUSY;
}
cmd->result = scb->tastat | (scb->hastat << 16);
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd); /* Notify system DONE */
+ scsi_done(cmd); /* Notify system DONE */
orc_release_scb(host, scb); /* Release SCB for current channel */
}
int sg_max, u64 sg_address);
static int aac_convert_sgraw2(struct aac_raw_io2 *rio2,
int pages, int nseg, int nseg_new);
+static void aac_probe_container_scsi_done(struct scsi_cmnd *scsi_cmnd);
static int aac_send_srb_fib(struct scsi_cmnd* scsicmd);
static int aac_send_hba_fib(struct scsi_cmnd *scsicmd);
#ifdef AAC_DETAILED_STATUS_INFO
struct fib *fibptr) {
struct scsi_device *device;
- if (unlikely(!scsicmd || !scsicmd->scsi_done)) {
+ if (unlikely(!scsicmd)) {
dprintk((KERN_WARNING "aac_valid_context: scsi command corrupt\n"));
aac_fib_complete(fibptr);
return 0;
return status;
}
+static void aac_scsi_done(struct scsi_cmnd *scmd)
+{
+ if (scmd->device->request_queue) {
+ /* SCSI command has been submitted by the SCSI mid-layer. */
+ scsi_done(scmd);
+ } else {
+ /* SCSI command has been submitted by aac_probe_container(). */
+ aac_probe_container_scsi_done(scmd);
+ }
+}
+
static void get_container_name_callback(void *context, struct fib * fibptr)
{
struct aac_get_name_resp * get_name_reply;
scsicmd->result = DID_OK << 16 | SAM_STAT_GOOD;
aac_fib_complete(fibptr);
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
/*
return aac_scsi_cmd(scsicmd);
scsicmd->result = DID_NO_CONNECT << 16;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
int aac_probe_container(struct aac_dev *dev, int cid)
{
- struct scsi_cmnd *scsicmd = kmalloc(sizeof(*scsicmd), GFP_KERNEL);
- struct scsi_device *scsidev = kmalloc(sizeof(*scsidev), GFP_KERNEL);
+ struct scsi_cmnd *scsicmd = kzalloc(sizeof(*scsicmd), GFP_KERNEL);
+ struct scsi_device *scsidev = kzalloc(sizeof(*scsidev), GFP_KERNEL);
int status;
if (!scsicmd || !scsidev) {
kfree(scsidev);
return -ENOMEM;
}
- scsicmd->scsi_done = aac_probe_container_scsi_done;
scsicmd->device = scsidev;
scsidev->sdev_state = 0;
scsicmd->result = DID_OK << 16 | SAM_STAT_GOOD;
aac_fib_complete(fibptr);
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
/*
memcpy(cmd->sense_buffer, &dev->fsa_dev[cid].sense_data,
min_t(size_t, sizeof(dev->fsa_dev[cid].sense_data),
SCSI_SENSE_BUFFERSIZE));
- cmd->scsi_done(cmd);
+ aac_scsi_done(cmd);
return 1;
}
return 0;
}
aac_fib_complete(fibptr);
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
static int aac_read(struct scsi_cmnd * scsicmd)
memcpy(scsicmd->sense_buffer, &dev->fsa_dev[cid].sense_data,
min_t(size_t, sizeof(dev->fsa_dev[cid].sense_data),
SCSI_SENSE_BUFFERSIZE));
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
* For some reason, the Fib didn't queue, return QUEUE_FULL
*/
scsicmd->result = DID_OK << 16 | SAM_STAT_TASK_SET_FULL;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
aac_fib_complete(cmd_fibcontext);
aac_fib_free(cmd_fibcontext);
return 0;
memcpy(scsicmd->sense_buffer, &dev->fsa_dev[cid].sense_data,
min_t(size_t, sizeof(dev->fsa_dev[cid].sense_data),
SCSI_SENSE_BUFFERSIZE));
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
* For some reason, the Fib didn't queue, return QUEUE_FULL
*/
scsicmd->result = DID_OK << 16 | SAM_STAT_TASK_SET_FULL;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
aac_fib_complete(cmd_fibcontext);
aac_fib_free(cmd_fibcontext);
aac_fib_complete(fibptr);
aac_fib_free(fibptr);
- cmd->scsi_done(cmd);
+ aac_scsi_done(cmd);
}
static int aac_synchronize(struct scsi_cmnd *scsicmd)
aac_fib_complete(fibptr);
aac_fib_free(fibptr);
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
static int aac_start_stop(struct scsi_cmnd *scsicmd)
if (!(aac->supplement_adapter_info.supported_options2 &
AAC_OPTION_POWER_MANAGEMENT)) {
scsicmd->result = DID_OK << 16 | SAM_STAT_GOOD;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
scsi_done_ret:
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
scsicmd->result |= le32_to_cpu(srbreply->scsi_status);
aac_fib_complete(fibptr);
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
static void hba_resp_task_complete(struct aac_dev *dev,
if (fibptr->flags & FIB_CONTEXT_FLAG_NATIVE_HBA_TMF)
scsicmd->SCp.sent_command = 1;
else
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
}
/**
if (scmd_id(scsicmd) >= dev->maximum_num_physicals ||
scsicmd->device->lun > 7) {
scsicmd->result = DID_NO_CONNECT << 16;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
if (scmd_id(scsicmd) >= dev->maximum_num_physicals ||
scsicmd->device->lun > AAC_MAX_LUN - 1) {
scsicmd->result = DID_NO_CONNECT << 16;
- scsicmd->scsi_done(scsicmd);
+ aac_scsi_done(scsicmd);
return 0;
}
-static struct device_attribute *aac_dev_attrs[] = {
- &aac_raid_level_attr,
- &aac_unique_id_attr,
+static struct attribute *aac_dev_attrs[] = {
+ &aac_raid_level_attr.attr,
+ &aac_unique_id_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(aac_dev);
+
static int aac_ioctl(struct scsi_device *sdev, unsigned int cmd,
void __user *arg)
{
.show = aac_show_reset_adapter,
};
-static struct device_attribute *aac_attrs[] = {
- &aac_model,
- &aac_vendor,
- &aac_flags,
- &aac_kernel_version,
- &aac_monitor_version,
- &aac_bios_version,
- &aac_lld_version,
- &aac_serial_number,
- &aac_max_channel,
- &aac_max_id,
- &aac_reset,
+static struct attribute *aac_host_attrs[] = {
+ &aac_model.attr,
+ &aac_vendor.attr,
+ &aac_flags.attr,
+ &aac_kernel_version.attr,
+ &aac_monitor_version.attr,
+ &aac_bios_version.attr,
+ &aac_lld_version.attr,
+ &aac_serial_number.attr,
+ &aac_max_channel.attr,
+ &aac_max_id.attr,
+ &aac_reset.attr,
NULL
};
+ATTRIBUTE_GROUPS(aac_host);
+
ssize_t aac_get_serial_number(struct device *device, char *buf)
{
return aac_show_serial_number(device, &aac_serial_number, buf);
#endif
.queuecommand = aac_queuecommand,
.bios_param = aac_biosparm,
- .shost_attrs = aac_attrs,
+ .shost_groups = aac_host_groups,
.slave_configure = aac_slave_configure,
.change_queue_depth = aac_change_queue_depth,
- .sdev_attrs = aac_dev_attrs,
+ .sdev_groups = aac_dev_groups,
.eh_abort_handler = aac_eh_abort,
.eh_device_reset_handler = aac_eh_dev_reset,
.eh_target_reset_handler = aac_eh_target_reset,
shost->host_no);
seq_printf(m,
- " iop_base 0x%lx, cable_detect: %X, err_code %u\n",
- (unsigned long)v->iop_base,
+ " iop_base 0x%p, cable_detect: %X, err_code %u\n",
+ v->iop_base,
AdvReadWordRegister(iop_base,IOPW_SCSI_CFG1) & CABLE_DETECT,
v->err_code);
{
scsi_dma_unmap(scp);
ASC_STATS(scp->device->host, done);
- scp->scsi_done(scp);
+ scsi_done(scp);
}
static void AscSetBank(PortAddr iop_base, uchar bank)
return ASC_ERROR;
}
- asc_sg_head = kzalloc(sizeof(asc_scsi_q->sg_head) +
- use_sg * sizeof(struct asc_sg_list), GFP_ATOMIC);
+ asc_sg_head = kzalloc(struct_size(asc_sg_head, sg_list, use_sg),
+ GFP_ATOMIC);
if (!asc_sg_head) {
scsi_dma_unmap(scp);
set_host_byte(scp, DID_SOFT_ERROR);
* This function always returns 0. Command return status is saved
* in the 'scp' result field.
*/
-static int
-advansys_queuecommand_lck(struct scsi_cmnd *scp, void (*done)(struct scsi_cmnd *))
+static int advansys_queuecommand_lck(struct scsi_cmnd *scp)
{
struct Scsi_Host *shost = scp->device->host;
int asc_res, result = 0;
ASC_STATS(shost, queuecommand);
- scp->scsi_done = done;
asc_res = asc_execute_scsi_cmnd(scp);
* Queue a command and setup interrupts for a free bus.
*/
static int aha152x_internal_queue(struct scsi_cmnd *SCpnt,
- struct completion *complete,
- int phase, void (*done)(struct scsi_cmnd *))
+ struct completion *complete, int phase)
{
struct Scsi_Host *shpnt = SCpnt->device->host;
unsigned long flags;
- SCpnt->scsi_done = done;
SCpnt->SCp.phase = not_issued | phase;
SCpnt->SCp.Status = 0x1; /* Ilegal status by SCSI standard */
SCpnt->SCp.Message = 0;
* queue a command
*
*/
-static int aha152x_queue_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int aha152x_queue_lck(struct scsi_cmnd *SCpnt)
{
- return aha152x_internal_queue(SCpnt, NULL, 0, done);
+ return aha152x_internal_queue(SCpnt, NULL, 0);
}
static DEF_SCSI_QCMD(aha152x_queue)
}
}
+static void aha152x_scsi_done(struct scsi_cmnd *SCpnt)
+{
+ if (SCpnt->SCp.phase & resetting)
+ reset_done(SCpnt);
+ else
+ scsi_done(SCpnt);
+}
+
/*
* Abort a command
*
SCpnt->cmd_len = 0;
- aha152x_internal_queue(SCpnt, &done, resetting, reset_done);
+ aha152x_internal_queue(SCpnt, &done, resetting);
timeleft = wait_for_completion_timeout(&done, 100*HZ);
if (!timeleft) {
scsi_eh_prep_cmnd(ptr, &sc->ses, NULL, 0, ~0);
DO_UNLOCK(flags);
- aha152x_internal_queue(ptr, NULL, check_condition, ptr->scsi_done);
+ aha152x_internal_queue(ptr, NULL, check_condition);
DO_LOCK(flags);
}
}
- if(DONE_SC && DONE_SC->scsi_done) {
+ if (DONE_SC) {
struct scsi_cmnd *ptr = DONE_SC;
DONE_SC=NULL;
if (!HOSTDATA(shpnt)->commands)
SETPORT(PORTA, 0); /* turn led off */
- if(ptr->scsi_done != reset_done) {
+ if (!(ptr->SCp.phase & resetting)) {
kfree(ptr->host_scribble);
ptr->host_scribble=NULL;
}
DO_UNLOCK(flags);
- ptr->scsi_done(ptr);
+ aha152x_scsi_done(ptr);
DO_LOCK(flags);
}
ptr->host_scribble=NULL;
set_host_byte(ptr, DID_RESET);
- ptr->scsi_done(ptr);
+ aha152x_scsi_done(ptr);
}
ptr = next;
struct bio_vec bv;
rq_for_each_segment(bv, rq, iter) {
- memcpy_to_page(bv.bv_page, bv.bv_offset, buf,
- bv.bv_len);
+ memcpy_to_bvec(&bv, buf);
buf += bv.bv_len;
}
}
{
struct Scsi_Host *sh = dev_id;
struct aha1542_hostdata *aha1542 = shost_priv(sh);
- void (*my_done)(struct scsi_cmnd *) = NULL;
int errstatus, mbi, mbo, mbistatus;
int number_serviced;
unsigned long flags;
tmp_cmd = aha1542->int_cmds[mbo];
- if (!tmp_cmd || !tmp_cmd->scsi_done) {
+ if (!tmp_cmd) {
spin_unlock_irqrestore(sh->host_lock, flags);
shost_printk(KERN_WARNING, sh, "Unexpected interrupt\n");
shost_printk(KERN_WARNING, sh, "tarstat=%x, hastat=%x idlun=%x ccb#=%d\n", ccb[mbo].tarstat,
ccb[mbo].hastat, ccb[mbo].idlun, mbo);
return IRQ_HANDLED;
}
- my_done = tmp_cmd->scsi_done;
aha1542_free_cmd(tmp_cmd);
/*
* Fetch the sense data, and tuck it away, in the required slot. The
aha1542->int_cmds[mbo] = NULL; /* This effectively frees up the mailbox slot, as
* far as queuecommand is concerned
*/
- my_done(tmp_cmd);
+ scsi_done(tmp_cmd);
number_serviced++;
};
}
if (*cmd->cmnd == REQUEST_SENSE) {
/* Don't do the command - we have the sense data already */
cmd->result = 0;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
#ifdef DEBUG
struct bio_vec bv;
rq_for_each_segment(bv, rq, iter) {
- memcpy_from_page(buf, bv.bv_page, bv.bv_offset,
- bv.bv_len);
+ memcpy_from_bvec(buf, &bv);
buf += bv.bv_len;
}
}
aha1542->aha1542_last_mbo_used = mbo;
#ifdef DEBUG
- shost_printk(KERN_DEBUG, sh, "Sending command (%d %p)...", mbo, cmd->scsi_done);
+ shost_printk(KERN_DEBUG, sh, "Sending command (%d)...", mbo);
#endif
/* This gets trashed for some reason */
return IRQ_RETVAL(handled);
}
-static int aha1740_queuecommand_lck(struct scsi_cmnd * SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int aha1740_queuecommand_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
unchar direction;
unchar *cmd = (unchar *) SCpnt->cmnd;
unchar target = scmd_id(SCpnt);
/*
* Queue an SCB to the controller.
*/
-static int
-ahd_linux_queue_lck(struct scsi_cmnd * cmd, void (*scsi_done) (struct scsi_cmnd *))
+static int ahd_linux_queue_lck(struct scsi_cmnd *cmd)
{
struct ahd_softc *ahd;
struct ahd_linux_device *dev = scsi_transport_device_data(cmd->device);
ahd = *(struct ahd_softc **)cmd->device->host->hostdata;
- cmd->scsi_done = scsi_done;
cmd->result = CAM_REQ_INPROG << 16;
rtn = ahd_linux_run_command(ahd, dev, cmd);
ahd_cmd_set_transaction_status(cmd, new_status);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
static void
/*
* XXX
* ahd_dmamap_sync is only used on buffers allocated with
- * the pci_alloc_consistent() API. Although I'm not sure how
+ * the dma_alloc_coherent() API. Although I'm not sure how
* this works on architectures with a write buffer, Linux does
* not have an API to sync "coherent" memory. Perhaps we need
* to do an mb()?
/*
* Queue an SCB to the controller.
*/
-static int
-ahc_linux_queue_lck(struct scsi_cmnd * cmd, void (*scsi_done) (struct scsi_cmnd *))
+static int ahc_linux_queue_lck(struct scsi_cmnd *cmd)
{
struct ahc_softc *ahc;
struct ahc_linux_device *dev = scsi_transport_device_data(cmd->device);
ahc_lock(ahc, &flags);
if (ahc->platform_data->qfrozen == 0) {
- cmd->scsi_done = scsi_done;
cmd->result = CAM_REQ_INPROG << 16;
rtn = ahc_linux_run_command(ahc, dev, cmd);
}
ahc_cmd_set_transaction_status(cmd, new_status);
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
static void
/*
* XXX
* ahc_dmamap_sync is only used on buffers allocated with
- * the pci_alloc_consistent() API. Although I'm not sure how
+ * the dma_alloc_coherent() API. Although I'm not sure how
* this works on architectures with a write buffer, Linux does
* not have an API to sync "coherent" memory. Perhaps we need
* to do an mb()?
struct QBUFFER __iomem *);
extern void arcmsr_clear_iop2drv_rqueue_buffer(struct AdapterControlBlock *);
extern struct QBUFFER __iomem *arcmsr_get_iop_rqbuffer(struct AdapterControlBlock *);
-extern struct device_attribute *arcmsr_host_attrs[];
+extern const struct attribute_group *arcmsr_host_groups[];
extern int arcmsr_alloc_sysfs_attr(struct AdapterControlBlock *);
void arcmsr_free_sysfs_attr(struct AdapterControlBlock *acb);
#include <scsi/scsi_transport.h>
#include "arcmsr.h"
-struct device_attribute *arcmsr_host_attrs[];
-
static ssize_t arcmsr_sysfs_iop_message_read(struct file *filp,
struct kobject *kobj,
struct bin_attribute *bin,
static DEVICE_ATTR(host_fw_sdram_size, S_IRUGO, arcmsr_attr_host_fw_sdram_size, NULL);
static DEVICE_ATTR(host_fw_hd_channels, S_IRUGO, arcmsr_attr_host_fw_hd_channels, NULL);
-struct device_attribute *arcmsr_host_attrs[] = {
- &dev_attr_host_driver_version,
- &dev_attr_host_driver_posted_cmd,
- &dev_attr_host_driver_reset,
- &dev_attr_host_driver_abort,
- &dev_attr_host_fw_model,
- &dev_attr_host_fw_version,
- &dev_attr_host_fw_request_len,
- &dev_attr_host_fw_numbers_queue,
- &dev_attr_host_fw_sdram_size,
- &dev_attr_host_fw_hd_channels,
+static struct attribute *arcmsr_host_attrs[] = {
+ &dev_attr_host_driver_version.attr,
+ &dev_attr_host_driver_posted_cmd.attr,
+ &dev_attr_host_driver_reset.attr,
+ &dev_attr_host_driver_abort.attr,
+ &dev_attr_host_fw_model.attr,
+ &dev_attr_host_fw_version.attr,
+ &dev_attr_host_fw_request_len.attr,
+ &dev_attr_host_fw_numbers_queue.attr,
+ &dev_attr_host_fw_sdram_size.attr,
+ &dev_attr_host_fw_hd_channels.attr,
NULL,
};
+
+static const struct attribute_group arcmsr_host_attr_group = {
+ .attrs = arcmsr_host_attrs,
+};
+
+const struct attribute_group *arcmsr_host_groups[] = {
+ &arcmsr_host_attr_group,
+ NULL
+};
.sg_tablesize = ARCMSR_DEFAULT_SG_ENTRIES,
.max_sectors = ARCMSR_MAX_XFER_SECTORS_C,
.cmd_per_lun = ARCMSR_DEFAULT_CMD_PERLUN,
- .shost_attrs = arcmsr_host_attrs,
+ .shost_groups = arcmsr_host_groups,
.no_write_same = 1,
};
spin_lock_irqsave(&acb->ccblist_lock, flags);
list_add_tail(&ccb->list, &acb->ccb_free_list);
spin_unlock_irqrestore(&acb->ccblist_lock, flags);
- pcmd->scsi_done(pcmd);
+ scsi_done(pcmd);
}
static void arcmsr_report_sense_info(struct CommandControlBlock *ccb)
if (ccb->startdone == ARCMSR_CCB_START) {
ccb->pcmd->result = DID_NO_CONNECT << 16;
arcmsr_pci_unmap_dma(ccb);
- ccb->pcmd->scsi_done(ccb->pcmd);
+ scsi_done(ccb->pcmd);
}
}
for (target = 0; target < ARCMSR_MAX_TARGETID; target++) {
if (cmd->device->lun) {
cmd->result = (DID_TIME_OUT << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return;
}
inqdata[0] = TYPE_PROCESSOR;
sg = scsi_sglist(cmd);
kunmap_atomic(buffer - sg->offset);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
break;
case WRITE_BUFFER:
case READ_BUFFER: {
if (arcmsr_iop_message_xfer(acb, cmd))
cmd->result = (DID_ERROR << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
break;
default:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
-static int arcmsr_queue_command_lck(struct scsi_cmnd *cmd,
- void (* done)(struct scsi_cmnd *))
+static int arcmsr_queue_command_lck(struct scsi_cmnd *cmd)
{
struct Scsi_Host *host = cmd->device->host;
struct AdapterControlBlock *acb = (struct AdapterControlBlock *) host->hostdata;
if (acb->acb_flags & ACB_F_ADAPTER_REMOVED) {
cmd->result = (DID_NO_CONNECT << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
- cmd->scsi_done = done;
cmd->host_scribble = NULL;
cmd->result = 0;
if (target == 16) {
return SCSI_MLQUEUE_HOST_BUSY;
if (arcmsr_build_ccb( acb, ccb, cmd ) == FAILED) {
cmd->result = (DID_ERROR << 16) | SAM_STAT_RESERVATION_CONFLICT;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
arcmsr_post_ccb(acb, ccb);
}
}
- if (!SCpnt->scsi_done)
- panic("scsi%d.H: null scsi_done function in acornscsi_done", host->host->host_no);
-
clear_bit(SCpnt->device->id * 8 +
(u8)(SCpnt->device->lun & 0x7), host->busyluns);
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
} else
printk("scsi%d: null command in acornscsi_done", host->host->host_no);
*/
/*
- * Function : acornscsi_queuecmd(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+ * Function : acornscsi_queuecmd(struct scsi_cmnd *cmd)
* Purpose : queues a SCSI command
* Params : cmd - SCSI command
- * done - function called on completion, with pointer to command descriptor
* Returns : 0, or < 0 on error.
*/
-static int acornscsi_queuecmd_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int acornscsi_queuecmd_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
AS_Host *host = (AS_Host *)SCpnt->device->host->hostdata;
- if (!done) {
- /* there should be some way of rejecting errors like this without panicing... */
- panic("scsi%d: queuecommand called with NULL done function [cmd=%p]",
- host->host->host_no, SCpnt);
- return -EINVAL;
- }
-
#if (DEBUG & DEBUG_NO_WRITE)
if (acornscsi_cmdtype(SCpnt->cmnd[0]) == CMD_WRITE && (NO_WRITE & (1 << SCpnt->device->id))) {
printk(KERN_CRIT "scsi%d.%c: WRITE attempted with NO_WRITE flag set\n",
}
#endif
- SCpnt->scsi_done = done;
SCpnt->host_scribble = NULL;
SCpnt->result = 0;
SCpnt->SCp.phase = (int)acornscsi_datadirection(SCpnt->cmnd[0]);
.eh_bus_reset_handler = fas216_eh_bus_reset,
.eh_device_reset_handler = fas216_eh_device_reset,
.eh_abort_handler = fas216_eh_abort,
+ .cmd_size = sizeof(struct fas216_cmd_priv),
.can_queue = 0,
.this_id = 7,
.sg_tablesize = SG_ALL,
.eh_bus_reset_handler = fas216_eh_bus_reset,
.eh_device_reset_handler = fas216_eh_device_reset,
.eh_abort_handler = fas216_eh_abort,
+ .cmd_size = sizeof(struct fas216_cmd_priv),
.can_queue = 1,
.this_id = 7,
.sg_tablesize = SG_MAX_SEGMENTS,
.eh_bus_reset_handler = fas216_eh_bus_reset,
.eh_device_reset_handler = fas216_eh_device_reset,
.eh_abort_handler = fas216_eh_abort,
+ .cmd_size = sizeof(struct fas216_cmd_priv),
.can_queue = 1,
.this_id = 7,
.sg_tablesize = SG_MAX_SEGMENTS,
* correctly by fas216_std_done.
*/
scsi_eh_restore_cmnd(SCpnt, &info->ses);
- SCpnt->scsi_done(SCpnt);
+ fas216_cmd_priv(SCpnt)->scsi_done(SCpnt);
}
/**
}
done:
- if (SCpnt->scsi_done) {
- SCpnt->scsi_done(SCpnt);
+ if (fas216_cmd_priv(SCpnt)->scsi_done) {
+ fas216_cmd_priv(SCpnt)->scsi_done(SCpnt);
return;
}
}
/**
- * fas216_queue_command - queue a command for adapter to process.
+ * fas216_queue_command_internal - queue a command for the adapter to process
* @SCpnt: Command to queue
* @done: done function to call once command is complete
*
* Returns: 0 on success, else error.
* Notes: io_request_lock is held, interrupts are disabled.
*/
-static int fas216_queue_command_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int fas216_queue_command_internal(struct scsi_cmnd *SCpnt,
+ void (*done)(struct scsi_cmnd *))
{
FAS216_Info *info = (FAS216_Info *)SCpnt->device->host->hostdata;
int result;
fas216_log_command(info, LOG_CONNECT, SCpnt,
"received command (%p)", SCpnt);
- SCpnt->scsi_done = done;
+ fas216_cmd_priv(SCpnt)->scsi_done = done;
SCpnt->host_scribble = (void *)fas216_std_done;
SCpnt->result = 0;
return result;
}
+static int fas216_queue_command_lck(struct scsi_cmnd *SCpnt)
+{
+ return fas216_queue_command_internal(SCpnt, scsi_done);
+}
+
DEF_SCSI_QCMD(fas216_queue_command)
/**
* Returns: scsi result code.
* Notes: io_request_lock is held, interrupts are disabled.
*/
-static int fas216_noqueue_command_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int fas216_noqueue_command_lck(struct scsi_cmnd *SCpnt)
{
FAS216_Info *info = (FAS216_Info *)SCpnt->device->host->hostdata;
BUG_ON(info->scsi.irq);
info->internal_done = 0;
- fas216_queue_command_lck(SCpnt, fas216_internal_done);
+ fas216_queue_command_internal(SCpnt, fas216_internal_done);
/*
* This wastes time, since we can't return until the command is
spin_lock_irq(info->host->host_lock);
- done(SCpnt);
+ scsi_done(SCpnt);
return 0;
}
unsigned long magic_end;
} FAS216_Info;
+/* driver-private data per SCSI command. */
+struct fas216_cmd_priv {
+ void (*scsi_done)(struct scsi_cmnd *cmd);
+};
+
+static inline struct fas216_cmd_priv *fas216_cmd_priv(struct scsi_cmnd *cmd)
+{
+ return scsi_cmd_priv(cmd);
+}
+
/* Function: int fas216_init (struct Scsi_Host *instance)
* Purpose : initialise FAS/NCR/AMD SCSI structures.
* Params : instance - a driver-specific filled-out structure
.eh_bus_reset_handler = fas216_eh_bus_reset,
.eh_device_reset_handler = fas216_eh_device_reset,
.eh_abort_handler = fas216_eh_abort,
-
+ .cmd_size = sizeof(struct fas216_cmd_priv),
.can_queue = 8,
.this_id = 7,
.sg_tablesize = SG_MAX_SEGMENTS,
scsi_dma_unmap(workreq);
spin_lock_irqsave(dev->host->host_lock, flags);
- (*workreq->scsi_done) (workreq);
+ scsi_done(workreq);
#ifdef ED_DBGP
printk("workreq->scsi_done\n");
#endif
*
* Queue a command to the ATP queue. Called with the host lock held.
*/
-static int atp870u_queuecommand_lck(struct scsi_cmnd *req_p,
- void (*done) (struct scsi_cmnd *))
+static int atp870u_queuecommand_lck(struct scsi_cmnd *req_p)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
unsigned char c;
unsigned int m;
struct atp_unit *dev;
return 0;
}
- if (done) {
- req_p->scsi_done = done;
- } else {
-#ifdef ED_DBGP
- printk( "atp870u_queuecommand: done can't be NULL\n");
-#endif
- req_p->result = 0;
- done(req_p);
- return 0;
- }
-
/*
* Count new command
*/
beiscsi_active_session_disp, NULL);
DEVICE_ATTR(beiscsi_free_session_count, S_IRUGO,
beiscsi_free_session_disp, NULL);
-static struct device_attribute *beiscsi_attrs[] = {
- &dev_attr_beiscsi_log_enable,
- &dev_attr_beiscsi_drvr_ver,
- &dev_attr_beiscsi_adapter_family,
- &dev_attr_beiscsi_fw_ver,
- &dev_attr_beiscsi_active_session_count,
- &dev_attr_beiscsi_free_session_count,
- &dev_attr_beiscsi_phys_port,
+
+static struct attribute *beiscsi_attrs[] = {
+ &dev_attr_beiscsi_log_enable.attr,
+ &dev_attr_beiscsi_drvr_ver.attr,
+ &dev_attr_beiscsi_adapter_family.attr,
+ &dev_attr_beiscsi_fw_ver.attr,
+ &dev_attr_beiscsi_active_session_count.attr,
+ &dev_attr_beiscsi_free_session_count.attr,
+ &dev_attr_beiscsi_phys_port.attr,
NULL,
};
+ATTRIBUTE_GROUPS(beiscsi);
+
static char const *cqe_desc[] = {
"RESERVED_DESC",
"SOL_CMD_COMPLETE",
.eh_abort_handler = beiscsi_eh_abort,
.eh_device_reset_handler = beiscsi_eh_device_reset,
.eh_target_reset_handler = iscsi_eh_session_reset,
- .shost_attrs = beiscsi_attrs,
+ .shost_groups = beiscsi_groups,
.sg_tablesize = BEISCSI_SGLIST_ELEMENTS,
.can_queue = BE2_IO_DEPTH,
.this_id = -1,
static DEVICE_ATTR(number_of_discovered_ports, S_IRUGO,
bfad_im_num_of_discovered_ports_show, NULL);
-struct device_attribute *bfad_im_host_attrs[] = {
- &dev_attr_serial_number,
- &dev_attr_model,
- &dev_attr_model_description,
- &dev_attr_node_name,
- &dev_attr_symbolic_name,
- &dev_attr_hardware_version,
- &dev_attr_driver_version,
- &dev_attr_option_rom_version,
- &dev_attr_firmware_version,
- &dev_attr_number_of_ports,
- &dev_attr_driver_name,
- &dev_attr_number_of_discovered_ports,
+static struct attribute *bfad_im_host_attrs[] = {
+ &dev_attr_serial_number.attr,
+ &dev_attr_model.attr,
+ &dev_attr_model_description.attr,
+ &dev_attr_node_name.attr,
+ &dev_attr_symbolic_name.attr,
+ &dev_attr_hardware_version.attr,
+ &dev_attr_driver_version.attr,
+ &dev_attr_option_rom_version.attr,
+ &dev_attr_firmware_version.attr,
+ &dev_attr_number_of_ports.attr,
+ &dev_attr_driver_name.attr,
+ &dev_attr_number_of_discovered_ports.attr,
NULL,
};
-struct device_attribute *bfad_im_vport_attrs[] = {
- &dev_attr_serial_number,
- &dev_attr_model,
- &dev_attr_model_description,
- &dev_attr_node_name,
- &dev_attr_symbolic_name,
- &dev_attr_hardware_version,
- &dev_attr_driver_version,
- &dev_attr_option_rom_version,
- &dev_attr_firmware_version,
- &dev_attr_number_of_ports,
- &dev_attr_driver_name,
- &dev_attr_number_of_discovered_ports,
+static const struct attribute_group bfad_im_host_attr_group = {
+ .attrs = bfad_im_host_attrs
+};
+
+const struct attribute_group *bfad_im_host_groups[] = {
+ &bfad_im_host_attr_group,
+ NULL
+};
+
+struct attribute *bfad_im_vport_attrs[] = {
+ &dev_attr_serial_number.attr,
+ &dev_attr_model.attr,
+ &dev_attr_model_description.attr,
+ &dev_attr_node_name.attr,
+ &dev_attr_symbolic_name.attr,
+ &dev_attr_hardware_version.attr,
+ &dev_attr_driver_version.attr,
+ &dev_attr_option_rom_version.attr,
+ &dev_attr_firmware_version.attr,
+ &dev_attr_number_of_ports.attr,
+ &dev_attr_driver_name.attr,
+ &dev_attr_number_of_discovered_ports.attr,
NULL,
};
+static const struct attribute_group bfad_im_vport_attr_group = {
+ .attrs = bfad_im_vport_attrs
+};
+const struct attribute_group *bfad_im_vport_groups[] = {
+ &bfad_im_vport_attr_group,
+ NULL
+};
}
}
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
}
void
}
}
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
}
void
timeout *= 2;
}
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
bfa_trc(bfad, hal_io->iotag);
BFA_LOG(KERN_INFO, bfad, bfa_log_level,
"scsi%d: complete abort 0x%p iotag 0x%x\n",
.this_id = -1,
.sg_tablesize = BFAD_IO_MAX_SGE,
.cmd_per_lun = 3,
- .shost_attrs = bfad_im_host_attrs,
+ .shost_groups = bfad_im_host_groups,
.max_sectors = BFAD_MAX_SECTORS,
.vendor_id = BFA_PCI_VENDOR_ID_BROCADE,
};
.this_id = -1,
.sg_tablesize = BFAD_IO_MAX_SGE,
.cmd_per_lun = 3,
- .shost_attrs = bfad_im_vport_attrs,
+ .shost_groups = bfad_im_vport_groups,
.max_sectors = BFAD_MAX_SECTORS,
};
/*
* Scsi_Host template entry, queue a SCSI command to the BFAD.
*/
-static int
-bfad_im_queuecommand_lck(struct scsi_cmnd *cmnd, void (*done) (struct scsi_cmnd *))
+static int bfad_im_queuecommand_lck(struct scsi_cmnd *cmnd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct bfad_im_port_s *im_port =
(struct bfad_im_port_s *) cmnd->device->host->hostdata[0];
struct bfad_s *bfad = im_port->bfad;
if (sg_cnt < 0)
return SCSI_MLQUEUE_HOST_BUSY;
- cmnd->scsi_done = done;
-
spin_lock_irqsave(&bfad->bfad_lock, flags);
if (!(bfad->bfad_flags & BFAD_HAL_START_DONE)) {
printk(KERN_WARNING
extern struct scsi_transport_template *bfad_im_scsi_transport_template;
extern struct scsi_transport_template *bfad_im_scsi_vport_transport_template;
-extern struct device_attribute *bfad_im_host_attrs[];
-extern struct device_attribute *bfad_im_vport_attrs[];
+extern const struct attribute_group *bfad_im_host_groups[];
+extern const struct attribute_group *bfad_im_vport_groups[];
irqreturn_t bfad_intx(int irq, void *dev_id);
static DEVICE_ATTR(tm_timeout, S_IRUGO|S_IWUSR, bnx2fc_tm_timeout_show,
bnx2fc_tm_timeout_store);
-static struct device_attribute *bnx2fc_host_attrs[] = {
- &dev_attr_tm_timeout,
+static struct attribute *bnx2fc_host_attrs[] = {
+ &dev_attr_tm_timeout.attr,
NULL,
};
+ATTRIBUTE_GROUPS(bnx2fc_host);
+
/*
* scsi_host_template structure used while registering with SCSI-ml
*/
.max_sectors = 0x3fbf,
.track_queue_depth = 1,
.slave_configure = bnx2fc_slave_configure,
- .shost_attrs = bnx2fc_host_attrs,
+ .shost_groups = bnx2fc_host_groups,
};
static struct libfc_function_template bnx2fc_libfc_fcn_templ = {
sc_cmd->allowed);
scsi_set_resid(sc_cmd, scsi_bufflen(sc_cmd));
sc_cmd->SCp.ptr = NULL;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
}
struct bnx2fc_cmd_mgr *bnx2fc_cmd_mgr_alloc(struct bnx2fc_hba *hba)
}
sc_cmd->SCp.ptr = NULL;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
kref_put(&io_req->refcount, bnx2fc_cmd_release);
if (io_req->wait_for_abts_comp) {
rval = fc_remote_port_chkready(rport);
if (rval) {
sc_cmd->result = rval;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
break;
}
sc_cmd->SCp.ptr = NULL;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
kref_put(&io_req->refcount, bnx2fc_cmd_release);
}
extern unsigned int sq_size;
extern unsigned int rq_size;
-extern struct device_attribute *bnx2i_dev_attributes[];
+extern const struct attribute_group *bnx2i_dev_groups[];
.cmd_per_lun = 128,
.this_id = -1,
.sg_tablesize = ISCSI_MAX_BDS_PER_CMD,
- .shost_attrs = bnx2i_dev_attributes,
+ .shost_groups = bnx2i_dev_groups,
.track_queue_depth = 1,
};
static DEVICE_ATTR(num_ccell, S_IRUGO | S_IWUSR,
bnx2i_show_ccell_info, bnx2i_set_ccell_info);
-struct device_attribute *bnx2i_dev_attributes[] = {
- &dev_attr_sq_size,
- &dev_attr_num_ccell,
+static struct attribute *bnx2i_dev_attributes[] = {
+ &dev_attr_sq_size.attr,
+ &dev_attr_num_ccell.attr,
+ NULL
+};
+
+static const struct attribute_group bnx2i_dev_attr_group = {
+ .attrs = bnx2i_dev_attributes
+};
+
+const struct attribute_group *bnx2i_dev_groups[] = {
+ &bnx2i_dev_attr_group,
NULL
};
struct fc_els_csp *csp;
struct fc_els_cssp *clsp;
enum fw_retval retval;
- __be32 nport_id;
+ __be32 nport_id = 0;
retval = FW_CMD_RETVAL_G(ntohl(rsp->alloc_to_len16));
if (retval != FW_SUCCESS) {
static DEVICE_ATTR(dbg_level, S_IRUGO | S_IWUSR, csio_show_dbg_level,
csio_store_dbg_level);
-static struct device_attribute *csio_fcoe_lport_attrs[] = {
- &dev_attr_hw_state,
- &dev_attr_device_reset,
- &dev_attr_disable_port,
- &dev_attr_dbg_level,
+static struct attribute *csio_fcoe_lport_attrs[] = {
+ &dev_attr_hw_state.attr,
+ &dev_attr_device_reset.attr,
+ &dev_attr_disable_port.attr,
+ &dev_attr_dbg_level.attr,
NULL,
};
+ATTRIBUTE_GROUPS(csio_fcoe_lport);
+
static ssize_t
csio_show_num_reg_rnodes(struct device *dev,
struct device_attribute *attr, char *buf)
static DEVICE_ATTR(num_reg_rnodes, S_IRUGO, csio_show_num_reg_rnodes, NULL);
-static struct device_attribute *csio_fcoe_vport_attrs[] = {
- &dev_attr_num_reg_rnodes,
- &dev_attr_dbg_level,
+static struct attribute *csio_fcoe_vport_attrs[] = {
+ &dev_attr_num_reg_rnodes.attr,
+ &dev_attr_dbg_level.attr,
NULL,
};
+ATTRIBUTE_GROUPS(csio_fcoe_vport);
+
static inline uint32_t
csio_scsi_copy_to_sgl(struct csio_hw *hw, struct csio_ioreq *req)
{
}
cmnd->result = (((host_status) << 16) | scsi_status);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
/* Wake up waiting threads */
csio_scsi_cmnd(req) = NULL;
}
cmnd->result = (((host_status) << 16) | scsi_status);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
csio_scsi_cmnd(req) = NULL;
CSIO_INC_STATS(csio_hw_to_scsim(hw), n_tot_success);
} else {
return rv;
err_done:
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
spin_unlock_irq(&hw->lock);
cmnd->result = (DID_ERROR << 16);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return FAILED;
}
.this_id = -1,
.sg_tablesize = CSIO_SCSI_MAX_SGE,
.cmd_per_lun = CSIO_MAX_CMD_PER_LUN,
- .shost_attrs = csio_fcoe_lport_attrs,
+ .shost_groups = csio_fcoe_lport_groups,
.max_sectors = CSIO_MAX_SECTOR_SIZE,
};
.this_id = -1,
.sg_tablesize = CSIO_SCSI_MAX_SGE,
.cmd_per_lun = CSIO_MAX_CMD_PER_LUN,
- .shost_attrs = csio_fcoe_vport_attrs,
+ .shost_groups = csio_fcoe_vport_groups,
.max_sectors = CSIO_MAX_SECTOR_SIZE,
};
dev_dbg_ratelimited(dev, "%s:scp=%p result=%08x ioasc=%08x\n",
__func__, scp, scp->result, cmd->sa.ioasc);
- scp->scsi_done(scp);
+ scsi_done(scp);
} else if (cmd->cmd_tmf) {
spin_lock_irqsave(&cfg->tmf_slock, lock_flags);
cfg->tmf_active = false;
if (cmd->scp) {
scp = cmd->scp;
scp->result = (DID_IMM_RETRY << 16);
- scp->scsi_done(scp);
+ scsi_done(scp);
} else {
cmd->cmd_aborted = true;
case STATE_FAILTERM:
dev_dbg_ratelimited(dev, "%s: device has failed\n", __func__);
scp->result = (DID_NO_CONNECT << 16);
- scp->scsi_done(scp);
+ scsi_done(scp);
rc = 0;
goto out;
default:
static DEVICE_ATTR_RW(num_hwqs);
static DEVICE_ATTR_RW(hwq_mode);
-static struct device_attribute *cxlflash_host_attrs[] = {
- &dev_attr_port0,
- &dev_attr_port1,
- &dev_attr_port2,
- &dev_attr_port3,
- &dev_attr_lun_mode,
- &dev_attr_ioctl_version,
- &dev_attr_port0_lun_table,
- &dev_attr_port1_lun_table,
- &dev_attr_port2_lun_table,
- &dev_attr_port3_lun_table,
- &dev_attr_irqpoll_weight,
- &dev_attr_num_hwqs,
- &dev_attr_hwq_mode,
+static struct attribute *cxlflash_host_attrs[] = {
+ &dev_attr_port0.attr,
+ &dev_attr_port1.attr,
+ &dev_attr_port2.attr,
+ &dev_attr_port3.attr,
+ &dev_attr_lun_mode.attr,
+ &dev_attr_ioctl_version.attr,
+ &dev_attr_port0_lun_table.attr,
+ &dev_attr_port1_lun_table.attr,
+ &dev_attr_port2_lun_table.attr,
+ &dev_attr_port3_lun_table.attr,
+ &dev_attr_irqpoll_weight.attr,
+ &dev_attr_num_hwqs.attr,
+ &dev_attr_hwq_mode.attr,
NULL
};
+ATTRIBUTE_GROUPS(cxlflash_host);
+
/*
* Device attributes
*/
static DEVICE_ATTR_RO(mode);
-static struct device_attribute *cxlflash_dev_attrs[] = {
- &dev_attr_mode,
+static struct attribute *cxlflash_dev_attrs[] = {
+ &dev_attr_mode.attr,
NULL
};
+ATTRIBUTE_GROUPS(cxlflash_dev);
+
/*
* Host template
*/
.this_id = -1,
.sg_tablesize = 1, /* No scatter gather support */
.max_sectors = CXLFLASH_MAX_SECTORS,
- .shost_attrs = cxlflash_host_attrs,
- .sdev_attrs = cxlflash_dev_attrs,
+ .shost_groups = cxlflash_host_groups,
+ .sdev_groups = cxlflash_dev_groups,
};
/*
* and is expected to be held on return.
*
**/
-static int dc395x_queue_command_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int dc395x_queue_command_lck(struct scsi_cmnd *cmd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct DeviceCtlBlk *dcb;
struct ScsiReqBlk *srb;
struct AdapterCtlBlk *acb =
goto complete;
}
- /* set callback and clear result in the command */
- cmd->scsi_done = done;
set_host_byte(cmd, DID_OK);
set_status_byte(cmd, SAM_STAT_GOOD);
dprintkl(KERN_ERR, "srb_done: ERROR! Completed cmd with tmp_srb\n");
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
waiting_process_next(acb);
}
if (force) {
/* For new EH, we normally don't need to give commands back,
* as they all complete or all time out */
- p->scsi_done(p);
+ scsi_done(p);
}
}
if (!list_empty(&dcb->srb_going_list))
if (force) {
/* For new EH, we normally don't need to give commands back,
* as they all complete or all time out */
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
if (!list_empty(&dcb->srb_waiting_list))
/* initialise the adapter and everything we need */
if (adapter_init(acb, io_port_base, io_port_len, irq)) {
dprintkl(KERN_INFO, "adapter init failed\n");
+ acb = NULL;
goto fail;
}
return 0;
}
-static int adpt_queue_lck(struct scsi_cmnd * cmd, void (*done) (struct scsi_cmnd *))
+static int adpt_queue_lck(struct scsi_cmnd *cmd)
{
adpt_hba* pHba = NULL;
struct adpt_device* pDev = NULL; /* dpt per device information */
- cmd->scsi_done = done;
/*
* SCSI REQUEST_SENSE commands will be executed automatically by the
* Host Adapter for any errors, so they should not be executed
if ((cmd->cmnd[0] == REQUEST_SENSE) && (cmd->sense_buffer[0] != 0)) {
cmd->result = (DID_OK << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
// TODO: if any luns are at this bus, scsi id then fake a TEST_UNIT_READY and INQUIRY response
// with type 7F (for all luns less than the max for this bus,id) so the lun scan will continue.
cmd->result = (DID_NO_CONNECT << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
cmd->device->hostdata = pDev;
printk(KERN_WARNING"%s: scsi opcode 0x%x not supported.\n",
pHba->name, cmd->cmnd[0]);
cmd->result = (DID_ERROR <<16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
}
cmd->result |= (dev_status);
- if(cmd->scsi_done != NULL){
- cmd->scsi_done(cmd);
- }
+ scsi_done(cmd);
}
pci_set_drvdata(pdev, efct);
- if (pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) != 0 ||
- pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)) != 0) {
+ if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)) != 0) {
dev_warn(&pdev->dev, "trying DMA_BIT_MASK(32)\n");
- if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32)) != 0 ||
- pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)) != 0) {
+ if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)) != 0) {
dev_err(&pdev->dev, "setting DMA_BIT_MASK failed\n");
rc = -1;
goto dma_mask_out;
struct efct_scsi_tgt_io *ocp = &io->tgt_io;
struct se_cmd *cmd = &ocp->cmd;
- ocp->seg_map_cnt = pci_map_sg(io->efct->pci, cmd->t_data_sg,
+ ocp->seg_map_cnt = dma_map_sg(&io->efct->pci->dev, cmd->t_data_sg,
cmd->t_data_nents, cmd->data_direction);
if (ocp->seg_map_cnt == 0)
return -EFAULT;
if (WARN_ON(!ocp->seg_map_cnt || !cmd->t_data_sg))
return;
- pci_unmap_sg(io->efct->pci, cmd->t_data_sg,
+ dma_unmap_sg(&io->efct->pci->dev, cmd->t_data_sg,
ocp->seg_map_cnt, cmd->data_direction);
ocp->seg_map_cnt = 0;
}
xport = efct->xport;
- spin_lock_irqsave(&node->active_ios_lock, flags);
-
io = efct_io_pool_io_alloc(efct->xport->io_pool);
if (!io) {
efc_log_err(efct, "IO alloc Failed\n");
/* Add to node's active_ios list */
INIT_LIST_HEAD(&io->list_entry);
+ spin_lock_irqsave(&node->active_ios_lock, flags);
list_add(&io->list_entry, &node->active_ios);
spin_unlock_irqrestore(&node->active_ios_lock, flags);
#define nport_sm_trace(nport) \
efc_log_debug(nport->efc, \
- "[%s] %-20s\n", nport->display_name, efc_sm_event_name(evt)) \
+ "[%s] %-20s %-20s\n", nport->display_name, __func__, efc_sm_event_name(evt)) \
#endif /* __EFC_H__ */
{
struct efc_nport *nport = arg;
+ nport->attaching = false;
if (efc_nport_get_mbox_status(nport, mqe, status)) {
efc_nport_free_resources(nport, EFC_EVT_NPORT_ATTACH_FAIL, mqe);
return -EIO;
if (rc) {
efc_log_err(efc, "REG_VPI command failure\n");
efc_nport_free_resources(nport, EFC_EVT_NPORT_ATTACH_FAIL, buf);
+ } else {
+ nport->attaching = true;
}
return rc;
/* Issue the UNREG_VPI command to free the assigned VPI context */
if (nport->attached)
efc_nport_free_unreg_vpi(nport);
- else
+ else if (nport->attaching)
nport->free_req_pending = true;
+ else
+ efc_sm_post_event(&nport->sm, EFC_EVT_NPORT_FREE_OK, NULL);
return 0;
}
}
/* Allocate a buffer for all nodes */
- active_nodes = kzalloc(port_count * sizeof(*active_nodes), GFP_ATOMIC);
+ active_nodes = kcalloc(port_count, sizeof(*active_nodes), GFP_ATOMIC);
if (!active_nodes) {
node_printf(node, "efc_malloc failed\n");
return -EIO;
bool is_vport;
bool free_req_pending;
bool attached;
+ bool attaching;
bool p2p_winner;
struct efc_domain *domain;
u64 wwpn;
sli_get_read_config(struct sli4 *sli4)
{
struct sli4_rsp_read_config *conf = sli4->bmbx.virt;
- u32 i, total, total_size;
+ u32 i, total;
u32 *base;
if (sli_cmd_read_config(sli4, sli4->bmbx.virt)) {
for (i = 0; i < SLI4_RSRC_MAX; i++) {
total = sli4->ext[i].number * sli4->ext[i].size;
- total_size = BITS_TO_LONGS(total) * sizeof(long);
- sli4->ext[i].use_map = kzalloc(total_size, GFP_KERNEL);
+ sli4->ext[i].use_map = bitmap_zalloc(total, GFP_KERNEL);
if (!sli4->ext[i].use_map) {
efc_log_err(sli4, "bitmap memory allocation failed %d\n",
i);
sli4->ext[0].base = NULL;
for (i = 0; i < SLI4_RSRC_MAX; i++) {
- kfree(sli4->ext[i].use_map);
+ bitmap_free(sli4->ext[i].use_map);
sli4->ext[i].use_map = NULL;
sli4->ext[i].base = NULL;
}
for (i = 0; i < SLI4_RSRC_MAX; i++) {
sli4->ext[i].base = NULL;
- kfree(sli4->ext[i].use_map);
+ bitmap_free(sli4->ext[i].use_map);
sli4->ext[i].use_map = NULL;
}
if (unlikely(test_bit(AF_DEGRADED_MODE, &a->flags))) {
cmd->result = DID_NO_CONNECT << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
scsi_set_resid(cmd, 0);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return SUCCESS;
}
scsi_set_resid(cmd, 0);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return SUCCESS;
}
scsi_set_resid(rq->cmd, 0);
}
- rq->cmd->scsi_done(rq->cmd);
+ scsi_done(rq->cmd);
esas2r_free_request(a, rq);
}
}
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
list_del(&ent->list);
esp_put_ent(esp, ent);
scsi_track_queue_full(dev, lp->num_tagged - 1);
}
-static int esp_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int esp_queuecommand_lck(struct scsi_cmnd *cmd)
{
struct scsi_device *dev = cmd->device;
struct esp *esp = shost_priv(dev->host);
ent->cmd = cmd;
- cmd->scsi_done = done;
-
spriv = ESP_CMD_PRIV(cmd);
spriv->num_sg = 0;
if (ent->flags & ESP_CMD_FLAG_AUTOSENSE)
esp_unmap_sense(esp, ent);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
list_del(&ent->list);
esp_put_ent(esp, ent);
}
list_del(&ent->list);
cmd->result = DID_RESET << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
esp_put_ent(esp, ent);
}
list_del(&ent->list);
cmd->result = DID_ABORT << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
esp_put_ent(esp, ent);
}
/* Do not support for bonding device */
- if (netdev->priv_flags & IFF_BONDING && netdev->flags & IFF_MASTER) {
+ if (netif_is_bond_master(netdev)) {
FCOE_NETDEV_DBG(netdev, "Bonded interfaces not supported\n");
return -EOPNOTSUPP;
}
{
outb(0, fd->base + REG_ICTL);
fdomain_make_bus_idle(fd);
- fd->cur_cmd->scsi_done(fd->cur_cmd);
+ scsi_done(fd->cur_cmd);
fd->cur_cmd = NULL;
}
extern struct workqueue_struct *fnic_event_queue;
extern struct workqueue_struct *fnic_fip_queue;
-extern struct device_attribute *fnic_attrs[];
+extern const struct attribute_group *fnic_host_groups[];
void fnic_clear_intr_mode(struct fnic *fnic);
int fnic_set_intr_mode(struct fnic *fnic);
static DEVICE_ATTR(drv_version, S_IRUGO, fnic_show_drv_version, NULL);
static DEVICE_ATTR(link_state, S_IRUGO, fnic_show_link_state, NULL);
-struct device_attribute *fnic_attrs[] = {
- &dev_attr_fnic_state,
- &dev_attr_drv_version,
- &dev_attr_link_state,
+static struct attribute *fnic_host_attrs[] = {
+ &dev_attr_fnic_state.attr,
+ &dev_attr_drv_version.attr,
+ &dev_attr_link_state.attr,
NULL,
};
+
+static const struct attribute_group fnic_host_attr_group = {
+ .attrs = fnic_host_attrs
+};
+
+const struct attribute_group *fnic_host_groups[] = {
+ &fnic_host_attr_group,
+ NULL
+};
.can_queue = FNIC_DFLT_IO_REQ,
.sg_tablesize = FNIC_MAX_SG_DESC_CNT,
.max_sectors = 0xffff,
- .shost_attrs = fnic_attrs,
+ .shost_groups = fnic_host_groups,
.track_queue_depth = 1,
};
* Routine to send a scsi cdb
* Called with host_lock held and interrupts disabled.
*/
-static int fnic_queuecommand_lck(struct scsi_cmnd *sc, void (*done)(struct scsi_cmnd *))
+static int fnic_queuecommand_lck(struct scsi_cmnd *sc)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
const int tag = scsi_cmd_to_rq(sc)->tag;
struct fc_lport *lp = shost_priv(sc->device->host);
struct fc_rport *rport;
CMD_STATE(sc) = FNIC_IOREQ_CMD_PENDING;
CMD_SP(sc) = (char *)io_req;
CMD_FLAGS(sc) |= FNIC_IO_INITIALIZED;
- sc->scsi_done = done;
/* create copy wq desc and enqueue it */
wq = &fnic->wq_copy[0];
}
/* Call SCSI completion function to complete the IO */
- if (sc->scsi_done)
- sc->scsi_done(sc);
+ scsi_done(sc);
}
/* fnic_fcpio_itmf_cmpl_handler
fnic_release_ioreq_buf(fnic, io_req, sc);
mempool_free(io_req, fnic->io_req_pool);
- if (sc->scsi_done) {
- FNIC_TRACE(fnic_fcpio_itmf_cmpl_handler,
- sc->device->host->host_no, id,
- sc,
- jiffies_to_msecs(jiffies - start_time),
- desc,
- (((u64)hdr_status << 40) |
- (u64)sc->cmnd[0] << 32 |
- (u64)sc->cmnd[2] << 24 |
- (u64)sc->cmnd[3] << 16 |
- (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
- (((u64)CMD_FLAGS(sc) << 32) |
- CMD_STATE(sc)));
- sc->scsi_done(sc);
- atomic64_dec(&fnic_stats->io_stats.active_ios);
- if (atomic64_read(&fnic->io_cmpl_skip))
- atomic64_dec(&fnic->io_cmpl_skip);
- else
- atomic64_inc(&fnic_stats->io_stats.io_completions);
- }
+ FNIC_TRACE(fnic_fcpio_itmf_cmpl_handler,
+ sc->device->host->host_no, id,
+ sc,
+ jiffies_to_msecs(jiffies - start_time),
+ desc,
+ (((u64)hdr_status << 40) |
+ (u64)sc->cmnd[0] << 32 |
+ (u64)sc->cmnd[2] << 24 |
+ (u64)sc->cmnd[3] << 16 |
+ (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
+ (((u64)CMD_FLAGS(sc) << 32) |
+ CMD_STATE(sc)));
+ scsi_done(sc);
+ atomic64_dec(&fnic_stats->io_stats.active_ios);
+ if (atomic64_read(&fnic->io_cmpl_skip))
+ atomic64_dec(&fnic->io_cmpl_skip);
+ else
+ atomic64_inc(&fnic_stats->io_stats.io_completions);
}
-
} else if (id & FNIC_TAG_DEV_RST) {
/* Completion of device reset */
CMD_LR_STATUS(sc) = hdr_status;
atomic64_inc(&fnic_stats->io_stats.io_completions);
/* Complete the command to SCSI */
- if (sc->scsi_done) {
- if (!(CMD_FLAGS(sc) & FNIC_IO_ISSUED))
- shost_printk(KERN_ERR, fnic->lport->host,
- "Calling done for IO not issued to fw: tag:0x%x sc:0x%p\n",
- tag, sc);
-
- FNIC_TRACE(fnic_cleanup_io,
- sc->device->host->host_no, tag, sc,
- jiffies_to_msecs(jiffies - start_time),
- 0, ((u64)sc->cmnd[0] << 32 |
- (u64)sc->cmnd[2] << 24 |
- (u64)sc->cmnd[3] << 16 |
- (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
- (((u64)CMD_FLAGS(sc) << 32) | CMD_STATE(sc)));
-
- sc->scsi_done(sc);
- }
+ if (!(CMD_FLAGS(sc) & FNIC_IO_ISSUED))
+ shost_printk(KERN_ERR, fnic->lport->host,
+ "Calling done for IO not issued to fw: tag:0x%x sc:0x%p\n",
+ tag, sc);
+
+ FNIC_TRACE(fnic_cleanup_io,
+ sc->device->host->host_no, tag, sc,
+ jiffies_to_msecs(jiffies - start_time),
+ 0, ((u64)sc->cmnd[0] << 32 |
+ (u64)sc->cmnd[2] << 24 |
+ (u64)sc->cmnd[3] << 16 |
+ (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
+ (((u64)CMD_FLAGS(sc) << 32) | CMD_STATE(sc)));
+
+ scsi_done(sc);
+
return true;
}
FNIC_SCSI_DBG(KERN_DEBUG, fnic->lport->host, "wq_copy_cleanup_handler:"
" DID_NO_CONNECT\n");
- if (sc->scsi_done) {
- FNIC_TRACE(fnic_wq_copy_cleanup_handler,
- sc->device->host->host_no, id, sc,
- jiffies_to_msecs(jiffies - start_time),
- 0, ((u64)sc->cmnd[0] << 32 |
- (u64)sc->cmnd[2] << 24 | (u64)sc->cmnd[3] << 16 |
- (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
- (((u64)CMD_FLAGS(sc) << 32) | CMD_STATE(sc)));
+ FNIC_TRACE(fnic_wq_copy_cleanup_handler,
+ sc->device->host->host_no, id, sc,
+ jiffies_to_msecs(jiffies - start_time),
+ 0, ((u64)sc->cmnd[0] << 32 |
+ (u64)sc->cmnd[2] << 24 | (u64)sc->cmnd[3] << 16 |
+ (u64)sc->cmnd[4] << 8 | sc->cmnd[5]),
+ (((u64)CMD_FLAGS(sc) << 32) | CMD_STATE(sc)));
- sc->scsi_done(sc);
- }
+ scsi_done(sc);
}
static inline int fnic_queue_abort_io_req(struct fnic *fnic, int tag,
fnic_release_ioreq_buf(fnic, io_req, sc);
mempool_free(io_req, fnic->io_req_pool);
- if (sc->scsi_done) {
/* Call SCSI completion function to complete the IO */
- sc->result = (DID_ABORT << 16);
- sc->scsi_done(sc);
- atomic64_dec(&fnic_stats->io_stats.active_ios);
- if (atomic64_read(&fnic->io_cmpl_skip))
- atomic64_dec(&fnic->io_cmpl_skip);
- else
- atomic64_inc(&fnic_stats->io_stats.io_completions);
- }
+ sc->result = DID_ABORT << 16;
+ scsi_done(sc);
+ atomic64_dec(&fnic_stats->io_stats.active_ios);
+ if (atomic64_read(&fnic->io_cmpl_skip))
+ atomic64_dec(&fnic->io_cmpl_skip);
+ else
+ atomic64_inc(&fnic_stats->io_stats.io_completions);
fnic_abort_cmd_end:
FNIC_TRACE(fnic_abort_cmd, sc->device->host->host_no, tag, sc,
* Any IO is returned during reset, it needs to call scsi_done
* to return the scsi_cmnd to upper layer.
*/
- if (sc->scsi_done) {
- /* Set result to let upper SCSI layer retry */
- sc->result = DID_RESET << 16;
- sc->scsi_done(sc);
- }
+ /* Set result to let upper SCSI layer retry */
+ sc->result = DID_RESET << 16;
+ scsi_done(sc);
+
return true;
}
#define HISI_SAS_QUEUE_SLOTS 4096
#define HISI_SAS_MAX_ITCT_ENTRIES 1024
#define HISI_SAS_MAX_DEVICES HISI_SAS_MAX_ITCT_ENTRIES
-#define HISI_SAS_RESET_BIT 0
+#define HISI_SAS_RESETTING_BIT 0
#define HISI_SAS_REJECT_CMD_BIT 1
#define HISI_SAS_PM_BIT 2
#define HISI_SAS_HW_FAULT_BIT 3
extern int hisi_sas_remove(struct platform_device *pdev);
extern int hisi_sas_slave_configure(struct scsi_device *sdev);
+extern int hisi_sas_slave_alloc(struct scsi_device *sdev);
extern int hisi_sas_scan_finished(struct Scsi_Host *shost, unsigned long time);
extern void hisi_sas_scan_start(struct Scsi_Host *shost);
extern int hisi_sas_host_reset(struct Scsi_Host *shost, int reset_type);
*/
local_phy = sas_get_local_phy(device);
if (!scsi_is_sas_phy_local(local_phy) &&
- !test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags)) {
+ !test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags)) {
unsigned long deadline = ata_deadline(jiffies, 20000);
struct sata_device *sata_dev = &device->sata_dev;
struct ata_host *ata_host = sata_dev->ata_host;
return rc;
}
+int hisi_sas_slave_alloc(struct scsi_device *sdev)
+{
+ struct domain_device *ddev;
+ int rc;
+
+ rc = sas_slave_alloc(sdev);
+ if (rc)
+ return rc;
+ ddev = sdev_to_domain_dev(sdev);
+
+ return hisi_sas_init_device(ddev);
+}
+EXPORT_SYMBOL_GPL(hisi_sas_slave_alloc);
+
static int hisi_sas_dev_found(struct domain_device *device)
{
struct hisi_hba *hisi_hba = dev_to_hisi_hba(device);
dev_info(dev, "dev[%d:%x] found\n",
sas_dev->device_id, sas_dev->dev_type);
- rc = hisi_sas_init_device(device);
- if (rc)
- goto err_out;
sas_dev->dev_status = HISI_SAS_DEV_NORMAL;
return 0;
sas_dev->device_id, sas_dev->dev_type);
down(&hisi_hba->sem);
- if (!test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags)) {
+ if (!test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags)) {
hisi_sas_internal_task_abort(hisi_hba, device,
HISI_SAS_INT_ABT_DEV, 0, true);
static int hisi_sas_control_phy(struct asd_sas_phy *sas_phy, enum phy_func func,
void *funcdata)
{
+ struct hisi_sas_phy *phy = container_of(sas_phy,
+ struct hisi_sas_phy, sas_phy);
struct sas_ha_struct *sas_ha = sas_phy->ha;
struct hisi_hba *hisi_hba = sas_ha->lldd_ha;
+ struct device *dev = hisi_hba->dev;
+ DECLARE_COMPLETION_ONSTACK(completion);
int phy_no = sas_phy->id;
+ u8 sts = phy->phy_attached;
+ int ret = 0;
+
+ phy->reset_completion = &completion;
switch (func) {
case PHY_FUNC_HARD_RESET:
case PHY_FUNC_DISABLE:
hisi_sas_phy_enable(hisi_hba, phy_no, 0);
- break;
+ goto out;
case PHY_FUNC_SET_LINK_RATE:
- return hisi_sas_phy_set_linkrate(hisi_hba, phy_no, funcdata);
+ ret = hisi_sas_phy_set_linkrate(hisi_hba, phy_no, funcdata);
+ break;
+
case PHY_FUNC_GET_EVENTS:
if (hisi_hba->hw->get_events) {
hisi_hba->hw->get_events(hisi_hba, phy_no);
- break;
+ goto out;
}
fallthrough;
case PHY_FUNC_RELEASE_SPINUP_HOLD:
default:
- return -EOPNOTSUPP;
+ ret = -EOPNOTSUPP;
+ goto out;
}
- return 0;
+
+ if (sts && !wait_for_completion_timeout(&completion, 2 * HZ)) {
+ dev_warn(dev, "phy%d wait phyup timed out for func %d\n",
+ phy_no, func);
+ if (phy->in_reset)
+ ret = -ETIMEDOUT;
+ }
+
+out:
+ phy->reset_completion = NULL;
+
+ return ret;
}
static void hisi_sas_task_done(struct sas_task *task)
{
- del_timer(&task->slow_task->timer);
+ del_timer_sync(&task->slow_task->timer);
complete(&task->slow_task->completion);
}
res = hisi_sas_task_exec(task, GFP_KERNEL, 1, tmf);
if (res) {
- del_timer(&task->slow_task->timer);
+ del_timer_sync(&task->slow_task->timer);
dev_err(dev, "abort tmf: executing internal task failed: %d\n",
res);
goto ex_err;
scsi_block_requests(shost);
hisi_hba->hw->wait_cmds_complete_timeout(hisi_hba, 100, 5000);
- if (timer_pending(&hisi_hba->timer))
- del_timer_sync(&hisi_hba->timer);
+ del_timer_sync(&hisi_hba->timer);
set_bit(HISI_SAS_REJECT_CMD_BIT, &hisi_hba->flags);
}
hisi_sas_reset_init_all_devices(hisi_hba);
up(&hisi_hba->sem);
scsi_unblock_requests(shost);
- clear_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags);
+ clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
hisi_sas_rescan_topology(hisi_hba, hisi_hba->phy_state);
}
if (!hisi_hba->hw->soft_reset)
return -1;
- if (test_and_set_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags))
+ if (test_and_set_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags))
return -1;
if (hisi_sas_debugfs_enable && hisi_hba->debugfs_itct[0].itct)
clear_bit(HISI_SAS_REJECT_CMD_BIT, &hisi_hba->flags);
up(&hisi_hba->sem);
scsi_unblock_requests(shost);
- clear_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags);
+ clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
return rc;
}
struct hisi_sas_device *sas_dev = device->lldd_dev;
struct hisi_hba *hisi_hba = dev_to_hisi_hba(device);
struct sas_ha_struct *sas_ha = &hisi_hba->sha;
- DECLARE_COMPLETION_ONSTACK(phyreset);
int rc, reset_type;
if (!local_phy->enabled) {
sas_ha->sas_phy[local_phy->number];
struct hisi_sas_phy *phy =
container_of(sas_phy, struct hisi_sas_phy, sas_phy);
+ unsigned long flags;
+
+ spin_lock_irqsave(&phy->lock, flags);
phy->in_reset = 1;
- phy->reset_completion = &phyreset;
+ spin_unlock_irqrestore(&phy->lock, flags);
}
reset_type = (sas_dev->dev_status == HISI_SAS_DEV_INIT ||
sas_ha->sas_phy[local_phy->number];
struct hisi_sas_phy *phy =
container_of(sas_phy, struct hisi_sas_phy, sas_phy);
- int ret = wait_for_completion_timeout(&phyreset,
- I_T_NEXUS_RESET_PHYUP_TIMEOUT);
unsigned long flags;
spin_lock_irqsave(&phy->lock, flags);
- phy->reset_completion = NULL;
phy->in_reset = 0;
spin_unlock_irqrestore(&phy->lock, flags);
/* report PHY down if timed out */
- if (!ret)
+ if (rc == -ETIMEDOUT)
hisi_sas_phy_down(hisi_hba, sas_phy->id, 0, GFP_KERNEL);
} else if (sas_dev->dev_status != HISI_SAS_DEV_INIT) {
/*
}
hisi_sas_dereg_device(hisi_hba, device);
- if (dev_is_sata(device)) {
+ rc = hisi_sas_debug_I_T_nexus_reset(device);
+ if (rc == TMF_RESP_FUNC_COMPLETE && dev_is_sata(device)) {
+ struct sas_phy *local_phy;
+
rc = hisi_sas_softreset_ata_disk(device);
- if (rc == TMF_RESP_FUNC_FAILED)
- return TMF_RESP_FUNC_FAILED;
+ switch (rc) {
+ case -ECOMM:
+ rc = -ENODEV;
+ break;
+ case TMF_RESP_FUNC_FAILED:
+ case -EMSGSIZE:
+ case -EIO:
+ local_phy = sas_get_local_phy(device);
+ rc = sas_phy_enable(local_phy, 0);
+ if (!rc) {
+ local_phy->enabled = 0;
+ dev_err(dev, "Disabled local phy of ATA disk %016llx due to softreset fail (%d)\n",
+ SAS_ADDR(device->sas_addr), rc);
+ rc = -ENODEV;
+ }
+ sas_put_local_phy(local_phy);
+ break;
+ default:
+ break;
+ }
}
- rc = hisi_sas_debug_I_T_nexus_reset(device);
-
if ((rc == TMF_RESP_FUNC_COMPLETE) || (rc == -ENODEV))
hisi_sas_release_task(hisi_hba, device);
res = hisi_sas_internal_abort_task_exec(hisi_hba, sas_dev->device_id,
task, abort_flag, tag, dq);
if (res) {
- del_timer(&task->slow_task->timer);
+ del_timer_sync(&task->slow_task->timer);
dev_err(dev, "internal task abort: executing internal task failed: %d\n",
res);
goto exit;
} else {
struct hisi_sas_port *port = phy->port;
- if (test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags) ||
+ if (test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags) ||
phy->in_reset) {
dev_info(dev, "ignore flutter phy%d down\n", phy_no);
return;
struct hisi_hba *hisi_hba = sha->lldd_ha;
struct Scsi_Host *shost = sha->core.shost;
- if (timer_pending(&hisi_hba->timer))
- del_timer(&hisi_hba->timer);
+ del_timer_sync(&hisi_hba->timer);
sas_unregister_ha(sha);
sas_remove_host(sha->core.shost);
u32 *frame_rcvd = (u32 *)sas_phy->frame_rcvd;
struct sas_identify_frame *id = (struct sas_identify_frame *)frame_rcvd;
irqreturn_t res = IRQ_HANDLED;
- unsigned long flags;
irq_value = hisi_sas_phy_read32(hisi_hba, phy_no, CHL_INT2);
if (!(irq_value & CHL_INT2_SL_PHY_ENA_MSK)) {
phy->identify.target_port_protocols =
SAS_PROTOCOL_SMP;
hisi_sas_notify_phy_event(phy, HISI_PHYE_PHY_UP);
-
- spin_lock_irqsave(&phy->lock, flags);
- if (phy->reset_completion) {
- phy->in_reset = 0;
- complete(phy->reset_completion);
- }
- spin_unlock_irqrestore(&phy->lock, flags);
-
end:
+ if (phy->reset_completion)
+ complete(phy->reset_completion);
hisi_sas_phy_write32(hisi_hba, phy_no, CHL_INT2,
CHL_INT2_SL_PHY_ENA_MSK);
goto end;
}
- if (!test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags))
+ if (!test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags))
sas_notify_port_event(sas_phy, PORTE_BROADCAST_RCVD,
GFP_ATOMIC);
return 0;
}
-static struct device_attribute *host_attrs_v1_hw[] = {
- &dev_attr_phy_event_threshold,
+static struct attribute *host_v1_hw_attrs[] = {
+ &dev_attr_phy_event_threshold.attr,
NULL
};
+ATTRIBUTE_GROUPS(host_v1_hw);
+
static struct scsi_host_template sht_v1_hw = {
.name = DRV_NAME,
.proc_name = DRV_NAME,
.max_sectors = SCSI_DEFAULT_MAX_SECTORS,
.eh_device_reset_handler = sas_eh_device_reset_handler,
.eh_target_reset_handler = sas_eh_target_reset_handler,
- .slave_alloc = sas_slave_alloc,
+ .slave_alloc = hisi_sas_slave_alloc,
.target_destroy = sas_target_destroy,
.ioctl = sas_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = host_attrs_v1_hw,
+ .shost_groups = host_v1_hw_groups,
.host_reset = hisi_sas_host_reset,
};
case STAT_IO_COMPLETE:
/* internal abort command complete */
ts->stat = TMF_RESP_FUNC_SUCC;
- del_timer(&slot->internal_abort_timer);
+ del_timer_sync(&slot->internal_abort_timer);
goto out;
case STAT_IO_NO_DEVICE:
ts->stat = TMF_RESP_FUNC_COMPLETE;
- del_timer(&slot->internal_abort_timer);
+ del_timer_sync(&slot->internal_abort_timer);
goto out;
case STAT_IO_NOT_VALID:
/* abort single io, controller don't find
* the io need to abort
*/
ts->stat = TMF_RESP_FUNC_FAILED;
- del_timer(&slot->internal_abort_timer);
+ del_timer_sync(&slot->internal_abort_timer);
goto out;
default:
break;
struct device *dev = hisi_hba->dev;
u32 *frame_rcvd = (u32 *)sas_phy->frame_rcvd;
struct sas_identify_frame *id = (struct sas_identify_frame *)frame_rcvd;
- unsigned long flags;
hisi_sas_phy_write32(hisi_hba, phy_no, PHYCTRL_PHY_ENA_MSK, 1);
set_link_timer_quirk(hisi_hba);
}
hisi_sas_notify_phy_event(phy, HISI_PHYE_PHY_UP);
- spin_lock_irqsave(&phy->lock, flags);
- if (phy->reset_completion) {
- phy->in_reset = 0;
- complete(phy->reset_completion);
- }
- spin_unlock_irqrestore(&phy->lock, flags);
-
end:
+ if (phy->reset_completion)
+ complete(phy->reset_completion);
hisi_sas_phy_write32(hisi_hba, phy_no, CHL_INT0,
CHL_INT0_SL_PHY_ENABLE_MSK);
hisi_sas_phy_write32(hisi_hba, phy_no, PHYCTRL_PHY_ENA_MSK, 0);
hisi_sas_phy_write32(hisi_hba, phy_no, SL_RX_BCAST_CHK_MSK, 1);
bcast_status = hisi_sas_phy_read32(hisi_hba, phy_no, RX_PRIMS_STATUS);
if ((bcast_status & RX_BCAST_CHG_MSK) &&
- !test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags))
+ !test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags))
sas_notify_port_event(sas_phy, PORTE_BROADCAST_RCVD,
GFP_ATOMIC);
hisi_sas_phy_write32(hisi_hba, phy_no, CHL_INT0,
u32 ent_tmp, ent_msk, ent_int, port_id, link_rate, hard_phy_linkrate;
irqreturn_t res = IRQ_HANDLED;
u8 attached_sas_addr[SAS_ADDR_SIZE] = {0};
- unsigned long flags;
int phy_no, offset;
del_timer(&phy->timer);
phy->identify.target_port_protocols = SAS_PROTOCOL_SATA;
hisi_sas_notify_phy_event(phy, HISI_PHYE_PHY_UP);
- spin_lock_irqsave(&phy->lock, flags);
- if (phy->reset_completion) {
- phy->in_reset = 0;
+ if (phy->reset_completion)
complete(phy->reset_completion);
- }
- spin_unlock_irqrestore(&phy->lock, flags);
end:
hisi_sas_write32(hisi_hba, ENT_INT_SRC1 + offset, ent_tmp);
hisi_sas_write32(hisi_hba, ENT_INT_SRC_MSK1 + offset, ent_msk);
}
-static struct device_attribute *host_attrs_v2_hw[] = {
- &dev_attr_phy_event_threshold,
+static struct attribute *host_v2_hw_attrs[] = {
+ &dev_attr_phy_event_threshold.attr,
NULL
};
+ATTRIBUTE_GROUPS(host_v2_hw);
+
static int map_queues_v2_hw(struct Scsi_Host *shost)
{
struct hisi_hba *hisi_hba = shost_priv(shost);
.max_sectors = SCSI_DEFAULT_MAX_SECTORS,
.eh_device_reset_handler = sas_eh_device_reset_handler,
.eh_target_reset_handler = sas_eh_target_reset_handler,
- .slave_alloc = sas_slave_alloc,
+ .slave_alloc = hisi_sas_slave_alloc,
.target_destroy = sas_target_destroy,
.ioctl = sas_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = host_attrs_v2_hw,
+ .shost_groups = host_v2_hw_groups,
.host_reset = hisi_sas_host_reset,
.map_queues = map_queues_v2_hw,
.host_tagset = 1,
#define CHNL_INT_STS_INT2_MSK BIT(3)
#define CHNL_WIDTH 4
+#define BAR_NO_V3_HW 5
+
enum {
DSM_FUNC_ERR_HANDLE_MSI = 0,
};
struct hisi_sas_phy *phy = &hisi_hba->phy[phy_no];
struct asd_sas_phy *sas_phy = &phy->sas_phy;
struct device *dev = hisi_hba->dev;
- unsigned long flags;
del_timer(&phy->timer);
hisi_sas_phy_write32(hisi_hba, phy_no, PHYCTRL_PHY_ENA_MSK, 1);
phy->phy_attached = 1;
hisi_sas_notify_phy_event(phy, HISI_PHYE_PHY_UP);
res = IRQ_HANDLED;
- spin_lock_irqsave(&phy->lock, flags);
- if (phy->reset_completion) {
- phy->in_reset = 0;
- complete(phy->reset_completion);
- }
- spin_unlock_irqrestore(&phy->lock, flags);
end:
+ if (phy->reset_completion)
+ complete(phy->reset_completion);
hisi_sas_phy_write32(hisi_hba, phy_no, CHL_INT0,
CHL_INT0_SL_PHY_ENABLE_MSK);
hisi_sas_phy_write32(hisi_hba, phy_no, PHYCTRL_PHY_ENA_MSK, 0);
hisi_sas_phy_write32(hisi_hba, phy_no, SL_RX_BCAST_CHK_MSK, 1);
bcast_status = hisi_sas_phy_read32(hisi_hba, phy_no, RX_PRIMS_STATUS);
if ((bcast_status & RX_BCAST_CHG_MSK) &&
- !test_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags))
+ !test_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags))
sas_notify_port_event(sas_phy, PORTE_BROADCAST_RCVD,
GFP_ATOMIC);
hisi_sas_phy_write32(hisi_hba, phy_no, CHL_INT0,
return 0;
}
-static struct device_attribute *host_attrs_v3_hw[] = {
- &dev_attr_phy_event_threshold,
- &dev_attr_intr_conv_v3_hw,
- &dev_attr_intr_coal_ticks_v3_hw,
- &dev_attr_intr_coal_count_v3_hw,
+static struct attribute *host_v3_hw_attrs[] = {
+ &dev_attr_phy_event_threshold.attr,
+ &dev_attr_intr_conv_v3_hw.attr,
+ &dev_attr_intr_coal_ticks_v3_hw.attr,
+ &dev_attr_intr_coal_count_v3_hw.attr,
NULL
};
+ATTRIBUTE_GROUPS(host_v3_hw);
+
#define HISI_SAS_DEBUGFS_REG(x) {#x, x}
struct hisi_sas_debugfs_reg_lu {
.max_sectors = SCSI_DEFAULT_MAX_SECTORS,
.eh_device_reset_handler = sas_eh_device_reset_handler,
.eh_target_reset_handler = sas_eh_target_reset_handler,
- .slave_alloc = sas_slave_alloc,
+ .slave_alloc = hisi_sas_slave_alloc,
.target_destroy = sas_target_destroy,
.ioctl = sas_ioctl,
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = host_attrs_v3_hw,
+ .shost_groups = host_v3_hw_groups,
.tag_alloc_policy = BLK_TAG_ALLOC_RR,
.host_reset = hisi_sas_host_reset,
.host_tagset = 1,
do_div(timestamp, NSEC_PER_MSEC);
hisi_hba->debugfs_timestamp[debugfs_dump_index] = timestamp;
- hisi_hba->debugfs_dump_index++;
debugfs_snapshot_prepare_v3_hw(hisi_hba);
debugfs_create_files_v3_hw(hisi_hba);
debugfs_snapshot_restore_v3_hw(hisi_hba);
+ hisi_hba->debugfs_dump_index++;
}
static ssize_t debugfs_trigger_dump_v3_hw_write(struct file *file,
struct sas_ha_struct *sha;
int rc, phy_nr, port_nr, i;
- rc = pci_enable_device(pdev);
+ rc = pcim_enable_device(pdev);
if (rc)
goto err_out;
pci_set_master(pdev);
- rc = pci_request_regions(pdev, DRV_NAME);
+ rc = pcim_iomap_regions(pdev, 1 << BAR_NO_V3_HW, DRV_NAME);
if (rc)
- goto err_out_disable_device;
+ goto err_out;
rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
if (rc)
if (rc) {
dev_err(dev, "No usable DMA addressing method\n");
rc = -ENODEV;
- goto err_out_regions;
+ goto err_out;
}
shost = hisi_sas_shost_alloc_pci(pdev);
if (!shost) {
rc = -ENOMEM;
- goto err_out_regions;
+ goto err_out;
}
sha = SHOST_TO_SAS_HA(shost);
hisi_hba = shost_priv(shost);
dev_set_drvdata(dev, sha);
- hisi_hba->regs = pcim_iomap(pdev, 5, 0);
+ hisi_hba->regs = pcim_iomap_table(pdev)[BAR_NO_V3_HW];
if (!hisi_hba->regs) {
dev_err(dev, "cannot map register\n");
rc = -ENOMEM;
rc = interrupt_preinit_v3_hw(hisi_hba);
if (rc)
goto err_out_debugfs;
- dev_err(dev, "%d hw queues\n", shost->nr_hw_queues);
+
rc = scsi_add_host(shost, dev);
if (rc)
goto err_out_free_irq_vectors;
err_out_ha:
hisi_sas_free(hisi_hba);
scsi_host_put(shost);
-err_out_regions:
- pci_release_regions(pdev);
-err_out_disable_device:
- pci_disable_device(pdev);
err_out:
return rc;
}
struct Scsi_Host *shost = sha->core.shost;
pm_runtime_get_noresume(dev);
- if (timer_pending(&hisi_hba->timer))
- del_timer(&hisi_hba->timer);
+ del_timer_sync(&hisi_hba->timer);
sas_unregister_ha(sha);
flush_workqueue(hisi_hba->wq);
sas_remove_host(sha->core.shost);
hisi_sas_v3_destroy_irqs(pdev, hisi_hba);
- pci_release_regions(pdev);
- pci_disable_device(pdev);
hisi_sas_free(hisi_hba);
debugfs_exit_v3_hw(hisi_hba);
scsi_host_put(shost);
int rc;
dev_info(dev, "FLR prepare\n");
- set_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags);
+ set_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
hisi_sas_controller_reset_prepare(hisi_hba);
rc = disable_host_v3_hw(hisi_hba);
return -ENODEV;
}
- if (test_and_set_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags))
+ if (test_and_set_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags))
return -1;
scsi_block_requests(shost);
if (rc) {
dev_err(dev, "PM suspend: disable host failed rc=%d\n", rc);
clear_bit(HISI_SAS_REJECT_CMD_BIT, &hisi_hba->flags);
- clear_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags);
+ clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
scsi_unblock_requests(shost);
return rc;
}
}
phys_init_v3_hw(hisi_hba);
sas_resume_ha(sha);
- clear_bit(HISI_SAS_RESET_BIT, &hisi_hba->flags);
+ clear_bit(HISI_SAS_RESETTING_BIT, &hisi_hba->flags);
return 0;
}
struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
{
struct Scsi_Host *shost;
- int index;
+ int index, i, j = 0;
shost = kzalloc(sizeof(struct Scsi_Host) + privsize, GFP_KERNEL);
if (!shost)
shost->shost_state = SHOST_CREATED;
INIT_LIST_HEAD(&shost->__devices);
INIT_LIST_HEAD(&shost->__targets);
+ INIT_LIST_HEAD(&shost->eh_abort_list);
INIT_LIST_HEAD(&shost->eh_cmd_q);
INIT_LIST_HEAD(&shost->starved_list);
init_waitqueue_head(&shost->host_wait);
dev_set_name(&shost->shost_gendev, "host%d", shost->host_no);
shost->shost_gendev.bus = &scsi_bus_type;
shost->shost_gendev.type = &scsi_host_type;
+ scsi_enable_async_suspend(&shost->shost_gendev);
device_initialize(&shost->shost_dev);
shost->shost_dev.parent = &shost->shost_gendev;
shost->shost_dev.class = &shost_class;
dev_set_name(&shost->shost_dev, "host%d", shost->host_no);
- shost->shost_dev.groups = scsi_sysfs_shost_attr_groups;
+ shost->shost_dev.groups = shost->shost_dev_attr_groups;
+ shost->shost_dev_attr_groups[j++] = &scsi_shost_attr_group;
+ if (sht->shost_groups) {
+ for (i = 0; sht->shost_groups[i] &&
+ j < ARRAY_SIZE(shost->shost_dev_attr_groups);
+ i++, j++) {
+ shost->shost_dev_attr_groups[j] =
+ sht->shost_groups[i];
+ }
+ }
+ WARN_ON_ONCE(j >= ARRAY_SIZE(shost->shost_dev_attr_groups));
shost->ehandler = kthread_run(scsi_error_handler, shost,
"scsi_eh_%d", shost->host_no);
scsi_dma_unmap(scmd);
scmd->result = 0;
set_host_byte(scmd, status);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return true;
}
static DEVICE_ATTR(legacy_board, S_IRUGO,
host_show_legacy_board, NULL);
-static struct device_attribute *hpsa_sdev_attrs[] = {
- &dev_attr_raid_level,
- &dev_attr_lunid,
- &dev_attr_unique_id,
- &dev_attr_hp_ssd_smart_path_enabled,
- &dev_attr_path_info,
- &dev_attr_sas_address,
+static struct attribute *hpsa_sdev_attrs[] = {
+ &dev_attr_raid_level.attr,
+ &dev_attr_lunid.attr,
+ &dev_attr_unique_id.attr,
+ &dev_attr_hp_ssd_smart_path_enabled.attr,
+ &dev_attr_path_info.attr,
+ &dev_attr_sas_address.attr,
NULL,
};
-static struct device_attribute *hpsa_shost_attrs[] = {
- &dev_attr_rescan,
- &dev_attr_firmware_revision,
- &dev_attr_commands_outstanding,
- &dev_attr_transport_mode,
- &dev_attr_resettable,
- &dev_attr_hp_ssd_smart_path_status,
- &dev_attr_raid_offload_debug,
- &dev_attr_lockup_detected,
- &dev_attr_ctlr_num,
- &dev_attr_legacy_board,
+ATTRIBUTE_GROUPS(hpsa_sdev);
+
+static struct attribute *hpsa_shost_attrs[] = {
+ &dev_attr_rescan.attr,
+ &dev_attr_firmware_revision.attr,
+ &dev_attr_commands_outstanding.attr,
+ &dev_attr_transport_mode.attr,
+ &dev_attr_resettable.attr,
+ &dev_attr_hp_ssd_smart_path_status.attr,
+ &dev_attr_raid_offload_debug.attr,
+ &dev_attr_lockup_detected.attr,
+ &dev_attr_ctlr_num.attr,
+ &dev_attr_legacy_board.attr,
NULL,
};
+ATTRIBUTE_GROUPS(hpsa_shost);
+
#define HPSA_NRESERVED_CMDS (HPSA_CMDS_RESERVED_FOR_DRIVER +\
HPSA_MAX_CONCURRENT_PASSTHRUS)
#ifdef CONFIG_COMPAT
.compat_ioctl = hpsa_compat_ioctl,
#endif
- .sdev_attrs = hpsa_sdev_attrs,
- .shost_attrs = hpsa_shost_attrs,
+ .sdev_groups = hpsa_sdev_groups,
+ .shost_groups = hpsa_shost_groups,
.max_sectors = 2048,
.no_write_same = 1,
};
struct CommandList *c, struct scsi_cmnd *cmd)
{
hpsa_cmd_resolve_and_free(h, c);
- if (cmd && cmd->scsi_done)
- cmd->scsi_done(cmd);
+ if (cmd)
+ scsi_done(cmd);
}
static void hpsa_retry_cmd(struct ctlr_info *h, struct CommandList *c)
* if it encountered a dma mapping failure.
*/
cmd->result = DID_IMM_RETRY << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
dev = cmd->device->hostdata;
if (!dev) {
cmd->result = DID_NO_CONNECT << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
if (dev->removed) {
cmd->result = DID_NO_CONNECT << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
if (unlikely(lockup_detected(h))) {
cmd->result = DID_NO_CONNECT << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
skip_resid:
dprintk("scsi_done(%p)\n", scp);
- scp->scsi_done(scp);
+ scsi_done(scp);
free_req(hba, &hba->reqs[tag]);
}
return 0;
}
-static int hptiop_queuecommand_lck(struct scsi_cmnd *scp,
- void (*done)(struct scsi_cmnd *))
+static int hptiop_queuecommand_lck(struct scsi_cmnd *scp)
{
struct Scsi_Host *host = scp->device->host;
struct hptiop_hba *hba = (struct hptiop_hba *)host->hostdata;
int sg_count = 0;
struct hptiop_request *_req;
- BUG_ON(!done);
- scp->scsi_done = done;
-
_req = get_req(hba);
if (_req == NULL) {
dprintk("hptiop_queuecmd : no free req\n");
cmd_done:
dprintk("scsi_done(scp=%p)\n", scp);
- scp->scsi_done(scp);
+ scsi_done(scp);
return 0;
}
.show = hptiop_show_fw_version,
};
-static struct device_attribute *hptiop_attrs[] = {
- &hptiop_attr_version,
- &hptiop_attr_fw_version,
+static struct attribute *hptiop_host_attrs[] = {
+ &hptiop_attr_version.attr,
+ &hptiop_attr_fw_version.attr,
NULL
};
+ATTRIBUTE_GROUPS(hptiop_host);
+
static int hptiop_slave_config(struct scsi_device *sdev)
{
if (sdev->type == TYPE_TAPE)
.info = hptiop_info,
.emulated = 0,
.proc_name = driver_name,
- .shost_attrs = hptiop_attrs,
+ .shost_groups = hptiop_host_groups,
.slave_configure = hptiop_slave_config,
.this_id = -1,
.change_queue_depth = hptiop_adjust_disk_queue_depth,
if (cmnd) {
scsi_dma_unmap(cmnd);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
}
ibmvfc_free_event(evt);
cmnd->result = (DID_ERROR << 16);
scsi_dma_unmap(cmnd);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
}
ibmvfc_free_event(evt);
if (unlikely((rc = fc_remote_port_chkready(rport))) ||
unlikely((rc = ibmvfc_host_chkready(vhost)))) {
cmnd->result = rc;
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
"Failed to map DMA buffer for command. rc=%d\n", rc);
cmnd->result = DID_ERROR << 16;
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
};
#endif
-static struct device_attribute *ibmvfc_attrs[] = {
- &dev_attr_partition_name,
- &dev_attr_device_name,
- &dev_attr_port_loc_code,
- &dev_attr_drc_name,
- &dev_attr_npiv_version,
- &dev_attr_capabilities,
- &dev_attr_log_level,
- &dev_attr_nr_scsi_channels,
+static struct attribute *ibmvfc_host_attrs[] = {
+ &dev_attr_partition_name.attr,
+ &dev_attr_device_name.attr,
+ &dev_attr_port_loc_code.attr,
+ &dev_attr_drc_name.attr,
+ &dev_attr_npiv_version.attr,
+ &dev_attr_capabilities.attr,
+ &dev_attr_log_level.attr,
+ &dev_attr_nr_scsi_channels.attr,
NULL
};
+ATTRIBUTE_GROUPS(ibmvfc_host);
+
static struct scsi_host_template driver_template = {
.module = THIS_MODULE,
.name = "IBM POWER Virtual FC Adapter",
.this_id = -1,
.sg_tablesize = SG_ALL,
.max_sectors = IBMVFC_MAX_SECTORS,
- .shost_attrs = ibmvfc_attrs,
+ .shost_groups = ibmvfc_host_groups,
.track_queue_depth = 1,
.host_tagset = 1,
};
pool->iu_storage =
dma_alloc_coherent(hostdata->dev,
pool->size * sizeof(*pool->iu_storage),
- &pool->iu_token, 0);
+ &pool->iu_token, GFP_KERNEL);
if (!pool->iu_storage) {
kfree(pool->events);
return -ENOMEM;
* @cmnd: struct scsi_cmnd to be executed
* @done: Callback function to be called when cmd is completed
*/
-static int ibmvscsi_queuecommand_lck(struct scsi_cmnd *cmnd,
- void (*done) (struct scsi_cmnd *))
+static int ibmvscsi_queuecommand_lck(struct scsi_cmnd *cmnd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct srp_cmd *srp_cmd;
struct srp_event_struct *evt_struct;
struct srp_indirect_buf *indirect;
return 0;
}
-static struct device_attribute *ibmvscsi_attrs[] = {
- &ibmvscsi_host_vhost_loc,
- &ibmvscsi_host_vhost_name,
- &ibmvscsi_host_srp_version,
- &ibmvscsi_host_partition_name,
- &ibmvscsi_host_partition_number,
- &ibmvscsi_host_mad_version,
- &ibmvscsi_host_os_type,
- &ibmvscsi_host_config,
+static struct attribute *ibmvscsi_host_attrs[] = {
+ &ibmvscsi_host_vhost_loc.attr,
+ &ibmvscsi_host_vhost_name.attr,
+ &ibmvscsi_host_srp_version.attr,
+ &ibmvscsi_host_partition_name.attr,
+ &ibmvscsi_host_partition_number.attr,
+ &ibmvscsi_host_mad_version.attr,
+ &ibmvscsi_host_os_type.attr,
+ &ibmvscsi_host_config.attr,
NULL
};
+ATTRIBUTE_GROUPS(ibmvscsi_host);
+
/* ------------------------------------------------------------
* SCSI driver registration
*/
.can_queue = IBMVSCSI_MAX_REQUESTS_DEFAULT,
.this_id = -1,
.sg_tablesize = SG_ALL,
- .shost_attrs = ibmvscsi_attrs,
+ .shost_groups = ibmvscsi_host_groups,
};
/**
NULL,
};
-static ssize_t ibmvscsis_tpg_enable_show(struct config_item *item,
- char *page)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct ibmvscsis_tport *tport = container_of(se_tpg,
- struct ibmvscsis_tport,
- se_tpg);
- return snprintf(page, PAGE_SIZE, "%d\n", (tport->enabled) ? 1 : 0);
-}
-
-static ssize_t ibmvscsis_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
+static int ibmvscsis_enable_tpg(struct se_portal_group *se_tpg, bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct ibmvscsis_tport *tport = container_of(se_tpg,
struct ibmvscsis_tport,
se_tpg);
struct scsi_info *vscsi = container_of(tport, struct scsi_info, tport);
- unsigned long tmp;
- int rc;
long lrc;
- rc = kstrtoul(page, 0, &tmp);
- if (rc < 0) {
- dev_err(&vscsi->dev, "Unable to extract srpt_tpg_store_enable\n");
- return -EINVAL;
- }
-
- if ((tmp != 0) && (tmp != 1)) {
- dev_err(&vscsi->dev, "Illegal value for srpt_tpg_store_enable\n");
- return -EINVAL;
- }
-
- if (tmp) {
+ if (enable) {
spin_lock_bh(&vscsi->intr_lock);
tport->enabled = true;
lrc = ibmvscsis_enable_change_state(vscsi);
spin_unlock_bh(&vscsi->intr_lock);
}
- dev_dbg(&vscsi->dev, "tpg_enable_store, tmp %ld, state %d\n", tmp,
- vscsi->state);
-
- return count;
+ return 0;
}
-CONFIGFS_ATTR(ibmvscsis_tpg_, enable);
-
-static struct configfs_attribute *ibmvscsis_tpg_attrs[] = {
- &ibmvscsis_tpg_attr_enable,
- NULL,
-};
static const struct target_core_fabric_ops ibmvscsis_ops = {
.module = THIS_MODULE,
.fabric_make_wwn = ibmvscsis_make_tport,
.fabric_drop_wwn = ibmvscsis_drop_tport,
.fabric_make_tpg = ibmvscsis_make_tpg,
+ .fabric_enable_tpg = ibmvscsis_enable_tpg,
.fabric_drop_tpg = ibmvscsis_drop_tpg,
.tfc_wwn_attrs = ibmvscsis_wwn_attrs,
- .tfc_tpg_base_attrs = ibmvscsis_tpg_attrs,
};
static void ibmvscsis_dev_release(struct device *dev) {};
spin_lock_irqsave(host->host_lock, flags);
dev->cur_cmd = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
spin_unlock_irqrestore(host->host_lock, flags);
return;
}
return 0;
}
-static int imm_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done)(struct scsi_cmnd *))
+static int imm_queuecommand_lck(struct scsi_cmnd *cmd)
{
imm_struct *dev = imm_dev(cmd->device->host);
dev->failed = 0;
dev->jstart = jiffies;
dev->cur_cmd = cmd;
- cmd->scsi_done = done;
cmd->result = DID_ERROR << 16; /* default return code */
cmd->SCp.phase = 0; /* bus free */
* will cause the mid layer to call us again later with the command)
*/
-static int i91u_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done)(struct scsi_cmnd *))
+static int i91u_queuecommand_lck(struct scsi_cmnd *cmd)
{
struct initio_host *host = (struct initio_host *) cmd->device->host->hostdata;
struct scsi_ctrl_blk *cmnd;
- cmd->scsi_done = done;
-
cmnd = initio_alloc_scb(host);
if (!cmnd)
return SCSI_MLQUEUE_HOST_BUSY;
cmnd->result = cblk->tastat | (cblk->hastat << 16);
i91u_unmap_scb(host->pci_dev, cmnd);
- cmnd->scsi_done(cmnd); /* Notify system DONE */
+ scsi_done(cmnd); /* Notify system DONE */
initio_release_scb(host, cblk); /* Release SCB for current channel */
}
scsi_cmd->result |= (DID_ERROR << 16);
scsi_dma_unmap(ipr_cmd->scsi_cmd);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
if (ipr_cmd->eh_comp)
complete(ipr_cmd->eh_comp);
list_add_tail(&ipr_cmd->queue, &ipr_cmd->hrrq->hrrq_free_q);
.write = ipr_next_async_err_log
};
-static struct device_attribute *ipr_ioa_attrs[] = {
- &ipr_fw_version_attr,
- &ipr_log_level_attr,
- &ipr_diagnostics_attr,
- &ipr_ioa_state_attr,
- &ipr_ioa_reset_attr,
- &ipr_update_fw_attr,
- &ipr_ioa_fw_type_attr,
- &ipr_iopoll_weight_attr,
+static struct attribute *ipr_ioa_attrs[] = {
+ &ipr_fw_version_attr.attr,
+ &ipr_log_level_attr.attr,
+ &ipr_diagnostics_attr.attr,
+ &ipr_ioa_state_attr.attr,
+ &ipr_ioa_reset_attr.attr,
+ &ipr_update_fw_attr.attr,
+ &ipr_ioa_fw_type_attr.attr,
+ &ipr_iopoll_weight_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(ipr_ioa);
+
#ifdef CONFIG_SCSI_IPR_DUMP
/**
* ipr_read_dump - Dump the adapter
.store = ipr_store_raw_mode
};
-static struct device_attribute *ipr_dev_attrs[] = {
- &ipr_adapter_handle_attr,
- &ipr_resource_path_attr,
- &ipr_device_id_attr,
- &ipr_resource_type_attr,
- &ipr_raw_mode_attr,
+static struct attribute *ipr_dev_attrs[] = {
+ &ipr_adapter_handle_attr.attr,
+ &ipr_resource_path_attr.attr,
+ &ipr_device_id_attr.attr,
+ &ipr_resource_type_attr.attr,
+ &ipr_raw_mode_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(ipr_dev);
+
/**
* ipr_biosparam - Return the HSC mapping
* @sdev: scsi device struct
res->in_erp = 0;
}
scsi_dma_unmap(ipr_cmd->scsi_cmd);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
if (ipr_cmd->eh_comp)
complete(ipr_cmd->eh_comp);
list_add_tail(&ipr_cmd->queue, &ipr_cmd->hrrq->hrrq_free_q);
}
scsi_dma_unmap(ipr_cmd->scsi_cmd);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
if (ipr_cmd->eh_comp)
complete(ipr_cmd->eh_comp);
list_add_tail(&ipr_cmd->queue, &ipr_cmd->hrrq->hrrq_free_q);
scsi_dma_unmap(scsi_cmd);
spin_lock_irqsave(ipr_cmd->hrrq->lock, lock_flags);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
if (ipr_cmd->eh_comp)
complete(ipr_cmd->eh_comp);
list_add_tail(&ipr_cmd->queue, &ipr_cmd->hrrq->hrrq_free_q);
spin_lock_irqsave(hrrq->lock, hrrq_flags);
memset(scsi_cmd->sense_buffer, 0, SCSI_SENSE_BUFFERSIZE);
scsi_cmd->result = (DID_NO_CONNECT << 16);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
spin_unlock_irqrestore(hrrq->lock, hrrq_flags);
return 0;
}
.sg_tablesize = IPR_MAX_SGLIST,
.max_sectors = IPR_IOA_MAX_SECTORS,
.cmd_per_lun = IPR_MAX_CMD_PER_LUN,
- .shost_attrs = ipr_ioa_attrs,
- .sdev_attrs = ipr_dev_attrs,
+ .shost_groups = ipr_ioa_groups,
+ .sdev_groups = ipr_dev_groups,
.proc_name = IPR_NAME,
};
while ((scb = ips_removeq_scb_head(&ha->scb_activelist))) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
ips_freescb(ha, scb);
}
while ((scsi_cmd = ips_removeq_wait_head(&ha->scb_waitlist))) {
scsi_cmd->result = DID_ERROR;
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
}
ha->active = FALSE;
while ((scb = ips_removeq_scb_head(&ha->scb_activelist))) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
ips_freescb(ha, scb);
}
while ((scsi_cmd = ips_removeq_wait_head(&ha->scb_waitlist))) {
scsi_cmd->result = DID_ERROR << 16;
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
}
ha->active = FALSE;
while ((scb = ips_removeq_scb_head(&ha->scb_activelist))) {
scb->scsi_cmd->result = DID_RESET << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
ips_freescb(ha, scb);
}
/* Linux obtains io_request_lock before calling this function */
/* */
/****************************************************************************/
-static int ips_queue_lck(struct scsi_cmnd *SC, void (*done) (struct scsi_cmnd *))
+static int ips_queue_lck(struct scsi_cmnd *SC)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
ips_ha_t *ha;
ips_passthru_t *pt;
return (0);
}
- SC->scsi_done = done;
-
DEBUG_VAR(2, "(%s%d): ips_queue: cmd 0x%X (%d %d %d)",
ips_name,
ha->host_num,
ha->ioctl_reset = 1; /* This reset request is from an IOCTL */
__ips_eh_reset(SC);
SC->result = DID_OK << 16;
- SC->scsi_done(SC);
+ scsi_done(SC);
return (0);
}
case IPS_FAILURE:
if (scb->scsi_cmd) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
}
ips_freescb(ha, scb);
case IPS_SUCCESS_IMM:
if (scb->scsi_cmd) {
scb->scsi_cmd->result = DID_OK << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
}
ips_freescb(ha, scb);
case IPS_FAILURE:
if (scb->scsi_cmd) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
}
if (scb->bus)
break;
case IPS_SUCCESS_IMM:
if (scb->scsi_cmd)
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
if (scb->bus)
ha->dcdb_active[scb->bus - 1] &=
case IPS_FAILURE:
if (scb->scsi_cmd) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
}
ips_freescb(ha, scb);
case IPS_SUCCESS_IMM:
if (scb->scsi_cmd) {
scb->scsi_cmd->result = DID_ERROR << 16;
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
}
ips_freescb(ha, scb);
ha->dcdb_active[scb->bus - 1] &= ~(1 << scb->target_id);
}
- scb->scsi_cmd->scsi_done(scb->scsi_cmd);
+ scsi_done(scb->scsi_cmd);
ips_freescb(ha, scb);
}
static DEVICE_ATTR(isci_id, S_IRUGO, isci_show_id, NULL);
-static struct device_attribute *isci_host_attrs[] = {
- &dev_attr_isci_id,
+static struct attribute *isci_host_attrs[] = {
+ &dev_attr_isci_id.attr,
NULL
};
+ATTRIBUTE_GROUPS(isci_host);
+
static struct scsi_host_template isci_sht = {
.module = THIS_MODULE,
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = isci_host_attrs,
+ .shost_groups = isci_host_groups,
.track_queue_depth = 1,
};
u32 isci_task_ssp_request_get_response_data_length(
struct isci_request *request);
-int isci_queuecommand(
- struct scsi_cmnd *scsi_cmd,
- void (*donefunc)(struct scsi_cmnd *));
-
#endif /* !defined(_SCI_TASK_H_) */
rval = fc_remote_port_chkready(rport);
if (rval) {
sc_cmd->result = rval;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
* online
*/
sc_cmd->result = DID_IMM_RETRY << 16;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
goto out;
}
list_del(&fsp->list);
sc_cmd->SCp.ptr = NULL;
spin_unlock_irqrestore(&si->scsi_queue_lock, flags);
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
/* release ref from initial allocation in queue command */
fc_fcp_pkt_release(fsp);
* it will decide how to return sc to scsi-ml.
*/
if (oldstate != ISCSI_TASK_REQUEUE_SCSIQ)
- sc->scsi_done(sc);
+ scsi_done(sc);
}
}
ISCSI_DBG_SESSION(session, "iscsi: cmd 0x%x is not queued (%d)\n",
sc->cmnd[0], reason);
scsi_set_resid(sc, scsi_bufflen(sc));
- sc->scsi_done(sc);
+ scsi_done(sc);
return 0;
}
EXPORT_SYMBOL_GPL(iscsi_queuecommand);
session->tmf_state = TMF_INITIAL;
timer_setup(&session->tmf_timer, iscsi_tmf_timedout, 0);
mutex_init(&session->eh_mutex);
+ init_waitqueue_head(&session->ehwait);
spin_lock_init(&session->frwd_lock);
spin_lock_init(&session->back_lock);
goto login_task_data_alloc_fail;
conn->login_task->data = conn->data = data;
- init_waitqueue_head(&session->ehwait);
-
return cls_conn;
login_task_data_alloc_fail:
return error;
}
+EXPORT_SYMBOL_GPL(sas_register_ha);
static void sas_disable_events(struct sas_ha_struct *sas_ha)
{
return 0;
}
+EXPORT_SYMBOL_GPL(sas_unregister_ha);
static int sas_get_linkerrors(struct sas_phy *phy)
{
}
}
-static int sas_phy_enable(struct sas_phy *phy, int enable)
+int sas_phy_enable(struct sas_phy *phy, int enable)
{
int ret;
enum phy_func cmd;
}
return ret;
}
+EXPORT_SYMBOL_GPL(sas_phy_enable);
int sas_phy_reset(struct sas_phy *phy, int hard_reset)
{
}
return ret;
}
+EXPORT_SYMBOL_GPL(sas_phy_reset);
int sas_set_phy_speed(struct sas_phy *phy,
struct sas_phy_linkrates *rates)
module_init(sas_class_init);
module_exit(sas_class_exit);
-EXPORT_SYMBOL_GPL(sas_register_ha);
-EXPORT_SYMBOL_GPL(sas_unregister_ha);
}
sas_end_task(sc, task);
- sc->scsi_done(sc);
+ scsi_done(sc);
}
static struct sas_task *sas_create_task(struct scsi_cmnd *cmd,
else
cmd->result = DID_ERROR << 16;
out_done:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
+EXPORT_SYMBOL_GPL(sas_queuecommand);
static void sas_eh_finish_cmd(struct scsi_cmnd *cmd)
{
return FAILED;
}
+EXPORT_SYMBOL_GPL(sas_eh_device_reset_handler);
int sas_eh_target_reset_handler(struct scsi_cmnd *cmd)
{
return FAILED;
}
+EXPORT_SYMBOL_GPL(sas_eh_target_reset_handler);
/* Try to reset a device */
static int try_to_reset_cmd_device(struct scsi_cmnd *cmd)
return -EINVAL;
}
+EXPORT_SYMBOL_GPL(sas_ioctl);
struct domain_device *sas_find_dev_by_rphy(struct sas_rphy *rphy)
{
starget->hostdata = found_dev;
return 0;
}
+EXPORT_SYMBOL_GPL(sas_target_alloc);
#define SAS_DEF_QD 256
return 0;
}
+EXPORT_SYMBOL_GPL(sas_slave_configure);
int sas_change_queue_depth(struct scsi_device *sdev, int depth)
{
depth = 1;
return scsi_change_queue_depth(sdev, depth);
}
+EXPORT_SYMBOL_GPL(sas_change_queue_depth);
int sas_bios_param(struct scsi_device *scsi_dev,
struct block_device *bdev,
return 0;
}
+EXPORT_SYMBOL_GPL(sas_bios_param);
/*
* Tell an upper layer that it needs to initiate an abort for a given task.
else
blk_abort_request(scsi_cmd_to_rq(sc));
}
+EXPORT_SYMBOL_GPL(sas_task_abort);
int sas_slave_alloc(struct scsi_device *sdev)
{
return 0;
}
+EXPORT_SYMBOL_GPL(sas_slave_alloc);
void sas_target_destroy(struct scsi_target *starget)
{
starget->hostdata = NULL;
sas_put_device(found_dev);
}
+EXPORT_SYMBOL_GPL(sas_target_destroy);
#define SAS_STRING_ADDR_SIZE 16
}
EXPORT_SYMBOL_GPL(sas_request_addr);
-EXPORT_SYMBOL_GPL(sas_queuecommand);
-EXPORT_SYMBOL_GPL(sas_target_alloc);
-EXPORT_SYMBOL_GPL(sas_slave_configure);
-EXPORT_SYMBOL_GPL(sas_change_queue_depth);
-EXPORT_SYMBOL_GPL(sas_bios_param);
-EXPORT_SYMBOL_GPL(sas_task_abort);
-EXPORT_SYMBOL_GPL(sas_phy_reset);
-EXPORT_SYMBOL_GPL(sas_eh_device_reset_handler);
-EXPORT_SYMBOL_GPL(sas_eh_target_reset_handler);
-EXPORT_SYMBOL_GPL(sas_slave_alloc);
-EXPORT_SYMBOL_GPL(sas_target_destroy);
-EXPORT_SYMBOL_GPL(sas_ioctl);
* Firmware supports Forced Link Speed
* capability
*/
+#define HBA_PCI_ERR 0x80000 /* The PCI slot is offline */
#define HBA_FLOGI_ISSUED 0x100000 /* FLOGI was issued */
#define HBA_CGN_RSVD1 0x200000 /* Reserved CGN flag */
#define HBA_CGN_DAY_WRAP 0x400000 /* HBA Congestion info day wraps */
LPFC_VMID_PRIO_TAG_ALL_TARGETS,
"Enable Priority Tagging VMID support");
-struct device_attribute *lpfc_hba_attrs[] = {
- &dev_attr_nvme_info,
- &dev_attr_scsi_stat,
- &dev_attr_bg_info,
- &dev_attr_bg_guard_err,
- &dev_attr_bg_apptag_err,
- &dev_attr_bg_reftag_err,
- &dev_attr_info,
- &dev_attr_serialnum,
- &dev_attr_modeldesc,
- &dev_attr_modelname,
- &dev_attr_programtype,
- &dev_attr_portnum,
- &dev_attr_fwrev,
- &dev_attr_hdw,
- &dev_attr_option_rom_version,
- &dev_attr_link_state,
- &dev_attr_num_discovered_ports,
- &dev_attr_menlo_mgmt_mode,
- &dev_attr_lpfc_drvr_version,
- &dev_attr_lpfc_enable_fip,
- &dev_attr_lpfc_temp_sensor,
- &dev_attr_lpfc_log_verbose,
- &dev_attr_lpfc_lun_queue_depth,
- &dev_attr_lpfc_tgt_queue_depth,
- &dev_attr_lpfc_hba_queue_depth,
- &dev_attr_lpfc_peer_port_login,
- &dev_attr_lpfc_nodev_tmo,
- &dev_attr_lpfc_devloss_tmo,
- &dev_attr_lpfc_enable_fc4_type,
- &dev_attr_lpfc_fcp_class,
- &dev_attr_lpfc_use_adisc,
- &dev_attr_lpfc_first_burst_size,
- &dev_attr_lpfc_ack0,
- &dev_attr_lpfc_xri_rebalancing,
- &dev_attr_lpfc_topology,
- &dev_attr_lpfc_scan_down,
- &dev_attr_lpfc_link_speed,
- &dev_attr_lpfc_fcp_io_sched,
- &dev_attr_lpfc_ns_query,
- &dev_attr_lpfc_fcp2_no_tgt_reset,
- &dev_attr_lpfc_cr_delay,
- &dev_attr_lpfc_cr_count,
- &dev_attr_lpfc_multi_ring_support,
- &dev_attr_lpfc_multi_ring_rctl,
- &dev_attr_lpfc_multi_ring_type,
- &dev_attr_lpfc_fdmi_on,
- &dev_attr_lpfc_enable_SmartSAN,
- &dev_attr_lpfc_max_luns,
- &dev_attr_lpfc_enable_npiv,
- &dev_attr_lpfc_fcf_failover_policy,
- &dev_attr_lpfc_enable_rrq,
- &dev_attr_lpfc_fcp_wait_abts_rsp,
- &dev_attr_nport_evt_cnt,
- &dev_attr_board_mode,
- &dev_attr_max_vpi,
- &dev_attr_used_vpi,
- &dev_attr_max_rpi,
- &dev_attr_used_rpi,
- &dev_attr_max_xri,
- &dev_attr_used_xri,
- &dev_attr_npiv_info,
- &dev_attr_issue_reset,
- &dev_attr_lpfc_poll,
- &dev_attr_lpfc_poll_tmo,
- &dev_attr_lpfc_task_mgmt_tmo,
- &dev_attr_lpfc_use_msi,
- &dev_attr_lpfc_nvme_oas,
- &dev_attr_lpfc_nvme_embed_cmd,
- &dev_attr_lpfc_fcp_imax,
- &dev_attr_lpfc_force_rscn,
- &dev_attr_lpfc_cq_poll_threshold,
- &dev_attr_lpfc_cq_max_proc_limit,
- &dev_attr_lpfc_fcp_cpu_map,
- &dev_attr_lpfc_fcp_mq_threshold,
- &dev_attr_lpfc_hdw_queue,
- &dev_attr_lpfc_irq_chann,
- &dev_attr_lpfc_suppress_rsp,
- &dev_attr_lpfc_nvmet_mrq,
- &dev_attr_lpfc_nvmet_mrq_post,
- &dev_attr_lpfc_nvme_enable_fb,
- &dev_attr_lpfc_nvmet_fb_size,
- &dev_attr_lpfc_enable_bg,
- &dev_attr_lpfc_soft_wwnn,
- &dev_attr_lpfc_soft_wwpn,
- &dev_attr_lpfc_soft_wwn_enable,
- &dev_attr_lpfc_enable_hba_reset,
- &dev_attr_lpfc_enable_hba_heartbeat,
- &dev_attr_lpfc_EnableXLane,
- &dev_attr_lpfc_XLanePriority,
- &dev_attr_lpfc_xlane_lun,
- &dev_attr_lpfc_xlane_tgt,
- &dev_attr_lpfc_xlane_vpt,
- &dev_attr_lpfc_xlane_lun_state,
- &dev_attr_lpfc_xlane_lun_status,
- &dev_attr_lpfc_xlane_priority,
- &dev_attr_lpfc_sg_seg_cnt,
- &dev_attr_lpfc_max_scsicmpl_time,
- &dev_attr_lpfc_stat_data_ctrl,
- &dev_attr_lpfc_aer_support,
- &dev_attr_lpfc_aer_state_cleanup,
- &dev_attr_lpfc_sriov_nr_virtfn,
- &dev_attr_lpfc_req_fw_upgrade,
- &dev_attr_lpfc_suppress_link_up,
- &dev_attr_iocb_hw,
- &dev_attr_pls,
- &dev_attr_pt,
- &dev_attr_txq_hw,
- &dev_attr_txcmplq_hw,
- &dev_attr_lpfc_sriov_hw_max_virtfn,
- &dev_attr_protocol,
- &dev_attr_lpfc_xlane_supported,
- &dev_attr_lpfc_enable_mds_diags,
- &dev_attr_lpfc_ras_fwlog_buffsize,
- &dev_attr_lpfc_ras_fwlog_level,
- &dev_attr_lpfc_ras_fwlog_func,
- &dev_attr_lpfc_enable_bbcr,
- &dev_attr_lpfc_enable_dpp,
- &dev_attr_lpfc_enable_mi,
- &dev_attr_cmf_info,
- &dev_attr_lpfc_max_vmid,
- &dev_attr_lpfc_vmid_inactivity_timeout,
- &dev_attr_lpfc_vmid_app_header,
- &dev_attr_lpfc_vmid_priority_tagging,
+static struct attribute *lpfc_hba_attrs[] = {
+ &dev_attr_nvme_info.attr,
+ &dev_attr_scsi_stat.attr,
+ &dev_attr_bg_info.attr,
+ &dev_attr_bg_guard_err.attr,
+ &dev_attr_bg_apptag_err.attr,
+ &dev_attr_bg_reftag_err.attr,
+ &dev_attr_info.attr,
+ &dev_attr_serialnum.attr,
+ &dev_attr_modeldesc.attr,
+ &dev_attr_modelname.attr,
+ &dev_attr_programtype.attr,
+ &dev_attr_portnum.attr,
+ &dev_attr_fwrev.attr,
+ &dev_attr_hdw.attr,
+ &dev_attr_option_rom_version.attr,
+ &dev_attr_link_state.attr,
+ &dev_attr_num_discovered_ports.attr,
+ &dev_attr_menlo_mgmt_mode.attr,
+ &dev_attr_lpfc_drvr_version.attr,
+ &dev_attr_lpfc_enable_fip.attr,
+ &dev_attr_lpfc_temp_sensor.attr,
+ &dev_attr_lpfc_log_verbose.attr,
+ &dev_attr_lpfc_lun_queue_depth.attr,
+ &dev_attr_lpfc_tgt_queue_depth.attr,
+ &dev_attr_lpfc_hba_queue_depth.attr,
+ &dev_attr_lpfc_peer_port_login.attr,
+ &dev_attr_lpfc_nodev_tmo.attr,
+ &dev_attr_lpfc_devloss_tmo.attr,
+ &dev_attr_lpfc_enable_fc4_type.attr,
+ &dev_attr_lpfc_fcp_class.attr,
+ &dev_attr_lpfc_use_adisc.attr,
+ &dev_attr_lpfc_first_burst_size.attr,
+ &dev_attr_lpfc_ack0.attr,
+ &dev_attr_lpfc_xri_rebalancing.attr,
+ &dev_attr_lpfc_topology.attr,
+ &dev_attr_lpfc_scan_down.attr,
+ &dev_attr_lpfc_link_speed.attr,
+ &dev_attr_lpfc_fcp_io_sched.attr,
+ &dev_attr_lpfc_ns_query.attr,
+ &dev_attr_lpfc_fcp2_no_tgt_reset.attr,
+ &dev_attr_lpfc_cr_delay.attr,
+ &dev_attr_lpfc_cr_count.attr,
+ &dev_attr_lpfc_multi_ring_support.attr,
+ &dev_attr_lpfc_multi_ring_rctl.attr,
+ &dev_attr_lpfc_multi_ring_type.attr,
+ &dev_attr_lpfc_fdmi_on.attr,
+ &dev_attr_lpfc_enable_SmartSAN.attr,
+ &dev_attr_lpfc_max_luns.attr,
+ &dev_attr_lpfc_enable_npiv.attr,
+ &dev_attr_lpfc_fcf_failover_policy.attr,
+ &dev_attr_lpfc_enable_rrq.attr,
+ &dev_attr_lpfc_fcp_wait_abts_rsp.attr,
+ &dev_attr_nport_evt_cnt.attr,
+ &dev_attr_board_mode.attr,
+ &dev_attr_max_vpi.attr,
+ &dev_attr_used_vpi.attr,
+ &dev_attr_max_rpi.attr,
+ &dev_attr_used_rpi.attr,
+ &dev_attr_max_xri.attr,
+ &dev_attr_used_xri.attr,
+ &dev_attr_npiv_info.attr,
+ &dev_attr_issue_reset.attr,
+ &dev_attr_lpfc_poll.attr,
+ &dev_attr_lpfc_poll_tmo.attr,
+ &dev_attr_lpfc_task_mgmt_tmo.attr,
+ &dev_attr_lpfc_use_msi.attr,
+ &dev_attr_lpfc_nvme_oas.attr,
+ &dev_attr_lpfc_nvme_embed_cmd.attr,
+ &dev_attr_lpfc_fcp_imax.attr,
+ &dev_attr_lpfc_force_rscn.attr,
+ &dev_attr_lpfc_cq_poll_threshold.attr,
+ &dev_attr_lpfc_cq_max_proc_limit.attr,
+ &dev_attr_lpfc_fcp_cpu_map.attr,
+ &dev_attr_lpfc_fcp_mq_threshold.attr,
+ &dev_attr_lpfc_hdw_queue.attr,
+ &dev_attr_lpfc_irq_chann.attr,
+ &dev_attr_lpfc_suppress_rsp.attr,
+ &dev_attr_lpfc_nvmet_mrq.attr,
+ &dev_attr_lpfc_nvmet_mrq_post.attr,
+ &dev_attr_lpfc_nvme_enable_fb.attr,
+ &dev_attr_lpfc_nvmet_fb_size.attr,
+ &dev_attr_lpfc_enable_bg.attr,
+ &dev_attr_lpfc_soft_wwnn.attr,
+ &dev_attr_lpfc_soft_wwpn.attr,
+ &dev_attr_lpfc_soft_wwn_enable.attr,
+ &dev_attr_lpfc_enable_hba_reset.attr,
+ &dev_attr_lpfc_enable_hba_heartbeat.attr,
+ &dev_attr_lpfc_EnableXLane.attr,
+ &dev_attr_lpfc_XLanePriority.attr,
+ &dev_attr_lpfc_xlane_lun.attr,
+ &dev_attr_lpfc_xlane_tgt.attr,
+ &dev_attr_lpfc_xlane_vpt.attr,
+ &dev_attr_lpfc_xlane_lun_state.attr,
+ &dev_attr_lpfc_xlane_lun_status.attr,
+ &dev_attr_lpfc_xlane_priority.attr,
+ &dev_attr_lpfc_sg_seg_cnt.attr,
+ &dev_attr_lpfc_max_scsicmpl_time.attr,
+ &dev_attr_lpfc_stat_data_ctrl.attr,
+ &dev_attr_lpfc_aer_support.attr,
+ &dev_attr_lpfc_aer_state_cleanup.attr,
+ &dev_attr_lpfc_sriov_nr_virtfn.attr,
+ &dev_attr_lpfc_req_fw_upgrade.attr,
+ &dev_attr_lpfc_suppress_link_up.attr,
+ &dev_attr_iocb_hw.attr,
+ &dev_attr_pls.attr,
+ &dev_attr_pt.attr,
+ &dev_attr_txq_hw.attr,
+ &dev_attr_txcmplq_hw.attr,
+ &dev_attr_lpfc_sriov_hw_max_virtfn.attr,
+ &dev_attr_protocol.attr,
+ &dev_attr_lpfc_xlane_supported.attr,
+ &dev_attr_lpfc_enable_mds_diags.attr,
+ &dev_attr_lpfc_ras_fwlog_buffsize.attr,
+ &dev_attr_lpfc_ras_fwlog_level.attr,
+ &dev_attr_lpfc_ras_fwlog_func.attr,
+ &dev_attr_lpfc_enable_bbcr.attr,
+ &dev_attr_lpfc_enable_dpp.attr,
+ &dev_attr_lpfc_enable_mi.attr,
+ &dev_attr_cmf_info.attr,
+ &dev_attr_lpfc_max_vmid.attr,
+ &dev_attr_lpfc_vmid_inactivity_timeout.attr,
+ &dev_attr_lpfc_vmid_app_header.attr,
+ &dev_attr_lpfc_vmid_priority_tagging.attr,
NULL,
};
-struct device_attribute *lpfc_vport_attrs[] = {
- &dev_attr_info,
- &dev_attr_link_state,
- &dev_attr_num_discovered_ports,
- &dev_attr_lpfc_drvr_version,
- &dev_attr_lpfc_log_verbose,
- &dev_attr_lpfc_lun_queue_depth,
- &dev_attr_lpfc_tgt_queue_depth,
- &dev_attr_lpfc_nodev_tmo,
- &dev_attr_lpfc_devloss_tmo,
- &dev_attr_lpfc_hba_queue_depth,
- &dev_attr_lpfc_peer_port_login,
- &dev_attr_lpfc_restrict_login,
- &dev_attr_lpfc_fcp_class,
- &dev_attr_lpfc_use_adisc,
- &dev_attr_lpfc_first_burst_size,
- &dev_attr_lpfc_max_luns,
- &dev_attr_nport_evt_cnt,
- &dev_attr_npiv_info,
- &dev_attr_lpfc_enable_da_id,
- &dev_attr_lpfc_max_scsicmpl_time,
- &dev_attr_lpfc_stat_data_ctrl,
- &dev_attr_lpfc_static_vport,
- &dev_attr_cmf_info,
+static const struct attribute_group lpfc_hba_attr_group = {
+ .attrs = lpfc_hba_attrs
+};
+
+const struct attribute_group *lpfc_hba_groups[] = {
+ &lpfc_hba_attr_group,
+ NULL
+};
+
+static struct attribute *lpfc_vport_attrs[] = {
+ &dev_attr_info.attr,
+ &dev_attr_link_state.attr,
+ &dev_attr_num_discovered_ports.attr,
+ &dev_attr_lpfc_drvr_version.attr,
+ &dev_attr_lpfc_log_verbose.attr,
+ &dev_attr_lpfc_lun_queue_depth.attr,
+ &dev_attr_lpfc_tgt_queue_depth.attr,
+ &dev_attr_lpfc_nodev_tmo.attr,
+ &dev_attr_lpfc_devloss_tmo.attr,
+ &dev_attr_lpfc_hba_queue_depth.attr,
+ &dev_attr_lpfc_peer_port_login.attr,
+ &dev_attr_lpfc_restrict_login.attr,
+ &dev_attr_lpfc_fcp_class.attr,
+ &dev_attr_lpfc_use_adisc.attr,
+ &dev_attr_lpfc_first_burst_size.attr,
+ &dev_attr_lpfc_max_luns.attr,
+ &dev_attr_nport_evt_cnt.attr,
+ &dev_attr_npiv_info.attr,
+ &dev_attr_lpfc_enable_da_id.attr,
+ &dev_attr_lpfc_max_scsicmpl_time.attr,
+ &dev_attr_lpfc_stat_data_ctrl.attr,
+ &dev_attr_lpfc_static_vport.attr,
+ &dev_attr_cmf_info.attr,
NULL,
};
+static const struct attribute_group lpfc_vport_attr_group = {
+ .attrs = lpfc_vport_attrs
+};
+
+const struct attribute_group *lpfc_vport_groups[] = {
+ &lpfc_vport_attr_group,
+ NULL
+};
+
/**
* sysfs_ctlreg_write - Write method for writing to ctlreg
* @filp: open sysfs file
struct lpfc_nodelist *lpfc_nlp_init(struct lpfc_vport *vport, uint32_t did);
struct lpfc_nodelist *lpfc_nlp_get(struct lpfc_nodelist *);
int lpfc_nlp_put(struct lpfc_nodelist *);
+void lpfc_check_nlp_post_devloss(struct lpfc_vport *vport,
+ struct lpfc_nodelist *ndlp);
void lpfc_ignore_els_cmpl(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb,
struct lpfc_iocbq *rspiocb);
int lpfc_nlp_not_used(struct lpfc_nodelist *ndlp);
int lpfc_config_port_prep(struct lpfc_hba *);
void lpfc_update_vport_wwn(struct lpfc_vport *vport);
int lpfc_config_port_post(struct lpfc_hba *);
+int lpfc_sli4_refresh_params(struct lpfc_hba *phba);
int lpfc_hba_down_prep(struct lpfc_hba *);
int lpfc_hba_down_post(struct lpfc_hba *);
void lpfc_hba_init(struct lpfc_hba *, uint32_t *);
void lpfc_get_vport_cfgparam(struct lpfc_vport *);
int lpfc_alloc_sysfs_attr(struct lpfc_vport *);
void lpfc_free_sysfs_attr(struct lpfc_vport *);
-extern struct device_attribute *lpfc_hba_attrs[];
-extern struct device_attribute *lpfc_vport_attrs[];
+extern const struct attribute_group *lpfc_hba_groups[];
+extern const struct attribute_group *lpfc_vport_groups[];
extern struct scsi_host_template lpfc_template;
extern struct scsi_host_template lpfc_template_nvme;
extern struct fc_function_template lpfc_transport_functions;
NLP_XPT_HAS_HH = 0x10
};
+enum lpfc_nlp_save_flags {
+ /* devloss occurred during recovery */
+ NLP_IN_RECOV_POST_DEV_LOSS = 0x1,
+ /* wait for outstanding LOGO to cmpl */
+ NLP_WAIT_FOR_LOGO = 0x2,
+};
+
struct lpfc_nodelist {
struct list_head nlp_listp;
struct serv_parm fc_sparam; /* buffer for service params */
unsigned long *active_rrqs_xri_bitmap;
struct lpfc_scsicmd_bkt *lat_data; /* Latency data */
uint32_t fc4_prli_sent;
- u32 upcall_flags;
-#define NLP_WAIT_FOR_LOGO 0x2
+
+ /* flags to keep ndlp alive until special conditions are met */
+ enum lpfc_nlp_save_flags save_flags;
enum lpfc_fc4_xpt_flags fc4_xpt_flags;
lpfc_printf_vlog(vport, KERN_WARNING, LOG_TRACE_EVENT,
"0150 FLOGI failure Status:x%x/x%x "
- "xri x%x TMO:x%x\n",
+ "xri x%x TMO:x%x refcnt %d\n",
irsp->ulpStatus, irsp->un.ulpWord[4],
- cmdiocb->sli4_xritag, irsp->ulpTimeout);
+ cmdiocb->sli4_xritag, irsp->ulpTimeout,
+ kref_read(&ndlp->kref));
/* If this is not a loop open failure, bail out */
if (!(irsp->ulpStatus == IOSTAT_LOCAL_REJECT &&
/* FLOGI completes successfully */
lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS,
"0101 FLOGI completes successfully, I/O tag:x%x, "
- "xri x%x Data: x%x x%x x%x x%x x%x x%x x%x\n",
+ "xri x%x Data: x%x x%x x%x x%x x%x x%x x%x %d\n",
cmdiocb->iotag, cmdiocb->sli4_xritag,
irsp->un.ulpWord[4], sp->cmn.e_d_tov,
sp->cmn.w2.r_a_tov, sp->cmn.edtovResolution,
vport->port_state, vport->fc_flag,
- sp->cmn.priority_tagging);
+ sp->cmn.priority_tagging, kref_read(&ndlp->kref));
if (sp->cmn.priority_tagging)
vport->vmid_flag |= LPFC_VMID_ISSUE_QFPA;
phba->fcf.fcf_flag &= ~FCF_DISCOVERY;
spin_unlock_irq(&phba->hbalock);
- if (!(ndlp->fc4_xpt_flags & (SCSI_XPT_REGD | NVME_XPT_REGD)))
- lpfc_nlp_put(ndlp);
if (!lpfc_error_lost_link(irsp)) {
/* FLOGI failed, so just use loop map to make discovery list */
lpfc_disc_list_loopmap(vport);
lpfc_disc_state_machine(vport, ndlp, cmdiocb,
NLP_EVT_CMPL_PRLI);
+ /*
+ * For P2P topology, retain the node so that PLOGI can be
+ * attempted on it again.
+ */
+ if (vport->fc_flag & FC_PT2PT)
+ goto out;
+
/* As long as this node is not registered with the SCSI
* or NVMe transport and no other PRLIs are outstanding,
* it is no longer an active node. Otherwise devloss
irsp = &(rspiocb->iocb);
spin_lock_irq(&ndlp->lock);
ndlp->nlp_flag &= ~NLP_LOGO_SND;
- if (ndlp->upcall_flags & NLP_WAIT_FOR_LOGO) {
+ if (ndlp->save_flags & NLP_WAIT_FOR_LOGO) {
wake_up_waiter = 1;
- ndlp->upcall_flags &= ~NLP_WAIT_FOR_LOGO;
+ ndlp->save_flags &= ~NLP_WAIT_FOR_LOGO;
}
spin_unlock_irq(&ndlp->lock);
retry = 1;
delay = 100;
break;
+ case IOERR_SLI_ABORTED:
+ /* Retry ELS PLOGI command?
+ * Possibly the rport just wasn't ready.
+ */
+ if (cmd == ELS_CMD_PLOGI) {
+ /* No retry if state change */
+ if (ndlp &&
+ ndlp->nlp_state != NLP_STE_PLOGI_ISSUE)
+ goto out_retry;
+ retry = 1;
+ maxretry = 2;
+ }
+ break;
}
break;
*/
if (phba->sli_rev == LPFC_SLI_REV4 &&
(vport && vport->port_type == LPFC_NPIV_PORT) &&
+ !(ndlp->fc4_xpt_flags & SCSI_XPT_REGD) &&
ndlp->nlp_flag & NLP_RELEASE_RPI) {
lpfc_sli4_free_rpi(phba, ndlp->nlp_rpi);
spin_lock_irq(&ndlp->lock);
}
/* The NPIV instance is rejecting this unsolicited ELS. Make sure the
- * node's assigned RPI needs to be released as this node will get
- * freed.
+ * node's assigned RPI gets released provided this node is not already
+ * registered with the transport.
*/
if (phba->sli_rev == LPFC_SLI_REV4 &&
- vport->port_type == LPFC_NPIV_PORT) {
+ vport->port_type == LPFC_NPIV_PORT &&
+ !(ndlp->fc4_xpt_flags & SCSI_XPT_REGD)) {
spin_lock_irq(&ndlp->lock);
ndlp->nlp_flag |= NLP_RELEASE_RPI;
spin_unlock_irq(&ndlp->lock);
* from backend
*/
lpfc_nlp_unreg_node(vport, ndlp);
+ lpfc_unreg_rpi(vport, ndlp);
continue;
}
irsp->ulpStatus, irsp->un.ulpWord[4]);
goto fdisc_failed;
}
+
+ lpfc_check_nlp_post_devloss(vport, ndlp);
+
spin_lock_irq(shost->host_lock);
vport->fc_flag &= ~FC_VPORT_CVL_RCVD;
vport->fc_flag &= ~FC_VPORT_LOGO_RCVD;
{
struct lpfc_hba *phba = vport->phba;
struct lpfc_sglq *sglq_entry = NULL, *sglq_next = NULL;
+ struct lpfc_nodelist *ndlp = NULL;
unsigned long iflag = 0;
spin_lock_irqsave(&phba->sli4_hba.sgl_list_lock, iflag);
&phba->sli4_hba.lpfc_abts_els_sgl_list, list) {
if (sglq_entry->ndlp && sglq_entry->ndlp->vport == vport) {
lpfc_nlp_put(sglq_entry->ndlp);
+ ndlp = sglq_entry->ndlp;
sglq_entry->ndlp = NULL;
+
+ /* If the xri on the abts_els_sgl list is for the Fport
+ * node and the vport is unloading, the xri aborted wcqe
+ * likely isn't coming back. Just release the sgl.
+ */
+ if ((vport->load_flag & FC_UNLOADING) &&
+ ndlp->nlp_DID == Fabric_DID) {
+ list_del(&sglq_entry->list);
+ sglq_entry->state = SGL_FREED;
+ list_add_tail(&sglq_entry->list,
+ &phba->sli4_hba.lpfc_els_sgl_list);
+ }
}
}
spin_unlock_irqrestore(&phba->sli4_hba.sgl_list_lock, iflag);
spin_lock_irqsave(&ndlp->lock, iflags);
ndlp->nlp_flag |= NLP_IN_DEV_LOSS;
- ndlp->nlp_flag &= ~NLP_NPR_2B_DISC;
+
+ /* If there is a PLOGI in progress, and we are in a
+ * NLP_NPR_2B_DISC state, don't turn off the flag.
+ */
+ if (ndlp->nlp_state != NLP_STE_PLOGI_ISSUE)
+ ndlp->nlp_flag &= ~NLP_NPR_2B_DISC;
/*
* The backend does not expect any more calls associated with this
}
/**
+ * lpfc_check_nlp_post_devloss - Check to restore ndlp refcnt after devloss
+ * @vport: Pointer to vport object.
+ * @ndlp: Pointer to remote node object.
+ *
+ * If NLP_IN_RECOV_POST_DEV_LOSS flag was set due to outstanding recovery of
+ * node during dev_loss_tmo processing, then this function restores the nlp_put
+ * kref decrement from lpfc_dev_loss_tmo_handler.
+ **/
+void
+lpfc_check_nlp_post_devloss(struct lpfc_vport *vport,
+ struct lpfc_nodelist *ndlp)
+{
+ unsigned long iflags;
+
+ spin_lock_irqsave(&ndlp->lock, iflags);
+ if (ndlp->save_flags & NLP_IN_RECOV_POST_DEV_LOSS) {
+ ndlp->save_flags &= ~NLP_IN_RECOV_POST_DEV_LOSS;
+ spin_unlock_irqrestore(&ndlp->lock, iflags);
+ lpfc_nlp_get(ndlp);
+ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY | LOG_NODE,
+ "8438 Devloss timeout reversed on DID x%x "
+ "refcnt %d ndlp %p flag x%x "
+ "port_state = x%x\n",
+ ndlp->nlp_DID, kref_read(&ndlp->kref), ndlp,
+ ndlp->nlp_flag, vport->port_state);
+ spin_lock_irqsave(&ndlp->lock, iflags);
+ }
+ spin_unlock_irqrestore(&ndlp->lock, iflags);
+}
+
+/**
* lpfc_dev_loss_tmo_handler - Remote node devloss timeout handler
* @ndlp: Pointer to remote node object.
*
uint8_t *name;
int warn_on = 0;
int fcf_inuse = 0;
+ bool recovering = false;
+ struct fc_vport *fc_vport = NULL;
unsigned long iflags;
vport = ndlp->vport;
/* Fabric nodes are done. */
if (ndlp->nlp_type & NLP_FABRIC) {
+ spin_lock_irqsave(&ndlp->lock, iflags);
+ /* In massive vport configuration settings, it's possible
+ * dev_loss_tmo fired during node recovery. So, check if
+ * fabric nodes are in discovery states outstanding.
+ */
+ switch (ndlp->nlp_DID) {
+ case Fabric_DID:
+ fc_vport = vport->fc_vport;
+ if (fc_vport &&
+ fc_vport->vport_state == FC_VPORT_INITIALIZING)
+ recovering = true;
+ break;
+ case Fabric_Cntl_DID:
+ if (ndlp->nlp_flag & NLP_REG_LOGIN_SEND)
+ recovering = true;
+ break;
+ case FDMI_DID:
+ fallthrough;
+ case NameServer_DID:
+ if (ndlp->nlp_state >= NLP_STE_PLOGI_ISSUE &&
+ ndlp->nlp_state <= NLP_STE_REG_LOGIN_ISSUE)
+ recovering = true;
+ break;
+ }
+ spin_unlock_irqrestore(&ndlp->lock, iflags);
+
+ /* Mark an NLP_IN_RECOV_POST_DEV_LOSS flag to know if reversing
+ * the following lpfc_nlp_put is necessary after fabric node is
+ * recovered.
+ */
+ if (recovering) {
+ lpfc_printf_vlog(vport, KERN_INFO,
+ LOG_DISCOVERY | LOG_NODE,
+ "8436 Devloss timeout marked on "
+ "DID x%x refcnt %d ndlp %p "
+ "flag x%x port_state = x%x\n",
+ ndlp->nlp_DID, kref_read(&ndlp->kref),
+ ndlp, ndlp->nlp_flag,
+ vport->port_state);
+ spin_lock_irqsave(&ndlp->lock, iflags);
+ ndlp->save_flags |= NLP_IN_RECOV_POST_DEV_LOSS;
+ spin_unlock_irqrestore(&ndlp->lock, iflags);
+ } else if (ndlp->nlp_state == NLP_STE_UNMAPPED_NODE) {
+ /* Fabric node fully recovered before this dev_loss_tmo
+ * queue work is processed. Thus, ignore the
+ * dev_loss_tmo event.
+ */
+ lpfc_printf_vlog(vport, KERN_INFO,
+ LOG_DISCOVERY | LOG_NODE,
+ "8437 Devloss timeout ignored on "
+ "DID x%x refcnt %d ndlp %p "
+ "flag x%x port_state = x%x\n",
+ ndlp->nlp_DID, kref_read(&ndlp->kref),
+ ndlp, ndlp->nlp_flag,
+ vport->port_state);
+ return fcf_inuse;
+ }
+
lpfc_nlp_put(ndlp);
return fcf_inuse;
}
ndlp->nlp_state, ndlp->nlp_rpi);
}
+ /* If we are devloss, but we are in the process of rediscovering the
+ * ndlp, don't issue a NLP_EVT_DEVICE_RM event.
+ */
+ if (ndlp->nlp_state >= NLP_STE_PLOGI_ISSUE &&
+ ndlp->nlp_state <= NLP_STE_PRLI_ISSUE) {
+ return fcf_inuse;
+ }
+
if (!(ndlp->fc4_xpt_flags & NVME_XPT_REGD))
lpfc_disc_state_machine(vport, ndlp, NULL, NLP_EVT_DEVICE_RM);
struct lpfc_nodelist *ndlp, *next_ndlp;
list_for_each_entry_safe(ndlp, next_ndlp, &vport->fc_nodes, nlp_listp) {
- if (ndlp->nlp_state == NLP_STE_UNUSED_NODE)
+ if (ndlp->nlp_state == NLP_STE_UNUSED_NODE) {
+ /* It's possible the FLOGI to the fabric node never
+ * successfully completed and never registered with the
+ * transport. In this case there is no way to clean up
+ * the node.
+ */
+ if (ndlp->nlp_DID == Fabric_DID) {
+ if (ndlp->nlp_prev_state ==
+ NLP_STE_UNUSED_NODE &&
+ !ndlp->fc4_xpt_flags)
+ lpfc_nlp_put(ndlp);
+ }
continue;
+ }
if ((phba->sli3_options & LPFC_SLI3_VPORT_TEARDOWN) ||
((vport->port_type == LPFC_NPIV_PORT) &&
goto out;
}
+ lpfc_check_nlp_post_devloss(vport, ndlp);
+
if (phba->sli_rev < LPFC_SLI_REV4)
ndlp->nlp_rpi = mb->un.varWords[0];
ndlp->nlp_state);
ndlp->nlp_flag |= NLP_RPI_REGISTERED;
+ ndlp->nlp_flag &= ~NLP_REG_LOGIN_SEND;
ndlp->nlp_type |= NLP_FABRIC;
lpfc_nlp_set_state(vport, ndlp, NLP_STE_UNMAPPED_NODE);
fc_remote_port_rolechg(rport, rport_ids.roles);
lpfc_printf_vlog(ndlp->vport, KERN_INFO, LOG_NODE,
- "3183 %s rport x%px DID x%x, role x%x\n",
- __func__, rport, rport->port_id, rport->roles);
+ "3183 %s rport x%px DID x%x, role x%x refcnt %d\n",
+ __func__, rport, rport->port_id, rport->roles,
+ kref_read(&ndlp->kref));
if ((rport->scsi_target_id != -1) &&
(rport->scsi_target_id < LPFC_MAX_TARGET)) {
lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE,
"3184 rport unregister x%06x, rport x%px "
- "xptflg x%x\n",
- ndlp->nlp_DID, rport, ndlp->fc4_xpt_flags);
+ "xptflg x%x refcnt %d\n",
+ ndlp->nlp_DID, rport, ndlp->fc4_xpt_flags,
+ kref_read(&ndlp->kref));
fc_remote_port_delete(rport);
lpfc_nlp_put(ndlp);
void
lpfc_nlp_reg_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp)
{
-
unsigned long iflags;
+ lpfc_check_nlp_post_devloss(vport, ndlp);
+
spin_lock_irqsave(&ndlp->lock, iflags);
if (ndlp->fc4_xpt_flags & NLP_XPT_REGD) {
/* Already registered with backend, trigger rescan */
/* Reg/Unreg for FCP and NVME Transport interface */
if ((old_state == NLP_STE_MAPPED_NODE ||
old_state == NLP_STE_UNMAPPED_NODE)) {
- /* For nodes marked for ADISC, Handle unreg in ADISC cmpl */
- if (!(ndlp->nlp_flag & NLP_NPR_ADISC))
+ /* For nodes marked for ADISC, Handle unreg in ADISC cmpl
+ * if linkup. In linkdown do unreg_node
+ */
+ if (!(ndlp->nlp_flag & NLP_NPR_ADISC) ||
+ !lpfc_is_link_up(vport->phba))
lpfc_nlp_unreg_node(vport, ndlp);
}
rc = lpfc_sli_issue_mbox(phba, mbox, MBX_NOWAIT);
if (rc == MBX_NOT_FINISHED) {
+ ndlp->nlp_flag &= ~NLP_UNREG_INP;
mempool_free(mbox, phba->mbox_mem_pool);
acc_plogi = 1;
}
#define lpfc_sliport_status_rdy_SHIFT 23
#define lpfc_sliport_status_rdy_MASK 0x1
#define lpfc_sliport_status_rdy_WORD word0
+#define lpfc_sliport_status_pldv_SHIFT 0
+#define lpfc_sliport_status_pldv_MASK 0x1
+#define lpfc_sliport_status_pldv_WORD word0
+#define CFG_PLD 0x3C
#define MAX_IF_TYPE_2_RESETS 6
#define LPFC_CTL_PORT_CTL_OFFSET 0x408
static enum cpuhp_state lpfc_cpuhp_state;
/* Used when mapping IRQ vectors in a driver centric manner */
static uint32_t lpfc_present_cpu;
+static bool lpfc_pldv_detect;
static void __lpfc_cpuhp_remove(struct lpfc_hba *phba);
static void lpfc_cpuhp_remove(struct lpfc_hba *phba);
}
/**
+ * lpfc_sli4_refresh_params - update driver copy of params.
+ * @phba: Pointer to HBA context object.
+ *
+ * This is called to refresh driver copy of dynamic fields from the
+ * common_get_sli4_parameters descriptor.
+ **/
+int
+lpfc_sli4_refresh_params(struct lpfc_hba *phba)
+{
+ LPFC_MBOXQ_t *mboxq;
+ struct lpfc_mqe *mqe;
+ struct lpfc_sli4_parameters *mbx_sli4_parameters;
+ int length, rc;
+
+ mboxq = (LPFC_MBOXQ_t *)mempool_alloc(phba->mbox_mem_pool, GFP_KERNEL);
+ if (!mboxq)
+ return -ENOMEM;
+
+ mqe = &mboxq->u.mqe;
+ /* Read the port's SLI4 Config Parameters */
+ length = (sizeof(struct lpfc_mbx_get_sli4_parameters) -
+ sizeof(struct lpfc_sli4_cfg_mhdr));
+ lpfc_sli4_config(phba, mboxq, LPFC_MBOX_SUBSYSTEM_COMMON,
+ LPFC_MBOX_OPCODE_GET_SLI4_PARAMETERS,
+ length, LPFC_SLI4_MBX_EMBED);
+
+ rc = lpfc_sli_issue_mbox(phba, mboxq, MBX_POLL);
+ if (unlikely(rc)) {
+ mempool_free(mboxq, phba->mbox_mem_pool);
+ return rc;
+ }
+ mbx_sli4_parameters = &mqe->un.get_sli4_parameters.sli4_parameters;
+ phba->sli4_hba.pc_sli4_params.mi_ver =
+ bf_get(cfg_mi_ver, mbx_sli4_parameters);
+ phba->sli4_hba.pc_sli4_params.cmf =
+ bf_get(cfg_cmf, mbx_sli4_parameters);
+ phba->sli4_hba.pc_sli4_params.pls =
+ bf_get(cfg_pvl, mbx_sli4_parameters);
+
+ mempool_free(mboxq, phba->mbox_mem_pool);
+ return rc;
+}
+
+/**
* lpfc_hba_init_link - Initialize the FC link
* @phba: pointer to lpfc hba data structure.
* @flag: mailbox command issue mode - either MBX_POLL or MBX_NOWAIT
lpfc_sli4_offline_eratt(struct lpfc_hba *phba)
{
spin_lock_irq(&phba->hbalock);
+ if (phba->link_state == LPFC_HBA_ERROR &&
+ phba->hba_flag & HBA_PCI_ERR) {
+ spin_unlock_irq(&phba->hbalock);
+ return;
+ }
phba->link_state = LPFC_HBA_ERROR;
spin_unlock_irq(&phba->hbalock);
if (pci_channel_offline(phba->pcidev)) {
lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
"3166 pci channel is offline\n");
- lpfc_sli4_offline_eratt(phba);
return;
}
struct lpfc_vport **vports;
struct Scsi_Host *shost;
int i;
+ int offline = 0;
if (vport->fc_flag & FC_OFFLINE_MODE)
return;
lpfc_linkdown(phba);
+ offline = pci_channel_offline(phba->pcidev);
+
/* Issue an unreg_login to all nodes on all vports */
vports = lpfc_create_vport_work_array(phba);
if (vports != NULL) {
ndlp->nlp_flag &= ~NLP_NPR_ADISC;
spin_unlock_irq(&ndlp->lock);
- lpfc_unreg_rpi(vports[i], ndlp);
+ if (offline) {
+ spin_lock_irq(&ndlp->lock);
+ ndlp->nlp_flag &= ~(NLP_UNREG_INP |
+ NLP_RPI_REGISTERED);
+ spin_unlock_irq(&ndlp->lock);
+ } else {
+ lpfc_unreg_rpi(vports[i], ndlp);
+ }
/*
* Whenever an SLI4 port goes offline, free the
* RPI. Get a new RPI when the adapter port
lpfc_disc_state_machine(vports[i], ndlp,
NULL, NLP_EVT_DEVICE_RECOVERY);
- /* Don't remove the node unless the
+ /* Don't remove the node unless the node
* has been unregistered with the
- * transport. If so, let dev_loss
- * take care of the node.
+ * transport, and we're not in recovery
+ * before dev_loss_tmo triggered.
+ * Otherwise, let dev_loss take care of
+ * the node.
*/
- if (!(ndlp->fc4_xpt_flags &
+ if (!(ndlp->save_flags &
+ NLP_IN_RECOV_POST_DEV_LOSS) &&
+ !(ndlp->fc4_xpt_flags &
(NVME_XPT_REGD | SCSI_XPT_REGD)))
lpfc_disc_state_machine
(vports[i], ndlp,
/* Template for all vports this physical port creates */
memcpy(&phba->vport_template, &lpfc_template,
sizeof(*template));
- phba->vport_template.shost_attrs = lpfc_vport_attrs;
+ phba->vport_template.shost_groups = lpfc_vport_groups;
phba->vport_template.eh_bus_reset_handler = NULL;
phba->vport_template.eh_host_reset_handler = NULL;
phba->vport_template.vendor_id = 0;
uint32_t io_cnt;
uint32_t head, tail;
uint32_t busy, max_read;
- uint64_t total, rcv, lat, mbpi;
+ uint64_t total, rcv, lat, mbpi, extra;
int timer_interval = LPFC_CMF_INTERVAL;
uint32_t ms;
struct lpfc_cgn_stat *cgs;
phba->hba_flag & HBA_SETUP) {
mbpi = phba->cmf_last_sync_bw;
phba->cmf_last_sync_bw = 0;
- lpfc_issue_cmf_sync_wqe(phba, LPFC_CMF_INTERVAL, total);
+ extra = 0;
+
+ /* Calculate any extra bytes needed to account for the
+ * timer accuracy. If we are less than LPFC_CMF_INTERVAL
+ * add an extra 3% slop factor, equal to LPFC_CMF_INTERVAL
+ * add an extra 2%. The goal is to equalize total with a
+ * time > LPFC_CMF_INTERVAL or <= LPFC_CMF_INTERVAL + 1
+ */
+ if (ms == LPFC_CMF_INTERVAL)
+ extra = div_u64(total, 50);
+ else if (ms < LPFC_CMF_INTERVAL)
+ extra = div_u64(total, 33);
+ lpfc_issue_cmf_sync_wqe(phba, LPFC_CMF_INTERVAL, total + extra);
} else {
/* For Monitor mode or link down we want mbpi
* to be the full link speed
"3194 Unable to retrieve supported "
"speeds, rc = 0x%x\n", rc);
}
+ rc = lpfc_sli4_refresh_params(phba);
+ if (rc) {
+ lpfc_printf_log(phba, KERN_ERR, LOG_SLI,
+ "3174 Unable to update pls support, "
+ "rc x%x\n", rc);
+ }
vports = lpfc_create_vport_work_array(phba);
if (vports != NULL) {
for (i = 0; i <= phba->max_vports && vports[i] != NULL;
/* Cannot find existing Fabric ndlp, so allocate a new one */
ndlp = lpfc_nlp_init(vport, Fabric_DID);
if (!ndlp)
- return 0;
+ return NULL;
/* Set the node type */
ndlp->nlp_type |= NLP_FABRIC;
/* Put ndlp onto node list */
out_disable_device:
pci_disable_device(pdev);
out_error:
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1401 Failed to enable pci device\n");
return -ENODEV;
}
phba->lpfc_stop_port = lpfc_stop_port_s4;
break;
default:
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1431 Invalid HBA PCI-device group: 0x%x\n",
dev_grp);
return -ENODEV;
phba->work_status[0],
phba->work_status[1]);
port_error = -ENODEV;
+ break;
}
+
+ if (lpfc_pldv_detect &&
+ bf_get(lpfc_sli_intf_sli_family,
+ &phba->sli4_hba.sli_intf) ==
+ LPFC_SLI_INTF_FAMILY_G6)
+ pci_write_config_byte(phba->pcidev,
+ LPFC_SLI_INTF, CFG_PLD);
break;
case LPFC_SLI_INTF_IF_TYPE_1:
default:
goto out;
}
+ if (bf_get(lpfc_sliport_status_pldv, ®_data))
+ lpfc_pldv_detect = true;
+
if (!port_reset) {
/*
* Reset the port now
/* There is no SLI3 failback for SLI4 devices. */
if (bf_get(lpfc_sli_intf_valid, &phba->sli4_hba.sli_intf) !=
LPFC_SLI_INTF_VALID) {
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"2894 SLI_INTF reg contents invalid "
"sli_intf reg 0x%x\n",
phba->sli4_hba.sli_intf.word0);
atomic_set(&phba->cgn_sync_alarm_cnt, 0);
atomic_set(&phba->cgn_sync_warn_cnt, 0);
- atomic64_set(&phba->cgn_acqe_stat.alarm, 0);
- atomic64_set(&phba->cgn_acqe_stat.warn, 0);
atomic_set(&phba->cgn_driver_evt_cnt, 0);
atomic_set(&phba->cgn_latency_evt_cnt, 0);
atomic64_set(&phba->cgn_latency_evt, 0);
return error;
}
+ /* Init cpu_map array */
+ lpfc_cpu_map_array_init(phba);
+ /* Init hba_eq_hdl array */
+ lpfc_hba_eq_hdl_array_init(phba);
/* Configure and enable interrupt */
intr_mode = lpfc_sli_enable_intr(phba, phba->intr_mode);
if (intr_mode == LPFC_INTR_ERROR) {
lpfc_sli4_prep_dev_for_recover(phba);
return PCI_ERS_RESULT_CAN_RECOVER;
case pci_channel_io_frozen:
+ phba->hba_flag |= HBA_PCI_ERR;
/* Fatal error, prepare for slot reset */
lpfc_sli4_prep_dev_for_reset(phba);
return PCI_ERS_RESULT_NEED_RESET;
case pci_channel_io_perm_failure:
+ phba->hba_flag |= HBA_PCI_ERR;
/* Permanent failure, prepare for device down */
lpfc_sli4_prep_dev_for_perm_failure(phba);
return PCI_ERS_RESULT_DISCONNECT;
default:
+ phba->hba_flag |= HBA_PCI_ERR;
/* Unknown state, prepare and request slot reset */
lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
"2825 Unknown PCI error state: x%x\n", state);
pci_restore_state(pdev);
+ phba->hba_flag &= ~HBA_PCI_ERR;
/*
* As the new kernel behavior of pci_restore_state() API call clears
* device saved_state flag, need to save the restored state again.
return PCI_ERS_RESULT_DISCONNECT;
} else
phba->intr_mode = intr_mode;
+ lpfc_cpu_affinity_check(phba, phba->cfg_irq_chann);
/* Log the current active interrupt mode */
lpfc_log_intr_mode(phba, phba->intr_mode);
struct lpfc_hba *phba = ((struct lpfc_vport *)shost->hostdata)->phba;
pci_ers_result_t rc = PCI_ERS_RESULT_DISCONNECT;
+ if (phba->link_state == LPFC_HBA_ERROR &&
+ phba->hba_flag & HBA_IOQ_FLUSH)
+ return PCI_ERS_RESULT_NEED_RESET;
+
switch (phba->pci_dev_grp) {
case LPFC_PCI_DEV_LP:
rc = lpfc_io_error_detected_s3(pdev, state);
/* Initialize in case vector mapping is needed */
lpfc_present_cpu = num_present_cpus();
+ lpfc_pldv_detect = false;
+
error = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
"lpfc/sli4:online",
lpfc_cpu_online, lpfc_cpu_offline);
* calling state machine to remove the node.
*/
lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME_DISC,
- "6146 remoteport delete of remoteport x%px\n",
- remoteport);
+ "6146 remoteport delete of remoteport x%px, ndlp x%px "
+ "DID x%x xflags x%x\n",
+ remoteport, ndlp, ndlp->nlp_DID, ndlp->fc4_xpt_flags);
spin_lock_irq(&ndlp->lock);
/* The register rebind might have occurred before the delete
#ifdef CONFIG_SCSI_LPFC_DEBUG_FS
int cpu;
#endif
+ int offline = 0;
/* Sanity check on return of outstanding command */
if (!lpfc_ncmd) {
nCmd->transferred_length = 0;
nCmd->rcv_rsplen = 0;
nCmd->status = NVME_SC_INTERNAL;
+ offline = pci_channel_offline(vport->phba->pcidev);
}
}
/* pick up SLI4 exhange busy condition */
- if (bf_get(lpfc_wcqe_c_xb, wcqe))
+ if (bf_get(lpfc_wcqe_c_xb, wcqe) && !offline)
lpfc_ncmd->flags |= LPFC_SBUF_XBUSY;
else
lpfc_ncmd->flags &= ~LPFC_SBUF_XBUSY;
struct sli4_sge *first_data_sgl;
struct ulp_bde64 *bde;
dma_addr_t physaddr = 0;
- uint32_t num_bde = 0;
uint32_t dma_len = 0;
uint32_t dma_offset = 0;
int nseg, i, j;
}
sgl->word2 = 0;
- if ((num_bde + 1) == nseg) {
+ if (nseg == 1) {
bf_set(lpfc_sli4_sge_last, sgl, 1);
bf_set(lpfc_sli4_sge_type, sgl,
LPFC_SGE_TYPE_DATA);
j++;
}
- if (phba->cfg_enable_pbde) {
- /* Use PBDE support for first SGL only, offset == 0 */
+
+ /* PBDE support for first data SGE only */
+ if (nseg == 1 && phba->cfg_enable_pbde) {
/* Words 13-15 */
bde = (struct ulp_bde64 *)
&wqe->words[13];
bde->tus.f.bdeFlags = BUFF_TYPE_BDE_64;
bde->tus.w = cpu_to_le32(bde->tus.w);
- /* Word 11 */
+ /* Word 11 - set PBDE bit */
bf_set(wqe_pbde, &wqe->generic.wqe_com, 1);
} else {
memset(&wqe->words[13], 0, (sizeof(uint32_t) * 3));
- bf_set(wqe_pbde, &wqe->generic.wqe_com, 0);
+ /* Word 11 - PBDE bit disabled by default template */
}
} else {
abts_nvme = 0;
for (i = 0; i < phba->cfg_hdw_queue; i++) {
qp = &phba->sli4_hba.hdwq[i];
+ if (!vport || !vport->localport ||
+ !qp || !qp->io_wq)
+ return;
+
pring = qp->io_wq->pring;
if (!pring)
continue;
abts_scsi += qp->abts_scsi_io_bufs;
abts_nvme += qp->abts_nvme_io_bufs;
}
+ if (!vport || !vport->localport ||
+ vport->phba->hba_flag & HBA_PCI_ERR)
+ return;
+
lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT,
"6176 Lport x%px Localport x%px wait "
"timed out. Pending %d [%d:%d]. "
return;
localport = vport->localport;
+ if (!localport)
+ return;
lport = (struct lpfc_nvme_lport *)localport->private;
lpfc_printf_vlog(vport, KERN_INFO, LOG_NVME,
* return values is ignored. The upcall is a courtesy to the
* transport.
*/
- if (vport->load_flag & FC_UNLOADING)
+ if (vport->load_flag & FC_UNLOADING ||
+ unlikely(vport->phba->hba_flag & HBA_PCI_ERR))
(void)nvme_fc_set_remoteport_devloss(remoteport, 0);
ret = nvme_fc_unregister_remoteport(remoteport);
}
/**
+ * lpfc_sli4_nvme_pci_offline_aborted - Fast-path process of NVME xri abort
+ * @phba: pointer to lpfc hba data structure.
+ * @lpfc_ncmd: The nvme job structure for the request being aborted.
+ *
+ * This routine is invoked by the worker thread to process a SLI4 fast-path
+ * NVME aborted xri. Aborted NVME IO commands are completed to the transport
+ * here.
+ **/
+void
+lpfc_sli4_nvme_pci_offline_aborted(struct lpfc_hba *phba,
+ struct lpfc_io_buf *lpfc_ncmd)
+{
+ struct nvmefc_fcp_req *nvme_cmd = NULL;
+
+ lpfc_printf_log(phba, KERN_INFO, LOG_NVME_ABTS,
+ "6533 %s nvme_cmd %p tag x%x abort complete and "
+ "xri released\n", __func__,
+ lpfc_ncmd->nvmeCmd,
+ lpfc_ncmd->cur_iocbq.iotag);
+
+ /* Aborted NVME commands are required to not complete
+ * before the abort exchange command fully completes.
+ * Once completed, it is available via the put list.
+ */
+ if (lpfc_ncmd->nvmeCmd) {
+ nvme_cmd = lpfc_ncmd->nvmeCmd;
+ nvme_cmd->transferred_length = 0;
+ nvme_cmd->rcv_rsplen = 0;
+ nvme_cmd->status = NVME_SC_INTERNAL;
+ nvme_cmd->done(nvme_cmd);
+ lpfc_ncmd->nvmeCmd = NULL;
+ }
+ lpfc_release_nvme_buf(phba, lpfc_ncmd);
+}
+
+/**
* lpfc_sli4_nvme_xri_aborted - Fast-path process of NVME xri abort
* @phba: pointer to lpfc hba data structure.
* @axri: pointer to the fcp xri abort wcqe structure.
struct ulp_bde64 *bde;
dma_addr_t physaddr;
int i, cnt, nsegs;
- int do_pbde;
+ bool use_pbde = false;
int xc = 1;
if (!lpfc_is_link_up(phba)) {
if (!xc)
bf_set(wqe_xc, &wqe->fcp_tsend.wqe_com, 0);
- /* Word 11 - set sup, irsp, irsplen later */
- do_pbde = 0;
-
/* Word 12 */
wqe->fcp_tsend.fcp_data_len = rsp->transfer_length;
if (!xc)
bf_set(wqe_xc, &wqe->fcp_treceive.wqe_com, 0);
- /* Word 11 - set pbde later */
- if (phba->cfg_enable_pbde) {
- do_pbde = 1;
+ /* Word 11 - check for pbde */
+ if (nsegs == 1 && phba->cfg_enable_pbde) {
+ use_pbde = true;
+ /* Word 11 - PBDE bit already preset by template */
} else {
+ /* Overwrite default template setting */
bf_set(wqe_pbde, &wqe->fcp_treceive.wqe_com, 0);
- do_pbde = 0;
}
/* Word 12 */
((rsp->rsplen >> 2) - 1));
memcpy(&wqe->words[16], rsp->rspaddr, rsp->rsplen);
}
- do_pbde = 0;
/* Word 12 */
wqe->fcp_trsp.rsvd_12_15[0] = 0;
bf_set(lpfc_sli4_sge_last, sgl, 1);
sgl->word2 = cpu_to_le32(sgl->word2);
sgl->sge_len = cpu_to_le32(cnt);
- if (i == 0) {
- bde = (struct ulp_bde64 *)&wqe->words[13];
- if (do_pbde) {
- /* Words 13-15 (PBDE) */
- bde->addrLow = sgl->addr_lo;
- bde->addrHigh = sgl->addr_hi;
- bde->tus.f.bdeSize =
- le32_to_cpu(sgl->sge_len);
- bde->tus.f.bdeFlags = BUFF_TYPE_BDE_64;
- bde->tus.w = cpu_to_le32(bde->tus.w);
- } else {
- memset(bde, 0, sizeof(struct ulp_bde64));
- }
- }
sgl++;
ctxp->offset += cnt;
}
+
+ bde = (struct ulp_bde64 *)&wqe->words[13];
+ if (use_pbde) {
+ /* decrement sgl ptr backwards once to first data sge */
+ sgl--;
+
+ /* Words 13-15 (PBDE) */
+ bde->addrLow = sgl->addr_lo;
+ bde->addrHigh = sgl->addr_hi;
+ bde->tus.f.bdeSize = le32_to_cpu(sgl->sge_len);
+ bde->tus.f.bdeFlags = BUFF_TYPE_BDE_64;
+ bde->tus.w = cpu_to_le32(bde->tus.w);
+ } else {
+ memset(bde, 0, sizeof(struct ulp_bde64));
+ }
ctxp->state = LPFC_NVME_STE_DATA;
ctxp->entry_cnt++;
return nvmewqe;
lpfc_sli4_io_xri_aborted(struct lpfc_hba *phba,
struct sli4_wcqe_xri_aborted *axri, int idx)
{
- uint16_t xri = bf_get(lpfc_wcqe_xa_xri, axri);
- uint16_t rxid = bf_get(lpfc_wcqe_xa_remote_xid, axri);
+ u16 xri = 0;
+ u16 rxid = 0;
struct lpfc_io_buf *psb, *next_psb;
struct lpfc_sli4_hdw_queue *qp;
unsigned long iflag = 0;
int rrq_empty = 0;
struct lpfc_sli_ring *pring = phba->sli4_hba.els_wq->pring;
struct scsi_cmnd *cmd;
+ int offline = 0;
if (!(phba->cfg_enable_fc4_type & LPFC_ENABLE_FCP))
return;
-
+ offline = pci_channel_offline(phba->pcidev);
+ if (!offline) {
+ xri = bf_get(lpfc_wcqe_xa_xri, axri);
+ rxid = bf_get(lpfc_wcqe_xa_remote_xid, axri);
+ }
qp = &phba->sli4_hba.hdwq[idx];
spin_lock_irqsave(&phba->hbalock, iflag);
spin_lock(&qp->abts_io_buf_list_lock);
list_for_each_entry_safe(psb, next_psb,
&qp->lpfc_abts_io_buf_list, list) {
+ if (offline)
+ xri = psb->cur_iocbq.sli4_xritag;
if (psb->cur_iocbq.sli4_xritag == xri) {
list_del_init(&psb->list);
psb->flags &= ~LPFC_SBUF_XBUSY;
qp->abts_nvme_io_bufs--;
spin_unlock(&qp->abts_io_buf_list_lock);
spin_unlock_irqrestore(&phba->hbalock, iflag);
- lpfc_sli4_nvme_xri_aborted(phba, axri, psb);
- return;
+ if (!offline) {
+ lpfc_sli4_nvme_xri_aborted(phba, axri,
+ psb);
+ return;
+ }
+ lpfc_sli4_nvme_pci_offline_aborted(phba, psb);
+ spin_lock_irqsave(&phba->hbalock, iflag);
+ spin_lock(&qp->abts_io_buf_list_lock);
+ continue;
}
qp->abts_scsi_io_bufs--;
spin_unlock(&qp->abts_io_buf_list_lock);
rrq_empty = list_empty(&phba->active_rrq_list);
spin_unlock_irqrestore(&phba->hbalock, iflag);
- if (ndlp) {
+ if (ndlp && !offline) {
lpfc_set_rrq_active(phba, ndlp,
psb->cur_iocbq.sli4_lxritag, rxid, 1);
lpfc_sli4_abts_err_handler(phba, ndlp, axri);
}
- if (phba->cfg_fcp_wait_abts_rsp) {
+ if (phba->cfg_fcp_wait_abts_rsp || offline) {
spin_lock_irqsave(&psb->buf_lock, iflag);
cmd = psb->pCmd;
psb->pCmd = NULL;
* scsi_done upcall.
*/
if (cmd)
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/*
* We expect there is an abort thread waiting
lpfc_release_scsi_buf_s4(phba, psb);
if (rrq_empty)
lpfc_worker_wake_up(phba);
- return;
+ if (!offline)
+ return;
+ spin_lock_irqsave(&phba->hbalock, iflag);
+ spin_lock(&qp->abts_io_buf_list_lock);
+ continue;
}
}
spin_unlock(&qp->abts_io_buf_list_lock);
- for (i = 1; i <= phba->sli.last_iotag; i++) {
- iocbq = phba->sli.iocbq_lookup[i];
-
- if (!(iocbq->iocb_flag & LPFC_IO_FCP) ||
- (iocbq->iocb_flag & LPFC_IO_LIBDFC))
- continue;
- if (iocbq->sli4_xritag != xri)
- continue;
- psb = container_of(iocbq, struct lpfc_io_buf, cur_iocbq);
- psb->flags &= ~LPFC_SBUF_XBUSY;
- spin_unlock_irqrestore(&phba->hbalock, iflag);
- if (!list_empty(&pring->txq))
- lpfc_worker_wake_up(phba);
- return;
+ if (!offline) {
+ for (i = 1; i <= phba->sli.last_iotag; i++) {
+ iocbq = phba->sli.iocbq_lookup[i];
+ if (!(iocbq->iocb_flag & LPFC_IO_FCP) ||
+ (iocbq->iocb_flag & LPFC_IO_LIBDFC))
+ continue;
+ if (iocbq->sli4_xritag != xri)
+ continue;
+ psb = container_of(iocbq, struct lpfc_io_buf, cur_iocbq);
+ psb->flags &= ~LPFC_SBUF_XBUSY;
+ spin_unlock_irqrestore(&phba->hbalock, iflag);
+ if (!list_empty(&pring->txq))
+ lpfc_worker_wake_up(phba);
+ return;
+ }
}
spin_unlock_irqrestore(&phba->hbalock, iflag);
}
bpl += 2;
if (scsi_sg_count(scsi_cmnd)) {
/*
- * The driver stores the segment count returned from pci_map_sg
+ * The driver stores the segment count returned from dma_map_sg
* because this a count of dma-mappings used to map the use_sg
* pages. They are not guaranteed to be the same for those
* architectures that implement an IOMMU.
bpl += 2;
if (scsi_sg_count(scsi_cmnd)) {
/*
- * The driver stores the segment count returned from pci_map_sg
+ * The driver stores the segment count returned from dma_map_sg
* because this a count of dma-mappings used to map the use_sg
* pages. They are not guaranteed to be the same for those
* architectures that implement an IOMMU.
struct lpfc_vport *vport = phba->pport;
union lpfc_wqe128 *wqe = &pwqeq->wqe;
dma_addr_t physaddr;
- uint32_t num_bde = 0;
uint32_t dma_len;
uint32_t dma_offset = 0;
int nseg, i, j;
*/
if (scsi_sg_count(scsi_cmnd)) {
/*
- * The driver stores the segment count returned from pci_map_sg
+ * The driver stores the segment count returned from dma_map_sg
* because this a count of dma-mappings used to map the use_sg
* pages. They are not guaranteed to be the same for those
* architectures that implement an IOMMU.
j = 2;
for (i = 0; i < nseg; i++) {
sgl->word2 = 0;
- if ((num_bde + 1) == nseg) {
+ if (nseg == 1) {
bf_set(lpfc_sli4_sge_last, sgl, 1);
bf_set(lpfc_sli4_sge_type, sgl,
LPFC_SGE_TYPE_DATA);
j++;
}
- /*
- * Setup the first Payload BDE. For FCoE we just key off
- * Performance Hints, for FC we use lpfc_enable_pbde.
- * We populate words 13-15 of IOCB/WQE.
+
+ /* PBDE support for first data SGE only.
+ * For FCoE, we key off Performance Hints.
+ * For FC, we key off lpfc_enable_pbde.
*/
- if ((phba->sli3_options & LPFC_SLI4_PERFH_ENABLED) ||
- phba->cfg_enable_pbde) {
+ if (nseg == 1 &&
+ ((phba->sli3_options & LPFC_SLI4_PERFH_ENABLED) ||
+ phba->cfg_enable_pbde)) {
+ /* Words 13-15 */
bde = (struct ulp_bde64 *)
&wqe->words[13];
bde->addrLow = first_data_sgl->addr_lo;
bde->tus.f.bdeFlags = BUFF_TYPE_BDE_64;
bde->tus.w = cpu_to_le32(bde->tus.w);
+ /* Word 11 - set PBDE bit */
+ bf_set(wqe_pbde, &wqe->generic.wqe_com, 1);
} else {
memset(&wqe->words[13], 0, (sizeof(uint32_t) * 3));
+ /* Word 11 - PBDE bit disabled by default template */
}
} else {
sgl += 1;
- /* clear the last flag in the fcp_rsp map entry */
+ /* set the last flag in the fcp_rsp map entry */
sgl->word2 = le32_to_cpu(sgl->word2);
bf_set(lpfc_sli4_sge_last, sgl, 1);
sgl->word2 = cpu_to_le32(sgl->word2);
}
}
- /* Word 11 */
- if (phba->cfg_enable_pbde)
- bf_set(wqe_pbde, &wqe->generic.wqe_com, 1);
-
/*
* Finish initializing those IOCB fields that are dependent on the
* scsi_cmnd request_buffer. Note that for SLI-2 the bdeSize is
*/
if (scsi_sg_count(scsi_cmnd)) {
/*
- * The driver stores the segment count returned from pci_map_sg
+ * The driver stores the segment count returned from dma_map_sg
* because this a count of dma-mappings used to map the use_sg
* pages. They are not guaranteed to be the same for those
* architectures that implement an IOMMU.
int cpu;
/* At this point we are either LPFC_CFG_MANAGED or LPFC_CFG_MONITOR */
- if (phba->cmf_active_mode == LPFC_CFG_MANAGED) {
+ if (phba->cmf_active_mode == LPFC_CFG_MANAGED &&
+ phba->cmf_max_bytes_per_interval) {
total = 0;
for_each_present_cpu(cpu) {
cgs = per_cpu_ptr(phba->cmf_stat, cpu);
goto out;
/* The sdev is not guaranteed to be valid post scsi_done upcall. */
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/*
* If there is an abort thread waiting for command completion
#endif
/* The sdev is not guaranteed to be valid post scsi_done upcall. */
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/*
* If there is an abort thread waiting for command completion
phba->lpfc_scsi_prep_cmnd_buf = lpfc_scsi_prep_cmnd_buf_s4;
break;
default:
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1418 Invalid HBA PCI-device group: 0x%x\n",
dev_grp);
return -ENODEV;
shost);
out_fail_command:
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
/* Issue LOGO, if no LOGO is outstanding */
spin_lock_irqsave(&pnode->lock, flags);
- if (!(pnode->upcall_flags & NLP_WAIT_FOR_LOGO) &&
+ if (!(pnode->save_flags & NLP_WAIT_FOR_LOGO) &&
!pnode->logo_waitq) {
pnode->logo_waitq = &waitq;
pnode->nlp_fcp_info &= ~NLP_FCP_2_DEVICE;
pnode->nlp_flag |= NLP_ISSUE_LOGO;
- pnode->upcall_flags |= NLP_WAIT_FOR_LOGO;
+ pnode->save_flags |= NLP_WAIT_FOR_LOGO;
spin_unlock_irqrestore(&pnode->lock, flags);
lpfc_unreg_rpi(vport, pnode);
wait_event_timeout(waitq,
- (!(pnode->upcall_flags &
+ (!(pnode->save_flags &
NLP_WAIT_FOR_LOGO)),
msecs_to_jiffies(dev_loss_tmo *
1000));
- if (pnode->upcall_flags & NLP_WAIT_FOR_LOGO) {
+ if (pnode->save_flags & NLP_WAIT_FOR_LOGO) {
lpfc_printf_vlog(vport, KERN_ERR, logit,
"0725 SCSI layer TGTRST "
"failed & LOGO TMO (%d, %llu) "
"return x%x\n",
tgt_id, lun_id, status);
spin_lock_irqsave(&pnode->lock, flags);
- pnode->upcall_flags &= ~NLP_WAIT_FOR_LOGO;
+ pnode->save_flags &= ~NLP_WAIT_FOR_LOGO;
} else {
spin_lock_irqsave(&pnode->lock, flags);
}
if (rc)
goto error;
+ /* Wait for successful restart of adapter */
+ if (phba->sli_rev < LPFC_SLI_REV4) {
+ rc = lpfc_sli_chipset_init(phba);
+ if (rc)
+ goto error;
+ }
+
rc = lpfc_online(phba);
if (rc)
goto error;
.this_id = -1,
.sg_tablesize = 1,
.cmd_per_lun = 1,
- .shost_attrs = lpfc_hba_attrs,
+ .shost_groups = lpfc_hba_groups,
.max_sectors = 0xFFFFFFFF,
.vendor_id = LPFC_NL_VENDOR_ID,
.track_queue_depth = 0,
.this_id = -1,
.sg_tablesize = LPFC_DEFAULT_SG_SEG_CNT,
.cmd_per_lun = LPFC_CMD_PER_LUN,
- .shost_attrs = lpfc_hba_attrs,
+ .shost_groups = lpfc_hba_groups,
.max_sectors = 0xFFFFFFFF,
.vendor_id = LPFC_NL_VENDOR_ID,
.change_queue_depth = scsi_change_queue_depth,
}
if ((iocbq->iocb_flag & LPFC_EXCHANGE_BUSY) &&
- (sglq->state != SGL_XRI_ABORTED)) {
+ (!(unlikely(pci_channel_offline(phba->pcidev)))) &&
+ sglq->state != SGL_XRI_ABORTED) {
spin_lock_irqsave(&phba->sli4_hba.sgl_list_lock,
iflag);
lpfc_sli_cancel_iocbs(phba, &txq,
IOSTAT_LOCAL_REJECT,
IOERR_SLI_DOWN);
- /* Flush the txcmpq */
+ /* Flush the txcmplq */
lpfc_sli_cancel_iocbs(phba, &txcmplq,
IOSTAT_LOCAL_REJECT,
IOERR_SLI_DOWN);
+ if (unlikely(pci_channel_offline(phba->pcidev)))
+ lpfc_sli4_io_xri_aborted(phba, NULL, 0);
}
} else {
pring = &psli->sli3_ring[LPFC_FCP_RING];
/* Zero out Congestion Signal ACQE counter */
phba->cgn_acqe_cnt = 0;
- atomic64_set(&phba->cgn_acqe_stat.warn, 0);
- atomic64_set(&phba->cgn_acqe_stat.alarm, 0);
acqe = bf_get(lpfc_mbx_set_feature_CGN_acqe_freq,
&pmb->u.mqe.un.set_feature);
lpfc_cmf_setup(struct lpfc_hba *phba)
{
LPFC_MBOXQ_t *mboxq;
- struct lpfc_mqe *mqe;
struct lpfc_dmabuf *mp;
struct lpfc_pc_sli4_params *sli4_params;
- struct lpfc_sli4_parameters *mbx_sli4_parameters;
- int length;
int rc, cmf, mi_ver;
+ rc = lpfc_sli4_refresh_params(phba);
+ if (unlikely(rc))
+ return rc;
+
mboxq = (LPFC_MBOXQ_t *)mempool_alloc(phba->mbox_mem_pool, GFP_KERNEL);
if (!mboxq)
return -ENOMEM;
- mqe = &mboxq->u.mqe;
- /* Read the port's SLI4 Config Parameters */
- length = (sizeof(struct lpfc_mbx_get_sli4_parameters) -
- sizeof(struct lpfc_sli4_cfg_mhdr));
- lpfc_sli4_config(phba, mboxq, LPFC_MBOX_SUBSYSTEM_COMMON,
- LPFC_MBOX_OPCODE_GET_SLI4_PARAMETERS,
- length, LPFC_SLI4_MBX_EMBED);
-
- rc = lpfc_sli_issue_mbox(phba, mboxq, MBX_POLL);
- if (unlikely(rc)) {
- mempool_free(mboxq, phba->mbox_mem_pool);
- return rc;
- }
-
- /* Gather info on CMF and MI support */
sli4_params = &phba->sli4_hba.pc_sli4_params;
- mbx_sli4_parameters = &mqe->un.get_sli4_parameters.sli4_parameters;
- sli4_params->mi_ver = bf_get(cfg_mi_ver, mbx_sli4_parameters);
- sli4_params->cmf = bf_get(cfg_cmf, mbx_sli4_parameters);
/* Are we forcing MI off via module parameter? */
if (!phba->cfg_enable_mi)
/* initialize congestion buffer info */
lpfc_init_congestion_buf(phba);
lpfc_init_congestion_stat(phba);
+
+ /* Zero out Congestion Signal counters */
+ atomic64_set(&phba->cgn_acqe_stat.alarm, 0);
+ atomic64_set(&phba->cgn_acqe_stat.warn, 0);
}
rc = lpfc_sli4_cgn_params_read(phba);
struct lpfc_vport *vport = phba->pport;
struct lpfc_dmabuf *mp;
struct lpfc_rqb *rqbp;
+ u32 flg;
/* Perform a PCI function reset to start from clean */
rc = lpfc_pci_function_reset(phba);
else {
spin_lock_irq(&phba->hbalock);
phba->sli.sli_flag |= LPFC_SLI_ACTIVE;
+ flg = phba->sli.sli_flag;
spin_unlock_irq(&phba->hbalock);
+ /* Allow a little time after setting SLI_ACTIVE for any polled
+ * MBX commands to complete via BSG.
+ */
+ for (i = 0; i < 50 && (flg & LPFC_SLI_MBOX_ACTIVE); i++) {
+ msleep(20);
+ spin_lock_irq(&phba->hbalock);
+ flg = phba->sli.sli_flag;
+ spin_unlock_irq(&phba->hbalock);
+ }
}
lpfc_sli4_dip(phba);
"(%d):2541 Mailbox command x%x "
"(x%x/x%x) failure: "
"mqe_sta: x%x mcqe_sta: x%x/x%x "
- "Data: x%x x%x\n,",
+ "Data: x%x x%x\n",
mboxq->vport ? mboxq->vport->vpi : 0,
mboxq->u.mb.mbxCommand,
lpfc_sli_config_mbox_subsys_get(phba,
"(%d):2597 Sync Mailbox command "
"x%x (x%x/x%x) failure: "
"mqe_sta: x%x mcqe_sta: x%x/x%x "
- "Data: x%x x%x\n,",
+ "Data: x%x x%x\n",
mboxq->vport ? mboxq->vport->vpi : 0,
mboxq->u.mb.mbxCommand,
lpfc_sli_config_mbox_subsys_get(phba,
phba->lpfc_sli_brdready = lpfc_sli_brdready_s4;
break;
default:
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1420 Invalid HBA PCI-device group: 0x%x\n",
dev_grp);
return -ENODEV;
phba->__lpfc_sli_issue_fcp_io = __lpfc_sli_issue_fcp_io_s4;
break;
default:
- lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT,
+ lpfc_printf_log(phba, KERN_ERR, LOG_INIT,
"1419 Invalid HBA PCI-device group: 0x%x\n",
dev_grp);
return -ENODEV;
/* ABTS WQE must go to the same WQ as the WQE to be aborted */
abtsiocbp->hba_wqidx = cmdiocb->hba_wqidx;
- if (cmdiocb->iocb_flag & LPFC_IO_FCP) {
- abtsiocbp->iocb_flag |= LPFC_IO_FCP;
- abtsiocbp->iocb_flag |= LPFC_USE_FCPWQIDX;
- }
+ if (cmdiocb->iocb_flag & LPFC_IO_FCP)
+ abtsiocbp->iocb_flag |= (LPFC_IO_FCP | LPFC_USE_FCPWQIDX);
if (cmdiocb->iocb_flag & LPFC_IO_FOF)
abtsiocbp->iocb_flag |= LPFC_IO_FOF;
- if (phba->link_state >= LPFC_LINK_UP)
- iabt->ulpCommand = CMD_ABORT_XRI_CN;
- else
+ if (phba->link_state < LPFC_LINK_UP ||
+ (phba->sli_rev == LPFC_SLI_REV4 &&
+ phba->sli4_hba.link_state.status == LPFC_FC_LA_TYPE_LINK_DOWN))
iabt->ulpCommand = CMD_CLOSE_XRI_CN;
+ else
+ iabt->ulpCommand = CMD_ABORT_XRI_CN;
if (cmpl)
abtsiocbp->iocb_cmpl = cmpl;
}
/**
- * lpfc_sli_validate_fcp_iocb - find commands associated with a vport or LUN
+ * lpfc_sli_validate_fcp_iocb_for_abort - filter iocbs appropriate for FCP aborts
+ * @iocbq: Pointer to iocb object.
+ * @vport: Pointer to driver virtual port object.
+ *
+ * This function acts as an iocb filter for functions which abort FCP iocbs.
+ *
+ * Return values
+ * -ENODEV, if a null iocb or vport ptr is encountered
+ * -EINVAL, if the iocb is not an FCP I/O, not on the TX cmpl queue, premarked as
+ * driver already started the abort process, or is an abort iocb itself
+ * 0, passes criteria for aborting the FCP I/O iocb
+ **/
+static int
+lpfc_sli_validate_fcp_iocb_for_abort(struct lpfc_iocbq *iocbq,
+ struct lpfc_vport *vport)
+{
+ IOCB_t *icmd = NULL;
+
+ /* No null ptr vports */
+ if (!iocbq || iocbq->vport != vport)
+ return -ENODEV;
+
+ /* iocb must be for FCP IO, already exists on the TX cmpl queue,
+ * can't be premarked as driver aborted, nor be an ABORT iocb itself
+ */
+ icmd = &iocbq->iocb;
+ if (!(iocbq->iocb_flag & LPFC_IO_FCP) ||
+ !(iocbq->iocb_flag & LPFC_IO_ON_TXCMPLQ) ||
+ (iocbq->iocb_flag & LPFC_DRIVER_ABORTED) ||
+ (icmd->ulpCommand == CMD_ABORT_XRI_CN ||
+ icmd->ulpCommand == CMD_CLOSE_XRI_CN))
+ return -EINVAL;
+
+ return 0;
+}
+
+/**
+ * lpfc_sli_validate_fcp_iocb - validate commands associated with a SCSI target
* @iocbq: Pointer to driver iocb object.
* @vport: Pointer to driver virtual port object.
* @tgt_id: SCSI ID of the target.
* @lun_id: LUN ID of the scsi device.
* @ctx_cmd: LPFC_CTX_LUN/LPFC_CTX_TGT/LPFC_CTX_HOST
*
- * This function acts as an iocb filter for functions which abort or count
- * all FCP iocbs pending on a lun/SCSI target/SCSI host. It will return
+ * This function acts as an iocb filter for validating a lun/SCSI target/SCSI
+ * host.
+ *
+ * It will return
* 0 if the filtering criteria is met for the given iocb and will return
* 1 if the filtering criteria is not met.
* If ctx_cmd == LPFC_CTX_LUN, the function returns 0 only if the
lpfc_ctx_cmd ctx_cmd)
{
struct lpfc_io_buf *lpfc_cmd;
- IOCB_t *icmd = NULL;
int rc = 1;
- if (!iocbq || iocbq->vport != vport)
- return rc;
-
- if (!(iocbq->iocb_flag & LPFC_IO_FCP) ||
- !(iocbq->iocb_flag & LPFC_IO_ON_TXCMPLQ) ||
- iocbq->iocb_flag & LPFC_DRIVER_ABORTED)
- return rc;
-
- icmd = &iocbq->iocb;
- if (icmd->ulpCommand == CMD_ABORT_XRI_CN ||
- icmd->ulpCommand == CMD_CLOSE_XRI_CN)
- return rc;
-
lpfc_cmd = container_of(iocbq, struct lpfc_io_buf, cur_iocbq);
if (lpfc_cmd->pCmd == NULL)
{
struct lpfc_hba *phba = vport->phba;
struct lpfc_iocbq *iocbq;
+ IOCB_t *icmd = NULL;
int sum, i;
+ unsigned long iflags;
- spin_lock_irq(&phba->hbalock);
+ spin_lock_irqsave(&phba->hbalock, iflags);
for (i = 1, sum = 0; i <= phba->sli.last_iotag; i++) {
iocbq = phba->sli.iocbq_lookup[i];
- if (lpfc_sli_validate_fcp_iocb (iocbq, vport, tgt_id, lun_id,
- ctx_cmd) == 0)
+ if (!iocbq || iocbq->vport != vport)
+ continue;
+ if (!(iocbq->iocb_flag & LPFC_IO_FCP) ||
+ !(iocbq->iocb_flag & LPFC_IO_ON_TXCMPLQ))
+ continue;
+
+ /* Include counting outstanding aborts */
+ icmd = &iocbq->iocb;
+ if (icmd->ulpCommand == CMD_ABORT_XRI_CN ||
+ icmd->ulpCommand == CMD_CLOSE_XRI_CN) {
+ sum++;
+ continue;
+ }
+
+ if (lpfc_sli_validate_fcp_iocb(iocbq, vport, tgt_id, lun_id,
+ ctx_cmd) == 0)
sum++;
}
- spin_unlock_irq(&phba->hbalock);
+ spin_unlock_irqrestore(&phba->hbalock, iflags);
return sum;
}
*
* This function sends an abort command for every SCSI command
* associated with the given virtual port pending on the ring
- * filtered by lpfc_sli_validate_fcp_iocb function.
+ * filtered by lpfc_sli_validate_fcp_iocb_for_abort and then
+ * lpfc_sli_validate_fcp_iocb function. The ordering for validation before
+ * submitting abort iocbs must be lpfc_sli_validate_fcp_iocb_for_abort
+ * followed by lpfc_sli_validate_fcp_iocb.
+ *
* When abort_cmd == LPFC_CTX_LUN, the function sends abort only to the
* FCP iocbs associated with lun specified by tgt_id and lun_id
* parameters
for (i = 1; i <= phba->sli.last_iotag; i++) {
iocbq = phba->sli.iocbq_lookup[i];
+ if (lpfc_sli_validate_fcp_iocb_for_abort(iocbq, vport))
+ continue;
+
if (lpfc_sli_validate_fcp_iocb(iocbq, vport, tgt_id, lun_id,
abort_cmd) != 0)
continue;
*
* This function sends an abort command for every SCSI command
* associated with the given virtual port pending on the ring
- * filtered by lpfc_sli_validate_fcp_iocb function.
+ * filtered by lpfc_sli_validate_fcp_iocb_for_abort and then
+ * lpfc_sli_validate_fcp_iocb function. The ordering for validation before
+ * submitting abort iocbs must be lpfc_sli_validate_fcp_iocb_for_abort
+ * followed by lpfc_sli_validate_fcp_iocb.
+ *
* When taskmgmt_cmd == LPFC_CTX_LUN, the function sends abort only to the
* FCP iocbs associated with lun specified by tgt_id and lun_id
* parameters
for (i = 1; i <= phba->sli.last_iotag; i++) {
iocbq = phba->sli.iocbq_lookup[i];
+ if (lpfc_sli_validate_fcp_iocb_for_abort(iocbq, vport))
+ continue;
+
if (lpfc_sli_validate_fcp_iocb(iocbq, vport, tgt_id, lun_id,
cmd) != 0)
continue;
fail_msg,
piocbq->iotag, piocbq->sli4_xritag);
list_add_tail(&piocbq->list, &completions);
+ fail_msg = NULL;
}
spin_unlock_irqrestore(&pring->ring_lock, iflags);
}
qp = &phba->sli4_hba.hdwq[hwqid];
lpfc_ncmd = NULL;
+ if (!qp) {
+ lpfc_printf_log(phba, KERN_INFO,
+ LOG_SLI | LOG_NVME_ABTS | LOG_FCP,
+ "5556 NULL qp for hwqid x%x\n", hwqid);
+ return lpfc_ncmd;
+ }
multixri_pool = qp->p_multixri_pool;
+ if (!multixri_pool) {
+ lpfc_printf_log(phba, KERN_INFO,
+ LOG_SLI | LOG_NVME_ABTS | LOG_FCP,
+ "5557 NULL multixri for hwqid x%x\n", hwqid);
+ return lpfc_ncmd;
+ }
pvt_pool = &multixri_pool->pvt_pool;
+ if (!pvt_pool) {
+ lpfc_printf_log(phba, KERN_INFO,
+ LOG_SLI | LOG_NVME_ABTS | LOG_FCP,
+ "5558 NULL pvt_pool for hwqid x%x\n", hwqid);
+ return lpfc_ncmd;
+ }
multixri_pool->io_req_count++;
/* If pvt_pool is empty, move some XRIs from public to private pool */
qp = &phba->sli4_hba.hdwq[hwqid];
lpfc_cmd = NULL;
+ if (!qp) {
+ lpfc_printf_log(phba, KERN_WARNING,
+ LOG_SLI | LOG_NVME_ABTS | LOG_FCP,
+ "5555 NULL qp for hwqid x%x\n", hwqid);
+ return lpfc_cmd;
+ }
if (phba->cfg_xri_rebalancing)
lpfc_cmd = lpfc_get_io_buf_from_multixri_pools(
int lpfc_sli4_resume_rpi(struct lpfc_nodelist *,
void (*)(struct lpfc_hba *, LPFC_MBOXQ_t *), void *);
void lpfc_sli4_els_xri_abort_event_proc(struct lpfc_hba *phba);
+void lpfc_sli4_nvme_pci_offline_aborted(struct lpfc_hba *phba,
+ struct lpfc_io_buf *lpfc_ncmd);
void lpfc_sli4_nvme_xri_aborted(struct lpfc_hba *phba,
struct sli4_wcqe_xri_aborted *axri,
struct lpfc_io_buf *lpfc_ncmd);
* included with this package. *
*******************************************************************/
-#define LPFC_DRIVER_VERSION "14.0.0.1"
+#define LPFC_DRIVER_VERSION "14.0.0.3"
#define LPFC_DRIVER_NAME "lpfc"
/* Used for SLI 2/3 */
static void cmd_done(struct fsc_state *, int result);
static void set_dma_cmds(struct fsc_state *, struct scsi_cmnd *);
-
-static int mac53c94_queue_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int mac53c94_queue_lck(struct scsi_cmnd *cmd)
{
struct fsc_state *state;
}
#endif
- cmd->scsi_done = done;
cmd->host_scribble = NULL;
state = (struct fsc_state *) cmd->device->host->hostdata;
cmd = state->current_req;
if (cmd) {
cmd->result = result;
- (*cmd->scsi_done)(cmd);
+ scsi_done(cmd);
state->current_req = NULL;
}
state->phase = idle;
*
* The command queuing entry point for the mid-layer.
*/
-static int
-megaraid_queue_lck(struct scsi_cmnd *scmd, void (*done)(struct scsi_cmnd *))
+static int megaraid_queue_lck(struct scsi_cmnd *scmd)
{
adapter_t *adapter;
scb_t *scb;
adapter = (adapter_t *)scmd->device->host->hostdata;
- scmd->scsi_done = done;
-
-
/*
* Allocate and build a SCB request
* busy flag will be set if mega_build_cmd() command could not
/* have just LUN 0 for each target on virtual channels */
if (cmd->device->lun) {
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
if(ldrv_num > max_ldrv_num ) {
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
* devices
*/
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
}
*/
if( !adapter->has_cluster ) {
cmd->result = (DID_OK << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
return scb;
#else
cmd->result = (DID_OK << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
#endif
kunmap_atomic(buf - sg->offset);
cmd->result = (DID_OK << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
if( ! adapter->has_cluster ) {
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
default:
cmd->result = (DID_BAD_TARGET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return NULL;
}
}
struct scsi_pointer* spos = (struct scsi_pointer *)pos;
cmd = list_entry(spos, struct scsi_cmnd, SCp);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
INIT_LIST_HEAD(&adapter->completed_list);
static DEVICE_ATTR_ADMIN_RO(megaraid_mbox_app_hndl);
// Host template initializer for megaraid mbox sysfs device attributes
-static struct device_attribute *megaraid_shost_attrs[] = {
- &dev_attr_megaraid_mbox_app_hndl,
+static struct attribute *megaraid_shost_attrs[] = {
+ &dev_attr_megaraid_mbox_app_hndl.attr,
NULL,
};
+ATTRIBUTE_GROUPS(megaraid_shost);
static DEVICE_ATTR_ADMIN_RO(megaraid_mbox_ld);
// Host template initializer for megaraid mbox sysfs device attributes
-static struct device_attribute *megaraid_sdev_attrs[] = {
- &dev_attr_megaraid_mbox_ld,
+static struct attribute *megaraid_sdev_attrs[] = {
+ &dev_attr_megaraid_mbox_ld.attr,
NULL,
};
+ATTRIBUTE_GROUPS(megaraid_sdev);
+
/*
* Scsi host template for megaraid unified driver
*/
.eh_host_reset_handler = megaraid_reset_handler,
.change_queue_depth = scsi_change_queue_depth,
.no_write_same = 1,
- .sdev_attrs = megaraid_sdev_attrs,
- .shost_attrs = megaraid_shost_attrs,
+ .sdev_groups = megaraid_sdev_groups,
+ .shost_groups = megaraid_shost_groups,
};
*
* Queue entry point for mailbox based controllers.
*/
-static int
-megaraid_queue_command_lck(struct scsi_cmnd *scp, void (*done)(struct scsi_cmnd *))
+static int megaraid_queue_command_lck(struct scsi_cmnd *scp)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
adapter_t *adapter;
scb_t *scb;
int if_busy;
adapter = SCP2ADAPTER(scp);
- scp->scsi_done = done;
scp->result = 0;
/*
megaraid_dealloc_scb(adapter, scb);
// send the scsi packet back to kernel
- scp->scsi_done(scp);
+ scsi_done(scp);
}
return;
scb->sno, scb->dev_channel, scb->dev_target));
scp->result = (DID_ABORT << 16);
- scp->scsi_done(scp);
+ scsi_done(scp);
megaraid_dealloc_scb(adapter, scb);
scb->dev_channel, scb->dev_target));
scp->result = (DID_ABORT << 16);
- scp->scsi_done(scp);
+ scsi_done(scp);
megaraid_dealloc_scb(adapter, scb);
}
scb->scp->result = (DID_RESET << 16);
- scb->scp->scsi_done(scb->scp);
+ scsi_done(scb->scp);
megaraid_dealloc_scb(adapter, scb);
}
/*
* MegaRAID SAS Driver meta data
*/
-#define MEGASAS_VERSION "07.717.02.00-rc1"
-#define MEGASAS_RELDATE "May 19, 2021"
+#define MEGASAS_VERSION "07.719.03.00-rc1"
+#define MEGASAS_RELDATE "Sep 29, 2021"
#define MEGASAS_MSIX_NAME_LEN 32
if (instance->unload == 1) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
return SCSI_MLQUEUE_HOST_BUSY;
} else {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
}
if (!mr_device_priv_data ||
(atomic_read(&instance->adprecovery) == MEGASAS_HW_CRITICAL_ERROR)) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
ld_tgt_id = MEGASAS_TARGET_ID(scmd->device);
if (instance->ld_tgtid_status[ld_tgt_id] == LD_TARGET_ID_DELETED) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
}
return instance->instancet->build_and_issue_cmd(instance, scmd);
out_done:
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
reset_index, reset_cmd,
reset_cmd->scmd->cmnd[0]);
- reset_cmd->scmd->scsi_done(reset_cmd->scmd);
+ scsi_done(reset_cmd->scmd);
megasas_return_cmd(instance, reset_cmd);
} else if (reset_cmd->sync_cmd) {
dev_notice(&instance->pdev->dev, "%p synch cmds"
static DEVICE_ATTR_RO(dump_system_regs);
static DEVICE_ATTR_RO(raid_map_id);
-static struct device_attribute *megaraid_host_attrs[] = {
- &dev_attr_fw_crash_buffer_size,
- &dev_attr_fw_crash_buffer,
- &dev_attr_fw_crash_state,
- &dev_attr_page_size,
- &dev_attr_ldio_outstanding,
- &dev_attr_fw_cmds_outstanding,
- &dev_attr_enable_sdev_max_qd,
- &dev_attr_dump_system_regs,
- &dev_attr_raid_map_id,
+static struct attribute *megaraid_host_attrs[] = {
+ &dev_attr_fw_crash_buffer_size.attr,
+ &dev_attr_fw_crash_buffer.attr,
+ &dev_attr_fw_crash_state.attr,
+ &dev_attr_page_size.attr,
+ &dev_attr_ldio_outstanding.attr,
+ &dev_attr_fw_cmds_outstanding.attr,
+ &dev_attr_enable_sdev_max_qd.attr,
+ &dev_attr_dump_system_regs.attr,
+ &dev_attr_raid_map_id.attr,
NULL,
};
+ATTRIBUTE_GROUPS(megaraid_host);
+
/*
* Scsi host template for megaraid_sas driver
*/
.eh_abort_handler = megasas_task_abort,
.eh_host_reset_handler = megasas_reset_bus_host,
.eh_timed_out = megasas_reset_timer,
- .shost_attrs = megaraid_host_attrs,
+ .shost_groups = megaraid_host_groups,
.bios_param = megasas_bios_param,
.map_queues = megasas_map_queues,
.mq_poll = megasas_blk_mq_poll,
atomic_dec(&instance->fw_outstanding);
scsi_dma_unmap(cmd->scmd);
- cmd->scmd->scsi_done(cmd->scmd);
+ scsi_done(cmd->scmd);
megasas_return_cmd(instance, cmd);
break;
atomic_dec(&instance->fw_outstanding);
scsi_dma_unmap(cmd->scmd);
- cmd->scmd->scsi_done(cmd->scmd);
+ scsi_done(cmd->scmd);
megasas_return_cmd(instance, cmd);
break;
megasas_return_cmd_fusion(instance, cmd);
scsi_dma_unmap(scmd_local);
megasas_sdev_busy_dec(instance, scmd_local);
- scmd_local->scsi_done(scmd_local);
+ scsi_done(scmd_local);
}
}
/**
+ * access_irq_context: Access to reply processing
+ * @irq_context: IRQ context
+ *
+ * Synchronize access to reply processing.
+ *
+ * Return: true on success, false on failure.
+ */
+static inline
+bool access_irq_context(struct megasas_irq_context *irq_context)
+{
+ if (!irq_context)
+ return true;
+
+ if (atomic_add_unless(&irq_context->in_used, 1, 1))
+ return true;
+
+ return false;
+}
+
+/**
+ * release_irq_context: Release reply processing
+ * @irq_context: IRQ context
+ *
+ * Release access of reply processing.
+ *
+ * Return: Nothing.
+ */
+static inline
+void release_irq_context(struct megasas_irq_context *irq_context)
+{
+ if (irq_context)
+ atomic_dec(&irq_context->in_used);
+}
+
+/**
* complete_cmd_fusion - Completes command
* @instance: Adapter soft state
* @MSIxIndex: MSI number
if (atomic_read(&instance->adprecovery) == MEGASAS_HW_CRITICAL_ERROR)
return IRQ_HANDLED;
+ if (!access_irq_context(irq_context))
+ return 0;
+
desc = fusion->reply_frames_desc[MSIxIndex] +
fusion->last_reply_idx[MSIxIndex];
reply_descript_type = reply_desc->ReplyFlags &
MPI2_RPY_DESCRIPT_FLAGS_TYPE_MASK;
- if (reply_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED)
+ if (reply_descript_type == MPI2_RPY_DESCRIPT_FLAGS_UNUSED) {
+ release_irq_context(irq_context);
return IRQ_NONE;
-
- if (irq_context && !atomic_add_unless(&irq_context->in_used, 1, 1))
- return 0;
+ }
num_completed = 0;
megasas_return_cmd_fusion(instance, cmd_fusion);
scsi_dma_unmap(scmd_local);
megasas_sdev_busy_dec(instance, scmd_local);
- scmd_local->scsi_done(scmd_local);
+ scsi_done(scmd_local);
} else /* Optimal VD - R1 FP command completion. */
megasas_complete_r1_command(instance, cmd_fusion);
break;
irq_context->irq_line_enable = true;
irq_poll_sched(&irq_context->irqpoll);
}
- atomic_dec(&irq_context->in_used);
+ release_irq_context(irq_context);
return num_completed;
}
}
megasas_check_and_restore_queue_depth(instance);
}
- if (irq_context)
- atomic_dec(&irq_context->in_used);
+ release_irq_context(irq_context);
return num_completed;
}
atomic_dec(&instance->ldio_outstanding);
megasas_return_cmd_fusion(instance, cmd_fusion);
scsi_dma_unmap(scmd_local);
- scmd_local->scsi_done(scmd_local);
+ scsi_done(scmd_local);
}
}
}
-/*
- * Complete a SCSI command
- */
-static void mesh_completed(struct mesh_state *ms, struct scsi_cmnd *cmd)
-{
- (*cmd->scsi_done)(cmd);
-}
-
-
/* Called with meshinterrupt disabled, initialize the chipset
* and eventually do the initial bus reset. The lock must not be
* held since we can schedule.
#endif
}
cmd->SCp.this_residual -= ms->data_ptr;
- mesh_completed(ms, cmd);
+ scsi_done(cmd);
}
if (start_next) {
out_8(&ms->mesh->sequence, SEQ_ENBRESEL);
if ((cmd = tp->current_req) != NULL) {
set_host_byte(cmd, DID_RESET);
tp->current_req = NULL;
- mesh_completed(ms, cmd);
+ scsi_done(cmd);
}
ms->tgts[tgt].sdtr_state = do_sdtr;
ms->tgts[tgt].sync_params = ASYNC_PARAMS;
while ((cmd = ms->request_q) != NULL) {
ms->request_q = (struct scsi_cmnd *) cmd->host_scribble;
set_host_byte(cmd, DID_RESET);
- mesh_completed(ms, cmd);
+ scsi_done(cmd);
}
ms->phase = idle;
ms->msgphase = msg_none;
* Called by midlayer with host locked to queue a new
* request
*/
-static int mesh_queue_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int mesh_queue_lck(struct scsi_cmnd *cmd)
{
struct mesh_state *ms;
- cmd->scsi_done = done;
cmd->host_scribble = NULL;
ms = (struct mesh_state *) cmd->device->host->hostdata;
static void
mpi3mr_print_ioc_info(struct mpi3mr_ioc *mrioc)
{
- int i = 0, bytes_wrote = 0;
+ int i = 0, bytes_written = 0;
char personality[16];
char protocol[50] = {0};
char capabilities[100] = {0};
- bool is_string_nonempty = false;
struct mpi3mr_compimg_ver *fwver = &mrioc->facts.fw_ver;
switch (mrioc->facts.personality) {
for (i = 0; i < ARRAY_SIZE(mpi3mr_protocols); i++) {
if (mrioc->facts.protocol_flags &
mpi3mr_protocols[i].protocol) {
- if (is_string_nonempty &&
- (bytes_wrote < sizeof(protocol)))
- bytes_wrote += snprintf(protocol + bytes_wrote,
- (sizeof(protocol) - bytes_wrote), ",");
-
- if (bytes_wrote < sizeof(protocol))
- bytes_wrote += snprintf(protocol + bytes_wrote,
- (sizeof(protocol) - bytes_wrote), "%s",
+ bytes_written += scnprintf(protocol + bytes_written,
+ sizeof(protocol) - bytes_written, "%s%s",
+ bytes_written ? "," : "",
mpi3mr_protocols[i].name);
- is_string_nonempty = true;
}
}
- bytes_wrote = 0;
- is_string_nonempty = false;
+ bytes_written = 0;
for (i = 0; i < ARRAY_SIZE(mpi3mr_capabilities); i++) {
if (mrioc->facts.protocol_flags &
mpi3mr_capabilities[i].capability) {
- if (is_string_nonempty &&
- (bytes_wrote < sizeof(capabilities)))
- bytes_wrote += snprintf(capabilities + bytes_wrote,
- (sizeof(capabilities) - bytes_wrote), ",");
-
- if (bytes_wrote < sizeof(capabilities))
- bytes_wrote += snprintf(capabilities + bytes_wrote,
- (sizeof(capabilities) - bytes_wrote), "%s",
+ bytes_written += scnprintf(capabilities + bytes_written,
+ sizeof(capabilities) - bytes_written, "%s%s",
+ bytes_written ? "," : "",
mpi3mr_capabilities[i].name);
- is_string_nonempty = true;
}
}
ioc_info(mrioc, "Protocol=(%s), Capabilities=(%s)\n",
- protocol, capabilities);
+ protocol, capabilities);
}
/**
scsi_dma_unmap(scmd);
scmd->result = DID_RESET << 16;
scsi_print_command(scmd);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
mrioc->flush_io_count++;
}
}
mpi3mr_clear_scmd_priv(mrioc, scmd);
scsi_dma_unmap(scmd);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
out:
if (sense_buf)
mpi3mr_repost_sense_buf(mrioc,
__func__);
scsi_print_command(scmd);
scmd->result = DID_OK << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return true;
}
scmd->result = SAM_STAT_CHECK_CONDITION;
scsi_build_sense_buffer(0, scmd->sense_buffer, ILLEGAL_REQUEST,
0x1A, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return true;
}
if (param_len != scsi_bufflen(scmd)) {
scmd->result = SAM_STAT_CHECK_CONDITION;
scsi_build_sense_buffer(0, scmd->sense_buffer, ILLEGAL_REQUEST,
0x1A, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return true;
}
buf = kzalloc(scsi_bufflen(scmd), GFP_ATOMIC);
scmd->result = SAM_STAT_CHECK_CONDITION;
scsi_build_sense_buffer(0, scmd->sense_buffer, ILLEGAL_REQUEST,
0x55, 0x03);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return true;
}
scsi_sg_copy_to_buffer(scmd, buf, scsi_bufflen(scmd));
scmd->result = SAM_STAT_CHECK_CONDITION;
scsi_build_sense_buffer(0, scmd->sense_buffer, ILLEGAL_REQUEST,
0x26, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
kfree(buf);
return true;
}
sdev_priv_data = scmd->device->hostdata;
if (!sdev_priv_data || !sdev_priv_data->tgt_priv_data) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
if (mrioc->stop_drv_processing &&
!(mpi3mr_allow_scmd_to_fw(scmd))) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
dev_handle = stgt_priv_data->dev_handle;
if (dev_handle == MPI3MR_INVALID_DEV_HANDLE) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
if (stgt_priv_data->dev_removed) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
if (atomic_read(&stgt_priv_data->block_io)) {
if (mrioc->stop_drv_processing) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
retval = SCSI_MLQUEUE_DEVICE_BUSY;
host_tag = mpi3mr_host_tag_for_scmd(mrioc, scmd);
if (host_tag == MPI3MR_HOSTTAG_INVALID) {
scmd->result = DID_ERROR << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
goto out;
}
struct SL_WH_MPI_TRIGGERS_T *mpi_tg, bool set);
/* ctl shared API */
-extern struct device_attribute *mpt3sas_host_attrs[];
-extern struct device_attribute *mpt3sas_dev_attrs[];
+extern const struct attribute_group *mpt3sas_host_groups[];
+extern const struct attribute_group *mpt3sas_dev_groups[];
void mpt3sas_ctl_init(ushort hbas_to_enumerate);
void mpt3sas_ctl_exit(ushort hbas_to_enumerate);
u8 mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index,
}
static DEVICE_ATTR_RW(enable_sdev_max_qd);
-struct device_attribute *mpt3sas_host_attrs[] = {
- &dev_attr_version_fw,
- &dev_attr_version_bios,
- &dev_attr_version_mpi,
- &dev_attr_version_product,
- &dev_attr_version_nvdata_persistent,
- &dev_attr_version_nvdata_default,
- &dev_attr_board_name,
- &dev_attr_board_assembly,
- &dev_attr_board_tracer,
- &dev_attr_io_delay,
- &dev_attr_device_delay,
- &dev_attr_logging_level,
- &dev_attr_fwfault_debug,
- &dev_attr_fw_queue_depth,
- &dev_attr_host_sas_address,
- &dev_attr_ioc_reset_count,
- &dev_attr_host_trace_buffer_size,
- &dev_attr_host_trace_buffer,
- &dev_attr_host_trace_buffer_enable,
- &dev_attr_reply_queue_count,
- &dev_attr_diag_trigger_master,
- &dev_attr_diag_trigger_event,
- &dev_attr_diag_trigger_scsi,
- &dev_attr_diag_trigger_mpi,
- &dev_attr_drv_support_bitmap,
- &dev_attr_BRM_status,
- &dev_attr_enable_sdev_max_qd,
+static struct attribute *mpt3sas_host_attrs[] = {
+ &dev_attr_version_fw.attr,
+ &dev_attr_version_bios.attr,
+ &dev_attr_version_mpi.attr,
+ &dev_attr_version_product.attr,
+ &dev_attr_version_nvdata_persistent.attr,
+ &dev_attr_version_nvdata_default.attr,
+ &dev_attr_board_name.attr,
+ &dev_attr_board_assembly.attr,
+ &dev_attr_board_tracer.attr,
+ &dev_attr_io_delay.attr,
+ &dev_attr_device_delay.attr,
+ &dev_attr_logging_level.attr,
+ &dev_attr_fwfault_debug.attr,
+ &dev_attr_fw_queue_depth.attr,
+ &dev_attr_host_sas_address.attr,
+ &dev_attr_ioc_reset_count.attr,
+ &dev_attr_host_trace_buffer_size.attr,
+ &dev_attr_host_trace_buffer.attr,
+ &dev_attr_host_trace_buffer_enable.attr,
+ &dev_attr_reply_queue_count.attr,
+ &dev_attr_diag_trigger_master.attr,
+ &dev_attr_diag_trigger_event.attr,
+ &dev_attr_diag_trigger_scsi.attr,
+ &dev_attr_diag_trigger_mpi.attr,
+ &dev_attr_drv_support_bitmap.attr,
+ &dev_attr_BRM_status.attr,
+ &dev_attr_enable_sdev_max_qd.attr,
NULL,
};
+static const struct attribute_group mpt3sas_host_attr_group = {
+ .attrs = mpt3sas_host_attrs
+};
+
+const struct attribute_group *mpt3sas_host_groups[] = {
+ &mpt3sas_host_attr_group,
+ NULL
+};
+
/* device attributes */
/**
}
static DEVICE_ATTR_RW(sas_ncq_prio_enable);
-struct device_attribute *mpt3sas_dev_attrs[] = {
- &dev_attr_sas_address,
- &dev_attr_sas_device_handle,
- &dev_attr_sas_ncq_prio_supported,
- &dev_attr_sas_ncq_prio_enable,
+static struct attribute *mpt3sas_dev_attrs[] = {
+ &dev_attr_sas_address.attr,
+ &dev_attr_sas_device_handle.attr,
+ &dev_attr_sas_ncq_prio_supported.attr,
+ &dev_attr_sas_ncq_prio_enable.attr,
NULL,
};
+static const struct attribute_group mpt3sas_dev_attr_group = {
+ .attrs = mpt3sas_dev_attrs
+};
+
+const struct attribute_group *mpt3sas_dev_groups[] = {
+ &mpt3sas_dev_attr_group,
+ NULL
+};
+
/* file operations table for mpt3ctl device */
static const struct file_operations ctl_fops = {
.owner = THIS_MODULE,
sdev_printk(KERN_INFO, scmd->device,
"device been deleted! scmd(0x%p)\n", scmd);
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
r = SUCCESS;
goto out;
}
sdev_printk(KERN_INFO, scmd->device,
"device been deleted! scmd(0x%p)\n", scmd);
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
r = SUCCESS;
goto out;
}
starget_printk(KERN_INFO, starget,
"target been deleted! scmd(0x%p)\n", scmd);
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
r = SUCCESS;
goto out;
}
scmd->result = DID_NO_CONNECT << 16;
else
scmd->result = DID_RESET << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
}
dtmprintk(ioc, ioc_info(ioc, "completing %d cmds\n", count));
}
sas_device_priv_data = scmd->device->hostdata;
if (!sas_device_priv_data || !sas_device_priv_data->sas_target) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
if (!(_scsih_allow_scmd_to_device(ioc, scmd))) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
handle = sas_target_priv_data->handle;
if (handle == MPT3SAS_INVALID_DEVICE_HANDLE) {
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
} else if (sas_target_priv_data->deleted) {
/* device has been deleted */
scmd->result = DID_NO_CONNECT << 16;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
} else if (sas_target_priv_data->tm_busy ||
sas_device_priv_data->block) {
scsi_dma_unmap(scmd);
mpt3sas_base_free_smid(ioc, smid);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
.sg_tablesize = MPT2SAS_SG_DEPTH,
.max_sectors = 32767,
.cmd_per_lun = 7,
- .shost_attrs = mpt3sas_host_attrs,
- .sdev_attrs = mpt3sas_dev_attrs,
+ .shost_groups = mpt3sas_host_groups,
+ .sdev_groups = mpt3sas_dev_groups,
.track_queue_depth = 1,
.cmd_size = sizeof(struct scsiio_tracker),
};
.max_sectors = 32767,
.max_segment_size = 0xffffffff,
.cmd_per_lun = 7,
- .shost_attrs = mpt3sas_host_attrs,
- .sdev_attrs = mpt3sas_dev_attrs,
+ .shost_groups = mpt3sas_host_groups,
+ .sdev_groups = mpt3sas_dev_groups,
.track_queue_depth = 1,
.cmd_size = sizeof(struct scsiio_tracker),
.map_queues = scsih_map_queues,
[chip_1320] = { 2, 4, 0x800, 17, 64, 8, 9, &mvs_94xx_dispatch, },
};
-static struct device_attribute *mvst_host_attrs[];
+static const struct attribute_group *mvst_host_groups[];
#define SOC_SAS_NUM 2
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = mvst_host_attrs,
+ .shost_groups = mvst_host_groups,
.track_queue_depth = 1,
};
sas_release_transport(mvs_stt);
}
-static struct device_attribute *mvst_host_attrs[] = {
- &dev_attr_driver_version,
- &dev_attr_interrupt_coalescing,
+static struct attribute *mvst_host_attrs[] = {
+ &dev_attr_driver_version.attr,
+ &dev_attr_interrupt_coalescing.attr,
NULL,
};
+ATTRIBUTE_GROUPS(mvst_host);
+
module_init(mvs_init);
module_exit(mvs_exit);
dma_unmap_sg(&mhba->pdev->dev, scsi_sglist(scmd),
scsi_sg_count(scmd),
scmd->sc_data_direction);
- cmd->scmd->scsi_done(scmd);
+ scsi_done(scmd);
mvumi_return_cmd(mhba, cmd);
}
out_return_cmd:
mvumi_return_cmd(mhba, cmd);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
spin_unlock_irqrestore(shost->host_lock, irq_flags);
return 0;
}
if (nsge > 1) {
dma_pool_free(cb->dcdb_pool, dcdb, dcdb_addr);
scmd->result = (DID_ERROR << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
dev_dbg(&shost->shost_gendev, "ldev %u in state %x, skip\n",
sdev->id, ldev_info ? ldev_info->state : 0xff);
scmd->result = (DID_BAD_TARGET << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
switch (scmd->cmnd[0]) {
case TEST_UNIT_READY:
scmd->result = (DID_OK << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case INQUIRY:
if (scmd->cmnd[1] & 1) {
myrb_inquiry(cb, scmd);
scmd->result = (DID_OK << 16);
}
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case SYNCHRONIZE_CACHE:
scmd->result = (DID_OK << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case MODE_SENSE:
if ((scmd->cmnd[2] & 0x3F) != 0x3F &&
myrb_mode_sense(cb, scmd, ldev_info);
scmd->result = (DID_OK << 16);
}
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case READ_CAPACITY:
if ((scmd->cmnd[1] & 1) ||
(scmd->cmnd[8] & 1)) {
/* Illegal request, invalid field in CDB */
scsi_build_sense(scmd, 0, ILLEGAL_REQUEST, 0x24, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
lba = get_unaligned_be32(&scmd->cmnd[2]);
if (lba) {
/* Illegal request, invalid field in CDB */
scsi_build_sense(scmd, 0, ILLEGAL_REQUEST, 0x24, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
myrb_read_capacity(cb, scmd, ldev_info);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case REQUEST_SENSE:
myrb_request_sense(cb, scmd);
/* Assume good status */
scmd->result = (DID_OK << 16);
}
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case READ_6:
if (ldev_info->state == MYRB_DEVICE_WO) {
/* Data protect, attempt to read invalid data */
scsi_build_sense(scmd, 0, DATA_PROTECT, 0x21, 0x06);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
fallthrough;
if (ldev_info->state == MYRB_DEVICE_WO) {
/* Data protect, attempt to read invalid data */
scsi_build_sense(scmd, 0, DATA_PROTECT, 0x21, 0x06);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
fallthrough;
if (ldev_info->state == MYRB_DEVICE_WO) {
/* Data protect, attempt to read invalid data */
scsi_build_sense(scmd, 0, DATA_PROTECT, 0x21, 0x06);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
fallthrough;
default:
/* Illegal request, invalid opcode */
scsi_build_sense(scmd, 0, ILLEGAL_REQUEST, 0x20, 0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
if (sdev->channel > myrb_logical_channel(shost)) {
scmd->result = (DID_BAD_TARGET << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
if (sdev->channel == myrb_logical_channel(shost))
}
static DEVICE_ATTR_WO(flush_cache);
-static struct device_attribute *myrb_sdev_attrs[] = {
- &dev_attr_rebuild,
- &dev_attr_consistency_check,
- &dev_attr_raid_state,
- &dev_attr_raid_level,
+static struct attribute *myrb_sdev_attrs[] = {
+ &dev_attr_rebuild.attr,
+ &dev_attr_consistency_check.attr,
+ &dev_attr_raid_state.attr,
+ &dev_attr_raid_level.attr,
NULL,
};
-static struct device_attribute *myrb_shost_attrs[] = {
- &dev_attr_ctlr_num,
- &dev_attr_model,
- &dev_attr_firmware,
- &dev_attr_flush_cache,
+ATTRIBUTE_GROUPS(myrb_sdev);
+
+static struct attribute *myrb_shost_attrs[] = {
+ &dev_attr_ctlr_num.attr,
+ &dev_attr_model.attr,
+ &dev_attr_firmware.attr,
+ &dev_attr_flush_cache.attr,
NULL,
};
+ATTRIBUTE_GROUPS(myrb_shost);
+
static struct scsi_host_template myrb_template = {
.module = THIS_MODULE,
.name = "DAC960",
.slave_destroy = myrb_slave_destroy,
.bios_param = myrb_biosparam,
.cmd_size = sizeof(struct myrb_cmdblk),
- .shost_attrs = myrb_shost_attrs,
- .sdev_attrs = myrb_sdev_attrs,
+ .shost_groups = myrb_shost_groups,
+ .sdev_groups = myrb_sdev_groups,
.this_id = -1,
};
scmd->result = (DID_ERROR << 16);
break;
}
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
}
static void myrb_handle_cmdblk(struct myrb_hba *cb, struct myrb_cmdblk *cmd_blk)
}
static DEVICE_ATTR_RW(consistency_check);
-static struct device_attribute *myrs_sdev_attrs[] = {
- &dev_attr_consistency_check,
- &dev_attr_rebuild,
- &dev_attr_raid_state,
- &dev_attr_raid_level,
+static struct attribute *myrs_sdev_attrs[] = {
+ &dev_attr_consistency_check.attr,
+ &dev_attr_rebuild.attr,
+ &dev_attr_raid_state.attr,
+ &dev_attr_raid_level.attr,
NULL,
};
+ATTRIBUTE_GROUPS(myrs_sdev);
+
static ssize_t serial_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
}
static DEVICE_ATTR_RW(disable_enclosure_messages);
-static struct device_attribute *myrs_shost_attrs[] = {
- &dev_attr_serial,
- &dev_attr_ctlr_num,
- &dev_attr_processor,
- &dev_attr_model,
- &dev_attr_ctlr_type,
- &dev_attr_cache_size,
- &dev_attr_firmware,
- &dev_attr_discovery,
- &dev_attr_flush_cache,
- &dev_attr_disable_enclosure_messages,
+static struct attribute *myrs_shost_attrs[] = {
+ &dev_attr_serial.attr,
+ &dev_attr_ctlr_num.attr,
+ &dev_attr_processor.attr,
+ &dev_attr_model.attr,
+ &dev_attr_ctlr_type.attr,
+ &dev_attr_cache_size.attr,
+ &dev_attr_firmware.attr,
+ &dev_attr_discovery.attr,
+ &dev_attr_flush_cache.attr,
+ &dev_attr_disable_enclosure_messages.attr,
NULL,
};
+ATTRIBUTE_GROUPS(myrs_shost);
+
/*
* SCSI midlayer interface
*/
if (!scmd->device->hostdata) {
scmd->result = (DID_NO_CONNECT << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
switch (scmd->cmnd[0]) {
case REPORT_LUNS:
scsi_build_sense(scmd, 0, ILLEGAL_REQUEST, 0x20, 0x0);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
case MODE_SENSE:
if (scmd->device->channel >= cs->ctlr_info->physchan_present) {
myrs_mode_sense(cs, scmd, ldev_info);
scmd->result = (DID_OK << 16);
}
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
break;
if (WARN_ON(!hw_sgl)) {
scsi_dma_unmap(scmd);
scmd->result = (DID_ERROR << 16);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
return 0;
}
hw_sgl->sge_addr = (u64)sg_dma_address(sgl);
.slave_configure = myrs_slave_configure,
.slave_destroy = myrs_slave_destroy,
.cmd_size = sizeof(struct myrs_cmdblk),
- .shost_attrs = myrs_shost_attrs,
- .sdev_attrs = myrs_sdev_attrs,
+ .shost_groups = myrs_shost_groups,
+ .sdev_groups = myrs_sdev_groups,
.this_id = -1,
};
scmd->result = (DID_BAD_TARGET << 16);
else
scmd->result = (DID_OK << 16) | status;
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
}
static void myrs_handle_cmdblk(struct myrs_hba *cs, struct myrs_cmdblk *cmd_blk)
while (lcmd) {
cmd = lcmd;
lcmd = (struct scsi_cmnd *) cmd->host_scribble;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
return 0;
}
-static int ncr53c8xx_queue_command_lck (struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int ncr53c8xx_queue_command_lck(struct scsi_cmnd *cmd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct ncb *np = ((struct host_data *) cmd->device->host->hostdata)->ncb;
unsigned long flags;
int sts;
printk("ncr53c8xx_queue_command\n");
#endif
- cmd->scsi_done = done;
cmd->host_scribble = NULL;
cmd->__data_mapped = 0;
cmd->__data_mapping = 0;
.show = show_ncr53c8xx_revision,
};
-static struct device_attribute *ncr53c8xx_host_attrs[] = {
- &ncr53c8xx_revision_attr,
+static struct attribute *ncr53c8xx_host_attrs[] = {
+ &ncr53c8xx_revision_attr.attr,
NULL
};
+ATTRIBUTE_GROUPS(ncr53c8xx_host);
+
/*==========================================================
**
** Boot command line.
if (!tpnt->name)
tpnt->name = SCSI_NCR_DRIVER_NAME;
- if (!tpnt->shost_attrs)
- tpnt->shost_attrs = ncr53c8xx_host_attrs;
+ if (!tpnt->shost_groups)
+ tpnt->shost_groups = ncr53c8xx_host_groups;
tpnt->queuecommand = ncr53c8xx_queue_command;
tpnt->slave_configure = ncr53c8xx_slave_configure;
return TRUE;
}
-static int nsp32_queuecommand_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int nsp32_queuecommand_lck(struct scsi_cmnd *SCpnt)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
nsp32_hw_data *data = (nsp32_hw_data *)SCpnt->device->host->hostdata;
nsp32_target *target;
nsp32_lunt *cur_lunt;
show_command(SCpnt);
- SCpnt->scsi_done = done;
data->CurrentSC = SCpnt;
SCpnt->SCp.Status = SAM_STAT_CHECK_CONDITION;
scsi_set_resid(SCpnt, scsi_bufflen(SCpnt));
/*
* call scsi_done
*/
- (*SCpnt->scsi_done)(SCpnt);
+ scsi_done(SCpnt);
/*
* reset parameters
data->CurrentSC = NULL;
- SCpnt->scsi_done(SCpnt);
+ scsi_done(SCpnt);
}
-static int nsp_queuecommand_lck(struct scsi_cmnd *SCpnt,
- void (*done)(struct scsi_cmnd *))
+static int nsp_queuecommand_lck(struct scsi_cmnd *SCpnt)
{
#ifdef NSP_DEBUG
/*unsigned int host_id = SCpnt->device->host->this_id;*/
scsi_bufflen(SCpnt), scsi_sg_count(SCpnt));
//nsp_dbg(NSP_DEBUG_QUEUECOMMAND, "before CurrentSC=0x%p", data->CurrentSC);
- SCpnt->scsi_done = done;
-
if (data->CurrentSC != NULL) {
nsp_msg(KERN_DEBUG, "CurrentSC!=NULL this can't be happen");
SCpnt->result = DID_BAD_TARGET << 16;
idle_out:
curSC->SCp.phase = idle;
- curSC->scsi_done(curSC);
+ scsi_done(curSC);
goto out;
}
return (info_msg);
}
-static int
-SYM53C500_queue_lck(struct scsi_cmnd *SCpnt, void (*done)(struct scsi_cmnd *))
+static int SYM53C500_queue_lck(struct scsi_cmnd *SCpnt)
{
int i;
int port_base = SCpnt->device->host->io_port;
VDEB(printk("\n"));
data->current_SC = SCpnt;
- data->current_SC->scsi_done = done;
data->current_SC->SCp.phase = command_ph;
data->current_SC->SCp.Status = 0;
data->current_SC->SCp.Message = 0;
.store = SYM53C500_store_pio,
};
-static struct device_attribute *SYM53C500_shost_attrs[] = {
- &SYM53C500_pio_attr,
+static struct attribute *SYM53C500_shost_attrs[] = {
+ &SYM53C500_pio_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(SYM53C500_shost);
+
/*
* scsi_host_template initializer
*/
.can_queue = 1,
.this_id = 7,
.sg_tablesize = 32,
- .shost_attrs = SYM53C500_shost_attrs
+ .shost_groups = SYM53C500_shost_groups
};
static int SYM53C500_config_check(struct pcmcia_device *p_dev, void *priv_data)
char *str = buf;
int start = 0;
u32 ib_offset = pm8001_ha->ib_offset;
+ u32 queue_size = pm8001_ha->max_q_num * PM8001_MPI_QUEUE * 128;
#define IB_MEMMAP(c) \
(*(u32 *)((u8 *)pm8001_ha-> \
memoryMap.region[ib_offset].virt_ptr + \
start = start + 4;
}
pm8001_ha->evtlog_ib_offset += SYSFS_OFFSET;
- if (((pm8001_ha->evtlog_ib_offset) % (PM80XX_IB_OB_QUEUE_SIZE)) == 0)
+ if (((pm8001_ha->evtlog_ib_offset) % queue_size) == 0)
pm8001_ha->evtlog_ib_offset = 0;
return str - buf;
char *str = buf;
int start = 0;
u32 ob_offset = pm8001_ha->ob_offset;
+ u32 queue_size = pm8001_ha->max_q_num * PM8001_MPI_QUEUE * 128;
#define OB_MEMMAP(c) \
(*(u32 *)((u8 *)pm8001_ha-> \
memoryMap.region[ob_offset].virt_ptr + \
start = start + 4;
}
pm8001_ha->evtlog_ob_offset += SYSFS_OFFSET;
- if (((pm8001_ha->evtlog_ob_offset) % (PM80XX_IB_OB_QUEUE_SIZE)) == 0)
+ if (((pm8001_ha->evtlog_ob_offset) % queue_size) == 0)
pm8001_ha->evtlog_ob_offset = 0;
return str - buf;
}
static DEVICE_ATTR_RO(ctl_iop1_count);
-struct device_attribute *pm8001_host_attrs[] = {
- &dev_attr_interface_rev,
- &dev_attr_controller_fatal_error,
- &dev_attr_fw_version,
- &dev_attr_update_fw,
- &dev_attr_aap_log,
- &dev_attr_iop_log,
- &dev_attr_fatal_log,
- &dev_attr_non_fatal_log,
- &dev_attr_non_fatal_count,
- &dev_attr_gsm_log,
- &dev_attr_max_out_io,
- &dev_attr_max_devices,
- &dev_attr_max_sg_list,
- &dev_attr_sas_spec_support,
- &dev_attr_logging_level,
- &dev_attr_event_log_size,
- &dev_attr_host_sas_address,
- &dev_attr_bios_version,
- &dev_attr_ib_log,
- &dev_attr_ob_log,
- &dev_attr_ila_version,
- &dev_attr_inc_fw_ver,
- &dev_attr_ctl_mpi_state,
- &dev_attr_ctl_hmi_error,
- &dev_attr_ctl_raae_count,
- &dev_attr_ctl_iop0_count,
- &dev_attr_ctl_iop1_count,
+static struct attribute *pm8001_host_attrs[] = {
+ &dev_attr_interface_rev.attr,
+ &dev_attr_controller_fatal_error.attr,
+ &dev_attr_fw_version.attr,
+ &dev_attr_update_fw.attr,
+ &dev_attr_aap_log.attr,
+ &dev_attr_iop_log.attr,
+ &dev_attr_fatal_log.attr,
+ &dev_attr_non_fatal_log.attr,
+ &dev_attr_non_fatal_count.attr,
+ &dev_attr_gsm_log.attr,
+ &dev_attr_max_out_io.attr,
+ &dev_attr_max_devices.attr,
+ &dev_attr_max_sg_list.attr,
+ &dev_attr_sas_spec_support.attr,
+ &dev_attr_logging_level.attr,
+ &dev_attr_event_log_size.attr,
+ &dev_attr_host_sas_address.attr,
+ &dev_attr_bios_version.attr,
+ &dev_attr_ib_log.attr,
+ &dev_attr_ob_log.attr,
+ &dev_attr_ila_version.attr,
+ &dev_attr_inc_fw_ver.attr,
+ &dev_attr_ctl_mpi_state.attr,
+ &dev_attr_ctl_hmi_error.attr,
+ &dev_attr_ctl_raae_count.attr,
+ &dev_attr_ctl_iop0_count.attr,
+ &dev_attr_ctl_iop1_count.attr,
NULL,
};
+static const struct attribute_group pm8001_host_attr_group = {
+ .attrs = pm8001_host_attrs
+};
+
+const struct attribute_group *pm8001_host_groups[] = {
+ &pm8001_host_attr_group,
+ NULL
+};
* fw_control_context->usrAddr
*/
complete(pm8001_ha->nvmd_completion);
- pm8001_dbg(pm8001_ha, MSG, "Set nvm data complete!\n");
+ pm8001_dbg(pm8001_ha, MSG, "Get nvmd data complete!\n");
ccb->task = NULL;
ccb->ccb_tag = 0xFFFFFFFF;
pm8001_tag_free(pm8001_ha, tag);
struct pm8001_phy *phy = &pm8001_ha->phy[phy_id];
unsigned long flags;
u8 deviceType = pPayload->sas_identify.dev_type;
+ phy->port = port;
+ port->port_id = port_id;
port->port_state = portstate;
phy->phy_state = PHY_STATE_LINK_UP_SPC;
pm8001_dbg(pm8001_ha, MSG,
unsigned long flags;
pm8001_dbg(pm8001_ha, DEVIO, "HW_EVENT_SATA_PHY_UP port id = %d, phy id = %d\n",
port_id, phy_id);
+ phy->port = port;
+ port->port_id = port_id;
port->port_state = portstate;
phy->phy_state = PHY_STATE_LINK_UP_SPC;
port->port_attached = 1;
u16 ITNT = 2000;
struct domain_device *dev = pm8001_dev->sas_device;
struct domain_device *parent_dev = dev->parent;
+ struct pm8001_port *port = dev->port->lldd_port;
circularQ = &pm8001_ha->inbnd_q_tbl[0];
memset(&payload, 0, sizeof(payload));
if (pm8001_dev->dev_type == SAS_SATA_DEV)
stp_sspsmp_sata = 0x00; /* stp*/
else if (pm8001_dev->dev_type == SAS_END_DEVICE ||
- pm8001_dev->dev_type == SAS_EDGE_EXPANDER_DEVICE ||
- pm8001_dev->dev_type == SAS_FANOUT_EXPANDER_DEVICE)
+ dev_is_expander(pm8001_dev->dev_type))
stp_sspsmp_sata = 0x01; /*ssp or smp*/
}
if (parent_dev && dev_is_expander(parent_dev->dev_type))
linkrate = (pm8001_dev->sas_device->linkrate < dev->port->linkrate) ?
pm8001_dev->sas_device->linkrate : dev->port->linkrate;
payload.phyid_portid =
- cpu_to_le32(((pm8001_dev->sas_device->port->id) & 0x0F) |
+ cpu_to_le32(((port->port_id) & 0x0F) |
((phy_id & 0x0F) << 4));
payload.dtype_dlr_retry = cpu_to_le32((retryFlag & 0x01) |
((linkrate & 0x0F) * 0x1000000) |
#ifdef CONFIG_COMPAT
.compat_ioctl = sas_ioctl,
#endif
- .shost_attrs = pm8001_host_attrs,
+ .shost_groups = pm8001_host_groups,
.track_queue_depth = 1,
};
.lldd_I_T_nexus_reset = pm8001_I_T_nexus_reset,
.lldd_lu_reset = pm8001_lu_reset,
.lldd_query_task = pm8001_query_task,
+ .lldd_port_formed = pm8001_port_formed,
};
/**
goto err_out;
/* Memory region for ccb_info*/
+ pm8001_ha->ccb_count = ccb_count;
pm8001_ha->ccb_info =
kcalloc(ccb_count, sizeof(struct pm8001_ccb_info), GFP_KERNEL);
if (!pm8001_ha->ccb_info) {
tasklet_kill(&pm8001_ha->tasklet[j]);
#endif
scsi_host_put(pm8001_ha->shost);
+
+ for (i = 0; i < pm8001_ha->ccb_count; i++) {
+ dma_free_coherent(&pm8001_ha->pdev->dev,
+ sizeof(struct pm8001_prd) * PM8001_MAX_DMA_SG,
+ pm8001_ha->ccb_info[i].buf_prd,
+ pm8001_ha->ccb_info[i].ccb_dma_handle);
+ }
+ kfree(pm8001_ha->ccb_info);
+ kfree(pm8001_ha->devices);
+
pm8001_free(pm8001_ha);
kfree(sha->sas_phy);
kfree(sha->sas_port);
tmf_task.tmf = TMF_CLEAR_TASK_SET;
return pm8001_issue_ssp_tmf(dev, lun, &tmf_task);
}
+
+void pm8001_port_formed(struct asd_sas_phy *sas_phy)
+{
+ struct sas_ha_struct *sas_ha = sas_phy->ha;
+ struct pm8001_hba_info *pm8001_ha = sas_ha->lldd_ha;
+ struct pm8001_phy *phy = sas_phy->lldd_phy;
+ struct asd_sas_port *sas_port = sas_phy->port;
+ struct pm8001_port *port = phy->port;
+
+ if (!sas_port) {
+ pm8001_dbg(pm8001_ha, FAIL, "Received null port\n");
+ return;
+ }
+ sas_port->lldd_port = port;
+}
u8 port_attached;
u16 wide_port_phymap;
u8 port_state;
+ u8 port_id;
struct list_head list;
};
__le32 producer_index;
u32 consumer_idx;
spinlock_t oq_lock;
+ unsigned long lock_flags;
};
struct pm8001_hba_memspace {
void __iomem *memvirtaddr;
u32 iomb_size; /* SPC and SPCV IOMB size */
struct pm8001_device *devices;
struct pm8001_ccb_info *ccb_info;
+ u32 ccb_count;
#ifdef PM8001_USE_MSIX
int number_of_intr;/*will be used in remove()*/
char intr_drvname[PM8001_MAX_MSIX_VEC]
int pm8001_I_T_nexus_reset(struct domain_device *dev);
int pm8001_I_T_nexus_event_handler(struct domain_device *dev);
int pm8001_query_task(struct sas_task *task);
+void pm8001_port_formed(struct asd_sas_phy *sas_phy);
void pm8001_open_reject_retry(
struct pm8001_hba_info *pm8001_ha,
struct sas_task *task_to_close,
int pm80xx_fatal_errors(struct pm8001_hba_info *pm8001_ha);
void pm8001_free_dev(struct pm8001_device *pm8001_dev);
/* ctl shared API */
-extern struct device_attribute *pm8001_host_attrs[];
+extern const struct attribute_group *pm8001_host_groups[];
static inline void
pm8001_ccb_task_free_done(struct pm8001_hba_info *pm8001_ha,
{
pm8001_ccb_task_free(pm8001_ha, task, ccb, ccb_idx);
smp_mb(); /*in order to force CPU ordering*/
- spin_unlock(&pm8001_ha->lock);
task->task_done(task);
- spin_lock(&pm8001_ha->lock);
}
#endif
/*See the comments for mpi_ssp_completion */
static void
-mpi_sata_completion(struct pm8001_hba_info *pm8001_ha, void *piomb)
+mpi_sata_completion(struct pm8001_hba_info *pm8001_ha,
+ struct outbound_queue_table *circularQ, void *piomb)
{
struct sas_task *t;
struct pm8001_ccb_info *ccb;
IO_OPEN_CNX_ERROR_IT_NEXUS_LOSS);
ts->resp = SAS_TASK_UNDELIVERED;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
IO_OPEN_CNX_ERROR_IT_NEXUS_LOSS);
ts->resp = SAS_TASK_UNDELIVERED;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
IO_OPEN_CNX_ERROR_STP_RESOURCES_BUSY);
ts->resp = SAS_TASK_UNDELIVERED;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
IO_DS_NON_OPERATIONAL);
ts->resp = SAS_TASK_UNDELIVERED;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
IO_DS_IN_ERROR);
ts->resp = SAS_TASK_UNDELIVERED;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
pm8001_ccb_task_free(pm8001_ha, t, ccb, tag);
} else {
spin_unlock_irqrestore(&t->task_state_lock, flags);
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
}
}
/*See the comments for mpi_ssp_completion */
-static void mpi_sata_event(struct pm8001_hba_info *pm8001_ha, void *piomb)
+static void mpi_sata_event(struct pm8001_hba_info *pm8001_ha,
+ struct outbound_queue_table *circularQ, void *piomb)
{
struct sas_task *t;
struct task_status_struct *ts;
IO_OPEN_CNX_ERROR_IT_NEXUS_LOSS);
ts->resp = SAS_TASK_COMPLETE;
ts->stat = SAS_QUEUE_FULL;
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
return;
}
break;
pm8001_ccb_task_free(pm8001_ha, t, ccb, tag);
} else {
spin_unlock_irqrestore(&t->task_state_lock, flags);
+ spin_unlock_irqrestore(&circularQ->oq_lock,
+ circularQ->lock_flags);
pm8001_ccb_task_free_done(pm8001_ha, t, ccb, tag);
+ spin_lock_irqsave(&circularQ->oq_lock,
+ circularQ->lock_flags);
}
}
struct pm8001_phy *phy = &pm8001_ha->phy[phy_id];
unsigned long flags;
u8 deviceType = pPayload->sas_identify.dev_type;
+ phy->port = port;
+ port->port_id = port_id;
port->port_state = portstate;
port->wide_port_phymap |= (1U << phy_id);
phy->phy_state = PHY_STATE_LINK_UP_SPCV;
"port id %d, phy id %d link_rate %d portstate 0x%x\n",
port_id, phy_id, link_rate, portstate);
+ phy->port = port;
+ port->port_id = port_id;
port->port_state = portstate;
phy->phy_state = PHY_STATE_LINK_UP_SPCV;
port->port_attached = 1;
* @pm8001_ha: our hba card information
* @piomb: IO message buffer
*/
-static void process_one_iomb(struct pm8001_hba_info *pm8001_ha, void *piomb)
+static void process_one_iomb(struct pm8001_hba_info *pm8001_ha,
+ struct outbound_queue_table *circularQ, void *piomb)
{
__le32 pHeader = *(__le32 *)piomb;
u32 opc = (u32)((le32_to_cpu(pHeader)) & 0xFFF);
break;
case OPC_OUB_SATA_COMP:
pm8001_dbg(pm8001_ha, MSG, "OPC_OUB_SATA_COMP\n");
- mpi_sata_completion(pm8001_ha, piomb);
+ mpi_sata_completion(pm8001_ha, circularQ, piomb);
break;
case OPC_OUB_SATA_EVENT:
pm8001_dbg(pm8001_ha, MSG, "OPC_OUB_SATA_EVENT\n");
- mpi_sata_event(pm8001_ha, piomb);
+ mpi_sata_event(pm8001_ha, circularQ, piomb);
break;
case OPC_OUB_SSP_EVENT:
pm8001_dbg(pm8001_ha, MSG, "OPC_OUB_SSP_EVENT\n");
void *pMsg1 = NULL;
u8 bc;
u32 ret = MPI_IO_STATUS_FAIL;
- unsigned long flags;
u32 regval;
if (vec == (pm8001_ha->max_q_num - 1)) {
}
}
circularQ = &pm8001_ha->outbnd_q_tbl[vec];
- spin_lock_irqsave(&circularQ->oq_lock, flags);
+ spin_lock_irqsave(&circularQ->oq_lock, circularQ->lock_flags);
do {
/* spurious interrupt during setup if kexec-ing and
* driver doing a doorbell access w/ the pre-kexec oq
ret = pm8001_mpi_msg_consume(pm8001_ha, circularQ, &pMsg1, &bc);
if (MPI_IO_STATUS_SUCCESS == ret) {
/* process the outbound message */
- process_one_iomb(pm8001_ha, (void *)(pMsg1 - 4));
+ process_one_iomb(pm8001_ha, circularQ,
+ (void *)(pMsg1 - 4));
/* free the message from the outbound circular buffer */
pm8001_mpi_msg_free_set(pm8001_ha, pMsg1,
circularQ, bc);
break;
}
} while (1);
- spin_unlock_irqrestore(&circularQ->oq_lock, flags);
+ spin_unlock_irqrestore(&circularQ->oq_lock, circularQ->lock_flags);
return ret;
}
u16 ITNT = 2000;
struct domain_device *dev = pm8001_dev->sas_device;
struct domain_device *parent_dev = dev->parent;
+ struct pm8001_port *port = dev->port->lldd_port;
circularQ = &pm8001_ha->inbnd_q_tbl[0];
memset(&payload, 0, sizeof(payload));
if (pm8001_dev->dev_type == SAS_SATA_DEV)
stp_sspsmp_sata = 0x00; /* stp*/
else if (pm8001_dev->dev_type == SAS_END_DEVICE ||
- pm8001_dev->dev_type == SAS_EDGE_EXPANDER_DEVICE ||
- pm8001_dev->dev_type == SAS_FANOUT_EXPANDER_DEVICE)
+ dev_is_expander(pm8001_dev->dev_type))
stp_sspsmp_sata = 0x01; /*ssp or smp*/
}
if (parent_dev && dev_is_expander(parent_dev->dev_type))
pm8001_dev->sas_device->linkrate : dev->port->linkrate;
payload.phyid_portid =
- cpu_to_le32(((pm8001_dev->sas_device->port->id) & 0xFF) |
+ cpu_to_le32(((port->port_id) & 0xFF) |
((phy_id & 0xFF) << 8));
payload.dtype_dlr_mcn_ir_retry = cpu_to_le32((retryFlag & 0x01) |
scsi_dma_unmap(scsi_cmd);
pmcraid_return_cmd(cmd);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
}
/**
le32_to_cpu(resp) >> 2,
cmd->ioa_cb->ioarcb.cdb[0],
scsi_cmd->result);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
} else if (cmd->cmd_done == pmcraid_internal_done ||
cmd->cmd_done == pmcraid_erp_done) {
cmd->cmd_done(cmd);
if (rc == 0) {
scsi_dma_unmap(scsi_cmd);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
}
return rc;
* SCSI_MLQUEUE_DEVICE_BUSY if device is busy
* SCSI_MLQUEUE_HOST_BUSY if host is busy
*/
-static int pmcraid_queuecommand_lck(
- struct scsi_cmnd *scsi_cmd,
- void (*done) (struct scsi_cmnd *)
-)
+static int pmcraid_queuecommand_lck(struct scsi_cmnd *scsi_cmd)
{
struct pmcraid_instance *pinstance;
struct pmcraid_resource_entry *res;
pinstance =
(struct pmcraid_instance *)scsi_cmd->device->host->hostdata;
fw_version = be16_to_cpu(pinstance->inq_data->fw_version);
- scsi_cmd->scsi_done = done;
res = scsi_cmd->device->hostdata;
scsi_cmd->result = (DID_OK << 16);
if (pinstance->ioa_state == IOA_STATE_DEAD) {
pmcraid_info("IOA is dead, but queuecommand is scheduled\n");
scsi_cmd->result = (DID_NO_CONNECT << 16);
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
return 0;
}
*/
if (scsi_cmd->cmnd[0] == SYNCHRONIZE_CACHE) {
pmcraid_info("SYNC_CACHE(0x35), completing in driver itself\n");
- scsi_cmd->scsi_done(scsi_cmd);
+ scsi_done(scsi_cmd);
return 0;
}
.show = pmcraid_show_adapter_id,
};
-static struct device_attribute *pmcraid_host_attrs[] = {
- &pmcraid_log_level_attr,
- &pmcraid_driver_version_attr,
- &pmcraid_adapter_id_attr,
+static struct attribute *pmcraid_host_attrs[] = {
+ &pmcraid_log_level_attr.attr,
+ &pmcraid_driver_version_attr.attr,
+ &pmcraid_adapter_id_attr.attr,
NULL,
};
+ATTRIBUTE_GROUPS(pmcraid_host);
/* host template structure for pmcraid driver */
static struct scsi_host_template pmcraid_host_template = {
.max_sectors = PMCRAID_IOA_MAX_SECTORS,
.no_write_same = 1,
.cmd_per_lun = PMCRAID_MAX_CMD_PER_LUN,
- .shost_attrs = pmcraid_host_attrs,
+ .shost_groups = pmcraid_host_groups,
.proc_name = PMCRAID_DRIVER_NAME,
};
dev->cur_cmd = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
static int ppa_engine(ppa_struct *dev, struct scsi_cmnd *cmd)
return 0;
}
-static int ppa_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done) (struct scsi_cmnd *))
+static int ppa_queuecommand_lck(struct scsi_cmnd *cmd)
{
ppa_struct *dev = ppa_dev(cmd->device->host);
dev->failed = 0;
dev->jstart = jiffies;
dev->cur_cmd = cmd;
- cmd->scsi_done = done;
cmd->result = DID_ERROR << 16; /* default return code */
cmd->SCp.phase = 0; /* bus free */
return 0;
}
-static int ps3rom_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done)(struct scsi_cmnd *))
+static int ps3rom_queuecommand_lck(struct scsi_cmnd *cmd)
{
struct ps3rom_private *priv = shost_priv(cmd->device->host);
struct ps3_storage_device *dev = priv->dev;
int res;
priv->curr_cmd = cmd;
- cmd->scsi_done = done;
opcode = cmd->cmnd[0];
/*
scsi_build_sense(cmd, 0, ILLEGAL_REQUEST, 0, 0);
cmd->result = res;
priv->curr_cmd = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
return 0;
done:
priv->curr_cmd = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return IRQ_HANDLED;
}
extern struct qedf_ioreq *qedf_alloc_cmd(struct qedf_rport *fcport,
u8 cmd_type);
-extern struct device_attribute *qedf_host_attrs[];
+extern const struct attribute_group *qedf_host_groups[];
extern void qedf_cmd_timer_set(struct qedf_ctx *qedf, struct qedf_ioreq *io_req,
unsigned int timer_msec);
extern int qedf_init_mp_req(struct qedf_ioreq *io_req);
static DEVICE_ATTR_RO(fcoe_mac);
static DEVICE_ATTR_RO(fka_period);
-struct device_attribute *qedf_host_attrs[] = {
- &dev_attr_fcoe_mac,
- &dev_attr_fka_period,
+static struct attribute *qedf_host_attrs[] = {
+ &dev_attr_fcoe_mac.attr,
+ &dev_attr_fka_period.attr,
NULL,
};
+static const struct attribute_group qedf_host_attr_group = {
+ .attrs = qedf_host_attrs
+};
+
+const struct attribute_group *qedf_host_groups[] = {
+ &qedf_host_attr_group,
+ NULL
+};
+
extern const struct qed_fcoe_ops *qed_ops;
void qedf_capture_grc_dump(struct qedf_ctx *qedf)
"Number of SG elements %d exceeds what hardware limitation of %d.\n",
num_sgs, QEDF_MAX_BDS_PER_CMD);
sc_cmd->result = DID_ERROR;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
"Returning DNC as unloading or stop io, flags 0x%lx.\n",
qedf->flags);
sc_cmd->result = DID_NO_CONNECT << 16;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
"Completing sc_cmd=%p DID_NO_CONNECT as MSI-X is not enabled.\n",
sc_cmd);
sc_cmd->result = DID_NO_CONNECT << 16;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
"fc_remote_port_chkready failed=0x%x for port_id=0x%06x.\n",
rval, rport->port_id);
sc_cmd->result = rval;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
return 0;
}
io_req->sc_cmd = NULL;
sc_cmd->SCp.ptr = NULL;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
kref_put(&io_req->refcount, qedf_release_cmd);
}
goto bad_scsi_ptr;
}
- if (!sc_cmd->scsi_done) {
- QEDF_ERR(&qedf->dbg_ctx,
- "sc_cmd->scsi_done for sc_cmd %p is NULL.\n",
- sc_cmd);
- goto bad_scsi_ptr;
- }
-
qedf_unmap_sg_list(qedf, io_req);
sc_cmd->result = result << 16;
io_req->sc_cmd = NULL;
sc_cmd->SCp.ptr = NULL;
- sc_cmd->scsi_done(sc_cmd);
+ scsi_done(sc_cmd);
kref_put(&io_req->refcount, qedf_release_cmd);
return;
.cmd_per_lun = 32,
.max_sectors = 0xffff,
.queuecommand = qedf_queuecommand,
- .shost_attrs = qedf_host_attrs,
+ .shost_groups = qedf_host_groups,
.eh_abort_handler = qedf_eh_abort,
.eh_device_reset_handler = qedf_eh_device_reset, /* lun reset */
.eh_target_reset_handler = qedf_eh_target_reset, /* target reset */
extern const struct qed_iscsi_ops *qedi_ops;
extern const struct qedi_debugfs_ops qedi_debugfs_ops[];
extern const struct file_operations qedi_dbg_fops[];
-extern struct device_attribute *qedi_shost_attrs[];
+extern const struct attribute_group *qedi_shost_groups[];
int qedi_alloc_sq(struct qedi_ctx *qedi, struct qedi_endpoint *ep);
void qedi_free_sq(struct qedi_ctx *qedi, struct qedi_endpoint *ep);
.max_sectors = 0xffff,
.dma_boundary = QEDI_HW_DMA_BOUNDARY,
.cmd_per_lun = 128,
- .shost_attrs = qedi_shost_attrs,
+ .shost_groups = qedi_shost_groups,
};
static void qedi_conn_free_login_resources(struct qedi_ctx *qedi,
static DEVICE_ATTR_RO(port_state);
static DEVICE_ATTR_RO(speed);
-struct device_attribute *qedi_shost_attrs[] = {
- &dev_attr_port_state,
- &dev_attr_speed,
+static struct attribute *qedi_shost_attrs[] = {
+ &dev_attr_port_state.attr,
+ &dev_attr_speed.attr,
+ NULL
+};
+
+static const struct attribute_group qedi_shost_attr_group = {
+ .attrs = qedi_shost_attrs
+};
+
+const struct attribute_group *qedi_shost_groups[] = {
+ &qedi_shost_attr_group,
NULL
};
* handling). Unfortunately, it sometimes calls the scheduler in interrupt
* context which is a big NO! NO!.
**************************************************************************/
-static int
-qla1280_queuecommand_lck(struct scsi_cmnd *cmd, void (*fn)(struct scsi_cmnd *))
+static int qla1280_queuecommand_lck(struct scsi_cmnd *cmd)
{
struct Scsi_Host *host = cmd->device->host;
struct scsi_qla_host *ha = (struct scsi_qla_host *)host->hostdata;
struct srb *sp = (struct srb *)CMD_SP(cmd);
int status;
- cmd->scsi_done = fn;
sp->cmd = cmd;
sp->flags = 0;
sp->wait = NULL;
sp->wait = NULL;
if(CMD_HANDLE(cmd) == COMPLETED_HANDLE) {
status = SUCCESS;
- (*cmd->scsi_done)(cmd);
+ scsi_done(cmd);
}
return status;
}
ha->actthreads--;
if (sp->wait == NULL)
- (*(cmd)->scsi_done)(cmd);
+ scsi_done(cmd);
else
complete(sp->wait);
}
return strlen(buf);
}
+static const struct {
+ u16 rate;
+ char *str;
+} port_speed_str[] = {
+ { PORT_SPEED_4GB, "4" },
+ { PORT_SPEED_8GB, "8" },
+ { PORT_SPEED_16GB, "16" },
+ { PORT_SPEED_32GB, "32" },
+ { PORT_SPEED_64GB, "64" },
+ { PORT_SPEED_10GB, "10" },
+};
+
static ssize_t
qla2x00_port_speed_show(struct device *dev, struct device_attribute *attr,
char *buf)
struct scsi_qla_host *vha = shost_priv(dev_to_shost(dev));
struct qla_hw_data *ha = vha->hw;
ssize_t rval;
- char *spd[7] = {"0", "0", "0", "4", "8", "16", "32"};
+ u16 i;
+ char *speed = "Unknown";
rval = qla2x00_get_data_rate(vha);
if (rval != QLA_SUCCESS) {
return -EINVAL;
}
- return scnprintf(buf, PAGE_SIZE, "%s\n", spd[ha->link_data_rate]);
+ for (i = 0; i < ARRAY_SIZE(port_speed_str); i++) {
+ if (port_speed_str[i].rate != ha->link_data_rate)
+ continue;
+ speed = port_speed_str[i].str;
+ break;
+ }
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", speed);
}
static ssize_t
static DEVICE_ATTR(fw_attr, 0444, qla2x00_fw_attr_show, NULL);
static DEVICE_ATTR_RO(edif_doorbell);
-
-struct device_attribute *qla2x00_host_attrs[] = {
- &dev_attr_driver_version,
- &dev_attr_fw_version,
- &dev_attr_serial_num,
- &dev_attr_isp_name,
- &dev_attr_isp_id,
- &dev_attr_model_name,
- &dev_attr_model_desc,
- &dev_attr_pci_info,
- &dev_attr_link_state,
- &dev_attr_zio,
- &dev_attr_zio_timer,
- &dev_attr_beacon,
- &dev_attr_beacon_config,
- &dev_attr_optrom_bios_version,
- &dev_attr_optrom_efi_version,
- &dev_attr_optrom_fcode_version,
- &dev_attr_optrom_fw_version,
- &dev_attr_84xx_fw_version,
- &dev_attr_total_isp_aborts,
- &dev_attr_serdes_version,
- &dev_attr_mpi_version,
- &dev_attr_phy_version,
- &dev_attr_flash_block_size,
- &dev_attr_vlan_id,
- &dev_attr_vn_port_mac_address,
- &dev_attr_fabric_param,
- &dev_attr_fw_state,
- &dev_attr_optrom_gold_fw_version,
- &dev_attr_thermal_temp,
- &dev_attr_diag_requests,
- &dev_attr_diag_megabytes,
- &dev_attr_fw_dump_size,
- &dev_attr_allow_cna_fw_dump,
- &dev_attr_pep_version,
- &dev_attr_min_supported_speed,
- &dev_attr_max_supported_speed,
- &dev_attr_zio_threshold,
- &dev_attr_dif_bundle_statistics,
- &dev_attr_port_speed,
- &dev_attr_port_no,
- &dev_attr_fw_attr,
- &dev_attr_dport_diagnostics,
- &dev_attr_edif_doorbell,
- &dev_attr_mpi_pause,
- NULL, /* reserve for qlini_mode */
- NULL, /* reserve for ql2xiniexchg */
- NULL, /* reserve for ql2xexchoffld */
+static struct attribute *qla2x00_host_attrs[] = {
+ &dev_attr_driver_version.attr,
+ &dev_attr_fw_version.attr,
+ &dev_attr_serial_num.attr,
+ &dev_attr_isp_name.attr,
+ &dev_attr_isp_id.attr,
+ &dev_attr_model_name.attr,
+ &dev_attr_model_desc.attr,
+ &dev_attr_pci_info.attr,
+ &dev_attr_link_state.attr,
+ &dev_attr_zio.attr,
+ &dev_attr_zio_timer.attr,
+ &dev_attr_beacon.attr,
+ &dev_attr_beacon_config.attr,
+ &dev_attr_optrom_bios_version.attr,
+ &dev_attr_optrom_efi_version.attr,
+ &dev_attr_optrom_fcode_version.attr,
+ &dev_attr_optrom_fw_version.attr,
+ &dev_attr_84xx_fw_version.attr,
+ &dev_attr_total_isp_aborts.attr,
+ &dev_attr_serdes_version.attr,
+ &dev_attr_mpi_version.attr,
+ &dev_attr_phy_version.attr,
+ &dev_attr_flash_block_size.attr,
+ &dev_attr_vlan_id.attr,
+ &dev_attr_vn_port_mac_address.attr,
+ &dev_attr_fabric_param.attr,
+ &dev_attr_fw_state.attr,
+ &dev_attr_optrom_gold_fw_version.attr,
+ &dev_attr_thermal_temp.attr,
+ &dev_attr_diag_requests.attr,
+ &dev_attr_diag_megabytes.attr,
+ &dev_attr_fw_dump_size.attr,
+ &dev_attr_allow_cna_fw_dump.attr,
+ &dev_attr_pep_version.attr,
+ &dev_attr_min_supported_speed.attr,
+ &dev_attr_max_supported_speed.attr,
+ &dev_attr_zio_threshold.attr,
+ &dev_attr_dif_bundle_statistics.attr,
+ &dev_attr_port_speed.attr,
+ &dev_attr_port_no.attr,
+ &dev_attr_fw_attr.attr,
+ &dev_attr_dport_diagnostics.attr,
+ &dev_attr_edif_doorbell.attr,
+ &dev_attr_mpi_pause.attr,
+ &dev_attr_qlini_mode.attr,
+ &dev_attr_ql2xiniexchg.attr,
+ &dev_attr_ql2xexchoffld.attr,
NULL,
};
-void qla_insert_tgt_attrs(void)
+static umode_t qla_host_attr_is_visible(struct kobject *kobj,
+ struct attribute *attr, int i)
{
- struct device_attribute **attr;
+ if (ql2x_ini_mode != QLA2XXX_INI_MODE_DUAL &&
+ (attr == &dev_attr_qlini_mode.attr ||
+ attr == &dev_attr_ql2xiniexchg.attr ||
+ attr == &dev_attr_ql2xexchoffld.attr))
+ return 0;
+ return attr->mode;
+}
- /* advance to empty slot */
- for (attr = &qla2x00_host_attrs[0]; *attr; ++attr)
- continue;
+static const struct attribute_group qla2x00_host_attr_group = {
+ .is_visible = qla_host_attr_is_visible,
+ .attrs = qla2x00_host_attrs
+};
- *attr = &dev_attr_qlini_mode;
- attr++;
- *attr = &dev_attr_ql2xiniexchg;
- attr++;
- *attr = &dev_attr_ql2xexchoffld;
-}
+const struct attribute_group *qla2x00_host_groups[] = {
+ &qla2x00_host_attr_group,
+ NULL
+};
/* Host attributes. */
if (fcport->loop_id != FC_NO_LOOP_ID)
fcport->logout_on_delete = 1;
- qlt_schedule_sess_for_deletion(fcport);
+ if (!EDIF_NEGOTIATION_PENDING(fcport)) {
+ ql_dbg(ql_dbg_disc, fcport->vha, 0x911e,
+ "%s %d schedule session deletion\n", __func__,
+ __LINE__);
+ qlt_schedule_sess_for_deletion(fcport);
+ }
} else {
qla2x00_port_logout(fcport->vha, fcport);
}
case QL_VND_MANAGE_HOST_PORT:
return qla2x00_manage_host_port(bsg_job);
+ case QL_VND_MBX_PASSTHRU:
+ return qla2x00_mailbox_passthru(bsg_job);
+
default:
return -ENOSYS;
}
sp->free(sp);
return 0;
}
+
+int qla2x00_mailbox_passthru(struct bsg_job *bsg_job)
+{
+ struct fc_bsg_reply *bsg_reply = bsg_job->reply;
+ scsi_qla_host_t *vha = shost_priv(fc_bsg_to_shost(bsg_job));
+ int ret = -EINVAL;
+ int ptsize = sizeof(struct qla_mbx_passthru);
+ struct qla_mbx_passthru *req_data = NULL;
+ uint32_t req_data_len;
+
+ req_data_len = bsg_job->request_payload.payload_len;
+ if (req_data_len != ptsize) {
+ ql_log(ql_log_warn, vha, 0xf0a3, "req_data_len invalid.\n");
+ return -EIO;
+ }
+ req_data = kzalloc(ptsize, GFP_KERNEL);
+ if (!req_data) {
+ ql_log(ql_log_warn, vha, 0xf0a4,
+ "req_data memory allocation failure.\n");
+ return -ENOMEM;
+ }
+
+ /* Copy the request buffer in req_data */
+ sg_copy_to_buffer(bsg_job->request_payload.sg_list,
+ bsg_job->request_payload.sg_cnt, req_data, ptsize);
+ ret = qla_mailbox_passthru(vha, req_data->mbx_in, req_data->mbx_out);
+
+ /* Copy the req_data in request buffer */
+ sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
+ bsg_job->reply_payload.sg_cnt, req_data, ptsize);
+
+ bsg_reply->reply_payload_rcv_len = ptsize;
+ if (ret == QLA_SUCCESS)
+ bsg_reply->reply_data.vendor_reply.vendor_rsp[0] = EXT_STATUS_OK;
+ else
+ bsg_reply->reply_data.vendor_reply.vendor_rsp[0] = EXT_STATUS_ERR;
+
+ bsg_job->reply_len = sizeof(*bsg_job->reply);
+ bsg_reply->result = DID_OK << 16;
+ bsg_job_done(bsg_job, bsg_reply->result, bsg_reply->reply_payload_rcv_len);
+
+ kfree(req_data);
+
+ return ret;
+}
#define QL_VND_GET_HOST_STATS 0x24
#define QL_VND_GET_TGT_STATS 0x25
#define QL_VND_MANAGE_HOST_PORT 0x26
+#define QL_VND_MBX_PASSTHRU 0x2B
/* BSG Vendor specific subcode returns */
#define EXT_STATUS_OK 0
uint16_t speed;
} __attribute__ ((packed));
+struct qla_mbx_passthru {
+ uint16_t reserved1[2];
+ uint16_t mbx_in[32];
+ uint16_t mbx_out[32];
+ uint32_t reserved2[16];
+} __packed;
/* FRU VPD */
u8 els_opcode;
u8 vp_idx;
__le16 nport_handle;
- u16 control_flags;
+ u16 control_flags, ox_id;
__le32 rx_xchg_address;
- port_id_t did;
+ port_id_t did, sid;
u32 tx_len, tx_byte_count, rx_len, rx_byte_count;
dma_addr_t tx_addr, rx_addr;
struct qla_fw_resources fwres ____cacheline_aligned;
u32 cmd_cnt;
u32 cmd_completion_cnt;
+ u32 prev_completion_cnt;
};
/* Place holder for FW buffer parameters */
struct qla_chip_state_84xx *cs84xx;
struct isp_operations *isp_ops;
struct workqueue_struct *wq;
+ struct work_struct heartbeat_work;
struct qlfc_fw fw_buf;
/* FCP_CMND priority support */
struct qla_hw_data_stat stat;
pci_error_state_t pci_error_state;
- u64 prev_cmd_cnt;
struct dma_pool *purex_dma_pool;
struct btree_head32 host_map;
#define SET_ZIO_THRESHOLD_NEEDED 32
#define ISP_ABORT_TO_ROM 33
#define VPORT_DELETE 34
-#define HEARTBEAT_CHK 38
#define PROCESS_PUREX_IOCB 63
"%s edif not enabled\n", __func__);
goto done;
}
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
ql_dbg(ql_dbg_edif, vha, 0x09102,
"%s doorbell not enabled\n", __func__);
goto done;
return false;
}
-static void qla_edif_reset_auth_wait(struct fc_port *fcport, int state,
- int waitonly)
-{
- int cnt, max_cnt = 200;
- bool traced = false;
-
- fcport->keep_nport_handle = 1;
-
- if (!waitonly) {
- qla2x00_set_fcport_disc_state(fcport, state);
- qlt_schedule_sess_for_deletion(fcport);
- } else {
- qla2x00_set_fcport_disc_state(fcport, state);
- }
-
- ql_dbg(ql_dbg_edif, fcport->vha, 0xf086,
- "%s: waiting for session, max_cnt=%u\n",
- __func__, max_cnt);
-
- cnt = 0;
-
- if (waitonly) {
- /* Marker wait min 10 msecs. */
- msleep(50);
- cnt += 50;
- }
- while (1) {
- if (!traced) {
- ql_dbg(ql_dbg_edif, fcport->vha, 0xf086,
- "%s: session sleep.\n",
- __func__);
- traced = true;
- }
- msleep(20);
- cnt++;
- if (waitonly && (fcport->disc_state == state ||
- fcport->disc_state == DSC_LOGIN_COMPLETE))
- break;
- if (fcport->disc_state == DSC_LOGIN_AUTH_PEND)
- break;
- if (cnt > max_cnt)
- break;
- }
-
- if (!waitonly) {
- ql_dbg(ql_dbg_edif, fcport->vha, 0xf086,
- "%s: waited for session - %8phC, loopid=%x portid=%06x fcport=%p state=%u, cnt=%u\n",
- __func__, fcport->port_name, fcport->loop_id,
- fcport->d_id.b24, fcport, fcport->disc_state, cnt);
- } else {
- ql_dbg(ql_dbg_edif, fcport->vha, 0xf086,
- "%s: waited ONLY for session - %8phC, loopid=%x portid=%06x fcport=%p state=%u, cnt=%u\n",
- __func__, fcport->port_name, fcport->loop_id,
- fcport->d_id.b24, fcport, fcport->disc_state, cnt);
- }
-}
-
static void
qla_edif_free_sa_ctl(fc_port_t *fcport, struct edif_sa_ctl *sa_ctl,
int index)
struct app_start_reply appreply;
struct fc_port *fcport, *tf;
- ql_dbg(ql_dbg_edif, vha, 0x911d, "%s app start\n", __func__);
+ ql_log(ql_log_info, vha, 0x1313,
+ "EDIF application registration with driver, FC device connections will be re-established.\n");
sg_copy_to_buffer(bsg_job->request_payload.sg_list,
bsg_job->request_payload.sg_cnt, &appstart,
ql_dbg(ql_dbg_edif, vha, 0x911d, "%s app_vid=%x app_start_flags %x\n",
__func__, appstart.app_info.app_vid, appstart.app_start_flags);
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
/* mark doorbell as active since an app is now present */
- vha->e_dbell.db_flags = EDB_ACTIVE;
+ vha->e_dbell.db_flags |= EDB_ACTIVE;
} else {
ql_dbg(ql_dbg_edif, vha, 0x911e, "%s doorbell already active\n",
__func__);
qla2xxx_wake_dpc(vha);
} else {
list_for_each_entry_safe(fcport, tf, &vha->vp_fcports, list) {
+ ql_dbg(ql_dbg_edif, vha, 0x2058,
+ "FCSP - nn %8phN pn %8phN portid=%06x.\n",
+ fcport->node_name, fcport->port_name,
+ fcport->d_id.b24);
ql_dbg(ql_dbg_edif, vha, 0xf084,
- "%s: sess %p %8phC lid %#04x s_id %06x logout %d\n",
- __func__, fcport, fcport->port_name,
- fcport->loop_id, fcport->d_id.b24,
- fcport->logout_on_delete);
-
- ql_dbg(ql_dbg_edif, vha, 0xf084,
- "keep %d els_logo %d disc state %d auth state %d stop state %d\n",
- fcport->keep_nport_handle,
- fcport->send_els_logo, fcport->disc_state,
- fcport->edif.auth_state, fcport->edif.app_stop);
+ "%s: se_sess %p / sess %p from port %8phC "
+ "loop_id %#04x s_id %06x logout %d "
+ "keep %d els_logo %d disc state %d auth state %d"
+ "stop state %d\n",
+ __func__, fcport->se_sess, fcport,
+ fcport->port_name, fcport->loop_id,
+ fcport->d_id.b24, fcport->logout_on_delete,
+ fcport->keep_nport_handle, fcport->send_els_logo,
+ fcport->disc_state, fcport->edif.auth_state,
+ fcport->edif.app_stop);
if (atomic_read(&vha->loop_state) == LOOP_DOWN)
break;
- if (!(fcport->flags & FCF_FCSP_DEVICE))
- continue;
fcport->edif.app_started = 1;
- if (fcport->edif.app_stop ||
- (fcport->disc_state != DSC_LOGIN_COMPLETE &&
- fcport->disc_state != DSC_LOGIN_PEND &&
- fcport->disc_state != DSC_DELETED)) {
- /* no activity */
- fcport->edif.app_stop = 0;
-
- ql_dbg(ql_dbg_edif, vha, 0x911e,
- "%s wwpn %8phC calling qla_edif_reset_auth_wait\n",
- __func__, fcport->port_name);
- fcport->edif.app_sess_online = 1;
- qla_edif_reset_auth_wait(fcport, DSC_LOGIN_PEND, 0);
- }
+ fcport->login_retry = vha->hw->login_retry_count;
+
+ /* no activity */
+ fcport->edif.app_stop = 0;
+
+ ql_dbg(ql_dbg_edif, vha, 0x911e,
+ "%s wwpn %8phC calling qla_edif_reset_auth_wait\n",
+ __func__, fcport->port_name);
+ fcport->edif.app_sess_online = 0;
+ qlt_schedule_sess_for_deletion(fcport);
qla_edif_sa_ctl_init(vha, fcport);
}
}
appreply.edif_enode_active = vha->pur_cinfo.enode_flags;
appreply.edif_edb_active = vha->e_dbell.db_flags;
- bsg_job->reply_len = sizeof(struct fc_bsg_reply) +
- sizeof(struct app_start_reply);
+ bsg_job->reply_len = sizeof(struct fc_bsg_reply);
SET_DID_STATUS(bsg_reply->result, DID_OK);
- sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
- bsg_job->reply_payload.sg_cnt, &appreply,
- sizeof(struct app_start_reply));
+ bsg_reply->reply_payload_rcv_len = sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
+ bsg_job->reply_payload.sg_cnt,
+ &appreply,
+ sizeof(struct app_start_reply));
ql_dbg(ql_dbg_edif, vha, 0x911d,
"%s app start completed with 0x%x\n",
ql_dbg(ql_dbg_edif, vha, 0x911e,
"%s AUTH complete - RESUME with prli for wwpn %8phC\n",
__func__, fcport->port_name);
- qla_edif_reset_auth_wait(fcport, DSC_LOGIN_PEND, 1);
qla24xx_post_prli_work(vha, fcport);
}
errstate_exit:
bsg_job->reply_len = sizeof(struct fc_bsg_reply);
- sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
- bsg_job->reply_payload.sg_cnt, &appplogireply,
- sizeof(struct app_plogi_reply));
+ bsg_reply->reply_payload_rcv_len = sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
+ bsg_job->reply_payload.sg_cnt,
+ &appplogireply,
+ sizeof(struct app_plogi_reply));
return rval;
}
if (qla_ini_mode_enabled(fcport->vha)) {
fcport->send_els_logo = 1;
- qla_edif_reset_auth_wait(fcport, DSC_LOGIN_PEND, 0);
+ qlt_schedule_sess_for_deletion(fcport);
}
}
qla_edif_app_getfcinfo(scsi_qla_host_t *vha, struct bsg_job *bsg_job)
{
int32_t rval = 0;
- int32_t num_cnt;
+ int32_t pcnt = 0;
struct fc_bsg_reply *bsg_reply = bsg_job->reply;
struct app_pinfo_req app_req;
struct app_pinfo_reply *app_reply;
bsg_job->request_payload.sg_cnt, &app_req,
sizeof(struct app_pinfo_req));
- num_cnt = app_req.num_ports; /* num of ports alloc'd by app */
-
app_reply = kzalloc((sizeof(struct app_pinfo_reply) +
- sizeof(struct app_pinfo) * num_cnt), GFP_KERNEL);
+ sizeof(struct app_pinfo) * app_req.num_ports), GFP_KERNEL);
+
if (!app_reply) {
SET_DID_STATUS(bsg_reply->result, DID_ERROR);
rval = -1;
} else {
struct fc_port *fcport = NULL, *tf;
- uint32_t pcnt = 0;
list_for_each_entry_safe(fcport, tf, &vha->vp_fcports, list) {
if (!(fcport->flags & FCF_FCSP_DEVICE))
SET_DID_STATUS(bsg_reply->result, DID_OK);
}
- sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
- bsg_job->reply_payload.sg_cnt, app_reply,
- sizeof(struct app_pinfo_reply) + sizeof(struct app_pinfo) * num_cnt);
+ bsg_job->reply_len = sizeof(struct fc_bsg_reply);
+ bsg_reply->reply_payload_rcv_len = sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
+ bsg_job->reply_payload.sg_cnt,
+ app_reply,
+ sizeof(struct app_pinfo_reply) + sizeof(struct app_pinfo) * pcnt);
kfree(app_reply);
{
int32_t rval = 0;
struct fc_bsg_reply *bsg_reply = bsg_job->reply;
- uint32_t ret_size, size;
+ uint32_t size;
struct app_sinfo_req app_req;
struct app_stats_reply *app_reply;
+ uint32_t pcnt = 0;
sg_copy_to_buffer(bsg_job->request_payload.sg_list,
bsg_job->request_payload.sg_cnt, &app_req,
size = sizeof(struct app_stats_reply) +
(sizeof(struct app_sinfo) * app_req.num_ports);
- if (size > bsg_job->reply_payload.payload_len)
- ret_size = bsg_job->reply_payload.payload_len;
- else
- ret_size = size;
-
app_reply = kzalloc(size, GFP_KERNEL);
if (!app_reply) {
SET_DID_STATUS(bsg_reply->result, DID_ERROR);
rval = -1;
} else {
struct fc_port *fcport = NULL, *tf;
- uint32_t pcnt = 0;
list_for_each_entry_safe(fcport, tf, &vha->vp_fcports, list) {
if (fcport->edif.enable) {
SET_DID_STATUS(bsg_reply->result, DID_OK);
}
+ bsg_job->reply_len = sizeof(struct fc_bsg_reply);
bsg_reply->reply_payload_rcv_len =
sg_copy_from_buffer(bsg_job->reply_payload.sg_list,
- bsg_job->reply_payload.sg_cnt, app_reply, ret_size);
+ bsg_job->reply_payload.sg_cnt, app_reply,
+ sizeof(struct app_stats_reply) + (sizeof(struct app_sinfo) * pcnt));
kfree(app_reply);
__func__,
bsg_request->rqst_data.h_vendor.vendor_cmd[1]);
rval = EXT_STATUS_INVALID_PARAM;
- bsg_job->reply_len = sizeof(struct fc_bsg_reply);
- SET_DID_STATUS(bsg_reply->result, DID_ERROR);
+ done = false;
break;
}
goto done;
}
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
ql_log(ql_log_warn, vha, 0x70a1, "App not started\n");
rval = -EIO;
SET_DID_STATUS(bsg_reply->result, DID_ERROR);
spin_unlock_irqrestore(&vha->pur_cinfo.pur_lock, flags);
}
+static void qla_enode_clear(scsi_qla_host_t *vha, port_id_t portid)
+{
+ unsigned long flags;
+ struct enode *e, *tmp;
+ struct purexevent *purex;
+ LIST_HEAD(enode_list);
+
+ if (vha->pur_cinfo.enode_flags != ENODE_ACTIVE) {
+ ql_dbg(ql_dbg_edif, vha, 0x09102,
+ "%s enode not active\n", __func__);
+ return;
+ }
+ spin_lock_irqsave(&vha->pur_cinfo.pur_lock, flags);
+ list_for_each_entry_safe(e, tmp, &vha->pur_cinfo.head, list) {
+ purex = &e->u.purexinfo;
+ if (purex->pur_info.pur_sid.b24 == portid.b24) {
+ ql_dbg(ql_dbg_edif, vha, 0x911d,
+ "%s free ELS sid=%06x. xchg %x, nb=%xh\n",
+ __func__, portid.b24,
+ purex->pur_info.pur_rx_xchg_address,
+ purex->pur_info.pur_bytes_rcvd);
+
+ list_del_init(&e->list);
+ list_add_tail(&e->list, &enode_list);
+ }
+ }
+ spin_unlock_irqrestore(&vha->pur_cinfo.pur_lock, flags);
+
+ list_for_each_entry_safe(e, tmp, &enode_list, list) {
+ list_del_init(&e->list);
+ qla_enode_free(vha, e);
+ }
+}
+
/*
* allocate enode struct and populate buffer
* returns: enode pointer with buffers
qla_enode_find(scsi_qla_host_t *vha, uint32_t ntype, uint32_t p1, uint32_t p2)
{
struct enode *node_rtn = NULL;
- struct enode *list_node = NULL;
+ struct enode *list_node, *q;
unsigned long flags;
- struct list_head *pos, *q;
uint32_t sid;
- uint32_t rw_flag;
struct purexevent *purex;
/* secure the list from moving under us */
spin_lock_irqsave(&vha->pur_cinfo.pur_lock, flags);
- list_for_each_safe(pos, q, &vha->pur_cinfo.head) {
- list_node = list_entry(pos, struct enode, list);
+ list_for_each_entry_safe(list_node, q, &vha->pur_cinfo.head, list) {
/* node type determines what p1 and p2 are */
purex = &list_node->u.purexinfo;
sid = p1;
- rw_flag = p2;
if (purex->pur_info.pur_sid.b24 == sid) {
- if (purex->pur_info.pur_pend == 1 &&
- rw_flag == PUR_GET) {
- /*
- * if the receive is in progress
- * and its a read/get then can't
- * transfer yet
- */
- ql_dbg(ql_dbg_edif, vha, 0x9106,
- "%s purex xfer in progress for sid=%x\n",
- __func__, sid);
- } else {
- /* found it and its complete */
- node_rtn = list_node;
- list_del(pos);
- break;
- }
+ /* found it and its complete */
+ node_rtn = list_node;
+ list_del(&list_node->list);
+ break;
}
}
qla_els_pt_iocb(vha, els_iocb, a);
ql_dbg(ql_dbg_edif, vha, 0x0183,
- "Sending ELS reject...\n");
+ "Sending ELS reject ox_id %04x s:%06x -> d:%06x\n",
+ a->ox_id, a->sid.b24, a->did.b24);
ql_dump_buffer(ql_dbg_edif + ql_dbg_verbose, vha, 0x0185,
vha->hw->elsrej.c, sizeof(*vha->hw->elsrej.c));
/* flush iocb to mem before notifying hw doorbell */
void
qla_edb_init(scsi_qla_host_t *vha)
{
- if (vha->e_dbell.db_flags == EDB_ACTIVE) {
+ if (DBELL_ACTIVE(vha)) {
/* list already init'd - error */
ql_dbg(ql_dbg_edif, vha, 0x09102,
"edif db already initialized, cannot reinit\n");
node->ntype = N_UNDEF;
}
+static void qla_edb_clear(scsi_qla_host_t *vha, port_id_t portid)
+{
+ unsigned long flags;
+ struct edb_node *e, *tmp;
+ port_id_t sid;
+ LIST_HEAD(edb_list);
+
+ if (DBELL_INACTIVE(vha)) {
+ /* doorbell list not enabled */
+ ql_dbg(ql_dbg_edif, vha, 0x09102,
+ "%s doorbell not enabled\n", __func__);
+ return;
+ }
+
+ /* grab lock so list doesn't move */
+ spin_lock_irqsave(&vha->e_dbell.db_lock, flags);
+ list_for_each_entry_safe(e, tmp, &vha->e_dbell.head, list) {
+ switch (e->ntype) {
+ case VND_CMD_AUTH_STATE_NEEDED:
+ case VND_CMD_AUTH_STATE_SESSION_SHUTDOWN:
+ sid = e->u.plogi_did;
+ break;
+ case VND_CMD_AUTH_STATE_ELS_RCVD:
+ sid = e->u.els_sid;
+ break;
+ case VND_CMD_AUTH_STATE_SAUPDATE_COMPL:
+ /* app wants to see this */
+ continue;
+ default:
+ ql_log(ql_log_warn, vha, 0x09102,
+ "%s unknown node type: %x\n", __func__, e->ntype);
+ sid.b24 = 0;
+ break;
+ }
+ if (sid.b24 == portid.b24) {
+ ql_dbg(ql_dbg_edif, vha, 0x910f,
+ "%s free doorbell event : node type = %x %p\n",
+ __func__, e->ntype, e);
+ list_del_init(&e->list);
+ list_add_tail(&e->list, &edb_list);
+ }
+ }
+ spin_unlock_irqrestore(&vha->e_dbell.db_lock, flags);
+
+ list_for_each_entry_safe(e, tmp, &edb_list, list) {
+ qla_edb_node_free(vha, e);
+ list_del_init(&e->list);
+ kfree(e);
+ }
+}
+
/* function called when app is stopping */
void
unsigned long flags;
struct edb_node *node, *q;
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
/* doorbell list not enabled */
ql_dbg(ql_dbg_edif, vha, 0x09102,
"%s doorbell not enabled\n", __func__);
{
unsigned long flags;
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
/* doorbell list not enabled */
ql_dbg(ql_dbg_edif, vha, 0x09102,
"%s doorbell not enabled\n", __func__);
return;
}
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
if (fcport)
fcport->edif.auth_state = dbtype;
/* doorbell list not enabled */
struct qla_hw_data *ha = vha->hw;
if (!vha->vp_idx && N2N_TOPO(ha) && ha->flags.n2n_fw_acc_sec) {
- if (vha->e_dbell.db_flags != EDB_ACTIVE &&
+ if (DBELL_INACTIVE(vha) &&
ha->edif_post_stop_cnt_down) {
ha->edif_post_stop_cnt_down--;
sz = 256;
/* stop new threads from waiting if we're not init'd */
- if (vha->e_dbell.db_flags != EDB_ACTIVE) {
+ if (DBELL_INACTIVE(vha)) {
ql_dbg(ql_dbg_edif + ql_dbg_verbose, vha, 0x09122,
"%s error - edif db not enabled\n", __func__);
return 0;
a.tx_addr = vha->hw->elsrej.cdma;
a.vp_idx = vha->vp_idx;
a.control_flags = EPD_ELS_RJT;
+ a.ox_id = le16_to_cpu(p->ox_id);
sid = p->s_id[0] | (p->s_id[1] << 8) | (p->s_id[2] << 16);
return;
}
- if (totlen > MAX_PAYLOAD) {
+ if (totlen > ELS_MAX_PAYLOAD) {
ql_dbg(ql_dbg_edif, vha, 0x0910d,
"%s WARNING: verbose ELS frame received (totlen=%x)\n",
__func__, totlen);
purex = &ptr->u.purexinfo;
purex->pur_info.pur_sid = a.did;
- purex->pur_info.pur_pend = 0;
purex->pur_info.pur_bytes_rcvd = totlen;
purex->pur_info.pur_rx_xchg_address = le32_to_cpu(p->rx_xchg_addr);
purex->pur_info.pur_nphdl = le16_to_cpu(p->nport_handle);
purex->pur_info.pur_did.b.al_pa = p->d_id[0];
purex->pur_info.vp_idx = p->vp_idx;
+ a.sid = purex->pur_info.pur_did;
+
rc = __qla_copy_purex_to_buffer(vha, pkt, rsp, purex->msgp,
purex->msgp_len);
if (rc) {
fcport = qla2x00_find_fcport_by_pid(host, &purex->pur_info.pur_sid);
- if (host->e_dbell.db_flags != EDB_ACTIVE ||
+ if (DBELL_INACTIVE(vha) ||
(fcport && EDIF_SESSION_DOWN(fcport))) {
ql_dbg(ql_dbg_edif, host, 0x0910c, "%s e_dbell.db_flags =%x %06x\n",
__func__, host->e_dbell.db_flags,
ql_dbg(ql_dbg_edif, host, 0x0910c,
"%s COMPLETE purex->pur_info.pur_bytes_rcvd =%xh s:%06x -> d:%06x xchg=%xh\n",
__func__, purex->pur_info.pur_bytes_rcvd, purex->pur_info.pur_sid.b24,
- purex->pur_info.pur_did.b24, p->rx_xchg_addr);
+ purex->pur_info.pur_did.b24, purex->pur_info.pur_rx_xchg_address);
qla_edb_eventcreate(host, VND_CMD_AUTH_STATE_ELS_RCVD, sid, 0, NULL);
}
/* release any sadb entries -- only done at teardown */
void qla_edif_sadb_release(struct qla_hw_data *ha)
{
- struct list_head *pos;
- struct list_head *tmp;
- struct edif_sa_index_entry *entry;
+ struct edif_sa_index_entry *entry, *tmp;
- list_for_each_safe(pos, tmp, &ha->sadb_rx_index_list) {
- entry = list_entry(pos, struct edif_sa_index_entry, next);
+ list_for_each_entry_safe(entry, tmp, &ha->sadb_rx_index_list, next) {
list_del(&entry->next);
kfree(entry);
}
- list_for_each_safe(pos, tmp, &ha->sadb_tx_index_list) {
- entry = list_entry(pos, struct edif_sa_index_entry, next);
+ list_for_each_entry_safe(entry, tmp, &ha->sadb_tx_index_list, next) {
list_del(&entry->next);
kfree(entry);
}
void qla_edif_sess_down(struct scsi_qla_host *vha, struct fc_port *sess)
{
- if (sess->edif.app_sess_online && vha->e_dbell.db_flags & EDB_ACTIVE) {
+ if (sess->edif.app_sess_online && DBELL_ACTIVE(vha)) {
ql_dbg(ql_dbg_disc, vha, 0xf09c,
"%s: sess %8phN send port_offline event\n",
__func__, sess->port_name);
qla2x00_post_aen_work(vha, FCH_EVT_PORT_OFFLINE, sess->d_id.b24);
}
}
+
+void qla_edif_clear_appdata(struct scsi_qla_host *vha, struct fc_port *fcport)
+{
+ if (!(fcport->flags & FCF_FCSP_DEVICE))
+ return;
+
+ qla_edb_clear(vha, fcport->d_id);
+ qla_enode_clear(vha, fcport->d_id);
+}
};
enum db_flags_t {
- EDB_ACTIVE = 0x1,
+ EDB_ACTIVE = BIT_0,
};
+#define DBELL_ACTIVE(_v) (_v->e_dbell.db_flags & EDB_ACTIVE)
+#define DBELL_INACTIVE(_v) (!(_v->e_dbell.db_flags & EDB_ACTIVE))
+
struct edif_dbell {
enum db_flags_t db_flags;
spinlock_t db_lock;
};
#define NUM_ENTRIES 256
-#define MAX_PAYLOAD 1024
#define PUR_GET 1
struct dinfo {
};
struct pur_ninfo {
- unsigned int pur_pend:1;
port_id_t pur_sid;
port_id_t pur_did;
uint8_t vp_idx;
} u;
};
+#define RX_ELS_SIZE (roundup(sizeof(struct enode) + ELS_MAX_PAYLOAD, SMP_CACHE_BYTES))
+
#define EDIF_SESSION_DOWN(_s) \
(qla_ini_mode_enabled(_s->vha) && (_s->disc_state == DSC_DELETE_PEND || \
_s->disc_state == DSC_DELETED || \
!_s->edif.app_sess_online))
+#define EDIF_NEGOTIATION_PENDING(_fcport) \
+ (DBELL_ACTIVE(_fcport->vha) && \
+ (_fcport->disc_state == DSC_LOGIN_AUTH_PEND))
+
#endif /* __QLA_EDIF_H */
#define __QLA_EDIF_BSG_H
/* BSG Vendor specific commands */
-#define ELS_MAX_PAYLOAD 1024
+#define ELS_MAX_PAYLOAD 2112
#ifndef WWN_SIZE
#define WWN_SIZE 8
#endif
void qla2x00_release_all_sadb(struct scsi_qla_host *vha, struct fc_port *fcport);
int qla_edif_process_els(scsi_qla_host_t *vha, struct bsg_job *bsgjob);
void qla_edif_sess_down(struct scsi_qla_host *vha, struct fc_port *sess);
+void qla_edif_clear_appdata(struct scsi_qla_host *vha,
+ struct fc_port *fcport);
const char *sc_to_str(uint16_t cmd);
/*
extern int ql2xgffidenable;
extern int ql2xenabledif;
extern int ql2xenablehba_err_chk;
-extern int ql2xtargetreset;
extern int ql2xdontresethba;
extern uint64_t ql2xmaxlun;
extern int ql2xmdcapmask;
extern void qla2xxx_flash_npiv_conf(scsi_qla_host_t *);
extern int qla24xx_read_fcp_prio_cfg(scsi_qla_host_t *);
+extern int qla2x00_mailbox_passthru(struct bsg_job *bsg_job);
int __qla_copy_purex_to_buffer(struct scsi_qla_host *vha, void **pkt,
struct rsp_que **rsp, u8 *buf, u32 buf_len);
+int qla_mailbox_passthru(scsi_qla_host_t *vha, uint16_t *mbx_in,
+ uint16_t *mbx_out);
+
/*
* Global Function Prototypes in qla_dbg.c source file.
*/
* Global Function Prototypes in qla_attr.c source file.
*/
struct device_attribute;
-extern struct device_attribute *qla2x00_host_attrs[];
-extern struct device_attribute *qla2x00_host_attrs_dm[];
+extern const struct attribute_group *qla2x00_host_groups[];
struct fc_function_template;
extern struct fc_function_template qla2xxx_transport_functions;
extern struct fc_function_template qla2xxx_transport_vport_functions;
extern int qla24xx_update_all_fcp_prio(scsi_qla_host_t *);
extern int qla24xx_fcp_prio_cfg_valid(scsi_qla_host_t *,
struct qla_fcp_prio_cfg *, uint8_t);
-void qla_insert_tgt_attrs(void);
/*
* Global Function Prototypes in qla_dfs.c source file.
*/
extern void qlafx00_fxdisc_iocb(srb_t *, struct fxdisc_entry_fx00 *);
extern void qlafx00_timer_routine(scsi_qla_host_t *);
extern int qlafx00_rescan_isp(scsi_qla_host_t *);
-extern int qlafx00_loop_reset(scsi_qla_host_t *vha);
/* qla82xx related functions */
}
if (IS_QLA2031(ha)) {
if ((ha->pdev->subsystem_vendor == 0x103C) &&
- (ha->pdev->subsystem_device == 0x8002)) {
+ ((ha->pdev->subsystem_device == 0x8002) ||
+ (ha->pdev->subsystem_device == 0x8086))) {
speeds = FDMI_PORT_SPEED_16GB;
} else {
speeds = FDMI_PORT_SPEED_16GB|FDMI_PORT_SPEED_8GB|
lio->u.logio.flags |= SRB_LOGIN_PRLI_ONLY;
} else {
if (vha->hw->flags.edif_enabled &&
- vha->e_dbell.db_flags & EDB_ACTIVE) {
+ DBELL_ACTIVE(vha)) {
lio->u.logio.flags |=
(SRB_LOGIN_FCSP | SRB_LOGIN_SKIP_PRLI);
- ql_dbg(ql_dbg_disc, vha, 0x2072,
- "Async-login: w/ FCSP %8phC hdl=%x, loopid=%x portid=%06x\n",
- fcport->port_name, sp->handle, fcport->loop_id, fcport->d_id.b24);
} else {
lio->u.logio.flags |= SRB_LOGIN_COND_PLOGI;
}
if (NVME_TARGET(vha->hw, fcport))
lio->u.logio.flags |= SRB_LOGIN_SKIP_PRLI;
+ rval = qla2x00_start_sp(sp);
+
ql_dbg(ql_dbg_disc, vha, 0x2072,
- "Async-login - %8phC hdl=%x, loopid=%x portid=%06x retries=%d.\n",
+ "Async-login - %8phC hdl=%x, loopid=%x portid=%06x retries=%d %s.\n",
fcport->port_name, sp->handle, fcport->loop_id,
- fcport->d_id.b24, fcport->login_retry);
+ fcport->d_id.b24, fcport->login_retry,
+ lio->u.logio.flags & SRB_LOGIN_FCSP ? "FCSP" : "");
- rval = qla2x00_start_sp(sp);
if (rval != QLA_SUCCESS) {
fcport->flags |= FCF_LOGIN_NEEDED;
set_bit(RELOGIN_NEEDED, &vha->dpc_flags);
break;
case DSC_LS_PLOGI_COMP:
if (vha->hw->flags.edif_enabled &&
- vha->e_dbell.db_flags & EDB_ACTIVE) {
+ DBELL_ACTIVE(vha)) {
/* check to see if App support secure or not */
qla24xx_post_gpdb_work(vha, fcport, 0);
break;
sp->name, res, sp->u.iocb_cmd.u.mbx.in_mb[1],
sp->u.iocb_cmd.u.mbx.in_mb[2]);
- if (res == QLA_FUNCTION_TIMEOUT)
- return;
sp->fcport->flags &= ~(FCF_ASYNC_SENT|FCF_ASYNC_ACTIVE);
memset(&ea, 0, sizeof(ea));
spin_unlock_irqrestore(&vha->hw->tgt.sess_lock, flags);
list_for_each_entry_safe(fcport, tf, &h, gnl_entry) {
- list_del_init(&fcport->gnl_entry);
spin_lock_irqsave(&vha->hw->tgt.sess_lock, flags);
+ list_del_init(&fcport->gnl_entry);
fcport->flags &= ~(FCF_ASYNC_SENT | FCF_ASYNC_ACTIVE);
spin_unlock_irqrestore(&vha->hw->tgt.sess_lock, flags);
ea.fcport = fcport;
qla2x00_post_aen_work(vha, FCH_EVT_PORT_ONLINE,
fcport->d_id.b24);
- if (vha->e_dbell.db_flags == EDB_ACTIVE) {
+ if (DBELL_ACTIVE(vha)) {
ql_dbg(ql_dbg_disc, vha, 0x20ef,
"%s %d %8phC EDIF: post DB_AUTH: AUTH needed\n",
__func__, __LINE__, fcport->port_name);
fc_port_t *fcport;
unsigned long flags;
- fcport = qla2x00_find_fcport_by_nportid(vha, &ea->id, 1);
- if (fcport) {
- if (fcport->flags & FCF_FCP2_DEVICE) {
- ql_dbg(ql_dbg_disc, vha, 0x2115,
- "Delaying session delete for FCP2 portid=%06x %8phC ",
- fcport->d_id.b24, fcport->port_name);
- return;
+ switch (ea->id.b.rsvd_1) {
+ case RSCN_PORT_ADDR:
+ fcport = qla2x00_find_fcport_by_nportid(vha, &ea->id, 1);
+ if (fcport) {
+ if (fcport->flags & FCF_FCP2_DEVICE) {
+ ql_dbg(ql_dbg_disc, vha, 0x2115,
+ "Delaying session delete for FCP2 portid=%06x %8phC ",
+ fcport->d_id.b24, fcport->port_name);
+ return;
+ }
+
+ if (vha->hw->flags.edif_enabled && DBELL_ACTIVE(vha)) {
+ /*
+ * On ipsec start by remote port, Target port
+ * may use RSCN to trigger initiator to
+ * relogin. If driver is already in the
+ * process of a relogin, then ignore the RSCN
+ * and allow the current relogin to continue.
+ * This reduces thrashing of the connection.
+ */
+ if (atomic_read(&fcport->state) == FCS_ONLINE) {
+ /*
+ * If state = online, then set scan_needed=1 to do relogin.
+ * Otherwise we're already in the middle of a relogin
+ */
+ fcport->scan_needed = 1;
+ fcport->rscn_gen++;
+ }
+ } else {
+ fcport->scan_needed = 1;
+ fcport->rscn_gen++;
+ }
}
- fcport->scan_needed = 1;
- fcport->rscn_gen++;
+ break;
+ case RSCN_AREA_ADDR:
+ list_for_each_entry(fcport, &vha->vp_fcports, list) {
+ if (fcport->flags & FCF_FCP2_DEVICE)
+ continue;
+
+ if ((ea->id.b24 & 0xffff00) == (fcport->d_id.b24 & 0xffff00)) {
+ fcport->scan_needed = 1;
+ fcport->rscn_gen++;
+ }
+ }
+ break;
+ case RSCN_DOM_ADDR:
+ list_for_each_entry(fcport, &vha->vp_fcports, list) {
+ if (fcport->flags & FCF_FCP2_DEVICE)
+ continue;
+
+ if ((ea->id.b24 & 0xff0000) == (fcport->d_id.b24 & 0xff0000)) {
+ fcport->scan_needed = 1;
+ fcport->rscn_gen++;
+ }
+ }
+ break;
+ case RSCN_FAB_ADDR:
+ default:
+ list_for_each_entry(fcport, &vha->vp_fcports, list) {
+ if (fcport->flags & FCF_FCP2_DEVICE)
+ continue;
+
+ fcport->scan_needed = 1;
+ fcport->rscn_gen++;
+ }
+ break;
}
spin_lock_irqsave(&vha->work_lock, flags);
* fw shal not send PRLI after PLOGI Acc
*/
if (ha->flags.edif_enabled &&
- vha->e_dbell.db_flags & EDB_ACTIVE) {
+ DBELL_ACTIVE(vha)) {
ha->fw_options[3] |= BIT_15;
ha->flags.n2n_fw_acc_sec = 1;
} else {
(ha->flags.fawwpn_enabled) ? "enabled" : "disabled");
}
+ /* ELS pass through payload is limit by frame size. */
+ if (ha->flags.edif_enabled)
+ mid_init_cb->init_cb.frame_payload_size = cpu_to_le16(ELS_MAX_PAYLOAD);
+
rval = qla2x00_init_firmware(vha, ha->init_cb_size);
next_check:
if (rval) {
"LOOP READY.\n");
ha->flags.fw_init_done = 1;
- if (ha->flags.edif_enabled &&
- !(vha->e_dbell.db_flags & EDB_ACTIVE) &&
- N2N_TOPO(vha->hw)) {
- /*
- * use port online to wake up app to get ready
- * for authentication
- */
- qla2x00_post_aen_work(vha, FCH_EVT_PORT_ONLINE, 0);
- }
+ /*
+ * use link up to wake up app to get ready for
+ * authentication.
+ */
+ if (ha->flags.edif_enabled && DBELL_INACTIVE(vha))
+ qla2x00_post_aen_work(vha, FCH_EVT_LINKUP,
+ ha->link_data_rate);
/*
* Process any ATIO queue entries that came in
qla2x00_update_fcport(fcport->vha, fcport);
+ ql_dbg(ql_dbg_disc, fcport->vha, 0x911e,
+ "%s rscn gen %d/%d next DS %d\n", __func__,
+ rscn_gen, fcport->rscn_gen, fcport->next_disc_state);
+
if (rscn_gen != fcport->rscn_gen) {
/* RSCN(s) came in while registration */
switch (fcport->next_disc_state) {
ha->chip_reset++;
ha->base_qpair->chip_reset = ha->chip_reset;
ha->base_qpair->cmd_cnt = ha->base_qpair->cmd_completion_cnt = 0;
+ ha->base_qpair->prev_completion_cnt = 0;
for (i = 0; i < ha->max_qpairs; i++) {
if (ha->queue_pair_map[i]) {
ha->queue_pair_map[i]->chip_reset =
ha->base_qpair->chip_reset;
ha->queue_pair_map[i]->cmd_cnt =
ha->queue_pair_map[i]->cmd_completion_cnt = 0;
+ ha->base_qpair->prev_completion_cnt = 0;
}
}
elsio->u.els_plogi.els_cmd = els_opcode;
elsio->u.els_plogi.els_plogi_pyld->opcode = els_opcode;
- if (els_opcode == ELS_DCMD_PLOGI && vha->hw->flags.edif_enabled &&
- vha->e_dbell.db_flags & EDB_ACTIVE) {
+ if (els_opcode == ELS_DCMD_PLOGI && DBELL_ACTIVE(vha)) {
struct fc_els_flogi *p = ptr;
p->fl_csp.sp_features |= cpu_to_be16(FC_SP_FT_SEC);
}
} else if (comp_status == CS_PORT_LOGGED_OUT) {
+ ql_dbg(ql_dbg_disc, vha, 0x911e,
+ "%s %d schedule session deletion\n",
+ __func__, __LINE__);
+
els->u.els_plogi.len = 0;
res = DID_IMM_RETRY << 16;
qlt_schedule_sess_for_deletion(sp->fcport);
mcp->in_mb |= MBX_13|MBX_12|MBX_11|MBX_10;
if (IS_FWI2_CAPABLE(vha->hw))
mcp->in_mb |= MBX_19|MBX_18|MBX_17|MBX_16;
- if (IS_QLA27XX(vha->hw) || IS_QLA28XX(vha->hw)) {
- mcp->in_mb |= MBX_15;
- mcp->out_mb |= MBX_7|MBX_21|MBX_22|MBX_23;
- }
+ if (IS_QLA27XX(vha->hw) || IS_QLA28XX(vha->hw))
+ mcp->in_mb |= MBX_15|MBX_21|MBX_22|MBX_23;
mcp->tov = MBX_TOV_SECONDS;
mcp->flags = 0;
fc_port_t *fcport = sp->fcport;
struct scsi_qla_host *vha = fcport->vha;
struct qla_hw_data *ha = vha->hw;
- struct req_que *req = vha->req;
+ struct req_que *req;
struct qla_qpair *qpair = sp->qpair;
ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x108c,
"Failed %s %x\n", __func__, rval);
}
}
+
+int qla_mailbox_passthru(scsi_qla_host_t *vha,
+ uint16_t *mbx_in, uint16_t *mbx_out)
+{
+ mbx_cmd_t mc;
+ mbx_cmd_t *mcp = &mc;
+ int rval = -EINVAL;
+
+ memset(&mc, 0, sizeof(mc));
+ /* Receiving all 32 register's contents */
+ memcpy(&mcp->mb, (char *)mbx_in, (32 * sizeof(uint16_t)));
+
+ mcp->out_mb = 0xFFFFFFFF;
+ mcp->in_mb = 0xFFFFFFFF;
+
+ mcp->tov = MBX_TOV_SECONDS;
+ mcp->flags = 0;
+ mcp->bufp = NULL;
+
+ rval = qla2x00_mailbox_command(vha, mcp);
+
+ if (rval != QLA_SUCCESS) {
+ ql_dbg(ql_dbg_mbx, vha, 0xf0a2,
+ "Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]);
+ } else {
+ ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0xf0a3, "Done %s.\n",
+ __func__);
+ /* passing all 32 register's contents */
+ memcpy(mbx_out, &mcp->mb, 32 * sizeof(uint16_t));
+ }
+
+ return rval;
+}
}
int
-qlafx00_loop_reset(scsi_qla_host_t *vha)
-{
- int ret;
- struct fc_port *fcport;
- struct qla_hw_data *ha = vha->hw;
-
- if (ql2xtargetreset) {
- list_for_each_entry(fcport, &vha->vp_fcports, list) {
- if (fcport->port_type != FCT_TARGET)
- continue;
-
- ret = ha->isp_ops->target_reset(fcport, 0, 0);
- if (ret != QLA_SUCCESS) {
- ql_dbg(ql_dbg_taskm, vha, 0x803d,
- "Bus Reset failed: Reset=%d "
- "d_id=%x.\n", ret, fcport->d_id.b24);
- }
- }
- }
- return QLA_SUCCESS;
-}
-
-int
qlafx00_iospace_config(struct qla_hw_data *ha)
{
if (pci_request_selected_regions(ha->pdev, ha->bars,
fc_port_t *fcport = sp->fcport;
struct qla_hw_data *ha = fcport->vha->hw;
int rval, abts_done_called = 1;
+ bool io_wait_for_abort_done;
+ uint32_t handle;
ql_dbg(ql_dbg_io, fcport->vha, 0xffff,
"%s called for sp=%p, hndl=%x on fcport=%p desc=%p deleted=%d\n",
goto out;
}
+ /*
+ * sp may not be valid after abort_command if return code is either
+ * SUCCESS or ERR_FROM_FW codes, so cache the value here.
+ */
+ io_wait_for_abort_done = ql2xabts_wait_nvme &&
+ QLA_ABTS_WAIT_ENABLED(sp);
+ handle = sp->handle;
+
rval = ha->isp_ops->abort_command(sp);
ql_dbg(ql_dbg_io, fcport->vha, 0x212b,
"%s: %s command for sp=%p, handle=%x on fcport=%p rval=%x\n",
__func__, (rval != QLA_SUCCESS) ? "Failed to abort" : "Aborted",
- sp, sp->handle, fcport, rval);
+ sp, handle, fcport, rval);
/*
* If async tmf is enabled, the abort callback is called only on
* are waited until ABTS complete. This kref is decreased
* at qla24xx_abort_sp_done function.
*/
- if (abts_done_called && ql2xabts_wait_nvme && QLA_ABTS_WAIT_ENABLED(sp))
+ if (abts_done_called && io_wait_for_abort_done)
return;
out:
/* kref_get was done before work was schedule. */
uint16_t avail_dsds;
struct dsd64 *cur_dsd;
struct req_que *req = NULL;
+ struct rsp_que *rsp = NULL;
struct scsi_qla_host *vha = sp->fcport->vha;
struct qla_hw_data *ha = vha->hw;
struct qla_qpair *qpair = sp->qpair;
/* Setup qpair pointers */
req = qpair->req;
+ rsp = qpair->rsp;
tot_dsds = fd->sg_cnt;
/* Acquire qpair specific lock */
/* Set chip new ring index. */
wrt_reg_dword(req->req_q_in, req->ring_index);
+ if (vha->flags.process_response_queue &&
+ rsp->ring_ptr->signature != RESPONSE_PROCESSED)
+ qla24xx_process_response_queue(vha, rsp);
+
queuing_error:
spin_unlock_irqrestore(&qpair->qp_lock, flags);
" 0 -- Regular doorbell.\n"
" 1 -- CAMRAM doorbell (faster).\n");
-int ql2xtargetreset = 1;
-module_param(ql2xtargetreset, int, S_IRUGO);
-MODULE_PARM_DESC(ql2xtargetreset,
- "Enable target reset."
- "Default is 1 - use hw defaults.");
-
int ql2xgffidenable;
module_param(ql2xgffidenable, int, S_IRUGO);
MODULE_PARM_DESC(ql2xgffidenable,
sp->free(sp);
cmd->result = res;
CMD_SP(cmd) = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
if (comp)
complete(comp);
}
sp->free(sp);
cmd->result = res;
CMD_SP(cmd) = NULL;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
if (comp)
complete(comp);
}
return SCSI_MLQUEUE_TARGET_BUSY;
qc24_fail_command:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
return SCSI_MLQUEUE_TARGET_BUSY;
qc24_fail_command:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
uint32_t ratov_j;
struct qla_qpair *qpair;
unsigned long flags;
+ int fast_fail_status = SUCCESS;
if (qla2x00_isp_reg_stat(ha)) {
ql_log(ql_log_info, vha, 0x8042,
return FAILED;
}
+ /* Save any FAST_IO_FAIL value to return later if abort succeeds */
ret = fc_block_scsi_eh(cmd);
if (ret != 0)
- return ret;
+ fast_fail_status = ret;
sp = scsi_cmd_priv(cmd);
qpair = sp->qpair;
vha->cmd_timeout_cnt++;
if ((sp->fcport && sp->fcport->deleted) || !qpair)
- return SUCCESS;
+ return fast_fail_status != SUCCESS ? fast_fail_status : FAILED;
spin_lock_irqsave(qpair->qp_lock_ptr, flags);
sp->comp = ∁
__func__, ha->r_a_tov/10);
ret = FAILED;
} else {
- ret = SUCCESS;
+ ret = fast_fail_status;
}
break;
default:
qla2x00_loop_reset(scsi_qla_host_t *vha)
{
int ret;
- struct fc_port *fcport;
struct qla_hw_data *ha = vha->hw;
- if (IS_QLAFX00(ha)) {
- return qlafx00_loop_reset(vha);
- }
-
- if (ql2xtargetreset == 1 && ha->flags.enable_target_reset) {
- list_for_each_entry(fcport, &vha->vp_fcports, list) {
- if (fcport->port_type != FCT_TARGET)
- continue;
-
- ret = ha->isp_ops->target_reset(fcport, 0, 0);
- if (ret != QLA_SUCCESS) {
- ql_dbg(ql_dbg_taskm, vha, 0x802c,
- "Bus Reset failed: Reset=%d "
- "d_id=%x.\n", ret, fcport->d_id.b24);
- }
- }
- }
-
+ if (IS_QLAFX00(ha))
+ return QLA_SUCCESS;
if (ha->flags.enable_lip_full_login && !IS_CNA_CAPABLE(ha)) {
atomic_set(&vha->loop_state, LOOP_DOWN);
return atomic_read(&vha->loop_state) == LOOP_READY;
}
+static void qla_heartbeat_work_fn(struct work_struct *work)
+{
+ struct qla_hw_data *ha = container_of(work,
+ struct qla_hw_data, heartbeat_work);
+ struct scsi_qla_host *base_vha = pci_get_drvdata(ha->pdev);
+
+ if (!ha->flags.mbox_busy && base_vha->flags.init_done)
+ qla_no_op_mb(base_vha);
+}
+
static void qla2x00_iocb_work_fn(struct work_struct *work)
{
struct scsi_qla_host *vha = container_of(work,
host->transportt, sht->vendor_id);
INIT_WORK(&base_vha->iocb_work, qla2x00_iocb_work_fn);
+ INIT_WORK(&ha->heartbeat_work, qla_heartbeat_work_fn);
/* Set up the irqs */
ret = qla2x00_request_irqs(ha, rsp);
host->can_queue, base_vha->req,
base_vha->mgmt_svr_loop_id, host->sg_tablesize);
+ /* Check if FW supports MQ or not for ISP25xx */
+ if (IS_QLA25XX(ha) && !(ha->fw_attributes & BIT_6))
+ ha->mqenable = 0;
+
if (ha->mqenable) {
bool startit = false;
static inline void
qla24xx_free_purex_list(struct purex_list *list)
{
- struct list_head *item, *next;
+ struct purex_item *item, *next;
ulong flags;
spin_lock_irqsave(&list->lock, flags);
- list_for_each_safe(item, next, &list->head) {
- list_del(item);
- kfree(list_entry(item, struct purex_item, list));
+ list_for_each_entry_safe(item, next, &list->head, list) {
+ list_del(&item->list);
+ kfree(item);
}
spin_unlock_irqrestore(&list->lock, flags);
}
/* allocate the purex dma pool */
ha->purex_dma_pool = dma_pool_create(name, &ha->pdev->dev,
- MAX_PAYLOAD, 8, 0);
+ ELS_MAX_PAYLOAD, 8, 0);
if (!ha->purex_dma_pool) {
ql_dbg_pci(ql_dbg_init, ha->pdev, 0x011b,
qla2x00_lip_reset(base_vha);
}
- if (test_bit(HEARTBEAT_CHK, &base_vha->dpc_flags)) {
- /*
- * if there is a mb in progress then that's
- * enough of a check to see if fw is still ticking.
- */
- if (!ha->flags.mbox_busy && base_vha->flags.init_done)
- qla_no_op_mb(base_vha);
-
- clear_bit(HEARTBEAT_CHK, &base_vha->dpc_flags);
- }
-
ha->dpc_active = 0;
end_loop:
set_current_state(TASK_INTERRUPTIBLE);
static bool qla_do_heartbeat(struct scsi_qla_host *vha)
{
- u64 cmd_cnt, prev_cmd_cnt;
- bool do_hb = false;
struct qla_hw_data *ha = vha->hw;
- int i;
+ u32 cmpl_cnt;
+ u16 i;
+ bool do_heartbeat = false;
- /* if cmds are still pending down in fw, then do hb */
- if (ha->base_qpair->cmd_cnt != ha->base_qpair->cmd_completion_cnt) {
- do_hb = true;
+ /*
+ * Allow do_heartbeat only if we don’t have any active interrupts,
+ * but there are still IOs outstanding with firmware.
+ */
+ cmpl_cnt = ha->base_qpair->cmd_completion_cnt;
+ if (cmpl_cnt == ha->base_qpair->prev_completion_cnt &&
+ cmpl_cnt != ha->base_qpair->cmd_cnt) {
+ do_heartbeat = true;
goto skip;
}
+ ha->base_qpair->prev_completion_cnt = cmpl_cnt;
for (i = 0; i < ha->max_qpairs; i++) {
- if (ha->queue_pair_map[i] &&
- ha->queue_pair_map[i]->cmd_cnt !=
- ha->queue_pair_map[i]->cmd_completion_cnt) {
- do_hb = true;
- break;
+ if (ha->queue_pair_map[i]) {
+ cmpl_cnt = ha->queue_pair_map[i]->cmd_completion_cnt;
+ if (cmpl_cnt == ha->queue_pair_map[i]->prev_completion_cnt &&
+ cmpl_cnt != ha->queue_pair_map[i]->cmd_cnt) {
+ do_heartbeat = true;
+ break;
+ }
+ ha->queue_pair_map[i]->prev_completion_cnt = cmpl_cnt;
}
}
skip:
- prev_cmd_cnt = ha->prev_cmd_cnt;
- cmd_cnt = ha->base_qpair->cmd_cnt;
- for (i = 0; i < ha->max_qpairs; i++) {
- if (ha->queue_pair_map[i])
- cmd_cnt += ha->queue_pair_map[i]->cmd_cnt;
- }
- ha->prev_cmd_cnt = cmd_cnt;
-
- if (!do_hb && ((cmd_cnt - prev_cmd_cnt) > 50))
- /*
- * IOs are completing before periodic hb check.
- * IOs seems to be running, do hb for sanity check.
- */
- do_hb = true;
-
- return do_hb;
+ return do_heartbeat;
}
static void qla_heart_beat(struct scsi_qla_host *vha)
{
+ struct qla_hw_data *ha = vha->hw;
+
if (vha->vp_idx)
return;
if (vha->hw->flags.eeh_busy || qla2x00_chip_is_down(vha))
return;
- if (qla_do_heartbeat(vha)) {
- set_bit(HEARTBEAT_CHK, &vha->dpc_flags);
- qla2xxx_wake_dpc(vha);
- }
+ if (qla_do_heartbeat(vha))
+ queue_work(ha->wq, &ha->heartbeat_work);
}
/**************************************************************************
.sg_tablesize = SG_ALL,
.max_sectors = 0xFFFF,
- .shost_attrs = qla2x00_host_attrs,
+ .shost_groups = qla2x00_host_groups,
.supported_mode = MODE_INITIATOR,
.track_queue_depth = 1,
if (ql2xextended_error_logging == 1)
ql2xextended_error_logging = QL_DBG_DEFAULT1_MASK;
- if (ql2x_ini_mode == QLA2XXX_INI_MODE_DUAL)
- qla_insert_tgt_attrs();
-
qla2xxx_transport_template =
fc_attach_transport(&qla2xxx_transport_functions);
if (!qla2xxx_transport_template) {
"%s bypassing release_all_sadb\n",
__func__);
}
+ qla_edif_clear_appdata(vha, sess);
qla_edif_sess_down(vha, sess);
}
qla2x00_mark_device_lost(vha, sess, 0);
}
if (vha->hw->flags.edif_enabled) {
- if (!(vha->e_dbell.db_flags & EDB_ACTIVE)) {
+ if (DBELL_INACTIVE(vha)) {
ql_dbg(ql_dbg_disc, vha, 0xffff,
"%s %d Term INOT due to app not started lid=%d, NportID %06X ",
__func__, __LINE__, loop_id, port_id.b24);
/*
* Driver version
*/
-#define QLA2XXX_VERSION "10.02.06.200-k"
+#define QLA2XXX_VERSION "10.02.07.200-k"
#define QLA_DRIVER_MAJOR_VER 10
#define QLA_DRIVER_MINOR_VER 2
-#define QLA_DRIVER_PATCH_VER 6
+#define QLA_DRIVER_PATCH_VER 7
#define QLA_DRIVER_BETA_VER 200
/* End items for tcm_qla2xxx_tpg_attrib_cit */
-static ssize_t tcm_qla2xxx_tpg_enable_show(struct config_item *item,
- char *page)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct tcm_qla2xxx_tpg *tpg = container_of(se_tpg,
- struct tcm_qla2xxx_tpg, se_tpg);
-
- return snprintf(page, PAGE_SIZE, "%d\n",
- atomic_read(&tpg->lport_tpg_enabled));
-}
-
-static ssize_t tcm_qla2xxx_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
+static int tcm_qla2xxx_enable_tpg(struct se_portal_group *se_tpg,
+ bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct se_wwn *se_wwn = se_tpg->se_tpg_wwn;
struct tcm_qla2xxx_lport *lport = container_of(se_wwn,
struct tcm_qla2xxx_lport, lport_wwn);
struct scsi_qla_host *vha = lport->qla_vha;
struct tcm_qla2xxx_tpg *tpg = container_of(se_tpg,
struct tcm_qla2xxx_tpg, se_tpg);
- unsigned long op;
- int rc;
- rc = kstrtoul(page, 0, &op);
- if (rc < 0) {
- pr_err("kstrtoul() returned %d\n", rc);
- return -EINVAL;
- }
- if ((op != 1) && (op != 0)) {
- pr_err("Illegal value for tpg_enable: %lu\n", op);
- return -EINVAL;
- }
- if (op) {
+ if (enable) {
if (atomic_read(&tpg->lport_tpg_enabled))
return -EEXIST;
qlt_enable_vha(vha);
} else {
if (!atomic_read(&tpg->lport_tpg_enabled))
- return count;
+ return 0;
atomic_set(&tpg->lport_tpg_enabled, 0);
qlt_stop_phase1(vha->vha_tgt.qla_tgt);
qlt_stop_phase2(vha->vha_tgt.qla_tgt);
}
- return count;
+ return 0;
}
static ssize_t tcm_qla2xxx_tpg_dynamic_sessions_show(struct config_item *item,
return sprintf(page, "%d\n", tpg->tpg_attrib.fabric_prot_type);
}
-CONFIGFS_ATTR(tcm_qla2xxx_tpg_, enable);
CONFIGFS_ATTR_RO(tcm_qla2xxx_tpg_, dynamic_sessions);
CONFIGFS_ATTR(tcm_qla2xxx_tpg_, fabric_prot_type);
static struct configfs_attribute *tcm_qla2xxx_tpg_attrs[] = {
- &tcm_qla2xxx_tpg_attr_enable,
&tcm_qla2xxx_tpg_attr_dynamic_sessions,
&tcm_qla2xxx_tpg_attr_fabric_prot_type,
NULL,
kfree(tpg);
}
-static ssize_t tcm_qla2xxx_npiv_tpg_enable_show(struct config_item *item,
- char *page)
-{
- return tcm_qla2xxx_tpg_enable_show(item, page);
-}
-
-static ssize_t tcm_qla2xxx_npiv_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
+static int tcm_qla2xxx_npiv_enable_tpg(struct se_portal_group *se_tpg,
+ bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct se_wwn *se_wwn = se_tpg->se_tpg_wwn;
struct tcm_qla2xxx_lport *lport = container_of(se_wwn,
struct tcm_qla2xxx_lport, lport_wwn);
struct scsi_qla_host *vha = lport->qla_vha;
struct tcm_qla2xxx_tpg *tpg = container_of(se_tpg,
struct tcm_qla2xxx_tpg, se_tpg);
- unsigned long op;
- int rc;
- rc = kstrtoul(page, 0, &op);
- if (rc < 0) {
- pr_err("kstrtoul() returned %d\n", rc);
- return -EINVAL;
- }
- if ((op != 1) && (op != 0)) {
- pr_err("Illegal value for tpg_enable: %lu\n", op);
- return -EINVAL;
- }
- if (op) {
+ if (enable) {
if (atomic_read(&tpg->lport_tpg_enabled))
return -EEXIST;
qlt_enable_vha(vha);
} else {
if (!atomic_read(&tpg->lport_tpg_enabled))
- return count;
+ return 0;
atomic_set(&tpg->lport_tpg_enabled, 0);
qlt_stop_phase1(vha->vha_tgt.qla_tgt);
qlt_stop_phase2(vha->vha_tgt.qla_tgt);
}
- return count;
+ return 0;
}
-CONFIGFS_ATTR(tcm_qla2xxx_npiv_tpg_, enable);
-
-static struct configfs_attribute *tcm_qla2xxx_npiv_tpg_attrs[] = {
- &tcm_qla2xxx_npiv_tpg_attr_enable,
- NULL,
-};
-
static struct se_portal_group *tcm_qla2xxx_npiv_make_tpg(struct se_wwn *wwn,
const char *name)
{
.fabric_make_wwn = tcm_qla2xxx_make_lport,
.fabric_drop_wwn = tcm_qla2xxx_drop_lport,
.fabric_make_tpg = tcm_qla2xxx_make_tpg,
+ .fabric_enable_tpg = tcm_qla2xxx_enable_tpg,
.fabric_drop_tpg = tcm_qla2xxx_drop_tpg,
.fabric_init_nodeacl = tcm_qla2xxx_init_nodeacl,
.fabric_make_wwn = tcm_qla2xxx_npiv_make_lport,
.fabric_drop_wwn = tcm_qla2xxx_npiv_drop_lport,
.fabric_make_tpg = tcm_qla2xxx_npiv_make_tpg,
+ .fabric_enable_tpg = tcm_qla2xxx_npiv_enable_tpg,
.fabric_drop_tpg = tcm_qla2xxx_drop_tpg,
.fabric_init_nodeacl = tcm_qla2xxx_init_nodeacl,
.tfc_wwn_attrs = tcm_qla2xxx_wwn_attrs,
- .tfc_tpg_base_attrs = tcm_qla2xxx_npiv_tpg_attrs,
};
static int tcm_qla2xxx_register_configfs(void)
static DEVICE_ATTR(fw_load_src, S_IRUGO, qla4xxx_fw_load_src_show, NULL);
static DEVICE_ATTR(fw_uptime, S_IRUGO, qla4xxx_fw_uptime_show, NULL);
-struct device_attribute *qla4xxx_host_attrs[] = {
- &dev_attr_fw_version,
- &dev_attr_serial_num,
- &dev_attr_iscsi_version,
- &dev_attr_optrom_version,
- &dev_attr_board_id,
- &dev_attr_fw_state,
- &dev_attr_phy_port_cnt,
- &dev_attr_phy_port_num,
- &dev_attr_iscsi_func_cnt,
- &dev_attr_hba_model,
- &dev_attr_fw_timestamp,
- &dev_attr_fw_build_user,
- &dev_attr_fw_ext_timestamp,
- &dev_attr_fw_load_src,
- &dev_attr_fw_uptime,
+static struct attribute *qla4xxx_host_attrs[] = {
+ &dev_attr_fw_version.attr,
+ &dev_attr_serial_num.attr,
+ &dev_attr_iscsi_version.attr,
+ &dev_attr_optrom_version.attr,
+ &dev_attr_board_id.attr,
+ &dev_attr_fw_state.attr,
+ &dev_attr_phy_port_cnt.attr,
+ &dev_attr_phy_port_num.attr,
+ &dev_attr_iscsi_func_cnt.attr,
+ &dev_attr_hba_model.attr,
+ &dev_attr_fw_timestamp.attr,
+ &dev_attr_fw_build_user.attr,
+ &dev_attr_fw_ext_timestamp.attr,
+ &dev_attr_fw_load_src.attr,
+ &dev_attr_fw_uptime.attr,
NULL,
};
+
+static const struct attribute_group qla4xxx_host_attr_group = {
+ .attrs = qla4xxx_host_attrs
+};
+
+const struct attribute_group *qla4xxx_host_groups[] = {
+ &qla4xxx_host_attr_group,
+ NULL
+};
extern int ql4xmdcapmask;
extern int ql4xenablemd;
-extern struct device_attribute *qla4xxx_host_attrs[];
+extern const struct attribute_group *qla4xxx_host_groups[];
+
#endif /* _QLA4x_GBL_H */
.sg_tablesize = SG_ALL,
.max_sectors = 0xFFFF,
- .shost_attrs = qla4xxx_host_attrs,
+ .shost_groups = qla4xxx_host_groups,
.host_reset = qla4xxx_host_reset,
.vendor_id = SCSI_NL_VID_TYPE_PCI | PCI_VENDOR_ID_QLOGIC,
};
mempool_free(srb, ha->srb_mempool);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
/**
return SCSI_MLQUEUE_HOST_BUSY;
qc_fail_command:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
* If result is CHECK CONDITION done calls qcommand to request
* sense
*/
- (icmd->scsi_done) (icmd);
+ scsi_done(icmd);
}
irqreturn_t qlogicfas408_ihandl(int irq, void *dev_id)
* Queued command
*/
-static int qlogicfas408_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done) (struct scsi_cmnd *))
+static int qlogicfas408_queuecommand_lck(struct scsi_cmnd *cmd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct qlogicfas408_priv *priv = get_priv_by_cmd(cmd);
set_host_byte(cmd, DID_OK);
return 0;
}
- cmd->scsi_done = done;
/* wait for the last command's interrupt to finish */
while (priv->qlcmd != NULL) {
barrier();
*
* "This code must fly." -davem
*/
-static int qlogicpti_queuecommand_lck(struct scsi_cmnd *Cmnd, void (*done)(struct scsi_cmnd *))
+static int qlogicpti_queuecommand_lck(struct scsi_cmnd *Cmnd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct Scsi_Host *host = Cmnd->device->host;
struct qlogicpti *qpti = (struct qlogicpti *) host->hostdata;
struct Command_Entry *cmd;
u_int out_ptr;
int in_ptr;
- Cmnd->scsi_done = done;
-
in_ptr = qpti->req_in_ptr;
cmd = (struct Command_Entry *) &qpti->req_cpu[in_ptr];
out_ptr = sbus_readw(qpti->qregs + MBOX4);
struct scsi_cmnd *next;
next = (struct scsi_cmnd *) dq->host_scribble;
- dq->scsi_done(dq);
+ scsi_done(dq);
dq = next;
} while (dq != NULL);
}
EXPORT_SYMBOL(scsi_logging_level);
#endif
-/*
- * Domain for asynchronous system resume operations. It is marked 'exclusive'
- * to avoid being included in the async_synchronize_full() that is invoked by
- * dpm_resume().
- */
-ASYNC_DOMAIN_EXCLUSIVE(scsi_sd_pm_domain);
-EXPORT_SYMBOL(scsi_sd_pm_domain);
-
#ifdef CONFIG_SCSI_LOGGING
void scsi_log_send(struct scsi_cmnd *cmd)
{
{
unsigned char *cmd = scp->cmnd;
unsigned char arr[SDEBUG_READCAP16_ARR_SZ];
- int alloc_len;
+ u32 alloc_len;
alloc_len = get_unaligned_be32(cmd + 10);
/* following just in case virtual_gb changed */
}
return fill_from_dev_buffer(scp, arr,
- min_t(int, alloc_len, SDEBUG_READCAP16_ARR_SZ));
+ min_t(u32, alloc_len, SDEBUG_READCAP16_ARR_SZ));
}
#define SDEBUG_MAX_TGTPGS_ARR_SZ 1412
unsigned char *cmd = scp->cmnd;
unsigned char *arr;
int host_no = devip->sdbg_host->shost->host_no;
- int n, ret, alen, rlen;
int port_group_a, port_group_b, port_a, port_b;
+ u32 alen, n, rlen;
+ int ret;
alen = get_unaligned_be32(cmd + 6);
arr = kzalloc(SDEBUG_MAX_TGTPGS_ARR_SZ, GFP_ATOMIC);
* - The constructed command length
* - The maximum array size
*/
- rlen = min_t(int, alen, n);
+ rlen = min(alen, n);
ret = fill_from_dev_buffer(scp, arr,
- min_t(int, rlen, SDEBUG_MAX_TGTPGS_ARR_SZ));
+ min_t(u32, rlen, SDEBUG_MAX_TGTPGS_ARR_SZ));
kfree(arr);
return ret;
}
mk_sense_invalid_opcode(scp);
return check_condition_result;
}
+ if (vnum == 0)
+ return 0; /* not an error */
a_num = is_bytchk3 ? 1 : vnum;
/* Treat following check like one for read (i.e. no write) access */
ret = check_device_access_params(scp, lba, a_num, false);
}
zs_lba = get_unaligned_be64(cmd + 2);
alloc_len = get_unaligned_be32(cmd + 10);
+ if (alloc_len == 0)
+ return 0; /* not an error */
rep_opts = cmd[14] & 0x3f;
partial = cmd[14] & 0x80;
pr_info("bypassing scsi_done() due to aborted cmd\n");
return;
}
- scp->scsi_done(scp); /* callback to mid level */
+ scsi_done(scp); /* callback to mid level */
}
/* When high resolution timer goes off this function is called. */
if (new_sd_dp)
kfree(sd_dp);
/* call scsi_done() from this thread */
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
/* otherwise reduce kt by elapsed time */
cmnd->result &= ~SDEG_RES_IMMED_MASK;
if (cmnd->result == 0 && scsi_result != 0)
cmnd->result = scsi_result;
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
}
sd_dp->defer_t = SDEB_DEFER_NONE;
spin_unlock_irqrestore(&sqp->qc_lock, iflags);
- scp->scsi_done(scp); /* callback to mid level */
+ scsi_done(scp); /* callback to mid level */
spin_lock_irqsave(&sqp->qc_lock, iflags);
num_entries++;
}
#include <asm/unaligned.h>
-static void scsi_eh_done(struct scsi_cmnd *scmd);
-
/*
* These should *probably* be handled by the host itself.
* Since it is allowed to sleep, it probably should.
return true;
}
+static void scsi_eh_complete_abort(struct scsi_cmnd *scmd, struct Scsi_Host *shost)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(shost->host_lock, flags);
+ list_del_init(&scmd->eh_entry);
+ /*
+ * If the abort succeeds, and there is no further
+ * EH action, clear the ->last_reset time.
+ */
+ if (list_empty(&shost->eh_abort_list) &&
+ list_empty(&shost->eh_cmd_q))
+ if (shost->eh_deadline != -1)
+ shost->last_reset = 0;
+ spin_unlock_irqrestore(shost->host_lock, flags);
+}
+
/**
* scmd_eh_abort_handler - Handle command aborts
* @work: command to be aborted.
container_of(work, struct scsi_cmnd, abort_work.work);
struct scsi_device *sdev = scmd->device;
enum scsi_disposition rtn;
+ unsigned long flags;
if (scsi_host_eh_past_deadline(sdev->host)) {
SCSI_LOG_ERROR_RECOVERY(3,
SCSI_LOG_ERROR_RECOVERY(3,
scmd_printk(KERN_WARNING, scmd,
"retry aborted command\n"));
+ scsi_eh_complete_abort(scmd, sdev->host);
scsi_queue_insert(scmd, SCSI_MLQUEUE_EH_RETRY);
return;
} else {
SCSI_LOG_ERROR_RECOVERY(3,
scmd_printk(KERN_WARNING, scmd,
"finish aborted command\n"));
+ scsi_eh_complete_abort(scmd, sdev->host);
scsi_finish_command(scmd);
return;
}
}
}
+ spin_lock_irqsave(sdev->host->host_lock, flags);
+ list_del_init(&scmd->eh_entry);
+ spin_unlock_irqrestore(sdev->host->host_lock, flags);
scsi_eh_scmd_add(scmd);
}
spin_lock_irqsave(shost->host_lock, flags);
if (shost->eh_deadline != -1 && !shost->last_reset)
shost->last_reset = jiffies;
+ BUG_ON(!list_empty(&scmd->eh_entry));
+ list_add_tail(&scmd->eh_entry, &shost->eh_abort_list);
spin_unlock_irqrestore(shost->host_lock, flags);
scmd->eh_eflags |= SCSI_EH_ABORT_SCHEDULED;
/* handler does not care. Drop down to default handling */
}
- if (scmd->cmnd[0] == TEST_UNIT_READY && scmd->scsi_done != scsi_eh_done)
+ if (scmd->cmnd[0] == TEST_UNIT_READY &&
+ scmd->submitter != SUBMITTED_BY_SCSI_ERROR_HANDLER)
/*
* nasty: for mid-layer issued TURs, we need to return the
* actual sense data without any recovery attempt. For eh
* scsi_eh_done - Completion function for error handling.
* @scmd: Cmd that is done.
*/
-static void scsi_eh_done(struct scsi_cmnd *scmd)
+void scsi_eh_done(struct scsi_cmnd *scmd)
{
struct completion *eh_action;
shost->eh_action = &done;
scsi_log_send(scmd);
- scmd->scsi_done = scsi_eh_done;
+ scmd->submitter = SUBMITTED_BY_SCSI_ERROR_HANDLER;
/*
* Lock sdev->state_mutex to avoid that scsi_device_quiesce() can
if (rtn) {
if (timeleft > stall_for) {
scsi_eh_restore_cmnd(scmd, &ses);
+
timeleft -= stall_for;
msleep(jiffies_to_msecs(stall_for));
goto retry;
}
EXPORT_SYMBOL(scsi_report_device_reset);
-static void
-scsi_reset_provider_done_command(struct scsi_cmnd *scmd)
-{
-}
-
/**
* scsi_ioctl_reset: explicitly reset a host/bus/target/device
* @dev: scsi_device to operate on
scsi_init_command(dev, scmd);
scmd->cmnd = scsi_req(rq)->cmd;
- scmd->scsi_done = scsi_reset_provider_done_command;
+ scmd->submitter = SUBMITTED_BY_SCSI_RESET_IOCTL;
memset(&scmd->sdb, 0, sizeof(scmd->sdb));
scmd->cmd_len = 0;
{
struct scsi_request *req = scsi_req(rq);
+ if (hdr->cmd_len < 6)
+ return -EMSGSIZE;
if (copy_from_user(req->cmd, hdr->cmdp, hdr->cmd_len))
return -EFAULT;
if (!scsi_cmd_allowed(req->cmd, mode))
/*
* If there had been no error, but we have leftover bytes in the
- * requeues just queue the command up again.
+ * request just queue the command up again.
*/
if (likely(result == 0))
scsi_io_completion_reprep(cmd, q);
cmd->sense_buffer = buf;
cmd->prot_sdb = prot;
cmd->flags = flags;
+ INIT_LIST_HEAD(&cmd->eh_entry);
INIT_DELAYED_WORK(&cmd->abort_work, scmd_eh_abort_handler);
cmd->jiffies_at_alloc = jiffies_at_alloc;
cmd->retries = retries;
}
cmd->cmd_len = scsi_req(req)->cmd_len;
- if (cmd->cmd_len == 0)
- cmd->cmd_len = scsi_command_size(cmd->cmnd);
cmd->cmnd = scsi_req(req)->cmd;
cmd->transfersize = blk_rq_bytes(req);
cmd->allowed = scsi_req(req)->retries;
return rtn;
done:
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
return scsi_cmd_to_driver(cmd)->init_command(cmd);
}
-static void scsi_mq_done(struct scsi_cmnd *cmd)
+void scsi_done(struct scsi_cmnd *cmd)
{
+ switch (cmd->submitter) {
+ case SUBMITTED_BY_BLOCK_LAYER:
+ break;
+ case SUBMITTED_BY_SCSI_ERROR_HANDLER:
+ return scsi_eh_done(cmd);
+ case SUBMITTED_BY_SCSI_RESET_IOCTL:
+ return;
+ }
+
if (unlikely(blk_should_fake_timeout(scsi_cmd_to_rq(cmd)->q)))
return;
if (unlikely(test_and_set_bit(SCMD_STATE_COMPLETE, &cmd->state)))
trace_scsi_dispatch_cmd_done(cmd);
blk_mq_complete_request(scsi_cmd_to_rq(cmd));
}
+EXPORT_SYMBOL(scsi_done);
static void scsi_mq_put_budget(struct request_queue *q, int budget_token)
{
scsi_set_resid(cmd, 0);
memset(cmd->sense_buffer, 0, SCSI_SENSE_BUFFERSIZE);
- cmd->scsi_done = scsi_mq_done;
+ cmd->submitter = SUBMITTED_BY_BLOCK_LAYER;
blk_mq_start_request(req);
reason = scsi_dispatch_cmd(cmd);
memset(cmd, 0, sizeof(cmd));
cmd[1] = (pf ? 0x10 : 0) | (sp ? 0x01 : 0);
- if (sdev->use_10_for_ms) {
- if (len > 65535)
+ /*
+ * Use MODE SELECT(10) if the device asked for it or if the mode page
+ * and the mode select header cannot fit within the maximumm 255 bytes
+ * of the MODE SELECT(6) command.
+ */
+ if (sdev->use_10_for_ms ||
+ len + 4 > 255 ||
+ data->block_descriptor_length > 255) {
+ if (len > 65535 - 8)
return -EINVAL;
real_buffer = kmalloc(8 + len, GFP_KERNEL);
if (!real_buffer)
real_buffer[3] = data->device_specific;
real_buffer[4] = data->longlba ? 0x01 : 0;
real_buffer[5] = 0;
- real_buffer[6] = data->block_descriptor_length >> 8;
- real_buffer[7] = data->block_descriptor_length;
+ put_unaligned_be16(data->block_descriptor_length,
+ &real_buffer[6]);
cmd[0] = MODE_SELECT_10;
- cmd[7] = len >> 8;
- cmd[8] = len;
+ put_unaligned_be16(len, &cmd[7]);
} else {
- if (len > 255 || data->block_descriptor_length > 255 ||
- data->longlba)
+ if (data->longlba)
return -EINVAL;
real_buffer = kmalloc(4 + len, GFP_KERNEL);
/**
* scsi_mode_sense - issue a mode sense, falling back from 10 to six bytes if necessary.
* @sdev: SCSI device to be queried
- * @dbd: set if mode sense will allow block descriptors to be returned
+ * @dbd: set to prevent mode sense from returning block descriptors
* @modepage: mode page being requested
* @buffer: request buffer (may not be smaller than eight bytes)
* @len: length of request buffer.
sshdr = &my_sshdr;
retry:
- use_10_for_ms = sdev->use_10_for_ms;
+ use_10_for_ms = sdev->use_10_for_ms || len > 255;
if (use_10_for_ms) {
- if (len < 8)
- len = 8;
+ if (len < 8 || len > 65535)
+ return -EINVAL;
cmd[0] = MODE_SENSE_10;
- cmd[8] = len;
+ put_unaligned_be16(len, &cmd[7]);
header_length = 8;
} else {
if (len < 4)
- len = 4;
+ return -EINVAL;
cmd[0] = MODE_SENSE;
cmd[4] = len;
if ((sshdr->sense_key == ILLEGAL_REQUEST) &&
(sshdr->asc == 0x20) && (sshdr->ascq == 0)) {
/*
- * Invalid command operation code
+ * Invalid command operation code: retry using
+ * MODE SENSE(6) if this was a MODE SENSE(10)
+ * request, except if the request mode page is
+ * too large for MODE SENSE single byte
+ * allocation length field.
*/
if (use_10_for_ms) {
+ if (len > 255)
+ return -EIO;
sdev->use_10_for_ms = 0;
goto retry;
}
data->longlba = 0;
data->block_descriptor_length = 0;
} else if (use_10_for_ms) {
- data->length = buffer[0]*256 + buffer[1] + 2;
+ data->length = get_unaligned_be16(&buffer[0]) + 2;
data->medium_type = buffer[2];
data->device_specific = buffer[3];
data->longlba = buffer[4] & 0x01;
- data->block_descriptor_length = buffer[6]*256
- + buffer[7];
+ data->block_descriptor_length = get_unaligned_be16(&buffer[6]);
} else {
data->length = buffer[0] + 1;
data->medium_type = buffer[1];
}
EXPORT_SYMBOL(scsi_target_resume);
+static int __scsi_internal_device_block_nowait(struct scsi_device *sdev)
+{
+ if (scsi_device_set_state(sdev, SDEV_BLOCK))
+ return scsi_device_set_state(sdev, SDEV_CREATED_BLOCK);
+
+ return 0;
+}
+
+void scsi_start_queue(struct scsi_device *sdev)
+{
+ if (cmpxchg(&sdev->queue_stopped, 1, 0))
+ blk_mq_unquiesce_queue(sdev->request_queue);
+}
+
+static void scsi_stop_queue(struct scsi_device *sdev, bool nowait)
+{
+ /*
+ * The atomic variable of ->queue_stopped covers that
+ * blk_mq_quiesce_queue* is balanced with blk_mq_unquiesce_queue.
+ *
+ * However, we still need to wait until quiesce is done
+ * in case that queue has been stopped.
+ */
+ if (!cmpxchg(&sdev->queue_stopped, 0, 1)) {
+ if (nowait)
+ blk_mq_quiesce_queue_nowait(sdev->request_queue);
+ else
+ blk_mq_quiesce_queue(sdev->request_queue);
+ } else {
+ if (!nowait)
+ blk_mq_wait_quiesce_done(sdev->request_queue);
+ }
+}
+
/**
* scsi_internal_device_block_nowait - try to transition to the SDEV_BLOCK state
* @sdev: device to block
*/
int scsi_internal_device_block_nowait(struct scsi_device *sdev)
{
- struct request_queue *q = sdev->request_queue;
- int err = 0;
-
- err = scsi_device_set_state(sdev, SDEV_BLOCK);
- if (err) {
- err = scsi_device_set_state(sdev, SDEV_CREATED_BLOCK);
-
- if (err)
- return err;
- }
+ int ret = __scsi_internal_device_block_nowait(sdev);
/*
* The device has transitioned to SDEV_BLOCK. Stop the
* block layer from calling the midlayer with this device's
* request queue.
*/
- blk_mq_quiesce_queue_nowait(q);
- return 0;
+ if (!ret)
+ scsi_stop_queue(sdev, true);
+ return ret;
}
EXPORT_SYMBOL_GPL(scsi_internal_device_block_nowait);
*/
static int scsi_internal_device_block(struct scsi_device *sdev)
{
- struct request_queue *q = sdev->request_queue;
int err;
mutex_lock(&sdev->state_mutex);
- err = scsi_internal_device_block_nowait(sdev);
+ err = __scsi_internal_device_block_nowait(sdev);
if (err == 0)
- blk_mq_quiesce_queue(q);
+ scsi_stop_queue(sdev, false);
mutex_unlock(&sdev->state_mutex);
return err;
}
-void scsi_start_queue(struct scsi_device *sdev)
-{
- struct request_queue *q = sdev->request_queue;
-
- blk_mq_unquiesce_queue(q);
-}
-
/**
* scsi_internal_device_unblock_nowait - resume a device after a block request
* @sdev: device to resume
const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
int err;
- /* flush pending in-flight resume operations, suspend is synchronous */
- async_synchronize_full_domain(&scsi_sd_pm_domain);
-
err = scsi_device_quiesce(to_scsi_device(dev));
if (err == 0) {
err = cb(dev, pm);
return err;
}
-static int scsi_dev_type_resume(struct device *dev,
- int (*cb)(struct device *, const struct dev_pm_ops *))
-{
- const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
- int err = 0;
-
- err = cb(dev, pm);
- scsi_device_resume(to_scsi_device(dev));
- dev_dbg(dev, "scsi resume: %d\n", err);
-
- if (err == 0) {
- pm_runtime_disable(dev);
- err = pm_runtime_set_active(dev);
- pm_runtime_enable(dev);
-
- /*
- * Forcibly set runtime PM status of request queue to "active"
- * to make sure we can again get requests from the queue
- * (see also blk_pm_peek_request()).
- *
- * The resume hook will correct runtime PM status of the disk.
- */
- if (!err && scsi_is_sdev_device(dev)) {
- struct scsi_device *sdev = to_scsi_device(dev);
-
- blk_set_runtime_active(sdev->request_queue);
- }
- }
-
- return err;
-}
-
static int
scsi_bus_suspend_common(struct device *dev,
int (*cb)(struct device *, const struct dev_pm_ops *))
{
- int err = 0;
-
- if (scsi_is_sdev_device(dev)) {
- /*
- * All the high-level SCSI drivers that implement runtime
- * PM treat runtime suspend, system suspend, and system
- * hibernate nearly identically. In all cases the requirements
- * for runtime suspension are stricter.
- */
- if (pm_runtime_suspended(dev))
- return 0;
-
- err = scsi_dev_type_suspend(dev, cb);
- }
-
- return err;
-}
-
-static void async_sdev_resume(void *dev, async_cookie_t cookie)
-{
- scsi_dev_type_resume(dev, do_scsi_resume);
-}
-
-static void async_sdev_thaw(void *dev, async_cookie_t cookie)
-{
- scsi_dev_type_resume(dev, do_scsi_thaw);
-}
+ if (!scsi_is_sdev_device(dev))
+ return 0;
-static void async_sdev_restore(void *dev, async_cookie_t cookie)
-{
- scsi_dev_type_resume(dev, do_scsi_restore);
+ return scsi_dev_type_suspend(dev, cb);
}
static int scsi_bus_resume_common(struct device *dev,
int (*cb)(struct device *, const struct dev_pm_ops *))
{
- async_func_t fn;
+ const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
+ int err;
if (!scsi_is_sdev_device(dev))
- fn = NULL;
- else if (cb == do_scsi_resume)
- fn = async_sdev_resume;
- else if (cb == do_scsi_thaw)
- fn = async_sdev_thaw;
- else if (cb == do_scsi_restore)
- fn = async_sdev_restore;
- else
- fn = NULL;
-
- if (fn) {
- async_schedule_domain(fn, dev, &scsi_sd_pm_domain);
-
- /*
- * If a user has disabled async probing a likely reason
- * is due to a storage enclosure that does not inject
- * staggered spin-ups. For safety, make resume
- * synchronous as well in that case.
- */
- if (strncmp(scsi_scan_type, "async", 5) != 0)
- async_synchronize_full_domain(&scsi_sd_pm_domain);
- } else {
- pm_runtime_disable(dev);
- pm_runtime_set_active(dev);
- pm_runtime_enable(dev);
- }
- return 0;
+ return 0;
+
+ err = cb(dev, pm);
+ scsi_device_resume(to_scsi_device(dev));
+ dev_dbg(dev, "scsi resume: %d\n", err);
+
+ return err;
}
static int scsi_bus_prepare(struct device *dev)
int scsi_eh_get_sense(struct list_head *work_q,
struct list_head *done_q);
int scsi_noretry_cmd(struct scsi_cmnd *scmd);
+void scsi_eh_done(struct scsi_cmnd *scmd);
/* scsi_lib.c */
extern int scsi_maybe_unblock_host(struct scsi_device *sdev);
#endif /* CONFIG_PROC_FS */
/* scsi_scan.c */
-extern char scsi_scan_type[];
+void scsi_enable_async_suspend(struct device *dev);
extern int scsi_complete_async_scans(void);
extern int scsi_scan_host_selected(struct Scsi_Host *, unsigned int,
unsigned int, u64, enum scsi_scan_mode);
extern void __scsi_remove_device(struct scsi_device *);
extern struct bus_type scsi_bus_type;
-extern const struct attribute_group *scsi_sysfs_shost_attr_groups[];
+extern const struct attribute_group scsi_shost_attr_group;
/* scsi_netlink.c */
#ifdef CONFIG_SCSI_NETLINK
static inline void scsi_autopm_put_host(struct Scsi_Host *h) {}
#endif /* CONFIG_PM */
-extern struct async_domain scsi_sd_pm_domain;
-
/* scsi_dh.c */
#ifdef CONFIG_SCSI_DH
void scsi_dh_add_device(struct scsi_device *sdev);
};
/**
+ * scsi_enable_async_suspend - Enable async suspend and resume
+ */
+void scsi_enable_async_suspend(struct device *dev)
+{
+ /*
+ * If a user has disabled async probing a likely reason is due to a
+ * storage enclosure that does not inject staggered spin-ups. For
+ * safety, make resume synchronous as well in that case.
+ */
+ if (strncmp(scsi_scan_type, "async", 5) != 0)
+ return;
+ /* Enable asynchronous suspend and resume. */
+ device_enable_async_suspend(dev);
+}
+
+/**
* scsi_complete_async_scans - Wait for asynchronous scans to complete
*
* When this function returns, any host which started scanning before
dev_set_name(dev, "target%d:%d:%d", shost->host_no, channel, id);
dev->bus = &scsi_bus_type;
dev->type = &scsi_target_type;
+ scsi_enable_async_suspend(dev);
starget->id = id;
starget->channel = channel;
starget->can_queue = 0;
spin_unlock_irqrestore(shost->host_lock, flags);
}
-/**
- * scsi_get_host_dev - Create a scsi_device that points to the host adapter itself
- * @shost: Host that needs a scsi_device
- *
- * Lock status: None assumed.
- *
- * Returns: The scsi_device or NULL
- *
- * Notes:
- * Attach a single scsi_device to the Scsi_Host - this should
- * be made to look like a "pseudo-device" that points to the
- * HA itself.
- *
- * Note - this device is not accessible from any high-level
- * drivers (including generics), which is probably not
- * optimal. We can add hooks later to attach.
- */
-struct scsi_device *scsi_get_host_dev(struct Scsi_Host *shost)
-{
- struct scsi_device *sdev = NULL;
- struct scsi_target *starget;
-
- mutex_lock(&shost->scan_mutex);
- if (!scsi_host_scan_allowed(shost))
- goto out;
- starget = scsi_alloc_target(&shost->shost_gendev, 0, shost->this_id);
- if (!starget)
- goto out;
-
- sdev = scsi_alloc_sdev(starget, 0, NULL);
- if (sdev)
- sdev->borken = 0;
- else
- scsi_target_reap(starget);
- put_device(&starget->dev);
- out:
- mutex_unlock(&shost->scan_mutex);
- return sdev;
-}
-EXPORT_SYMBOL(scsi_get_host_dev);
-
-/**
- * scsi_free_host_dev - Free a scsi_device that points to the host adapter itself
- * @sdev: Host device to be freed
- *
- * Lock status: None assumed.
- *
- * Returns: Nothing
- */
-void scsi_free_host_dev(struct scsi_device *sdev)
-{
- BUG_ON(sdev->id != sdev->host->this_id);
-
- __scsi_remove_device(sdev);
-}
-EXPORT_SYMBOL(scsi_free_host_dev);
-
NULL
};
-static struct attribute_group scsi_shost_attr_group = {
+const struct attribute_group scsi_shost_attr_group = {
.attrs = scsi_sysfs_shost_attrs,
};
-const struct attribute_group *scsi_sysfs_shost_attr_groups[] = {
- &scsi_shost_attr_group,
- NULL
-};
-
static void scsi_device_cls_release(struct device *class_dev)
{
struct scsi_device *sdev;
int i, ret;
struct scsi_device *sdev = to_scsi_device(dev);
enum scsi_device_state state = 0;
+ bool rescan_dev = false;
for (i = 0; i < ARRAY_SIZE(sdev_states); i++) {
const int len = strlen(sdev_states[i].name);
}
mutex_lock(&sdev->state_mutex);
- ret = scsi_device_set_state(sdev, state);
- /*
- * If the device state changes to SDEV_RUNNING, we need to
- * run the queue to avoid I/O hang, and rescan the device
- * to revalidate it. Running the queue first is necessary
- * because another thread may be waiting inside
- * blk_mq_freeze_queue_wait() and because that call may be
- * waiting for pending I/O to finish.
- */
- if (ret == 0 && state == SDEV_RUNNING) {
+ if (sdev->sdev_state == SDEV_RUNNING && state == SDEV_RUNNING) {
+ ret = count;
+ } else {
+ ret = scsi_device_set_state(sdev, state);
+ if (ret == 0 && state == SDEV_RUNNING)
+ rescan_dev = true;
+ }
+ mutex_unlock(&sdev->state_mutex);
+
+ if (rescan_dev) {
+ /*
+ * If the device state changes to SDEV_RUNNING, we need to
+ * run the queue to avoid I/O hang, and rescan the device
+ * to revalidate it. Running the queue first is necessary
+ * because another thread may be waiting inside
+ * blk_mq_freeze_queue_wait() and because that call may be
+ * waiting for pending I/O to finish.
+ */
blk_mq_run_hw_queues(sdev->request_queue, true);
scsi_rescan_device(dev);
}
- mutex_unlock(&sdev->state_mutex);
return ret == 0 ? count : -EINVAL;
}
**/
int scsi_sysfs_add_sdev(struct scsi_device *sdev)
{
- int error, i;
+ int error;
struct scsi_target *starget = sdev->sdev_target;
error = scsi_target_add(starget);
* We're treating error on bsg register as non-fatal, so
* pretend nothing went wrong.
*/
+ error = PTR_ERR(sdev->bsg_dev);
sdev_printk(KERN_INFO, sdev,
"Failed to register bsg queue, errno=%d\n",
error);
}
}
- /* add additional host specific attributes */
- if (sdev->host->hostt->sdev_attrs) {
- for (i = 0; sdev->host->hostt->sdev_attrs[i]; i++) {
- error = device_create_file(&sdev->sdev_gendev,
- sdev->host->hostt->sdev_attrs[i]);
- if (error)
- return error;
- }
- }
-
- if (sdev->host->hostt->sdev_groups) {
- error = sysfs_create_groups(&sdev->sdev_gendev.kobj,
- sdev->host->hostt->sdev_groups);
- if (error)
- return error;
- }
-
scsi_autopm_put_device(sdev);
return error;
}
if (res != 0)
return;
- if (sdev->host->hostt->sdev_groups)
- sysfs_remove_groups(&sdev->sdev_gendev.kobj,
- sdev->host->hostt->sdev_groups);
-
if (IS_ENABLED(CONFIG_BLK_DEV_BSG) && sdev->bsg_dev)
bsg_unregister_queue(sdev->bsg_dev);
device_unregister(&sdev->sdev_dev);
**/
int scsi_sysfs_add_host(struct Scsi_Host *shost)
{
- int error, i;
-
- /* add host specific attributes */
- if (shost->hostt->shost_attrs) {
- for (i = 0; shost->hostt->shost_attrs[i]; i++) {
- error = device_create_file(&shost->shost_dev,
- shost->hostt->shost_attrs[i]);
- if (error)
- return error;
- }
- }
-
transport_register_device(&shost->shost_gendev);
transport_configure_device(&shost->shost_gendev);
return 0;
{
unsigned long flags;
struct Scsi_Host *shost = sdev->host;
+ struct scsi_host_template *hostt = shost->hostt;
struct scsi_target *starget = sdev->sdev_target;
device_initialize(&sdev->sdev_gendev);
sdev->sdev_gendev.bus = &scsi_bus_type;
sdev->sdev_gendev.type = &scsi_dev_type;
+ scsi_enable_async_suspend(&sdev->sdev_gendev);
dev_set_name(&sdev->sdev_gendev, "%d:%d:%d:%llu",
sdev->host->host_no, sdev->channel, sdev->id, sdev->lun);
+ sdev->sdev_gendev.groups = hostt->sdev_groups;
device_initialize(&sdev->sdev_dev);
sdev->sdev_dev.parent = get_device(&sdev->sdev_gendev);
}
spin_unlock_irqrestore(&session->lock, flags);
- if (session->transport->session_recovery_timedout)
- session->transport->session_recovery_timedout(session);
-
ISCSI_DBG_TRANS_SESSION(session, "Unblocking SCSI target\n");
scsi_target_unblock(&session->dev, SDEV_TRANSPORT_OFFLINE);
ISCSI_DBG_TRANS_SESSION(session, "Completed unblocking SCSI target\n");
+
+ if (session->transport->session_recovery_timedout)
+ session->transport->session_recovery_timedout(session);
}
static void __iscsi_unblock_session(struct work_struct *work)
{ SAS_LINK_RATE_3_0_GBPS, "3.0 Gbit" },
{ SAS_LINK_RATE_6_0_GBPS, "6.0 Gbit" },
{ SAS_LINK_RATE_12_0_GBPS, "12.0 Gbit" },
+ { SAS_LINK_RATE_22_5_GBPS, "22.5 Gbit" },
};
sas_bitfield_name_search(linkspeed, sas_linkspeed_names)
sas_bitfield_name_set(linkspeed, sas_linkspeed_names)
static void sd_shutdown(struct device *);
static int sd_suspend_system(struct device *);
static int sd_suspend_runtime(struct device *);
-static int sd_resume(struct device *);
+static int sd_resume_system(struct device *);
static int sd_resume_runtime(struct device *);
static void sd_rescan(struct device *);
static blk_status_t sd_init_command(struct scsi_cmnd *SCpnt);
static const struct dev_pm_ops sd_pm_ops = {
.suspend = sd_suspend_system,
- .resume = sd_resume,
+ .resume = sd_resume_system,
.poweroff = sd_suspend_system,
- .restore = sd_resume,
+ .restore = sd_resume_system,
.runtime_suspend = sd_suspend_runtime,
.runtime_resume = sd_resume_runtime,
};
unsigned char *buffer, int len, struct scsi_mode_data *data,
struct scsi_sense_hdr *sshdr)
{
+ /*
+ * If we must use MODE SENSE(10), make sure that the buffer length
+ * is at least 8 bytes so that the mode sense header fits.
+ */
+ if (sdkp->device->use_10_for_ms && len < 8)
+ len = 8;
+
return scsi_mode_sense(sdkp->device, dbd, modepage, buffer, len,
SD_TIMEOUT, sdkp->max_retries, data,
sshdr);
}
}
- sd_first_printk(KERN_ERR, sdkp, "No Caching mode page found\n");
+ sd_first_printk(KERN_WARNING, sdkp,
+ "No Caching mode page found\n");
goto defaults;
Page_found:
"Assuming drive cache: write back\n");
sdkp->WCE = 1;
} else {
- sd_first_printk(KERN_ERR, sdkp,
+ sd_first_printk(KERN_WARNING, sdkp,
"Assuming drive cache: write through\n");
sdkp->WCE = 0;
}
pm_runtime_set_autosuspend_delay(dev,
sdp->host->hostt->rpm_autosuspend_delay);
}
- device_add_disk(dev, gd, NULL);
+
+ error = device_add_disk(dev, gd, NULL);
+ if (error) {
+ put_device(&sdkp->dev);
+ goto out;
+ }
+
if (sdkp->capacity)
sd_dif_config_host(sdkp);
sdkp = dev_get_drvdata(dev);
scsi_autopm_get_device(sdkp->device);
- async_synchronize_full_domain(&scsi_sd_pm_domain);
device_del(&sdkp->dev);
del_gendisk(sdkp->disk);
sd_shutdown(dev);
static int sd_suspend_system(struct device *dev)
{
+ if (pm_runtime_suspended(dev))
+ return 0;
+
return sd_suspend_common(dev, true);
}
return ret;
}
+static int sd_resume_system(struct device *dev)
+{
+ if (pm_runtime_suspended(dev))
+ return 0;
+
+ return sd_resume(dev);
+}
+
static int sd_resume_runtime(struct device *dev)
{
struct scsi_disk *sdkp = dev_get_drvdata(dev);
__le32 sis_product_identifier; /* B4h */
u8 reserved5[0xbc - (0xb4 + sizeof(__le32))];
__le32 sis_firmware_status; /* BCh */
- u8 reserved6[0x1000 - (0xbc + sizeof(__le32))];
+ u8 reserved6[0xcc - (0xbc + sizeof(__le32))];
+ __le32 sis_ctrl_shutdown_reason_code; /* CCh */
+ u8 reserved7[0x1000 - (0xcc + sizeof(__le32))];
__le32 sis_mailbox[8]; /* 1000h */
- u8 reserved7[0x4000 - (0x1000 + (sizeof(__le32) * 8))];
+ u8 reserved8[0x4000 - (0x1000 + (sizeof(__le32) * 8))];
/*
* The PQI spec states that the PQI registers should be at
* offset 0 from the PCIe BAR 0. However, we can't map
#define PQI_DEVICE_REGISTERS_OFFSET 0x4000
+/* shutdown reasons for taking the controller offline */
+enum pqi_ctrl_shutdown_reason {
+ PQI_IQ_NOT_DRAINED_TIMEOUT = 1,
+ PQI_LUN_RESET_TIMEOUT = 2,
+ PQI_IO_PENDING_POST_LUN_RESET_TIMEOUT = 3,
+ PQI_NO_HEARTBEAT = 4,
+ PQI_FIRMWARE_KERNEL_NOT_UP = 5,
+ PQI_OFA_RESPONSE_TIMEOUT = 6,
+ PQI_INVALID_REQ_ID = 7,
+ PQI_UNMATCHED_REQ_ID = 8,
+ PQI_IO_PI_OUT_OF_RANGE = 9,
+ PQI_EVENT_PI_OUT_OF_RANGE = 10,
+ PQI_UNEXPECTED_IU_TYPE = 11
+};
+
enum pqi_io_path {
RAID_PATH = 0,
AIO_PATH = 1
#define PQI_FIRMWARE_FEATURE_TMF_IU_TIMEOUT 14
#define PQI_FIRMWARE_FEATURE_RAID_BYPASS_ON_ENCRYPTED_NVME 15
#define PQI_FIRMWARE_FEATURE_UNIQUE_WWID_IN_REPORT_PHYS_LUN 16
-#define PQI_FIRMWARE_FEATURE_MAXIMUM 16
+#define PQI_FIRMWARE_FEATURE_FW_TRIAGE 17
+#define PQI_FIRMWARE_FEATURE_RPL_EXTENDED_FORMAT_4_5 18
+#define PQI_FIRMWARE_FEATURE_MAXIMUM 18
struct pqi_config_table_debug {
struct pqi_config_table_section_header header;
#define CISS_REPORT_LOG_FLAG_QUEUE_DEPTH (1 << 5)
#define CISS_REPORT_LOG_FLAG_DRIVE_TYPE_MIX (1 << 6)
-#define CISS_REPORT_PHYS_FLAG_OTHER (1 << 1)
+#define CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_2 0x2
+#define CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_4 0x4
+#define CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_MASK 0xf
-struct report_log_lun_extended_entry {
+struct report_log_lun {
u8 lunid[8];
u8 volume_id[16];
};
-struct report_log_lun_extended {
+struct report_log_lun_list {
struct report_lun_header header;
- struct report_log_lun_extended_entry lun_entries[1];
+ struct report_log_lun lun_entries[1];
};
-struct report_phys_lun_extended_entry {
+struct report_phys_lun_8byte_wwid {
u8 lunid[8];
__be64 wwid;
u8 device_type;
u32 aio_handle;
};
+struct report_phys_lun_16byte_wwid {
+ u8 lunid[8];
+ u8 wwid[16];
+ u8 device_type;
+ u8 device_flags;
+ u8 lun_count; /* number of LUNs in a multi-LUN device */
+ u8 redundant_paths;
+ u32 aio_handle;
+};
+
/* for device_flags field of struct report_phys_lun_extended_entry */
#define CISS_REPORT_PHYS_DEV_FLAG_AIO_ENABLED 0x8
-struct report_phys_lun_extended {
+struct report_phys_lun_8byte_wwid_list {
struct report_lun_header header;
- struct report_phys_lun_extended_entry lun_entries[1];
+ struct report_phys_lun_8byte_wwid lun_entries[1];
+};
+
+struct report_phys_lun_16byte_wwid_list {
+ struct report_lun_header header;
+ struct report_phys_lun_16byte_wwid lun_entries[1];
};
struct raid_map_disk_data {
int target;
int lun;
u8 scsi3addr[8];
- __be64 wwid;
+ u8 wwid[16];
u8 volume_id[16];
u8 is_physical_device : 1;
u8 is_external_raid_device : 1;
u8 keep_device : 1;
u8 volume_offline : 1;
u8 rescan : 1;
+ u8 ignore_device : 1;
bool aio_enabled; /* only valid for physical disks */
bool in_remove;
bool device_offline;
u8 raid_iu_timeout_supported : 1;
u8 tmf_iu_timeout_supported : 1;
u8 unique_wwid_in_report_phys_lun_supported : 1;
+ u8 firmware_triage_supported : 1;
+ u8 rpl_extended_format_4_5_supported : 1;
u8 enable_r1_writes : 1;
u8 enable_r5_writes : 1;
u8 enable_r6_writes : 1;
#define BUILD_TIMESTAMP
#endif
-#define DRIVER_VERSION "2.1.10-020"
+#define DRIVER_VERSION "2.1.12-055"
#define DRIVER_MAJOR 2
#define DRIVER_MINOR 1
-#define DRIVER_RELEASE 10
-#define DRIVER_REVISION 20
+#define DRIVER_RELEASE 12
+#define DRIVER_REVISION 55
#define DRIVER_NAME "Microchip SmartPQI Driver (v" \
DRIVER_VERSION BUILD_TIMESTAMP ")"
MODULE_VERSION(DRIVER_VERSION);
MODULE_LICENSE("GPL");
-static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info);
+static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info,
+ enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason);
static void pqi_ctrl_offline_worker(struct work_struct *work);
static int pqi_scan_scsi_devices(struct pqi_ctrl_info *ctrl_info);
static void pqi_scan_start(struct Scsi_Host *shost);
static inline void pqi_scsi_done(struct scsi_cmnd *scmd)
{
pqi_prep_for_scsi_done(scmd);
- scmd->scsi_done(scmd);
+ scsi_done(scmd);
}
static inline void pqi_disable_write_same(struct scsi_device *sdev)
{
if (ctrl_info->controller_online)
if (!sis_is_firmware_running(ctrl_info))
- pqi_take_ctrl_offline(ctrl_info);
+ pqi_take_ctrl_offline(ctrl_info, PQI_FIRMWARE_KERNEL_NOT_UP);
}
static inline bool pqi_is_hba_lunid(u8 *scsi3addr)
return pqi_scsi3addr_equal(scsi3addr, RAID_CTLR_LUNID);
}
+#define PQI_DRIVER_SCRATCH_PQI_MODE 0x1
+#define PQI_DRIVER_SCRATCH_FW_TRIAGE_SUPPORTED 0x2
+
static inline enum pqi_ctrl_mode pqi_get_ctrl_mode(struct pqi_ctrl_info *ctrl_info)
{
- return sis_read_driver_scratch(ctrl_info);
+ return sis_read_driver_scratch(ctrl_info) & PQI_DRIVER_SCRATCH_PQI_MODE ? PQI_MODE : SIS_MODE;
}
static inline void pqi_save_ctrl_mode(struct pqi_ctrl_info *ctrl_info,
enum pqi_ctrl_mode mode)
{
- sis_write_driver_scratch(ctrl_info, mode);
+ u32 driver_scratch;
+
+ driver_scratch = sis_read_driver_scratch(ctrl_info);
+
+ if (mode == PQI_MODE)
+ driver_scratch |= PQI_DRIVER_SCRATCH_PQI_MODE;
+ else
+ driver_scratch &= ~PQI_DRIVER_SCRATCH_PQI_MODE;
+
+ sis_write_driver_scratch(ctrl_info, driver_scratch);
+}
+
+static inline bool pqi_is_fw_triage_supported(struct pqi_ctrl_info *ctrl_info)
+{
+ return (sis_read_driver_scratch(ctrl_info) & PQI_DRIVER_SCRATCH_FW_TRIAGE_SUPPORTED) != 0;
+}
+
+static inline void pqi_save_fw_triage_setting(struct pqi_ctrl_info *ctrl_info, bool is_supported)
+{
+ u32 driver_scratch;
+
+ driver_scratch = sis_read_driver_scratch(ctrl_info);
+
+ if (is_supported)
+ driver_scratch |= PQI_DRIVER_SCRATCH_FW_TRIAGE_SUPPORTED;
+ else
+ driver_scratch &= ~PQI_DRIVER_SCRATCH_FW_TRIAGE_SUPPORTED;
+
+ sis_write_driver_scratch(ctrl_info, driver_scratch);
}
static inline void pqi_ctrl_block_scan(struct pqi_ctrl_info *ctrl_info)
cdb = request->cdb;
switch (cmd) {
+ case TEST_UNIT_READY:
+ request->data_direction = SOP_READ_FLAG;
+ cdb[0] = TEST_UNIT_READY;
+ break;
case INQUIRY:
request->data_direction = SOP_READ_FLAG;
cdb[0] = INQUIRY;
case CISS_REPORT_PHYS:
request->data_direction = SOP_READ_FLAG;
cdb[0] = cmd;
- if (cmd == CISS_REPORT_PHYS)
- cdb[1] = CISS_REPORT_PHYS_FLAG_OTHER;
- else
+ if (cmd == CISS_REPORT_PHYS) {
+ if (ctrl_info->rpl_extended_format_4_5_supported)
+ cdb[1] = CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_4;
+ else
+ cdb[1] = CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_2;
+ } else {
cdb[1] = ctrl_info->ciss_report_log_flags;
+ }
put_unaligned_be32(cdb_length, &cdb[6]);
break;
case CISS_GET_RAID_MAP:
static inline int pqi_report_phys_luns(struct pqi_ctrl_info *ctrl_info, void **buffer)
{
- return pqi_report_phys_logical_luns(ctrl_info, CISS_REPORT_PHYS, buffer);
+ int rc;
+ unsigned int i;
+ u8 rpl_response_format;
+ u32 num_physicals;
+ size_t rpl_16byte_wwid_list_length;
+ void *rpl_list;
+ struct report_lun_header *rpl_header;
+ struct report_phys_lun_8byte_wwid_list *rpl_8byte_wwid_list;
+ struct report_phys_lun_16byte_wwid_list *rpl_16byte_wwid_list;
+
+ rc = pqi_report_phys_logical_luns(ctrl_info, CISS_REPORT_PHYS, &rpl_list);
+ if (rc)
+ return rc;
+
+ if (ctrl_info->rpl_extended_format_4_5_supported) {
+ rpl_header = rpl_list;
+ rpl_response_format = rpl_header->flags & CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_MASK;
+ if (rpl_response_format == CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_4) {
+ *buffer = rpl_list;
+ return 0;
+ } else if (rpl_response_format != CISS_REPORT_PHYS_FLAG_EXTENDED_FORMAT_2) {
+ dev_err(&ctrl_info->pci_dev->dev,
+ "RPL returned unsupported data format %u\n",
+ rpl_response_format);
+ return -EINVAL;
+ } else {
+ dev_warn(&ctrl_info->pci_dev->dev,
+ "RPL returned extended format 2 instead of 4\n");
+ }
+ }
+
+ rpl_8byte_wwid_list = rpl_list;
+ num_physicals = get_unaligned_be32(&rpl_8byte_wwid_list->header.list_length) / sizeof(rpl_8byte_wwid_list->lun_entries[0]);
+ rpl_16byte_wwid_list_length = sizeof(struct report_lun_header) + (num_physicals * sizeof(struct report_phys_lun_16byte_wwid));
+
+ rpl_16byte_wwid_list = kmalloc(rpl_16byte_wwid_list_length, GFP_KERNEL);
+ if (!rpl_16byte_wwid_list)
+ return -ENOMEM;
+
+ put_unaligned_be32(num_physicals * sizeof(struct report_phys_lun_16byte_wwid),
+ &rpl_16byte_wwid_list->header.list_length);
+ rpl_16byte_wwid_list->header.flags = rpl_8byte_wwid_list->header.flags;
+
+ for (i = 0; i < num_physicals; i++) {
+ memcpy(&rpl_16byte_wwid_list->lun_entries[i].lunid, &rpl_8byte_wwid_list->lun_entries[i].lunid, sizeof(rpl_8byte_wwid_list->lun_entries[i].lunid));
+ memset(&rpl_16byte_wwid_list->lun_entries[i].wwid, 0, 8);
+ memcpy(&rpl_16byte_wwid_list->lun_entries[i].wwid[8], &rpl_8byte_wwid_list->lun_entries[i].wwid, sizeof(rpl_8byte_wwid_list->lun_entries[i].wwid));
+ rpl_16byte_wwid_list->lun_entries[i].device_type = rpl_8byte_wwid_list->lun_entries[i].device_type;
+ rpl_16byte_wwid_list->lun_entries[i].device_flags = rpl_8byte_wwid_list->lun_entries[i].device_flags;
+ rpl_16byte_wwid_list->lun_entries[i].lun_count = rpl_8byte_wwid_list->lun_entries[i].lun_count;
+ rpl_16byte_wwid_list->lun_entries[i].redundant_paths = rpl_8byte_wwid_list->lun_entries[i].redundant_paths;
+ rpl_16byte_wwid_list->lun_entries[i].aio_handle = rpl_8byte_wwid_list->lun_entries[i].aio_handle;
+ }
+
+ kfree(rpl_8byte_wwid_list);
+ *buffer = rpl_16byte_wwid_list;
+
+ return 0;
}
static inline int pqi_report_logical_luns(struct pqi_ctrl_info *ctrl_info, void **buffer)
}
static int pqi_get_device_lists(struct pqi_ctrl_info *ctrl_info,
- struct report_phys_lun_extended **physdev_list,
- struct report_log_lun_extended **logdev_list)
+ struct report_phys_lun_16byte_wwid_list **physdev_list,
+ struct report_log_lun_list **logdev_list)
{
int rc;
size_t logdev_list_length;
size_t logdev_data_length;
- struct report_log_lun_extended *internal_logdev_list;
- struct report_log_lun_extended *logdev_data;
+ struct report_log_lun_list *internal_logdev_list;
+ struct report_log_lun_list *logdev_data;
struct report_lun_header report_lun_header;
rc = pqi_report_phys_luns(ctrl_info, (void **)physdev_list);
} else {
memset(&report_lun_header, 0, sizeof(report_lun_header));
logdev_data =
- (struct report_log_lun_extended *)&report_lun_header;
+ (struct report_log_lun_list *)&report_lun_header;
logdev_list_length = 0;
}
logdev_list_length;
internal_logdev_list = kmalloc(logdev_data_length +
- sizeof(struct report_log_lun_extended), GFP_KERNEL);
+ sizeof(struct report_log_lun), GFP_KERNEL);
if (!internal_logdev_list) {
kfree(*logdev_list);
*logdev_list = NULL;
memcpy(internal_logdev_list, logdev_data, logdev_data_length);
memset((u8 *)internal_logdev_list + logdev_data_length, 0,
- sizeof(struct report_log_lun_extended_entry));
+ sizeof(struct report_log_lun));
put_unaligned_be32(logdev_list_length +
- sizeof(struct report_log_lun_extended_entry),
+ sizeof(struct report_log_lun),
&internal_logdev_list->header.list_length);
kfree(*logdev_list);
return rc;
}
+/*
+ * Prevent adding drive to OS for some corner cases such as a drive
+ * undergoing a sanitize operation. Some OSes will continue to poll
+ * the drive until the sanitize completes, which can take hours,
+ * resulting in long bootup delays. Commands such as TUR, READ_CAP
+ * are allowed, but READ/WRITE cause check condition. So the OS
+ * cannot check/read the partition table.
+ * Note: devices that have completed sanitize must be re-enabled
+ * using the management utility.
+ */
+static bool pqi_keep_device_offline(struct pqi_ctrl_info *ctrl_info,
+ struct pqi_scsi_dev *device)
+{
+ u8 scsi_status;
+ int rc;
+ enum dma_data_direction dir;
+ char *buffer;
+ int buffer_length = 64;
+ size_t sense_data_length;
+ struct scsi_sense_hdr sshdr;
+ struct pqi_raid_path_request request;
+ struct pqi_raid_error_info error_info;
+ bool offline = false; /* Assume keep online */
+
+ /* Do not check controllers. */
+ if (pqi_is_hba_lunid(device->scsi3addr))
+ return false;
+
+ /* Do not check LVs. */
+ if (pqi_is_logical_device(device))
+ return false;
+
+ buffer = kmalloc(buffer_length, GFP_KERNEL);
+ if (!buffer)
+ return false; /* Assume not offline */
+
+ /* Check for SANITIZE in progress using TUR */
+ rc = pqi_build_raid_path_request(ctrl_info, &request,
+ TEST_UNIT_READY, RAID_CTLR_LUNID, buffer,
+ buffer_length, 0, &dir);
+ if (rc)
+ goto out; /* Assume not offline */
+
+ memcpy(request.lun_number, device->scsi3addr, sizeof(request.lun_number));
+
+ rc = pqi_submit_raid_request_synchronous(ctrl_info, &request.header, 0, &error_info);
+
+ if (rc)
+ goto out; /* Assume not offline */
+
+ scsi_status = error_info.status;
+ sense_data_length = get_unaligned_le16(&error_info.sense_data_length);
+ if (sense_data_length == 0)
+ sense_data_length =
+ get_unaligned_le16(&error_info.response_data_length);
+ if (sense_data_length) {
+ if (sense_data_length > sizeof(error_info.data))
+ sense_data_length = sizeof(error_info.data);
+
+ /*
+ * Check for sanitize in progress: asc:0x04, ascq: 0x1b
+ */
+ if (scsi_status == SAM_STAT_CHECK_CONDITION &&
+ scsi_normalize_sense(error_info.data,
+ sense_data_length, &sshdr) &&
+ sshdr.sense_key == NOT_READY &&
+ sshdr.asc == 0x04 &&
+ sshdr.ascq == 0x1b) {
+ device->device_offline = true;
+ offline = true;
+ goto out; /* Keep device offline */
+ }
+ }
+
+out:
+ kfree(buffer);
+ return offline;
+}
+
static int pqi_get_device_info(struct pqi_ctrl_info *ctrl_info,
struct pqi_scsi_dev *device,
struct bmic_identify_physical_device *id_phys)
{
int rc;
- pqi_device_remove_start(device);
-
rc = pqi_device_wait_for_pending_io(ctrl_info, device,
PQI_REMOVE_DEVICE_PENDING_IO_TIMEOUT_MSECS);
if (rc)
scsi_remove_device(device->sdev);
else
pqi_remove_sas_device(device);
+
+ pqi_device_remove_start(device);
}
/* Assumes the SCSI device list lock is held. */
return false;
if (dev1->is_physical_device)
- return dev1->wwid == dev2->wwid;
+ return memcmp(dev1->wwid, dev2->wwid, sizeof(dev1->wwid)) == 0;
return memcmp(dev1->volume_id, dev2->volume_id, sizeof(dev1->volume_id)) == 0;
}
else
count += scnprintf(buffer + count,
PQI_DEV_INFO_BUFFER_LENGTH - count,
- " %016llx", device->sas_address);
+ " %016llx%016llx",
+ get_unaligned_be64(&device->wwid[0]),
+ get_unaligned_be64(&device->wwid[8]));
count += scnprintf(buffer + count, PQI_DEV_INFO_BUFFER_LENGTH - count,
" %s %.8s %.16s ",
list_for_each_entry_safe(device, next, &ctrl_info->scsi_device_list,
scsi_device_list_entry) {
if (device->device_gone) {
- list_del_init(&device->scsi_device_list_entry);
+ list_del(&device->scsi_device_list_entry);
list_add_tail(&device->delete_list_entry, &delete_list);
}
}
if (device->volume_offline) {
pqi_dev_info(ctrl_info, "offline", device);
pqi_show_volume_status(ctrl_info, device);
- }
- list_del(&device->delete_list_entry);
- if (pqi_is_device_added(device)) {
- pqi_remove_device(ctrl_info, device);
} else {
- if (!device->volume_offline)
- pqi_dev_info(ctrl_info, "removed", device);
- pqi_free_device(device);
+ pqi_dev_info(ctrl_info, "removed", device);
}
+ if (pqi_is_device_added(device))
+ pqi_remove_device(ctrl_info, device);
+ list_del(&device->delete_list_entry);
+ pqi_free_device(device);
}
/*
}
static inline void pqi_set_physical_device_wwid(struct pqi_ctrl_info *ctrl_info,
- struct pqi_scsi_dev *device, struct report_phys_lun_extended_entry *phys_lun_ext_entry)
+ struct pqi_scsi_dev *device, struct report_phys_lun_16byte_wwid *phys_lun)
{
if (ctrl_info->unique_wwid_in_report_phys_lun_supported ||
+ ctrl_info->rpl_extended_format_4_5_supported ||
pqi_is_device_with_sas_address(device))
- device->wwid = phys_lun_ext_entry->wwid;
+ memcpy(device->wwid, phys_lun->wwid, sizeof(device->wwid));
else
- device->wwid = cpu_to_be64(get_unaligned_be64(&device->page_83_identifier));
+ memcpy(&device->wwid[8], device->page_83_identifier, 8);
}
static int pqi_update_scsi_devices(struct pqi_ctrl_info *ctrl_info)
int i;
int rc;
LIST_HEAD(new_device_list_head);
- struct report_phys_lun_extended *physdev_list = NULL;
- struct report_log_lun_extended *logdev_list = NULL;
- struct report_phys_lun_extended_entry *phys_lun_ext_entry;
- struct report_log_lun_extended_entry *log_lun_ext_entry;
+ struct report_phys_lun_16byte_wwid_list *physdev_list = NULL;
+ struct report_log_lun_list *logdev_list = NULL;
+ struct report_phys_lun_16byte_wwid *phys_lun;
+ struct report_log_lun *log_lun;
struct bmic_identify_physical_device *id_phys = NULL;
u32 num_physicals;
u32 num_logicals;
if (pqi_hide_vsep) {
for (i = num_physicals - 1; i >= 0; i--) {
- phys_lun_ext_entry =
- &physdev_list->lun_entries[i];
- if (CISS_GET_DRIVE_NUMBER(phys_lun_ext_entry->lunid) == PQI_VSEP_CISS_BTL) {
- pqi_mask_device(phys_lun_ext_entry->lunid);
+ phys_lun = &physdev_list->lun_entries[i];
+ if (CISS_GET_DRIVE_NUMBER(phys_lun->lunid) == PQI_VSEP_CISS_BTL) {
+ pqi_mask_device(phys_lun->lunid);
break;
}
}
if ((!pqi_expose_ld_first && i < num_physicals) ||
(pqi_expose_ld_first && i >= num_logicals)) {
is_physical_device = true;
- phys_lun_ext_entry =
- &physdev_list->lun_entries[physical_index++];
- log_lun_ext_entry = NULL;
- scsi3addr = phys_lun_ext_entry->lunid;
+ phys_lun = &physdev_list->lun_entries[physical_index++];
+ log_lun = NULL;
+ scsi3addr = phys_lun->lunid;
} else {
is_physical_device = false;
- phys_lun_ext_entry = NULL;
- log_lun_ext_entry =
- &logdev_list->lun_entries[logical_index++];
- scsi3addr = log_lun_ext_entry->lunid;
+ phys_lun = NULL;
+ log_lun = &logdev_list->lun_entries[logical_index++];
+ scsi3addr = log_lun->lunid;
}
if (is_physical_device && pqi_skip_device(scsi3addr))
memcpy(device->scsi3addr, scsi3addr, sizeof(device->scsi3addr));
device->is_physical_device = is_physical_device;
if (is_physical_device) {
- device->device_type = phys_lun_ext_entry->device_type;
+ device->device_type = phys_lun->device_type;
if (device->device_type == SA_DEVICE_TYPE_EXPANDER_SMP)
device->is_expander_smp_device = true;
} else {
if (!pqi_is_supported_device(device))
continue;
+ /* Do not present disks that the OS cannot fully probe */
+ if (pqi_keep_device_offline(ctrl_info, device))
+ continue;
+
/* Gather information about the device. */
rc = pqi_get_device_info(ctrl_info, device, id_phys);
if (rc == -ENOMEM) {
if (rc) {
if (device->is_physical_device)
dev_warn(&ctrl_info->pci_dev->dev,
- "obtaining device info failed, skipping physical device %016llx\n",
- get_unaligned_be64(&phys_lun_ext_entry->wwid));
+ "obtaining device info failed, skipping physical device %016llx%016llx\n",
+ get_unaligned_be64(&phys_lun->wwid[0]),
+ get_unaligned_be64(&phys_lun->wwid[8]));
else
dev_warn(&ctrl_info->pci_dev->dev,
"obtaining device info failed, skipping logical device %08x%08x\n",
pqi_assign_bus_target_lun(device);
if (device->is_physical_device) {
- pqi_set_physical_device_wwid(ctrl_info, device, phys_lun_ext_entry);
- if ((phys_lun_ext_entry->device_flags &
+ pqi_set_physical_device_wwid(ctrl_info, device, phys_lun);
+ if ((phys_lun->device_flags &
CISS_REPORT_PHYS_DEV_FLAG_AIO_ENABLED) &&
- phys_lun_ext_entry->aio_handle) {
+ phys_lun->aio_handle) {
device->aio_enabled = true;
device->aio_handle =
- phys_lun_ext_entry->aio_handle;
+ phys_lun->aio_handle;
}
} else {
- memcpy(device->volume_id, log_lun_ext_entry->volume_id,
+ memcpy(device->volume_id, log_lun->volume_id,
sizeof(device->volume_id));
}
if (pqi_is_device_with_sas_address(device))
- device->sas_address = get_unaligned_be64(&device->wwid);
+ device->sas_address = get_unaligned_be64(&device->wwid[8]);
new_device_list[num_valid_devices++] = device;
}
return rc;
}
+static void pqi_remove_all_scsi_devices(struct pqi_ctrl_info *ctrl_info)
+{
+ unsigned long flags;
+ struct pqi_scsi_dev *device;
+ struct pqi_scsi_dev *next;
+
+ spin_lock_irqsave(&ctrl_info->scsi_device_list_lock, flags);
+
+ list_for_each_entry_safe(device, next, &ctrl_info->scsi_device_list,
+ scsi_device_list_entry) {
+ if (pqi_is_device_added(device))
+ pqi_remove_device(ctrl_info, device);
+ list_del(&device->scsi_device_list_entry);
+ pqi_free_device(device);
+ }
+
+ spin_unlock_irqrestore(&ctrl_info->scsi_device_list_lock, flags);
+}
+
static int pqi_scan_scsi_devices(struct pqi_ctrl_info *ctrl_info)
{
int rc;
return rc;
}
-static inline void pqi_invalid_response(struct pqi_ctrl_info *ctrl_info)
+static inline void pqi_invalid_response(struct pqi_ctrl_info *ctrl_info,
+ enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason)
{
- pqi_take_ctrl_offline(ctrl_info);
+ pqi_take_ctrl_offline(ctrl_info, ctrl_shutdown_reason);
}
static int pqi_process_io_intr(struct pqi_ctrl_info *ctrl_info, struct pqi_queue_group *queue_group)
while (1) {
oq_pi = readl(queue_group->oq_pi);
if (oq_pi >= ctrl_info->num_elements_per_oq) {
- pqi_invalid_response(ctrl_info);
+ pqi_invalid_response(ctrl_info, PQI_IO_PI_OUT_OF_RANGE);
dev_err(&ctrl_info->pci_dev->dev,
"I/O interrupt: producer index (%u) out of range (0-%u): consumer index: %u\n",
oq_pi, ctrl_info->num_elements_per_oq - 1, oq_ci);
request_id = get_unaligned_le16(&response->request_id);
if (request_id >= ctrl_info->max_io_slots) {
- pqi_invalid_response(ctrl_info);
+ pqi_invalid_response(ctrl_info, PQI_INVALID_REQ_ID);
dev_err(&ctrl_info->pci_dev->dev,
"request ID in response (%u) out of range (0-%u): producer index: %u consumer index: %u\n",
request_id, ctrl_info->max_io_slots - 1, oq_pi, oq_ci);
io_request = &ctrl_info->io_request_pool[request_id];
if (atomic_read(&io_request->refcount) == 0) {
- pqi_invalid_response(ctrl_info);
+ pqi_invalid_response(ctrl_info, PQI_UNMATCHED_REQ_ID);
dev_err(&ctrl_info->pci_dev->dev,
"request ID in response (%u) does not match an outstanding I/O request: producer index: %u consumer index: %u\n",
request_id, oq_pi, oq_ci);
pqi_process_io_error(response->header.iu_type, io_request);
break;
default:
- pqi_invalid_response(ctrl_info);
+ pqi_invalid_response(ctrl_info, PQI_UNEXPECTED_IU_TYPE);
dev_err(&ctrl_info->pci_dev->dev,
"unexpected IU type: 0x%x: producer index: %u consumer index: %u\n",
response->header.iu_type, oq_pi, oq_ci);
pqi_ofa_free_host_buffer(ctrl_info);
pqi_ctrl_ofa_done(ctrl_info);
pqi_ofa_ctrl_unquiesce(ctrl_info);
- pqi_take_ctrl_offline(ctrl_info);
+ pqi_take_ctrl_offline(ctrl_info, PQI_OFA_RESPONSE_TIMEOUT);
break;
}
}
dev_err(&ctrl_info->pci_dev->dev,
"no heartbeat detected - last heartbeat count: %u\n",
heartbeat_count);
- pqi_take_ctrl_offline(ctrl_info);
+ pqi_take_ctrl_offline(ctrl_info, PQI_NO_HEARTBEAT);
return;
}
} else {
while (1) {
oq_pi = readl(event_queue->oq_pi);
if (oq_pi >= PQI_NUM_EVENT_QUEUE_ELEMENTS) {
- pqi_invalid_response(ctrl_info);
+ pqi_invalid_response(ctrl_info, PQI_EVENT_PI_OUT_OF_RANGE);
dev_err(&ctrl_info->pci_dev->dev,
"event interrupt: producer index (%u) out of range (0-%u): consumer index: %u\n",
oq_pi, PQI_NUM_EVENT_QUEUE_ELEMENTS - 1, oq_ci);
timeout = PQI_ADMIN_QUEUE_CREATE_TIMEOUT_JIFFIES + jiffies;
while (1) {
+ msleep(PQI_ADMIN_QUEUE_CREATE_POLL_INTERVAL_MSECS);
status = readb(&pqi_registers->function_and_status_code);
if (status == PQI_STATUS_IDLE)
break;
if (time_after(jiffies, timeout))
return -ETIMEDOUT;
- msleep(PQI_ADMIN_QUEUE_CREATE_POLL_INTERVAL_MSECS);
}
/*
return rc;
}
-static int pqi_wait_until_queued_io_drained(struct pqi_ctrl_info *ctrl_info,
- struct pqi_queue_group *queue_group)
+static unsigned int pqi_queued_io_count(struct pqi_ctrl_info *ctrl_info)
{
+ unsigned int i;
unsigned int path;
unsigned long flags;
- bool list_is_empty;
+ unsigned int queued_io_count;
+ struct pqi_queue_group *queue_group;
+ struct pqi_io_request *io_request;
- for (path = 0; path < 2; path++) {
- while (1) {
- spin_lock_irqsave(
- &queue_group->submit_lock[path], flags);
- list_is_empty =
- list_empty(&queue_group->request_list[path]);
- spin_unlock_irqrestore(
- &queue_group->submit_lock[path], flags);
- if (list_is_empty)
- break;
- pqi_check_ctrl_health(ctrl_info);
- if (pqi_ctrl_offline(ctrl_info))
- return -ENXIO;
- usleep_range(1000, 2000);
+ queued_io_count = 0;
+
+ for (i = 0; i < ctrl_info->num_queue_groups; i++) {
+ queue_group = &ctrl_info->queue_groups[i];
+ for (path = 0; path < 2; path++) {
+ spin_lock_irqsave(&queue_group->submit_lock[path], flags);
+ list_for_each_entry(io_request, &queue_group->request_list[path], request_list_entry)
+ queued_io_count++;
+ spin_unlock_irqrestore(&queue_group->submit_lock[path], flags);
}
}
- return 0;
+ return queued_io_count;
}
-static int pqi_wait_until_inbound_queues_empty(struct pqi_ctrl_info *ctrl_info)
+static unsigned int pqi_nonempty_inbound_queue_count(struct pqi_ctrl_info *ctrl_info)
{
- int rc;
unsigned int i;
unsigned int path;
+ unsigned int nonempty_inbound_queue_count;
struct pqi_queue_group *queue_group;
pqi_index_t iq_pi;
pqi_index_t iq_ci;
+ nonempty_inbound_queue_count = 0;
+
for (i = 0; i < ctrl_info->num_queue_groups; i++) {
queue_group = &ctrl_info->queue_groups[i];
-
- rc = pqi_wait_until_queued_io_drained(ctrl_info, queue_group);
- if (rc)
- return rc;
-
for (path = 0; path < 2; path++) {
iq_pi = queue_group->iq_pi_copy[path];
+ iq_ci = readl(queue_group->iq_ci[path]);
+ if (iq_ci != iq_pi)
+ nonempty_inbound_queue_count++;
+ }
+ }
- while (1) {
- iq_ci = readl(queue_group->iq_ci[path]);
- if (iq_ci == iq_pi)
- break;
- pqi_check_ctrl_health(ctrl_info);
- if (pqi_ctrl_offline(ctrl_info))
- return -ENXIO;
- usleep_range(1000, 2000);
- }
+ return nonempty_inbound_queue_count;
+}
+
+#define PQI_INBOUND_QUEUES_NONEMPTY_WARNING_TIMEOUT_SECS 10
+
+static int pqi_wait_until_inbound_queues_empty(struct pqi_ctrl_info *ctrl_info)
+{
+ unsigned long start_jiffies;
+ unsigned long warning_timeout;
+ unsigned int queued_io_count;
+ unsigned int nonempty_inbound_queue_count;
+ bool displayed_warning;
+
+ displayed_warning = false;
+ start_jiffies = jiffies;
+ warning_timeout = (PQI_INBOUND_QUEUES_NONEMPTY_WARNING_TIMEOUT_SECS * PQI_HZ) + start_jiffies;
+
+ while (1) {
+ queued_io_count = pqi_queued_io_count(ctrl_info);
+ nonempty_inbound_queue_count = pqi_nonempty_inbound_queue_count(ctrl_info);
+ if (queued_io_count == 0 && nonempty_inbound_queue_count == 0)
+ break;
+ pqi_check_ctrl_health(ctrl_info);
+ if (pqi_ctrl_offline(ctrl_info))
+ return -ENXIO;
+ if (time_after(jiffies, warning_timeout)) {
+ dev_warn(&ctrl_info->pci_dev->dev,
+ "waiting %u seconds for queued I/O to drain (queued I/O count: %u; non-empty inbound queue count: %u)\n",
+ jiffies_to_msecs(jiffies - start_jiffies) / 1000, queued_io_count, nonempty_inbound_queue_count);
+ displayed_warning = true;
+ warning_timeout = (PQI_INBOUND_QUEUES_NONEMPTY_WARNING_TIMEOUT_SECS * PQI_HZ) + jiffies;
}
+ usleep_range(1000, 2000);
}
+ if (displayed_warning)
+ dev_warn(&ctrl_info->pci_dev->dev,
+ "queued I/O drained after waiting for %u seconds\n",
+ jiffies_to_msecs(jiffies - start_jiffies) / 1000);
+
return 0;
}
if (pqi_ctrl_offline(ctrl_info))
return -ENXIO;
msecs_waiting = jiffies_to_msecs(jiffies - start_jiffies);
- if (msecs_waiting > timeout_msecs) {
+ if (msecs_waiting >= timeout_msecs) {
dev_err(&ctrl_info->pci_dev->dev,
"scsi %d:%d:%d:%d: timed out after %lu seconds waiting for %d outstanding command(s)\n",
ctrl_info->scsi_host->host_no, device->bus, device->target,
{
int rc;
unsigned int wait_secs;
+ int cmds_outstanding;
wait_secs = 0;
}
wait_secs += PQI_LUN_RESET_POLL_COMPLETION_SECS;
-
+ cmds_outstanding = atomic_read(&device->scsi_cmds_outstanding);
dev_warn(&ctrl_info->pci_dev->dev,
- "scsi %d:%d:%d:%d: waiting %u seconds for LUN reset to complete\n",
- ctrl_info->scsi_host->host_no, device->bus, device->target, device->lun,
- wait_secs);
+ "scsi %d:%d:%d:%d: waiting %u seconds for LUN reset to complete (%d command(s) outstanding)\n",
+ ctrl_info->scsi_host->host_no, device->bus, device->target, device->lun, wait_secs, cmds_outstanding);
}
return rc;
rphy = target_to_rphy(starget);
device = pqi_find_device_by_sas_rphy(ctrl_info, rphy);
if (device) {
- device->target = sdev_id(sdev);
- device->lun = sdev->lun;
- device->target_lun_valid = true;
+ if (device->target_lun_valid) {
+ device->ignore_device = true;
+ } else {
+ device->target = sdev_id(sdev);
+ device->lun = sdev->lun;
+ device->target_lun_valid = true;
+ }
}
} else {
device = pqi_find_scsi_dev(ctrl_info, sdev_channel(sdev),
ctrl_info->pci_dev, 0);
}
-static int pqi_slave_configure(struct scsi_device *sdev)
+static inline bool pqi_is_tape_changer_device(struct pqi_scsi_dev *device)
{
- struct pqi_scsi_dev *device;
-
- device = sdev->hostdata;
- device->devtype = sdev->type;
-
- return 0;
+ return device->devtype == TYPE_TAPE || device->devtype == TYPE_MEDIUM_CHANGER;
}
-static void pqi_slave_destroy(struct scsi_device *sdev)
+static int pqi_slave_configure(struct scsi_device *sdev)
{
- unsigned long flags;
+ int rc = 0;
struct pqi_scsi_dev *device;
- struct pqi_ctrl_info *ctrl_info;
-
- ctrl_info = shost_to_hba(sdev->host);
-
- spin_lock_irqsave(&ctrl_info->scsi_device_list_lock, flags);
device = sdev->hostdata;
- if (device) {
- sdev->hostdata = NULL;
- if (!list_empty(&device->scsi_device_list_entry))
- list_del(&device->scsi_device_list_entry);
- }
-
- spin_unlock_irqrestore(&ctrl_info->scsi_device_list_lock, flags);
+ device->devtype = sdev->type;
- if (device) {
- pqi_dev_info(ctrl_info, "removed", device);
- pqi_free_device(device);
+ if (pqi_is_tape_changer_device(device) && device->ignore_device) {
+ rc = -ENXIO;
+ device->ignore_device = false;
}
+
+ return rc;
}
static int pqi_getpciinfo_ioctl(struct pqi_ctrl_info *ctrl_info, void __user *arg)
static DEVICE_ATTR(enable_r6_writes, 0644,
pqi_host_enable_r6_writes_show, pqi_host_enable_r6_writes_store);
-static struct device_attribute *pqi_shost_attrs[] = {
- &dev_attr_driver_version,
- &dev_attr_firmware_version,
- &dev_attr_model,
- &dev_attr_serial_number,
- &dev_attr_vendor,
- &dev_attr_rescan,
- &dev_attr_lockup_action,
- &dev_attr_enable_stream_detection,
- &dev_attr_enable_r5_writes,
- &dev_attr_enable_r6_writes,
+static struct attribute *pqi_shost_attrs[] = {
+ &dev_attr_driver_version.attr,
+ &dev_attr_firmware_version.attr,
+ &dev_attr_model.attr,
+ &dev_attr_serial_number.attr,
+ &dev_attr_vendor.attr,
+ &dev_attr_rescan.attr,
+ &dev_attr_lockup_action.attr,
+ &dev_attr_enable_stream_detection.attr,
+ &dev_attr_enable_r5_writes.attr,
+ &dev_attr_enable_r6_writes.attr,
NULL
};
+ATTRIBUTE_GROUPS(pqi_shost);
+
static ssize_t pqi_unique_id_show(struct device *dev,
struct device_attribute *attr, char *buffer)
{
return -ENODEV;
}
- if (device->is_physical_device) {
- memset(unique_id, 0, 8);
- memcpy(unique_id + 8, &device->wwid, sizeof(device->wwid));
- } else {
+ if (device->is_physical_device)
+ memcpy(unique_id, device->wwid, sizeof(device->wwid));
+ else
memcpy(unique_id, device->volume_id, sizeof(device->volume_id));
- }
spin_unlock_irqrestore(&ctrl_info->scsi_device_list_lock, flags);
static DEVICE_ATTR(raid_level, 0444, pqi_raid_level_show, NULL);
static DEVICE_ATTR(raid_bypass_cnt, 0444, pqi_raid_bypass_cnt_show, NULL);
-static struct device_attribute *pqi_sdev_attrs[] = {
- &dev_attr_lunid,
- &dev_attr_unique_id,
- &dev_attr_path_info,
- &dev_attr_sas_address,
- &dev_attr_ssd_smart_path_enabled,
- &dev_attr_raid_level,
- &dev_attr_raid_bypass_cnt,
+static struct attribute *pqi_sdev_attrs[] = {
+ &dev_attr_lunid.attr,
+ &dev_attr_unique_id.attr,
+ &dev_attr_path_info.attr,
+ &dev_attr_sas_address.attr,
+ &dev_attr_ssd_smart_path_enabled.attr,
+ &dev_attr_raid_level.attr,
+ &dev_attr_raid_bypass_cnt.attr,
NULL
};
+ATTRIBUTE_GROUPS(pqi_sdev);
+
static struct scsi_host_template pqi_driver_template = {
.module = THIS_MODULE,
.name = DRIVER_NAME_SHORT,
.ioctl = pqi_ioctl,
.slave_alloc = pqi_slave_alloc,
.slave_configure = pqi_slave_configure,
- .slave_destroy = pqi_slave_destroy,
.map_queues = pqi_map_queues,
- .sdev_attrs = pqi_sdev_attrs,
- .shost_attrs = pqi_shost_attrs,
+ .sdev_groups = pqi_sdev_groups,
+ .shost_groups = pqi_shost_groups,
};
static int pqi_register_scsi(struct pqi_ctrl_info *ctrl_info)
ctrl_info->unique_wwid_in_report_phys_lun_supported =
firmware_feature->enabled;
break;
+ case PQI_FIRMWARE_FEATURE_FW_TRIAGE:
+ ctrl_info->firmware_triage_supported = firmware_feature->enabled;
+ pqi_save_fw_triage_setting(ctrl_info, firmware_feature->enabled);
+ break;
+ case PQI_FIRMWARE_FEATURE_RPL_EXTENDED_FORMAT_4_5:
+ ctrl_info->rpl_extended_format_4_5_supported = firmware_feature->enabled;
+ break;
}
pqi_firmware_feature_status(ctrl_info, firmware_feature);
.feature_bit = PQI_FIRMWARE_FEATURE_UNIQUE_WWID_IN_REPORT_PHYS_LUN,
.feature_status = pqi_ctrl_update_feature_flags,
},
+ {
+ .feature_name = "Firmware Triage",
+ .feature_bit = PQI_FIRMWARE_FEATURE_FW_TRIAGE,
+ .feature_status = pqi_ctrl_update_feature_flags,
+ },
+ {
+ .feature_name = "RPL Extended Formats 4 and 5",
+ .feature_bit = PQI_FIRMWARE_FEATURE_RPL_EXTENDED_FORMAT_4_5,
+ .feature_status = pqi_ctrl_update_feature_flags,
+ },
};
static void pqi_process_firmware_features(
ctrl_info->raid_iu_timeout_supported = false;
ctrl_info->tmf_iu_timeout_supported = false;
ctrl_info->unique_wwid_in_report_phys_lun_supported = false;
+ ctrl_info->firmware_triage_supported = false;
+ ctrl_info->rpl_extended_format_4_5_supported = false;
}
static int pqi_process_config_table(struct pqi_ctrl_info *ctrl_info)
u32 product_id;
if (reset_devices) {
+ if (pqi_is_fw_triage_supported(ctrl_info)) {
+ rc = sis_wait_for_fw_triage_completion(ctrl_info);
+ if (rc)
+ return rc;
+ }
sis_soft_reset(ctrl_info);
msleep(PQI_POST_RESET_DELAY_SECS * PQI_HZ);
} else {
{
pqi_cancel_rescan_worker(ctrl_info);
pqi_cancel_update_time_worker(ctrl_info);
+ pqi_remove_all_scsi_devices(ctrl_info);
pqi_unregister_scsi(ctrl_info);
if (ctrl_info->pqi_mode_enabled)
pqi_revert_to_sis_mode(ctrl_info);
unsigned int i;
struct pqi_io_request *io_request;
struct scsi_cmnd *scmd;
+ struct scsi_device *sdev;
for (i = 0; i < ctrl_info->max_io_slots; i++) {
io_request = &ctrl_info->io_request_pool[i];
scmd = io_request->scmd;
if (scmd) {
- set_host_byte(scmd, DID_NO_CONNECT);
+ sdev = scmd->device;
+ if (!sdev || !scsi_device_online(sdev)) {
+ pqi_free_io_request(io_request);
+ continue;
+ } else {
+ set_host_byte(scmd, DID_NO_CONNECT);
+ }
} else {
io_request->status = -ENXIO;
io_request->error_info =
pqi_take_ctrl_offline_deferred(ctrl_info);
}
-static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info)
+static void pqi_take_ctrl_offline(struct pqi_ctrl_info *ctrl_info,
+ enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason)
{
if (!ctrl_info->controller_online)
return;
ctrl_info->pqi_mode_enabled = false;
pqi_ctrl_block_requests(ctrl_info);
if (!pqi_disable_ctrl_shutdown)
- sis_shutdown_ctrl(ctrl_info);
+ sis_shutdown_ctrl(ctrl_info, ctrl_shutdown_reason);
pci_disable_device(ctrl_info->pci_dev);
dev_err(&ctrl_info->pci_dev->dev, "controller offline\n");
schedule_work(&ctrl_info->ctrl_offline_work);
},
{
PCI_DEVICE_SUB(PCI_VENDOR_ID_ADAPTEC2, 0x028f,
+ PCI_VENDOR_ID_ADAPTEC2, 0x14a2)
+ },
+ {
+ PCI_DEVICE_SUB(PCI_VENDOR_ID_ADAPTEC2, 0x028f,
PCI_VENDOR_ID_ADAPTEC2, 0x14b0)
},
{
BUILD_BUG_ON(offsetof(struct pqi_ctrl_registers,
sis_firmware_status) != 0xbc);
BUILD_BUG_ON(offsetof(struct pqi_ctrl_registers,
+ sis_ctrl_shutdown_reason_code) != 0xcc);
+ BUILD_BUG_ON(offsetof(struct pqi_ctrl_registers,
sis_mailbox) != 0x1000);
BUILD_BUG_ON(offsetof(struct pqi_ctrl_registers,
pqi_registers) != 0x4000);
}
if (found_device->devtype == TYPE_ENCLOSURE) {
- *identifier = get_unaligned_be64(&found_device->wwid);
+ *identifier = get_unaligned_be64(&found_device->wwid[8]);
rc = 0;
goto out;
}
memcmp(device->phys_connector,
found_device->phys_connector, 2) == 0) {
*identifier =
- get_unaligned_be64(&device->wwid);
+ get_unaligned_be64(&device->wwid[8]);
rc = 0;
goto out;
}
if (device->devtype == TYPE_ENCLOSURE &&
CISS_GET_DRIVE_NUMBER(device->scsi3addr) ==
PQI_VSEP_CISS_BTL) {
- *identifier = get_unaligned_be64(&device->wwid);
+ *identifier = get_unaligned_be64(&device->wwid[8]);
rc = 0;
goto out;
}
#define SIS_BASE_STRUCT_REVISION 9
#define SIS_BASE_STRUCT_ALIGNMENT 16
+#define SIS_CTRL_KERNEL_FW_TRIAGE 0x3
#define SIS_CTRL_KERNEL_UP 0x80
#define SIS_CTRL_KERNEL_PANIC 0x100
#define SIS_CTRL_READY_TIMEOUT_SECS 180
#define SIS_CTRL_READY_RESUME_TIMEOUT_SECS 90
#define SIS_CTRL_READY_POLL_INTERVAL_MSECS 10
+enum sis_fw_triage_status {
+ FW_TRIAGE_NOT_STARTED = 0,
+ FW_TRIAGE_STARTED,
+ FW_TRIAGE_COND_INVALID,
+ FW_TRIAGE_COMPLETED
+};
+
#pragma pack(1)
/* for use with SIS_CMD_INIT_BASE_STRUCT_ADDRESS command */
sis_set_doorbell_bit(ctrl_info, SIS_ENABLE_INTX);
}
-void sis_shutdown_ctrl(struct pqi_ctrl_info *ctrl_info)
+void sis_shutdown_ctrl(struct pqi_ctrl_info *ctrl_info,
+ enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason)
{
if (readl(&ctrl_info->registers->sis_firmware_status) &
SIS_CTRL_KERNEL_PANIC)
return;
- writel(SIS_TRIGGER_SHUTDOWN,
- &ctrl_info->registers->sis_host_to_ctrl_doorbell);
+ if (ctrl_info->firmware_triage_supported)
+ writel(ctrl_shutdown_reason, &ctrl_info->registers->sis_ctrl_shutdown_reason_code);
+
+ writel(SIS_TRIGGER_SHUTDOWN, &ctrl_info->registers->sis_host_to_ctrl_doorbell);
}
int sis_pqi_reset_quiesce(struct pqi_ctrl_info *ctrl_info)
return readl(&ctrl_info->registers->sis_driver_scratch);
}
+static inline enum sis_fw_triage_status
+ sis_read_firmware_triage_status(struct pqi_ctrl_info *ctrl_info)
+{
+ return ((enum sis_fw_triage_status)(readl(&ctrl_info->registers->sis_firmware_status) &
+ SIS_CTRL_KERNEL_FW_TRIAGE));
+}
+
void sis_soft_reset(struct pqi_ctrl_info *ctrl_info)
{
writel(SIS_SOFT_RESET,
&ctrl_info->registers->sis_host_to_ctrl_doorbell);
}
+#define SIS_FW_TRIAGE_STATUS_TIMEOUT_SECS 300
+#define SIS_FW_TRIAGE_STATUS_POLL_INTERVAL_SECS 1
+
+int sis_wait_for_fw_triage_completion(struct pqi_ctrl_info *ctrl_info)
+{
+ int rc;
+ enum sis_fw_triage_status status;
+ unsigned long timeout;
+
+ timeout = (SIS_FW_TRIAGE_STATUS_TIMEOUT_SECS * PQI_HZ) + jiffies;
+ while (1) {
+ status = sis_read_firmware_triage_status(ctrl_info);
+ if (status == FW_TRIAGE_COND_INVALID) {
+ dev_err(&ctrl_info->pci_dev->dev,
+ "firmware triage condition invalid\n");
+ rc = -EINVAL;
+ break;
+ } else if (status == FW_TRIAGE_NOT_STARTED ||
+ status == FW_TRIAGE_COMPLETED) {
+ rc = 0;
+ break;
+ }
+
+ if (time_after(jiffies, timeout)) {
+ dev_err(&ctrl_info->pci_dev->dev,
+ "timed out waiting for firmware triage status\n");
+ rc = -ETIMEDOUT;
+ break;
+ }
+
+ ssleep(SIS_FW_TRIAGE_STATUS_POLL_INTERVAL_SECS);
+ }
+
+ return rc;
+}
+
static void __attribute__((unused)) verify_structures(void)
{
BUILD_BUG_ON(offsetof(struct sis_base_struct,
int sis_init_base_struct_addr(struct pqi_ctrl_info *ctrl_info);
void sis_enable_msix(struct pqi_ctrl_info *ctrl_info);
void sis_enable_intx(struct pqi_ctrl_info *ctrl_info);
-void sis_shutdown_ctrl(struct pqi_ctrl_info *ctrl_info);
+void sis_shutdown_ctrl(struct pqi_ctrl_info *ctrl_info,
+ enum pqi_ctrl_shutdown_reason ctrl_shutdown_reason);
int sis_pqi_reset_quiesce(struct pqi_ctrl_info *ctrl_info);
int sis_reenable_sis_mode(struct pqi_ctrl_info *ctrl_info);
void sis_write_driver_scratch(struct pqi_ctrl_info *ctrl_info, u32 value);
u32 sis_read_driver_scratch(struct pqi_ctrl_info *ctrl_info);
void sis_soft_reset(struct pqi_ctrl_info *ctrl_info);
u32 sis_get_product_id(struct pqi_ctrl_info *ctrl_info);
+int sis_wait_for_fw_triage_completion(struct pqi_ctrl_info *ctrl_info);
#endif /* _SMARTPQI_SIS_H */
void snic_glob_cleanup(void);
extern struct workqueue_struct *snic_event_queue;
-extern struct device_attribute *snic_attrs[];
+extern const struct attribute_group *snic_host_groups[];
int snic_queuecommand(struct Scsi_Host *, struct scsi_cmnd *);
int snic_abort_cmd(struct scsi_cmnd *);
static DEVICE_ATTR(drv_version, S_IRUGO, snic_show_drv_version, NULL);
static DEVICE_ATTR(link_state, S_IRUGO, snic_show_link_state, NULL);
-struct device_attribute *snic_attrs[] = {
- &dev_attr_snic_sym_name,
- &dev_attr_snic_state,
- &dev_attr_drv_version,
- &dev_attr_link_state,
+static struct attribute *snic_host_attrs[] = {
+ &dev_attr_snic_sym_name.attr,
+ &dev_attr_snic_state.attr,
+ &dev_attr_drv_version.attr,
+ &dev_attr_link_state.attr,
NULL,
};
+
+static const struct attribute_group snic_host_attr_group = {
+ .attrs = snic_host_attrs
+};
+
+const struct attribute_group *snic_host_groups[] = {
+ &snic_host_attr_group,
+ NULL
+};
.can_queue = SNIC_MAX_IO_REQ,
.sg_tablesize = SNIC_MAX_SG_DESC_CNT,
.max_sectors = 0x800,
- .shost_attrs = snic_attrs,
+ .shost_groups = snic_host_groups,
.track_queue_depth = 1,
.cmd_size = sizeof(struct snic_internal_io_state),
.proc_name = "snic_scsi",
SNIC_HOST_ERR(shost, "Tgt %p id %d Not Ready.\n", tgt, tgt->id);
atomic64_inc(&snic->s_stats.misc.tgt_not_rdy);
sc->result = ret;
- sc->scsi_done(sc);
+ scsi_done(sc);
return 0;
}
SNIC_TRC_CMD(sc), SNIC_TRC_CMD_STATE_FLAGS(sc));
- if (sc->scsi_done)
- sc->scsi_done(sc);
+ scsi_done(sc);
snic_stats_update_io_cmpl(&snic->s_stats);
} /* end of snic_icmnd_cmpl_handler */
snic_release_req_buf(snic, rqi, sc);
- if (sc->scsi_done) {
- SNIC_TRC(snic->shost->host_no, cmnd_id, (ulong) sc,
- jiffies_to_msecs(jiffies - start_time),
- (ulong) fwreq, SNIC_TRC_CMD(sc),
- SNIC_TRC_CMD_STATE_FLAGS(sc));
+ SNIC_TRC(snic->shost->host_no, cmnd_id, (ulong) sc,
+ jiffies_to_msecs(jiffies - start_time),
+ (ulong) fwreq, SNIC_TRC_CMD(sc),
+ SNIC_TRC_CMD_STATE_FLAGS(sc));
- sc->scsi_done(sc);
- }
+ scsi_done(sc);
break;
* Call scsi_done to complete the IO.
*/
sc->result = (DID_ERROR << 16);
- sc->scsi_done(sc);
+ scsi_done(sc);
break;
default:
snic_release_req_buf(snic, rqi, sc);
sc->result = (DID_ERROR << 16);
- sc->scsi_done(sc);
+ scsi_done(sc);
ret = 0;
/* Update IO stats */
snic_stats_update_io_cmpl(&snic->s_stats);
- if (sc->scsi_done) {
- SNIC_TRC(snic->shost->host_no, tag, (ulong) sc,
- jiffies_to_msecs(jiffies - st_time), 0,
- SNIC_TRC_CMD(sc),
- SNIC_TRC_CMD_STATE_FLAGS(sc));
+ SNIC_TRC(snic->shost->host_no, tag, (ulong) sc,
+ jiffies_to_msecs(jiffies - st_time), 0,
+ SNIC_TRC_CMD(sc),
+ SNIC_TRC_CMD_STATE_FLAGS(sc));
- sc->scsi_done(sc);
- }
+ scsi_done(sc);
}
} /* end of snic_scsi_cleanup */
cd->device = sdev;
cd->disk = disk;
cd->driver = &sr_template;
- cd->disk = disk;
cd->capacity = 0x1fffff;
cd->device->changed = 1; /* force recheck CD type */
cd->media_present = 1;
dev_set_drvdata(dev, cd);
disk->flags |= GENHD_FL_REMOVABLE;
sr_revalidate_disk(cd);
- device_add_disk(&sdev->sdev_gendev, disk, NULL);
+
+ error = device_add_disk(&sdev->sdev_gendev, disk, NULL);
+ if (error) {
+ kref_put(&cd->kref, sr_kref_release);
+ goto fail;
+ }
sdev_printk(KERN_DEBUG, sdev,
"Attached scsi CD-ROM %s\n", cd->cdi.name);
if (ccb->cmd) {
scsi_dma_unmap(ccb->cmd);
ccb->cmd->result = status << 16;
- ccb->cmd->scsi_done(ccb->cmd);
+ scsi_done(ccb->cmd);
ccb->cmd = NULL;
}
}
return 0;
}
-static int
-stex_queuecommand_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int stex_queuecommand_lck(struct scsi_cmnd *cmd)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct st_hba *hba;
struct Scsi_Host *host;
unsigned int id, lun;
break;
}
- cmd->scsi_done = done;
-
tag = scsi_cmd_to_rq(cmd)->tag;
if (unlikely(tag >= host->can_queue))
}
cmd->result = result;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
static void stex_copy_data(struct st_ccb *ccb,
scsi_set_resid(scmnd,
cmd_request->payload->range.len - data_transfer_length);
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
if (payload_sz >
sizeof(struct vmbus_channel_packet_multipage_buffer))
* future versions of the host.
*/
if (!storvsc_scsi_cmd_ok(scmnd)) {
- scmnd->scsi_done(scmnd);
+ scsi_done(scmnd);
return 0;
}
}
complete(ucmd->eh_done);
scsi_dma_unmap(cmd);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
/*
* queuecommand method. Entered with the host adapter lock held and
* interrupts disabled.
*/
-static int sym53c8xx_queue_command_lck(struct scsi_cmnd *cmd,
- void (*done)(struct scsi_cmnd *))
+static int sym53c8xx_queue_command_lck(struct scsi_cmnd *cmd)
{
struct sym_hcb *np = SYM_SOFTC_PTR(cmd);
struct sym_ucmd *ucp = SYM_UCMD_PTR(cmd);
int sts = 0;
- cmd->scsi_done = done;
memset(ucp, 0, sizeof(*ucp));
/*
If unsure, say N.
config SCSI_UFS_EXYNOS
- tristate "EXYNOS specific hooks to UFS controller platform driver"
+ tristate "Exynos specific hooks to UFS controller platform driver"
depends on SCSI_UFSHCD_PLATFORM && (ARCH_EXYNOS || COMPILE_TEST)
help
- This selects the EXYNOS specific additions to UFSHCD platform driver.
- UFS host on EXYNOS includes HCI and UNIPRO layer, and associates with
- UFS-PHY driver.
+ This selects the Samsung Exynos SoC specific additions to UFSHCD
+ platform driver. UFS host on Samsung Exynos SoC includes HCI and
+ UNIPRO layer, and associates with UFS-PHY driver.
- Select this if you have UFS host controller on EXYNOS chipset.
+ Select this if you have UFS host controller on Samsung Exynos SoC.
If unsure, say N.
config SCSI_UFS_CRYPTO
help
Enable fault injection support in the UFS driver. This makes it easier
to test the UFS error handler and abort handler.
+
+config SCSI_UFS_HWMON
+ bool "UFS Temperature Notification"
+ depends on SCSI_UFSHCD=HWMON || HWMON=y
+ help
+ This provides support for UFS hardware monitoring. If enabled,
+ a hardware monitoring device will be created for the UFS device.
+
+ If unsure, say N.
ufshcd-core-$(CONFIG_SCSI_UFS_CRYPTO) += ufshcd-crypto.o
ufshcd-core-$(CONFIG_SCSI_UFS_HPB) += ufshpb.o
ufshcd-core-$(CONFIG_SCSI_UFS_FAULT_INJECTION) += ufs-fault-injection.o
+ufshcd-core-$(CONFIG_SCSI_UFS_HWMON) += ufs-hwmon.o
obj-$(CONFIG_SCSI_UFS_DWC_TC_PCI) += tc-dwc-g210-pci.o ufshcd-dwc.o tc-dwc-g210.o
obj-$(CONFIG_SCSI_UFS_DWC_TC_PLATFORM) += tc-dwc-g210-pltfrm.o ufshcd-dwc.o tc-dwc-g210.o
static struct dentry *ufs_debugfs_root;
+struct ufs_debugfs_attr {
+ const char *name;
+ mode_t mode;
+ const struct file_operations *fops;
+};
+
+/* @file corresponds to a debugfs attribute in directory hba->debugfs_root. */
+static inline struct ufs_hba *hba_from_file(const struct file *file)
+{
+ return d_inode(file->f_path.dentry->d_parent)->i_private;
+}
+
void __init ufs_debugfs_init(void)
{
ufs_debugfs_root = debugfs_create_dir("ufshcd", NULL);
static int ufs_debugfs_stats_show(struct seq_file *s, void *data)
{
- struct ufs_hba *hba = s->private;
+ struct ufs_hba *hba = hba_from_file(s->file);
struct ufs_event_hist *e = hba->ufs_stats.event;
#define PRT(fmt, typ) \
ufs_debugfs_put_user_access(hba);
}
+static int ufs_saved_err_show(struct seq_file *s, void *data)
+{
+ struct ufs_debugfs_attr *attr = s->private;
+ struct ufs_hba *hba = hba_from_file(s->file);
+ const int *p;
+
+ if (strcmp(attr->name, "saved_err") == 0) {
+ p = &hba->saved_err;
+ } else if (strcmp(attr->name, "saved_uic_err") == 0) {
+ p = &hba->saved_uic_err;
+ } else {
+ return -ENOENT;
+ }
+
+ seq_printf(s, "%d\n", *p);
+ return 0;
+}
+
+static ssize_t ufs_saved_err_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct ufs_debugfs_attr *attr = file->f_inode->i_private;
+ struct ufs_hba *hba = hba_from_file(file);
+ char val_str[16] = { };
+ int val, ret;
+
+ if (count > sizeof(val_str))
+ return -EINVAL;
+ if (copy_from_user(val_str, buf, count))
+ return -EFAULT;
+ ret = kstrtoint(val_str, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ spin_lock_irq(hba->host->host_lock);
+ if (strcmp(attr->name, "saved_err") == 0) {
+ hba->saved_err = val;
+ } else if (strcmp(attr->name, "saved_uic_err") == 0) {
+ hba->saved_uic_err = val;
+ } else {
+ ret = -ENOENT;
+ }
+ if (ret == 0)
+ ufshcd_schedule_eh_work(hba);
+ spin_unlock_irq(hba->host->host_lock);
+
+ return ret < 0 ? ret : count;
+}
+
+static int ufs_saved_err_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, ufs_saved_err_show, inode->i_private);
+}
+
+static const struct file_operations ufs_saved_err_fops = {
+ .owner = THIS_MODULE,
+ .open = ufs_saved_err_open,
+ .read = seq_read,
+ .write = ufs_saved_err_write,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const struct ufs_debugfs_attr ufs_attrs[] = {
+ { "stats", 0400, &ufs_debugfs_stats_fops },
+ { "saved_err", 0600, &ufs_saved_err_fops },
+ { "saved_uic_err", 0600, &ufs_saved_err_fops },
+ { }
+};
+
void ufs_debugfs_hba_init(struct ufs_hba *hba)
{
+ const struct ufs_debugfs_attr *attr;
+ struct dentry *root;
+
/* Set default exception event rate limit period to 20ms */
hba->debugfs_ee_rate_limit_ms = 20;
INIT_DELAYED_WORK(&hba->debugfs_ee_work, ufs_debugfs_restart_ee);
- hba->debugfs_root = debugfs_create_dir(dev_name(hba->dev), ufs_debugfs_root);
- debugfs_create_file("stats", 0400, hba->debugfs_root, hba, &ufs_debugfs_stats_fops);
+
+ root = debugfs_create_dir(dev_name(hba->dev), ufs_debugfs_root);
+ if (IS_ERR_OR_NULL(root))
+ return;
+ hba->debugfs_root = root;
+ d_inode(root)->i_private = hba;
+ for (attr = ufs_attrs; attr->name; attr++)
+ debugfs_create_file(attr->name, attr->mode, root, (void *)attr,
+ attr->fops);
debugfs_create_file("exception_event_mask", 0600, hba->debugfs_root,
hba, &ee_usr_mask_fops);
debugfs_create_u32("exception_event_rate_limit_ms", 0600, hba->debugfs_root,
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_address.h>
+#include <linux/mfd/syscon.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
+#include <linux/regmap.h>
#include "ufshcd.h"
#include "ufshcd-pltfrm.h"
#define HCI_ERR_EN_T_LAYER 0x84
#define HCI_ERR_EN_DME_LAYER 0x88
#define HCI_CLKSTOP_CTRL 0xB0
+#define REFCLKOUT_STOP BIT(4)
#define REFCLK_STOP BIT(2)
#define UNIPRO_MCLK_STOP BIT(1)
#define UNIPRO_PCLK_STOP BIT(0)
-#define CLK_STOP_MASK (REFCLK_STOP |\
+#define CLK_STOP_MASK (REFCLKOUT_STOP | REFCLK_STOP |\
UNIPRO_MCLK_STOP |\
UNIPRO_PCLK_STOP)
#define HCI_MISC 0xB4
UIC_TRANSPORT_NO_CONNECTION_RX |\
UIC_TRANSPORT_BAD_TC)
+/* FSYS UFS Shareability */
+#define UFS_WR_SHARABLE BIT(2)
+#define UFS_RD_SHARABLE BIT(1)
+#define UFS_SHARABLE (UFS_WR_SHARABLE | UFS_RD_SHARABLE)
+#define UFS_SHAREABILITY_OFFSET 0x710
+
+/* Multi-host registers */
+#define MHCTRL 0xC4
+#define MHCTRL_EN_VH_MASK (0xE)
+#define MHCTRL_EN_VH(vh) (vh << 1)
+#define PH2VH_MBOX 0xD8
+
+#define MH_MSG_MASK (0xFF)
+
+#define MH_MSG(id, msg) ((id << 8) | (msg & 0xFF))
+#define MH_MSG_PH_READY 0x1
+#define MH_MSG_VH_READY 0x2
+
+#define ALLOW_INQUIRY BIT(25)
+#define ALLOW_MODE_SELECT BIT(24)
+#define ALLOW_MODE_SENSE BIT(23)
+#define ALLOW_PRE_FETCH GENMASK(22, 21)
+#define ALLOW_READ_CMD_ALL GENMASK(20, 18) /* read_6/10/16 */
+#define ALLOW_READ_BUFFER BIT(17)
+#define ALLOW_READ_CAPACITY GENMASK(16, 15)
+#define ALLOW_REPORT_LUNS BIT(14)
+#define ALLOW_REQUEST_SENSE BIT(13)
+#define ALLOW_SYNCHRONIZE_CACHE GENMASK(8, 7)
+#define ALLOW_TEST_UNIT_READY BIT(6)
+#define ALLOW_UNMAP BIT(5)
+#define ALLOW_VERIFY BIT(4)
+#define ALLOW_WRITE_CMD_ALL GENMASK(3, 1) /* write_6/10/16 */
+
+#define ALLOW_TRANS_VH_DEFAULT (ALLOW_INQUIRY | ALLOW_MODE_SELECT | \
+ ALLOW_MODE_SENSE | ALLOW_PRE_FETCH | \
+ ALLOW_READ_CMD_ALL | ALLOW_READ_BUFFER | \
+ ALLOW_READ_CAPACITY | ALLOW_REPORT_LUNS | \
+ ALLOW_REQUEST_SENSE | ALLOW_SYNCHRONIZE_CACHE | \
+ ALLOW_TEST_UNIT_READY | ALLOW_UNMAP | \
+ ALLOW_VERIFY | ALLOW_WRITE_CMD_ALL)
+
+#define HCI_MH_ALLOWABLE_TRAN_OF_VH 0x30C
+#define HCI_MH_IID_IN_TASK_TAG 0X308
+
+#define PH_READY_TIMEOUT_MS (5 * MSEC_PER_SEC)
+
enum {
UNIPRO_L1_5 = 0,/* PHY Adapter */
UNIPRO_L2, /* Data Link */
return 0;
}
+static int exynosauto_ufs_drv_init(struct device *dev, struct exynos_ufs *ufs)
+{
+ struct exynos_ufs_uic_attr *attr = ufs->drv_data->uic_attr;
+
+ /* IO Coherency setting */
+ if (ufs->sysreg) {
+ return regmap_update_bits(ufs->sysreg,
+ ufs->shareability_reg_offset,
+ UFS_SHARABLE, UFS_SHARABLE);
+ }
+
+ attr->tx_dif_p_nsec = 3200000;
+
+ return 0;
+}
+
+static int exynosauto_ufs_post_hce_enable(struct exynos_ufs *ufs)
+{
+ struct ufs_hba *hba = ufs->hba;
+
+ /* Enable Virtual Host #1 */
+ ufshcd_rmwl(hba, MHCTRL_EN_VH_MASK, MHCTRL_EN_VH(1), MHCTRL);
+ /* Default VH Transfer permissions */
+ hci_writel(ufs, ALLOW_TRANS_VH_DEFAULT, HCI_MH_ALLOWABLE_TRAN_OF_VH);
+ /* IID information is replaced in TASKTAG[7:5] instead of IID in UCD */
+ hci_writel(ufs, 0x1, HCI_MH_IID_IN_TASK_TAG);
+
+ return 0;
+}
+
+static int exynosauto_ufs_pre_link(struct exynos_ufs *ufs)
+{
+ struct ufs_hba *hba = ufs->hba;
+ int i;
+ u32 tx_line_reset_period, rx_line_reset_period;
+
+ rx_line_reset_period = (RX_LINE_RESET_TIME * ufs->mclk_rate) / NSEC_PER_MSEC;
+ tx_line_reset_period = (TX_LINE_RESET_TIME * ufs->mclk_rate) / NSEC_PER_MSEC;
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB(0x200), 0x40);
+ for_each_ufs_rx_lane(ufs, i) {
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_RX_CLK_PRD, i),
+ DIV_ROUND_UP(NSEC_PER_SEC, ufs->mclk_rate));
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_RX_CLK_PRD_EN, i), 0x0);
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_RX_LINERESET_VALUE2, i),
+ (rx_line_reset_period >> 16) & 0xFF);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_RX_LINERESET_VALUE1, i),
+ (rx_line_reset_period >> 8) & 0xFF);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_RX_LINERESET_VALUE0, i),
+ (rx_line_reset_period) & 0xFF);
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(0x2f, i), 0x79);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(0x84, i), 0x1);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(0x25, i), 0xf6);
+ }
+
+ for_each_ufs_tx_lane(ufs, i) {
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_TX_CLK_PRD, i),
+ DIV_ROUND_UP(NSEC_PER_SEC, ufs->mclk_rate));
+ /* Not to affect VND_TX_LINERESET_PVALUE to VND_TX_CLK_PRD */
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_TX_CLK_PRD_EN, i),
+ 0x02);
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_TX_LINERESET_PVALUE2, i),
+ (tx_line_reset_period >> 16) & 0xFF);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_TX_LINERESET_PVALUE1, i),
+ (tx_line_reset_period >> 8) & 0xFF);
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(VND_TX_LINERESET_PVALUE0, i),
+ (tx_line_reset_period) & 0xFF);
+
+ /* TX PWM Gear Capability / PWM_G1_ONLY */
+ ufshcd_dme_set(hba, UIC_ARG_MIB_SEL(0x04, i), 0x1);
+ }
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB(0x200), 0x0);
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB(PA_LOCAL_TX_LCC_ENABLE), 0x0);
+
+ ufshcd_dme_set(hba, UIC_ARG_MIB(0xa011), 0x8000);
+
+ return 0;
+}
+
+static int exynosauto_ufs_pre_pwr_change(struct exynos_ufs *ufs,
+ struct ufs_pa_layer_attr *pwr)
+{
+ struct ufs_hba *hba = ufs->hba;
+
+ /* PACP_PWR_req and delivered to the remote DME */
+ ufshcd_dme_set(hba, UIC_ARG_MIB(PA_PWRMODEUSERDATA0), 12000);
+ ufshcd_dme_set(hba, UIC_ARG_MIB(PA_PWRMODEUSERDATA1), 32000);
+ ufshcd_dme_set(hba, UIC_ARG_MIB(PA_PWRMODEUSERDATA2), 16000);
+
+ return 0;
+}
+
+static int exynosauto_ufs_post_pwr_change(struct exynos_ufs *ufs,
+ struct ufs_pa_layer_attr *pwr)
+{
+ struct ufs_hba *hba = ufs->hba;
+ u32 enabled_vh;
+
+ enabled_vh = ufshcd_readl(hba, MHCTRL) & MHCTRL_EN_VH_MASK;
+
+ /* Send physical host ready message to virtual hosts */
+ ufshcd_writel(hba, MH_MSG(enabled_vh, MH_MSG_PH_READY), PH2VH_MBOX);
+
+ return 0;
+}
+
static int exynos7_ufs_pre_link(struct exynos_ufs *ufs)
{
struct ufs_hba *hba = ufs->hba;
}
}
+static int exynos_ufs_setup_clocks(struct ufs_hba *hba, bool on,
+ enum ufs_notify_change_status status)
+{
+ struct exynos_ufs *ufs = ufshcd_get_variant(hba);
+
+ if (!ufs)
+ return 0;
+
+ if (on && status == PRE_CHANGE) {
+ if (ufs->opts & EXYNOS_UFS_OPT_BROKEN_AUTO_CLK_CTRL)
+ exynos_ufs_disable_auto_ctrl_hcc(ufs);
+ exynos_ufs_ungate_clks(ufs);
+ } else if (!on && status == POST_CHANGE) {
+ exynos_ufs_gate_clks(ufs);
+ if (ufs->opts & EXYNOS_UFS_OPT_BROKEN_AUTO_CLK_CTRL)
+ exynos_ufs_enable_auto_ctrl_hcc(ufs);
+ }
+
+ return 0;
+}
+
static int exynos_ufs_pre_link(struct ufs_hba *hba)
{
struct exynos_ufs *ufs = ufshcd_get_variant(hba);
/* m-phy */
exynos_ufs_phy_init(ufs);
- exynos_ufs_config_phy_time_attr(ufs);
- exynos_ufs_config_phy_cap_attr(ufs);
+ if (!(ufs->opts & EXYNOS_UFS_OPT_SKIP_CONFIG_PHY_ATTR)) {
+ exynos_ufs_config_phy_time_attr(ufs);
+ exynos_ufs_config_phy_cap_attr(ufs);
+ }
+
+ exynos_ufs_setup_clocks(hba, true, PRE_CHANGE);
if (ufs->drv_data->pre_link)
ufs->drv_data->pre_link(ufs);
static int exynos_ufs_parse_dt(struct device *dev, struct exynos_ufs *ufs)
{
struct device_node *np = dev->of_node;
- struct exynos_ufs_drv_data *drv_data = &exynos_ufs_drvs;
struct exynos_ufs_uic_attr *attr;
int ret = 0;
- while (drv_data->compatible) {
- if (of_device_is_compatible(np, drv_data->compatible)) {
- ufs->drv_data = drv_data;
- break;
- }
- drv_data++;
- }
+ ufs->drv_data = device_get_match_data(dev);
if (ufs->drv_data && ufs->drv_data->uic_attr) {
attr = ufs->drv_data->uic_attr;
goto out;
}
+ ufs->sysreg = syscon_regmap_lookup_by_phandle(np, "samsung,sysreg");
+ if (IS_ERR(ufs->sysreg))
+ ufs->sysreg = NULL;
+ else {
+ if (of_property_read_u32_index(np, "samsung,sysreg", 1,
+ &ufs->shareability_reg_offset)) {
+ dev_warn(dev, "can't get an offset from sysreg. Set to default value\n");
+ ufs->shareability_reg_offset = UFS_SHAREABILITY_OFFSET;
+ }
+ }
+
ufs->pclk_avail_min = PCLK_AVAIL_MIN;
ufs->pclk_avail_max = PCLK_AVAIL_MAX;
return ret;
}
+static inline void exynos_ufs_priv_init(struct ufs_hba *hba,
+ struct exynos_ufs *ufs)
+{
+ ufs->hba = hba;
+ ufs->opts = ufs->drv_data->opts;
+ ufs->rx_sel_idx = PA_MAXDATALANES;
+ if (ufs->opts & EXYNOS_UFS_OPT_BROKEN_RX_SEL_IDX)
+ ufs->rx_sel_idx = 0;
+ hba->priv = (void *)ufs;
+ hba->quirks = ufs->drv_data->quirks;
+}
+
static int exynos_ufs_init(struct ufs_hba *hba)
{
struct device *dev = hba->dev;
if (ret)
goto phy_off;
- ufs->hba = hba;
- ufs->opts = ufs->drv_data->opts;
- ufs->rx_sel_idx = PA_MAXDATALANES;
- if (ufs->opts & EXYNOS_UFS_OPT_BROKEN_RX_SEL_IDX)
- ufs->rx_sel_idx = 0;
- hba->priv = (void *)ufs;
- hba->quirks = ufs->drv_data->quirks;
+ exynos_ufs_priv_init(hba, ufs);
+
if (ufs->drv_data->drv_init) {
ret = ufs->drv_data->drv_init(dev, ufs);
if (ret) {
switch (status) {
case PRE_CHANGE:
+ if (ufs->drv_data->pre_hce_enable) {
+ ret = ufs->drv_data->pre_hce_enable(ufs);
+ if (ret)
+ return ret;
+ }
+
ret = exynos_ufs_host_reset(hba);
if (ret)
return ret;
exynos_ufs_calc_pwm_clk_div(ufs);
if (!(ufs->opts & EXYNOS_UFS_OPT_BROKEN_AUTO_CLK_CTRL))
exynos_ufs_enable_auto_ctrl_hcc(ufs);
+
+ if (ufs->drv_data->post_hce_enable)
+ ret = ufs->drv_data->post_hce_enable(ufs);
+
break;
}
}
}
-static int exynos_ufs_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
+static int exynos_ufs_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
+ enum ufs_notify_change_status status)
{
struct exynos_ufs *ufs = ufshcd_get_variant(hba);
+ if (status == PRE_CHANGE)
+ return 0;
+
if (!ufshcd_is_link_active(hba))
phy_power_off(ufs->phy);
return 0;
}
+static int exynosauto_ufs_vh_link_startup_notify(struct ufs_hba *hba,
+ enum ufs_notify_change_status status)
+{
+ if (status == POST_CHANGE) {
+ ufshcd_set_link_active(hba);
+ ufshcd_set_ufs_dev_active(hba);
+ }
+
+ return 0;
+}
+
+static int exynosauto_ufs_vh_wait_ph_ready(struct ufs_hba *hba)
+{
+ u32 mbox;
+ ktime_t start, stop;
+
+ start = ktime_get();
+ stop = ktime_add(start, ms_to_ktime(PH_READY_TIMEOUT_MS));
+
+ do {
+ mbox = ufshcd_readl(hba, PH2VH_MBOX);
+ /* TODO: Mailbox message protocols between the PH and VHs are
+ * not implemented yet. This will be supported later
+ */
+ if ((mbox & MH_MSG_MASK) == MH_MSG_PH_READY)
+ return 0;
+
+ usleep_range(40, 50);
+ } while (ktime_before(ktime_get(), stop));
+
+ return -ETIME;
+}
+
+static int exynosauto_ufs_vh_init(struct ufs_hba *hba)
+{
+ struct device *dev = hba->dev;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct exynos_ufs *ufs;
+ int ret;
+
+ ufs = devm_kzalloc(dev, sizeof(*ufs), GFP_KERNEL);
+ if (!ufs)
+ return -ENOMEM;
+
+ /* exynos-specific hci */
+ ufs->reg_hci = devm_platform_ioremap_resource_byname(pdev, "vs_hci");
+ if (IS_ERR(ufs->reg_hci)) {
+ dev_err(dev, "cannot ioremap for hci vendor register\n");
+ return PTR_ERR(ufs->reg_hci);
+ }
+
+ ret = exynosauto_ufs_vh_wait_ph_ready(hba);
+ if (ret)
+ return ret;
+
+ ufs->drv_data = device_get_match_data(dev);
+ if (!ufs->drv_data)
+ return -ENODEV;
+
+ exynos_ufs_priv_init(hba, ufs);
+
+ return 0;
+}
+
static struct ufs_hba_variant_ops ufs_hba_exynos_ops = {
.name = "exynos_ufs",
.init = exynos_ufs_init,
.hce_enable_notify = exynos_ufs_hce_enable_notify,
.link_startup_notify = exynos_ufs_link_startup_notify,
.pwr_change_notify = exynos_ufs_pwr_change_notify,
+ .setup_clocks = exynos_ufs_setup_clocks,
.setup_xfer_req = exynos_ufs_specify_nexus_t_xfer_req,
.setup_task_mgmt = exynos_ufs_specify_nexus_t_tm_req,
.hibern8_notify = exynos_ufs_hibern8_notify,
.resume = exynos_ufs_resume,
};
+static struct ufs_hba_variant_ops ufs_hba_exynosauto_vh_ops = {
+ .name = "exynosauto_ufs_vh",
+ .init = exynosauto_ufs_vh_init,
+ .link_startup_notify = exynosauto_ufs_vh_link_startup_notify,
+};
+
static int exynos_ufs_probe(struct platform_device *pdev)
{
int err;
struct device *dev = &pdev->dev;
+ const struct ufs_hba_variant_ops *vops = &ufs_hba_exynos_ops;
+ const struct exynos_ufs_drv_data *drv_data =
+ device_get_match_data(dev);
- err = ufshcd_pltfrm_init(pdev, &ufs_hba_exynos_ops);
+ if (drv_data && drv_data->vops)
+ vops = drv_data->vops;
+
+ err = ufshcd_pltfrm_init(pdev, vops);
if (err)
dev_err(dev, "ufshcd_pltfrm_init() failed %d\n", err);
.pa_dbg_option_suite = 0x30103,
};
+static struct exynos_ufs_drv_data exynosauto_ufs_drvs = {
+ .uic_attr = &exynos7_uic_attr,
+ .quirks = UFSHCD_QUIRK_PRDT_BYTE_GRAN |
+ UFSHCI_QUIRK_SKIP_RESET_INTR_AGGR |
+ UFSHCD_QUIRK_BROKEN_OCS_FATAL_ERROR |
+ UFSHCD_QUIRK_SKIP_DEF_UNIPRO_TIMEOUT_SETTING,
+ .opts = EXYNOS_UFS_OPT_BROKEN_AUTO_CLK_CTRL |
+ EXYNOS_UFS_OPT_SKIP_CONFIG_PHY_ATTR |
+ EXYNOS_UFS_OPT_BROKEN_RX_SEL_IDX,
+ .drv_init = exynosauto_ufs_drv_init,
+ .post_hce_enable = exynosauto_ufs_post_hce_enable,
+ .pre_link = exynosauto_ufs_pre_link,
+ .pre_pwr_change = exynosauto_ufs_pre_pwr_change,
+ .post_pwr_change = exynosauto_ufs_post_pwr_change,
+};
+
+static struct exynos_ufs_drv_data exynosauto_ufs_vh_drvs = {
+ .vops = &ufs_hba_exynosauto_vh_ops,
+ .quirks = UFSHCD_QUIRK_PRDT_BYTE_GRAN |
+ UFSHCI_QUIRK_SKIP_RESET_INTR_AGGR |
+ UFSHCD_QUIRK_BROKEN_OCS_FATAL_ERROR |
+ UFSHCI_QUIRK_BROKEN_HCE |
+ UFSHCD_QUIRK_BROKEN_UIC_CMD |
+ UFSHCD_QUIRK_SKIP_PH_CONFIGURATION |
+ UFSHCD_QUIRK_SKIP_DEF_UNIPRO_TIMEOUT_SETTING,
+ .opts = EXYNOS_UFS_OPT_BROKEN_RX_SEL_IDX,
+};
+
static struct exynos_ufs_drv_data exynos_ufs_drvs = {
- .compatible = "samsung,exynos7-ufs",
.uic_attr = &exynos7_uic_attr,
.quirks = UFSHCD_QUIRK_PRDT_BYTE_GRAN |
UFSHCI_QUIRK_BROKEN_REQ_LIST_CLR |
static const struct of_device_id exynos_ufs_of_match[] = {
{ .compatible = "samsung,exynos7-ufs",
.data = &exynos_ufs_drvs },
+ { .compatible = "samsung,exynosautov9-ufs",
+ .data = &exynosauto_ufs_drvs },
+ { .compatible = "samsung,exynosautov9-ufs-vh",
+ .data = &exynosauto_ufs_vh_drvs },
{},
};
#define TX_GRAN_NVAL_10_08 0x0296
#define TX_GRAN_NVAL_H(v) (((v) >> 8) & 0x3)
+#define VND_TX_CLK_PRD 0xAA
+#define VND_TX_CLK_PRD_EN 0xA9
+#define VND_TX_LINERESET_PVALUE0 0xAD
+#define VND_TX_LINERESET_PVALUE1 0xAC
+#define VND_TX_LINERESET_PVALUE2 0xAB
+
+#define TX_LINE_RESET_TIME 3200
+
+#define VND_RX_CLK_PRD 0x12
+#define VND_RX_CLK_PRD_EN 0x11
+#define VND_RX_LINERESET_VALUE0 0x1D
+#define VND_RX_LINERESET_VALUE1 0x1C
+#define VND_RX_LINERESET_VALUE2 0x1B
+
+#define RX_LINE_RESET_TIME 1000
+
#define RX_FILLER_ENABLE 0x0316
#define RX_FILLER_EN (1 << 1)
#define RX_LINERESET_VAL 0x0317
#define PA_HIBERN8TIME_VAL 0x20
#define PCLK_AVAIL_MIN 70000000
-#define PCLK_AVAIL_MAX 133000000
+#define PCLK_AVAIL_MAX 167000000
struct exynos_ufs_uic_attr {
/* TX Attributes */
};
struct exynos_ufs_drv_data {
- char *compatible;
+ const struct ufs_hba_variant_ops *vops;
struct exynos_ufs_uic_attr *uic_attr;
unsigned int quirks;
unsigned int opts;
struct ufs_pa_layer_attr *pwr);
int (*post_pwr_change)(struct exynos_ufs *ufs,
struct ufs_pa_layer_attr *pwr);
+ int (*pre_hce_enable)(struct exynos_ufs *ufs);
+ int (*post_hce_enable)(struct exynos_ufs *ufs);
};
struct ufs_phy_time_cfg {
struct ufs_pa_layer_attr dev_req_params;
struct ufs_phy_time_cfg t_cfg;
ktime_t entry_hibern8_t;
- struct exynos_ufs_drv_data *drv_data;
+ const struct exynos_ufs_drv_data *drv_data;
+ struct regmap *sysreg;
+ u32 shareability_reg_offset;
u32 opts;
#define EXYNOS_UFS_OPT_HAS_APB_CLK_CTRL BIT(0)
#define EXYNOS_UFS_OPT_BROKEN_AUTO_CLK_CTRL BIT(2)
#define EXYNOS_UFS_OPT_BROKEN_RX_SEL_IDX BIT(3)
#define EXYNOS_UFS_OPT_USE_SW_HIBERN8_TIMER BIT(4)
+#define EXYNOS_UFS_OPT_SKIP_CONFIG_PHY_ATTR BIT(5)
};
#define for_each_ufs_rx_lane(ufs, i) \
return ret;
}
-static int ufs_hisi_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
+static int ufs_hisi_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
+ enum ufs_notify_change_status status)
{
struct ufs_hisi_host *host = ufshcd_get_variant(hba);
+ if (status == PRE_CHANGE)
+ return 0;
+
if (pm_op == UFS_RUNTIME_PM)
return 0;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * UFS hardware monitoring support
+ * Copyright (c) 2021, Western Digital Corporation
+ */
+
+#include <linux/hwmon.h>
+#include <linux/units.h>
+
+#include "ufshcd.h"
+
+struct ufs_hwmon_data {
+ struct ufs_hba *hba;
+ u8 mask;
+};
+
+static int ufs_read_temp_enable(struct ufs_hba *hba, u8 mask, long *val)
+{
+ u32 ee_mask;
+ int err;
+
+ err = ufshcd_query_attr(hba, UPIU_QUERY_OPCODE_READ_ATTR, QUERY_ATTR_IDN_EE_CONTROL, 0, 0,
+ &ee_mask);
+ if (err)
+ return err;
+
+ *val = (mask & ee_mask & MASK_EE_TOO_HIGH_TEMP) || (mask & ee_mask & MASK_EE_TOO_LOW_TEMP);
+
+ return 0;
+}
+
+static int ufs_get_temp(struct ufs_hba *hba, enum attr_idn idn, long *val)
+{
+ u32 value;
+ int err;
+
+ err = ufshcd_query_attr(hba, UPIU_QUERY_OPCODE_READ_ATTR, idn, 0, 0, &value);
+ if (err)
+ return err;
+
+ if (value == 0)
+ return -ENODATA;
+
+ *val = ((long)value - 80) * MILLIDEGREE_PER_DEGREE;
+
+ return 0;
+}
+
+static int ufs_hwmon_read(struct device *dev, enum hwmon_sensor_types type, u32 attr, int channel,
+ long *val)
+{
+ struct ufs_hwmon_data *data = dev_get_drvdata(dev);
+ struct ufs_hba *hba = data->hba;
+ int err;
+
+ down(&hba->host_sem);
+
+ if (!ufshcd_is_user_access_allowed(hba)) {
+ up(&hba->host_sem);
+ return -EBUSY;
+ }
+
+ ufshcd_rpm_get_sync(hba);
+
+ switch (attr) {
+ case hwmon_temp_enable:
+ err = ufs_read_temp_enable(hba, data->mask, val);
+
+ break;
+ case hwmon_temp_crit:
+ err = ufs_get_temp(hba, QUERY_ATTR_IDN_HIGH_TEMP_BOUND, val);
+
+ break;
+ case hwmon_temp_lcrit:
+ err = ufs_get_temp(hba, QUERY_ATTR_IDN_LOW_TEMP_BOUND, val);
+
+ break;
+ case hwmon_temp_input:
+ err = ufs_get_temp(hba, QUERY_ATTR_IDN_CASE_ROUGH_TEMP, val);
+
+ break;
+ default:
+ err = -EOPNOTSUPP;
+
+ break;
+ }
+
+ ufshcd_rpm_put_sync(hba);
+
+ up(&hba->host_sem);
+
+ return err;
+}
+
+static int ufs_hwmon_write(struct device *dev, enum hwmon_sensor_types type, u32 attr, int channel,
+ long val)
+{
+ struct ufs_hwmon_data *data = dev_get_drvdata(dev);
+ struct ufs_hba *hba = data->hba;
+ int err;
+
+ if (attr != hwmon_temp_enable)
+ return -EINVAL;
+
+ if (val != 0 && val != 1)
+ return -EINVAL;
+
+ down(&hba->host_sem);
+
+ if (!ufshcd_is_user_access_allowed(hba)) {
+ up(&hba->host_sem);
+ return -EBUSY;
+ }
+
+ ufshcd_rpm_get_sync(hba);
+
+ if (val == 1)
+ err = ufshcd_update_ee_usr_mask(hba, MASK_EE_URGENT_TEMP, 0);
+ else
+ err = ufshcd_update_ee_usr_mask(hba, 0, MASK_EE_URGENT_TEMP);
+
+ ufshcd_rpm_put_sync(hba);
+
+ up(&hba->host_sem);
+
+ return err;
+}
+
+static umode_t ufs_hwmon_is_visible(const void *_data, enum hwmon_sensor_types type, u32 attr,
+ int channel)
+{
+ if (type != hwmon_temp)
+ return 0;
+
+ switch (attr) {
+ case hwmon_temp_enable:
+ return 0644;
+ case hwmon_temp_crit:
+ case hwmon_temp_lcrit:
+ case hwmon_temp_input:
+ return 0444;
+ default:
+ break;
+ }
+ return 0;
+}
+
+static const struct hwmon_channel_info *ufs_hwmon_info[] = {
+ HWMON_CHANNEL_INFO(temp, HWMON_T_ENABLE | HWMON_T_INPUT | HWMON_T_CRIT | HWMON_T_LCRIT),
+ NULL
+};
+
+static const struct hwmon_ops ufs_hwmon_ops = {
+ .is_visible = ufs_hwmon_is_visible,
+ .read = ufs_hwmon_read,
+ .write = ufs_hwmon_write,
+};
+
+static const struct hwmon_chip_info ufs_hwmon_hba_info = {
+ .ops = &ufs_hwmon_ops,
+ .info = ufs_hwmon_info,
+};
+
+void ufs_hwmon_probe(struct ufs_hba *hba, u8 mask)
+{
+ struct device *dev = hba->dev;
+ struct ufs_hwmon_data *data;
+ struct device *hwmon;
+
+ data = kzalloc(sizeof(*data), GFP_KERNEL);
+ if (!data)
+ return;
+
+ data->hba = hba;
+ data->mask = mask;
+
+ hwmon = hwmon_device_register_with_info(dev, "ufs", data, &ufs_hwmon_hba_info, NULL);
+ if (IS_ERR(hwmon)) {
+ dev_warn(dev, "Failed to instantiate hwmon device\n");
+ kfree(data);
+ return;
+ }
+
+ hba->hwmon_device = hwmon;
+}
+
+void ufs_hwmon_remove(struct ufs_hba *hba)
+{
+ struct ufs_hwmon_data *data;
+
+ if (!hba->hwmon_device)
+ return;
+
+ data = dev_get_drvdata(hba->hwmon_device);
+ hwmon_device_unregister(hba->hwmon_device);
+ hba->hwmon_device = NULL;
+ kfree(data);
+}
+
+void ufs_hwmon_notify_event(struct ufs_hba *hba, u8 ee_mask)
+{
+ if (!hba->hwmon_device)
+ return;
+
+ if (ee_mask & MASK_EE_TOO_HIGH_TEMP)
+ hwmon_notify_event(hba->hwmon_device, hwmon_temp, hwmon_temp_max_alarm, 0);
+
+ if (ee_mask & MASK_EE_TOO_LOW_TEMP)
+ hwmon_notify_event(hba->hwmon_device, hwmon_temp, hwmon_temp_min_alarm, 0);
+}
#include <linux/platform_device.h>
#include <linux/regulator/consumer.h>
#include <linux/reset.h>
+#include <linux/sched/clock.h>
#include <linux/soc/mediatek/mtk_sip_svc.h>
#include "ufshcd.h"
if (on) {
ufs_mtk_ref_clk_notify(on, res);
- ufshcd_delay_us(host->ref_clk_ungating_wait_us, 10);
ufshcd_writel(hba, REFCLK_REQUEST, REG_UFS_REFCLK_CTRL);
} else {
+ ufshcd_delay_us(host->ref_clk_gating_wait_us, 10);
ufshcd_writel(hba, REFCLK_RELEASE, REG_UFS_REFCLK_CTRL);
}
out:
host->ref_clk_enabled = on;
- if (!on) {
- ufshcd_delay_us(host->ref_clk_gating_wait_us, 10);
+ if (on)
+ ufshcd_delay_us(host->ref_clk_ungating_wait_us, 10);
+ else
ufs_mtk_ref_clk_notify(on, res);
- }
return 0;
}
static void ufs_mtk_setup_ref_clk_wait_us(struct ufs_hba *hba,
- u16 gating_us, u16 ungating_us)
+ u16 gating_us)
{
struct ufs_mtk_host *host = ufshcd_get_variant(hba);
host->ref_clk_gating_wait_us = gating_us;
}
- host->ref_clk_ungating_wait_us = ungating_us;
+ host->ref_clk_ungating_wait_us = REFCLK_DEFAULT_WAIT_US;
+}
+
+static void ufs_mtk_dbg_sel(struct ufs_hba *hba)
+{
+ struct ufs_mtk_host *host = ufshcd_get_variant(hba);
+
+ if (((host->ip_ver >> 16) & 0xFF) >= 0x36) {
+ ufshcd_writel(hba, 0x820820, REG_UFS_DEBUG_SEL);
+ ufshcd_writel(hba, 0x0, REG_UFS_DEBUG_SEL_B0);
+ ufshcd_writel(hba, 0x55555555, REG_UFS_DEBUG_SEL_B1);
+ ufshcd_writel(hba, 0xaaaaaaaa, REG_UFS_DEBUG_SEL_B2);
+ ufshcd_writel(hba, 0xffffffff, REG_UFS_DEBUG_SEL_B3);
+ } else {
+ ufshcd_writel(hba, 0x20, REG_UFS_DEBUG_SEL);
+ }
+}
+
+static void ufs_mtk_wait_idle_state(struct ufs_hba *hba,
+ unsigned long retry_ms)
+{
+ u64 timeout, time_checked;
+ u32 val, sm;
+ bool wait_idle;
+
+ /* cannot use plain ktime_get() in suspend */
+ timeout = ktime_get_mono_fast_ns() + retry_ms * 1000000UL;
+
+ /* wait a specific time after check base */
+ udelay(10);
+ wait_idle = false;
+
+ do {
+ time_checked = ktime_get_mono_fast_ns();
+ ufs_mtk_dbg_sel(hba);
+ val = ufshcd_readl(hba, REG_UFS_PROBE);
+
+ sm = val & 0x1f;
+
+ /*
+ * if state is in H8 enter and H8 enter confirm
+ * wait until return to idle state.
+ */
+ if ((sm >= VS_HIB_ENTER) && (sm <= VS_HIB_EXIT)) {
+ wait_idle = true;
+ udelay(50);
+ continue;
+ } else if (!wait_idle)
+ break;
+
+ if (wait_idle && (sm == VS_HCE_BASE))
+ break;
+ } while (time_checked < timeout);
+
+ if (wait_idle && sm != VS_HCE_BASE)
+ dev_info(hba->dev, "wait idle tmo: 0x%x\n", val);
}
static int ufs_mtk_wait_link_state(struct ufs_hba *hba, u32 state,
timeout = ktime_add_ms(ktime_get(), max_wait_ms);
do {
time_checked = ktime_get();
- ufshcd_writel(hba, 0x20, REG_UFS_DEBUG_SEL);
+ ufs_mtk_dbg_sel(hba);
val = ufshcd_readl(hba, REG_UFS_PROBE);
val = val >> 28;
ufs_mtk_mphy_power_on(hba, true);
ufs_mtk_setup_clocks(hba, true, POST_CHANGE);
+ host->ip_ver = ufshcd_readl(hba, REG_UFS_MTK_IP_VER);
+
goto out;
out_variant_clear:
REGULATOR_MODE_NORMAL);
}
-static int ufs_mtk_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
+static void ufs_mtk_auto_hibern8_disable(struct ufs_hba *hba)
+{
+ unsigned long flags;
+ int ret;
+
+ /* disable auto-hibern8 */
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ ufshcd_writel(hba, 0, REG_AUTO_HIBERNATE_IDLE_TIMER);
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ /* wait host return to idle state when auto-hibern8 off */
+ ufs_mtk_wait_idle_state(hba, 5);
+
+ ret = ufs_mtk_wait_link_state(hba, VS_LINK_UP, 100);
+ if (ret)
+ dev_warn(hba->dev, "exit h8 state fail, ret=%d\n", ret);
+}
+
+static int ufs_mtk_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
+ enum ufs_notify_change_status status)
{
int err;
struct arm_smccc_res res;
+ if (status == PRE_CHANGE) {
+ if (!ufshcd_is_auto_hibern8_supported(hba))
+ return 0;
+ ufs_mtk_auto_hibern8_disable(hba);
+ return 0;
+ }
+
if (ufshcd_is_link_hibern8(hba)) {
err = ufs_mtk_link_set_lpm(hba);
if (err)
"MPHY Ctrl ");
/* Direct debugging information to REG_MTK_PROBE */
- ufshcd_writel(hba, 0x20, REG_UFS_DEBUG_SEL);
+ ufs_mtk_dbg_sel(hba);
ufshcd_dump_regs(hba, REG_UFS_PROBE, 0x4, "Debug Probe ");
}
* requirements.
*/
if (mid == UFS_VENDOR_SAMSUNG)
- ufs_mtk_setup_ref_clk_wait_us(hba, 1, 1);
+ ufs_mtk_setup_ref_clk_wait_us(hba, 1);
else if (mid == UFS_VENDOR_SKHYNIX)
- ufs_mtk_setup_ref_clk_wait_us(hba, 30, 30);
+ ufs_mtk_setup_ref_clk_wait_us(hba, 30);
else if (mid == UFS_VENDOR_TOSHIBA)
- ufs_mtk_setup_ref_clk_wait_us(hba, 100, 32);
+ ufs_mtk_setup_ref_clk_wait_us(hba, 100);
+ else
+ ufs_mtk_setup_ref_clk_wait_us(hba,
+ REFCLK_DEFAULT_WAIT_US);
return 0;
}
#define REG_UFS_REFCLK_CTRL 0x144
#define REG_UFS_EXTREG 0x2100
#define REG_UFS_MPHYCTRL 0x2200
+#define REG_UFS_MTK_IP_VER 0x2240
#define REG_UFS_REJECT_MON 0x22AC
#define REG_UFS_DEBUG_SEL 0x22C0
#define REG_UFS_PROBE 0x22C8
+#define REG_UFS_DEBUG_SEL_B0 0x22D0
+#define REG_UFS_DEBUG_SEL_B1 0x22D4
+#define REG_UFS_DEBUG_SEL_B2 0x22D8
+#define REG_UFS_DEBUG_SEL_B3 0x22DC
/*
* Ref-clk control
#define REFCLK_ACK BIT(1)
#define REFCLK_REQ_TIMEOUT_US 3000
+#define REFCLK_DEFAULT_WAIT_US 32
/*
* Other attributes
};
/*
+ * Vendor specific host controller state
+ */
+enum {
+ VS_HCE_RESET = 0,
+ VS_HCE_BASE = 1,
+ VS_HCE_OOCPR_WAIT = 2,
+ VS_HCE_DME_RESET = 3,
+ VS_HCE_MIDDLE = 4,
+ VS_HCE_DME_ENABLE = 5,
+ VS_HCE_DEFAULTS = 6,
+ VS_HIB_IDLEEN = 7,
+ VS_HIB_ENTER = 8,
+ VS_HIB_ENTER_CONF = 9,
+ VS_HIB_MIDDLE = 10,
+ VS_HIB_WAITTIMER = 11,
+ VS_HIB_EXIT_CONF = 12,
+ VS_HIB_EXIT = 13,
+};
+
+/*
* SiP commands
*/
#define MTK_SIP_UFS_CONTROL MTK_SIP_SMC_CMD(0x276)
bool ref_clk_enabled;
u16 ref_clk_ungating_wait_us;
u16 ref_clk_gating_wait_us;
+ u32 ip_ver;
};
#endif /* !_UFS_MEDIATEK_H */
gpiod_set_value_cansleep(host->device_reset, asserted);
}
-static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
+static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
+ enum ufs_notify_change_status status)
{
struct ufs_qcom_host *host = ufshcd_get_variant(hba);
struct phy *phy = host->generic_phy;
+ if (status == PRE_CHANGE)
+ return 0;
+
if (ufs_qcom_is_link_off(hba)) {
/*
* Disable the tx/rx lane symbol clocks before PHY is
enum ufs_notify_change_status status)
{
struct ufs_qcom_host *host = ufshcd_get_variant(hba);
- int err = 0;
/*
* In case ufs_qcom_init() is not yet done, simply ignore.
break;
}
- return err;
+ return 0;
}
static int
int err = 0;
if (status == PRE_CHANGE) {
+ err = ufshcd_uic_hibern8_enter(hba);
+ if (err)
+ return err;
if (scale_up)
err = ufs_qcom_clk_scale_up_pre_change(hba);
else
err = ufs_qcom_clk_scale_down_pre_change(hba);
+ if (err)
+ ufshcd_uic_hibern8_exit(hba);
+
} else {
if (scale_up)
err = ufs_qcom_clk_scale_up_post_change(hba);
else
err = ufs_qcom_clk_scale_down_post_change(hba);
- if (err || !dev_req_params)
+
+ if (err || !dev_req_params) {
+ ufshcd_uic_hibern8_exit(hba);
goto out;
+ }
ufs_qcom_cfg_timers(hba,
dev_req_params->gear_rx,
dev_req_params->pwr_rx,
dev_req_params->hs_rate,
false);
+ ufshcd_uic_hibern8_exit(hba);
}
out:
QUERY_ATTR_IDN_PSA_STATE = 0x15,
QUERY_ATTR_IDN_PSA_DATA_SIZE = 0x16,
QUERY_ATTR_IDN_REF_CLK_GATING_WAIT_TIME = 0x17,
+ QUERY_ATTR_IDN_CASE_ROUGH_TEMP = 0x18,
+ QUERY_ATTR_IDN_HIGH_TEMP_BOUND = 0x19,
+ QUERY_ATTR_IDN_LOW_TEMP_BOUND = 0x1A,
QUERY_ATTR_IDN_WB_FLUSH_STATUS = 0x1C,
QUERY_ATTR_IDN_AVAIL_WB_BUFF_SIZE = 0x1D,
QUERY_ATTR_IDN_WB_BUFF_LIFE_TIME_EST = 0x1E,
/* Possible values for dExtendedUFSFeaturesSupport */
enum {
+ UFS_DEV_LOW_TEMP_NOTIF = BIT(4),
+ UFS_DEV_HIGH_TEMP_NOTIF = BIT(5),
+ UFS_DEV_EXT_TEMP_NOTIF = BIT(6),
UFS_DEV_HPB_SUPPORT = BIT(7),
UFS_DEV_WRITE_BOOSTER_SUP = BIT(8),
};
MASK_EE_WRITEBOOSTER_EVENT = BIT(5),
MASK_EE_PERFORMANCE_THROTTLING = BIT(6),
};
+#define MASK_EE_URGENT_TEMP (MASK_EE_TOO_HIGH_TEMP | MASK_EE_TOO_LOW_TEMP)
/* Background operation status */
enum bkops_status {
clki->min_freq = clkfreq[i];
clki->max_freq = clkfreq[i+1];
- clki->name = kstrdup(name, GFP_KERNEL);
+ clki->name = devm_kstrdup(dev, name, GFP_KERNEL);
if (!strcmp(name, "ref_clk"))
clki->keep_link_active = true;
dev_dbg(dev, "%s: min %u max %u name %s\n", "freq-table-hz",
if (!vreg)
return -ENOMEM;
- vreg->name = kstrdup(name, GFP_KERNEL);
+ vreg->name = devm_kstrdup(dev, name, GFP_KERNEL);
snprintf(prop_name, MAX_PROP_SIZE, "%s-max-microamp", name);
if (of_property_read_u32(np, prop_name, &vreg->max_uA)) {
/* maximum number of reset retries before giving up */
#define MAX_HOST_RESET_RETRIES 5
+/* Maximum number of error handler retries before giving up */
+#define MAX_ERR_HANDLER_RETRIES 5
+
/* Expose the flag value from utp_upiu_query.value */
#define MASK_QUERY_UPIU_FLAG_LOC 0xFF
UFSHCD_CAN_QUEUE = 32,
};
+static const char *const ufshcd_state_name[] = {
+ [UFSHCD_STATE_RESET] = "reset",
+ [UFSHCD_STATE_OPERATIONAL] = "operational",
+ [UFSHCD_STATE_ERROR] = "error",
+ [UFSHCD_STATE_EH_SCHEDULED_FATAL] = "eh_fatal",
+ [UFSHCD_STATE_EH_SCHEDULED_NON_FATAL] = "eh_non_fatal",
+};
+
/* UFSHCD error handling flags */
enum {
UFSHCD_EH_IN_PROGRESS = (1 << 0),
static int ufshcd_eh_host_reset_handler(struct scsi_cmnd *cmd);
static int ufshcd_clear_tm_cmd(struct ufs_hba *hba, int tag);
static void ufshcd_hba_exit(struct ufs_hba *hba);
-static int ufshcd_clear_ua_wluns(struct ufs_hba *hba);
-static int ufshcd_probe_hba(struct ufs_hba *hba, bool async);
+static int ufshcd_probe_hba(struct ufs_hba *hba, bool init_dev_params);
static int ufshcd_setup_clocks(struct ufs_hba *hba, bool on);
-static int ufshcd_uic_hibern8_enter(struct ufs_hba *hba);
static inline void ufshcd_add_delay_before_dme_cmd(struct ufs_hba *hba);
static int ufshcd_host_reset_and_restore(struct ufs_hba *hba);
static void ufshcd_resume_clkscaling(struct ufs_hba *hba);
static irqreturn_t ufshcd_intr(int irq, void *__hba);
static int ufshcd_change_power_mode(struct ufs_hba *hba,
struct ufs_pa_layer_attr *pwr_mode);
-static void ufshcd_schedule_eh_work(struct ufs_hba *hba);
static int ufshcd_setup_hba_vreg(struct ufs_hba *hba, bool on);
static int ufshcd_setup_vreg(struct ufs_hba *hba, bool on);
static inline int ufshcd_config_vreg_hpm(struct ufs_hba *hba,
* This function is used to get the OCS field from UTRD
* Returns the OCS field in the UTRD
*/
-static inline int ufshcd_get_tr_ocs(struct ufshcd_lrb *lrbp)
+static enum utp_ocs ufshcd_get_tr_ocs(struct ufshcd_lrb *lrbp)
{
return le32_to_cpu(lrbp->utr_descriptor_ptr->header.dword_2) & MASK_OCS;
}
int ret;
unsigned long flags;
+ if (hba->quirks & UFSHCD_QUIRK_BROKEN_UIC_CMD)
+ return 0;
+
ufshcd_hold(hba, false);
mutex_lock(&hba->uic_cmd_mutex);
ufshcd_add_delay_before_dme_cmd(hba);
sizeof(struct ufshcd_sg_entry)));
else
lrbp->utr_descriptor_ptr->prd_table_length =
- cpu_to_le16((u16) (sg_segments));
+ cpu_to_le16(sg_segments);
- prd_table = (struct ufshcd_sg_entry *)lrbp->ucd_prdt_ptr;
+ prd_table = lrbp->ucd_prdt_ptr;
scsi_for_each_sg(cmd, sg, sg_segments, i) {
- prd_table[i].size =
- cpu_to_le32(((u32) sg_dma_len(sg))-1);
- prd_table[i].base_addr =
- cpu_to_le32(lower_32_bits(sg->dma_address));
- prd_table[i].upper_addr =
- cpu_to_le32(upper_32_bits(sg->dma_address));
+ const unsigned int len = sg_dma_len(sg);
+
+ /*
+ * From the UFSHCI spec: "Data Byte Count (DBC): A '0'
+ * based value that indicates the length, in bytes, of
+ * the data block. A maximum of length of 256KB may
+ * exist for any entry. Bits 1:0 of this field shall be
+ * 11b to indicate Dword granularity. A value of '3'
+ * indicates 4 bytes, '7' indicates 8 bytes, etc."
+ */
+ WARN_ONCE(len > 256 * 1024, "len = %#x\n", len);
+ prd_table[i].size = cpu_to_le32(len - 1);
+ prd_table[i].addr = cpu_to_le64(sg->dma_address);
prd_table[i].reserved = 0;
}
} else {
lrb->ucd_req_dma_addr = cmd_desc_element_addr;
lrb->ucd_rsp_ptr = (struct utp_upiu_rsp *)cmd_descp[i].response_upiu;
lrb->ucd_rsp_dma_addr = cmd_desc_element_addr + response_offset;
- lrb->ucd_prdt_ptr = (struct ufshcd_sg_entry *)cmd_descp[i].prd_table;
+ lrb->ucd_prdt_ptr = cmd_descp[i].prd_table;
lrb->ucd_prdt_dma_addr = cmd_desc_element_addr + prdt_offset;
}
switch (hba->ufshcd_state) {
case UFSHCD_STATE_OPERATIONAL:
+ break;
case UFSHCD_STATE_EH_SCHEDULED_NON_FATAL:
+ /*
+ * SCSI error handler can call ->queuecommand() while UFS error
+ * handler is in progress. Error interrupts could change the
+ * state from UFSHCD_STATE_RESET to
+ * UFSHCD_STATE_EH_SCHEDULED_NON_FATAL. Prevent requests
+ * being issued in that case.
+ */
+ if (ufshcd_eh_in_progress(hba)) {
+ err = SCSI_MLQUEUE_HOST_BUSY;
+ goto out;
+ }
break;
case UFSHCD_STATE_EH_SCHEDULED_FATAL:
/*
if (hba->pm_op_in_progress) {
hba->force_reset = true;
set_host_byte(cmd, DID_BAD_TARGET);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
goto out;
}
fallthrough;
goto out;
case UFSHCD_STATE_ERROR:
set_host_byte(cmd, DID_ERROR);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
goto out;
}
if (ret)
dev_err(hba->dev, "%s: link recovery failed, err %d",
__func__, ret);
- else
- ufshcd_clear_ua_wluns(hba);
return ret;
}
EXPORT_SYMBOL_GPL(ufshcd_link_recovery);
-static int ufshcd_uic_hibern8_enter(struct ufs_hba *hba)
+int ufshcd_uic_hibern8_enter(struct ufs_hba *hba)
{
int ret;
struct uic_command uic_cmd = {0};
return ret;
}
+EXPORT_SYMBOL_GPL(ufshcd_uic_hibern8_enter);
int ufshcd_uic_hibern8_exit(struct ufs_hba *hba)
{
{
int result = 0;
int scsi_status;
- int ocs;
+ enum utp_ocs ocs;
/* overall command status of utrd */
ocs = ufshcd_get_tr_ocs(lrbp);
* __ufshcd_transfer_req_compl - handle SCSI and query command completion
* @hba: per adapter instance
* @completed_reqs: bitmask that indicates which requests to complete
- * @retry_requests: whether to ask the SCSI core to retry completed requests
*/
static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,
- unsigned long completed_reqs,
- bool retry_requests)
+ unsigned long completed_reqs)
{
struct ufshcd_lrb *lrbp;
struct scsi_cmnd *cmd;
if (unlikely(ufshcd_should_inform_monitor(hba, lrbp)))
ufshcd_update_monitor(hba, lrbp);
ufshcd_add_command_trace(hba, index, UFS_CMD_COMP);
- result = retry_requests ? DID_BUS_BUSY << 16 :
- ufshcd_transfer_rsp_status(hba, lrbp);
+ result = ufshcd_transfer_rsp_status(hba, lrbp);
scsi_dma_unmap(cmd);
cmd->result = result;
/* Mark completed command as NULL in LRB */
lrbp->cmd = NULL;
/* Do not touch lrbp after scsi done */
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
ufshcd_release(hba);
update_scaling = true;
} else if (lrbp->command_type == UTP_CMD_TYPE_DEV_MANAGE ||
/**
* ufshcd_transfer_req_compl - handle SCSI and query command completion
* @hba: per adapter instance
- * @retry_requests: whether or not to ask to retry requests
*
* Returns
* IRQ_HANDLED - If interrupt is valid
* IRQ_NONE - If invalid interrupt
*/
-static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba,
- bool retry_requests)
+static irqreturn_t ufshcd_transfer_req_compl(struct ufs_hba *hba)
{
unsigned long completed_reqs, flags;
u32 tr_doorbell;
spin_unlock_irqrestore(&hba->outstanding_lock, flags);
if (completed_reqs) {
- __ufshcd_transfer_req_compl(hba, completed_reqs,
- retry_requests);
+ __ufshcd_transfer_req_compl(hba, completed_reqs);
return IRQ_HANDLED;
} else {
return IRQ_NONE;
__func__, err);
}
+static void ufshcd_temp_exception_event_handler(struct ufs_hba *hba, u16 status)
+{
+ u32 value;
+
+ if (ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_READ_ATTR,
+ QUERY_ATTR_IDN_CASE_ROUGH_TEMP, 0, 0, &value))
+ return;
+
+ dev_info(hba->dev, "exception Tcase %d\n", value - 80);
+
+ ufs_hwmon_notify_event(hba, status & MASK_EE_URGENT_TEMP);
+
+ /*
+ * A placeholder for the platform vendors to add whatever additional
+ * steps required
+ */
+}
+
static int __ufshcd_wb_toggle(struct ufs_hba *hba, bool set, enum flag_idn idn)
{
u8 index;
if (status & hba->ee_drv_mask & MASK_EE_URGENT_BKOPS)
ufshcd_bkops_exception_event_handler(hba);
+ if (status & hba->ee_drv_mask & MASK_EE_URGENT_TEMP)
+ ufshcd_temp_exception_event_handler(hba, status);
+
ufs_debugfs_exception_event(hba, status);
out:
ufshcd_scsi_unblock_requests(hba);
- return;
}
/* Complete requests that have door-bell cleared */
static void ufshcd_complete_requests(struct ufs_hba *hba)
{
- ufshcd_transfer_req_compl(hba, /*retry_requests=*/false);
- ufshcd_tmc_handler(hba);
-}
-
-static void ufshcd_retry_aborted_requests(struct ufs_hba *hba)
-{
- ufshcd_transfer_req_compl(hba, /*retry_requests=*/true);
+ ufshcd_transfer_req_compl(hba);
ufshcd_tmc_handler(hba);
}
(hba->saved_err & (INT_FATAL_ERRORS | UFSHCD_UIC_HIBERN8_MASK));
}
-/* host lock must be held before calling this func */
-static inline void ufshcd_schedule_eh_work(struct ufs_hba *hba)
+void ufshcd_schedule_eh_work(struct ufs_hba *hba)
{
+ lockdep_assert_held(hba->host->host_lock);
+
/* handle fatal errors only when link is not in error state */
if (hba->ufshcd_state != UFSHCD_STATE_ERROR) {
if (hba->force_reset || ufshcd_is_link_broken(hba) ||
ufshcd_release(hba);
if (ufshcd_is_clkscaling_supported(hba))
ufshcd_clk_scaling_suspend(hba, false);
- ufshcd_clear_ua_wluns(hba);
ufshcd_rpm_put(hba);
}
*/
static void ufshcd_err_handler(struct work_struct *work)
{
+ int retries = MAX_ERR_HANDLER_RETRIES;
struct ufs_hba *hba;
unsigned long flags;
- bool err_xfer = false;
- bool err_tm = false;
- int err = 0, pmc_err;
+ bool needs_restore;
+ bool needs_reset;
+ bool err_xfer;
+ bool err_tm;
+ int pmc_err;
int tag;
- bool needs_reset = false, needs_restore = false;
hba = container_of(work, struct ufs_hba, eh_work);
+ dev_info(hba->dev,
+ "%s started; HBA state %s; powered %d; shutting down %d; saved_err = %d; saved_uic_err = %d; force_reset = %d%s\n",
+ __func__, ufshcd_state_name[hba->ufshcd_state],
+ hba->is_powered, hba->shutting_down, hba->saved_err,
+ hba->saved_uic_err, hba->force_reset,
+ ufshcd_is_link_broken(hba) ? "; link is broken" : "");
+
down(&hba->host_sem);
spin_lock_irqsave(hba->host->host_lock, flags);
if (ufshcd_err_handling_should_stop(hba)) {
/* Complete requests that have door-bell cleared by h/w */
ufshcd_complete_requests(hba);
spin_lock_irqsave(hba->host->host_lock, flags);
+again:
+ needs_restore = false;
+ needs_reset = false;
+ err_xfer = false;
+ err_tm = false;
+
if (hba->ufshcd_state != UFSHCD_STATE_ERROR)
hba->ufshcd_state = UFSHCD_STATE_RESET;
/*
err_xfer = true;
goto lock_skip_pending_xfer_clear;
}
+ dev_err(hba->dev, "Aborted tag %d / CDB %#02x\n", tag,
+ hba->lrb[tag].cmd ? hba->lrb[tag].cmd->cmnd[0] : -1);
}
/* Clear pending task management requests */
}
lock_skip_pending_xfer_clear:
- ufshcd_retry_aborted_requests(hba);
+ /* Complete the requests that are cleared by s/w */
+ ufshcd_complete_requests(hba);
spin_lock_irqsave(hba->host->host_lock, flags);
hba->silence_err_logs = false;
do_reset:
/* Fatal errors need reset */
if (needs_reset) {
+ int err;
+
hba->force_reset = false;
spin_unlock_irqrestore(hba->host->host_lock, flags);
err = ufshcd_reset_and_restore(hba);
dev_err_ratelimited(hba->dev, "%s: exit: saved_err 0x%x saved_uic_err 0x%x",
__func__, hba->saved_err, hba->saved_uic_err);
}
+ /* Exit in an operational state or dead */
+ if (hba->ufshcd_state != UFSHCD_STATE_OPERATIONAL &&
+ hba->ufshcd_state != UFSHCD_STATE_ERROR) {
+ if (--retries)
+ goto again;
+ hba->ufshcd_state = UFSHCD_STATE_ERROR;
+ }
ufshcd_clear_eh_in_progress(hba);
spin_unlock_irqrestore(hba->host->host_lock, flags);
ufshcd_err_handling_unprepare(hba);
up(&hba->host_sem);
+
+ dev_info(hba->dev, "%s finished; HBA state %s\n", __func__,
+ ufshcd_state_name[hba->ufshcd_state]);
}
/**
irqreturn_t ret = IRQ_NONE;
int tag;
- pending = ufshcd_readl(hba, REG_UTP_TASK_REQ_DOOR_BELL);
-
spin_lock_irqsave(hba->host->host_lock, flags);
+ pending = ufshcd_readl(hba, REG_UTP_TASK_REQ_DOOR_BELL);
issued = hba->outstanding_tasks & ~pending;
for_each_set_bit(tag, &issued, hba->nutmrs) {
struct request *req = hba->tmf_rqs[tag];
retval |= ufshcd_tmc_handler(hba);
if (intr_status & UTP_TRANSFER_REQ_COMPL)
- retval |= ufshcd_transfer_req_compl(hba, /*retry_requests=*/false);
+ retval |= ufshcd_transfer_req_compl(hba);
return retval;
}
err = ufshcd_wait_for_register(hba,
REG_UTP_TASK_REQ_DOOR_BELL,
mask, 0, 1000, 1000);
+
+ dev_err(hba->dev, "Clearing task management function with tag %d %s\n",
+ tag, err ? "succeeded" : "failed");
+
out:
return err;
}
err = wait_for_completion_io_timeout(&wait,
msecs_to_jiffies(TM_CMD_TIMEOUT));
if (!err) {
- /*
- * Make sure that ufshcd_compl_tm() does not trigger a
- * use-after-free.
- */
- req->end_io_data = NULL;
ufshcd_add_tm_upiu_trace(hba, task_tag, UFS_TM_ERR);
dev_err(hba->dev, "%s: task management cmd 0x%.2x timed-out\n",
__func__, tm_function);
u8 tm_function, u8 *tm_response)
{
struct utp_task_req_desc treq = { { 0 }, };
- int ocs_value, err;
+ enum utp_ocs ocs_value;
+ int err;
/* Configure task request descriptor */
treq.header.dword_0 = cpu_to_le32(UTP_REQ_DESC_INT_CMD);
int err;
enum dev_cmd_type cmd_type = DEV_CMD_TYPE_QUERY;
struct utp_task_req_desc treq = { { 0 }, };
- int ocs_value;
+ enum utp_ocs ocs_value;
u8 tm_f = be32_to_cpu(req_upiu->header.dword_1) >> 16 & MASK_TM_FUNC;
switch (msgcode) {
err = ufshcd_clear_cmd(hba, pos);
if (err)
break;
- __ufshcd_transfer_req_compl(hba, 1U << pos, false);
+ __ufshcd_transfer_req_compl(hba, 1U << pos);
}
}
dev_err(hba->dev,
"%s: cmd was completed, but without a notifying intr, tag = %d",
__func__, tag);
- __ufshcd_transfer_req_compl(hba, 1UL << tag, /*retry_requests=*/false);
+ __ufshcd_transfer_req_compl(hba, 1UL << tag);
goto release;
}
goto release;
}
+ lrbp->cmd = NULL;
err = SUCCESS;
release:
ufshpb_reset_host(hba);
ufshcd_hba_stop(hba);
hba->silence_err_logs = true;
- ufshcd_retry_aborted_requests(hba);
+ ufshcd_complete_requests(hba);
hba->silence_err_logs = false;
/* scale up clocks to max frequency before full reinitialization */
*/
static int ufshcd_reset_and_restore(struct ufs_hba *hba)
{
- u32 saved_err;
- u32 saved_uic_err;
+ u32 saved_err = 0;
+ u32 saved_uic_err = 0;
int err = 0;
unsigned long flags;
int retries = MAX_HOST_RESET_RETRIES;
- /*
- * This is a fresh start, cache and clear saved error first,
- * in case new error generated during reset and restore.
- */
spin_lock_irqsave(hba->host->host_lock, flags);
- saved_err = hba->saved_err;
- saved_uic_err = hba->saved_uic_err;
- hba->saved_err = 0;
- hba->saved_uic_err = 0;
- spin_unlock_irqrestore(hba->host->host_lock, flags);
-
do {
+ /*
+ * This is a fresh start, cache and clear saved error first,
+ * in case new error generated during reset and restore.
+ */
+ saved_err |= hba->saved_err;
+ saved_uic_err |= hba->saved_uic_err;
+ hba->saved_err = 0;
+ hba->saved_uic_err = 0;
+ hba->force_reset = false;
+ hba->ufshcd_state = UFSHCD_STATE_RESET;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
/* Reset the attached device */
ufshcd_device_reset(hba);
err = ufshcd_host_reset_and_restore(hba);
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ if (err)
+ continue;
+ /* Do not exit unless operational or dead */
+ if (hba->ufshcd_state != UFSHCD_STATE_OPERATIONAL &&
+ hba->ufshcd_state != UFSHCD_STATE_ERROR &&
+ hba->ufshcd_state != UFSHCD_STATE_EH_SCHEDULED_NON_FATAL)
+ err = -EAGAIN;
} while (err && --retries);
- spin_lock_irqsave(hba->host->host_lock, flags);
/*
* Inform scsi mid-layer that we did reset and allow to handle
* Unit Attention properly.
hba->caps &= ~UFSHCD_CAP_WB_EN;
}
+static void ufshcd_temp_notif_probe(struct ufs_hba *hba, u8 *desc_buf)
+{
+ struct ufs_dev_info *dev_info = &hba->dev_info;
+ u32 ext_ufs_feature;
+ u8 mask = 0;
+
+ if (!(hba->caps & UFSHCD_CAP_TEMP_NOTIF) || dev_info->wspecversion < 0x300)
+ return;
+
+ ext_ufs_feature = get_unaligned_be32(desc_buf + DEVICE_DESC_PARAM_EXT_UFS_FEATURE_SUP);
+
+ if (ext_ufs_feature & UFS_DEV_LOW_TEMP_NOTIF)
+ mask |= MASK_EE_TOO_LOW_TEMP;
+
+ if (ext_ufs_feature & UFS_DEV_HIGH_TEMP_NOTIF)
+ mask |= MASK_EE_TOO_HIGH_TEMP;
+
+ if (mask) {
+ ufshcd_enable_ee(hba, mask);
+ ufs_hwmon_probe(hba, mask);
+ }
+}
+
void ufshcd_fixup_dev_quirks(struct ufs_hba *hba, struct ufs_dev_fix *fixups)
{
struct ufs_dev_fix *f;
ufshcd_wb_probe(hba, desc_buf);
+ ufshcd_temp_notif_probe(hba, desc_buf);
+
/*
* ufshcd_read_string_desc returns size of the string
* reset the error value
if (ret)
goto out;
- ufshcd_clear_ua_wluns(hba);
-
/* Initialize devfreq after UFS device is detected */
if (ufshcd_is_clkscaling_supported(hba)) {
memcpy(&hba->clk_scaling.saved_pwr_info.info,
return ret;
}
-static void ufshcd_request_sense_done(struct request *rq, blk_status_t error)
-{
- if (error != BLK_STS_OK)
- pr_err("%s: REQUEST SENSE failed (%d)\n", __func__, error);
- kfree(rq->end_io_data);
- blk_mq_free_request(rq);
-}
-
-static int
-ufshcd_request_sense_async(struct ufs_hba *hba, struct scsi_device *sdev)
-{
- /*
- * Some UFS devices clear unit attention condition only if the sense
- * size used (UFS_SENSE_SIZE in this case) is non-zero.
- */
- static const u8 cmd[6] = {REQUEST_SENSE, 0, 0, 0, UFS_SENSE_SIZE, 0};
- struct scsi_request *rq;
- struct request *req;
- char *buffer;
- int ret;
-
- buffer = kzalloc(UFS_SENSE_SIZE, GFP_KERNEL);
- if (!buffer)
- return -ENOMEM;
-
- req = blk_mq_alloc_request(sdev->request_queue, REQ_OP_DRV_IN,
- /*flags=*/BLK_MQ_REQ_PM);
- if (IS_ERR(req)) {
- ret = PTR_ERR(req);
- goto out_free;
- }
-
- ret = blk_rq_map_kern(sdev->request_queue, req,
- buffer, UFS_SENSE_SIZE, GFP_NOIO);
- if (ret)
- goto out_put;
-
- rq = scsi_req(req);
- rq->cmd_len = ARRAY_SIZE(cmd);
- memcpy(rq->cmd, cmd, rq->cmd_len);
- rq->retries = 3;
- req->timeout = 1 * HZ;
- req->rq_flags |= RQF_PM | RQF_QUIET;
- req->end_io_data = buffer;
-
- blk_execute_rq_nowait(/*bd_disk=*/NULL, req, /*at_head=*/true,
- ufshcd_request_sense_done);
- return 0;
-
-out_put:
- blk_mq_free_request(req);
-out_free:
- kfree(buffer);
- return ret;
-}
-
-static int ufshcd_clear_ua_wlun(struct ufs_hba *hba, u8 wlun)
-{
- struct scsi_device *sdp;
- unsigned long flags;
- int ret = 0;
-
- spin_lock_irqsave(hba->host->host_lock, flags);
- if (wlun == UFS_UPIU_UFS_DEVICE_WLUN)
- sdp = hba->sdev_ufs_device;
- else if (wlun == UFS_UPIU_RPMB_WLUN)
- sdp = hba->sdev_rpmb;
- else
- BUG();
- if (sdp) {
- ret = scsi_device_get(sdp);
- if (!ret && !scsi_device_online(sdp)) {
- ret = -ENODEV;
- scsi_device_put(sdp);
- }
- } else {
- ret = -ENODEV;
- }
- spin_unlock_irqrestore(hba->host->host_lock, flags);
- if (ret)
- goto out_err;
-
- ret = ufshcd_request_sense_async(hba, sdp);
- scsi_device_put(sdp);
-out_err:
- if (ret)
- dev_err(hba->dev, "%s: UAC clear LU=%x ret = %d\n",
- __func__, wlun, ret);
- return ret;
-}
-
-static int ufshcd_clear_ua_wluns(struct ufs_hba *hba)
-{
- int ret = 0;
-
- if (!hba->wlun_dev_clr_ua)
- goto out;
-
- ret = ufshcd_clear_ua_wlun(hba, UFS_UPIU_UFS_DEVICE_WLUN);
- if (!ret)
- ret = ufshcd_clear_ua_wlun(hba, UFS_UPIU_RPMB_WLUN);
- if (!ret)
- hba->wlun_dev_clr_ua = false;
-out:
- if (ret)
- dev_err(hba->dev, "%s: Failed to clear UAC WLUNS ret = %d\n",
- __func__, ret);
- return ret;
-}
-
/**
* ufshcd_probe_hba - probe hba to detect device and initialize it
* @hba: per-adapter instance
if (ret)
goto out;
+ if (hba->quirks & UFSHCD_QUIRK_SKIP_PH_CONFIGURATION)
+ goto out;
+
/* Debug counters initialization */
ufshcd_clear_dbg_ufs_stats(hba);
/* UFS device is also active now */
ufshcd_set_ufs_dev_active(hba);
ufshcd_force_reset_auto_bkops(hba);
- hba->wlun_dev_clr_ua = true;
- hba->wlun_rpmb_clr_ua = true;
/* Gear up to HS gear if supported */
if (hba->max_pwr_info.is_valid) {
struct scsi_sense_hdr sshdr;
struct scsi_device *sdp;
unsigned long flags;
- int ret;
+ int ret, retries;
spin_lock_irqsave(hba->host->host_lock, flags);
sdp = hba->sdev_ufs_device;
* handling context.
*/
hba->host->eh_noresume = 1;
- if (hba->wlun_dev_clr_ua)
- ufshcd_clear_ua_wlun(hba, UFS_UPIU_UFS_DEVICE_WLUN);
cmd[4] = pwr_mode << 4;
* callbacks hence set the RQF_PM flag so that it doesn't resume the
* already suspended childs.
*/
- ret = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
- START_STOP_TIMEOUT, 0, 0, RQF_PM, NULL);
+ for (retries = 3; retries > 0; --retries) {
+ ret = scsi_execute(sdp, cmd, DMA_NONE, NULL, 0, NULL, &sshdr,
+ START_STOP_TIMEOUT, 0, 0, RQF_PM, NULL);
+ if (!scsi_status_is_check_condition(ret) ||
+ !scsi_sense_valid(&sshdr) ||
+ sshdr.sense_key != UNIT_ATTENTION)
+ break;
+ }
if (ret) {
sdev_printk(KERN_WARNING, sdp,
"START_STOP failed for power mode: %d, result %x\n",
flush_work(&hba->eeh_work);
+ ret = ufshcd_vops_suspend(hba, pm_op, PRE_CHANGE);
+ if (ret)
+ goto enable_scaling;
+
if (req_dev_pwr_mode != hba->curr_dev_pwr_mode) {
if (pm_op != UFS_RUNTIME_PM)
/* ensure that bkops is disabled */
* vendor specific host controller register space call them before the
* host clocks are ON.
*/
- ret = ufshcd_vops_suspend(hba, pm_op);
+ ret = ufshcd_vops_suspend(hba, pm_op, POST_CHANGE);
if (ret)
goto set_link_active;
goto out;
set_old_link_state:
ufshcd_link_state_transition(hba, old_link_state, 0);
vendor_suspend:
- ufshcd_vops_suspend(hba, pm_op);
+ ufshcd_vops_suspend(hba, pm_op, PRE_CHANGE);
+ ufshcd_vops_suspend(hba, pm_op, POST_CHANGE);
out:
if (ret)
ufshcd_update_evt_hist(hba, UFS_EVT_WL_RES_ERR, (u32)ret);
{
if (hba->sdev_ufs_device)
ufshcd_rpm_get_sync(hba);
+ ufs_hwmon_remove(hba);
ufs_bsg_remove(hba);
ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
ufshcd_rpm_put(hba);
hba->complete_put = false;
}
- if (hba->rpmb_complete_put) {
- ufshcd_rpmb_rpm_put(hba);
- hba->rpmb_complete_put = false;
- }
}
EXPORT_SYMBOL_GPL(ufshcd_resume_complete);
}
hba->complete_put = true;
}
- if (hba->sdev_rpmb) {
- ufshcd_rpmb_rpm_get_sync(hba);
- hba->rpmb_complete_put = true;
- }
return 0;
}
EXPORT_SYMBOL_GPL(ufshcd_suspend_prepare);
},
};
-static int ufshcd_rpmb_probe(struct device *dev)
-{
- return is_rpmb_wlun(to_scsi_device(dev)) ? 0 : -ENODEV;
-}
-
-static inline int ufshcd_clear_rpmb_uac(struct ufs_hba *hba)
-{
- int ret = 0;
-
- if (!hba->wlun_rpmb_clr_ua)
- return 0;
- ret = ufshcd_clear_ua_wlun(hba, UFS_UPIU_RPMB_WLUN);
- if (!ret)
- hba->wlun_rpmb_clr_ua = 0;
- return ret;
-}
-
-#ifdef CONFIG_PM
-static int ufshcd_rpmb_resume(struct device *dev)
-{
- struct ufs_hba *hba = wlun_dev_to_hba(dev);
-
- if (hba->sdev_rpmb)
- ufshcd_clear_rpmb_uac(hba);
- return 0;
-}
-#endif
-
-static const struct dev_pm_ops ufs_rpmb_pm_ops = {
- SET_RUNTIME_PM_OPS(NULL, ufshcd_rpmb_resume, NULL)
- SET_SYSTEM_SLEEP_PM_OPS(NULL, ufshcd_rpmb_resume)
-};
-
-/* ufs_rpmb_wlun_template - Describes UFS RPMB WLUN. Used only to send UAC. */
-static struct scsi_driver ufs_rpmb_wlun_template = {
- .gendrv = {
- .name = "ufs_rpmb_wlun",
- .owner = THIS_MODULE,
- .probe = ufshcd_rpmb_probe,
- .pm = &ufs_rpmb_pm_ops,
- },
-};
-
static int __init ufshcd_core_init(void)
{
int ret;
+ /* Verify that there are no gaps in struct utp_transfer_cmd_desc. */
+ static_assert(sizeof(struct utp_transfer_cmd_desc) ==
+ 2 * ALIGNED_UPIU_SIZE +
+ SG_ALL * sizeof(struct ufshcd_sg_entry));
+
ufs_debugfs_init();
ret = scsi_register_driver(&ufs_dev_wlun_template.gendrv);
if (ret)
- goto debugfs_exit;
-
- ret = scsi_register_driver(&ufs_rpmb_wlun_template.gendrv);
- if (ret)
- goto unregister;
-
- return ret;
-unregister:
- scsi_unregister_driver(&ufs_dev_wlun_template.gendrv);
-debugfs_exit:
- ufs_debugfs_exit();
+ ufs_debugfs_exit();
return ret;
}
static void __exit ufshcd_core_exit(void)
{
ufs_debugfs_exit();
- scsi_unregister_driver(&ufs_rpmb_wlun_template.gendrv);
scsi_unregister_driver(&ufs_dev_wlun_template.gendrv);
}
enum ufs_notify_change_status);
int (*apply_dev_quirks)(struct ufs_hba *hba);
void (*fixup_dev_quirks)(struct ufs_hba *hba);
- int (*suspend)(struct ufs_hba *, enum ufs_pm_op);
+ int (*suspend)(struct ufs_hba *, enum ufs_pm_op,
+ enum ufs_notify_change_status);
int (*resume)(struct ufs_hba *, enum ufs_pm_op);
void (*dbg_register_dump)(struct ufs_hba *hba);
int (*phy_initialization)(struct ufs_hba *);
* This quirk allows only sg entries aligned with page size.
*/
UFSHCD_QUIRK_ALIGN_SG_WITH_PAGE_SIZE = 1 << 14,
+
+ /*
+ * This quirk needs to be enabled if the host controller does not
+ * support UIC command
+ */
+ UFSHCD_QUIRK_BROKEN_UIC_CMD = 1 << 15,
+
+ /*
+ * This quirk needs to be enabled if the host controller cannot
+ * support physical host configuration.
+ */
+ UFSHCD_QUIRK_SKIP_PH_CONFIGURATION = 1 << 16,
};
enum ufshcd_caps {
* in order to exit DeepSleep state.
*/
UFSHCD_CAP_DEEPSLEEP = 1 << 10,
+
+ /*
+ * This capability allows the host controller driver to use temperature
+ * notification if it is supported by the UFS device.
+ */
+ UFSHCD_CAP_TEMP_NOTIF = 1 << 11,
};
struct ufs_hba_variant_params {
struct scsi_device *sdev_ufs_device;
struct scsi_device *sdev_rpmb;
+#ifdef CONFIG_SCSI_UFS_HWMON
+ struct device *hwmon_device;
+#endif
+
enum ufs_dev_pwr_mode curr_dev_pwr_mode;
enum uic_link_state uic_link_state;
/* Desired UFS power management level during runtime PM */
struct ufs_vreg_info vreg_info;
struct list_head clk_list_head;
- bool wlun_dev_clr_ua;
- bool wlun_rpmb_clr_ua;
-
/* Number of requests aborts */
int req_abort_count;
#endif
u32 luns_avail;
bool complete_put;
- bool rpmb_complete_put;
};
/* Returns true if clocks can be gated. Otherwise false */
int ufshcd_link_recovery(struct ufs_hba *hba);
int ufshcd_make_hba_operational(struct ufs_hba *hba);
void ufshcd_remove(struct ufs_hba *);
+int ufshcd_uic_hibern8_enter(struct ufs_hba *hba);
int ufshcd_uic_hibern8_exit(struct ufs_hba *hba);
void ufshcd_delay_us(unsigned long us, unsigned long tolerance);
int ufshcd_wait_for_register(struct ufs_hba *hba, u32 reg, u32 mask,
void ufshcd_parse_dev_ref_clk_freq(struct ufs_hba *hba, struct clk *refclk);
void ufshcd_update_evt_hist(struct ufs_hba *hba, u32 id, u32 val);
void ufshcd_hba_stop(struct ufs_hba *hba);
+void ufshcd_schedule_eh_work(struct ufs_hba *hba);
static inline void check_upiu_size(void)
{
return 0;
}
+#ifdef CONFIG_SCSI_UFS_HWMON
+void ufs_hwmon_probe(struct ufs_hba *hba, u8 mask);
+void ufs_hwmon_remove(struct ufs_hba *hba);
+void ufs_hwmon_notify_event(struct ufs_hba *hba, u8 ee_mask);
+#else
+static inline void ufs_hwmon_probe(struct ufs_hba *hba, u8 mask) {}
+static inline void ufs_hwmon_remove(struct ufs_hba *hba) {}
+static inline void ufs_hwmon_notify_event(struct ufs_hba *hba, u8 ee_mask) {}
+#endif
+
#ifdef CONFIG_PM
extern int ufshcd_runtime_suspend(struct device *dev);
extern int ufshcd_runtime_resume(struct device *dev);
hba->vops->fixup_dev_quirks(hba);
}
-static inline int ufshcd_vops_suspend(struct ufs_hba *hba, enum ufs_pm_op op)
+static inline int ufshcd_vops_suspend(struct ufs_hba *hba, enum ufs_pm_op op,
+ enum ufs_notify_change_status status)
{
if (hba->vops && hba->vops->suspend)
- return hba->vops->suspend(hba, op);
+ return hba->vops->suspend(hba, op, status);
return 0;
}
return pm_runtime_put(&hba->sdev_ufs_device->sdev_gendev);
}
-static inline int ufshcd_rpmb_rpm_get_sync(struct ufs_hba *hba)
-{
- return pm_runtime_get_sync(&hba->sdev_rpmb->sdev_gendev);
-}
-
-static inline int ufshcd_rpmb_rpm_put(struct ufs_hba *hba)
-{
- return pm_runtime_put(&hba->sdev_rpmb->sdev_gendev);
-}
-
#endif /* End of Header */
};
/* Overall command status values */
-enum {
+enum utp_ocs {
OCS_SUCCESS = 0x0,
OCS_INVALID_CMD_TABLE_ATTR = 0x1,
OCS_INVALID_PRDT_ATTR = 0x2,
OCS_INVALID_CRYPTO_CONFIG = 0x9,
OCS_GENERAL_CRYPTO_ERROR = 0xA,
OCS_INVALID_COMMAND_STATUS = 0x0F,
+};
+
+enum {
MASK_OCS = 0x0F,
};
/**
* struct ufshcd_sg_entry - UFSHCI PRD Entry
- * @base_addr: Lower 32bit physical address DW-0
- * @upper_addr: Upper 32bit physical address DW-1
+ * @addr: Physical address; DW-0 and DW-1.
* @reserved: Reserved for future use DW-2
* @size: size of physical segment DW-3
*/
struct ufshcd_sg_entry {
- __le32 base_addr;
- __le32 upper_addr;
+ __le64 addr;
__le32 reserved;
__le32 size;
};
/**
- * struct utp_transfer_cmd_desc - UFS Command Descriptor structure
+ * struct utp_transfer_cmd_desc - UTP Command Descriptor (UCD)
* @command_upiu: Command UPIU Frame address
* @response_upiu: Response UPIU Frame address
* @prd_table: Physical Region Descriptor
};
/**
- * struct utp_transfer_req_desc - UTRD structure
+ * struct utp_transfer_req_desc - UTP Transfer Request Descriptor (UTRD)
* @header: UTRD header DW-0 to DW-3
* @command_desc_base_addr_lo: UCD base address low DW-4
* @command_desc_base_addr_hi: UCD base address high DW-5
if (!ufshpb_is_supported_chunk(hpb, transfer_len))
return 0;
- WARN_ON_ONCE(transfer_len > HPB_MULTI_CHUNK_HIGH);
-
if (hpb->is_hcm) {
/*
* in host control mode, reads are the main source for
if (ufshpb_is_legacy(hba))
hpb->pre_req_max_tr_len = HPB_LEGACY_CHUNK_HIGH;
else
- hpb->pre_req_max_tr_len = HPB_MULTI_CHUNK_HIGH;
+ hpb->pre_req_max_tr_len = hpb_dev_info->max_hpb_single_cmd;
hpb->lu_pinned_start = hpb_lu_info->pinned_start;
hpb->lu_pinned_end = hpb_lu_info->num_pinned ?
ufshcd_map_desc_id_to_length(hba, QUERY_DESC_IDN_UNIT, &size);
- pm_runtime_get_sync(hba->dev);
+ ufshcd_rpm_get_sync(hba);
ret = ufshcd_query_descriptor_retry(hba, UPIU_QUERY_OPCODE_READ_DESC,
QUERY_DESC_IDN_UNIT, lun, 0,
desc_buf, &size);
- pm_runtime_put_sync(hba->dev);
+ ufshcd_rpm_put_sync(hba);
if (ret) {
dev_err(hba->dev,
{
struct ufshpb_dev_info *hpb_dev_info = &hba->ufshpb_dev;
int version, ret;
- u32 max_hpb_single_cmd = HPB_MULTI_CHUNK_LOW;
+ int max_single_cmd;
hpb_dev_info->control_mode = desc_buf[DEVICE_DESC_PARAM_HPB_CONTROL];
if (version == HPB_SUPPORT_LEGACY_VERSION)
hpb_dev_info->is_legacy = true;
- pm_runtime_get_sync(hba->dev);
- ret = ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_READ_ATTR,
- QUERY_ATTR_IDN_MAX_HPB_SINGLE_CMD, 0, 0, &max_hpb_single_cmd);
- pm_runtime_put_sync(hba->dev);
-
- if (ret)
- dev_err(hba->dev, "%s: idn: read max size of single hpb cmd query request failed",
- __func__);
- hpb_dev_info->max_hpb_single_cmd = max_hpb_single_cmd;
-
/*
* Get the number of user logical unit to check whether all
* scsi_device finish initialization
*/
hpb_dev_info->num_lu = desc_buf[DEVICE_DESC_PARAM_NUM_LU];
+
+ if (hpb_dev_info->is_legacy)
+ return;
+
+ ret = ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_READ_ATTR,
+ QUERY_ATTR_IDN_MAX_HPB_SINGLE_CMD, 0, 0, &max_single_cmd);
+
+ if (ret)
+ hpb_dev_info->max_hpb_single_cmd = HPB_LEGACY_CHUNK_HIGH;
+ else
+ hpb_dev_info->max_hpb_single_cmd = min(max_single_cmd + 1, HPB_MULTI_CHUNK_HIGH);
}
void ufshpb_init(struct ufs_hba *hba)
/* hpb support chunk size */
#define HPB_LEGACY_CHUNK_HIGH 1
-#define HPB_MULTI_CHUNK_LOW 7
#define HPB_MULTI_CHUNK_HIGH 255
/* hpb vender defined opcode */
VIRTIO_SCSI_SENSE_SIZE));
}
- sc->scsi_done(sc);
+ scsi_done(sc);
}
static void virtscsi_vq_done(struct virtio_scsi *vscsi,
* we're using independent interrupts (e.g. MSI). Poll the
* virtqueues once.
*
- * In the abort case, sc->scsi_done will do nothing, because
- * the block layer must have detected a timeout and as a result
- * REQ_ATOM_COMPLETE has been set.
+ * In the abort case, scsi_done() will do nothing, because the
+ * command timed out and hence SCMD_STATE_COMPLETE has been set.
*/
virtscsi_poll_requests(vscsi);
"cmd=%p %x ctx=%p result=0x%x status=0x%x,%x\n",
cmd, cmd->cmnd[0], ctx, cmd->result, btstat, sdstat);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
/*
return 0;
}
-static int pvscsi_queue_lck(struct scsi_cmnd *cmd, void (*done)(struct scsi_cmnd *))
+static int pvscsi_queue_lck(struct scsi_cmnd *cmd)
{
struct Scsi_Host *host = cmd->device->host;
struct pvscsi_adapter *adapter = shost_priv(host);
return SCSI_MLQUEUE_HOST_BUSY;
}
- cmd->scsi_done = done;
op = cmd->cmnd[0];
dev_dbg(&cmd->device->sdev_gendev,
* Successfully aborted the command.
*/
cmd->result = (DID_ABORT << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
out:
spin_unlock_irqrestore(&adapter->hw_lock, flags);
pvscsi_patch_sense(cmd);
pvscsi_release_context(adapter, ctx);
cmd->result = (DID_RESET << 16);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
}
}
msg[1] = offset;
}
-static int
-wd33c93_queuecommand_lck(struct scsi_cmnd *cmd,
- void (*done)(struct scsi_cmnd *))
+static int wd33c93_queuecommand_lck(struct scsi_cmnd *cmd)
{
struct WD33C93_hostdata *hostdata;
struct scsi_cmnd *tmp;
/* Set up a few fields in the scsi_cmnd structure for our own use:
* - host_scribble is the pointer to the next cmd in the input queue
- * - scsi_done points to the routine we call when a cmd is finished
* - result is what you'd expect
*/
cmd->host_scribble = NULL;
- cmd->scsi_done = done;
cmd->result = 0;
/* We use the Scsi_Pointer structure that's included with each command
cmd->result = DID_NO_CONNECT << 16;
hostdata->busy[cmd->device->id] &= ~(1 << (cmd->device->lun & 0xff));
hostdata->state = S_UNCONNECTED;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/* From esp.c:
* There is a window of time within the scsi_done() path
scsi_msg_to_host_byte(cmd, cmd->SCp.Message);
set_status_byte(cmd, cmd->SCp.Status);
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/* We are no longer connected to a target - check to see if
* there are commands waiting to be executed.
scsi_msg_to_host_byte(cmd, cmd->SCp.Message);
set_status_byte(cmd, cmd->SCp.Status);
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
/* We are no longer connected to a target - check to see if
* there are commands waiting to be executed.
scsi_msg_to_host_byte(cmd, cmd->SCp.Message);
set_status_byte(cmd, cmd->SCp.Status);
}
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
break;
case S_PRE_TMP_DISC:
case S_RUNNING_LEVEL2:
("scsi%d: Abort - removing command from input_Q. ",
instance->host_no);
enable_irq(cmd->device->host->irq);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return SUCCESS;
}
prev = tmp;
wd33c93_execute(instance);
enable_irq(cmd->device->host->irq);
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return SUCCESS;
}
SCSI_SENSE_BUFFERSIZE, DMA_FROM_DEVICE);
cmd->result = result << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
}
/* Build a SCB and send it to the card */
DMA_BIDIRECTIONAL);
out_error:
cmd->result = DID_ERROR << 16;
- cmd->scsi_done(cmd);
+ scsi_done(cmd);
return 0;
}
if (sense_len)
memcpy(sc->sense_buffer, ring_rsp->sense_buffer, sense_len);
- sc->scsi_done(sc);
+ scsi_done(sc);
}
static void scsifront_sync_cmd_done(struct vscsifrnt_info *info,
if (err == -ENOMEM)
return SCSI_MLQUEUE_HOST_BUSY;
sc->result = DID_ERROR << 16;
- sc->scsi_done(sc);
+ scsi_done(sc);
return 0;
}
maple_queue_cache = KMEM_CACHE(maple_buffer, SLAB_HWCACHE_ALIGN);
- if (!maple_queue_cache)
+ if (!maple_queue_cache) {
+ retval = -ENOMEM;
goto cleanup_bothirqs;
+ }
INIT_LIST_HEAD(&maple_waitq);
INIT_LIST_HEAD(&maple_sentq);
if (!mdev[i]) {
while (i-- > 0)
maple_free_dev(mdev[i]);
+ retval = -ENOMEM;
goto cleanup_cache;
}
baseunits[i] = mdev[i];
cd->cur_ptr += bytes;
written += bytes;
+ kfree(kbuf);
return written;
err_free_buf:
* potentially being migrated away.
*/
if (cpu < 0)
- cpu = smp_processor_id();
+ cpu = raw_smp_processor_id();
/* If a specific cpu was requested, pick it up immediately */
return dpio_by_cpu[cpu];
int i, num_enqueued = 0;
unsigned long irq_flags;
- spin_lock(&s->access_spinlock);
- local_irq_save(irq_flags);
+ spin_lock_irqsave(&s->access_spinlock, irq_flags);
half_mask = (s->eqcr.pi_ci_mask>>1);
full_mask = s->eqcr.pi_ci_mask;
s->eqcr.available = qm_cyc_diff(s->eqcr.pi_ring_size,
eqcr_ci, s->eqcr.ci);
if (!s->eqcr.available) {
- local_irq_restore(irq_flags);
- spin_unlock(&s->access_spinlock);
+ spin_unlock_irqrestore(&s->access_spinlock, irq_flags);
return 0;
}
}
dma_wmb();
qbman_write_register(s, QBMAN_CINH_SWP_EQCR_PI,
(QB_RT_BIT)|(s->eqcr.pi)|s->eqcr.pi_vb);
- local_irq_restore(irq_flags);
- spin_unlock(&s->access_spinlock);
+ spin_unlock_irqrestore(&s->access_spinlock, irq_flags);
return num_enqueued;
}
}
EXPORT_SYMBOL_GPL(wkup_m3_ipc_put);
-static void wkup_m3_rproc_boot_thread(struct wkup_m3_ipc *m3_ipc)
+static int wkup_m3_rproc_boot_thread(void *arg)
{
+ struct wkup_m3_ipc *m3_ipc = arg;
struct device *dev = m3_ipc->dev;
int ret;
else
m3_ipc_state = m3_ipc;
- do_exit(0);
+ return 0;
}
static int wkup_m3_ipc_probe(struct platform_device *pdev)
* can boot the wkup_m3 as soon as it's ready without holding
* up kernel boot
*/
- task = kthread_run((void *)wkup_m3_rproc_boot_thread, m3_ipc,
+ task = kthread_run(wkup_m3_rproc_boot_thread, m3_ipc,
"wkup_m3_rproc_loader");
if (IS_ERR(task)) {
#define CQSPI_NEEDS_WR_DELAY BIT(0)
#define CQSPI_DISABLE_DAC_MODE BIT(1)
#define CQSPI_SUPPORT_EXTERNAL_DMA BIT(2)
+#define CQSPI_NO_SUPPORT_WR_COMPLETION BIT(3)
/* Capabilities */
#define CQSPI_SUPPORTS_OCTAL BIT(0)
struct cqspi_flash_pdata f_pdata[CQSPI_MAX_CHIPSELECT];
bool use_dma_read;
u32 pd_dev_id;
+ bool wr_completion;
};
struct cqspi_driver_platdata {
* polling on the controller's side. spinand and spi-nor will take
* care of polling the status register.
*/
- reg = readl(reg_base + CQSPI_REG_WR_COMPLETION_CTRL);
- reg |= CQSPI_REG_WR_DISABLE_AUTO_POLL;
- writel(reg, reg_base + CQSPI_REG_WR_COMPLETION_CTRL);
+ if (cqspi->wr_completion) {
+ reg = readl(reg_base + CQSPI_REG_WR_COMPLETION_CTRL);
+ reg |= CQSPI_REG_WR_DISABLE_AUTO_POLL;
+ writel(reg, reg_base + CQSPI_REG_WR_COMPLETION_CTRL);
+ }
reg = readl(reg_base + CQSPI_REG_SIZE);
reg &= ~CQSPI_REG_SIZE_ADDRESS_MASK;
cqspi->master_ref_clk_hz = clk_get_rate(cqspi->clk);
master->max_speed_hz = cqspi->master_ref_clk_hz;
+
+ /* write completion is supported by default */
+ cqspi->wr_completion = true;
+
ddata = of_device_get_match_data(dev);
if (ddata) {
if (ddata->quirks & CQSPI_NEEDS_WR_DELAY)
cqspi->use_direct_mode = true;
if (ddata->quirks & CQSPI_SUPPORT_EXTERNAL_DMA)
cqspi->use_dma_read = true;
+ if (ddata->quirks & CQSPI_NO_SUPPORT_WR_COMPLETION)
+ cqspi->wr_completion = false;
if (of_device_is_compatible(pdev->dev.of_node,
"xlnx,versal-ospi-1.0"))
.quirks = CQSPI_DISABLE_DAC_MODE,
};
+static const struct cqspi_driver_platdata socfpga_qspi = {
+ .quirks = CQSPI_NO_SUPPORT_WR_COMPLETION,
+};
+
static const struct cqspi_driver_platdata versal_ospi = {
.hwcaps_mask = CQSPI_SUPPORTS_OCTAL,
.quirks = CQSPI_DISABLE_DAC_MODE | CQSPI_SUPPORT_EXTERNAL_DMA,
.compatible = "xlnx,versal-ospi-1.0",
.data = (void *)&versal_ospi,
},
+ {
+ .compatible = "intel,socfpga-qspi",
+ .data = (void *)&socfpga_qspi,
+ },
{ /* end of table */ }
};
ret = devm_spi_register_controller(&pdev->dev, controller);
if (ret < 0) {
- dev_err(&pdev->dev, "spi_register_controller error.\n");
+ dev_err_probe(&pdev->dev, ret, "spi_register_controller error: %i\n", ret);
goto out_pm_get;
}
int ret;
mas->tx = dma_request_chan(mas->dev, "tx");
- ret = dev_err_probe(mas->dev, IS_ERR(mas->tx), "Failed to get tx DMA ch\n");
- if (ret < 0)
+ if (IS_ERR(mas->tx)) {
+ ret = dev_err_probe(mas->dev, PTR_ERR(mas->tx),
+ "Failed to get tx DMA ch\n");
goto err_tx;
+ }
mas->rx = dma_request_chan(mas->dev, "rx");
- ret = dev_err_probe(mas->dev, IS_ERR(mas->rx), "Failed to get rx DMA ch\n");
- if (ret < 0)
+ if (IS_ERR(mas->rx)) {
+ ret = dev_err_probe(mas->dev, PTR_ERR(mas->rx),
+ "Failed to get rx DMA ch\n");
goto err_rx;
+ }
return 0;
err_rx:
+ mas->rx = NULL;
dma_release_channel(mas->tx);
- mas->tx = NULL;
err_tx:
- mas->rx = NULL;
+ mas->tx = NULL;
return ret;
}
device_del(&ctlr->dev);
- /* Release the last reference on the controller if its driver
- * has not yet been converted to devm_spi_alloc_master/slave().
- */
- if (!ctlr->devm_allocated)
- put_device(&ctlr->dev);
-
/* free bus id */
mutex_lock(&board_lock);
if (found == ctlr)
if (IS_ENABLED(CONFIG_SPI_DYNAMIC))
mutex_unlock(&ctlr->add_lock);
+
+ /* Release the last reference on the controller if its driver
+ * has not yet been converted to devm_spi_alloc_master/slave().
+ */
+ if (!ctlr->devm_allocated)
+ put_device(&ctlr->dev);
}
EXPORT_SYMBOL_GPL(spi_unregister_controller);
{
struct ssb_bus *ssb;
int err = -ENOMEM;
- const char *name;
u32 val;
ssb = kzalloc(sizeof(*ssb), GFP_KERNEL);
err = pci_enable_device(dev);
if (err)
goto err_kfree_ssb;
- name = dev_name(&dev->dev);
- if (dev->driver && dev->driver->name)
- name = dev->driver->name;
- err = pci_request_regions(dev, name);
+ err = pci_request_regions(dev, dev_driver_string(&dev->dev));
if (err)
goto err_pci_disable;
pci_set_master(dev);
source "drivers/staging/pi433/Kconfig"
-source "drivers/staging/mt7621-pci/Kconfig"
-
source "drivers/staging/mt7621-dma/Kconfig"
source "drivers/staging/ralink-gdma/Kconfig"
obj-$(CONFIG_GREYBUS) += greybus/
obj-$(CONFIG_BCM2835_VCHIQ) += vc04_services/
obj-$(CONFIG_PI433) += pi433/
-obj-$(CONFIG_PCI_MT7621) += mt7621-pci/
obj-$(CONFIG_SOC_MT7621) += mt7621-dma/
obj-$(CONFIG_DMA_RALINK) += ralink-gdma/
obj-$(CONFIG_SOC_MT7621) += mt7621-dts/
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0
-config PCI_MT7621
- tristate "MediaTek MT7621 PCI Controller"
- depends on RALINK
- select PCI_DRIVERS_GENERIC
- help
- This selects a driver for the MediaTek MT7621 PCI Controller.
-
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_PCI_MT7621) += pci-mt7621.o
+++ /dev/null
-
-- general code review and cleanup
-
-Cc: NeilBrown <neil@brown.name>
+++ /dev/null
-MediaTek MT7621 PCIe controller
-
-Required properties:
-- compatible: "mediatek,mt7621-pci"
-- device_type: Must be "pci"
-- reg: Base addresses and lengths of the PCIe subsys and root ports.
-- bus-range: Range of bus numbers associated with this controller.
-- #address-cells: Address representation for root ports (must be 3)
-- pinctrl-names : The pin control state names.
-- pinctrl-0: The "default" pinctrl state.
-- #size-cells: Size representation for root ports (must be 2)
-- ranges: Ranges for the PCI memory and I/O regions.
-- #interrupt-cells: Must be 1
-- interrupt-map-mask and interrupt-map: Standard PCI IRQ mapping properties.
- Please refer to the standard PCI bus binding document for a more detailed
- explanation.
-- status: either "disabled" or "okay".
-- resets: Must contain an entry for each entry in reset-names.
- See ../reset/reset.txt for details.
-- reset-names: Must be "pcie0", "pcie1", "pcieN"... based on the number of
- root ports.
-- clocks: Must contain an entry for each entry in clock-names.
- See ../clocks/clock-bindings.txt for details.
-- clock-names: Must be "pcie0", "pcie1", "pcieN"... based on the number of
- root ports.
-- reset-gpios: GPIO specs for the reset pins.
-
-In addition, the device tree node must have sub-nodes describing each PCIe port
-interface, having the following mandatory properties:
-
-Required properties:
-- reg: Only the first four bytes are used to refer to the correct bus number
- and device number.
-- #address-cells: Must be 3
-- #size-cells: Must be 2
-- ranges: Sub-ranges distributed from the PCIe controller node. An empty
- property is sufficient.
-- bus-range: Range of bus numbers associated with this port.
-
-Example for MT7621:
-
- pcie: pcie@1e140000 {
- compatible = "mediatek,mt7621-pci";
- reg = <0x1e140000 0x100 /* host-pci bridge registers */
- 0x1e142000 0x100 /* pcie port 0 RC control registers */
- 0x1e143000 0x100 /* pcie port 1 RC control registers */
- 0x1e144000 0x100>; /* pcie port 2 RC control registers */
-
- #address-cells = <3>;
- #size-cells = <2>;
-
- pinctrl-names = "default";
- pinctrl-0 = <&pcie_pins>;
-
- device_type = "pci";
-
- bus-range = <0 255>;
- ranges = <
- 0x02000000 0 0x00000000 0x60000000 0 0x10000000 /* pci memory */
- 0x01000000 0 0x00000000 0x1e160000 0 0x00010000 /* io space */
- >;
-
- #interrupt-cells = <1>;
- interrupt-map-mask = <0xF0000 0 0 1>;
- interrupt-map = <0x10000 0 0 1 &gic GIC_SHARED 4 IRQ_TYPE_LEVEL_HIGH>,
- <0x20000 0 0 1 &gic GIC_SHARED 24 IRQ_TYPE_LEVEL_HIGH>,
- <0x30000 0 0 1 &gic GIC_SHARED 25 IRQ_TYPE_LEVEL_HIGH>;
-
- status = "disabled";
-
- resets = <&rstctrl 24 &rstctrl 25 &rstctrl 26>;
- reset-names = "pcie0", "pcie1", "pcie2";
- clocks = <&clkctrl 24 &clkctrl 25 &clkctrl 26>;
- clock-names = "pcie0", "pcie1", "pcie2";
-
- reset-gpios = <&gpio 19 GPIO_ACTIVE_LOW>,
- <&gpio 8 GPIO_ACTIVE_LOW>,
- <&gpio 7 GPIO_ACTIVE_LOW>;
-
- pcie@0,0 {
- reg = <0x0000 0 0 0 0>;
- #address-cells = <3>;
- #size-cells = <2>;
- ranges;
- bus-range = <0x00 0xff>;
- };
-
- pcie@1,0 {
- reg = <0x0800 0 0 0 0>;
- #address-cells = <3>;
- #size-cells = <2>;
- ranges;
- bus-range = <0x00 0xff>;
- };
-
- pcie@2,0 {
- reg = <0x1000 0 0 0 0>;
- #address-cells = <3>;
- #size-cells = <2>;
- ranges;
- bus-range = <0x00 0xff>;
- };
- };
-
complete(&pcmdpriv->stop_cmd_thread);
- thread_exit();
+ return 0;
}
/*
spinlock_t lock;
};
-#define thread_exit() complete_and_exit(NULL, 0)
-
static inline struct list_head *get_list_head(struct __queue *queue)
{
return (&(queue->queue));
#define _pkt struct sk_buff
#define _buffer unsigned char
-#define thread_exit() complete_and_exit(NULL, 0)
#define _init_queue(pqueue) \
do { \
r8712_free_cmd_obj(pcmd);
} while (1);
complete(&pcmdpriv->terminate_cmdthread_comp);
- thread_exit();
+ return 0;
}
void r8712_event_handle(struct _adapter *padapter, __le32 *peventbuf)
complete(&pcmdpriv->terminate_cmdthread_comp);
atomic_set(&pcmdpriv->cmdthd_running, false);
- thread_exit();
+ return 0;
}
/*
complete(&padapter->xmitpriv.terminate_xmitthread_comp);
- thread_exit();
+ return 0;
}
void rtw_sctx_init(struct submit_ctx *sctx, int timeout_ms)
complete(&pxmitpriv->SdioXmitTerminate);
- thread_exit();
+ return 0;
}
s32 rtl8723bs_mgnt_xmit(
spinlock_t lock;
};
- #define thread_exit() complete_and_exit(NULL, 0)
-
static inline struct list_head *get_next(struct list_head *list)
{
return list->next;
/* queue a command */
/* This is always called with scsi_lock(host) held */
-static int queuecommand_lck(struct scsi_cmnd *srb,
- void (*done)(struct scsi_cmnd *))
+static int queuecommand_lck(struct scsi_cmnd *srb)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct rtsx_dev *dev = host_to_rtsx(srb->device->host);
struct rtsx_chip *chip = dev->chip;
}
/* enqueue the command and wake up the control thread */
- srb->scsi_done = done;
chip->srb = srb;
complete(&dev->cmnd_ready);
/* indicate that the command is done */
else if (chip->srb->result != DID_ABORT << 16) {
- chip->srb->scsi_done(chip->srb);
+ scsi_done(chip->srb);
} else {
skip_for_abort:
dev_err(&dev->pci->dev, "scsi command aborted\n");
if (chip->srb) {
chip->srb->result = DID_NO_CONNECT << 16;
scsi_lock(host);
- chip->srb->scsi_done(dev->chip->srb);
+ scsi_done(dev->chip->srb);
chip->srb = NULL;
scsi_unlock(host);
}
rtn = forward_taskmgmt_command(TASK_MGMT_ABORT_TASK, scsidev);
if (rtn == SUCCESS) {
scsicmd->result = DID_ABORT << 16;
- scsicmd->scsi_done(scsicmd);
+ scsi_done(scsicmd);
}
return rtn;
}
rtn = forward_taskmgmt_command(TASK_MGMT_LUN_RESET, scsidev);
if (rtn == SUCCESS) {
scsicmd->result = DID_RESET << 16;
- scsicmd->scsi_done(scsicmd);
+ scsi_done(scsicmd);
}
return rtn;
}
rtn = forward_taskmgmt_command(TASK_MGMT_BUS_RESET, scsidev);
if (rtn == SUCCESS) {
scsicmd->result = DID_RESET << 16;
- scsicmd->scsi_done(scsicmd);
+ scsi_done(scsicmd);
}
return rtn;
}
* Return: 0 if successfully queued to the Service Partition, otherwise
* error code
*/
-static int visorhba_queue_command_lck(struct scsi_cmnd *scsicmd,
- void (*visorhba_cmnd_done)
- (struct scsi_cmnd *))
+static int visorhba_queue_command_lck(struct scsi_cmnd *scsicmd)
{
+ void (*visorhba_cmnd_done)(struct scsi_cmnd *) = scsi_done;
struct uiscmdrsp *cmdrsp;
struct scsi_device *scsidev = scsicmd->device;
int insert_location;
*/
cmdrsp->scsi.handle = insert_location;
- /* save done function that we have call when cmd is complete */
- scsicmd->scsi_done = visorhba_cmnd_done;
+ WARN_ON_ONCE(visorhba_cmnd_done != scsi_done);
/* save destination */
cmdrsp->scsi.vdest.channel = scsidev->channel;
cmdrsp->scsi.vdest.id = scsidev->id;
.eh_device_reset_handler = visorhba_device_reset_handler,
.eh_bus_reset_handler = visorhba_bus_reset_handler,
.eh_host_reset_handler = visorhba_host_reset_handler,
- .shost_attrs = NULL,
#define visorhba_MAX_CMNDS 128
.can_queue = visorhba_MAX_CMNDS,
.sg_tablesize = 64,
case CMD_SCSI_TYPE:
scsicmd = pendingdel->sent;
scsicmd->result = DID_RESET << 16;
- if (scsicmd->scsi_done)
- scsicmd->scsi_done(scsicmd);
+ scsi_done(scsicmd);
break;
case CMD_SCSITASKMGMT_TYPE:
cmdrsp = pendingdel->sent;
else
do_scsi_nolinuxstat(cmdrsp, scsicmd);
- scsicmd->scsi_done(scsicmd);
+ scsi_done(scsicmd);
}
/*
csk->rcv_win = CXGBIT_10G_RCV_WIN;
if (scale)
csk->rcv_win *= scale;
+ csk->rcv_win = min(csk->rcv_win, RCV_BUFSIZ_M << 10);
#define CXGBIT_10G_SND_WIN (256 * 1024)
csk->snd_win = CXGBIT_10G_SND_WIN;
if (scale)
csk->snd_win *= scale;
+ csk->snd_win = min(csk->snd_win, 512U * 1024);
pr_debug("%s snd_win %d rcv_win %d\n",
__func__, csk->snd_win, csk->rcv_win);
if (!skb)
return -1;
- credit_dack = RX_DACK_CHANGE_F | RX_DACK_MODE_V(1) |
+ credit_dack = RX_DACK_CHANGE_F | RX_DACK_MODE_V(3) |
RX_CREDITS_V(csk->rx_credits);
cxgb_mk_rx_data_ack(skb, len, csk->tid, csk->ctrlq_idx,
if (tcph->ece && tcph->cwr)
opt2 |= CCTRL_ECN_V(1);
- opt2 |= RX_COALESCE_V(3);
opt2 |= CONG_CNTRL_V(CONG_ALG_NEWRENO);
opt2 |= T5_ISS_F;
csk->rcv_nxt = rcv_isn;
- if (csk->rcv_win > (RCV_BUFSIZ_M << 10))
- csk->rx_credits = (csk->rcv_win - (RCV_BUFSIZ_M << 10));
-
csk->snd_wscale = TCPOPT_SND_WSCALE_G(tcp_opt);
cxgbit_set_emss(csk, tcp_opt);
dst_confirm(csk->dst);
struct cxgb4_lld_info *lldi = &cdev->lldi;
u32 mdsl;
-#define ULP2_MAX_PKT_LEN 16224
-#define ISCSI_PDU_NONPAYLOAD_LEN 312
- mdsl = min_t(u32, lldi->iscsi_iolen - ISCSI_PDU_NONPAYLOAD_LEN,
- ULP2_MAX_PKT_LEN - ISCSI_PDU_NONPAYLOAD_LEN);
- mdsl = min_t(u32, mdsl, 8192);
+#define CXGBIT_T5_MAX_PDU_LEN 16224
+#define CXGBIT_PDU_NONPAYLOAD_LEN 312 /* 48(BHS) + 256(AHS) + 8(Digest) */
+ if (is_t5(lldi->adapter_type)) {
+ mdsl = min_t(u32, lldi->iscsi_iolen - CXGBIT_PDU_NONPAYLOAD_LEN,
+ CXGBIT_T5_MAX_PDU_LEN - CXGBIT_PDU_NONPAYLOAD_LEN);
+ } else {
+ mdsl = lldi->iscsi_iolen - CXGBIT_PDU_NONPAYLOAD_LEN;
+ mdsl = min(mdsl, 16384U);
+ }
+
+ mdsl = round_down(mdsl, 4);
+ mdsl = min_t(u32, mdsl, 4 * PAGE_SIZE);
mdsl = min_t(u32, mdsl, (MAX_SKB_FRAGS - 1) * PAGE_SIZE);
cdev->mdsl = mdsl;
wr_ulp_mode = FW_OFLD_TX_DATA_WR_ULPMODE_V(ULP_MODE_ISCSI) |
FW_OFLD_TX_DATA_WR_ULPSUBMODE_V(submode);
- req->tunnel_to_proxy = htonl((wr_ulp_mode) | force |
- FW_OFLD_TX_DATA_WR_SHOVE_V(skb_peek(&csk->txq) ? 0 : 1));
+ req->tunnel_to_proxy = htonl(wr_ulp_mode | force |
+ FW_OFLD_TX_DATA_WR_SHOVE_F);
}
static void cxgbit_arp_failure_skb_discard(void *handle, struct sk_buff *skb)
return ret;
}
-static int cxgbit_rx_lro_skb(struct cxgbit_sock *csk, struct sk_buff *skb)
+static int cxgbit_t5_rx_lro_skb(struct cxgbit_sock *csk, struct sk_buff *skb)
{
struct cxgbit_lro_cb *lro_cb = cxgbit_skb_lro_cb(skb);
struct cxgbit_lro_pdu_cb *pdu_cb = cxgbit_skb_lro_pdu_cb(skb, 0);
return ret;
}
+static int cxgbit_rx_lro_skb(struct cxgbit_sock *csk, struct sk_buff *skb)
+{
+ struct cxgbit_lro_cb *lro_cb = cxgbit_skb_lro_cb(skb);
+ int ret;
+
+ ret = cxgbit_process_lro_skb(csk, skb);
+ if (ret)
+ return ret;
+
+ csk->rx_credits += lro_cb->pdu_totallen;
+ if (csk->rx_credits >= csk->rcv_win) {
+ csk->rx_credits = 0;
+ cxgbit_rx_data_ack(csk);
+ }
+
+ return 0;
+}
+
static int cxgbit_rx_skb(struct cxgbit_sock *csk, struct sk_buff *skb)
{
struct cxgb4_lld_info *lldi = &csk->com.cdev->lldi;
if (likely(cxgbit_skcb_flags(skb) & SKCBF_RX_LRO)) {
if (is_t5(lldi->adapter_type))
- ret = cxgbit_rx_lro_skb(csk, skb);
+ ret = cxgbit_t5_rx_lro_skb(csk, skb);
else
- ret = cxgbit_process_lro_skb(csk, skb);
+ ret = cxgbit_rx_lro_skb(csk, skb);
}
__kfree_skb(skb);
/* Start items for lio_target_tpg_cit */
-static ssize_t lio_target_tpg_enable_show(struct config_item *item, char *page)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct iscsi_portal_group *tpg = container_of(se_tpg,
- struct iscsi_portal_group, tpg_se_tpg);
- ssize_t len;
-
- spin_lock(&tpg->tpg_state_lock);
- len = sprintf(page, "%d\n",
- (tpg->tpg_state == TPG_STATE_ACTIVE) ? 1 : 0);
- spin_unlock(&tpg->tpg_state_lock);
-
- return len;
-}
-
-static ssize_t lio_target_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct iscsi_portal_group *tpg = container_of(se_tpg,
- struct iscsi_portal_group, tpg_se_tpg);
- u32 op;
- int ret;
-
- ret = kstrtou32(page, 0, &op);
- if (ret)
- return ret;
- if ((op != 1) && (op != 0)) {
- pr_err("Illegal value for tpg_enable: %u\n", op);
- return -EINVAL;
- }
-
- ret = iscsit_get_tpg(tpg);
- if (ret < 0)
- return -EINVAL;
-
- if (op) {
- ret = iscsit_tpg_enable_portal_group(tpg);
- if (ret < 0)
- goto out;
- } else {
- /*
- * iscsit_tpg_disable_portal_group() assumes force=1
- */
- ret = iscsit_tpg_disable_portal_group(tpg, 1);
- if (ret < 0)
- goto out;
- }
-
- iscsit_put_tpg(tpg);
- return count;
-out:
- iscsit_put_tpg(tpg);
- return -EINVAL;
-}
-
-
static ssize_t lio_target_tpg_dynamic_sessions_show(struct config_item *item,
char *page)
{
return target_show_dynamic_sessions(to_tpg(item), page);
}
-CONFIGFS_ATTR(lio_target_tpg_, enable);
CONFIGFS_ATTR_RO(lio_target_tpg_, dynamic_sessions);
static struct configfs_attribute *lio_target_tpg_attrs[] = {
- &lio_target_tpg_attr_enable,
&lio_target_tpg_attr_dynamic_sessions,
NULL,
};
return NULL;
}
+static int lio_target_tiqn_enabletpg(struct se_portal_group *se_tpg,
+ bool enable)
+{
+ struct iscsi_portal_group *tpg = container_of(se_tpg,
+ struct iscsi_portal_group, tpg_se_tpg);
+ int ret;
+
+ ret = iscsit_get_tpg(tpg);
+ if (ret < 0)
+ return -EINVAL;
+
+ if (enable) {
+ ret = iscsit_tpg_enable_portal_group(tpg);
+ if (ret < 0)
+ goto out;
+ } else {
+ /*
+ * iscsit_tpg_disable_portal_group() assumes force=1
+ */
+ ret = iscsit_tpg_disable_portal_group(tpg, 1);
+ if (ret < 0)
+ goto out;
+ }
+
+ iscsit_put_tpg(tpg);
+ return 0;
+out:
+ iscsit_put_tpg(tpg);
+ return -EINVAL;
+}
+
static void lio_target_tiqn_deltpg(struct se_portal_group *se_tpg)
{
struct iscsi_portal_group *tpg;
.fabric_drop_wwn = lio_target_call_coredeltiqn,
.add_wwn_groups = lio_target_add_wwn_groups,
.fabric_make_tpg = lio_target_tiqn_addtpg,
+ .fabric_enable_tpg = lio_target_tiqn_enabletpg,
.fabric_drop_tpg = lio_target_tiqn_deltpg,
.fabric_make_np = lio_target_call_addnptotpg,
.fabric_drop_np = lio_target_call_delnpfromtpg,
if (se_cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)
kmem_cache_free(tcm_loop_cmd_cache, tl_cmd);
else
- sc->scsi_done(sc);
+ scsi_done(sc);
}
static int tcm_loop_show_info(struct seq_file *m, struct Scsi_Host *host)
return;
out_done:
- sc->scsi_done(sc);
+ scsi_done(sc);
}
/*
return count;
}
-static ssize_t sbp_tpg_enable_show(struct config_item *item, char *page)
+static int sbp_enable_tpg(struct se_portal_group *se_tpg, bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct sbp_tpg *tpg = container_of(se_tpg, struct sbp_tpg, se_tpg);
struct sbp_tport *tport = tpg->tport;
- return sprintf(page, "%d\n", tport->enable);
-}
-
-static ssize_t sbp_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct sbp_tpg *tpg = container_of(se_tpg, struct sbp_tpg, se_tpg);
- struct sbp_tport *tport = tpg->tport;
- unsigned long val;
int ret;
- if (kstrtoul(page, 0, &val) < 0)
- return -EINVAL;
- if ((val != 0) && (val != 1))
- return -EINVAL;
-
- if (tport->enable == val)
- return count;
-
- if (val) {
+ if (enable) {
if (sbp_count_se_tpg_luns(&tpg->se_tpg) == 0) {
pr_err("Cannot enable a target with no LUNs!\n");
return -EINVAL;
spin_unlock_bh(&se_tpg->session_lock);
}
- tport->enable = val;
+ tport->enable = enable;
ret = sbp_update_unit_directory(tport);
if (ret < 0) {
return ret;
}
- return count;
+ return 0;
}
CONFIGFS_ATTR(sbp_tpg_, directory_id);
-CONFIGFS_ATTR(sbp_tpg_, enable);
static struct configfs_attribute *sbp_tpg_base_attrs[] = {
&sbp_tpg_attr_directory_id,
- &sbp_tpg_attr_enable,
NULL,
};
.fabric_make_wwn = sbp_make_tport,
.fabric_drop_wwn = sbp_drop_tport,
.fabric_make_tpg = sbp_make_tpg,
+ .fabric_enable_tpg = sbp_enable_tpg,
.fabric_drop_tpg = sbp_drop_tpg,
.fabric_post_link = sbp_post_link_lun,
.fabric_pre_unlink = sbp_pre_unlink_lun,
* this CDB was received upon to determine this value individually
* for ALUA target port group.
*/
- spin_lock(&cmd->se_lun->lun_tg_pt_gp_lock);
- tg_pt_gp = cmd->se_lun->lun_tg_pt_gp;
+ rcu_read_lock();
+ tg_pt_gp = rcu_dereference(cmd->se_lun->lun_tg_pt_gp);
if (tg_pt_gp)
buf[5] = tg_pt_gp->tg_pt_gp_implicit_trans_secs;
- spin_unlock(&cmd->se_lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
}
transport_kunmap_data_sg(cmd);
* Determine if explicit ALUA via SET_TARGET_PORT_GROUPS is allowed
* for the local tg_pt_gp.
*/
- spin_lock(&l_lun->lun_tg_pt_gp_lock);
- l_tg_pt_gp = l_lun->lun_tg_pt_gp;
+ rcu_read_lock();
+ l_tg_pt_gp = rcu_dereference(l_lun->lun_tg_pt_gp);
if (!l_tg_pt_gp) {
- spin_unlock(&l_lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
pr_err("Unable to access l_lun->tg_pt_gp\n");
rc = TCM_UNSUPPORTED_SCSI_OPCODE;
goto out;
}
if (!(l_tg_pt_gp->tg_pt_gp_alua_access_type & TPGS_EXPLICIT_ALUA)) {
- spin_unlock(&l_lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
pr_debug("Unable to process SET_TARGET_PORT_GROUPS"
" while TPGS_EXPLICIT_ALUA is disabled\n");
rc = TCM_UNSUPPORTED_SCSI_OPCODE;
goto out;
}
valid_states = l_tg_pt_gp->tg_pt_gp_alua_supported_states;
- spin_unlock(&l_lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
ptr = &buf[4]; /* Skip over RESERVED area in header */
" target port\n");
return TCM_ALUA_OFFLINE;
}
-
- if (!lun->lun_tg_pt_gp)
+ rcu_read_lock();
+ tg_pt_gp = rcu_dereference(lun->lun_tg_pt_gp);
+ if (!tg_pt_gp) {
+ rcu_read_unlock();
return 0;
+ }
- spin_lock(&lun->lun_tg_pt_gp_lock);
- tg_pt_gp = lun->lun_tg_pt_gp;
out_alua_state = tg_pt_gp->tg_pt_gp_alua_access_state;
nonop_delay_msecs = tg_pt_gp->tg_pt_gp_nonop_delay_msecs;
tg_pt_gp_id = tg_pt_gp->tg_pt_gp_id;
-
- spin_unlock(&lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
/*
* Process ALUA_ACCESS_STATE_ACTIVE_OPTIMIZED in a separate conditional
* statement so the compiler knows explicitly to check this case first.
struct t10_alua_tg_pt_gp *tg_pt_gp;
int trans_delay_msecs;
- spin_lock(&lun->lun_tg_pt_gp_lock);
- tg_pt_gp = lun->lun_tg_pt_gp;
+ rcu_read_lock();
+ tg_pt_gp = rcu_dereference(lun->lun_tg_pt_gp);
if (!tg_pt_gp) {
- spin_unlock(&lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
pr_err("Unable to complete secondary state"
" transition\n");
return -EINVAL;
"implicit", config_item_name(&tg_pt_gp->tg_pt_gp_group.cg_item),
tg_pt_gp->tg_pt_gp_id, (offline) ? "OFFLINE" : "ONLINE");
- spin_unlock(&lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
/*
* Do the optional transition delay after we set the secondary
* ALUA access state.
pr_err("Maximum ALUA alua_tg_pt_gps_count:"
" 0x0000ffff reached\n");
spin_unlock(&dev->t10_alua.tg_pt_gps_lock);
- kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
return -ENOSPC;
}
again:
__target_attach_tg_pt_gp(lun,
dev->t10_alua.default_tg_pt_gp);
} else
- lun->lun_tg_pt_gp = NULL;
+ rcu_assign_pointer(lun->lun_tg_pt_gp, NULL);
spin_unlock(&lun->lun_tg_pt_gp_lock);
spin_lock(&tg_pt_gp->tg_pt_gp_lock);
}
spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
+ synchronize_rcu();
kmem_cache_free(t10_alua_tg_pt_gp_cache, tg_pt_gp);
}
assert_spin_locked(&lun->lun_tg_pt_gp_lock);
spin_lock(&tg_pt_gp->tg_pt_gp_lock);
- lun->lun_tg_pt_gp = tg_pt_gp;
+ rcu_assign_pointer(lun->lun_tg_pt_gp, tg_pt_gp);
list_add_tail(&lun->lun_tg_pt_gp_link, &tg_pt_gp->tg_pt_gp_lun_list);
tg_pt_gp->tg_pt_gp_members++;
spin_lock(&lun->lun_deve_lock);
spin_lock(&lun->lun_tg_pt_gp_lock);
__target_attach_tg_pt_gp(lun, tg_pt_gp);
spin_unlock(&lun->lun_tg_pt_gp_lock);
+ synchronize_rcu();
}
static void __target_detach_tg_pt_gp(struct se_lun *lun,
list_del_init(&lun->lun_tg_pt_gp_link);
tg_pt_gp->tg_pt_gp_members--;
spin_unlock(&tg_pt_gp->tg_pt_gp_lock);
-
- lun->lun_tg_pt_gp = NULL;
}
void target_detach_tg_pt_gp(struct se_lun *lun)
struct t10_alua_tg_pt_gp *tg_pt_gp;
spin_lock(&lun->lun_tg_pt_gp_lock);
- tg_pt_gp = lun->lun_tg_pt_gp;
- if (tg_pt_gp)
+ tg_pt_gp = rcu_dereference_check(lun->lun_tg_pt_gp,
+ lockdep_is_held(&lun->lun_tg_pt_gp_lock));
+ if (tg_pt_gp) {
__target_detach_tg_pt_gp(lun, tg_pt_gp);
+ rcu_assign_pointer(lun->lun_tg_pt_gp, NULL);
+ }
spin_unlock(&lun->lun_tg_pt_gp_lock);
+ synchronize_rcu();
+}
+
+static void target_swap_tg_pt_gp(struct se_lun *lun,
+ struct t10_alua_tg_pt_gp *old_tg_pt_gp,
+ struct t10_alua_tg_pt_gp *new_tg_pt_gp)
+{
+ assert_spin_locked(&lun->lun_tg_pt_gp_lock);
+
+ if (old_tg_pt_gp)
+ __target_detach_tg_pt_gp(lun, old_tg_pt_gp);
+ __target_attach_tg_pt_gp(lun, new_tg_pt_gp);
}
ssize_t core_alua_show_tg_pt_gp_info(struct se_lun *lun, char *page)
struct t10_alua_tg_pt_gp *tg_pt_gp;
ssize_t len = 0;
- spin_lock(&lun->lun_tg_pt_gp_lock);
- tg_pt_gp = lun->lun_tg_pt_gp;
+ rcu_read_lock();
+ tg_pt_gp = rcu_dereference(lun->lun_tg_pt_gp);
if (tg_pt_gp) {
tg_pt_ci = &tg_pt_gp->tg_pt_gp_group.cg_item;
len += sprintf(page, "TG Port Alias: %s\nTG Port Group ID:"
"Offline" : "None",
core_alua_dump_status(lun->lun_tg_pt_secondary_stat));
}
- spin_unlock(&lun->lun_tg_pt_gp_lock);
+ rcu_read_unlock();
return len;
}
}
spin_lock(&lun->lun_tg_pt_gp_lock);
- tg_pt_gp = lun->lun_tg_pt_gp;
+ tg_pt_gp = rcu_dereference_check(lun->lun_tg_pt_gp,
+ lockdep_is_held(&lun->lun_tg_pt_gp_lock));
if (tg_pt_gp) {
/*
* Clearing an existing tg_pt_gp association, and replacing
&tg_pt_gp->tg_pt_gp_group.cg_item),
tg_pt_gp->tg_pt_gp_id);
- __target_detach_tg_pt_gp(lun, tg_pt_gp);
- __target_attach_tg_pt_gp(lun,
+ target_swap_tg_pt_gp(lun, tg_pt_gp,
dev->t10_alua.default_tg_pt_gp);
spin_unlock(&lun->lun_tg_pt_gp_lock);
- return count;
+ goto sync_rcu;
}
- __target_detach_tg_pt_gp(lun, tg_pt_gp);
move = 1;
}
- __target_attach_tg_pt_gp(lun, tg_pt_gp_new);
+ target_swap_tg_pt_gp(lun, tg_pt_gp, tg_pt_gp_new);
spin_unlock(&lun->lun_tg_pt_gp_lock);
pr_debug("Target_Core_ConfigFS: %s %s/tpgt_%hu/%s to ALUA"
" Target Port Group: alua/%s, ID: %hu\n", (move) ?
tg_pt_gp_new->tg_pt_gp_id);
core_alua_put_tg_pt_gp_from_name(tg_pt_gp_new);
+sync_rcu:
+ synchronize_rcu();
return count;
}
* fabric driver unload of TFO->module to proceed.
*/
rcu_barrier();
+ kfree(t->tf_tpg_base_cit.ct_attrs);
kfree(t);
return;
}
INIT_LIST_HEAD(&dev->t10_alua.lba_map_list);
spin_lock_init(&dev->t10_alua.lba_map_lock);
+ INIT_WORK(&dev->delayed_cmd_work, target_do_delayed_work);
+
dev->t10_wwn.t10_dev = dev;
/*
* Use OpenFabrics IEEE Company ID: 00 14 05
.release = target_fabric_tpg_release,
};
-TF_CIT_SETUP_DRV(tpg_base, &target_fabric_tpg_base_item_ops, NULL);
+static ssize_t target_fabric_tpg_base_enable_show(struct config_item *item,
+ char *page)
+{
+ return sysfs_emit(page, "%d\n", to_tpg(item)->enabled);
+}
+
+static ssize_t target_fabric_tpg_base_enable_store(struct config_item *item,
+ const char *page,
+ size_t count)
+{
+ struct se_portal_group *se_tpg = to_tpg(item);
+ int ret;
+ bool op;
+
+ ret = strtobool(page, &op);
+ if (ret)
+ return ret;
+
+ if (se_tpg->enabled == op)
+ return count;
+
+ ret = se_tpg->se_tpg_tfo->fabric_enable_tpg(se_tpg, op);
+ if (ret)
+ return ret;
+
+ se_tpg->enabled = op;
+
+ return count;
+}
+
+CONFIGFS_ATTR(target_fabric_tpg_base_, enable);
+static int
+target_fabric_setup_tpg_base_cit(struct target_fabric_configfs *tf)
+{
+ struct config_item_type *cit = &tf->tf_tpg_base_cit;
+ struct configfs_attribute **attrs = NULL;
+ size_t nr_attrs = 0;
+ int i = 0;
+
+ if (tf->tf_ops->tfc_tpg_base_attrs)
+ while (tf->tf_ops->tfc_tpg_base_attrs[nr_attrs] != NULL)
+ nr_attrs++;
+
+ if (tf->tf_ops->fabric_enable_tpg)
+ nr_attrs++;
+
+ if (nr_attrs == 0)
+ goto done;
+
+ /* + 1 for final NULL in the array */
+ attrs = kcalloc(nr_attrs + 1, sizeof(*attrs), GFP_KERNEL);
+ if (!attrs)
+ return -ENOMEM;
+
+ if (tf->tf_ops->tfc_tpg_base_attrs)
+ for (; tf->tf_ops->tfc_tpg_base_attrs[i] != NULL; i++)
+ attrs[i] = tf->tf_ops->tfc_tpg_base_attrs[i];
+
+ if (tf->tf_ops->fabric_enable_tpg)
+ attrs[i] = &target_fabric_tpg_base_attr_enable;
+
+done:
+ cit->ct_item_ops = &target_fabric_tpg_base_item_ops;
+ cit->ct_attrs = attrs;
+ cit->ct_owner = tf->tf_ops->module;
+ pr_debug("Setup generic tpg_base\n");
+
+ return 0;
+}
/* End of tfc_tpg_base_cit */
/* Start of tfc_tpg_cit */
int target_fabric_setup_cits(struct target_fabric_configfs *tf)
{
+ int ret;
+
target_fabric_setup_discovery_cit(tf);
target_fabric_setup_wwn_cit(tf);
target_fabric_setup_wwn_fabric_stats_cit(tf);
target_fabric_setup_wwn_param_cit(tf);
target_fabric_setup_tpg_cit(tf);
- target_fabric_setup_tpg_base_cit(tf);
+
+ ret = target_fabric_setup_tpg_base_cit(tf);
+ if (ret)
+ return ret;
+
target_fabric_setup_tpg_port_cit(tf);
target_fabric_setup_tpg_port_stat_cit(tf);
target_fabric_setup_tpg_lun_cit(tf);
{
struct iblock_dev *ib_dev = IBLOCK_DEV(dev);
struct block_device *bd = ib_dev->ibd_bd;
- char buf[BDEVNAME_SIZE];
ssize_t bl = 0;
if (bd)
- bl += sprintf(b + bl, "iBlock device: %s",
- bdevname(bd, buf));
+ bl += sprintf(b + bl, "iBlock device: %pg", bd);
if (ib_dev->ibd_flags & IBDF_HAS_UDEV_PATH)
bl += sprintf(b + bl, " UDEV PATH: %s",
ib_dev->ibd_udev_path);
void transport_clear_lun_ref(struct se_lun *);
sense_reason_t target_cmd_size_check(struct se_cmd *cmd, unsigned int size);
void target_qf_do_work(struct work_struct *work);
+void target_do_delayed_work(struct work_struct *work);
bool target_check_wce(struct se_device *dev);
bool target_check_fua(struct se_device *dev);
void __target_execute_cmd(struct se_cmd *, bool);
void core_tmr_release_req(struct se_tmr_req *tmr)
{
- struct se_device *dev = tmr->tmr_dev;
- unsigned long flags;
-
- if (dev) {
- spin_lock_irqsave(&dev->se_tmr_lock, flags);
- list_del_init(&tmr->tmr_list);
- spin_unlock_irqrestore(&dev->se_tmr_lock, flags);
- }
-
kfree(tmr);
}
se_cmd->state_active = false;
spin_unlock_irqrestore(&dev->queues[i].lock, flags);
- /*
- * Ensure that this ABORT request is visible to the LU
- * RESET code.
- */
- if (!tmr->tmr_dev)
- WARN_ON_ONCE(transport_lookup_tmr_lun(tmr->task_cmd) < 0);
-
if (dev->transport->tmr_notify)
dev->transport->tmr_notify(dev, TMR_ABORT_TASK,
&aborted_list);
}
list_move_tail(&tmr_p->tmr_list, &drain_tmr_list);
+ tmr_p->tmr_dev = NULL;
}
spin_unlock_irqrestore(&dev->se_tmr_lock, flags);
spin_unlock_irqrestore(&dev->queues[cmd->cpuid].lock, flags);
}
+static void target_remove_from_tmr_list(struct se_cmd *cmd)
+{
+ struct se_device *dev = NULL;
+ unsigned long flags;
+
+ if (cmd->se_cmd_flags & SCF_SCSI_TMR_CDB)
+ dev = cmd->se_tmr_req->tmr_dev;
+
+ if (dev) {
+ spin_lock_irqsave(&dev->se_tmr_lock, flags);
+ if (cmd->se_tmr_req->tmr_dev)
+ list_del_init(&cmd->se_tmr_req->tmr_list);
+ spin_unlock_irqrestore(&dev->se_tmr_lock, flags);
+ }
+}
/*
* This function is called by the target core after the target core has
* finished processing a SCSI command or SCSI TMF. Both the regular command
{
unsigned long flags;
- target_remove_from_state_list(cmd);
-
- /*
- * Clear struct se_cmd->se_lun before the handoff to FE.
- */
- cmd->se_lun = NULL;
-
spin_lock_irqsave(&cmd->t_state_lock, flags);
/*
* Determine if frontend context caller is requesting the stopping of
if (!lun)
return;
+ target_remove_from_state_list(cmd);
+ target_remove_from_tmr_list(cmd);
+
if (cmpxchg(&cmd->lun_ref_active, true, false))
percpu_ref_put(&lun->lun_ref);
+
+ /*
+ * Clear struct se_cmd->se_lun before the handoff to FE.
+ */
+ cmd->se_lun = NULL;
}
static void target_complete_failure_work(struct work_struct *work)
ret = dev->transport->parse_cdb(cmd);
if (ret == TCM_UNSUPPORTED_SCSI_OPCODE)
- pr_warn_ratelimited("%s/%s: Unsupported SCSI Opcode 0x%02x, sending CHECK_CONDITION.\n",
- cmd->se_tfo->fabric_name,
- cmd->se_sess->se_node_acl->initiatorname,
- cmd->t_task_cdb[0]);
+ pr_debug_ratelimited("%s/%s: Unsupported SCSI Opcode 0x%02x, sending CHECK_CONDITION.\n",
+ cmd->se_tfo->fabric_name,
+ cmd->se_sess->se_node_acl->initiatorname,
+ cmd->t_task_cdb[0]);
if (ret)
return ret;
*/
switch (cmd->sam_task_attr) {
case TCM_HEAD_TAG:
+ atomic_inc_mb(&dev->non_ordered);
pr_debug("Added HEAD_OF_QUEUE for CDB: 0x%02x\n",
cmd->t_task_cdb[0]);
return false;
case TCM_ORDERED_TAG:
- atomic_inc_mb(&dev->dev_ordered_sync);
+ atomic_inc_mb(&dev->delayed_cmd_count);
pr_debug("Added ORDERED for CDB: 0x%02x to ordered list\n",
cmd->t_task_cdb[0]);
-
- /*
- * Execute an ORDERED command if no other older commands
- * exist that need to be completed first.
- */
- if (!atomic_read(&dev->simple_cmds))
- return false;
break;
default:
/*
* For SIMPLE and UNTAGGED Task Attribute commands
*/
- atomic_inc_mb(&dev->simple_cmds);
+ atomic_inc_mb(&dev->non_ordered);
+
+ if (atomic_read(&dev->delayed_cmd_count) == 0)
+ return false;
break;
}
- if (atomic_read(&dev->dev_ordered_sync) == 0)
- return false;
+ if (cmd->sam_task_attr != TCM_ORDERED_TAG) {
+ atomic_inc_mb(&dev->delayed_cmd_count);
+ /*
+ * We will account for this when we dequeue from the delayed
+ * list.
+ */
+ atomic_dec_mb(&dev->non_ordered);
+ }
+
+ spin_lock_irq(&cmd->t_state_lock);
+ cmd->transport_state &= ~CMD_T_SENT;
+ spin_unlock_irq(&cmd->t_state_lock);
spin_lock(&dev->delayed_cmd_lock);
list_add_tail(&cmd->se_delayed_node, &dev->delayed_cmd_list);
pr_debug("Added CDB: 0x%02x Task Attr: 0x%02x to delayed CMD listn",
cmd->t_task_cdb[0], cmd->sam_task_attr);
+ /*
+ * We may have no non ordered cmds when this function started or we
+ * could have raced with the last simple/head cmd completing, so kick
+ * the delayed handler here.
+ */
+ schedule_work(&dev->delayed_cmd_work);
return true;
}
if (target_write_prot_action(cmd))
return;
- if (target_handle_task_attr(cmd)) {
- spin_lock_irq(&cmd->t_state_lock);
- cmd->transport_state &= ~CMD_T_SENT;
- spin_unlock_irq(&cmd->t_state_lock);
+ if (target_handle_task_attr(cmd))
return;
- }
__target_execute_cmd(cmd, true);
}
* Process all commands up to the last received ORDERED task attribute which
* requires another blocking boundary
*/
-static void target_restart_delayed_cmds(struct se_device *dev)
+void target_do_delayed_work(struct work_struct *work)
{
- for (;;) {
+ struct se_device *dev = container_of(work, struct se_device,
+ delayed_cmd_work);
+
+ spin_lock(&dev->delayed_cmd_lock);
+ while (!dev->ordered_sync_in_progress) {
struct se_cmd *cmd;
- spin_lock(&dev->delayed_cmd_lock);
- if (list_empty(&dev->delayed_cmd_list)) {
- spin_unlock(&dev->delayed_cmd_lock);
+ if (list_empty(&dev->delayed_cmd_list))
break;
- }
cmd = list_entry(dev->delayed_cmd_list.next,
struct se_cmd, se_delayed_node);
+
+ if (cmd->sam_task_attr == TCM_ORDERED_TAG) {
+ /*
+ * Check if we started with:
+ * [ordered] [simple] [ordered]
+ * and we are now at the last ordered so we have to wait
+ * for the simple cmd.
+ */
+ if (atomic_read(&dev->non_ordered) > 0)
+ break;
+
+ dev->ordered_sync_in_progress = true;
+ }
+
list_del(&cmd->se_delayed_node);
+ atomic_dec_mb(&dev->delayed_cmd_count);
spin_unlock(&dev->delayed_cmd_lock);
+ if (cmd->sam_task_attr != TCM_ORDERED_TAG)
+ atomic_inc_mb(&dev->non_ordered);
+
cmd->transport_state |= CMD_T_SENT;
__target_execute_cmd(cmd, true);
- if (cmd->sam_task_attr == TCM_ORDERED_TAG)
- break;
+ spin_lock(&dev->delayed_cmd_lock);
}
+ spin_unlock(&dev->delayed_cmd_lock);
}
/*
goto restart;
if (cmd->sam_task_attr == TCM_SIMPLE_TAG) {
- atomic_dec_mb(&dev->simple_cmds);
+ atomic_dec_mb(&dev->non_ordered);
dev->dev_cur_ordered_id++;
} else if (cmd->sam_task_attr == TCM_HEAD_TAG) {
+ atomic_dec_mb(&dev->non_ordered);
dev->dev_cur_ordered_id++;
pr_debug("Incremented dev_cur_ordered_id: %u for HEAD_OF_QUEUE\n",
dev->dev_cur_ordered_id);
} else if (cmd->sam_task_attr == TCM_ORDERED_TAG) {
- atomic_dec_mb(&dev->dev_ordered_sync);
+ spin_lock(&dev->delayed_cmd_lock);
+ dev->ordered_sync_in_progress = false;
+ spin_unlock(&dev->delayed_cmd_lock);
dev->dev_cur_ordered_id++;
pr_debug("Incremented dev_cur_ordered_id: %u for ORDERED\n",
cmd->se_cmd_flags &= ~SCF_TASK_ATTR_SET;
restart:
- target_restart_delayed_cmds(dev);
+ if (atomic_read(&dev->delayed_cmd_count) > 0)
+ schedule_work(&dev->delayed_cmd_work);
}
static void transport_complete_qf(struct se_cmd *cmd)
rcu_read_unlock();
for (i = cnt; i < page_cnt; i++) {
- /* try to get new page from the mm */
- page = alloc_page(GFP_NOIO);
+ /* try to get new zeroed page from the mm */
+ page = alloc_page(GFP_NOIO | __GFP_ZERO);
if (!page)
break;
{
int i = 0, cmd_cnt = 0;
bool unqueued = false;
- uint16_t *cmd_ids = NULL;
struct tcmu_cmd *cmd;
struct se_cmd *se_cmd;
struct tcmu_tmr *tmr;
pr_debug("TMR event %d on dev %s, aborted cmds %d, afflicted cmd_ids %d\n",
tcmu_tmr_type(tmf), udev->name, i, cmd_cnt);
- tmr = kmalloc(sizeof(*tmr) + cmd_cnt * sizeof(*cmd_ids), GFP_NOIO);
+ tmr = kmalloc(struct_size(tmr, tmr_cmd_ids, cmd_cnt), GFP_NOIO);
if (!tmr)
goto unlock;
return -EINVAL;
}
-static int target_xcopy_parse_segdesc_02(struct se_cmd *se_cmd, struct xcopy_op *xop,
- unsigned char *p)
+static int target_xcopy_parse_segdesc_02(struct xcopy_op *xop, unsigned char *p)
{
unsigned char *desc = p;
int dc = (desc[1] & 0x02);
return 0;
}
-static int target_xcopy_parse_segment_descriptors(struct se_cmd *se_cmd,
- struct xcopy_op *xop, unsigned char *p,
- unsigned int sdll, sense_reason_t *sense_ret)
+static int target_xcopy_parse_segment_descriptors(struct xcopy_op *xop,
+ unsigned char *p, unsigned int sdll,
+ sense_reason_t *sense_ret)
{
unsigned char *desc = p;
unsigned int start = 0;
*/
switch (desc[0]) {
case 0x02:
- rc = target_xcopy_parse_segdesc_02(se_cmd, xop, desc);
+ rc = target_xcopy_parse_segdesc_02(xop, desc);
if (rc < 0)
goto out;
*/
seg_desc = &p[16] + tdll;
- rc = target_xcopy_parse_segment_descriptors(se_cmd, xop, seg_desc,
- sdll, &ret);
+ rc = target_xcopy_parse_segment_descriptors(xop, seg_desc, sdll, &ret);
if (rc <= 0)
goto out;
static int user_space_bind(struct thermal_zone_device *tz)
{
- pr_warn("Userspace governor deprecated: use thermal netlink " \
- "notification instead\n");
+ pr_warn_once("Userspace governor deprecated: use thermal netlink " \
+ "notification instead\n");
return 0;
}
config INT340X_THERMAL
tristate "ACPI INT340X thermal drivers"
- depends on X86 && ACPI && PCI
+ depends on X86_64 && ACPI && PCI
select THERMAL_GOV_USER_SPACE
select ACPI_THERMAL_REL
select ACPI_FAN
select INTEL_SOC_DTS_IOSF_CORE
- select PROC_THERMAL_MMIO_RAPL if X86_64 && POWERCAP
+ select PROC_THERMAL_MMIO_RAPL if POWERCAP
help
Newer laptops and tablets that use ACPI may have thermal sensors and
other devices with thermal control capabilities outside the core
int proc_thermal_mbox_add(struct pci_dev *pdev, struct proc_thermal_device *proc_priv);
void proc_thermal_mbox_remove(struct pci_dev *pdev);
-int processor_thermal_send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u32 *cmd_resp);
+int processor_thermal_send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u64 *cmd_resp);
int proc_thermal_add(struct device *dev, struct proc_thermal_device *priv);
void proc_thermal_remove(struct proc_thermal_device *proc_priv);
int proc_thermal_suspend(struct device *dev);
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/pci.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include "processor_thermal_device.h"
#define MBOX_CMD_WORKLOAD_TYPE_READ 0x0E
static DEFINE_MUTEX(mbox_lock);
-static int send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u32 *cmd_resp)
+static int send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u64 *cmd_resp)
{
struct proc_thermal_device *proc_priv;
u32 retries, data;
goto unlock_mbox;
}
- if (cmd_id == MBOX_CMD_WORKLOAD_TYPE_READ) {
- data = readl((void __iomem *) (proc_priv->mmio_base + MBOX_OFFSET_DATA));
- *cmd_resp = data & 0xff;
- }
-
ret = 0;
+
+ if (!cmd_resp)
+ break;
+
+ if (cmd_id == MBOX_CMD_WORKLOAD_TYPE_READ)
+ *cmd_resp = readl((void __iomem *) (proc_priv->mmio_base + MBOX_OFFSET_DATA));
+ else
+ *cmd_resp = readq((void __iomem *) (proc_priv->mmio_base + MBOX_OFFSET_DATA));
+
break;
} while (--retries);
return ret;
}
-int processor_thermal_send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u32 *cmd_resp)
+int processor_thermal_send_mbox_cmd(struct pci_dev *pdev, u16 cmd_id, u32 cmd_data, u64 *cmd_resp)
{
return send_mbox_cmd(pdev, cmd_id, cmd_data, cmd_resp);
}
char *buf)
{
struct pci_dev *pdev = to_pci_dev(dev);
- u32 cmd_resp;
+ u64 cmd_resp;
int ret;
ret = send_mbox_cmd(pdev, MBOX_CMD_WORKLOAD_TYPE_READ, 0, &cmd_resp);
int proc_thermal_mbox_add(struct pci_dev *pdev, struct proc_thermal_device *proc_priv)
{
- u32 cmd_resp;
+ u64 cmd_resp;
int ret;
/* Check if there is a mailbox support, if fails return success */
const char *buf, size_t count)
{
u16 cmd_id = 0x0008;
- u32 cmd_resp;
+ u64 cmd_resp;
u32 input;
int ret;
char *buf)
{
u16 cmd_id = 0x0007;
- u32 cmd_resp;
+ u64 cmd_resp;
int ret;
ret = processor_thermal_send_mbox_cmd(to_pci_dev(dev), cmd_id, 0, &cmd_resp);
if (ret)
return ret;
- return sprintf(buf, "%u\n", cmd_resp);
+ return sprintf(buf, "%llu\n", cmd_resp);
}
static ssize_t ddr_data_rate_show(struct device *dev,
char *buf)
{
u16 cmd_id = 0x0107;
- u32 cmd_resp;
+ u64 cmd_resp;
int ret;
ret = processor_thermal_send_mbox_cmd(to_pci_dev(dev), cmd_id, 0, &cmd_resp);
if (ret)
return ret;
- return sprintf(buf, "%u\n", cmd_resp);
+ return sprintf(buf, "%llu\n", cmd_resp);
}
static DEVICE_ATTR_RW(rfi_restriction);
{
struct thermal_instance *pos;
tz->temperature = THERMAL_TEMP_INVALID;
+ tz->prev_low_trip = -INT_MAX;
+ tz->prev_high_trip = INT_MAX;
list_for_each_entry(pos, &tz->thermal_instances, tz_node)
pos->initialized = false;
}
{
struct __thermal_zone *data = tz->devdata;
- if (!data->ops->get_temp)
+ if (!data->ops || !data->ops->get_temp)
return -EINVAL;
return data->ops->get_temp(data->sensor_data, temp);
{
struct __thermal_zone *data = tz->devdata;
+ if (!data->ops || !data->ops->set_emul_temp)
+ return -EINVAL;
+
return data->ops->set_emul_temp(data->sensor_data, temp);
}
{
struct __thermal_zone *data = tz->devdata;
- if (!data->ops->get_trend)
+ if (!data->ops || !data->ops->get_trend)
return -EINVAL;
return data->ops->get_trend(data->sensor_data, trip, trend);
if (trip >= data->ntrips || trip < 0)
return -EDOM;
- if (data->ops->set_trip_temp) {
+ if (data->ops && data->ops->set_trip_temp) {
int ret;
ret = data->ops->set_trip_temp(data->sensor_data, trip, temp);
{
int retval;
struct usb_device *rhdev;
- struct usb_hcd *shared_hcd;
if (!hcd->skip_phy_initialization && usb_hcd_is_primary_hcd(hcd)) {
hcd->phy_roothub = usb_phy_roothub_alloc(hcd->self.sysdev);
goto err_hcd_driver_start;
}
- /* starting here, usbcore will pay attention to the shared HCD roothub */
- shared_hcd = hcd->shared_hcd;
- if (!usb_hcd_is_primary_hcd(hcd) && shared_hcd && HCD_DEFER_RH_REGISTER(shared_hcd)) {
- retval = register_root_hub(shared_hcd);
- if (retval != 0)
- goto err_register_root_hub;
-
- if (shared_hcd->uses_new_polling && HCD_POLL_RH(shared_hcd))
- usb_hcd_poll_rh_status(shared_hcd);
- }
-
/* starting here, usbcore will pay attention to this root hub */
- if (!HCD_DEFER_RH_REGISTER(hcd)) {
- retval = register_root_hub(hcd);
- if (retval != 0)
- goto err_register_root_hub;
+ retval = register_root_hub(hcd);
+ if (retval != 0)
+ goto err_register_root_hub;
- if (hcd->uses_new_polling && HCD_POLL_RH(hcd))
- usb_hcd_poll_rh_status(hcd);
- }
+ if (hcd->uses_new_polling && HCD_POLL_RH(hcd))
+ usb_hcd_poll_rh_status(hcd);
return retval;
void usb_remove_hcd(struct usb_hcd *hcd)
{
struct usb_device *rhdev = hcd->self.root_hub;
- bool rh_registered;
dev_info(hcd->self.controller, "remove, state %x\n", hcd->state);
dev_dbg(hcd->self.controller, "roothub graceful disconnect\n");
spin_lock_irq (&hcd_root_hub_lock);
- rh_registered = hcd->rh_registered;
hcd->rh_registered = 0;
spin_unlock_irq (&hcd_root_hub_lock);
cancel_work_sync(&hcd->died_work);
mutex_lock(&usb_bus_idr_lock);
- if (rh_registered)
- usb_disconnect(&rhdev); /* Sets rhdev to NULL */
+ usb_disconnect(&rhdev); /* Sets rhdev to NULL */
mutex_unlock(&usb_bus_idr_lock);
/*
if (!seg)
return;
- memblock_free(seg->dma, PAGE_SIZE);
+ memblock_phys_free(seg->dma, PAGE_SIZE);
ring->segment = NULL;
}
xdbc_free_ring(&xdbc.in_ring);
if (xdbc.table_dma)
- memblock_free(xdbc.table_dma, PAGE_SIZE);
+ memblock_phys_free(xdbc.table_dma, PAGE_SIZE);
if (xdbc.out_dma)
- memblock_free(xdbc.out_dma, PAGE_SIZE);
+ memblock_phys_free(xdbc.out_dma, PAGE_SIZE);
xdbc.table_base = NULL;
xdbc.out_buf = NULL;
xdbc_free_ring(&xdbc.evt_ring);
xdbc_free_ring(&xdbc.out_ring);
xdbc_free_ring(&xdbc.in_ring);
- memblock_free(xdbc.table_dma, PAGE_SIZE);
- memblock_free(xdbc.out_dma, PAGE_SIZE);
+ memblock_phys_free(xdbc.table_dma, PAGE_SIZE);
+ memblock_phys_free(xdbc.out_dma, PAGE_SIZE);
writel(0, &xdbc.xdbc_reg->control);
early_iounmap(xdbc.xhci_base, xdbc.xhci_length);
NULL,
};
-static ssize_t tcm_usbg_tpg_enable_show(struct config_item *item, char *page)
-{
- struct se_portal_group *se_tpg = to_tpg(item);
- struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);
-
- return snprintf(page, PAGE_SIZE, "%u\n", tpg->gadget_connect);
-}
-
static int usbg_attach(struct usbg_tpg *);
static void usbg_detach(struct usbg_tpg *);
-static ssize_t tcm_usbg_tpg_enable_store(struct config_item *item,
- const char *page, size_t count)
+static int usbg_enable_tpg(struct se_portal_group *se_tpg, bool enable)
{
- struct se_portal_group *se_tpg = to_tpg(item);
struct usbg_tpg *tpg = container_of(se_tpg, struct usbg_tpg, se_tpg);
- bool op;
- ssize_t ret;
-
- ret = strtobool(page, &op);
- if (ret)
- return ret;
-
- if ((op && tpg->gadget_connect) || (!op && !tpg->gadget_connect))
- return -EINVAL;
+ int ret = 0;
- if (op)
+ if (enable)
ret = usbg_attach(tpg);
else
usbg_detach(tpg);
if (ret)
return ret;
- tpg->gadget_connect = op;
+ tpg->gadget_connect = enable;
- return count;
+ return 0;
}
static ssize_t tcm_usbg_tpg_nexus_show(struct config_item *item, char *page)
return count;
}
-CONFIGFS_ATTR(tcm_usbg_tpg_, enable);
CONFIGFS_ATTR(tcm_usbg_tpg_, nexus);
static struct configfs_attribute *usbg_base_attrs[] = {
- &tcm_usbg_tpg_attr_enable,
&tcm_usbg_tpg_attr_nexus,
NULL,
};
.fabric_make_wwn = usbg_make_tport,
.fabric_drop_wwn = usbg_drop_tport,
.fabric_make_tpg = usbg_make_tpg,
+ .fabric_enable_tpg = usbg_enable_tpg,
.fabric_drop_tpg = usbg_drop_tpg,
.fabric_post_link = usbg_port_link,
.fabric_pre_unlink = usbg_port_unlink,
{
u16 temp;
- desc->bPwrOn2PwrGood = 10; /* xhci section 5.4.9 says 20ms max */
desc->bHubContrCurrent = 0;
desc->bNbrPorts = ports;
desc->bDescriptorType = USB_DT_HUB;
temp = 1 + (ports / 8);
desc->bDescLength = USB_DT_HUB_NONVAR_SIZE + 2 * temp;
+ desc->bPwrOn2PwrGood = 10; /* xhci section 5.4.8 says 20ms */
/* The Device Removable bits are reported on a byte granularity.
* If the port doesn't exist within that byte, the bit is set to 0.
xhci_common_hub_descriptor(xhci, desc, ports);
desc->bDescriptorType = USB_DT_SS_HUB;
desc->bDescLength = USB_DT_SS_HUB_SIZE;
+ desc->bPwrOn2PwrGood = 50; /* usb 3.1 may fail if less than 100ms */
/* header decode latency should be zero for roothubs,
* see section 4.23.5.2.
struct xhci_driver_data *driver_data;
const struct pci_device_id *id;
- id = pci_match_id(pdev->driver->id_table, pdev);
+ id = pci_match_id(to_pci_driver(pdev->dev.driver)->id_table, pdev);
if (id && id->driver_data) {
driver_data = (struct xhci_driver_data *)id->driver_data;
if (ret)
xhci_free_command(xhci, command);
}
- set_bit(HCD_FLAG_DEFER_RH_REGISTER, &hcd->flags);
xhci_dbg_trace(xhci, trace_xhci_dbg_init,
"Finished xhci_run for USB2 roothub");
desc->context.data_pipe = pipe;
}
-
-static int
-mts_scsi_queuecommand_lck(struct scsi_cmnd *srb, mts_scsi_cmnd_callback callback)
+static int mts_scsi_queuecommand_lck(struct scsi_cmnd *srb)
{
+ mts_scsi_cmnd_callback callback = scsi_done;
struct mts_desc* desc = (struct mts_desc*)(srb->device->host->hostdata[0]);
int res;
/* queue a command */
/* This is always called with scsi_lock(host) held */
-static int queuecommand_lck(struct scsi_cmnd *srb,
- void (*done)(struct scsi_cmnd *))
+static int queuecommand_lck(struct scsi_cmnd *srb)
{
+ void (*done)(struct scsi_cmnd *) = scsi_done;
struct us_data *us = host_to_us(srb->device->host);
/* check for state-transition errors */
}
/* enqueue the command and wake up the control thread */
- srb->scsi_done = done;
us->srb = srb;
complete(&us->cmnd_ready);
}
static DEVICE_ATTR_RW(max_sectors);
-static struct device_attribute *sysfs_device_attr_list[] = {
- &dev_attr_max_sectors,
+static struct attribute *usb_sdev_attrs[] = {
+ &dev_attr_max_sectors.attr,
NULL,
};
+ATTRIBUTE_GROUPS(usb_sdev);
+
/*
* this defines our host template, with which we'll allocate hosts
*/
.skip_settle_delay = 1,
/* sysfs device attributes */
- .sdev_attrs = sysfs_device_attr_list,
+ .sdev_groups = usb_sdev_groups,
/* module management */
.module = THIS_MODULE
return -EBUSY;
devinfo->cmnd[cmdinfo->uas_tag - 1] = NULL;
uas_free_unsubmitted_urbs(cmnd);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
return 0;
}
-static int uas_queuecommand_lck(struct scsi_cmnd *cmnd,
- void (*done)(struct scsi_cmnd *))
+static int uas_queuecommand_lck(struct scsi_cmnd *cmnd)
{
struct scsi_device *sdev = cmnd->device;
struct uas_dev_info *devinfo = sdev->hostdata;
memcpy(cmnd->sense_buffer, usb_stor_sense_invalidCDB,
sizeof(usb_stor_sense_invalidCDB));
cmnd->result = SAM_STAT_CHECK_CONDITION;
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
return 0;
}
if (devinfo->resetting) {
set_host_byte(cmnd, DID_ERROR);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
goto zombie;
}
return SCSI_MLQUEUE_DEVICE_BUSY;
}
- cmnd->scsi_done = done;
-
memset(cmdinfo, 0, sizeof(*cmdinfo));
cmdinfo->uas_tag = idx + 1; /* uas-tag == usb-stream-id, so 1 based */
cmdinfo->state = SUBMIT_STATUS_URB | ALLOC_CMD_URB | SUBMIT_CMD_URB;
*/
if (err == -ENODEV) {
set_host_byte(cmnd, DID_ERROR);
- cmnd->scsi_done(cmnd);
+ scsi_done(cmnd);
goto zombie;
}
if (err) {
if (srb->result == DID_ABORT << 16) {
SkipForAbort:
usb_stor_dbg(us, "scsi command aborted\n");
- srb = NULL; /* Don't call srb->scsi_done() */
+ srb = NULL; /* Don't call scsi_done() */
}
/*
if (srb) {
usb_stor_dbg(us, "scsi cmd done, result=0x%x\n",
srb->result);
- srb->scsi_done(srb);
+ scsi_done(srb);
}
} /* for (;;) */
struct backlight_device *bd = to_backlight_device(dev);
mutex_lock(&bd->ops_lock);
- if (bd->ops && bd->ops->get_brightness)
- rc = sprintf(buf, "%d\n", bd->ops->get_brightness(bd));
- else
+ if (bd->ops && bd->ops->get_brightness) {
+ rc = bd->ops->get_brightness(bd);
+ if (rc >= 0)
+ rc = sprintf(buf, "%d\n", rc);
+ } else {
rc = sprintf(buf, "%d\n", bd->props.brightness);
+ }
mutex_unlock(&bd->ops_lock);
return rc;
void backlight_force_update(struct backlight_device *bd,
enum backlight_update_reason reason)
{
+ int brightness;
+
mutex_lock(&bd->ops_lock);
- if (bd->ops && bd->ops->get_brightness)
- bd->props.brightness = bd->ops->get_brightness(bd);
+ if (bd->ops && bd->ops->get_brightness) {
+ brightness = bd->ops->get_brightness(bd);
+ if (brightness >= 0)
+ bd->props.brightness = brightness;
+ else
+ dev_err(&bd->dev,
+ "Could not update brightness from device: %pe\n",
+ ERR_PTR(brightness));
+ }
mutex_unlock(&bd->ops_lock);
backlight_generate_event(bd, reason);
}
of_node_put(np);
if (!bd)
return ERR_PTR(-EPROBE_DEFER);
- /*
- * Note: gpio_backlight uses brightness as
- * power state during probe
- */
- if (!bd->props.brightness)
- bd->props.brightness = bd->props.max_brightness;
}
}
}
EXPORT_SYMBOL_GPL(ili9320_probe_spi);
-int ili9320_remove(struct ili9320 *ili)
+void ili9320_remove(struct ili9320 *ili)
{
ili9320_power(ili, FB_BLANK_POWERDOWN);
- return 0;
}
EXPORT_SYMBOL_GPL(ili9320_remove);
extern int ili9320_probe_spi(struct spi_device *spi,
struct ili9320_client *cli);
-extern int ili9320_remove(struct ili9320 *lcd);
+extern void ili9320_remove(struct ili9320 *lcd);
extern void ili9320_shutdown(struct ili9320 *lcd);
/* PM */
static int vgg2432a4_remove(struct spi_device *spi)
{
- return ili9320_remove(spi_get_drvdata(spi));
+ ili9320_remove(spi_get_drvdata(spi));
+
+ return 0;
}
static void vgg2432a4_shutdown(struct spi_device *spi)
bool blink, bool underline, bool reverse,
bool italic)
{
- u8 attr = ((color & 0x70) >> 1) | ((color & 7));
+ u8 fg = color & 7;
+ u8 bg = (color & 0x70) >> 4;
- if (reverse) {
- color = ((color >> 3) & 0x7) | ((color & 0x7) << 3);
- }
-
- return attr;
+ if (reverse)
+ return (fg << 3) | bg;
+ else
+ return (bg << 3) | fg;
}
static void sticon_invert_region(struct vc_data *conp, u16 *p, int count)
}
}
-static void bit_bmove(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width)
-{
- struct fb_copyarea area;
-
- area.sx = sx * vc->vc_font.width;
- area.sy = sy * vc->vc_font.height;
- area.dx = dx * vc->vc_font.width;
- area.dy = dy * vc->vc_font.height;
- area.height = height * vc->vc_font.height;
- area.width = width * vc->vc_font.width;
-
- info->fbops->fb_copyarea(info, &area);
-}
-
static void bit_clear(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width)
{
void fbcon_set_bitops(struct fbcon_ops *ops)
{
- ops->bmove = bit_bmove;
ops->clear = bit_clear;
ops->putcs = bit_putcs;
ops->clear_margins = bit_clear_margins;
int count, int ypos, int xpos);
static void fbcon_clear_margins(struct vc_data *vc, int bottom_only);
static void fbcon_cursor(struct vc_data *vc, int mode);
-static void fbcon_bmove(struct vc_data *vc, int sy, int sx, int dy, int dx,
- int height, int width);
static int fbcon_switch(struct vc_data *vc);
static int fbcon_blank(struct vc_data *vc, int blank, int mode_switch);
static void fbcon_set_palette(struct vc_data *vc, const unsigned char *table);
/*
* Internal routines
*/
-static __inline__ void ywrap_up(struct vc_data *vc, int count);
-static __inline__ void ywrap_down(struct vc_data *vc, int count);
-static __inline__ void ypan_up(struct vc_data *vc, int count);
-static __inline__ void ypan_down(struct vc_data *vc, int count);
-static void fbcon_bmove_rec(struct vc_data *vc, struct fbcon_display *p, int sy, int sx,
- int dy, int dx, int height, int width, u_int y_break);
static void fbcon_set_disp(struct fb_info *info, struct fb_var_screeninfo *var,
int unit);
-static void fbcon_redraw_move(struct vc_data *vc, struct fbcon_display *p,
- int line, int count, int dy);
static void fbcon_modechanged(struct fb_info *info);
static void fbcon_set_all_vcs(struct fb_info *info);
static void fbcon_start(void);
ops->graphics = 0;
/*
- * No more hw acceleration for fbcon.
- *
- * FIXME: Garbage collect all the now dead code after sufficient time
- * has passed.
- */
- p->scrollmode = SCROLL_REDRAW;
-
- /*
* ++guenther: console.c:vc_allocate() relies on initializing
* vc_{cols,rows}, but we must not set those if we are only
* resizing the console.
* This system is now divided into two levels because of complications
* caused by hardware scrolling. Top level functions:
*
- * fbcon_bmove(), fbcon_clear(), fbcon_putc(), fbcon_clear_margins()
+ * fbcon_clear(), fbcon_putc(), fbcon_clear_margins()
*
* handles y values in range [0, scr_height-1] that correspond to real
* screen positions. y_wrap shift means that first line of bitmap may be
* anywhere on this display. These functions convert lineoffsets to
* bitmap offsets and deal with the wrap-around case by splitting blits.
*
- * fbcon_bmove_physical_8() -- These functions fast implementations
* fbcon_clear_physical_8() -- of original fbcon_XXX fns.
* fbcon_putc_physical_8() -- (font width != 8) may be added later
*
}
}
-static __inline__ void ywrap_up(struct vc_data *vc, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_ops *ops = info->fbcon_par;
- struct fbcon_display *p = &fb_display[vc->vc_num];
-
- p->yscroll += count;
- if (p->yscroll >= p->vrows) /* Deal with wrap */
- p->yscroll -= p->vrows;
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode |= FB_VMODE_YWRAP;
- ops->update_start(info);
- scrollback_max += count;
- if (scrollback_max > scrollback_phys_max)
- scrollback_max = scrollback_phys_max;
- scrollback_current = 0;
-}
-
-static __inline__ void ywrap_down(struct vc_data *vc, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_ops *ops = info->fbcon_par;
- struct fbcon_display *p = &fb_display[vc->vc_num];
-
- p->yscroll -= count;
- if (p->yscroll < 0) /* Deal with wrap */
- p->yscroll += p->vrows;
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode |= FB_VMODE_YWRAP;
- ops->update_start(info);
- scrollback_max -= count;
- if (scrollback_max < 0)
- scrollback_max = 0;
- scrollback_current = 0;
-}
-
-static __inline__ void ypan_up(struct vc_data *vc, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_display *p = &fb_display[vc->vc_num];
- struct fbcon_ops *ops = info->fbcon_par;
-
- p->yscroll += count;
- if (p->yscroll > p->vrows - vc->vc_rows) {
- ops->bmove(vc, info, p->vrows - vc->vc_rows,
- 0, 0, 0, vc->vc_rows, vc->vc_cols);
- p->yscroll -= p->vrows - vc->vc_rows;
- }
-
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode &= ~FB_VMODE_YWRAP;
- ops->update_start(info);
- fbcon_clear_margins(vc, 1);
- scrollback_max += count;
- if (scrollback_max > scrollback_phys_max)
- scrollback_max = scrollback_phys_max;
- scrollback_current = 0;
-}
-
-static __inline__ void ypan_up_redraw(struct vc_data *vc, int t, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_ops *ops = info->fbcon_par;
- struct fbcon_display *p = &fb_display[vc->vc_num];
-
- p->yscroll += count;
-
- if (p->yscroll > p->vrows - vc->vc_rows) {
- p->yscroll -= p->vrows - vc->vc_rows;
- fbcon_redraw_move(vc, p, t + count, vc->vc_rows - count, t);
- }
-
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode &= ~FB_VMODE_YWRAP;
- ops->update_start(info);
- fbcon_clear_margins(vc, 1);
- scrollback_max += count;
- if (scrollback_max > scrollback_phys_max)
- scrollback_max = scrollback_phys_max;
- scrollback_current = 0;
-}
-
-static __inline__ void ypan_down(struct vc_data *vc, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_display *p = &fb_display[vc->vc_num];
- struct fbcon_ops *ops = info->fbcon_par;
-
- p->yscroll -= count;
- if (p->yscroll < 0) {
- ops->bmove(vc, info, 0, 0, p->vrows - vc->vc_rows,
- 0, vc->vc_rows, vc->vc_cols);
- p->yscroll += p->vrows - vc->vc_rows;
- }
-
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode &= ~FB_VMODE_YWRAP;
- ops->update_start(info);
- fbcon_clear_margins(vc, 1);
- scrollback_max -= count;
- if (scrollback_max < 0)
- scrollback_max = 0;
- scrollback_current = 0;
-}
-
-static __inline__ void ypan_down_redraw(struct vc_data *vc, int t, int count)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_ops *ops = info->fbcon_par;
- struct fbcon_display *p = &fb_display[vc->vc_num];
-
- p->yscroll -= count;
-
- if (p->yscroll < 0) {
- p->yscroll += p->vrows - vc->vc_rows;
- fbcon_redraw_move(vc, p, t, vc->vc_rows - count, t + count);
- }
-
- ops->var.xoffset = 0;
- ops->var.yoffset = p->yscroll * vc->vc_font.height;
- ops->var.vmode &= ~FB_VMODE_YWRAP;
- ops->update_start(info);
- fbcon_clear_margins(vc, 1);
- scrollback_max -= count;
- if (scrollback_max < 0)
- scrollback_max = 0;
- scrollback_current = 0;
-}
-
-static void fbcon_redraw_move(struct vc_data *vc, struct fbcon_display *p,
- int line, int count, int dy)
-{
- unsigned short *s = (unsigned short *)
- (vc->vc_origin + vc->vc_size_row * line);
-
- while (count--) {
- unsigned short *start = s;
- unsigned short *le = advance_row(s, 1);
- unsigned short c;
- int x = 0;
- unsigned short attr = 1;
-
- do {
- c = scr_readw(s);
- if (attr != (c & 0xff00)) {
- attr = c & 0xff00;
- if (s > start) {
- fbcon_putcs(vc, start, s - start,
- dy, x);
- x += s - start;
- start = s;
- }
- }
- console_conditional_schedule();
- s++;
- } while (s < le);
- if (s > start)
- fbcon_putcs(vc, start, s - start, dy, x);
- console_conditional_schedule();
- dy++;
- }
-}
-
-static void fbcon_redraw_blit(struct vc_data *vc, struct fb_info *info,
- struct fbcon_display *p, int line, int count, int ycount)
-{
- int offset = ycount * vc->vc_cols;
- unsigned short *d = (unsigned short *)
- (vc->vc_origin + vc->vc_size_row * line);
- unsigned short *s = d + offset;
- struct fbcon_ops *ops = info->fbcon_par;
-
- while (count--) {
- unsigned short *start = s;
- unsigned short *le = advance_row(s, 1);
- unsigned short c;
- int x = 0;
-
- do {
- c = scr_readw(s);
-
- if (c == scr_readw(d)) {
- if (s > start) {
- ops->bmove(vc, info, line + ycount, x,
- line, x, 1, s-start);
- x += s - start + 1;
- start = s + 1;
- } else {
- x++;
- start++;
- }
- }
-
- scr_writew(c, d);
- console_conditional_schedule();
- s++;
- d++;
- } while (s < le);
- if (s > start)
- ops->bmove(vc, info, line + ycount, x, line, x, 1,
- s-start);
- console_conditional_schedule();
- if (ycount > 0)
- line++;
- else {
- line--;
- /* NOTE: We subtract two lines from these pointers */
- s -= vc->vc_size_row;
- d -= vc->vc_size_row;
- }
- }
-}
-
static void fbcon_redraw(struct vc_data *vc, struct fbcon_display *p,
int line, int count, int offset)
{
{
struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
struct fbcon_display *p = &fb_display[vc->vc_num];
- int scroll_partial = info->flags & FBINFO_PARTIAL_PAN_OK;
if (fbcon_is_inactive(vc, info))
return true;
case SM_UP:
if (count > vc->vc_rows) /* Maximum realistic size */
count = vc->vc_rows;
- if (logo_shown >= 0)
- goto redraw_up;
- switch (p->scrollmode) {
- case SCROLL_MOVE:
- fbcon_redraw_blit(vc, info, p, t, b - t - count,
- count);
- fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
- scr_memsetw((unsigned short *) (vc->vc_origin +
- vc->vc_size_row *
- (b - count)),
- vc->vc_video_erase_char,
- vc->vc_size_row * count);
- return true;
-
- case SCROLL_WRAP_MOVE:
- if (b - t - count > 3 * vc->vc_rows >> 2) {
- if (t > 0)
- fbcon_bmove(vc, 0, 0, count, 0, t,
- vc->vc_cols);
- ywrap_up(vc, count);
- if (vc->vc_rows - b > 0)
- fbcon_bmove(vc, b - count, 0, b, 0,
- vc->vc_rows - b,
- vc->vc_cols);
- } else if (info->flags & FBINFO_READS_FAST)
- fbcon_bmove(vc, t + count, 0, t, 0,
- b - t - count, vc->vc_cols);
- else
- goto redraw_up;
- fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_PAN_REDRAW:
- if ((p->yscroll + count <=
- 2 * (p->vrows - vc->vc_rows))
- && ((!scroll_partial && (b - t == vc->vc_rows))
- || (scroll_partial
- && (b - t - count >
- 3 * vc->vc_rows >> 2)))) {
- if (t > 0)
- fbcon_redraw_move(vc, p, 0, t, count);
- ypan_up_redraw(vc, t, count);
- if (vc->vc_rows - b > 0)
- fbcon_redraw_move(vc, p, b,
- vc->vc_rows - b, b);
- } else
- fbcon_redraw_move(vc, p, t + count, b - t - count, t);
- fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_PAN_MOVE:
- if ((p->yscroll + count <=
- 2 * (p->vrows - vc->vc_rows))
- && ((!scroll_partial && (b - t == vc->vc_rows))
- || (scroll_partial
- && (b - t - count >
- 3 * vc->vc_rows >> 2)))) {
- if (t > 0)
- fbcon_bmove(vc, 0, 0, count, 0, t,
- vc->vc_cols);
- ypan_up(vc, count);
- if (vc->vc_rows - b > 0)
- fbcon_bmove(vc, b - count, 0, b, 0,
- vc->vc_rows - b,
- vc->vc_cols);
- } else if (info->flags & FBINFO_READS_FAST)
- fbcon_bmove(vc, t + count, 0, t, 0,
- b - t - count, vc->vc_cols);
- else
- goto redraw_up;
- fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_REDRAW:
- redraw_up:
- fbcon_redraw(vc, p, t, b - t - count,
- count * vc->vc_cols);
- fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
- scr_memsetw((unsigned short *) (vc->vc_origin +
- vc->vc_size_row *
- (b - count)),
- vc->vc_video_erase_char,
- vc->vc_size_row * count);
- return true;
- }
- break;
+ fbcon_redraw(vc, p, t, b - t - count,
+ count * vc->vc_cols);
+ fbcon_clear(vc, b - count, 0, count, vc->vc_cols);
+ scr_memsetw((unsigned short *) (vc->vc_origin +
+ vc->vc_size_row *
+ (b - count)),
+ vc->vc_video_erase_char,
+ vc->vc_size_row * count);
+ return true;
case SM_DOWN:
if (count > vc->vc_rows) /* Maximum realistic size */
count = vc->vc_rows;
- if (logo_shown >= 0)
- goto redraw_down;
- switch (p->scrollmode) {
- case SCROLL_MOVE:
- fbcon_redraw_blit(vc, info, p, b - 1, b - t - count,
- -count);
- fbcon_clear(vc, t, 0, count, vc->vc_cols);
- scr_memsetw((unsigned short *) (vc->vc_origin +
- vc->vc_size_row *
- t),
- vc->vc_video_erase_char,
- vc->vc_size_row * count);
- return true;
-
- case SCROLL_WRAP_MOVE:
- if (b - t - count > 3 * vc->vc_rows >> 2) {
- if (vc->vc_rows - b > 0)
- fbcon_bmove(vc, b, 0, b - count, 0,
- vc->vc_rows - b,
- vc->vc_cols);
- ywrap_down(vc, count);
- if (t > 0)
- fbcon_bmove(vc, count, 0, 0, 0, t,
- vc->vc_cols);
- } else if (info->flags & FBINFO_READS_FAST)
- fbcon_bmove(vc, t, 0, t + count, 0,
- b - t - count, vc->vc_cols);
- else
- goto redraw_down;
- fbcon_clear(vc, t, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_PAN_MOVE:
- if ((count - p->yscroll <= p->vrows - vc->vc_rows)
- && ((!scroll_partial && (b - t == vc->vc_rows))
- || (scroll_partial
- && (b - t - count >
- 3 * vc->vc_rows >> 2)))) {
- if (vc->vc_rows - b > 0)
- fbcon_bmove(vc, b, 0, b - count, 0,
- vc->vc_rows - b,
- vc->vc_cols);
- ypan_down(vc, count);
- if (t > 0)
- fbcon_bmove(vc, count, 0, 0, 0, t,
- vc->vc_cols);
- } else if (info->flags & FBINFO_READS_FAST)
- fbcon_bmove(vc, t, 0, t + count, 0,
- b - t - count, vc->vc_cols);
- else
- goto redraw_down;
- fbcon_clear(vc, t, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_PAN_REDRAW:
- if ((count - p->yscroll <= p->vrows - vc->vc_rows)
- && ((!scroll_partial && (b - t == vc->vc_rows))
- || (scroll_partial
- && (b - t - count >
- 3 * vc->vc_rows >> 2)))) {
- if (vc->vc_rows - b > 0)
- fbcon_redraw_move(vc, p, b, vc->vc_rows - b,
- b - count);
- ypan_down_redraw(vc, t, count);
- if (t > 0)
- fbcon_redraw_move(vc, p, count, t, 0);
- } else
- fbcon_redraw_move(vc, p, t, b - t - count, t + count);
- fbcon_clear(vc, t, 0, count, vc->vc_cols);
- break;
-
- case SCROLL_REDRAW:
- redraw_down:
- fbcon_redraw(vc, p, b - 1, b - t - count,
- -count * vc->vc_cols);
- fbcon_clear(vc, t, 0, count, vc->vc_cols);
- scr_memsetw((unsigned short *) (vc->vc_origin +
- vc->vc_size_row *
- t),
- vc->vc_video_erase_char,
- vc->vc_size_row * count);
- return true;
- }
+ fbcon_redraw(vc, p, b - 1, b - t - count,
+ -count * vc->vc_cols);
+ fbcon_clear(vc, t, 0, count, vc->vc_cols);
+ scr_memsetw((unsigned short *) (vc->vc_origin +
+ vc->vc_size_row *
+ t),
+ vc->vc_video_erase_char,
+ vc->vc_size_row * count);
+ return true;
}
return false;
}
-
-static void fbcon_bmove(struct vc_data *vc, int sy, int sx, int dy, int dx,
- int height, int width)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_display *p = &fb_display[vc->vc_num];
-
- if (fbcon_is_inactive(vc, info))
- return;
-
- if (!width || !height)
- return;
-
- /* Split blits that cross physical y_wrap case.
- * Pathological case involves 4 blits, better to use recursive
- * code rather than unrolled case
- *
- * Recursive invocations don't need to erase the cursor over and
- * over again, so we use fbcon_bmove_rec()
- */
- fbcon_bmove_rec(vc, p, sy, sx, dy, dx, height, width,
- p->vrows - p->yscroll);
-}
-
-static void fbcon_bmove_rec(struct vc_data *vc, struct fbcon_display *p, int sy, int sx,
- int dy, int dx, int height, int width, u_int y_break)
-{
- struct fb_info *info = registered_fb[con2fb_map[vc->vc_num]];
- struct fbcon_ops *ops = info->fbcon_par;
- u_int b;
-
- if (sy < y_break && sy + height > y_break) {
- b = y_break - sy;
- if (dy < sy) { /* Avoid trashing self */
- fbcon_bmove_rec(vc, p, sy, sx, dy, dx, b, width,
- y_break);
- fbcon_bmove_rec(vc, p, sy + b, sx, dy + b, dx,
- height - b, width, y_break);
- } else {
- fbcon_bmove_rec(vc, p, sy + b, sx, dy + b, dx,
- height - b, width, y_break);
- fbcon_bmove_rec(vc, p, sy, sx, dy, dx, b, width,
- y_break);
- }
- return;
- }
-
- if (dy < y_break && dy + height > y_break) {
- b = y_break - dy;
- if (dy < sy) { /* Avoid trashing self */
- fbcon_bmove_rec(vc, p, sy, sx, dy, dx, b, width,
- y_break);
- fbcon_bmove_rec(vc, p, sy + b, sx, dy + b, dx,
- height - b, width, y_break);
- } else {
- fbcon_bmove_rec(vc, p, sy + b, sx, dy + b, dx,
- height - b, width, y_break);
- fbcon_bmove_rec(vc, p, sy, sx, dy, dx, b, width,
- y_break);
- }
- return;
- }
- ops->bmove(vc, info, real_y(p, sy), sx, real_y(p, dy), dx,
- height, width);
-}
-
static void updatescrollmode(struct fbcon_display *p,
struct fb_info *info,
struct vc_data *vc)
updatescrollmode(p, info, vc);
- switch (p->scrollmode) {
- case SCROLL_WRAP_MOVE:
- scrollback_phys_max = p->vrows - vc->vc_rows;
- break;
- case SCROLL_PAN_MOVE:
- case SCROLL_PAN_REDRAW:
- scrollback_phys_max = p->vrows - 2 * vc->vc_rows;
- if (scrollback_phys_max < 0)
- scrollback_phys_max = 0;
- break;
- default:
- scrollback_phys_max = 0;
- break;
- }
-
+ scrollback_phys_max = 0;
scrollback_max = 0;
scrollback_current = 0;
/* Filled in by the low-level console driver */
const u_char *fontdata;
int userfont; /* != 0 if fontdata kmalloc()ed */
- u_short scrollmode; /* Scroll Method */
u_short inverse; /* != 0 text black on white as default */
short yscroll; /* Hardware scrolling */
int vrows; /* number of virtual rows */
};
struct fbcon_ops {
- void (*bmove)(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width);
void (*clear)(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width);
void (*putcs)(struct vc_data *vc, struct fb_info *info,
#define attr_bgcol_ec(bgshift, vc, info) attr_col_ec(bgshift, vc, info, 0)
#define attr_fgcol_ec(fgshift, vc, info) attr_col_ec(fgshift, vc, info, 1)
- /*
- * Scroll Method
- */
-
-/* There are several methods fbcon can use to move text around the screen:
- *
- * Operation Pan Wrap
- *---------------------------------------------
- * SCROLL_MOVE copyarea No No
- * SCROLL_PAN_MOVE copyarea Yes No
- * SCROLL_WRAP_MOVE copyarea No Yes
- * SCROLL_REDRAW imageblit No No
- * SCROLL_PAN_REDRAW imageblit Yes No
- * SCROLL_WRAP_REDRAW imageblit No Yes
- *
- * (SCROLL_WRAP_REDRAW is not implemented yet)
- *
- * In general, fbcon will choose the best scrolling
- * method based on the rule below:
- *
- * Pan/Wrap > accel imageblit > accel copyarea >
- * soft imageblit > (soft copyarea)
- *
- * Exception to the rule: Pan + accel copyarea is
- * preferred over Pan + accel imageblit.
- *
- * The above is typical for PCI/AGP cards. Unless
- * overridden, fbcon will never use soft copyarea.
- *
- * If you need to override the above rule, set the
- * appropriate flags in fb_info->flags. For example,
- * to prefer copyarea over imageblit, set
- * FBINFO_READS_FAST.
- *
- * Other notes:
- * + use the hardware engine to move the text
- * (hw-accelerated copyarea() and fillrect())
- * + use hardware-supported panning on a large virtual screen
- * + amifb can not only pan, but also wrap the display by N lines
- * (i.e. visible line i = physical line (i+N) % yres).
- * + read what's already rendered on the screen and
- * write it in a different place (this is cfb_copyarea())
- * + re-render the text to the screen
- *
- * Whether to use wrapping or panning can only be figured out at
- * runtime (when we know whether our font height is a multiple
- * of the pan/wrap step)
- *
- */
-
-#define SCROLL_MOVE 0x001
-#define SCROLL_PAN_MOVE 0x002
-#define SCROLL_WRAP_MOVE 0x003
-#define SCROLL_REDRAW 0x004
-#define SCROLL_PAN_REDRAW 0x005
-
#ifdef CONFIG_FB_TILEBLITTING
extern void fbcon_set_tileops(struct vc_data *vc, struct fb_info *info);
#endif
}
}
-
-static void ccw_bmove(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width)
-{
- struct fbcon_ops *ops = info->fbcon_par;
- struct fb_copyarea area;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
-
- area.sx = sy * vc->vc_font.height;
- area.sy = vyres - ((sx + width) * vc->vc_font.width);
- area.dx = dy * vc->vc_font.height;
- area.dy = vyres - ((dx + width) * vc->vc_font.width);
- area.width = height * vc->vc_font.height;
- area.height = width * vc->vc_font.width;
-
- info->fbops->fb_copyarea(info, &area);
-}
-
static void ccw_clear(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width)
{
- struct fbcon_ops *ops = info->fbcon_par;
struct fb_fillrect region;
int bgshift = (vc->vc_hi_font_mask) ? 13 : 12;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
region.color = attr_bgcol_ec(bgshift,vc,info);
region.dx = sy * vc->vc_font.height;
u32 cnt, pitch, size;
u32 attribute = get_attribute(info, scr_readw(s));
u8 *dst, *buf = NULL;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
if (!ops->fontbuffer)
return;
int attribute, use_sw = vc->vc_cursor_type & CUR_SW;
int err = 1, dx, dy;
char *src;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
if (!ops->fontbuffer)
return;
{
struct fbcon_ops *ops = info->fbcon_par;
u32 yoffset;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
int err;
yoffset = (vyres - info->var.yres) - ops->var.xoffset;
void fbcon_rotate_ccw(struct fbcon_ops *ops)
{
- ops->bmove = ccw_bmove;
ops->clear = ccw_clear;
ops->putcs = ccw_putcs;
ops->clear_margins = ccw_clear_margins;
}
}
-
-static void cw_bmove(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width)
-{
- struct fbcon_ops *ops = info->fbcon_par;
- struct fb_copyarea area;
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
-
- area.sx = vxres - ((sy + height) * vc->vc_font.height);
- area.sy = sx * vc->vc_font.width;
- area.dx = vxres - ((dy + height) * vc->vc_font.height);
- area.dy = dx * vc->vc_font.width;
- area.width = height * vc->vc_font.height;
- area.height = width * vc->vc_font.width;
-
- info->fbops->fb_copyarea(info, &area);
-}
-
static void cw_clear(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width)
{
- struct fbcon_ops *ops = info->fbcon_par;
struct fb_fillrect region;
int bgshift = (vc->vc_hi_font_mask) ? 13 : 12;
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vxres = info->var.xres;
region.color = attr_bgcol_ec(bgshift,vc,info);
region.dx = vxres - ((sy + height) * vc->vc_font.height);
u32 cnt, pitch, size;
u32 attribute = get_attribute(info, scr_readw(s));
u8 *dst, *buf = NULL;
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vxres = info->var.xres;
if (!ops->fontbuffer)
return;
int attribute, use_sw = vc->vc_cursor_type & CUR_SW;
int err = 1, dx, dy;
char *src;
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vxres = info->var.xres;
if (!ops->fontbuffer)
return;
static int cw_update_start(struct fb_info *info)
{
struct fbcon_ops *ops = info->fbcon_par;
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vxres = info->var.xres;
u32 xoffset;
int err;
void fbcon_rotate_cw(struct fbcon_ops *ops)
{
- ops->bmove = cw_bmove;
ops->clear = cw_clear;
ops->putcs = cw_putcs;
ops->clear_margins = cw_clear_margins;
#ifndef _FBCON_ROTATE_H
#define _FBCON_ROTATE_H
-#define GETVYRES(s,i) ({ \
- (s == SCROLL_REDRAW || s == SCROLL_MOVE) ? \
- (i)->var.yres : (i)->var.yres_virtual; })
-
-#define GETVXRES(s,i) ({ \
- (s == SCROLL_REDRAW || s == SCROLL_MOVE || !(i)->fix.xpanstep) ? \
- (i)->var.xres : (i)->var.xres_virtual; })
-
-
static inline int pattern_test_bit(u32 x, u32 y, u32 pitch, const char *pat)
{
u32 tmp = (y * pitch) + x, index = tmp / 8, bit = tmp % 8;
}
}
-
-static void ud_bmove(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width)
-{
- struct fbcon_ops *ops = info->fbcon_par;
- struct fb_copyarea area;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
-
- area.sy = vyres - ((sy + height) * vc->vc_font.height);
- area.sx = vxres - ((sx + width) * vc->vc_font.width);
- area.dy = vyres - ((dy + height) * vc->vc_font.height);
- area.dx = vxres - ((dx + width) * vc->vc_font.width);
- area.height = height * vc->vc_font.height;
- area.width = width * vc->vc_font.width;
-
- info->fbops->fb_copyarea(info, &area);
-}
-
static void ud_clear(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width)
{
- struct fbcon_ops *ops = info->fbcon_par;
struct fb_fillrect region;
int bgshift = (vc->vc_hi_font_mask) ? 13 : 12;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
+ u32 vxres = info->var.xres;
region.color = attr_bgcol_ec(bgshift,vc,info);
region.dy = vyres - ((sy + height) * vc->vc_font.height);
u32 mod = vc->vc_font.width % 8, cnt, pitch, size;
u32 attribute = get_attribute(info, scr_readw(s));
u8 *dst, *buf = NULL;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
+ u32 vxres = info->var.xres;
if (!ops->fontbuffer)
return;
int attribute, use_sw = vc->vc_cursor_type & CUR_SW;
int err = 1, dx, dy;
char *src;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
+ u32 vxres = info->var.xres;
if (!ops->fontbuffer)
return;
{
struct fbcon_ops *ops = info->fbcon_par;
int xoffset, yoffset;
- u32 vyres = GETVYRES(ops->p->scrollmode, info);
- u32 vxres = GETVXRES(ops->p->scrollmode, info);
+ u32 vyres = info->var.yres;
+ u32 vxres = info->var.xres;
int err;
xoffset = vxres - info->var.xres - ops->var.xoffset;
void fbcon_rotate_ud(struct fbcon_ops *ops)
{
- ops->bmove = ud_bmove;
ops->clear = ud_clear;
ops->putcs = ud_putcs;
ops->clear_margins = ud_clear_margins;
{
unlink_framebuffer(fb_info);
if (fb_info->pixmap.addr &&
- (fb_info->pixmap.flags & FB_PIXMAP_DEFAULT))
+ (fb_info->pixmap.flags & FB_PIXMAP_DEFAULT)) {
kfree(fb_info->pixmap.addr);
+ fb_info->pixmap.addr = NULL;
+ }
+
fb_destroy_modelist(&fb_info->modelist);
registered_fb[fb_info->node] = NULL;
num_registered_fb--;
#include <asm/types.h>
#include "fbcon.h"
-static void tile_bmove(struct vc_data *vc, struct fb_info *info, int sy,
- int sx, int dy, int dx, int height, int width)
-{
- struct fb_tilearea area;
-
- area.sx = sx;
- area.sy = sy;
- area.dx = dx;
- area.dy = dy;
- area.height = height;
- area.width = width;
-
- info->tileops->fb_tilecopy(info, &area);
-}
-
static void tile_clear(struct vc_data *vc, struct fb_info *info, int sy,
int sx, int height, int width)
{
struct fb_tilemap map;
struct fbcon_ops *ops = info->fbcon_par;
- ops->bmove = tile_bmove;
ops->clear = tile_clear;
ops->putcs = tile_putcs;
ops->clear_margins = tile_clear_margins;
char *option = NULL;
efi_memory_desc_t md;
+ /*
+ * Generic drivers must not be registered if a framebuffer exists.
+ * If a native driver was probed, the display hardware was already
+ * taken and attempting to use the system framebuffer is dangerous.
+ */
+ if (num_registered_fb > 0) {
+ dev_err(&dev->dev,
+ "efifb: a framebuffer is already registered\n");
+ return -EINVAL;
+ }
+
if (screen_info.orig_video_isVGA != VIDEO_TYPE_EFI || pci_dev_disabled)
return -ENODEV;
struct simplefb_par *par;
struct resource *mem;
+ /*
+ * Generic drivers must not be registered if a framebuffer exists.
+ * If a native driver was probed, the display hardware was already
+ * taken and attempting to use the system framebuffer is dangerous.
+ */
+ if (num_registered_fb > 0) {
+ dev_err(&pdev->dev,
+ "simplefb: a framebuffer is already registered\n");
+ return -EINVAL;
+ }
+
if (fb_get_options("simplefb", NULL))
return -ENODEV;
}
/**
- * xxxfb_copyarea - REQUIRED function. Can use generic routines if
- * non acclerated hardware and packed pixel based.
+ * xxxfb_copyarea - OBSOLETE function.
* Copies one area of the screen to another area.
+ * Will be deleted in a future version
*
* @info: frame buffer structure that represents a single frame buffer
* @area: Structure providing the data to copy the framebuffer contents
* from one region to another.
*
- * This drawing operation copies a rectangular area from one area of the
+ * This drawing operation copied a rectangular area from one area of the
* screen to another area.
*/
void xxxfb_copyarea(struct fb_info *p, const struct fb_copyarea *area)
.fb_setcolreg = xxxfb_setcolreg,
.fb_blank = xxxfb_blank,
.fb_pan_display = xxxfb_pan_display,
- .fb_fillrect = xxxfb_fillrect, /* Needed !!! */
- .fb_copyarea = xxxfb_copyarea, /* Needed !!! */
- .fb_imageblit = xxxfb_imageblit, /* Needed !!! */
+ .fb_fillrect = xxxfb_fillrect, /* Needed !!! */
+ .fb_copyarea = xxxfb_copyarea, /* Obsolete */
+ .fb_imageblit = xxxfb_imageblit, /* Needed !!! */
.fb_cursor = xxxfb_cursor, /* Optional !!! */
.fb_sync = xxxfb_sync,
.fb_ioctl = xxxfb_ioctl,
default m
depends on X86_64
depends on VIRTIO
- depends on MEMORY_HOTPLUG_SPARSE
+ depends on MEMORY_HOTPLUG
depends on MEMORY_HOTREMOVE
depends on CONTIG_ALLOC
+ depends on EXCLUSIVE_SYSTEM_RAM
help
This driver provides access to virtio-mem paravirtualized memory
devices, allowing to hotplug and hotunplug memory.
* When this lock is held the pointers can't change, ONLINE and
* OFFLINE blocks can't change the state and no subblocks will get
* plugged/unplugged.
+ *
+ * In kdump mode, used to serialize requests, last_block_addr and
+ * last_block_plugged.
*/
struct mutex hotplug_mutex;
bool hotplug_active;
/* An error occurred we cannot handle - stop processing requests. */
bool broken;
+ /* Cached valued of is_kdump_kernel() when the device was probed. */
+ bool in_kdump;
+
/* The driver is being removed. */
spinlock_t removal_lock;
bool removing;
/* Memory notifier (online/offline events). */
struct notifier_block memory_notifier;
+#ifdef CONFIG_PROC_VMCORE
+ /* vmcore callback for /proc/vmcore handling in kdump mode */
+ struct vmcore_cb vmcore_cb;
+ uint64_t last_block_addr;
+ bool last_block_plugged;
+#endif /* CONFIG_PROC_VMCORE */
+
/* Next device in the list of virtio-mem devices. */
struct list_head next;
};
static void virtio_mem_fake_offline_cancel_offline(unsigned long pfn,
unsigned long nr_pages);
static void virtio_mem_retry(struct virtio_mem *vm);
+static int virtio_mem_create_resource(struct virtio_mem *vm);
+static void virtio_mem_delete_resource(struct virtio_mem *vm);
/*
* Register a virtio-mem device so it will be considered for the online_page
uint64_t diff;
int rc;
+ if (unlikely(vm->in_kdump)) {
+ dev_warn_once(&vm->vdev->dev,
+ "unexpected workqueue run in kdump kernel\n");
+ return;
+ }
+
hrtimer_cancel(&vm->retry_timer);
if (vm->broken)
return 0;
}
-static int virtio_mem_init(struct virtio_mem *vm)
+static int virtio_mem_init_hotplug(struct virtio_mem *vm)
{
const struct range pluggable_range = mhp_get_pluggable_range(true);
- uint64_t sb_size, addr;
- uint16_t node_id;
-
- if (!vm->vdev->config->get) {
- dev_err(&vm->vdev->dev, "config access disabled\n");
- return -EINVAL;
- }
-
- /*
- * We don't want to (un)plug or reuse any memory when in kdump. The
- * memory is still accessible (but not mapped).
- */
- if (is_kdump_kernel()) {
- dev_warn(&vm->vdev->dev, "disabled in kdump kernel\n");
- return -EBUSY;
- }
-
- /* Fetch all properties that can't change. */
- virtio_cread_le(vm->vdev, struct virtio_mem_config, plugged_size,
- &vm->plugged_size);
- virtio_cread_le(vm->vdev, struct virtio_mem_config, block_size,
- &vm->device_block_size);
- virtio_cread_le(vm->vdev, struct virtio_mem_config, node_id,
- &node_id);
- vm->nid = virtio_mem_translate_node_id(vm, node_id);
- virtio_cread_le(vm->vdev, struct virtio_mem_config, addr, &vm->addr);
- virtio_cread_le(vm->vdev, struct virtio_mem_config, region_size,
- &vm->region_size);
-
- /* Determine the nid for the device based on the lowest address. */
- if (vm->nid == NUMA_NO_NODE)
- vm->nid = memory_add_physaddr_to_nid(vm->addr);
+ uint64_t unit_pages, sb_size, addr;
+ int rc;
/* bad device setup - warn only */
if (!IS_ALIGNED(vm->addr, memory_block_size_bytes()))
vm->offline_threshold);
}
- dev_info(&vm->vdev->dev, "start address: 0x%llx", vm->addr);
- dev_info(&vm->vdev->dev, "region size: 0x%llx", vm->region_size);
- dev_info(&vm->vdev->dev, "device block size: 0x%llx",
- (unsigned long long)vm->device_block_size);
dev_info(&vm->vdev->dev, "memory block size: 0x%lx",
memory_block_size_bytes());
if (vm->in_sbm)
else
dev_info(&vm->vdev->dev, "big block size: 0x%llx",
(unsigned long long)vm->bbm.bb_size);
+
+ /* create the parent resource for all memory */
+ rc = virtio_mem_create_resource(vm);
+ if (rc)
+ return rc;
+
+ /* use a single dynamic memory group to cover the whole memory device */
+ if (vm->in_sbm)
+ unit_pages = PHYS_PFN(memory_block_size_bytes());
+ else
+ unit_pages = PHYS_PFN(vm->bbm.bb_size);
+ rc = memory_group_register_dynamic(vm->nid, unit_pages);
+ if (rc < 0)
+ goto out_del_resource;
+ vm->mgid = rc;
+
+ /*
+ * If we still have memory plugged, we have to unplug all memory first.
+ * Registering our parent resource makes sure that this memory isn't
+ * actually in use (e.g., trying to reload the driver).
+ */
+ if (vm->plugged_size) {
+ vm->unplug_all_required = true;
+ dev_info(&vm->vdev->dev, "unplugging all memory is required\n");
+ }
+
+ /* register callbacks */
+ vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
+ rc = register_memory_notifier(&vm->memory_notifier);
+ if (rc)
+ goto out_unreg_group;
+ rc = register_virtio_mem_device(vm);
+ if (rc)
+ goto out_unreg_mem;
+
+ return 0;
+out_unreg_mem:
+ unregister_memory_notifier(&vm->memory_notifier);
+out_unreg_group:
+ memory_group_unregister(vm->mgid);
+out_del_resource:
+ virtio_mem_delete_resource(vm);
+ return rc;
+}
+
+#ifdef CONFIG_PROC_VMCORE
+static int virtio_mem_send_state_request(struct virtio_mem *vm, uint64_t addr,
+ uint64_t size)
+{
+ const uint64_t nb_vm_blocks = size / vm->device_block_size;
+ const struct virtio_mem_req req = {
+ .type = cpu_to_virtio16(vm->vdev, VIRTIO_MEM_REQ_STATE),
+ .u.state.addr = cpu_to_virtio64(vm->vdev, addr),
+ .u.state.nb_blocks = cpu_to_virtio16(vm->vdev, nb_vm_blocks),
+ };
+ int rc = -ENOMEM;
+
+ dev_dbg(&vm->vdev->dev, "requesting state: 0x%llx - 0x%llx\n", addr,
+ addr + size - 1);
+
+ switch (virtio_mem_send_request(vm, &req)) {
+ case VIRTIO_MEM_RESP_ACK:
+ return virtio16_to_cpu(vm->vdev, vm->resp.u.state.state);
+ case VIRTIO_MEM_RESP_ERROR:
+ rc = -EINVAL;
+ break;
+ default:
+ break;
+ }
+
+ dev_dbg(&vm->vdev->dev, "requesting state failed: %d\n", rc);
+ return rc;
+}
+
+static bool virtio_mem_vmcore_pfn_is_ram(struct vmcore_cb *cb,
+ unsigned long pfn)
+{
+ struct virtio_mem *vm = container_of(cb, struct virtio_mem,
+ vmcore_cb);
+ uint64_t addr = PFN_PHYS(pfn);
+ bool is_ram;
+ int rc;
+
+ if (!virtio_mem_contains_range(vm, addr, PAGE_SIZE))
+ return true;
+ if (!vm->plugged_size)
+ return false;
+
+ /*
+ * We have to serialize device requests and access to the information
+ * about the block queried last.
+ */
+ mutex_lock(&vm->hotplug_mutex);
+
+ addr = ALIGN_DOWN(addr, vm->device_block_size);
+ if (addr != vm->last_block_addr) {
+ rc = virtio_mem_send_state_request(vm, addr,
+ vm->device_block_size);
+ /* On any kind of error, we're going to signal !ram. */
+ if (rc == VIRTIO_MEM_STATE_PLUGGED)
+ vm->last_block_plugged = true;
+ else
+ vm->last_block_plugged = false;
+ vm->last_block_addr = addr;
+ }
+
+ is_ram = vm->last_block_plugged;
+ mutex_unlock(&vm->hotplug_mutex);
+ return is_ram;
+}
+#endif /* CONFIG_PROC_VMCORE */
+
+static int virtio_mem_init_kdump(struct virtio_mem *vm)
+{
+#ifdef CONFIG_PROC_VMCORE
+ dev_info(&vm->vdev->dev, "memory hot(un)plug disabled in kdump kernel\n");
+ vm->vmcore_cb.pfn_is_ram = virtio_mem_vmcore_pfn_is_ram;
+ register_vmcore_cb(&vm->vmcore_cb);
+ return 0;
+#else /* CONFIG_PROC_VMCORE */
+ dev_warn(&vm->vdev->dev, "disabled in kdump kernel without vmcore\n");
+ return -EBUSY;
+#endif /* CONFIG_PROC_VMCORE */
+}
+
+static int virtio_mem_init(struct virtio_mem *vm)
+{
+ uint16_t node_id;
+
+ if (!vm->vdev->config->get) {
+ dev_err(&vm->vdev->dev, "config access disabled\n");
+ return -EINVAL;
+ }
+
+ /* Fetch all properties that can't change. */
+ virtio_cread_le(vm->vdev, struct virtio_mem_config, plugged_size,
+ &vm->plugged_size);
+ virtio_cread_le(vm->vdev, struct virtio_mem_config, block_size,
+ &vm->device_block_size);
+ virtio_cread_le(vm->vdev, struct virtio_mem_config, node_id,
+ &node_id);
+ vm->nid = virtio_mem_translate_node_id(vm, node_id);
+ virtio_cread_le(vm->vdev, struct virtio_mem_config, addr, &vm->addr);
+ virtio_cread_le(vm->vdev, struct virtio_mem_config, region_size,
+ &vm->region_size);
+
+ /* Determine the nid for the device based on the lowest address. */
+ if (vm->nid == NUMA_NO_NODE)
+ vm->nid = memory_add_physaddr_to_nid(vm->addr);
+
+ dev_info(&vm->vdev->dev, "start address: 0x%llx", vm->addr);
+ dev_info(&vm->vdev->dev, "region size: 0x%llx", vm->region_size);
+ dev_info(&vm->vdev->dev, "device block size: 0x%llx",
+ (unsigned long long)vm->device_block_size);
if (vm->nid != NUMA_NO_NODE && IS_ENABLED(CONFIG_NUMA))
dev_info(&vm->vdev->dev, "nid: %d", vm->nid);
- return 0;
+ /*
+ * We don't want to (un)plug or reuse any memory when in kdump. The
+ * memory is still accessible (but not exposed to Linux).
+ */
+ if (vm->in_kdump)
+ return virtio_mem_init_kdump(vm);
+ return virtio_mem_init_hotplug(vm);
}
static int virtio_mem_create_resource(struct virtio_mem *vm)
if (!name)
return -ENOMEM;
+ /* Disallow mapping device memory via /dev/mem completely. */
vm->parent_resource = __request_mem_region(vm->addr, vm->region_size,
- name, IORESOURCE_SYSTEM_RAM);
+ name, IORESOURCE_SYSTEM_RAM |
+ IORESOURCE_EXCLUSIVE);
if (!vm->parent_resource) {
kfree(name);
dev_warn(&vm->vdev->dev, "could not reserve device region\n");
static int virtio_mem_probe(struct virtio_device *vdev)
{
struct virtio_mem *vm;
- uint64_t unit_pages;
int rc;
BUILD_BUG_ON(sizeof(struct virtio_mem_req) != 24);
hrtimer_init(&vm->retry_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
vm->retry_timer.function = virtio_mem_timer_expired;
vm->retry_timer_ms = VIRTIO_MEM_RETRY_TIMER_MIN_MS;
+ vm->in_kdump = is_kdump_kernel();
/* register the virtqueue */
rc = virtio_mem_init_vq(vm);
if (rc)
goto out_del_vq;
- /* create the parent resource for all memory */
- rc = virtio_mem_create_resource(vm);
- if (rc)
- goto out_del_vq;
-
- /* use a single dynamic memory group to cover the whole memory device */
- if (vm->in_sbm)
- unit_pages = PHYS_PFN(memory_block_size_bytes());
- else
- unit_pages = PHYS_PFN(vm->bbm.bb_size);
- rc = memory_group_register_dynamic(vm->nid, unit_pages);
- if (rc < 0)
- goto out_del_resource;
- vm->mgid = rc;
-
- /*
- * If we still have memory plugged, we have to unplug all memory first.
- * Registering our parent resource makes sure that this memory isn't
- * actually in use (e.g., trying to reload the driver).
- */
- if (vm->plugged_size) {
- vm->unplug_all_required = true;
- dev_info(&vm->vdev->dev, "unplugging all memory is required\n");
- }
-
- /* register callbacks */
- vm->memory_notifier.notifier_call = virtio_mem_memory_notifier_cb;
- rc = register_memory_notifier(&vm->memory_notifier);
- if (rc)
- goto out_unreg_group;
- rc = register_virtio_mem_device(vm);
- if (rc)
- goto out_unreg_mem;
-
virtio_device_ready(vdev);
/* trigger a config update to start processing the requested_size */
- atomic_set(&vm->config_changed, 1);
- queue_work(system_freezable_wq, &vm->wq);
+ if (!vm->in_kdump) {
+ atomic_set(&vm->config_changed, 1);
+ queue_work(system_freezable_wq, &vm->wq);
+ }
return 0;
-out_unreg_mem:
- unregister_memory_notifier(&vm->memory_notifier);
-out_unreg_group:
- memory_group_unregister(vm->mgid);
-out_del_resource:
- virtio_mem_delete_resource(vm);
out_del_vq:
vdev->config->del_vqs(vdev);
out_free_vm:
return rc;
}
-static void virtio_mem_remove(struct virtio_device *vdev)
+static void virtio_mem_deinit_hotplug(struct virtio_mem *vm)
{
- struct virtio_mem *vm = vdev->priv;
unsigned long mb_id;
int rc;
* away. Warn at least.
*/
if (virtio_mem_has_memory_added(vm)) {
- dev_warn(&vdev->dev, "device still has system memory added\n");
+ dev_warn(&vm->vdev->dev,
+ "device still has system memory added\n");
} else {
virtio_mem_delete_resource(vm);
kfree_const(vm->resource_name);
} else {
vfree(vm->bbm.bb_states);
}
+}
+
+static void virtio_mem_deinit_kdump(struct virtio_mem *vm)
+{
+#ifdef CONFIG_PROC_VMCORE
+ unregister_vmcore_cb(&vm->vmcore_cb);
+#endif /* CONFIG_PROC_VMCORE */
+}
+
+static void virtio_mem_remove(struct virtio_device *vdev)
+{
+ struct virtio_mem *vm = vdev->priv;
+
+ if (vm->in_kdump)
+ virtio_mem_deinit_kdump(vm);
+ else
+ virtio_mem_deinit_hotplug(vm);
/* reset the device and cleanup the queues */
vdev->config->reset(vdev);
{
struct virtio_mem *vm = vdev->priv;
+ if (unlikely(vm->in_kdump))
+ return;
+
atomic_set(&vm->config_changed, 1);
virtio_mem_retry(vm);
}
#if defined(CONFIG_NUMA) && defined(CONFIG_ACPI_NUMA)
VIRTIO_MEM_F_ACPI_PXM,
#endif
+ VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE,
};
static const struct virtio_device_id virtio_mem_id_table[] = {
select WATCHDOG_CORE
select MFD_SYSCON if ARCH_EXYNOS
help
- Watchdog timer block in the Samsung SoCs. This will reboot
- the system when the timer expires with the watchdog enabled.
+ Watchdog timer block in the Samsung S3C24xx, S3C64xx, S5Pv210 and
+ Exynos SoCs. This will reboot the system when the timer expires with
+ the watchdog enabled.
The driver is limited by the speed of the system's PCLK
signal, so with reasonably fast systems (PCLK around 50-66MHz)
then watchdog intervals of over approximately 20seconds are
unavailable.
+ Choose Y/M here only if you build for such Samsung SoC.
The driver can be built as a module by choosing M, and will
- be called s3c2410_wdt
+ be called s3c2410_wdt.
config SA1100_WATCHDOG
tristate "SA1100/PXA2xx watchdog"
Say N if you are unsure.
-config IOP_WATCHDOG
- tristate "IOP Watchdog"
- depends on ARCH_IOP13XX
- select WATCHDOG_NOWAYOUT if (ARCH_IOP32X || ARCH_IOP33X)
- help
- Say Y here if to include support for the watchdog timer
- in the Intel IOP3XX & IOP13XX I/O Processors. This driver can
- be built as a module by choosing M. The module will
- be called iop_wdt.
-
- Note: The IOP13XX watchdog does an Internal Bus Reset which will
- affect both cores and the peripherals of the IOP. The ATU-X
- and/or ATUe configuration registers will remain intact, but if
- operating as an Root Complex and/or Central Resource, the PCI-X
- and/or PCIe busses will also be reset. THIS IS A VERY BIG HAMMER.
-
config DAVINCI_WATCHDOG
tristate "DaVinci watchdog"
depends on ARCH_DAVINCI || ARCH_KEYSTONE || COMPILE_TEST
To compile this driver as a module, choose M here: the
module will be called imx7ulp_wdt.
-config UX500_WATCHDOG
- tristate "ST-Ericsson Ux500 watchdog"
+config DB500_WATCHDOG
+ tristate "ST-Ericsson DB800 watchdog"
depends on MFD_DB8500_PRCMU
select WATCHDOG_CORE
default y
help
Say Y here to include Watchdog timer support for the watchdog
- existing in the prcmu of ST-Ericsson Ux500 series platforms.
+ existing in the prcmu of ST-Ericsson DB8500 platform.
To compile this driver as a module, choose M here: the
- module will be called ux500_wdt.
+ module will be called db500_wdt.
config RETU_WATCHDOG
tristate "Retu watchdog"
config F71808E_WDT
tristate "Fintek F718xx, F818xx Super I/O Watchdog"
depends on X86
+ select WATCHDOG_CORE
help
This is the driver for the hardware watchdog on the Fintek F71808E,
F71862FG, F71868, F71869, F71882FG, F71889FG, F81803, F81865, and
config AR7_WDT
tristate "TI AR7 Watchdog Timer"
- depends on AR7 || (MIPS && COMPILE_TEST)
+ depends on AR7 || (MIPS && 32BIT && COMPILE_TEST)
help
Hardware driver for the TI AR7 Watchdog Timer.
obj-$(CONFIG_DW_WATCHDOG) += dw_wdt.o
obj-$(CONFIG_EP93XX_WATCHDOG) += ep93xx_wdt.o
obj-$(CONFIG_PNX4008_WATCHDOG) += pnx4008_wdt.o
-obj-$(CONFIG_IOP_WATCHDOG) += iop_wdt.o
obj-$(CONFIG_DAVINCI_WATCHDOG) += davinci_wdt.o
obj-$(CONFIG_K3_RTI_WATCHDOG) += rti_wdt.o
obj-$(CONFIG_ORION_WATCHDOG) += orion_wdt.o
obj-$(CONFIG_IMX2_WDT) += imx2_wdt.o
obj-$(CONFIG_IMX_SC_WDT) += imx_sc_wdt.o
obj-$(CONFIG_IMX7ULP_WDT) += imx7ulp_wdt.o
-obj-$(CONFIG_UX500_WATCHDOG) += ux500_wdt.o
+obj-$(CONFIG_DB500_WATCHDOG) += db8500_wdt.o
obj-$(CONFIG_RETU_WATCHDOG) += retu_wdt.o
obj-$(CONFIG_BCM2835_WDT) += bcm2835_wdt.o
obj-$(CONFIG_MOXART_WDT) += moxart_wdt.o
/* XXX currently fixed, allows max margin ~68.72 secs */
#define prescale_value 0xffff
-/* Resource of the WDT registers */
-static struct resource *ar7_regs_wdt;
/* Pointer to the remapped WDT IO space */
static struct ar7_wdt *ar7_wdt;
{
int rc;
- ar7_regs_wdt =
- platform_get_resource_byname(pdev, IORESOURCE_MEM, "regs");
- ar7_wdt = devm_ioremap_resource(&pdev->dev, ar7_regs_wdt);
+ ar7_wdt = devm_platform_ioremap_resource_byname(pdev, "regs");
if (IS_ERR(ar7_wdt))
return PTR_ERR(ar7_wdt);
bcm63xx_wdt_pet();
+ fallthrough;
+
case WDIOC_GETTIMEOUT:
return put_user(wdt_time, p);
struct da9062_watchdog *wdt = watchdog_get_drvdata(wdd);
int ret;
+ /*
+ * Prevent pings from occurring late in system poweroff/reboot sequence
+ * and possibly locking out restart handler from accessing i2c bus.
+ */
+ if (system_state > SYSTEM_RUNNING)
+ return 0;
+
ret = da9062_reset_watchdog_timer(wdt);
if (ret)
dev_err(wdt->hw->dev, "Failed to ping the watchdog (err = %d)\n",
struct da9063 *da9063 = watchdog_get_drvdata(wdd);
int ret;
+ /*
+ * Prevent pings from occurring late in system poweroff/reboot sequence
+ * and possibly locking out restart handler from accessing i2c bus.
+ */
+ if (system_state > SYSTEM_RUNNING)
+ return 0;
+
ret = regmap_write(da9063->regmap, DA9063_REG_CONTROL_F,
DA9063_WATCHDOG);
if (ret)
#include <linux/uaccess.h>
#include <linux/watchdog.h>
#include <linux/platform_device.h>
-#include <linux/platform_data/ux500_wdt.h>
#include <linux/mfd/dbx500-prcmu.h>
#define WATCHDOG_MIN 0
#define WATCHDOG_MAX28 268435 /* 28 bit resolution in ms == 268435.455 s */
-#define WATCHDOG_MAX32 4294967 /* 32 bit resolution in ms == 4294967.295 s */
static unsigned int timeout = WATCHDOG_TIMEOUT;
module_param(timeout, uint, 0);
"Watchdog cannot be stopped once started (default="
__MODULE_STRING(WATCHDOG_NOWAYOUT) ")");
-static int ux500_wdt_start(struct watchdog_device *wdd)
+static int db8500_wdt_start(struct watchdog_device *wdd)
{
return prcmu_enable_a9wdog(PRCMU_WDOG_ALL);
}
-static int ux500_wdt_stop(struct watchdog_device *wdd)
+static int db8500_wdt_stop(struct watchdog_device *wdd)
{
return prcmu_disable_a9wdog(PRCMU_WDOG_ALL);
}
-static int ux500_wdt_keepalive(struct watchdog_device *wdd)
+static int db8500_wdt_keepalive(struct watchdog_device *wdd)
{
return prcmu_kick_a9wdog(PRCMU_WDOG_ALL);
}
-static int ux500_wdt_set_timeout(struct watchdog_device *wdd,
+static int db8500_wdt_set_timeout(struct watchdog_device *wdd,
unsigned int timeout)
{
- ux500_wdt_stop(wdd);
+ db8500_wdt_stop(wdd);
prcmu_load_a9wdog(PRCMU_WDOG_ALL, timeout * 1000);
- ux500_wdt_start(wdd);
+ db8500_wdt_start(wdd);
return 0;
}
-static const struct watchdog_info ux500_wdt_info = {
+static const struct watchdog_info db8500_wdt_info = {
.options = WDIOF_SETTIMEOUT | WDIOF_KEEPALIVEPING | WDIOF_MAGICCLOSE,
- .identity = "Ux500 WDT",
+ .identity = "DB8500 WDT",
.firmware_version = 1,
};
-static const struct watchdog_ops ux500_wdt_ops = {
+static const struct watchdog_ops db8500_wdt_ops = {
.owner = THIS_MODULE,
- .start = ux500_wdt_start,
- .stop = ux500_wdt_stop,
- .ping = ux500_wdt_keepalive,
- .set_timeout = ux500_wdt_set_timeout,
+ .start = db8500_wdt_start,
+ .stop = db8500_wdt_stop,
+ .ping = db8500_wdt_keepalive,
+ .set_timeout = db8500_wdt_set_timeout,
};
-static struct watchdog_device ux500_wdt = {
- .info = &ux500_wdt_info,
- .ops = &ux500_wdt_ops,
+static struct watchdog_device db8500_wdt = {
+ .info = &db8500_wdt_info,
+ .ops = &db8500_wdt_ops,
.min_timeout = WATCHDOG_MIN,
- .max_timeout = WATCHDOG_MAX32,
+ .max_timeout = WATCHDOG_MAX28,
};
-static int ux500_wdt_probe(struct platform_device *pdev)
+static int db8500_wdt_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
int ret;
- struct ux500_wdt_data *pdata = dev_get_platdata(dev);
- if (pdata) {
- if (pdata->timeout > 0)
- timeout = pdata->timeout;
- if (pdata->has_28_bits_resolution)
- ux500_wdt.max_timeout = WATCHDOG_MAX28;
- }
-
- ux500_wdt.parent = dev;
- watchdog_set_nowayout(&ux500_wdt, nowayout);
+ timeout = 600; /* Default to 10 minutes */
+ db8500_wdt.parent = dev;
+ watchdog_set_nowayout(&db8500_wdt, nowayout);
/* disable auto off on sleep */
prcmu_config_a9wdog(PRCMU_WDOG_CPU1, false);
/* set HW initial value */
prcmu_load_a9wdog(PRCMU_WDOG_ALL, timeout * 1000);
- ret = devm_watchdog_register_device(dev, &ux500_wdt);
+ ret = devm_watchdog_register_device(dev, &db8500_wdt);
if (ret)
return ret;
}
#ifdef CONFIG_PM
-static int ux500_wdt_suspend(struct platform_device *pdev,
+static int db8500_wdt_suspend(struct platform_device *pdev,
pm_message_t state)
{
- if (watchdog_active(&ux500_wdt)) {
- ux500_wdt_stop(&ux500_wdt);
+ if (watchdog_active(&db8500_wdt)) {
+ db8500_wdt_stop(&db8500_wdt);
prcmu_config_a9wdog(PRCMU_WDOG_CPU1, true);
prcmu_load_a9wdog(PRCMU_WDOG_ALL, timeout * 1000);
- ux500_wdt_start(&ux500_wdt);
+ db8500_wdt_start(&db8500_wdt);
}
return 0;
}
-static int ux500_wdt_resume(struct platform_device *pdev)
+static int db8500_wdt_resume(struct platform_device *pdev)
{
- if (watchdog_active(&ux500_wdt)) {
- ux500_wdt_stop(&ux500_wdt);
+ if (watchdog_active(&db8500_wdt)) {
+ db8500_wdt_stop(&db8500_wdt);
prcmu_config_a9wdog(PRCMU_WDOG_CPU1, false);
prcmu_load_a9wdog(PRCMU_WDOG_ALL, timeout * 1000);
- ux500_wdt_start(&ux500_wdt);
+ db8500_wdt_start(&db8500_wdt);
}
return 0;
}
#else
-#define ux500_wdt_suspend NULL
-#define ux500_wdt_resume NULL
+#define db8500_wdt_suspend NULL
+#define db8500_wdt_resume NULL
#endif
-static struct platform_driver ux500_wdt_driver = {
- .probe = ux500_wdt_probe,
- .suspend = ux500_wdt_suspend,
- .resume = ux500_wdt_resume,
+static struct platform_driver db8500_wdt_driver = {
+ .probe = db8500_wdt_probe,
+ .suspend = db8500_wdt_suspend,
+ .resume = db8500_wdt_resume,
.driver = {
- .name = "ux500_wdt",
+ .name = "db8500_wdt",
},
};
-module_platform_driver(ux500_wdt_driver);
+module_platform_driver(db8500_wdt_driver);
MODULE_AUTHOR("Jonas Aaberg <jonas.aberg@stericsson.com>");
-MODULE_DESCRIPTION("Ux500 Watchdog Driver");
+MODULE_DESCRIPTION("DB8500 Watchdog Driver");
MODULE_LICENSE("GPL");
-MODULE_ALIAS("platform:ux500_wdt");
+MODULE_ALIAS("platform:db8500_wdt");
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/err.h>
-#include <linux/fs.h>
#include <linux/init.h>
#include <linux/io.h>
#include <linux/ioport.h>
-#include <linux/miscdevice.h>
#include <linux/module.h>
-#include <linux/mutex.h>
-#include <linux/notifier.h>
-#include <linux/reboot.h>
-#include <linux/uaccess.h>
+#include <linux/platform_device.h>
#include <linux/watchdog.h>
#define DRVNAME "f71808e_wdt"
module_param(force_id, ushort, 0);
MODULE_PARM_DESC(force_id, "Override the detected device ID");
-static const int max_timeout = WATCHDOG_MAX_TIMEOUT;
static int timeout = WATCHDOG_TIMEOUT; /* default timeout in seconds */
module_param(timeout, int, 0);
MODULE_PARM_DESC(timeout,
enum chips { f71808fg, f71858fg, f71862fg, f71868, f71869, f71882fg, f71889fg,
f81803, f81865, f81866};
-static const char *f71808e_names[] = {
+static const char * const fintek_wdt_names[] = {
"f71808fg",
"f71858fg",
"f71862fg",
static inline void superio_select(int base, int ld);
static inline void superio_exit(int base);
-struct watchdog_data {
+struct fintek_wdt {
+ struct watchdog_device wdd;
unsigned short sioaddr;
enum chips type;
- unsigned long opened;
- struct mutex lock;
- char expect_close;
struct watchdog_info ident;
- unsigned short timeout;
u8 timer_val; /* content for the wd_time register */
char minutes_mode;
u8 pulse_val; /* pulse width flag */
char pulse_mode; /* enable pulse output mode? */
- char caused_reboot; /* last reboot was by the watchdog */
};
-static struct watchdog_data watchdog = {
- .lock = __MUTEX_INITIALIZER(watchdog.lock),
+struct fintek_wdt_pdata {
+ enum chips type;
};
/* Super I/O functions */
release_region(base, 2);
}
-static int watchdog_set_timeout(int timeout)
+static int fintek_wdt_set_timeout(struct watchdog_device *wdd, unsigned int timeout)
{
- if (timeout <= 0
- || timeout > max_timeout) {
- pr_err("watchdog timeout out of range\n");
- return -EINVAL;
- }
-
- mutex_lock(&watchdog.lock);
+ struct fintek_wdt *wd = watchdog_get_drvdata(wdd);
- watchdog.timeout = timeout;
if (timeout > 0xff) {
- watchdog.timer_val = DIV_ROUND_UP(timeout, 60);
- watchdog.minutes_mode = true;
+ wd->timer_val = DIV_ROUND_UP(timeout, 60);
+ wd->minutes_mode = true;
+ timeout = wd->timer_val * 60;
} else {
- watchdog.timer_val = timeout;
- watchdog.minutes_mode = false;
+ wd->timer_val = timeout;
+ wd->minutes_mode = false;
}
- mutex_unlock(&watchdog.lock);
+ wdd->timeout = timeout;
return 0;
}
-static int watchdog_set_pulse_width(unsigned int pw)
+static int fintek_wdt_set_pulse_width(struct fintek_wdt *wd, unsigned int pw)
{
- int err = 0;
unsigned int t1 = 25, t2 = 125, t3 = 5000;
- if (watchdog.type == f71868) {
+ if (wd->type == f71868) {
t1 = 30;
t2 = 150;
t3 = 6000;
}
- mutex_lock(&watchdog.lock);
-
if (pw <= 1) {
- watchdog.pulse_val = 0;
+ wd->pulse_val = 0;
} else if (pw <= t1) {
- watchdog.pulse_val = 1;
+ wd->pulse_val = 1;
} else if (pw <= t2) {
- watchdog.pulse_val = 2;
+ wd->pulse_val = 2;
} else if (pw <= t3) {
- watchdog.pulse_val = 3;
+ wd->pulse_val = 3;
} else {
pr_err("pulse width out of range\n");
- err = -EINVAL;
- goto exit_unlock;
+ return -EINVAL;
}
- watchdog.pulse_mode = pw;
+ wd->pulse_mode = pw;
-exit_unlock:
- mutex_unlock(&watchdog.lock);
- return err;
+ return 0;
}
-static int watchdog_keepalive(void)
+static int fintek_wdt_keepalive(struct watchdog_device *wdd)
{
- int err = 0;
+ struct fintek_wdt *wd = watchdog_get_drvdata(wdd);
+ int err;
- mutex_lock(&watchdog.lock);
- err = superio_enter(watchdog.sioaddr);
+ err = superio_enter(wd->sioaddr);
if (err)
- goto exit_unlock;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
+ return err;
+ superio_select(wd->sioaddr, SIO_F71808FG_LD_WDT);
- if (watchdog.minutes_mode)
+ if (wd->minutes_mode)
/* select minutes for timer units */
- superio_set_bit(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
+ superio_set_bit(wd->sioaddr, F71808FG_REG_WDT_CONF,
F71808FG_FLAG_WD_UNIT);
else
/* select seconds for timer units */
- superio_clear_bit(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
+ superio_clear_bit(wd->sioaddr, F71808FG_REG_WDT_CONF,
F71808FG_FLAG_WD_UNIT);
/* Set timer value */
- superio_outb(watchdog.sioaddr, F71808FG_REG_WD_TIME,
- watchdog.timer_val);
+ superio_outb(wd->sioaddr, F71808FG_REG_WD_TIME,
+ wd->timer_val);
- superio_exit(watchdog.sioaddr);
+ superio_exit(wd->sioaddr);
-exit_unlock:
- mutex_unlock(&watchdog.lock);
- return err;
+ return 0;
}
-static int watchdog_start(void)
+static int fintek_wdt_start(struct watchdog_device *wdd)
{
+ struct fintek_wdt *wd = watchdog_get_drvdata(wdd);
int err;
u8 tmp;
/* Make sure we don't die as soon as the watchdog is enabled below */
- err = watchdog_keepalive();
+ err = fintek_wdt_keepalive(wdd);
if (err)
return err;
- mutex_lock(&watchdog.lock);
- err = superio_enter(watchdog.sioaddr);
+ err = superio_enter(wd->sioaddr);
if (err)
- goto exit_unlock;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
+ return err;
+ superio_select(wd->sioaddr, SIO_F71808FG_LD_WDT);
/* Watchdog pin configuration */
- switch (watchdog.type) {
+ switch (wd->type) {
case f71808fg:
/* Set pin 21 to GPIO23/WDTRST#, then to WDTRST# */
- superio_clear_bit(watchdog.sioaddr, SIO_REG_MFUNCT2, 3);
- superio_clear_bit(watchdog.sioaddr, SIO_REG_MFUNCT3, 3);
+ superio_clear_bit(wd->sioaddr, SIO_REG_MFUNCT2, 3);
+ superio_clear_bit(wd->sioaddr, SIO_REG_MFUNCT3, 3);
break;
case f71862fg:
if (f71862fg_pin == 63) {
/* SPI must be disabled first to use this pin! */
- superio_clear_bit(watchdog.sioaddr, SIO_REG_ROM_ADDR_SEL, 6);
- superio_set_bit(watchdog.sioaddr, SIO_REG_MFUNCT3, 4);
+ superio_clear_bit(wd->sioaddr, SIO_REG_ROM_ADDR_SEL, 6);
+ superio_set_bit(wd->sioaddr, SIO_REG_MFUNCT3, 4);
} else if (f71862fg_pin == 56) {
- superio_set_bit(watchdog.sioaddr, SIO_REG_MFUNCT1, 1);
+ superio_set_bit(wd->sioaddr, SIO_REG_MFUNCT1, 1);
}
break;
case f71868:
case f71869:
/* GPIO14 --> WDTRST# */
- superio_clear_bit(watchdog.sioaddr, SIO_REG_MFUNCT1, 4);
+ superio_clear_bit(wd->sioaddr, SIO_REG_MFUNCT1, 4);
break;
case f71882fg:
/* Set pin 56 to WDTRST# */
- superio_set_bit(watchdog.sioaddr, SIO_REG_MFUNCT1, 1);
+ superio_set_bit(wd->sioaddr, SIO_REG_MFUNCT1, 1);
break;
case f71889fg:
/* set pin 40 to WDTRST# */
- superio_outb(watchdog.sioaddr, SIO_REG_MFUNCT3,
- superio_inb(watchdog.sioaddr, SIO_REG_MFUNCT3) & 0xcf);
+ superio_outb(wd->sioaddr, SIO_REG_MFUNCT3,
+ superio_inb(wd->sioaddr, SIO_REG_MFUNCT3) & 0xcf);
break;
case f81803:
/* Enable TSI Level register bank */
- superio_clear_bit(watchdog.sioaddr, SIO_REG_CLOCK_SEL, 3);
+ superio_clear_bit(wd->sioaddr, SIO_REG_CLOCK_SEL, 3);
/* Set pin 27 to WDTRST# */
- superio_outb(watchdog.sioaddr, SIO_REG_TSI_LEVEL_SEL, 0x5f &
- superio_inb(watchdog.sioaddr, SIO_REG_TSI_LEVEL_SEL));
+ superio_outb(wd->sioaddr, SIO_REG_TSI_LEVEL_SEL, 0x5f &
+ superio_inb(wd->sioaddr, SIO_REG_TSI_LEVEL_SEL));
break;
case f81865:
/* Set pin 70 to WDTRST# */
- superio_clear_bit(watchdog.sioaddr, SIO_REG_MFUNCT3, 5);
+ superio_clear_bit(wd->sioaddr, SIO_REG_MFUNCT3, 5);
break;
case f81866:
* BIT5: 0 -> WDTRST#
* 1 -> GPIO15
*/
- tmp = superio_inb(watchdog.sioaddr, SIO_F81866_REG_PORT_SEL);
+ tmp = superio_inb(wd->sioaddr, SIO_F81866_REG_PORT_SEL);
tmp &= ~(BIT(3) | BIT(0));
tmp |= BIT(2);
- superio_outb(watchdog.sioaddr, SIO_F81866_REG_PORT_SEL, tmp);
+ superio_outb(wd->sioaddr, SIO_F81866_REG_PORT_SEL, tmp);
- superio_clear_bit(watchdog.sioaddr, SIO_F81866_REG_GPIO1, 5);
+ superio_clear_bit(wd->sioaddr, SIO_F81866_REG_GPIO1, 5);
break;
default:
goto exit_superio;
}
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
- superio_set_bit(watchdog.sioaddr, SIO_REG_ENABLE, 0);
+ superio_select(wd->sioaddr, SIO_F71808FG_LD_WDT);
+ superio_set_bit(wd->sioaddr, SIO_REG_ENABLE, 0);
- if (watchdog.type == f81865 || watchdog.type == f81866)
- superio_set_bit(watchdog.sioaddr, F81865_REG_WDO_CONF,
+ if (wd->type == f81865 || wd->type == f81866)
+ superio_set_bit(wd->sioaddr, F81865_REG_WDO_CONF,
F81865_FLAG_WDOUT_EN);
else
- superio_set_bit(watchdog.sioaddr, F71808FG_REG_WDO_CONF,
+ superio_set_bit(wd->sioaddr, F71808FG_REG_WDO_CONF,
F71808FG_FLAG_WDOUT_EN);
- superio_set_bit(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
+ superio_set_bit(wd->sioaddr, F71808FG_REG_WDT_CONF,
F71808FG_FLAG_WD_EN);
- if (watchdog.pulse_mode) {
+ if (wd->pulse_mode) {
/* Select "pulse" output mode with given duration */
- u8 wdt_conf = superio_inb(watchdog.sioaddr,
+ u8 wdt_conf = superio_inb(wd->sioaddr,
F71808FG_REG_WDT_CONF);
/* Set WD_PSWIDTH bits (1:0) */
- wdt_conf = (wdt_conf & 0xfc) | (watchdog.pulse_val & 0x03);
+ wdt_conf = (wdt_conf & 0xfc) | (wd->pulse_val & 0x03);
/* Set WD_PULSE to "pulse" mode */
wdt_conf |= BIT(F71808FG_FLAG_WD_PULSE);
- superio_outb(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
+ superio_outb(wd->sioaddr, F71808FG_REG_WDT_CONF,
wdt_conf);
} else {
/* Select "level" output mode */
- superio_clear_bit(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
+ superio_clear_bit(wd->sioaddr, F71808FG_REG_WDT_CONF,
F71808FG_FLAG_WD_PULSE);
}
exit_superio:
- superio_exit(watchdog.sioaddr);
-exit_unlock:
- mutex_unlock(&watchdog.lock);
+ superio_exit(wd->sioaddr);
return err;
}
-static int watchdog_stop(void)
-{
- int err = 0;
-
- mutex_lock(&watchdog.lock);
- err = superio_enter(watchdog.sioaddr);
- if (err)
- goto exit_unlock;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
-
- superio_clear_bit(watchdog.sioaddr, F71808FG_REG_WDT_CONF,
- F71808FG_FLAG_WD_EN);
-
- superio_exit(watchdog.sioaddr);
-
-exit_unlock:
- mutex_unlock(&watchdog.lock);
-
- return err;
-}
-
-static int watchdog_get_status(void)
-{
- int status = 0;
-
- mutex_lock(&watchdog.lock);
- status = (watchdog.caused_reboot) ? WDIOF_CARDRESET : 0;
- mutex_unlock(&watchdog.lock);
-
- return status;
-}
-
-static bool watchdog_is_running(void)
-{
- /*
- * if we fail to determine the watchdog's status assume it to be
- * running to be on the safe side
- */
- bool is_running = true;
-
- mutex_lock(&watchdog.lock);
- if (superio_enter(watchdog.sioaddr))
- goto exit_unlock;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
-
- is_running = (superio_inb(watchdog.sioaddr, SIO_REG_ENABLE) & BIT(0))
- && (superio_inb(watchdog.sioaddr, F71808FG_REG_WDT_CONF)
- & BIT(F71808FG_FLAG_WD_EN));
-
- superio_exit(watchdog.sioaddr);
-
-exit_unlock:
- mutex_unlock(&watchdog.lock);
- return is_running;
-}
-
-/* /dev/watchdog api */
-
-static int watchdog_open(struct inode *inode, struct file *file)
+static int fintek_wdt_stop(struct watchdog_device *wdd)
{
+ struct fintek_wdt *wd = watchdog_get_drvdata(wdd);
int err;
- /* If the watchdog is alive we don't need to start it again */
- if (test_and_set_bit(0, &watchdog.opened))
- return -EBUSY;
-
- err = watchdog_start();
- if (err) {
- clear_bit(0, &watchdog.opened);
+ err = superio_enter(wd->sioaddr);
+ if (err)
return err;
- }
-
- if (nowayout)
- __module_get(THIS_MODULE);
+ superio_select(wd->sioaddr, SIO_F71808FG_LD_WDT);
- watchdog.expect_close = 0;
- return stream_open(inode, file);
-}
+ superio_clear_bit(wd->sioaddr, F71808FG_REG_WDT_CONF,
+ F71808FG_FLAG_WD_EN);
-static int watchdog_release(struct inode *inode, struct file *file)
-{
- clear_bit(0, &watchdog.opened);
+ superio_exit(wd->sioaddr);
- if (!watchdog.expect_close) {
- watchdog_keepalive();
- pr_crit("Unexpected close, not stopping watchdog!\n");
- } else if (!nowayout) {
- watchdog_stop();
- }
return 0;
}
-/*
- * watchdog_write:
- * @file: file handle to the watchdog
- * @buf: buffer to write
- * @count: count of bytes
- * @ppos: pointer to the position to write. No seeks allowed
- *
- * A write to a watchdog device is defined as a keepalive signal. Any
- * write of data will do, as we we don't define content meaning.
- */
-
-static ssize_t watchdog_write(struct file *file, const char __user *buf,
- size_t count, loff_t *ppos)
+static bool fintek_wdt_is_running(struct fintek_wdt *wd, u8 wdt_conf)
{
- if (count) {
- if (!nowayout) {
- size_t i;
-
- /* In case it was set long ago */
- bool expect_close = false;
-
- for (i = 0; i != count; i++) {
- char c;
- if (get_user(c, buf + i))
- return -EFAULT;
- if (c == 'V')
- expect_close = true;
- }
-
- /* Properly order writes across fork()ed processes */
- mutex_lock(&watchdog.lock);
- watchdog.expect_close = expect_close;
- mutex_unlock(&watchdog.lock);
- }
-
- /* someone wrote to us, we should restart timer */
- watchdog_keepalive();
- }
- return count;
+ return (superio_inb(wd->sioaddr, SIO_REG_ENABLE) & BIT(0))
+ && (wdt_conf & BIT(F71808FG_FLAG_WD_EN));
}
-/*
- * watchdog_ioctl:
- * @inode: inode of the device
- * @file: file handle to the device
- * @cmd: watchdog command
- * @arg: argument pointer
- *
- * The watchdog API defines a common set of functions for all watchdogs
- * according to their available features.
- */
-static long watchdog_ioctl(struct file *file, unsigned int cmd,
- unsigned long arg)
-{
- int status;
- int new_options;
- int new_timeout;
- union {
- struct watchdog_info __user *ident;
- int __user *i;
- } uarg;
-
- uarg.i = (int __user *)arg;
-
- switch (cmd) {
- case WDIOC_GETSUPPORT:
- return copy_to_user(uarg.ident, &watchdog.ident,
- sizeof(watchdog.ident)) ? -EFAULT : 0;
-
- case WDIOC_GETSTATUS:
- status = watchdog_get_status();
- if (status < 0)
- return status;
- return put_user(status, uarg.i);
-
- case WDIOC_GETBOOTSTATUS:
- return put_user(0, uarg.i);
-
- case WDIOC_SETOPTIONS:
- if (get_user(new_options, uarg.i))
- return -EFAULT;
-
- if (new_options & WDIOS_DISABLECARD)
- watchdog_stop();
-
- if (new_options & WDIOS_ENABLECARD)
- return watchdog_start();
- fallthrough;
-
- case WDIOC_KEEPALIVE:
- watchdog_keepalive();
- return 0;
-
- case WDIOC_SETTIMEOUT:
- if (get_user(new_timeout, uarg.i))
- return -EFAULT;
-
- if (watchdog_set_timeout(new_timeout))
- return -EINVAL;
-
- watchdog_keepalive();
- fallthrough;
-
- case WDIOC_GETTIMEOUT:
- return put_user(watchdog.timeout, uarg.i);
-
- default:
- return -ENOTTY;
-
- }
-}
+static const struct watchdog_ops fintek_wdt_ops = {
+ .owner = THIS_MODULE,
+ .start = fintek_wdt_start,
+ .stop = fintek_wdt_stop,
+ .ping = fintek_wdt_keepalive,
+ .set_timeout = fintek_wdt_set_timeout,
+};
-static int watchdog_notify_sys(struct notifier_block *this, unsigned long code,
- void *unused)
+static int fintek_wdt_probe(struct platform_device *pdev)
{
- if (code == SYS_DOWN || code == SYS_HALT)
- watchdog_stop();
- return NOTIFY_DONE;
-}
+ struct device *dev = &pdev->dev;
+ struct fintek_wdt_pdata *pdata;
+ struct watchdog_device *wdd;
+ struct fintek_wdt *wd;
+ int wdt_conf, err = 0;
+ struct resource *res;
+ int sioaddr;
-static const struct file_operations watchdog_fops = {
- .owner = THIS_MODULE,
- .llseek = no_llseek,
- .open = watchdog_open,
- .release = watchdog_release,
- .write = watchdog_write,
- .unlocked_ioctl = watchdog_ioctl,
- .compat_ioctl = compat_ptr_ioctl,
-};
+ res = platform_get_resource(pdev, IORESOURCE_IO, 0);
+ if (!res)
+ return -ENXIO;
-static struct miscdevice watchdog_miscdev = {
- .minor = WATCHDOG_MINOR,
- .name = "watchdog",
- .fops = &watchdog_fops,
-};
+ sioaddr = res->start;
-static struct notifier_block watchdog_notifier = {
- .notifier_call = watchdog_notify_sys,
-};
+ wd = devm_kzalloc(dev, sizeof(*wd), GFP_KERNEL);
+ if (!wd)
+ return -ENOMEM;
-static int __init watchdog_init(int sioaddr)
-{
- int wdt_conf, err = 0;
+ pdata = dev->platform_data;
- /* No need to lock watchdog.lock here because no entry points
- * into the module have been registered yet.
- */
- watchdog.sioaddr = sioaddr;
- watchdog.ident.options = WDIOF_MAGICCLOSE
- | WDIOF_KEEPALIVEPING
- | WDIOF_CARDRESET;
+ wd->type = pdata->type;
+ wd->sioaddr = sioaddr;
+ wd->ident.options = WDIOF_SETTIMEOUT
+ | WDIOF_MAGICCLOSE
+ | WDIOF_KEEPALIVEPING
+ | WDIOF_CARDRESET;
- snprintf(watchdog.ident.identity,
- sizeof(watchdog.ident.identity), "%s watchdog",
- f71808e_names[watchdog.type]);
+ snprintf(wd->ident.identity,
+ sizeof(wd->ident.identity), "%s watchdog",
+ fintek_wdt_names[wd->type]);
err = superio_enter(sioaddr);
if (err)
return err;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
+ superio_select(wd->sioaddr, SIO_F71808FG_LD_WDT);
wdt_conf = superio_inb(sioaddr, F71808FG_REG_WDT_CONF);
- watchdog.caused_reboot = wdt_conf & BIT(F71808FG_FLAG_WDTMOUT_STS);
/*
* We don't want WDTMOUT_STS to stick around till regular reboot.
superio_outb(sioaddr, F71808FG_REG_WDT_CONF,
wdt_conf | BIT(F71808FG_FLAG_WDTMOUT_STS));
- superio_exit(sioaddr);
+ wdd = &wd->wdd;
- err = watchdog_set_timeout(timeout);
- if (err)
- return err;
- err = watchdog_set_pulse_width(pulse_width);
- if (err)
- return err;
+ if (fintek_wdt_is_running(wd, wdt_conf))
+ set_bit(WDOG_HW_RUNNING, &wdd->status);
- err = register_reboot_notifier(&watchdog_notifier);
- if (err)
- return err;
+ superio_exit(sioaddr);
- err = misc_register(&watchdog_miscdev);
- if (err) {
- pr_err("cannot register miscdev on minor=%d\n",
- watchdog_miscdev.minor);
- goto exit_reboot;
- }
+ wdd->parent = dev;
+ wdd->info = &wd->ident;
+ wdd->ops = &fintek_wdt_ops;
+ wdd->min_timeout = 1;
+ wdd->max_timeout = WATCHDOG_MAX_TIMEOUT;
- if (start_withtimeout) {
- if (start_withtimeout <= 0
- || start_withtimeout > max_timeout) {
- pr_err("starting timeout out of range\n");
- err = -EINVAL;
- goto exit_miscdev;
- }
+ watchdog_set_drvdata(wdd, wd);
+ watchdog_set_nowayout(wdd, nowayout);
+ watchdog_stop_on_unregister(wdd);
+ watchdog_stop_on_reboot(wdd);
+ watchdog_init_timeout(wdd, start_withtimeout ?: timeout, NULL);
- err = watchdog_start();
- if (err) {
- pr_err("cannot start watchdog timer\n");
- goto exit_miscdev;
- }
+ if (wdt_conf & BIT(F71808FG_FLAG_WDTMOUT_STS))
+ wdd->bootstatus = WDIOF_CARDRESET;
- mutex_lock(&watchdog.lock);
- err = superio_enter(sioaddr);
- if (err)
- goto exit_unlock;
- superio_select(watchdog.sioaddr, SIO_F71808FG_LD_WDT);
+ /*
+ * WATCHDOG_HANDLE_BOOT_ENABLED can result in keepalive being directly
+ * called without a set_timeout before, so it needs to be done here
+ * unconditionally.
+ */
+ fintek_wdt_set_timeout(wdd, wdd->timeout);
+ fintek_wdt_set_pulse_width(wd, pulse_width);
- if (start_withtimeout > 0xff) {
- /* select minutes for timer units */
- superio_set_bit(sioaddr, F71808FG_REG_WDT_CONF,
- F71808FG_FLAG_WD_UNIT);
- superio_outb(sioaddr, F71808FG_REG_WD_TIME,
- DIV_ROUND_UP(start_withtimeout, 60));
- } else {
- /* select seconds for timer units */
- superio_clear_bit(sioaddr, F71808FG_REG_WDT_CONF,
- F71808FG_FLAG_WD_UNIT);
- superio_outb(sioaddr, F71808FG_REG_WD_TIME,
- start_withtimeout);
+ if (start_withtimeout) {
+ err = fintek_wdt_start(wdd);
+ if (err) {
+ dev_err(dev, "cannot start watchdog timer\n");
+ return err;
}
- superio_exit(sioaddr);
- mutex_unlock(&watchdog.lock);
-
- if (nowayout)
- __module_get(THIS_MODULE);
-
- pr_info("watchdog started with initial timeout of %u sec\n",
- start_withtimeout);
+ set_bit(WDOG_HW_RUNNING, &wdd->status);
+ dev_info(dev, "watchdog started with initial timeout of %u sec\n",
+ start_withtimeout);
}
- return 0;
-
-exit_unlock:
- mutex_unlock(&watchdog.lock);
-exit_miscdev:
- misc_deregister(&watchdog_miscdev);
-exit_reboot:
- unregister_reboot_notifier(&watchdog_notifier);
-
- return err;
+ return devm_watchdog_register_device(dev, wdd);
}
-static int __init f71808e_find(int sioaddr)
+static int __init fintek_wdt_find(int sioaddr)
{
+ enum chips type;
u16 devid;
int err = superio_enter(sioaddr);
if (err)
devid = force_id ? force_id : superio_inw(sioaddr, SIO_REG_DEVID);
switch (devid) {
case SIO_F71808_ID:
- watchdog.type = f71808fg;
+ type = f71808fg;
break;
case SIO_F71862_ID:
- watchdog.type = f71862fg;
+ type = f71862fg;
break;
case SIO_F71868_ID:
- watchdog.type = f71868;
+ type = f71868;
break;
case SIO_F71869_ID:
case SIO_F71869A_ID:
- watchdog.type = f71869;
+ type = f71869;
break;
case SIO_F71882_ID:
- watchdog.type = f71882fg;
+ type = f71882fg;
break;
case SIO_F71889_ID:
- watchdog.type = f71889fg;
+ type = f71889fg;
break;
case SIO_F71858_ID:
/* Confirmed (by datasheet) not to have a watchdog. */
err = -ENODEV;
goto exit;
case SIO_F81803_ID:
- watchdog.type = f81803;
+ type = f81803;
break;
case SIO_F81865_ID:
- watchdog.type = f81865;
+ type = f81865;
break;
case SIO_F81866_ID:
- watchdog.type = f81866;
+ type = f81866;
break;
default:
pr_info("Unrecognized Fintek device: %04x\n",
}
pr_info("Found %s watchdog chip, revision %d\n",
- f71808e_names[watchdog.type],
+ fintek_wdt_names[type],
(int)superio_inb(sioaddr, SIO_REG_DEVREV));
+
exit:
superio_exit(sioaddr);
- return err;
+ return err ? err : type;
}
-static int __init f71808e_init(void)
+static struct platform_driver fintek_wdt_driver = {
+ .probe = fintek_wdt_probe,
+ .driver = {
+ .name = DRVNAME,
+ },
+};
+
+static struct platform_device *fintek_wdt_pdev;
+
+static int __init fintek_wdt_init(void)
{
static const unsigned short addrs[] = { 0x2e, 0x4e };
- int err = -ENODEV;
+ struct fintek_wdt_pdata pdata;
+ struct resource wdt_res = {};
+ int ret;
int i;
if (f71862fg_pin != 63 && f71862fg_pin != 56) {
}
for (i = 0; i < ARRAY_SIZE(addrs); i++) {
- err = f71808e_find(addrs[i]);
- if (err == 0)
+ ret = fintek_wdt_find(addrs[i]);
+ if (ret >= 0)
break;
}
if (i == ARRAY_SIZE(addrs))
- return err;
+ return ret;
- return watchdog_init(addrs[i]);
+ pdata.type = ret;
+
+ platform_driver_register(&fintek_wdt_driver);
+
+ wdt_res.name = "superio port";
+ wdt_res.flags = IORESOURCE_IO;
+ wdt_res.start = addrs[i];
+ wdt_res.end = addrs[i] + 1;
+
+ fintek_wdt_pdev = platform_device_register_resndata(NULL, DRVNAME, -1,
+ &wdt_res, 1,
+ &pdata, sizeof(pdata));
+ if (IS_ERR(fintek_wdt_pdev)) {
+ platform_driver_unregister(&fintek_wdt_driver);
+ return PTR_ERR(fintek_wdt_pdev);
+ }
+
+ return 0;
}
-static void __exit f71808e_exit(void)
+static void __exit fintek_wdt_exit(void)
{
- if (watchdog_is_running()) {
- pr_warn("Watchdog timer still running, stopping it\n");
- watchdog_stop();
- }
- misc_deregister(&watchdog_miscdev);
- unregister_reboot_notifier(&watchdog_notifier);
+ platform_device_unregister(fintek_wdt_pdev);
+ platform_driver_unregister(&fintek_wdt_driver);
}
MODULE_DESCRIPTION("F71808E Watchdog Driver");
MODULE_AUTHOR("Giel van Schijndel <me@mortis.eu>");
MODULE_LICENSE("GPL");
-module_init(f71808e_init);
-module_exit(f71808e_exit);
+module_init(fintek_wdt_init);
+module_exit(fintek_wdt_exit);
* NO_REBOOT flag is Memory-Mapped GCS register bit 5 (TCO version 2),
* or memory-mapped PMC register bit 4 (TCO version 3).
*/
- struct resource *gcs_pmc_res;
unsigned long __iomem *gcs_pmc;
/* the lock for io operations */
spinlock_t io_lock;
return time_left;
}
+static void iTCO_wdt_set_running(struct iTCO_wdt_private *p)
+{
+ u16 val;
+
+ /* Bit 11: TCO Timer Halt -> 0 = The TCO timer is * enabled */
+ val = inw(TCO1_CNT(p));
+ if (!(val & BIT(11)))
+ set_bit(WDOG_HW_RUNNING, &p->wddev.status);
+}
+
/*
* Kernel Interfaces
*/
*/
if (p->iTCO_version >= 2 && p->iTCO_version < 6 &&
!pdata->no_reboot_use_pmc) {
- p->gcs_pmc_res = platform_get_resource(pdev,
- IORESOURCE_MEM,
- ICH_RES_MEM_GCS_PMC);
- p->gcs_pmc = devm_ioremap_resource(dev, p->gcs_pmc_res);
+ p->gcs_pmc = devm_platform_ioremap_resource(pdev, ICH_RES_MEM_GCS_PMC);
if (IS_ERR(p->gcs_pmc))
return PTR_ERR(p->gcs_pmc);
}
watchdog_set_drvdata(&p->wddev, p);
platform_set_drvdata(pdev, p);
- /* Make sure the watchdog is not running */
- iTCO_wdt_stop(&p->wddev);
+ iTCO_wdt_set_running(p);
/* Check that the heartbeat value is within it's range;
if not reset to the default */
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * drivers/char/watchdog/iop_wdt.c
- *
- * WDT driver for Intel I/O Processors
- * Copyright (C) 2005, Intel Corporation.
- *
- * Based on ixp4xx driver, Copyright 2004 (c) MontaVista, Software, Inc.
- *
- * Curt E Bruns <curt.e.bruns@intel.com>
- * Peter Milne <peter.milne@d-tacq.com>
- * Dan Williams <dan.j.williams@intel.com>
- */
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/fs.h>
-#include <linux/init.h>
-#include <linux/device.h>
-#include <linux/miscdevice.h>
-#include <linux/watchdog.h>
-#include <linux/uaccess.h>
-#include <mach/hardware.h>
-
-static bool nowayout = WATCHDOG_NOWAYOUT;
-static unsigned long wdt_status;
-static unsigned long boot_status;
-static DEFINE_SPINLOCK(wdt_lock);
-
-#define WDT_IN_USE 0
-#define WDT_OK_TO_CLOSE 1
-#define WDT_ENABLED 2
-
-static unsigned long iop_watchdog_timeout(void)
-{
- return (0xffffffffUL / get_iop_tick_rate());
-}
-
-/**
- * wdt_supports_disable - determine if we are accessing a iop13xx watchdog
- * or iop3xx by whether it has a disable command
- */
-static int wdt_supports_disable(void)
-{
- int can_disable;
-
- if (IOP_WDTCR_EN_ARM != IOP_WDTCR_DIS_ARM)
- can_disable = 1;
- else
- can_disable = 0;
-
- return can_disable;
-}
-
-static void wdt_enable(void)
-{
- /* Arm and enable the Timer to starting counting down from 0xFFFF.FFFF
- * Takes approx. 10.7s to timeout
- */
- spin_lock(&wdt_lock);
- write_wdtcr(IOP_WDTCR_EN_ARM);
- write_wdtcr(IOP_WDTCR_EN);
- spin_unlock(&wdt_lock);
-}
-
-/* returns 0 if the timer was successfully disabled */
-static int wdt_disable(void)
-{
- /* Stop Counting */
- if (wdt_supports_disable()) {
- spin_lock(&wdt_lock);
- write_wdtcr(IOP_WDTCR_DIS_ARM);
- write_wdtcr(IOP_WDTCR_DIS);
- clear_bit(WDT_ENABLED, &wdt_status);
- spin_unlock(&wdt_lock);
- pr_info("Disabled\n");
- return 0;
- } else
- return 1;
-}
-
-static int iop_wdt_open(struct inode *inode, struct file *file)
-{
- if (test_and_set_bit(WDT_IN_USE, &wdt_status))
- return -EBUSY;
-
- clear_bit(WDT_OK_TO_CLOSE, &wdt_status);
- wdt_enable();
- set_bit(WDT_ENABLED, &wdt_status);
- return stream_open(inode, file);
-}
-
-static ssize_t iop_wdt_write(struct file *file, const char *data, size_t len,
- loff_t *ppos)
-{
- if (len) {
- if (!nowayout) {
- size_t i;
-
- clear_bit(WDT_OK_TO_CLOSE, &wdt_status);
-
- for (i = 0; i != len; i++) {
- char c;
-
- if (get_user(c, data + i))
- return -EFAULT;
- if (c == 'V')
- set_bit(WDT_OK_TO_CLOSE, &wdt_status);
- }
- }
- wdt_enable();
- }
- return len;
-}
-
-static const struct watchdog_info ident = {
- .options = WDIOF_CARDRESET | WDIOF_MAGICCLOSE | WDIOF_KEEPALIVEPING,
- .identity = "iop watchdog",
-};
-
-static long iop_wdt_ioctl(struct file *file,
- unsigned int cmd, unsigned long arg)
-{
- int options;
- int ret = -ENOTTY;
- int __user *argp = (int __user *)arg;
-
- switch (cmd) {
- case WDIOC_GETSUPPORT:
- if (copy_to_user(argp, &ident, sizeof(ident)))
- ret = -EFAULT;
- else
- ret = 0;
- break;
-
- case WDIOC_GETSTATUS:
- ret = put_user(0, argp);
- break;
-
- case WDIOC_GETBOOTSTATUS:
- ret = put_user(boot_status, argp);
- break;
-
- case WDIOC_SETOPTIONS:
- if (get_user(options, (int *)arg))
- return -EFAULT;
-
- if (options & WDIOS_DISABLECARD) {
- if (!nowayout) {
- if (wdt_disable() == 0) {
- set_bit(WDT_OK_TO_CLOSE, &wdt_status);
- ret = 0;
- } else
- ret = -ENXIO;
- } else
- ret = 0;
- }
- if (options & WDIOS_ENABLECARD) {
- wdt_enable();
- ret = 0;
- }
- break;
-
- case WDIOC_KEEPALIVE:
- wdt_enable();
- ret = 0;
- break;
-
- case WDIOC_GETTIMEOUT:
- ret = put_user(iop_watchdog_timeout(), argp);
- break;
- }
- return ret;
-}
-
-static int iop_wdt_release(struct inode *inode, struct file *file)
-{
- int state = 1;
- if (test_bit(WDT_OK_TO_CLOSE, &wdt_status))
- if (test_bit(WDT_ENABLED, &wdt_status))
- state = wdt_disable();
-
- /* if the timer is not disabled reload and notify that we are still
- * going down
- */
- if (state != 0) {
- wdt_enable();
- pr_crit("Device closed unexpectedly - reset in %lu seconds\n",
- iop_watchdog_timeout());
- }
-
- clear_bit(WDT_IN_USE, &wdt_status);
- clear_bit(WDT_OK_TO_CLOSE, &wdt_status);
-
- return 0;
-}
-
-static const struct file_operations iop_wdt_fops = {
- .owner = THIS_MODULE,
- .llseek = no_llseek,
- .write = iop_wdt_write,
- .unlocked_ioctl = iop_wdt_ioctl,
- .compat_ioctl = compat_ptr_ioctl,
- .open = iop_wdt_open,
- .release = iop_wdt_release,
-};
-
-static struct miscdevice iop_wdt_miscdev = {
- .minor = WATCHDOG_MINOR,
- .name = "watchdog",
- .fops = &iop_wdt_fops,
-};
-
-static int __init iop_wdt_init(void)
-{
- int ret;
-
- /* check if the reset was caused by the watchdog timer */
- boot_status = (read_rcsr() & IOP_RCSR_WDT) ? WDIOF_CARDRESET : 0;
-
- /* Configure Watchdog Timeout to cause an Internal Bus (IB) Reset
- * NOTE: An IB Reset will Reset both cores in the IOP342
- */
- write_wdtsr(IOP13XX_WDTCR_IB_RESET);
-
- /* Register after we have the device set up so we cannot race
- with an open */
- ret = misc_register(&iop_wdt_miscdev);
- if (ret == 0)
- pr_info("timeout %lu sec\n", iop_watchdog_timeout());
-
- return ret;
-}
-
-static void __exit iop_wdt_exit(void)
-{
- misc_deregister(&iop_wdt_miscdev);
-}
-
-module_init(iop_wdt_init);
-module_exit(iop_wdt_exit);
-
-module_param(nowayout, bool, 0);
-MODULE_PARM_DESC(nowayout, "Watchdog cannot be stopped once started");
-
-MODULE_AUTHOR("Curt E Bruns <curt.e.bruns@intel.com>");
-MODULE_DESCRIPTION("iop watchdog timer driver");
-MODULE_LICENSE("GPL");
#define GXBB_WDT_TCNT_SETUP_MASK (BIT(16) - 1)
#define GXBB_WDT_TCNT_CNT_SHIFT 16
+static bool nowayout = WATCHDOG_NOWAYOUT;
+module_param(nowayout, bool, 0);
+MODULE_PARM_DESC(nowayout, "Watchdog cannot be stopped once started default="
+ __MODULE_STRING(WATCHDOG_NOWAYOUT) ")");
+
+static unsigned int timeout;
+module_param(timeout, uint, 0);
+MODULE_PARM_DESC(timeout, "Watchdog heartbeat in seconds="
+ __MODULE_STRING(DEFAULT_TIMEOUT) ")");
+
struct meson_gxbb_wdt {
void __iomem *reg_base;
struct watchdog_device wdt_dev;
data->wdt_dev.max_hw_heartbeat_ms = GXBB_WDT_TCNT_SETUP_MASK;
data->wdt_dev.min_timeout = 1;
data->wdt_dev.timeout = DEFAULT_TIMEOUT;
+ watchdog_init_timeout(&data->wdt_dev, timeout, dev);
+ watchdog_set_nowayout(&data->wdt_dev, nowayout);
watchdog_set_drvdata(&data->wdt_dev, data);
/* Setup with 1ms timebase */
struct mlxreg_wdt *wdt = watchdog_get_drvdata(wdd);
struct mlxreg_core_data *reg_data = &wdt->pdata->data[wdt->ping_idx];
- return regmap_update_bits_base(wdt->regmap, reg_data->reg,
- ~reg_data->mask, BIT(reg_data->bit),
- NULL, false, true);
+ return regmap_write_bits(wdt->regmap, reg_data->reg, ~reg_data->mask,
+ BIT(reg_data->bit));
}
static int mlxreg_wdt_set_timeout(struct watchdog_device *wdd,
void __iomem *wdt_base;
spinlock_t lock; /* protects WDT_SWSYSRST reg */
struct reset_controller_dev rcdev;
+ bool disable_wdt_extrst;
};
struct mtk_wdt_data {
reg |= (WDT_MODE_IRQ_EN | WDT_MODE_DUAL_EN);
else
reg &= ~(WDT_MODE_IRQ_EN | WDT_MODE_DUAL_EN);
+ if (mtk_wdt->disable_wdt_extrst)
+ reg &= ~WDT_MODE_EXRST_EN;
reg |= (WDT_MODE_EN | WDT_MODE_KEY);
iowrite32(reg, wdt_base + WDT_MODE);
if (err)
return err;
}
+
+ mtk_wdt->disable_wdt_extrst =
+ of_property_read_bool(dev->of_node, "mediatek,disable-extrst");
+
return 0;
}
{
int ret = 0;
struct device *dev = &pdev->dev;
- struct resource *wdt_mem;
struct watchdog_device *wdd;
struct rti_wdt_device *wdt;
struct clk *clk;
watchdog_set_nowayout(wdd, 1);
watchdog_set_restart_priority(wdd, 128);
- wdt_mem = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- wdt->base = devm_ioremap_resource(dev, wdt_mem);
+ wdt->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(wdt->base)) {
ret = PTR_ERR(wdt->base);
goto err_iomap;
return -ENOENT;
}
- priv->wdev.info = &rza_wdt_ident,
- priv->wdev.ops = &rza_wdt_ops,
+ priv->wdev.info = &rza_wdt_ident;
+ priv->wdev.ops = &rza_wdt_ops;
priv->wdev.parent = dev;
priv->cks = (u8)(uintptr_t) of_device_get_match_data(dev);
* https://www.kernelconcepts.de
*
* See AMD Publication 43009 "AMD SB700/710/750 Register Reference Guide",
+ * AMD Publication 44413 "AMD SP5100 Register Reference Guide"
* AMD Publication 45482 "AMD SB800-Series Southbridges Register
* Reference Guide"
* AMD Publication 48751 "BIOS and Kernel Developer’s Guide (BKDG)
return 0;
}
+static unsigned int tco_timer_get_timeleft(struct watchdog_device *wdd)
+{
+ struct sp5100_tco *tco = watchdog_get_drvdata(wdd);
+
+ return readl(SP5100_WDT_COUNT(tco->tcobase));
+}
+
static u8 sp5100_tco_read_pm_reg8(u8 index)
{
outb(index, SP5100_IO_PM_INDEX_REG);
.stop = tco_timer_stop,
.ping = tco_timer_ping,
.set_timeout = tco_timer_set_timeout,
+ .get_timeleft = tco_timer_get_timeleft,
};
static int sp5100_tco_probe(struct platform_device *pdev)
/* This is the timer base. */
wdt->regs = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(wdt->regs)) {
- dev_err(dev, "Could not get resource\n");
+ if (IS_ERR(wdt->regs))
return PTR_ERR(wdt->regs);
- }
ret = stm32_iwdg_clk_init(pdev, wdt);
if (ret)
u8 wdt_timeout_shift;
u8 wdt_reset_mask;
u8 wdt_reset_val;
+ u32 wdt_key_val;
};
struct sunxi_wdt_dev {
val = readl(wdt_base + regs->wdt_cfg);
val &= ~(regs->wdt_reset_mask);
val |= regs->wdt_reset_val;
+ val |= regs->wdt_key_val;
writel(val, wdt_base + regs->wdt_cfg);
/* Set lowest timeout and enable watchdog */
val = readl(wdt_base + regs->wdt_mode);
val &= ~(WDT_TIMEOUT_MASK << regs->wdt_timeout_shift);
val |= WDT_MODE_EN;
+ val |= regs->wdt_key_val;
writel(val, wdt_base + regs->wdt_mode);
/*
mdelay(5);
val = readl(wdt_base + regs->wdt_mode);
val |= WDT_MODE_EN;
+ val |= regs->wdt_key_val;
writel(val, wdt_base + regs->wdt_mode);
}
return 0;
reg = readl(wdt_base + regs->wdt_mode);
reg &= ~(WDT_TIMEOUT_MASK << regs->wdt_timeout_shift);
reg |= wdt_timeout_map[timeout] << regs->wdt_timeout_shift;
+ reg |= regs->wdt_key_val;
writel(reg, wdt_base + regs->wdt_mode);
sunxi_wdt_ping(wdt_dev);
void __iomem *wdt_base = sunxi_wdt->wdt_base;
const struct sunxi_wdt_reg *regs = sunxi_wdt->wdt_regs;
- writel(0, wdt_base + regs->wdt_mode);
+ writel(regs->wdt_key_val, wdt_base + regs->wdt_mode);
return 0;
}
reg = readl(wdt_base + regs->wdt_cfg);
reg &= ~(regs->wdt_reset_mask);
reg |= regs->wdt_reset_val;
+ reg |= regs->wdt_key_val;
writel(reg, wdt_base + regs->wdt_cfg);
/* Enable watchdog */
reg = readl(wdt_base + regs->wdt_mode);
reg |= WDT_MODE_EN;
+ reg |= regs->wdt_key_val;
writel(reg, wdt_base + regs->wdt_mode);
return 0;
.wdt_reset_val = 0x01,
};
+static const struct sunxi_wdt_reg sun20i_wdt_reg = {
+ .wdt_ctrl = 0x10,
+ .wdt_cfg = 0x14,
+ .wdt_mode = 0x18,
+ .wdt_timeout_shift = 4,
+ .wdt_reset_mask = 0x03,
+ .wdt_reset_val = 0x01,
+ .wdt_key_val = 0x16aa0000,
+};
+
static const struct of_device_id sunxi_wdt_dt_ids[] = {
{ .compatible = "allwinner,sun4i-a10-wdt", .data = &sun4i_wdt_reg },
{ .compatible = "allwinner,sun6i-a31-wdt", .data = &sun6i_wdt_reg },
+ { .compatible = "allwinner,sun20i-d1-wdt", .data = &sun20i_wdt_reg },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, sunxi_wdt_dt_ids);
select DMA_OPS
select SWIOTLB
+config XEN_PCI_STUB
+ bool
+
+config XEN_PCIDEV_STUB
+ tristate "Xen PCI-device stub driver"
+ depends on PCI && !X86 && XEN
+ depends on XEN_BACKEND
+ select XEN_PCI_STUB
+ default m
+ help
+ The PCI device stub driver provides limited version of the PCI
+ device backend driver without para-virtualized support for guests.
+ If you select this to be a module, you will need to make sure no
+ other driver has bound to the device(s) you want to make visible to
+ other guests.
+
+ The "hide" parameter (only applicable if backend driver is compiled
+ into the kernel) allows you to bind the PCI devices to this module
+ from the default device drivers. The argument is the list of PCI BDFs:
+ xen-pciback.hide=(03:00.0)(04:00.0)
+
+ If in doubt, say m.
+
config XEN_PCIDEV_BACKEND
tristate "Xen PCI-device backend driver"
depends on PCI && X86 && XEN
depends on XEN_BACKEND
+ select XEN_PCI_STUB
default m
help
The PCI device backend driver allows the kernel to export arbitrary
obj-$(CONFIG_XEN_PVHVM_GUEST) += platform-pci.o
obj-$(CONFIG_SWIOTLB_XEN) += swiotlb-xen.o
obj-$(CONFIG_XEN_MCE_LOG) += mcelog.o
-obj-$(CONFIG_XEN_PCIDEV_BACKEND) += xen-pciback/
+obj-$(CONFIG_XEN_PCI_STUB) += xen-pciback/
obj-$(CONFIG_XEN_PRIVCMD) += xen-privcmd.o
obj-$(CONFIG_XEN_ACPI_PROCESSOR) += xen-acpi-processor.o
obj-$(CONFIG_XEN_EFI) += efi.o
#include <linux/percpu-defs.h>
#include <linux/slab.h>
#include <linux/sysctl.h>
+#include <linux/moduleparam.h>
#include <asm/page.h>
#include <asm/tlb.h>
#include <xen/page.h>
#include <xen/mem-reservation.h>
-static int xen_hotplug_unpopulated;
+#undef MODULE_PARAM_PREFIX
+#define MODULE_PARAM_PREFIX "xen."
+
+static uint __read_mostly balloon_boot_timeout = 180;
+module_param(balloon_boot_timeout, uint, 0444);
#ifdef CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
+static int xen_hotplug_unpopulated;
static struct ctl_table balloon_table[] = {
{
{ }
};
+#else
+#define xen_hotplug_unpopulated 0
#endif
/*
* BP_ECANCELED: error, balloon operation canceled.
*/
-enum bp_state {
+static enum bp_state {
BP_DONE,
BP_WAIT,
BP_EAGAIN,
BP_ECANCELED
-};
+} balloon_state = BP_DONE;
/* Main waiting point for xen-balloon thread. */
static DECLARE_WAIT_QUEUE_HEAD(balloon_thread_wq);
return list_entry(next, struct page, lru);
}
-static enum bp_state update_schedule(enum bp_state state)
+static void update_schedule(void)
{
- if (state == BP_WAIT)
- return BP_WAIT;
-
- if (state == BP_ECANCELED)
- return BP_ECANCELED;
+ if (balloon_state == BP_WAIT || balloon_state == BP_ECANCELED)
+ return;
- if (state == BP_DONE) {
+ if (balloon_state == BP_DONE) {
balloon_stats.schedule_delay = 1;
balloon_stats.retry_count = 1;
- return BP_DONE;
+ return;
}
++balloon_stats.retry_count;
balloon_stats.retry_count > balloon_stats.max_retry_count) {
balloon_stats.schedule_delay = 1;
balloon_stats.retry_count = 1;
- return BP_ECANCELED;
+ balloon_state = BP_ECANCELED;
+ return;
}
balloon_stats.schedule_delay <<= 1;
if (balloon_stats.schedule_delay > balloon_stats.max_schedule_delay)
balloon_stats.schedule_delay = balloon_stats.max_schedule_delay;
- return BP_EAGAIN;
+ balloon_state = BP_EAGAIN;
}
#ifdef CONFIG_XEN_BALLOON_MEMORY_HOTPLUG
* Stop waiting if either state is BP_DONE and ballooning action is
* needed, or if the credit has changed while state is not BP_DONE.
*/
-static bool balloon_thread_cond(enum bp_state state, long credit)
+static bool balloon_thread_cond(long credit)
{
- if (state == BP_DONE)
+ if (balloon_state == BP_DONE)
credit = 0;
return current_credit() != credit || kthread_should_stop();
*/
static int balloon_thread(void *unused)
{
- enum bp_state state = BP_DONE;
long credit;
unsigned long timeout;
set_freezable();
for (;;) {
- switch (state) {
+ switch (balloon_state) {
case BP_DONE:
case BP_ECANCELED:
timeout = 3600 * HZ;
credit = current_credit();
wait_event_freezable_timeout(balloon_thread_wq,
- balloon_thread_cond(state, credit), timeout);
+ balloon_thread_cond(credit), timeout);
if (kthread_should_stop())
return 0;
if (credit > 0) {
if (balloon_is_inflated())
- state = increase_reservation(credit);
+ balloon_state = increase_reservation(credit);
else
- state = reserve_additional_memory();
+ balloon_state = reserve_additional_memory();
}
if (credit < 0) {
long n_pages;
n_pages = min(-credit, si_mem_available());
- state = decrease_reservation(n_pages, GFP_BALLOON);
- if (state == BP_DONE && n_pages != -credit &&
+ balloon_state = decrease_reservation(n_pages,
+ GFP_BALLOON);
+ if (balloon_state == BP_DONE && n_pages != -credit &&
n_pages < totalreserve_pages)
- state = BP_EAGAIN;
+ balloon_state = BP_EAGAIN;
}
- state = update_schedule(state);
+ update_schedule();
mutex_unlock(&balloon_mutex);
}
EXPORT_SYMBOL_GPL(balloon_set_new_target);
-static int add_ballooned_pages(int nr_pages)
+#ifndef CONFIG_XEN_UNPOPULATED_ALLOC
+static int add_ballooned_pages(unsigned int nr_pages)
{
enum bp_state st;
}
/**
- * alloc_xenballooned_pages - get pages that have been ballooned out
+ * xen_alloc_unpopulated_pages - get pages that have been ballooned out
* @nr_pages: Number of pages to get
* @pages: pages returned
* @return 0 on success, error otherwise
*/
-int alloc_xenballooned_pages(int nr_pages, struct page **pages)
+int xen_alloc_unpopulated_pages(unsigned int nr_pages, struct page **pages)
{
- int pgno = 0;
+ unsigned int pgno = 0;
struct page *page;
int ret;
return 0;
out_undo:
mutex_unlock(&balloon_mutex);
- free_xenballooned_pages(pgno, pages);
+ xen_free_unpopulated_pages(pgno, pages);
/*
* NB: free_xenballooned_pages will only subtract pgno pages, but since
* target_unpopulated is incremented with nr_pages at the start we need
balloon_stats.target_unpopulated -= nr_pages - pgno;
return ret;
}
-EXPORT_SYMBOL(alloc_xenballooned_pages);
+EXPORT_SYMBOL(xen_alloc_unpopulated_pages);
/**
- * free_xenballooned_pages - return pages retrieved with get_ballooned_pages
+ * xen_free_unpopulated_pages - return pages retrieved with get_ballooned_pages
* @nr_pages: Number of pages
* @pages: pages to return
*/
-void free_xenballooned_pages(int nr_pages, struct page **pages)
+void xen_free_unpopulated_pages(unsigned int nr_pages, struct page **pages)
{
- int i;
+ unsigned int i;
mutex_lock(&balloon_mutex);
mutex_unlock(&balloon_mutex);
}
-EXPORT_SYMBOL(free_xenballooned_pages);
+EXPORT_SYMBOL(xen_free_unpopulated_pages);
-#if defined(CONFIG_XEN_PV) && !defined(CONFIG_XEN_UNPOPULATED_ALLOC)
+#if defined(CONFIG_XEN_PV)
static void __init balloon_add_region(unsigned long start_pfn,
unsigned long pages)
{
balloon_stats.total_pages += extra_pfn_end - start_pfn;
}
#endif
+#endif
static int __init balloon_init(void)
{
return 0;
}
subsys_initcall(balloon_init);
+
+static int __init balloon_wait_finish(void)
+{
+ long credit, last_credit = 0;
+ unsigned long last_changed = 0;
+
+ if (!xen_domain())
+ return -ENODEV;
+
+ /* PV guests don't need to wait. */
+ if (xen_pv_domain() || !current_credit())
+ return 0;
+
+ pr_notice("Waiting for initial ballooning down having finished.\n");
+
+ while ((credit = current_credit()) < 0) {
+ if (credit != last_credit) {
+ last_changed = jiffies;
+ last_credit = credit;
+ }
+ if (balloon_state == BP_ECANCELED) {
+ pr_warn_once("Initial ballooning failed, %ld pages need to be freed.\n",
+ -credit);
+ if (jiffies - last_changed >= HZ * balloon_boot_timeout)
+ panic("Initial ballooning failed!\n");
+ }
+
+ schedule_timeout_interruptible(HZ / 10);
+ }
+
+ pr_notice("Initial ballooning down finished.\n");
+
+ return 0;
+}
+late_initcall_sync(balloon_wait_finish);
for (i = 0; i < count; i++) {
struct page *page = pages[i];
unsigned long pfn = page_to_pfn(page);
+ int ret;
BUG_ON(!page);
set_phys_to_machine(pfn, frames[i]);
- /* Link back into the page tables if not highmem. */
- if (!PageHighMem(page)) {
- int ret;
-
- ret = HYPERVISOR_update_va_mapping(
- (unsigned long)__va(pfn << PAGE_SHIFT),
- mfn_pte(frames[i], PAGE_KERNEL),
- 0);
- BUG_ON(ret);
- }
+ ret = HYPERVISOR_update_va_mapping(
+ (unsigned long)__va(pfn << PAGE_SHIFT),
+ mfn_pte(frames[i], PAGE_KERNEL), 0);
+ BUG_ON(ret);
}
}
EXPORT_SYMBOL_GPL(__xenmem_reservation_va_mapping_update);
for (i = 0; i < count; i++) {
struct page *page = pages[i];
unsigned long pfn = page_to_pfn(page);
+ int ret;
/*
* We don't support PV MMU when Linux and Xen are using
*/
BUILD_BUG_ON(XEN_PAGE_SIZE != PAGE_SIZE);
- if (!PageHighMem(page)) {
- int ret;
+ ret = HYPERVISOR_update_va_mapping(
+ (unsigned long)__va(pfn << PAGE_SHIFT),
+ __pte_ma(0), 0);
+ BUG_ON(ret);
- ret = HYPERVISOR_update_va_mapping(
- (unsigned long)__va(pfn << PAGE_SHIFT),
- __pte_ma(0), 0);
- BUG_ON(ret);
- }
__set_phys_to_machine(pfn, INVALID_P2M_ENTRY);
}
}
#include <linux/pci.h>
#include <linux/acpi.h>
#include <linux/pci-acpi.h>
+#include <xen/pci.h>
#include <xen/xen.h>
#include <xen/interface/physdev.h>
#include <xen/interface/xen.h>
return 0;
}
#endif
+
+#ifdef CONFIG_XEN_DOM0
+struct xen_device_domain_owner {
+ domid_t domain;
+ struct pci_dev *dev;
+ struct list_head list;
+};
+
+static DEFINE_SPINLOCK(dev_domain_list_spinlock);
+static struct list_head dev_domain_list = LIST_HEAD_INIT(dev_domain_list);
+
+static struct xen_device_domain_owner *find_device(struct pci_dev *dev)
+{
+ struct xen_device_domain_owner *owner;
+
+ list_for_each_entry(owner, &dev_domain_list, list) {
+ if (owner->dev == dev)
+ return owner;
+ }
+ return NULL;
+}
+
+int xen_find_device_domain_owner(struct pci_dev *dev)
+{
+ struct xen_device_domain_owner *owner;
+ int domain = -ENODEV;
+
+ spin_lock(&dev_domain_list_spinlock);
+ owner = find_device(dev);
+ if (owner)
+ domain = owner->domain;
+ spin_unlock(&dev_domain_list_spinlock);
+ return domain;
+}
+EXPORT_SYMBOL_GPL(xen_find_device_domain_owner);
+
+int xen_register_device_domain_owner(struct pci_dev *dev, uint16_t domain)
+{
+ struct xen_device_domain_owner *owner;
+
+ owner = kzalloc(sizeof(struct xen_device_domain_owner), GFP_KERNEL);
+ if (!owner)
+ return -ENODEV;
+
+ spin_lock(&dev_domain_list_spinlock);
+ if (find_device(dev)) {
+ spin_unlock(&dev_domain_list_spinlock);
+ kfree(owner);
+ return -EEXIST;
+ }
+ owner->domain = domain;
+ owner->dev = dev;
+ list_add_tail(&owner->list, &dev_domain_list);
+ spin_unlock(&dev_domain_list_spinlock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xen_register_device_domain_owner);
+
+int xen_unregister_device_domain_owner(struct pci_dev *dev)
+{
+ struct xen_device_domain_owner *owner;
+
+ spin_lock(&dev_domain_list_spinlock);
+ owner = find_device(dev);
+ if (!owner) {
+ spin_unlock(&dev_domain_list_spinlock);
+ return -ENODEV;
+ }
+ list_del(&owner->list);
+ spin_unlock(&dev_domain_list_spinlock);
+ kfree(owner);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xen_unregister_device_domain_owner);
+#endif
write_unlock_bh(&mappass->sock->sk->sk_callback_lock);
}
sock_release(mappass->sock);
- flush_workqueue(mappass->wq);
destroy_workqueue(mappass->wq);
kfree(mappass);
*/
rc = xen_swiotlb_fixup(start, nslabs);
if (rc) {
- memblock_free(__pa(start), PAGE_ALIGN(bytes));
+ memblock_free(start, PAGE_ALIGN(bytes));
if (nslabs > 1024 && repeat--) {
/* Min is 2MB */
nslabs = max(1024UL, ALIGN(nslabs >> 1, IO_TLB_SEGSIZE));
static void free_acpi_perf_data(void)
{
- unsigned int i;
+ int i;
/* Freeing a NULL pointer is OK, and alloc_percpu zeroes. */
for_each_possible_cpu(i)
static int xen_upload_processor_pm_data(void)
{
struct acpi_processor *pr_backup = NULL;
- unsigned int i;
+ int i;
int rc = 0;
pr_info("Uploading Xen processor PM info\n");
static int __init xen_acpi_processor_init(void)
{
- unsigned int i;
+ int i;
int rc;
if (!xen_initial_domain())
# SPDX-License-Identifier: GPL-2.0
+
+# N.B. The below cannot be expressed with a single line using
+# CONFIG_XEN_PCI_STUB as it always remains in "y" state,
+# thus preventing the driver to be built as a module.
+# Please note, that CONFIG_XEN_PCIDEV_BACKEND and
+# CONFIG_XEN_PCIDEV_STUB are mutually exclusive.
obj-$(CONFIG_XEN_PCIDEV_BACKEND) += xen-pciback.o
+obj-$(CONFIG_XEN_PCIDEV_STUB) += xen-pciback.o
xen-pciback-y := pci_stub.o pciback_ops.o xenbus.o
xen-pciback-y += conf_space.o conf_space_header.o \
}
out:
- return ERR_PTR(err);
+ return err ? ERR_PTR(err) : NULL;
}
static const struct config_field caplist_pm[] = {
else {
pos = (offset - PCI_BASE_ADDRESS_0) / 4;
if (pos && (res[pos - 1].flags & IORESOURCE_MEM_64)) {
- bar->val = res[pos - 1].start >> 32;
- bar->len_val = -resource_size(&res[pos - 1]) >> 32;
+ /*
+ * Use ">> 16 >> 16" instead of direct ">> 32" shift
+ * to avoid warnings on 32-bit architectures.
+ */
+ bar->val = res[pos - 1].start >> 16 >> 16;
+ bar->len_val = -resource_size(&res[pos - 1]) >> 16 >> 16;
return bar;
}
}
#include <linux/sched.h>
#include <linux/atomic.h>
#include <xen/events.h>
-#include <asm/xen/pci.h>
+#include <xen/pci.h>
+#include <xen/xen.h>
#include <asm/xen/hypervisor.h>
#include <xen/interface/physdev.h>
#include "pciback.h"
struct pci_dev *dev);
void pcistub_put_pci_dev(struct pci_dev *dev);
+static inline bool xen_pcibk_pv_support(void)
+{
+ return IS_ENABLED(CONFIG_XEN_PCIDEV_BACKEND);
+}
+
/* Ensure a device is turned off or reset */
void xen_pcibk_reset_device(struct pci_dev *pdev);
#include <linux/workqueue.h>
#include <xen/xenbus.h>
#include <xen/events.h>
-#include <asm/xen/pci.h>
+#include <xen/pci.h>
#include "pciback.h"
#define INVALID_EVTCHN_IRQ (-1)
int __init xen_pcibk_xenbus_register(void)
{
+ if (!xen_pcibk_pv_support())
+ return 0;
+
xen_pcibk_backend = &xen_pcibk_vpci_backend;
if (passthrough)
xen_pcibk_backend = &xen_pcibk_passthrough_backend;
void __exit xen_pcibk_xenbus_unregister(void)
{
- xenbus_unregister_driver(&xen_pcibk_driver);
+ if (xen_pcibk_pv_support())
+ xenbus_unregister_driver(&xen_pcibk_driver);
}
config 9P_FS
tristate "Plan 9 Resource Sharing Support (9P2000)"
depends on INET && NET_9P
+ select NETFS_SUPPORT
help
If you say Y here, you will get experimental support for
Plan 9 resource sharing via the 9P2000 protocol.
+// SPDX-License-Identifier: LGPL-2.1
/*
* Copyright IBM Corporation, 2010
* Author Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
#include <linux/module.h>
char *name;
size_t size;
void *buffer;
+
if (!acl)
return 0;
+/* SPDX-License-Identifier: LGPL-2.1 */
/*
* Copyright IBM Corporation, 2010
* Author Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
#ifndef FS_9P_ACL_H
#define FS_9P_ACL_H
#ifdef CONFIG_9P_FS_POSIX_ACL
-extern int v9fs_get_acl(struct inode *, struct p9_fid *);
-extern struct posix_acl *v9fs_iop_get_acl(struct inode *inode, int type, bool rcu);
-extern int v9fs_acl_chmod(struct inode *, struct p9_fid *);
-extern int v9fs_set_create_acl(struct inode *, struct p9_fid *,
- struct posix_acl *, struct posix_acl *);
-extern int v9fs_acl_mode(struct inode *dir, umode_t *modep,
- struct posix_acl **dpacl, struct posix_acl **pacl);
-extern void v9fs_put_acl(struct posix_acl *dacl, struct posix_acl *acl);
+int v9fs_get_acl(struct inode *inode, struct p9_fid *fid);
+struct posix_acl *v9fs_iop_get_acl(struct inode *inode, int type,
+ bool rcu);
+int v9fs_acl_chmod(struct inode *inode, struct p9_fid *fid);
+int v9fs_set_create_acl(struct inode *inode, struct p9_fid *fid,
+ struct posix_acl *dacl, struct posix_acl *acl);
+int v9fs_acl_mode(struct inode *dir, umode_t *modep,
+ struct posix_acl **dpacl, struct posix_acl **pacl);
+void v9fs_put_acl(struct posix_acl *dacl, struct posix_acl *acl);
#else
#define v9fs_iop_get_acl NULL
static inline int v9fs_get_acl(struct inode *inode, struct p9_fid *fid)
#define CACHETAG_LEN 11
struct fscache_netfs v9fs_cache_netfs = {
- .name = "9p",
- .version = 0,
+ .name = "9p",
+ .version = 0,
};
/*
mutex_unlock(&v9inode->fscache_lock);
}
-
-int __v9fs_fscache_release_page(struct page *page, gfp_t gfp)
-{
- struct inode *inode = page->mapping->host;
- struct v9fs_inode *v9inode = V9FS_I(inode);
-
- BUG_ON(!v9inode->fscache);
-
- return fscache_maybe_release_page(v9inode->fscache, page, gfp);
-}
-
-void __v9fs_fscache_invalidate_page(struct page *page)
-{
- struct inode *inode = page->mapping->host;
- struct v9fs_inode *v9inode = V9FS_I(inode);
-
- BUG_ON(!v9inode->fscache);
-
- if (PageFsCache(page)) {
- fscache_wait_on_page_write(v9inode->fscache, page);
- BUG_ON(!PageLocked(page));
- fscache_uncache_page(v9inode->fscache, page);
- }
-}
-
-static void v9fs_vfs_readpage_complete(struct page *page, void *data,
- int error)
-{
- if (!error)
- SetPageUptodate(page);
-
- unlock_page(page);
-}
-
-/*
- * __v9fs_readpage_from_fscache - read a page from cache
- *
- * Returns 0 if the pages are in cache and a BIO is submitted,
- * 1 if the pages are not in cache and -error otherwise.
- */
-
-int __v9fs_readpage_from_fscache(struct inode *inode, struct page *page)
-{
- int ret;
- const struct v9fs_inode *v9inode = V9FS_I(inode);
-
- p9_debug(P9_DEBUG_FSC, "inode %p page %p\n", inode, page);
- if (!v9inode->fscache)
- return -ENOBUFS;
-
- ret = fscache_read_or_alloc_page(v9inode->fscache,
- page,
- v9fs_vfs_readpage_complete,
- NULL,
- GFP_KERNEL);
- switch (ret) {
- case -ENOBUFS:
- case -ENODATA:
- p9_debug(P9_DEBUG_FSC, "page/inode not in cache %d\n", ret);
- return 1;
- case 0:
- p9_debug(P9_DEBUG_FSC, "BIO submitted\n");
- return ret;
- default:
- p9_debug(P9_DEBUG_FSC, "ret %d\n", ret);
- return ret;
- }
-}
-
-/*
- * __v9fs_readpages_from_fscache - read multiple pages from cache
- *
- * Returns 0 if the pages are in cache and a BIO is submitted,
- * 1 if the pages are not in cache and -error otherwise.
- */
-
-int __v9fs_readpages_from_fscache(struct inode *inode,
- struct address_space *mapping,
- struct list_head *pages,
- unsigned *nr_pages)
-{
- int ret;
- const struct v9fs_inode *v9inode = V9FS_I(inode);
-
- p9_debug(P9_DEBUG_FSC, "inode %p pages %u\n", inode, *nr_pages);
- if (!v9inode->fscache)
- return -ENOBUFS;
-
- ret = fscache_read_or_alloc_pages(v9inode->fscache,
- mapping, pages, nr_pages,
- v9fs_vfs_readpage_complete,
- NULL,
- mapping_gfp_mask(mapping));
- switch (ret) {
- case -ENOBUFS:
- case -ENODATA:
- p9_debug(P9_DEBUG_FSC, "pages/inodes not in cache %d\n", ret);
- return 1;
- case 0:
- BUG_ON(!list_empty(pages));
- BUG_ON(*nr_pages != 0);
- p9_debug(P9_DEBUG_FSC, "BIO submitted\n");
- return ret;
- default:
- p9_debug(P9_DEBUG_FSC, "ret %d\n", ret);
- return ret;
- }
-}
-
-/*
- * __v9fs_readpage_to_fscache - write a page to the cache
- *
- */
-
-void __v9fs_readpage_to_fscache(struct inode *inode, struct page *page)
-{
- int ret;
- const struct v9fs_inode *v9inode = V9FS_I(inode);
-
- p9_debug(P9_DEBUG_FSC, "inode %p page %p\n", inode, page);
- ret = fscache_write_page(v9inode->fscache, page,
- i_size_read(&v9inode->vfs_inode), GFP_KERNEL);
- p9_debug(P9_DEBUG_FSC, "ret = %d\n", ret);
- if (ret != 0)
- v9fs_uncache_page(inode, page);
-}
-
-/*
- * wait for a page to complete writing to the cache
- */
-void __v9fs_fscache_wait_on_page_write(struct inode *inode, struct page *page)
-{
- const struct v9fs_inode *v9inode = V9FS_I(inode);
- p9_debug(P9_DEBUG_FSC, "inode %p page %p\n", inode, page);
- if (PageFsCache(page))
- fscache_wait_on_page_write(v9inode->fscache, page);
-}
#ifndef _9P_CACHE_H
#define _9P_CACHE_H
-#ifdef CONFIG_9P_FSCACHE
+#define FSCACHE_USE_NEW_IO_API
#include <linux/fscache.h>
-#include <linux/spinlock.h>
+
+#ifdef CONFIG_9P_FSCACHE
extern struct fscache_netfs v9fs_cache_netfs;
extern const struct fscache_cookie_def v9fs_cache_session_index_def;
extern int __v9fs_cache_register(void);
extern void __v9fs_cache_unregister(void);
-extern int __v9fs_fscache_release_page(struct page *page, gfp_t gfp);
-extern void __v9fs_fscache_invalidate_page(struct page *page);
-extern int __v9fs_readpage_from_fscache(struct inode *inode,
- struct page *page);
-extern int __v9fs_readpages_from_fscache(struct inode *inode,
- struct address_space *mapping,
- struct list_head *pages,
- unsigned *nr_pages);
-extern void __v9fs_readpage_to_fscache(struct inode *inode, struct page *page);
-extern void __v9fs_fscache_wait_on_page_write(struct inode *inode,
- struct page *page);
-
-static inline int v9fs_fscache_release_page(struct page *page,
- gfp_t gfp)
-{
- return __v9fs_fscache_release_page(page, gfp);
-}
-
-static inline void v9fs_fscache_invalidate_page(struct page *page)
-{
- __v9fs_fscache_invalidate_page(page);
-}
-
-static inline int v9fs_readpage_from_fscache(struct inode *inode,
- struct page *page)
-{
- return __v9fs_readpage_from_fscache(inode, page);
-}
-
-static inline int v9fs_readpages_from_fscache(struct inode *inode,
- struct address_space *mapping,
- struct list_head *pages,
- unsigned *nr_pages)
-{
- return __v9fs_readpages_from_fscache(inode, mapping, pages,
- nr_pages);
-}
-
-static inline void v9fs_readpage_to_fscache(struct inode *inode,
- struct page *page)
-{
- if (PageFsCache(page))
- __v9fs_readpage_to_fscache(inode, page);
-}
-
-static inline void v9fs_uncache_page(struct inode *inode, struct page *page)
-{
- struct v9fs_inode *v9inode = V9FS_I(inode);
- fscache_uncache_page(v9inode->fscache, page);
- BUG_ON(PageFsCache(page));
-}
-
-static inline void v9fs_fscache_wait_on_page_write(struct inode *inode,
- struct page *page)
-{
- return __v9fs_fscache_wait_on_page_write(inode, page);
-}
-
#else /* CONFIG_9P_FSCACHE */
static inline void v9fs_cache_inode_get_cookie(struct inode *inode)
{
}
-static inline int v9fs_fscache_release_page(struct page *page,
- gfp_t gfp) {
- return 1;
-}
-
-static inline void v9fs_fscache_invalidate_page(struct page *page) {}
-
-static inline int v9fs_readpage_from_fscache(struct inode *inode,
- struct page *page)
-{
- return -ENOBUFS;
-}
-
-static inline int v9fs_readpages_from_fscache(struct inode *inode,
- struct address_space *mapping,
- struct list_head *pages,
- unsigned *nr_pages)
-{
- return -ENOBUFS;
-}
-
-static inline void v9fs_readpage_to_fscache(struct inode *inode,
- struct page *page)
-{}
-
-static inline void v9fs_uncache_page(struct inode *inode, struct page *page)
-{}
-
-static inline void v9fs_fscache_wait_on_page_write(struct inode *inode,
- struct page *page)
-{
- return;
-}
-
#endif /* CONFIG_9P_FSCACHE */
#endif /* _9P_CACHE_H */
/* we'll recheck under lock if there's anything to look in */
if (!ret && dentry->d_fsdata) {
struct hlist_head *h = (struct hlist_head *)&dentry->d_fsdata;
+
spin_lock(&dentry->d_lock);
hlist_for_each_entry(fid, h, dlist) {
if (any || uid_eq(fid->uid, uid)) {
return ERR_PTR(-EPERM);
if (v9fs_proto_dotu(v9ses) || v9fs_proto_dotl(v9ses))
- uname = NULL;
+ uname = NULL;
else
uname = v9ses->uname;
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/v9fs.c
- *
* This file contains functions assisting in mapping VFS to 9P2000
*
* Copyright (C) 2004-2008 by Eric Van Hensbergen <ericvh@gmail.com>
substring_t args[MAX_OPT_ARGS];
char *p;
int option = 0;
- char *s, *e;
+ char *s;
int ret = 0;
/* setup defaults */
while ((p = strsep(&options, ",")) != NULL) {
int token, r;
+
if (!*p)
continue;
+
token = match_token(p, tokens, args);
switch (token) {
case Opt_debug:
v9ses->flags |= V9FS_ACCESS_CLIENT;
} else {
uid_t uid;
+
v9ses->flags |= V9FS_ACCESS_SINGLE;
- uid = simple_strtoul(s, &e, 10);
- if (*e != '\0') {
- ret = -EINVAL;
- pr_info("Unknown access argument %s\n",
- s);
+ r = kstrtouint(s, 10, &uid);
+ if (r) {
+ ret = r;
+ pr_info("Unknown access argument %s: %d\n",
+ s, r);
kfree(s);
continue;
}
* mark transport as disconnected and cancel all pending requests.
*/
-void v9fs_session_cancel(struct v9fs_session_info *v9ses) {
+void v9fs_session_cancel(struct v9fs_session_info *v9ses)
+{
p9_debug(P9_DEBUG_ERROR, "cancel session %p\n", v9ses);
p9_client_disconnect(v9ses->clnt);
}
static int v9fs_cache_register(void)
{
int ret;
+
ret = v9fs_init_inode_cache();
if (ret < 0)
return ret;
static int __init init_v9fs(void)
{
int err;
+
pr_info("Installing v9fs 9p2000 file system support\n");
/* TODO: Setup list of registered trasnport modules */
return container_of(inode, struct v9fs_inode, vfs_inode);
}
+static inline struct fscache_cookie *v9fs_inode_cookie(struct v9fs_inode *v9inode)
+{
+#ifdef CONFIG_9P_FSCACHE
+ return v9inode->fscache;
+#else
+ return NULL;
+#endif
+}
+
extern int v9fs_show_options(struct seq_file *m, struct dentry *root);
-struct p9_fid *v9fs_session_init(struct v9fs_session_info *, const char *,
- char *);
+struct p9_fid *v9fs_session_init(struct v9fs_session_info *v9ses,
+ const char *dev_name, char *data);
extern void v9fs_session_close(struct v9fs_session_info *v9ses);
extern void v9fs_session_cancel(struct v9fs_session_info *v9ses);
extern void v9fs_session_begin_cancel(struct v9fs_session_info *v9ses);
extern struct dentry *v9fs_vfs_lookup(struct inode *dir, struct dentry *dentry,
- unsigned int flags);
+ unsigned int flags);
extern int v9fs_vfs_unlink(struct inode *i, struct dentry *d);
extern int v9fs_vfs_rmdir(struct inode *i, struct dentry *d);
extern int v9fs_vfs_rename(struct user_namespace *mnt_userns,
static inline struct v9fs_session_info *v9fs_inode2v9ses(struct inode *inode)
{
- return (inode->i_sb->s_fs_info);
+ return inode->i_sb->s_fs_info;
}
static inline struct v9fs_session_info *v9fs_dentry2v9ses(struct dentry *dentry)
struct inode *v9fs_alloc_inode(struct super_block *sb);
void v9fs_free_inode(struct inode *inode);
-struct inode *v9fs_get_inode(struct super_block *sb, umode_t mode, dev_t);
+struct inode *v9fs_get_inode(struct super_block *sb, umode_t mode,
+ dev_t rdev);
int v9fs_init_inode(struct v9fs_session_info *v9ses,
- struct inode *inode, umode_t mode, dev_t);
+ struct inode *inode, umode_t mode, dev_t rdev);
void v9fs_evict_inode(struct inode *inode);
ino_t v9fs_qid2ino(struct p9_qid *qid);
void v9fs_stat2inode(struct p9_wstat *stat, struct inode *inode,
int v9fs_uflags2omode(int uflags, int extended);
void v9fs_blank_wstat(struct p9_wstat *wstat);
-int v9fs_vfs_setattr_dotl(struct user_namespace *, struct dentry *,
- struct iattr *);
+int v9fs_vfs_setattr_dotl(struct user_namespace *mnt_userns,
+ struct dentry *dentry, struct iattr *iattr);
int v9fs_file_fsync_dotl(struct file *filp, loff_t start, loff_t end,
int datasync);
int v9fs_refresh_inode(struct p9_fid *fid, struct inode *inode);
static inline void v9fs_invalidate_inode_attr(struct inode *inode)
{
struct v9fs_inode *v9inode;
+
v9inode = V9FS_I(inode);
v9inode->cache_validity |= V9FS_INO_INVALID_ATTR;
- return;
}
int v9fs_open_to_dotl_flags(int flags);
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_addr.c
- *
* This file contians vfs address (mmap) ops for 9P2000.
*
* Copyright (C) 2005 by Eric Van Hensbergen <ericvh@gmail.com>
#include <linux/idr.h>
#include <linux/sched.h>
#include <linux/uio.h>
-#include <linux/bvec.h>
+#include <linux/netfs.h>
#include <net/9p/9p.h>
#include <net/9p/client.h>
#include "fid.h"
/**
- * v9fs_fid_readpage - read an entire page in from 9P
- * @data: Opaque pointer to the fid being read
- * @page: structure to page
- *
+ * v9fs_req_issue_op - Issue a read from 9P
+ * @subreq: The read to make
*/
-static int v9fs_fid_readpage(void *data, struct page *page)
+static void v9fs_req_issue_op(struct netfs_read_subrequest *subreq)
{
- struct p9_fid *fid = data;
- struct inode *inode = page->mapping->host;
- struct bio_vec bvec = {.bv_page = page, .bv_len = PAGE_SIZE};
+ struct netfs_read_request *rreq = subreq->rreq;
+ struct p9_fid *fid = rreq->netfs_priv;
struct iov_iter to;
- int retval, err;
+ loff_t pos = subreq->start + subreq->transferred;
+ size_t len = subreq->len - subreq->transferred;
+ int total, err;
- p9_debug(P9_DEBUG_VFS, "\n");
+ iov_iter_xarray(&to, READ, &rreq->mapping->i_pages, pos, len);
- BUG_ON(!PageLocked(page));
+ total = p9_client_read(fid, pos, &to, &err);
+ netfs_subreq_terminated(subreq, err ?: total, false);
+}
- retval = v9fs_readpage_from_fscache(inode, page);
- if (retval == 0)
- return retval;
+/**
+ * v9fs_init_rreq - Initialise a read request
+ * @rreq: The read request
+ * @file: The file being read from
+ */
+static void v9fs_init_rreq(struct netfs_read_request *rreq, struct file *file)
+{
+ struct p9_fid *fid = file->private_data;
- iov_iter_bvec(&to, READ, &bvec, 1, PAGE_SIZE);
+ refcount_inc(&fid->count);
+ rreq->netfs_priv = fid;
+}
- retval = p9_client_read(fid, page_offset(page), &to, &err);
- if (err) {
- v9fs_uncache_page(inode, page);
- retval = err;
- goto done;
- }
+/**
+ * v9fs_req_cleanup - Cleanup request initialized by v9fs_init_rreq
+ * @mapping: unused mapping of request to cleanup
+ * @priv: private data to cleanup, a fid, guaranted non-null.
+ */
+static void v9fs_req_cleanup(struct address_space *mapping, void *priv)
+{
+ struct p9_fid *fid = priv;
- zero_user(page, retval, PAGE_SIZE - retval);
- flush_dcache_page(page);
- SetPageUptodate(page);
+ p9_client_clunk(fid);
+}
- v9fs_readpage_to_fscache(inode, page);
- retval = 0;
+/**
+ * v9fs_is_cache_enabled - Determine if caching is enabled for an inode
+ * @inode: The inode to check
+ */
+static bool v9fs_is_cache_enabled(struct inode *inode)
+{
+ struct fscache_cookie *cookie = v9fs_inode_cookie(V9FS_I(inode));
-done:
- unlock_page(page);
- return retval;
+ return fscache_cookie_enabled(cookie) && !hlist_empty(&cookie->backing_objects);
}
/**
- * v9fs_vfs_readpage - read an entire page in from 9P
- *
- * @filp: file being read
- * @page: structure to page
- *
+ * v9fs_begin_cache_operation - Begin a cache operation for a read
+ * @rreq: The read request
*/
-
-static int v9fs_vfs_readpage(struct file *filp, struct page *page)
+static int v9fs_begin_cache_operation(struct netfs_read_request *rreq)
{
- return v9fs_fid_readpage(filp->private_data, page);
+ struct fscache_cookie *cookie = v9fs_inode_cookie(V9FS_I(rreq->inode));
+
+ return fscache_begin_read_operation(rreq, cookie);
}
+static const struct netfs_read_request_ops v9fs_req_ops = {
+ .init_rreq = v9fs_init_rreq,
+ .is_cache_enabled = v9fs_is_cache_enabled,
+ .begin_cache_operation = v9fs_begin_cache_operation,
+ .issue_op = v9fs_req_issue_op,
+ .cleanup = v9fs_req_cleanup,
+};
+
/**
- * v9fs_vfs_readpages - read a set of pages from 9P
- *
- * @filp: file being read
- * @mapping: the address space
- * @pages: list of pages to read
- * @nr_pages: count of pages to read
+ * v9fs_vfs_readpage - read an entire page in from 9P
+ * @file: file being read
+ * @page: structure to page
*
*/
-
-static int v9fs_vfs_readpages(struct file *filp, struct address_space *mapping,
- struct list_head *pages, unsigned nr_pages)
+static int v9fs_vfs_readpage(struct file *file, struct page *page)
{
- int ret = 0;
- struct inode *inode;
-
- inode = mapping->host;
- p9_debug(P9_DEBUG_VFS, "inode: %p file: %p\n", inode, filp);
+ struct folio *folio = page_folio(page);
- ret = v9fs_readpages_from_fscache(inode, mapping, pages, &nr_pages);
- if (ret == 0)
- return ret;
+ return netfs_readpage(file, folio, &v9fs_req_ops, NULL);
+}
- ret = read_cache_pages(mapping, pages, v9fs_fid_readpage,
- filp->private_data);
- p9_debug(P9_DEBUG_VFS, " = %d\n", ret);
- return ret;
+/**
+ * v9fs_vfs_readahead - read a set of pages from 9P
+ * @ractl: The readahead parameters
+ */
+static void v9fs_vfs_readahead(struct readahead_control *ractl)
+{
+ netfs_readahead(ractl, &v9fs_req_ops, NULL);
}
/**
static int v9fs_release_page(struct page *page, gfp_t gfp)
{
- if (PagePrivate(page))
+ struct folio *folio = page_folio(page);
+
+ if (folio_test_private(folio))
return 0;
- return v9fs_fscache_release_page(page, gfp);
+#ifdef CONFIG_9P_FSCACHE
+ if (folio_test_fscache(folio)) {
+ if (!(gfp & __GFP_DIRECT_RECLAIM) || !(gfp & __GFP_FS))
+ return 0;
+ folio_wait_fscache(folio);
+ }
+#endif
+ return 1;
}
/**
static void v9fs_invalidate_page(struct page *page, unsigned int offset,
unsigned int length)
{
- /*
- * If called with zero offset, we should release
- * the private state assocated with the page
- */
- if (offset == 0 && length == PAGE_SIZE)
- v9fs_fscache_invalidate_page(page);
+ struct folio *folio = page_folio(page);
+
+ folio_wait_fscache(folio);
}
-static int v9fs_vfs_writepage_locked(struct page *page)
+static int v9fs_vfs_write_folio_locked(struct folio *folio)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = folio_inode(folio);
struct v9fs_inode *v9inode = V9FS_I(inode);
- loff_t size = i_size_read(inode);
+ loff_t start = folio_pos(folio);
+ loff_t i_size = i_size_read(inode);
struct iov_iter from;
- struct bio_vec bvec;
- int err, len;
+ size_t len = folio_size(folio);
+ int err;
- if (page->index == size >> PAGE_SHIFT)
- len = size & ~PAGE_MASK;
- else
- len = PAGE_SIZE;
+ if (start >= i_size)
+ return 0; /* Simultaneous truncation occurred */
- bvec.bv_page = page;
- bvec.bv_offset = 0;
- bvec.bv_len = len;
- iov_iter_bvec(&from, WRITE, &bvec, 1, len);
+ len = min_t(loff_t, i_size - start, len);
+
+ iov_iter_xarray(&from, WRITE, &folio_mapping(folio)->i_pages, start, len);
/* We should have writeback_fid always set */
BUG_ON(!v9inode->writeback_fid);
- set_page_writeback(page);
+ folio_start_writeback(folio);
- p9_client_write(v9inode->writeback_fid, page_offset(page), &from, &err);
+ p9_client_write(v9inode->writeback_fid, start, &from, &err);
- end_page_writeback(page);
+ folio_end_writeback(folio);
return err;
}
static int v9fs_vfs_writepage(struct page *page, struct writeback_control *wbc)
{
+ struct folio *folio = page_folio(page);
int retval;
- p9_debug(P9_DEBUG_VFS, "page %p\n", page);
+ p9_debug(P9_DEBUG_VFS, "folio %p\n", folio);
- retval = v9fs_vfs_writepage_locked(page);
+ retval = v9fs_vfs_write_folio_locked(folio);
if (retval < 0) {
if (retval == -EAGAIN) {
- redirty_page_for_writepage(wbc, page);
+ folio_redirty_for_writepage(wbc, folio);
retval = 0;
} else {
- SetPageError(page);
- mapping_set_error(page->mapping, retval);
+ mapping_set_error(folio_mapping(folio), retval);
}
} else
retval = 0;
- unlock_page(page);
+ folio_unlock(folio);
return retval;
}
static int v9fs_launder_page(struct page *page)
{
+ struct folio *folio = page_folio(page);
int retval;
- struct inode *inode = page->mapping->host;
- v9fs_fscache_wait_on_page_write(inode, page);
- if (clear_page_dirty_for_io(page)) {
- retval = v9fs_vfs_writepage_locked(page);
+ if (folio_clear_dirty_for_io(folio)) {
+ retval = v9fs_vfs_write_folio_locked(folio);
if (retval)
return retval;
}
+ folio_wait_fscache(folio);
return 0;
}
loff_t pos = iocb->ki_pos;
ssize_t n;
int err = 0;
+
if (iov_iter_rw(iter) == WRITE) {
n = p9_client_write(file->private_data, pos, iter, &err);
if (n) {
struct inode *inode = file_inode(file);
loff_t i_size = i_size_read(inode);
+
if (pos + n > i_size)
inode_add_bytes(inode, pos + n - i_size);
}
}
static int v9fs_write_begin(struct file *filp, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned flags,
- struct page **pagep, void **fsdata)
+ loff_t pos, unsigned int len, unsigned int flags,
+ struct page **subpagep, void **fsdata)
{
- int retval = 0;
- struct page *page;
- struct v9fs_inode *v9inode;
- pgoff_t index = pos >> PAGE_SHIFT;
- struct inode *inode = mapping->host;
-
+ int retval;
+ struct folio *folio;
+ struct v9fs_inode *v9inode = V9FS_I(mapping->host);
p9_debug(P9_DEBUG_VFS, "filp %p, mapping %p\n", filp, mapping);
- v9inode = V9FS_I(inode);
-start:
- page = grab_cache_page_write_begin(mapping, index, flags);
- if (!page) {
- retval = -ENOMEM;
- goto out;
- }
BUG_ON(!v9inode->writeback_fid);
- if (PageUptodate(page))
- goto out;
- if (len == PAGE_SIZE)
- goto out;
+ /* Prefetch area to be written into the cache if we're caching this
+ * file. We need to do this before we get a lock on the page in case
+ * there's more than one writer competing for the same cache block.
+ */
+ retval = netfs_write_begin(filp, mapping, pos, len, flags, &folio, fsdata,
+ &v9fs_req_ops, NULL);
+ if (retval < 0)
+ return retval;
- retval = v9fs_fid_readpage(v9inode->writeback_fid, page);
- put_page(page);
- if (!retval)
- goto start;
-out:
- *pagep = page;
+ *subpagep = &folio->page;
return retval;
}
static int v9fs_write_end(struct file *filp, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned copied,
- struct page *page, void *fsdata)
+ loff_t pos, unsigned int len, unsigned int copied,
+ struct page *subpage, void *fsdata)
{
loff_t last_pos = pos + copied;
- struct inode *inode = page->mapping->host;
+ struct folio *folio = page_folio(subpage);
+ struct inode *inode = mapping->host;
p9_debug(P9_DEBUG_VFS, "filp %p, mapping %p\n", filp, mapping);
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
if (unlikely(copied < len)) {
copied = 0;
goto out;
- } else if (len == PAGE_SIZE) {
- SetPageUptodate(page);
}
+
+ folio_mark_uptodate(folio);
}
+
/*
* No need to use i_size_read() here, the i_size
* cannot change under us because we hold the i_mutex.
inode_add_bytes(inode, last_pos - inode->i_size);
i_size_write(inode, last_pos);
}
- set_page_dirty(page);
+ folio_mark_dirty(folio);
out:
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
return copied;
}
const struct address_space_operations v9fs_addr_operations = {
.readpage = v9fs_vfs_readpage,
- .readpages = v9fs_vfs_readpages,
+ .readahead = v9fs_vfs_readahead,
.set_page_dirty = __set_page_dirty_nobuffers,
.writepage = v9fs_vfs_writepage,
.write_begin = v9fs_write_begin,
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_dentry.c
- *
* This file contians vfs dentry ops for the 9P2000 protocol.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
static void v9fs_dentry_release(struct dentry *dentry)
{
struct hlist_node *p, *n;
+
p9_debug(P9_DEBUG_VFS, " dentry: %pd (%p)\n",
dentry, dentry);
hlist_for_each_safe(p, n, (struct hlist_head *)&dentry->d_fsdata)
if (v9inode->cache_validity & V9FS_INO_INVALID_ATTR) {
int retval;
struct v9fs_session_info *v9ses;
+
fid = v9fs_fid_lookup(dentry);
if (IS_ERR(fid))
return PTR_ERR(fid);
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_dir.c
- *
* This file contains vfs directory ops for the 9P2000 protocol.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
static struct p9_rdir *v9fs_alloc_rdir_buf(struct file *filp, int buflen)
{
struct p9_fid *fid = filp->private_data;
+
if (!fid->rdir)
fid->rdir = kzalloc(sizeof(struct p9_rdir) + buflen, GFP_KERNEL);
return fid->rdir;
if (rdir->tail == rdir->head) {
struct iov_iter to;
int n;
+
iov_iter_kvec(&to, READ, &kvec, 1, buflen);
n = p9_client_read(file->private_data, ctx->pos, &to,
&err);
.iterate_shared = v9fs_dir_readdir_dotl,
.open = v9fs_file_open,
.release = v9fs_dir_release,
- .fsync = v9fs_file_fsync_dotl,
+ .fsync = v9fs_file_fsync_dotl,
};
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_file.c
- *
* This file contians vfs file ops for 9P2000.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
struct inode *inode = file_inode(file);
loff_t i_size;
unsigned long pg_start, pg_end;
+
pg_start = origin >> PAGE_SHIFT;
pg_end = (origin + retval - 1) >> PAGE_SHIFT;
if (inode->i_mapping && inode->i_mapping->nrpages)
v9fs_vm_page_mkwrite(struct vm_fault *vmf)
{
struct v9fs_inode *v9inode;
- struct page *page = vmf->page;
+ struct folio *folio = page_folio(vmf->page);
struct file *filp = vmf->vma->vm_file;
struct inode *inode = file_inode(filp);
- p9_debug(P9_DEBUG_VFS, "page %p fid %lx\n",
- page, (unsigned long)filp->private_data);
+ p9_debug(P9_DEBUG_VFS, "folio %p fid %lx\n",
+ folio, (unsigned long)filp->private_data);
+
+ v9inode = V9FS_I(inode);
+
+ /* Wait for the page to be written to the cache before we allow it to
+ * be modified. We then assume the entire page will need writing back.
+ */
+#ifdef CONFIG_9P_FSCACHE
+ if (folio_test_fscache(folio) &&
+ folio_wait_fscache_killable(folio) < 0)
+ return VM_FAULT_NOPAGE;
+#endif
/* Update file times before taking page lock */
file_update_time(filp);
- v9inode = V9FS_I(inode);
- /* make sure the cache has finished storing the page */
- v9fs_fscache_wait_on_page_write(inode, page);
BUG_ON(!v9inode->writeback_fid);
- lock_page(page);
- if (page->mapping != inode->i_mapping)
+ if (folio_lock_killable(folio) < 0)
+ return VM_FAULT_RETRY;
+ if (folio_mapping(folio) != inode->i_mapping)
goto out_unlock;
- wait_for_stable_page(page);
+ folio_wait_stable(folio);
return VM_FAULT_LOCKED;
out_unlock:
- unlock_page(page);
+ folio_unlock(folio);
return VM_FAULT_NOPAGE;
}
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_inode.c
- *
* This file contains vfs inode ops for the 9P2000 protocol.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
static u32 unixmode2p9mode(struct v9fs_session_info *v9ses, umode_t mode)
{
int res;
+
res = mode & 0777;
if (S_ISDIR(mode))
res |= P9_DMDIR;
static umode_t p9mode2unixmode(struct v9fs_session_info *v9ses,
struct p9_wstat *stat, dev_t *rdev)
{
- int res;
+ int res, r;
u32 mode = stat->mode;
*rdev = 0;
res |= S_IFIFO;
else if ((mode & P9_DMDEVICE) && (v9fs_proto_dotu(v9ses))
&& (v9ses->nodev == 0)) {
- char type = 0, ext[32];
+ char type = 0;
int major = -1, minor = -1;
- strlcpy(ext, stat->extension, sizeof(ext));
- sscanf(ext, "%c %i %i", &type, &major, &minor);
+ r = sscanf(stat->extension, "%c %i %i", &type, &major, &minor);
+ if (r != 3) {
+ p9_debug(P9_DEBUG_ERROR,
+ "invalid device string, umode will be bogus: %s\n",
+ stat->extension);
+ return res;
+ }
switch (type) {
case 'c':
res |= S_IFCHR;
struct inode *v9fs_alloc_inode(struct super_block *sb)
{
struct v9fs_inode *v9inode;
+
v9inode = kmem_cache_alloc(v9fs_inode_cache, GFP_KERNEL);
if (!v9inode)
return NULL;
{
int err = 0;
- inode_init_owner(&init_user_ns,inode, NULL, mode);
+ inode_init_owner(&init_user_ns, inode, NULL, mode);
inode->i_blocks = 0;
inode->i_rdev = rdev;
inode->i_atime = inode->i_mtime = inode->i_ctime = current_time(inode);
unsigned long i_ino;
struct inode *inode;
struct v9fs_session_info *v9ses = sb->s_fs_info;
- int (*test)(struct inode *, void *);
+ int (*test)(struct inode *inode, void *data);
if (new)
test = v9fs_test_new_inode;
static int v9fs_at_to_dotl_flags(int flags)
{
int rflags = 0;
+
if (flags & AT_REMOVEDIR)
rflags |= P9_DOTL_AT_REMOVEDIR;
+
return rflags;
}
static int
v9fs_vfs_atomic_open(struct inode *dir, struct dentry *dentry,
- struct file *file, unsigned flags, umode_t mode)
+ struct file *file, unsigned int flags, umode_t mode)
{
int err;
u32 perm;
fid = v9fs_fid_lookup(dentry);
use_dentry = 1;
}
- if(IS_ERR(fid))
+ if (IS_ERR(fid))
return PTR_ERR(fid);
v9fs_blank_wstat(&wstat);
char name[2 + U32_MAX_DIGITS + 1 + U32_MAX_DIGITS + 1];
u32 perm;
- p9_debug(P9_DEBUG_VFS, " %lu,%pd mode: %hx MAJOR: %u MINOR: %u\n",
+ p9_debug(P9_DEBUG_VFS, " %lu,%pd mode: %x MAJOR: %u MINOR: %u\n",
dir->i_ino, dentry, mode,
MAJOR(rdev), MINOR(rdev));
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_inode_dotl.c
- *
* This file contains vfs inode ops for the 9P2000.L protocol.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
unsigned long i_ino;
struct inode *inode;
struct v9fs_session_info *v9ses = sb->s_fs_info;
- int (*test)(struct inode *, void *);
+ int (*test)(struct inode *inode, void *data);
if (new)
test = v9fs_test_new_inode_dotl;
static int
v9fs_vfs_atomic_open_dotl(struct inode *dir, struct dentry *dentry,
- struct file *file, unsigned flags, umode_t omode)
+ struct file *file, unsigned int flags, umode_t omode)
{
int err = 0;
kgid_t gid;
v9ses = v9fs_inode2v9ses(dir);
name = dentry->d_name.name;
- p9_debug(P9_DEBUG_VFS, "name:%s flags:0x%x mode:0x%hx\n",
+ p9_debug(P9_DEBUG_VFS, "name:%s flags:0x%x mode:0x%x\n",
name, flags, omode);
dfid = v9fs_parent_fid(dentry);
if (v9ses->cache == CACHE_LOOSE || v9ses->cache == CACHE_FSCACHE) {
/* Get the latest stat info from server. */
struct p9_fid *fid;
+
fid = v9fs_fid_lookup(old_dentry);
if (IS_ERR(fid))
return PTR_ERR(fid);
struct p9_qid qid;
struct posix_acl *dacl = NULL, *pacl = NULL;
- p9_debug(P9_DEBUG_VFS, " %lu,%pd mode: %hx MAJOR: %u MINOR: %u\n",
+ p9_debug(P9_DEBUG_VFS, " %lu,%pd mode: %x MAJOR: %u MINOR: %u\n",
dir->i_ino, dentry, omode,
MAJOR(rdev), MINOR(rdev));
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/vfs_super.c
- *
- * This file contians superblock ops for 9P2000. It is intended that
- * you mount this file system on directories.
*
* Copyright (C) 2004 by Eric Van Hensbergen <ericvh@gmail.com>
* Copyright (C) 2002 by Ron Minnich <rminnich@lanl.gov>
if (!v9ses->cache) {
sb->s_bdi->ra_pages = 0;
sb->s_bdi->io_pages = 0;
+ } else {
+ sb->s_bdi->ra_pages = v9ses->maxdata >> PAGE_SHIFT;
+ sb->s_bdi->io_pages = v9ses->maxdata >> PAGE_SHIFT;
}
sb->s_flags |= SB_ACTIVE | SB_DIRSYNC;
struct inode *inode = NULL;
struct dentry *root = NULL;
struct v9fs_session_info *v9ses = NULL;
- umode_t mode = S_IRWXUGO | S_ISVTX;
+ umode_t mode = 0777 | S_ISVTX;
struct p9_fid *fid;
int retval = 0;
sb->s_root = root;
if (v9fs_proto_dotl(v9ses)) {
struct p9_stat_dotl *st = NULL;
+
st = p9_client_getattr_dotl(fid, P9_STATS_BASIC);
if (IS_ERR(st)) {
retval = PTR_ERR(st);
kfree(st);
} else {
struct p9_wstat *st = NULL;
+
st = p9_client_stat(fid);
if (IS_ERR(st)) {
retval = PTR_ERR(st);
static int v9fs_drop_inode(struct inode *inode)
{
struct v9fs_session_info *v9ses;
+
v9ses = v9fs_inode2v9ses(inode);
if (v9ses->cache == CACHE_LOOSE || v9ses->cache == CACHE_FSCACHE)
return generic_drop_inode(inode);
/*
* in case of non cached mode always drop the
- * the inode because we want the inode attribute
+ * inode because we want the inode attribute
* to always match that on the server.
*/
return 1;
+// SPDX-License-Identifier: LGPL-2.1
/*
* Copyright IBM Corporation, 2010
* Author Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
#include <linux/module.h>
+/* SPDX-License-Identifier: LGPL-2.1 */
/*
* Copyright IBM Corporation, 2010
* Author Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
#ifndef FS_9P_XATTR_H
#define FS_9P_XATTR_H
extern const struct xattr_handler v9fs_xattr_acl_access_handler;
extern const struct xattr_handler v9fs_xattr_acl_default_handler;
-extern ssize_t v9fs_fid_xattr_get(struct p9_fid *, const char *,
- void *, size_t);
-extern ssize_t v9fs_xattr_get(struct dentry *, const char *,
- void *, size_t);
-extern int v9fs_fid_xattr_set(struct p9_fid *, const char *,
- const void *, size_t, int);
-extern int v9fs_xattr_set(struct dentry *, const char *,
- const void *, size_t, int);
-extern ssize_t v9fs_listxattr(struct dentry *, char *, size_t);
+ssize_t v9fs_fid_xattr_get(struct p9_fid *fid, const char *name,
+ void *buffer, size_t buffer_size);
+ssize_t v9fs_xattr_get(struct dentry *dentry, const char *name,
+ void *buffer, size_t buffer_size);
+int v9fs_fid_xattr_set(struct p9_fid *fid, const char *name,
+ const void *value, size_t value_len, int flags);
+int v9fs_xattr_set(struct dentry *dentry, const char *name,
+ const void *value, size_t value_len, int flags);
+ssize_t v9fs_listxattr(struct dentry *dentry, char *buffer,
+ size_t buffer_size);
#endif /* FS_9P_XATTR_H */
};
/*
- * Drop the refs that we're holding on the pages we were reading into. We've
+ * Drop the refs that we're holding on the folios we were reading into. We've
* got refs on the first nr_pages pages.
*/
static void afs_dir_read_cleanup(struct afs_read *req)
{
struct address_space *mapping = req->vnode->vfs_inode.i_mapping;
- struct page *page;
+ struct folio *folio;
pgoff_t last = req->nr_pages - 1;
XA_STATE(xas, &mapping->i_pages, 0);
return;
rcu_read_lock();
- xas_for_each(&xas, page, last) {
- if (xas_retry(&xas, page))
+ xas_for_each(&xas, folio, last) {
+ if (xas_retry(&xas, folio))
continue;
- BUG_ON(xa_is_value(page));
- BUG_ON(PageCompound(page));
- ASSERTCMP(page->mapping, ==, mapping);
+ BUG_ON(xa_is_value(folio));
+ ASSERTCMP(folio_file_mapping(folio), ==, mapping);
- put_page(page);
+ folio_put(folio);
}
rcu_read_unlock();
}
/*
- * check that a directory page is valid
+ * check that a directory folio is valid
*/
-static bool afs_dir_check_page(struct afs_vnode *dvnode, struct page *page,
- loff_t i_size)
+static bool afs_dir_check_folio(struct afs_vnode *dvnode, struct folio *folio,
+ loff_t i_size)
{
- struct afs_xdr_dir_page *dbuf;
- loff_t latter, off;
- int tmp, qty;
+ union afs_xdr_dir_block *block;
+ size_t offset, size;
+ loff_t pos;
- /* Determine how many magic numbers there should be in this page, but
+ /* Determine how many magic numbers there should be in this folio, but
* we must take care because the directory may change size under us.
*/
- off = page_offset(page);
- if (i_size <= off)
+ pos = folio_pos(folio);
+ if (i_size <= pos)
goto checked;
- latter = i_size - off;
- if (latter >= PAGE_SIZE)
- qty = PAGE_SIZE;
- else
- qty = latter;
- qty /= sizeof(union afs_xdr_dir_block);
-
- /* check them */
- dbuf = kmap_atomic(page);
- for (tmp = 0; tmp < qty; tmp++) {
- if (dbuf->blocks[tmp].hdr.magic != AFS_DIR_MAGIC) {
- printk("kAFS: %s(%lx): bad magic %d/%d is %04hx\n",
- __func__, dvnode->vfs_inode.i_ino, tmp, qty,
- ntohs(dbuf->blocks[tmp].hdr.magic));
- trace_afs_dir_check_failed(dvnode, off, i_size);
- kunmap(page);
+ size = min_t(loff_t, folio_size(folio), i_size - pos);
+ for (offset = 0; offset < size; offset += sizeof(*block)) {
+ block = kmap_local_folio(folio, offset);
+ if (block->hdr.magic != AFS_DIR_MAGIC) {
+ printk("kAFS: %s(%lx): [%llx] bad magic %zx/%zx is %04hx\n",
+ __func__, dvnode->vfs_inode.i_ino,
+ pos, offset, size, ntohs(block->hdr.magic));
+ trace_afs_dir_check_failed(dvnode, pos + offset, i_size);
+ kunmap_local(block);
trace_afs_file_error(dvnode, -EIO, afs_file_error_dir_bad_magic);
goto error;
}
/* Make sure each block is NUL terminated so we can reasonably
- * use string functions on it. The filenames in the page
+ * use string functions on it. The filenames in the folio
* *should* be NUL-terminated anyway.
*/
- ((u8 *)&dbuf->blocks[tmp])[AFS_DIR_BLOCK_SIZE - 1] = 0;
- }
-
- kunmap_atomic(dbuf);
+ ((u8 *)block)[AFS_DIR_BLOCK_SIZE - 1] = 0;
+ kunmap_local(block);
+ }
checked:
afs_stat_v(dvnode, n_read_dir);
return true;
*/
static void afs_dir_dump(struct afs_vnode *dvnode, struct afs_read *req)
{
- struct afs_xdr_dir_page *dbuf;
+ union afs_xdr_dir_block *block;
struct address_space *mapping = dvnode->vfs_inode.i_mapping;
- struct page *page;
- unsigned int i, qty = PAGE_SIZE / sizeof(union afs_xdr_dir_block);
+ struct folio *folio;
pgoff_t last = req->nr_pages - 1;
+ size_t offset, size;
XA_STATE(xas, &mapping->i_pages, 0);
req->pos, req->nr_pages,
req->iter->iov_offset, iov_iter_count(req->iter));
- xas_for_each(&xas, page, last) {
- if (xas_retry(&xas, page))
+ xas_for_each(&xas, folio, last) {
+ if (xas_retry(&xas, folio))
continue;
- BUG_ON(PageCompound(page));
- BUG_ON(page->mapping != mapping);
-
- dbuf = kmap_atomic(page);
- for (i = 0; i < qty; i++) {
- union afs_xdr_dir_block *block = &dbuf->blocks[i];
+ BUG_ON(folio_file_mapping(folio) != mapping);
- pr_warn("[%02lx] %32phN\n", page->index * qty + i, block);
+ size = min_t(loff_t, folio_size(folio), req->actual_len - folio_pos(folio));
+ for (offset = 0; offset < size; offset += sizeof(*block)) {
+ block = kmap_local_folio(folio, offset);
+ pr_warn("[%02lx] %32phN\n", folio_index(folio) + offset, block);
+ kunmap_local(block);
}
- kunmap_atomic(dbuf);
}
}
/*
- * Check all the pages in a directory. All the pages are held pinned.
+ * Check all the blocks in a directory. All the folios are held pinned.
*/
static int afs_dir_check(struct afs_vnode *dvnode, struct afs_read *req)
{
struct address_space *mapping = dvnode->vfs_inode.i_mapping;
- struct page *page;
+ struct folio *folio;
pgoff_t last = req->nr_pages - 1;
int ret = 0;
return 0;
rcu_read_lock();
- xas_for_each(&xas, page, last) {
- if (xas_retry(&xas, page))
+ xas_for_each(&xas, folio, last) {
+ if (xas_retry(&xas, folio))
continue;
- BUG_ON(PageCompound(page));
- BUG_ON(page->mapping != mapping);
+ BUG_ON(folio_file_mapping(folio) != mapping);
- if (!afs_dir_check_page(dvnode, page, req->file_size)) {
+ if (!afs_dir_check_folio(dvnode, folio, req->actual_len)) {
afs_dir_dump(dvnode, req);
ret = -EIO;
break;
/*
* Read the directory into the pagecache in one go, scrubbing the previous
- * contents. The list of pages is returned, pinning them so that they don't
+ * contents. The list of folios is returned, pinning them so that they don't
* get reclaimed during the iteration.
*/
static struct afs_read *afs_read_dir(struct afs_vnode *dvnode, struct key *key)
__acquires(&dvnode->validate_lock)
{
+ struct address_space *mapping = dvnode->vfs_inode.i_mapping;
struct afs_read *req;
loff_t i_size;
- int nr_pages, i, n;
+ int nr_pages, i;
int ret;
_enter("");
req->iter = &req->def_iter;
/* Fill in any gaps that we might find where the memory reclaimer has
- * been at work and pin all the pages. If there are any gaps, we will
+ * been at work and pin all the folios. If there are any gaps, we will
* need to reread the entire directory contents.
*/
i = req->nr_pages;
while (i < nr_pages) {
- struct page *pages[8], *page;
-
- n = find_get_pages_contig(dvnode->vfs_inode.i_mapping, i,
- min_t(unsigned int, nr_pages - i,
- ARRAY_SIZE(pages)),
- pages);
- _debug("find %u at %u/%u", n, i, nr_pages);
-
- if (n == 0) {
- gfp_t gfp = dvnode->vfs_inode.i_mapping->gfp_mask;
+ struct folio *folio;
+ folio = filemap_get_folio(mapping, i);
+ if (!folio) {
if (test_and_clear_bit(AFS_VNODE_DIR_VALID, &dvnode->flags))
afs_stat_v(dvnode, n_inval);
ret = -ENOMEM;
- page = __page_cache_alloc(gfp);
- if (!page)
+ folio = __filemap_get_folio(mapping,
+ i, FGP_LOCK | FGP_CREAT,
+ mapping->gfp_mask);
+ if (!folio)
goto error;
- ret = add_to_page_cache_lru(page,
- dvnode->vfs_inode.i_mapping,
- i, gfp);
- if (ret < 0)
- goto error;
-
- attach_page_private(page, (void *)1);
- unlock_page(page);
- req->nr_pages++;
- i++;
- } else {
- req->nr_pages += n;
- i += n;
+ folio_attach_private(folio, (void *)1);
+ folio_unlock(folio);
}
+
+ req->nr_pages += folio_nr_pages(folio);
+ i += folio_nr_pages(folio);
}
/* If we're going to reload, we need to lock all the pages to prevent
size_t nlen;
int tmp;
- _enter("%u,%x,%p,,",(unsigned)ctx->pos,blkoff,block);
+ _enter("%llx,%x", ctx->pos, blkoff);
curr = (ctx->pos - blkoff) / sizeof(union afs_xdr_dirent);
struct key *key, afs_dataversion_t *_dir_version)
{
struct afs_vnode *dvnode = AFS_FS_I(dir);
- struct afs_xdr_dir_page *dbuf;
union afs_xdr_dir_block *dblock;
struct afs_read *req;
- struct page *page;
- unsigned blkoff, limit;
- void __rcu **slot;
+ struct folio *folio;
+ unsigned offset, size;
int ret;
_enter("{%lu},%u,,", dir->i_ino, (unsigned)ctx->pos);
/* walk through the blocks in sequence */
ret = 0;
while (ctx->pos < req->actual_len) {
- blkoff = ctx->pos & ~(sizeof(union afs_xdr_dir_block) - 1);
-
- /* Fetch the appropriate page from the directory and re-add it
+ /* Fetch the appropriate folio from the directory and re-add it
* to the LRU. We have all the pages pinned with an extra ref.
*/
- rcu_read_lock();
- page = NULL;
- slot = radix_tree_lookup_slot(&dvnode->vfs_inode.i_mapping->i_pages,
- blkoff / PAGE_SIZE);
- if (slot)
- page = radix_tree_deref_slot(slot);
- rcu_read_unlock();
- if (!page) {
+ folio = __filemap_get_folio(dir->i_mapping, ctx->pos / PAGE_SIZE,
+ FGP_ACCESSED, 0);
+ if (!folio) {
ret = afs_bad(dvnode, afs_file_error_dir_missing_page);
break;
}
- mark_page_accessed(page);
- limit = blkoff & ~(PAGE_SIZE - 1);
+ offset = round_down(ctx->pos, sizeof(*dblock)) - folio_file_pos(folio);
+ size = min_t(loff_t, folio_size(folio),
+ req->actual_len - folio_file_pos(folio));
- dbuf = kmap(page);
-
- /* deal with the individual blocks stashed on this page */
do {
- dblock = &dbuf->blocks[(blkoff % PAGE_SIZE) /
- sizeof(union afs_xdr_dir_block)];
- ret = afs_dir_iterate_block(dvnode, ctx, dblock, blkoff);
- if (ret != 1) {
- kunmap(page);
+ dblock = kmap_local_folio(folio, offset);
+ ret = afs_dir_iterate_block(dvnode, ctx, dblock,
+ folio_file_pos(folio) + offset);
+ kunmap_local(dblock);
+ if (ret != 1)
goto out;
- }
- blkoff += sizeof(union afs_xdr_dir_block);
+ } while (offset += sizeof(*dblock), offset < size);
- } while (ctx->pos < dir->i_size && blkoff < limit);
-
- kunmap(page);
ret = 0;
}
}
/*
- * Release a directory page and clean up its private state if it's not busy
- * - return true if the page can now be released, false if not
+ * Release a directory folio and clean up its private state if it's not busy
+ * - return true if the folio can now be released, false if not
*/
-static int afs_dir_releasepage(struct page *page, gfp_t gfp_flags)
+static int afs_dir_releasepage(struct page *subpage, gfp_t gfp_flags)
{
- struct afs_vnode *dvnode = AFS_FS_I(page->mapping->host);
+ struct folio *folio = page_folio(subpage);
+ struct afs_vnode *dvnode = AFS_FS_I(folio_inode(folio));
- _enter("{{%llx:%llu}[%lu]}", dvnode->fid.vid, dvnode->fid.vnode, page->index);
+ _enter("{{%llx:%llu}[%lu]}", dvnode->fid.vid, dvnode->fid.vnode, folio_index(folio));
- detach_page_private(page);
+ folio_detach_private(folio);
/* The directory will need reloading. */
if (test_and_clear_bit(AFS_VNODE_DIR_VALID, &dvnode->flags))
afs_stat_v(dvnode, n_relpg);
- return 1;
+ return true;
}
/*
- * invalidate part or all of a page
- * - release a page and clean up its private data if offset is 0 (indicating
- * the entire page)
+ * Invalidate part or all of a folio.
*/
-static void afs_dir_invalidatepage(struct page *page, unsigned int offset,
+static void afs_dir_invalidatepage(struct page *subpage, unsigned int offset,
unsigned int length)
{
- struct afs_vnode *dvnode = AFS_FS_I(page->mapping->host);
+ struct folio *folio = page_folio(subpage);
+ struct afs_vnode *dvnode = AFS_FS_I(folio_inode(folio));
- _enter("{%lu},%u,%u", page->index, offset, length);
+ _enter("{%lu},%u,%u", folio_index(folio), offset, length);
- BUG_ON(!PageLocked(page));
+ BUG_ON(!folio_test_locked(folio));
/* The directory will need reloading. */
if (test_and_clear_bit(AFS_VNODE_DIR_VALID, &dvnode->flags))
afs_stat_v(dvnode, n_inval);
- /* we clean up only if the entire page is being invalidated */
- if (offset == 0 && length == thp_size(page))
- detach_page_private(page);
+ /* we clean up only if the entire folio is being invalidated */
+ if (offset == 0 && length == folio_size(folio))
+ folio_detach_private(folio);
}
}
/*
+ * Get a new directory folio.
+ */
+static struct folio *afs_dir_get_folio(struct afs_vnode *vnode, pgoff_t index)
+{
+ struct address_space *mapping = vnode->vfs_inode.i_mapping;
+ struct folio *folio;
+
+ folio = __filemap_get_folio(mapping, index,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT,
+ mapping->gfp_mask);
+ if (!folio)
+ clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
+ else if (folio && !folio_test_private(folio))
+ folio_attach_private(folio, (void *)1);
+
+ return folio;
+}
+
+/*
* Scan a directory block looking for a dirent of the right name.
*/
static int afs_dir_scan_block(union afs_xdr_dir_block *block, struct qstr *name,
enum afs_edit_dir_reason why)
{
union afs_xdr_dir_block *meta, *block;
- struct afs_xdr_dir_page *meta_page, *dir_page;
union afs_xdr_dirent *de;
- struct page *page0, *page;
+ struct folio *folio0, *folio;
unsigned int need_slots, nr_blocks, b;
pgoff_t index;
loff_t i_size;
- gfp_t gfp;
int slot;
_enter(",,{%d,%s},", name->len, name->name);
return;
}
- gfp = vnode->vfs_inode.i_mapping->gfp_mask;
- page0 = find_or_create_page(vnode->vfs_inode.i_mapping, 0, gfp);
- if (!page0) {
- clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
+ folio0 = afs_dir_get_folio(vnode, 0);
+ if (!folio0) {
_leave(" [fgp]");
return;
}
/* Work out how many slots we're going to need. */
need_slots = afs_dir_calc_slots(name->len);
- meta_page = kmap(page0);
- meta = &meta_page->blocks[0];
+ meta = kmap_local_folio(folio0, 0);
if (i_size == 0)
goto new_directory;
nr_blocks = i_size / AFS_DIR_BLOCK_SIZE;
- /* Find a block that has sufficient slots available. Each VM page
+ /* Find a block that has sufficient slots available. Each folio
* contains two or more directory blocks.
*/
for (b = 0; b < nr_blocks + 1; b++) {
- /* If the directory extended into a new page, then we need to
- * tack a new page on the end.
+ /* If the directory extended into a new folio, then we need to
+ * tack a new folio on the end.
*/
index = b / AFS_DIR_BLOCKS_PER_PAGE;
- if (index == 0) {
- page = page0;
- dir_page = meta_page;
- } else {
- if (nr_blocks >= AFS_DIR_MAX_BLOCKS)
- goto error;
- gfp = vnode->vfs_inode.i_mapping->gfp_mask;
- page = find_or_create_page(vnode->vfs_inode.i_mapping,
- index, gfp);
- if (!page)
+ if (nr_blocks >= AFS_DIR_MAX_BLOCKS)
+ goto error;
+ if (index >= folio_nr_pages(folio0)) {
+ folio = afs_dir_get_folio(vnode, index);
+ if (!folio)
goto error;
- if (!PagePrivate(page))
- attach_page_private(page, (void *)1);
- dir_page = kmap(page);
+ } else {
+ folio = folio0;
}
+ block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio));
+
/* Abandon the edit if we got a callback break. */
if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))
goto invalidated;
- block = &dir_page->blocks[b % AFS_DIR_BLOCKS_PER_PAGE];
-
_debug("block %u: %2u %3u %u",
b,
(b < AFS_DIR_BLOCKS_WITH_CTR) ? meta->meta.alloc_ctrs[b] : 99,
afs_set_i_size(vnode, (b + 1) * AFS_DIR_BLOCK_SIZE);
}
- /* Only lower dir pages have a counter in the header. */
+ /* Only lower dir blocks have a counter in the header. */
if (b >= AFS_DIR_BLOCKS_WITH_CTR ||
meta->meta.alloc_ctrs[b] >= need_slots) {
/* We need to try and find one or more consecutive
}
}
- if (page != page0) {
- unlock_page(page);
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
}
i_size = AFS_DIR_BLOCK_SIZE;
afs_set_i_size(vnode, i_size);
slot = AFS_DIR_RESV_BLOCKS0;
- page = page0;
- block = meta;
+ folio = folio0;
+ block = kmap_local_folio(folio, 0);
nr_blocks = 1;
b = 0;
/* Adjust the bitmap. */
afs_set_contig_bits(block, slot, need_slots);
- if (page != page0) {
- unlock_page(page);
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
/* Adjust the allocation counter. */
_debug("Insert %s in %u[%u]", name->name, b, slot);
out_unmap:
- unlock_page(page0);
- kunmap(page0);
- put_page(page0);
+ kunmap_local(meta);
+ folio_unlock(folio0);
+ folio_put(folio0);
_leave("");
return;
invalidated:
trace_afs_edit_dir(vnode, why, afs_edit_dir_create_inval, 0, 0, 0, 0, name->name);
clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
- if (page != page0) {
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
goto out_unmap;
void afs_edit_dir_remove(struct afs_vnode *vnode,
struct qstr *name, enum afs_edit_dir_reason why)
{
- struct afs_xdr_dir_page *meta_page, *dir_page;
union afs_xdr_dir_block *meta, *block;
union afs_xdr_dirent *de;
- struct page *page0, *page;
+ struct folio *folio0, *folio;
unsigned int need_slots, nr_blocks, b;
pgoff_t index;
loff_t i_size;
}
nr_blocks = i_size / AFS_DIR_BLOCK_SIZE;
- page0 = find_lock_page(vnode->vfs_inode.i_mapping, 0);
- if (!page0) {
- clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
+ folio0 = afs_dir_get_folio(vnode, 0);
+ if (!folio0) {
_leave(" [fgp]");
return;
}
/* Work out how many slots we're going to discard. */
need_slots = afs_dir_calc_slots(name->len);
- meta_page = kmap(page0);
- meta = &meta_page->blocks[0];
+ meta = kmap_local_folio(folio0, 0);
- /* Find a page that has sufficient slots available. Each VM page
+ /* Find a block that has sufficient slots available. Each folio
* contains two or more directory blocks.
*/
for (b = 0; b < nr_blocks; b++) {
index = b / AFS_DIR_BLOCKS_PER_PAGE;
- if (index != 0) {
- page = find_lock_page(vnode->vfs_inode.i_mapping, index);
- if (!page)
+ if (index >= folio_nr_pages(folio0)) {
+ folio = afs_dir_get_folio(vnode, index);
+ if (!folio)
goto error;
- dir_page = kmap(page);
} else {
- page = page0;
- dir_page = meta_page;
+ folio = folio0;
}
+ block = kmap_local_folio(folio, b * AFS_DIR_BLOCK_SIZE - folio_file_pos(folio));
+
/* Abandon the edit if we got a callback break. */
if (!test_bit(AFS_VNODE_DIR_VALID, &vnode->flags))
goto invalidated;
- block = &dir_page->blocks[b % AFS_DIR_BLOCKS_PER_PAGE];
-
if (b > AFS_DIR_BLOCKS_WITH_CTR ||
meta->meta.alloc_ctrs[b] <= AFS_DIR_SLOTS_PER_BLOCK - 1 - need_slots) {
slot = afs_dir_scan_block(block, name, b);
goto found_dirent;
}
- if (page != page0) {
- unlock_page(page);
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
}
/* Adjust the bitmap. */
afs_clear_contig_bits(block, slot, need_slots);
- if (page != page0) {
- unlock_page(page);
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
/* Adjust the allocation counter. */
_debug("Remove %s from %u[%u]", name->name, b, slot);
out_unmap:
- unlock_page(page0);
- kunmap(page0);
- put_page(page0);
+ kunmap_local(meta);
+ folio_unlock(folio0);
+ folio_put(folio0);
_leave("");
return;
trace_afs_edit_dir(vnode, why, afs_edit_dir_delete_inval,
0, 0, 0, 0, name->name);
clear_bit(AFS_VNODE_DIR_VALID, &vnode->flags);
- if (page != page0) {
- unlock_page(page);
- kunmap(page);
- put_page(page);
+ kunmap_local(block);
+ if (folio != folio0) {
+ folio_unlock(folio);
+ folio_put(folio);
}
goto out_unmap;
{
struct afs_vnode *vnode = AFS_FS_I(page->mapping->host);
struct afs_read *fsreq;
+ struct folio *folio = page_folio(page);
int ret;
fsreq = afs_alloc_read(GFP_NOFS);
if (!fsreq)
return -ENOMEM;
- fsreq->pos = page->index * PAGE_SIZE;
- fsreq->len = PAGE_SIZE;
+ fsreq->pos = folio_pos(folio);
+ fsreq->len = folio_size(folio);
fsreq->vnode = vnode;
fsreq->iter = &fsreq->def_iter;
iov_iter_xarray(&fsreq->def_iter, READ, &page->mapping->i_pages,
fsreq->pos, fsreq->len);
ret = afs_fetch_data(fsreq->vnode, fsreq);
- page_endio(page, false, ret);
+ if (ret == 0)
+ SetPageUptodate(page);
+ unlock_page(page);
return ret;
}
}
static int afs_check_write_begin(struct file *file, loff_t pos, unsigned len,
- struct page *page, void **_fsdata)
+ struct folio *folio, void **_fsdata)
{
struct afs_vnode *vnode = AFS_FS_I(file_inode(file));
static int afs_readpage(struct file *file, struct page *page)
{
- return netfs_readpage(file, page, &afs_req_ops, NULL);
+ struct folio *folio = page_folio(page);
+
+ return netfs_readpage(file, folio, &afs_req_ops, NULL);
}
static void afs_readahead(struct readahead_control *ractl)
* Adjust the dirty region of the page on truncation or full invalidation,
* getting rid of the markers altogether if the region is entirely invalidated.
*/
-static void afs_invalidate_dirty(struct page *page, unsigned int offset,
+static void afs_invalidate_dirty(struct folio *folio, unsigned int offset,
unsigned int length)
{
- struct afs_vnode *vnode = AFS_FS_I(page->mapping->host);
+ struct afs_vnode *vnode = AFS_FS_I(folio_inode(folio));
unsigned long priv;
unsigned int f, t, end = offset + length;
- priv = page_private(page);
+ priv = (unsigned long)folio_get_private(folio);
/* we clean up only if the entire page is being invalidated */
- if (offset == 0 && length == thp_size(page))
+ if (offset == 0 && length == folio_size(folio))
goto full_invalidate;
/* If the page was dirtied by page_mkwrite(), the PTE stays writable
* and we don't get another notification to tell us to expand it
* again.
*/
- if (afs_is_page_dirty_mmapped(priv))
+ if (afs_is_folio_dirty_mmapped(priv))
return;
/* We may need to shorten the dirty region */
- f = afs_page_dirty_from(page, priv);
- t = afs_page_dirty_to(page, priv);
+ f = afs_folio_dirty_from(folio, priv);
+ t = afs_folio_dirty_to(folio, priv);
if (t <= offset || f >= end)
return; /* Doesn't overlap */
if (f == t)
goto undirty;
- priv = afs_page_dirty(page, f, t);
- set_page_private(page, priv);
- trace_afs_page_dirty(vnode, tracepoint_string("trunc"), page);
+ priv = afs_folio_dirty(folio, f, t);
+ folio_change_private(folio, (void *)priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("trunc"), folio);
return;
undirty:
- trace_afs_page_dirty(vnode, tracepoint_string("undirty"), page);
- clear_page_dirty_for_io(page);
+ trace_afs_folio_dirty(vnode, tracepoint_string("undirty"), folio);
+ folio_clear_dirty_for_io(folio);
full_invalidate:
- trace_afs_page_dirty(vnode, tracepoint_string("inval"), page);
- detach_page_private(page);
+ trace_afs_folio_dirty(vnode, tracepoint_string("inval"), folio);
+ folio_detach_private(folio);
}
/*
static void afs_invalidatepage(struct page *page, unsigned int offset,
unsigned int length)
{
- _enter("{%lu},%u,%u", page->index, offset, length);
+ struct folio *folio = page_folio(page);
+
+ _enter("{%lu},%u,%u", folio_index(folio), offset, length);
BUG_ON(!PageLocked(page));
if (PagePrivate(page))
- afs_invalidate_dirty(page, offset, length);
+ afs_invalidate_dirty(folio, offset, length);
- wait_on_page_fscache(page);
+ folio_wait_fscache(folio);
_leave("");
}
*/
static int afs_releasepage(struct page *page, gfp_t gfp_flags)
{
- struct afs_vnode *vnode = AFS_FS_I(page->mapping->host);
+ struct folio *folio = page_folio(page);
+ struct afs_vnode *vnode = AFS_FS_I(folio_inode(folio));
_enter("{{%llx:%llu}[%lu],%lx},%x",
- vnode->fid.vid, vnode->fid.vnode, page->index, page->flags,
+ vnode->fid.vid, vnode->fid.vnode, folio_index(folio), folio->flags,
gfp_flags);
/* deny if page is being written to the cache and the caller hasn't
* elected to wait */
#ifdef CONFIG_AFS_FSCACHE
- if (PageFsCache(page)) {
+ if (folio_test_fscache(folio)) {
if (!(gfp_flags & __GFP_DIRECT_RECLAIM) || !(gfp_flags & __GFP_FS))
return false;
- wait_on_page_fscache(page);
+ folio_wait_fscache(folio);
}
#endif
- if (PagePrivate(page)) {
- trace_afs_page_dirty(vnode, tracepoint_string("rel"), page);
- detach_page_private(page);
+ if (folio_test_private(folio)) {
+ trace_afs_folio_dirty(vnode, tracepoint_string("rel"), folio);
+ folio_detach_private(folio);
}
- /* indicate that the page can be released */
+ /* Indicate that the folio can be released */
_leave(" = T");
- return 1;
+ return true;
}
static void afs_add_open_mmap(struct afs_vnode *vnode)
} __packed;
/*
- * We use page->private to hold the amount of the page that we've written to,
+ * We use folio->private to hold the amount of the folio that we've written to,
* splitting the field into two parts. However, we need to represent a range
- * 0...PAGE_SIZE, so we reduce the resolution if the size of the page
+ * 0...FOLIO_SIZE, so we reduce the resolution if the size of the folio
* exceeds what we can encode.
*/
#ifdef CONFIG_64BIT
-#define __AFS_PAGE_PRIV_MASK 0x7fffffffUL
-#define __AFS_PAGE_PRIV_SHIFT 32
-#define __AFS_PAGE_PRIV_MMAPPED 0x80000000UL
+#define __AFS_FOLIO_PRIV_MASK 0x7fffffffUL
+#define __AFS_FOLIO_PRIV_SHIFT 32
+#define __AFS_FOLIO_PRIV_MMAPPED 0x80000000UL
#else
-#define __AFS_PAGE_PRIV_MASK 0x7fffUL
-#define __AFS_PAGE_PRIV_SHIFT 16
-#define __AFS_PAGE_PRIV_MMAPPED 0x8000UL
+#define __AFS_FOLIO_PRIV_MASK 0x7fffUL
+#define __AFS_FOLIO_PRIV_SHIFT 16
+#define __AFS_FOLIO_PRIV_MMAPPED 0x8000UL
#endif
-static inline unsigned int afs_page_dirty_resolution(struct page *page)
+static inline unsigned int afs_folio_dirty_resolution(struct folio *folio)
{
- int shift = thp_order(page) + PAGE_SHIFT - (__AFS_PAGE_PRIV_SHIFT - 1);
+ int shift = folio_shift(folio) - (__AFS_FOLIO_PRIV_SHIFT - 1);
return (shift > 0) ? shift : 0;
}
-static inline size_t afs_page_dirty_from(struct page *page, unsigned long priv)
+static inline size_t afs_folio_dirty_from(struct folio *folio, unsigned long priv)
{
- unsigned long x = priv & __AFS_PAGE_PRIV_MASK;
+ unsigned long x = priv & __AFS_FOLIO_PRIV_MASK;
/* The lower bound is inclusive */
- return x << afs_page_dirty_resolution(page);
+ return x << afs_folio_dirty_resolution(folio);
}
-static inline size_t afs_page_dirty_to(struct page *page, unsigned long priv)
+static inline size_t afs_folio_dirty_to(struct folio *folio, unsigned long priv)
{
- unsigned long x = (priv >> __AFS_PAGE_PRIV_SHIFT) & __AFS_PAGE_PRIV_MASK;
+ unsigned long x = (priv >> __AFS_FOLIO_PRIV_SHIFT) & __AFS_FOLIO_PRIV_MASK;
/* The upper bound is immediately beyond the region */
- return (x + 1) << afs_page_dirty_resolution(page);
+ return (x + 1) << afs_folio_dirty_resolution(folio);
}
-static inline unsigned long afs_page_dirty(struct page *page, size_t from, size_t to)
+static inline unsigned long afs_folio_dirty(struct folio *folio, size_t from, size_t to)
{
- unsigned int res = afs_page_dirty_resolution(page);
+ unsigned int res = afs_folio_dirty_resolution(folio);
from >>= res;
to = (to - 1) >> res;
- return (to << __AFS_PAGE_PRIV_SHIFT) | from;
+ return (to << __AFS_FOLIO_PRIV_SHIFT) | from;
}
-static inline unsigned long afs_page_dirty_mmapped(unsigned long priv)
+static inline unsigned long afs_folio_dirty_mmapped(unsigned long priv)
{
- return priv | __AFS_PAGE_PRIV_MMAPPED;
+ return priv | __AFS_FOLIO_PRIV_MMAPPED;
}
-static inline bool afs_is_page_dirty_mmapped(unsigned long priv)
+static inline bool afs_is_folio_dirty_mmapped(unsigned long priv)
{
- return priv & __AFS_PAGE_PRIV_MMAPPED;
+ return priv & __AFS_FOLIO_PRIV_MMAPPED;
}
#include <trace/events/afs.h>
struct page **_page, void **fsdata)
{
struct afs_vnode *vnode = AFS_FS_I(file_inode(file));
- struct page *page;
+ struct folio *folio;
unsigned long priv;
unsigned f, from;
unsigned t, to;
* file. We need to do this before we get a lock on the page in case
* there's more than one writer competing for the same cache block.
*/
- ret = netfs_write_begin(file, mapping, pos, len, flags, &page, fsdata,
+ ret = netfs_write_begin(file, mapping, pos, len, flags, &folio, fsdata,
&afs_req_ops, NULL);
if (ret < 0)
return ret;
- index = page->index;
+ index = folio_index(folio);
from = pos - index * PAGE_SIZE;
to = from + len;
/* See if this page is already partially written in a way that we can
* merge the new write with.
*/
- if (PagePrivate(page)) {
- priv = page_private(page);
- f = afs_page_dirty_from(page, priv);
- t = afs_page_dirty_to(page, priv);
+ if (folio_test_private(folio)) {
+ priv = (unsigned long)folio_get_private(folio);
+ f = afs_folio_dirty_from(folio, priv);
+ t = afs_folio_dirty_to(folio, priv);
ASSERTCMP(f, <=, t);
- if (PageWriteback(page)) {
- trace_afs_page_dirty(vnode, tracepoint_string("alrdy"), page);
+ if (folio_test_writeback(folio)) {
+ trace_afs_folio_dirty(vnode, tracepoint_string("alrdy"), folio);
goto flush_conflicting_write;
}
/* If the file is being filled locally, allow inter-write
goto flush_conflicting_write;
}
- *_page = page;
+ *_page = &folio->page;
_leave(" = 0");
return 0;
*/
flush_conflicting_write:
_debug("flush conflict");
- ret = write_one_page(page);
+ ret = folio_write_one(folio);
if (ret < 0)
goto error;
- ret = lock_page_killable(page);
+ ret = folio_lock_killable(folio);
if (ret < 0)
goto error;
goto try_again;
error:
- put_page(page);
+ folio_put(folio);
_leave(" = %d", ret);
return ret;
}
*/
int afs_write_end(struct file *file, struct address_space *mapping,
loff_t pos, unsigned len, unsigned copied,
- struct page *page, void *fsdata)
+ struct page *subpage, void *fsdata)
{
+ struct folio *folio = page_folio(subpage);
struct afs_vnode *vnode = AFS_FS_I(file_inode(file));
unsigned long priv;
- unsigned int f, from = pos & (thp_size(page) - 1);
+ unsigned int f, from = offset_in_folio(folio, pos);
unsigned int t, to = from + copied;
loff_t i_size, maybe_i_size;
_enter("{%llx:%llu},{%lx}",
- vnode->fid.vid, vnode->fid.vnode, page->index);
+ vnode->fid.vid, vnode->fid.vnode, folio_index(folio));
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
if (copied < len) {
copied = 0;
goto out;
}
- SetPageUptodate(page);
+ folio_mark_uptodate(folio);
}
if (copied == 0)
write_sequnlock(&vnode->cb_lock);
}
- if (PagePrivate(page)) {
- priv = page_private(page);
- f = afs_page_dirty_from(page, priv);
- t = afs_page_dirty_to(page, priv);
+ if (folio_test_private(folio)) {
+ priv = (unsigned long)folio_get_private(folio);
+ f = afs_folio_dirty_from(folio, priv);
+ t = afs_folio_dirty_to(folio, priv);
if (from < f)
f = from;
if (to > t)
t = to;
- priv = afs_page_dirty(page, f, t);
- set_page_private(page, priv);
- trace_afs_page_dirty(vnode, tracepoint_string("dirty+"), page);
+ priv = afs_folio_dirty(folio, f, t);
+ folio_change_private(folio, (void *)priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("dirty+"), folio);
} else {
- priv = afs_page_dirty(page, from, to);
- attach_page_private(page, (void *)priv);
- trace_afs_page_dirty(vnode, tracepoint_string("dirty"), page);
+ priv = afs_folio_dirty(folio, from, to);
+ folio_attach_private(folio, (void *)priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("dirty"), folio);
}
- if (set_page_dirty(page))
- _debug("dirtied %lx", page->index);
+ if (folio_mark_dirty(folio))
+ _debug("dirtied %lx", folio_index(folio));
out:
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
return copied;
}
loff_t start, loff_t len)
{
struct afs_vnode *vnode = AFS_FS_I(mapping->host);
- struct pagevec pv;
- unsigned int loop, psize;
+ struct folio *folio;
+ pgoff_t index = start / PAGE_SIZE;
+ pgoff_t last = (start + len - 1) / PAGE_SIZE, next;
_enter("{%llx:%llu},%llx @%llx",
vnode->fid.vid, vnode->fid.vnode, len, start);
- pagevec_init(&pv);
-
do {
- _debug("kill %llx @%llx", len, start);
-
- pv.nr = find_get_pages_contig(mapping, start / PAGE_SIZE,
- PAGEVEC_SIZE, pv.pages);
- if (pv.nr == 0)
- break;
+ _debug("kill %lx (to %lx)", index, last);
- for (loop = 0; loop < pv.nr; loop++) {
- struct page *page = pv.pages[loop];
+ folio = filemap_get_folio(mapping, index);
+ if (!folio) {
+ next = index + 1;
+ continue;
+ }
- if (page->index * PAGE_SIZE >= start + len)
- break;
+ next = folio_next_index(folio);
- psize = thp_size(page);
- start += psize;
- len -= psize;
- ClearPageUptodate(page);
- end_page_writeback(page);
- lock_page(page);
- generic_error_remove_page(mapping, page);
- unlock_page(page);
- }
+ folio_clear_uptodate(folio);
+ folio_end_writeback(folio);
+ folio_lock(folio);
+ generic_error_remove_page(mapping, &folio->page);
+ folio_unlock(folio);
+ folio_put(folio);
- __pagevec_release(&pv);
- } while (len > 0);
+ } while (index = next, index <= last);
_leave("");
}
loff_t start, loff_t len)
{
struct afs_vnode *vnode = AFS_FS_I(mapping->host);
- struct pagevec pv;
- unsigned int loop, psize;
+ struct folio *folio;
+ pgoff_t index = start / PAGE_SIZE;
+ pgoff_t last = (start + len - 1) / PAGE_SIZE, next;
_enter("{%llx:%llu},%llx @%llx",
vnode->fid.vid, vnode->fid.vnode, len, start);
- pagevec_init(&pv);
-
do {
_debug("redirty %llx @%llx", len, start);
- pv.nr = find_get_pages_contig(mapping, start / PAGE_SIZE,
- PAGEVEC_SIZE, pv.pages);
- if (pv.nr == 0)
- break;
-
- for (loop = 0; loop < pv.nr; loop++) {
- struct page *page = pv.pages[loop];
-
- if (page->index * PAGE_SIZE >= start + len)
- break;
-
- psize = thp_size(page);
- start += psize;
- len -= psize;
- redirty_page_for_writepage(wbc, page);
- end_page_writeback(page);
+ folio = filemap_get_folio(mapping, index);
+ if (!folio) {
+ next = index + 1;
+ continue;
}
- __pagevec_release(&pv);
- } while (len > 0);
+ next = index + folio_nr_pages(folio);
+ folio_redirty_for_writepage(wbc, folio);
+ folio_end_writeback(folio);
+ folio_put(folio);
+ } while (index = next, index <= last);
_leave("");
}
static void afs_pages_written_back(struct afs_vnode *vnode, loff_t start, unsigned int len)
{
struct address_space *mapping = vnode->vfs_inode.i_mapping;
- struct page *page;
+ struct folio *folio;
pgoff_t end;
XA_STATE(xas, &mapping->i_pages, start / PAGE_SIZE);
rcu_read_lock();
end = (start + len - 1) / PAGE_SIZE;
- xas_for_each(&xas, page, end) {
- if (!PageWriteback(page)) {
- kdebug("bad %x @%llx page %lx %lx", len, start, page->index, end);
- ASSERT(PageWriteback(page));
+ xas_for_each(&xas, folio, end) {
+ if (!folio_test_writeback(folio)) {
+ kdebug("bad %x @%llx page %lx %lx",
+ len, start, folio_index(folio), end);
+ ASSERT(folio_test_writeback(folio));
}
- trace_afs_page_dirty(vnode, tracepoint_string("clear"), page);
- detach_page_private(page);
- page_endio(page, true, 0);
+ trace_afs_folio_dirty(vnode, tracepoint_string("clear"), folio);
+ folio_detach_private(folio);
+ folio_end_writeback(folio);
}
rcu_read_unlock();
unsigned int *_len)
{
struct pagevec pvec;
- struct page *page;
+ struct folio *folio;
unsigned long priv;
unsigned int psize, filler = 0;
unsigned int f, t;
*/
rcu_read_lock();
- xas_for_each(&xas, page, ULONG_MAX) {
+ xas_for_each(&xas, folio, ULONG_MAX) {
stop = true;
- if (xas_retry(&xas, page))
+ if (xas_retry(&xas, folio))
continue;
- if (xa_is_value(page))
+ if (xa_is_value(folio))
break;
- if (page->index != index)
+ if (folio_index(folio) != index)
break;
- if (!page_cache_get_speculative(page)) {
+ if (!folio_try_get_rcu(folio)) {
xas_reset(&xas);
continue;
}
/* Has the page moved or been split? */
- if (unlikely(page != xas_reload(&xas))) {
- put_page(page);
+ if (unlikely(folio != xas_reload(&xas))) {
+ folio_put(folio);
break;
}
- if (!trylock_page(page)) {
- put_page(page);
+ if (!folio_trylock(folio)) {
+ folio_put(folio);
break;
}
- if (!PageDirty(page) || PageWriteback(page)) {
- unlock_page(page);
- put_page(page);
+ if (!folio_test_dirty(folio) || folio_test_writeback(folio)) {
+ folio_unlock(folio);
+ folio_put(folio);
break;
}
- psize = thp_size(page);
- priv = page_private(page);
- f = afs_page_dirty_from(page, priv);
- t = afs_page_dirty_to(page, priv);
+ psize = folio_size(folio);
+ priv = (unsigned long)folio_get_private(folio);
+ f = afs_folio_dirty_from(folio, priv);
+ t = afs_folio_dirty_to(folio, priv);
if (f != 0 && !new_content) {
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
break;
}
else if (t == psize || new_content)
stop = false;
- index += thp_nr_pages(page);
- if (!pagevec_add(&pvec, page))
+ index += folio_nr_pages(folio);
+ if (!pagevec_add(&pvec, &folio->page))
break;
if (stop)
break;
break;
for (i = 0; i < pagevec_count(&pvec); i++) {
- page = pvec.pages[i];
- trace_afs_page_dirty(vnode, tracepoint_string("store+"), page);
+ folio = page_folio(pvec.pages[i]);
+ trace_afs_folio_dirty(vnode, tracepoint_string("store+"), folio);
- if (!clear_page_dirty_for_io(page))
+ if (!folio_clear_dirty_for_io(folio))
BUG();
- if (test_set_page_writeback(page))
+ if (folio_start_writeback(folio))
BUG();
- *_count -= thp_nr_pages(page);
- unlock_page(page);
+ *_count -= folio_nr_pages(folio);
+ folio_unlock(folio);
}
pagevec_release(&pvec);
* Synchronously write back the locked page and any subsequent non-locked dirty
* pages.
*/
-static ssize_t afs_write_back_from_locked_page(struct address_space *mapping,
- struct writeback_control *wbc,
- struct page *page,
- loff_t start, loff_t end)
+static ssize_t afs_write_back_from_locked_folio(struct address_space *mapping,
+ struct writeback_control *wbc,
+ struct folio *folio,
+ loff_t start, loff_t end)
{
struct afs_vnode *vnode = AFS_FS_I(mapping->host);
struct iov_iter iter;
long count = wbc->nr_to_write;
int ret;
- _enter(",%lx,%llx-%llx", page->index, start, end);
+ _enter(",%lx,%llx-%llx", folio_index(folio), start, end);
- if (test_set_page_writeback(page))
+ if (folio_start_writeback(folio))
BUG();
- count -= thp_nr_pages(page);
+ count -= folio_nr_pages(folio);
/* Find all consecutive lockable dirty pages that have contiguous
* written regions, stopping when we find a page that is not
* immediately lockable, is not dirty or is missing, or we reach the
* end of the range.
*/
- priv = page_private(page);
- offset = afs_page_dirty_from(page, priv);
- to = afs_page_dirty_to(page, priv);
- trace_afs_page_dirty(vnode, tracepoint_string("store"), page);
+ priv = (unsigned long)folio_get_private(folio);
+ offset = afs_folio_dirty_from(folio, priv);
+ to = afs_folio_dirty_to(folio, priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("store"), folio);
len = to - offset;
start += offset;
max_len = min_t(unsigned long long, max_len, i_size - start);
if (len < max_len &&
- (to == thp_size(page) || new_content))
+ (to == folio_size(folio) || new_content))
afs_extend_writeback(mapping, vnode, &count,
start, max_len, new_content, &len);
len = min_t(loff_t, len, max_len);
* set; the first page is still locked at this point, but all the rest
* have been unlocked.
*/
- unlock_page(page);
+ folio_unlock(folio);
if (start < i_size) {
_debug("write back %x @%llx [%llx]", len, start, i_size);
* write a page back to the server
* - the caller locked the page for us
*/
-int afs_writepage(struct page *page, struct writeback_control *wbc)
+int afs_writepage(struct page *subpage, struct writeback_control *wbc)
{
+ struct folio *folio = page_folio(subpage);
ssize_t ret;
loff_t start;
- _enter("{%lx},", page->index);
+ _enter("{%lx},", folio_index(folio));
- start = page->index * PAGE_SIZE;
- ret = afs_write_back_from_locked_page(page->mapping, wbc, page,
- start, LLONG_MAX - start);
+ start = folio_index(folio) * PAGE_SIZE;
+ ret = afs_write_back_from_locked_folio(folio_mapping(folio), wbc,
+ folio, start, LLONG_MAX - start);
if (ret < 0) {
_leave(" = %zd", ret);
return ret;
struct writeback_control *wbc,
loff_t start, loff_t end, loff_t *_next)
{
- struct page *page;
+ struct folio *folio;
+ struct page *head_page;
ssize_t ret;
int n;
pgoff_t index = start / PAGE_SIZE;
n = find_get_pages_range_tag(mapping, &index, end / PAGE_SIZE,
- PAGECACHE_TAG_DIRTY, 1, &page);
+ PAGECACHE_TAG_DIRTY, 1, &head_page);
if (!n)
break;
- start = (loff_t)page->index * PAGE_SIZE; /* May regress with THPs */
+ folio = page_folio(head_page);
+ start = folio_pos(folio); /* May regress with THPs */
- _debug("wback %lx", page->index);
+ _debug("wback %lx", folio_index(folio));
/* At this point we hold neither the i_pages lock nor the
* page lock: the page may be truncated or invalidated
* back from swapper_space to tmpfs file mapping
*/
if (wbc->sync_mode != WB_SYNC_NONE) {
- ret = lock_page_killable(page);
+ ret = folio_lock_killable(folio);
if (ret < 0) {
- put_page(page);
+ folio_put(folio);
return ret;
}
} else {
- if (!trylock_page(page)) {
- put_page(page);
+ if (!folio_trylock(folio)) {
+ folio_put(folio);
return 0;
}
}
- if (page->mapping != mapping || !PageDirty(page)) {
- start += thp_size(page);
- unlock_page(page);
- put_page(page);
+ if (folio_mapping(folio) != mapping ||
+ !folio_test_dirty(folio)) {
+ start += folio_size(folio);
+ folio_unlock(folio);
+ folio_put(folio);
continue;
}
- if (PageWriteback(page)) {
- unlock_page(page);
+ if (folio_test_writeback(folio)) {
+ folio_unlock(folio);
if (wbc->sync_mode != WB_SYNC_NONE)
- wait_on_page_writeback(page);
- put_page(page);
+ folio_wait_writeback(folio);
+ folio_put(folio);
continue;
}
- if (!clear_page_dirty_for_io(page))
+ if (!folio_clear_dirty_for_io(folio))
BUG();
- ret = afs_write_back_from_locked_page(mapping, wbc, page, start, end);
- put_page(page);
+ ret = afs_write_back_from_locked_folio(mapping, wbc, folio, start, end);
+ folio_put(folio);
if (ret < 0) {
_leave(" = %zd", ret);
return ret;
vm_fault_t afs_page_mkwrite(struct vm_fault *vmf)
{
struct folio *folio = page_folio(vmf->page);
- struct page *page = &folio->page;
struct file *file = vmf->vma->vm_file;
struct inode *inode = file_inode(file);
struct afs_vnode *vnode = AFS_FS_I(inode);
unsigned long priv;
vm_fault_t ret = VM_FAULT_RETRY;
- _enter("{{%llx:%llu}},{%lx}", vnode->fid.vid, vnode->fid.vnode, page->index);
+ _enter("{{%llx:%llu}},{%lx}", vnode->fid.vid, vnode->fid.vnode, folio_index(folio));
afs_validate(vnode, af->key);
* be modified. We then assume the entire page will need writing back.
*/
#ifdef CONFIG_AFS_FSCACHE
- if (PageFsCache(page) &&
- wait_on_page_fscache_killable(page) < 0)
+ if (folio_test_fscache(folio) &&
+ folio_wait_fscache_killable(folio) < 0)
goto out;
#endif
if (folio_wait_writeback_killable(folio))
goto out;
- if (lock_page_killable(page) < 0)
+ if (folio_lock_killable(folio) < 0)
goto out;
- /* We mustn't change page->private until writeback is complete as that
+ /* We mustn't change folio->private until writeback is complete as that
* details the portion of the page we need to write back and we might
* need to redirty the page if there's a problem.
*/
goto out;
}
- priv = afs_page_dirty(page, 0, thp_size(page));
- priv = afs_page_dirty_mmapped(priv);
- if (PagePrivate(page)) {
- set_page_private(page, priv);
- trace_afs_page_dirty(vnode, tracepoint_string("mkwrite+"), page);
+ priv = afs_folio_dirty(folio, 0, folio_size(folio));
+ priv = afs_folio_dirty_mmapped(priv);
+ if (folio_test_private(folio)) {
+ folio_change_private(folio, (void *)priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("mkwrite+"), folio);
} else {
- attach_page_private(page, (void *)priv);
- trace_afs_page_dirty(vnode, tracepoint_string("mkwrite"), page);
+ folio_attach_private(folio, (void *)priv);
+ trace_afs_folio_dirty(vnode, tracepoint_string("mkwrite"), folio);
}
file_update_time(file);
/*
* Clean up a page during invalidation.
*/
-int afs_launder_page(struct page *page)
+int afs_launder_page(struct page *subpage)
{
- struct address_space *mapping = page->mapping;
- struct afs_vnode *vnode = AFS_FS_I(mapping->host);
+ struct folio *folio = page_folio(subpage);
+ struct afs_vnode *vnode = AFS_FS_I(folio_inode(folio));
struct iov_iter iter;
struct bio_vec bv[1];
unsigned long priv;
unsigned int f, t;
int ret = 0;
- _enter("{%lx}", page->index);
+ _enter("{%lx}", folio_index(folio));
- priv = page_private(page);
- if (clear_page_dirty_for_io(page)) {
+ priv = (unsigned long)folio_get_private(folio);
+ if (folio_clear_dirty_for_io(folio)) {
f = 0;
- t = thp_size(page);
- if (PagePrivate(page)) {
- f = afs_page_dirty_from(page, priv);
- t = afs_page_dirty_to(page, priv);
+ t = folio_size(folio);
+ if (folio_test_private(folio)) {
+ f = afs_folio_dirty_from(folio, priv);
+ t = afs_folio_dirty_to(folio, priv);
}
- bv[0].bv_page = page;
+ bv[0].bv_page = &folio->page;
bv[0].bv_offset = f;
bv[0].bv_len = t - f;
iov_iter_bvec(&iter, WRITE, bv, 1, bv[0].bv_len);
- trace_afs_page_dirty(vnode, tracepoint_string("launder"), page);
- ret = afs_store_data(vnode, &iter, page_offset(page) + f, true);
+ trace_afs_folio_dirty(vnode, tracepoint_string("launder"), folio);
+ ret = afs_store_data(vnode, &iter, folio_pos(folio) + f, true);
}
- trace_afs_page_dirty(vnode, tracepoint_string("laundered"), page);
- detach_page_private(page);
- wait_on_page_fscache(page);
+ trace_afs_folio_dirty(vnode, tracepoint_string("laundered"), folio);
+ folio_detach_private(folio);
+ folio_wait_fscache(folio);
return ret;
}
kuid_t uid)
{
kuid_t kuid = i_uid_into_mnt(mnt_userns, inode);
- if (uid_eq(current_fsuid(), kuid) && uid_eq(uid, kuid))
+ if (uid_eq(current_fsuid(), kuid) && uid_eq(uid, inode->i_uid))
return true;
if (capable_wrt_inode_uidgid(mnt_userns, inode, CAP_CHOWN))
return true;
{
kgid_t kgid = i_gid_into_mnt(mnt_userns, inode);
if (uid_eq(current_fsuid(), i_uid_into_mnt(mnt_userns, inode)) &&
- (in_group_p(gid) || gid_eq(gid, kgid)))
+ (in_group_p(gid) || gid_eq(gid, inode->i_gid)))
return true;
if (capable_wrt_inode_uidgid(mnt_userns, inode, CAP_CHOWN))
return true;
#define STACK_ADD(sp, items) ((elf_addr_t __user *)(sp) - (items))
#define STACK_ROUND(sp, items) \
(((unsigned long) (sp - items)) &~ 15UL)
-#define STACK_ALLOC(sp, len) ({ sp -= len ; sp; })
+#define STACK_ALLOC(sp, len) (sp -= len)
#endif
#ifndef ELF_BASE_PLATFORM
vaddr = elf_ppnt->p_vaddr;
/*
- * If we are loading ET_EXEC or we have already performed
- * the ET_DYN load_addr calculations, proceed normally.
+ * The first time through the loop, load_addr_set is false:
+ * layout will be calculated. Once set, use MAP_FIXED since
+ * we know we've already safely mapped the entire region with
+ * MAP_FIXED_NOREPLACE in the once-per-binary logic following.
*/
- if (elf_ex->e_type == ET_EXEC || load_addr_set) {
+ if (load_addr_set) {
elf_flags |= MAP_FIXED;
+ } else if (elf_ex->e_type == ET_EXEC) {
+ /*
+ * This logic is run once for the first LOAD Program
+ * Header for ET_EXEC binaries. No special handling
+ * is needed.
+ */
+ elf_flags |= MAP_FIXED_NOREPLACE;
} else if (elf_ex->e_type == ET_DYN) {
/*
* This logic is run once for the first LOAD Program
* Header for ET_DYN binaries to calculate the
* randomization (load_bias) for all the LOAD
- * Program Headers, and to calculate the entire
- * size of the ELF mapping (total_size). (Note that
- * load_addr_set is set to true later once the
- * initial mapping is performed.)
+ * Program Headers.
*
* There are effectively two types of ET_DYN
* binaries: programs (i.e. PIE: ET_DYN with INTERP)
* Therefore, programs are loaded offset from
* ELF_ET_DYN_BASE and loaders are loaded into the
* independently randomized mmap region (0 load_bias
- * without MAP_FIXED).
+ * without MAP_FIXED nor MAP_FIXED_NOREPLACE).
*/
if (interpreter) {
load_bias = ELF_ET_DYN_BASE;
alignment = maximum_alignment(elf_phdata, elf_ex->e_phnum);
if (alignment)
load_bias &= ~(alignment - 1);
- elf_flags |= MAP_FIXED;
+ elf_flags |= MAP_FIXED_NOREPLACE;
} else
load_bias = 0;
* is then page aligned.
*/
load_bias = ELF_PAGESTART(load_bias - vaddr);
+ }
+ /*
+ * Calculate the entire size of the ELF mapping (total_size).
+ * (Note that load_addr_set is set to true later once the
+ * initial mapping is performed.)
+ */
+ if (!load_addr_set) {
total_size = total_mapping_size(elf_phdata,
elf_ex->e_phnum);
if (!total_size) {
ordered_list);
if (!test_bit(WORK_DONE_BIT, &work->flags))
break;
+ /*
+ * Orders all subsequent loads after reading WORK_DONE_BIT,
+ * paired with the smp_mb__before_atomic in btrfs_work_helper
+ * this guarantees that the ordered function will see all
+ * updates from ordinary work function.
+ */
+ smp_rmb();
/*
* we are going to call the ordered done function, but
thresh_exec_hook(wq);
work->func(work);
if (need_order) {
+ /*
+ * Ensures all memory accesses done in the work function are
+ * ordered before setting the WORK_DONE_BIT. Ensuring the thread
+ * which is going to executed the ordered work sees them.
+ * Pairs with the smp_rmb in run_ordered_work.
+ */
+ smp_mb__before_atomic();
set_bit(WORK_DONE_BIT, &work->flags);
run_ordered_work(wq, work);
} else {
*/
static void write_dev_flush(struct btrfs_device *device)
{
- struct request_queue *q = bdev_get_queue(device->bdev);
struct bio *bio = device->flush_bio;
+#ifndef CONFIG_BTRFS_FS_CHECK_INTEGRITY
+ /*
+ * When a disk has write caching disabled, we skip submission of a bio
+ * with flush and sync requests before writing the superblock, since
+ * it's not needed. However when the integrity checker is enabled, this
+ * results in reports that there are metadata blocks referred by a
+ * superblock that were not properly flushed. So don't skip the bio
+ * submission only when the integrity checker is enabled for the sake
+ * of simplicity, since this is a debug tool and not meant for use in
+ * non-debug builds.
+ */
+ struct request_queue *q = bdev_get_queue(device->bdev);
if (!test_bit(QUEUE_FLAG_WC, &q->queue_flags))
return;
+#endif
bio_reset(bio);
bio->bi_end_io = btrfs_end_empty_barrier;
static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from)
{
+ const bool is_sync_write = (iocb->ki_flags & IOCB_DSYNC);
struct file *file = iocb->ki_filp;
struct inode *inode = file_inode(file);
struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb);
loff_t pos;
ssize_t written = 0;
ssize_t written_buffered;
+ size_t prev_left = 0;
loff_t endbyte;
ssize_t err;
unsigned int ilock_flags = 0;
- struct iomap_dio *dio = NULL;
if (iocb->ki_flags & IOCB_NOWAIT)
ilock_flags |= BTRFS_ILOCK_TRY;
goto buffered;
}
- dio = __iomap_dio_rw(iocb, from, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
- 0, 0);
+ /*
+ * We remove IOCB_DSYNC so that we don't deadlock when iomap_dio_rw()
+ * calls generic_write_sync() (through iomap_dio_complete()), because
+ * that results in calling fsync (btrfs_sync_file()) which will try to
+ * lock the inode in exclusive/write mode.
+ */
+ if (is_sync_write)
+ iocb->ki_flags &= ~IOCB_DSYNC;
- btrfs_inode_unlock(inode, ilock_flags);
+ /*
+ * The iov_iter can be mapped to the same file range we are writing to.
+ * If that's the case, then we will deadlock in the iomap code, because
+ * it first calls our callback btrfs_dio_iomap_begin(), which will create
+ * an ordered extent, and after that it will fault in the pages that the
+ * iov_iter refers to. During the fault in we end up in the readahead
+ * pages code (starting at btrfs_readahead()), which will lock the range,
+ * find that ordered extent and then wait for it to complete (at
+ * btrfs_lock_and_flush_ordered_range()), resulting in a deadlock since
+ * obviously the ordered extent can never complete as we didn't submit
+ * yet the respective bio(s). This always happens when the buffer is
+ * memory mapped to the same file range, since the iomap DIO code always
+ * invalidates pages in the target file range (after starting and waiting
+ * for any writeback).
+ *
+ * So here we disable page faults in the iov_iter and then retry if we
+ * got -EFAULT, faulting in the pages before the retry.
+ */
+again:
+ from->nofault = true;
+ err = iomap_dio_rw(iocb, from, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
+ IOMAP_DIO_PARTIAL, written);
+ from->nofault = false;
- if (IS_ERR_OR_NULL(dio)) {
- err = PTR_ERR_OR_ZERO(dio);
- if (err < 0 && err != -ENOTBLK)
- goto out;
- } else {
- written = iomap_dio_complete(dio);
+ /* No increment (+=) because iomap returns a cumulative value. */
+ if (err > 0)
+ written = err;
+
+ if (iov_iter_count(from) > 0 && (err == -EFAULT || err > 0)) {
+ const size_t left = iov_iter_count(from);
+ /*
+ * We have more data left to write. Try to fault in as many as
+ * possible of the remainder pages and retry. We do this without
+ * releasing and locking again the inode, to prevent races with
+ * truncate.
+ *
+ * Also, in case the iov refers to pages in the file range of the
+ * file we want to write to (due to a mmap), we could enter an
+ * infinite loop if we retry after faulting the pages in, since
+ * iomap will invalidate any pages in the range early on, before
+ * it tries to fault in the pages of the iov. So we keep track of
+ * how much was left of iov in the previous EFAULT and fallback
+ * to buffered IO in case we haven't made any progress.
+ */
+ if (left == prev_left) {
+ err = -ENOTBLK;
+ } else {
+ fault_in_iov_iter_readable(from, left);
+ prev_left = left;
+ goto again;
+ }
}
- if (written < 0 || !iov_iter_count(from)) {
- err = written;
+ btrfs_inode_unlock(inode, ilock_flags);
+
+ /*
+ * Add back IOCB_DSYNC. Our caller, btrfs_file_write_iter(), will do
+ * the fsync (call generic_write_sync()).
+ */
+ if (is_sync_write)
+ iocb->ki_flags |= IOCB_DSYNC;
+
+ /* If 'err' is -ENOTBLK then it means we must fallback to buffered IO. */
+ if ((err < 0 && err != -ENOTBLK) || !iov_iter_count(from))
goto out;
- }
buffered:
pos = iocb->ki_pos;
invalidate_mapping_pages(file->f_mapping, pos >> PAGE_SHIFT,
endbyte >> PAGE_SHIFT);
out:
- return written ? written : err;
+ return err < 0 ? err : written;
}
static ssize_t btrfs_file_write_iter(struct kiocb *iocb,
static ssize_t btrfs_direct_read(struct kiocb *iocb, struct iov_iter *to)
{
struct inode *inode = file_inode(iocb->ki_filp);
+ size_t prev_left = 0;
+ ssize_t read = 0;
ssize_t ret;
if (fsverity_active(inode))
return 0;
btrfs_inode_lock(inode, BTRFS_ILOCK_SHARED);
+again:
+ /*
+ * This is similar to what we do for direct IO writes, see the comment
+ * at btrfs_direct_write(), but we also disable page faults in addition
+ * to disabling them only at the iov_iter level. This is because when
+ * reading from a hole or prealloc extent, iomap calls iov_iter_zero(),
+ * which can still trigger page fault ins despite having set ->nofault
+ * to true of our 'to' iov_iter.
+ *
+ * The difference to direct IO writes is that we deadlock when trying
+ * to lock the extent range in the inode's tree during he page reads
+ * triggered by the fault in (while for writes it is due to waiting for
+ * our own ordered extent). This is because for direct IO reads,
+ * btrfs_dio_iomap_begin() returns with the extent range locked, which
+ * is only unlocked in the endio callback (end_bio_extent_readpage()).
+ */
+ pagefault_disable();
+ to->nofault = true;
ret = iomap_dio_rw(iocb, to, &btrfs_dio_iomap_ops, &btrfs_dio_ops,
- 0, 0);
+ IOMAP_DIO_PARTIAL, read);
+ to->nofault = false;
+ pagefault_enable();
+
+ /* No increment (+=) because iomap returns a cumulative value. */
+ if (ret > 0)
+ read = ret;
+
+ if (iov_iter_count(to) > 0 && (ret == -EFAULT || ret > 0)) {
+ const size_t left = iov_iter_count(to);
+
+ if (left == prev_left) {
+ /*
+ * We didn't make any progress since the last attempt,
+ * fallback to a buffered read for the remainder of the
+ * range. This is just to avoid any possibility of looping
+ * for too long.
+ */
+ ret = read;
+ } else {
+ /*
+ * We made some progress since the last retry or this is
+ * the first time we are retrying. Fault in as many pages
+ * as possible and retry.
+ */
+ fault_in_iov_iter_writeable(to, left);
+ prev_left = left;
+ goto again;
+ }
+ }
btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED);
- return ret;
+ return ret < 0 ? ret : read;
}
static ssize_t btrfs_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
bool need_unlock; /* for mut. excl. ops lock */
int ret;
+ if (!arg)
+ btrfs_warn(fs_info,
+ "IOC_BALANCE ioctl (v1) is deprecated and will be removed in kernel 5.18");
+
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
static int copy_compressed_data_to_page(char *compressed_data,
size_t compressed_size,
struct page **out_pages,
+ unsigned long max_nr_page,
u32 *cur_out,
const u32 sectorsize)
{
struct page *cur_page;
char *kaddr;
+ if ((*cur_out / PAGE_SIZE) >= max_nr_page)
+ return -E2BIG;
+
/*
* We never allow a segment header crossing sector boundary, previous
* run should ensure we have enough space left inside the sector.
orig_out + compressed_size - *cur_out);
kunmap(cur_page);
+
+ if ((*cur_out / PAGE_SIZE) >= max_nr_page)
+ return -E2BIG;
+
cur_page = out_pages[*cur_out / PAGE_SIZE];
/* Allocate a new page */
if (!cur_page) {
const u32 sectorsize = btrfs_sb(mapping->host->i_sb)->sectorsize;
struct page *page_in = NULL;
char *sizes_ptr;
+ const unsigned long max_nr_page = *out_pages;
int ret = 0;
/* Points to the file offset of input data */
u64 cur_in = start;
u32 cur_out = 0;
u32 len = *total_out;
+ ASSERT(max_nr_page > 0);
*out_pages = 0;
*total_out = 0;
*total_in = 0;
}
ret = copy_compressed_data_to_page(workspace->cbuf, out_len,
- pages, &cur_out, sectorsize);
+ pages, max_nr_page,
+ &cur_out, sectorsize);
if (ret < 0)
goto out;
u64 physical_for_dev_replace;
atomic_t refs;
u8 mirror_num;
- int have_csum:1;
- int io_error:1;
+ unsigned int have_csum:1;
+ unsigned int io_error:1;
u8 csum[BTRFS_CSUM_SIZE];
struct scrub_recover *recover;
fs_info->fs_devices->total_rw_bytes = 0;
/*
+ * Lockdep complains about possible circular locking dependency between
+ * a disk's open_mutex (struct gendisk.open_mutex), the rw semaphores
+ * used for freeze procection of a fs (struct super_block.s_writers),
+ * which we take when starting a transaction, and extent buffers of the
+ * chunk tree if we call read_one_dev() while holding a lock on an
+ * extent buffer of the chunk tree. Since we are mounting the filesystem
+ * and at this point there can't be any concurrent task modifying the
+ * chunk tree, to keep it simple, just skip locking on the chunk tree.
+ */
+ ASSERT(!test_bit(BTRFS_FS_OPEN, &fs_info->flags));
+ path->skip_locking = 1;
+
+ /*
* Read all device items, and then all the chunk items. All
* device items are found before any chunk item (their object id
* is smaller than the lowest possible object id for a chunk
goto error;
break;
}
- /*
- * The nodes on level 1 are not locked but we don't need to do
- * that during mount time as nothing else can access the tree
- */
node = path->nodes[1];
if (node) {
if (last_ra_node != node->start) {
* requirement for chunk allocation, see the comment on
* top of btrfs_chunk_alloc() for details.
*/
- ASSERT(!test_bit(BTRFS_FS_OPEN, &fs_info->flags));
chunk = btrfs_item_ptr(leaf, slot, struct btrfs_chunk);
ret = read_one_chunk(&found_key, leaf, chunk);
if (ret)
/* 307s to avoid pathologically clashing with transaction commit */
#define ZSTD_BTRFS_RECLAIM_JIFFIES (307 * HZ)
-static ZSTD_parameters zstd_get_btrfs_parameters(unsigned int level,
+static zstd_parameters zstd_get_btrfs_parameters(unsigned int level,
size_t src_len)
{
- ZSTD_parameters params = ZSTD_getParams(level, src_len, 0);
+ zstd_parameters params = zstd_get_params(level, src_len);
if (params.cParams.windowLog > ZSTD_BTRFS_MAX_WINDOWLOG)
params.cParams.windowLog = ZSTD_BTRFS_MAX_WINDOWLOG;
unsigned long last_used; /* jiffies */
struct list_head list;
struct list_head lru_list;
- ZSTD_inBuffer in_buf;
- ZSTD_outBuffer out_buf;
+ zstd_in_buffer in_buf;
+ zstd_out_buffer out_buf;
};
/*
unsigned int level;
for (level = 1; level <= ZSTD_BTRFS_MAX_LEVEL; level++) {
- ZSTD_parameters params =
+ zstd_parameters params =
zstd_get_btrfs_parameters(level, ZSTD_BTRFS_MAX_INPUT);
size_t level_size =
max_t(size_t,
- ZSTD_CStreamWorkspaceBound(params.cParams),
- ZSTD_DStreamWorkspaceBound(ZSTD_BTRFS_MAX_INPUT));
+ zstd_cstream_workspace_bound(¶ms.cParams),
+ zstd_dstream_workspace_bound(ZSTD_BTRFS_MAX_INPUT));
max_size = max_t(size_t, max_size, level_size);
zstd_ws_mem_sizes[level - 1] = max_size;
unsigned long *total_in, unsigned long *total_out)
{
struct workspace *workspace = list_entry(ws, struct workspace, list);
- ZSTD_CStream *stream;
+ zstd_cstream *stream;
int ret = 0;
int nr_pages = 0;
struct page *in_page = NULL; /* The current page to read */
unsigned long len = *total_out;
const unsigned long nr_dest_pages = *out_pages;
unsigned long max_out = nr_dest_pages * PAGE_SIZE;
- ZSTD_parameters params = zstd_get_btrfs_parameters(workspace->req_level,
+ zstd_parameters params = zstd_get_btrfs_parameters(workspace->req_level,
len);
*out_pages = 0;
*total_in = 0;
/* Initialize the stream */
- stream = ZSTD_initCStream(params, len, workspace->mem,
+ stream = zstd_init_cstream(¶ms, len, workspace->mem,
workspace->size);
if (!stream) {
- pr_warn("BTRFS: ZSTD_initCStream failed\n");
+ pr_warn("BTRFS: zstd_init_cstream failed\n");
ret = -EIO;
goto out;
}
while (1) {
size_t ret2;
- ret2 = ZSTD_compressStream(stream, &workspace->out_buf,
+ ret2 = zstd_compress_stream(stream, &workspace->out_buf,
&workspace->in_buf);
- if (ZSTD_isError(ret2)) {
- pr_debug("BTRFS: ZSTD_compressStream returned %d\n",
- ZSTD_getErrorCode(ret2));
+ if (zstd_is_error(ret2)) {
+ pr_debug("BTRFS: zstd_compress_stream returned %d\n",
+ zstd_get_error_code(ret2));
ret = -EIO;
goto out;
}
while (1) {
size_t ret2;
- ret2 = ZSTD_endStream(stream, &workspace->out_buf);
- if (ZSTD_isError(ret2)) {
- pr_debug("BTRFS: ZSTD_endStream returned %d\n",
- ZSTD_getErrorCode(ret2));
+ ret2 = zstd_end_stream(stream, &workspace->out_buf);
+ if (zstd_is_error(ret2)) {
+ pr_debug("BTRFS: zstd_end_stream returned %d\n",
+ zstd_get_error_code(ret2));
ret = -EIO;
goto out;
}
struct workspace *workspace = list_entry(ws, struct workspace, list);
struct page **pages_in = cb->compressed_pages;
size_t srclen = cb->compressed_len;
- ZSTD_DStream *stream;
+ zstd_dstream *stream;
int ret = 0;
unsigned long page_in_index = 0;
unsigned long total_pages_in = DIV_ROUND_UP(srclen, PAGE_SIZE);
unsigned long buf_start;
unsigned long total_out = 0;
- stream = ZSTD_initDStream(
+ stream = zstd_init_dstream(
ZSTD_BTRFS_MAX_INPUT, workspace->mem, workspace->size);
if (!stream) {
- pr_debug("BTRFS: ZSTD_initDStream failed\n");
+ pr_debug("BTRFS: zstd_init_dstream failed\n");
ret = -EIO;
goto done;
}
while (1) {
size_t ret2;
- ret2 = ZSTD_decompressStream(stream, &workspace->out_buf,
+ ret2 = zstd_decompress_stream(stream, &workspace->out_buf,
&workspace->in_buf);
- if (ZSTD_isError(ret2)) {
- pr_debug("BTRFS: ZSTD_decompressStream returned %d\n",
- ZSTD_getErrorCode(ret2));
+ if (zstd_is_error(ret2)) {
+ pr_debug("BTRFS: zstd_decompress_stream returned %d\n",
+ zstd_get_error_code(ret2));
ret = -EIO;
goto done;
}
size_t destlen)
{
struct workspace *workspace = list_entry(ws, struct workspace, list);
- ZSTD_DStream *stream;
+ zstd_dstream *stream;
int ret = 0;
size_t ret2;
unsigned long total_out = 0;
unsigned long pg_offset = 0;
- stream = ZSTD_initDStream(
+ stream = zstd_init_dstream(
ZSTD_BTRFS_MAX_INPUT, workspace->mem, workspace->size);
if (!stream) {
- pr_warn("BTRFS: ZSTD_initDStream failed\n");
+ pr_warn("BTRFS: zstd_init_dstream failed\n");
ret = -EIO;
goto finish;
}
/* Check if the frame is over and we still need more input */
if (ret2 == 0) {
- pr_debug("BTRFS: ZSTD_decompressStream ended early\n");
+ pr_debug("BTRFS: zstd_decompress_stream ended early\n");
ret = -EIO;
goto finish;
}
- ret2 = ZSTD_decompressStream(stream, &workspace->out_buf,
+ ret2 = zstd_decompress_stream(stream, &workspace->out_buf,
&workspace->in_buf);
- if (ZSTD_isError(ret2)) {
- pr_debug("BTRFS: ZSTD_decompressStream returned %d\n",
- ZSTD_getErrorCode(ret2));
+ if (zstd_is_error(ret2)) {
+ pr_debug("BTRFS: zstd_decompress_stream returned %d\n",
+ zstd_get_error_code(ret2));
ret = -EIO;
goto finish;
}
(CONGESTION_ON_THRESH(congestion_kb) >> 2))
static int ceph_netfs_check_write_begin(struct file *file, loff_t pos, unsigned int len,
- struct page *page, void **_fsdata);
+ struct folio *folio, void **_fsdata);
static inline struct ceph_snap_context *page_snap_context(struct page *page)
{
};
/* read a single page, without unlocking it. */
-static int ceph_readpage(struct file *file, struct page *page)
+static int ceph_readpage(struct file *file, struct page *subpage)
{
+ struct folio *folio = page_folio(subpage);
struct inode *inode = file_inode(file);
struct ceph_inode_info *ci = ceph_inode(inode);
struct ceph_vino vino = ceph_vino(inode);
- u64 off = page_offset(page);
- u64 len = thp_size(page);
+ size_t len = folio_size(folio);
+ u64 off = folio_file_pos(folio);
if (ci->i_inline_version != CEPH_INLINE_NONE) {
/*
* into page cache while getting Fcr caps.
*/
if (off == 0) {
- unlock_page(page);
+ folio_unlock(folio);
return -EINVAL;
}
- zero_user_segment(page, 0, thp_size(page));
- SetPageUptodate(page);
- unlock_page(page);
+ zero_user_segment(&folio->page, 0, folio_size(folio));
+ folio_mark_uptodate(folio);
+ folio_unlock(folio);
return 0;
}
- dout("readpage ino %llx.%llx file %p off %llu len %llu page %p index %lu\n",
- vino.ino, vino.snap, file, off, len, page, page->index);
+ dout("readpage ino %llx.%llx file %p off %llu len %zu folio %p index %lu\n",
+ vino.ino, vino.snap, file, off, len, folio, folio_index(folio));
- return netfs_readpage(file, page, &ceph_netfs_read_ops, NULL);
+ return netfs_readpage(file, folio, &ceph_netfs_read_ops, NULL);
}
static void ceph_readahead(struct readahead_control *ractl)
wbc->sync_mode == WB_SYNC_NONE ? "NONE" :
(wbc->sync_mode == WB_SYNC_ALL ? "ALL" : "HOLD"));
- if (READ_ONCE(fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN) {
+ if (ceph_inode_is_shutdown(inode)) {
if (ci->i_wrbuffer_ref > 0) {
pr_warn_ratelimited(
"writepage_start %p %lld forced umount\n",
ceph_find_incompatible(struct page *page)
{
struct inode *inode = page->mapping->host;
- struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
struct ceph_inode_info *ci = ceph_inode(inode);
- if (READ_ONCE(fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN) {
- dout(" page %p forced umount\n", page);
- return ERR_PTR(-EIO);
+ if (ceph_inode_is_shutdown(inode)) {
+ dout(" page %p %llx:%llx is shutdown\n", page,
+ ceph_vinop(inode));
+ return ERR_PTR(-ESTALE);
}
for (;;) {
}
static int ceph_netfs_check_write_begin(struct file *file, loff_t pos, unsigned int len,
- struct page *page, void **_fsdata)
+ struct folio *folio, void **_fsdata)
{
struct inode *inode = file_inode(file);
struct ceph_inode_info *ci = ceph_inode(inode);
struct ceph_snap_context *snapc;
- snapc = ceph_find_incompatible(page);
+ snapc = ceph_find_incompatible(folio_page(folio, 0));
if (snapc) {
int r;
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
if (IS_ERR(snapc))
return PTR_ERR(snapc);
* clean, or already dirty within the same snap context.
*/
static int ceph_write_begin(struct file *file, struct address_space *mapping,
- loff_t pos, unsigned len, unsigned flags,
+ loff_t pos, unsigned len, unsigned aop_flags,
struct page **pagep, void **fsdata)
{
struct inode *inode = file_inode(file);
struct ceph_inode_info *ci = ceph_inode(inode);
- struct page *page = NULL;
+ struct folio *folio = NULL;
pgoff_t index = pos >> PAGE_SHIFT;
int r;
* for inline_version sent to the MDS.
*/
if (ci->i_inline_version != CEPH_INLINE_NONE) {
- page = grab_cache_page_write_begin(mapping, index, flags);
- if (!page)
+ unsigned int fgp_flags = FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE;
+ if (aop_flags & AOP_FLAG_NOFS)
+ fgp_flags |= FGP_NOFS;
+ folio = __filemap_get_folio(mapping, index, fgp_flags,
+ mapping_gfp_mask(mapping));
+ if (!folio)
return -ENOMEM;
/*
* The inline_version on a new inode is set to 1. If that's the
- * case, then the page is brand new and isn't yet Uptodate.
+ * case, then the folio is brand new and isn't yet Uptodate.
*/
r = 0;
if (index == 0 && ci->i_inline_version != 1) {
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
WARN_ONCE(1, "ceph: write_begin called on still-inlined inode (inline_version %llu)!\n",
ci->i_inline_version);
r = -EINVAL;
}
goto out;
}
- zero_user_segment(page, 0, thp_size(page));
- SetPageUptodate(page);
+ zero_user_segment(&folio->page, 0, folio_size(folio));
+ folio_mark_uptodate(folio);
goto out;
}
- r = netfs_write_begin(file, inode->i_mapping, pos, len, 0, &page, NULL,
+ r = netfs_write_begin(file, inode->i_mapping, pos, len, 0, &folio, NULL,
&ceph_netfs_read_ops, NULL);
out:
if (r == 0)
- wait_on_page_fscache(page);
+ folio_wait_fscache(folio);
if (r < 0) {
- if (page)
- put_page(page);
+ if (folio)
+ folio_put(folio);
} else {
- WARN_ON_ONCE(!PageLocked(page));
- *pagep = page;
+ WARN_ON_ONCE(!folio_test_locked(folio));
+ *pagep = &folio->page;
}
return r;
}
*/
static int ceph_write_end(struct file *file, struct address_space *mapping,
loff_t pos, unsigned len, unsigned copied,
- struct page *page, void *fsdata)
+ struct page *subpage, void *fsdata)
{
+ struct folio *folio = page_folio(subpage);
struct inode *inode = file_inode(file);
bool check_cap = false;
- dout("write_end file %p inode %p page %p %d~%d (%d)\n", file,
- inode, page, (int)pos, (int)copied, (int)len);
+ dout("write_end file %p inode %p folio %p %d~%d (%d)\n", file,
+ inode, folio, (int)pos, (int)copied, (int)len);
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
/* just return that nothing was copied on a short copy */
if (copied < len) {
copied = 0;
goto out;
}
- SetPageUptodate(page);
+ folio_mark_uptodate(folio);
}
/* did file size increase? */
if (pos+copied > i_size_read(inode))
check_cap = ceph_inode_set_size(inode, pos+copied);
- set_page_dirty(page);
+ folio_mark_dirty(folio);
out:
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
if (check_cap)
ceph_check_caps(ceph_inode(inode), CHECK_CAPS_AUTHONLY, NULL);
return copied;
}
-/*
- * we set .direct_IO to indicate direct io is supported, but since we
- * intercept O_DIRECT reads and writes early, this function should
- * never get called.
- */
-static ssize_t ceph_direct_io(struct kiocb *iocb, struct iov_iter *iter)
-{
- WARN_ON(1);
- return -EINVAL;
-}
-
const struct address_space_operations ceph_aops = {
.readpage = ceph_readpage,
.readahead = ceph_readahead,
.set_page_dirty = ceph_set_page_dirty,
.invalidatepage = ceph_invalidatepage,
.releasepage = ceph_releasepage,
- .direct_IO = ceph_direct_io,
+ .direct_IO = noop_direct_IO,
};
static void ceph_block_sigs(sigset_t *oldset)
sigset_t oldset;
vm_fault_t ret = VM_FAULT_SIGBUS;
+ if (ceph_inode_is_shutdown(inode))
+ return ret;
+
ceph_block_sigs(&oldset);
dout("filemap_fault %p %llx.%llx %llu trying to get caps\n",
sigset_t oldset;
vm_fault_t ret = VM_FAULT_SIGBUS;
+ if (ceph_inode_is_shutdown(inode))
+ return ret;
+
prealloc_cf = ceph_alloc_cap_flush();
if (!prealloc_cf)
return VM_FAULT_OOM;
#include "super.h"
#include "cache.h"
-struct ceph_aux_inode {
- u64 version;
- u64 mtime_sec;
- u64 mtime_nsec;
-};
-
struct fscache_netfs ceph_cache_netfs = {
.name = "ceph",
.version = 0,
void *cookie_netfs_data, const void *data, uint16_t dlen,
loff_t object_size)
{
- struct ceph_aux_inode aux;
struct ceph_inode_info* ci = cookie_netfs_data;
struct inode* inode = &ci->vfs_inode;
- if (dlen != sizeof(aux) ||
+ if (dlen != sizeof(ci->i_version) ||
i_size_read(inode) != object_size)
return FSCACHE_CHECKAUX_OBSOLETE;
- memset(&aux, 0, sizeof(aux));
- aux.version = ci->i_version;
- aux.mtime_sec = inode->i_mtime.tv_sec;
- aux.mtime_nsec = inode->i_mtime.tv_nsec;
-
- if (memcmp(data, &aux, sizeof(aux)) != 0)
+ if (*(u64 *)data != ci->i_version)
return FSCACHE_CHECKAUX_OBSOLETE;
dout("ceph inode 0x%p cached okay\n", ci);
{
struct ceph_inode_info *ci = ceph_inode(inode);
struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
- struct ceph_aux_inode aux;
/* No caching for filesystem */
if (!fsc->fscache)
inode_lock_nested(inode, I_MUTEX_CHILD);
if (!ci->fscache) {
- memset(&aux, 0, sizeof(aux));
- aux.version = ci->i_version;
- aux.mtime_sec = inode->i_mtime.tv_sec;
- aux.mtime_nsec = inode->i_mtime.tv_nsec;
ci->fscache = fscache_acquire_cookie(fsc->fscache,
&ceph_fscache_inode_object_def,
&ci->i_vino, sizeof(ci->i_vino),
- &aux, sizeof(aux),
+ &ci->i_version, sizeof(ci->i_version),
ci, i_size_read(inode), false);
}
inode_unlock(inode);
lockdep_assert_held(&ci->i_ceph_lock);
- fsc = ceph_sb_to_client(ci->vfs_inode.i_sb);
+ fsc = ceph_inode_to_client(&ci->vfs_inode);
WARN_ON_ONCE(ci->i_auth_cap == cap &&
!list_empty(&ci->i_dirty_item) &&
!fsc->blocklisted &&
- READ_ONCE(fsc->mount_state) != CEPH_MOUNT_SHUTDOWN);
+ !ceph_inode_is_shutdown(&ci->vfs_inode));
__ceph_remove_cap(cap, queue_release);
}
}
}
- dout("check_caps %p file_want %s used %s dirty %s flushing %s"
- " issued %s revoking %s retain %s %s%s\n", inode,
+ dout("check_caps %llx.%llx file_want %s used %s dirty %s flushing %s"
+ " issued %s revoking %s retain %s %s%s\n", ceph_vinop(inode),
ceph_cap_string(file_wanted),
ceph_cap_string(used), ceph_cap_string(ci->i_dirty_caps),
ceph_cap_string(ci->i_flushing_caps),
(revoking & (CEPH_CAP_FILE_CACHE|
CEPH_CAP_FILE_LAZYIO)) && /* or revoking cache */
!tried_invalidate) {
- dout("check_caps trying to invalidate on %p\n", inode);
+ dout("check_caps trying to invalidate on %llx.%llx\n",
+ ceph_vinop(inode));
if (try_nonblocking_invalidate(inode) < 0) {
dout("check_caps queuing invalidate\n");
queue_invalidate = true;
*
* Returns 0 if caps were not able to be acquired (yet), 1 if succeed,
* or a negative error code. There are 3 speical error codes:
- * -EAGAIN: need to sleep but non-blocking is specified
- * -EFBIG: ask caller to call check_max_size() and try again.
- * -ESTALE: ask caller to call ceph_renew_caps() and try again.
+ * -EAGAIN: need to sleep but non-blocking is specified
+ * -EFBIG: ask caller to call check_max_size() and try again.
+ * -EUCLEAN: ask caller to call ceph_renew_caps() and try again.
*/
enum {
/* first 8 bits are reserved for CEPH_FILE_MODE_FOO */
dout("get_cap_refs %p endoff %llu > maxsize %llu\n",
inode, endoff, ci->i_max_size);
if (endoff > ci->i_requested_max_size)
- ret = ci->i_auth_cap ? -EFBIG : -ESTALE;
+ ret = ci->i_auth_cap ? -EFBIG : -EUCLEAN;
goto out_unlock;
}
/*
goto out_unlock;
}
- if (READ_ONCE(mdsc->fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN) {
- dout("get_cap_refs %p forced umount\n", inode);
- ret = -EIO;
+ if (ceph_inode_is_shutdown(inode)) {
+ dout("get_cap_refs %p inode is shutdown\n", inode);
+ ret = -ESTALE;
goto out_unlock;
}
mds_wanted = __ceph_caps_mds_wanted(ci, false);
dout("get_cap_refs %p need %s > mds_wanted %s\n",
inode, ceph_cap_string(need),
ceph_cap_string(mds_wanted));
- ret = -ESTALE;
+ ret = -EUCLEAN;
goto out_unlock;
}
ret = try_get_cap_refs(inode, need, want, 0, flags, got);
/* three special error codes */
- if (ret == -EAGAIN || ret == -EFBIG || ret == -ESTALE)
+ if (ret == -EAGAIN || ret == -EFBIG || ret == -EUCLEAN)
ret = 0;
return ret;
}
}
if (ret < 0) {
- if (ret == -EFBIG || ret == -ESTALE) {
+ if (ret == -EFBIG || ret == -EUCLEAN) {
int ret2 = ceph_wait_on_async_create(inode);
if (ret2 < 0)
return ret2;
check_max_size(inode, endoff);
continue;
}
- if (ret == -ESTALE) {
+ if (ret == -EUCLEAN) {
/* session was killed, try renew caps */
ret = ceph_renew_caps(inode, flags);
if (ret == 0)
i_dirty_item);
inode = &ci->vfs_inode;
ihold(inode);
- dout("flush_dirty_caps %p\n", inode);
+ dout("flush_dirty_caps %llx.%llx\n", ceph_vinop(inode));
spin_unlock(&mdsc->cap_dirty_lock);
ceph_check_caps(ci, CHECK_CAPS_FLUSH, NULL);
iput(inode);
spin_unlock(&dentry->d_lock);
return ret;
}
+
+static int remove_capsnaps(struct ceph_mds_client *mdsc, struct inode *inode)
+{
+ struct ceph_inode_info *ci = ceph_inode(inode);
+ struct ceph_cap_snap *capsnap;
+ int capsnap_release = 0;
+
+ lockdep_assert_held(&ci->i_ceph_lock);
+
+ dout("removing capsnaps, ci is %p, inode is %p\n", ci, inode);
+
+ while (!list_empty(&ci->i_cap_snaps)) {
+ capsnap = list_first_entry(&ci->i_cap_snaps,
+ struct ceph_cap_snap, ci_item);
+ __ceph_remove_capsnap(inode, capsnap, NULL, NULL);
+ ceph_put_snap_context(capsnap->context);
+ ceph_put_cap_snap(capsnap);
+ capsnap_release++;
+ }
+ wake_up_all(&ci->i_cap_wq);
+ wake_up_all(&mdsc->cap_flushing_wq);
+ return capsnap_release;
+}
+
+int ceph_purge_inode_cap(struct inode *inode, struct ceph_cap *cap, bool *invalidate)
+{
+ struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
+ struct ceph_mds_client *mdsc = fsc->mdsc;
+ struct ceph_inode_info *ci = ceph_inode(inode);
+ bool is_auth;
+ bool dirty_dropped = false;
+ int iputs = 0;
+
+ lockdep_assert_held(&ci->i_ceph_lock);
+
+ dout("removing cap %p, ci is %p, inode is %p\n",
+ cap, ci, &ci->vfs_inode);
+
+ is_auth = (cap == ci->i_auth_cap);
+ __ceph_remove_cap(cap, false);
+ if (is_auth) {
+ struct ceph_cap_flush *cf;
+
+ if (ceph_inode_is_shutdown(inode)) {
+ if (inode->i_data.nrpages > 0)
+ *invalidate = true;
+ if (ci->i_wrbuffer_ref > 0)
+ mapping_set_error(&inode->i_data, -EIO);
+ }
+
+ spin_lock(&mdsc->cap_dirty_lock);
+
+ /* trash all of the cap flushes for this inode */
+ while (!list_empty(&ci->i_cap_flush_list)) {
+ cf = list_first_entry(&ci->i_cap_flush_list,
+ struct ceph_cap_flush, i_list);
+ list_del_init(&cf->g_list);
+ list_del_init(&cf->i_list);
+ if (!cf->is_capsnap)
+ ceph_free_cap_flush(cf);
+ }
+
+ if (!list_empty(&ci->i_dirty_item)) {
+ pr_warn_ratelimited(
+ " dropping dirty %s state for %p %lld\n",
+ ceph_cap_string(ci->i_dirty_caps),
+ inode, ceph_ino(inode));
+ ci->i_dirty_caps = 0;
+ list_del_init(&ci->i_dirty_item);
+ dirty_dropped = true;
+ }
+ if (!list_empty(&ci->i_flushing_item)) {
+ pr_warn_ratelimited(
+ " dropping dirty+flushing %s state for %p %lld\n",
+ ceph_cap_string(ci->i_flushing_caps),
+ inode, ceph_ino(inode));
+ ci->i_flushing_caps = 0;
+ list_del_init(&ci->i_flushing_item);
+ mdsc->num_cap_flushing--;
+ dirty_dropped = true;
+ }
+ spin_unlock(&mdsc->cap_dirty_lock);
+
+ if (dirty_dropped) {
+ mapping_set_error(inode->i_mapping, -EIO);
+
+ if (ci->i_wrbuffer_ref_head == 0 &&
+ ci->i_wr_ref == 0 &&
+ ci->i_dirty_caps == 0 &&
+ ci->i_flushing_caps == 0) {
+ ceph_put_snap_context(ci->i_head_snapc);
+ ci->i_head_snapc = NULL;
+ }
+ }
+
+ if (atomic_read(&ci->i_filelock_ref) > 0) {
+ /* make further file lock syscall return -EIO */
+ ci->i_ceph_flags |= CEPH_I_ERROR_FILELOCK;
+ pr_warn_ratelimited(" dropping file locks for %p %lld\n",
+ inode, ceph_ino(inode));
+ }
+
+ if (!ci->i_dirty_caps && ci->i_prealloc_cap_flush) {
+ cf = ci->i_prealloc_cap_flush;
+ ci->i_prealloc_cap_flush = NULL;
+ if (!cf->is_capsnap)
+ ceph_free_cap_flush(cf);
+ }
+
+ if (!list_empty(&ci->i_cap_snaps))
+ iputs = remove_capsnaps(mdsc, inode);
+ }
+ if (dirty_dropped)
+ ++iputs;
+ return iputs;
+}
name, total, avg, _min, max, sum); \
}
-static int metric_show(struct seq_file *s, void *p)
+static int metrics_file_show(struct seq_file *s, void *p)
{
struct ceph_fs_client *fsc = s->private;
- struct ceph_mds_client *mdsc = fsc->mdsc;
- struct ceph_client_metric *m = &mdsc->metric;
- int nr_caps = 0;
- s64 total, sum, avg, min, max, sq;
- u64 sum_sz, avg_sz, min_sz, max_sz;
+ struct ceph_client_metric *m = &fsc->mdsc->metric;
- sum = percpu_counter_sum(&m->total_inodes);
seq_printf(s, "item total\n");
seq_printf(s, "------------------------------------------\n");
- seq_printf(s, "%-35s%lld / %lld\n", "opened files / total inodes",
- atomic64_read(&m->opened_files), sum);
- seq_printf(s, "%-35s%lld / %lld\n", "pinned i_caps / total inodes",
- atomic64_read(&m->total_caps), sum);
- seq_printf(s, "%-35s%lld / %lld\n", "opened inodes / total inodes",
- percpu_counter_sum(&m->opened_inodes), sum);
-
- seq_printf(s, "\n");
+ seq_printf(s, "%-35s%lld\n", "total inodes",
+ percpu_counter_sum(&m->total_inodes));
+ seq_printf(s, "%-35s%lld\n", "opened files",
+ atomic64_read(&m->opened_files));
+ seq_printf(s, "%-35s%lld\n", "pinned i_caps",
+ atomic64_read(&m->total_caps));
+ seq_printf(s, "%-35s%lld\n", "opened inodes",
+ percpu_counter_sum(&m->opened_inodes));
+ return 0;
+}
+
+static const char * const metric_str[] = {
+ "read",
+ "write",
+ "metadata",
+ "copyfrom"
+};
+static int metrics_latency_show(struct seq_file *s, void *p)
+{
+ struct ceph_fs_client *fsc = s->private;
+ struct ceph_client_metric *cm = &fsc->mdsc->metric;
+ struct ceph_metric *m;
+ s64 total, sum, avg, min, max, sq;
+ int i;
+
seq_printf(s, "item total avg_lat(us) min_lat(us) max_lat(us) stdev(us)\n");
seq_printf(s, "-----------------------------------------------------------------------------------\n");
- spin_lock(&m->read_metric_lock);
- total = m->total_reads;
- sum = m->read_latency_sum;
- avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
- min = m->read_latency_min;
- max = m->read_latency_max;
- sq = m->read_latency_sq_sum;
- spin_unlock(&m->read_metric_lock);
- CEPH_LAT_METRIC_SHOW("read", total, avg, min, max, sq);
-
- spin_lock(&m->write_metric_lock);
- total = m->total_writes;
- sum = m->write_latency_sum;
- avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
- min = m->write_latency_min;
- max = m->write_latency_max;
- sq = m->write_latency_sq_sum;
- spin_unlock(&m->write_metric_lock);
- CEPH_LAT_METRIC_SHOW("write", total, avg, min, max, sq);
-
- spin_lock(&m->metadata_metric_lock);
- total = m->total_metadatas;
- sum = m->metadata_latency_sum;
- avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
- min = m->metadata_latency_min;
- max = m->metadata_latency_max;
- sq = m->metadata_latency_sq_sum;
- spin_unlock(&m->metadata_metric_lock);
- CEPH_LAT_METRIC_SHOW("metadata", total, avg, min, max, sq);
-
- seq_printf(s, "\n");
+ for (i = 0; i < METRIC_MAX; i++) {
+ m = &cm->metric[i];
+ spin_lock(&m->lock);
+ total = m->total;
+ sum = m->latency_sum;
+ avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
+ min = m->latency_min;
+ max = m->latency_max;
+ sq = m->latency_sq_sum;
+ spin_unlock(&m->lock);
+ CEPH_LAT_METRIC_SHOW(metric_str[i], total, avg, min, max, sq);
+ }
+
+ return 0;
+}
+
+static int metrics_size_show(struct seq_file *s, void *p)
+{
+ struct ceph_fs_client *fsc = s->private;
+ struct ceph_client_metric *cm = &fsc->mdsc->metric;
+ struct ceph_metric *m;
+ s64 total;
+ u64 sum, avg, min, max;
+ int i;
+
seq_printf(s, "item total avg_sz(bytes) min_sz(bytes) max_sz(bytes) total_sz(bytes)\n");
seq_printf(s, "----------------------------------------------------------------------------------------\n");
- spin_lock(&m->read_metric_lock);
- total = m->total_reads;
- sum_sz = m->read_size_sum;
- avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
- min_sz = m->read_size_min;
- max_sz = m->read_size_max;
- spin_unlock(&m->read_metric_lock);
- CEPH_SZ_METRIC_SHOW("read", total, avg_sz, min_sz, max_sz, sum_sz);
-
- spin_lock(&m->write_metric_lock);
- total = m->total_writes;
- sum_sz = m->write_size_sum;
- avg_sz = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum_sz, total) : 0;
- min_sz = m->write_size_min;
- max_sz = m->write_size_max;
- spin_unlock(&m->write_metric_lock);
- CEPH_SZ_METRIC_SHOW("write", total, avg_sz, min_sz, max_sz, sum_sz);
-
- seq_printf(s, "\n");
+ for (i = 0; i < METRIC_MAX; i++) {
+ /* skip 'metadata' as it doesn't use the size metric */
+ if (i == METRIC_METADATA)
+ continue;
+ m = &cm->metric[i];
+ spin_lock(&m->lock);
+ total = m->total;
+ sum = m->size_sum;
+ avg = total > 0 ? DIV64_U64_ROUND_CLOSEST(sum, total) : 0;
+ min = m->size_min;
+ max = m->size_max;
+ spin_unlock(&m->lock);
+ CEPH_SZ_METRIC_SHOW(metric_str[i], total, avg, min, max, sum);
+ }
+
+ return 0;
+}
+
+static int metrics_caps_show(struct seq_file *s, void *p)
+{
+ struct ceph_fs_client *fsc = s->private;
+ struct ceph_client_metric *m = &fsc->mdsc->metric;
+ int nr_caps = 0;
+
seq_printf(s, "item total miss hit\n");
seq_printf(s, "-------------------------------------------------\n");
DEFINE_SHOW_ATTRIBUTE(mdsc);
DEFINE_SHOW_ATTRIBUTE(caps);
DEFINE_SHOW_ATTRIBUTE(mds_sessions);
-DEFINE_SHOW_ATTRIBUTE(metric);
DEFINE_SHOW_ATTRIBUTE(status);
+DEFINE_SHOW_ATTRIBUTE(metrics_file);
+DEFINE_SHOW_ATTRIBUTE(metrics_latency);
+DEFINE_SHOW_ATTRIBUTE(metrics_size);
+DEFINE_SHOW_ATTRIBUTE(metrics_caps);
/*
debugfs_remove(fsc->debugfs_mdsmap);
debugfs_remove(fsc->debugfs_mds_sessions);
debugfs_remove(fsc->debugfs_caps);
- debugfs_remove(fsc->debugfs_metric);
+ debugfs_remove(fsc->debugfs_status);
debugfs_remove(fsc->debugfs_mdsc);
+ debugfs_remove_recursive(fsc->debugfs_metrics_dir);
}
void ceph_fs_debugfs_init(struct ceph_fs_client *fsc)
fsc,
&mdsc_fops);
- fsc->debugfs_metric = debugfs_create_file("metrics",
- 0400,
- fsc->client->debugfs_dir,
- fsc,
- &metric_fops);
-
fsc->debugfs_caps = debugfs_create_file("caps",
0400,
fsc->client->debugfs_dir,
fsc->client->debugfs_dir,
fsc,
&status_fops);
+
+ fsc->debugfs_metrics_dir = debugfs_create_dir("metrics",
+ fsc->client->debugfs_dir);
+
+ debugfs_create_file("file", 0400, fsc->debugfs_metrics_dir, fsc,
+ &metrics_file_fops);
+ debugfs_create_file("latency", 0400, fsc->debugfs_metrics_dir, fsc,
+ &metrics_latency_fops);
+ debugfs_create_file("size", 0400, fsc->debugfs_metrics_dir, fsc,
+ &metrics_size_fops);
+ debugfs_create_file("caps", 0400, fsc->debugfs_metrics_dir, fsc,
+ &metrics_caps_fops);
}
ceph_mdsc_put_request(req);
if (!inode)
return err < 0 ? ERR_PTR(err) : ERR_PTR(-ESTALE);
+ } else {
+ if (ceph_inode_is_shutdown(inode)) {
+ iput(inode);
+ return ERR_PTR(-ESTALE);
+ }
}
return inode;
}
return ERR_PTR(-ESTALE);
inode = ceph_find_inode(sb, vino);
- if (inode)
+ if (inode) {
+ if (ceph_inode_is_shutdown(inode)) {
+ iput(inode);
+ return ERR_PTR(-ESTALE);
+ }
return d_obtain_alias(inode);
+ }
req = ceph_mdsc_create_request(mdsc, CEPH_MDS_OP_LOOKUPINO,
USE_ANY_MDS);
if (result) {
struct dentry *dentry = req->r_dentry;
+ struct inode *inode = d_inode(dentry);
int pathlen = 0;
u64 base = 0;
char *path = ceph_mdsc_build_path(req->r_dentry, &pathlen,
if (!d_unhashed(dentry))
d_drop(dentry);
- /* FIXME: start returning I/O errors on all accesses? */
+ ceph_inode_shutdown(inode);
+
pr_warn("ceph: async create failure path=(%llx)%s result=%d!\n",
base, IS_ERR(path) ? "<<bad>>" : path, result);
ceph_mdsc_free_path(path, pathlen);
}
ceph_kick_flushing_inode_caps(req->r_session, ci);
spin_unlock(&ci->i_ceph_lock);
- } else {
+ } else if (!result) {
pr_warn("%s: no req->r_target_inode for 0x%llx\n", __func__,
req->r_deleg_ino);
}
ssize_t ret;
u64 off = iocb->ki_pos;
u64 len = iov_iter_count(to);
+ u64 i_size;
dout("sync_read on file %p %llu~%u %s\n", file, off, (unsigned)len,
(file->f_flags & O_DIRECT) ? "O_DIRECT" : "");
struct page **pages;
int num_pages;
size_t page_off;
- u64 i_size;
bool more;
int idx;
size_t left;
}
if (off > iocb->ki_pos) {
- if (ret >= 0 &&
- iov_iter_count(to) > 0 && off >= i_size_read(inode))
+ if (off >= i_size) {
*retry_op = CHECK_EOF;
- ret = off - iocb->ki_pos;
- iocb->ki_pos = off;
+ ret = i_size - iocb->ki_pos;
+ iocb->ki_pos = i_size;
+ } else {
+ ret = off - iocb->ki_pos;
+ iocb->ki_pos = off;
+ }
}
dout("sync_read result %zd retry_op %d\n", ret, *retry_op);
dout("aio_read %p %llx.%llx %llu~%u trying to get caps on %p\n",
inode, ceph_vinop(inode), iocb->ki_pos, (unsigned)len, inode);
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
if (direct_lock)
ceph_start_io_direct(inode);
else
loff_t pos;
loff_t limit = max(i_size_read(inode), fsc->max_file_size);
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
if (ceph_snap(inode) != CEPH_NOSNAP)
return -EROFS;
return 0;
}
+static struct ceph_osd_request *
+ceph_alloc_copyfrom_request(struct ceph_osd_client *osdc,
+ u64 src_snapid,
+ struct ceph_object_id *src_oid,
+ struct ceph_object_locator *src_oloc,
+ struct ceph_object_id *dst_oid,
+ struct ceph_object_locator *dst_oloc,
+ u32 truncate_seq, u64 truncate_size)
+{
+ struct ceph_osd_request *req;
+ int ret;
+ u32 src_fadvise_flags =
+ CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
+ CEPH_OSD_OP_FLAG_FADVISE_NOCACHE;
+ u32 dst_fadvise_flags =
+ CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
+ CEPH_OSD_OP_FLAG_FADVISE_DONTNEED;
+
+ req = ceph_osdc_alloc_request(osdc, NULL, 1, false, GFP_KERNEL);
+ if (!req)
+ return ERR_PTR(-ENOMEM);
+
+ req->r_flags = CEPH_OSD_FLAG_WRITE;
+
+ ceph_oloc_copy(&req->r_t.base_oloc, dst_oloc);
+ ceph_oid_copy(&req->r_t.base_oid, dst_oid);
+
+ ret = osd_req_op_copy_from_init(req, src_snapid, 0,
+ src_oid, src_oloc,
+ src_fadvise_flags,
+ dst_fadvise_flags,
+ truncate_seq,
+ truncate_size,
+ CEPH_OSD_COPY_FROM_FLAG_TRUNCATE_SEQ);
+ if (ret)
+ goto out;
+
+ ret = ceph_osdc_alloc_messages(req, GFP_KERNEL);
+ if (ret)
+ goto out;
+
+ return req;
+
+out:
+ ceph_osdc_put_request(req);
+ return ERR_PTR(ret);
+}
+
static ssize_t ceph_do_objects_copy(struct ceph_inode_info *src_ci, u64 *src_off,
struct ceph_inode_info *dst_ci, u64 *dst_off,
struct ceph_fs_client *fsc,
{
struct ceph_object_locator src_oloc, dst_oloc;
struct ceph_object_id src_oid, dst_oid;
+ struct ceph_osd_client *osdc;
+ struct ceph_osd_request *req;
size_t bytes = 0;
u64 src_objnum, src_objoff, dst_objnum, dst_objoff;
u32 src_objlen, dst_objlen;
src_oloc.pool_ns = ceph_try_get_string(src_ci->i_layout.pool_ns);
dst_oloc.pool = dst_ci->i_layout.pool_id;
dst_oloc.pool_ns = ceph_try_get_string(dst_ci->i_layout.pool_ns);
+ osdc = &fsc->client->osdc;
while (len >= object_size) {
ceph_calc_file_object_mapping(&src_ci->i_layout, *src_off,
ceph_oid_printf(&dst_oid, "%llx.%08llx",
dst_ci->i_vino.ino, dst_objnum);
/* Do an object remote copy */
- ret = ceph_osdc_copy_from(&fsc->client->osdc,
- src_ci->i_vino.snap, 0,
- &src_oid, &src_oloc,
- CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
- CEPH_OSD_OP_FLAG_FADVISE_NOCACHE,
- &dst_oid, &dst_oloc,
- CEPH_OSD_OP_FLAG_FADVISE_SEQUENTIAL |
- CEPH_OSD_OP_FLAG_FADVISE_DONTNEED,
- dst_ci->i_truncate_seq,
- dst_ci->i_truncate_size,
- CEPH_OSD_COPY_FROM_FLAG_TRUNCATE_SEQ);
+ req = ceph_alloc_copyfrom_request(osdc, src_ci->i_vino.snap,
+ &src_oid, &src_oloc,
+ &dst_oid, &dst_oloc,
+ dst_ci->i_truncate_seq,
+ dst_ci->i_truncate_size);
+ if (IS_ERR(req))
+ ret = PTR_ERR(req);
+ else {
+ ceph_osdc_start_request(osdc, req, false);
+ ret = ceph_osdc_wait_request(osdc, req);
+ ceph_update_copyfrom_metrics(&fsc->mdsc->metric,
+ req->r_start_latency,
+ req->r_end_latency,
+ object_size, ret);
+ ceph_osdc_put_request(req);
+ }
if (ret) {
if (ret == -EOPNOTSUPP) {
fsc->have_copy_from2 = false;
static void ceph_do_invalidate_pages(struct inode *inode)
{
struct ceph_inode_info *ci = ceph_inode(inode);
- struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
u32 orig_gen;
int check = 0;
mutex_lock(&ci->i_truncate_mutex);
- if (READ_ONCE(fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN) {
- pr_warn_ratelimited("invalidate_pages %p %lld forced umount\n",
- inode, ceph_ino(inode));
+ if (ceph_inode_is_shutdown(inode)) {
+ pr_warn_ratelimited("%s: inode %llx.%llx is shut down\n",
+ __func__, ceph_vinop(inode));
mapping_set_error(inode->i_mapping, -EIO);
truncate_pagecache(inode, 0);
mutex_unlock(&ci->i_truncate_mutex);
ceph_fscache_invalidate(inode);
if (invalidate_inode_pages2(inode->i_mapping) < 0) {
- pr_err("invalidate_pages %p fails\n", inode);
+ pr_err("invalidate_inode_pages2 %llx.%llx failed\n",
+ ceph_vinop(inode));
}
spin_lock(&ci->i_ceph_lock);
loff_t isize = i_size_read(inode);
dout("setattr %p size %lld -> %lld\n", inode, isize, attr->ia_size);
- if ((issued & CEPH_CAP_FILE_EXCL) && attr->ia_size > isize) {
- i_size_write(inode, attr->ia_size);
- inode->i_blocks = calc_inode_blocks(attr->ia_size);
- ci->i_reported_size = attr->ia_size;
- dirtied |= CEPH_CAP_FILE_EXCL;
- ia_valid |= ATTR_MTIME;
+ if ((issued & CEPH_CAP_FILE_EXCL) && attr->ia_size >= isize) {
+ if (attr->ia_size > isize) {
+ i_size_write(inode, attr->ia_size);
+ inode->i_blocks = calc_inode_blocks(attr->ia_size);
+ ci->i_reported_size = attr->ia_size;
+ dirtied |= CEPH_CAP_FILE_EXCL;
+ ia_valid |= ATTR_MTIME;
+ }
} else if ((issued & CEPH_CAP_FILE_SHARED) == 0 ||
attr->ia_size != isize) {
req->r_args.setattr.size = cpu_to_le64(attr->ia_size);
if (ceph_snap(inode) != CEPH_NOSNAP)
return -EROFS;
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
err = setattr_prepare(&init_user_ns, dentry, attr);
if (err != 0)
return err;
u32 valid_mask = STATX_BASIC_STATS;
int err = 0;
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
/* Skip the getattr altogether if we're asked not to sync */
if (!(flags & AT_STATX_DONT_SYNC)) {
err = ceph_do_getattr(inode,
stat->result_mask = request_mask & valid_mask;
return err;
}
+
+void ceph_inode_shutdown(struct inode *inode)
+{
+ struct ceph_inode_info *ci = ceph_inode(inode);
+ struct rb_node *p;
+ int iputs = 0;
+ bool invalidate = false;
+
+ spin_lock(&ci->i_ceph_lock);
+ ci->i_ceph_flags |= CEPH_I_SHUTDOWN;
+ p = rb_first(&ci->i_caps);
+ while (p) {
+ struct ceph_cap *cap = rb_entry(p, struct ceph_cap, ci_node);
+
+ p = rb_next(p);
+ iputs += ceph_purge_inode_cap(inode, cap, &invalidate);
+ }
+ spin_unlock(&ci->i_ceph_lock);
+
+ if (invalidate)
+ ceph_queue_invalidate(inode);
+ while (iputs--)
+ iput(inode);
+}
if (!(fl->fl_flags & FL_POSIX))
return -ENOLCK;
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
dout("ceph_lock, fl_owner: %p\n", fl->fl_owner);
/* set wait bit as appropriate, then make command as Ceph expects it*/
if (!(fl->fl_flags & FL_FLOCK))
return -ENOLCK;
+ if (ceph_inode_is_shutdown(inode))
+ return -ESTALE;
+
dout("ceph_flock, fl_file: %p\n", fl->fl_file);
spin_lock(&ci->i_ceph_lock);
return ret;
}
-static int remove_capsnaps(struct ceph_mds_client *mdsc, struct inode *inode)
-{
- struct ceph_inode_info *ci = ceph_inode(inode);
- struct ceph_cap_snap *capsnap;
- int capsnap_release = 0;
-
- lockdep_assert_held(&ci->i_ceph_lock);
-
- dout("removing capsnaps, ci is %p, inode is %p\n", ci, inode);
-
- while (!list_empty(&ci->i_cap_snaps)) {
- capsnap = list_first_entry(&ci->i_cap_snaps,
- struct ceph_cap_snap, ci_item);
- __ceph_remove_capsnap(inode, capsnap, NULL, NULL);
- ceph_put_snap_context(capsnap->context);
- ceph_put_cap_snap(capsnap);
- capsnap_release++;
- }
- wake_up_all(&ci->i_cap_wq);
- wake_up_all(&mdsc->cap_flushing_wq);
- return capsnap_release;
-}
-
static int remove_session_caps_cb(struct inode *inode, struct ceph_cap *cap,
void *arg)
{
- struct ceph_fs_client *fsc = (struct ceph_fs_client *)arg;
- struct ceph_mds_client *mdsc = fsc->mdsc;
struct ceph_inode_info *ci = ceph_inode(inode);
- LIST_HEAD(to_remove);
- bool dirty_dropped = false;
bool invalidate = false;
- int capsnap_release = 0;
+ int iputs;
dout("removing cap %p, ci is %p, inode is %p\n",
cap, ci, &ci->vfs_inode);
spin_lock(&ci->i_ceph_lock);
- __ceph_remove_cap(cap, false);
- if (!ci->i_auth_cap) {
- struct ceph_cap_flush *cf;
-
- if (READ_ONCE(fsc->mount_state) >= CEPH_MOUNT_SHUTDOWN) {
- if (inode->i_data.nrpages > 0)
- invalidate = true;
- if (ci->i_wrbuffer_ref > 0)
- mapping_set_error(&inode->i_data, -EIO);
- }
-
- while (!list_empty(&ci->i_cap_flush_list)) {
- cf = list_first_entry(&ci->i_cap_flush_list,
- struct ceph_cap_flush, i_list);
- list_move(&cf->i_list, &to_remove);
- }
-
- spin_lock(&mdsc->cap_dirty_lock);
-
- list_for_each_entry(cf, &to_remove, i_list)
- list_del_init(&cf->g_list);
-
- if (!list_empty(&ci->i_dirty_item)) {
- pr_warn_ratelimited(
- " dropping dirty %s state for %p %lld\n",
- ceph_cap_string(ci->i_dirty_caps),
- inode, ceph_ino(inode));
- ci->i_dirty_caps = 0;
- list_del_init(&ci->i_dirty_item);
- dirty_dropped = true;
- }
- if (!list_empty(&ci->i_flushing_item)) {
- pr_warn_ratelimited(
- " dropping dirty+flushing %s state for %p %lld\n",
- ceph_cap_string(ci->i_flushing_caps),
- inode, ceph_ino(inode));
- ci->i_flushing_caps = 0;
- list_del_init(&ci->i_flushing_item);
- mdsc->num_cap_flushing--;
- dirty_dropped = true;
- }
- spin_unlock(&mdsc->cap_dirty_lock);
-
- if (dirty_dropped) {
- mapping_set_error(inode->i_mapping, -EIO);
-
- if (ci->i_wrbuffer_ref_head == 0 &&
- ci->i_wr_ref == 0 &&
- ci->i_dirty_caps == 0 &&
- ci->i_flushing_caps == 0) {
- ceph_put_snap_context(ci->i_head_snapc);
- ci->i_head_snapc = NULL;
- }
- }
-
- if (atomic_read(&ci->i_filelock_ref) > 0) {
- /* make further file lock syscall return -EIO */
- ci->i_ceph_flags |= CEPH_I_ERROR_FILELOCK;
- pr_warn_ratelimited(" dropping file locks for %p %lld\n",
- inode, ceph_ino(inode));
- }
-
- if (!ci->i_dirty_caps && ci->i_prealloc_cap_flush) {
- list_add(&ci->i_prealloc_cap_flush->i_list, &to_remove);
- ci->i_prealloc_cap_flush = NULL;
- }
-
- if (!list_empty(&ci->i_cap_snaps))
- capsnap_release = remove_capsnaps(mdsc, inode);
- }
+ iputs = ceph_purge_inode_cap(inode, cap, &invalidate);
spin_unlock(&ci->i_ceph_lock);
- while (!list_empty(&to_remove)) {
- struct ceph_cap_flush *cf;
- cf = list_first_entry(&to_remove,
- struct ceph_cap_flush, i_list);
- list_del_init(&cf->i_list);
- if (!cf->is_capsnap)
- ceph_free_cap_flush(cf);
- }
wake_up_all(&ci->i_cap_wq);
if (invalidate)
ceph_queue_invalidate(inode);
- if (dirty_dropped)
- iput(inode);
- while (capsnap_release--)
+ while (iputs--)
iput(inode);
return 0;
}
if (msg_version >= 3) {
u32 len;
- /* version >= 2, metadata */
- if (__decode_session_metadata(&p, end, &blocklisted) < 0)
+ /* version >= 2 and < 5, decode metadata, skip otherwise
+ * as it's handled via flags.
+ */
+ if (msg_version >= 5)
+ ceph_decode_skip_map(&p, end, string, string, bad);
+ else if (__decode_session_metadata(&p, end, &blocklisted) < 0)
goto bad;
+
/* version >= 3, feature bits */
ceph_decode_32_safe(&p, end, len, bad);
if (len) {
}
}
+ if (msg_version >= 5) {
+ u32 flags;
+ /* version >= 4, struct_v, struct_cv, len, metric_spec */
+ ceph_decode_skip_n(&p, end, 2 + sizeof(u32) * 2, bad);
+ /* version >= 5, flags */
+ ceph_decode_32_safe(&p, end, flags, bad);
+ if (flags & CEPH_SESSION_BLOCKLISTED) {
+ pr_warn("mds%d session blocklisted\n", session->s_mds);
+ blocklisted = true;
+ }
+ }
+
mutex_lock(&mdsc->mutex);
if (op == CEPH_SESSION_CLOSE) {
ceph_get_mds_session(session);
return;
bad:
- pr_err("error decoding fsmap\n");
+ pr_err("error decoding fsmap %d. Shutting down mount.\n", err);
+ ceph_umount_begin(mdsc->fsc->sb);
err_out:
mutex_lock(&mdsc->mutex);
mdsc->mdsmap_err = err;
bad_unlock:
mutex_unlock(&mdsc->mutex);
bad:
- pr_err("error decoding mdsmap %d\n", err);
+ pr_err("error decoding mdsmap %d. Shutting down mount.\n", err);
+ ceph_umount_begin(mdsc->fsc->sb);
return;
}
goto nomem;
for (j = 0; j < num_export_targets; j++) {
target = ceph_decode_32(&pexport_targets);
- if (target >= m->possible_max_rank) {
- err = -EIO;
- goto corrupt;
- }
info->export_targets[j] = target;
}
} else {
read->header.ver = 1;
read->header.compat = 1;
read->header.data_len = cpu_to_le32(sizeof(*read) - header_len);
- sum = m->read_latency_sum;
+ sum = m->metric[METRIC_READ].latency_sum;
jiffies_to_timespec64(sum, &ts);
read->sec = cpu_to_le32(ts.tv_sec);
read->nsec = cpu_to_le32(ts.tv_nsec);
write->header.ver = 1;
write->header.compat = 1;
write->header.data_len = cpu_to_le32(sizeof(*write) - header_len);
- sum = m->write_latency_sum;
+ sum = m->metric[METRIC_WRITE].latency_sum;
jiffies_to_timespec64(sum, &ts);
write->sec = cpu_to_le32(ts.tv_sec);
write->nsec = cpu_to_le32(ts.tv_nsec);
meta->header.ver = 1;
meta->header.compat = 1;
meta->header.data_len = cpu_to_le32(sizeof(*meta) - header_len);
- sum = m->metadata_latency_sum;
+ sum = m->metric[METRIC_METADATA].latency_sum;
jiffies_to_timespec64(sum, &ts);
meta->sec = cpu_to_le32(ts.tv_sec);
meta->nsec = cpu_to_le32(ts.tv_nsec);
rsize->header.ver = 1;
rsize->header.compat = 1;
rsize->header.data_len = cpu_to_le32(sizeof(*rsize) - header_len);
- rsize->total_ops = cpu_to_le64(m->total_reads);
- rsize->total_size = cpu_to_le64(m->read_size_sum);
+ rsize->total_ops = cpu_to_le64(m->metric[METRIC_READ].total);
+ rsize->total_size = cpu_to_le64(m->metric[METRIC_READ].size_sum);
items++;
/* encode the write io size metric */
wsize->header.ver = 1;
wsize->header.compat = 1;
wsize->header.data_len = cpu_to_le32(sizeof(*wsize) - header_len);
- wsize->total_ops = cpu_to_le64(m->total_writes);
- wsize->total_size = cpu_to_le64(m->write_size_sum);
+ wsize->total_ops = cpu_to_le64(m->metric[METRIC_WRITE].total);
+ wsize->total_size = cpu_to_le64(m->metric[METRIC_WRITE].size_sum);
items++;
put_unaligned_le32(items, &head->num);
int ceph_metric_init(struct ceph_client_metric *m)
{
- int ret;
+ struct ceph_metric *metric;
+ int ret, i;
if (!m)
return -EINVAL;
if (ret)
goto err_i_caps_mis;
- spin_lock_init(&m->read_metric_lock);
- m->read_latency_sq_sum = 0;
- m->read_latency_min = KTIME_MAX;
- m->read_latency_max = 0;
- m->total_reads = 0;
- m->read_latency_sum = 0;
- m->read_size_min = U64_MAX;
- m->read_size_max = 0;
- m->read_size_sum = 0;
-
- spin_lock_init(&m->write_metric_lock);
- m->write_latency_sq_sum = 0;
- m->write_latency_min = KTIME_MAX;
- m->write_latency_max = 0;
- m->total_writes = 0;
- m->write_latency_sum = 0;
- m->write_size_min = U64_MAX;
- m->write_size_max = 0;
- m->write_size_sum = 0;
-
- spin_lock_init(&m->metadata_metric_lock);
- m->metadata_latency_sq_sum = 0;
- m->metadata_latency_min = KTIME_MAX;
- m->metadata_latency_max = 0;
- m->total_metadatas = 0;
- m->metadata_latency_sum = 0;
+ for (i = 0; i < METRIC_MAX; i++) {
+ metric = &m->metric[i];
+ spin_lock_init(&metric->lock);
+ metric->size_sum = 0;
+ metric->size_min = U64_MAX;
+ metric->size_max = 0;
+ metric->total = 0;
+ metric->latency_sum = 0;
+ metric->latency_sq_sum = 0;
+ metric->latency_min = KTIME_MAX;
+ metric->latency_max = 0;
+ }
atomic64_set(&m->opened_files, 0);
ret = percpu_counter_init(&m->opened_inodes, 0, GFP_KERNEL);
*sq_sump += sq;
}
-void ceph_update_read_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- unsigned int size, int rc)
+void ceph_update_metrics(struct ceph_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ unsigned int size, int rc)
{
ktime_t lat = ktime_sub(r_end, r_start);
ktime_t total;
if (unlikely(rc < 0 && rc != -ENOENT && rc != -ETIMEDOUT))
return;
- spin_lock(&m->read_metric_lock);
- total = ++m->total_reads;
- m->read_size_sum += size;
- m->read_latency_sum += lat;
- METRIC_UPDATE_MIN_MAX(m->read_size_min,
- m->read_size_max,
- size);
- METRIC_UPDATE_MIN_MAX(m->read_latency_min,
- m->read_latency_max,
- lat);
- __update_stdev(total, m->read_latency_sum,
- &m->read_latency_sq_sum, lat);
- spin_unlock(&m->read_metric_lock);
-}
-
-void ceph_update_write_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- unsigned int size, int rc)
-{
- ktime_t lat = ktime_sub(r_end, r_start);
- ktime_t total;
-
- if (unlikely(rc && rc != -ETIMEDOUT))
- return;
-
- spin_lock(&m->write_metric_lock);
- total = ++m->total_writes;
- m->write_size_sum += size;
- m->write_latency_sum += lat;
- METRIC_UPDATE_MIN_MAX(m->write_size_min,
- m->write_size_max,
- size);
- METRIC_UPDATE_MIN_MAX(m->write_latency_min,
- m->write_latency_max,
- lat);
- __update_stdev(total, m->write_latency_sum,
- &m->write_latency_sq_sum, lat);
- spin_unlock(&m->write_metric_lock);
-}
-
-void ceph_update_metadata_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- int rc)
-{
- ktime_t lat = ktime_sub(r_end, r_start);
- ktime_t total;
-
- if (unlikely(rc && rc != -ENOENT))
- return;
-
- spin_lock(&m->metadata_metric_lock);
- total = ++m->total_metadatas;
- m->metadata_latency_sum += lat;
- METRIC_UPDATE_MIN_MAX(m->metadata_latency_min,
- m->metadata_latency_max,
- lat);
- __update_stdev(total, m->metadata_latency_sum,
- &m->metadata_latency_sq_sum, lat);
- spin_unlock(&m->metadata_metric_lock);
+ spin_lock(&m->lock);
+ total = ++m->total;
+ m->size_sum += size;
+ METRIC_UPDATE_MIN_MAX(m->size_min, m->size_max, size);
+ m->latency_sum += lat;
+ METRIC_UPDATE_MIN_MAX(m->latency_min, m->latency_max, lat);
+ __update_stdev(total, m->latency_sum, &m->latency_sq_sum, lat);
+ spin_unlock(&m->lock);
}
__le32 num; /* the number of metrics that will be sent */
} __packed;
+enum metric_type {
+ METRIC_READ,
+ METRIC_WRITE,
+ METRIC_METADATA,
+ METRIC_COPYFROM,
+ METRIC_MAX
+};
+
+struct ceph_metric {
+ spinlock_t lock;
+ u64 total;
+ u64 size_sum;
+ u64 size_min;
+ u64 size_max;
+ ktime_t latency_sum;
+ ktime_t latency_sq_sum;
+ ktime_t latency_min;
+ ktime_t latency_max;
+};
+
/* This is the global metrics */
struct ceph_client_metric {
atomic64_t total_dentries;
struct percpu_counter i_caps_hit;
struct percpu_counter i_caps_mis;
- spinlock_t read_metric_lock;
- u64 total_reads;
- u64 read_size_sum;
- u64 read_size_min;
- u64 read_size_max;
- ktime_t read_latency_sum;
- ktime_t read_latency_sq_sum;
- ktime_t read_latency_min;
- ktime_t read_latency_max;
-
- spinlock_t write_metric_lock;
- u64 total_writes;
- u64 write_size_sum;
- u64 write_size_min;
- u64 write_size_max;
- ktime_t write_latency_sum;
- ktime_t write_latency_sq_sum;
- ktime_t write_latency_min;
- ktime_t write_latency_max;
-
- spinlock_t metadata_metric_lock;
- u64 total_metadatas;
- ktime_t metadata_latency_sum;
- ktime_t metadata_latency_sq_sum;
- ktime_t metadata_latency_min;
- ktime_t metadata_latency_max;
+ struct ceph_metric metric[METRIC_MAX];
/* The total number of directories and files that are opened */
atomic64_t opened_files;
percpu_counter_inc(&m->i_caps_mis);
}
-extern void ceph_update_read_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- unsigned int size, int rc);
-extern void ceph_update_write_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- unsigned int size, int rc);
-extern void ceph_update_metadata_metrics(struct ceph_client_metric *m,
- ktime_t r_start, ktime_t r_end,
- int rc);
+extern void ceph_update_metrics(struct ceph_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ unsigned int size, int rc);
+
+static inline void ceph_update_read_metrics(struct ceph_client_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ unsigned int size, int rc)
+{
+ ceph_update_metrics(&m->metric[METRIC_READ],
+ r_start, r_end, size, rc);
+}
+static inline void ceph_update_write_metrics(struct ceph_client_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ unsigned int size, int rc)
+{
+ ceph_update_metrics(&m->metric[METRIC_WRITE],
+ r_start, r_end, size, rc);
+}
+static inline void ceph_update_metadata_metrics(struct ceph_client_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ int rc)
+{
+ ceph_update_metrics(&m->metric[METRIC_METADATA],
+ r_start, r_end, 0, rc);
+}
+static inline void ceph_update_copyfrom_metrics(struct ceph_client_metric *m,
+ ktime_t r_start, ktime_t r_end,
+ unsigned int size, int rc)
+{
+ ceph_update_metrics(&m->metric[METRIC_COPYFROM],
+ r_start, r_end, size, rc);
+}
#endif /* _FS_CEPH_MDS_METRIC_H */
struct ceph_fs_client *fsc = ceph_inode_to_client(d_inode(dentry));
struct ceph_mon_client *monc = &fsc->client->monc;
struct ceph_statfs st;
- u64 fsid;
- int err;
+ int i, err;
u64 data_pool;
if (fsc->mdsc->mdsmap->m_num_data_pg_pools == 1) {
buf->f_namelen = NAME_MAX;
/* Must convert the fsid, for consistent values across arches */
+ buf->f_fsid.val[0] = 0;
mutex_lock(&monc->mutex);
- fsid = le64_to_cpu(*(__le64 *)(&monc->monmap->fsid)) ^
- le64_to_cpu(*((__le64 *)&monc->monmap->fsid + 1));
+ for (i = 0 ; i < sizeof(monc->monmap->fsid) / sizeof(__le32) ; ++i)
+ buf->f_fsid.val[0] ^= le32_to_cpu(((__le32 *)&monc->monmap->fsid)[i]);
mutex_unlock(&monc->mutex);
- buf->f_fsid = u64_to_fsid(fsid);
+ /* fold the fs_cluster_id into the upper bits */
+ buf->f_fsid.val[1] = monc->fs_cluster_id;
return 0;
}
if (fsopt->flags & CEPH_MOUNT_OPT_CLEANRECOVER)
seq_show_option(m, "recover_session", "clean");
- if (fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS)
- seq_puts(m, ",nowsync");
+ if (!(fsopt->flags & CEPH_MOUNT_OPT_ASYNC_DIROPS))
+ seq_puts(m, ",wsync");
if (fsopt->wsize != CEPH_MAX_WRITE_SIZE)
seq_printf(m, ",wsize=%u", fsopt->wsize);
* ceph_umount_begin - initiate forced umount. Tear down the
* mount, skipping steps that may hang while waiting for server(s).
*/
-static void ceph_umount_begin(struct super_block *sb)
+void ceph_umount_begin(struct super_block *sb)
{
struct ceph_fs_client *fsc = ceph_sb_to_client(sb);
#define CEPH_MOUNT_OPT_DEFAULT \
(CEPH_MOUNT_OPT_DCACHE | \
- CEPH_MOUNT_OPT_NOCOPYFROM)
+ CEPH_MOUNT_OPT_NOCOPYFROM | \
+ CEPH_MOUNT_OPT_ASYNC_DIROPS)
#define ceph_set_mount_opt(fsc, opt) \
(fsc)->mount_options->flags |= CEPH_MOUNT_OPT_##opt
struct dentry *debugfs_congestion_kb;
struct dentry *debugfs_bdi;
struct dentry *debugfs_mdsc, *debugfs_mdsmap;
- struct dentry *debugfs_metric;
struct dentry *debugfs_status;
struct dentry *debugfs_mds_sessions;
+ struct dentry *debugfs_metrics_dir;
#endif
#ifdef CONFIG_CEPH_FSCACHE
#define CEPH_I_ODIRECT (1 << 11) /* inode in direct I/O mode */
#define CEPH_ASYNC_CREATE_BIT (12) /* async create in flight for this */
#define CEPH_I_ASYNC_CREATE (1 << CEPH_ASYNC_CREATE_BIT)
+#define CEPH_I_SHUTDOWN (1 << 13) /* inode is no longer usable */
/*
* Masks of ceph inode work.
struct ceph_snapid_map *sm);
extern void ceph_trim_snapid_map(struct ceph_mds_client *mdsc);
extern void ceph_cleanup_snapid_map(struct ceph_mds_client *mdsc);
+void ceph_umount_begin(struct super_block *sb);
/*
extern int ceph_getattr(struct user_namespace *mnt_userns,
const struct path *path, struct kstat *stat,
u32 request_mask, unsigned int flags);
+void ceph_inode_shutdown(struct inode *inode);
+
+static inline bool ceph_inode_is_shutdown(struct inode *inode)
+{
+ unsigned long flags = READ_ONCE(ceph_inode(inode)->i_ceph_flags);
+ struct ceph_fs_client *fsc = ceph_inode_to_client(inode);
+ int state = READ_ONCE(fsc->mount_state);
+
+ return (flags & CEPH_I_SHUTDOWN) || state >= CEPH_MOUNT_SHUTDOWN;
+}
/* xattr.c */
int __ceph_setxattr(struct inode *, const char *, const void *, size_t, int);
extern int ceph_uninline_data(struct file *filp, struct page *locked_page);
extern int ceph_pool_perm_check(struct inode *inode, int need);
extern void ceph_pool_perm_destroy(struct ceph_mds_client* mdsc);
+int ceph_purge_inode_cap(struct inode *inode, struct ceph_cap *cap, bool *invalidate);
/* file.c */
extern const struct file_operations ceph_file_fops;
c = 0;
spin_lock(&cifs_tcp_ses_lock);
list_for_each_entry(server, &cifs_tcp_ses_list, tcp_ses_list) {
- if (server->is_channel)
+ /* channel info will be printed as a part of sessions below */
+ if (CIFS_SERVER_IS_CHAN(server))
continue;
c++;
seq_printf(m, " signed");
if (server->posix_ext_supported)
seq_printf(m, " posix");
+ if (server->nosharesock)
+ seq_printf(m, " nosharesock");
if (server->rdma)
seq_printf(m, "\nRDMA ");
from_kuid(&init_user_ns, ses->linux_uid),
from_kuid(&init_user_ns, ses->cred_uid));
+ spin_lock(&ses->chan_lock);
if (ses->chan_count > 1) {
seq_printf(m, "\n\n\tExtra Channels: %zu ",
ses->chan_count-1);
for (j = 1; j < ses->chan_count; j++)
cifs_dump_channel(m, j, &ses->chans[j]);
}
+ spin_unlock(&ses->chan_lock);
seq_puts(m, "\n\n\tShares: ");
j = 0;
static struct vfsmount *cifs_dfs_do_automount(struct dentry *mntpt)
{
struct cifs_sb_info *cifs_sb;
- struct cifs_ses *ses;
- struct cifs_tcon *tcon;
void *page;
- char *full_path, *root_path;
- unsigned int xid;
- int rc;
+ char *full_path;
struct vfsmount *mnt;
cifs_dbg(FYI, "in %s\n", __func__);
* the double backslashes usually used in the UNC. This function
* gives us the latter, so we must adjust the result.
*/
- mnt = ERR_PTR(-ENOMEM);
-
cifs_sb = CIFS_SB(mntpt->d_sb);
if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_DFS) {
mnt = ERR_PTR(-EREMOTE);
}
convert_delimiter(full_path, '\\');
-
cifs_dbg(FYI, "%s: full_path: %s\n", __func__, full_path);
- if (!cifs_sb_master_tlink(cifs_sb)) {
- cifs_dbg(FYI, "%s: master tlink is NULL\n", __func__);
- goto free_full_path;
- }
-
- tcon = cifs_sb_master_tcon(cifs_sb);
- if (!tcon) {
- cifs_dbg(FYI, "%s: master tcon is NULL\n", __func__);
- goto free_full_path;
- }
-
- root_path = kstrdup(tcon->treeName, GFP_KERNEL);
- if (!root_path) {
- mnt = ERR_PTR(-ENOMEM);
- goto free_full_path;
- }
- cifs_dbg(FYI, "%s: root path: %s\n", __func__, root_path);
-
- ses = tcon->ses;
- xid = get_xid();
-
- /*
- * If DFS root has been expired, then unconditionally fetch it again to
- * refresh DFS referral cache.
- */
- rc = dfs_cache_find(xid, ses, cifs_sb->local_nls, cifs_remap(cifs_sb),
- root_path + 1, NULL, NULL);
- if (!rc) {
- rc = dfs_cache_find(xid, ses, cifs_sb->local_nls,
- cifs_remap(cifs_sb), full_path + 1,
- NULL, NULL);
- }
-
- free_xid(xid);
-
- if (rc) {
- mnt = ERR_PTR(rc);
- goto free_root_path;
- }
- /*
- * OK - we were able to get and cache a referral for @full_path.
- *
- * Now, pass it down to cifs_mount() and it will retry every available
- * node server in case of failures - no need to do it here.
- */
mnt = cifs_dfs_do_mount(mntpt, cifs_sb, full_path);
- cifs_dbg(FYI, "%s: cifs_dfs_do_mount:%s , mnt:%p\n", __func__,
- full_path + 1, mnt);
+ cifs_dbg(FYI, "%s: cifs_dfs_do_mount:%s , mnt:%p\n", __func__, full_path + 1, mnt);
-free_root_path:
- kfree(root_path);
free_full_path:
free_dentry_path(page);
cdda_exit:
/* only used when CIFS_MOUNT_USE_PREFIX_PATH is set */
char *prepath;
- /*
- * Canonical DFS path initially provided by the mount call. We might connect to something
- * different via DFS but we want to keep it to do failover properly.
- */
- char *origin_fullpath; /* \\HOST\SHARE\[OPTIONAL PATH] */
/* randomly generated 128-bit number for indexing dfs mount groups in referral cache */
uuid_t dfs_mount_id;
/*
static int cifs_swn_resource_state_changed(struct cifs_swn_reg *swnreg, const char *name, int state)
{
- int i;
-
switch (state) {
case CIFS_SWN_RESOURCE_STATE_UNAVAILABLE:
cifs_dbg(FYI, "%s: resource name '%s' become unavailable\n", __func__, name);
- for (i = 0; i < swnreg->tcon->ses->chan_count; i++) {
- spin_lock(&GlobalMid_Lock);
- if (swnreg->tcon->ses->chans[i].server->tcpStatus != CifsExiting)
- swnreg->tcon->ses->chans[i].server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&GlobalMid_Lock);
- }
+ cifs_ses_mark_for_reconnect(swnreg->tcon->ses);
break;
case CIFS_SWN_RESOURCE_STATE_AVAILABLE:
cifs_dbg(FYI, "%s: resource name '%s' become available\n", __func__, name);
- for (i = 0; i < swnreg->tcon->ses->chan_count; i++) {
- spin_lock(&GlobalMid_Lock);
- if (swnreg->tcon->ses->chans[i].server->tcpStatus != CifsExiting)
- swnreg->tcon->ses->chans[i].server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&GlobalMid_Lock);
- }
+ cifs_ses_mark_for_reconnect(swnreg->tcon->ses);
break;
case CIFS_SWN_RESOURCE_STATE_UNKNOWN:
cifs_dbg(FYI, "%s: resource name '%s' changed to unknown state\n", __func__, name);
#include <linux/key-type.h>
#include "cifs_spnego.h"
#include "fscache.h"
-#include "smb2pdu.h"
#ifdef CONFIG_CIFS_DFS_UPCALL
#include "dfs_cache.h"
#endif
#include <linux/slab.h>
#include <linux/mempool.h>
#include <linux/workqueue.h>
+#include <linux/utsname.h>
#include "cifs_fs_sb.h"
#include "cifsacl.h"
#include <crypto/internal/hash.h>
#include <linux/scatterlist.h>
#include <uapi/linux/cifs/cifs_mount.h>
+#include "../smbfs_common/smb2pdu.h"
#include "smb2pdu.h"
#define CIFS_MAGIC_NUMBER 0xFF534D42 /* the first four bytes of SMB PDUs */
#define SMB_ECHO_INTERVAL_MAX 600
#define SMB_ECHO_INTERVAL_DEFAULT 60
-/* dns resolution interval in seconds */
+/* dns resolution intervals in seconds */
+#define SMB_DNS_RESOLVE_INTERVAL_MIN 120
#define SMB_DNS_RESOLVE_INTERVAL_DEFAULT 600
/* maximum number of PDUs in one compound */
#define XATTR_DOS_ATTRIB "user.DOSATTRIB"
#endif
+#define CIFS_MAX_WORKSTATION_LEN (__NEW_UTS_LEN + 1) /* reasonable max for client */
+
/*
* CIFS vfs client Status information (based on what we know.)
*/
struct list_head pending_mid_q;
bool noblocksnd; /* use blocking sendmsg */
bool noautotune; /* do not autotune send buf sizes */
+ bool nosharesock;
bool tcp_nodelay;
unsigned int credits; /* send no more requests at once */
unsigned int max_credits; /* can override large 32000 default at mnt */
*/
int nr_targets;
bool noblockcnt; /* use non-blocking connect() */
- bool is_channel; /* if a session channel */
+
+ /*
+ * If this is a session channel,
+ * primary_server holds the ref-counted
+ * pointer to primary channel connection for the session.
+ */
+#define CIFS_SERVER_IS_CHAN(server) (!!(server)->primary_server)
+ struct TCP_Server_Info *primary_server;
+
#ifdef CONFIG_CIFS_SWN_UPCALL
bool use_swn_dstaddr;
struct sockaddr_storage swn_dstaddr;
#endif
#ifdef CONFIG_CIFS_DFS_UPCALL
bool is_dfs_conn; /* if a dfs connection */
+ struct mutex refpath_lock; /* protects leaf_fullpath */
+ /*
+ * Canonical DFS full paths that were used to chase referrals in mount and reconnect.
+ *
+ * origin_fullpath: first or original referral path
+ * leaf_fullpath: last referral path (might be changed due to nested links in reconnect)
+ *
+ * current_fullpath: pointer to either origin_fullpath or leaf_fullpath
+ * NOTE: cannot be accessed outside cifs_reconnect() and smb2_reconnect()
+ *
+ * format: \\HOST\SHARE\[OPTIONAL PATH]
+ */
+ char *origin_fullpath, *leaf_fullpath, *current_fullpath;
#endif
};
static inline void
revert_current_mid_from_hdr(struct TCP_Server_Info *server,
- const struct smb2_sync_hdr *shdr)
+ const struct smb2_hdr *shdr)
{
unsigned int num = le16_to_cpu(shdr->CreditCharge);
and after mount option parsing we fill it */
char *domainName;
char *password;
+ char *workstation_name;
struct session_key auth_key;
struct ntlmssp_auth *ntlmssp; /* ciphertext, flags, server challenge */
enum securityEnum sectype; /* what security flavor was specified? */
* iface_lock should be taken when accessing any of these fields
*/
spinlock_t iface_lock;
+ /* ========= begin: protected by iface_lock ======== */
struct cifs_server_iface *iface_list;
size_t iface_count;
unsigned long iface_last_update; /* jiffies */
+ /* ========= end: protected by iface_lock ======== */
+ spinlock_t chan_lock;
+ /* ========= begin: protected by chan_lock ======== */
#define CIFS_MAX_CHANNELS 16
struct cifs_chan chans[CIFS_MAX_CHANNELS];
struct cifs_chan *binding_chan;
size_t chan_count;
size_t chan_max;
atomic_t chan_seq; /* round robin state */
+ /* ========= end: protected by chan_lock ======== */
};
/*
struct cached_fid crfid; /* Cached root fid */
/* BB add field for back pointer to sb struct(s)? */
#ifdef CONFIG_CIFS_DFS_UPCALL
- char *dfs_path; /* canonical DFS path */
struct list_head ulist; /* cache update list */
#endif
};
tcon->share_flags & (SHI1005_FLAGS_DFS | SHI1005_FLAGS_DFS_ROOT);
}
+static inline bool cifs_is_referral_server(struct cifs_tcon *tcon,
+ const struct dfs_info3_param *ref)
+{
+ /*
+ * Check if all targets are capable of handling DFS referrals as per
+ * MS-DFSC 2.2.4 RESP_GET_DFS_REFERRAL.
+ */
+ return is_tcon_dfs(tcon) || (ref && (ref->flags & DFSREF_REFERRAL_SERVER));
+}
+
#endif /* _CIFS_GLOB_H */
extern void cifs_close_deferred_file_under_dentry(struct cifs_tcon *cifs_tcon,
const char *path);
-
-extern struct TCP_Server_Info *cifs_get_tcp_session(struct smb3_fs_context *ctx);
+extern struct TCP_Server_Info *
+cifs_get_tcp_session(struct smb3_fs_context *ctx,
+ struct TCP_Server_Info *primary_server);
extern void cifs_put_tcp_session(struct TCP_Server_Info *server,
int from_reconnect);
extern void cifs_put_tcon(struct cifs_tcon *tcon);
bool is_server_using_iface(struct TCP_Server_Info *server,
struct cifs_server_iface *iface);
bool is_ses_using_iface(struct cifs_ses *ses, struct cifs_server_iface *iface);
+void cifs_ses_mark_for_reconnect(struct cifs_ses *ses);
void extract_unc_hostname(const char *unc, const char **h, size_t *len);
int copy_path_name(char *dst, const char *src);
struct super_block *cifs_get_tcp_super(struct TCP_Server_Info *server);
void cifs_put_tcp_super(struct super_block *sb);
-int update_super_prepath(struct cifs_tcon *tcon, char *prefix);
+int cifs_update_super_prepath(struct cifs_sb_info *cifs_sb, char *prefix);
char *extract_hostname(const char *unc);
char *extract_sharename(const char *unc);
return options;
}
+struct super_block *cifs_get_tcon_super(struct cifs_tcon *tcon);
+void cifs_put_tcon_super(struct super_block *sb);
+
#endif /* _CIFSPROTO_H */
/* Drop the connection to not overload the server */
#define NUM_STATUS_IO_TIMEOUT 5
+struct mount_ctx {
+ struct cifs_sb_info *cifs_sb;
+ struct smb3_fs_context *fs_ctx;
+ unsigned int xid;
+ struct TCP_Server_Info *server;
+ struct cifs_ses *ses;
+ struct cifs_tcon *tcon;
+#ifdef CONFIG_CIFS_DFS_UPCALL
+ struct cifs_ses *root_ses;
+ uuid_t mount_id;
+ char *origin_fullpath, *leaf_fullpath;
+#endif
+};
+
static int ip_connect(struct TCP_Server_Info *server);
static int generic_ip_connect(struct TCP_Server_Info *server);
static void tlink_rb_insert(struct rb_root *root, struct tcon_link *new_tlink);
* To make sure we don't use the cached entry, retry 1s
* after expiry.
*/
- ttl = (expiry - now + 1);
+ ttl = max_t(unsigned long, expiry - now, SMB_DNS_RESOLVE_INTERVAL_MIN) + 1;
}
rc = !rc ? -1 : 0;
mutex_unlock(&server->srv_mutex);
}
-#ifdef CONFIG_CIFS_DFS_UPCALL
-/* These functions must be called with server->srv_mutex held */
-static void reconn_set_next_dfs_target(struct TCP_Server_Info *server,
- struct cifs_sb_info *cifs_sb,
- struct dfs_cache_tgt_list *tgt_list,
- struct dfs_cache_tgt_iterator **tgt_it)
-{
- const char *name;
- int rc;
-
- if (!cifs_sb || !cifs_sb->origin_fullpath)
- return;
-
- if (!*tgt_it) {
- *tgt_it = dfs_cache_get_tgt_iterator(tgt_list);
- } else {
- *tgt_it = dfs_cache_get_next_tgt(tgt_list, *tgt_it);
- if (!*tgt_it)
- *tgt_it = dfs_cache_get_tgt_iterator(tgt_list);
- }
-
- cifs_dbg(FYI, "%s: UNC: %s\n", __func__, cifs_sb->origin_fullpath);
-
- name = dfs_cache_get_tgt_name(*tgt_it);
-
- kfree(server->hostname);
-
- server->hostname = extract_hostname(name);
- if (IS_ERR(server->hostname)) {
- cifs_dbg(FYI,
- "%s: failed to extract hostname from target: %ld\n",
- __func__, PTR_ERR(server->hostname));
- return;
- }
-
- rc = reconn_set_ipaddr_from_hostname(server);
- if (rc) {
- cifs_dbg(FYI, "%s: failed to resolve hostname: %d\n",
- __func__, rc);
- }
-}
-
-static inline int reconn_setup_dfs_targets(struct cifs_sb_info *cifs_sb,
- struct dfs_cache_tgt_list *tl)
-{
- if (!cifs_sb->origin_fullpath)
- return -EOPNOTSUPP;
- return dfs_cache_noreq_find(cifs_sb->origin_fullpath + 1, NULL, tl);
-}
-#endif
-
-/*
- * cifs tcp session reconnection
+/**
+ * Mark all sessions and tcons for reconnect.
*
- * mark tcp session as reconnecting so temporarily locked
- * mark all smb sessions as reconnecting for tcp session
- * reconnect tcp session
- * wake up waiters on reconnection? - (not needed currently)
+ * @server needs to be previously set to CifsNeedReconnect.
*/
-int
-cifs_reconnect(struct TCP_Server_Info *server)
+static void cifs_mark_tcp_ses_conns_for_reconnect(struct TCP_Server_Info *server)
{
- int rc = 0;
- struct list_head *tmp, *tmp2;
struct cifs_ses *ses;
struct cifs_tcon *tcon;
- struct mid_q_entry *mid_entry;
+ struct mid_q_entry *mid, *nmid;
struct list_head retry_list;
-#ifdef CONFIG_CIFS_DFS_UPCALL
- struct super_block *sb = NULL;
- struct cifs_sb_info *cifs_sb = NULL;
- struct dfs_cache_tgt_list tgt_list = DFS_CACHE_TGT_LIST_INIT(tgt_list);
- struct dfs_cache_tgt_iterator *tgt_it = NULL;
-#endif
+ struct TCP_Server_Info *pserver;
- spin_lock(&GlobalMid_Lock);
- server->nr_targets = 1;
-#ifdef CONFIG_CIFS_DFS_UPCALL
- spin_unlock(&GlobalMid_Lock);
- sb = cifs_get_tcp_super(server);
- if (IS_ERR(sb)) {
- rc = PTR_ERR(sb);
- cifs_dbg(FYI, "%s: will not do DFS failover: rc = %d\n",
- __func__, rc);
- sb = NULL;
- } else {
- cifs_sb = CIFS_SB(sb);
- rc = reconn_setup_dfs_targets(cifs_sb, &tgt_list);
- if (rc) {
- cifs_sb = NULL;
- if (rc != -EOPNOTSUPP) {
- cifs_server_dbg(VFS, "%s: no target servers for DFS failover\n",
- __func__);
- }
- } else {
- server->nr_targets = dfs_cache_get_nr_tgts(&tgt_list);
- }
- }
- cifs_dbg(FYI, "%s: will retry %d target(s)\n", __func__,
- server->nr_targets);
- spin_lock(&GlobalMid_Lock);
-#endif
- if (server->tcpStatus == CifsExiting) {
- /* the demux thread will exit normally
- next time through the loop */
- spin_unlock(&GlobalMid_Lock);
-#ifdef CONFIG_CIFS_DFS_UPCALL
- dfs_cache_free_tgts(&tgt_list);
- cifs_put_tcp_super(sb);
-#endif
- wake_up(&server->response_q);
- return rc;
- } else
- server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&GlobalMid_Lock);
server->maxBuf = 0;
server->max_read = 0;
cifs_dbg(FYI, "Mark tcp session as need reconnect\n");
trace_smb3_reconnect(server->CurrentMid, server->conn_id, server->hostname);
+ /*
+ * before reconnecting the tcp session, mark the smb session (uid) and the tid bad so they
+ * are not used until reconnected.
+ */
+ cifs_dbg(FYI, "%s: marking sessions and tcons for reconnect\n", __func__);
+
+ /* If server is a channel, select the primary channel */
+ pserver = CIFS_SERVER_IS_CHAN(server) ? server->primary_server : server;
- /* before reconnecting the tcp session, mark the smb session (uid)
- and the tid bad so they are not used until reconnected */
- cifs_dbg(FYI, "%s: marking sessions and tcons for reconnect\n",
- __func__);
spin_lock(&cifs_tcp_ses_lock);
- list_for_each(tmp, &server->smb_ses_list) {
- ses = list_entry(tmp, struct cifs_ses, smb_ses_list);
+ list_for_each_entry(ses, &pserver->smb_ses_list, smb_ses_list) {
ses->need_reconnect = true;
- list_for_each(tmp2, &ses->tcon_list) {
- tcon = list_entry(tmp2, struct cifs_tcon, tcon_list);
+ list_for_each_entry(tcon, &ses->tcon_list, tcon_list)
tcon->need_reconnect = true;
- }
if (ses->tcon_ipc)
ses->tcon_ipc->need_reconnect = true;
}
cifs_dbg(FYI, "%s: tearing down socket\n", __func__);
mutex_lock(&server->srv_mutex);
if (server->ssocket) {
- cifs_dbg(FYI, "State: 0x%x Flags: 0x%lx\n",
- server->ssocket->state, server->ssocket->flags);
+ cifs_dbg(FYI, "State: 0x%x Flags: 0x%lx\n", server->ssocket->state,
+ server->ssocket->flags);
kernel_sock_shutdown(server->ssocket, SHUT_WR);
- cifs_dbg(FYI, "Post shutdown state: 0x%x Flags: 0x%lx\n",
- server->ssocket->state, server->ssocket->flags);
+ cifs_dbg(FYI, "Post shutdown state: 0x%x Flags: 0x%lx\n", server->ssocket->state,
+ server->ssocket->flags);
sock_release(server->ssocket);
server->ssocket = NULL;
}
INIT_LIST_HEAD(&retry_list);
cifs_dbg(FYI, "%s: moving mids to private list\n", __func__);
spin_lock(&GlobalMid_Lock);
- list_for_each_safe(tmp, tmp2, &server->pending_mid_q) {
- mid_entry = list_entry(tmp, struct mid_q_entry, qhead);
- kref_get(&mid_entry->refcount);
- if (mid_entry->mid_state == MID_REQUEST_SUBMITTED)
- mid_entry->mid_state = MID_RETRY_NEEDED;
- list_move(&mid_entry->qhead, &retry_list);
- mid_entry->mid_flags |= MID_DELETED;
+ list_for_each_entry_safe(mid, nmid, &server->pending_mid_q, qhead) {
+ kref_get(&mid->refcount);
+ if (mid->mid_state == MID_REQUEST_SUBMITTED)
+ mid->mid_state = MID_RETRY_NEEDED;
+ list_move(&mid->qhead, &retry_list);
+ mid->mid_flags |= MID_DELETED;
}
spin_unlock(&GlobalMid_Lock);
mutex_unlock(&server->srv_mutex);
cifs_dbg(FYI, "%s: issuing mid callbacks\n", __func__);
- list_for_each_safe(tmp, tmp2, &retry_list) {
- mid_entry = list_entry(tmp, struct mid_q_entry, qhead);
- list_del_init(&mid_entry->qhead);
- mid_entry->callback(mid_entry);
- cifs_mid_q_entry_release(mid_entry);
+ list_for_each_entry_safe(mid, nmid, &retry_list, qhead) {
+ list_del_init(&mid->qhead);
+ mid->callback(mid);
+ cifs_mid_q_entry_release(mid);
}
if (cifs_rdma_enabled(server)) {
smbd_destroy(server);
mutex_unlock(&server->srv_mutex);
}
+}
+
+static bool cifs_tcp_ses_needs_reconnect(struct TCP_Server_Info *server, int num_targets)
+{
+ spin_lock(&GlobalMid_Lock);
+ server->nr_targets = num_targets;
+ if (server->tcpStatus == CifsExiting) {
+ /* the demux thread will exit normally next time through the loop */
+ spin_unlock(&GlobalMid_Lock);
+ wake_up(&server->response_q);
+ return false;
+ }
+ server->tcpStatus = CifsNeedReconnect;
+ spin_unlock(&GlobalMid_Lock);
+ return true;
+}
+
+/*
+ * cifs tcp session reconnection
+ *
+ * mark tcp session as reconnecting so temporarily locked
+ * mark all smb sessions as reconnecting for tcp session
+ * reconnect tcp session
+ * wake up waiters on reconnection? - (not needed currently)
+ */
+static int __cifs_reconnect(struct TCP_Server_Info *server)
+{
+ int rc = 0;
+
+ if (!cifs_tcp_ses_needs_reconnect(server, 1))
+ return 0;
+
+ cifs_mark_tcp_ses_conns_for_reconnect(server);
do {
try_to_freeze();
-
mutex_lock(&server->srv_mutex);
-
if (!cifs_swn_set_server_dstaddr(server)) {
-#ifdef CONFIG_CIFS_DFS_UPCALL
- if (cifs_sb && cifs_sb->origin_fullpath)
- /*
- * Set up next DFS target server (if any) for reconnect. If DFS
- * feature is disabled, then we will retry last server we
- * connected to before.
- */
- reconn_set_next_dfs_target(server, cifs_sb, &tgt_list, &tgt_it);
- else {
-#endif
- /*
- * Resolve the hostname again to make sure that IP address is up-to-date.
- */
+ /* resolve the hostname again to make sure that IP address is up-to-date */
rc = reconn_set_ipaddr_from_hostname(server);
- if (rc) {
- cifs_dbg(FYI, "%s: failed to resolve hostname: %d\n",
- __func__, rc);
- }
-
-#ifdef CONFIG_CIFS_DFS_UPCALL
- }
-#endif
-
-
+ cifs_dbg(FYI, "%s: reconn_set_ipaddr_from_hostname: rc=%d\n", __func__, rc);
}
if (cifs_rdma_enabled(server))
else
rc = generic_ip_connect(server);
if (rc) {
- cifs_dbg(FYI, "reconnect error %d\n", rc);
mutex_unlock(&server->srv_mutex);
+ cifs_dbg(FYI, "%s: reconnect error %d\n", __func__, rc);
msleep(3000);
} else {
atomic_inc(&tcpSesReconnectCount);
}
} while (server->tcpStatus == CifsNeedReconnect);
+ if (server->tcpStatus == CifsNeedNegotiate)
+ mod_delayed_work(cifsiod_wq, &server->echo, 0);
+
+ wake_up(&server->response_q);
+ return rc;
+}
+
#ifdef CONFIG_CIFS_DFS_UPCALL
- if (tgt_it) {
- rc = dfs_cache_noreq_update_tgthint(cifs_sb->origin_fullpath + 1,
- tgt_it);
- if (rc) {
- cifs_server_dbg(VFS, "%s: failed to update DFS target hint: rc = %d\n",
- __func__, rc);
+static int __reconnect_target_unlocked(struct TCP_Server_Info *server, const char *target)
+{
+ int rc;
+ char *hostname;
+
+ if (!cifs_swn_set_server_dstaddr(server)) {
+ if (server->hostname != target) {
+ hostname = extract_hostname(target);
+ if (!IS_ERR(hostname)) {
+ kfree(server->hostname);
+ server->hostname = hostname;
+ } else {
+ cifs_dbg(FYI, "%s: couldn't extract hostname or address from dfs target: %ld\n",
+ __func__, PTR_ERR(hostname));
+ cifs_dbg(FYI, "%s: default to last target server: %s\n", __func__,
+ server->hostname);
+ }
}
- dfs_cache_free_tgts(&tgt_list);
+ /* resolve the hostname again to make sure that IP address is up-to-date. */
+ rc = reconn_set_ipaddr_from_hostname(server);
+ cifs_dbg(FYI, "%s: reconn_set_ipaddr_from_hostname: rc=%d\n", __func__, rc);
}
+ /* Reconnect the socket */
+ if (cifs_rdma_enabled(server))
+ rc = smbd_reconnect(server);
+ else
+ rc = generic_ip_connect(server);
- cifs_put_tcp_super(sb);
-#endif
+ return rc;
+}
+
+static int reconnect_target_unlocked(struct TCP_Server_Info *server, struct dfs_cache_tgt_list *tl,
+ struct dfs_cache_tgt_iterator **target_hint)
+{
+ int rc;
+ struct dfs_cache_tgt_iterator *tit;
+
+ *target_hint = NULL;
+
+ /* If dfs target list is empty, then reconnect to last server */
+ tit = dfs_cache_get_tgt_iterator(tl);
+ if (!tit)
+ return __reconnect_target_unlocked(server, server->hostname);
+
+ /* Otherwise, try every dfs target in @tl */
+ for (; tit; tit = dfs_cache_get_next_tgt(tl, tit)) {
+ rc = __reconnect_target_unlocked(server, dfs_cache_get_tgt_name(tit));
+ if (!rc) {
+ *target_hint = tit;
+ break;
+ }
+ }
+ return rc;
+}
+
+static int reconnect_dfs_server(struct TCP_Server_Info *server)
+{
+ int rc = 0;
+ const char *refpath = server->current_fullpath + 1;
+ struct dfs_cache_tgt_list tl = DFS_CACHE_TGT_LIST_INIT(tl);
+ struct dfs_cache_tgt_iterator *target_hint = NULL;
+ int num_targets = 0;
+
+ /*
+ * Determine the number of dfs targets the referral path in @cifs_sb resolves to.
+ *
+ * smb2_reconnect() needs to know how long it should wait based upon the number of dfs
+ * targets (server->nr_targets). It's also possible that the cached referral was cleared
+ * through /proc/fs/cifs/dfscache or the target list is empty due to server settings after
+ * refreshing the referral, so, in this case, default it to 1.
+ */
+ if (!dfs_cache_noreq_find(refpath, NULL, &tl))
+ num_targets = dfs_cache_get_nr_tgts(&tl);
+ if (!num_targets)
+ num_targets = 1;
+
+ if (!cifs_tcp_ses_needs_reconnect(server, num_targets))
+ return 0;
+
+ cifs_mark_tcp_ses_conns_for_reconnect(server);
+
+ do {
+ try_to_freeze();
+ mutex_lock(&server->srv_mutex);
+
+ rc = reconnect_target_unlocked(server, &tl, &target_hint);
+ if (rc) {
+ /* Failed to reconnect socket */
+ mutex_unlock(&server->srv_mutex);
+ cifs_dbg(FYI, "%s: reconnect error %d\n", __func__, rc);
+ msleep(3000);
+ continue;
+ }
+ /*
+ * Socket was created. Update tcp session status to CifsNeedNegotiate so that a
+ * process waiting for reconnect will know it needs to re-establish session and tcon
+ * through the reconnected target server.
+ */
+ atomic_inc(&tcpSesReconnectCount);
+ set_credits(server, 1);
+ spin_lock(&GlobalMid_Lock);
+ if (server->tcpStatus != CifsExiting)
+ server->tcpStatus = CifsNeedNegotiate;
+ spin_unlock(&GlobalMid_Lock);
+ cifs_swn_reset_server_dstaddr(server);
+ mutex_unlock(&server->srv_mutex);
+ } while (server->tcpStatus == CifsNeedReconnect);
+
+ if (target_hint)
+ dfs_cache_noreq_update_tgthint(refpath, target_hint);
+
+ dfs_cache_free_tgts(&tl);
+
+ /* Need to set up echo worker again once connection has been established */
if (server->tcpStatus == CifsNeedNegotiate)
mod_delayed_work(cifsiod_wq, &server->echo, 0);
return rc;
}
+int cifs_reconnect(struct TCP_Server_Info *server)
+{
+ /* If tcp session is not an dfs connection, then reconnect to last target server */
+ spin_lock(&cifs_tcp_ses_lock);
+ if (!server->is_dfs_conn || !server->origin_fullpath || !server->leaf_fullpath) {
+ spin_unlock(&cifs_tcp_ses_lock);
+ return __cifs_reconnect(server);
+ }
+ spin_unlock(&cifs_tcp_ses_lock);
+
+ return reconnect_dfs_server(server);
+}
+#else
+int cifs_reconnect(struct TCP_Server_Info *server)
+{
+ return __cifs_reconnect(server);
+}
+#endif
+
static void
cifs_echo_request(struct work_struct *work)
{
* Trying to handle/dequeue a mid after the send_recv()
* function has finished processing it is a bug.
*/
- if (mid->mid_flags & MID_DELETED)
+ if (mid->mid_flags & MID_DELETED) {
+ spin_unlock(&GlobalMid_Lock);
pr_warn_once("trying to dequeue a deleted mid\n");
- else {
+ } else {
list_del_init(&mid->qhead);
mid->mid_flags |= MID_DELETED;
+ spin_unlock(&GlobalMid_Lock);
}
- spin_unlock(&GlobalMid_Lock);
}
static unsigned int
smb2_get_credits_from_hdr(char *buffer, struct TCP_Server_Info *server)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buffer;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buffer;
/*
* SMB1 does not use credits.
*/
}
- kfree(server->hostname);
+#ifdef CONFIG_CIFS_DFS_UPCALL
+ kfree(server->origin_fullpath);
+ kfree(server->leaf_fullpath);
+#endif
kfree(server);
length = atomic_dec_return(&tcpSesAllocCount);
static void
smb2_add_credits_from_hdr(char *buffer, struct TCP_Server_Info *server)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buffer;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buffer;
int scredits, in_flight;
/*
{
struct sockaddr *addr = (struct sockaddr *)&ctx->dstaddr;
- if (ctx->nosharesock)
+ if (ctx->nosharesock) {
+ server->nosharesock = true;
+ return 0;
+ }
+
+ /* this server does not share socket */
+ if (server->nosharesock)
return 0;
/* If multidialect negotiation see if existing sessions match one */
if (!net_eq(cifs_net_ns(server), current->nsproxy->net_ns))
return 0;
+ if (strcasecmp(server->hostname, ctx->server_hostname))
+ return 0;
+
if (!match_address(server, addr,
(struct sockaddr *)&ctx->srcaddr))
return 0;
* Skip ses channels since they're only handled in lower layers
* (e.g. cifs_send_recv).
*/
- if (server->is_channel || !match_server(server, ctx))
+ if (CIFS_SERVER_IS_CHAN(server) || !match_server(server, ctx))
continue;
++server->srv_count;
list_del_init(&server->tcp_ses_list);
spin_unlock(&cifs_tcp_ses_lock);
+ /* For secondary channels, we pick up ref-count on the primary server */
+ if (CIFS_SERVER_IS_CHAN(server))
+ cifs_put_tcp_session(server->primary_server, from_reconnect);
+
cancel_delayed_work_sync(&server->echo);
cancel_delayed_work_sync(&server->resolve);
spin_unlock(&GlobalMid_Lock);
cifs_crypto_secmech_release(server);
- cifs_fscache_release_client_cookie(server);
+
+ /* fscache server cookies are based on primary channel only */
+ if (!CIFS_SERVER_IS_CHAN(server))
+ cifs_fscache_release_client_cookie(server);
kfree(server->session_key.response);
server->session_key.response = NULL;
server->session_key.len = 0;
+ kfree(server->hostname);
task = xchg(&server->tsk, NULL);
if (task)
}
struct TCP_Server_Info *
-cifs_get_tcp_session(struct smb3_fs_context *ctx)
+cifs_get_tcp_session(struct smb3_fs_context *ctx,
+ struct TCP_Server_Info *primary_server)
{
struct TCP_Server_Info *tcp_ses = NULL;
int rc;
goto out_err;
}
+ tcp_ses->hostname = kstrdup(ctx->server_hostname, GFP_KERNEL);
+ if (!tcp_ses->hostname) {
+ rc = -ENOMEM;
+ goto out_err;
+ }
+
tcp_ses->ops = ctx->ops;
tcp_ses->vals = ctx->vals;
cifs_set_net_ns(tcp_ses, get_net(current->nsproxy->net_ns));
- tcp_ses->hostname = extract_hostname(ctx->UNC);
- if (IS_ERR(tcp_ses->hostname)) {
- rc = PTR_ERR(tcp_ses->hostname);
- goto out_err_crypto_release;
- }
tcp_ses->conn_id = atomic_inc_return(&tcpSesNextId);
tcp_ses->noblockcnt = ctx->rootfs;
tcp_ses->in_flight = 0;
tcp_ses->max_in_flight = 0;
tcp_ses->credits = 1;
+ if (primary_server) {
+ spin_lock(&cifs_tcp_ses_lock);
+ ++primary_server->srv_count;
+ tcp_ses->primary_server = primary_server;
+ spin_unlock(&cifs_tcp_ses_lock);
+ }
init_waitqueue_head(&tcp_ses->response_q);
init_waitqueue_head(&tcp_ses->request_q);
INIT_LIST_HEAD(&tcp_ses->pending_mid_q);
INIT_DELAYED_WORK(&tcp_ses->resolve, cifs_resolve_server);
INIT_DELAYED_WORK(&tcp_ses->reconnect, smb2_reconnect_server);
mutex_init(&tcp_ses->reconnect_mutex);
+#ifdef CONFIG_CIFS_DFS_UPCALL
+ mutex_init(&tcp_ses->refpath_lock);
+#endif
memcpy(&tcp_ses->srcaddr, &ctx->srcaddr,
sizeof(tcp_ses->srcaddr));
memcpy(&tcp_ses->dstaddr, &ctx->dstaddr,
list_add(&tcp_ses->tcp_ses_list, &cifs_tcp_ses_list);
spin_unlock(&cifs_tcp_ses_lock);
- cifs_fscache_get_client_cookie(tcp_ses);
+ /* fscache server cookies are based on primary channel only */
+ if (!CIFS_SERVER_IS_CHAN(tcp_ses))
+ cifs_fscache_get_client_cookie(tcp_ses);
/* queue echo request delayed work */
queue_delayed_work(cifsiod_wq, &tcp_ses->echo, tcp_ses->echo_interval);
out_err:
if (tcp_ses) {
- if (!IS_ERR(tcp_ses->hostname))
- kfree(tcp_ses->hostname);
+ if (CIFS_SERVER_IS_CHAN(tcp_ses))
+ cifs_put_tcp_session(tcp_ses->primary_server, false);
+ kfree(tcp_ses->hostname);
if (tcp_ses->ssocket)
sock_release(tcp_ses->ssocket);
kfree(tcp_ses);
* If an existing session is limited to less channels than
* requested, it should not be reused
*/
- if (ses->chan_max < ctx->max_channels)
+ spin_lock(&ses->chan_lock);
+ if (ses->chan_max < ctx->max_channels) {
+ spin_unlock(&ses->chan_lock);
return 0;
+ }
+ spin_unlock(&ses->chan_lock);
switch (ses->sectype) {
case Kerberos:
void cifs_put_smb_ses(struct cifs_ses *ses)
{
unsigned int rc, xid;
+ unsigned int chan_count;
struct TCP_Server_Info *server = ses->server;
cifs_dbg(FYI, "%s: ses_count=%d\n", __func__, ses->ses_count);
list_del_init(&ses->smb_ses_list);
spin_unlock(&cifs_tcp_ses_lock);
+ spin_lock(&ses->chan_lock);
+ chan_count = ses->chan_count;
+ spin_unlock(&ses->chan_lock);
+
/* close any extra channels */
- if (ses->chan_count > 1) {
+ if (chan_count > 1) {
int i;
- for (i = 1; i < ses->chan_count; i++)
+ for (i = 1; i < chan_count; i++) {
+ /*
+ * note: for now, we're okay accessing ses->chans
+ * without chan_lock. But when chans can go away, we'll
+ * need to introduce ref counting to make sure that chan
+ * is not freed from under us.
+ */
cifs_put_tcp_session(ses->chans[i].server, 0);
+ ses->chans[i].server = NULL;
+ }
}
sesInfoFree(ses);
ses->status);
mutex_lock(&ses->session_mutex);
- rc = cifs_negotiate_protocol(xid, ses);
- if (rc) {
- mutex_unlock(&ses->session_mutex);
- /* problem -- put our ses reference */
- cifs_put_smb_ses(ses);
- free_xid(xid);
- return ERR_PTR(rc);
- }
if (ses->need_reconnect) {
cifs_dbg(FYI, "Session needs reconnect\n");
+
+ rc = cifs_negotiate_protocol(xid, ses);
+ if (rc) {
+ mutex_unlock(&ses->session_mutex);
+ /* problem -- put our ses reference */
+ cifs_put_smb_ses(ses);
+ free_xid(xid);
+ return ERR_PTR(rc);
+ }
+
rc = cifs_setup_session(xid, ses,
ctx->local_nls);
if (rc) {
if (!ses->domainName)
goto get_ses_fail;
}
+ if (ctx->workstation_name) {
+ ses->workstation_name = kstrdup(ctx->workstation_name,
+ GFP_KERNEL);
+ if (!ses->workstation_name)
+ goto get_ses_fail;
+ }
if (ctx->domainauto)
ses->domainAuto = ctx->domainauto;
ses->cred_uid = ctx->cred_uid;
mutex_lock(&ses->session_mutex);
/* add server as first channel */
+ spin_lock(&ses->chan_lock);
ses->chans[0].server = server;
ses->chan_count = 1;
ses->chan_max = ctx->multichannel ? ctx->max_channels:1;
+ spin_unlock(&ses->chan_lock);
rc = cifs_negotiate_protocol(xid, ses);
if (!rc)
list_add(&tcon->tcon_list, &ses->tcon_list);
spin_unlock(&cifs_tcp_ses_lock);
- cifs_fscache_get_super_cookie(tcon);
-
return tcon;
out_fail:
rc = 0;
if (rc < 0) {
cifs_dbg(FYI, "Error %d connecting to server\n", rc);
+ trace_smb3_connect_err(server->hostname, server->conn_id, &server->dstaddr, rc);
sock_release(socket);
server->ssocket = NULL;
return rc;
}
-
+ trace_smb3_connect_done(server->hostname, server->conn_id, &server->dstaddr);
if (sport == htons(RFC1001_PORT))
rc = ip_rfc1001_connect(server);
}
/* Release all succeed connections */
-static inline void mount_put_conns(struct cifs_sb_info *cifs_sb,
- unsigned int xid,
- struct TCP_Server_Info *server,
- struct cifs_ses *ses, struct cifs_tcon *tcon)
+static inline void mount_put_conns(struct mount_ctx *mnt_ctx)
{
int rc = 0;
- if (tcon)
- cifs_put_tcon(tcon);
- else if (ses)
- cifs_put_smb_ses(ses);
- else if (server)
- cifs_put_tcp_session(server, 0);
- cifs_sb->mnt_cifs_flags &= ~CIFS_MOUNT_POSIX_PATHS;
- free_xid(xid);
+ if (mnt_ctx->tcon)
+ cifs_put_tcon(mnt_ctx->tcon);
+ else if (mnt_ctx->ses)
+ cifs_put_smb_ses(mnt_ctx->ses);
+ else if (mnt_ctx->server)
+ cifs_put_tcp_session(mnt_ctx->server, 0);
+ mnt_ctx->cifs_sb->mnt_cifs_flags &= ~CIFS_MOUNT_POSIX_PATHS;
+ free_xid(mnt_ctx->xid);
}
/* Get connections for tcp, ses and tcon */
-static int mount_get_conns(struct smb3_fs_context *ctx, struct cifs_sb_info *cifs_sb,
- unsigned int *xid,
- struct TCP_Server_Info **nserver,
- struct cifs_ses **nses, struct cifs_tcon **ntcon)
+static int mount_get_conns(struct mount_ctx *mnt_ctx)
{
int rc = 0;
- struct TCP_Server_Info *server;
- struct cifs_ses *ses;
- struct cifs_tcon *tcon;
-
- *nserver = NULL;
- *nses = NULL;
- *ntcon = NULL;
+ struct TCP_Server_Info *server = NULL;
+ struct cifs_ses *ses = NULL;
+ struct cifs_tcon *tcon = NULL;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ unsigned int xid;
- *xid = get_xid();
+ xid = get_xid();
/* get a reference to a tcp session */
- server = cifs_get_tcp_session(ctx);
+ server = cifs_get_tcp_session(ctx, NULL);
if (IS_ERR(server)) {
rc = PTR_ERR(server);
- return rc;
+ server = NULL;
+ goto out;
}
- *nserver = server;
-
/* get a reference to a SMB session */
ses = cifs_get_smb_ses(server, ctx);
if (IS_ERR(ses)) {
rc = PTR_ERR(ses);
- return rc;
+ ses = NULL;
+ goto out;
}
- *nses = ses;
-
if ((ctx->persistent == true) && (!(ses->server->capabilities &
SMB2_GLOBAL_CAP_PERSISTENT_HANDLES))) {
cifs_server_dbg(VFS, "persistent handles not supported by server\n");
- return -EOPNOTSUPP;
+ rc = -EOPNOTSUPP;
+ goto out;
}
/* search for existing tcon to this server share */
tcon = cifs_get_tcon(ses, ctx);
if (IS_ERR(tcon)) {
rc = PTR_ERR(tcon);
- return rc;
+ tcon = NULL;
+ goto out;
}
- *ntcon = tcon;
-
/* if new SMB3.11 POSIX extensions are supported do not remap / and \ */
if (tcon->posix_extensions)
cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_POSIX_PATHS;
* reset of caps checks mount to see if unix extensions disabled
* for just this mount.
*/
- reset_cifs_unix_caps(*xid, tcon, cifs_sb, ctx);
+ reset_cifs_unix_caps(xid, tcon, cifs_sb, ctx);
if ((tcon->ses->server->tcpStatus == CifsNeedReconnect) &&
(le64_to_cpu(tcon->fsUnixInfo.Capability) &
- CIFS_UNIX_TRANSPORT_ENCRYPTION_MANDATORY_CAP))
- return -EACCES;
+ CIFS_UNIX_TRANSPORT_ENCRYPTION_MANDATORY_CAP)) {
+ rc = -EACCES;
+ goto out;
+ }
} else
tcon->unix_ext = 0; /* server does not support them */
/* do not care if a following call succeed - informational */
if (!tcon->pipe && server->ops->qfs_tcon) {
- server->ops->qfs_tcon(*xid, tcon, cifs_sb);
+ server->ops->qfs_tcon(xid, tcon, cifs_sb);
if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_RO_CACHE) {
if (tcon->fsDevInfo.DeviceCharacteristics &
cpu_to_le32(FILE_READ_ONLY_DEVICE))
cifs_dbg(VFS, "read only mount of RW share\n");
/* no need to log a RW mount of a typical RW share */
}
+ /*
+ * The cookie is initialized from volume info returned above.
+ * Inside cifs_fscache_get_super_cookie it checks
+ * that we do not get super cookie twice.
+ */
+ cifs_fscache_get_super_cookie(tcon);
}
/*
(cifs_sb->ctx->rsize > server->ops->negotiate_rsize(tcon, ctx)))
cifs_sb->ctx->rsize = server->ops->negotiate_rsize(tcon, ctx);
- return 0;
+out:
+ mnt_ctx->server = server;
+ mnt_ctx->ses = ses;
+ mnt_ctx->tcon = tcon;
+ mnt_ctx->xid = xid;
+
+ return rc;
}
static int mount_setup_tlink(struct cifs_sb_info *cifs_sb, struct cifs_ses *ses,
}
#ifdef CONFIG_CIFS_DFS_UPCALL
-static int mount_get_dfs_conns(struct smb3_fs_context *ctx, struct cifs_sb_info *cifs_sb,
- unsigned int *xid, struct TCP_Server_Info **nserver,
- struct cifs_ses **nses, struct cifs_tcon **ntcon)
+/* Get unique dfs connections */
+static int mount_get_dfs_conns(struct mount_ctx *mnt_ctx)
{
int rc;
- ctx->nosharesock = true;
- rc = mount_get_conns(ctx, cifs_sb, xid, nserver, nses, ntcon);
- if (*nserver) {
+ mnt_ctx->fs_ctx->nosharesock = true;
+ rc = mount_get_conns(mnt_ctx);
+ if (mnt_ctx->server) {
cifs_dbg(FYI, "%s: marking tcp session as a dfs connection\n", __func__);
spin_lock(&cifs_tcp_ses_lock);
- (*nserver)->is_dfs_conn = true;
+ mnt_ctx->server->is_dfs_conn = true;
spin_unlock(&cifs_tcp_ses_lock);
}
return rc;
}
/*
- * expand_dfs_referral - Perform a dfs referral query and update the cifs_sb
- *
- * If a referral is found, cifs_sb->ctx->mount_options will be (re-)allocated
- * to a string containing updated options for the submount. Otherwise it
- * will be left untouched.
+ * expand_dfs_referral - Update cifs_sb from dfs referral path
*
- * Returns the rc from get_dfs_path to the caller, which can be used to
- * determine whether there were referrals.
+ * cifs_sb->ctx->mount_options will be (re-)allocated to a string containing updated options for the
+ * submount. Otherwise it will be left untouched.
*/
-static int
-expand_dfs_referral(const unsigned int xid, struct cifs_ses *ses,
- struct smb3_fs_context *ctx, struct cifs_sb_info *cifs_sb,
- char *ref_path)
-{
- int rc;
- struct dfs_info3_param referral = {0};
- char *full_path = NULL, *mdata = NULL;
-
- if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_DFS)
- return -EREMOTE;
-
- full_path = build_unc_path_to_root(ctx, cifs_sb, true);
- if (IS_ERR(full_path))
- return PTR_ERR(full_path);
-
- rc = dfs_cache_find(xid, ses, cifs_sb->local_nls, cifs_remap(cifs_sb),
- ref_path, &referral, NULL);
- if (!rc) {
- char *fake_devname = NULL;
-
- mdata = cifs_compose_mount_options(cifs_sb->ctx->mount_options,
- full_path + 1, &referral,
- &fake_devname);
- free_dfs_info_param(&referral);
-
- if (IS_ERR(mdata)) {
- rc = PTR_ERR(mdata);
- mdata = NULL;
- } else {
- /*
- * We can not clear out the whole structure since we
- * no longer have an explicit function to parse
- * a mount-string. Instead we need to clear out the
- * individual fields that are no longer valid.
- */
- kfree(ctx->prepath);
- ctx->prepath = NULL;
- rc = cifs_setup_volume_info(ctx, mdata, fake_devname);
- }
- kfree(fake_devname);
- kfree(cifs_sb->ctx->mount_options);
- cifs_sb->ctx->mount_options = mdata;
- }
- kfree(full_path);
- return rc;
-}
-
-static int get_next_dfs_tgt(struct dfs_cache_tgt_list *tgt_list,
- struct dfs_cache_tgt_iterator **tgt_it)
-{
- if (!*tgt_it)
- *tgt_it = dfs_cache_get_tgt_iterator(tgt_list);
- else
- *tgt_it = dfs_cache_get_next_tgt(tgt_list, *tgt_it);
- return !*tgt_it ? -EHOSTDOWN : 0;
-}
-
-static int update_vol_info(const struct dfs_cache_tgt_iterator *tgt_it,
- struct smb3_fs_context *fake_ctx, struct smb3_fs_context *ctx)
-{
- const char *tgt = dfs_cache_get_tgt_name(tgt_it);
- int len = strlen(tgt) + 2;
- char *new_unc;
-
- new_unc = kmalloc(len, GFP_KERNEL);
- if (!new_unc)
- return -ENOMEM;
- scnprintf(new_unc, len, "\\%s", tgt);
-
- kfree(ctx->UNC);
- ctx->UNC = new_unc;
-
- if (fake_ctx->prepath) {
- kfree(ctx->prepath);
- ctx->prepath = fake_ctx->prepath;
- fake_ctx->prepath = NULL;
- }
- memcpy(&ctx->dstaddr, &fake_ctx->dstaddr, sizeof(ctx->dstaddr));
-
- return 0;
-}
-
-static int do_dfs_failover(const char *path, const char *full_path, struct cifs_sb_info *cifs_sb,
- struct smb3_fs_context *ctx, struct cifs_ses *root_ses,
- unsigned int *xid, struct TCP_Server_Info **server,
- struct cifs_ses **ses, struct cifs_tcon **tcon)
+static int expand_dfs_referral(struct mount_ctx *mnt_ctx, const char *full_path,
+ struct dfs_info3_param *referral)
{
int rc;
- char *npath = NULL;
- struct dfs_cache_tgt_list tgt_list = DFS_CACHE_TGT_LIST_INIT(tgt_list);
- struct dfs_cache_tgt_iterator *tgt_it = NULL;
- struct smb3_fs_context tmp_ctx = {NULL};
-
- if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_DFS)
- return -EOPNOTSUPP;
-
- npath = dfs_cache_canonical_path(path, cifs_sb->local_nls, cifs_remap(cifs_sb));
- if (IS_ERR(npath))
- return PTR_ERR(npath);
-
- cifs_dbg(FYI, "%s: path=%s full_path=%s\n", __func__, npath, full_path);
-
- rc = dfs_cache_noreq_find(npath, NULL, &tgt_list);
- if (rc)
- goto out;
- /*
- * We use a 'tmp_ctx' here because we need pass it down to the mount_{get,put} functions to
- * test connection against new DFS targets.
- */
- rc = smb3_fs_context_dup(&tmp_ctx, ctx);
- if (rc)
- goto out;
-
- for (;;) {
- struct dfs_info3_param ref = {0};
- char *fake_devname = NULL, *mdata = NULL;
-
- /* Get next DFS target server - if any */
- rc = get_next_dfs_tgt(&tgt_list, &tgt_it);
- if (rc)
- break;
-
- rc = dfs_cache_get_tgt_referral(npath, tgt_it, &ref);
- if (rc)
- break;
-
- cifs_dbg(FYI, "%s: old ctx: UNC=%s prepath=%s\n", __func__, tmp_ctx.UNC,
- tmp_ctx.prepath);
-
- mdata = cifs_compose_mount_options(cifs_sb->ctx->mount_options, full_path + 1, &ref,
- &fake_devname);
- free_dfs_info_param(&ref);
-
- if (IS_ERR(mdata)) {
- rc = PTR_ERR(mdata);
- mdata = NULL;
- } else
- rc = cifs_setup_volume_info(&tmp_ctx, mdata, fake_devname);
-
- kfree(mdata);
- kfree(fake_devname);
-
- if (rc)
- break;
-
- cifs_dbg(FYI, "%s: new ctx: UNC=%s prepath=%s\n", __func__, tmp_ctx.UNC,
- tmp_ctx.prepath);
-
- mount_put_conns(cifs_sb, *xid, *server, *ses, *tcon);
- rc = mount_get_dfs_conns(&tmp_ctx, cifs_sb, xid, server, ses, tcon);
- if (!rc || (*server && *ses)) {
- /*
- * We were able to connect to new target server. Update current context with
- * new target server.
- */
- rc = update_vol_info(tgt_it, &tmp_ctx, ctx);
- break;
- }
- }
- if (!rc) {
- cifs_dbg(FYI, "%s: final ctx: UNC=%s prepath=%s\n", __func__, tmp_ctx.UNC,
- tmp_ctx.prepath);
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
+ char *fake_devname = NULL, *mdata = NULL;
+
+ mdata = cifs_compose_mount_options(cifs_sb->ctx->mount_options, full_path + 1, referral,
+ &fake_devname);
+ if (IS_ERR(mdata)) {
+ rc = PTR_ERR(mdata);
+ mdata = NULL;
+ } else {
/*
- * Update DFS target hint in DFS referral cache with the target server we
- * successfully reconnected to.
+ * We can not clear out the whole structure since we no longer have an explicit
+ * function to parse a mount-string. Instead we need to clear out the individual
+ * fields that are no longer valid.
*/
- rc = dfs_cache_update_tgthint(*xid, root_ses ? root_ses : *ses, cifs_sb->local_nls,
- cifs_remap(cifs_sb), path, tgt_it);
+ kfree(ctx->prepath);
+ ctx->prepath = NULL;
+ rc = cifs_setup_volume_info(ctx, mdata, fake_devname);
}
+ kfree(fake_devname);
+ kfree(cifs_sb->ctx->mount_options);
+ cifs_sb->ctx->mount_options = mdata;
-out:
- kfree(npath);
- smb3_cleanup_fs_context_contents(&tmp_ctx);
- dfs_cache_free_tgts(&tgt_list);
return rc;
}
#endif
* Check if path is remote (e.g. a DFS share). Return -EREMOTE if it is,
* otherwise 0.
*/
-static int is_path_remote(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx,
- const unsigned int xid,
- struct TCP_Server_Info *server,
- struct cifs_tcon *tcon)
+static int is_path_remote(struct mount_ctx *mnt_ctx)
{
int rc;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct TCP_Server_Info *server = mnt_ctx->server;
+ unsigned int xid = mnt_ctx->xid;
+ struct cifs_tcon *tcon = mnt_ctx->tcon;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
char *full_path;
if (!server->ops->is_path_accessible)
}
#ifdef CONFIG_CIFS_DFS_UPCALL
-static void set_root_ses(struct cifs_sb_info *cifs_sb, const uuid_t *mount_id, struct cifs_ses *ses,
- struct cifs_ses **root_ses)
+static void set_root_ses(struct mount_ctx *mnt_ctx)
{
- if (ses) {
+ if (mnt_ctx->ses) {
spin_lock(&cifs_tcp_ses_lock);
- ses->ses_count++;
+ mnt_ctx->ses->ses_count++;
spin_unlock(&cifs_tcp_ses_lock);
- dfs_cache_add_refsrv_session(mount_id, ses);
+ dfs_cache_add_refsrv_session(&mnt_ctx->mount_id, mnt_ctx->ses);
}
- *root_ses = ses;
+ mnt_ctx->root_ses = mnt_ctx->ses;
}
-/* Set up next dfs prefix path in @dfs_path */
-static int next_dfs_prepath(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx,
- const unsigned int xid, struct TCP_Server_Info *server,
- struct cifs_tcon *tcon, char **dfs_path)
+static int is_dfs_mount(struct mount_ctx *mnt_ctx, bool *isdfs, struct dfs_cache_tgt_list *root_tl)
{
- char *path, *npath;
- int added_treename = is_tcon_dfs(tcon);
int rc;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
- path = cifs_build_path_to_root(ctx, cifs_sb, tcon, added_treename);
- if (!path)
- return -ENOMEM;
+ *isdfs = true;
- rc = is_path_remote(cifs_sb, ctx, xid, server, tcon);
- if (rc == -EREMOTE) {
- struct smb3_fs_context v = {NULL};
- /* if @path contains a tree name, skip it in the prefix path */
- if (added_treename) {
- rc = smb3_parse_devname(path, &v);
- if (rc)
- goto out;
- npath = build_unc_path_to_root(&v, cifs_sb, true);
- smb3_cleanup_fs_context_contents(&v);
- } else {
- v.UNC = ctx->UNC;
- v.prepath = path + 1;
- npath = build_unc_path_to_root(&v, cifs_sb, true);
- }
+ rc = mount_get_conns(mnt_ctx);
+ /*
+ * If called with 'nodfs' mount option, then skip DFS resolving. Otherwise unconditionally
+ * try to get an DFS referral (even cached) to determine whether it is an DFS mount.
+ *
+ * Skip prefix path to provide support for DFS referrals from w2k8 servers which don't seem
+ * to respond with PATH_NOT_COVERED to requests that include the prefix.
+ */
+ if ((cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_DFS) ||
+ dfs_cache_find(mnt_ctx->xid, mnt_ctx->ses, cifs_sb->local_nls, cifs_remap(cifs_sb),
+ ctx->UNC + 1, NULL, root_tl)) {
+ if (rc)
+ return rc;
+ /* Check if it is fully accessible and then mount it */
+ rc = is_path_remote(mnt_ctx);
+ if (!rc)
+ *isdfs = false;
+ else if (rc != -EREMOTE)
+ return rc;
+ }
+ return 0;
+}
- if (IS_ERR(npath)) {
- rc = PTR_ERR(npath);
- goto out;
- }
+static int connect_dfs_target(struct mount_ctx *mnt_ctx, const char *full_path,
+ const char *ref_path, struct dfs_cache_tgt_iterator *tit)
+{
+ int rc;
+ struct dfs_info3_param ref = {};
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ char *oldmnt = cifs_sb->ctx->mount_options;
- kfree(*dfs_path);
- *dfs_path = npath;
- rc = -EREMOTE;
+ rc = dfs_cache_get_tgt_referral(ref_path, tit, &ref);
+ if (rc)
+ goto out;
+
+ rc = expand_dfs_referral(mnt_ctx, full_path, &ref);
+ if (rc)
+ goto out;
+
+ /* Connect to new target only if we were redirected (e.g. mount options changed) */
+ if (oldmnt != cifs_sb->ctx->mount_options) {
+ mount_put_conns(mnt_ctx);
+ rc = mount_get_dfs_conns(mnt_ctx);
+ }
+ if (!rc) {
+ if (cifs_is_referral_server(mnt_ctx->tcon, &ref))
+ set_root_ses(mnt_ctx);
+ rc = dfs_cache_update_tgthint(mnt_ctx->xid, mnt_ctx->root_ses, cifs_sb->local_nls,
+ cifs_remap(cifs_sb), ref_path, tit);
}
out:
- kfree(path);
+ free_dfs_info_param(&ref);
return rc;
}
-/* Check if resolved targets can handle any DFS referrals */
-static int is_referral_server(const char *ref_path, struct cifs_sb_info *cifs_sb,
- struct cifs_tcon *tcon, bool *ref_server)
+static int connect_dfs_root(struct mount_ctx *mnt_ctx, struct dfs_cache_tgt_list *root_tl)
{
int rc;
- struct dfs_info3_param ref = {0};
+ char *full_path;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
+ struct dfs_cache_tgt_iterator *tit;
- cifs_dbg(FYI, "%s: ref_path=%s\n", __func__, ref_path);
+ /* Put initial connections as they might be shared with other mounts. We need unique dfs
+ * connections per mount to properly failover, so mount_get_dfs_conns() must be used from
+ * now on.
+ */
+ mount_put_conns(mnt_ctx);
+ mount_get_dfs_conns(mnt_ctx);
- if (is_tcon_dfs(tcon)) {
- *ref_server = true;
- } else {
- char *npath;
+ full_path = build_unc_path_to_root(ctx, cifs_sb, true);
+ if (IS_ERR(full_path))
+ return PTR_ERR(full_path);
- npath = dfs_cache_canonical_path(ref_path, cifs_sb->local_nls, cifs_remap(cifs_sb));
- if (IS_ERR(npath))
- return PTR_ERR(npath);
+ mnt_ctx->origin_fullpath = dfs_cache_canonical_path(ctx->UNC, cifs_sb->local_nls,
+ cifs_remap(cifs_sb));
+ if (IS_ERR(mnt_ctx->origin_fullpath)) {
+ rc = PTR_ERR(mnt_ctx->origin_fullpath);
+ mnt_ctx->origin_fullpath = NULL;
+ goto out;
+ }
- rc = dfs_cache_noreq_find(npath, &ref, NULL);
- kfree(npath);
- if (rc) {
- cifs_dbg(VFS, "%s: dfs_cache_noreq_find: failed (rc=%d)\n", __func__, rc);
- return rc;
+ /* Try all dfs root targets */
+ for (rc = -ENOENT, tit = dfs_cache_get_tgt_iterator(root_tl);
+ tit; tit = dfs_cache_get_next_tgt(root_tl, tit)) {
+ rc = connect_dfs_target(mnt_ctx, full_path, mnt_ctx->origin_fullpath + 1, tit);
+ if (!rc) {
+ mnt_ctx->leaf_fullpath = kstrdup(mnt_ctx->origin_fullpath, GFP_KERNEL);
+ if (!mnt_ctx->leaf_fullpath)
+ rc = -ENOMEM;
+ break;
}
- cifs_dbg(FYI, "%s: ref.flags=0x%x\n", __func__, ref.flags);
- /*
- * Check if all targets are capable of handling DFS referrals as per
- * MS-DFSC 2.2.4 RESP_GET_DFS_REFERRAL.
- */
- *ref_server = !!(ref.flags & DFSREF_REFERRAL_SERVER);
- free_dfs_info_param(&ref);
}
- return 0;
+
+out:
+ kfree(full_path);
+ return rc;
}
-int cifs_mount(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx)
+static int __follow_dfs_link(struct mount_ctx *mnt_ctx)
{
- int rc = 0;
- unsigned int xid;
- struct TCP_Server_Info *server = NULL;
- struct cifs_ses *ses = NULL, *root_ses = NULL;
- struct cifs_tcon *tcon = NULL;
- int count = 0;
- uuid_t mount_id = {0};
- char *ref_path = NULL, *full_path = NULL;
- char *oldmnt = NULL;
- bool ref_server = false;
+ int rc;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
+ char *full_path;
+ struct dfs_cache_tgt_list tl = DFS_CACHE_TGT_LIST_INIT(tl);
+ struct dfs_cache_tgt_iterator *tit;
- rc = mount_get_conns(ctx, cifs_sb, &xid, &server, &ses, &tcon);
- /*
- * If called with 'nodfs' mount option, then skip DFS resolving. Otherwise unconditionally
- * try to get an DFS referral (even cached) to determine whether it is an DFS mount.
- *
- * Skip prefix path to provide support for DFS referrals from w2k8 servers which don't seem
- * to respond with PATH_NOT_COVERED to requests that include the prefix.
- */
- if ((cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NO_DFS) ||
- dfs_cache_find(xid, ses, cifs_sb->local_nls, cifs_remap(cifs_sb), ctx->UNC + 1, NULL,
- NULL)) {
- if (rc)
- goto error;
- /* Check if it is fully accessible and then mount it */
- rc = is_path_remote(cifs_sb, ctx, xid, server, tcon);
- if (!rc)
- goto out;
- if (rc != -EREMOTE)
- goto error;
+ full_path = build_unc_path_to_root(ctx, cifs_sb, true);
+ if (IS_ERR(full_path))
+ return PTR_ERR(full_path);
+
+ kfree(mnt_ctx->leaf_fullpath);
+ mnt_ctx->leaf_fullpath = dfs_cache_canonical_path(full_path, cifs_sb->local_nls,
+ cifs_remap(cifs_sb));
+ if (IS_ERR(mnt_ctx->leaf_fullpath)) {
+ rc = PTR_ERR(mnt_ctx->leaf_fullpath);
+ mnt_ctx->leaf_fullpath = NULL;
+ goto out;
}
- mount_put_conns(cifs_sb, xid, server, ses, tcon);
- /*
- * Ignore error check here because we may failover to other targets from cached a
- * referral.
- */
- (void)mount_get_dfs_conns(ctx, cifs_sb, &xid, &server, &ses, &tcon);
+ /* Get referral from dfs link */
+ rc = dfs_cache_find(mnt_ctx->xid, mnt_ctx->root_ses, cifs_sb->local_nls,
+ cifs_remap(cifs_sb), mnt_ctx->leaf_fullpath + 1, NULL, &tl);
+ if (rc)
+ goto out;
- /* Get path of DFS root */
- ref_path = build_unc_path_to_root(ctx, cifs_sb, false);
- if (IS_ERR(ref_path)) {
- rc = PTR_ERR(ref_path);
- ref_path = NULL;
- goto error;
+ /* Try all dfs link targets */
+ for (rc = -ENOENT, tit = dfs_cache_get_tgt_iterator(&tl);
+ tit; tit = dfs_cache_get_next_tgt(&tl, tit)) {
+ rc = connect_dfs_target(mnt_ctx, full_path, mnt_ctx->leaf_fullpath + 1, tit);
+ if (!rc) {
+ rc = is_path_remote(mnt_ctx);
+ break;
+ }
+ }
+
+out:
+ kfree(full_path);
+ dfs_cache_free_tgts(&tl);
+ return rc;
+}
+
+static int follow_dfs_link(struct mount_ctx *mnt_ctx)
+{
+ int rc;
+ struct cifs_sb_info *cifs_sb = mnt_ctx->cifs_sb;
+ struct smb3_fs_context *ctx = mnt_ctx->fs_ctx;
+ char *full_path;
+ int num_links = 0;
+
+ full_path = build_unc_path_to_root(ctx, cifs_sb, true);
+ if (IS_ERR(full_path))
+ return PTR_ERR(full_path);
+
+ kfree(mnt_ctx->origin_fullpath);
+ mnt_ctx->origin_fullpath = dfs_cache_canonical_path(full_path, cifs_sb->local_nls,
+ cifs_remap(cifs_sb));
+ kfree(full_path);
+
+ if (IS_ERR(mnt_ctx->origin_fullpath)) {
+ rc = PTR_ERR(mnt_ctx->origin_fullpath);
+ mnt_ctx->origin_fullpath = NULL;
+ return rc;
}
- uuid_gen(&mount_id);
- set_root_ses(cifs_sb, &mount_id, ses, &root_ses);
do {
- /* Save full path of last DFS path we used to resolve final target server */
- kfree(full_path);
- full_path = build_unc_path_to_root(ctx, cifs_sb, !!count);
- if (IS_ERR(full_path)) {
- rc = PTR_ERR(full_path);
- full_path = NULL;
+ rc = __follow_dfs_link(mnt_ctx);
+ if (!rc || rc != -EREMOTE)
break;
- }
- /* Chase referral */
- oldmnt = cifs_sb->ctx->mount_options;
- rc = expand_dfs_referral(xid, root_ses, ctx, cifs_sb, ref_path + 1);
- if (rc)
- break;
- /* Connect to new DFS target only if we were redirected */
- if (oldmnt != cifs_sb->ctx->mount_options) {
- mount_put_conns(cifs_sb, xid, server, ses, tcon);
- rc = mount_get_dfs_conns(ctx, cifs_sb, &xid, &server, &ses, &tcon);
- }
- if (rc && !server && !ses) {
- /* Failed to connect. Try to connect to other targets in the referral. */
- rc = do_dfs_failover(ref_path + 1, full_path, cifs_sb, ctx, root_ses, &xid,
- &server, &ses, &tcon);
- }
- if (rc == -EACCES || rc == -EOPNOTSUPP || !server || !ses)
- break;
- if (!tcon)
- continue;
+ } while (rc = -ELOOP, ++num_links < MAX_NESTED_LINKS);
- /* Make sure that requests go through new root servers */
- rc = is_referral_server(ref_path + 1, cifs_sb, tcon, &ref_server);
- if (rc)
- break;
- if (ref_server)
- set_root_ses(cifs_sb, &mount_id, ses, &root_ses);
+ return rc;
+}
- /* Get next dfs path and then continue chasing them if -EREMOTE */
- rc = next_dfs_prepath(cifs_sb, ctx, xid, server, tcon, &ref_path);
- /* Prevent recursion on broken link referrals */
- if (rc == -EREMOTE && ++count > MAX_NESTED_LINKS)
- rc = -ELOOP;
- } while (rc == -EREMOTE);
+/* Set up DFS referral paths for failover */
+static void setup_server_referral_paths(struct mount_ctx *mnt_ctx)
+{
+ struct TCP_Server_Info *server = mnt_ctx->server;
+
+ server->origin_fullpath = mnt_ctx->origin_fullpath;
+ server->leaf_fullpath = mnt_ctx->leaf_fullpath;
+ server->current_fullpath = mnt_ctx->leaf_fullpath;
+ mnt_ctx->origin_fullpath = mnt_ctx->leaf_fullpath = NULL;
+}
- if (rc || !tcon || !ses)
+int cifs_mount(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx)
+{
+ int rc;
+ struct mount_ctx mnt_ctx = { .cifs_sb = cifs_sb, .fs_ctx = ctx, };
+ struct dfs_cache_tgt_list tl = DFS_CACHE_TGT_LIST_INIT(tl);
+ bool isdfs;
+
+ rc = is_dfs_mount(&mnt_ctx, &isdfs, &tl);
+ if (rc)
goto error;
+ if (!isdfs)
+ goto out;
- kfree(ref_path);
- /*
- * Store DFS full path in both superblock and tree connect structures.
- *
- * For DFS root mounts, the prefix path (cifs_sb->prepath) is preserved during reconnect so
- * only the root path is set in cifs_sb->origin_fullpath and tcon->dfs_path. And for DFS
- * links, the prefix path is included in both and may be changed during reconnect. See
- * cifs_tree_connect().
- */
- ref_path = dfs_cache_canonical_path(full_path, cifs_sb->local_nls, cifs_remap(cifs_sb));
- kfree(full_path);
- full_path = NULL;
+ uuid_gen(&mnt_ctx.mount_id);
+ rc = connect_dfs_root(&mnt_ctx, &tl);
+ dfs_cache_free_tgts(&tl);
- if (IS_ERR(ref_path)) {
- rc = PTR_ERR(ref_path);
- ref_path = NULL;
+ if (rc)
goto error;
- }
- cifs_sb->origin_fullpath = ref_path;
- ref_path = kstrdup(cifs_sb->origin_fullpath, GFP_KERNEL);
- if (!ref_path) {
- rc = -ENOMEM;
+ rc = is_path_remote(&mnt_ctx);
+ if (rc == -EREMOTE)
+ rc = follow_dfs_link(&mnt_ctx);
+ if (rc)
goto error;
- }
- spin_lock(&cifs_tcp_ses_lock);
- tcon->dfs_path = ref_path;
- ref_path = NULL;
- spin_unlock(&cifs_tcp_ses_lock);
+ setup_server_referral_paths(&mnt_ctx);
/*
- * After reconnecting to a different server, unique ids won't
- * match anymore, so we disable serverino. This prevents
- * dentry revalidation to think the dentry are stale (ESTALE).
+ * After reconnecting to a different server, unique ids won't match anymore, so we disable
+ * serverino. This prevents dentry revalidation to think the dentry are stale (ESTALE).
*/
cifs_autodisable_serverino(cifs_sb);
/*
- * Force the use of prefix path to support failover on DFS paths that
- * resolve to targets that have different prefix paths.
+ * Force the use of prefix path to support failover on DFS paths that resolve to targets
+ * that have different prefix paths.
*/
cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_USE_PREFIX_PATH;
kfree(cifs_sb->prepath);
cifs_sb->prepath = ctx->prepath;
ctx->prepath = NULL;
- uuid_copy(&cifs_sb->dfs_mount_id, &mount_id);
+ uuid_copy(&cifs_sb->dfs_mount_id, &mnt_ctx.mount_id);
out:
- free_xid(xid);
- cifs_try_adding_channels(cifs_sb, ses);
- return mount_setup_tlink(cifs_sb, ses, tcon);
+ free_xid(mnt_ctx.xid);
+ cifs_try_adding_channels(cifs_sb, mnt_ctx.ses);
+ return mount_setup_tlink(cifs_sb, mnt_ctx.ses, mnt_ctx.tcon);
error:
- kfree(ref_path);
- kfree(full_path);
- kfree(cifs_sb->origin_fullpath);
- dfs_cache_put_refsrv_sessions(&mount_id);
- mount_put_conns(cifs_sb, xid, server, ses, tcon);
+ dfs_cache_put_refsrv_sessions(&mnt_ctx.mount_id);
+ kfree(mnt_ctx.origin_fullpath);
+ kfree(mnt_ctx.leaf_fullpath);
+ mount_put_conns(&mnt_ctx);
return rc;
}
#else
int cifs_mount(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx)
{
int rc = 0;
- unsigned int xid;
- struct cifs_ses *ses;
- struct cifs_tcon *tcon;
- struct TCP_Server_Info *server;
+ struct mount_ctx mnt_ctx = { .cifs_sb = cifs_sb, .fs_ctx = ctx, };
- rc = mount_get_conns(ctx, cifs_sb, &xid, &server, &ses, &tcon);
+ rc = mount_get_conns(&mnt_ctx);
if (rc)
goto error;
- if (tcon) {
- rc = is_path_remote(cifs_sb, ctx, xid, server, tcon);
+ if (mnt_ctx.tcon) {
+ rc = is_path_remote(&mnt_ctx);
if (rc == -EREMOTE)
rc = -EOPNOTSUPP;
if (rc)
goto error;
}
- free_xid(xid);
-
- return mount_setup_tlink(cifs_sb, ses, tcon);
+ free_xid(mnt_ctx.xid);
+ return mount_setup_tlink(cifs_sb, mnt_ctx.ses, mnt_ctx.tcon);
error:
- mount_put_conns(cifs_sb, xid, server, ses, tcon);
+ mount_put_conns(&mnt_ctx);
return rc;
}
#endif
kfree(cifs_sb->prepath);
#ifdef CONFIG_CIFS_DFS_UPCALL
dfs_cache_put_refsrv_sessions(&cifs_sb->dfs_mount_id);
- kfree(cifs_sb->origin_fullpath);
#endif
call_rcu(&cifs_sb->rcu, delayed_free);
}
}
#ifdef CONFIG_CIFS_DFS_UPCALL
-int cifs_tree_connect(const unsigned int xid, struct cifs_tcon *tcon, const struct nls_table *nlsc)
+/* Update dfs referral path of superblock */
+static int update_server_fullpath(struct TCP_Server_Info *server, struct cifs_sb_info *cifs_sb,
+ const char *target)
+{
+ int rc = 0;
+ size_t len = strlen(target);
+ char *refpath, *npath;
+
+ if (unlikely(len < 2 || *target != '\\'))
+ return -EINVAL;
+
+ if (target[1] == '\\') {
+ len += 1;
+ refpath = kmalloc(len, GFP_KERNEL);
+ if (!refpath)
+ return -ENOMEM;
+
+ scnprintf(refpath, len, "%s", target);
+ } else {
+ len += sizeof("\\");
+ refpath = kmalloc(len, GFP_KERNEL);
+ if (!refpath)
+ return -ENOMEM;
+
+ scnprintf(refpath, len, "\\%s", target);
+ }
+
+ npath = dfs_cache_canonical_path(refpath, cifs_sb->local_nls, cifs_remap(cifs_sb));
+ kfree(refpath);
+
+ if (IS_ERR(npath)) {
+ rc = PTR_ERR(npath);
+ } else {
+ mutex_lock(&server->refpath_lock);
+ kfree(server->leaf_fullpath);
+ server->leaf_fullpath = npath;
+ mutex_unlock(&server->refpath_lock);
+ server->current_fullpath = server->leaf_fullpath;
+ }
+ return rc;
+}
+
+static int target_share_matches_server(struct TCP_Server_Info *server, const char *tcp_host,
+ size_t tcp_host_len, char *share, bool *target_match)
+{
+ int rc = 0;
+ const char *dfs_host;
+ size_t dfs_host_len;
+
+ *target_match = true;
+ extract_unc_hostname(share, &dfs_host, &dfs_host_len);
+
+ /* Check if hostnames or addresses match */
+ if (dfs_host_len != tcp_host_len || strncasecmp(dfs_host, tcp_host, dfs_host_len) != 0) {
+ cifs_dbg(FYI, "%s: %.*s doesn't match %.*s\n", __func__, (int)dfs_host_len,
+ dfs_host, (int)tcp_host_len, tcp_host);
+ rc = match_target_ip(server, dfs_host, dfs_host_len, target_match);
+ if (rc)
+ cifs_dbg(VFS, "%s: failed to match target ip: %d\n", __func__, rc);
+ }
+ return rc;
+}
+
+static int __tree_connect_dfs_target(const unsigned int xid, struct cifs_tcon *tcon,
+ struct cifs_sb_info *cifs_sb, char *tree, bool islink,
+ struct dfs_cache_tgt_list *tl)
{
int rc;
struct TCP_Server_Info *server = tcon->ses->server;
const struct smb_version_operations *ops = server->ops;
- struct dfs_cache_tgt_list tl;
- struct dfs_cache_tgt_iterator *it = NULL;
- char *tree;
+ struct cifs_tcon *ipc = tcon->ses->tcon_ipc;
+ char *share = NULL, *prefix = NULL;
const char *tcp_host;
size_t tcp_host_len;
- const char *dfs_host;
- size_t dfs_host_len;
- char *share = NULL, *prefix = NULL;
- struct dfs_info3_param ref = {0};
- bool isroot;
+ struct dfs_cache_tgt_iterator *tit;
+ bool target_match;
- tree = kzalloc(MAX_TREE_SIZE, GFP_KERNEL);
- if (!tree)
- return -ENOMEM;
+ extract_unc_hostname(server->hostname, &tcp_host, &tcp_host_len);
- /* If it is not dfs or there was no cached dfs referral, then reconnect to same share */
- if (!tcon->dfs_path || dfs_cache_noreq_find(tcon->dfs_path + 1, &ref, &tl)) {
- if (tcon->ipc) {
- scnprintf(tree, MAX_TREE_SIZE, "\\\\%s\\IPC$", server->hostname);
- rc = ops->tree_connect(xid, tcon->ses, tree, tcon, nlsc);
- } else {
- rc = ops->tree_connect(xid, tcon->ses, tcon->treeName, tcon, nlsc);
- }
+ tit = dfs_cache_get_tgt_iterator(tl);
+ if (!tit) {
+ rc = -ENOENT;
goto out;
}
- isroot = ref.server_type == DFS_TYPE_ROOT;
- free_dfs_info_param(&ref);
-
- extract_unc_hostname(server->hostname, &tcp_host, &tcp_host_len);
-
- for (it = dfs_cache_get_tgt_iterator(&tl); it; it = dfs_cache_get_next_tgt(&tl, it)) {
- bool target_match;
+ /* Try to tree connect to all dfs targets */
+ for (; tit; tit = dfs_cache_get_next_tgt(tl, tit)) {
+ const char *target = dfs_cache_get_tgt_name(tit);
+ struct dfs_cache_tgt_list ntl = DFS_CACHE_TGT_LIST_INIT(ntl);
kfree(share);
kfree(prefix);
- share = NULL;
- prefix = NULL;
+ share = prefix = NULL;
- rc = dfs_cache_get_tgt_share(tcon->dfs_path + 1, it, &share, &prefix);
+ /* Check if share matches with tcp ses */
+ rc = dfs_cache_get_tgt_share(server->current_fullpath + 1, tit, &share, &prefix);
if (rc) {
- cifs_dbg(VFS, "%s: failed to parse target share %d\n",
- __func__, rc);
- continue;
+ cifs_dbg(VFS, "%s: failed to parse target share: %d\n", __func__, rc);
+ break;
}
- extract_unc_hostname(share, &dfs_host, &dfs_host_len);
-
- if (dfs_host_len != tcp_host_len
- || strncasecmp(dfs_host, tcp_host, dfs_host_len) != 0) {
- cifs_dbg(FYI, "%s: %.*s doesn't match %.*s\n", __func__, (int)dfs_host_len,
- dfs_host, (int)tcp_host_len, tcp_host);
+ rc = target_share_matches_server(server, tcp_host, tcp_host_len, share,
+ &target_match);
+ if (rc)
+ break;
+ if (!target_match) {
+ rc = -EHOSTUNREACH;
+ continue;
+ }
- rc = match_target_ip(server, dfs_host, dfs_host_len, &target_match);
- if (rc) {
- cifs_dbg(VFS, "%s: failed to match target ip: %d\n", __func__, rc);
+ if (ipc->need_reconnect) {
+ scnprintf(tree, MAX_TREE_SIZE, "\\\\%s\\IPC$", server->hostname);
+ rc = ops->tree_connect(xid, ipc->ses, tree, ipc, cifs_sb->local_nls);
+ if (rc)
break;
- }
-
- if (!target_match) {
- cifs_dbg(FYI, "%s: skipping target\n", __func__);
- continue;
- }
}
- if (tcon->ipc) {
- scnprintf(tree, MAX_TREE_SIZE, "\\\\%s\\IPC$", share);
- rc = ops->tree_connect(xid, tcon->ses, tree, tcon, nlsc);
+ scnprintf(tree, MAX_TREE_SIZE, "\\%s", share);
+ if (!islink) {
+ rc = ops->tree_connect(xid, tcon->ses, tree, tcon, cifs_sb->local_nls);
+ break;
+ }
+ /*
+ * If no dfs referrals were returned from link target, then just do a TREE_CONNECT
+ * to it. Otherwise, cache the dfs referral and then mark current tcp ses for
+ * reconnect so either the demultiplex thread or the echo worker will reconnect to
+ * newly resolved target.
+ */
+ if (dfs_cache_find(xid, tcon->ses, cifs_sb->local_nls, cifs_remap(cifs_sb), target,
+ NULL, &ntl)) {
+ rc = ops->tree_connect(xid, tcon->ses, tree, tcon, cifs_sb->local_nls);
+ if (rc)
+ continue;
+ rc = dfs_cache_noreq_update_tgthint(server->current_fullpath + 1, tit);
+ if (!rc)
+ rc = cifs_update_super_prepath(cifs_sb, prefix);
} else {
- scnprintf(tree, MAX_TREE_SIZE, "\\%s", share);
- rc = ops->tree_connect(xid, tcon->ses, tree, tcon, nlsc);
- /* Only handle prefix paths of DFS link targets */
- if (!rc && !isroot) {
- rc = update_super_prepath(tcon, prefix);
- break;
- }
+ /* Target is another dfs share */
+ rc = update_server_fullpath(server, cifs_sb, target);
+ dfs_cache_free_tgts(tl);
+
+ if (!rc) {
+ rc = -EREMOTE;
+ list_replace_init(&ntl.tl_list, &tl->tl_list);
+ } else
+ dfs_cache_free_tgts(&ntl);
}
- if (rc == -EREMOTE)
- break;
+ break;
}
+out:
kfree(share);
kfree(prefix);
- if (!rc) {
- if (it)
- rc = dfs_cache_noreq_update_tgthint(tcon->dfs_path + 1, it);
- else
- rc = -ENOENT;
+ return rc;
+}
+
+static int tree_connect_dfs_target(const unsigned int xid, struct cifs_tcon *tcon,
+ struct cifs_sb_info *cifs_sb, char *tree, bool islink,
+ struct dfs_cache_tgt_list *tl)
+{
+ int rc;
+ int num_links = 0;
+ struct TCP_Server_Info *server = tcon->ses->server;
+
+ do {
+ rc = __tree_connect_dfs_target(xid, tcon, cifs_sb, tree, islink, tl);
+ if (!rc || rc != -EREMOTE)
+ break;
+ } while (rc = -ELOOP, ++num_links < MAX_NESTED_LINKS);
+ /*
+ * If we couldn't tree connect to any targets from last referral path, then retry from
+ * original referral path.
+ */
+ if (rc && server->current_fullpath != server->origin_fullpath) {
+ server->current_fullpath = server->origin_fullpath;
+ cifs_ses_mark_for_reconnect(tcon->ses);
}
- dfs_cache_free_tgts(&tl);
+
+ dfs_cache_free_tgts(tl);
+ return rc;
+}
+
+int cifs_tree_connect(const unsigned int xid, struct cifs_tcon *tcon, const struct nls_table *nlsc)
+{
+ int rc;
+ struct TCP_Server_Info *server = tcon->ses->server;
+ const struct smb_version_operations *ops = server->ops;
+ struct super_block *sb = NULL;
+ struct cifs_sb_info *cifs_sb;
+ struct dfs_cache_tgt_list tl = DFS_CACHE_TGT_LIST_INIT(tl);
+ char *tree;
+ struct dfs_info3_param ref = {0};
+
+ tree = kzalloc(MAX_TREE_SIZE, GFP_KERNEL);
+ if (!tree)
+ return -ENOMEM;
+
+ if (tcon->ipc) {
+ scnprintf(tree, MAX_TREE_SIZE, "\\\\%s\\IPC$", server->hostname);
+ rc = ops->tree_connect(xid, tcon->ses, tree, tcon, nlsc);
+ goto out;
+ }
+
+ sb = cifs_get_tcp_super(server);
+ if (IS_ERR(sb)) {
+ rc = PTR_ERR(sb);
+ cifs_dbg(VFS, "%s: could not find superblock: %d\n", __func__, rc);
+ goto out;
+ }
+
+ cifs_sb = CIFS_SB(sb);
+
+ /* If it is not dfs or there was no cached dfs referral, then reconnect to same share */
+ if (!server->current_fullpath ||
+ dfs_cache_noreq_find(server->current_fullpath + 1, &ref, &tl)) {
+ rc = ops->tree_connect(xid, tcon->ses, tcon->treeName, tcon, cifs_sb->local_nls);
+ goto out;
+ }
+
+ rc = tree_connect_dfs_target(xid, tcon, cifs_sb, tree, ref.server_type == DFS_TYPE_LINK,
+ &tl);
+ free_dfs_info_param(&ref);
+
out:
kfree(tree);
+ cifs_put_tcp_super(sb);
+
return rc;
}
#else
seq_printf(m,
"cache entry: path=%s,type=%s,ttl=%d,etime=%ld,hdr_flags=0x%x,ref_flags=0x%x,interlink=%s,path_consumed=%d,expired=%s\n",
ce->path, ce->srvtype == DFS_TYPE_ROOT ? "root" : "link",
- ce->ttl, ce->etime.tv_nsec, ce->ref_flags, ce->hdr_flags,
+ ce->ttl, ce->etime.tv_nsec, ce->hdr_flags, ce->ref_flags,
IS_DFS_INTERLINK(ce->hdr_flags) ? "yes" : "no",
ce->path_consumed, cache_entry_expired(ce) ? "yes" : "no");
}
cifs_dbg(FYI, "%s: no cached or matched targets. mark dfs share for reconnect.\n", __func__);
- for (i = 0; i < tcon->ses->chan_count; i++) {
- spin_lock(&GlobalMid_Lock);
- if (tcon->ses->chans[i].server->tcpStatus != CifsExiting)
- tcon->ses->chans[i].server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&GlobalMid_Lock);
- }
+ cifs_ses_mark_for_reconnect(tcon->ses);
}
/* Refresh dfs referral of tcon and mark it for reconnect if needed */
-static int refresh_tcon(struct cifs_ses **sessions, struct cifs_tcon *tcon, bool force_refresh)
+static int __refresh_tcon(const char *path, struct cifs_ses **sessions, struct cifs_tcon *tcon,
+ bool force_refresh)
{
- const char *path = tcon->dfs_path + 1;
struct cifs_ses *ses;
struct cache_entry *ce;
struct dfs_info3_param *refs = NULL;
return rc;
}
+static int refresh_tcon(struct cifs_ses **sessions, struct cifs_tcon *tcon, bool force_refresh)
+{
+ struct TCP_Server_Info *server = tcon->ses->server;
+
+ mutex_lock(&server->refpath_lock);
+ if (strcasecmp(server->leaf_fullpath, server->origin_fullpath))
+ __refresh_tcon(server->leaf_fullpath + 1, sessions, tcon, force_refresh);
+ mutex_unlock(&server->refpath_lock);
+
+ __refresh_tcon(server->origin_fullpath + 1, sessions, tcon, force_refresh);
+
+ return 0;
+}
+
/**
* dfs_cache_remount_fs - remount a DFS share
*
int dfs_cache_remount_fs(struct cifs_sb_info *cifs_sb)
{
struct cifs_tcon *tcon;
+ struct TCP_Server_Info *server;
struct mount_group *mg;
struct cifs_ses *sessions[CACHE_MAX_ENTRIES + 1] = {NULL};
int rc;
return -EINVAL;
tcon = cifs_sb_master_tcon(cifs_sb);
- if (!tcon->dfs_path) {
- cifs_dbg(FYI, "%s: not a dfs tcon\n", __func__);
+ server = tcon->ses->server;
+
+ if (!server->origin_fullpath) {
+ cifs_dbg(FYI, "%s: not a dfs mount\n", __func__);
return 0;
}
if (uuid_is_null(&cifs_sb->dfs_mount_id)) {
- cifs_dbg(FYI, "%s: tcon has no dfs mount group id\n", __func__);
+ cifs_dbg(FYI, "%s: no dfs mount group id\n", __func__);
return -EINVAL;
}
mg = find_mount_group_locked(&cifs_sb->dfs_mount_id);
if (IS_ERR(mg)) {
mutex_unlock(&mount_group_list_lock);
- cifs_dbg(FYI, "%s: tcon has ipc session to refresh referral\n", __func__);
+ cifs_dbg(FYI, "%s: no ipc session for refreshing referral\n", __func__);
return PTR_ERR(mg);
}
kref_get(&mg->refcount);
spin_lock(&cifs_tcp_ses_lock);
list_for_each_entry(server, &cifs_tcp_ses_list, tcp_ses_list) {
+ if (!server->is_dfs_conn)
+ continue;
+
list_for_each_entry(ses, &server->smb_ses_list, smb_ses_list) {
list_for_each_entry(tcon, &ses->tcon_list, tcon_list) {
- if (tcon->dfs_path) {
+ if (!tcon->ipc && !tcon->need_reconnect) {
tcon->tc_count++;
list_add_tail(&tcon->ulist, &tcons);
}
spin_unlock(&cifs_tcp_ses_lock);
list_for_each_entry_safe(tcon, ntcon, &tcons, ulist) {
+ struct TCP_Server_Info *server = tcon->ses->server;
+
list_del_init(&tcon->ulist);
- refresh_tcon(sessions, tcon, false);
+
+ mutex_lock(&server->refpath_lock);
+ if (strcasecmp(server->leaf_fullpath, server->origin_fullpath))
+ __refresh_tcon(server->leaf_fullpath + 1, sessions, tcon, false);
+ mutex_unlock(&server->refpath_lock);
+
+ __refresh_tcon(server->origin_fullpath + 1, sessions, tcon, false);
cifs_put_tcon(tcon);
}
}
tcon = tlink_tcon(smbfile->tlink);
if (!(cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NOSSYNC)) {
server = tcon->ses->server;
- if (server->ops->flush)
- rc = server->ops->flush(xid, tcon, &smbfile->fid);
- else
+ if (server->ops->flush == NULL) {
rc = -ENOSYS;
+ goto strict_fsync_exit;
+ }
+
+ if ((OPEN_FMODE(smbfile->f_flags) & FMODE_WRITE) == 0) {
+ smbfile = find_writable_file(CIFS_I(inode), FIND_WR_ANY);
+ if (smbfile) {
+ rc = server->ops->flush(xid, tcon, &smbfile->fid);
+ cifsFileInfo_put(smbfile);
+ } else
+ cifs_dbg(FYI, "ignore fsync for file not open for write\n");
+ } else
+ rc = server->ops->flush(xid, tcon, &smbfile->fid);
}
+strict_fsync_exit:
free_xid(xid);
return rc;
}
struct cifs_tcon *tcon;
struct TCP_Server_Info *server;
struct cifsFileInfo *smbfile = file->private_data;
+ struct inode *inode = file_inode(file);
struct cifs_sb_info *cifs_sb = CIFS_FILE_SB(file);
rc = file_write_and_wait_range(file, start, end);
tcon = tlink_tcon(smbfile->tlink);
if (!(cifs_sb->mnt_cifs_flags & CIFS_MOUNT_NOSSYNC)) {
server = tcon->ses->server;
- if (server->ops->flush)
- rc = server->ops->flush(xid, tcon, &smbfile->fid);
- else
+ if (server->ops->flush == NULL) {
rc = -ENOSYS;
+ goto fsync_exit;
+ }
+
+ if ((OPEN_FMODE(smbfile->f_flags) & FMODE_WRITE) == 0) {
+ smbfile = find_writable_file(CIFS_I(inode), FIND_WR_ANY);
+ if (smbfile) {
+ rc = server->ops->flush(xid, tcon, &smbfile->fid);
+ cifsFileInfo_put(smbfile);
+ } else
+ cifs_dbg(FYI, "ignore fsync for file not open for write\n");
+ } else
+ rc = server->ops->flush(xid, tcon, &smbfile->fid);
}
+fsync_exit:
free_xid(xid);
return rc;
}
fsparam_flag("nosharesock", Opt_nosharesock),
fsparam_flag_no("persistenthandles", Opt_persistent),
fsparam_flag_no("resilienthandles", Opt_resilient),
+ fsparam_flag_no("tcpnodelay", Opt_tcp_nodelay),
fsparam_flag("domainauto", Opt_domainauto),
fsparam_flag("rdma", Opt_rdma),
fsparam_flag("modesid", Opt_modesid),
new_ctx->nodename = NULL;
new_ctx->username = NULL;
new_ctx->password = NULL;
+ new_ctx->server_hostname = NULL;
new_ctx->domainname = NULL;
+ new_ctx->workstation_name = NULL;
new_ctx->UNC = NULL;
new_ctx->source = NULL;
new_ctx->iocharset = NULL;
DUP_CTX_STR(mount_options);
DUP_CTX_STR(username);
DUP_CTX_STR(password);
+ DUP_CTX_STR(server_hostname);
DUP_CTX_STR(UNC);
DUP_CTX_STR(source);
DUP_CTX_STR(domainname);
+ DUP_CTX_STR(workstation_name);
DUP_CTX_STR(nodename);
DUP_CTX_STR(iocharset);
if (!pos)
return -EINVAL;
+ /* record the server hostname */
+ kfree(ctx->server_hostname);
+ ctx->server_hostname = kstrndup(devname + 2, pos - devname - 2, GFP_KERNEL);
+ if (!ctx->server_hostname)
+ return -ENOMEM;
+
/* skip past delimiter */
++pos;
cifs_errorf(fc, "can not change domainname during remount\n");
return -EINVAL;
}
+ if (new_ctx->workstation_name &&
+ (!old_ctx->workstation_name || strcmp(new_ctx->workstation_name, old_ctx->workstation_name))) {
+ cifs_errorf(fc, "can not change workstation_name during remount\n");
+ return -EINVAL;
+ }
if (new_ctx->nodename &&
(!old_ctx->nodename || strcmp(new_ctx->nodename, old_ctx->nodename))) {
cifs_errorf(fc, "can not change nodename during remount\n");
return rc;
/*
- * We can not change UNC/username/password/domainname/nodename/iocharset
+ * We can not change UNC/username/password/domainname/
+ * workstation_name/nodename/iocharset
* during reconnect so ignore what we have in the new context and
* just use what we already have in cifs_sb->ctx.
*/
STEAL_STRING(cifs_sb, ctx, username);
STEAL_STRING(cifs_sb, ctx, password);
STEAL_STRING(cifs_sb, ctx, domainname);
+ STEAL_STRING(cifs_sb, ctx, workstation_name);
STEAL_STRING(cifs_sb, ctx, nodename);
STEAL_STRING(cifs_sb, ctx, iocharset);
}
}
break;
+ case Opt_tcp_nodelay:
+ /* tcp nodelay should not usually be needed since we CORK/UNCORK the socket */
+ if (result.negated)
+ ctx->sockopt_tcp_nodelay = false;
+ else
+ ctx->sockopt_tcp_nodelay = true;
+ break;
case Opt_domainauto:
ctx->domainauto = true;
break;
int smb3_init_fs_context(struct fs_context *fc)
{
+ int rc;
struct smb3_fs_context *ctx;
char *nodename = utsname()->nodename;
int i;
ctx = kzalloc(sizeof(struct smb3_fs_context), GFP_KERNEL);
- if (unlikely(!ctx))
- return -ENOMEM;
+ if (unlikely(!ctx)) {
+ rc = -ENOMEM;
+ goto err_exit;
+ }
+
+ ctx->workstation_name = kstrdup(nodename, GFP_KERNEL);
+ if (unlikely(!ctx->workstation_name)) {
+ rc = -ENOMEM;
+ goto err_exit;
+ }
/*
* does not have to be perfect mapping since field is
fc->fs_private = ctx;
fc->ops = &smb3_fs_context_ops;
return 0;
+
+err_exit:
+ if (ctx) {
+ kfree(ctx->workstation_name);
+ kfree(ctx);
+ }
+
+ return rc;
}
void
ctx->username = NULL;
kfree_sensitive(ctx->password);
ctx->password = NULL;
+ kfree(ctx->server_hostname);
+ ctx->server_hostname = NULL;
kfree(ctx->UNC);
ctx->UNC = NULL;
kfree(ctx->source);
ctx->source = NULL;
kfree(ctx->domainname);
ctx->domainname = NULL;
+ kfree(ctx->workstation_name);
+ ctx->workstation_name = NULL;
kfree(ctx->nodename);
ctx->nodename = NULL;
kfree(ctx->iocharset);
Opt_nosharesock,
Opt_persistent,
Opt_resilient,
+ Opt_tcp_nodelay,
Opt_domainauto,
Opt_rdma,
Opt_modesid,
char *password;
char *domainname;
char *source;
+ char *server_hostname;
char *UNC;
char *nodename;
+ char *workstation_name;
char *iocharset; /* local code page for mapping to and from Unicode */
char source_rfc1001_name[RFC1001_NAME_LEN_WITH_NULL]; /* clnt nb name */
char target_rfc1001_name[RFC1001_NAME_LEN_WITH_NULL]; /* srvr nb name */
char *sharename;
struct cifs_fscache_super_auxdata auxdata;
+ /*
+ * Check if cookie was already initialized so don't reinitialize it.
+ * In the future, as we integrate with newer fscache features,
+ * we may want to instead add a check if cookie has changed
+ */
+ if (tcon->fscache == NULL)
+ return;
+
sharename = extract_sharename(tcon->treeName);
if (IS_ERR(sharename)) {
cifs_dbg(FYI, "%s: couldn't extract sharename\n", __func__);
INIT_LIST_HEAD(&ret_buf->tcon_list);
mutex_init(&ret_buf->session_mutex);
spin_lock_init(&ret_buf->iface_lock);
+ spin_lock_init(&ret_buf->chan_lock);
}
return ret_buf;
}
kfree_sensitive(buf_to_free->password);
kfree(buf_to_free->user_name);
kfree(buf_to_free->domainName);
+ kfree(buf_to_free->workstation_name);
kfree_sensitive(buf_to_free->auth_key.response);
kfree(buf_to_free->iface_list);
kfree_sensitive(buf_to_free);
kfree(buf_to_free->nativeFileSystem);
kfree_sensitive(buf_to_free->password);
kfree(buf_to_free->crfid.fid);
-#ifdef CONFIG_CIFS_DFS_UPCALL
- kfree(buf_to_free->dfs_path);
-#endif
kfree(buf_to_free);
}
* SMB2 header is bigger than CIFS one - no problems to clean some
* more bytes for CIFS.
*/
- size_t buf_size = sizeof(struct smb2_sync_hdr);
+ size_t buf_size = sizeof(struct smb2_hdr);
/*
* We could use negotiated size instead of max_msgsize -
return rc;
}
-static void tcon_super_cb(struct super_block *sb, void *arg)
+int cifs_update_super_prepath(struct cifs_sb_info *cifs_sb, char *prefix)
{
- struct super_cb_data *sd = arg;
- struct cifs_tcon *tcon = sd->data;
- struct cifs_sb_info *cifs_sb;
-
- if (sd->sb)
- return;
-
- cifs_sb = CIFS_SB(sb);
- if (tcon->dfs_path && cifs_sb->origin_fullpath &&
- !strcasecmp(tcon->dfs_path, cifs_sb->origin_fullpath))
- sd->sb = sb;
-}
-
-static inline struct super_block *cifs_get_tcon_super(struct cifs_tcon *tcon)
-{
- return __cifs_get_super(tcon_super_cb, tcon);
-}
-
-static inline void cifs_put_tcon_super(struct super_block *sb)
-{
- __cifs_put_super(sb);
-}
-#else
-static inline struct super_block *cifs_get_tcon_super(struct cifs_tcon *tcon)
-{
- return ERR_PTR(-EOPNOTSUPP);
-}
-
-static inline void cifs_put_tcon_super(struct super_block *sb)
-{
-}
-#endif
-
-int update_super_prepath(struct cifs_tcon *tcon, char *prefix)
-{
- struct super_block *sb;
- struct cifs_sb_info *cifs_sb;
- int rc = 0;
-
- sb = cifs_get_tcon_super(tcon);
- if (IS_ERR(sb))
- return PTR_ERR(sb);
-
- cifs_sb = CIFS_SB(sb);
-
kfree(cifs_sb->prepath);
if (prefix && *prefix) {
cifs_sb->prepath = kstrdup(prefix, GFP_ATOMIC);
- if (!cifs_sb->prepath) {
- rc = -ENOMEM;
- goto out;
- }
+ if (!cifs_sb->prepath)
+ return -ENOMEM;
convert_delimiter(cifs_sb->prepath, CIFS_DIR_SEP(cifs_sb));
} else
cifs_sb->prepath = NULL;
cifs_sb->mnt_cifs_flags |= CIFS_MOUNT_USE_PREFIX_PATH;
-
-out:
- cifs_put_tcon_super(sb);
- return rc;
+ return 0;
}
+#endif
*/
int decode_ntlmssp_challenge(char *bcc_ptr, int blob_len, struct cifs_ses *ses);
-void build_ntlmssp_negotiate_blob(unsigned char *pbuffer, struct cifs_ses *ses);
+int build_ntlmssp_negotiate_blob(unsigned char **pbuffer, u16 *buflen,
+ struct cifs_ses *ses,
+ const struct nls_table *nls_cp);
int build_ntlmssp_auth_blob(unsigned char **pbuffer, u16 *buflen,
struct cifs_ses *ses,
const struct nls_table *nls_cp);
{
int i;
+ spin_lock(&ses->chan_lock);
for (i = 0; i < ses->chan_count; i++) {
- if (is_server_using_iface(ses->chans[i].server, iface))
+ if (is_server_using_iface(ses->chans[i].server, iface)) {
+ spin_unlock(&ses->chan_lock);
return true;
+ }
}
+ spin_unlock(&ses->chan_lock);
return false;
}
/* returns number of channels added */
int cifs_try_adding_channels(struct cifs_sb_info *cifs_sb, struct cifs_ses *ses)
{
- int old_chan_count = ses->chan_count;
- int left = ses->chan_max - ses->chan_count;
+ int old_chan_count, new_chan_count;
+ int left;
int i = 0;
int rc = 0;
int tries = 0;
struct cifs_server_iface *ifaces = NULL;
size_t iface_count;
+ if (ses->server->dialect < SMB30_PROT_ID) {
+ cifs_dbg(VFS, "multichannel is not supported on this protocol version, use 3.0 or above\n");
+ return 0;
+ }
+
+ spin_lock(&ses->chan_lock);
+
+ new_chan_count = old_chan_count = ses->chan_count;
+ left = ses->chan_max - ses->chan_count;
+
if (left <= 0) {
cifs_dbg(FYI,
"ses already at max_channels (%zu), nothing to open\n",
ses->chan_max);
- return 0;
- }
-
- if (ses->server->dialect < SMB30_PROT_ID) {
- cifs_dbg(VFS, "multichannel is not supported on this protocol version, use 3.0 or above\n");
+ spin_unlock(&ses->chan_lock);
return 0;
}
if (!(ses->server->capabilities & SMB2_GLOBAL_CAP_MULTI_CHANNEL)) {
- cifs_dbg(VFS, "server %s does not support multichannel\n", ses->server->hostname);
ses->chan_max = 1;
+ spin_unlock(&ses->chan_lock);
+ cifs_dbg(VFS, "server %s does not support multichannel\n", ses->server->hostname);
return 0;
}
+ spin_unlock(&ses->chan_lock);
/*
* Make a copy of the iface list at the time and use that
cifs_dbg(FYI, "successfully opened new channel on iface#%d\n",
i);
left--;
+ new_chan_count++;
}
kfree(ifaces);
- return ses->chan_count - old_chan_count;
+ return new_chan_count - old_chan_count;
}
/*
{
int i;
+ spin_lock(&ses->chan_lock);
for (i = 0; i < ses->chan_count; i++) {
- if (ses->chans[i].server == server)
+ if (ses->chans[i].server == server) {
+ spin_unlock(&ses->chan_lock);
return &ses->chans[i];
+ }
}
+ spin_unlock(&ses->chan_lock);
return NULL;
}
cifs_ses_add_channel(struct cifs_sb_info *cifs_sb, struct cifs_ses *ses,
struct cifs_server_iface *iface)
{
+ struct TCP_Server_Info *chan_server;
struct cifs_chan *chan;
struct smb3_fs_context ctx = {NULL};
static const char unc_fmt[] = "\\%s\\foo";
SMB2_CLIENT_GUID_SIZE);
ctx.use_client_guid = true;
- mutex_lock(&ses->session_mutex);
+ chan_server = cifs_get_tcp_session(&ctx, ses->server);
+ mutex_lock(&ses->session_mutex);
+ spin_lock(&ses->chan_lock);
chan = ses->binding_chan = &ses->chans[ses->chan_count];
- chan->server = cifs_get_tcp_session(&ctx);
+ chan->server = chan_server;
if (IS_ERR(chan->server)) {
rc = PTR_ERR(chan->server);
chan->server = NULL;
+ spin_unlock(&ses->chan_lock);
goto out;
}
- spin_lock(&cifs_tcp_ses_lock);
- chan->server->is_channel = true;
- spin_unlock(&cifs_tcp_ses_lock);
+ spin_unlock(&ses->chan_lock);
/*
* We need to allocate the server crypto now as we will need
* ses to the new server.
*/
+ spin_lock(&ses->chan_lock);
ses->chan_count++;
atomic_set(&ses->chan_seq, 0);
+ spin_unlock(&ses->chan_lock);
+
out:
ses->binding = false;
ses->binding_chan = NULL;
return rc;
}
+/* Mark all session channels for reconnect */
+void cifs_ses_mark_for_reconnect(struct cifs_ses *ses)
+{
+ int i;
+
+ for (i = 0; i < ses->chan_count; i++) {
+ spin_lock(&GlobalMid_Lock);
+ if (ses->chans[i].server->tcpStatus != CifsExiting)
+ ses->chans[i].server->tcpStatus = CifsNeedReconnect;
+ spin_unlock(&GlobalMid_Lock);
+ }
+}
+
static __u32 cifs_ssetup_hdr(struct cifs_ses *ses, SESSION_SETUP_ANDX *pSMB)
{
__u32 capabilities = 0;
return 0;
}
+static int size_of_ntlmssp_blob(struct cifs_ses *ses, int base_size)
+{
+ int sz = base_size + ses->auth_key.len
+ - CIFS_SESS_KEY_SIZE + CIFS_CPHTXT_SIZE + 2;
+
+ if (ses->domainName)
+ sz += sizeof(__le16) * strnlen(ses->domainName, CIFS_MAX_DOMAINNAME_LEN);
+ else
+ sz += sizeof(__le16);
+
+ if (ses->user_name)
+ sz += sizeof(__le16) * strnlen(ses->user_name, CIFS_MAX_USERNAME_LEN);
+ else
+ sz += sizeof(__le16);
+
+ sz += sizeof(__le16) * strnlen(ses->workstation_name, CIFS_MAX_WORKSTATION_LEN);
+
+ return sz;
+}
+
+static inline void cifs_security_buffer_from_str(SECURITY_BUFFER *pbuf,
+ char *str_value,
+ int str_length,
+ unsigned char *pstart,
+ unsigned char **pcur,
+ const struct nls_table *nls_cp)
+{
+ unsigned char *tmp = pstart;
+ int len;
+
+ if (!pbuf)
+ return;
+
+ if (!pcur)
+ pcur = &tmp;
+
+ if (!str_value) {
+ pbuf->BufferOffset = cpu_to_le32(*pcur - pstart);
+ pbuf->Length = 0;
+ pbuf->MaximumLength = 0;
+ *pcur += sizeof(__le16);
+ } else {
+ len = cifs_strtoUTF16((__le16 *)*pcur,
+ str_value,
+ str_length,
+ nls_cp);
+ len *= sizeof(__le16);
+ pbuf->BufferOffset = cpu_to_le32(*pcur - pstart);
+ pbuf->Length = cpu_to_le16(len);
+ pbuf->MaximumLength = cpu_to_le16(len);
+ *pcur += len;
+ }
+}
+
/* BB Move to ntlmssp.c eventually */
-/* We do not malloc the blob, it is passed in pbuffer, because
- it is fixed size, and small, making this approach cleaner */
-void build_ntlmssp_negotiate_blob(unsigned char *pbuffer,
- struct cifs_ses *ses)
+int build_ntlmssp_negotiate_blob(unsigned char **pbuffer,
+ u16 *buflen,
+ struct cifs_ses *ses,
+ const struct nls_table *nls_cp)
{
+ int rc = 0;
struct TCP_Server_Info *server = cifs_ses_server(ses);
- NEGOTIATE_MESSAGE *sec_blob = (NEGOTIATE_MESSAGE *)pbuffer;
+ NEGOTIATE_MESSAGE *sec_blob;
__u32 flags;
+ unsigned char *tmp;
+ int len;
+
+ len = size_of_ntlmssp_blob(ses, sizeof(NEGOTIATE_MESSAGE));
+ *pbuffer = kmalloc(len, GFP_KERNEL);
+ if (!*pbuffer) {
+ rc = -ENOMEM;
+ cifs_dbg(VFS, "Error %d during NTLMSSP allocation\n", rc);
+ *buflen = 0;
+ goto setup_ntlm_neg_ret;
+ }
+ sec_blob = (NEGOTIATE_MESSAGE *)*pbuffer;
- memset(pbuffer, 0, sizeof(NEGOTIATE_MESSAGE));
+ memset(*pbuffer, 0, sizeof(NEGOTIATE_MESSAGE));
memcpy(sec_blob->Signature, NTLMSSP_SIGNATURE, 8);
sec_blob->MessageType = NtLmNegotiate;
if (!server->session_estab || ses->ntlmssp->sesskey_per_smbsess)
flags |= NTLMSSP_NEGOTIATE_KEY_XCH;
+ tmp = *pbuffer + sizeof(NEGOTIATE_MESSAGE);
sec_blob->NegotiateFlags = cpu_to_le32(flags);
- sec_blob->WorkstationName.BufferOffset = 0;
- sec_blob->WorkstationName.Length = 0;
- sec_blob->WorkstationName.MaximumLength = 0;
-
- /* Domain name is sent on the Challenge not Negotiate NTLMSSP request */
- sec_blob->DomainName.BufferOffset = 0;
- sec_blob->DomainName.Length = 0;
- sec_blob->DomainName.MaximumLength = 0;
-}
-
-static int size_of_ntlmssp_blob(struct cifs_ses *ses)
-{
- int sz = sizeof(AUTHENTICATE_MESSAGE) + ses->auth_key.len
- - CIFS_SESS_KEY_SIZE + CIFS_CPHTXT_SIZE + 2;
-
- if (ses->domainName)
- sz += 2 * strnlen(ses->domainName, CIFS_MAX_DOMAINNAME_LEN);
- else
- sz += 2;
+ /* these fields should be null in negotiate phase MS-NLMP 3.1.5.1.1 */
+ cifs_security_buffer_from_str(&sec_blob->DomainName,
+ NULL,
+ CIFS_MAX_DOMAINNAME_LEN,
+ *pbuffer, &tmp,
+ nls_cp);
- if (ses->user_name)
- sz += 2 * strnlen(ses->user_name, CIFS_MAX_USERNAME_LEN);
- else
- sz += 2;
+ cifs_security_buffer_from_str(&sec_blob->WorkstationName,
+ NULL,
+ CIFS_MAX_WORKSTATION_LEN,
+ *pbuffer, &tmp,
+ nls_cp);
- return sz;
+ *buflen = tmp - *pbuffer;
+setup_ntlm_neg_ret:
+ return rc;
}
int build_ntlmssp_auth_blob(unsigned char **pbuffer,
AUTHENTICATE_MESSAGE *sec_blob;
__u32 flags;
unsigned char *tmp;
+ int len;
rc = setup_ntlmv2_rsp(ses, nls_cp);
if (rc) {
*buflen = 0;
goto setup_ntlmv2_ret;
}
- *pbuffer = kmalloc(size_of_ntlmssp_blob(ses), GFP_KERNEL);
+
+ len = size_of_ntlmssp_blob(ses, sizeof(AUTHENTICATE_MESSAGE));
+ *pbuffer = kmalloc(len, GFP_KERNEL);
if (!*pbuffer) {
rc = -ENOMEM;
cifs_dbg(VFS, "Error %d during NTLMSSP allocation\n", rc);
NTLMSSP_REQUEST_TARGET | NTLMSSP_NEGOTIATE_TARGET_INFO |
NTLMSSP_NEGOTIATE_128 | NTLMSSP_NEGOTIATE_UNICODE |
NTLMSSP_NEGOTIATE_NTLM | NTLMSSP_NEGOTIATE_EXTENDED_SEC |
- NTLMSSP_NEGOTIATE_SEAL;
+ NTLMSSP_NEGOTIATE_SEAL | NTLMSSP_NEGOTIATE_WORKSTATION_SUPPLIED;
if (ses->server->sign)
flags |= NTLMSSP_NEGOTIATE_SIGN;
if (!ses->server->session_estab || ses->ntlmssp->sesskey_per_smbsess)
sec_blob->NtChallengeResponse.MaximumLength = 0;
}
- if (ses->domainName == NULL) {
- sec_blob->DomainName.BufferOffset = cpu_to_le32(tmp - *pbuffer);
- sec_blob->DomainName.Length = 0;
- sec_blob->DomainName.MaximumLength = 0;
- tmp += 2;
- } else {
- int len;
- len = cifs_strtoUTF16((__le16 *)tmp, ses->domainName,
- CIFS_MAX_DOMAINNAME_LEN, nls_cp);
- len *= 2; /* unicode is 2 bytes each */
- sec_blob->DomainName.BufferOffset = cpu_to_le32(tmp - *pbuffer);
- sec_blob->DomainName.Length = cpu_to_le16(len);
- sec_blob->DomainName.MaximumLength = cpu_to_le16(len);
- tmp += len;
- }
+ cifs_security_buffer_from_str(&sec_blob->DomainName,
+ ses->domainName,
+ CIFS_MAX_DOMAINNAME_LEN,
+ *pbuffer, &tmp,
+ nls_cp);
- if (ses->user_name == NULL) {
- sec_blob->UserName.BufferOffset = cpu_to_le32(tmp - *pbuffer);
- sec_blob->UserName.Length = 0;
- sec_blob->UserName.MaximumLength = 0;
- tmp += 2;
- } else {
- int len;
- len = cifs_strtoUTF16((__le16 *)tmp, ses->user_name,
- CIFS_MAX_USERNAME_LEN, nls_cp);
- len *= 2; /* unicode is 2 bytes each */
- sec_blob->UserName.BufferOffset = cpu_to_le32(tmp - *pbuffer);
- sec_blob->UserName.Length = cpu_to_le16(len);
- sec_blob->UserName.MaximumLength = cpu_to_le16(len);
- tmp += len;
- }
+ cifs_security_buffer_from_str(&sec_blob->UserName,
+ ses->user_name,
+ CIFS_MAX_USERNAME_LEN,
+ *pbuffer, &tmp,
+ nls_cp);
- sec_blob->WorkstationName.BufferOffset = cpu_to_le32(tmp - *pbuffer);
- sec_blob->WorkstationName.Length = 0;
- sec_blob->WorkstationName.MaximumLength = 0;
- tmp += 2;
+ cifs_security_buffer_from_str(&sec_blob->WorkstationName,
+ ses->workstation_name,
+ CIFS_MAX_WORKSTATION_LEN,
+ *pbuffer, &tmp,
+ nls_cp);
if (((ses->ntlmssp->server_flags & NTLMSSP_NEGOTIATE_KEY_XCH) ||
(ses->ntlmssp->server_flags & NTLMSSP_NEGOTIATE_EXTENDED_SEC))
struct cifs_ses *ses = sess_data->ses;
__u16 bytes_remaining;
char *bcc_ptr;
+ unsigned char *ntlmsspblob = NULL;
u16 blob_len;
cifs_dbg(FYI, "rawntlmssp session setup negotiate phase\n");
pSMB = (SESSION_SETUP_ANDX *)sess_data->iov[0].iov_base;
/* Build security blob before we assemble the request */
- build_ntlmssp_negotiate_blob(pSMB->req.SecurityBlob, ses);
- sess_data->iov[1].iov_len = sizeof(NEGOTIATE_MESSAGE);
- sess_data->iov[1].iov_base = pSMB->req.SecurityBlob;
- pSMB->req.SecurityBlobLength = cpu_to_le16(sizeof(NEGOTIATE_MESSAGE));
+ rc = build_ntlmssp_negotiate_blob(&ntlmsspblob,
+ &blob_len, ses,
+ sess_data->nls_cp);
+ if (rc)
+ goto out;
+
+ sess_data->iov[1].iov_len = blob_len;
+ sess_data->iov[1].iov_base = ntlmsspblob;
+ pSMB->req.SecurityBlobLength = cpu_to_le16(blob_len);
rc = _sess_auth_rawntlmssp_assemble_req(sess_data);
if (rc)
struct smb2_file_link_info link_info;
};
+/*
+ * note: If cfile is passed, the reference to it is dropped here.
+ * So make sure that you do not reuse cfile after return from this func.
+ */
static int
smb2_compound_op(const unsigned int xid, struct cifs_tcon *tcon,
struct cifs_sb_info *cifs_sb, const char *full_path,
create_options |= OPEN_REPARSE_POINT;
/* Failed on a symbolic link - query a reparse point info */
+ cifs_get_readable_path(tcon, full_path, &cfile);
rc = smb2_compound_op(xid, tcon, cifs_sb, full_path,
FILE_READ_ATTRIBUTES, FILE_OPEN,
create_options, ACL_NO_MODE,
- smb2_data, SMB2_OP_QUERY_INFO, NULL);
+ smb2_data, SMB2_OP_QUERY_INFO, cfile);
}
if (rc)
goto out;
create_options |= OPEN_REPARSE_POINT;
/* Failed on a symbolic link - query a reparse point info */
+ cifs_get_readable_path(tcon, full_path, &cfile);
rc = smb2_compound_op(xid, tcon, cifs_sb, full_path,
FILE_READ_ATTRIBUTES, FILE_OPEN,
create_options, ACL_NO_MODE,
- smb2_data, SMB2_OP_POSIX_QUERY_INFO, NULL);
+ smb2_data, SMB2_OP_POSIX_QUERY_INFO, cfile);
}
if (rc)
goto out;
struct cifs_sb_info *cifs_sb, bool set_alloc)
{
__le64 eof = cpu_to_le64(size);
+ struct cifsFileInfo *cfile;
+ cifs_get_writable_path(tcon, full_path, FIND_WR_ANY, &cfile);
return smb2_compound_op(xid, tcon, cifs_sb, full_path,
FILE_WRITE_DATA, FILE_OPEN, 0, ACL_NO_MODE,
- &eof, SMB2_OP_SET_EOF, NULL);
+ &eof, SMB2_OP_SET_EOF, cfile);
}
int
{
struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb);
struct tcon_link *tlink;
+ struct cifs_tcon *tcon;
+ struct cifsFileInfo *cfile;
int rc;
if ((buf->CreationTime == 0) && (buf->LastAccessTime == 0) &&
tlink = cifs_sb_tlink(cifs_sb);
if (IS_ERR(tlink))
return PTR_ERR(tlink);
+ tcon = tlink_tcon(tlink);
- rc = smb2_compound_op(xid, tlink_tcon(tlink), cifs_sb, full_path,
+ cifs_get_writable_path(tcon, full_path, FIND_WR_ANY, &cfile);
+ rc = smb2_compound_op(xid, tcon, cifs_sb, full_path,
FILE_WRITE_ATTRIBUTES, FILE_OPEN,
- 0, ACL_NO_MODE, buf, SMB2_OP_SET_INFO, NULL);
+ 0, ACL_NO_MODE, buf, SMB2_OP_SET_INFO, cfile);
cifs_put_tlink(tlink);
return rc;
}
int
map_smb2_to_linux_error(char *buf, bool log_err)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
unsigned int i;
int rc = -EIO;
__le32 smb2err = shdr->Status;
if (smb2err == 0) {
- trace_smb3_cmd_done(shdr->TreeId, shdr->SessionId,
- le16_to_cpu(shdr->Command), le64_to_cpu(shdr->MessageId));
+ trace_smb3_cmd_done(le32_to_cpu(shdr->Id.SyncId.TreeId),
+ le64_to_cpu(shdr->SessionId),
+ le16_to_cpu(shdr->Command),
+ le64_to_cpu(shdr->MessageId));
return 0;
}
cifs_dbg(FYI, "Mapping SMB2 status code 0x%08x to POSIX err %d\n",
__le32_to_cpu(smb2err), rc);
- trace_smb3_cmd_err(shdr->TreeId, shdr->SessionId,
- le16_to_cpu(shdr->Command),
- le64_to_cpu(shdr->MessageId), le32_to_cpu(smb2err), rc);
+ trace_smb3_cmd_err(le32_to_cpu(shdr->Id.SyncId.TreeId),
+ le64_to_cpu(shdr->SessionId),
+ le16_to_cpu(shdr->Command),
+ le64_to_cpu(shdr->MessageId),
+ le32_to_cpu(smb2err), rc);
return rc;
}
*
*/
#include <linux/ctype.h>
-#include "smb2pdu.h"
#include "cifsglob.h"
#include "cifsproto.h"
#include "smb2proto.h"
#include "nterr.h"
static int
-check_smb2_hdr(struct smb2_sync_hdr *shdr, __u64 mid)
+check_smb2_hdr(struct smb2_hdr *shdr, __u64 mid)
{
__u64 wire_mid = le64_to_cpu(shdr->MessageId);
/* SMB2_OPLOCK_BREAK */ cpu_to_le16(24)
};
-#define SMB311_NEGPROT_BASE_SIZE (sizeof(struct smb2_sync_hdr) + sizeof(struct smb2_negotiate_rsp))
+#define SMB311_NEGPROT_BASE_SIZE (sizeof(struct smb2_hdr) + sizeof(struct smb2_negotiate_rsp))
-static __u32 get_neg_ctxt_len(struct smb2_sync_hdr *hdr, __u32 len,
+static __u32 get_neg_ctxt_len(struct smb2_hdr *hdr, __u32 len,
__u32 non_ctxlen)
{
__u16 neg_count;
int
smb2_check_message(char *buf, unsigned int len, struct TCP_Server_Info *srvr)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
- struct smb2_sync_pdu *pdu = (struct smb2_sync_pdu *)shdr;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
+ struct smb2_pdu *pdu = (struct smb2_pdu *)shdr;
__u64 mid;
__u32 clc_len; /* calculated length */
int command;
- int pdu_size = sizeof(struct smb2_sync_pdu);
- int hdr_size = sizeof(struct smb2_sync_hdr);
+ int pdu_size = sizeof(struct smb2_pdu);
+ int hdr_size = sizeof(struct smb2_hdr);
/*
* Add function to do table lookup of StructureSize by command
/* decrypt frame now that it is completely read in */
spin_lock(&cifs_tcp_ses_lock);
list_for_each_entry(ses, &srvr->smb_ses_list, smb_ses_list) {
- if (ses->Suid == thdr->SessionId)
+ if (ses->Suid == le64_to_cpu(thdr->SessionId))
break;
}
spin_unlock(&cifs_tcp_ses_lock);
* area and the offset to it (from the beginning of the smb are also returned.
*/
char *
-smb2_get_data_area_len(int *off, int *len, struct smb2_sync_hdr *shdr)
+smb2_get_data_area_len(int *off, int *len, struct smb2_hdr *shdr)
{
*off = 0;
*len = 0;
unsigned int
smb2_calc_size(void *buf, struct TCP_Server_Info *srvr)
{
- struct smb2_sync_pdu *pdu = (struct smb2_sync_pdu *)buf;
- struct smb2_sync_hdr *shdr = &pdu->sync_hdr;
+ struct smb2_pdu *pdu = (struct smb2_pdu *)buf;
+ struct smb2_hdr *shdr = &pdu->hdr;
int offset; /* the offset from the beginning of SMB to data area */
int data_length; /* the length of the variable length data area */
/* Structure Size has already been checked to make sure it is 64 */
cifs_dbg(FYI, "Checking for oplock break\n");
- if (rsp->sync_hdr.Command != SMB2_OPLOCK_BREAK)
+ if (rsp->hdr.Command != SMB2_OPLOCK_BREAK)
return false;
if (rsp->StructureSize !=
int
smb2_handle_cancelled_mid(struct mid_q_entry *mid, struct TCP_Server_Info *server)
{
- struct smb2_sync_hdr *sync_hdr = mid->resp_buf;
+ struct smb2_hdr *hdr = mid->resp_buf;
struct smb2_create_rsp *rsp = mid->resp_buf;
struct cifs_tcon *tcon;
int rc;
- if ((mid->optype & CIFS_CP_CREATE_CLOSE_OP) || sync_hdr->Command != SMB2_CREATE ||
- sync_hdr->Status != STATUS_SUCCESS)
+ if ((mid->optype & CIFS_CP_CREATE_CLOSE_OP) || hdr->Command != SMB2_CREATE ||
+ hdr->Status != STATUS_SUCCESS)
return 0;
- tcon = smb2_find_smb_tcon(server, sync_hdr->SessionId,
- sync_hdr->TreeId);
+ tcon = smb2_find_smb_tcon(server, le64_to_cpu(hdr->SessionId),
+ le32_to_cpu(hdr->Id.SyncId.TreeId));
if (!tcon)
return -ENOENT;
rc = __smb2_handle_cancelled_cmd(tcon,
- le16_to_cpu(sync_hdr->Command),
- le64_to_cpu(sync_hdr->MessageId),
- rsp->PersistentFileId,
- rsp->VolatileFileId);
+ le16_to_cpu(hdr->Command),
+ le64_to_cpu(hdr->MessageId),
+ le64_to_cpu(rsp->PersistentFileId),
+ le64_to_cpu(rsp->VolatileFileId));
if (rc)
cifs_put_tcon(tcon);
{
int i, rc;
struct sdesc *d;
- struct smb2_sync_hdr *hdr;
+ struct smb2_hdr *hdr;
struct TCP_Server_Info *server = cifs_ses_server(ses);
- hdr = (struct smb2_sync_hdr *)iov[0].iov_base;
+ hdr = (struct smb2_hdr *)iov[0].iov_base;
/* neg prot are always taken */
if (hdr->Command == SMB2_NEGOTIATE)
goto ok;
__smb2_find_mid(struct TCP_Server_Info *server, char *buf, bool dequeue)
{
struct mid_q_entry *mid;
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
__u64 wire_mid = le64_to_cpu(shdr->MessageId);
if (shdr->ProtocolId == SMB2_TRANSFORM_PROTO_NUM) {
smb2_dump_detail(void *buf, struct TCP_Server_Info *server)
{
#ifdef CONFIG_CIFS_DEBUG2
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
cifs_server_dbg(VFS, "Cmd: %d Err: 0x%x Flags: 0x%x Mid: %llu Pid: %d\n",
shdr->Command, shdr->Status, shdr->Flags, shdr->MessageId,
- shdr->ProcessId);
+ shdr->Id.SyncId.ProcessId);
cifs_server_dbg(VFS, "smb buf %p len %u\n", buf,
server->ops->calc_smb_size(buf, server));
#endif
atomic_inc(&tcon->num_remote_opens);
o_rsp = (struct smb2_create_rsp *)rsp_iov[0].iov_base;
- oparms.fid->persistent_fid = o_rsp->PersistentFileId;
- oparms.fid->volatile_fid = o_rsp->VolatileFileId;
+ oparms.fid->persistent_fid = le64_to_cpu(o_rsp->PersistentFileId);
+ oparms.fid->volatile_fid = le64_to_cpu(o_rsp->VolatileFileId);
#ifdef CONFIG_CIFS_DEBUG2
- oparms.fid->mid = le64_to_cpu(o_rsp->sync_hdr.MessageId);
+ oparms.fid->mid = le64_to_cpu(o_rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
tcon->crfid.tcon = tcon;
/* If the open failed there is nothing to do */
op_rsp = (struct smb2_create_rsp *)rsp_iov[0].iov_base;
- if (op_rsp == NULL || op_rsp->sync_hdr.Status != STATUS_SUCCESS) {
+ if (op_rsp == NULL || op_rsp->hdr.Status != STATUS_SUCCESS) {
cifs_dbg(FYI, "query_dir_first: open failed rc=%d\n", rc);
goto qdf_free;
}
- fid->persistent_fid = op_rsp->PersistentFileId;
- fid->volatile_fid = op_rsp->VolatileFileId;
+ fid->persistent_fid = le64_to_cpu(op_rsp->PersistentFileId);
+ fid->volatile_fid = le64_to_cpu(op_rsp->VolatileFileId);
/* Anything else than ENODATA means a genuine error */
if (rc && rc != -ENODATA) {
atomic_inc(&tcon->num_remote_opens);
qd_rsp = (struct smb2_query_directory_rsp *)rsp_iov[1].iov_base;
- if (qd_rsp->sync_hdr.Status == STATUS_NO_MORE_FILES) {
+ if (qd_rsp->hdr.Status == STATUS_NO_MORE_FILES) {
trace_smb3_query_dir_done(xid, fid->persistent_fid,
tcon->tid, tcon->ses->Suid, 0, 0);
srch_inf->endOfSearch = true;
static bool
smb2_is_status_pending(char *buf, struct TCP_Server_Info *server)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
int scredits, in_flight;
if (shdr->Status != STATUS_PENDING)
static bool
smb2_is_session_expired(char *buf)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
if (shdr->Status != STATUS_NETWORK_SESSION_EXPIRED &&
shdr->Status != STATUS_USER_SESSION_DELETED)
return false;
- trace_smb3_ses_expired(shdr->TreeId, shdr->SessionId,
+ trace_smb3_ses_expired(le32_to_cpu(shdr->Id.SyncId.TreeId),
+ le64_to_cpu(shdr->SessionId),
le16_to_cpu(shdr->Command),
le64_to_cpu(shdr->MessageId));
cifs_dbg(FYI, "Session expired or deleted\n");
static bool
smb2_is_status_io_timeout(char *buf)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
if (shdr->Status == STATUS_IO_TIMEOUT)
return true;
static void
smb2_is_network_name_deleted(char *buf, struct TCP_Server_Info *server)
{
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
struct list_head *tmp, *tmp1;
struct cifs_ses *ses;
struct cifs_tcon *tcon;
ses = list_entry(tmp, struct cifs_ses, smb_ses_list);
list_for_each(tmp1, &ses->tcon_list) {
tcon = list_entry(tmp1, struct cifs_tcon, tcon_list);
- if (tcon->tid == shdr->TreeId) {
+ if (tcon->tid == le32_to_cpu(shdr->Id.SyncId.TreeId)) {
tcon->need_reconnect = true;
spin_unlock(&cifs_tcp_ses_lock);
pr_warn_once("Server share %s deleted.\n",
void
smb2_set_related(struct smb_rqst *rqst)
{
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
- shdr = (struct smb2_sync_hdr *)(rqst->rq_iov[0].iov_base);
+ shdr = (struct smb2_hdr *)(rqst->rq_iov[0].iov_base);
if (shdr == NULL) {
cifs_dbg(FYI, "shdr NULL in smb2_set_related\n");
return;
void
smb2_set_next_command(struct cifs_tcon *tcon, struct smb_rqst *rqst)
{
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
struct cifs_ses *ses = tcon->ses;
struct TCP_Server_Info *server = ses->server;
unsigned long len = smb_rqst_len(server, rqst);
int i, num_padding;
- shdr = (struct smb2_sync_hdr *)(rqst->rq_iov[0].iov_base);
+ shdr = (struct smb2_hdr *)(rqst->rq_iov[0].iov_base);
if (shdr == NULL) {
cifs_dbg(FYI, "shdr NULL in smb2_set_next_command\n");
return;
struct fsctl_get_dfs_referral_req *dfs_req = NULL;
struct get_dfs_referral_rsp *dfs_rsp = NULL;
u32 dfs_req_size = 0, dfs_rsp_size = 0;
+ int retry_count = 0;
cifs_dbg(FYI, "%s: path: %s\n", __func__, search_name);
true /* is_fsctl */,
(char *)dfs_req, dfs_req_size, CIFSMaxBufSize,
(char **)&dfs_rsp, &dfs_rsp_size);
- } while (rc == -EAGAIN);
+ if (!is_retryable_error(rc))
+ break;
+ usleep_range(512, 2048);
+ } while (++retry_count < 5);
if (rc) {
- if ((rc != -ENOENT) && (rc != -EOPNOTSUPP))
- cifs_tcon_dbg(VFS, "ioctl error in %s rc=%d\n", __func__, rc);
+ if (!is_retryable_error(rc) && rc != -ENOENT && rc != -EOPNOTSUPP)
+ cifs_tcon_dbg(VFS, "%s: ioctl error: rc=%d\n", __func__, rc);
goto out;
}
resp_buftype, rsp_iov);
create_rsp = rsp_iov[0].iov_base;
- if (create_rsp && create_rsp->sync_hdr.Status)
+ if (create_rsp && create_rsp->hdr.Status)
err_iov = rsp_iov[0];
ioctl_rsp = rsp_iov[1].iov_base;
fill_transform_hdr(struct smb2_transform_hdr *tr_hdr, unsigned int orig_len,
struct smb_rqst *old_rq, __le16 cipher_type)
{
- struct smb2_sync_hdr *shdr =
- (struct smb2_sync_hdr *)old_rq->rq_iov[0].iov_base;
+ struct smb2_hdr *shdr =
+ (struct smb2_hdr *)old_rq->rq_iov[0].iov_base;
memset(tr_hdr, 0, sizeof(struct smb2_transform_hdr));
tr_hdr->ProtocolId = SMB2_TRANSFORM_PROTO_NUM;
struct crypto_aead *tfm;
unsigned int crypt_len = le32_to_cpu(tr_hdr->OriginalMessageSize);
- rc = smb2_get_enc_key(server, tr_hdr->SessionId, enc, key);
+ rc = smb2_get_enc_key(server, le64_to_cpu(tr_hdr->SessionId), enc, key);
if (rc) {
cifs_server_dbg(VFS, "%s: Could not get %scryption key\n", __func__,
enc ? "en" : "de");
unsigned int cur_page_idx;
unsigned int pad_len;
struct cifs_readdata *rdata = mid->callback_data;
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)buf;
struct bio_vec *bvec = NULL;
struct iov_iter iter;
struct kvec iov;
{
int ret, length;
char *buf = server->smallbuf;
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
unsigned int pdu_length = server->pdu_size;
unsigned int buf_size;
struct mid_q_entry *mid_entry;
next_is_large = server->large_buf;
one_more:
- shdr = (struct smb2_sync_hdr *)buf;
+ shdr = (struct smb2_hdr *)buf;
if (shdr->NextCommand) {
if (next_is_large)
next_buffer = (char *)cifs_buf_get();
unsigned int orig_len = le32_to_cpu(tr_hdr->OriginalMessageSize);
if (pdu_length < sizeof(struct smb2_transform_hdr) +
- sizeof(struct smb2_sync_hdr)) {
+ sizeof(struct smb2_hdr)) {
cifs_server_dbg(VFS, "Transform message is too small (%u)\n",
pdu_length);
cifs_reconnect(server);
static int
smb2_next_header(char *buf)
{
- struct smb2_sync_hdr *hdr = (struct smb2_sync_hdr *)buf;
+ struct smb2_hdr *hdr = (struct smb2_hdr *)buf;
struct smb2_transform_hdr *t_hdr = (struct smb2_transform_hdr *)buf;
if (hdr->ProtocolId == SMB2_TRANSFORM_PROTO_NUM)
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
.exclusive_lock_type = SMB2_LOCKFLAG_EXCLUSIVE_LOCK,
.shared_lock_type = SMB2_LOCKFLAG_SHARED_LOCK,
.unlock_lock_type = SMB2_LOCKFLAG_UNLOCK,
- .header_size = sizeof(struct smb2_sync_hdr),
+ .header_size = sizeof(struct smb2_hdr),
.header_preamble_size = 0,
.max_header_size = MAX_SMB2_HDR_SIZE,
.read_rsp_size = sizeof(struct smb2_read_rsp) - 1,
#include <linux/uuid.h>
#include <linux/pagemap.h>
#include <linux/xattr.h>
-#include "smb2pdu.h"
#include "cifsglob.h"
#include "cifsacl.h"
#include "cifsproto.h"
}
static void
-smb2_hdr_assemble(struct smb2_sync_hdr *shdr, __le16 smb2_cmd,
+smb2_hdr_assemble(struct smb2_hdr *shdr, __le16 smb2_cmd,
const struct cifs_tcon *tcon,
struct TCP_Server_Info *server)
{
} else {
shdr->CreditRequest = cpu_to_le16(2);
}
- shdr->ProcessId = cpu_to_le32((__u16)current->tgid);
+ shdr->Id.SyncId.ProcessId = cpu_to_le32((__u16)current->tgid);
if (!tcon)
goto out;
shdr->CreditCharge = cpu_to_le16(1);
/* else CreditCharge MBZ */
- shdr->TreeId = tcon->tid;
+ shdr->Id.SyncId.TreeId = cpu_to_le32(tcon->tid);
/* Uid is not converted */
if (tcon->ses)
- shdr->SessionId = tcon->ses->Suid;
+ shdr->SessionId = cpu_to_le64(tcon->ses->Suid);
/*
* If we would set SMB2_FLAGS_DFS_OPERATIONS on open we also would have
if (tcon == NULL)
return 0;
- if (smb2_command == SMB2_TREE_CONNECT)
+ /*
+ * Need to also skip SMB2_IOCTL because it is used for checking nested dfs links in
+ * cifs_tree_connect().
+ */
+ if (smb2_command == SMB2_TREE_CONNECT || smb2_command == SMB2_IOCTL)
return 0;
if (tcon->tidStatus == CifsExiting) {
/*
* If we are reconnecting an extra channel, bind
*/
- if (server->is_channel) {
+ if (CIFS_SERVER_IS_CHAN(server)) {
ses->binding = true;
ses->binding_chan = cifs_ses_find_chan(ses, server);
}
void *buf,
unsigned int *total_len)
{
- struct smb2_sync_pdu *spdu = (struct smb2_sync_pdu *)buf;
+ struct smb2_pdu *spdu = (struct smb2_pdu *)buf;
/* lookup word count ie StructureSize from table */
__u16 parmsize = smb2_req_struct_sizes[le16_to_cpu(smb2_command)];
*/
memset(buf, 0, 256);
- smb2_hdr_assemble(&spdu->sync_hdr, smb2_command, tcon, server);
+ smb2_hdr_assemble(&spdu->hdr, smb2_command, tcon, server);
spdu->StructureSize2 = cpu_to_le16(parmsize);
- *total_len = parmsize + sizeof(struct smb2_sync_hdr);
+ *total_len = parmsize + sizeof(struct smb2_hdr);
}
/*
}
fill_small_buf(smb2_command, tcon, server,
- (struct smb2_sync_hdr *)(*request_buf),
+ (struct smb2_hdr *)(*request_buf),
total_len);
if (tcon != NULL) {
pneg_ctxt->ContextType = SMB2_PREAUTH_INTEGRITY_CAPABILITIES;
pneg_ctxt->DataLength = cpu_to_le16(38);
pneg_ctxt->HashAlgorithmCount = cpu_to_le16(1);
- pneg_ctxt->SaltLength = cpu_to_le16(SMB311_LINUX_CLIENT_SALT_SIZE);
- get_random_bytes(pneg_ctxt->Salt, SMB311_LINUX_CLIENT_SALT_SIZE);
+ pneg_ctxt->SaltLength = cpu_to_le16(SMB311_SALT_SIZE);
+ get_random_bytes(pneg_ctxt->Salt, SMB311_SALT_SIZE);
pneg_ctxt->HashAlgorithms = SMB2_PREAUTH_INTEGRITY_SHA512;
}
if (rc)
return rc;
- req->sync_hdr.SessionId = 0;
+ req->hdr.SessionId = 0;
memset(server->preauth_sha_hash, 0, SMB2_PREAUTH_HASH_SIZE);
memset(ses->preauth_sha_hash, 0, SMB2_PREAUTH_HASH_SIZE);
server->cipher_type = SMB2_ENCRYPTION_AES128_CCM;
security_blob = smb2_get_data_area_len(&blob_offset, &blob_length,
- (struct smb2_sync_hdr *)rsp);
+ (struct smb2_hdr *)rsp);
/*
* See MS-SMB2 section 2.2.4: if no blob, client picks default which
* for us will be
return rc;
if (sess_data->ses->binding) {
- req->sync_hdr.SessionId = sess_data->ses->Suid;
- req->sync_hdr.Flags |= SMB2_FLAGS_SIGNED;
+ req->hdr.SessionId = cpu_to_le64(sess_data->ses->Suid);
+ req->hdr.Flags |= SMB2_FLAGS_SIGNED;
req->PreviousSessionId = 0;
req->Flags = SMB2_SESSION_REQ_FLAG_BINDING;
} else {
/* First session, not a reauthenticate */
- req->sync_hdr.SessionId = 0;
+ req->hdr.SessionId = 0;
/*
* if reconnect, we need to send previous sess id
* otherwise it is 0
*/
- req->PreviousSessionId = sess_data->previous_session;
+ req->PreviousSessionId = cpu_to_le64(sess_data->previous_session);
req->Flags = 0; /* MBZ */
}
/* enough to enable echos and oplocks and one max size write */
- req->sync_hdr.CreditRequest = cpu_to_le16(130);
+ req->hdr.CreditRequest = cpu_to_le16(130);
/* only one of SMB2 signing flags may be set in SMB2 request */
if (server->sign)
rsp = (struct smb2_sess_setup_rsp *)sess_data->iov[0].iov_base;
/* keep session id and flags if binding */
if (!ses->binding) {
- ses->Suid = rsp->sync_hdr.SessionId;
+ ses->Suid = le64_to_cpu(rsp->hdr.SessionId);
ses->session_flags = le16_to_cpu(rsp->SessionFlags);
}
int rc;
struct cifs_ses *ses = sess_data->ses;
struct smb2_sess_setup_rsp *rsp = NULL;
- char *ntlmssp_blob = NULL;
+ unsigned char *ntlmssp_blob = NULL;
bool use_spnego = false; /* else use raw ntlmssp */
u16 blob_length = 0;
if (rc)
goto out_err;
- ntlmssp_blob = kmalloc(sizeof(struct _NEGOTIATE_MESSAGE),
- GFP_KERNEL);
- if (ntlmssp_blob == NULL) {
- rc = -ENOMEM;
- goto out;
- }
+ rc = build_ntlmssp_negotiate_blob(&ntlmssp_blob,
+ &blob_length, ses,
+ sess_data->nls_cp);
+ if (rc)
+ goto out_err;
- build_ntlmssp_negotiate_blob(ntlmssp_blob, ses);
if (use_spnego) {
/* BB eventually need to add this */
cifs_dbg(VFS, "spnego not supported for SMB2 yet\n");
rc = -EOPNOTSUPP;
goto out;
- } else {
- blob_length = sizeof(struct _NEGOTIATE_MESSAGE);
- /* with raw NTLMSSP we don't encapsulate in SPNEGO */
}
sess_data->iov[1].iov_base = ntlmssp_blob;
sess_data->iov[1].iov_len = blob_length;
/* If true, rc here is expected and not an error */
if (sess_data->buf0_type != CIFS_NO_BUFFER &&
- rsp->sync_hdr.Status == STATUS_MORE_PROCESSING_REQUIRED)
+ rsp->hdr.Status == STATUS_MORE_PROCESSING_REQUIRED)
rc = 0;
if (rc)
/* keep existing ses id and flags if binding */
if (!ses->binding) {
- ses->Suid = rsp->sync_hdr.SessionId;
+ ses->Suid = le64_to_cpu(rsp->hdr.SessionId);
ses->session_flags = le16_to_cpu(rsp->SessionFlags);
}
goto out;
req = (struct smb2_sess_setup_req *) sess_data->iov[0].iov_base;
- req->sync_hdr.SessionId = ses->Suid;
+ req->hdr.SessionId = cpu_to_le64(ses->Suid);
rc = build_ntlmssp_auth_blob(&ntlmssp_blob, &blob_length, ses,
sess_data->nls_cp);
/* keep existing ses id and flags if binding */
if (!ses->binding) {
- ses->Suid = rsp->sync_hdr.SessionId;
+ ses->Suid = le64_to_cpu(rsp->hdr.SessionId);
ses->session_flags = le16_to_cpu(rsp->SessionFlags);
}
return rc;
/* since no tcon, smb2_init can not do this, so do here */
- req->sync_hdr.SessionId = ses->Suid;
+ req->hdr.SessionId = cpu_to_le64(ses->Suid);
if (ses->session_flags & SMB2_SESSION_FLAG_ENCRYPT_DATA)
flags |= CIFS_TRANSFORM_REQ;
else if (server->sign)
- req->sync_hdr.Flags |= SMB2_FLAGS_SIGNED;
+ req->hdr.Flags |= SMB2_FLAGS_SIGNED;
flags |= CIFS_NO_RSP_BUF;
!(ses->session_flags &
(SMB2_SESSION_FLAG_IS_GUEST|SMB2_SESSION_FLAG_IS_NULL)) &&
((ses->user_name != NULL) || (ses->sectype == Kerberos)))
- req->sync_hdr.Flags |= SMB2_FLAGS_SIGNED;
+ req->hdr.Flags |= SMB2_FLAGS_SIGNED;
memset(&rqst, 0, sizeof(struct smb_rqst));
rqst.rq_iov = iov;
rqst.rq_nvec = 2;
/* Need 64 for max size write so ask for more in case not there yet */
- req->sync_hdr.CreditRequest = cpu_to_le16(64);
+ req->hdr.CreditRequest = cpu_to_le16(64);
rc = cifs_send_recv(xid, ses, server,
&rqst, &resp_buftype, flags, &rsp_iov);
cifs_small_buf_release(req);
rsp = (struct smb2_tree_connect_rsp *)rsp_iov.iov_base;
trace_smb3_tcon(xid, tcon->tid, ses->Suid, tree, rc);
- if (rc != 0) {
+ if ((rc != 0) || (rsp == NULL)) {
cifs_stats_fail_inc(tcon, SMB2_TREE_CONNECT_HE);
tcon->need_reconnect = true;
goto tcon_error_exit;
tcon->maximal_access = le32_to_cpu(rsp->MaximalAccess);
tcon->tidStatus = CifsGood;
tcon->need_reconnect = false;
- tcon->tid = rsp->sync_hdr.TreeId;
+ tcon->tid = le32_to_cpu(rsp->hdr.Id.SyncId.TreeId);
strlcpy(tcon->treeName, tree, sizeof(tcon->treeName));
if ((rsp->Capabilities & SMB2_SHARE_CAP_DFS) &&
return rc;
tcon_error_exit:
- if (rsp && rsp->sync_hdr.Status == STATUS_BAD_NETWORK_NAME) {
+ if (rsp && rsp->hdr.Status == STATUS_BAD_NETWORK_NAME)
cifs_tcon_dbg(VFS, "BAD_NETWORK_NAME: %s\n", tree);
- }
goto tcon_exit;
}
if (tcon->share_flags & SHI1005_FLAGS_DFS) {
int name_len;
- req->sync_hdr.Flags |= SMB2_FLAGS_DFS_OPERATIONS;
+ req->hdr.Flags |= SMB2_FLAGS_DFS_OPERATIONS;
rc = alloc_path_with_tree_prefix(©_path, ©_size,
&name_len,
tcon->treeName, utf16_path);
goto err_free_rsp_buf;
}
+ /*
+ * Although unlikely to be possible for rsp to be null and rc not set,
+ * adding check below is slightly safer long term (and quiets Coverity
+ * warning)
+ */
rsp = (struct smb2_create_rsp *)rsp_iov.iov_base;
- trace_smb3_posix_mkdir_done(xid, rsp->PersistentFileId, tcon->tid,
+ if (rsp == NULL) {
+ rc = -EIO;
+ kfree(pc_buf);
+ goto err_free_req;
+ }
+
+ trace_smb3_posix_mkdir_done(xid, le64_to_cpu(rsp->PersistentFileId),
+ tcon->tid,
ses->Suid, CREATE_NOT_FILE,
FILE_WRITE_ATTRIBUTES);
- SMB2_close(xid, tcon, rsp->PersistentFileId, rsp->VolatileFileId);
+ SMB2_close(xid, tcon, le64_to_cpu(rsp->PersistentFileId),
+ le64_to_cpu(rsp->VolatileFileId));
/* Eventually save off posix specific response info and timestaps */
if (tcon->share_flags & SHI1005_FLAGS_DFS) {
int name_len;
- req->sync_hdr.Flags |= SMB2_FLAGS_DFS_OPERATIONS;
+ req->hdr.Flags |= SMB2_FLAGS_DFS_OPERATIONS;
rc = alloc_path_with_tree_prefix(©_path, ©_size,
&name_len,
tcon->treeName, path);
tcon->need_reconnect = true;
}
goto creat_exit;
- } else
- trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid,
+ } else if (rsp == NULL) /* unlikely to happen, but safer to check */
+ goto creat_exit;
+ else
+ trace_smb3_open_done(xid, le64_to_cpu(rsp->PersistentFileId),
+ tcon->tid,
ses->Suid, oparms->create_options,
oparms->desired_access);
atomic_inc(&tcon->num_remote_opens);
- oparms->fid->persistent_fid = rsp->PersistentFileId;
- oparms->fid->volatile_fid = rsp->VolatileFileId;
+ oparms->fid->persistent_fid = le64_to_cpu(rsp->PersistentFileId);
+ oparms->fid->volatile_fid = le64_to_cpu(rsp->VolatileFileId);
oparms->fid->access = oparms->desired_access;
#ifdef CONFIG_CIFS_DEBUG2
- oparms->fid->mid = le64_to_cpu(rsp->sync_hdr.MessageId);
+ oparms->fid->mid = le64_to_cpu(rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
if (buf) {
* response size smaller.
*/
req->MaxOutputResponse = cpu_to_le32(max_response_size);
- req->sync_hdr.CreditCharge =
+ req->hdr.CreditCharge =
cpu_to_le16(DIV_ROUND_UP(max(indatalen, max_response_size),
SMB2_MAX_BUFFER_SIZE));
if (is_fsctl)
/* validate negotiate request must be signed - see MS-SMB2 3.2.5.5 */
if (opcode == FSCTL_VALIDATE_NEGOTIATE_INFO)
- req->sync_hdr.Flags |= SMB2_FLAGS_SIGNED;
+ req->hdr.Flags |= SMB2_FLAGS_SIGNED;
return 0;
}
if ((plen == NULL) || (out_data == NULL))
goto ioctl_exit;
+ /*
+ * Although unlikely to be possible for rsp to be null and rc not set,
+ * adding check below is slightly safer long term (and quiets Coverity
+ * warning)
+ */
+ if (rsp == NULL) {
+ rc = -EIO;
+ goto ioctl_exit;
+ }
+
*plen = le32_to_cpu(rsp->OutputCount);
/* We check for obvious errors in the output buffer length and offset */
if (rc)
return rc;
- req->PersistentFileId = persistent_fid;
- req->VolatileFileId = volatile_fid;
+ req->PersistentFileId = cpu_to_le64(persistent_fid);
+ req->VolatileFileId = cpu_to_le64(volatile_fid);
if (query_attrs)
req->Flags = SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB;
else
if (rc)
return rc;
- req->PersistentFileId = persistent_fid;
- req->VolatileFileId = volatile_fid;
+ req->PersistentFileId = cpu_to_le64(persistent_fid);
+ req->VolatileFileId = cpu_to_le64(volatile_fid);
/* See note 354 of MS-SMB2, 64K max */
req->OutputBufferLength =
cpu_to_le32(SMB2_MAX_BUFFER_SIZE - MAX_SMB2_HDR_SIZE);
if (mid->mid_state == MID_RESPONSE_RECEIVED
|| mid->mid_state == MID_RESPONSE_MALFORMED) {
- credits.value = le16_to_cpu(rsp->sync_hdr.CreditRequest);
+ credits.value = le16_to_cpu(rsp->hdr.CreditRequest);
credits.instance = server->reconnect_instance;
}
if (rc)
return rc;
- req->sync_hdr.CreditRequest = cpu_to_le16(1);
+ req->hdr.CreditRequest = cpu_to_le16(1);
iov[0].iov_len = total_len;
iov[0].iov_base = (char *)req;
if (rc)
return rc;
- req->PersistentFileId = persistent_fid;
- req->VolatileFileId = volatile_fid;
+ req->PersistentFileId = cpu_to_le64(persistent_fid);
+ req->VolatileFileId = cpu_to_le64(volatile_fid);
iov[0].iov_base = (char *)req;
iov[0].iov_len = total_len;
unsigned int remaining_bytes, int request_type)
{
int rc = -EACCES;
- struct smb2_read_plain_req *req = NULL;
- struct smb2_sync_hdr *shdr;
+ struct smb2_read_req *req = NULL;
+ struct smb2_hdr *shdr;
struct TCP_Server_Info *server = io_parms->server;
rc = smb2_plain_req_init(SMB2_READ, io_parms->tcon, server,
if (server == NULL)
return -ECONNABORTED;
- shdr = &req->sync_hdr;
- shdr->ProcessId = cpu_to_le32(io_parms->pid);
+ shdr = &req->hdr;
+ shdr->Id.SyncId.ProcessId = cpu_to_le32(io_parms->pid);
- req->PersistentFileId = io_parms->persistent_fid;
- req->VolatileFileId = io_parms->volatile_fid;
+ req->PersistentFileId = cpu_to_le64(io_parms->persistent_fid);
+ req->VolatileFileId = cpu_to_le64(io_parms->volatile_fid);
req->ReadChannelInfoOffset = 0; /* reserved */
req->ReadChannelInfoLength = 0; /* reserved */
req->Channel = 0; /* reserved */
if (need_invalidate)
req->Channel = SMB2_CHANNEL_RDMA_V1;
req->ReadChannelInfoOffset =
- cpu_to_le16(offsetof(struct smb2_read_plain_req, Buffer));
+ cpu_to_le16(offsetof(struct smb2_read_req, Buffer));
req->ReadChannelInfoLength =
cpu_to_le16(sizeof(struct smbd_buffer_descriptor_v1));
v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0];
* Related requests use info from previous read request
* in chain.
*/
- shdr->SessionId = 0xFFFFFFFFFFFFFFFF;
- shdr->TreeId = 0xFFFFFFFF;
- req->PersistentFileId = 0xFFFFFFFFFFFFFFFF;
- req->VolatileFileId = 0xFFFFFFFFFFFFFFFF;
+ shdr->SessionId = cpu_to_le64(0xFFFFFFFFFFFFFFFF);
+ shdr->Id.SyncId.TreeId = cpu_to_le32(0xFFFFFFFF);
+ req->PersistentFileId = cpu_to_le64(0xFFFFFFFFFFFFFFFF);
+ req->VolatileFileId = cpu_to_le64(0xFFFFFFFFFFFFFFFF);
}
}
if (remaining_bytes > io_parms->length)
struct cifs_readdata *rdata = mid->callback_data;
struct cifs_tcon *tcon = tlink_tcon(rdata->cfile->tlink);
struct TCP_Server_Info *server = rdata->server;
- struct smb2_sync_hdr *shdr =
- (struct smb2_sync_hdr *)rdata->iov[0].iov_base;
+ struct smb2_hdr *shdr =
+ (struct smb2_hdr *)rdata->iov[0].iov_base;
struct cifs_credits credits = { .value = 0, .instance = 0 };
struct smb_rqst rqst = { .rq_iov = &rdata->iov[1],
.rq_nvec = 1,
{
int rc, flags = 0;
char *buf;
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
struct cifs_io_parms io_parms;
struct smb_rqst rqst = { .rq_iov = rdata->iov,
.rq_nvec = 1 };
rdata->iov[0].iov_base = buf;
rdata->iov[0].iov_len = total_len;
- shdr = (struct smb2_sync_hdr *)buf;
+ shdr = (struct smb2_hdr *)buf;
if (rdata->credits.value > 0) {
shdr->CreditCharge = cpu_to_le16(DIV_ROUND_UP(rdata->bytes,
{
struct smb_rqst rqst;
int resp_buftype, rc;
- struct smb2_read_plain_req *req = NULL;
+ struct smb2_read_req *req = NULL;
struct smb2_read_rsp *rsp = NULL;
struct kvec iov[1];
struct kvec rsp_iov;
if (rc != -ENODATA) {
cifs_stats_fail_inc(io_parms->tcon, SMB2_READ_HE);
cifs_dbg(VFS, "Send error in read = %d\n", rc);
- trace_smb3_read_err(xid, req->PersistentFileId,
+ trace_smb3_read_err(xid,
+ le64_to_cpu(req->PersistentFileId),
io_parms->tcon->tid, ses->Suid,
io_parms->offset, io_parms->length,
rc);
} else
- trace_smb3_read_done(xid, req->PersistentFileId,
- io_parms->tcon->tid, ses->Suid,
- io_parms->offset, 0);
+ trace_smb3_read_done(xid,
+ le64_to_cpu(req->PersistentFileId),
+ io_parms->tcon->tid, ses->Suid,
+ io_parms->offset, 0);
free_rsp_buf(resp_buftype, rsp_iov.iov_base);
cifs_small_buf_release(req);
return rc == -ENODATA ? 0 : rc;
} else
- trace_smb3_read_done(xid, req->PersistentFileId,
+ trace_smb3_read_done(xid,
+ le64_to_cpu(req->PersistentFileId),
io_parms->tcon->tid, ses->Suid,
io_parms->offset, io_parms->length);
switch (mid->mid_state) {
case MID_RESPONSE_RECEIVED:
- credits.value = le16_to_cpu(rsp->sync_hdr.CreditRequest);
+ credits.value = le16_to_cpu(rsp->hdr.CreditRequest);
credits.instance = server->reconnect_instance;
wdata->result = smb2_check_receive(mid, server, 0);
if (wdata->result != 0)
wdata->result = -EAGAIN;
break;
case MID_RESPONSE_MALFORMED:
- credits.value = le16_to_cpu(rsp->sync_hdr.CreditRequest);
+ credits.value = le16_to_cpu(rsp->hdr.CreditRequest);
credits.instance = server->reconnect_instance;
fallthrough;
default:
{
int rc = -EACCES, flags = 0;
struct smb2_write_req *req = NULL;
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
struct cifs_tcon *tcon = tlink_tcon(wdata->cfile->tlink);
struct TCP_Server_Info *server = wdata->server;
struct kvec iov[1];
if (smb3_encryption_required(tcon))
flags |= CIFS_TRANSFORM_REQ;
- shdr = (struct smb2_sync_hdr *)req;
- shdr->ProcessId = cpu_to_le32(wdata->cfile->pid);
+ shdr = (struct smb2_hdr *)req;
+ shdr->Id.SyncId.ProcessId = cpu_to_le32(wdata->cfile->pid);
- req->PersistentFileId = wdata->cfile->fid.persistent_fid;
- req->VolatileFileId = wdata->cfile->fid.volatile_fid;
+ req->PersistentFileId = cpu_to_le64(wdata->cfile->fid.persistent_fid);
+ req->VolatileFileId = cpu_to_le64(wdata->cfile->fid.volatile_fid);
req->WriteChannelInfoOffset = 0;
req->WriteChannelInfoLength = 0;
req->Channel = 0;
wdata, flags, &wdata->credits);
if (rc) {
- trace_smb3_write_err(0 /* no xid */, req->PersistentFileId,
+ trace_smb3_write_err(0 /* no xid */,
+ le64_to_cpu(req->PersistentFileId),
tcon->tid, tcon->ses->Suid, wdata->offset,
wdata->bytes, rc);
kref_put(&wdata->refcount, release);
if (smb3_encryption_required(io_parms->tcon))
flags |= CIFS_TRANSFORM_REQ;
- req->sync_hdr.ProcessId = cpu_to_le32(io_parms->pid);
+ req->hdr.Id.SyncId.ProcessId = cpu_to_le32(io_parms->pid);
- req->PersistentFileId = io_parms->persistent_fid;
- req->VolatileFileId = io_parms->volatile_fid;
+ req->PersistentFileId = cpu_to_le64(io_parms->persistent_fid);
+ req->VolatileFileId = cpu_to_le64(io_parms->volatile_fid);
req->WriteChannelInfoOffset = 0;
req->WriteChannelInfoLength = 0;
req->Channel = 0;
rsp = (struct smb2_write_rsp *)rsp_iov.iov_base;
if (rc) {
- trace_smb3_write_err(xid, req->PersistentFileId,
+ trace_smb3_write_err(xid,
+ le64_to_cpu(req->PersistentFileId),
io_parms->tcon->tid,
io_parms->tcon->ses->Suid,
io_parms->offset, io_parms->length, rc);
cifs_dbg(VFS, "Send error in write = %d\n", rc);
} else {
*nbytes = le32_to_cpu(rsp->DataLength);
- trace_smb3_write_done(xid, req->PersistentFileId,
- io_parms->tcon->tid,
- io_parms->tcon->ses->Suid,
- io_parms->offset, *nbytes);
+ trace_smb3_write_done(xid,
+ le64_to_cpu(req->PersistentFileId),
+ io_parms->tcon->tid,
+ io_parms->tcon->ses->Suid,
+ io_parms->offset, *nbytes);
}
cifs_small_buf_release(req);
if (rc) {
if (rc == -ENODATA &&
- rsp->sync_hdr.Status == STATUS_NO_MORE_FILES) {
+ rsp->hdr.Status == STATUS_NO_MORE_FILES) {
trace_smb3_query_dir_done(xid, persistent_fid,
tcon->tid, tcon->ses->Suid, index, 0);
srch_inf->endOfSearch = true;
if (rc)
return rc;
- req->sync_hdr.ProcessId = cpu_to_le32(pid);
+ req->hdr.Id.SyncId.ProcessId = cpu_to_le32(pid);
req->InfoType = info_type;
req->FileInfoClass = info_class;
req->PersistentFileId = persistent_fid;
req->VolatileFid = volatile_fid;
req->PersistentFid = persistent_fid;
req->OplockLevel = oplock_level;
- req->sync_hdr.CreditRequest = cpu_to_le16(1);
+ req->hdr.CreditRequest = cpu_to_le16(1);
flags |= CIFS_NO_RSP_BUF;
if (smb3_encryption_required(tcon))
flags |= CIFS_TRANSFORM_REQ;
- req->sync_hdr.ProcessId = cpu_to_le32(pid);
+ req->hdr.Id.SyncId.ProcessId = cpu_to_le32(pid);
req->LockCount = cpu_to_le16(num_lock);
req->PersistentFileId = persist_fid;
if (smb3_encryption_required(tcon))
flags |= CIFS_TRANSFORM_REQ;
- req->sync_hdr.CreditRequest = cpu_to_le16(1);
+ req->hdr.CreditRequest = cpu_to_le16(1);
req->StructureSize = cpu_to_le16(36);
total_len += 12;
#include <net/sock.h>
#include "cifsacl.h"
-/*
- * Note that, due to trying to use names similar to the protocol specifications,
- * there are many mixed case field names in the structures below. Although
- * this does not match typical Linux kernel style, it is necessary to be
- * able to match against the protocol specfication.
- *
- * SMB2 commands
- * Some commands have minimal (wct=0,bcc=0), or uninteresting, responses
- * (ie no useful data other than the SMB error code itself) and are marked such.
- * Knowing this helps avoid response buffer allocations and copy in some cases.
- */
-
-/* List of commands in host endian */
-#define SMB2_NEGOTIATE_HE 0x0000
-#define SMB2_SESSION_SETUP_HE 0x0001
-#define SMB2_LOGOFF_HE 0x0002 /* trivial request/resp */
-#define SMB2_TREE_CONNECT_HE 0x0003
-#define SMB2_TREE_DISCONNECT_HE 0x0004 /* trivial req/resp */
-#define SMB2_CREATE_HE 0x0005
-#define SMB2_CLOSE_HE 0x0006
-#define SMB2_FLUSH_HE 0x0007 /* trivial resp */
-#define SMB2_READ_HE 0x0008
-#define SMB2_WRITE_HE 0x0009
-#define SMB2_LOCK_HE 0x000A
-#define SMB2_IOCTL_HE 0x000B
-#define SMB2_CANCEL_HE 0x000C
-#define SMB2_ECHO_HE 0x000D
-#define SMB2_QUERY_DIRECTORY_HE 0x000E
-#define SMB2_CHANGE_NOTIFY_HE 0x000F
-#define SMB2_QUERY_INFO_HE 0x0010
-#define SMB2_SET_INFO_HE 0x0011
-#define SMB2_OPLOCK_BREAK_HE 0x0012
-
-/* The same list in little endian */
-#define SMB2_NEGOTIATE cpu_to_le16(SMB2_NEGOTIATE_HE)
-#define SMB2_SESSION_SETUP cpu_to_le16(SMB2_SESSION_SETUP_HE)
-#define SMB2_LOGOFF cpu_to_le16(SMB2_LOGOFF_HE)
-#define SMB2_TREE_CONNECT cpu_to_le16(SMB2_TREE_CONNECT_HE)
-#define SMB2_TREE_DISCONNECT cpu_to_le16(SMB2_TREE_DISCONNECT_HE)
-#define SMB2_CREATE cpu_to_le16(SMB2_CREATE_HE)
-#define SMB2_CLOSE cpu_to_le16(SMB2_CLOSE_HE)
-#define SMB2_FLUSH cpu_to_le16(SMB2_FLUSH_HE)
-#define SMB2_READ cpu_to_le16(SMB2_READ_HE)
-#define SMB2_WRITE cpu_to_le16(SMB2_WRITE_HE)
-#define SMB2_LOCK cpu_to_le16(SMB2_LOCK_HE)
-#define SMB2_IOCTL cpu_to_le16(SMB2_IOCTL_HE)
-#define SMB2_CANCEL cpu_to_le16(SMB2_CANCEL_HE)
-#define SMB2_ECHO cpu_to_le16(SMB2_ECHO_HE)
-#define SMB2_QUERY_DIRECTORY cpu_to_le16(SMB2_QUERY_DIRECTORY_HE)
-#define SMB2_CHANGE_NOTIFY cpu_to_le16(SMB2_CHANGE_NOTIFY_HE)
-#define SMB2_QUERY_INFO cpu_to_le16(SMB2_QUERY_INFO_HE)
-#define SMB2_SET_INFO cpu_to_le16(SMB2_SET_INFO_HE)
-#define SMB2_OPLOCK_BREAK cpu_to_le16(SMB2_OPLOCK_BREAK_HE)
-
-#define SMB2_INTERNAL_CMD cpu_to_le16(0xFFFF)
-
-#define NUMBER_OF_SMB2_COMMANDS 0x0013
-
/* 52 transform hdr + 64 hdr + 88 create rsp */
#define SMB2_TRANSFORM_HEADER_SIZE 52
#define MAX_SMB2_HDR_SIZE 204
-#define SMB2_PROTO_NUMBER cpu_to_le32(0x424d53fe)
-#define SMB2_TRANSFORM_PROTO_NUM cpu_to_le32(0x424d53fd)
-#define SMB2_COMPRESSION_TRANSFORM_ID cpu_to_le32(0x424d53fc)
-
-/*
- * SMB2 Header Definition
- *
- * "MBZ" : Must be Zero
- * "BB" : BugBug, Something to check/review/analyze later
- * "PDU" : "Protocol Data Unit" (ie a network "frame")
- *
- */
-
-#define SMB2_HEADER_STRUCTURE_SIZE cpu_to_le16(64)
-
-struct smb2_sync_hdr {
- __le32 ProtocolId; /* 0xFE 'S' 'M' 'B' */
- __le16 StructureSize; /* 64 */
- __le16 CreditCharge; /* MBZ */
- __le32 Status; /* Error from server */
- __le16 Command;
- __le16 CreditRequest; /* CreditResponse */
- __le32 Flags;
- __le32 NextCommand;
- __le64 MessageId;
- __le32 ProcessId;
- __u32 TreeId; /* opaque - so do not make little endian */
- __u64 SessionId; /* opaque - so do not make little endian */
- __u8 Signature[16];
-} __packed;
-
/* The total header size for SMB2 read and write */
-#define SMB2_READWRITE_PDU_HEADER_SIZE (48 + sizeof(struct smb2_sync_hdr))
-
-struct smb2_sync_pdu {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize2; /* size of wct area (varies, request specific) */
-} __packed;
-
-#define SMB3_AES_CCM_NONCE 11
-#define SMB3_AES_GCM_NONCE 12
-
-/* Transform flags (for 3.0 dialect this flag indicates CCM */
-#define TRANSFORM_FLAG_ENCRYPTED 0x0001
-struct smb2_transform_hdr {
- __le32 ProtocolId; /* 0xFD 'S' 'M' 'B' */
- __u8 Signature[16];
- __u8 Nonce[16];
- __le32 OriginalMessageSize;
- __u16 Reserved1;
- __le16 Flags; /* EncryptionAlgorithm for 3.0, enc enabled for 3.1.1 */
- __u64 SessionId;
-} __packed;
-
-/* See MS-SMB2 2.2.42 */
-struct smb2_compression_transform_hdr_unchained {
- __le32 ProtocolId; /* 0xFC 'S' 'M' 'B' */
- __le32 OriginalCompressedSegmentSize;
- __le16 CompressionAlgorithm;
- __le16 Flags;
- __le16 Length; /* if chained it is length, else offset */
-} __packed;
-
-/* See MS-SMB2 2.2.42.1 */
-#define SMB2_COMPRESSION_FLAG_NONE 0x0000
-#define SMB2_COMPRESSION_FLAG_CHAINED 0x0001
-
-struct compression_payload_header {
- __le16 CompressionAlgorithm;
- __le16 Flags;
- __le32 Length; /* length of compressed playload including field below if present */
- /* __le32 OriginalPayloadSize; */ /* optional, present when LZNT1, LZ77, LZ77+Huffman */
-} __packed;
-
-/* See MS-SMB2 2.2.42.2 */
-struct smb2_compression_transform_hdr_chained {
- __le32 ProtocolId; /* 0xFC 'S' 'M' 'B' */
- __le32 OriginalCompressedSegmentSize;
- /* struct compression_payload_header[] */
-} __packed;
-
-/* See MS-SMB2 2.2.42.2.2 */
-struct compression_pattern_payload_v1 {
- __le16 Pattern;
- __le16 Reserved1;
- __le16 Reserved2;
- __le32 Repetitions;
-} __packed;
+#define SMB2_READWRITE_PDU_HEADER_SIZE (48 + sizeof(struct smb2_hdr))
/* See MS-SMB2 2.2.43 */
struct smb2_rdma_transform {
} __packed;
/*
- * SMB2 flag definitions
- */
-#define SMB2_FLAGS_SERVER_TO_REDIR cpu_to_le32(0x00000001)
-#define SMB2_FLAGS_ASYNC_COMMAND cpu_to_le32(0x00000002)
-#define SMB2_FLAGS_RELATED_OPERATIONS cpu_to_le32(0x00000004)
-#define SMB2_FLAGS_SIGNED cpu_to_le32(0x00000008)
-#define SMB2_FLAGS_PRIORITY_MASK cpu_to_le32(0x00000070) /* SMB3.1.1 */
-#define SMB2_FLAGS_DFS_OPERATIONS cpu_to_le32(0x10000000)
-#define SMB2_FLAGS_REPLAY_OPERATION cpu_to_le32(0x20000000) /* SMB3 & up */
-
-/*
* Definitions for SMB2 Protocol Data Units (network frames)
*
* See MS-SMB2.PDF specification for protocol details.
#define SMB2_ERROR_STRUCTURE_SIZE2 cpu_to_le16(9)
struct smb2_err_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize;
__le16 Reserved; /* MBZ */
__le32 ByteCount; /* even if zero, at least one byte follows */
/* __u8 ResourceName[] */ /* Name of share as counted Unicode string */
} __packed;
-#define SMB2_CLIENT_GUID_SIZE 16
-
-struct smb2_negotiate_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 36 */
- __le16 DialectCount;
- __le16 SecurityMode;
- __le16 Reserved; /* MBZ */
- __le32 Capabilities;
- __u8 ClientGUID[SMB2_CLIENT_GUID_SIZE];
- /* In SMB3.02 and earlier next three were MBZ le64 ClientStartTime */
- __le32 NegotiateContextOffset; /* SMB3.1.1 only. MBZ earlier */
- __le16 NegotiateContextCount; /* SMB3.1.1 only. MBZ earlier */
- __le16 Reserved2;
- __le16 Dialects[4]; /* BB expand this if autonegotiate > 4 dialects */
-} __packed;
-
-/* Dialects */
-#define SMB10_PROT_ID 0x0000 /* local only, not sent on wire w/CIFS negprot */
-#define SMB20_PROT_ID 0x0202
-#define SMB21_PROT_ID 0x0210
-#define SMB30_PROT_ID 0x0300
-#define SMB302_PROT_ID 0x0302
-#define SMB311_PROT_ID 0x0311
-#define BAD_PROT_ID 0xFFFF
-
-/* SecurityMode flags */
-#define SMB2_NEGOTIATE_SIGNING_ENABLED 0x0001
-#define SMB2_NEGOTIATE_SIGNING_REQUIRED 0x0002
-#define SMB2_SEC_MODE_FLAGS_ALL 0x0003
-
-/* Capabilities flags */
-#define SMB2_GLOBAL_CAP_DFS 0x00000001
-#define SMB2_GLOBAL_CAP_LEASING 0x00000002 /* Resp only New to SMB2.1 */
-#define SMB2_GLOBAL_CAP_LARGE_MTU 0X00000004 /* Resp only New to SMB2.1 */
-#define SMB2_GLOBAL_CAP_MULTI_CHANNEL 0x00000008 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_PERSISTENT_HANDLES 0x00000010 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_DIRECTORY_LEASING 0x00000020 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_ENCRYPTION 0x00000040 /* New to SMB3 */
-/* Internal types */
-#define SMB2_NT_FIND 0x00100000
-#define SMB2_LARGE_FILES 0x00200000
-
-
-/* Negotiate Contexts - ContextTypes. See MS-SMB2 section 2.2.3.1 for details */
-#define SMB2_PREAUTH_INTEGRITY_CAPABILITIES cpu_to_le16(1)
-#define SMB2_ENCRYPTION_CAPABILITIES cpu_to_le16(2)
-#define SMB2_COMPRESSION_CAPABILITIES cpu_to_le16(3)
-#define SMB2_NETNAME_NEGOTIATE_CONTEXT_ID cpu_to_le16(5)
-#define SMB2_TRANSPORT_CAPABILITIES cpu_to_le16(6)
-#define SMB2_RDMA_TRANSFORM_CAPABILITIES cpu_to_le16(7)
-#define SMB2_SIGNING_CAPABILITIES cpu_to_le16(8)
-#define SMB2_POSIX_EXTENSIONS_AVAILABLE cpu_to_le16(0x100)
-
-struct smb2_neg_context {
- __le16 ContextType;
- __le16 DataLength;
- __le32 Reserved;
- /* Followed by array of data. NOTE: some servers require padding to 8 byte boundary */
-} __packed;
-
-#define SMB311_LINUX_CLIENT_SALT_SIZE 32
-/* Hash Algorithm Types */
-#define SMB2_PREAUTH_INTEGRITY_SHA512 cpu_to_le16(0x0001)
-#define SMB2_PREAUTH_HASH_SIZE 64
-
-/*
- * SaltLength that the server send can be zero, so the only three required
- * fields (all __le16) end up six bytes total, so the minimum context data len
- * in the response is six bytes which accounts for
- *
- * HashAlgorithmCount, SaltLength, and 1 HashAlgorithm.
- */
-#define MIN_PREAUTH_CTXT_DATA_LEN 6
-
-struct smb2_preauth_neg_context {
- __le16 ContextType; /* 1 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 HashAlgorithmCount; /* 1 */
- __le16 SaltLength;
- __le16 HashAlgorithms; /* HashAlgorithms[0] since only one defined */
- __u8 Salt[SMB311_LINUX_CLIENT_SALT_SIZE];
-} __packed;
-
-/* Encryption Algorithms Ciphers */
-#define SMB2_ENCRYPTION_AES128_CCM cpu_to_le16(0x0001)
-#define SMB2_ENCRYPTION_AES128_GCM cpu_to_le16(0x0002)
-/* we currently do not request AES256_CCM since presumably GCM faster */
-#define SMB2_ENCRYPTION_AES256_CCM cpu_to_le16(0x0003)
-#define SMB2_ENCRYPTION_AES256_GCM cpu_to_le16(0x0004)
-
-/* Min encrypt context data is one cipher so 2 bytes + 2 byte count field */
-#define MIN_ENCRYPT_CTXT_DATA_LEN 4
-struct smb2_encryption_neg_context {
- __le16 ContextType; /* 2 */
- __le16 DataLength;
- __le32 Reserved;
- /* CipherCount usally 2, but can be 3 when AES256-GCM enabled */
- __le16 CipherCount; /* AES128-GCM and AES128-CCM by default */
- __le16 Ciphers[3];
-} __packed;
-
-/* See MS-SMB2 2.2.3.1.3 */
-#define SMB3_COMPRESS_NONE cpu_to_le16(0x0000)
-#define SMB3_COMPRESS_LZNT1 cpu_to_le16(0x0001)
-#define SMB3_COMPRESS_LZ77 cpu_to_le16(0x0002)
-#define SMB3_COMPRESS_LZ77_HUFF cpu_to_le16(0x0003)
-/* Pattern scanning algorithm See MS-SMB2 3.1.4.4.1 */
-#define SMB3_COMPRESS_PATTERN cpu_to_le16(0x0004) /* Pattern_V1 */
-
-/* Compression Flags */
-#define SMB2_COMPRESSION_CAPABILITIES_FLAG_NONE cpu_to_le32(0x00000000)
-#define SMB2_COMPRESSION_CAPABILITIES_FLAG_CHAINED cpu_to_le32(0x00000001)
-
-struct smb2_compression_capabilities_context {
- __le16 ContextType; /* 3 */
- __le16 DataLength;
- __u32 Reserved;
- __le16 CompressionAlgorithmCount;
- __u16 Padding;
- __u32 Flags;
- __le16 CompressionAlgorithms[3];
- __u16 Pad; /* Some servers require pad to DataLen multiple of 8 */
- /* Check if pad needed */
-} __packed;
-
-/*
- * For smb2_netname_negotiate_context_id See MS-SMB2 2.2.3.1.4.
- * Its struct simply contains NetName, an array of Unicode characters
- */
-struct smb2_netname_neg_context {
- __le16 ContextType; /* 5 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 NetName[]; /* hostname of target converted to UCS-2 */
-} __packed;
-
-/*
- * For smb2_transport_capabilities context see MS-SMB2 2.2.3.1.5
- * and 2.2.4.1.5
- */
-
-/* Flags */
-#define SMB2_ACCEPT_TRANSFORM_LEVEL_SECURITY 0x00000001
-
-struct smb2_transport_capabilities_context {
- __le16 ContextType; /* 6 */
- __le16 DataLength;
- __u32 Reserved;
- __le32 Flags;
- __u32 Pad;
-} __packed;
-
-/*
- * For rdma transform capabilities context see MS-SMB2 2.2.3.1.6
- * and 2.2.4.1.6
- */
-
-/* RDMA Transform IDs */
-#define SMB2_RDMA_TRANSFORM_NONE 0x0000
-#define SMB2_RDMA_TRANSFORM_ENCRYPTION 0x0001
-#define SMB2_RDMA_TRANSFORM_SIGNING 0x0002
-
-struct smb2_rdma_transform_capabilities_context {
- __le16 ContextType; /* 7 */
- __le16 DataLength;
- __u32 Reserved;
- __le16 TransformCount;
- __u16 Reserved1;
- __u32 Reserved2;
- __le16 RDMATransformIds[];
-} __packed;
-
-/*
- * For signing capabilities context see MS-SMB2 2.2.3.1.7
- * and 2.2.4.1.7
- */
-
-/* Signing algorithms */
-#define SIGNING_ALG_HMAC_SHA256 0
-#define SIGNING_ALG_AES_CMAC 1
-#define SIGNING_ALG_AES_GMAC 2
-
-struct smb2_signing_capabilities {
- __le16 ContextType; /* 8 */
- __le16 DataLength;
- __u32 Reserved;
- __le16 SigningAlgorithmCount;
- __le16 SigningAlgorithms[];
- /* Followed by padding to 8 byte boundary (required by some servers) */
-} __packed;
-
-#define POSIX_CTXT_DATA_LEN 16
-struct smb2_posix_neg_context {
- __le16 ContextType; /* 0x100 */
- __le16 DataLength;
- __le32 Reserved;
- __u8 Name[16]; /* POSIX ctxt GUID 93AD25509CB411E7B42383DE968BCD7C */
-} __packed;
-
-struct smb2_negotiate_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 65 */
- __le16 SecurityMode;
- __le16 DialectRevision;
- __le16 NegotiateContextCount; /* Prior to SMB3.1.1 was Reserved & MBZ */
- __u8 ServerGUID[16];
- __le32 Capabilities;
- __le32 MaxTransactSize;
- __le32 MaxReadSize;
- __le32 MaxWriteSize;
- __le64 SystemTime; /* MBZ */
- __le64 ServerStartTime;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __le32 NegotiateContextOffset; /* Pre:SMB3.1.1 was reserved/ignored */
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-/* Flags */
-#define SMB2_SESSION_REQ_FLAG_BINDING 0x01
-#define SMB2_SESSION_REQ_FLAG_ENCRYPT_DATA 0x04
-
-struct smb2_sess_setup_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 25 */
- __u8 Flags;
- __u8 SecurityMode;
- __le32 Capabilities;
- __le32 Channel;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __u64 PreviousSessionId;
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-/* Currently defined SessionFlags */
-#define SMB2_SESSION_FLAG_IS_GUEST 0x0001
-#define SMB2_SESSION_FLAG_IS_NULL 0x0002
-#define SMB2_SESSION_FLAG_ENCRYPT_DATA 0x0004
-struct smb2_sess_setup_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 SessionFlags;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-struct smb2_logoff_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-struct smb2_logoff_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-/* Flags/Reserved for SMB3.1.1 */
-#define SMB2_TREE_CONNECT_FLAG_CLUSTER_RECONNECT cpu_to_le16(0x0001)
-#define SMB2_TREE_CONNECT_FLAG_REDIRECT_TO_OWNER cpu_to_le16(0x0002)
-#define SMB2_TREE_CONNECT_FLAG_EXTENSION_PRESENT cpu_to_le16(0x0004)
-
-struct smb2_tree_connect_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 Flags; /* Reserved MBZ for dialects prior to SMB3.1.1 */
- __le16 PathOffset;
- __le16 PathLength;
- __u8 Buffer[1]; /* variable length */
-} __packed;
-
-/* See MS-SMB2 section 2.2.9.2 */
-/* Context Types */
-#define SMB2_RESERVED_TREE_CONNECT_CONTEXT_ID 0x0000
-#define SMB2_REMOTED_IDENTITY_TREE_CONNECT_CONTEXT_ID cpu_to_le16(0x0001)
-
-struct tree_connect_contexts {
- __le16 ContextType;
- __le16 DataLength;
- __le32 Reserved;
- __u8 Data[];
-} __packed;
-
-/* Remoted identity tree connect context structures - see MS-SMB2 2.2.9.2.1 */
-struct smb3_blob_data {
- __le16 BlobSize;
- __u8 BlobData[];
-} __packed;
-
-/* Valid values for Attr */
-#define SE_GROUP_MANDATORY 0x00000001
-#define SE_GROUP_ENABLED_BY_DEFAULT 0x00000002
-#define SE_GROUP_ENABLED 0x00000004
-#define SE_GROUP_OWNER 0x00000008
-#define SE_GROUP_USE_FOR_DENY_ONLY 0x00000010
-#define SE_GROUP_INTEGRITY 0x00000020
-#define SE_GROUP_INTEGRITY_ENABLED 0x00000040
-#define SE_GROUP_RESOURCE 0x20000000
-#define SE_GROUP_LOGON_ID 0xC0000000
-
-/* struct sid_attr_data is SidData array in BlobData format then le32 Attr */
-
-struct sid_array_data {
- __le16 SidAttrCount;
- /* SidAttrList - array of sid_attr_data structs */
-} __packed;
-
-struct luid_attr_data {
-
-} __packed;
-
-/*
- * struct privilege_data is the same as BLOB_DATA - see MS-SMB2 2.2.9.2.1.5
- * but with size of LUID_ATTR_DATA struct and BlobData set to LUID_ATTR DATA
- */
-
-struct privilege_array_data {
- __le16 PrivilegeCount;
- /* array of privilege_data structs */
-} __packed;
-
-struct remoted_identity_tcon_context {
- __le16 TicketType; /* must be 0x0001 */
- __le16 TicketSize; /* total size of this struct */
- __le16 User; /* offset to SID_ATTR_DATA struct with user info */
- __le16 UserName; /* offset to null terminated Unicode username string */
- __le16 Domain; /* offset to null terminated Unicode domain name */
- __le16 Groups; /* offset to SID_ARRAY_DATA struct with group info */
- __le16 RestrictedGroups; /* similar to above */
- __le16 Privileges; /* offset to PRIVILEGE_ARRAY_DATA struct */
- __le16 PrimaryGroup; /* offset to SID_ARRAY_DATA struct */
- __le16 Owner; /* offset to BLOB_DATA struct */
- __le16 DefaultDacl; /* offset to BLOB_DATA struct */
- __le16 DeviceGroups; /* offset to SID_ARRAY_DATA struct */
- __le16 UserClaims; /* offset to BLOB_DATA struct */
- __le16 DeviceClaims; /* offset to BLOB_DATA struct */
- __u8 TicketInfo[]; /* variable length buf - remoted identity data */
-} __packed;
-
-struct smb2_tree_connect_req_extension {
- __le32 TreeConnectContextOffset;
- __le16 TreeConnectContextCount;
- __u8 Reserved[10];
- __u8 PathName[]; /* variable sized array */
- /* followed by array of TreeConnectContexts */
-} __packed;
-
-struct smb2_tree_connect_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 16 */
- __u8 ShareType; /* see below */
- __u8 Reserved;
- __le32 ShareFlags; /* see below */
- __le32 Capabilities; /* see below */
- __le32 MaximalAccess;
-} __packed;
-
-/* Possible ShareType values */
-#define SMB2_SHARE_TYPE_DISK 0x01
-#define SMB2_SHARE_TYPE_PIPE 0x02
-#define SMB2_SHARE_TYPE_PRINT 0x03
-
-/*
- * Possible ShareFlags - exactly one and only one of the first 4 caching flags
- * must be set (any of the remaining, SHI1005, flags may be set individually
- * or in combination.
- */
-#define SMB2_SHAREFLAG_MANUAL_CACHING 0x00000000
-#define SMB2_SHAREFLAG_AUTO_CACHING 0x00000010
-#define SMB2_SHAREFLAG_VDO_CACHING 0x00000020
-#define SMB2_SHAREFLAG_NO_CACHING 0x00000030
-#define SHI1005_FLAGS_DFS 0x00000001
-#define SHI1005_FLAGS_DFS_ROOT 0x00000002
-#define SHI1005_FLAGS_RESTRICT_EXCLUSIVE_OPENS 0x00000100
-#define SHI1005_FLAGS_FORCE_SHARED_DELETE 0x00000200
-#define SHI1005_FLAGS_ALLOW_NAMESPACE_CACHING 0x00000400
-#define SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM 0x00000800
-#define SHI1005_FLAGS_FORCE_LEVELII_OPLOCK 0x00001000
-#define SHI1005_FLAGS_ENABLE_HASH_V1 0x00002000
-#define SHI1005_FLAGS_ENABLE_HASH_V2 0x00004000
-#define SHI1005_FLAGS_ENCRYPT_DATA 0x00008000
-#define SMB2_SHAREFLAG_IDENTITY_REMOTING 0x00040000 /* 3.1.1 */
-#define SMB2_SHAREFLAG_COMPRESS_DATA 0x00100000 /* 3.1.1 */
-#define SHI1005_FLAGS_ALL 0x0014FF33
-
-/* Possible share capabilities */
-#define SMB2_SHARE_CAP_DFS cpu_to_le32(0x00000008) /* all dialects */
-#define SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY cpu_to_le32(0x00000010) /* 3.0 */
-#define SMB2_SHARE_CAP_SCALEOUT cpu_to_le32(0x00000020) /* 3.0 */
-#define SMB2_SHARE_CAP_CLUSTER cpu_to_le32(0x00000040) /* 3.0 */
-#define SMB2_SHARE_CAP_ASYMMETRIC cpu_to_le32(0x00000080) /* 3.02 */
-#define SMB2_SHARE_CAP_REDIRECT_TO_OWNER cpu_to_le32(0x00000100) /* 3.1.1 */
-
-struct smb2_tree_disconnect_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-struct smb2_tree_disconnect_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-/* File Attrubutes */
-#define FILE_ATTRIBUTE_READONLY 0x00000001
-#define FILE_ATTRIBUTE_HIDDEN 0x00000002
-#define FILE_ATTRIBUTE_SYSTEM 0x00000004
-#define FILE_ATTRIBUTE_DIRECTORY 0x00000010
-#define FILE_ATTRIBUTE_ARCHIVE 0x00000020
-#define FILE_ATTRIBUTE_NORMAL 0x00000080
-#define FILE_ATTRIBUTE_TEMPORARY 0x00000100
-#define FILE_ATTRIBUTE_SPARSE_FILE 0x00000200
-#define FILE_ATTRIBUTE_REPARSE_POINT 0x00000400
-#define FILE_ATTRIBUTE_COMPRESSED 0x00000800
-#define FILE_ATTRIBUTE_OFFLINE 0x00001000
-#define FILE_ATTRIBUTE_NOT_CONTENT_INDEXED 0x00002000
-#define FILE_ATTRIBUTE_ENCRYPTED 0x00004000
-#define FILE_ATTRIBUTE_INTEGRITY_STREAM 0x00008000
-#define FILE_ATTRIBUTE_NO_SCRUB_DATA 0x00020000
-
-/* Oplock levels */
-#define SMB2_OPLOCK_LEVEL_NONE 0x00
-#define SMB2_OPLOCK_LEVEL_II 0x01
-#define SMB2_OPLOCK_LEVEL_EXCLUSIVE 0x08
-#define SMB2_OPLOCK_LEVEL_BATCH 0x09
-#define SMB2_OPLOCK_LEVEL_LEASE 0xFF
-/* Non-spec internal type */
-#define SMB2_OPLOCK_LEVEL_NOCHANGE 0x99
-
-/* Desired Access Flags */
-#define FILE_READ_DATA_LE cpu_to_le32(0x00000001)
-#define FILE_WRITE_DATA_LE cpu_to_le32(0x00000002)
-#define FILE_APPEND_DATA_LE cpu_to_le32(0x00000004)
-#define FILE_READ_EA_LE cpu_to_le32(0x00000008)
-#define FILE_WRITE_EA_LE cpu_to_le32(0x00000010)
-#define FILE_EXECUTE_LE cpu_to_le32(0x00000020)
-#define FILE_READ_ATTRIBUTES_LE cpu_to_le32(0x00000080)
-#define FILE_WRITE_ATTRIBUTES_LE cpu_to_le32(0x00000100)
-#define FILE_DELETE_LE cpu_to_le32(0x00010000)
-#define FILE_READ_CONTROL_LE cpu_to_le32(0x00020000)
-#define FILE_WRITE_DAC_LE cpu_to_le32(0x00040000)
-#define FILE_WRITE_OWNER_LE cpu_to_le32(0x00080000)
-#define FILE_SYNCHRONIZE_LE cpu_to_le32(0x00100000)
-#define FILE_ACCESS_SYSTEM_SECURITY_LE cpu_to_le32(0x01000000)
-#define FILE_MAXIMAL_ACCESS_LE cpu_to_le32(0x02000000)
-#define FILE_GENERIC_ALL_LE cpu_to_le32(0x10000000)
-#define FILE_GENERIC_EXECUTE_LE cpu_to_le32(0x20000000)
-#define FILE_GENERIC_WRITE_LE cpu_to_le32(0x40000000)
-#define FILE_GENERIC_READ_LE cpu_to_le32(0x80000000)
-
-/* ShareAccess Flags */
-#define FILE_SHARE_READ_LE cpu_to_le32(0x00000001)
-#define FILE_SHARE_WRITE_LE cpu_to_le32(0x00000002)
-#define FILE_SHARE_DELETE_LE cpu_to_le32(0x00000004)
-#define FILE_SHARE_ALL_LE cpu_to_le32(0x00000007)
-
-/* CreateDisposition Flags */
-#define FILE_SUPERSEDE_LE cpu_to_le32(0x00000000)
-#define FILE_OPEN_LE cpu_to_le32(0x00000001)
-#define FILE_CREATE_LE cpu_to_le32(0x00000002)
-#define FILE_OPEN_IF_LE cpu_to_le32(0x00000003)
-#define FILE_OVERWRITE_LE cpu_to_le32(0x00000004)
-#define FILE_OVERWRITE_IF_LE cpu_to_le32(0x00000005)
-
-/* CreateOptions Flags */
-#define FILE_DIRECTORY_FILE_LE cpu_to_le32(0x00000001)
-/* same as #define CREATE_NOT_FILE_LE cpu_to_le32(0x00000001) */
-#define FILE_WRITE_THROUGH_LE cpu_to_le32(0x00000002)
-#define FILE_SEQUENTIAL_ONLY_LE cpu_to_le32(0x00000004)
-#define FILE_NO_INTERMEDIATE_BUFFERRING_LE cpu_to_le32(0x00000008)
-#define FILE_SYNCHRONOUS_IO_ALERT_LE cpu_to_le32(0x00000010)
-#define FILE_SYNCHRONOUS_IO_NON_ALERT_LE cpu_to_le32(0x00000020)
-#define FILE_NON_DIRECTORY_FILE_LE cpu_to_le32(0x00000040)
-#define FILE_COMPLETE_IF_OPLOCKED_LE cpu_to_le32(0x00000100)
-#define FILE_NO_EA_KNOWLEDGE_LE cpu_to_le32(0x00000200)
-#define FILE_RANDOM_ACCESS_LE cpu_to_le32(0x00000800)
-#define FILE_DELETE_ON_CLOSE_LE cpu_to_le32(0x00001000)
-#define FILE_OPEN_BY_FILE_ID_LE cpu_to_le32(0x00002000)
-#define FILE_OPEN_FOR_BACKUP_INTENT_LE cpu_to_le32(0x00004000)
-#define FILE_NO_COMPRESSION_LE cpu_to_le32(0x00008000)
-#define FILE_RESERVE_OPFILTER_LE cpu_to_le32(0x00100000)
-#define FILE_OPEN_REPARSE_POINT_LE cpu_to_le32(0x00200000)
-#define FILE_OPEN_NO_RECALL_LE cpu_to_le32(0x00400000)
-#define FILE_OPEN_FOR_FREE_SPACE_QUERY_LE cpu_to_le32(0x00800000)
-
-#define FILE_READ_RIGHTS_LE (FILE_READ_DATA_LE | FILE_READ_EA_LE \
- | FILE_READ_ATTRIBUTES_LE)
-#define FILE_WRITE_RIGHTS_LE (FILE_WRITE_DATA_LE | FILE_APPEND_DATA_LE \
- | FILE_WRITE_EA_LE | FILE_WRITE_ATTRIBUTES_LE)
-#define FILE_EXEC_RIGHTS_LE (FILE_EXECUTE_LE)
-
-/* Impersonation Levels. See MS-WPO section 9.7 and MSDN-IMPERS */
-#define IL_ANONYMOUS cpu_to_le32(0x00000000)
-#define IL_IDENTIFICATION cpu_to_le32(0x00000001)
-#define IL_IMPERSONATION cpu_to_le32(0x00000002)
-#define IL_DELEGATE cpu_to_le32(0x00000003)
-
-/* Create Context Values */
-#define SMB2_CREATE_EA_BUFFER "ExtA" /* extended attributes */
-#define SMB2_CREATE_SD_BUFFER "SecD" /* security descriptor */
-#define SMB2_CREATE_DURABLE_HANDLE_REQUEST "DHnQ"
-#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT "DHnC"
-#define SMB2_CREATE_ALLOCATION_SIZE "AISi"
-#define SMB2_CREATE_QUERY_MAXIMAL_ACCESS_REQUEST "MxAc"
-#define SMB2_CREATE_TIMEWARP_REQUEST "TWrp"
-#define SMB2_CREATE_QUERY_ON_DISK_ID "QFid"
-#define SMB2_CREATE_REQUEST_LEASE "RqLs"
-#define SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 "DH2Q"
-#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT_V2 "DH2C"
-#define SMB2_CREATE_APP_INSTANCE_ID 0x45BCA66AEFA7F74A9008FA462E144D74
-#define SMB2_CREATE_APP_INSTANCE_VERSION 0xB982D0B73B56074FA07B524A8116A010
-#define SVHDX_OPEN_DEVICE_CONTEX 0x9CCBCF9E04C1E643980E158DA1F6EC83
-#define SMB2_CREATE_TAG_POSIX 0x93AD25509CB411E7B42383DE968BCD7C
-
-/* Flag (SMB3 open response) values */
-#define SMB2_CREATE_FLAG_REPARSEPOINT 0x01
-
/*
* Maximum number of iovs we need for an open/create request.
* [0] : struct smb2_create_req
*/
#define SMB2_CREATE_IOV_SIZE 8
-struct smb2_create_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 57 */
- __u8 SecurityFlags;
- __u8 RequestedOplockLevel;
- __le32 ImpersonationLevel;
- __le64 SmbCreateFlags;
- __le64 Reserved;
- __le32 DesiredAccess;
- __le32 FileAttributes;
- __le32 ShareAccess;
- __le32 CreateDisposition;
- __le32 CreateOptions;
- __le16 NameOffset;
- __le16 NameLength;
- __le32 CreateContextsOffset;
- __le32 CreateContextsLength;
- __u8 Buffer[];
-} __packed;
-
/*
* Maximum size of a SMB2_CREATE response is 64 (smb2 header) +
* 88 (fixed part of create response) + 520 (path) + 208 (contexts) +
*/
#define MAX_SMB2_CREATE_RESPONSE_SIZE 880
-struct smb2_create_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 89 */
- __u8 OplockLevel;
- __u8 Flag; /* 0x01 if reparse point */
- __le32 CreateAction;
- __le64 CreationTime;
- __le64 LastAccessTime;
- __le64 LastWriteTime;
- __le64 ChangeTime;
- __le64 AllocationSize;
- __le64 EndofFile;
- __le32 FileAttributes;
- __le32 Reserved2;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
- __le32 CreateContextsOffset;
- __le32 CreateContextsLength;
- __u8 Buffer[1];
-} __packed;
-
-struct create_context {
- __le32 Next;
- __le16 NameOffset;
- __le16 NameLength;
- __le16 Reserved;
- __le16 DataOffset;
- __le32 DataLength;
- __u8 Buffer[];
-} __packed;
-
#define SMB2_LEASE_READ_CACHING_HE 0x01
#define SMB2_LEASE_HANDLE_CACHING_HE 0x02
#define SMB2_LEASE_WRITE_CACHING_HE 0x04
#define SMB2_IOCTL_IOV_SIZE 2
struct smb2_ioctl_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 57 */
__u16 Reserved;
__le32 CtlCode;
} __packed;
struct smb2_ioctl_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 57 */
__u16 Reserved;
__le32 CtlCode;
/* char * buffer[] */
} __packed;
-/* Currently defined values for close flags */
-#define SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB cpu_to_le16(0x0001)
-struct smb2_close_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 24 */
- __le16 Flags;
- __le32 Reserved;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
-} __packed;
-
-/*
- * Maximum size of a SMB2_CLOSE response is 64 (smb2 header) + 60 (data)
- */
-#define MAX_SMB2_CLOSE_RESPONSE_SIZE 124
-
-struct smb2_close_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* 60 */
- __le16 Flags;
- __le32 Reserved;
- __le64 CreationTime;
- __le64 LastAccessTime;
- __le64 LastWriteTime;
- __le64 ChangeTime;
- __le64 AllocationSize; /* Beginning of FILE_STANDARD_INFO equivalent */
- __le64 EndOfFile;
- __le32 Attributes;
-} __packed;
-
-struct smb2_flush_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 24 */
- __le16 Reserved1;
- __le32 Reserved2;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
-} __packed;
-
-struct smb2_flush_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize;
- __le16 Reserved;
-} __packed;
-
-/* For read request Flags field below, following flag is defined for SMB3.02 */
-#define SMB2_READFLAG_READ_UNBUFFERED 0x01
-#define SMB2_READFLAG_REQUEST_COMPRESSED 0x02 /* See MS-SMB2 2.2.19 */
-
-/* Channel field for read and write: exactly one of following flags can be set*/
-#define SMB2_CHANNEL_NONE cpu_to_le32(0x00000000)
-#define SMB2_CHANNEL_RDMA_V1 cpu_to_le32(0x00000001) /* SMB3 or later */
-#define SMB2_CHANNEL_RDMA_V1_INVALIDATE cpu_to_le32(0x00000002) /* >= SMB3.02 */
-#define SMB2_CHANNEL_RDMA_TRANSFORM cpu_to_le32(0x00000003) /* >= SMB3.02, only used on write */
-
-/* SMB2 read request without RFC1001 length at the beginning */
-struct smb2_read_plain_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 49 */
- __u8 Padding; /* offset from start of SMB2 header to place read */
- __u8 Flags; /* MBZ unless SMB3.02 or later */
- __le32 Length;
- __le64 Offset;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
- __le32 MinimumCount;
- __le32 Channel; /* MBZ except for SMB3 or later */
- __le32 RemainingBytes;
- __le16 ReadChannelInfoOffset;
- __le16 ReadChannelInfoLength;
- __u8 Buffer[1];
-} __packed;
-
-/* Read flags */
-#define SMB2_READFLAG_RESPONSE_NONE 0x00000000
-#define SMB2_READFLAG_RESPONSE_RDMA_TRANSFORM 0x00000001
-
-struct smb2_read_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 17 */
- __u8 DataOffset;
- __u8 Reserved;
- __le32 DataLength;
- __le32 DataRemaining;
- __u32 Flags;
- __u8 Buffer[1];
-} __packed;
-
-/* For write request Flags field below the following flags are defined: */
-#define SMB2_WRITEFLAG_WRITE_THROUGH 0x00000001 /* SMB2.1 or later */
-#define SMB2_WRITEFLAG_WRITE_UNBUFFERED 0x00000002 /* SMB3.02 or later */
-
-struct smb2_write_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 49 */
- __le16 DataOffset; /* offset from start of SMB2 header to write data */
- __le32 Length;
- __le64 Offset;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
- __le32 Channel; /* MBZ unless SMB3.02 or later */
- __le32 RemainingBytes;
- __le16 WriteChannelInfoOffset;
- __le16 WriteChannelInfoLength;
- __le32 Flags;
- __u8 Buffer[1];
-} __packed;
-
-struct smb2_write_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 17 */
- __u8 DataOffset;
- __u8 Reserved;
- __le32 DataLength;
- __le32 DataRemaining;
- __u32 Reserved2;
- __u8 Buffer[1];
-} __packed;
-
-/* notify flags */
-#define SMB2_WATCH_TREE 0x0001
-
-/* notify completion filter flags. See MS-FSCC 2.6 and MS-SMB2 2.2.35 */
-#define FILE_NOTIFY_CHANGE_FILE_NAME 0x00000001
-#define FILE_NOTIFY_CHANGE_DIR_NAME 0x00000002
-#define FILE_NOTIFY_CHANGE_ATTRIBUTES 0x00000004
-#define FILE_NOTIFY_CHANGE_SIZE 0x00000008
-#define FILE_NOTIFY_CHANGE_LAST_WRITE 0x00000010
-#define FILE_NOTIFY_CHANGE_LAST_ACCESS 0x00000020
-#define FILE_NOTIFY_CHANGE_CREATION 0x00000040
-#define FILE_NOTIFY_CHANGE_EA 0x00000080
-#define FILE_NOTIFY_CHANGE_SECURITY 0x00000100
-#define FILE_NOTIFY_CHANGE_STREAM_NAME 0x00000200
-#define FILE_NOTIFY_CHANGE_STREAM_SIZE 0x00000400
-#define FILE_NOTIFY_CHANGE_STREAM_WRITE 0x00000800
-
-struct smb2_change_notify_req {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize;
- __le16 Flags;
- __le32 OutputBufferLength;
- __u64 PersistentFileId; /* opaque endianness */
- __u64 VolatileFileId; /* opaque endianness */
- __le32 CompletionFilter;
- __u32 Reserved;
-} __packed;
-
-struct smb2_change_notify_rsp {
- struct smb2_sync_hdr sync_hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 OutputBufferOffset;
- __le32 OutputBufferLength;
- __u8 Buffer[1]; /* array of file notify structs */
-} __packed;
-
#define SMB2_LOCKFLAG_SHARED_LOCK 0x0001
#define SMB2_LOCKFLAG_EXCLUSIVE_LOCK 0x0002
#define SMB2_LOCKFLAG_UNLOCK 0x0004
} __packed;
struct smb2_lock_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 48 */
__le16 LockCount;
/*
} __packed;
struct smb2_lock_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 4 */
__le16 Reserved;
} __packed;
struct smb2_echo_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 4 */
__u16 Reserved;
} __packed;
struct smb2_echo_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 4 */
__u16 Reserved;
} __packed;
*/
struct smb2_query_directory_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 33 */
__u8 FileInformationClass;
__u8 Flags;
} __packed;
struct smb2_query_directory_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 9 */
__le16 OutputBufferOffset;
__le32 OutputBufferLength;
#define SL_INDEX_SPECIFIED 0x00000004
struct smb2_query_info_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 41 */
__u8 InfoType;
__u8 FileInfoClass;
} __packed;
struct smb2_query_info_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 9 */
__le16 OutputBufferOffset;
__le32 OutputBufferLength;
#define SMB2_SET_INFO_IOV_SIZE 3
struct smb2_set_info_req {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 33 */
__u8 InfoType;
__u8 FileInfoClass;
} __packed;
struct smb2_set_info_rsp {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 2 */
} __packed;
struct smb2_oplock_break {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 24 */
__u8 OplockLevel;
__u8 Reserved;
#define SMB2_NOTIFY_BREAK_LEASE_FLAG_ACK_REQUIRED cpu_to_le32(0x01)
struct smb2_lease_break {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 44 */
__le16 Epoch;
__le32 Flags;
} __packed;
struct smb2_lease_ack {
- struct smb2_sync_hdr sync_hdr;
+ struct smb2_hdr hdr;
__le16 StructureSize; /* Must be 36 */
__le16 Reserved;
__le32 Flags;
struct TCP_Server_Info *server);
extern unsigned int smb2_calc_size(void *buf, struct TCP_Server_Info *server);
extern char *smb2_get_data_area_len(int *off, int *len,
- struct smb2_sync_hdr *shdr);
+ struct smb2_hdr *shdr);
extern __le16 *cifs_convert_path_to_utf16(const char *from,
struct cifs_sb_info *cifs_sb);
#include <linux/mempool.h>
#include <linux/highmem.h>
#include <crypto/aead.h>
-#include "smb2pdu.h"
#include "cifsglob.h"
#include "cifsproto.h"
#include "smb2proto.h"
unsigned char smb2_signature[SMB2_HMACSHA256_SIZE];
unsigned char *sigptr = smb2_signature;
struct kvec *iov = rqst->rq_iov;
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)iov[0].iov_base;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)iov[0].iov_base;
struct cifs_ses *ses;
struct shash_desc *shash;
struct crypto_shash *hash;
struct sdesc *sdesc = NULL;
struct smb_rqst drqst;
- ses = smb2_find_smb_ses(server, shdr->SessionId);
+ ses = smb2_find_smb_ses(server, le64_to_cpu(shdr->SessionId));
if (!ses) {
cifs_server_dbg(VFS, "%s: Could not find session\n", __func__);
return 0;
unsigned char smb3_signature[SMB2_CMACAES_SIZE];
unsigned char *sigptr = smb3_signature;
struct kvec *iov = rqst->rq_iov;
- struct smb2_sync_hdr *shdr = (struct smb2_sync_hdr *)iov[0].iov_base;
+ struct smb2_hdr *shdr = (struct smb2_hdr *)iov[0].iov_base;
struct shash_desc *shash;
struct crypto_shash *hash;
struct sdesc *sdesc = NULL;
struct smb_rqst drqst;
u8 key[SMB3_SIGN_KEY_SIZE];
- rc = smb2_get_sign_key(shdr->SessionId, server, key);
+ rc = smb2_get_sign_key(le64_to_cpu(shdr->SessionId), server, key);
if (rc)
return 0;
smb2_sign_rqst(struct smb_rqst *rqst, struct TCP_Server_Info *server)
{
int rc = 0;
- struct smb2_sync_hdr *shdr;
+ struct smb2_hdr *shdr;
struct smb2_sess_setup_req *ssr;
bool is_binding;
bool is_signed;
- shdr = (struct smb2_sync_hdr *)rqst->rq_iov[0].iov_base;
+ shdr = (struct smb2_hdr *)rqst->rq_iov[0].iov_base;
ssr = (struct smb2_sess_setup_req *)shdr;
is_binding = shdr->Command == SMB2_SESSION_SETUP &&
{
unsigned int rc;
char server_response_sig[SMB2_SIGNATURE_SIZE];
- struct smb2_sync_hdr *shdr =
- (struct smb2_sync_hdr *)rqst->rq_iov[0].iov_base;
+ struct smb2_hdr *shdr =
+ (struct smb2_hdr *)rqst->rq_iov[0].iov_base;
if ((shdr->Command == SMB2_NEGOTIATE) ||
(shdr->Command == SMB2_SESSION_SETUP) ||
*/
static inline void
smb2_seq_num_into_buf(struct TCP_Server_Info *server,
- struct smb2_sync_hdr *shdr)
+ struct smb2_hdr *shdr)
{
unsigned int i, num = le16_to_cpu(shdr->CreditCharge);
}
static struct mid_q_entry *
-smb2_mid_entry_alloc(const struct smb2_sync_hdr *shdr,
+smb2_mid_entry_alloc(const struct smb2_hdr *shdr,
struct TCP_Server_Info *server)
{
struct mid_q_entry *temp;
atomic_inc(&midCount);
temp->mid_state = MID_REQUEST_ALLOCATED;
- trace_smb3_cmd_enter(shdr->TreeId, shdr->SessionId,
- le16_to_cpu(shdr->Command), temp->mid);
+ trace_smb3_cmd_enter(le32_to_cpu(shdr->Id.SyncId.TreeId),
+ le64_to_cpu(shdr->SessionId),
+ le16_to_cpu(shdr->Command), temp->mid);
return temp;
}
static int
smb2_get_mid_entry(struct cifs_ses *ses, struct TCP_Server_Info *server,
- struct smb2_sync_hdr *shdr, struct mid_q_entry **mid)
+ struct smb2_hdr *shdr, struct mid_q_entry **mid)
{
if (server->tcpStatus == CifsExiting)
return -ENOENT;
struct smb_rqst *rqst)
{
int rc;
- struct smb2_sync_hdr *shdr =
- (struct smb2_sync_hdr *)rqst->rq_iov[0].iov_base;
+ struct smb2_hdr *shdr =
+ (struct smb2_hdr *)rqst->rq_iov[0].iov_base;
struct mid_q_entry *mid;
smb2_seq_num_into_buf(server, shdr);
smb2_setup_async_request(struct TCP_Server_Info *server, struct smb_rqst *rqst)
{
int rc;
- struct smb2_sync_hdr *shdr =
- (struct smb2_sync_hdr *)rqst->rq_iov[0].iov_base;
+ struct smb2_hdr *shdr =
+ (struct smb2_hdr *)rqst->rq_iov[0].iov_base;
struct mid_q_entry *mid;
if (server->tcpStatus == CifsNeedNegotiate &&
#define _CIFS_TRACE_H
#include <linux/tracepoint.h>
+#include <linux/net.h>
+#include <linux/inet.h>
/*
* Please use this 3-part article as a reference for writing new tracepoints:
DEFINE_SMB3_LEASE_ERR_EVENT(lease_err);
+DECLARE_EVENT_CLASS(smb3_connect_class,
+ TP_PROTO(char *hostname,
+ __u64 conn_id,
+ const struct __kernel_sockaddr_storage *dst_addr),
+ TP_ARGS(hostname, conn_id, dst_addr),
+ TP_STRUCT__entry(
+ __string(hostname, hostname)
+ __field(__u64, conn_id)
+ __array(__u8, dst_addr, sizeof(struct sockaddr_storage))
+ ),
+ TP_fast_assign(
+ struct sockaddr_storage *pss = NULL;
+
+ __entry->conn_id = conn_id;
+ pss = (struct sockaddr_storage *)__entry->dst_addr;
+ *pss = *dst_addr;
+ __assign_str(hostname, hostname);
+ ),
+ TP_printk("conn_id=0x%llx server=%s addr=%pISpsfc",
+ __entry->conn_id,
+ __get_str(hostname),
+ __entry->dst_addr)
+)
+
+#define DEFINE_SMB3_CONNECT_EVENT(name) \
+DEFINE_EVENT(smb3_connect_class, smb3_##name, \
+ TP_PROTO(char *hostname, \
+ __u64 conn_id, \
+ const struct __kernel_sockaddr_storage *addr), \
+ TP_ARGS(hostname, conn_id, addr))
+
+DEFINE_SMB3_CONNECT_EVENT(connect_done);
+
+DECLARE_EVENT_CLASS(smb3_connect_err_class,
+ TP_PROTO(char *hostname, __u64 conn_id,
+ const struct __kernel_sockaddr_storage *dst_addr, int rc),
+ TP_ARGS(hostname, conn_id, dst_addr, rc),
+ TP_STRUCT__entry(
+ __string(hostname, hostname)
+ __field(__u64, conn_id)
+ __array(__u8, dst_addr, sizeof(struct sockaddr_storage))
+ __field(int, rc)
+ ),
+ TP_fast_assign(
+ struct sockaddr_storage *pss = NULL;
+
+ __entry->conn_id = conn_id;
+ __entry->rc = rc;
+ pss = (struct sockaddr_storage *)__entry->dst_addr;
+ *pss = *dst_addr;
+ __assign_str(hostname, hostname);
+ ),
+ TP_printk("rc=%d conn_id=0x%llx server=%s addr=%pISpsfc",
+ __entry->rc,
+ __entry->conn_id,
+ __get_str(hostname),
+ __entry->dst_addr)
+)
+
+#define DEFINE_SMB3_CONNECT_ERR_EVENT(name) \
+DEFINE_EVENT(smb3_connect_err_class, smb3_##name, \
+ TP_PROTO(char *hostname, \
+ __u64 conn_id, \
+ const struct __kernel_sockaddr_storage *addr, \
+ int rc), \
+ TP_ARGS(hostname, conn_id, addr, rc))
+
+DEFINE_SMB3_CONNECT_ERR_EVENT(connect_err);
+
DECLARE_EVENT_CLASS(smb3_reconnect_class,
TP_PROTO(__u64 currmid,
__u64 conn_id,
if (!ses)
return NULL;
+ spin_lock(&ses->chan_lock);
if (!ses->binding) {
/* round robin */
if (ses->chan_count > 1) {
index = (uint)atomic_inc_return(&ses->chan_seq);
index %= ses->chan_count;
}
+ spin_unlock(&ses->chan_lock);
return ses->chans[index].server;
} else {
+ spin_unlock(&ses->chan_lock);
return cifs_ses_server(ses);
}
}
struct inode *inode;
struct coda_inode_info *cii;
unsigned long hash = coda_f2i(fid);
+ umode_t inode_type = coda_inode_type(attr);
+retry:
inode = iget5_locked(sb, hash, coda_test_inode, coda_set_inode, fid);
-
if (!inode)
return ERR_PTR(-ENOMEM);
inode->i_ino = hash;
/* inode is locked and unique, no need to grab cii->c_lock */
cii->c_mapcount = 0;
+ coda_fill_inode(inode, attr);
unlock_new_inode(inode);
+ } else if ((inode->i_mode & S_IFMT) != inode_type) {
+ /* Inode has changed type, mark bad and grab a new one */
+ remove_inode_hash(inode);
+ coda_flag_inode(inode, C_PURGE);
+ iput(inode);
+ goto retry;
}
-
- /* always replace the attributes, type might have changed */
- coda_fill_inode(inode, attr);
return inode;
}
}
/* utility functions below */
+umode_t coda_inode_type(struct coda_vattr *attr)
+{
+ switch (attr->va_type) {
+ case C_VREG:
+ return S_IFREG;
+ case C_VDIR:
+ return S_IFDIR;
+ case C_VLNK:
+ return S_IFLNK;
+ case C_VNON:
+ default:
+ return 0;
+ }
+}
+
void coda_vattr_to_iattr(struct inode *inode, struct coda_vattr *attr)
{
- int inode_type;
- /* inode's i_flags, i_ino are set by iget
- XXX: is this all we need ??
- */
- switch (attr->va_type) {
- case C_VNON:
- inode_type = 0;
- break;
- case C_VREG:
- inode_type = S_IFREG;
- break;
- case C_VDIR:
- inode_type = S_IFDIR;
- break;
- case C_VLNK:
- inode_type = S_IFLNK;
- break;
- default:
- inode_type = 0;
- }
+ /* inode's i_flags, i_ino are set by iget
+ * XXX: is this all we need ??
+ */
+ umode_t inode_type = coda_inode_type(attr);
inode->i_mode |= inode_type;
if (attr->va_mode != (u_short) -1)
u32, unsigned int);
int coda_setattr(struct user_namespace *, struct dentry *, struct iattr *);
-/* this file: heloers */
+/* this file: helpers */
char *coda_f2s(struct CodaFid *f);
int coda_iscontrol(const char *name, size_t length);
+umode_t coda_inode_type(struct coda_vattr *attr);
void coda_vattr_to_iattr(struct inode *, struct coda_vattr *);
void coda_iattr_to_vattr(struct iattr *, struct coda_vattr *);
unsigned short coda_flags_to_cflags(unsigned short);
{
struct coda_inode_info *cii = ITOC(inode);
+ if (!inode)
+ return;
+
spin_lock(&cii->c_lock);
cii->c_flags |= flag;
spin_unlock(&cii->c_lock);
coda_dir_drop_nlink(old_dir);
coda_dir_inc_nlink(new_dir);
}
- coda_dir_update_mtime(old_dir);
- coda_dir_update_mtime(new_dir);
coda_flag_inode(d_inode(new_dentry), C_VATTR);
- } else {
- coda_flag_inode(old_dir, C_VATTR);
- coda_flag_inode(new_dir, C_VATTR);
}
+ coda_dir_update_mtime(old_dir);
+ coda_dir_update_mtime(new_dir);
}
return error;
}
*/
static int coda_dentry_delete(const struct dentry * dentry)
{
- int flags;
+ struct inode *inode;
+ struct coda_inode_info *cii;
if (d_really_is_negative(dentry))
return 0;
- flags = (ITOC(d_inode(dentry))->c_flags) & C_PURGE;
- if (is_bad_inode(d_inode(dentry)) || flags) {
+ inode = d_inode(dentry);
+ if (!inode || is_bad_inode(inode))
return 1;
- }
+
+ cii = ITOC(inode);
+ if (cii->c_flags & C_PURGE)
+ return 1;
+
return 0;
}
* to the Coda project. Contact Peter Braam <coda@cs.cmu.edu>.
*/
+#include <linux/refcount.h>
#include <linux/types.h>
#include <linux/kernel.h>
#include <linux/time.h>
#include "coda_int.h"
struct coda_vm_ops {
- atomic_t refcnt;
+ refcount_t refcnt;
struct file *coda_file;
const struct vm_operations_struct *host_vm_ops;
struct vm_operations_struct vm_ops;
struct coda_vm_ops *cvm_ops =
container_of(vma->vm_ops, struct coda_vm_ops, vm_ops);
- atomic_inc(&cvm_ops->refcnt);
+ refcount_inc(&cvm_ops->refcnt);
if (cvm_ops->host_vm_ops && cvm_ops->host_vm_ops->open)
cvm_ops->host_vm_ops->open(vma);
if (cvm_ops->host_vm_ops && cvm_ops->host_vm_ops->close)
cvm_ops->host_vm_ops->close(vma);
- if (atomic_dec_and_test(&cvm_ops->refcnt)) {
+ if (refcount_dec_and_test(&cvm_ops->refcnt)) {
vma->vm_ops = cvm_ops->host_vm_ops;
fput(cvm_ops->coda_file);
kfree(cvm_ops);
cvm_ops->vm_ops.open = coda_vm_open;
cvm_ops->vm_ops.close = coda_vm_close;
cvm_ops->coda_file = coda_file;
- atomic_set(&cvm_ops->refcnt, 1);
+ refcount_set(&cvm_ops->refcnt, 1);
vma->vm_ops = &cvm_ops->vm_ops;
}
struct coda_file_info *cfi;
struct coda_inode_info *cii;
struct inode *host_inode;
- int err;
cfi = coda_ftoc(coda_file);
- err = venus_close(coda_inode->i_sb, coda_i2f(coda_inode),
+ venus_close(coda_inode->i_sb, coda_i2f(coda_inode),
coda_flags, coda_file->f_cred->fsuid);
host_inode = file_inode(cfi->cfi_container);
hdr.opcode, hdr.unique);
nbytes = size;
}
- dcbuf = kvmalloc(nbytes, GFP_KERNEL);
- if (!dcbuf) {
- retval = -ENOMEM;
- goto out;
- }
- if (copy_from_user(dcbuf, buf, nbytes)) {
- kvfree(dcbuf);
- retval = -EFAULT;
+
+ dcbuf = vmemdup_user(buf, nbytes);
+ if (IS_ERR(dcbuf)) {
+ retval = PTR_ERR(dcbuf);
goto out;
}
MODULE_DESCRIPTION("Coda Distributed File System VFS interface");
MODULE_ALIAS_CHARDEV_MAJOR(CODA_PSDEV_MAJOR);
MODULE_LICENSE("GPL");
-MODULE_VERSION("7.0");
+MODULE_VERSION("7.2");
static int __init init_coda(void)
{
list_add_tail(&req->uc_chain, &vcp->vc_pending);
wake_up_interruptible(&vcp->vc_waitq);
- if (req->uc_flags & CODA_REQ_ASYNC) {
+ /* We can return early on asynchronous requests */
+ if (outSize == NULL) {
mutex_unlock(&vcp->vc_mutex);
return 0;
}
/**
* prepend_name - prepend a pathname in front of current buffer pointer
- * @buffer: buffer pointer
- * @buflen: allocated length of the buffer
- * @name: name string and length qstr structure
+ * @p: prepend buffer which contains buffer pointer and allocated length
+ * @name: name string and length qstr structure
*
* With RCU path tracing, it may race with d_move(). Use READ_ONCE() to
* make sure that either the old or the new name pointer and length are
* prepend_path - Prepend path string to a buffer
* @path: the dentry/vfsmount to report
* @root: root vfsmnt/dentry
- * @buffer: pointer to the end of the buffer
- * @buflen: pointer to buffer length
+ * @p: prepend buffer which contains buffer pointer and allocated length
*
* The function will first try to write out the pathname without taking any
* lock other than the RCU read lock to make sure that dentries won't go away.
DBG_BUGON(1);
}
-/*
- * a compressed_pages[] placeholder in order to avoid
- * being filled with file pages for in-place decompression.
- */
-#define PAGE_UNALLOCATED ((void *)0x5F0E4B1D)
-
/* how to allocate cached pages for a pcluster */
enum z_erofs_cache_alloctype {
DONTALLOC, /* don't allocate any cached pages */
- DELAYEDALLOC, /* delayed allocation (at the time of submitting io) */
/*
* try to use cached I/O if page allocation succeeds or fallback
* to in-place I/O instead to avoid any direct reclaim.
/* I/O is needed, no possible to decompress directly */
standalone = false;
switch (type) {
- case DELAYEDALLOC:
- t = tagptr_init(compressed_page_t,
- PAGE_UNALLOCATED);
- break;
case TRYALLOC:
newpage = erofs_allocpage(pagepool, gfp);
if (!newpage)
/* callers must be with collection lock held */
static int z_erofs_attach_page(struct z_erofs_collector *clt,
- struct page *page,
- enum z_erofs_page_type type)
+ struct page *page, enum z_erofs_page_type type,
+ bool pvec_safereuse)
{
int ret;
z_erofs_try_inplace_io(clt, page))
return 0;
- ret = z_erofs_pagevec_enqueue(&clt->vector, page, type);
+ ret = z_erofs_pagevec_enqueue(&clt->vector, page, type,
+ pvec_safereuse);
clt->cl->vcnt += (unsigned int)ret;
-
return ret ? 0 : -EAGAIN;
}
tight &= (clt->mode >= COLLECT_PRIMARY_FOLLOWED);
retry:
- err = z_erofs_attach_page(clt, page, page_type);
+ err = z_erofs_attach_page(clt, page, page_type,
+ clt->mode >= COLLECT_PRIMARY_FOLLOWED);
/* should allocate an additional short-lived page for pagevec */
if (err == -EAGAIN) {
struct page *const newpage =
set_page_private(newpage, Z_EROFS_SHORTLIVED_PAGE);
err = z_erofs_attach_page(clt, newpage,
- Z_EROFS_PAGE_TYPE_EXCLUSIVE);
+ Z_EROFS_PAGE_TYPE_EXCLUSIVE, true);
if (!err)
goto retry;
}
if (!page)
goto out_allocpage;
- /*
- * the cached page has not been allocated and
- * an placeholder is out there, prepare it now.
- */
- if (page == PAGE_UNALLOCATED) {
- tocache = true;
- goto out_allocpage;
- }
-
/* process the target tagged pointer */
t = tagptr_init(compressed_page_t, page);
justfound = tagptr_unfold_tags(t);
#define Z_EROFS_VMAP_GLOBAL_PAGES 2048
#endif
-
static inline bool z_erofs_pagevec_enqueue(struct z_erofs_pagevec_ctor *ctor,
struct page *page,
- enum z_erofs_page_type type)
+ enum z_erofs_page_type type,
+ bool pvec_safereuse)
{
- if (!ctor->next && type)
- if (ctor->index + 1 == ctor->nr)
+ if (!ctor->next) {
+ /* some pages cannot be reused as pvec safely without I/O */
+ if (type == Z_EROFS_PAGE_TYPE_EXCLUSIVE && !pvec_safereuse)
+ type = Z_EROFS_VLE_PAGE_TYPE_TAIL_SHARED;
+
+ if (type != Z_EROFS_PAGE_TYPE_EXCLUSIVE &&
+ ctor->index + 1 == ctor->nr)
return false;
+ }
if (ctor->index >= ctor->nr)
z_erofs_pagevec_ctor_pagedown(ctor, false);
* SIGSEGV.
*/
if (bprm->point_of_no_return && !fatal_signal_pending(current))
- force_sigsegv(SIGSEGV);
+ force_fatal_sig(SIGSEGV);
out_unmark:
current->fs->in_exec = 0;
#ifndef _EXT4_H
#define _EXT4_H
+#include <linux/refcount.h>
#include <linux/types.h>
#include <linux/blkdev.h>
#include <linux/magic.h>
struct bio *bio; /* Linked list of completed
* bios covering the extent */
unsigned int flag; /* unwritten or not */
- atomic_t count; /* reference counter */
+ refcount_t count; /* reference counter */
struct list_head list_vec; /* list of ext4_io_end_vec */
} ext4_io_end_t;
static int ext4_ext_get_access(handle_t *handle, struct inode *inode,
struct ext4_ext_path *path)
{
+ int err = 0;
+
if (path->p_bh) {
/* path points to block */
BUFFER_TRACE(path->p_bh, "get_write_access");
- return ext4_journal_get_write_access(handle, inode->i_sb,
- path->p_bh, EXT4_JTR_NONE);
+ err = ext4_journal_get_write_access(handle, inode->i_sb,
+ path->p_bh, EXT4_JTR_NONE);
+ /*
+ * The extent buffer's verified bit will be set again in
+ * __ext4_ext_dirty(). We could leave an inconsistent
+ * buffer if the extents updating procudure break off du
+ * to some error happens, force to check it again.
+ */
+ if (!err)
+ clear_buffer_verified(path->p_bh);
}
/* path points to leaf/index in inode body */
/* we use in-core data, no need to protect them */
- return 0;
+ return err;
}
/*
/* path points to block */
err = __ext4_handle_dirty_metadata(where, line, handle,
inode, path->p_bh);
+ /* Extents updating done, re-set verified flag */
+ if (!err)
+ set_buffer_verified(path->p_bh);
} else {
/* path points to leaf/index in inode body */
err = ext4_mark_inode_dirty(handle, inode);
static int ext4_valid_extent_entries(struct inode *inode,
struct ext4_extent_header *eh,
- ext4_fsblk_t *pblk, int depth)
+ ext4_lblk_t lblk, ext4_fsblk_t *pblk,
+ int depth)
{
unsigned short entries;
+ ext4_lblk_t lblock = 0;
+ ext4_lblk_t prev = 0;
+
if (eh->eh_entries == 0)
return 1;
if (depth == 0) {
/* leaf entries */
struct ext4_extent *ext = EXT_FIRST_EXTENT(eh);
- ext4_lblk_t lblock = 0;
- ext4_lblk_t prev = 0;
- int len = 0;
+
+ /*
+ * The logical block in the first entry should equal to
+ * the number in the index block.
+ */
+ if (depth != ext_depth(inode) &&
+ lblk != le32_to_cpu(ext->ee_block))
+ return 0;
while (entries) {
if (!ext4_valid_extent(inode, ext))
return 0;
/* Check for overlapping extents */
lblock = le32_to_cpu(ext->ee_block);
- len = ext4_ext_get_actual_len(ext);
if ((lblock <= prev) && prev) {
*pblk = ext4_ext_pblock(ext);
return 0;
}
+ prev = lblock + ext4_ext_get_actual_len(ext) - 1;
ext++;
entries--;
- prev = lblock + len - 1;
}
} else {
struct ext4_extent_idx *ext_idx = EXT_FIRST_INDEX(eh);
+
+ /*
+ * The logical block in the first entry should equal to
+ * the number in the parent index block.
+ */
+ if (depth != ext_depth(inode) &&
+ lblk != le32_to_cpu(ext_idx->ei_block))
+ return 0;
while (entries) {
if (!ext4_valid_extent_idx(inode, ext_idx))
return 0;
+
+ /* Check for overlapping index extents */
+ lblock = le32_to_cpu(ext_idx->ei_block);
+ if ((lblock <= prev) && prev) {
+ *pblk = ext4_idx_pblock(ext_idx);
+ return 0;
+ }
ext_idx++;
entries--;
+ prev = lblock;
}
}
return 1;
static int __ext4_ext_check(const char *function, unsigned int line,
struct inode *inode, struct ext4_extent_header *eh,
- int depth, ext4_fsblk_t pblk)
+ int depth, ext4_fsblk_t pblk, ext4_lblk_t lblk)
{
const char *error_msg;
int max = 0, err = -EFSCORRUPTED;
error_msg = "invalid eh_entries";
goto corrupted;
}
- if (!ext4_valid_extent_entries(inode, eh, &pblk, depth)) {
+ if (!ext4_valid_extent_entries(inode, eh, lblk, &pblk, depth)) {
error_msg = "invalid extent entries";
goto corrupted;
}
}
#define ext4_ext_check(inode, eh, depth, pblk) \
- __ext4_ext_check(__func__, __LINE__, (inode), (eh), (depth), (pblk))
+ __ext4_ext_check(__func__, __LINE__, (inode), (eh), (depth), (pblk), 0)
int ext4_ext_check_inode(struct inode *inode)
{
static struct buffer_head *
__read_extent_tree_block(const char *function, unsigned int line,
- struct inode *inode, ext4_fsblk_t pblk, int depth,
- int flags)
+ struct inode *inode, struct ext4_extent_idx *idx,
+ int depth, int flags)
{
struct buffer_head *bh;
int err;
gfp_t gfp_flags = __GFP_MOVABLE | GFP_NOFS;
+ ext4_fsblk_t pblk;
if (flags & EXT4_EX_NOFAIL)
gfp_flags |= __GFP_NOFAIL;
+ pblk = ext4_idx_pblock(idx);
bh = sb_getblk_gfp(inode->i_sb, pblk, gfp_flags);
if (unlikely(!bh))
return ERR_PTR(-ENOMEM);
}
if (buffer_verified(bh) && !(flags & EXT4_EX_FORCE_CACHE))
return bh;
- err = __ext4_ext_check(function, line, inode,
- ext_block_hdr(bh), depth, pblk);
+ err = __ext4_ext_check(function, line, inode, ext_block_hdr(bh),
+ depth, pblk, le32_to_cpu(idx->ei_block));
if (err)
goto errout;
set_buffer_verified(bh);
}
-#define read_extent_tree_block(inode, pblk, depth, flags) \
- __read_extent_tree_block(__func__, __LINE__, (inode), (pblk), \
+#define read_extent_tree_block(inode, idx, depth, flags) \
+ __read_extent_tree_block(__func__, __LINE__, (inode), (idx), \
(depth), (flags))
/*
i--;
continue;
}
- bh = read_extent_tree_block(inode,
- ext4_idx_pblock(path[i].p_idx++),
+ bh = read_extent_tree_block(inode, path[i].p_idx++,
depth - i - 1,
EXT4_EX_FORCE_CACHE);
if (IS_ERR(bh)) {
r = EXT_LAST_INDEX(eh);
while (l <= r) {
m = l + (r - l) / 2;
+ ext_debug(inode, "%p(%u):%p(%u):%p(%u) ", l,
+ le32_to_cpu(l->ei_block), m, le32_to_cpu(m->ei_block),
+ r, le32_to_cpu(r->ei_block));
+
if (block < le32_to_cpu(m->ei_block))
r = m - 1;
else
l = m + 1;
- ext_debug(inode, "%p(%u):%p(%u):%p(%u) ", l,
- le32_to_cpu(l->ei_block), m, le32_to_cpu(m->ei_block),
- r, le32_to_cpu(r->ei_block));
}
path->p_idx = l - 1;
while (l <= r) {
m = l + (r - l) / 2;
+ ext_debug(inode, "%p(%u):%p(%u):%p(%u) ", l,
+ le32_to_cpu(l->ee_block), m, le32_to_cpu(m->ee_block),
+ r, le32_to_cpu(r->ee_block));
+
if (block < le32_to_cpu(m->ee_block))
r = m - 1;
else
l = m + 1;
- ext_debug(inode, "%p(%u):%p(%u):%p(%u) ", l,
- le32_to_cpu(l->ee_block), m, le32_to_cpu(m->ee_block),
- r, le32_to_cpu(r->ee_block));
}
path->p_ext = l - 1;
path[ppos].p_depth = i;
path[ppos].p_ext = NULL;
- bh = read_extent_tree_block(inode, path[ppos].p_block, --i,
- flags);
+ bh = read_extent_tree_block(inode, path[ppos].p_idx, --i, flags);
if (IS_ERR(bh)) {
ret = PTR_ERR(bh);
goto err;
struct ext4_extent_header *eh;
struct ext4_extent_idx *ix;
struct ext4_extent *ex;
- ext4_fsblk_t block;
int depth; /* Note, NOT eh_depth; depth from top of tree */
int ee_len;
* follow it and find the closest allocated
* block to the right */
ix++;
- block = ext4_idx_pblock(ix);
while (++depth < path->p_depth) {
/* subtract from p_depth to get proper eh_depth */
- bh = read_extent_tree_block(inode, block,
- path->p_depth - depth, 0);
+ bh = read_extent_tree_block(inode, ix, path->p_depth - depth, 0);
if (IS_ERR(bh))
return PTR_ERR(bh);
eh = ext_block_hdr(bh);
ix = EXT_FIRST_INDEX(eh);
- block = ext4_idx_pblock(ix);
put_bh(bh);
}
- bh = read_extent_tree_block(inode, block, path->p_depth - depth, 0);
+ bh = read_extent_tree_block(inode, ix, path->p_depth - depth, 0);
if (IS_ERR(bh))
return PTR_ERR(bh);
eh = ext_block_hdr(bh);
ext_debug(inode, "move to level %d (block %llu)\n",
i + 1, ext4_idx_pblock(path[i].p_idx));
memset(path + i + 1, 0, sizeof(*path));
- bh = read_extent_tree_block(inode,
- ext4_idx_pblock(path[i].p_idx), depth - i - 1,
- EXT4_EX_NOCACHE);
+ bh = read_extent_tree_block(inode, path[i].p_idx,
+ depth - i - 1,
+ EXT4_EX_NOCACHE);
if (IS_ERR(bh)) {
/* should we reset i_size? */
err = PTR_ERR(bh);
}
/*
- * ext4_access_path:
- * Function to access the path buffer for marking it dirty.
- * It also checks if there are sufficient credits left in the journal handle
- * to update path.
- */
-static int
-ext4_access_path(handle_t *handle, struct inode *inode,
- struct ext4_ext_path *path)
-{
- int credits, err;
-
- if (!ext4_handle_valid(handle))
- return 0;
-
- /*
- * Check if need to extend journal credits
- * 3 for leaf, sb, and inode plus 2 (bmap and group
- * descriptor) for each block group; assume two block
- * groups
- */
- credits = ext4_writepage_trans_blocks(inode);
- err = ext4_datasem_ensure_credits(handle, inode, 7, credits, 0);
- if (err < 0)
- return err;
-
- err = ext4_ext_get_access(handle, inode, path);
- return err;
-}
-
-/*
* ext4_ext_shift_path_extents:
* Shift the extents of a path structure lying between path[depth].p_ext
* and EXT_LAST_EXTENT(path[depth].p_hdr), by @shift blocks. @SHIFT tells
int depth, err = 0;
struct ext4_extent *ex_start, *ex_last;
bool update = false;
+ int credits, restart_credits;
depth = path->p_depth;
while (depth >= 0) {
return -EFSCORRUPTED;
ex_last = EXT_LAST_EXTENT(path[depth].p_hdr);
+ /* leaf + sb + inode */
+ credits = 3;
+ if (ex_start == EXT_FIRST_EXTENT(path[depth].p_hdr)) {
+ update = true;
+ /* extent tree + sb + inode */
+ credits = depth + 2;
+ }
- err = ext4_access_path(handle, inode, path + depth);
- if (err)
+ restart_credits = ext4_writepage_trans_blocks(inode);
+ err = ext4_datasem_ensure_credits(handle, inode, credits,
+ restart_credits, 0);
+ if (err) {
+ if (err > 0)
+ err = -EAGAIN;
goto out;
+ }
- if (ex_start == EXT_FIRST_EXTENT(path[depth].p_hdr))
- update = true;
+ err = ext4_ext_get_access(handle, inode, path + depth);
+ if (err)
+ goto out;
while (ex_start <= ex_last) {
if (SHIFT == SHIFT_LEFT) {
}
/* Update index too */
- err = ext4_access_path(handle, inode, path + depth);
+ err = ext4_ext_get_access(handle, inode, path + depth);
if (err)
goto out;
int ret = 0, depth;
struct ext4_extent *extent;
ext4_lblk_t stop, *iterator, ex_start, ex_end;
+ ext4_lblk_t tmp = EXT_MAX_BLOCKS;
/* Let path point to the last extent */
path = ext4_find_extent(inode, EXT_MAX_BLOCKS - 1, NULL,
* till we reach stop. In case of right shift, iterator points to stop
* and it is decreased till we reach start.
*/
+again:
if (SHIFT == SHIFT_LEFT)
iterator = &start;
else
iterator = &stop;
+ if (tmp != EXT_MAX_BLOCKS)
+ *iterator = tmp;
+
/*
* Its safe to start updating extents. Start and stop are unsigned, so
* in case of right shift if extent with 0 block is reached, iterator
}
}
+ tmp = *iterator;
if (SHIFT == SHIFT_LEFT) {
extent = EXT_LAST_EXTENT(path[depth].p_hdr);
*iterator = le32_to_cpu(extent->ee_block) +
}
ret = ext4_ext_shift_path_extents(path, shift, inode,
handle, SHIFT);
+ /* iterator can be NULL which means we should break */
+ if (ret == -EAGAIN)
+ goto again;
if (ret)
break;
}
int j, ret = 0;
struct ext4_map_blocks map;
+ if (ext4_test_inode_flag(inode, EXT4_INODE_INLINE_DATA))
+ return 0;
+
/* Determin the size of the file first */
path = ext4_find_extent(inode, EXT_MAX_BLOCKS - 1, NULL,
EXT4_EX_NOCACHE);
if (ret)
return ret;
- if (EXT4_INODE_SIZE(inode->i_sb) > EXT4_GOOD_OLD_INODE_SIZE)
+ if (ext4_test_inode_flag(inode, EXT4_INODE_INLINE_DATA))
+ inode_len = EXT4_INODE_SIZE(inode->i_sb);
+ else if (EXT4_INODE_SIZE(inode->i_sb) > EXT4_GOOD_OLD_INODE_SIZE)
inode_len += ei->i_extra_isize;
fc_inode.fc_ino = cpu_to_le32(inode->i_ino);
* crashing. This should be fixed but until then, we calculate
* the number of blocks the inode.
*/
- ext4_ext_replay_set_iblocks(inode);
+ if (!ext4_test_inode_flag(inode, EXT4_INODE_INLINE_DATA))
+ ext4_ext_replay_set_iblocks(inode);
inode->i_generation = le32_to_cpu(ext4_raw_inode(&iloc)->i_generation);
ext4_reset_inode_seed(inode);
}
cur = 0;
end = EXT_MAX_BLOCKS;
+ if (ext4_test_inode_flag(inode, EXT4_INODE_INLINE_DATA)) {
+ iput(inode);
+ continue;
+ }
while (cur < end) {
map.m_lblk = cur;
map.m_len = end - cur;
}
/*
- * the buffer head associated with a delayed and not unwritten
- * block found in the extent status cache must contain an
- * invalid block number and have its BH_New and BH_Delay bits
- * set, reflecting the state assigned when the block was
- * initially delayed allocated
+ * Delayed extent could be allocated by fallocate.
+ * So we need to check it.
*/
- if (ext4_es_is_delonly(&es)) {
- BUG_ON(bh->b_blocknr != invalid_block);
- BUG_ON(!buffer_new(bh));
- BUG_ON(!buffer_delay(bh));
+ if (ext4_es_is_delayed(&es) && !ext4_es_is_unwritten(&es)) {
+ map_bh(bh, inode->i_sb, invalid_block);
+ set_buffer_new(bh);
+ set_buffer_delay(bh);
return 0;
}
return err;
}
+static inline u64 ext4_inode_peek_iversion(const struct inode *inode)
+{
+ if (unlikely(EXT4_I(inode)->i_flags & EXT4_EA_INODE_FL))
+ return inode_peek_iversion_raw(inode);
+ else
+ return inode_peek_iversion(inode);
+}
+
+static int ext4_inode_blocks_set(struct ext4_inode *raw_inode,
+ struct ext4_inode_info *ei)
+{
+ struct inode *inode = &(ei->vfs_inode);
+ u64 i_blocks = READ_ONCE(inode->i_blocks);
+ struct super_block *sb = inode->i_sb;
+
+ if (i_blocks <= ~0U) {
+ /*
+ * i_blocks can be represented in a 32 bit variable
+ * as multiple of 512 bytes
+ */
+ raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
+ raw_inode->i_blocks_high = 0;
+ ext4_clear_inode_flag(inode, EXT4_INODE_HUGE_FILE);
+ return 0;
+ }
+
+ /*
+ * This should never happen since sb->s_maxbytes should not have
+ * allowed this, sb->s_maxbytes was set according to the huge_file
+ * feature in ext4_fill_super().
+ */
+ if (!ext4_has_feature_huge_file(sb))
+ return -EFSCORRUPTED;
+
+ if (i_blocks <= 0xffffffffffffULL) {
+ /*
+ * i_blocks can be represented in a 48 bit variable
+ * as multiple of 512 bytes
+ */
+ raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
+ raw_inode->i_blocks_high = cpu_to_le16(i_blocks >> 32);
+ ext4_clear_inode_flag(inode, EXT4_INODE_HUGE_FILE);
+ } else {
+ ext4_set_inode_flag(inode, EXT4_INODE_HUGE_FILE);
+ /* i_block is stored in file system block size */
+ i_blocks = i_blocks >> (inode->i_blkbits - 9);
+ raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
+ raw_inode->i_blocks_high = cpu_to_le16(i_blocks >> 32);
+ }
+ return 0;
+}
+
+static int ext4_fill_raw_inode(struct inode *inode, struct ext4_inode *raw_inode)
+{
+ struct ext4_inode_info *ei = EXT4_I(inode);
+ uid_t i_uid;
+ gid_t i_gid;
+ projid_t i_projid;
+ int block;
+ int err;
+
+ err = ext4_inode_blocks_set(raw_inode, ei);
+
+ raw_inode->i_mode = cpu_to_le16(inode->i_mode);
+ i_uid = i_uid_read(inode);
+ i_gid = i_gid_read(inode);
+ i_projid = from_kprojid(&init_user_ns, ei->i_projid);
+ if (!(test_opt(inode->i_sb, NO_UID32))) {
+ raw_inode->i_uid_low = cpu_to_le16(low_16_bits(i_uid));
+ raw_inode->i_gid_low = cpu_to_le16(low_16_bits(i_gid));
+ /*
+ * Fix up interoperability with old kernels. Otherwise,
+ * old inodes get re-used with the upper 16 bits of the
+ * uid/gid intact.
+ */
+ if (ei->i_dtime && list_empty(&ei->i_orphan)) {
+ raw_inode->i_uid_high = 0;
+ raw_inode->i_gid_high = 0;
+ } else {
+ raw_inode->i_uid_high =
+ cpu_to_le16(high_16_bits(i_uid));
+ raw_inode->i_gid_high =
+ cpu_to_le16(high_16_bits(i_gid));
+ }
+ } else {
+ raw_inode->i_uid_low = cpu_to_le16(fs_high2lowuid(i_uid));
+ raw_inode->i_gid_low = cpu_to_le16(fs_high2lowgid(i_gid));
+ raw_inode->i_uid_high = 0;
+ raw_inode->i_gid_high = 0;
+ }
+ raw_inode->i_links_count = cpu_to_le16(inode->i_nlink);
+
+ EXT4_INODE_SET_XTIME(i_ctime, inode, raw_inode);
+ EXT4_INODE_SET_XTIME(i_mtime, inode, raw_inode);
+ EXT4_INODE_SET_XTIME(i_atime, inode, raw_inode);
+ EXT4_EINODE_SET_XTIME(i_crtime, ei, raw_inode);
+
+ raw_inode->i_dtime = cpu_to_le32(ei->i_dtime);
+ raw_inode->i_flags = cpu_to_le32(ei->i_flags & 0xFFFFFFFF);
+ if (likely(!test_opt2(inode->i_sb, HURD_COMPAT)))
+ raw_inode->i_file_acl_high =
+ cpu_to_le16(ei->i_file_acl >> 32);
+ raw_inode->i_file_acl_lo = cpu_to_le32(ei->i_file_acl);
+ ext4_isize_set(raw_inode, ei->i_disksize);
+
+ raw_inode->i_generation = cpu_to_le32(inode->i_generation);
+ if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) {
+ if (old_valid_dev(inode->i_rdev)) {
+ raw_inode->i_block[0] =
+ cpu_to_le32(old_encode_dev(inode->i_rdev));
+ raw_inode->i_block[1] = 0;
+ } else {
+ raw_inode->i_block[0] = 0;
+ raw_inode->i_block[1] =
+ cpu_to_le32(new_encode_dev(inode->i_rdev));
+ raw_inode->i_block[2] = 0;
+ }
+ } else if (!ext4_has_inline_data(inode)) {
+ for (block = 0; block < EXT4_N_BLOCKS; block++)
+ raw_inode->i_block[block] = ei->i_data[block];
+ }
+
+ if (likely(!test_opt2(inode->i_sb, HURD_COMPAT))) {
+ u64 ivers = ext4_inode_peek_iversion(inode);
+
+ raw_inode->i_disk_version = cpu_to_le32(ivers);
+ if (ei->i_extra_isize) {
+ if (EXT4_FITS_IN_INODE(raw_inode, ei, i_version_hi))
+ raw_inode->i_version_hi =
+ cpu_to_le32(ivers >> 32);
+ raw_inode->i_extra_isize =
+ cpu_to_le16(ei->i_extra_isize);
+ }
+ }
+
+ if (i_projid != EXT4_DEF_PROJID &&
+ !ext4_has_feature_project(inode->i_sb))
+ err = err ?: -EFSCORRUPTED;
+
+ if (EXT4_INODE_SIZE(inode->i_sb) > EXT4_GOOD_OLD_INODE_SIZE &&
+ EXT4_FITS_IN_INODE(raw_inode, ei, i_projid))
+ raw_inode->i_projid = cpu_to_le32(i_projid);
+
+ ext4_inode_csum_set(inode, raw_inode, ei);
+ return err;
+}
+
/*
* ext4_get_inode_loc returns with an extra refcount against the inode's
- * underlying buffer_head on success. If 'in_mem' is true, we have all
- * data in memory that is needed to recreate the on-disk version of this
- * inode.
+ * underlying buffer_head on success. If we pass 'inode' and it does not
+ * have in-inode xattr, we have all inode data in memory that is needed
+ * to recreate the on-disk version of this inode.
*/
static int __ext4_get_inode_loc(struct super_block *sb, unsigned long ino,
- struct ext4_iloc *iloc, int in_mem,
+ struct inode *inode, struct ext4_iloc *iloc,
ext4_fsblk_t *ret_block)
{
struct ext4_group_desc *gdp;
* is the only valid inode in the block, we need not read the
* block.
*/
- if (in_mem) {
+ if (inode && !ext4_test_inode_state(inode, EXT4_STATE_XATTR)) {
struct buffer_head *bitmap_bh;
int i, start;
}
brelse(bitmap_bh);
if (i == start + inodes_per_block) {
+ struct ext4_inode *raw_inode =
+ (struct ext4_inode *) (bh->b_data + iloc->offset);
+
/* all other inodes are free, so skip I/O */
memset(bh->b_data, 0, bh->b_size);
+ if (!ext4_test_inode_state(inode, EXT4_STATE_NEW))
+ ext4_fill_raw_inode(inode, raw_inode);
set_buffer_uptodate(bh);
unlock_buffer(bh);
goto has_buffer;
ext4_fsblk_t err_blk;
int ret;
- ret = __ext4_get_inode_loc(inode->i_sb, inode->i_ino, iloc, 0,
+ ret = __ext4_get_inode_loc(inode->i_sb, inode->i_ino, NULL, iloc,
&err_blk);
if (ret == -EIO)
ext4_fsblk_t err_blk;
int ret;
- /* We have all inode data except xattrs in memory here. */
- ret = __ext4_get_inode_loc(inode->i_sb, inode->i_ino, iloc,
- !ext4_test_inode_state(inode, EXT4_STATE_XATTR), &err_blk);
+ ret = __ext4_get_inode_loc(inode->i_sb, inode->i_ino, inode, iloc,
+ &err_blk);
if (ret == -EIO)
ext4_error_inode_block(inode, err_blk, EIO,
int ext4_get_fc_inode_loc(struct super_block *sb, unsigned long ino,
struct ext4_iloc *iloc)
{
- return __ext4_get_inode_loc(sb, ino, iloc, 0, NULL);
+ return __ext4_get_inode_loc(sb, ino, NULL, iloc, NULL);
}
static bool ext4_should_enable_dax(struct inode *inode)
else
inode_set_iversion_queried(inode, val);
}
-static inline u64 ext4_inode_peek_iversion(const struct inode *inode)
-{
- if (unlikely(EXT4_I(inode)->i_flags & EXT4_EA_INODE_FL))
- return inode_peek_iversion_raw(inode);
- else
- return inode_peek_iversion(inode);
-}
struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
ext4_iget_flags flags, const char *function,
return ERR_PTR(ret);
}
-static int ext4_inode_blocks_set(handle_t *handle,
- struct ext4_inode *raw_inode,
- struct ext4_inode_info *ei)
-{
- struct inode *inode = &(ei->vfs_inode);
- u64 i_blocks = READ_ONCE(inode->i_blocks);
- struct super_block *sb = inode->i_sb;
-
- if (i_blocks <= ~0U) {
- /*
- * i_blocks can be represented in a 32 bit variable
- * as multiple of 512 bytes
- */
- raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
- raw_inode->i_blocks_high = 0;
- ext4_clear_inode_flag(inode, EXT4_INODE_HUGE_FILE);
- return 0;
- }
-
- /*
- * This should never happen since sb->s_maxbytes should not have
- * allowed this, sb->s_maxbytes was set according to the huge_file
- * feature in ext4_fill_super().
- */
- if (!ext4_has_feature_huge_file(sb))
- return -EFSCORRUPTED;
-
- if (i_blocks <= 0xffffffffffffULL) {
- /*
- * i_blocks can be represented in a 48 bit variable
- * as multiple of 512 bytes
- */
- raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
- raw_inode->i_blocks_high = cpu_to_le16(i_blocks >> 32);
- ext4_clear_inode_flag(inode, EXT4_INODE_HUGE_FILE);
- } else {
- ext4_set_inode_flag(inode, EXT4_INODE_HUGE_FILE);
- /* i_block is stored in file system block size */
- i_blocks = i_blocks >> (inode->i_blkbits - 9);
- raw_inode->i_blocks_lo = cpu_to_le32(i_blocks);
- raw_inode->i_blocks_high = cpu_to_le16(i_blocks >> 32);
- }
- return 0;
-}
-
static void __ext4_update_other_inode_time(struct super_block *sb,
unsigned long orig_ino,
unsigned long ino,
struct ext4_inode_info *ei = EXT4_I(inode);
struct buffer_head *bh = iloc->bh;
struct super_block *sb = inode->i_sb;
- int err = 0, block;
+ int err;
int need_datasync = 0, set_large_file = 0;
- uid_t i_uid;
- gid_t i_gid;
- projid_t i_projid;
spin_lock(&ei->i_raw_lock);
if (ext4_test_inode_state(inode, EXT4_STATE_NEW))
memset(raw_inode, 0, EXT4_SB(inode->i_sb)->s_inode_size);
- err = ext4_inode_blocks_set(handle, raw_inode, ei);
-
- raw_inode->i_mode = cpu_to_le16(inode->i_mode);
- i_uid = i_uid_read(inode);
- i_gid = i_gid_read(inode);
- i_projid = from_kprojid(&init_user_ns, ei->i_projid);
- if (!(test_opt(inode->i_sb, NO_UID32))) {
- raw_inode->i_uid_low = cpu_to_le16(low_16_bits(i_uid));
- raw_inode->i_gid_low = cpu_to_le16(low_16_bits(i_gid));
- /*
- * Fix up interoperability with old kernels. Otherwise,
- * old inodes get re-used with the upper 16 bits of the
- * uid/gid intact.
- */
- if (ei->i_dtime && list_empty(&ei->i_orphan)) {
- raw_inode->i_uid_high = 0;
- raw_inode->i_gid_high = 0;
- } else {
- raw_inode->i_uid_high =
- cpu_to_le16(high_16_bits(i_uid));
- raw_inode->i_gid_high =
- cpu_to_le16(high_16_bits(i_gid));
- }
- } else {
- raw_inode->i_uid_low = cpu_to_le16(fs_high2lowuid(i_uid));
- raw_inode->i_gid_low = cpu_to_le16(fs_high2lowgid(i_gid));
- raw_inode->i_uid_high = 0;
- raw_inode->i_gid_high = 0;
- }
- raw_inode->i_links_count = cpu_to_le16(inode->i_nlink);
-
- EXT4_INODE_SET_XTIME(i_ctime, inode, raw_inode);
- EXT4_INODE_SET_XTIME(i_mtime, inode, raw_inode);
- EXT4_INODE_SET_XTIME(i_atime, inode, raw_inode);
- EXT4_EINODE_SET_XTIME(i_crtime, ei, raw_inode);
-
- raw_inode->i_dtime = cpu_to_le32(ei->i_dtime);
- raw_inode->i_flags = cpu_to_le32(ei->i_flags & 0xFFFFFFFF);
- if (likely(!test_opt2(inode->i_sb, HURD_COMPAT)))
- raw_inode->i_file_acl_high =
- cpu_to_le16(ei->i_file_acl >> 32);
- raw_inode->i_file_acl_lo = cpu_to_le32(ei->i_file_acl);
- if (READ_ONCE(ei->i_disksize) != ext4_isize(inode->i_sb, raw_inode)) {
- ext4_isize_set(raw_inode, ei->i_disksize);
+ if (READ_ONCE(ei->i_disksize) != ext4_isize(inode->i_sb, raw_inode))
need_datasync = 1;
- }
if (ei->i_disksize > 0x7fffffffULL) {
if (!ext4_has_feature_large_file(sb) ||
- EXT4_SB(sb)->s_es->s_rev_level ==
- cpu_to_le32(EXT4_GOOD_OLD_REV))
+ EXT4_SB(sb)->s_es->s_rev_level == cpu_to_le32(EXT4_GOOD_OLD_REV))
set_large_file = 1;
}
- raw_inode->i_generation = cpu_to_le32(inode->i_generation);
- if (S_ISCHR(inode->i_mode) || S_ISBLK(inode->i_mode)) {
- if (old_valid_dev(inode->i_rdev)) {
- raw_inode->i_block[0] =
- cpu_to_le32(old_encode_dev(inode->i_rdev));
- raw_inode->i_block[1] = 0;
- } else {
- raw_inode->i_block[0] = 0;
- raw_inode->i_block[1] =
- cpu_to_le32(new_encode_dev(inode->i_rdev));
- raw_inode->i_block[2] = 0;
- }
- } else if (!ext4_has_inline_data(inode)) {
- for (block = 0; block < EXT4_N_BLOCKS; block++)
- raw_inode->i_block[block] = ei->i_data[block];
- }
-
- if (likely(!test_opt2(inode->i_sb, HURD_COMPAT))) {
- u64 ivers = ext4_inode_peek_iversion(inode);
-
- raw_inode->i_disk_version = cpu_to_le32(ivers);
- if (ei->i_extra_isize) {
- if (EXT4_FITS_IN_INODE(raw_inode, ei, i_version_hi))
- raw_inode->i_version_hi =
- cpu_to_le32(ivers >> 32);
- raw_inode->i_extra_isize =
- cpu_to_le16(ei->i_extra_isize);
- }
- }
- if (i_projid != EXT4_DEF_PROJID &&
- !ext4_has_feature_project(inode->i_sb))
- err = err ?: -EFSCORRUPTED;
-
- if (EXT4_INODE_SIZE(inode->i_sb) > EXT4_GOOD_OLD_INODE_SIZE &&
- EXT4_FITS_IN_INODE(raw_inode, ei, i_projid))
- raw_inode->i_projid = cpu_to_le32(i_projid);
-
- ext4_inode_csum_set(inode, raw_inode, ei);
+ err = ext4_fill_raw_inode(inode, raw_inode);
spin_unlock(&ei->i_raw_lock);
if (err) {
EXT4_ERROR_INODE(inode, "corrupted inode contents");
{
ext4_grpblk_t next, count, free_count;
void *bitmap;
- int ret = 0;
bitmap = e4b->bd_bitmap;
start = (e4b->bd_info->bb_first_free > start) ?
next = mb_find_next_bit(bitmap, max + 1, start);
if ((next - start) >= minblocks) {
- ret = ext4_trim_extent(sb, start, next - start, e4b);
+ int ret = ext4_trim_extent(sb, start, next - start, e4b);
+
if (ret && ret != -EOPNOTSUPP)
break;
- ret = 0;
count += next - start;
}
free_count += next - start;
fname->hinfo.minor_hash !=
EXT4_DIRENT_MINOR_HASH(de)) {
- return 0;
+ return false;
}
}
return !ext4_ci_compare(parent, &cf, de->name,
io_end->inode = inode;
INIT_LIST_HEAD(&io_end->list);
INIT_LIST_HEAD(&io_end->list_vec);
- atomic_set(&io_end->count, 1);
+ refcount_set(&io_end->count, 1);
}
return io_end;
}
void ext4_put_io_end_defer(ext4_io_end_t *io_end)
{
- if (atomic_dec_and_test(&io_end->count)) {
+ if (refcount_dec_and_test(&io_end->count)) {
if (!(io_end->flag & EXT4_IO_END_UNWRITTEN) ||
list_empty(&io_end->list_vec)) {
ext4_release_io_end(io_end);
{
int err = 0;
- if (atomic_dec_and_test(&io_end->count)) {
+ if (refcount_dec_and_test(&io_end->count)) {
if (io_end->flag & EXT4_IO_END_UNWRITTEN) {
err = ext4_convert_unwritten_io_end_vec(io_end->handle,
io_end);
ext4_io_end_t *ext4_get_io_end(ext4_io_end_t *io_end)
{
- atomic_inc(&io_end->count);
+ refcount_inc(&io_end->count);
return io_end;
}
#include <linux/part_stat.h>
#include <linux/kthread.h>
#include <linux/freezer.h>
+#include <linux/fsnotify.h>
#include "ext4.h"
#include "ext4_extents.h" /* Needed for trace points definition */
sb->s_id, function, line, current->comm, &vaf);
va_end(args);
}
+ fsnotify_sb_error(sb, NULL, error ? error : EFSCORRUPTED);
+
ext4_handle_error(sb, force_ro, error, 0, block, function, line);
}
current->comm, &vaf);
va_end(args);
}
+ fsnotify_sb_error(inode->i_sb, inode, error ? error : EFSCORRUPTED);
+
ext4_handle_error(inode->i_sb, false, error, inode->i_ino, block,
function, line);
}
current->comm, path, &vaf);
va_end(args);
}
+ fsnotify_sb_error(inode->i_sb, inode, EFSCORRUPTED);
+
ext4_handle_error(inode->i_sb, false, EFSCORRUPTED, inode->i_ino, block,
function, line);
}
printk(KERN_CRIT "EXT4-fs error (device %s) in %s:%d: %s\n",
sb->s_id, function, line, errstr);
}
+ fsnotify_sb_error(sb, NULL, errno ? errno : EFSCORRUPTED);
ext4_handle_error(sb, false, -errno, 0, 0, function, line);
}
struct super_block *sb = elr->lr_super;
ext4_group_t ngroups = EXT4_SB(sb)->s_groups_count;
ext4_group_t group = elr->lr_next_group;
- unsigned long timeout = 0;
unsigned int prefetch_ios = 0;
int ret = 0;
+ u64 start_time;
if (elr->lr_mode == EXT4_LI_MODE_PREFETCH_BBITMAP) {
elr->lr_next_group = ext4_mb_prefetch(sb, group,
ret = 1;
if (!ret) {
- timeout = jiffies;
+ start_time = ktime_get_real_ns();
ret = ext4_init_inode_table(sb, group,
elr->lr_timeout ? 0 : 1);
trace_ext4_lazy_itable_init(sb, group);
if (elr->lr_timeout == 0) {
- timeout = (jiffies - timeout) *
- EXT4_SB(elr->lr_super)->s_li_wait_mult;
- elr->lr_timeout = timeout;
+ elr->lr_timeout = nsecs_to_jiffies((ktime_get_real_ns() - start_time) *
+ EXT4_SB(elr->lr_super)->s_li_wait_mult);
}
elr->lr_next_sched = jiffies + elr->lr_timeout;
elr->lr_next_group = group + 1;
struct ext4_sb_info *sbi = EXT4_SB(sb);
unsigned long old_sb_flags, vfs_flags;
struct ext4_mount_options old_opts;
- int enable_quota = 0;
ext4_group_t g;
int err = 0;
#ifdef CONFIG_QUOTA
+ int enable_quota = 0;
int i, j;
char *to_free[EXT4_MAXQUOTAS];
#endif
}
if (ext4_test_mount_flag(sb, EXT4_MF_FS_ABORTED))
- ext4_abort(sb, EXT4_ERR_ESHUTDOWN, "Abort forced by user");
+ ext4_abort(sb, ESHUTDOWN, "Abort forced by user");
sb->s_flags = (sb->s_flags & ~SB_POSIXACL) |
(test_opt(sb, POSIX_ACL) ? SB_POSIXACL : 0);
err = -EROFS;
goto restore_opts;
}
+#ifdef CONFIG_QUOTA
enable_quota = 1;
+#endif
}
}
return PTR_ERR(inode);
}
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err) {
iput(inode);
goto err_out;
}
#ifdef CONFIG_QUOTA
- /* Needed for iput() to work correctly and not trash data */
- sbi->sb->s_flags |= SB_ACTIVE;
-
/*
* Turn on quotas which were not enabled for read-only mounts if
* filesystem has quota feature, so that they are updated correctly.
if (!is_journalled_quota(sbi))
return false;
- down_write(&sbi->quota_sem);
+ if (!down_write_trylock(&sbi->quota_sem))
+ return true;
if (is_sbi_flag_set(sbi, SBI_QUOTA_SKIP_FLUSH)) {
ret = false;
} else if (is_sbi_flag_set(sbi, SBI_QUOTA_NEED_REPAIR)) {
static int zstd_init_compress_ctx(struct compress_ctx *cc)
{
- ZSTD_parameters params;
- ZSTD_CStream *stream;
+ zstd_parameters params;
+ zstd_cstream *stream;
void *workspace;
unsigned int workspace_size;
unsigned char level = F2FS_I(cc->inode)->i_compress_flag >>
if (!level)
level = F2FS_ZSTD_DEFAULT_CLEVEL;
- params = ZSTD_getParams(level, cc->rlen, 0);
- workspace_size = ZSTD_CStreamWorkspaceBound(params.cParams);
+ params = zstd_get_params(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen);
+ workspace_size = zstd_cstream_workspace_bound(¶ms.cParams);
workspace = f2fs_kvmalloc(F2FS_I_SB(cc->inode),
workspace_size, GFP_NOFS);
if (!workspace)
return -ENOMEM;
- stream = ZSTD_initCStream(params, 0, workspace, workspace_size);
+ stream = zstd_init_cstream(¶ms, 0, workspace, workspace_size);
if (!stream) {
- printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initCStream failed\n",
+ printk_ratelimited("%sF2FS-fs (%s): %s zstd_init_cstream failed\n",
KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id,
__func__);
kvfree(workspace);
static int zstd_compress_pages(struct compress_ctx *cc)
{
- ZSTD_CStream *stream = cc->private2;
- ZSTD_inBuffer inbuf;
- ZSTD_outBuffer outbuf;
+ zstd_cstream *stream = cc->private2;
+ zstd_in_buffer inbuf;
+ zstd_out_buffer outbuf;
int src_size = cc->rlen;
int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
int ret;
outbuf.dst = cc->cbuf->cdata;
outbuf.size = dst_size;
- ret = ZSTD_compressStream(stream, &outbuf, &inbuf);
- if (ZSTD_isError(ret)) {
- printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n",
+ ret = zstd_compress_stream(stream, &outbuf, &inbuf);
+ if (zstd_is_error(ret)) {
+ printk_ratelimited("%sF2FS-fs (%s): %s zstd_compress_stream failed, ret: %d\n",
KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id,
- __func__, ZSTD_getErrorCode(ret));
+ __func__, zstd_get_error_code(ret));
return -EIO;
}
- ret = ZSTD_endStream(stream, &outbuf);
- if (ZSTD_isError(ret)) {
- printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_endStream returned %d\n",
+ ret = zstd_end_stream(stream, &outbuf);
+ if (zstd_is_error(ret)) {
+ printk_ratelimited("%sF2FS-fs (%s): %s zstd_end_stream returned %d\n",
KERN_ERR, F2FS_I_SB(cc->inode)->sb->s_id,
- __func__, ZSTD_getErrorCode(ret));
+ __func__, zstd_get_error_code(ret));
return -EIO;
}
static int zstd_init_decompress_ctx(struct decompress_io_ctx *dic)
{
- ZSTD_DStream *stream;
+ zstd_dstream *stream;
void *workspace;
unsigned int workspace_size;
unsigned int max_window_size =
MAX_COMPRESS_WINDOW_SIZE(dic->log_cluster_size);
- workspace_size = ZSTD_DStreamWorkspaceBound(max_window_size);
+ workspace_size = zstd_dstream_workspace_bound(max_window_size);
workspace = f2fs_kvmalloc(F2FS_I_SB(dic->inode),
workspace_size, GFP_NOFS);
if (!workspace)
return -ENOMEM;
- stream = ZSTD_initDStream(max_window_size, workspace, workspace_size);
+ stream = zstd_init_dstream(max_window_size, workspace, workspace_size);
if (!stream) {
- printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_initDStream failed\n",
+ printk_ratelimited("%sF2FS-fs (%s): %s zstd_init_dstream failed\n",
KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id,
__func__);
kvfree(workspace);
static int zstd_decompress_pages(struct decompress_io_ctx *dic)
{
- ZSTD_DStream *stream = dic->private2;
- ZSTD_inBuffer inbuf;
- ZSTD_outBuffer outbuf;
+ zstd_dstream *stream = dic->private2;
+ zstd_in_buffer inbuf;
+ zstd_out_buffer outbuf;
int ret;
inbuf.pos = 0;
outbuf.dst = dic->rbuf;
outbuf.size = dic->rlen;
- ret = ZSTD_decompressStream(stream, &outbuf, &inbuf);
- if (ZSTD_isError(ret)) {
- printk_ratelimited("%sF2FS-fs (%s): %s ZSTD_compressStream failed, ret: %d\n",
+ ret = zstd_decompress_stream(stream, &outbuf, &inbuf);
+ if (zstd_is_error(ret)) {
+ printk_ratelimited("%sF2FS-fs (%s): %s zstd_decompress_stream failed, ret: %d\n",
KERN_ERR, F2FS_I_SB(dic->inode)->sb->s_id,
- __func__, ZSTD_getErrorCode(ret));
+ __func__, zstd_get_error_code(ret));
return -EIO;
}
return is_page_in_cluster(cc, index);
}
+bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
+ int index, int nr_pages)
+{
+ unsigned long pgidx;
+ int i;
+
+ if (nr_pages - index < cc->cluster_size)
+ return false;
+
+ pgidx = pvec->pages[index]->index;
+
+ for (i = 1; i < cc->cluster_size; i++) {
+ if (pvec->pages[index + i]->index != pgidx + i)
+ return false;
+ }
+
+ return true;
+}
+
static bool cluster_has_invalid_data(struct compress_ctx *cc)
{
loff_t i_size = i_size_read(cc->inode);
if (cluster_may_compress(cc)) {
err = f2fs_compress_pages(cc);
if (err == -EAGAIN) {
+ add_compr_block_stat(cc->inode, cc->cluster_size);
goto write;
} else if (err) {
f2fs_put_rpages_wbc(cc, wbc, true, 1);
struct extent_info ei = {0, };
block_t blkaddr;
unsigned int start_pgofs;
+ int bidx = 0;
if (!maxblocks)
return 0;
+ map->m_bdev = inode->i_sb->s_bdev;
+ map->m_multidev_dio =
+ f2fs_allow_multi_device_dio(F2FS_I_SB(inode), flag);
+
map->m_len = 0;
map->m_flags = 0;
if (flag == F2FS_GET_BLOCK_DIO)
f2fs_wait_on_block_writeback_range(inode,
map->m_pblk, map->m_len);
+
+ if (map->m_multidev_dio) {
+ block_t blk_addr = map->m_pblk;
+
+ bidx = f2fs_target_device_index(sbi, map->m_pblk);
+
+ map->m_bdev = FDEV(bidx).bdev;
+ map->m_pblk -= FDEV(bidx).start_blk;
+ map->m_len = min(map->m_len,
+ FDEV(bidx).end_blk + 1 - map->m_pblk);
+
+ if (map->m_may_create)
+ f2fs_update_device_state(sbi, inode->i_ino,
+ blk_addr, map->m_len);
+ }
goto out;
}
if (flag == F2FS_GET_BLOCK_PRE_AIO)
goto skip;
+ if (map->m_multidev_dio)
+ bidx = f2fs_target_device_index(sbi, blkaddr);
+
if (map->m_len == 0) {
/* preallocated unwritten block should be mapped for fiemap. */
if (blkaddr == NEW_ADDR)
map->m_pblk = blkaddr;
map->m_len = 1;
+
+ if (map->m_multidev_dio)
+ map->m_bdev = FDEV(bidx).bdev;
} else if ((map->m_pblk != NEW_ADDR &&
blkaddr == (map->m_pblk + ofs)) ||
(map->m_pblk == NEW_ADDR && blkaddr == NEW_ADDR) ||
flag == F2FS_GET_BLOCK_PRE_DIO) {
+ if (map->m_multidev_dio && map->m_bdev != FDEV(bidx).bdev)
+ goto sync_out;
ofs++;
map->m_len++;
} else {
sync_out:
- /* for hardware encryption, but to avoid potential issue in future */
- if (flag == F2FS_GET_BLOCK_DIO && map->m_flags & F2FS_MAP_MAPPED)
+ if (flag == F2FS_GET_BLOCK_DIO && map->m_flags & F2FS_MAP_MAPPED) {
+ /*
+ * for hardware encryption, but to avoid potential issue
+ * in future
+ */
f2fs_wait_on_block_writeback_range(inode,
map->m_pblk, map->m_len);
+ invalidate_mapping_pages(META_MAPPING(sbi),
+ map->m_pblk, map->m_pblk);
+
+ if (map->m_multidev_dio) {
+ block_t blk_addr = map->m_pblk;
+
+ bidx = f2fs_target_device_index(sbi, map->m_pblk);
+
+ map->m_bdev = FDEV(bidx).bdev;
+ map->m_pblk -= FDEV(bidx).start_blk;
+
+ if (map->m_may_create)
+ f2fs_update_device_state(sbi, inode->i_ino,
+ blk_addr, map->m_len);
+
+ f2fs_bug_on(sbi, blk_addr + map->m_len >
+ FDEV(bidx).end_blk + 1);
+ }
+ }
if (flag == F2FS_GET_BLOCK_PRECACHE) {
if (map->m_flags & F2FS_MAP_MAPPED) {
f2fs_balance_fs(sbi, dn.node_changed);
}
out:
- trace_f2fs_map_blocks(inode, map, err);
+ trace_f2fs_map_blocks(inode, map, create, flag, err);
return err;
}
map_bh(bh, inode->i_sb, map.m_pblk);
bh->b_state = (bh->b_state & ~F2FS_MAP_FLAGS) | map.m_flags;
bh->b_size = blks_to_bytes(inode, map.m_len);
+
+ if (map.m_multidev_dio)
+ bh->b_bdev = map.m_bdev;
}
return err;
}
need_readd = false;
#ifdef CONFIG_F2FS_FS_COMPRESSION
if (f2fs_compressed_file(inode)) {
+ void *fsdata = NULL;
+ struct page *pagep;
+ int ret2;
+
ret = f2fs_init_compress_ctx(&cc);
if (ret) {
done = 1;
if (unlikely(f2fs_cp_error(sbi)))
goto lock_page;
- if (f2fs_cluster_is_empty(&cc)) {
- void *fsdata = NULL;
- struct page *pagep;
- int ret2;
+ if (!f2fs_cluster_is_empty(&cc))
+ goto lock_page;
- ret2 = f2fs_prepare_compress_overwrite(
+ ret2 = f2fs_prepare_compress_overwrite(
inode, &pagep,
page->index, &fsdata);
- if (ret2 < 0) {
- ret = ret2;
- done = 1;
- break;
- } else if (ret2 &&
- !f2fs_compress_write_end(inode,
- fsdata, page->index,
- 1)) {
- retry = 1;
- break;
- }
- } else {
- goto lock_page;
+ if (ret2 < 0) {
+ ret = ret2;
+ done = 1;
+ break;
+ } else if (ret2 &&
+ (!f2fs_compress_write_end(inode,
+ fsdata, page->index, 1) ||
+ !f2fs_all_cluster_page_loaded(&cc,
+ &pvec, i, nr_pages))) {
+ retry = 1;
+ break;
}
}
#endif
FAULT_DISCARD,
FAULT_WRITE_IO,
FAULT_SLAB_ALLOC,
+ FAULT_DQUOT_INIT,
FAULT_MAX,
};
#define MAX_DIR_RA_PAGES 4 /* maximum ra pages of dir */
+/* dirty segments threshold for triggering CP */
+#define DEFAULT_DIRTY_THRESHOLD 4
+
/* for in-memory extent cache entry */
#define F2FS_MIN_EXTENT_LEN 64 /* minimum extent length */
F2FS_MAP_UNWRITTEN)
struct f2fs_map_blocks {
+ struct block_device *m_bdev; /* for multi-device dio */
block_t m_pblk;
block_t m_lblk;
unsigned int m_len;
pgoff_t *m_next_extent; /* point to next possible extent */
int m_seg_type;
bool m_may_create; /* indicate it is from write path */
+ bool m_multidev_dio; /* indicate it allows multi-device dio */
};
/* for flag in get_data_block */
};
enum {
- FS_MODE_ADAPTIVE, /* use both lfs/ssr allocation */
- FS_MODE_LFS, /* use lfs allocation only */
+ FS_MODE_ADAPTIVE, /* use both lfs/ssr allocation */
+ FS_MODE_LFS, /* use lfs allocation only */
+ FS_MODE_FRAGMENT_SEG, /* segment fragmentation mode */
+ FS_MODE_FRAGMENT_BLK, /* block fragmentation mode */
};
enum {
/* For shrinker support */
struct list_head s_list;
+ struct mutex umount_mutex;
+ unsigned int shrinker_run_no;
+
+ /* For multi devices */
int s_ndevs; /* number of devices */
struct f2fs_dev_info *devs; /* for device list */
unsigned int dirty_device; /* for checkpoint data flush */
spinlock_t dev_lock; /* protect dirty_device */
- struct mutex umount_mutex;
- unsigned int shrinker_run_no;
+ bool aligned_blksize; /* all devices has the same logical blksize */
/* For write statistics */
u64 sectors_written_start;
unsigned long seq_file_ra_mul; /* multiplier for ra_pages of seq. files in fadvise */
+ int max_fragment_chunk; /* max chunk size for block fragmentation mode */
+ int max_fragment_hole; /* max hole size for block fragmentation mode */
+
#ifdef CONFIG_F2FS_FS_COMPRESSION
struct kmem_cache *page_array_slab; /* page array entry */
unsigned int page_array_slab_size; /* default page array slab size */
*/
int f2fs_inode_dirtied(struct inode *inode, bool sync);
void f2fs_inode_synced(struct inode *inode);
+int f2fs_dquot_initialize(struct inode *inode);
int f2fs_enable_quota_files(struct f2fs_sb_info *sbi, bool rdonly);
int f2fs_quota_sync(struct super_block *sb, int type);
loff_t max_file_blocks(struct inode *inode);
block_t old_blkaddr, block_t *new_blkaddr,
struct f2fs_summary *sum, int type,
struct f2fs_io_info *fio);
+void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino,
+ block_t blkaddr, unsigned int blkcnt);
void f2fs_wait_on_page_writeback(struct page *page,
enum page_type type, bool ordered, bool locked);
void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr);
unsigned int f2fs_usable_blks_in_seg(struct f2fs_sb_info *sbi,
unsigned int segno);
+#define DEF_FRAGMENT_SIZE 4
+#define MIN_FRAGMENT_SIZE 1
+#define MAX_FRAGMENT_SIZE 512
+
+static inline bool f2fs_need_rand_seg(struct f2fs_sb_info *sbi)
+{
+ return F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG ||
+ F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK;
+}
+
/*
* checkpoint.c
*/
block_t blkaddr);
bool f2fs_cluster_is_empty(struct compress_ctx *cc);
bool f2fs_cluster_can_merge_page(struct compress_ctx *cc, pgoff_t index);
+bool f2fs_all_cluster_page_loaded(struct compress_ctx *cc, struct pagevec *pvec,
+ int index, int nr_pages);
bool f2fs_sanity_check_cluster(struct dnode_of_data *dn);
void f2fs_compress_ctx_add_page(struct compress_ctx *cc, struct page *page);
int f2fs_write_multi_pages(struct compress_ctx *cc,
if (!f2fs_compressed_file(inode))
return true;
- if (S_ISREG(inode->i_mode) &&
- (get_dirty_pages(inode) || atomic_read(&fi->i_compr_blocks)))
+ if (S_ISREG(inode->i_mode) && F2FS_HAS_BLOCKS(inode))
return false;
fi->i_flags &= ~F2FS_COMPR_FL;
return align & blocksize_mask;
}
+static inline bool f2fs_allow_multi_device_dio(struct f2fs_sb_info *sbi,
+ int flag)
+{
+ if (!f2fs_is_multi_device(sbi))
+ return false;
+ if (flag != F2FS_GET_BLOCK_DIO)
+ return false;
+ return sbi->aligned_blksize;
+}
+
static inline bool f2fs_force_buffered_io(struct inode *inode,
struct kiocb *iocb, struct iov_iter *iter)
{
if (f2fs_post_read_required(inode))
return true;
- if (f2fs_is_multi_device(sbi))
+
+ /* disallow direct IO if any of devices has unaligned blksize */
+ if (f2fs_is_multi_device(sbi) && !sbi->aligned_blksize)
return true;
/*
* for blkzoned device, fallback direct IO to buffered IO, so
return -EIO;
}
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
return err;
if (is_quota_modification(inode, attr)) {
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
}
}
f2fs_put_page(ipage, 1);
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
#include <linux/delay.h>
#include <linux/freezer.h>
#include <linux/sched/signal.h>
+#include <linux/random.h>
#include "f2fs.h"
#include "node.h"
p->max_search = sbi->max_victim_search;
/* let's select beginning hot/small space first in no_heap mode*/
- if (test_opt(sbi, NOHEAP) &&
+ if (f2fs_need_rand_seg(sbi))
+ p->offset = prandom_u32() % (MAIN_SECS(sbi) * sbi->segs_per_sec);
+ else if (test_opt(sbi, NOHEAP) &&
(type == CURSEG_HOT_DATA || IS_NODESEG(type)))
p->offset = 0;
else
f2fs_hw_is_readonly(sbi) || f2fs_readonly(sbi->sb))
return 0;
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
inode->i_op = &f2fs_dir_inode_operations;
inode->i_fop = &f2fs_dir_operations;
inode->i_mapping->a_ops = &f2fs_dblock_aops;
- inode_nohighmem(inode);
+ mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
} else if (S_ISLNK(inode->i_mode)) {
if (file_is_encrypt(inode))
inode->i_op = &f2fs_encrypted_symlink_inode_operations;
if (inode->i_nlink || is_bad_inode(inode))
goto no_delete;
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err) {
err = 0;
set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
if (err)
goto fail_drop;
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
goto fail_drop;
if (!f2fs_is_checkpoint_ready(sbi))
return -ENOSPC;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
F2FS_I(old_dentry->d_inode)->i_projid)))
return -EXDEV;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
return 0;
}
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
goto fail;
}
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
goto fail;
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
goto fail;
if (err)
return err;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
inode->i_op = &f2fs_dir_inode_operations;
inode->i_fop = &f2fs_dir_operations;
inode->i_mapping->a_ops = &f2fs_dblock_aops;
- inode_nohighmem(inode);
+ mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
set_inode_flag(inode, FI_INC_LINK);
f2fs_lock_op(sbi);
if (!f2fs_is_checkpoint_ready(sbi))
return -ENOSPC;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
struct inode *inode;
int err;
- err = dquot_initialize(dir);
+ err = f2fs_dquot_initialize(dir);
if (err)
return err;
return err;
}
- err = dquot_initialize(old_dir);
+ err = f2fs_dquot_initialize(old_dir);
if (err)
goto out;
- err = dquot_initialize(new_dir);
+ err = f2fs_dquot_initialize(new_dir);
if (err)
goto out;
if (new_inode) {
- err = dquot_initialize(new_inode);
+ err = f2fs_dquot_initialize(new_inode);
if (err)
goto out;
}
F2FS_I(new_dentry->d_inode)->i_projid)))
return -EXDEV;
- err = dquot_initialize(old_dir);
+ err = f2fs_dquot_initialize(old_dir);
if (err)
goto out;
- err = dquot_initialize(new_dir);
+ err = f2fs_dquot_initialize(new_dir);
if (err)
goto out;
nid, nid_of_node(page), ino_of_node(page),
ofs_of_node(page), cpver_of_node(page),
next_blkaddr_of_node(page));
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
err = -EINVAL;
out_err:
ClearPageUptodate(page);
return NM_I(sbi)->nat_cnt[TOTAL_NAT] >= DEF_NAT_CACHE_THRESHOLD;
}
-static inline bool excess_dirty_nodes(struct f2fs_sb_info *sbi)
-{
- return get_pages(sbi, F2FS_DIRTY_NODES) >= sbi->blocks_per_seg * 8;
-}
-
enum mem_type {
FREE_NIDS, /* indicates the free nid list */
NAT_ENTRIES, /* indicates the cached nat entry */
if (IS_ERR(inode))
return ERR_CAST(inode);
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
goto err_out;
goto out_put;
}
- err = dquot_initialize(einode);
+ err = f2fs_dquot_initialize(einode);
if (err) {
iput(einode);
goto out_put;
if (IS_ERR(inode))
return PTR_ERR(inode);
- ret = dquot_initialize(inode);
+ ret = f2fs_dquot_initialize(inode);
if (ret) {
iput(inode);
return ret;
}
#ifdef CONFIG_QUOTA
- /* Needed for iput() to work correctly and not trash data */
- sbi->sb->s_flags |= SB_ACTIVE;
/* Turn on quotas so that they are updated correctly */
quota_enabled = f2fs_enable_quota_files(sbi, s_flags & SB_RDONLY);
#endif
err = recover_data(sbi, &inode_list, &tmp_inode_list, &dir_list);
if (!err)
f2fs_bug_on(sbi, !list_empty(&inode_list));
- else {
- /* restore s_flags to let iput() trash data */
- sbi->sb->s_flags = s_flags;
- }
+ else
+ f2fs_bug_on(sbi, sbi->sb->s_flags & SB_ACTIVE);
skip:
fix_curseg_write_pointer = !check_only || list_empty(&inode_list);
#include <linux/timer.h>
#include <linux/freezer.h>
#include <linux/sched/signal.h>
+#include <linux/random.h>
#include "f2fs.h"
#include "segment.h"
}
}
+static inline bool excess_dirty_threshold(struct f2fs_sb_info *sbi)
+{
+ int factor = rwsem_is_locked(&sbi->cp_rwsem) ? 3 : 2;
+ unsigned int dents = get_pages(sbi, F2FS_DIRTY_DENTS);
+ unsigned int qdata = get_pages(sbi, F2FS_DIRTY_QDATA);
+ unsigned int nodes = get_pages(sbi, F2FS_DIRTY_NODES);
+ unsigned int meta = get_pages(sbi, F2FS_DIRTY_META);
+ unsigned int imeta = get_pages(sbi, F2FS_DIRTY_IMETA);
+ unsigned int threshold = sbi->blocks_per_seg * factor *
+ DEFAULT_DIRTY_THRESHOLD;
+ unsigned int global_threshold = threshold * 3 / 2;
+
+ if (dents >= threshold || qdata >= threshold ||
+ nodes >= threshold || meta >= threshold ||
+ imeta >= threshold)
+ return true;
+ return dents + qdata + nodes + meta + imeta > global_threshold;
+}
+
void f2fs_balance_fs_bg(struct f2fs_sb_info *sbi, bool from_bg)
{
if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
else
f2fs_build_free_nids(sbi, false, false);
- if (excess_dirty_nats(sbi) || excess_dirty_nodes(sbi) ||
- excess_prefree_segs(sbi))
+ if (excess_dirty_nats(sbi) || excess_dirty_threshold(sbi) ||
+ excess_prefree_segs(sbi) || !f2fs_space_for_roll_forward(sbi))
goto do_sync;
/* there is background inflight IO or foreground operation recently */
goto do_sync;
/* checkpoint is the only way to shrink partial cached entries */
- if (f2fs_available_free_memory(sbi, NAT_ENTRIES) ||
+ if (f2fs_available_free_memory(sbi, NAT_ENTRIES) &&
f2fs_available_free_memory(sbi, INO_ENTRIES))
return;
unsigned short seg_type = curseg->seg_type;
sanity_check_seg_type(sbi, seg_type);
+ if (f2fs_need_rand_seg(sbi))
+ return prandom_u32() % (MAIN_SECS(sbi) * sbi->segs_per_sec);
/* if segs_per_sec is large than 1, we need to keep original policy. */
if (__is_large_section(sbi))
curseg->next_segno = segno;
reset_curseg(sbi, type, 1);
curseg->alloc_type = LFS;
+ if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK)
+ curseg->fragment_remained_chunk =
+ prandom_u32() % sbi->max_fragment_chunk + 1;
}
static int __next_free_blkoff(struct f2fs_sb_info *sbi,
static void __refresh_next_blkoff(struct f2fs_sb_info *sbi,
struct curseg_info *seg)
{
- if (seg->alloc_type == SSR)
+ if (seg->alloc_type == SSR) {
seg->next_blkoff =
__next_free_blkoff(sbi, seg->segno,
seg->next_blkoff + 1);
- else
+ } else {
seg->next_blkoff++;
+ if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK) {
+ /* To allocate block chunks in different sizes, use random number */
+ if (--seg->fragment_remained_chunk <= 0) {
+ seg->fragment_remained_chunk =
+ prandom_u32() % sbi->max_fragment_chunk + 1;
+ seg->next_blkoff +=
+ prandom_u32() % sbi->max_fragment_hole + 1;
+ }
+ }
+ }
}
bool f2fs_segment_has_free_slot(struct f2fs_sb_info *sbi, int segno)
up_read(&SM_I(sbi)->curseg_lock);
}
-static void update_device_state(struct f2fs_io_info *fio)
+void f2fs_update_device_state(struct f2fs_sb_info *sbi, nid_t ino,
+ block_t blkaddr, unsigned int blkcnt)
{
- struct f2fs_sb_info *sbi = fio->sbi;
- unsigned int devidx;
-
if (!f2fs_is_multi_device(sbi))
return;
- devidx = f2fs_target_device_index(sbi, fio->new_blkaddr);
+ while (1) {
+ unsigned int devidx = f2fs_target_device_index(sbi, blkaddr);
+ unsigned int blks = FDEV(devidx).end_blk - blkaddr + 1;
- /* update device state for fsync */
- f2fs_set_dirty_device(sbi, fio->ino, devidx, FLUSH_INO);
+ /* update device state for fsync */
+ f2fs_set_dirty_device(sbi, ino, devidx, FLUSH_INO);
- /* update device state for checkpoint */
- if (!f2fs_test_bit(devidx, (char *)&sbi->dirty_device)) {
- spin_lock(&sbi->dev_lock);
- f2fs_set_bit(devidx, (char *)&sbi->dirty_device);
- spin_unlock(&sbi->dev_lock);
+ /* update device state for checkpoint */
+ if (!f2fs_test_bit(devidx, (char *)&sbi->dirty_device)) {
+ spin_lock(&sbi->dev_lock);
+ f2fs_set_bit(devidx, (char *)&sbi->dirty_device);
+ spin_unlock(&sbi->dev_lock);
+ }
+
+ if (blkcnt <= blks)
+ break;
+ blkcnt -= blks;
+ blkaddr += blks;
}
}
goto reallocate;
}
- update_device_state(fio);
+ f2fs_update_device_state(fio->sbi, fio->ino, fio->new_blkaddr, 1);
if (keep_order)
up_read(&fio->sbi->io_order_lock);
goto drop_bio;
}
+ invalidate_mapping_pages(META_MAPPING(sbi),
+ fio->new_blkaddr, fio->new_blkaddr);
+
stat_inc_inplace_blocks(fio->sbi);
if (fio->bio && !(SM_I(sbi)->ipu_policy & (1 << F2FS_IPU_NOCACHE)))
else
err = f2fs_submit_page_bio(fio);
if (!err) {
- update_device_state(fio);
+ f2fs_update_device_state(fio->sbi, fio->ino,
+ fio->new_blkaddr, 1);
f2fs_update_iostat(fio->sbi, fio->io_type, F2FS_BLKSIZE);
}
unsigned short next_blkoff; /* next block offset to write */
unsigned int zone; /* current zone number */
unsigned int next_segno; /* preallocated segment */
+ int fragment_remained_chunk; /* remained block size in a chunk for block fragmentation mode */
bool inited; /* indicate inmem log is inited */
};
[FAULT_DISCARD] = "discard error",
[FAULT_WRITE_IO] = "write IO error",
[FAULT_SLAB_ALLOC] = "slab alloc",
+ [FAULT_DQUOT_INIT] = "dquot initialize",
};
void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate,
if (kstrtouint(str + 1, 10, &level))
return -EINVAL;
- if (!level || level > ZSTD_maxCLevel()) {
+ if (!level || level > zstd_max_clevel()) {
f2fs_info(sbi, "invalid zstd compress level: %d", level);
return -EINVAL;
}
F2FS_OPTION(sbi).fs_mode = FS_MODE_ADAPTIVE;
} else if (!strcmp(name, "lfs")) {
F2FS_OPTION(sbi).fs_mode = FS_MODE_LFS;
+ } else if (!strcmp(name, "fragment:segment")) {
+ F2FS_OPTION(sbi).fs_mode = FS_MODE_FRAGMENT_SEG;
+ } else if (!strcmp(name, "fragment:block")) {
+ F2FS_OPTION(sbi).fs_mode = FS_MODE_FRAGMENT_BLK;
} else {
kfree(name);
return -EINVAL;
/* Not pass down write hints if the number of active logs is lesser
* than NR_CURSEG_PERSIST_TYPE.
*/
- if (F2FS_OPTION(sbi).active_logs != NR_CURSEG_TYPE)
+ if (F2FS_OPTION(sbi).active_logs != NR_CURSEG_PERSIST_TYPE)
F2FS_OPTION(sbi).whint_mode = WHINT_MODE_OFF;
if (f2fs_sb_has_readonly(sbi) && !f2fs_readonly(sbi->sb)) {
seq_puts(seq, "adaptive");
else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_LFS)
seq_puts(seq, "lfs");
+ else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_SEG)
+ seq_puts(seq, "fragment:segment");
+ else if (F2FS_OPTION(sbi).fs_mode == FS_MODE_FRAGMENT_BLK)
+ seq_puts(seq, "fragment:block");
seq_printf(seq, ",active_logs=%u", F2FS_OPTION(sbi).active_logs);
if (test_opt(sbi, RESERVE_ROOT))
seq_printf(seq, ",reserve_root=%u,resuid=%u,resgid=%u",
return len - towrite;
}
+int f2fs_dquot_initialize(struct inode *inode)
+{
+ if (time_to_inject(F2FS_I_SB(inode), FAULT_DQUOT_INIT)) {
+ f2fs_show_injection_info(F2FS_I_SB(inode), FAULT_DQUOT_INIT);
+ return -ESRCH;
+ }
+
+ return dquot_initialize(inode);
+}
+
static struct dquot **f2fs_get_dquots(struct inode *inode)
{
return F2FS_I(inode)->i_dquot;
.get_nextdqblk = dquot_get_next_dqblk,
};
#else
+int f2fs_dquot_initialize(struct inode *inode)
+{
+ return 0;
+}
+
int f2fs_quota_sync(struct super_block *sb, int type)
{
return 0;
NR_CURSEG_PERSIST_TYPE + nat_bits_blocks >= blocks_per_seg)) {
f2fs_warn(sbi, "Insane cp_payload: %u, nat_bits_blocks: %u)",
cp_payload, nat_bits_blocks);
- return -EFSCORRUPTED;
+ return 1;
}
if (unlikely(f2fs_cp_error(sbi))) {
sbi->max_victim_search = DEF_MAX_VICTIM_SEARCH;
sbi->migration_granularity = sbi->segs_per_sec;
sbi->seq_file_ra_mul = MIN_RA_MUL;
+ sbi->max_fragment_chunk = DEF_FRAGMENT_SIZE;
+ sbi->max_fragment_hole = DEF_FRAGMENT_SIZE;
sbi->dir_level = DEF_DIR_LEVEL;
sbi->interval_time[CP_TIME] = DEF_CP_INTERVAL;
{
struct f2fs_super_block *raw_super = F2FS_RAW_SUPER(sbi);
unsigned int max_devices = MAX_DEVICES;
+ unsigned int logical_blksize;
int i;
/* Initialize single device information */
if (!sbi->devs)
return -ENOMEM;
+ logical_blksize = bdev_logical_block_size(sbi->sb->s_bdev);
+ sbi->aligned_blksize = true;
+
for (i = 0; i < max_devices; i++) {
if (i > 0 && !RDEV(i).path[0])
/* to release errored devices */
sbi->s_ndevs = i + 1;
+ if (logical_blksize != bdev_logical_block_size(FDEV(i).bdev))
+ sbi->aligned_blksize = false;
+
#ifdef CONFIG_BLK_DEV_ZONED
if (bdev_zoned_model(FDEV(i).bdev) == BLK_ZONED_HM &&
!f2fs_sb_has_blkzoned(sbi)) {
free_stats:
f2fs_destroy_stats(sbi);
free_nm:
+ /* stop discard thread before destroying node manager */
+ f2fs_stop_discard_thread(sbi);
f2fs_destroy_node_manager(sbi);
free_sm:
f2fs_destroy_segment_manager(sbi);
struct super_block *sb = sbi->sb;
if (f2fs_sb_has_casefold(sbi))
- return snprintf(buf, PAGE_SIZE, "%s (%d.%d.%d)\n",
+ return sysfs_emit(buf, "%s (%d.%d.%d)\n",
sb->s_encoding->charset,
(sb->s_encoding->version >> 16) & 0xff,
(sb->s_encoding->version >> 8) & 0xff,
static ssize_t main_blkaddr_show(struct f2fs_attr *a,
struct f2fs_sb_info *sbi, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)MAIN_BLKADDR(sbi));
}
return count;
}
+ if (!strcmp(a->attr.name, "max_fragment_chunk")) {
+ if (t >= MIN_FRAGMENT_SIZE && t <= MAX_FRAGMENT_SIZE)
+ sbi->max_fragment_chunk = t;
+ else
+ return -EINVAL;
+ return count;
+ }
+
+ if (!strcmp(a->attr.name, "max_fragment_hole")) {
+ if (t >= MIN_FRAGMENT_SIZE && t <= MAX_FRAGMENT_SIZE)
+ sbi->max_fragment_hole = t;
+ else
+ return -EINVAL;
+ return count;
+ }
+
*ui = (unsigned int)t;
return count;
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, seq_file_ra_mul, seq_file_ra_mul);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_segment_mode, gc_segment_mode);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_reclaimed_segments, gc_reclaimed_segs);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, max_fragment_chunk, max_fragment_chunk);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, max_fragment_hole, max_fragment_hole);
#define ATTR_LIST(name) (&f2fs_attr_##name.attr)
static struct attribute *f2fs_attrs[] = {
ATTR_LIST(seq_file_ra_mul),
ATTR_LIST(gc_segment_mode),
ATTR_LIST(gc_reclaimed_segments),
+ ATTR_LIST(max_fragment_chunk),
+ ATTR_LIST(max_fragment_hole),
NULL,
};
ATTRIBUTE_GROUPS(f2fs);
* here and not rely on ->open() doing it. This must be done before
* evicting the inline data.
*/
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
if (!f2fs_is_checkpoint_ready(sbi))
return -ENOSPC;
- err = dquot_initialize(inode);
+ err = f2fs_dquot_initialize(inode);
if (err)
return err;
ssize_t ret;
ret = fuse_direct_io(&io, from, &iocb->ki_pos, FUSE_DIO_WRITE);
- if (ret < 0)
- return ret;
- fuse_invalidate_attr(inode);
- fuse_write_update_size(inode, iocb->ki_pos);
+ fuse_write_update_attr(inode, iocb->ki_pos, ret);
return ret;
}
{
unsigned ncpy = min(*size, cs->len);
if (val) {
- void *pgaddr = kmap_atomic(cs->pg);
+ void *pgaddr = kmap_local_page(cs->pg);
void *buf = pgaddr + cs->offset;
if (cs->write)
else
memcpy(*val, buf, ncpy);
- kunmap_atomic(pgaddr);
+ kunmap_local(pgaddr);
*val += ncpy;
}
*size -= ncpy;
replace_page_cache_page(oldpage, newpage);
+ /*
+ * Release while we have extra ref on stolen page. Otherwise
+ * anon_pipe_buf_release() might think the page can be reused.
+ */
+ pipe_buf_release(cs->pipe, buf);
+
get_page(newpage);
if (!(buf->flags & PIPE_BUF_FLAG_LRU))
}
}
if (page) {
- void *mapaddr = kmap_atomic(page);
+ void *mapaddr = kmap_local_page(page);
void *buf = mapaddr + offset;
offset += fuse_copy_do(cs, &buf, &count);
- kunmap_atomic(mapaddr);
+ kunmap_local(mapaddr);
} else
offset += fuse_copy_do(cs, NULL, &count);
}
end = outarg.offset + outarg.size;
if (end > file_size) {
file_size = end;
- fuse_write_update_size(inode, file_size);
+ fuse_write_update_attr(inode, file_size, outarg.size);
}
num = outarg.size;
pipe_lock(pipe);
out_free:
- for (idx = 0; idx < nbuf; idx++)
- pipe_buf_release(pipe, &bufs[idx]);
+ for (idx = 0; idx < nbuf; idx++) {
+ struct pipe_buffer *buf = &bufs[idx];
+
+ if (buf->ops)
+ pipe_buf_release(pipe, buf);
+ }
pipe_unlock(pipe);
kvfree(bufs);
return time_to_jiffies(o->attr_valid, o->attr_valid_nsec);
}
-static void fuse_invalidate_attr_mask(struct inode *inode, u32 mask)
+void fuse_invalidate_attr_mask(struct inode *inode, u32 mask)
{
set_mask_bits(&get_fuse_inode(inode)->inval_mask, 0, mask);
}
return create_new_entry(fm, &args, dir, entry, S_IFLNK);
}
-void fuse_update_ctime(struct inode *inode)
+void fuse_flush_time_update(struct inode *inode)
+{
+ int err = sync_inode_metadata(inode, 1);
+
+ mapping_set_error(inode->i_mapping, err);
+}
+
+static void fuse_update_ctime_in_cache(struct inode *inode)
{
if (!IS_NOCMTIME(inode)) {
inode->i_ctime = current_time(inode);
mark_inode_dirty_sync(inode);
+ fuse_flush_time_update(inode);
}
}
+void fuse_update_ctime(struct inode *inode)
+{
+ fuse_invalidate_attr_mask(inode, STATX_CTIME);
+ fuse_update_ctime_in_cache(inode);
+}
+
+static void fuse_entry_unlinked(struct dentry *entry)
+{
+ struct inode *inode = d_inode(entry);
+ struct fuse_conn *fc = get_fuse_conn(inode);
+ struct fuse_inode *fi = get_fuse_inode(inode);
+
+ spin_lock(&fi->lock);
+ fi->attr_version = atomic64_inc_return(&fc->attr_version);
+ /*
+ * If i_nlink == 0 then unlink doesn't make sense, yet this can
+ * happen if userspace filesystem is careless. It would be
+ * difficult to enforce correct nlink usage so just ignore this
+ * condition here
+ */
+ if (S_ISDIR(inode->i_mode))
+ clear_nlink(inode);
+ else if (inode->i_nlink > 0)
+ drop_nlink(inode);
+ spin_unlock(&fi->lock);
+ fuse_invalidate_entry_cache(entry);
+ fuse_update_ctime(inode);
+}
+
static int fuse_unlink(struct inode *dir, struct dentry *entry)
{
int err;
args.in_args[0].value = entry->d_name.name;
err = fuse_simple_request(fm, &args);
if (!err) {
- struct inode *inode = d_inode(entry);
- struct fuse_inode *fi = get_fuse_inode(inode);
-
- spin_lock(&fi->lock);
- fi->attr_version = atomic64_inc_return(&fm->fc->attr_version);
- /*
- * If i_nlink == 0 then unlink doesn't make sense, yet this can
- * happen if userspace filesystem is careless. It would be
- * difficult to enforce correct nlink usage so just ignore this
- * condition here
- */
- if (inode->i_nlink > 0)
- drop_nlink(inode);
- spin_unlock(&fi->lock);
- fuse_invalidate_attr(inode);
fuse_dir_changed(dir);
- fuse_invalidate_entry_cache(entry);
- fuse_update_ctime(inode);
+ fuse_entry_unlinked(entry);
} else if (err == -EINTR)
fuse_invalidate_entry(entry);
return err;
args.in_args[0].value = entry->d_name.name;
err = fuse_simple_request(fm, &args);
if (!err) {
- clear_nlink(d_inode(entry));
fuse_dir_changed(dir);
- fuse_invalidate_entry_cache(entry);
+ fuse_entry_unlinked(entry);
} else if (err == -EINTR)
fuse_invalidate_entry(entry);
return err;
err = fuse_simple_request(fm, &args);
if (!err) {
/* ctime changes */
- fuse_invalidate_attr(d_inode(oldent));
fuse_update_ctime(d_inode(oldent));
- if (flags & RENAME_EXCHANGE) {
- fuse_invalidate_attr(d_inode(newent));
+ if (flags & RENAME_EXCHANGE)
fuse_update_ctime(d_inode(newent));
- }
fuse_dir_changed(olddir);
if (olddir != newdir)
fuse_dir_changed(newdir);
/* newent will end up negative */
- if (!(flags & RENAME_EXCHANGE) && d_really_is_positive(newent)) {
- fuse_invalidate_attr(d_inode(newent));
- fuse_invalidate_entry_cache(newent);
- fuse_update_ctime(d_inode(newent));
- }
+ if (!(flags & RENAME_EXCHANGE) && d_really_is_positive(newent))
+ fuse_entry_unlinked(newent);
} else if (err == -EINTR) {
/* If request was interrupted, DEITY only knows if the
rename actually took place. If the invalidation
args.in_args[1].size = newent->d_name.len + 1;
args.in_args[1].value = newent->d_name.name;
err = create_new_entry(fm, &args, newdir, newent, inode->i_mode);
- /* Contrary to "normal" filesystems it can happen that link
- makes two "logical" inodes point to the same "physical"
- inode. We invalidate the attributes of the old one, so it
- will reflect changes in the backing inode (link count,
- etc.)
- */
- if (!err) {
- struct fuse_inode *fi = get_fuse_inode(inode);
-
- spin_lock(&fi->lock);
- fi->attr_version = atomic64_inc_return(&fm->fc->attr_version);
- if (likely(inode->i_nlink < UINT_MAX))
- inc_nlink(inode);
- spin_unlock(&fi->lock);
- fuse_invalidate_attr(inode);
- fuse_update_ctime(inode);
- } else if (err == -EINTR) {
+ if (!err)
+ fuse_update_ctime_in_cache(inode);
+ else if (err == -EINTR)
fuse_invalidate_attr(inode);
- }
+
return err;
}
unsigned int blkbits;
struct fuse_conn *fc = get_fuse_conn(inode);
- /* see the comment in fuse_change_attributes() */
- if (fc->writeback_cache && S_ISREG(inode->i_mode)) {
- attr->size = i_size_read(inode);
- attr->mtime = inode->i_mtime.tv_sec;
- attr->mtimensec = inode->i_mtime.tv_nsec;
- attr->ctime = inode->i_ctime.tv_sec;
- attr->ctimensec = inode->i_ctime.tv_nsec;
- }
-
stat->dev = inode->i_sb->s_dev;
stat->ino = attr->ino;
stat->mode = (inode->i_mode & S_IFMT) | (attr->mode & 07777);
struct fuse_inode *fi = get_fuse_inode(inode);
int err = 0;
bool sync;
+ u32 inval_mask = READ_ONCE(fi->inval_mask);
+ u32 cache_mask = fuse_get_cache_mask(inode);
if (flags & AT_STATX_FORCE_SYNC)
sync = true;
else if (flags & AT_STATX_DONT_SYNC)
sync = false;
- else if (request_mask & READ_ONCE(fi->inval_mask))
+ else if (request_mask & inval_mask & ~cache_mask)
sync = true;
else
sync = time_before64(fi->i_time, get_jiffies_64());
return err;
}
-int fuse_update_attributes(struct inode *inode, struct file *file)
+int fuse_update_attributes(struct inode *inode, struct file *file, u32 mask)
{
- /* Do *not* need to get atime for internal purposes */
- return fuse_update_get_attr(inode, file, NULL,
- STATX_BASIC_STATS & ~STATX_ATIME, 0);
+ return fuse_update_get_attr(inode, file, NULL, mask, 0);
}
int fuse_reverse_inval_entry(struct fuse_conn *fc, u64 parent_nodeid,
if (!parent)
return -ENOENT;
- inode_lock(parent);
+ inode_lock_nested(parent, I_MUTEX_PARENT);
if (!S_ISDIR(parent->i_mode))
goto unlock;
struct fuse_setattr_in inarg;
struct fuse_attr_out outarg;
bool is_truncate = false;
- bool is_wb = fc->writeback_cache;
+ bool is_wb = fc->writeback_cache && S_ISREG(inode->i_mode);
loff_t oldsize;
int err;
- bool trust_local_cmtime = is_wb && S_ISREG(inode->i_mode);
+ bool trust_local_cmtime = is_wb;
bool fault_blocked = false;
if (!fc->default_permissions)
}
/* Flush dirty data/metadata before non-truncate SETATTR */
- if (is_wb && S_ISREG(inode->i_mode) &&
+ if (is_wb &&
attr->ia_valid &
(ATTR_MODE | ATTR_UID | ATTR_GID | ATTR_MTIME_SET |
ATTR_TIMES_SET)) {
}
fuse_change_attributes_common(inode, &outarg.attr,
- attr_timeout(&outarg));
+ attr_timeout(&outarg),
+ fuse_get_cache_mask(inode));
oldsize = inode->i_size;
/* see the comment in fuse_change_attributes() */
- if (!is_wb || is_truncate || !S_ISREG(inode->i_mode))
+ if (!is_wb || is_truncate)
i_size_write(inode, outarg.attr.size);
if (is_truncate) {
i_size_write(inode, 0);
spin_unlock(&fi->lock);
truncate_pagecache(inode, 0);
- fuse_invalidate_attr(inode);
- if (fc->writeback_cache)
- file_update_time(file);
+ file_update_time(file);
+ fuse_invalidate_attr_mask(inode, FUSE_STATX_MODSIZE);
} else if (!(ff->open_flags & FOPEN_KEEP_CACHE)) {
invalidate_inode_pages2(inode->i_mapping);
}
static int fuse_release(struct inode *inode, struct file *file)
{
- struct fuse_conn *fc = get_fuse_conn(inode);
-
- /* see fuse_vma_close() for !writeback_cache case */
- if (fc->writeback_cache)
- write_inode_now(inode, 1);
-
fuse_release_common(file, false);
/* return value is ignored by VFS */
if (fuse_is_bad(inode))
return -EIO;
+ if (ff->open_flags & FOPEN_NOFLUSH && !fm->fc->writeback_cache)
+ return 0;
+
err = write_inode_now(inode, 1);
if (err)
return err;
* enabled, i_blocks from cached attr may not be accurate.
*/
if (!err && fm->fc->writeback_cache)
- fuse_invalidate_attr(inode);
+ fuse_invalidate_attr_mask(inode, STATX_BLOCKS);
return err;
}
struct fuse_inode *fi = get_fuse_inode(inode);
spin_lock(&fi->lock);
- if (attr_ver == fi->attr_version && size < inode->i_size &&
+ if (attr_ver >= fi->attr_version && size < inode->i_size &&
!test_bit(FUSE_I_SIZE_UNSTABLE, &fi->state)) {
fi->attr_version = atomic64_inc_return(&fc->attr_version);
i_size_write(inode, size);
if (fc->auto_inval_data ||
(iocb->ki_pos + iov_iter_count(to) > i_size_read(inode))) {
int err;
- err = fuse_update_attributes(inode, iocb->ki_filp);
+ err = fuse_update_attributes(inode, iocb->ki_filp, STATX_SIZE);
if (err)
return err;
}
return err ?: ia->write.out.size;
}
-bool fuse_write_update_size(struct inode *inode, loff_t pos)
+bool fuse_write_update_attr(struct inode *inode, loff_t pos, ssize_t written)
{
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_inode *fi = get_fuse_inode(inode);
spin_lock(&fi->lock);
fi->attr_version = atomic64_inc_return(&fc->attr_version);
- if (pos > inode->i_size) {
+ if (written > 0 && pos > inode->i_size) {
i_size_write(inode, pos);
ret = true;
}
spin_unlock(&fi->lock);
+ fuse_invalidate_attr_mask(inode, FUSE_STATX_MODSIZE);
+
return ret;
}
kfree(ap->pages);
} while (!err && iov_iter_count(ii));
- if (res > 0)
- fuse_write_update_size(inode, pos);
-
+ fuse_write_update_attr(inode, pos, res);
clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state);
- fuse_invalidate_attr(inode);
return res > 0 ? res : err;
}
if (fc->writeback_cache) {
/* Update size (EOF optimization) and mode (SUID clearing) */
- err = fuse_update_attributes(mapping->host, file);
+ err = fuse_update_attributes(mapping->host, file,
+ STATX_SIZE | STATX_MODE);
if (err)
return err;
if (!ia)
return -ENOMEM;
- ia->io = io;
if (!cuse && fuse_range_is_writeback(inode, idx_from, idx_to)) {
if (!write)
inode_lock(inode);
} else {
res = fuse_direct_io(&io, from, &iocb->ki_pos,
FUSE_DIO_WRITE);
+ fuse_write_update_attr(inode, iocb->ki_pos, res);
}
}
- fuse_invalidate_attr(inode);
- if (res > 0)
- fuse_write_update_size(inode, iocb->ki_pos);
inode_unlock(inode);
return res;
* is enabled, we trust local ctime/mtime.
*/
if (!fc->writeback_cache)
- fuse_invalidate_attr(inode);
+ fuse_invalidate_attr_mask(inode, FUSE_STATX_MODIFY);
spin_lock(&fi->lock);
rb_erase(&wpa->writepages_entry, &fi->writepages);
while (wpa->next) {
static struct fuse_file *__fuse_write_file_get(struct fuse_inode *fi)
{
- struct fuse_file *ff = NULL;
+ struct fuse_file *ff;
spin_lock(&fi->lock);
- if (!list_empty(&fi->write_files)) {
- ff = list_entry(fi->write_files.next, struct fuse_file,
- write_entry);
+ ff = list_first_entry_or_null(&fi->write_files, struct fuse_file,
+ write_entry);
+ if (ff)
fuse_file_get(ff);
- }
spin_unlock(&fi->lock);
return ff;
struct fuse_file *ff;
int err;
+ /*
+ * Inode is always written before the last reference is dropped and
+ * hence this should not be reached from reclaim.
+ *
+ * Writing back the inode from reclaim can deadlock if the request
+ * processing itself needs an allocation. Allocations triggering
+ * reclaim while serving a request can't be prevented, because it can
+ * involve any number of unrelated userspace processes.
+ */
+ WARN_ON(wbc->for_reclaim);
+
ff = __fuse_write_file_get(fi);
err = fuse_flush_times(inode, ff);
if (ff)
if (!copied)
goto unlock;
+ pos += copied;
if (!PageUptodate(page)) {
/* Zero any unwritten bytes at the end of the page */
- size_t endoff = (pos + copied) & ~PAGE_MASK;
+ size_t endoff = pos & ~PAGE_MASK;
if (endoff)
zero_user_segment(page, endoff, PAGE_SIZE);
SetPageUptodate(page);
}
- fuse_write_update_size(inode, pos + copied);
+ if (pos > inode->i_size)
+ i_size_write(inode, pos);
+
set_page_dirty(page);
unlock:
}
/*
- * Write back dirty pages now, because there may not be any suitable
- * open files later
+ * Write back dirty data/metadata now (there may not be any suitable
+ * open files later for data)
*/
static void fuse_vma_close(struct vm_area_struct *vma)
{
- filemap_write_and_wait(vma->vm_file->f_mapping);
+ int err;
+
+ err = write_inode_now(vma->vm_file->f_mapping->host, 1);
+ mapping_set_error(vma->vm_file->f_mapping, err);
}
/*
return vfs_setpos(file, outarg.offset, inode->i_sb->s_maxbytes);
fallback:
- err = fuse_update_attributes(inode, file);
+ err = fuse_update_attributes(inode, file, STATX_SIZE);
if (!err)
return generic_file_llseek(file, offset, whence);
else
break;
case SEEK_END:
inode_lock(inode);
- retval = fuse_update_attributes(inode, file);
+ retval = fuse_update_attributes(inode, file, STATX_SIZE);
if (!retval)
retval = generic_file_llseek(file, offset, whence);
inode_unlock(inode);
if (iov_iter_rw(iter) == WRITE) {
ret = fuse_direct_io(io, iter, &pos, FUSE_DIO_WRITE);
- fuse_invalidate_attr(inode);
+ fuse_invalidate_attr_mask(inode, FUSE_STATX_MODSIZE);
} else {
ret = __fuse_direct_read(io, iter, &pos);
}
kref_put(&io->refcnt, fuse_io_release);
if (iov_iter_rw(iter) == WRITE) {
- if (ret > 0)
- fuse_write_update_size(inode, pos);
- else if (ret < 0 && offset + count > i_size)
+ fuse_write_update_attr(inode, pos, ret);
+ if (ret < 0 && offset + count > i_size)
fuse_do_truncate(file);
}
/* we could have extended the file */
if (!(mode & FALLOC_FL_KEEP_SIZE)) {
- bool changed = fuse_write_update_size(inode, offset + length);
-
- if (changed && fm->fc->writeback_cache)
+ if (fuse_write_update_attr(inode, offset + length, length))
file_update_time(file);
}
if (mode & (FALLOC_FL_PUNCH_HOLE | FALLOC_FL_ZERO_RANGE))
truncate_pagecache_range(inode, offset, offset + length - 1);
- fuse_invalidate_attr(inode);
+ fuse_invalidate_attr_mask(inode, FUSE_STATX_MODSIZE);
out:
if (!(mode & FALLOC_FL_KEEP_SIZE))
if (lock_inode)
inode_unlock(inode);
+ fuse_flush_time_update(inode);
+
return err;
}
ALIGN_DOWN(pos_out, PAGE_SIZE),
ALIGN(pos_out + outarg.size, PAGE_SIZE) - 1);
- if (fc->writeback_cache) {
- fuse_write_update_size(inode_out, pos_out + outarg.size);
- file_update_time(file_out);
- }
-
- fuse_invalidate_attr(inode_out);
+ file_update_time(file_out);
+ fuse_write_update_attr(inode_out, pos_out + outarg.size, outarg.size);
err = outarg.size;
out:
inode_unlock(inode_out);
file_accessed(file_in);
+ fuse_flush_time_update(inode_out);
+
return err;
}
u64 attr_valid, u64 attr_version);
void fuse_change_attributes_common(struct inode *inode, struct fuse_attr *attr,
- u64 attr_valid);
+ u64 attr_valid, u32 cache_mask);
+
+u32 fuse_get_cache_mask(struct inode *inode);
/**
* Initialize the client device
/**
* Invalidate inode attributes
*/
+
+/* Attributes possibly changed on data modification */
+#define FUSE_STATX_MODIFY (STATX_MTIME | STATX_CTIME | STATX_BLOCKS)
+
+/* Attributes possibly changed on data and/or size modification */
+#define FUSE_STATX_MODSIZE (FUSE_STATX_MODIFY | STATX_SIZE)
+
void fuse_invalidate_attr(struct inode *inode);
+void fuse_invalidate_attr_mask(struct inode *inode, u32 mask);
void fuse_invalidate_entry_cache(struct dentry *entry);
u64 fuse_lock_owner_id(struct fuse_conn *fc, fl_owner_t id);
+void fuse_flush_time_update(struct inode *inode);
void fuse_update_ctime(struct inode *inode);
-int fuse_update_attributes(struct inode *inode, struct file *file);
+int fuse_update_attributes(struct inode *inode, struct file *file, u32 mask);
void fuse_flush_writepages(struct inode *inode);
__poll_t fuse_file_poll(struct file *file, poll_table *wait);
int fuse_dev_release(struct inode *inode, struct file *file);
-bool fuse_write_update_size(struct inode *inode, loff_t pos);
+bool fuse_write_update_attr(struct inode *inode, loff_t pos, ssize_t written);
int fuse_flush_times(struct inode *inode, struct fuse_file *ff);
int fuse_write_inode(struct inode *inode, struct writeback_control *wbc);
{
struct fuse_inode *fi = get_fuse_inode(inode);
+ /* Will write inode on close/munmap and in all other dirtiers */
+ WARN_ON(inode->i_state & I_DIRTY_INODE);
+
truncate_inode_pages_final(&inode->i_data);
clear_inode(inode);
if (inode->i_sb->s_flags & SB_ACTIVE) {
}
void fuse_change_attributes_common(struct inode *inode, struct fuse_attr *attr,
- u64 attr_valid)
+ u64 attr_valid, u32 cache_mask)
{
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_inode *fi = get_fuse_inode(inode);
inode->i_atime.tv_sec = attr->atime;
inode->i_atime.tv_nsec = attr->atimensec;
/* mtime from server may be stale due to local buffered write */
- if (!fc->writeback_cache || !S_ISREG(inode->i_mode)) {
+ if (!(cache_mask & STATX_MTIME)) {
inode->i_mtime.tv_sec = attr->mtime;
inode->i_mtime.tv_nsec = attr->mtimensec;
+ }
+ if (!(cache_mask & STATX_CTIME)) {
inode->i_ctime.tv_sec = attr->ctime;
inode->i_ctime.tv_nsec = attr->ctimensec;
}
inode->i_flags &= ~S_NOSEC;
}
+u32 fuse_get_cache_mask(struct inode *inode)
+{
+ struct fuse_conn *fc = get_fuse_conn(inode);
+
+ if (!fc->writeback_cache || !S_ISREG(inode->i_mode))
+ return 0;
+
+ return STATX_MTIME | STATX_CTIME | STATX_SIZE;
+}
+
void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,
u64 attr_valid, u64 attr_version)
{
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_inode *fi = get_fuse_inode(inode);
- bool is_wb = fc->writeback_cache;
+ u32 cache_mask;
loff_t oldsize;
struct timespec64 old_mtime;
spin_lock(&fi->lock);
+ /*
+ * In case of writeback_cache enabled, writes update mtime, ctime and
+ * may update i_size. In these cases trust the cached value in the
+ * inode.
+ */
+ cache_mask = fuse_get_cache_mask(inode);
+ if (cache_mask & STATX_SIZE)
+ attr->size = i_size_read(inode);
+
+ if (cache_mask & STATX_MTIME) {
+ attr->mtime = inode->i_mtime.tv_sec;
+ attr->mtimensec = inode->i_mtime.tv_nsec;
+ }
+ if (cache_mask & STATX_CTIME) {
+ attr->ctime = inode->i_ctime.tv_sec;
+ attr->ctimensec = inode->i_ctime.tv_nsec;
+ }
+
if ((attr_version != 0 && fi->attr_version > attr_version) ||
test_bit(FUSE_I_SIZE_UNSTABLE, &fi->state)) {
spin_unlock(&fi->lock);
}
old_mtime = inode->i_mtime;
- fuse_change_attributes_common(inode, attr, attr_valid);
+ fuse_change_attributes_common(inode, attr, attr_valid, cache_mask);
oldsize = inode->i_size;
/*
* extend local i_size without keeping userspace server in sync. So,
* attr->size coming from server can be stale. We cannot trust it.
*/
- if (!is_wb || !S_ISREG(inode->i_mode))
+ if (!(cache_mask & STATX_SIZE))
i_size_write(inode, attr->size);
spin_unlock(&fi->lock);
- if (!is_wb && S_ISREG(inode->i_mode)) {
+ if (!cache_mask && S_ISREG(inode->i_mode)) {
bool inval = false;
if (oldsize != attr->size) {
in_iovs + out_iovs > FUSE_IOCTL_MAX_IOV)
goto out;
- vaddr = kmap_atomic(ap.pages[0]);
+ vaddr = kmap_local_page(ap.pages[0]);
err = fuse_copy_ioctl_iovec(fm->fc, iov_page, vaddr,
transferred, in_iovs + out_iovs,
(flags & FUSE_IOCTL_COMPAT) != 0);
- kunmap_atomic(vaddr);
+ kunmap_local(vaddr);
if (err)
goto out;
WARN_ON(fi->rdc.pos != pos))
goto unlock;
- addr = kmap_atomic(page);
+ addr = kmap_local_page(page);
if (!offset)
clear_page(addr);
memcpy(addr + offset, dirent, reclen);
- kunmap_atomic(addr);
+ kunmap_local(addr);
fi->rdc.size = (index << PAGE_SHIFT) + offset + reclen;
fi->rdc.pos = dirent->off;
unlock:
* cache; both cases require an up-to-date mtime value.
*/
if (!ctx->pos && fc->auto_inval_data) {
- int err = fuse_update_attributes(inode, file);
+ int err = fuse_update_attributes(inode, file, STATX_MTIME);
if (err)
return err;
static void virtio_fs_init_vq(struct virtio_fs_vq *fsvq, char *name,
int vq_type)
{
- strncpy(fsvq->name, name, VQ_NAME_LEN);
+ strscpy(fsvq->name, name, VQ_NAME_LEN);
spin_lock_init(&fsvq->lock);
INIT_LIST_HEAD(&fsvq->queued_reqs);
INIT_LIST_HEAD(&fsvq->end_reqs);
fm->fc->no_setxattr = 1;
err = -EOPNOTSUPP;
}
- if (!err) {
- fuse_invalidate_attr(inode);
+ if (!err)
fuse_update_ctime(inode);
- }
+
return err;
}
fm->fc->no_removexattr = 1;
err = -EOPNOTSUPP;
}
- if (!err) {
- fuse_invalidate_attr(inode);
+ if (!err)
fuse_update_ctime(inode);
- }
+
return err;
}
else if (height == ip->i_height)
ret = gfs2_hole_size(inode, lblock, len, mp, iomap);
else
- iomap->length = size - pos;
+ iomap->length = size - iomap->offset;
} else if (flags & IOMAP_WRITE) {
u64 alloc_size;
size_t *prev_count,
size_t *window_size)
{
- char __user *p = i->iov[0].iov_base + i->iov_offset;
size_t count = iov_iter_count(i);
+ char __user *p;
int pages = 1;
if (likely(!count))
if (*prev_count != count || !*window_size) {
int pages, nr_dirtied;
- pages = min_t(int, BIO_MAX_VECS,
- DIV_ROUND_UP(iov_iter_count(i), PAGE_SIZE));
+ pages = min_t(int, BIO_MAX_VECS, DIV_ROUND_UP(count, PAGE_SIZE));
nr_dirtied = max(current->nr_dirtied_pause -
current->nr_dirtied, 1);
pages = min(pages, nr_dirtied);
}
*prev_count = count;
+ p = i->iov[0].iov_base + i->iov_offset;
*window_size = (size_t)PAGE_SIZE * pages - offset_in_page(p);
return true;
}
struct gfs2_sbd *sdp = GFS2_SB(inode);
struct gfs2_holder *statfs_gh = NULL;
size_t prev_count = 0, window_size = 0;
+ size_t orig_count = iov_iter_count(from);
size_t read = 0;
ssize_t ret;
if (inode == sdp->sd_rindex)
gfs2_glock_dq_uninit(statfs_gh);
+ from->count = orig_count - read;
if (should_fault_in_pages(ret, from, &prev_count, &window_size)) {
size_t leftover;
leftover = fault_in_iov_iter_readable(from, window_size);
gfs2_holder_disallow_demote(gh);
if (leftover != window_size) {
+ from->count = min(from->count, window_size - leftover);
if (!gfs2_holder_queued(gh)) {
if (read)
goto out_uninit;
static void demote_incompat_holders(struct gfs2_glock *gl,
struct gfs2_holder *new_gh)
{
- struct gfs2_holder *gh;
+ struct gfs2_holder *gh, *tmp;
/*
* Demote incompatible holders before we make ourselves eligible.
* (This holder may or may not allow auto-demoting, but we don't want
* to demote the new holder before it's even granted.)
*/
- list_for_each_entry(gh, &gl->gl_holders, gh_list) {
+ list_for_each_entry_safe(gh, tmp, &gl->gl_holders, gh_list) {
/*
* Since holders are at the front of the list, we stop when we
* find the first non-holder.
* Since we unlock the lockref lock, we set a flag to indicate
* instantiate is in progress.
*/
- if (test_bit(GLF_INSTANTIATE_IN_PROG, &gl->gl_flags)) {
+ if (test_and_set_bit(GLF_INSTANTIATE_IN_PROG, &gl->gl_flags)) {
wait_on_bit(&gl->gl_flags, GLF_INSTANTIATE_IN_PROG,
TASK_UNINTERRUPTIBLE);
/*
goto again;
}
- set_bit(GLF_INSTANTIATE_IN_PROG, &gl->gl_flags);
-
ret = glops->go_instantiate(gh);
if (!ret)
clear_bit(GLF_INSTANTIATE_NEEDED, &gl->gl_flags);
- clear_bit(GLF_INSTANTIATE_IN_PROG, &gl->gl_flags);
- smp_mb__after_atomic();
- wake_up_bit(&gl->gl_flags, GLF_INSTANTIATE_IN_PROG);
+ clear_and_wake_up_bit(GLF_INSTANTIATE_IN_PROG, &gl->gl_flags);
return ret;
}
gfs2_ordered_del_inode(ip);
clear_inode(inode);
gfs2_dir_hash_inval(ip);
- if (ip->i_gl) {
- glock_clear_object(ip->i_gl, ip);
- wait_on_bit_io(&ip->i_flags, GIF_GLOP_PENDING, TASK_UNINTERRUPTIBLE);
- gfs2_glock_add_to_lru(ip->i_gl);
- gfs2_glock_put_eventually(ip->i_gl);
- ip->i_gl = NULL;
- }
if (gfs2_holder_initialized(&ip->i_iopen_gh)) {
struct gfs2_glock *gl = ip->i_iopen_gh.gh_gl;
gfs2_holder_uninit(&ip->i_iopen_gh);
gfs2_glock_put_eventually(gl);
}
+ if (ip->i_gl) {
+ glock_clear_object(ip->i_gl, ip);
+ wait_on_bit_io(&ip->i_flags, GIF_GLOP_PENDING, TASK_UNINTERRUPTIBLE);
+ gfs2_glock_add_to_lru(ip->i_gl);
+ gfs2_glock_put_eventually(ip->i_gl);
+ ip->i_gl = NULL;
+ }
}
static struct inode *gfs2_alloc_inode(struct super_block *sb)
goto out;
if (S_ISDIR(main_inode->i_mode)) {
- if (fd.entrylength < sizeof(struct hfs_cat_dir))
- /* panic? */;
+ WARN_ON(fd.entrylength < sizeof(struct hfs_cat_dir));
hfs_bnode_read(fd.bnode, &rec, fd.entryoffset,
sizeof(struct hfs_cat_dir));
if (rec.type != HFS_CDR_DIR ||
hfs_bnode_write(fd.bnode, &rec, fd.entryoffset,
sizeof(struct hfs_cat_file));
} else {
- if (fd.entrylength < sizeof(struct hfs_cat_file))
- /* panic? */;
+ WARN_ON(fd.entrylength < sizeof(struct hfs_cat_file));
hfs_bnode_read(fd.bnode, &rec, fd.entryoffset,
sizeof(struct hfs_cat_file));
if (rec.type != HFS_CDR_FIL ||
if (type == HFSPLUS_FOLDER) {
struct hfsplus_cat_folder *folder = &entry.folder;
- if (fd->entrylength < sizeof(struct hfsplus_cat_folder))
- /* panic? */;
+ WARN_ON(fd->entrylength < sizeof(struct hfsplus_cat_folder));
hfs_bnode_read(fd->bnode, &entry, fd->entryoffset,
sizeof(struct hfsplus_cat_folder));
hfsplus_get_perms(inode, &folder->permissions, 1);
} else if (type == HFSPLUS_FILE) {
struct hfsplus_cat_file *file = &entry.file;
- if (fd->entrylength < sizeof(struct hfsplus_cat_file))
- /* panic? */;
+ WARN_ON(fd->entrylength < sizeof(struct hfsplus_cat_file));
hfs_bnode_read(fd->bnode, &entry, fd->entryoffset,
sizeof(struct hfsplus_cat_file));
if (S_ISDIR(main_inode->i_mode)) {
struct hfsplus_cat_folder *folder = &entry.folder;
- if (fd.entrylength < sizeof(struct hfsplus_cat_folder))
- /* panic? */;
+ WARN_ON(fd.entrylength < sizeof(struct hfsplus_cat_folder));
hfs_bnode_read(fd.bnode, &entry, fd.entryoffset,
sizeof(struct hfsplus_cat_folder));
/* simple node checks? */
} else {
struct hfsplus_cat_file *file = &entry.file;
- if (fd.entrylength < sizeof(struct hfsplus_cat_file))
- /* panic? */;
+ WARN_ON(fd.entrylength < sizeof(struct hfsplus_cat_file));
hfs_bnode_read(fd.bnode, &entry, fd.entryoffset,
sizeof(struct hfsplus_cat_file));
hfsplus_inode_write_fork(inode, &file->data_fork);
* otherwise hugetlb_reserve_pages reserves one less hugepages than intended.
*/
struct file *hugetlb_file_setup(const char *name, size_t size,
- vm_flags_t acctflag, struct ucounts **ucounts,
- int creat_flags, int page_size_log)
+ vm_flags_t acctflag, int creat_flags,
+ int page_size_log)
{
struct inode *inode;
struct vfsmount *mnt;
if (hstate_idx < 0)
return ERR_PTR(-ENODEV);
- *ucounts = NULL;
mnt = hugetlbfs_vfsmount[hstate_idx];
if (!mnt)
return ERR_PTR(-ENOENT);
if (creat_flags == HUGETLB_SHMFS_INODE && !can_do_hugetlb_shm()) {
- *ucounts = current_ucounts();
- if (user_shm_lock(size, *ucounts)) {
- task_lock(current);
- pr_warn_once("%s (%d): Using mlock ulimits for SHM_HUGETLB is deprecated\n",
+ struct ucounts *ucounts = current_ucounts();
+
+ if (user_shm_lock(size, ucounts)) {
+ pr_warn_once("%s (%d): Using mlock ulimits for SHM_HUGETLB is obsolete\n",
current->comm, current->pid);
- task_unlock(current);
- } else {
- *ucounts = NULL;
- return ERR_PTR(-EPERM);
+ user_shm_unlock(size, ucounts);
}
+ return ERR_PTR(-EPERM);
}
file = ERR_PTR(-ENOSPC);
iput(inode);
out:
- if (*ucounts) {
- user_shm_unlock(size, *ucounts);
- *ucounts = NULL;
- }
return file;
}
}
EXPORT_SYMBOL(ihold);
-static void inode_lru_list_add(struct inode *inode)
+static void __inode_add_lru(struct inode *inode, bool rotate)
{
+ if (inode->i_state & (I_DIRTY_ALL | I_SYNC | I_FREEING | I_WILL_FREE))
+ return;
+ if (atomic_read(&inode->i_count))
+ return;
+ if (!(inode->i_sb->s_flags & SB_ACTIVE))
+ return;
+ if (!mapping_shrinkable(&inode->i_data))
+ return;
+
if (list_lru_add(&inode->i_sb->s_inode_lru, &inode->i_lru))
this_cpu_inc(nr_unused);
- else
+ else if (rotate)
inode->i_state |= I_REFERENCED;
}
*/
void inode_add_lru(struct inode *inode)
{
- if (!(inode->i_state & (I_DIRTY_ALL | I_SYNC |
- I_FREEING | I_WILL_FREE)) &&
- !atomic_read(&inode->i_count) && inode->i_sb->s_flags & SB_ACTIVE)
- inode_lru_list_add(inode);
+ __inode_add_lru(inode, false);
}
-
static void inode_lru_list_del(struct inode *inode)
{
-
if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru))
this_cpu_dec(nr_unused);
}
/*
* Isolate the inode from the LRU in preparation for freeing it.
*
- * Any inodes which are pinned purely because of attached pagecache have their
- * pagecache removed. If the inode has metadata buffers attached to
- * mapping->private_list then try to remove them.
- *
* If the inode has the I_REFERENCED flag set, then it means that it has been
* used recently - the flag is set in iput_final(). When we encounter such an
* inode, clear the flag and move it to the back of the LRU so it gets another
struct inode *inode = container_of(item, struct inode, i_lru);
/*
- * we are inverting the lru lock/inode->i_lock here, so use a trylock.
- * If we fail to get the lock, just skip it.
+ * We are inverting the lru lock/inode->i_lock here, so use a
+ * trylock. If we fail to get the lock, just skip it.
*/
if (!spin_trylock(&inode->i_lock))
return LRU_SKIP;
/*
- * Referenced or dirty inodes are still in use. Give them another pass
- * through the LRU as we canot reclaim them now.
+ * Inodes can get referenced, redirtied, or repopulated while
+ * they're already on the LRU, and this can make them
+ * unreclaimable for a while. Remove them lazily here; iput,
+ * sync, or the last page cache deletion will requeue them.
*/
if (atomic_read(&inode->i_count) ||
- (inode->i_state & ~I_REFERENCED)) {
+ (inode->i_state & ~I_REFERENCED) ||
+ !mapping_shrinkable(&inode->i_data)) {
list_lru_isolate(lru, &inode->i_lru);
spin_unlock(&inode->i_lock);
this_cpu_dec(nr_unused);
return LRU_REMOVED;
}
- /* recently referenced inodes get one more pass */
+ /* Recently referenced inodes get one more pass */
if (inode->i_state & I_REFERENCED) {
inode->i_state &= ~I_REFERENCED;
spin_unlock(&inode->i_lock);
return LRU_ROTATE;
}
+ /*
+ * On highmem systems, mapping_shrinkable() permits dropping
+ * page cache in order to free up struct inodes: lowmem might
+ * be under pressure before the cache inside the highmem zone.
+ */
if (inode_has_buffers(inode) || !mapping_empty(&inode->i_data)) {
__iget(inode);
spin_unlock(&inode->i_lock);
if (!drop &&
!(inode->i_state & I_DONTCACHE) &&
(sb->s_flags & SB_ACTIVE)) {
- inode_add_lru(inode);
+ __inode_add_lru(inode, true);
spin_unlock(&inode->i_lock);
return;
}
* inode.c
*/
extern long prune_icache_sb(struct super_block *sb, struct shrink_control *sc);
-extern void inode_add_lru(struct inode *inode);
extern int dentry_needs_remove_privs(struct dentry *dentry);
/*
return work->flags >> IO_WQ_HASH_SHIFT;
}
-static void io_wait_on_hash(struct io_wqe *wqe, unsigned int hash)
+static bool io_wait_on_hash(struct io_wqe *wqe, unsigned int hash)
{
struct io_wq *wq = wqe->wq;
+ bool ret = false;
spin_lock_irq(&wq->hash->wait.lock);
if (list_empty(&wqe->wait.entry)) {
if (!test_bit(hash, &wq->hash->map)) {
__set_current_state(TASK_RUNNING);
list_del_init(&wqe->wait.entry);
+ ret = true;
}
}
spin_unlock_irq(&wq->hash->wait.lock);
+ return ret;
}
static struct io_wq_work *io_get_next_work(struct io_wqe_acct *acct,
}
if (stall_hash != -1U) {
+ bool unstalled;
+
/*
* Set this before dropping the lock to avoid racing with new
* work being added and clearing the stalled bit.
*/
set_bit(IO_ACCT_STALLED_BIT, &acct->flags);
raw_spin_unlock(&wqe->lock);
- io_wait_on_hash(wqe, stall_hash);
+ unstalled = io_wait_on_hash(wqe, stall_hash);
raw_spin_lock(&wqe->lock);
+ if (unstalled) {
+ clear_bit(IO_ACCT_STALLED_BIT, &acct->flags);
+ if (wq_has_sleeper(&wqe->wq->hash->wait))
+ wake_up(&wqe->wq->hash->wait);
+ }
}
return NULL;
io_wqe_enqueue(wqe, linked);
if (hash != -1U && !next_hashed) {
+ /* serialize hash clear with wake_up() */
+ spin_lock_irq(&wq->hash->wait.lock);
clear_bit(hash, &wq->hash->map);
clear_bit(IO_ACCT_STALLED_BIT, &acct->flags);
+ spin_unlock_irq(&wq->hash->wait.lock);
if (wq_has_sleeper(&wq->hash->wait))
wake_up(&wq->hash->wait);
raw_spin_lock(&wqe->lock);
*/
int io_wq_max_workers(struct io_wq *wq, int *new_count)
{
- int i, node, prev = 0;
+ int prev[IO_WQ_ACCT_NR];
+ bool first_node = true;
+ int i, node;
BUILD_BUG_ON((int) IO_WQ_ACCT_BOUND != (int) IO_WQ_BOUND);
BUILD_BUG_ON((int) IO_WQ_ACCT_UNBOUND != (int) IO_WQ_UNBOUND);
new_count[i] = task_rlimit(current, RLIMIT_NPROC);
}
+ for (i = 0; i < IO_WQ_ACCT_NR; i++)
+ prev[i] = 0;
+
rcu_read_lock();
for_each_node(node) {
struct io_wqe *wqe = wq->wqes[node];
raw_spin_lock(&wqe->lock);
for (i = 0; i < IO_WQ_ACCT_NR; i++) {
acct = &wqe->acct[i];
- prev = max_t(int, acct->max_workers, prev);
+ if (first_node)
+ prev[i] = max_t(int, acct->max_workers, prev[i]);
if (new_count[i])
acct->max_workers = new_count[i];
- new_count[i] = prev;
}
raw_spin_unlock(&wqe->lock);
+ first_node = false;
}
rcu_read_unlock();
+
+ for (i = 0; i < IO_WQ_ACCT_NR; i++)
+ new_count[i] = prev[i];
+
return 0;
}
switch (io_arm_poll_handler(req)) {
case IO_APOLL_READY:
- if (linked_timeout) {
- io_queue_linked_timeout(linked_timeout);
- linked_timeout = NULL;
- }
io_req_task_queue(req);
break;
case IO_APOLL_ABORTED:
for (i = 0; i < sq_entries; i++) {
unsigned int entry = i + sq_head;
unsigned int sq_idx = READ_ONCE(ctx->sq_array[entry & sq_mask]);
- struct io_uring_sqe *sqe = &ctx->sq_sqes[sq_idx];
+ struct io_uring_sqe *sqe;
if (sq_idx > sq_mask)
continue;
BUILD_BUG_ON(sizeof(new_count) != sizeof(ctx->iowq_limits));
- memcpy(ctx->iowq_limits, new_count, sizeof(new_count));
+ for (i = 0; i < ARRAY_SIZE(new_count); i++)
+ if (new_count[i])
+ ctx->iowq_limits[i] = new_count[i];
ctx->iowq_limits_set = true;
- ret = -EINVAL;
if (tctx && tctx->io_wq) {
ret = io_wq_max_workers(tctx->io_wq, new_count);
if (ret)
de = (struct iso_directory_record *) (bh->b_data + offset);
de_len = *(unsigned char *) de;
+ if (de_len < sizeof(struct iso_directory_record))
+ goto fail;
if (offset + de_len > bufsize) {
int frag1 = bufsize - offset;
select NLS
select NLS_UTF8
select CRYPTO
- select CRYPTO_MD4
select CRYPTO_MD5
select CRYPTO_HMAC
select CRYPTO_ECB
select CRYPTO_GCM
select ASN1
select OID_REGISTRY
+ select CRC32
default n
help
Choose Y here if you want to allow SMB3 compliant clients
__u8 *pi_hash)
{
int rc;
- struct smb2_hdr *rcv_hdr = (struct smb2_hdr *)buf;
+ struct smb2_hdr *rcv_hdr = smb2_get_msg(buf);
char *all_bytes_msg = (char *)&rcv_hdr->ProtocolId;
- int msg_size = be32_to_cpu(rcv_hdr->smb2_buf_length);
+ int msg_size = get_rfc1002_len(buf);
struct ksmbd_crypto_ctx *ctx = NULL;
if (conn->preauth_info->Preauth_HashId !=
u8 *sign)
{
struct scatterlist *sg;
- unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 24;
+ unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 20;
int i, nr_entries[3] = {0}, total_entries = 0, sg_idx = 0;
if (!nvec)
int ksmbd_crypt_message(struct ksmbd_conn *conn, struct kvec *iov,
unsigned int nvec, int enc)
{
- struct smb2_transform_hdr *tr_hdr =
- (struct smb2_transform_hdr *)iov[0].iov_base;
- unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 24;
+ struct smb2_transform_hdr *tr_hdr = smb2_get_msg(iov[0].iov_base);
+ unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 20;
int rc;
struct scatterlist *sg;
u8 sign[SMB2_SIGNATURE_SIZE] = {};
int ksmbd_conn_write(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb_hdr *rsp_hdr = work->response_buf;
size_t len = 0;
int sent;
struct kvec iov[3];
int iov_idx = 0;
ksmbd_conn_try_dequeue_request(work);
- if (!rsp_hdr) {
+ if (!work->response_buf) {
pr_err("NULL response header\n");
return -EINVAL;
}
if (work->tr_buf) {
iov[iov_idx] = (struct kvec) { work->tr_buf,
- sizeof(struct smb2_transform_hdr) };
+ sizeof(struct smb2_transform_hdr) + 4 };
len += iov[iov_idx++].iov_len;
}
if (work->aux_payload_sz) {
- iov[iov_idx] = (struct kvec) { rsp_hdr, work->resp_hdr_sz };
+ iov[iov_idx] = (struct kvec) { work->response_buf, work->resp_hdr_sz };
len += iov[iov_idx++].iov_len;
iov[iov_idx] = (struct kvec) { work->aux_payload_buf, work->aux_payload_sz };
len += iov[iov_idx++].iov_len;
if (work->tr_buf)
iov[iov_idx].iov_len = work->resp_hdr_sz;
else
- iov[iov_idx].iov_len = get_rfc1002_len(rsp_hdr) + 4;
- iov[iov_idx].iov_base = rsp_hdr;
+ iov[iov_idx].iov_len = get_rfc1002_len(work->response_buf) + 4;
+ iov[iov_idx].iov_base = work->response_buf;
len += iov[iov_idx++].iov_len;
}
void ksmbd_workqueue_destroy(void)
{
- flush_workqueue(ksmbd_wq);
destroy_workqueue(ksmbd_wq);
ksmbd_wq = NULL;
}
*/
static inline void *ksmbd_resp_buf_next(struct ksmbd_work *work)
{
- return work->response_buf + work->next_smb2_rsp_hdr_off;
+ return work->response_buf + work->next_smb2_rsp_hdr_off + 4;
}
/**
*/
static inline void *ksmbd_req_buf_next(struct ksmbd_work *work)
{
- return work->request_buf + work->next_smb2_rcv_hdr_off;
+ return work->request_buf + work->next_smb2_rcv_hdr_off + 4;
}
struct ksmbd_work *ksmbd_alloc_work_struct(void);
return;
}
- rsp_hdr = work->response_buf;
+ rsp_hdr = smb2_get_msg(work->response_buf);
memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2);
- rsp_hdr->smb2_buf_length =
- cpu_to_be32(smb2_hdr_size_no_buflen(conn->vals));
+ *(__be32 *)work->response_buf =
+ cpu_to_be32(conn->vals->header_size);
rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER;
rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE;
rsp_hdr->CreditRequest = cpu_to_le16(0);
rsp_hdr->SessionId = 0;
memset(rsp_hdr->Signature, 0, 16);
- rsp = work->response_buf;
+ rsp = smb2_get_msg(work->response_buf);
rsp->StructureSize = cpu_to_le16(24);
if (!br_info->open_trunc &&
rsp->PersistentFid = cpu_to_le64(fp->persistent_id);
rsp->VolatileFid = cpu_to_le64(fp->volatile_id);
- inc_rfc1001_len(rsp, 24);
+ inc_rfc1001_len(work->response_buf, 24);
ksmbd_debug(OPLOCK,
"sending oplock break v_id %llu p_id = %llu lock level = %d\n",
return;
}
- rsp_hdr = work->response_buf;
+ rsp_hdr = smb2_get_msg(work->response_buf);
memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2);
- rsp_hdr->smb2_buf_length =
- cpu_to_be32(smb2_hdr_size_no_buflen(conn->vals));
+ *(__be32 *)work->response_buf =
+ cpu_to_be32(conn->vals->header_size);
rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER;
rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE;
rsp_hdr->CreditRequest = cpu_to_le16(0);
rsp_hdr->SessionId = 0;
memset(rsp_hdr->Signature, 0, 16);
- rsp = work->response_buf;
+ rsp = smb2_get_msg(work->response_buf);
rsp->StructureSize = cpu_to_le16(44);
rsp->Epoch = br_info->epoch;
rsp->Flags = 0;
rsp->AccessMaskHint = 0;
rsp->ShareMaskHint = 0;
- inc_rfc1001_len(rsp, 44);
+ inc_rfc1001_len(work->response_buf, 44);
ksmbd_conn_write(work);
ksmbd_free_work_struct(work);
*/
void create_lease_buf(u8 *rbuf, struct lease *lease)
{
- char *LeaseKey = (char *)&lease->lease_key;
-
if (lease->version == 2) {
struct create_lease_v2 *buf = (struct create_lease_v2 *)rbuf;
- char *ParentLeaseKey = (char *)&lease->parent_lease_key;
memset(buf, 0, sizeof(struct create_lease_v2));
- buf->lcontext.LeaseKeyLow = *((__le64 *)LeaseKey);
- buf->lcontext.LeaseKeyHigh = *((__le64 *)(LeaseKey + 8));
+ memcpy(buf->lcontext.LeaseKey, lease->lease_key,
+ SMB2_LEASE_KEY_SIZE);
buf->lcontext.LeaseFlags = lease->flags;
buf->lcontext.LeaseState = lease->state;
- buf->lcontext.ParentLeaseKeyLow = *((__le64 *)ParentLeaseKey);
- buf->lcontext.ParentLeaseKeyHigh = *((__le64 *)(ParentLeaseKey + 8));
+ memcpy(buf->lcontext.ParentLeaseKey, lease->parent_lease_key,
+ SMB2_LEASE_KEY_SIZE);
buf->ccontext.DataOffset = cpu_to_le16(offsetof
(struct create_lease_v2, lcontext));
buf->ccontext.DataLength = cpu_to_le32(sizeof(struct lease_context_v2));
struct create_lease *buf = (struct create_lease *)rbuf;
memset(buf, 0, sizeof(struct create_lease));
- buf->lcontext.LeaseKeyLow = *((__le64 *)LeaseKey);
- buf->lcontext.LeaseKeyHigh = *((__le64 *)(LeaseKey + 8));
+ memcpy(buf->lcontext.LeaseKey, lease->lease_key, SMB2_LEASE_KEY_SIZE);
buf->lcontext.LeaseFlags = lease->flags;
buf->lcontext.LeaseState = lease->state;
buf->ccontext.DataOffset = cpu_to_le16(offsetof
if (!lreq)
return NULL;
- data_offset = (char *)req + 4 + le32_to_cpu(req->CreateContextsOffset);
+ data_offset = (char *)req + le32_to_cpu(req->CreateContextsOffset);
cc = (struct create_context *)data_offset;
do {
cc = (struct create_context *)((char *)cc + next);
if (sizeof(struct lease_context_v2) == le32_to_cpu(cc->DataLength)) {
struct create_lease_v2 *lc = (struct create_lease_v2 *)cc;
- *((__le64 *)lreq->lease_key) = lc->lcontext.LeaseKeyLow;
- *((__le64 *)(lreq->lease_key + 8)) = lc->lcontext.LeaseKeyHigh;
+ memcpy(lreq->lease_key, lc->lcontext.LeaseKey, SMB2_LEASE_KEY_SIZE);
lreq->req_state = lc->lcontext.LeaseState;
lreq->flags = lc->lcontext.LeaseFlags;
lreq->duration = lc->lcontext.LeaseDuration;
- *((__le64 *)lreq->parent_lease_key) = lc->lcontext.ParentLeaseKeyLow;
- *((__le64 *)(lreq->parent_lease_key + 8)) = lc->lcontext.ParentLeaseKeyHigh;
+ memcpy(lreq->parent_lease_key, lc->lcontext.ParentLeaseKey,
+ SMB2_LEASE_KEY_SIZE);
lreq->version = 2;
} else {
struct create_lease *lc = (struct create_lease *)cc;
- *((__le64 *)lreq->lease_key) = lc->lcontext.LeaseKeyLow;
- *((__le64 *)(lreq->lease_key + 8)) = lc->lcontext.LeaseKeyHigh;
+ memcpy(lreq->lease_key, lc->lcontext.LeaseKey, SMB2_LEASE_KEY_SIZE);
lreq->req_state = lc->lcontext.LeaseState;
lreq->flags = lc->lcontext.LeaseFlags;
lreq->duration = lc->lcontext.LeaseDuration;
* CreateContextsOffset and CreateContextsLength are guaranteed to
* be valid because of ksmbd_smb2_check_message().
*/
- cc = (struct create_context *)((char *)req + 4 +
+ cc = (struct create_context *)((char *)req +
le32_to_cpu(req->CreateContextsOffset));
remain_len = le32_to_cpu(req->CreateContextsLength);
do {
#define OPLOCK_WRITE_TO_NONE 0x04
#define OPLOCK_READ_TO_NONE 0x08
-#define SMB2_LEASE_KEY_SIZE 16
-
struct lease_ctx_info {
__u8 lease_key[SMB2_LEASE_KEY_SIZE];
__le32 req_state;
MODULE_LICENSE("GPL");
MODULE_SOFTDEP("pre: ecb");
MODULE_SOFTDEP("pre: hmac");
-MODULE_SOFTDEP("pre: md4");
MODULE_SOFTDEP("pre: md5");
MODULE_SOFTDEP("pre: nls");
MODULE_SOFTDEP("pre: aes");
MODULE_SOFTDEP("pre: aead2");
MODULE_SOFTDEP("pre: ccm");
MODULE_SOFTDEP("pre: gcm");
+MODULE_SOFTDEP("pre: crc32");
module_init(ksmbd_server_init)
module_exit(ksmbd_server_exit)
#include "glob.h"
#include "nterr.h"
-#include "smb2pdu.h"
#include "smb_common.h"
#include "smbstatus.h"
#include "mgmt/user_session.h"
int ksmbd_smb2_check_message(struct ksmbd_work *work)
{
- struct smb2_pdu *pdu = work->request_buf;
+ struct smb2_pdu *pdu = ksmbd_req_buf_next(work);
struct smb2_hdr *hdr = &pdu->hdr;
int command;
__u32 clc_len; /* calculated length */
- __u32 len = get_rfc1002_len(pdu);
+ __u32 len = get_rfc1002_len(work->request_buf);
- if (work->next_smb2_rcv_hdr_off) {
- pdu = ksmbd_req_buf_next(work);
- hdr = &pdu->hdr;
- }
-
- if (le32_to_cpu(hdr->NextCommand) > 0) {
+ if (le32_to_cpu(hdr->NextCommand) > 0)
len = le32_to_cpu(hdr->NextCommand);
- } else if (work->next_smb2_rcv_hdr_off) {
+ else if (work->next_smb2_rcv_hdr_off)
len -= work->next_smb2_rcv_hdr_off;
- len = round_up(len, 8);
- }
if (check_smb2_hdr(hdr))
return 1;
#include <linux/slab.h>
#include "glob.h"
-#include "smb2pdu.h"
#include "auth.h"
#include "connection.h"
conn->cmds = smb2_0_server_cmds;
conn->max_cmds = ARRAY_SIZE(smb2_0_server_cmds);
conn->max_credits = SMB2_MAX_CREDITS;
- conn->signing_algorithm = SIGNING_ALG_HMAC_SHA256;
+ conn->signing_algorithm = SIGNING_ALG_HMAC_SHA256_LE;
if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB2_LEASES)
conn->vals->capabilities |= SMB2_GLOBAL_CAP_LEASING;
conn->cmds = smb2_0_server_cmds;
conn->max_cmds = ARRAY_SIZE(smb2_0_server_cmds);
conn->max_credits = SMB2_MAX_CREDITS;
- conn->signing_algorithm = SIGNING_ALG_AES_CMAC;
+ conn->signing_algorithm = SIGNING_ALG_AES_CMAC_LE;
if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB2_LEASES)
conn->vals->capabilities |= SMB2_GLOBAL_CAP_LEASING;
conn->cmds = smb2_0_server_cmds;
conn->max_cmds = ARRAY_SIZE(smb2_0_server_cmds);
conn->max_credits = SMB2_MAX_CREDITS;
- conn->signing_algorithm = SIGNING_ALG_AES_CMAC;
+ conn->signing_algorithm = SIGNING_ALG_AES_CMAC_LE;
if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB2_LEASES)
conn->vals->capabilities |= SMB2_GLOBAL_CAP_LEASING;
conn->cmds = smb2_0_server_cmds;
conn->max_cmds = ARRAY_SIZE(smb2_0_server_cmds);
conn->max_credits = SMB2_MAX_CREDITS;
- conn->signing_algorithm = SIGNING_ALG_AES_CMAC;
+ conn->signing_algorithm = SIGNING_ALG_AES_CMAC_LE;
if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB2_LEASES)
conn->vals->capabilities |= SMB2_GLOBAL_CAP_LEASING;
#include <linux/falloc.h>
#include "glob.h"
-#include "smb2pdu.h"
#include "smbfsctl.h"
#include "oplock.h"
#include "smbacl.h"
*req = ksmbd_req_buf_next(work);
*rsp = ksmbd_resp_buf_next(work);
} else {
- *req = work->request_buf;
- *rsp = work->response_buf;
+ *req = smb2_get_msg(work->request_buf);
+ *rsp = smb2_get_msg(work->response_buf);
}
}
*/
int smb2_get_ksmbd_tcon(struct ksmbd_work *work)
{
- struct smb2_hdr *req_hdr = work->request_buf;
+ struct smb2_hdr *req_hdr = smb2_get_msg(work->request_buf);
+ unsigned int cmd = le16_to_cpu(req_hdr->Command);
int tree_id;
work->tcon = NULL;
- if (work->conn->ops->get_cmd_val(work) == SMB2_TREE_CONNECT_HE ||
- work->conn->ops->get_cmd_val(work) == SMB2_CANCEL_HE ||
- work->conn->ops->get_cmd_val(work) == SMB2_LOGOFF_HE) {
+ if (cmd == SMB2_TREE_CONNECT_HE ||
+ cmd == SMB2_CANCEL_HE ||
+ cmd == SMB2_LOGOFF_HE) {
ksmbd_debug(SMB, "skip to check tree connect request\n");
return 0;
}
if (work->next_smb2_rcv_hdr_off)
err_rsp = ksmbd_resp_buf_next(work);
else
- err_rsp = work->response_buf;
+ err_rsp = smb2_get_msg(work->response_buf);
if (err_rsp->hdr.Status != STATUS_STOPPED_ON_SYMLINK) {
err_rsp->StructureSize = SMB2_ERROR_STRUCTURE_SIZE2_LE;
*/
bool is_smb2_neg_cmd(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr = work->request_buf;
+ struct smb2_hdr *hdr = smb2_get_msg(work->request_buf);
/* is it SMB2 header ? */
if (hdr->ProtocolId != SMB2_PROTO_NUMBER)
*/
bool is_smb2_rsp(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr = work->response_buf;
+ struct smb2_hdr *hdr = smb2_get_msg(work->response_buf);
/* is it SMB2 header ? */
if (hdr->ProtocolId != SMB2_PROTO_NUMBER)
if (work->next_smb2_rcv_hdr_off)
rcv_hdr = ksmbd_req_buf_next(work);
else
- rcv_hdr = work->request_buf;
+ rcv_hdr = smb2_get_msg(work->request_buf);
return le16_to_cpu(rcv_hdr->Command);
}
if (work->next_smb2_rcv_hdr_off)
rsp_hdr = ksmbd_resp_buf_next(work);
else
- rsp_hdr = work->response_buf;
+ rsp_hdr = smb2_get_msg(work->response_buf);
rsp_hdr->Status = err;
smb2_set_err_rsp(work);
}
if (conn->need_neg == false)
return -EINVAL;
- rsp_hdr = work->response_buf;
+ *(__be32 *)work->response_buf =
+ cpu_to_be32(conn->vals->header_size);
+ rsp_hdr = smb2_get_msg(work->response_buf);
memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2);
-
- rsp_hdr->smb2_buf_length =
- cpu_to_be32(smb2_hdr_size_no_buflen(conn->vals));
-
rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER;
rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE;
rsp_hdr->CreditRequest = cpu_to_le16(2);
rsp_hdr->SessionId = 0;
memset(rsp_hdr->Signature, 0, 16);
- rsp = work->response_buf;
+ rsp = smb2_get_msg(work->response_buf);
WARN_ON(ksmbd_conn_good(work));
rsp->SecurityBufferOffset = cpu_to_le16(128);
rsp->SecurityBufferLength = cpu_to_le16(AUTH_GSS_LENGTH);
- ksmbd_copy_gss_neg_header(((char *)(&rsp->hdr) +
- sizeof(rsp->hdr.smb2_buf_length)) +
+ ksmbd_copy_gss_neg_header((char *)(&rsp->hdr) +
le16_to_cpu(rsp->SecurityBufferOffset));
- inc_rfc1001_len(rsp, sizeof(struct smb2_negotiate_rsp) -
- sizeof(struct smb2_hdr) - sizeof(rsp->Buffer) +
- AUTH_GSS_LENGTH);
+ inc_rfc1001_len(work->response_buf,
+ sizeof(struct smb2_negotiate_rsp) -
+ sizeof(struct smb2_hdr) - sizeof(rsp->Buffer) +
+ AUTH_GSS_LENGTH);
rsp->SecurityMode = SMB2_NEGOTIATE_SIGNING_ENABLED_LE;
if (server_conf.signing == KSMBD_CONFIG_OPT_MANDATORY)
rsp->SecurityMode |= SMB2_NEGOTIATE_SIGNING_REQUIRED_LE;
next_hdr_offset = le32_to_cpu(req->NextCommand);
new_len = ALIGN(len, 8);
- inc_rfc1001_len(work->response_buf, ((sizeof(struct smb2_hdr) - 4)
- + new_len - len));
+ inc_rfc1001_len(work->response_buf,
+ sizeof(struct smb2_hdr) + new_len - len);
rsp->NextCommand = cpu_to_le32(new_len);
work->next_smb2_rcv_hdr_off += next_hdr_offset;
work->compound_fid = KSMBD_NO_FID;
work->compound_pfid = KSMBD_NO_FID;
}
- memset((char *)rsp_hdr + 4, 0, sizeof(struct smb2_hdr) + 2);
+ memset((char *)rsp_hdr, 0, sizeof(struct smb2_hdr) + 2);
rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER;
rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE;
rsp_hdr->Command = rcv_hdr->Command;
*/
bool is_chained_smb2_message(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr = work->request_buf;
+ struct smb2_hdr *hdr = smb2_get_msg(work->request_buf);
unsigned int len, next_cmd;
if (hdr->ProtocolId != SMB2_PROTO_NUMBER)
*/
int init_smb2_rsp_hdr(struct ksmbd_work *work)
{
- struct smb2_hdr *rsp_hdr = work->response_buf;
- struct smb2_hdr *rcv_hdr = work->request_buf;
+ struct smb2_hdr *rsp_hdr = smb2_get_msg(work->response_buf);
+ struct smb2_hdr *rcv_hdr = smb2_get_msg(work->request_buf);
struct ksmbd_conn *conn = work->conn;
memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2);
- rsp_hdr->smb2_buf_length =
- cpu_to_be32(smb2_hdr_size_no_buflen(conn->vals));
+ *(__be32 *)work->response_buf =
+ cpu_to_be32(conn->vals->header_size);
rsp_hdr->ProtocolId = rcv_hdr->ProtocolId;
rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE;
rsp_hdr->Command = rcv_hdr->Command;
*/
int smb2_allocate_rsp_buf(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr = work->request_buf;
+ struct smb2_hdr *hdr = smb2_get_msg(work->request_buf);
size_t small_sz = MAX_CIFS_SMALL_BUFFER_SIZE;
size_t large_sz = small_sz + work->conn->vals->max_trans_size;
size_t sz = small_sz;
if (cmd == SMB2_QUERY_INFO_HE) {
struct smb2_query_info_req *req;
- req = work->request_buf;
+ req = smb2_get_msg(work->request_buf);
if (req->InfoType == SMB2_O_INFO_FILE &&
(req->FileInfoClass == FILE_FULL_EA_INFORMATION ||
req->FileInfoClass == FILE_ALL_INFORMATION))
*/
int smb2_check_user_session(struct ksmbd_work *work)
{
- struct smb2_hdr *req_hdr = work->request_buf;
+ struct smb2_hdr *req_hdr = smb2_get_msg(work->request_buf);
struct ksmbd_conn *conn = work->conn;
unsigned int cmd = conn->ops->get_cmd_val(work);
unsigned long long sess_id;
struct ksmbd_conn *conn = work->conn;
int id;
- rsp_hdr = work->response_buf;
+ rsp_hdr = smb2_get_msg(work->response_buf);
rsp_hdr->Flags |= SMB2_FLAGS_ASYNC_COMMAND;
id = ksmbd_acquire_async_msg_id(&conn->async_ida);
{
struct smb2_hdr *rsp_hdr;
- rsp_hdr = work->response_buf;
+ rsp_hdr = smb2_get_msg(work->response_buf);
smb2_set_err_rsp(work);
rsp_hdr->Status = status;
int attr = 0;
if (S_ISDIR(stat->mode)) {
- attr = ATTR_DIRECTORY |
- (attribute & (ATTR_HIDDEN | ATTR_SYSTEM));
+ attr = FILE_ATTRIBUTE_DIRECTORY |
+ (attribute & (FILE_ATTRIBUTE_HIDDEN | FILE_ATTRIBUTE_SYSTEM));
} else {
- attr = (attribute & 0x00005137) | ATTR_ARCHIVE;
- attr &= ~(ATTR_DIRECTORY);
+ attr = (attribute & 0x00005137) | FILE_ATTRIBUTE_ARCHIVE;
+ attr &= ~(FILE_ATTRIBUTE_DIRECTORY);
if (S_ISREG(stat->mode) && (server_conf.share_fake_fscaps &
FILE_SUPPORTS_SPARSE_FILES))
- attr |= ATTR_SPARSE;
+ attr |= FILE_ATTRIBUTE_SPARSE_FILE;
if (smb2_get_reparse_tag_special_file(stat->mode))
- attr |= ATTR_REPARSE;
+ attr |= FILE_ATTRIBUTE_REPARSE_POINT;
}
return attr;
pneg_ctxt->Ciphers[0] = cipher_type;
}
-static void build_compression_ctxt(struct smb2_compression_ctx *pneg_ctxt,
+static void build_compression_ctxt(struct smb2_compression_capabilities_context *pneg_ctxt,
__le16 comp_algo)
{
pneg_ctxt->ContextType = SMB2_COMPRESSION_CAPABILITIES;
pneg_ctxt->DataLength =
- cpu_to_le16(sizeof(struct smb2_compression_ctx)
+ cpu_to_le16(sizeof(struct smb2_compression_capabilities_context)
- sizeof(struct smb2_neg_context));
pneg_ctxt->Reserved = cpu_to_le32(0);
pneg_ctxt->CompressionAlgorithmCount = cpu_to_le16(1);
- pneg_ctxt->Reserved1 = cpu_to_le32(0);
+ pneg_ctxt->Flags = cpu_to_le32(0);
pneg_ctxt->CompressionAlgorithms[0] = comp_algo;
}
}
static void assemble_neg_contexts(struct ksmbd_conn *conn,
- struct smb2_negotiate_rsp *rsp)
+ struct smb2_negotiate_rsp *rsp,
+ void *smb2_buf_len)
{
- /* +4 is to account for the RFC1001 len field */
char *pneg_ctxt = (char *)rsp +
- le32_to_cpu(rsp->NegotiateContextOffset) + 4;
+ le32_to_cpu(rsp->NegotiateContextOffset);
int neg_ctxt_cnt = 1;
int ctxt_size;
build_preauth_ctxt((struct smb2_preauth_neg_context *)pneg_ctxt,
conn->preauth_info->Preauth_HashId);
rsp->NegotiateContextCount = cpu_to_le16(neg_ctxt_cnt);
- inc_rfc1001_len(rsp, AUTH_GSS_PADDING);
+ inc_rfc1001_len(smb2_buf_len, AUTH_GSS_PADDING);
ctxt_size = sizeof(struct smb2_preauth_neg_context);
/* Round to 8 byte boundary */
pneg_ctxt += round_up(sizeof(struct smb2_preauth_neg_context), 8);
ksmbd_debug(SMB,
"assemble SMB2_COMPRESSION_CAPABILITIES context\n");
/* Temporarily set to SMB3_COMPRESS_NONE */
- build_compression_ctxt((struct smb2_compression_ctx *)pneg_ctxt,
+ build_compression_ctxt((struct smb2_compression_capabilities_context *)pneg_ctxt,
conn->compress_algorithm);
rsp->NegotiateContextCount = cpu_to_le16(++neg_ctxt_cnt);
- ctxt_size += sizeof(struct smb2_compression_ctx) + 2;
+ ctxt_size += sizeof(struct smb2_compression_capabilities_context) + 2;
/* Round to 8 byte boundary */
- pneg_ctxt += round_up(sizeof(struct smb2_compression_ctx) + 2,
+ pneg_ctxt += round_up(sizeof(struct smb2_compression_capabilities_context) + 2,
8);
}
ctxt_size += sizeof(struct smb2_signing_capabilities) + 2;
}
- inc_rfc1001_len(rsp, ctxt_size);
+ inc_rfc1001_len(smb2_buf_len, ctxt_size);
}
static __le32 decode_preauth_ctxt(struct ksmbd_conn *conn,
}
static void decode_compress_ctxt(struct ksmbd_conn *conn,
- struct smb2_compression_ctx *pneg_ctxt)
+ struct smb2_compression_capabilities_context *pneg_ctxt)
{
conn->compress_algorithm = SMB3_COMPRESS_NONE;
}
}
for (i = 0; i < sign_algo_cnt; i++) {
- if (pneg_ctxt->SigningAlgorithms[i] == SIGNING_ALG_HMAC_SHA256 ||
- pneg_ctxt->SigningAlgorithms[i] == SIGNING_ALG_AES_CMAC) {
+ if (pneg_ctxt->SigningAlgorithms[i] == SIGNING_ALG_HMAC_SHA256_LE ||
+ pneg_ctxt->SigningAlgorithms[i] == SIGNING_ALG_AES_CMAC_LE) {
ksmbd_debug(SMB, "Signing Algorithm ID = 0x%x\n",
pneg_ctxt->SigningAlgorithms[i]);
conn->signing_negotiated = true;
}
static __le32 deassemble_neg_contexts(struct ksmbd_conn *conn,
- struct smb2_negotiate_req *req)
+ struct smb2_negotiate_req *req,
+ int len_of_smb)
{
/* +4 is to account for the RFC1001 len field */
- struct smb2_neg_context *pctx = (struct smb2_neg_context *)((char *)req + 4);
+ struct smb2_neg_context *pctx = (struct smb2_neg_context *)req;
int i = 0, len_of_ctxts;
int offset = le32_to_cpu(req->NegotiateContextOffset);
int neg_ctxt_cnt = le16_to_cpu(req->NegotiateContextCount);
- int len_of_smb = be32_to_cpu(req->hdr.smb2_buf_length);
__le32 status = STATUS_INVALID_PARAMETER;
ksmbd_debug(SMB, "decoding %d negotiate contexts\n", neg_ctxt_cnt);
break;
decode_compress_ctxt(conn,
- (struct smb2_compression_ctx *)pctx);
+ (struct smb2_compression_capabilities_context *)pctx);
} else if (pctx->ContextType == SMB2_NETNAME_NEGOTIATE_CONTEXT_ID) {
ksmbd_debug(SMB,
"deassemble SMB2_NETNAME_NEGOTIATE_CONTEXT_ID context\n");
int smb2_handle_negotiate(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_negotiate_req *req = work->request_buf;
- struct smb2_negotiate_rsp *rsp = work->response_buf;
+ struct smb2_negotiate_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_negotiate_rsp *rsp = smb2_get_msg(work->response_buf);
int rc = 0;
unsigned int smb2_buf_len, smb2_neg_size;
__le32 status;
}
smb2_buf_len = get_rfc1002_len(work->request_buf);
- smb2_neg_size = offsetof(struct smb2_negotiate_req, Dialects) - 4;
+ smb2_neg_size = offsetof(struct smb2_negotiate_req, Dialects);
if (smb2_neg_size > smb2_buf_len) {
rsp->hdr.Status = STATUS_INVALID_PARAMETER;
rc = -EINVAL;
goto err_out;
}
- status = deassemble_neg_contexts(conn, req);
+ status = deassemble_neg_contexts(conn, req,
+ get_rfc1002_len(work->request_buf));
if (status != STATUS_SUCCESS) {
pr_err("deassemble_neg_contexts error(0x%x)\n",
status);
conn->preauth_info->Preauth_HashValue);
rsp->NegotiateContextOffset =
cpu_to_le32(OFFSET_OF_NEG_CONTEXT);
- assemble_neg_contexts(conn, rsp);
+ assemble_neg_contexts(conn, rsp, work->response_buf);
break;
case SMB302_PROT_ID:
init_smb3_02_server(conn);
rsp->SecurityBufferOffset = cpu_to_le16(128);
rsp->SecurityBufferLength = cpu_to_le16(AUTH_GSS_LENGTH);
- ksmbd_copy_gss_neg_header(((char *)(&rsp->hdr) +
- sizeof(rsp->hdr.smb2_buf_length)) +
- le16_to_cpu(rsp->SecurityBufferOffset));
- inc_rfc1001_len(rsp, sizeof(struct smb2_negotiate_rsp) -
+ ksmbd_copy_gss_neg_header((char *)(&rsp->hdr) +
+ le16_to_cpu(rsp->SecurityBufferOffset));
+ inc_rfc1001_len(work->response_buf, sizeof(struct smb2_negotiate_rsp) -
sizeof(struct smb2_hdr) - sizeof(rsp->Buffer) +
AUTH_GSS_LENGTH);
rsp->SecurityMode = SMB2_NEGOTIATE_SIGNING_ENABLED_LE;
struct negotiate_message *negblob,
size_t negblob_len)
{
- struct smb2_sess_setup_rsp *rsp = work->response_buf;
+ struct smb2_sess_setup_rsp *rsp = smb2_get_msg(work->response_buf);
struct challenge_message *chgblob;
unsigned char *spnego_blob = NULL;
u16 spnego_blob_len;
static int ntlm_authenticate(struct ksmbd_work *work)
{
- struct smb2_sess_setup_req *req = work->request_buf;
- struct smb2_sess_setup_rsp *rsp = work->response_buf;
+ struct smb2_sess_setup_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_sess_setup_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_conn *conn = work->conn;
struct ksmbd_session *sess = work->sess;
struct channel *chann = NULL;
memcpy((char *)&rsp->hdr.ProtocolId + sz, spnego_blob, spnego_blob_len);
rsp->SecurityBufferLength = cpu_to_le16(spnego_blob_len);
kfree(spnego_blob);
- inc_rfc1001_len(rsp, spnego_blob_len - 1);
+ inc_rfc1001_len(work->response_buf, spnego_blob_len - 1);
}
user = session_user(conn, req);
#ifdef CONFIG_SMB_SERVER_KERBEROS5
static int krb5_authenticate(struct ksmbd_work *work)
{
- struct smb2_sess_setup_req *req = work->request_buf;
- struct smb2_sess_setup_rsp *rsp = work->response_buf;
+ struct smb2_sess_setup_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_sess_setup_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_conn *conn = work->conn;
struct ksmbd_session *sess = work->sess;
char *in_blob, *out_blob;
out_blob = (char *)&rsp->hdr.ProtocolId +
le16_to_cpu(rsp->SecurityBufferOffset);
out_len = work->response_sz -
- offsetof(struct smb2_hdr, smb2_buf_length) -
- le16_to_cpu(rsp->SecurityBufferOffset);
+ (le16_to_cpu(rsp->SecurityBufferOffset) + 4);
/* Check previous session */
prev_sess_id = le64_to_cpu(req->PreviousSessionId);
return -EINVAL;
}
rsp->SecurityBufferLength = cpu_to_le16(out_len);
- inc_rfc1001_len(rsp, out_len - 1);
+ inc_rfc1001_len(work->response_buf, out_len - 1);
if ((conn->sign || server_conf.enforced_signing) ||
(req->SecurityMode & SMB2_NEGOTIATE_SIGNING_REQUIRED))
int smb2_sess_setup(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_sess_setup_req *req = work->request_buf;
- struct smb2_sess_setup_rsp *rsp = work->response_buf;
+ struct smb2_sess_setup_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_sess_setup_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_session *sess;
struct negotiate_message *negblob;
unsigned int negblob_len, negblob_off;
rsp->SessionFlags = 0;
rsp->SecurityBufferOffset = cpu_to_le16(72);
rsp->SecurityBufferLength = 0;
- inc_rfc1001_len(rsp, 9);
+ inc_rfc1001_len(work->response_buf, 9);
if (!req->hdr.SessionId) {
sess = ksmbd_smb2_session_create();
negblob_off = le16_to_cpu(req->SecurityBufferOffset);
negblob_len = le16_to_cpu(req->SecurityBufferLength);
- if (negblob_off < (offsetof(struct smb2_sess_setup_req, Buffer) - 4) ||
+ if (negblob_off < offsetof(struct smb2_sess_setup_req, Buffer) ||
negblob_len < offsetof(struct negotiate_message, NegotiateFlags))
return -EINVAL;
* Note: here total size -1 is done as an
* adjustment for 0 size blob
*/
- inc_rfc1001_len(rsp, le16_to_cpu(rsp->SecurityBufferLength) - 1);
+ inc_rfc1001_len(work->response_buf,
+ le16_to_cpu(rsp->SecurityBufferLength) - 1);
} else if (negblob->MessageType == NtLmAuthenticate) {
rc = ntlm_authenticate(work);
int smb2_tree_connect(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_tree_connect_req *req = work->request_buf;
- struct smb2_tree_connect_rsp *rsp = work->response_buf;
+ struct smb2_tree_connect_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_tree_connect_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_session *sess = work->sess;
char *treename = NULL, *name = NULL;
struct ksmbd_tree_conn_status status;
rsp->Reserved = 0;
/* default manual caching */
rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING;
- inc_rfc1001_len(rsp, 16);
+ inc_rfc1001_len(work->response_buf, 16);
if (!IS_ERR(treename))
kfree(treename);
*/
int smb2_tree_disconnect(struct ksmbd_work *work)
{
- struct smb2_tree_disconnect_rsp *rsp = work->response_buf;
+ struct smb2_tree_disconnect_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_session *sess = work->sess;
struct ksmbd_tree_connect *tcon = work->tcon;
rsp->StructureSize = cpu_to_le16(4);
- inc_rfc1001_len(rsp, 4);
+ inc_rfc1001_len(work->response_buf, 4);
ksmbd_debug(SMB, "request\n");
if (!tcon) {
- struct smb2_tree_disconnect_req *req = work->request_buf;
+ struct smb2_tree_disconnect_req *req =
+ smb2_get_msg(work->request_buf);
ksmbd_debug(SMB, "Invalid tid %d\n", req->hdr.Id.SyncId.TreeId);
rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
int smb2_session_logoff(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_logoff_rsp *rsp = work->response_buf;
+ struct smb2_logoff_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_session *sess = work->sess;
rsp->StructureSize = cpu_to_le16(4);
- inc_rfc1001_len(rsp, 4);
+ inc_rfc1001_len(work->response_buf, 4);
ksmbd_debug(SMB, "request\n");
ksmbd_conn_wait_idle(conn);
if (ksmbd_tree_conn_session_logoff(sess)) {
- struct smb2_logoff_req *req = work->request_buf;
+ struct smb2_logoff_req *req = smb2_get_msg(work->request_buf);
ksmbd_debug(SMB, "Invalid tid %d\n", req->hdr.Id.SyncId.TreeId);
rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
*/
static noinline int create_smb2_pipe(struct ksmbd_work *work)
{
- struct smb2_create_rsp *rsp = work->response_buf;
- struct smb2_create_req *req = work->request_buf;
+ struct smb2_create_rsp *rsp = smb2_get_msg(work->response_buf);
+ struct smb2_create_req *req = smb2_get_msg(work->request_buf);
int id;
int err;
char *name;
rsp->hdr.Status = STATUS_SUCCESS;
rsp->StructureSize = cpu_to_le16(89);
rsp->OplockLevel = SMB2_OPLOCK_LEVEL_NONE;
- rsp->Reserved = 0;
+ rsp->Flags = 0;
rsp->CreateAction = cpu_to_le32(FILE_OPENED);
rsp->CreationTime = cpu_to_le64(0);
rsp->ChangeTime = cpu_to_le64(0);
rsp->AllocationSize = cpu_to_le64(0);
rsp->EndofFile = cpu_to_le64(0);
- rsp->FileAttributes = ATTR_NORMAL_LE;
+ rsp->FileAttributes = FILE_ATTRIBUTE_NORMAL_LE;
rsp->Reserved2 = 0;
rsp->VolatileFileId = cpu_to_le64(id);
rsp->PersistentFileId = 0;
rsp->CreateContextsOffset = 0;
rsp->CreateContextsLength = 0;
- inc_rfc1001_len(rsp, 88); /* StructureSize - 1*/
+ inc_rfc1001_len(work->response_buf, 88); /* StructureSize - 1*/
kfree(name);
return 0;
struct xattr_dos_attrib da;
int rc;
- fp->f_ci->m_fattr &= ~(ATTR_HIDDEN_LE | ATTR_SYSTEM_LE);
+ fp->f_ci->m_fattr &= ~(FILE_ATTRIBUTE_HIDDEN_LE | FILE_ATTRIBUTE_SYSTEM_LE);
/* get FileAttributes from XATTR_NAME_DOS_ATTRIBUTE */
if (!test_share_config_flag(tcon->share_conf,
struct ksmbd_session *sess = work->sess;
struct ksmbd_tree_connect *tcon = work->tcon;
struct smb2_create_req *req;
- struct smb2_create_rsp *rsp, *rsp_org;
+ struct smb2_create_rsp *rsp;
struct path path;
struct ksmbd_share_config *share = tcon->share_conf;
struct ksmbd_file *fp = NULL;
umode_t posix_mode = 0;
__le32 daccess, maximal_access = 0;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
if (req->hdr.NextCommand && !work->next_smb2_rcv_hdr_off &&
if (req_op_level == SMB2_OPLOCK_LEVEL_LEASE)
lc = parse_lease_state(req);
- if (le32_to_cpu(req->ImpersonationLevel) > le32_to_cpu(IL_DELEGATE_LE)) {
+ if (le32_to_cpu(req->ImpersonationLevel) > le32_to_cpu(IL_DELEGATE)) {
pr_err("Invalid impersonationlevel : 0x%x\n",
le32_to_cpu(req->ImpersonationLevel));
rc = -EIO;
goto err_out1;
}
- if (req->CreateOptions && !(req->CreateOptions & CREATE_OPTIONS_MASK)) {
+ if (req->CreateOptions && !(req->CreateOptions & CREATE_OPTIONS_MASK_LE)) {
pr_err("Invalid create options : 0x%x\n",
le32_to_cpu(req->CreateOptions));
rc = -EINVAL;
goto err_out1;
}
- if (req->FileAttributes && !(req->FileAttributes & ATTR_MASK_LE)) {
+ if (req->FileAttributes && !(req->FileAttributes & FILE_ATTRIBUTE_MASK_LE)) {
pr_err("Invalid file attribute : 0x%x\n",
le32_to_cpu(req->FileAttributes));
rc = -EINVAL;
}
if (req->CreateOptions & FILE_DIRECTORY_FILE_LE &&
- req->FileAttributes & ATTR_NORMAL_LE) {
+ req->FileAttributes & FILE_ATTRIBUTE_NORMAL_LE) {
rsp->hdr.Status = STATUS_NOT_A_DIRECTORY;
rc = -EIO;
}
opinfo = rcu_dereference(fp->f_opinfo);
rsp->OplockLevel = opinfo != NULL ? opinfo->level : 0;
rcu_read_unlock();
- rsp->Reserved = 0;
+ rsp->Flags = 0;
rsp->CreateAction = cpu_to_le32(file_info);
rsp->CreationTime = cpu_to_le64(fp->create_time);
time = ksmbd_UnixTimeToNT(stat.atime);
rsp->CreateContextsOffset = 0;
rsp->CreateContextsLength = 0;
- inc_rfc1001_len(rsp_org, 88); /* StructureSize - 1*/
+ inc_rfc1001_len(work->response_buf, 88); /* StructureSize - 1*/
/* If lease is request send lease context response */
if (opinfo && opinfo->is_lease) {
create_lease_buf(rsp->Buffer, opinfo->o_lease);
le32_add_cpu(&rsp->CreateContextsLength,
conn->vals->create_lease_size);
- inc_rfc1001_len(rsp_org, conn->vals->create_lease_size);
+ inc_rfc1001_len(work->response_buf,
+ conn->vals->create_lease_size);
next_ptr = &lease_ccontext->Next;
next_off = conn->vals->create_lease_size;
}
le32_to_cpu(maximal_access));
le32_add_cpu(&rsp->CreateContextsLength,
conn->vals->create_mxac_size);
- inc_rfc1001_len(rsp_org, conn->vals->create_mxac_size);
+ inc_rfc1001_len(work->response_buf,
+ conn->vals->create_mxac_size);
if (next_ptr)
*next_ptr = cpu_to_le32(next_off);
next_ptr = &mxac_ccontext->Next;
stat.ino, tcon->id);
le32_add_cpu(&rsp->CreateContextsLength,
conn->vals->create_disk_id_size);
- inc_rfc1001_len(rsp_org, conn->vals->create_disk_id_size);
+ inc_rfc1001_len(work->response_buf,
+ conn->vals->create_disk_id_size);
if (next_ptr)
*next_ptr = cpu_to_le32(next_off);
next_ptr = &disk_id_ccontext->Next;
fp);
le32_add_cpu(&rsp->CreateContextsLength,
conn->vals->create_posix_size);
- inc_rfc1001_len(rsp_org, conn->vals->create_posix_size);
+ inc_rfc1001_len(work->response_buf,
+ conn->vals->create_posix_size);
if (next_ptr)
*next_ptr = cpu_to_le32(next_off);
}
if (contxt_cnt > 0) {
rsp->CreateContextsOffset =
- cpu_to_le32(offsetof(struct smb2_create_rsp, Buffer)
- - 4);
+ cpu_to_le32(offsetof(struct smb2_create_rsp, Buffer));
}
err_out:
ffdinfo->EaSize =
smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode);
if (ffdinfo->EaSize)
- ffdinfo->ExtFileAttributes = ATTR_REPARSE_POINT_LE;
+ ffdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE;
if (d_info->hide_dot_file && d_info->name[0] == '.')
- ffdinfo->ExtFileAttributes |= ATTR_HIDDEN_LE;
+ ffdinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
memcpy(ffdinfo->FileName, conv_name, conv_len);
ffdinfo->NextEntryOffset = cpu_to_le32(next_entry_offset);
break;
fbdinfo->EaSize =
smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode);
if (fbdinfo->EaSize)
- fbdinfo->ExtFileAttributes = ATTR_REPARSE_POINT_LE;
+ fbdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE;
fbdinfo->ShortNameLength = 0;
fbdinfo->Reserved = 0;
if (d_info->hide_dot_file && d_info->name[0] == '.')
- fbdinfo->ExtFileAttributes |= ATTR_HIDDEN_LE;
+ fbdinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
memcpy(fbdinfo->FileName, conv_name, conv_len);
fbdinfo->NextEntryOffset = cpu_to_le32(next_entry_offset);
break;
fdinfo = (struct file_directory_info *)kstat;
fdinfo->FileNameLength = cpu_to_le32(conv_len);
if (d_info->hide_dot_file && d_info->name[0] == '.')
- fdinfo->ExtFileAttributes |= ATTR_HIDDEN_LE;
+ fdinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
memcpy(fdinfo->FileName, conv_name, conv_len);
fdinfo->NextEntryOffset = cpu_to_le32(next_entry_offset);
break;
dinfo->EaSize =
smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode);
if (dinfo->EaSize)
- dinfo->ExtFileAttributes = ATTR_REPARSE_POINT_LE;
+ dinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE;
dinfo->Reserved = 0;
dinfo->UniqueId = cpu_to_le64(ksmbd_kstat->kstat->ino);
if (d_info->hide_dot_file && d_info->name[0] == '.')
- dinfo->ExtFileAttributes |= ATTR_HIDDEN_LE;
+ dinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
memcpy(dinfo->FileName, conv_name, conv_len);
dinfo->NextEntryOffset = cpu_to_le32(next_entry_offset);
break;
fibdinfo->EaSize =
smb2_get_reparse_tag_special_file(ksmbd_kstat->kstat->mode);
if (fibdinfo->EaSize)
- fibdinfo->ExtFileAttributes = ATTR_REPARSE_POINT_LE;
+ fibdinfo->ExtFileAttributes = FILE_ATTRIBUTE_REPARSE_POINT_LE;
fibdinfo->UniqueId = cpu_to_le64(ksmbd_kstat->kstat->ino);
fibdinfo->ShortNameLength = 0;
fibdinfo->Reserved = 0;
fibdinfo->Reserved2 = cpu_to_le16(0);
if (d_info->hide_dot_file && d_info->name[0] == '.')
- fibdinfo->ExtFileAttributes |= ATTR_HIDDEN_LE;
+ fibdinfo->ExtFileAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
memcpy(fibdinfo->FileName, conv_name, conv_len);
fibdinfo->NextEntryOffset = cpu_to_le32(next_entry_offset);
break;
posix_info->Mode = cpu_to_le32(ksmbd_kstat->kstat->mode);
posix_info->Inode = cpu_to_le64(ksmbd_kstat->kstat->ino);
posix_info->DosAttributes =
- S_ISDIR(ksmbd_kstat->kstat->mode) ? ATTR_DIRECTORY_LE : ATTR_ARCHIVE_LE;
+ S_ISDIR(ksmbd_kstat->kstat->mode) ?
+ FILE_ATTRIBUTE_DIRECTORY_LE : FILE_ATTRIBUTE_ARCHIVE_LE;
if (d_info->hide_dot_file && d_info->name[0] == '.')
- posix_info->DosAttributes |= ATTR_HIDDEN_LE;
+ posix_info->DosAttributes |= FILE_ATTRIBUTE_HIDDEN_LE;
id_to_sid(from_kuid_munged(&init_user_ns, ksmbd_kstat->kstat->uid),
SIDNFS_USER, (struct smb_sid *)&posix_info->SidBuffer[0]);
id_to_sid(from_kgid_munged(&init_user_ns, ksmbd_kstat->kstat->gid),
{
struct ksmbd_conn *conn = work->conn;
struct smb2_query_directory_req *req;
- struct smb2_query_directory_rsp *rsp, *rsp_org;
+ struct smb2_query_directory_rsp *rsp;
struct ksmbd_share_config *share = work->tcon->share_conf;
struct ksmbd_file *dir_fp = NULL;
struct ksmbd_dir_info d_info;
int buffer_sz;
struct smb2_query_dir_private query_dir_private = {NULL, };
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
if (ksmbd_override_fsids(work)) {
rsp->OutputBufferOffset = cpu_to_le16(0);
rsp->OutputBufferLength = cpu_to_le32(0);
rsp->Buffer[0] = 0;
- inc_rfc1001_len(rsp_org, 9);
+ inc_rfc1001_len(work->response_buf, 9);
} else {
((struct file_directory_info *)
((char *)rsp->Buffer + d_info.last_entry_offset))
rsp->StructureSize = cpu_to_le16(9);
rsp->OutputBufferOffset = cpu_to_le16(72);
rsp->OutputBufferLength = cpu_to_le32(d_info.data_count);
- inc_rfc1001_len(rsp_org, 8 + d_info.data_count);
+ inc_rfc1001_len(work->response_buf, 8 + d_info.data_count);
}
kfree(srch_ptr);
* Return: 0 on success, otherwise error
*/
static int buffer_check_err(int reqOutputBufferLength,
- struct smb2_query_info_rsp *rsp, int infoclass_size)
+ struct smb2_query_info_rsp *rsp,
+ void *rsp_org, int infoclass_size)
{
if (reqOutputBufferLength < le32_to_cpu(rsp->OutputBufferLength)) {
if (reqOutputBufferLength < infoclass_size) {
pr_err("Invalid Buffer Size Requested\n");
rsp->hdr.Status = STATUS_INFO_LENGTH_MISMATCH;
- rsp->hdr.smb2_buf_length = cpu_to_be32(sizeof(struct smb2_hdr) - 4);
+ *(__be32 *)rsp_org = cpu_to_be32(sizeof(struct smb2_hdr));
return -EINVAL;
}
ksmbd_debug(SMB, "Buffer Overflow\n");
rsp->hdr.Status = STATUS_BUFFER_OVERFLOW;
- rsp->hdr.smb2_buf_length = cpu_to_be32(sizeof(struct smb2_hdr) - 4 +
+ *(__be32 *)rsp_org = cpu_to_be32(sizeof(struct smb2_hdr) +
reqOutputBufferLength);
rsp->OutputBufferLength = cpu_to_le32(reqOutputBufferLength);
}
return 0;
}
-static void get_standard_info_pipe(struct smb2_query_info_rsp *rsp)
+static void get_standard_info_pipe(struct smb2_query_info_rsp *rsp,
+ void *rsp_org)
{
struct smb2_file_standard_info *sinfo;
sinfo->Directory = 0;
rsp->OutputBufferLength =
cpu_to_le32(sizeof(struct smb2_file_standard_info));
- inc_rfc1001_len(rsp, sizeof(struct smb2_file_standard_info));
+ inc_rfc1001_len(rsp_org, sizeof(struct smb2_file_standard_info));
}
-static void get_internal_info_pipe(struct smb2_query_info_rsp *rsp, u64 num)
+static void get_internal_info_pipe(struct smb2_query_info_rsp *rsp, u64 num,
+ void *rsp_org)
{
struct smb2_file_internal_info *file_info;
file_info->IndexNumber = cpu_to_le64(num | (1ULL << 63));
rsp->OutputBufferLength =
cpu_to_le32(sizeof(struct smb2_file_internal_info));
- inc_rfc1001_len(rsp, sizeof(struct smb2_file_internal_info));
+ inc_rfc1001_len(rsp_org, sizeof(struct smb2_file_internal_info));
}
static int smb2_get_info_file_pipe(struct ksmbd_session *sess,
struct smb2_query_info_req *req,
- struct smb2_query_info_rsp *rsp)
+ struct smb2_query_info_rsp *rsp,
+ void *rsp_org)
{
u64 id;
int rc;
switch (req->FileInfoClass) {
case FILE_STANDARD_INFORMATION:
- get_standard_info_pipe(rsp);
+ get_standard_info_pipe(rsp, rsp_org);
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, FILE_STANDARD_INFORMATION_SIZE);
+ rsp, rsp_org,
+ FILE_STANDARD_INFORMATION_SIZE);
break;
case FILE_INTERNAL_INFORMATION:
- get_internal_info_pipe(rsp, id);
+ get_internal_info_pipe(rsp, id, rsp_org);
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp, FILE_INTERNAL_INFORMATION_SIZE);
+ rsp, rsp_org,
+ FILE_INTERNAL_INFORMATION_SIZE);
break;
default:
ksmbd_debug(SMB, "smb2_info_file_pipe for %u not supported\n",
static int smb2_get_info_file(struct ksmbd_work *work,
struct smb2_query_info_req *req,
- struct smb2_query_info_rsp *rsp, void *rsp_org)
+ struct smb2_query_info_rsp *rsp)
{
struct ksmbd_file *fp;
int fileinfoclass = 0;
if (test_share_config_flag(work->tcon->share_conf,
KSMBD_SHARE_FLAG_PIPE)) {
/* smb2 info file called for pipe */
- return smb2_get_info_file_pipe(work->sess, req, rsp);
+ return smb2_get_info_file_pipe(work->sess, req, rsp,
+ work->response_buf);
}
if (work->next_smb2_rcv_hdr_off) {
switch (fileinfoclass) {
case FILE_ACCESS_INFORMATION:
- get_file_access_info(rsp, fp, rsp_org);
+ get_file_access_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_ACCESS_INFORMATION_SIZE;
break;
case FILE_BASIC_INFORMATION:
- rc = get_file_basic_info(rsp, fp, rsp_org);
+ rc = get_file_basic_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_BASIC_INFORMATION_SIZE;
break;
case FILE_STANDARD_INFORMATION:
- get_file_standard_info(rsp, fp, rsp_org);
+ get_file_standard_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_STANDARD_INFORMATION_SIZE;
break;
case FILE_ALIGNMENT_INFORMATION:
- get_file_alignment_info(rsp, rsp_org);
+ get_file_alignment_info(rsp, work->response_buf);
file_infoclass_size = FILE_ALIGNMENT_INFORMATION_SIZE;
break;
case FILE_ALL_INFORMATION:
- rc = get_file_all_info(work, rsp, fp, rsp_org);
+ rc = get_file_all_info(work, rsp, fp, work->response_buf);
file_infoclass_size = FILE_ALL_INFORMATION_SIZE;
break;
case FILE_ALTERNATE_NAME_INFORMATION:
- get_file_alternate_info(work, rsp, fp, rsp_org);
+ get_file_alternate_info(work, rsp, fp, work->response_buf);
file_infoclass_size = FILE_ALTERNATE_NAME_INFORMATION_SIZE;
break;
case FILE_STREAM_INFORMATION:
- get_file_stream_info(work, rsp, fp, rsp_org);
+ get_file_stream_info(work, rsp, fp, work->response_buf);
file_infoclass_size = FILE_STREAM_INFORMATION_SIZE;
break;
case FILE_INTERNAL_INFORMATION:
- get_file_internal_info(rsp, fp, rsp_org);
+ get_file_internal_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_INTERNAL_INFORMATION_SIZE;
break;
case FILE_NETWORK_OPEN_INFORMATION:
- rc = get_file_network_open_info(rsp, fp, rsp_org);
+ rc = get_file_network_open_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_NETWORK_OPEN_INFORMATION_SIZE;
break;
case FILE_EA_INFORMATION:
- get_file_ea_info(rsp, rsp_org);
+ get_file_ea_info(rsp, work->response_buf);
file_infoclass_size = FILE_EA_INFORMATION_SIZE;
break;
case FILE_FULL_EA_INFORMATION:
- rc = smb2_get_ea(work, fp, req, rsp, rsp_org);
+ rc = smb2_get_ea(work, fp, req, rsp, work->response_buf);
file_infoclass_size = FILE_FULL_EA_INFORMATION_SIZE;
break;
case FILE_POSITION_INFORMATION:
- get_file_position_info(rsp, fp, rsp_org);
+ get_file_position_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_POSITION_INFORMATION_SIZE;
break;
case FILE_MODE_INFORMATION:
- get_file_mode_info(rsp, fp, rsp_org);
+ get_file_mode_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_MODE_INFORMATION_SIZE;
break;
case FILE_COMPRESSION_INFORMATION:
- get_file_compression_info(rsp, fp, rsp_org);
+ get_file_compression_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_COMPRESSION_INFORMATION_SIZE;
break;
case FILE_ATTRIBUTE_TAG_INFORMATION:
- rc = get_file_attribute_tag_info(rsp, fp, rsp_org);
+ rc = get_file_attribute_tag_info(rsp, fp, work->response_buf);
file_infoclass_size = FILE_ATTRIBUTE_TAG_INFORMATION_SIZE;
break;
case SMB_FIND_FILE_POSIX_INFO:
pr_err("client doesn't negotiate with SMB3.1.1 POSIX Extensions\n");
rc = -EOPNOTSUPP;
} else {
- rc = find_file_posix_info(rsp, fp, rsp_org);
+ rc = find_file_posix_info(rsp, fp, work->response_buf);
file_infoclass_size = sizeof(struct smb311_posix_qinfo);
}
break;
}
if (!rc)
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp,
+ rsp, work->response_buf,
file_infoclass_size);
ksmbd_fd_put(work, fp);
return rc;
static int smb2_get_info_filesystem(struct ksmbd_work *work,
struct smb2_query_info_req *req,
- struct smb2_query_info_rsp *rsp, void *rsp_org)
+ struct smb2_query_info_rsp *rsp)
{
struct ksmbd_session *sess = work->sess;
struct ksmbd_conn *conn = sess->conn;
info->DeviceType = cpu_to_le32(stfs.f_type);
info->DeviceCharacteristics = cpu_to_le32(0x00000020);
rsp->OutputBufferLength = cpu_to_le32(8);
- inc_rfc1001_len(rsp_org, 8);
+ inc_rfc1001_len(work->response_buf, 8);
fs_infoclass_size = FS_DEVICE_INFORMATION_SIZE;
break;
}
info->FileSystemNameLen = cpu_to_le32(len);
sz = sizeof(struct filesystem_attribute_info) - 2 + len;
rsp->OutputBufferLength = cpu_to_le32(sz);
- inc_rfc1001_len(rsp_org, sz);
+ inc_rfc1001_len(work->response_buf, sz);
fs_infoclass_size = FS_ATTRIBUTE_INFORMATION_SIZE;
break;
}
{
struct filesystem_vol_info *info;
size_t sz;
+ unsigned int serial_crc = 0;
info = (struct filesystem_vol_info *)(rsp->Buffer);
info->VolumeCreationTime = 0;
+ serial_crc = crc32_le(serial_crc, share->name,
+ strlen(share->name));
+ serial_crc = crc32_le(serial_crc, share->path,
+ strlen(share->path));
+ serial_crc = crc32_le(serial_crc, ksmbd_netbios_name(),
+ strlen(ksmbd_netbios_name()));
/* Taking dummy value of serial number*/
- info->SerialNumber = cpu_to_le32(0xbc3ac512);
+ info->SerialNumber = cpu_to_le32(serial_crc);
len = smbConvertToUTF16((__le16 *)info->VolumeLabel,
share->name, PATH_MAX,
conn->local_nls, 0);
info->Reserved = 0;
sz = sizeof(struct filesystem_vol_info) - 2 + len;
rsp->OutputBufferLength = cpu_to_le32(sz);
- inc_rfc1001_len(rsp_org, sz);
+ inc_rfc1001_len(work->response_buf, sz);
fs_infoclass_size = FS_VOLUME_INFORMATION_SIZE;
break;
}
info->SectorsPerAllocationUnit = cpu_to_le32(1);
info->BytesPerSector = cpu_to_le32(stfs.f_bsize);
rsp->OutputBufferLength = cpu_to_le32(24);
- inc_rfc1001_len(rsp_org, 24);
+ inc_rfc1001_len(work->response_buf, 24);
fs_infoclass_size = FS_SIZE_INFORMATION_SIZE;
break;
}
info->SectorsPerAllocationUnit = cpu_to_le32(1);
info->BytesPerSector = cpu_to_le32(stfs.f_bsize);
rsp->OutputBufferLength = cpu_to_le32(32);
- inc_rfc1001_len(rsp_org, 32);
+ inc_rfc1001_len(work->response_buf, 32);
fs_infoclass_size = FS_FULL_SIZE_INFORMATION_SIZE;
break;
}
info->extended_info.rel_date = 0;
memcpy(info->extended_info.version_string, "1.1.0", strlen("1.1.0"));
rsp->OutputBufferLength = cpu_to_le32(64);
- inc_rfc1001_len(rsp_org, 64);
+ inc_rfc1001_len(work->response_buf, 64);
fs_infoclass_size = FS_OBJECT_ID_INFORMATION_SIZE;
break;
}
info->ByteOffsetForSectorAlignment = 0;
info->ByteOffsetForPartitionAlignment = 0;
rsp->OutputBufferLength = cpu_to_le32(28);
- inc_rfc1001_len(rsp_org, 28);
+ inc_rfc1001_len(work->response_buf, 28);
fs_infoclass_size = FS_SECTOR_SIZE_INFORMATION_SIZE;
break;
}
info->DefaultQuotaLimit = cpu_to_le64(SMB2_NO_FID);
info->Padding = 0;
rsp->OutputBufferLength = cpu_to_le32(48);
- inc_rfc1001_len(rsp_org, 48);
+ inc_rfc1001_len(work->response_buf, 48);
fs_infoclass_size = FS_CONTROL_INFORMATION_SIZE;
break;
}
info->TotalFileNodes = cpu_to_le64(stfs.f_files);
info->FreeFileNodes = cpu_to_le64(stfs.f_ffree);
rsp->OutputBufferLength = cpu_to_le32(56);
- inc_rfc1001_len(rsp_org, 56);
+ inc_rfc1001_len(work->response_buf, 56);
fs_infoclass_size = FS_POSIX_INFORMATION_SIZE;
}
break;
return -EOPNOTSUPP;
}
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
- rsp,
+ rsp, work->response_buf,
fs_infoclass_size);
path_put(&path);
return rc;
static int smb2_get_info_sec(struct ksmbd_work *work,
struct smb2_query_info_req *req,
- struct smb2_query_info_rsp *rsp, void *rsp_org)
+ struct smb2_query_info_rsp *rsp)
{
struct ksmbd_file *fp;
struct user_namespace *user_ns;
secdesclen = sizeof(struct smb_ntsd);
rsp->OutputBufferLength = cpu_to_le32(secdesclen);
- inc_rfc1001_len(rsp_org, secdesclen);
+ inc_rfc1001_len(work->response_buf, secdesclen);
return 0;
}
return rc;
rsp->OutputBufferLength = cpu_to_le32(secdesclen);
- inc_rfc1001_len(rsp_org, secdesclen);
+ inc_rfc1001_len(work->response_buf, secdesclen);
return 0;
}
int smb2_query_info(struct ksmbd_work *work)
{
struct smb2_query_info_req *req;
- struct smb2_query_info_rsp *rsp, *rsp_org;
+ struct smb2_query_info_rsp *rsp;
int rc = 0;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
ksmbd_debug(SMB, "GOT query info request\n");
switch (req->InfoType) {
case SMB2_O_INFO_FILE:
ksmbd_debug(SMB, "GOT SMB2_O_INFO_FILE\n");
- rc = smb2_get_info_file(work, req, rsp, (void *)rsp_org);
+ rc = smb2_get_info_file(work, req, rsp);
break;
case SMB2_O_INFO_FILESYSTEM:
ksmbd_debug(SMB, "GOT SMB2_O_INFO_FILESYSTEM\n");
- rc = smb2_get_info_filesystem(work, req, rsp, (void *)rsp_org);
+ rc = smb2_get_info_filesystem(work, req, rsp);
break;
case SMB2_O_INFO_SECURITY:
ksmbd_debug(SMB, "GOT SMB2_O_INFO_SECURITY\n");
- rc = smb2_get_info_sec(work, req, rsp, (void *)rsp_org);
+ rc = smb2_get_info_sec(work, req, rsp);
break;
default:
ksmbd_debug(SMB, "InfoType %d not supported yet\n",
}
rsp->StructureSize = cpu_to_le16(9);
rsp->OutputBufferOffset = cpu_to_le16(72);
- inc_rfc1001_len(rsp_org, 8);
+ inc_rfc1001_len(work->response_buf, 8);
return 0;
}
static noinline int smb2_close_pipe(struct ksmbd_work *work)
{
u64 id;
- struct smb2_close_req *req = work->request_buf;
- struct smb2_close_rsp *rsp = work->response_buf;
+ struct smb2_close_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_close_rsp *rsp = smb2_get_msg(work->response_buf);
id = le64_to_cpu(req->VolatileFileId);
ksmbd_session_rpc_close(work->sess, id);
rsp->AllocationSize = 0;
rsp->EndOfFile = 0;
rsp->Attributes = 0;
- inc_rfc1001_len(rsp, 60);
+ inc_rfc1001_len(work->response_buf, 60);
return 0;
}
u64 sess_id;
struct smb2_close_req *req;
struct smb2_close_rsp *rsp;
- struct smb2_close_rsp *rsp_org;
struct ksmbd_conn *conn = work->conn;
struct ksmbd_file *fp;
struct inode *inode;
u64 time;
int err = 0;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
if (test_share_config_flag(work->tcon->share_conf,
rsp->hdr.Status = STATUS_FILE_CLOSED;
smb2_set_err_rsp(work);
} else {
- inc_rfc1001_len(rsp_org, 60);
+ inc_rfc1001_len(work->response_buf, 60);
}
return 0;
*/
int smb2_echo(struct ksmbd_work *work)
{
- struct smb2_echo_rsp *rsp = work->response_buf;
+ struct smb2_echo_rsp *rsp = smb2_get_msg(work->response_buf);
rsp->StructureSize = cpu_to_le16(4);
rsp->Reserved = 0;
- inc_rfc1001_len(rsp, 4);
+ inc_rfc1001_len(work->response_buf, 4);
return 0;
}
if (file_info->Attributes) {
if (!S_ISDIR(inode->i_mode) &&
- file_info->Attributes & ATTR_DIRECTORY_LE) {
+ file_info->Attributes & FILE_ATTRIBUTE_DIRECTORY_LE) {
pr_err("can't change a file to a directory\n");
return -EINVAL;
}
- if (!(S_ISDIR(inode->i_mode) && file_info->Attributes == ATTR_NORMAL_LE))
+ if (!(S_ISDIR(inode->i_mode) && file_info->Attributes == FILE_ATTRIBUTE_NORMAL_LE))
fp->f_ci->m_fattr = file_info->Attributes |
- (fp->f_ci->m_fattr & ATTR_DIRECTORY_LE);
+ (fp->f_ci->m_fattr & FILE_ATTRIBUTE_DIRECTORY_LE);
}
if (test_share_config_flag(share, KSMBD_SHARE_FLAG_STORE_DOS_ATTRS) &&
mode = file_info->Mode;
- if ((mode & ~FILE_MODE_INFO_MASK) ||
- (mode & FILE_SYNCHRONOUS_IO_ALERT_LE &&
- mode & FILE_SYNCHRONOUS_IO_NONALERT_LE)) {
+ if ((mode & ~FILE_MODE_INFO_MASK)) {
pr_err("Mode is not valid : 0x%x\n", le32_to_cpu(mode));
return -EINVAL;
}
int smb2_set_info(struct ksmbd_work *work)
{
struct smb2_set_info_req *req;
- struct smb2_set_info_rsp *rsp, *rsp_org;
+ struct smb2_set_info_rsp *rsp;
struct ksmbd_file *fp;
int rc = 0;
unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID;
ksmbd_debug(SMB, "Received set info request\n");
- rsp_org = work->response_buf;
if (work->next_smb2_rcv_hdr_off) {
req = ksmbd_req_buf_next(work);
rsp = ksmbd_resp_buf_next(work);
pid = work->compound_pfid;
}
} else {
- req = work->request_buf;
- rsp = work->response_buf;
+ req = smb2_get_msg(work->request_buf);
+ rsp = smb2_get_msg(work->response_buf);
}
if (!has_file_id(id)) {
goto err_out;
rsp->StructureSize = cpu_to_le16(2);
- inc_rfc1001_len(rsp_org, 2);
+ inc_rfc1001_len(work->response_buf, 2);
ksmbd_fd_put(work, fp);
return 0;
int nbytes = 0, err;
u64 id;
struct ksmbd_rpc_command *rpc_resp;
- struct smb2_read_req *req = work->request_buf;
- struct smb2_read_rsp *rsp = work->response_buf;
+ struct smb2_read_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_read_rsp *rsp = smb2_get_msg(work->response_buf);
id = le64_to_cpu(req->VolatileFileId);
- inc_rfc1001_len(rsp, 16);
+ inc_rfc1001_len(work->response_buf, 16);
rpc_resp = ksmbd_rpc_read(work->sess, id);
if (rpc_resp) {
if (rpc_resp->flags != KSMBD_RPC_OK) {
rpc_resp->payload_sz);
nbytes = rpc_resp->payload_sz;
- work->resp_hdr_sz = get_rfc1002_len(rsp) + 4;
+ work->resp_hdr_sz = get_rfc1002_len(work->response_buf) + 4;
work->aux_payload_sz = nbytes;
kvfree(rpc_resp);
}
rsp->Reserved = 0;
rsp->DataLength = cpu_to_le32(nbytes);
rsp->DataRemaining = 0;
- rsp->Reserved2 = 0;
- inc_rfc1001_len(rsp, nbytes);
+ rsp->Flags = 0;
+ inc_rfc1001_len(work->response_buf, nbytes);
return 0;
out:
{
struct ksmbd_conn *conn = work->conn;
struct smb2_read_req *req;
- struct smb2_read_rsp *rsp, *rsp_org;
+ struct smb2_read_rsp *rsp;
struct ksmbd_file *fp;
loff_t offset;
size_t length, mincount;
ssize_t nbytes = 0, remain_bytes = 0;
int err = 0;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
if (test_share_config_flag(work->tcon->share_conf,
rsp->Reserved = 0;
rsp->DataLength = cpu_to_le32(nbytes);
rsp->DataRemaining = cpu_to_le32(remain_bytes);
- rsp->Reserved2 = 0;
- inc_rfc1001_len(rsp_org, 16);
- work->resp_hdr_sz = get_rfc1002_len(rsp_org) + 4;
+ rsp->Flags = 0;
+ inc_rfc1001_len(work->response_buf, 16);
+ work->resp_hdr_sz = get_rfc1002_len(work->response_buf) + 4;
work->aux_payload_sz = nbytes;
- inc_rfc1001_len(rsp_org, nbytes);
+ inc_rfc1001_len(work->response_buf, nbytes);
ksmbd_fd_put(work, fp);
return 0;
*/
static noinline int smb2_write_pipe(struct ksmbd_work *work)
{
- struct smb2_write_req *req = work->request_buf;
- struct smb2_write_rsp *rsp = work->response_buf;
+ struct smb2_write_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_write_rsp *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_rpc_command *rpc_resp;
u64 id = 0;
int err = 0, ret = 0;
id = le64_to_cpu(req->VolatileFileId);
if (le16_to_cpu(req->DataOffset) ==
- (offsetof(struct smb2_write_req, Buffer) - 4)) {
+ offsetof(struct smb2_write_req, Buffer)) {
data_buf = (char *)&req->Buffer[0];
} else {
- if ((u64)le16_to_cpu(req->DataOffset) + length > get_rfc1002_len(req)) {
+ if ((u64)le16_to_cpu(req->DataOffset) + length >
+ get_rfc1002_len(work->request_buf)) {
pr_err("invalid write data offset %u, smb_len %u\n",
le16_to_cpu(req->DataOffset),
- get_rfc1002_len(req));
+ get_rfc1002_len(work->request_buf));
err = -EINVAL;
goto out;
}
rsp->DataLength = cpu_to_le32(length);
rsp->DataRemaining = 0;
rsp->Reserved2 = 0;
- inc_rfc1001_len(rsp, 16);
+ inc_rfc1001_len(work->response_buf, 16);
return 0;
out:
if (err) {
int smb2_write(struct ksmbd_work *work)
{
struct smb2_write_req *req;
- struct smb2_write_rsp *rsp, *rsp_org;
+ struct smb2_write_rsp *rsp;
struct ksmbd_file *fp = NULL;
loff_t offset;
size_t length;
bool writethrough = false;
int err = 0;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
if (test_share_config_flag(work->tcon->share_conf, KSMBD_SHARE_FLAG_PIPE)) {
if (req->Channel != SMB2_CHANNEL_RDMA_V1 &&
req->Channel != SMB2_CHANNEL_RDMA_V1_INVALIDATE) {
if (le16_to_cpu(req->DataOffset) ==
- (offsetof(struct smb2_write_req, Buffer) - 4)) {
+ offsetof(struct smb2_write_req, Buffer)) {
data_buf = (char *)&req->Buffer[0];
} else {
- if ((u64)le16_to_cpu(req->DataOffset) + length > get_rfc1002_len(req)) {
+ if ((u64)le16_to_cpu(req->DataOffset) + length >
+ get_rfc1002_len(work->request_buf)) {
pr_err("invalid write data offset %u, smb_len %u\n",
le16_to_cpu(req->DataOffset),
- get_rfc1002_len(req));
+ get_rfc1002_len(work->request_buf));
err = -EINVAL;
goto out;
}
rsp->DataLength = cpu_to_le32(nbytes);
rsp->DataRemaining = 0;
rsp->Reserved2 = 0;
- inc_rfc1001_len(rsp_org, 16);
+ inc_rfc1001_len(work->response_buf, 16);
ksmbd_fd_put(work, fp);
return 0;
int smb2_flush(struct ksmbd_work *work)
{
struct smb2_flush_req *req;
- struct smb2_flush_rsp *rsp, *rsp_org;
+ struct smb2_flush_rsp *rsp;
int err;
- rsp_org = work->response_buf;
WORK_BUFFERS(work, req, rsp);
ksmbd_debug(SMB, "SMB2_FLUSH called for fid %llu\n",
rsp->StructureSize = cpu_to_le16(4);
rsp->Reserved = 0;
- inc_rfc1001_len(rsp_org, 4);
+ inc_rfc1001_len(work->response_buf, 4);
return 0;
out:
int smb2_cancel(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_hdr *hdr = work->request_buf;
+ struct smb2_hdr *hdr = smb2_get_msg(work->request_buf);
struct smb2_hdr *chdr;
struct ksmbd_work *cancel_work = NULL;
int canceled = 0;
spin_lock(&conn->request_lock);
list_for_each_entry(cancel_work, command_list,
async_request_entry) {
- chdr = cancel_work->request_buf;
+ chdr = smb2_get_msg(cancel_work->request_buf);
if (cancel_work->async_id !=
le64_to_cpu(hdr->Id.AsyncId))
spin_lock(&conn->request_lock);
list_for_each_entry(cancel_work, command_list, request_entry) {
- chdr = cancel_work->request_buf;
+ chdr = smb2_get_msg(cancel_work->request_buf);
if (chdr->MessageId != hdr->MessageId ||
cancel_work == work)
*/
int smb2_lock(struct ksmbd_work *work)
{
- struct smb2_lock_req *req = work->request_buf;
- struct smb2_lock_rsp *rsp = work->response_buf;
+ struct smb2_lock_req *req = smb2_get_msg(work->request_buf);
+ struct smb2_lock_rsp *rsp = smb2_get_msg(work->response_buf);
struct smb2_lock_element *lock_ele;
struct ksmbd_file *fp = NULL;
struct file_lock *flock = NULL;
ksmbd_debug(SMB, "successful in taking lock\n");
rsp->hdr.Status = STATUS_SUCCESS;
rsp->Reserved = 0;
- inc_rfc1001_len(rsp, 4);
+ inc_rfc1001_len(work->response_buf, 4);
ksmbd_fd_put(work, fp);
return 0;
int ret = 0;
int dialect;
- if (in_buf_len < sizeof(struct validate_negotiate_info_req) +
+ if (in_buf_len < offsetof(struct validate_negotiate_info_req, Dialects) +
le16_to_cpu(neg_req->DialectCount) * sizeof(__le16))
return -EINVAL;
old_fattr = fp->f_ci->m_fattr;
if (sparse->SetSparse)
- fp->f_ci->m_fattr |= ATTR_SPARSE_FILE_LE;
+ fp->f_ci->m_fattr |= FILE_ATTRIBUTE_SPARSE_FILE_LE;
else
- fp->f_ci->m_fattr &= ~ATTR_SPARSE_FILE_LE;
+ fp->f_ci->m_fattr &= ~FILE_ATTRIBUTE_SPARSE_FILE_LE;
if (fp->f_ci->m_fattr != old_fattr &&
test_share_config_flag(work->tcon->share_conf,
int smb2_ioctl(struct ksmbd_work *work)
{
struct smb2_ioctl_req *req;
- struct smb2_ioctl_rsp *rsp, *rsp_org;
+ struct smb2_ioctl_rsp *rsp;
unsigned int cnt_code, nbytes = 0, out_buf_len, in_buf_len;
u64 id = KSMBD_NO_FID;
struct ksmbd_conn *conn = work->conn;
int ret = 0;
- rsp_org = work->response_buf;
if (work->next_smb2_rcv_hdr_off) {
req = ksmbd_req_buf_next(work);
rsp = ksmbd_resp_buf_next(work);
id = work->compound_fid;
}
} else {
- req = work->request_buf;
- rsp = work->response_buf;
+ req = smb2_get_msg(work->request_buf);
+ rsp = smb2_get_msg(work->response_buf);
}
if (!has_file_id(id))
rsp->Reserved = cpu_to_le16(0);
rsp->Flags = cpu_to_le32(0);
rsp->Reserved2 = cpu_to_le32(0);
- inc_rfc1001_len(rsp_org, 48 + nbytes);
+ inc_rfc1001_len(work->response_buf, 48 + nbytes);
return 0;
*/
static void smb20_oplock_break_ack(struct ksmbd_work *work)
{
- struct smb2_oplock_break *req = work->request_buf;
- struct smb2_oplock_break *rsp = work->response_buf;
+ struct smb2_oplock_break *req = smb2_get_msg(work->request_buf);
+ struct smb2_oplock_break *rsp = smb2_get_msg(work->response_buf);
struct ksmbd_file *fp;
struct oplock_info *opinfo = NULL;
__le32 err = 0;
rsp->Reserved2 = 0;
rsp->VolatileFid = cpu_to_le64(volatile_id);
rsp->PersistentFid = cpu_to_le64(persistent_id);
- inc_rfc1001_len(rsp, 24);
+ inc_rfc1001_len(work->response_buf, 24);
return;
err_out:
static void smb21_lease_break_ack(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_lease_ack *req = work->request_buf;
- struct smb2_lease_ack *rsp = work->response_buf;
+ struct smb2_lease_ack *req = smb2_get_msg(work->request_buf);
+ struct smb2_lease_ack *rsp = smb2_get_msg(work->response_buf);
struct oplock_info *opinfo;
__le32 err = 0;
int ret = 0;
memcpy(rsp->LeaseKey, req->LeaseKey, 16);
rsp->LeaseState = lease_state;
rsp->LeaseDuration = 0;
- inc_rfc1001_len(rsp, 36);
+ inc_rfc1001_len(work->response_buf, 36);
return;
err_out:
*/
int smb2_oplock_break(struct ksmbd_work *work)
{
- struct smb2_oplock_break *req = work->request_buf;
- struct smb2_oplock_break *rsp = work->response_buf;
+ struct smb2_oplock_break *req = smb2_get_msg(work->request_buf);
+ struct smb2_oplock_break *rsp = smb2_get_msg(work->response_buf);
switch (le16_to_cpu(req->StructureSize)) {
case OP_BREAK_STRUCT_SIZE_20:
*/
int smb2_notify(struct ksmbd_work *work)
{
- struct smb2_notify_req *req;
- struct smb2_notify_rsp *rsp;
+ struct smb2_change_notify_req *req;
+ struct smb2_change_notify_rsp *rsp;
WORK_BUFFERS(work, req, rsp);
*/
bool smb2_is_sign_req(struct ksmbd_work *work, unsigned int command)
{
- struct smb2_hdr *rcv_hdr2 = work->request_buf;
+ struct smb2_hdr *rcv_hdr2 = smb2_get_msg(work->request_buf);
if ((rcv_hdr2->Flags & SMB2_FLAGS_SIGNED) &&
command != SMB2_NEGOTIATE_HE &&
*/
int smb2_check_sign_req(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr, *hdr_org;
+ struct smb2_hdr *hdr;
char signature_req[SMB2_SIGNATURE_SIZE];
char signature[SMB2_HMACSHA256_SIZE];
struct kvec iov[1];
size_t len;
- hdr_org = hdr = work->request_buf;
+ hdr = smb2_get_msg(work->request_buf);
if (work->next_smb2_rcv_hdr_off)
hdr = ksmbd_req_buf_next(work);
if (!hdr->NextCommand && !work->next_smb2_rcv_hdr_off)
- len = be32_to_cpu(hdr_org->smb2_buf_length);
+ len = get_rfc1002_len(work->request_buf);
else if (hdr->NextCommand)
len = le32_to_cpu(hdr->NextCommand);
else
- len = be32_to_cpu(hdr_org->smb2_buf_length) -
+ len = get_rfc1002_len(work->request_buf) -
work->next_smb2_rcv_hdr_off;
memcpy(signature_req, hdr->Signature, SMB2_SIGNATURE_SIZE);
*/
void smb2_set_sign_rsp(struct ksmbd_work *work)
{
- struct smb2_hdr *hdr, *hdr_org;
+ struct smb2_hdr *hdr;
struct smb2_hdr *req_hdr;
char signature[SMB2_HMACSHA256_SIZE];
struct kvec iov[2];
size_t len;
int n_vec = 1;
- hdr_org = hdr = work->response_buf;
+ hdr = smb2_get_msg(work->response_buf);
if (work->next_smb2_rsp_hdr_off)
hdr = ksmbd_resp_buf_next(work);
req_hdr = ksmbd_req_buf_next(work);
if (!work->next_smb2_rsp_hdr_off) {
- len = get_rfc1002_len(hdr_org);
+ len = get_rfc1002_len(work->response_buf);
if (req_hdr->NextCommand)
len = ALIGN(len, 8);
} else {
- len = get_rfc1002_len(hdr_org) - work->next_smb2_rsp_hdr_off;
+ len = get_rfc1002_len(work->response_buf) -
+ work->next_smb2_rsp_hdr_off;
len = ALIGN(len, 8);
}
{
struct ksmbd_conn *conn = work->conn;
char *signing_key;
- struct smb2_hdr *hdr, *hdr_org;
+ struct smb2_hdr *hdr;
struct channel *chann;
char signature_req[SMB2_SIGNATURE_SIZE];
char signature[SMB2_CMACAES_SIZE];
struct kvec iov[1];
size_t len;
- hdr_org = hdr = work->request_buf;
+ hdr = smb2_get_msg(work->request_buf);
if (work->next_smb2_rcv_hdr_off)
hdr = ksmbd_req_buf_next(work);
if (!hdr->NextCommand && !work->next_smb2_rcv_hdr_off)
- len = be32_to_cpu(hdr_org->smb2_buf_length);
+ len = get_rfc1002_len(work->request_buf);
else if (hdr->NextCommand)
len = le32_to_cpu(hdr->NextCommand);
else
- len = be32_to_cpu(hdr_org->smb2_buf_length) -
+ len = get_rfc1002_len(work->request_buf) -
work->next_smb2_rcv_hdr_off;
if (le16_to_cpu(hdr->Command) == SMB2_SESSION_SETUP_HE) {
void smb3_set_sign_rsp(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_hdr *req_hdr;
- struct smb2_hdr *hdr, *hdr_org;
+ struct smb2_hdr *req_hdr, *hdr;
struct channel *chann;
char signature[SMB2_CMACAES_SIZE];
struct kvec iov[2];
size_t len;
char *signing_key;
- hdr_org = hdr = work->response_buf;
+ hdr = smb2_get_msg(work->response_buf);
if (work->next_smb2_rsp_hdr_off)
hdr = ksmbd_resp_buf_next(work);
req_hdr = ksmbd_req_buf_next(work);
if (!work->next_smb2_rsp_hdr_off) {
- len = get_rfc1002_len(hdr_org);
+ len = get_rfc1002_len(work->response_buf);
if (req_hdr->NextCommand)
len = ALIGN(len, 8);
} else {
- len = get_rfc1002_len(hdr_org) - work->next_smb2_rsp_hdr_off;
+ len = get_rfc1002_len(work->response_buf) -
+ work->next_smb2_rsp_hdr_off;
len = ALIGN(len, 8);
}
if (le16_to_cpu(req->Command) == SMB2_NEGOTIATE_HE &&
conn->preauth_info)
- ksmbd_gen_preauth_integrity_hash(conn, (char *)rsp,
+ ksmbd_gen_preauth_integrity_hash(conn, work->response_buf,
conn->preauth_info->Preauth_HashValue);
if (le16_to_cpu(rsp->Command) == SMB2_SESSION_SETUP_HE && sess) {
if (!hash_value)
return;
}
- ksmbd_gen_preauth_integrity_hash(conn, (char *)rsp,
+ ksmbd_gen_preauth_integrity_hash(conn, work->response_buf,
hash_value);
}
}
-static void fill_transform_hdr(struct smb2_transform_hdr *tr_hdr, char *old_buf,
- __le16 cipher_type)
+static void fill_transform_hdr(void *tr_buf, char *old_buf, __le16 cipher_type)
{
- struct smb2_hdr *hdr = (struct smb2_hdr *)old_buf;
+ struct smb2_transform_hdr *tr_hdr = tr_buf + 4;
+ struct smb2_hdr *hdr = smb2_get_msg(old_buf);
unsigned int orig_len = get_rfc1002_len(old_buf);
- memset(tr_hdr, 0, sizeof(struct smb2_transform_hdr));
+ memset(tr_buf, 0, sizeof(struct smb2_transform_hdr) + 4);
tr_hdr->ProtocolId = SMB2_TRANSFORM_PROTO_NUM;
tr_hdr->OriginalMessageSize = cpu_to_le32(orig_len);
- tr_hdr->Flags = cpu_to_le16(0x01);
+ tr_hdr->Flags = cpu_to_le16(TRANSFORM_FLAG_ENCRYPTED);
if (cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
cipher_type == SMB2_ENCRYPTION_AES256_GCM)
get_random_bytes(&tr_hdr->Nonce, SMB3_AES_GCM_NONCE);
else
get_random_bytes(&tr_hdr->Nonce, SMB3_AES_CCM_NONCE);
memcpy(&tr_hdr->SessionId, &hdr->SessionId, 8);
- inc_rfc1001_len(tr_hdr, sizeof(struct smb2_transform_hdr) - 4);
- inc_rfc1001_len(tr_hdr, orig_len);
+ inc_rfc1001_len(tr_buf, sizeof(struct smb2_transform_hdr));
+ inc_rfc1001_len(tr_buf, orig_len);
}
int smb3_encrypt_resp(struct ksmbd_work *work)
{
char *buf = work->response_buf;
- struct smb2_transform_hdr *tr_hdr;
struct kvec iov[3];
int rc = -ENOMEM;
int buf_size = 0, rq_nvec = 2 + (work->aux_payload_sz ? 1 : 0);
if (ARRAY_SIZE(iov) < rq_nvec)
return -ENOMEM;
- tr_hdr = kzalloc(sizeof(struct smb2_transform_hdr), GFP_KERNEL);
- if (!tr_hdr)
+ work->tr_buf = kzalloc(sizeof(struct smb2_transform_hdr) + 4, GFP_KERNEL);
+ if (!work->tr_buf)
return rc;
/* fill transform header */
- fill_transform_hdr(tr_hdr, buf, work->conn->cipher_type);
+ fill_transform_hdr(work->tr_buf, buf, work->conn->cipher_type);
- iov[0].iov_base = tr_hdr;
- iov[0].iov_len = sizeof(struct smb2_transform_hdr);
+ iov[0].iov_base = work->tr_buf;
+ iov[0].iov_len = sizeof(struct smb2_transform_hdr) + 4;
buf_size += iov[0].iov_len - 4;
iov[1].iov_base = buf + 4;
return rc;
memmove(buf, iov[1].iov_base, iov[1].iov_len);
- tr_hdr->smb2_buf_length = cpu_to_be32(buf_size);
- work->tr_buf = tr_hdr;
+ *(__be32 *)work->tr_buf = cpu_to_be32(buf_size);
return rc;
}
bool smb3_is_transform_hdr(void *buf)
{
- struct smb2_transform_hdr *trhdr = buf;
+ struct smb2_transform_hdr *trhdr = smb2_get_msg(buf);
return trhdr->ProtocolId == SMB2_TRANSFORM_PROTO_NUM;
}
struct ksmbd_conn *conn = work->conn;
struct ksmbd_session *sess;
char *buf = work->request_buf;
- struct smb2_hdr *hdr;
unsigned int pdu_length = get_rfc1002_len(buf);
struct kvec iov[2];
- int buf_data_size = pdu_length + 4 -
- sizeof(struct smb2_transform_hdr);
- struct smb2_transform_hdr *tr_hdr = (struct smb2_transform_hdr *)buf;
+ int buf_data_size = pdu_length - sizeof(struct smb2_transform_hdr);
+ struct smb2_transform_hdr *tr_hdr = smb2_get_msg(buf);
int rc = 0;
if (buf_data_size < sizeof(struct smb2_hdr)) {
}
iov[0].iov_base = buf;
- iov[0].iov_len = sizeof(struct smb2_transform_hdr);
- iov[1].iov_base = buf + sizeof(struct smb2_transform_hdr);
+ iov[0].iov_len = sizeof(struct smb2_transform_hdr) + 4;
+ iov[1].iov_base = buf + sizeof(struct smb2_transform_hdr) + 4;
iov[1].iov_len = buf_data_size;
rc = ksmbd_crypt_message(conn, iov, 2, 0);
if (rc)
return rc;
memmove(buf + 4, iov[1].iov_base, buf_data_size);
- hdr = (struct smb2_hdr *)buf;
- hdr->smb2_buf_length = cpu_to_be32(buf_data_size);
+ *(__be32 *)buf = cpu_to_be32(buf_data_size);
return rc;
}
bool smb3_11_final_sess_setup_resp(struct ksmbd_work *work)
{
struct ksmbd_conn *conn = work->conn;
- struct smb2_hdr *rsp = work->response_buf;
+ struct smb2_hdr *rsp = smb2_get_msg(work->response_buf);
if (conn->dialect < SMB30_PROT_ID)
return false;
#include "ntlmssp.h"
#include "smbacl.h"
-/*
- * Note that, due to trying to use names similar to the protocol specifications,
- * there are many mixed case field names in the structures below. Although
- * this does not match typical Linux kernel style, it is necessary to be
- * able to match against the protocol specfication.
- *
- * SMB2 commands
- * Some commands have minimal (wct=0,bcc=0), or uninteresting, responses
- * (ie no useful data other than the SMB error code itself) and are marked such.
- * Knowing this helps avoid response buffer allocations and copy in some cases.
- */
-
-/* List of commands in host endian */
-#define SMB2_NEGOTIATE_HE 0x0000
-#define SMB2_SESSION_SETUP_HE 0x0001
-#define SMB2_LOGOFF_HE 0x0002 /* trivial request/resp */
-#define SMB2_TREE_CONNECT_HE 0x0003
-#define SMB2_TREE_DISCONNECT_HE 0x0004 /* trivial req/resp */
-#define SMB2_CREATE_HE 0x0005
-#define SMB2_CLOSE_HE 0x0006
-#define SMB2_FLUSH_HE 0x0007 /* trivial resp */
-#define SMB2_READ_HE 0x0008
-#define SMB2_WRITE_HE 0x0009
-#define SMB2_LOCK_HE 0x000A
-#define SMB2_IOCTL_HE 0x000B
-#define SMB2_CANCEL_HE 0x000C
-#define SMB2_ECHO_HE 0x000D
-#define SMB2_QUERY_DIRECTORY_HE 0x000E
-#define SMB2_CHANGE_NOTIFY_HE 0x000F
-#define SMB2_QUERY_INFO_HE 0x0010
-#define SMB2_SET_INFO_HE 0x0011
-#define SMB2_OPLOCK_BREAK_HE 0x0012
-
-/* The same list in little endian */
-#define SMB2_NEGOTIATE cpu_to_le16(SMB2_NEGOTIATE_HE)
-#define SMB2_SESSION_SETUP cpu_to_le16(SMB2_SESSION_SETUP_HE)
-#define SMB2_LOGOFF cpu_to_le16(SMB2_LOGOFF_HE)
-#define SMB2_TREE_CONNECT cpu_to_le16(SMB2_TREE_CONNECT_HE)
-#define SMB2_TREE_DISCONNECT cpu_to_le16(SMB2_TREE_DISCONNECT_HE)
-#define SMB2_CREATE cpu_to_le16(SMB2_CREATE_HE)
-#define SMB2_CLOSE cpu_to_le16(SMB2_CLOSE_HE)
-#define SMB2_FLUSH cpu_to_le16(SMB2_FLUSH_HE)
-#define SMB2_READ cpu_to_le16(SMB2_READ_HE)
-#define SMB2_WRITE cpu_to_le16(SMB2_WRITE_HE)
-#define SMB2_LOCK cpu_to_le16(SMB2_LOCK_HE)
-#define SMB2_IOCTL cpu_to_le16(SMB2_IOCTL_HE)
-#define SMB2_CANCEL cpu_to_le16(SMB2_CANCEL_HE)
-#define SMB2_ECHO cpu_to_le16(SMB2_ECHO_HE)
-#define SMB2_QUERY_DIRECTORY cpu_to_le16(SMB2_QUERY_DIRECTORY_HE)
-#define SMB2_CHANGE_NOTIFY cpu_to_le16(SMB2_CHANGE_NOTIFY_HE)
-#define SMB2_QUERY_INFO cpu_to_le16(SMB2_QUERY_INFO_HE)
-#define SMB2_SET_INFO cpu_to_le16(SMB2_SET_INFO_HE)
-#define SMB2_OPLOCK_BREAK cpu_to_le16(SMB2_OPLOCK_BREAK_HE)
-
/*Create Action Flags*/
#define FILE_SUPERSEDED 0x00000000
#define FILE_OPENED 0x00000001
/* SMB2 Max Credits */
#define SMB2_MAX_CREDITS 8192
-#define SMB2_CLIENT_GUID_SIZE 16
-#define SMB2_CREATE_GUID_SIZE 16
-
/* Maximum buffer size value we can send with 1 credit */
#define SMB2_MAX_BUFFER_SIZE 65536
/* BB FIXME - analyze following length BB */
#define MAX_SMB2_HDR_SIZE 0x78 /* 4 len + 64 hdr + (2*24 wct) + 2 bct + 2 pad */
-#define SMB2_PROTO_NUMBER cpu_to_le32(0x424d53fe) /* 'B''M''S' */
-#define SMB2_TRANSFORM_PROTO_NUM cpu_to_le32(0x424d53fd)
-
#define SMB21_DEFAULT_IOSIZE (1024 * 1024)
#define SMB3_DEFAULT_IOSIZE (4 * 1024 * 1024)
#define SMB3_DEFAULT_TRANS_SIZE (1024 * 1024)
#define SMB3_MAX_IOSIZE (8 * 1024 * 1024)
/*
- * SMB2 Header Definition
- *
- * "MBZ" : Must be Zero
- * "BB" : BugBug, Something to check/review/analyze later
- * "PDU" : "Protocol Data Unit" (ie a network "frame")
- *
- */
-
-#define __SMB2_HEADER_STRUCTURE_SIZE 64
-#define SMB2_HEADER_STRUCTURE_SIZE \
- cpu_to_le16(__SMB2_HEADER_STRUCTURE_SIZE)
-
-struct smb2_hdr {
- __be32 smb2_buf_length; /* big endian on wire */
- /*
- * length is only two or three bytes - with
- * one or two byte type preceding it that MBZ
- */
- __le32 ProtocolId; /* 0xFE 'S' 'M' 'B' */
- __le16 StructureSize; /* 64 */
- __le16 CreditCharge; /* MBZ */
- __le32 Status; /* Error from server */
- __le16 Command;
- __le16 CreditRequest; /* CreditResponse */
- __le32 Flags;
- __le32 NextCommand;
- __le64 MessageId;
- union {
- struct {
- __le32 ProcessId;
- __le32 TreeId;
- } __packed SyncId;
- __le64 AsyncId;
- } __packed Id;
- __le64 SessionId;
- __u8 Signature[16];
-} __packed;
-
-struct smb2_pdu {
- struct smb2_hdr hdr;
- __le16 StructureSize2; /* size of wct area (varies, request specific) */
-} __packed;
-
-#define SMB3_AES_CCM_NONCE 11
-#define SMB3_AES_GCM_NONCE 12
-
-struct smb2_transform_hdr {
- __be32 smb2_buf_length; /* big endian on wire */
- /*
- * length is only two or three bytes - with
- * one or two byte type preceding it that MBZ
- */
- __le32 ProtocolId; /* 0xFD 'S' 'M' 'B' */
- __u8 Signature[16];
- __u8 Nonce[16];
- __le32 OriginalMessageSize;
- __u16 Reserved1;
- __le16 Flags; /* EncryptionAlgorithm */
- __le64 SessionId;
-} __packed;
-
-/*
- * SMB2 flag definitions
- */
-#define SMB2_FLAGS_SERVER_TO_REDIR cpu_to_le32(0x00000001)
-#define SMB2_FLAGS_ASYNC_COMMAND cpu_to_le32(0x00000002)
-#define SMB2_FLAGS_RELATED_OPERATIONS cpu_to_le32(0x00000004)
-#define SMB2_FLAGS_SIGNED cpu_to_le32(0x00000008)
-#define SMB2_FLAGS_DFS_OPERATIONS cpu_to_le32(0x10000000)
-#define SMB2_FLAGS_REPLAY_OPERATIONS cpu_to_le32(0x20000000)
-
-/*
* Definitions for SMB2 Protocol Data Units (network frames)
*
* See MS-SMB2.PDF specification for protocol details.
__u8 ErrorData[1]; /* variable length */
} __packed;
-struct smb2_negotiate_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 36 */
- __le16 DialectCount;
- __le16 SecurityMode;
- __le16 Reserved; /* MBZ */
- __le32 Capabilities;
- __u8 ClientGUID[SMB2_CLIENT_GUID_SIZE];
- /* In SMB3.02 and earlier next three were MBZ le64 ClientStartTime */
- __le32 NegotiateContextOffset; /* SMB3.1.1 only. MBZ earlier */
- __le16 NegotiateContextCount; /* SMB3.1.1 only. MBZ earlier */
- __le16 Reserved2;
- __le16 Dialects[1]; /* One dialect (vers=) at a time for now */
-} __packed;
-
-/* SecurityMode flags */
-#define SMB2_NEGOTIATE_SIGNING_ENABLED_LE cpu_to_le16(0x0001)
-#define SMB2_NEGOTIATE_SIGNING_REQUIRED 0x0002
-#define SMB2_NEGOTIATE_SIGNING_REQUIRED_LE cpu_to_le16(0x0002)
-/* Capabilities flags */
-#define SMB2_GLOBAL_CAP_DFS 0x00000001
-#define SMB2_GLOBAL_CAP_LEASING 0x00000002 /* Resp only New to SMB2.1 */
-#define SMB2_GLOBAL_CAP_LARGE_MTU 0X00000004 /* Resp only New to SMB2.1 */
-#define SMB2_GLOBAL_CAP_MULTI_CHANNEL 0x00000008 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_PERSISTENT_HANDLES 0x00000010 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_DIRECTORY_LEASING 0x00000020 /* New to SMB3 */
-#define SMB2_GLOBAL_CAP_ENCRYPTION 0x00000040 /* New to SMB3 */
-/* Internal types */
-#define SMB2_NT_FIND 0x00100000
-#define SMB2_LARGE_FILES 0x00200000
-
-#define SMB311_SALT_SIZE 32
-/* Hash Algorithm Types */
-#define SMB2_PREAUTH_INTEGRITY_SHA512 cpu_to_le16(0x0001)
-
-#define PREAUTH_HASHVALUE_SIZE 64
-
struct preauth_integrity_info {
/* PreAuth integrity Hash ID */
__le16 Preauth_HashId;
/* PreAuth integrity Hash Value */
- __u8 Preauth_HashValue[PREAUTH_HASHVALUE_SIZE];
+ __u8 Preauth_HashValue[SMB2_PREAUTH_HASH_SIZE];
};
-/* offset is sizeof smb2_negotiate_rsp - 4 but rounded up to 8 bytes. */
+/* offset is sizeof smb2_negotiate_rsp but rounded up to 8 bytes. */
#ifdef CONFIG_SMB_SERVER_KERBEROS5
-/* sizeof(struct smb2_negotiate_rsp) - 4 =
+/* sizeof(struct smb2_negotiate_rsp) =
* header(64) + response(64) + GSS_LENGTH(96) + GSS_PADDING(0)
*/
#define OFFSET_OF_NEG_CONTEXT 0xe0
#else
-/* sizeof(struct smb2_negotiate_rsp) - 4 =
+/* sizeof(struct smb2_negotiate_rsp) =
* header(64) + response(64) + GSS_LENGTH(74) + GSS_PADDING(6)
*/
#define OFFSET_OF_NEG_CONTEXT 0xd0
#endif
-#define SMB2_PREAUTH_INTEGRITY_CAPABILITIES cpu_to_le16(1)
-#define SMB2_ENCRYPTION_CAPABILITIES cpu_to_le16(2)
-#define SMB2_COMPRESSION_CAPABILITIES cpu_to_le16(3)
-#define SMB2_NETNAME_NEGOTIATE_CONTEXT_ID cpu_to_le16(5)
-#define SMB2_SIGNING_CAPABILITIES cpu_to_le16(8)
-#define SMB2_POSIX_EXTENSIONS_AVAILABLE cpu_to_le16(0x100)
-
-struct smb2_neg_context {
- __le16 ContextType;
- __le16 DataLength;
- __le32 Reserved;
- /* Followed by array of data */
-} __packed;
-
-struct smb2_preauth_neg_context {
- __le16 ContextType; /* 1 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 HashAlgorithmCount; /* 1 */
- __le16 SaltLength;
- __le16 HashAlgorithms; /* HashAlgorithms[0] since only one defined */
- __u8 Salt[SMB311_SALT_SIZE];
-} __packed;
-
-/* Encryption Algorithms Ciphers */
-#define SMB2_ENCRYPTION_AES128_CCM cpu_to_le16(0x0001)
-#define SMB2_ENCRYPTION_AES128_GCM cpu_to_le16(0x0002)
-#define SMB2_ENCRYPTION_AES256_CCM cpu_to_le16(0x0003)
-#define SMB2_ENCRYPTION_AES256_GCM cpu_to_le16(0x0004)
-
-struct smb2_encryption_neg_context {
- __le16 ContextType; /* 2 */
- __le16 DataLength;
- __le32 Reserved;
- /* CipherCount usally 2, but can be 3 when AES256-GCM enabled */
- __le16 CipherCount; /* AES-128-GCM and AES-128-CCM by default */
- __le16 Ciphers[];
-} __packed;
-
-#define SMB3_COMPRESS_NONE cpu_to_le16(0x0000)
-#define SMB3_COMPRESS_LZNT1 cpu_to_le16(0x0001)
-#define SMB3_COMPRESS_LZ77 cpu_to_le16(0x0002)
-#define SMB3_COMPRESS_LZ77_HUFF cpu_to_le16(0x0003)
-
-struct smb2_compression_ctx {
- __le16 ContextType; /* 3 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 CompressionAlgorithmCount;
- __u16 Padding;
- __le32 Reserved1;
- __le16 CompressionAlgorithms[];
-} __packed;
-
-#define POSIX_CTXT_DATA_LEN 16
-struct smb2_posix_neg_context {
- __le16 ContextType; /* 0x100 */
- __le16 DataLength;
- __le32 Reserved;
- __u8 Name[16]; /* POSIX ctxt GUID 93AD25509CB411E7B42383DE968BCD7C */
-} __packed;
-
-struct smb2_netname_neg_context {
- __le16 ContextType; /* 0x100 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 NetName[]; /* hostname of target converted to UCS-2 */
-} __packed;
-
-/* Signing algorithms */
-#define SIGNING_ALG_HMAC_SHA256 cpu_to_le16(0)
-#define SIGNING_ALG_AES_CMAC cpu_to_le16(1)
-#define SIGNING_ALG_AES_GMAC cpu_to_le16(2)
-
-struct smb2_signing_capabilities {
- __le16 ContextType; /* 8 */
- __le16 DataLength;
- __le32 Reserved;
- __le16 SigningAlgorithmCount;
- __le16 SigningAlgorithms[];
-} __packed;
-
-struct smb2_negotiate_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 65 */
- __le16 SecurityMode;
- __le16 DialectRevision;
- __le16 NegotiateContextCount; /* Prior to SMB3.1.1 was Reserved & MBZ */
- __u8 ServerGUID[16];
- __le32 Capabilities;
- __le32 MaxTransactSize;
- __le32 MaxReadSize;
- __le32 MaxWriteSize;
- __le64 SystemTime; /* MBZ */
- __le64 ServerStartTime;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __le32 NegotiateContextOffset; /* Pre:SMB3.1.1 was reserved/ignored */
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-/* Flags */
-#define SMB2_SESSION_REQ_FLAG_BINDING 0x01
-#define SMB2_SESSION_REQ_FLAG_ENCRYPT_DATA 0x04
-
#define SMB2_SESSION_EXPIRED (0)
#define SMB2_SESSION_IN_PROGRESS BIT(0)
#define SMB2_SESSION_VALID BIT(1)
-/* Flags */
-#define SMB2_SESSION_REQ_FLAG_BINDING 0x01
-#define SMB2_SESSION_REQ_FLAG_ENCRYPT_DATA 0x04
-
-struct smb2_sess_setup_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 25 */
- __u8 Flags;
- __u8 SecurityMode;
- __le32 Capabilities;
- __le32 Channel;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __le64 PreviousSessionId;
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-/* Flags/Reserved for SMB3.1.1 */
-#define SMB2_SHAREFLAG_CLUSTER_RECONNECT 0x0001
-
-/* Currently defined SessionFlags */
-#define SMB2_SESSION_FLAG_IS_GUEST_LE cpu_to_le16(0x0001)
-#define SMB2_SESSION_FLAG_IS_NULL_LE cpu_to_le16(0x0002)
-#define SMB2_SESSION_FLAG_ENCRYPT_DATA_LE cpu_to_le16(0x0004)
-struct smb2_sess_setup_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 SessionFlags;
- __le16 SecurityBufferOffset;
- __le16 SecurityBufferLength;
- __u8 Buffer[1]; /* variable length GSS security buffer */
-} __packed;
-
-struct smb2_logoff_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-struct smb2_logoff_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-struct smb2_tree_connect_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 Reserved; /* Flags in SMB3.1.1 */
- __le16 PathOffset;
- __le16 PathLength;
- __u8 Buffer[1]; /* variable length */
-} __packed;
-
-struct smb2_tree_connect_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 16 */
- __u8 ShareType; /* see below */
- __u8 Reserved;
- __le32 ShareFlags; /* see below */
- __le32 Capabilities; /* see below */
- __le32 MaximalAccess;
-} __packed;
-
-/* Possible ShareType values */
-#define SMB2_SHARE_TYPE_DISK 0x01
-#define SMB2_SHARE_TYPE_PIPE 0x02
-#define SMB2_SHARE_TYPE_PRINT 0x03
-
-/*
- * Possible ShareFlags - exactly one and only one of the first 4 caching flags
- * must be set (any of the remaining, SHI1005, flags may be set individually
- * or in combination.
- */
-#define SMB2_SHAREFLAG_MANUAL_CACHING 0x00000000
-#define SMB2_SHAREFLAG_AUTO_CACHING 0x00000010
-#define SMB2_SHAREFLAG_VDO_CACHING 0x00000020
-#define SMB2_SHAREFLAG_NO_CACHING 0x00000030
-#define SHI1005_FLAGS_DFS 0x00000001
-#define SHI1005_FLAGS_DFS_ROOT 0x00000002
-#define SHI1005_FLAGS_RESTRICT_EXCLUSIVE_OPENS 0x00000100
-#define SHI1005_FLAGS_FORCE_SHARED_DELETE 0x00000200
-#define SHI1005_FLAGS_ALLOW_NAMESPACE_CACHING 0x00000400
-#define SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM 0x00000800
-#define SHI1005_FLAGS_FORCE_LEVELII_OPLOCK 0x00001000
-#define SHI1005_FLAGS_ENABLE_HASH 0x00002000
-
-/* Possible share capabilities */
-#define SMB2_SHARE_CAP_DFS cpu_to_le32(0x00000008)
-
-struct smb2_tree_disconnect_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-struct smb2_tree_disconnect_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 4 */
- __le16 Reserved;
-} __packed;
-
-#define ATTR_READONLY_LE cpu_to_le32(ATTR_READONLY)
-#define ATTR_HIDDEN_LE cpu_to_le32(ATTR_HIDDEN)
-#define ATTR_SYSTEM_LE cpu_to_le32(ATTR_SYSTEM)
-#define ATTR_DIRECTORY_LE cpu_to_le32(ATTR_DIRECTORY)
-#define ATTR_ARCHIVE_LE cpu_to_le32(ATTR_ARCHIVE)
-#define ATTR_NORMAL_LE cpu_to_le32(ATTR_NORMAL)
-#define ATTR_TEMPORARY_LE cpu_to_le32(ATTR_TEMPORARY)
-#define ATTR_SPARSE_FILE_LE cpu_to_le32(ATTR_SPARSE)
-#define ATTR_REPARSE_POINT_LE cpu_to_le32(ATTR_REPARSE)
-#define ATTR_COMPRESSED_LE cpu_to_le32(ATTR_COMPRESSED)
-#define ATTR_OFFLINE_LE cpu_to_le32(ATTR_OFFLINE)
-#define ATTR_NOT_CONTENT_INDEXED_LE cpu_to_le32(ATTR_NOT_CONTENT_INDEXED)
-#define ATTR_ENCRYPTED_LE cpu_to_le32(ATTR_ENCRYPTED)
-#define ATTR_INTEGRITY_STREAML_LE cpu_to_le32(0x00008000)
-#define ATTR_NO_SCRUB_DATA_LE cpu_to_le32(0x00020000)
-#define ATTR_MASK_LE cpu_to_le32(0x00007FB7)
-
-/* Oplock levels */
-#define SMB2_OPLOCK_LEVEL_NONE 0x00
-#define SMB2_OPLOCK_LEVEL_II 0x01
-#define SMB2_OPLOCK_LEVEL_EXCLUSIVE 0x08
-#define SMB2_OPLOCK_LEVEL_BATCH 0x09
-#define SMB2_OPLOCK_LEVEL_LEASE 0xFF
-/* Non-spec internal type */
-#define SMB2_OPLOCK_LEVEL_NOCHANGE 0x99
-
-/* Desired Access Flags */
-#define FILE_READ_DATA_LE cpu_to_le32(0x00000001)
-#define FILE_LIST_DIRECTORY_LE cpu_to_le32(0x00000001)
-#define FILE_WRITE_DATA_LE cpu_to_le32(0x00000002)
-#define FILE_ADD_FILE_LE cpu_to_le32(0x00000002)
-#define FILE_APPEND_DATA_LE cpu_to_le32(0x00000004)
-#define FILE_ADD_SUBDIRECTORY_LE cpu_to_le32(0x00000004)
-#define FILE_READ_EA_LE cpu_to_le32(0x00000008)
-#define FILE_WRITE_EA_LE cpu_to_le32(0x00000010)
-#define FILE_EXECUTE_LE cpu_to_le32(0x00000020)
-#define FILE_TRAVERSE_LE cpu_to_le32(0x00000020)
-#define FILE_DELETE_CHILD_LE cpu_to_le32(0x00000040)
-#define FILE_READ_ATTRIBUTES_LE cpu_to_le32(0x00000080)
-#define FILE_WRITE_ATTRIBUTES_LE cpu_to_le32(0x00000100)
-#define FILE_DELETE_LE cpu_to_le32(0x00010000)
-#define FILE_READ_CONTROL_LE cpu_to_le32(0x00020000)
-#define FILE_WRITE_DAC_LE cpu_to_le32(0x00040000)
-#define FILE_WRITE_OWNER_LE cpu_to_le32(0x00080000)
-#define FILE_SYNCHRONIZE_LE cpu_to_le32(0x00100000)
-#define FILE_ACCESS_SYSTEM_SECURITY_LE cpu_to_le32(0x01000000)
-#define FILE_MAXIMAL_ACCESS_LE cpu_to_le32(0x02000000)
-#define FILE_GENERIC_ALL_LE cpu_to_le32(0x10000000)
-#define FILE_GENERIC_EXECUTE_LE cpu_to_le32(0x20000000)
-#define FILE_GENERIC_WRITE_LE cpu_to_le32(0x40000000)
-#define FILE_GENERIC_READ_LE cpu_to_le32(0x80000000)
-#define DESIRED_ACCESS_MASK cpu_to_le32(0xF21F01FF)
-
-/* ShareAccess Flags */
-#define FILE_SHARE_READ_LE cpu_to_le32(0x00000001)
-#define FILE_SHARE_WRITE_LE cpu_to_le32(0x00000002)
-#define FILE_SHARE_DELETE_LE cpu_to_le32(0x00000004)
-#define FILE_SHARE_ALL_LE cpu_to_le32(0x00000007)
-
-/* CreateDisposition Flags */
-#define FILE_SUPERSEDE_LE cpu_to_le32(0x00000000)
-#define FILE_OPEN_LE cpu_to_le32(0x00000001)
-#define FILE_CREATE_LE cpu_to_le32(0x00000002)
-#define FILE_OPEN_IF_LE cpu_to_le32(0x00000003)
-#define FILE_OVERWRITE_LE cpu_to_le32(0x00000004)
-#define FILE_OVERWRITE_IF_LE cpu_to_le32(0x00000005)
-#define FILE_CREATE_MASK_LE cpu_to_le32(0x00000007)
-
-#define FILE_READ_DESIRED_ACCESS_LE (FILE_READ_DATA_LE | \
- FILE_READ_EA_LE | \
- FILE_GENERIC_READ_LE)
-#define FILE_WRITE_DESIRE_ACCESS_LE (FILE_WRITE_DATA_LE | \
- FILE_APPEND_DATA_LE | \
- FILE_WRITE_EA_LE | \
- FILE_WRITE_ATTRIBUTES_LE | \
- FILE_GENERIC_WRITE_LE)
-
-/* Impersonation Levels */
-#define IL_ANONYMOUS_LE cpu_to_le32(0x00000000)
-#define IL_IDENTIFICATION_LE cpu_to_le32(0x00000001)
-#define IL_IMPERSONATION_LE cpu_to_le32(0x00000002)
-#define IL_DELEGATE_LE cpu_to_le32(0x00000003)
-
-/* Create Context Values */
-#define SMB2_CREATE_EA_BUFFER "ExtA" /* extended attributes */
-#define SMB2_CREATE_SD_BUFFER "SecD" /* security descriptor */
-#define SMB2_CREATE_DURABLE_HANDLE_REQUEST "DHnQ"
-#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT "DHnC"
-#define SMB2_CREATE_ALLOCATION_SIZE "AlSi"
-#define SMB2_CREATE_QUERY_MAXIMAL_ACCESS_REQUEST "MxAc"
-#define SMB2_CREATE_TIMEWARP_REQUEST "TWrp"
-#define SMB2_CREATE_QUERY_ON_DISK_ID "QFid"
-#define SMB2_CREATE_REQUEST_LEASE "RqLs"
-#define SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 "DH2Q"
-#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT_V2 "DH2C"
-#define SMB2_CREATE_APP_INSTANCE_ID "\x45\xBC\xA6\x6A\xEF\xA7\xF7\x4A\x90\x08\xFA\x46\x2E\x14\x4D\x74"
- #define SMB2_CREATE_APP_INSTANCE_VERSION "\xB9\x82\xD0\xB7\x3B\x56\x07\x4F\xA0\x7B\x52\x4A\x81\x16\xA0\x10"
-#define SVHDX_OPEN_DEVICE_CONTEXT 0x83CE6F1AD851E0986E34401CC9BCFCE9
-#define SMB2_CREATE_TAG_POSIX "\x93\xAD\x25\x50\x9C\xB4\x11\xE7\xB4\x23\x83\xDE\x96\x8B\xCD\x7C"
-
-struct smb2_create_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 57 */
- __u8 SecurityFlags;
- __u8 RequestedOplockLevel;
- __le32 ImpersonationLevel;
- __le64 SmbCreateFlags;
- __le64 Reserved;
- __le32 DesiredAccess;
- __le32 FileAttributes;
- __le32 ShareAccess;
- __le32 CreateDisposition;
- __le32 CreateOptions;
- __le16 NameOffset;
- __le16 NameLength;
- __le32 CreateContextsOffset;
- __le32 CreateContextsLength;
- __u8 Buffer[0];
-} __packed;
-
-struct smb2_create_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 89 */
- __u8 OplockLevel;
- __u8 Reserved;
- __le32 CreateAction;
- __le64 CreationTime;
- __le64 LastAccessTime;
- __le64 LastWriteTime;
- __le64 ChangeTime;
- __le64 AllocationSize;
- __le64 EndofFile;
- __le32 FileAttributes;
- __le32 Reserved2;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
- __le32 CreateContextsOffset;
- __le32 CreateContextsLength;
- __u8 Buffer[1];
-} __packed;
-
-struct create_context {
- __le32 Next;
- __le16 NameOffset;
- __le16 NameLength;
- __le16 Reserved;
- __le16 DataOffset;
- __le32 DataLength;
- __u8 Buffer[0];
-} __packed;
-
struct create_durable_req_v2 {
struct create_context ccontext;
__u8 Name[8];
#define SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE cpu_to_le32(0x02)
+#define SMB2_LEASE_KEY_SIZE 16
+
struct lease_context {
- __le64 LeaseKeyLow;
- __le64 LeaseKeyHigh;
+ __u8 LeaseKey[SMB2_LEASE_KEY_SIZE];
__le32 LeaseState;
__le32 LeaseFlags;
__le64 LeaseDuration;
} __packed;
struct lease_context_v2 {
- __le64 LeaseKeyLow;
- __le64 LeaseKeyHigh;
+ __u8 LeaseKey[SMB2_LEASE_KEY_SIZE];
__le32 LeaseState;
__le32 LeaseFlags;
__le64 LeaseDuration;
- __le64 ParentLeaseKeyLow;
- __le64 ParentLeaseKeyHigh;
+ __u8 ParentLeaseKey[SMB2_LEASE_KEY_SIZE];
__le16 Epoch;
__le16 Reserved;
} __packed;
__u8 Pad[4];
} __packed;
-/* Currently defined values for close flags */
-#define SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB cpu_to_le16(0x0001)
-struct smb2_close_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 24 */
- __le16 Flags;
- __le32 Reserved;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
-} __packed;
-
-struct smb2_close_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* 60 */
- __le16 Flags;
- __le32 Reserved;
- __le64 CreationTime;
- __le64 LastAccessTime;
- __le64 LastWriteTime;
- __le64 ChangeTime;
- __le64 AllocationSize; /* Beginning of FILE_STANDARD_INFO equivalent */
- __le64 EndOfFile;
- __le32 Attributes;
-} __packed;
-
-struct smb2_flush_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 24 */
- __le16 Reserved1;
- __le32 Reserved2;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
-} __packed;
-
-struct smb2_flush_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize;
- __le16 Reserved;
-} __packed;
-
struct smb2_buffer_desc_v1 {
__le64 offset;
__le32 token;
__le32 length;
} __packed;
-#define SMB2_CHANNEL_NONE cpu_to_le32(0x00000000)
-#define SMB2_CHANNEL_RDMA_V1 cpu_to_le32(0x00000001)
-#define SMB2_CHANNEL_RDMA_V1_INVALIDATE cpu_to_le32(0x00000002)
-
-struct smb2_read_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 49 */
- __u8 Padding; /* offset from start of SMB2 header to place read */
- __u8 Reserved;
- __le32 Length;
- __le64 Offset;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
- __le32 MinimumCount;
- __le32 Channel; /* Reserved MBZ */
- __le32 RemainingBytes;
- __le16 ReadChannelInfoOffset; /* Reserved MBZ */
- __le16 ReadChannelInfoLength; /* Reserved MBZ */
- __u8 Buffer[1];
-} __packed;
-
-struct smb2_read_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 17 */
- __u8 DataOffset;
- __u8 Reserved;
- __le32 DataLength;
- __le32 DataRemaining;
- __u32 Reserved2;
- __u8 Buffer[1];
-} __packed;
-
-/* For write request Flags field below the following flag is defined: */
-#define SMB2_WRITEFLAG_WRITE_THROUGH 0x00000001
-
-struct smb2_write_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 49 */
- __le16 DataOffset; /* offset from start of SMB2 header to write data */
- __le32 Length;
- __le64 Offset;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
- __le32 Channel; /* Reserved MBZ */
- __le32 RemainingBytes;
- __le16 WriteChannelInfoOffset; /* Reserved MBZ */
- __le16 WriteChannelInfoLength; /* Reserved MBZ */
- __le32 Flags;
- __u8 Buffer[1];
-} __packed;
-
-struct smb2_write_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 17 */
- __u8 DataOffset;
- __u8 Reserved;
- __le32 DataLength;
- __le32 DataRemaining;
- __u32 Reserved2;
- __u8 Buffer[1];
-} __packed;
-
#define SMB2_0_IOCTL_IS_FSCTL 0x00000001
struct duplicate_extents_to_file {
__u8 DataBuffer[]; /* Variable Length */
} __packed;
-/* Completion Filter flags for Notify */
-#define FILE_NOTIFY_CHANGE_FILE_NAME 0x00000001
-#define FILE_NOTIFY_CHANGE_DIR_NAME 0x00000002
-#define FILE_NOTIFY_CHANGE_NAME 0x00000003
-#define FILE_NOTIFY_CHANGE_ATTRIBUTES 0x00000004
-#define FILE_NOTIFY_CHANGE_SIZE 0x00000008
-#define FILE_NOTIFY_CHANGE_LAST_WRITE 0x00000010
-#define FILE_NOTIFY_CHANGE_LAST_ACCESS 0x00000020
-#define FILE_NOTIFY_CHANGE_CREATION 0x00000040
-#define FILE_NOTIFY_CHANGE_EA 0x00000080
-#define FILE_NOTIFY_CHANGE_SECURITY 0x00000100
-#define FILE_NOTIFY_CHANGE_STREAM_NAME 0x00000200
-#define FILE_NOTIFY_CHANGE_STREAM_SIZE 0x00000400
-#define FILE_NOTIFY_CHANGE_STREAM_WRITE 0x00000800
-
-/* Flags */
-#define SMB2_WATCH_TREE 0x0001
-
-struct smb2_notify_req {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 32 */
- __le16 Flags;
- __le32 OutputBufferLength;
- __le64 PersistentFileId;
- __le64 VolatileFileId;
- __u32 CompletionFileter;
- __u32 Reserved;
-} __packed;
-
-struct smb2_notify_rsp {
- struct smb2_hdr hdr;
- __le16 StructureSize; /* Must be 9 */
- __le16 OutputBufferOffset;
- __le32 OutputBufferLength;
- __u8 Buffer[1];
-} __packed;
-
/* SMB2 Notify Action Flags */
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
__le64 CurrentByteOffset;
} __packed;
-#define FILE_MODE_INFO_MASK cpu_to_le32(0x0000103e)
+#define FILE_MODE_INFO_MASK cpu_to_le32(0x0000100e)
struct smb2_file_mode_info {
__le32 Mode;
int smb2_oplock_break(struct ksmbd_work *work);
int smb2_notify(struct ksmbd_work *ksmbd_work);
+/*
+ * Get the body of the smb2 message excluding the 4 byte rfc1002 headers
+ * from request/response buffer.
+ */
+static inline void *smb2_get_msg(void *buf)
+{
+ return buf + 4;
+}
+
#endif /* _SMB2PDU_H */
*/
int ksmbd_verify_smb_message(struct ksmbd_work *work)
{
- struct smb2_hdr *smb2_hdr = work->request_buf + work->next_smb2_rcv_hdr_off;
+ struct smb2_hdr *smb2_hdr = ksmbd_req_buf_next(work);
struct smb_hdr *hdr;
if (smb2_hdr->ProtocolId == SMB2_PROTO_NUMBER)
static int ksmbd_negotiate_smb_dialect(void *buf)
{
int smb_buf_length = get_rfc1002_len(buf);
- __le32 proto = ((struct smb2_hdr *)buf)->ProtocolId;
+ __le32 proto = ((struct smb2_hdr *)smb2_get_msg(buf))->ProtocolId;
if (proto == SMB2_PROTO_NUMBER) {
struct smb2_negotiate_req *req;
int smb2_neg_size =
- offsetof(struct smb2_negotiate_req, Dialects) - 4;
+ offsetof(struct smb2_negotiate_req, Dialects);
- req = (struct smb2_negotiate_req *)buf;
+ req = (struct smb2_negotiate_req *)smb2_get_msg(buf);
if (smb2_neg_size > smb_buf_length)
goto err_out;
struct ksmbd_conn *conn = work->conn;
int ret;
- conn->dialect = ksmbd_negotiate_smb_dialect(work->request_buf);
+ conn->dialect =
+ ksmbd_negotiate_smb_dialect(work->request_buf);
ksmbd_debug(SMB, "conn->dialect 0x%x\n", conn->dialect);
if (command == SMB2_NEGOTIATE_HE) {
- struct smb2_hdr *smb2_hdr = work->request_buf;
+ struct smb2_hdr *smb2_hdr = smb2_get_msg(work->request_buf);
if (smb2_hdr->ProtocolId != SMB2_PROTO_NUMBER) {
ksmbd_debug(SMB, "Downgrade to SMB1 negotiation\n");
#include "glob.h"
#include "nterr.h"
+#include "../smbfs_common/smb2pdu.h"
#include "smb2pdu.h"
/* ksmbd's Specific ERRNO */
#define SMB302_VERSION_STRING "3.02"
#define SMB311_VERSION_STRING "3.1.1"
-/* Dialects */
-#define SMB10_PROT_ID 0x00
-#define SMB20_PROT_ID 0x0202
-#define SMB21_PROT_ID 0x0210
-/* multi-protocol negotiate request */
-#define SMB2X_PROT_ID 0x02FF
-#define SMB30_PROT_ID 0x0300
-#define SMB302_PROT_ID 0x0302
-#define SMB311_PROT_ID 0x0311
-#define BAD_PROT_ID 0xFFFF
-
#define SMB_ECHO_INTERVAL (60 * HZ)
#define CIFS_DEFAULT_IOSIZE (64 * 1024)
/*
* File Attribute flags
*/
-#define ATTR_READONLY 0x0001
-#define ATTR_HIDDEN 0x0002
-#define ATTR_SYSTEM 0x0004
-#define ATTR_VOLUME 0x0008
-#define ATTR_DIRECTORY 0x0010
-#define ATTR_ARCHIVE 0x0020
-#define ATTR_DEVICE 0x0040
-#define ATTR_NORMAL 0x0080
-#define ATTR_TEMPORARY 0x0100
-#define ATTR_SPARSE 0x0200
-#define ATTR_REPARSE 0x0400
-#define ATTR_COMPRESSED 0x0800
-#define ATTR_OFFLINE 0x1000
-#define ATTR_NOT_CONTENT_INDEXED 0x2000
-#define ATTR_ENCRYPTED 0x4000
#define ATTR_POSIX_SEMANTICS 0x01000000
#define ATTR_BACKUP_SEMANTICS 0x02000000
#define ATTR_DELETE_ON_CLOSE 0x04000000
#define ATTR_NO_BUFFERING 0x20000000
#define ATTR_WRITE_THROUGH 0x80000000
-#define ATTR_READONLY_LE cpu_to_le32(ATTR_READONLY)
-#define ATTR_HIDDEN_LE cpu_to_le32(ATTR_HIDDEN)
-#define ATTR_SYSTEM_LE cpu_to_le32(ATTR_SYSTEM)
-#define ATTR_DIRECTORY_LE cpu_to_le32(ATTR_DIRECTORY)
-#define ATTR_ARCHIVE_LE cpu_to_le32(ATTR_ARCHIVE)
-#define ATTR_NORMAL_LE cpu_to_le32(ATTR_NORMAL)
-#define ATTR_TEMPORARY_LE cpu_to_le32(ATTR_TEMPORARY)
-#define ATTR_SPARSE_FILE_LE cpu_to_le32(ATTR_SPARSE)
-#define ATTR_REPARSE_POINT_LE cpu_to_le32(ATTR_REPARSE)
-#define ATTR_COMPRESSED_LE cpu_to_le32(ATTR_COMPRESSED)
-#define ATTR_OFFLINE_LE cpu_to_le32(ATTR_OFFLINE)
-#define ATTR_NOT_CONTENT_INDEXED_LE cpu_to_le32(ATTR_NOT_CONTENT_INDEXED)
-#define ATTR_ENCRYPTED_LE cpu_to_le32(ATTR_ENCRYPTED)
-#define ATTR_INTEGRITY_STREAML_LE cpu_to_le32(0x00008000)
-#define ATTR_NO_SCRUB_DATA_LE cpu_to_le32(0x00020000)
-#define ATTR_MASK_LE cpu_to_le32(0x00007FB7)
-
/* List of FileSystemAttributes - see 2.5.1 of MS-FSCC */
#define FILE_SUPPORTS_SPARSE_VDL 0x10000000 /* faster nonsparse extend */
#define FILE_SUPPORTS_BLOCK_REFCOUNTING 0x08000000 /* allow ioctl dup extents */
/* file_execute, file_read_attributes*/
/* write_dac, and delete. */
-#define FILE_READ_RIGHTS (FILE_READ_DATA | FILE_READ_EA | FILE_READ_ATTRIBUTES)
-#define FILE_WRITE_RIGHTS (FILE_WRITE_DATA | FILE_APPEND_DATA \
- | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES)
-#define FILE_EXEC_RIGHTS (FILE_EXECUTE)
-
#define SET_FILE_READ_RIGHTS (FILE_READ_DATA | FILE_READ_EA \
| FILE_READ_ATTRIBUTES \
| DELETE | READ_CONTROL | WRITE_DAC \
int (*proc)(struct ksmbd_work *swork);
};
-static inline size_t
-smb2_hdr_size_no_buflen(struct smb_version_values *vals)
-{
- return vals->header_size - 4;
-}
-
int ksmbd_min_protocol(void);
int ksmbd_max_protocol(void);
struct smb_direct_data_transfer *req =
(struct smb_direct_data_transfer *)recvmsg->packet;
struct smb2_hdr *hdr = (struct smb2_hdr *)(recvmsg->packet
- + le32_to_cpu(req->data_offset) - 4);
+ + le32_to_cpu(req->data_offset));
ksmbd_debug(RDMA,
"CreditGranted: %u, CreditRequested: %u, DataLength: %u, RemainingDataLength: %u, SMB: %x, Command: %u\n",
le16_to_cpu(req->credits_granted),
smb_direct_listener.cm_id = NULL;
if (smb_direct_wq) {
- flush_workqueue(smb_direct_wq);
destroy_workqueue(smb_direct_wq);
smb_direct_wq = NULL;
}
loff_t off, loff_t len)
{
smb_break_all_levII_oplock(work, fp, 1);
- if (fp->f_ci->m_fattr & ATTR_SPARSE_FILE_LE)
+ if (fp->f_ci->m_fattr & FILE_ATTRIBUTE_SPARSE_FILE_LE)
return vfs_fallocate(fp->filp,
FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
off, len);
time = ksmbd_UnixTimeToNT(kstat->ctime);
info->ChangeTime = cpu_to_le64(time);
- if (ksmbd_kstat->file_attributes & ATTR_DIRECTORY_LE) {
+ if (ksmbd_kstat->file_attributes & FILE_ATTRIBUTE_DIRECTORY_LE) {
info->EndOfFile = 0;
info->AllocationSize = 0;
} else {
* or that acl is disable in server's filesystem and the config is yes.
*/
if (S_ISDIR(ksmbd_kstat->kstat->mode))
- ksmbd_kstat->file_attributes = ATTR_DIRECTORY_LE;
+ ksmbd_kstat->file_attributes = FILE_ATTRIBUTE_DIRECTORY_LE;
else
- ksmbd_kstat->file_attributes = ATTR_ARCHIVE_LE;
+ ksmbd_kstat->file_attributes = FILE_ATTRIBUTE_ARCHIVE_LE;
if (test_share_config_flag(work->tcon->share_conf,
KSMBD_SHARE_FLAG_STORE_DOS_ATTRS)) {
};
/* CreateOptions */
-/* Flag is set, it must not be a file , valid for directory only */
-#define FILE_DIRECTORY_FILE_LE cpu_to_le32(0x00000001)
-#define FILE_WRITE_THROUGH_LE cpu_to_le32(0x00000002)
-#define FILE_SEQUENTIAL_ONLY_LE cpu_to_le32(0x00000004)
-
-/* Should not buffer on server*/
-#define FILE_NO_INTERMEDIATE_BUFFERING_LE cpu_to_le32(0x00000008)
-/* MBZ */
-#define FILE_SYNCHRONOUS_IO_ALERT_LE cpu_to_le32(0x00000010)
-/* MBZ */
-#define FILE_SYNCHRONOUS_IO_NONALERT_LE cpu_to_le32(0x00000020)
-
-/* Flaf must not be set for directory */
-#define FILE_NON_DIRECTORY_FILE_LE cpu_to_le32(0x00000040)
-
-/* Should be zero */
#define CREATE_TREE_CONNECTION cpu_to_le32(0x00000080)
-#define FILE_COMPLETE_IF_OPLOCKED_LE cpu_to_le32(0x00000100)
-#define FILE_NO_EA_KNOWLEDGE_LE cpu_to_le32(0x00000200)
-#define FILE_OPEN_REMOTE_INSTANCE cpu_to_le32(0x00000400)
-
-/**
- * Doc says this is obsolete "open for recovery" flag should be zero
- * in any case.
- */
-#define CREATE_OPEN_FOR_RECOVERY cpu_to_le32(0x00000400)
-#define FILE_RANDOM_ACCESS_LE cpu_to_le32(0x00000800)
-#define FILE_DELETE_ON_CLOSE_LE cpu_to_le32(0x00001000)
-#define FILE_OPEN_BY_FILE_ID_LE cpu_to_le32(0x00002000)
-#define FILE_OPEN_FOR_BACKUP_INTENT_LE cpu_to_le32(0x00004000)
-#define FILE_NO_COMPRESSION_LE cpu_to_le32(0x00008000)
-
-/* Should be zero*/
-#define FILE_OPEN_REQUIRING_OPLOCK cpu_to_le32(0x00010000)
-#define FILE_DISALLOW_EXCLUSIVE cpu_to_le32(0x00020000)
#define FILE_RESERVE_OPFILTER_LE cpu_to_le32(0x00100000)
-#define FILE_OPEN_REPARSE_POINT_LE cpu_to_le32(0x00200000)
-#define FILE_OPEN_NO_RECALL_LE cpu_to_le32(0x00400000)
-/* Should be zero */
-#define FILE_OPEN_FOR_FREE_SPACE_QUERY_LE cpu_to_le32(0x00800000)
-#define CREATE_OPTIONS_MASK cpu_to_le32(0x00FFFFFF)
#define CREATE_OPTION_READONLY 0x10000000
/* system. NB not sent over wire */
#define CREATE_OPTION_SPECIAL 0x20000000
}
EXPORT_SYMBOL(simple_rmdir);
+int simple_rename_exchange(struct inode *old_dir, struct dentry *old_dentry,
+ struct inode *new_dir, struct dentry *new_dentry)
+{
+ bool old_is_dir = d_is_dir(old_dentry);
+ bool new_is_dir = d_is_dir(new_dentry);
+
+ if (old_dir != new_dir && old_is_dir != new_is_dir) {
+ if (old_is_dir) {
+ drop_nlink(old_dir);
+ inc_nlink(new_dir);
+ } else {
+ drop_nlink(new_dir);
+ inc_nlink(old_dir);
+ }
+ }
+ old_dir->i_ctime = old_dir->i_mtime =
+ new_dir->i_ctime = new_dir->i_mtime =
+ d_inode(old_dentry)->i_ctime =
+ d_inode(new_dentry)->i_ctime = current_time(old_dir);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(simple_rename_exchange);
+
int simple_rename(struct user_namespace *mnt_userns, struct inode *old_dir,
struct dentry *old_dentry, struct inode *new_dir,
struct dentry *new_dentry, unsigned int flags)
struct inode *inode = d_inode(old_dentry);
int they_are_dirs = d_is_dir(old_dentry);
- if (flags & ~RENAME_NOREPLACE)
+ if (flags & ~(RENAME_NOREPLACE | RENAME_EXCHANGE))
return -EINVAL;
+ if (flags & RENAME_EXCHANGE)
+ return simple_rename_exchange(old_dir, old_dentry, new_dir, new_dentry);
+
if (!simple_empty(new_dentry))
return -ENOTEMPTY;
goto retry_cancel;
}
- dprintk("lockd: cancel status %u (task %u)\n",
- status, task->tk_pid);
-
switch (status) {
case NLM_LCK_GRANTED:
case NLM_LCK_DENIED_GRACE_PERIOD:
static int nlmsvc_dispatch(struct svc_rqst *rqstp, __be32 *statp)
{
const struct svc_procedure *procp = rqstp->rq_procinfo;
- struct kvec *argv = rqstp->rq_arg.head;
- struct kvec *resv = rqstp->rq_res.head;
svcxdr_init_decode(rqstp);
- if (!procp->pc_decode(rqstp, argv->iov_base))
+ if (!procp->pc_decode(rqstp, &rqstp->rq_arg_stream))
goto out_decode_err;
*statp = procp->pc_func(rqstp);
return 1;
svcxdr_init_encode(rqstp);
- if (!procp->pc_encode(rqstp, resv->iov_base + resv->iov_len))
+ if (!procp->pc_encode(rqstp, &rqstp->rq_res_stream))
goto out_encode_err;
return 1;
*/
static void nlm4svc_callback_exit(struct rpc_task *task, void *data)
{
- dprintk("lockd: %5u callback returned %d\n", task->tk_pid,
- -task->tk_status);
}
static void nlm4svc_callback_release(void *data)
*/
static void nlmsvc_callback_exit(struct rpc_task *task, void *data)
{
- dprintk("lockd: %5u callback returned %d\n", task->tk_pid,
- -task->tk_status);
}
void nlmsvc_release_call(struct nlm_rqst *call)
* Decode Call arguments
*/
-int
-nlmsvc_decode_void(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
-int
-nlmsvc_decode_testargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
- return 1;
+ return true;
}
-int
-nlmsvc_decode_lockargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &argp->block) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
if (xdr_stream_decode_bool(xdr, &argp->reclaim) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
argp->monitor = 1; /* monitor client by default */
- return 1;
+ return true;
}
-int
-nlmsvc_decode_cancargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &argp->block) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
- return 1;
+ return true;
}
-int
-nlmsvc_decode_unlockargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
argp->lock.fl.fl_type = F_UNLCK;
- return 1;
+ return true;
}
-int
-nlmsvc_decode_res(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_res *resp = rqstp->rq_argp;
if (!svcxdr_decode_cookie(xdr, &resp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_stats(xdr, &resp->status))
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nlmsvc_decode_reboot(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_reboot(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_reboot *argp = rqstp->rq_argp;
+ __be32 *p;
u32 len;
if (xdr_stream_decode_u32(xdr, &len) < 0)
- return 0;
+ return false;
if (len > SM_MAXSTRLEN)
- return 0;
+ return false;
p = xdr_inline_decode(xdr, len);
if (!p)
- return 0;
+ return false;
argp->len = len;
argp->mon = (char *)p;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
p = xdr_inline_decode(xdr, SM_PRIV_SIZE);
if (!p)
- return 0;
+ return false;
memcpy(&argp->priv.data, p, sizeof(argp->priv.data));
- return 1;
+ return true;
}
-int
-nlmsvc_decode_shareargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_shareargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
struct nlm_lock *lock = &argp->lock;
lock->svid = ~(u32)0;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_string(xdr, &lock->caller, &lock->len))
- return 0;
+ return false;
if (!svcxdr_decode_fhandle(xdr, &lock->fh))
- return 0;
+ return false;
if (!svcxdr_decode_owner(xdr, &lock->oh))
- return 0;
+ return false;
/* XXX: Range checks are missing in the original code */
if (xdr_stream_decode_u32(xdr, &argp->fsm_mode) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->fsm_access) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nlmsvc_decode_notify(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_decode_notify(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
struct nlm_lock *lock = &argp->lock;
if (!svcxdr_decode_string(xdr, &lock->caller, &lock->len))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
* Encode Reply results
*/
-int
-nlmsvc_encode_void(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_encode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
-int
-nlmsvc_encode_testres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_encode_testres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
return svcxdr_encode_cookie(xdr, &resp->cookie) &&
svcxdr_encode_testrply(xdr, resp);
}
-int
-nlmsvc_encode_res(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_encode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
return svcxdr_encode_cookie(xdr, &resp->cookie) &&
svcxdr_encode_stats(xdr, resp->status);
}
-int
-nlmsvc_encode_shareres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlmsvc_encode_shareres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
if (!svcxdr_encode_cookie(xdr, &resp->cookie))
- return 0;
+ return false;
if (!svcxdr_encode_stats(xdr, resp->status))
- return 0;
+ return false;
/* sequence */
if (xdr_stream_encode_u32(xdr, 0) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
* Decode Call arguments
*/
-int
-nlm4svc_decode_void(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
-int
-nlm4svc_decode_testargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
- return 1;
+ return true;
}
-int
-nlm4svc_decode_lockargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &argp->block) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
if (xdr_stream_decode_bool(xdr, &argp->reclaim) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
argp->monitor = 1; /* monitor client by default */
- return 1;
+ return true;
}
-int
-nlm4svc_decode_cancargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
u32 exclusive;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &argp->block) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_bool(xdr, &exclusive) < 0)
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
if (exclusive)
argp->lock.fl.fl_type = F_WRLCK;
- return 1;
+
+ return true;
}
-int
-nlm4svc_decode_unlockargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_lock(xdr, &argp->lock))
- return 0;
+ return false;
argp->lock.fl.fl_type = F_UNLCK;
- return 1;
+ return true;
}
-int
-nlm4svc_decode_res(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_res *resp = rqstp->rq_argp;
if (!svcxdr_decode_cookie(xdr, &resp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_stats(xdr, &resp->status))
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nlm4svc_decode_reboot(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_reboot(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_reboot *argp = rqstp->rq_argp;
+ __be32 *p;
u32 len;
if (xdr_stream_decode_u32(xdr, &len) < 0)
- return 0;
+ return false;
if (len > SM_MAXSTRLEN)
- return 0;
+ return false;
p = xdr_inline_decode(xdr, len);
if (!p)
- return 0;
+ return false;
argp->len = len;
argp->mon = (char *)p;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
p = xdr_inline_decode(xdr, SM_PRIV_SIZE);
if (!p)
- return 0;
+ return false;
memcpy(&argp->priv.data, p, sizeof(argp->priv.data));
- return 1;
+ return true;
}
-int
-nlm4svc_decode_shareargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_shareargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
struct nlm_lock *lock = &argp->lock;
lock->svid = ~(u32)0;
if (!svcxdr_decode_cookie(xdr, &argp->cookie))
- return 0;
+ return false;
if (!svcxdr_decode_string(xdr, &lock->caller, &lock->len))
- return 0;
+ return false;
if (!svcxdr_decode_fhandle(xdr, &lock->fh))
- return 0;
+ return false;
if (!svcxdr_decode_owner(xdr, &lock->oh))
- return 0;
+ return false;
/* XXX: Range checks are missing in the original code */
if (xdr_stream_decode_u32(xdr, &argp->fsm_mode) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->fsm_access) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nlm4svc_decode_notify(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_decode_notify(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nlm_args *argp = rqstp->rq_argp;
struct nlm_lock *lock = &argp->lock;
if (!svcxdr_decode_string(xdr, &lock->caller, &lock->len))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->state) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
* Encode Reply results
*/
-int
-nlm4svc_encode_void(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_encode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
-int
-nlm4svc_encode_testres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_encode_testres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
return svcxdr_encode_cookie(xdr, &resp->cookie) &&
svcxdr_encode_testrply(xdr, resp);
}
-int
-nlm4svc_encode_res(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_encode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
return svcxdr_encode_cookie(xdr, &resp->cookie) &&
svcxdr_encode_stats(xdr, resp->status);
}
-int
-nlm4svc_encode_shareres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nlm4svc_encode_shareres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nlm_res *resp = rqstp->rq_resp;
if (!svcxdr_encode_cookie(xdr, &resp->cookie))
- return 0;
+ return false;
if (!svcxdr_encode_stats(xdr, resp->status))
- return 0;
+ return false;
/* sequence */
if (xdr_stream_encode_u32(xdr, 0) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
/*
* Deal with the completion of writing the data to the cache. We have to clear
- * the PG_fscache bits on the pages involved and release the caller's ref.
+ * the PG_fscache bits on the folios involved and release the caller's ref.
*
* May be called in softirq mode and we inherit a ref from the caller.
*/
bool was_async)
{
struct netfs_read_subrequest *subreq;
- struct page *page;
+ struct folio *folio;
pgoff_t unlocked = 0;
bool have_unlocked = false;
list_for_each_entry(subreq, &rreq->subrequests, rreq_link) {
XA_STATE(xas, &rreq->mapping->i_pages, subreq->start / PAGE_SIZE);
- xas_for_each(&xas, page, (subreq->start + subreq->len - 1) / PAGE_SIZE) {
+ xas_for_each(&xas, folio, (subreq->start + subreq->len - 1) / PAGE_SIZE) {
/* We might have multiple writes from the same huge
- * page, but we mustn't unlock a page more than once.
+ * folio, but we mustn't unlock a folio more than once.
*/
- if (have_unlocked && page->index <= unlocked)
+ if (have_unlocked && folio_index(folio) <= unlocked)
continue;
- unlocked = page->index;
- end_page_fscache(page);
+ unlocked = folio_index(folio);
+ folio_end_fscache(folio);
have_unlocked = true;
}
}
}
/*
- * Unlock the pages in a read operation. We need to set PG_fscache on any
- * pages we're going to write back before we unlock them.
+ * Unlock the folios in a read operation. We need to set PG_fscache on any
+ * folios we're going to write back before we unlock them.
*/
static void netfs_rreq_unlock(struct netfs_read_request *rreq)
{
struct netfs_read_subrequest *subreq;
- struct page *page;
+ struct folio *folio;
unsigned int iopos, account = 0;
pgoff_t start_page = rreq->start / PAGE_SIZE;
pgoff_t last_page = ((rreq->start + rreq->len) / PAGE_SIZE) - 1;
bool subreq_failed = false;
- int i;
XA_STATE(xas, &rreq->mapping->i_pages, start_page);
trace_netfs_rreq(rreq, netfs_rreq_trace_unlock);
rcu_read_lock();
- xas_for_each(&xas, page, last_page) {
- unsigned int pgpos = (page->index - start_page) * PAGE_SIZE;
- unsigned int pgend = pgpos + thp_size(page);
+ xas_for_each(&xas, folio, last_page) {
+ unsigned int pgpos = (folio_index(folio) - start_page) * PAGE_SIZE;
+ unsigned int pgend = pgpos + folio_size(folio);
bool pg_failed = false;
for (;;) {
break;
}
if (test_bit(NETFS_SREQ_WRITE_TO_CACHE, &subreq->flags))
- set_page_fscache(page);
+ folio_start_fscache(folio);
pg_failed |= subreq_failed;
if (pgend < iopos + subreq->len)
break;
}
if (!pg_failed) {
- for (i = 0; i < thp_nr_pages(page); i++)
- flush_dcache_page(page);
- SetPageUptodate(page);
+ flush_dcache_folio(folio);
+ folio_mark_uptodate(folio);
}
- if (!test_bit(NETFS_RREQ_DONT_UNLOCK_PAGES, &rreq->flags)) {
- if (page->index == rreq->no_unlock_page &&
- test_bit(NETFS_RREQ_NO_UNLOCK_PAGE, &rreq->flags))
+ if (!test_bit(NETFS_RREQ_DONT_UNLOCK_FOLIOS, &rreq->flags)) {
+ if (folio_index(folio) == rreq->no_unlock_folio &&
+ test_bit(NETFS_RREQ_NO_UNLOCK_FOLIO, &rreq->flags))
_debug("no unlock");
else
- unlock_page(page);
+ folio_unlock(folio);
}
}
rcu_read_unlock();
void *netfs_priv)
{
struct netfs_read_request *rreq;
- struct page *page;
unsigned int debug_index = 0;
int ret;
} while (rreq->submitted < rreq->len);
- /* Drop the refs on the pages here rather than in the cache or
+ /* Drop the refs on the folios here rather than in the cache or
* filesystem. The locks will be dropped in netfs_rreq_unlock().
*/
- while ((page = readahead_page(ractl)))
- put_page(page);
+ while (readahead_folio(ractl))
+ ;
/* If we decrement nr_rd_ops to 0, the ref belongs to us. */
if (atomic_dec_and_test(&rreq->nr_rd_ops))
/**
* netfs_readpage - Helper to manage a readpage request
* @file: The file to read from
- * @page: The page to read
+ * @folio: The folio to read
* @ops: The network filesystem's operations for the helper to use
* @netfs_priv: Private netfs data to be retained in the request
*
* This is usable whether or not caching is enabled.
*/
int netfs_readpage(struct file *file,
- struct page *page,
+ struct folio *folio,
const struct netfs_read_request_ops *ops,
void *netfs_priv)
{
unsigned int debug_index = 0;
int ret;
- _enter("%lx", page_index(page));
+ _enter("%lx", folio_index(folio));
rreq = netfs_alloc_read_request(ops, netfs_priv, file);
if (!rreq) {
if (netfs_priv)
- ops->cleanup(netfs_priv, page_file_mapping(page));
- unlock_page(page);
+ ops->cleanup(netfs_priv, folio_file_mapping(folio));
+ folio_unlock(folio);
return -ENOMEM;
}
- rreq->mapping = page_file_mapping(page);
- rreq->start = page_file_offset(page);
- rreq->len = thp_size(page);
+ rreq->mapping = folio_file_mapping(folio);
+ rreq->start = folio_file_pos(folio);
+ rreq->len = folio_size(folio);
if (ops->begin_cache_operation) {
ret = ops->begin_cache_operation(rreq);
if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS) {
- unlock_page(page);
+ folio_unlock(folio);
goto out;
}
}
EXPORT_SYMBOL(netfs_readpage);
/**
- * netfs_skip_page_read - prep a page for writing without reading first
- * @page: page being prepared
+ * netfs_skip_folio_read - prep a folio for writing without reading first
+ * @folio: The folio being prepared
* @pos: starting position for the write
* @len: length of write
*
* In some cases, write_begin doesn't need to read at all:
- * - full page write
- * - write that lies in a page that is completely beyond EOF
- * - write that covers the the page from start to EOF or beyond it
+ * - full folio write
+ * - write that lies in a folio that is completely beyond EOF
+ * - write that covers the folio from start to EOF or beyond it
*
* If any of these criteria are met, then zero out the unwritten parts
- * of the page and return true. Otherwise, return false.
+ * of the folio and return true. Otherwise, return false.
*/
-static bool netfs_skip_page_read(struct page *page, loff_t pos, size_t len)
+static bool netfs_skip_folio_read(struct folio *folio, loff_t pos, size_t len)
{
- struct inode *inode = page->mapping->host;
+ struct inode *inode = folio_inode(folio);
loff_t i_size = i_size_read(inode);
- size_t offset = offset_in_thp(page, pos);
+ size_t offset = offset_in_folio(folio, pos);
- /* Full page write */
- if (offset == 0 && len >= thp_size(page))
+ /* Full folio write */
+ if (offset == 0 && len >= folio_size(folio))
return true;
- /* pos beyond last page in the file */
+ /* pos beyond last folio in the file */
if (pos - offset >= i_size)
goto zero_out;
- /* Write that covers from the start of the page to EOF or beyond */
+ /* Write that covers from the start of the folio to EOF or beyond */
if (offset == 0 && (pos + len) >= i_size)
goto zero_out;
return false;
zero_out:
- zero_user_segments(page, 0, offset, offset + len, thp_size(page));
+ zero_user_segments(&folio->page, 0, offset, offset + len, folio_size(folio));
return true;
}
* @file: The file to read from
* @mapping: The mapping to read from
* @pos: File position at which the write will begin
- * @len: The length of the write (may extend beyond the end of the page chosen)
- * @flags: AOP_* flags
- * @_page: Where to put the resultant page
+ * @len: The length of the write (may extend beyond the end of the folio chosen)
+ * @aop_flags: AOP_* flags
+ * @_folio: Where to put the resultant folio
* @_fsdata: Place for the netfs to store a cookie
* @ops: The network filesystem's operations for the helper to use
* @netfs_priv: Private netfs data to be retained in the request
* issue_op, is mandatory.
*
* The check_write_begin() operation can be provided to check for and flush
- * conflicting writes once the page is grabbed and locked. It is passed a
+ * conflicting writes once the folio is grabbed and locked. It is passed a
* pointer to the fsdata cookie that gets returned to the VM to be passed to
* write_end. It is permitted to sleep. It should return 0 if the request
- * should go ahead; unlock the page and return -EAGAIN to cause the page to be
- * regot; or return an error.
+ * should go ahead; unlock the folio and return -EAGAIN to cause the folio to
+ * be regot; or return an error.
*
* This is usable whether or not caching is enabled.
*/
int netfs_write_begin(struct file *file, struct address_space *mapping,
- loff_t pos, unsigned int len, unsigned int flags,
- struct page **_page, void **_fsdata,
+ loff_t pos, unsigned int len, unsigned int aop_flags,
+ struct folio **_folio, void **_fsdata,
const struct netfs_read_request_ops *ops,
void *netfs_priv)
{
struct netfs_read_request *rreq;
- struct page *page, *xpage;
+ struct folio *folio;
struct inode *inode = file_inode(file);
- unsigned int debug_index = 0;
+ unsigned int debug_index = 0, fgp_flags;
pgoff_t index = pos >> PAGE_SHIFT;
int ret;
DEFINE_READAHEAD(ractl, file, NULL, mapping, index);
retry:
- page = grab_cache_page_write_begin(mapping, index, flags);
- if (!page)
+ fgp_flags = FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE;
+ if (aop_flags & AOP_FLAG_NOFS)
+ fgp_flags |= FGP_NOFS;
+ folio = __filemap_get_folio(mapping, index, fgp_flags,
+ mapping_gfp_mask(mapping));
+ if (!folio)
return -ENOMEM;
if (ops->check_write_begin) {
/* Allow the netfs (eg. ceph) to flush conflicts. */
- ret = ops->check_write_begin(file, pos, len, page, _fsdata);
+ ret = ops->check_write_begin(file, pos, len, folio, _fsdata);
if (ret < 0) {
trace_netfs_failure(NULL, NULL, ret, netfs_fail_check_write_begin);
if (ret == -EAGAIN)
}
}
- if (PageUptodate(page))
- goto have_page;
+ if (folio_test_uptodate(folio))
+ goto have_folio;
/* If the page is beyond the EOF, we want to clear it - unless it's
* within the cache granule containing the EOF, in which case we need
* to preload the granule.
*/
if (!ops->is_cache_enabled(inode) &&
- netfs_skip_page_read(page, pos, len)) {
+ netfs_skip_folio_read(folio, pos, len)) {
netfs_stat(&netfs_n_rh_write_zskip);
- goto have_page_no_wait;
+ goto have_folio_no_wait;
}
ret = -ENOMEM;
rreq = netfs_alloc_read_request(ops, netfs_priv, file);
if (!rreq)
goto error;
- rreq->mapping = page->mapping;
- rreq->start = page_offset(page);
- rreq->len = thp_size(page);
- rreq->no_unlock_page = page->index;
- __set_bit(NETFS_RREQ_NO_UNLOCK_PAGE, &rreq->flags);
+ rreq->mapping = folio_file_mapping(folio);
+ rreq->start = folio_file_pos(folio);
+ rreq->len = folio_size(folio);
+ rreq->no_unlock_folio = folio_index(folio);
+ __set_bit(NETFS_RREQ_NO_UNLOCK_FOLIO, &rreq->flags);
netfs_priv = NULL;
if (ops->begin_cache_operation) {
/* Expand the request to meet caching requirements and download
* preferences.
*/
- ractl._nr_pages = thp_nr_pages(page);
+ ractl._nr_pages = folio_nr_pages(folio);
netfs_rreq_expand(rreq, &ractl);
netfs_get_read_request(rreq);
- /* We hold the page locks, so we can drop the references */
- while ((xpage = readahead_page(&ractl)))
- if (xpage != page)
- put_page(xpage);
+ /* We hold the folio locks, so we can drop the references */
+ folio_get(folio);
+ while (readahead_folio(&ractl))
+ ;
atomic_set(&rreq->nr_rd_ops, 1);
do {
if (ret < 0)
goto error;
-have_page:
- ret = wait_on_page_fscache_killable(page);
+have_folio:
+ ret = folio_wait_fscache_killable(folio);
if (ret < 0)
goto error;
-have_page_no_wait:
+have_folio_no_wait:
if (netfs_priv)
ops->cleanup(netfs_priv, mapping);
- *_page = page;
+ *_folio = folio;
_leave(" = 0");
return 0;
error_put:
netfs_put_read_request(rreq, false);
error:
- unlock_page(page);
- put_page(page);
+ folio_unlock(folio);
+ folio_put(folio);
if (netfs_priv)
ops->cleanup(netfs_priv, mapping);
_leave(" = %d", ret);
kfree(copy);
spin_unlock(&cps->clp->cl_lock);
+ trace_nfs4_cb_offload(&args->coa_fh, &args->coa_stateid,
+ args->wr_count, args->error,
+ args->wr_writeverf.committed);
return 0;
}
#endif /* CONFIG_NFS_V4_2 */
* svc_process_common() looks for an XDR encoder to know when
* not to drop a Reply.
*/
-static int nfs4_encode_void(struct svc_rqst *rqstp, __be32 *p)
+static bool nfs4_encode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return xdr_ressize_check(rqstp, p);
+ return true;
}
static __be32 decode_string(struct xdr_stream *xdr, unsigned int *len,
/*
* Probe filesystem information, including the FSID on v2/v3
*/
-int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, struct nfs_fattr *fattr)
+static int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, struct nfs_fattr *fattr)
{
struct nfs_fsinfo fsinfo;
struct nfs_client *clp = server->nfs_client;
return 0;
}
-EXPORT_SYMBOL_GPL(nfs_probe_fsinfo);
+
+/*
+ * Grab the destination's particulars, including lease expiry time.
+ *
+ * Returns zero if probe succeeded and retrieved FSID matches the FSID
+ * we have cached.
+ */
+int nfs_probe_server(struct nfs_server *server, struct nfs_fh *mntfh)
+{
+ struct nfs_fattr *fattr;
+ int error;
+
+ fattr = nfs_alloc_fattr();
+ if (fattr == NULL)
+ return -ENOMEM;
+
+ /* Sanity: the probe won't work if the destination server
+ * does not recognize the migrated FH. */
+ error = nfs_probe_fsinfo(server, mntfh, fattr);
+
+ nfs_free_fattr(fattr);
+ return error;
+}
+EXPORT_SYMBOL_GPL(nfs_probe_server);
/*
* Copy useful information when duplicating a server record
if (!(fattr->valid & NFS_ATTR_FATTR)) {
error = ctx->nfs_mod->rpc_ops->getattr(server, ctx->mntfh,
- fattr, NULL, NULL);
+ fattr, NULL);
if (error < 0) {
dprintk("nfs_create_server: getattr error = %d\n", -error);
goto error;
rpc_authflavor_t flavor)
{
struct nfs_server *server;
- struct nfs_fattr *fattr_fsinfo;
int error;
server = nfs_alloc_server();
server->cred = get_cred(source->cred);
- error = -ENOMEM;
- fattr_fsinfo = nfs_alloc_fattr();
- if (fattr_fsinfo == NULL)
- goto out_free_server;
-
/* Copy data from the source */
server->nfs_client = source->nfs_client;
server->destroy = source->destroy;
goto out_free_server;
/* probe the filesystem info for this server filesystem */
- error = nfs_probe_fsinfo(server, fh, fattr_fsinfo);
+ error = nfs_probe_server(server, fh);
if (error < 0)
goto out_free_server;
nfs_server_insert_lists(server);
server->mount_time = jiffies;
- nfs_free_fattr(fattr_fsinfo);
return server;
out_free_server:
- nfs_free_fattr(fattr_fsinfo);
nfs_free_server(server);
return ERR_PTR(error);
}
struct nfs_delegation *delegation;
delegation = nfs_start_delegation_return(nfsi);
- /* Synchronous recall of any application leases */
- break_lease(inode, O_WRONLY | O_RDWR);
- nfs_wb_all(inode);
- if (delegation != NULL)
+ if (delegation != NULL) {
+ /* Synchronous recall of any application leases */
+ break_lease(inode, O_WRONLY | O_RDWR);
+ if (S_ISREG(inode->i_mode))
+ nfs_wb_all(inode);
return nfs_end_delegation_return(inode, delegation, 1);
+ }
return 0;
}
ctx->attr_gencount = nfsi->attr_gencount;
ctx->dir_cookie = 0;
ctx->dup_cookie = 0;
+ ctx->page_index = 0;
spin_lock(&dir->i_lock);
if (list_empty(&nfsi->open_files) &&
(nfsi->cache_validity & NFS_INO_DATA_INVAL_DEFER))
NFS_INO_INVALID_DATA |
NFS_INO_REVAL_FORCED);
list_add(&ctx->list, &nfsi->open_files);
+ clear_bit(NFS_INO_FORCE_READDIR, &nfsi->flags);
spin_unlock(&dir->i_lock);
return ctx;
}
static bool
nfs_readdir_inode_mapping_valid(struct nfs_inode *nfsi)
{
- if (nfsi->cache_validity & (NFS_INO_INVALID_ATTR|NFS_INO_INVALID_DATA))
+ if (nfsi->cache_validity & (NFS_INO_INVALID_CHANGE |
+ NFS_INO_INVALID_DATA))
return false;
smp_rmb();
return !test_bit(NFS_INO_INVALIDATING, &nfsi->flags);
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
!list_empty(&nfsi->open_files)) {
set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
- invalidate_mapping_pages(dir->i_mapping,
- nfsi->page_index + 1, -1);
+ set_bit(NFS_INO_FORCE_READDIR, &nfsi->flags);
}
}
nfs_set_verifier(dentry, dir_verifier);
status = nfs_refresh_inode(d_inode(dentry), entry->fattr);
if (!status)
- nfs_setsecurity(d_inode(dentry), entry->fattr, entry->label);
+ nfs_setsecurity(d_inode(dentry), entry->fattr);
goto out;
} else {
d_invalidate(dentry);
goto out;
}
- inode = nfs_fhget(dentry->d_sb, entry->fh, entry->fattr, entry->label);
+ inode = nfs_fhget(dentry->d_sb, entry->fh, entry->fattr);
alias = d_splice_alias(inode, dentry);
d_lookup_done(dentry);
if (alias) {
xdr_set_scratch_page(&stream, scratch);
do {
- if (entry->label)
- entry->label->len = NFS4_MAXLABELLEN;
+ if (entry->fattr->label)
+ entry->fattr->label->len = NFS4_MAXLABELLEN;
status = xdr_decode(desc, entry, &stream);
if (status != 0)
return -ENOMEM;
entry->cookie = nfs_readdir_page_last_cookie(page);
entry->fh = nfs_alloc_fhandle();
- entry->fattr = nfs_alloc_fattr();
+ entry->fattr = nfs_alloc_fattr_with_label(NFS_SERVER(inode));
entry->server = NFS_SERVER(inode);
if (entry->fh == NULL || entry->fattr == NULL)
goto out;
- entry->label = nfs4_label_alloc(NFS_SERVER(inode), GFP_NOWAIT);
- if (IS_ERR(entry->label)) {
- status = PTR_ERR(entry->label);
- goto out;
- }
-
array_size = (dtsize + PAGE_SIZE - 1) >> PAGE_SHIFT;
pages = nfs_readdir_alloc_pages(array_size);
if (!pages)
- goto out_release_label;
+ goto out;
do {
unsigned int pglen;
} while (!status && nfs_readdir_page_needs_filling(page));
nfs_readdir_free_pages(pages, array_size);
-out_release_label:
- nfs4_label_free(entry->label);
out:
nfs_free_fattr(entry->fattr);
nfs_free_fhandle(entry->fh);
sizeof(nfsi->cookieverf));
}
res = nfs_readdir_search_array(desc);
- if (res == 0) {
- nfsi->page_index = desc->page_index;
+ if (res == 0)
return 0;
- }
nfs_readdir_page_unlock_and_put_cached(desc);
return res;
}
struct nfs_inode *nfsi = NFS_I(inode);
struct nfs_open_dir_context *dir_ctx = file->private_data;
struct nfs_readdir_descriptor *desc;
+ pgoff_t page_index;
int res;
dfprintk(FILE, "NFS: readdir(%pD2) starting at cookie %llu\n",
desc->dir_cookie = dir_ctx->dir_cookie;
desc->dup_cookie = dir_ctx->dup_cookie;
desc->duped = dir_ctx->duped;
+ page_index = dir_ctx->page_index;
desc->attr_gencount = dir_ctx->attr_gencount;
memcpy(desc->verf, dir_ctx->verf, sizeof(desc->verf));
spin_unlock(&file->f_lock);
+ if (test_and_clear_bit(NFS_INO_FORCE_READDIR, &nfsi->flags) &&
+ list_is_singular(&nfsi->open_files))
+ invalidate_mapping_pages(inode->i_mapping, page_index + 1, -1);
+
do {
res = readdir_search_pagecache(desc);
dir_ctx->dup_cookie = desc->dup_cookie;
dir_ctx->duped = desc->duped;
dir_ctx->attr_gencount = desc->attr_gencount;
+ dir_ctx->page_index = desc->page_index;
memcpy(dir_ctx->verf, desc->verf, sizeof(dir_ctx->verf));
spin_unlock(&file->f_lock);
static void nfs_set_verifier_locked(struct dentry *dentry, unsigned long verf)
{
struct inode *inode = d_inode(dentry);
+ struct inode *dir = d_inode(dentry->d_parent);
- if (!nfs_verifier_is_delegated(dentry) &&
- !nfs_verify_change_attribute(d_inode(dentry->d_parent), verf))
- goto out;
+ if (!nfs_verify_change_attribute(dir, verf))
+ return;
if (inode && NFS_PROTO(inode)->have_delegation(inode, FMODE_READ))
nfs_set_verifier_delegated(&verf);
-out:
dentry->d_time = verf;
}
static void nfs_mark_dir_for_revalidate(struct inode *inode)
{
spin_lock(&inode->i_lock);
- nfs_set_cache_invalid(inode, NFS_INO_REVAL_PAGECACHE);
+ nfs_set_cache_invalid(inode, NFS_INO_INVALID_CHANGE);
spin_unlock(&inode->i_lock);
}
{
struct nfs_fh *fhandle;
struct nfs_fattr *fattr;
- struct nfs4_label *label;
unsigned long dir_verifier;
int ret;
ret = -ENOMEM;
fhandle = nfs_alloc_fhandle();
- fattr = nfs_alloc_fattr();
- label = nfs4_label_alloc(NFS_SERVER(inode), GFP_KERNEL);
- if (fhandle == NULL || fattr == NULL || IS_ERR(label))
+ fattr = nfs_alloc_fattr_with_label(NFS_SERVER(inode));
+ if (fhandle == NULL || fattr == NULL)
goto out;
dir_verifier = nfs_save_change_attribute(dir);
- ret = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr, label);
+ ret = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr);
if (ret < 0) {
switch (ret) {
case -ESTALE:
if (nfs_refresh_inode(inode, fattr) < 0)
goto out;
- nfs_setsecurity(inode, fattr, label);
+ nfs_setsecurity(inode, fattr);
nfs_set_verifier(dentry, dir_verifier);
/* set a readdirplus hint that we had a cache miss */
out:
nfs_free_fattr(fattr);
nfs_free_fhandle(fhandle);
- nfs4_label_free(label);
/*
* If the lookup failed despite the dentry change attribute being
*/
static void nfs_dentry_iput(struct dentry *dentry, struct inode *inode)
{
- if (S_ISDIR(inode->i_mode))
- /* drop any readdir cache as it could easily be old */
- nfs_set_cache_invalid(inode, NFS_INO_INVALID_DATA);
-
if (dentry->d_flags & DCACHE_NFSFS_RENAMED) {
nfs_complete_unlink(dentry, inode);
nfs_drop_nlink(inode);
struct inode *inode = NULL;
struct nfs_fh *fhandle = NULL;
struct nfs_fattr *fattr = NULL;
- struct nfs4_label *label = NULL;
unsigned long dir_verifier;
int error;
res = ERR_PTR(-ENOMEM);
fhandle = nfs_alloc_fhandle();
- fattr = nfs_alloc_fattr();
+ fattr = nfs_alloc_fattr_with_label(NFS_SERVER(dir));
if (fhandle == NULL || fattr == NULL)
goto out;
- label = nfs4_label_alloc(NFS_SERVER(dir), GFP_NOWAIT);
- if (IS_ERR(label))
- goto out;
-
dir_verifier = nfs_save_change_attribute(dir);
trace_nfs_lookup_enter(dir, dentry, flags);
- error = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr, label);
+ error = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr);
if (error == -ENOENT)
goto no_entry;
if (error < 0) {
res = ERR_PTR(error);
- goto out_label;
+ goto out;
}
- inode = nfs_fhget(dentry->d_sb, fhandle, fattr, label);
+ inode = nfs_fhget(dentry->d_sb, fhandle, fattr);
res = ERR_CAST(inode);
if (IS_ERR(res))
- goto out_label;
+ goto out;
/* Notify readdir to use READDIRPLUS */
nfs_force_use_readdirplus(dir);
res = d_splice_alias(inode, dentry);
if (res != NULL) {
if (IS_ERR(res))
- goto out_label;
+ goto out;
dentry = res;
}
nfs_set_verifier(dentry, dir_verifier);
-out_label:
- trace_nfs_lookup_exit(dir, dentry, flags, error);
- nfs4_label_free(label);
out:
+ trace_nfs_lookup_exit(dir, dentry, flags, PTR_ERR_OR_ZERO(res));
nfs_free_fattr(fattr);
nfs_free_fhandle(fhandle);
return res;
struct dentry *
nfs_add_or_obtain(struct dentry *dentry, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
struct dentry *parent = dget_parent(dentry);
struct inode *dir = d_inode(parent);
d_drop(dentry);
if (fhandle->size == 0) {
- error = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr, NULL);
+ error = NFS_PROTO(dir)->lookup(dir, dentry, fhandle, fattr);
if (error)
goto out_error;
}
if (!(fattr->valid & NFS_ATTR_FATTR)) {
struct nfs_server *server = NFS_SB(dentry->d_sb);
error = server->nfs_client->rpc_ops->getattr(server, fhandle,
- fattr, NULL, NULL);
+ fattr, NULL);
if (error < 0)
goto out_error;
}
- inode = nfs_fhget(dentry->d_sb, fhandle, fattr, label);
+ inode = nfs_fhget(dentry->d_sb, fhandle, fattr);
d = d_splice_alias(inode, dentry);
out:
dput(parent);
* Code common to create, mkdir, and mknod.
*/
int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
struct dentry *d;
- d = nfs_add_or_obtain(dentry, fhandle, fattr, label);
+ d = nfs_add_or_obtain(dentry, fhandle, fattr);
if (IS_ERR(d))
return PTR_ERR(d);
d_delete(dentry);
}
+static void nfs_dentry_remove_handle_error(struct inode *dir,
+ struct dentry *dentry, int error)
+{
+ switch (error) {
+ case -ENOENT:
+ d_delete(dentry);
+ fallthrough;
+ case 0:
+ nfs_set_verifier(dentry, nfs_save_change_attribute(dir));
+ }
+}
+
int nfs_rmdir(struct inode *dir, struct dentry *dentry)
{
int error;
up_write(&NFS_I(d_inode(dentry))->rmdir_sem);
} else
error = NFS_PROTO(dir)->rmdir(dir, &dentry->d_name);
+ nfs_dentry_remove_handle_error(dir, dentry, error);
trace_nfs_rmdir_exit(dir, dentry, error);
return error;
}
spin_unlock(&dentry->d_lock);
error = nfs_safe_remove(dentry);
- if (!error || error == -ENOENT) {
- nfs_set_verifier(dentry, nfs_save_change_attribute(dir));
- } else if (need_rehash)
+ nfs_dentry_remove_handle_error(dir, dentry, error);
+ if (need_rehash)
d_rehash(dentry);
out:
trace_nfs_unlink_exit(dir, dentry, error);
return error;
}
+ nfs_set_verifier(dentry, nfs_save_change_attribute(dir));
+
/*
* No big deal if we can't add this page to the page cache here.
* READLINK will get the missing page from the server if needed.
d_drop(dentry);
error = NFS_PROTO(dir)->link(inode, dir, &dentry->d_name);
if (error == 0) {
+ nfs_set_verifier(dentry, nfs_save_change_attribute(dir));
ihold(inode);
d_add(dentry, inode);
}
nfs_unlock_and_release_request(req);
}
- if (atomic_dec_and_test(&cinfo.mds->rpcs_out))
+ if (nfs_commit_end(cinfo.mds))
nfs_direct_write_complete(dreq);
}
nfs_fh_to_dentry(struct super_block *sb, struct fid *fid,
int fh_len, int fh_type)
{
- struct nfs4_label *label = NULL;
struct nfs_fattr *fattr = NULL;
struct nfs_fh *server_fh = nfs_exp_embedfh(fid->raw);
size_t fh_size = offsetof(struct nfs_fh, data) + server_fh->size;
if (fh_len < len || fh_type != len)
return NULL;
- fattr = nfs_alloc_fattr();
+ fattr = nfs_alloc_fattr_with_label(NFS_SB(sb));
if (fattr == NULL) {
dentry = ERR_PTR(-ENOMEM);
goto out;
if (inode)
goto out_found;
- label = nfs4_label_alloc(NFS_SB(sb), GFP_KERNEL);
- if (IS_ERR(label)) {
- dentry = ERR_CAST(label);
- goto out_free_fattr;
- }
-
rpc_ops = NFS_SB(sb)->nfs_client->rpc_ops;
- ret = rpc_ops->getattr(NFS_SB(sb), server_fh, fattr, label, NULL);
+ ret = rpc_ops->getattr(NFS_SB(sb), server_fh, fattr, NULL);
if (ret) {
dprintk("%s: getattr failed %d\n", __func__, ret);
trace_nfs_fh_to_dentry(sb, server_fh, fattr->fileid, ret);
dentry = ERR_PTR(ret);
- goto out_free_label;
+ goto out_free_fattr;
}
- inode = nfs_fhget(sb, server_fh, fattr, label);
+ inode = nfs_fhget(sb, server_fh, fattr);
out_found:
dentry = d_obtain_alias(inode);
-
-out_free_label:
- nfs4_label_free(label);
out_free_fattr:
nfs_free_fattr(fattr);
out:
struct super_block *sb = inode->i_sb;
struct nfs_server *server = NFS_SB(sb);
struct nfs_fattr *fattr = NULL;
- struct nfs4_label *label = NULL;
struct dentry *parent;
struct nfs_rpc_ops const *ops = server->nfs_client->rpc_ops;
struct nfs_fh fh;
if (!ops->lookupp)
return ERR_PTR(-EACCES);
- fattr = nfs_alloc_fattr();
- if (fattr == NULL) {
- parent = ERR_PTR(-ENOMEM);
- goto out;
- }
+ fattr = nfs_alloc_fattr_with_label(server);
+ if (fattr == NULL)
+ return ERR_PTR(-ENOMEM);
- label = nfs4_label_alloc(server, GFP_KERNEL);
- if (IS_ERR(label)) {
- parent = ERR_CAST(label);
- goto out_free_fattr;
- }
-
- ret = ops->lookupp(inode, &fh, fattr, label);
+ ret = ops->lookupp(inode, &fh, fattr);
if (ret) {
parent = ERR_PTR(ret);
- goto out_free_label;
+ goto out;
}
- pinode = nfs_fhget(sb, &fh, fattr, label);
+ pinode = nfs_fhget(sb, &fh, fattr);
parent = d_obtain_alias(pinode);
-out_free_label:
- nfs4_label_free(label);
-out_free_fattr:
- nfs_free_fattr(fattr);
out:
+ nfs_free_fattr(fattr);
return parent;
}
{
struct nfs_pgio_header *hdr = data;
- dprintk("--> %s task->tk_status %d\n", __func__, task->tk_status);
-
if (test_bit(NFS_IOHDR_REDO, &hdr->flags) &&
task->tk_status == 0) {
nfs41_sequence_done(task, &hdr->res.seq_res);
{
struct nfs_pgio_header *hdr = data;
- dprintk("--> %s task->tk_status %d\n", __func__, task->tk_status);
-
if (test_bit(NFS_IOHDR_REDO, &hdr->flags) &&
task->tk_status == 0) {
nfs4_sequence_done(task, &hdr->res.seq_res);
goto noconnect;
ds = mirror->mirror_ds->ds;
+ if (READ_ONCE(ds->ds_clp))
+ goto out;
/* matching smp_wmb() in _nfs4_pnfs_v3/4_ds_connect */
smp_rmb();
- if (ds->ds_clp)
- goto out;
/* FIXME: For now we assume the server sent only one version of NFS
* to use for the DS.
goto out;
/* get the actual root for this mount */
- fsinfo.fattr = nfs_alloc_fattr();
+ fsinfo.fattr = nfs_alloc_fattr_with_label(server);
if (fsinfo.fattr == NULL)
goto out_name;
- fsinfo.fattr->label = nfs4_label_alloc(server, GFP_KERNEL);
- if (IS_ERR(fsinfo.fattr->label))
- goto out_fattr;
error = server->nfs_client->rpc_ops->getroot(server, ctx->mntfh, &fsinfo);
if (error < 0) {
dprintk("nfs_get_root: getattr error = %d\n", -error);
nfs_errorf(fc, "NFS: Couldn't getattr on root");
- goto out_label;
+ goto out_fattr;
}
- inode = nfs_fhget(s, ctx->mntfh, fsinfo.fattr, NULL);
+ inode = nfs_fhget(s, ctx->mntfh, fsinfo.fattr);
if (IS_ERR(inode)) {
dprintk("nfs_get_root: get root inode failed\n");
error = PTR_ERR(inode);
nfs_errorf(fc, "NFS: Couldn't get root inode");
- goto out_label;
+ goto out_fattr;
}
error = nfs_superblock_set_dummy_root(s, inode);
if (error != 0)
- goto out_label;
+ goto out_fattr;
/* root dentries normally start off anonymous and get spliced in later
* if the dentry tree reaches them; however if the dentry already
dprintk("nfs_get_root: get root dentry failed\n");
error = PTR_ERR(root);
nfs_errorf(fc, "NFS: Couldn't get root dentry");
- goto out_label;
+ goto out_fattr;
}
security_d_instantiate(root, inode);
!(kflags_out & SECURITY_LSM_NATIVE_LABELS))
server->caps &= ~NFS_CAP_SECURITY_LABEL;
- nfs_setsecurity(inode, fsinfo.fattr, fsinfo.fattr->label);
+ nfs_setsecurity(inode, fsinfo.fattr);
error = 0;
-out_label:
- nfs4_label_free(fsinfo.fattr->label);
out_fattr:
nfs_free_fattr(fsinfo.fattr);
out_name:
error_splat_root:
dput(fc->root);
fc->root = NULL;
- goto out_label;
+ goto out_fattr;
}
flags &= ~NFS_INO_INVALID_XATTR;
if (flags & NFS_INO_INVALID_DATA)
nfs_fscache_invalidate(inode);
- if (inode->i_mapping->nrpages == 0)
- flags &= ~(NFS_INO_INVALID_DATA|NFS_INO_DATA_INVAL_DEFER);
flags &= ~(NFS_INO_REVAL_PAGECACHE | NFS_INO_REVAL_FORCED);
+
nfsi->cache_validity |= flags;
+
+ if (inode->i_mapping->nrpages == 0)
+ nfsi->cache_validity &= ~(NFS_INO_INVALID_DATA |
+ NFS_INO_DATA_INVAL_DEFER);
+ else if (nfsi->cache_validity & NFS_INO_INVALID_DATA)
+ nfsi->cache_validity &= ~NFS_INO_DATA_INVAL_DEFER;
}
EXPORT_SYMBOL_GPL(nfs_set_cache_invalid);
spin_unlock(&inode->i_lock);
}
-void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr)
{
int error;
- if (label == NULL)
+ if (fattr->label == NULL)
return;
if ((fattr->valid & NFS_ATTR_FATTR_V4_SECURITY_LABEL) && inode->i_security) {
- error = security_inode_notifysecctx(inode, label->label,
- label->len);
+ error = security_inode_notifysecctx(inode, fattr->label->label,
+ fattr->label->len);
if (error)
printk(KERN_ERR "%s() %s %d "
"security_inode_notifysecctx() %d\n",
__func__,
- (char *)label->label,
- label->len, error);
+ (char *)fattr->label->label,
+ fattr->label->len, error);
nfs_clear_label_invalid(inode);
}
}
struct nfs4_label *nfs4_label_alloc(struct nfs_server *server, gfp_t flags)
{
- struct nfs4_label *label = NULL;
- int minor_version = server->nfs_client->cl_minorversion;
-
- if (minor_version < 2)
- return label;
+ struct nfs4_label *label;
if (!(server->caps & NFS_CAP_SECURITY_LABEL))
- return label;
+ return NULL;
label = kzalloc(sizeof(struct nfs4_label), flags);
if (label == NULL)
}
EXPORT_SYMBOL_GPL(nfs4_label_alloc);
#else
-void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr)
{
}
#endif
return inode;
}
+static void nfs_inode_init_regular(struct nfs_inode *nfsi)
+{
+ atomic_long_set(&nfsi->nrequests, 0);
+ INIT_LIST_HEAD(&nfsi->commit_info.list);
+ atomic_long_set(&nfsi->commit_info.ncommit, 0);
+ atomic_set(&nfsi->commit_info.rpcs_out, 0);
+ mutex_init(&nfsi->commit_mutex);
+}
+
+static void nfs_inode_init_dir(struct nfs_inode *nfsi)
+{
+ nfsi->cache_change_attribute = 0;
+ memset(nfsi->cookieverf, 0, sizeof(nfsi->cookieverf));
+ init_rwsem(&nfsi->rmdir_sem);
+}
+
/*
* This is our front-end to iget that looks up inodes by file handle
* instead of inode number.
*/
struct inode *
-nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr, struct nfs4_label *label)
+nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr)
{
struct nfs_find_desc desc = {
.fh = fh,
if (S_ISREG(inode->i_mode)) {
inode->i_fop = NFS_SB(sb)->nfs_client->rpc_ops->file_ops;
inode->i_data.a_ops = &nfs_file_aops;
+ nfs_inode_init_regular(nfsi);
} else if (S_ISDIR(inode->i_mode)) {
inode->i_op = NFS_SB(sb)->nfs_client->rpc_ops->dir_inode_ops;
inode->i_fop = &nfs_dir_operations;
inode->i_data.a_ops = &nfs_dir_aops;
+ nfs_inode_init_dir(nfsi);
/* Deal with crossing mountpoints */
if (fattr->valid & NFS_ATTR_FATTR_MOUNTPOINT ||
fattr->valid & NFS_ATTR_FATTR_V4_REFERRAL) {
inode->i_uid = make_kuid(&init_user_ns, -2);
inode->i_gid = make_kgid(&init_user_ns, -2);
inode->i_blocks = 0;
- memset(nfsi->cookieverf, 0, sizeof(nfsi->cookieverf));
nfsi->write_io = 0;
nfsi->read_io = 0;
fattr->size != 0)
nfs_set_cache_invalid(inode, NFS_INO_INVALID_BLOCKS);
- nfs_setsecurity(inode, fattr, label);
+ nfs_setsecurity(inode, fattr);
nfsi->attrtimeo = NFS_MINATTRTIMEO(inode);
nfsi->attrtimeo_timestamp = now;
if (S_ISREG(inode->i_mode))
nfs_sync_inode(inode);
- fattr = nfs_alloc_fattr();
+ fattr = nfs_alloc_fattr_with_label(NFS_SERVER(inode));
if (fattr == NULL) {
error = -ENOMEM;
goto out;
if (err)
goto out;
+ trace_nfs_size_truncate(inode, offset);
i_size_write(inode, offset);
/* Optimisation */
if (offset == 0)
ctx->cred = get_cred(filp->f_cred);
else
ctx->cred = get_current_cred();
- ctx->ll_cred = NULL;
+ rcu_assign_pointer(ctx->ll_cred, NULL);
ctx->state = NULL;
ctx->mode = f_mode;
ctx->flags = 0;
put_cred(ctx->cred);
dput(ctx->dentry);
nfs_sb_deactive(sb);
- put_rpccred(ctx->ll_cred);
+ put_rpccred(rcu_dereference_protected(ctx->ll_cred, 1));
kfree(ctx->mdsthreshold);
kfree_rcu(ctx, rcu_head);
}
__nfs_revalidate_inode(struct nfs_server *server, struct inode *inode)
{
int status = -ESTALE;
- struct nfs4_label *label = NULL;
struct nfs_fattr *fattr = NULL;
struct nfs_inode *nfsi = NFS_I(inode);
}
status = -ENOMEM;
- fattr = nfs_alloc_fattr();
+ fattr = nfs_alloc_fattr_with_label(NFS_SERVER(inode));
if (fattr == NULL)
goto out;
nfs_inc_stats(inode, NFSIOS_INODEREVALIDATE);
- label = nfs4_label_alloc(NFS_SERVER(inode), GFP_KERNEL);
- if (IS_ERR(label)) {
- status = PTR_ERR(label);
- goto out;
- }
-
- status = NFS_PROTO(inode)->getattr(server, NFS_FH(inode), fattr,
- label, inode);
+ status = NFS_PROTO(inode)->getattr(server, NFS_FH(inode), fattr, inode);
if (status != 0) {
dfprintk(PAGECACHE, "nfs_revalidate_inode: (%s/%Lu) getattr failed, error=%d\n",
inode->i_sb->s_id,
else
nfs_zap_caches(inode);
}
- goto err_out;
+ goto out;
}
status = nfs_refresh_inode(inode, fattr);
dfprintk(PAGECACHE, "nfs_revalidate_inode: (%s/%Lu) refresh failed, error=%d\n",
inode->i_sb->s_id,
(unsigned long long)NFS_FILEID(inode), status);
- goto err_out;
+ goto out;
}
if (nfsi->cache_validity & NFS_INO_INVALID_ACL)
nfs_zap_acl_cache(inode);
- nfs_setsecurity(inode, fattr, label);
+ nfs_setsecurity(inode, fattr);
dfprintk(PAGECACHE, "NFS: (%s/%Lu) revalidation complete\n",
inode->i_sb->s_id,
(unsigned long long)NFS_FILEID(inode));
-err_out:
- nfs4_label_free(label);
out:
nfs_free_fattr(fattr);
trace_nfs_revalidate_inode_exit(inode, status);
&& (fattr->valid & NFS_ATTR_FATTR_MTIME)
&& timespec64_equal(&ts, &fattr->pre_mtime)) {
inode->i_mtime = fattr->mtime;
- if (S_ISDIR(inode->i_mode))
- nfs_set_cache_invalid(inode, NFS_INO_INVALID_DATA);
}
if ((fattr->valid & NFS_ATTR_FATTR_PRESIZE)
&& (fattr->valid & NFS_ATTR_FATTR_SIZE)
&& i_size_read(inode) == nfs_size_to_loff_t(fattr->pre_size)
&& !nfs_have_writebacks(inode)) {
+ trace_nfs_size_wcc(inode, fattr->size);
i_size_write(inode, nfs_size_to_loff_t(fattr->size));
}
}
struct nfs_fattr *fattr;
fattr = kmalloc(sizeof(*fattr), GFP_NOFS);
- if (fattr != NULL)
+ if (fattr != NULL) {
nfs_fattr_init(fattr);
+ fattr->label = NULL;
+ }
return fattr;
}
EXPORT_SYMBOL_GPL(nfs_alloc_fattr);
+struct nfs_fattr *nfs_alloc_fattr_with_label(struct nfs_server *server)
+{
+ struct nfs_fattr *fattr = nfs_alloc_fattr();
+
+ if (!fattr)
+ return NULL;
+
+ fattr->label = nfs4_label_alloc(server, GFP_NOFS);
+ if (IS_ERR(fattr->label)) {
+ kfree(fattr);
+ return NULL;
+ }
+
+ return fattr;
+}
+EXPORT_SYMBOL_GPL(nfs_alloc_fattr_with_label);
+
struct nfs_fh *nfs_alloc_fhandle(void)
{
struct nfs_fh *fh;
NFS_INO_INVALID_BLOCKS | NFS_INO_INVALID_OTHER |
NFS_INO_INVALID_NLINK;
unsigned long cache_validity = NFS_I(inode)->cache_validity;
+ enum nfs4_change_attr_type ctype = NFS_SERVER(inode)->change_attr_type;
- if (!(cache_validity & NFS_INO_INVALID_CHANGE) &&
+ if (ctype != NFS4_CHANGE_TYPE_IS_UNDEFINED &&
+ !(cache_validity & NFS_INO_INVALID_CHANGE) &&
(cache_validity & check_valid) != 0 &&
(fattr->valid & NFS_ATTR_FATTR_CHANGE) != 0 &&
nfs_inode_attrs_cmp_monotonic(fattr, inode) == 0)
/* Do we perhaps have any outstanding writes, or has
* the file grown beyond our last write? */
if (!nfs_have_writebacks(inode) || new_isize > cur_isize) {
+ trace_nfs_size_update(inode, new_isize);
i_size_write(inode, new_isize);
if (!have_writers)
invalid |= NFS_INO_INVALID_DATA;
}
- dprintk("NFS: isize change on server for file %s/%ld "
- "(%Ld to %Ld)\n",
- inode->i_sb->s_id,
- inode->i_ino,
- (long long)cur_isize,
- (long long)new_isize);
}
if (new_isize == 0 &&
!(fattr->valid & (NFS_ATTR_FATTR_SPACE_USED |
save_cache_validity & NFS_INO_INVALID_OTHER;
if (fattr->valid & NFS_ATTR_FATTR_NLINK) {
- if (inode->i_nlink != fattr->nlink) {
- if (S_ISDIR(inode->i_mode))
- invalid |= NFS_INO_INVALID_DATA;
+ if (inode->i_nlink != fattr->nlink)
set_nlink(inode, fattr->nlink);
- }
} else if (fattr_supported & NFS_ATTR_FATTR_NLINK)
nfsi->cache_validity |=
save_cache_validity & NFS_INO_INVALID_NLINK;
INIT_LIST_HEAD(&nfsi->open_files);
INIT_LIST_HEAD(&nfsi->access_cache_entry_lru);
INIT_LIST_HEAD(&nfsi->access_cache_inode_lru);
- INIT_LIST_HEAD(&nfsi->commit_info.list);
- atomic_long_set(&nfsi->nrequests, 0);
- atomic_long_set(&nfsi->commit_info.ncommit, 0);
- atomic_set(&nfsi->commit_info.rpcs_out, 0);
- init_rwsem(&nfsi->rmdir_sem);
- mutex_init(&nfsi->commit_mutex);
nfs4_init_once(nfsi);
- nfsi->cache_change_attribute = 0;
}
static int __init nfs_init_inodecache(void)
extern struct nfs_client *nfs_alloc_client(const struct nfs_client_initdata *);
int nfs_create_rpc_client(struct nfs_client *, const struct nfs_client_initdata *, rpc_authflavor_t);
struct nfs_client *nfs_get_client(const struct nfs_client_initdata *);
-int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *, struct nfs_fattr *);
+int nfs_probe_server(struct nfs_server *, struct nfs_fh *);
void nfs_server_insert_lists(struct nfs_server *);
void nfs_server_remove_lists(struct nfs_server *);
void nfs_init_timeout_values(struct rpc_timeout *to, int proto, int timeo, int retrans);
nfs4_find_client_sessionid(struct net *, const struct sockaddr *,
struct nfs4_sessionid *, u32);
extern struct nfs_server *nfs_create_server(struct fs_context *);
+extern void nfs4_server_set_init_caps(struct nfs_server *);
extern struct nfs_server *nfs4_create_server(struct fs_context *);
extern struct nfs_server *nfs4_create_referral_server(struct fs_context *);
extern int nfs4_update_server(struct nfs_server *server, const char *hostname,
return dst;
}
-static inline void nfs4_label_free(struct nfs4_label *label)
-{
- if (label) {
- kfree(label->label);
- kfree(label);
- }
- return;
-}
static inline void nfs_zap_label_cache_locked(struct nfs_inode *nfsi)
{
}
#else
static inline struct nfs4_label *nfs4_label_alloc(struct nfs_server *server, gfp_t flags) { return NULL; }
-static inline void nfs4_label_free(void *label) {}
static inline void nfs_zap_label_cache_locked(struct nfs_inode *nfsi)
{
}
/* Look it up again to get its attributes */
err = server->nfs_client->rpc_ops->lookup(d_inode(parent), dentry,
- ctx->mntfh, ctx->clone_data.fattr,
- NULL);
+ ctx->mntfh, ctx->clone_data.fattr);
dput(parent);
if (err != 0)
return err;
*/
static int
nfs3_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label,
- struct inode *inode)
+ struct nfs_fattr *fattr, struct inode *inode)
{
struct rpc_message msg = {
.rpc_proc = &nfs3_procedures[NFS3PROC_GETATTR],
static int
nfs3_proc_lookup(struct inode *dir, struct dentry *dentry,
- struct nfs_fh *fhandle, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fh *fhandle, struct nfs_fattr *fattr)
{
unsigned short task_flags = 0;
}
static int nfs3_proc_lookupp(struct inode *inode, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
const char dotdot[] = "..";
const size_t len = strlen(dotdot);
if (status != 0)
return ERR_PTR(status);
- return nfs_add_or_obtain(dentry, data->res.fh, data->res.fattr, NULL);
+ return nfs_add_or_obtain(dentry, data->res.fh, data->res.fattr);
}
static void nfs3_free_createdata(struct nfs3_createdata *data)
/* ignore properties */
result->lease_time = 0;
- result->change_attr_type = NFS4_CHANGE_TYPE_IS_TIME_METADATA;
+ result->change_attr_type = NFS4_CHANGE_TYPE_IS_UNDEFINED;
return 0;
}
status = nfs_post_op_update_inode_force_wcc(inode,
res.falloc_fattr);
+ if (msg->rpc_proc == &nfs4_procedures[NFSPROC4_CLNT_ALLOCATE])
+ trace_nfs4_fallocate(inode, &args, status);
+ else
+ trace_nfs4_deallocate(inode, &args, status);
kfree(res.falloc_fattr);
return status;
}
status = nfs4_call_sync(dst_server->client, dst_server, &msg,
&args->seq_args, &res->seq_res, 0);
+ trace_nfs4_copy(src_inode, dst_inode, args, res, nss, status);
if (status == -ENOTSUPP)
dst_server->caps &= ~NFS_CAP_COPY;
if (status)
{
struct nfs42_offloadcancel_data *data = calldata;
+ trace_nfs4_offload_cancel(&data->args, task->tk_status);
nfs41_sequence_done(task, &data->res.osr_seq_res);
if (task->tk_status &&
nfs4_async_handle_error(task, data->seq_server, NULL,
status = nfs4_call_sync(src_server->client, src_server, &msg,
&args->cna_seq_args, &res->cnr_seq_res, 0);
+ trace_nfs4_copy_notify(file_inode(src), args, res, status);
if (status == -ENOTSUPP)
src_server->caps &= ~NFS_CAP_COPY_NOTIFY;
status = nfs4_call_sync(server->client, server, &msg,
&args.seq_args, &res.seq_res, 0);
+ trace_nfs4_llseek(inode, &args, &res, status);
if (status == -ENOTSUPP)
server->caps &= ~NFS_CAP_SEEK;
if (status)
status = nfs4_call_sync(server->client, server, msg,
&args.seq_args, &res.seq_res, 0);
+ trace_nfs4_clone(src_inode, dst_inode, &args, status);
if (status == 0) {
nfs42_copy_dest_done(dst_inode, dst_offset, count);
status = nfs_post_op_update_inode(dst_inode, res.dst_fattr);
struct nfs4_string group_name;
struct nfs4_label *a_label;
struct nfs_fattr f_attr;
- struct nfs4_label *f_label;
struct dentry *dir;
struct dentry *dentry;
struct nfs4_state_owner *owner;
const struct nfs_lock_context *l_ctx,
fmode_t fmode);
extern int nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label,
- struct inode *inode);
+ struct nfs_fattr *fattr, struct inode *inode);
extern int update_open_stateid(struct nfs4_state *state,
const nfs4_stateid *open_stateid,
const nfs4_stateid *deleg_stateid,
#endif
}
-static int nfs4_server_common_setup(struct nfs_server *server,
- struct nfs_fh *mntfh, bool auth_probe)
+void nfs4_server_set_init_caps(struct nfs_server *server)
{
- struct nfs_fattr *fattr;
- int error;
-
- /* data servers support only a subset of NFSv4.1 */
- if (is_ds_only_client(server->nfs_client))
- return -EPROTONOSUPPORT;
-
- fattr = nfs_alloc_fattr();
- if (fattr == NULL)
- return -ENOMEM;
-
- /* We must ensure the session is initialised first */
- error = nfs4_init_session(server->nfs_client);
- if (error < 0)
- goto out;
-
/* Set the basic capabilities */
server->caps |= server->nfs_client->cl_mvops->init_caps;
if (server->flags & NFS_MOUNT_NORDIRPLUS)
server->caps &= ~NFS_CAP_READDIRPLUS;
if (server->nfs_client->cl_proto == XPRT_TRANSPORT_RDMA)
server->caps &= ~NFS_CAP_READ_PLUS;
+
/*
* Don't use NFS uid/gid mapping if we're using AUTH_SYS or lower
* authentication.
if (nfs4_disable_idmapping &&
server->client->cl_auth->au_flavor == RPC_AUTH_UNIX)
server->caps |= NFS_CAP_UIDGID_NOMAP;
+}
+static int nfs4_server_common_setup(struct nfs_server *server,
+ struct nfs_fh *mntfh, bool auth_probe)
+{
+ int error;
+
+ /* data servers support only a subset of NFSv4.1 */
+ if (is_ds_only_client(server->nfs_client))
+ return -EPROTONOSUPPORT;
+
+ /* We must ensure the session is initialised first */
+ error = nfs4_init_session(server->nfs_client);
+ if (error < 0)
+ goto out;
+
+ nfs4_server_set_init_caps(server);
/* Probe the root fh to retrieve its FSID and filehandle */
error = nfs4_get_rootfh(server, mntfh, auth_probe);
(unsigned long long) server->fsid.minor);
nfs_display_fhandle(mntfh, "Pseudo-fs root FH");
- error = nfs_probe_fsinfo(server, mntfh, fattr);
+ error = nfs_probe_server(server, mntfh);
if (error < 0)
goto out;
server->mount_time = jiffies;
server->destroy = nfs4_destroy_server;
out:
- nfs_free_fattr(fattr);
return error;
}
return ERR_PTR(error);
}
-/*
- * Grab the destination's particulars, including lease expiry time.
- *
- * Returns zero if probe succeeded and retrieved FSID matches the FSID
- * we have cached.
- */
-static int nfs_probe_destination(struct nfs_server *server)
-{
- struct inode *inode = d_inode(server->super->s_root);
- struct nfs_fattr *fattr;
- int error;
-
- fattr = nfs_alloc_fattr();
- if (fattr == NULL)
- return -ENOMEM;
-
- /* Sanity: the probe won't work if the destination server
- * does not recognize the migrated FH. */
- error = nfs_probe_fsinfo(server, NFS_FH(inode), fattr);
-
- nfs_free_fattr(fattr);
- return error;
-}
-
/**
* nfs4_update_server - Move an nfs_server to a different nfs_client
*
server->nfs_client->cl_hostname = kstrdup(hostname, GFP_KERNEL);
nfs_server_insert_lists(server);
- return nfs_probe_destination(server);
+ return nfs_probe_server(server, NFS_FH(d_inode(server->super->s_root)));
}
static struct file *__nfs42_ssc_open(struct vfsmount *ss_mnt,
struct nfs_fh *src_fh, nfs4_stateid *stateid)
{
- struct nfs_fattr fattr;
+ struct nfs_fattr *fattr = nfs_alloc_fattr();
struct file *filep, *res;
struct nfs_server *server;
struct inode *r_ino = NULL;
server = NFS_SERVER(ss_mnt->mnt_root->d_inode);
- nfs_fattr_init(&fattr);
+ if (!fattr)
+ return ERR_PTR(-ENOMEM);
- status = nfs4_proc_getattr(server, src_fh, &fattr, NULL, NULL);
+ status = nfs4_proc_getattr(server, src_fh, fattr, NULL);
if (status < 0) {
res = ERR_PTR(status);
goto out;
goto out;
snprintf(read_name, len, SSC_READ_NAME_BODY, read_name_gen++);
- r_ino = nfs_fhget(ss_mnt->mnt_root->d_inode->i_sb, src_fh, &fattr,
- NULL);
+ r_ino = nfs_fhget(ss_mnt->mnt_root->d_inode->i_sb, src_fh, fattr);
if (IS_ERR(r_ino)) {
res = ERR_CAST(r_ino);
goto out_free_name;
}
- filep = alloc_file_pseudo(r_ino, ss_mnt, read_name, FMODE_READ,
+ filep = alloc_file_pseudo(r_ino, ss_mnt, read_name, O_RDONLY,
r_ino->i_fop);
if (IS_ERR(filep)) {
res = ERR_CAST(filep);
goto out_free_name;
}
- filep->f_mode |= FMODE_READ;
ctx = alloc_nfs_open_context(filep->f_path.dentry, filep->f_mode,
filep);
out_free_name:
kfree(read_name);
out:
+ nfs_free_fattr(fattr);
return res;
out_stateowner:
nfs4_put_state_owner(sp);
err_destroy_pipe:
rpc_destroy_pipe_data(idmap->idmap_pipe);
err:
- get_user_ns(idmap->user_ns);
+ put_user_ns(idmap->user_ns);
kfree(idmap);
return error;
}
static int _nfs4_recover_proc_open(struct nfs4_opendata *data);
static int nfs4_do_fsinfo(struct nfs_server *, struct nfs_fh *, struct nfs_fsinfo *);
static void nfs_fixup_referral_attributes(struct nfs_fattr *fattr);
-static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle, struct nfs_fattr *fattr, struct nfs4_label *label, struct inode *inode);
+static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
+ struct nfs_fattr *fattr, struct inode *inode);
static int nfs4_do_setattr(struct inode *inode, const struct cred *cred,
struct nfs_fattr *fattr, struct iattr *sattr,
- struct nfs_open_context *ctx, struct nfs4_label *ilabel,
- struct nfs4_label *olabel);
+ struct nfs_open_context *ctx, struct nfs4_label *ilabel);
#ifdef CONFIG_NFS_V4_1
static struct rpc_task *_nfs41_proc_sequence(struct nfs_client *clp,
const struct cred *cred,
static void nfs4_init_opendata_res(struct nfs4_opendata *p)
{
p->o_res.f_attr = &p->f_attr;
- p->o_res.f_label = p->f_label;
p->o_res.seqid = p->o_arg.seqid;
p->c_res.seqid = p->c_arg.seqid;
p->o_res.server = p->o_arg.server;
if (p == NULL)
goto err;
- p->f_label = nfs4_label_alloc(server, gfp_mask);
- if (IS_ERR(p->f_label))
+ p->f_attr.label = nfs4_label_alloc(server, gfp_mask);
+ if (IS_ERR(p->f_attr.label))
goto err_free_p;
p->a_label = nfs4_label_alloc(server, gfp_mask);
sizeof(p->o_arg.u.verifier.data));
}
}
- /* don't put an ACCESS op in OPEN compound if O_EXCL, because ACCESS
- * will return permission denied for all bits until close */
- if (!(flags & O_EXCL)) {
- /* ask server to check for all possible rights as results
- * are cached */
- switch (p->o_arg.claim) {
- default:
- break;
- case NFS4_OPEN_CLAIM_NULL:
- case NFS4_OPEN_CLAIM_FH:
- p->o_arg.access = NFS4_ACCESS_READ |
- NFS4_ACCESS_MODIFY |
- NFS4_ACCESS_EXTEND |
- NFS4_ACCESS_EXECUTE;
+ /* ask server to check for all possible rights as results
+ * are cached */
+ switch (p->o_arg.claim) {
+ default:
+ break;
+ case NFS4_OPEN_CLAIM_NULL:
+ case NFS4_OPEN_CLAIM_FH:
+ p->o_arg.access = NFS4_ACCESS_READ | NFS4_ACCESS_MODIFY |
+ NFS4_ACCESS_EXTEND | NFS4_ACCESS_DELETE |
+ NFS4_ACCESS_EXECUTE;
#ifdef CONFIG_NFS_V4_2
- if (server->caps & NFS_CAP_XATTR)
- p->o_arg.access |= NFS4_ACCESS_XAREAD |
- NFS4_ACCESS_XAWRITE |
- NFS4_ACCESS_XALIST;
+ if (!(server->caps & NFS_CAP_XATTR))
+ break;
+ p->o_arg.access |= NFS4_ACCESS_XAREAD | NFS4_ACCESS_XAWRITE |
+ NFS4_ACCESS_XALIST;
#endif
- }
}
p->o_arg.clientid = server->nfs_client->cl_clientid;
p->o_arg.id.create_time = ktime_to_ns(sp->so_seqid.create_time);
err_free_label:
nfs4_label_free(p->a_label);
err_free_f:
- nfs4_label_free(p->f_label);
+ nfs4_label_free(p->f_attr.label);
err_free_p:
kfree(p);
err:
nfs4_put_state_owner(p->owner);
nfs4_label_free(p->a_label);
- nfs4_label_free(p->f_label);
+ nfs4_label_free(p->f_attr.label);
dput(p->dir);
dput(p->dentry);
{
if (test_bit(NFS_OPEN_STATE, &state->flags)) {
/* The common case - we're updating to a new sequence number */
- if (nfs4_stateid_match_other(stateid, &state->open_stateid) &&
- nfs4_stateid_is_next(&state->open_stateid, stateid)) {
- return true;
+ if (nfs4_stateid_match_other(stateid, &state->open_stateid)) {
+ if (nfs4_stateid_is_next(&state->open_stateid, stateid))
+ return true;
+ return false;
}
- } else {
- /* This is the first OPEN in this generation */
- if (stateid->seqid == cpu_to_be32(1))
- return true;
+ /* The server returned a new stateid */
}
+ /* This is the first OPEN in this generation */
+ if (stateid->seqid == cpu_to_be32(1))
+ return true;
return false;
}
if (!(data->f_attr.valid & NFS_ATTR_FATTR))
return ERR_PTR(-EAGAIN);
inode = nfs_fhget(data->dir->d_sb, &data->o_res.fh,
- &data->f_attr, data->f_label);
+ &data->f_attr);
break;
default:
inode = d_inode(data->dentry);
/* Set the create mode (note dependency on the session type) */
data->o_arg.createmode = NFS4_CREATE_UNCHECKED;
if (data->o_arg.open_flags & O_EXCL) {
- data->o_arg.createmode = NFS4_CREATE_EXCLUSIVE;
- if (nfs4_has_persistent_session(clp))
+ data->o_arg.createmode = NFS4_CREATE_EXCLUSIVE4_1;
+ if (clp->cl_mvops->minor_version == 0) {
+ data->o_arg.createmode = NFS4_CREATE_EXCLUSIVE;
+ /* don't put an ACCESS op in OPEN compound if O_EXCL,
+ * because ACCESS will return permission denied for
+ * all bits until close */
+ data->o_res.access_request = data->o_arg.access = 0;
+ } else if (nfs4_has_persistent_session(clp))
data->o_arg.createmode = NFS4_CREATE_GUARDED;
- else if (clp->cl_mvops->minor_version > 0)
- data->o_arg.createmode = NFS4_CREATE_EXCLUSIVE4_1;
}
return;
unlock_no_action:
}
if (!(o_res->f_attr->valid & NFS_ATTR_FATTR)) {
nfs4_sequence_free_slot(&o_res->seq_res);
- nfs4_proc_getattr(server, &o_res->fh, o_res->f_attr,
- o_res->f_label, NULL);
+ nfs4_proc_getattr(server, &o_res->fh, o_res->f_attr, NULL);
}
return 0;
}
enum open_claim_type4 claim = NFS4_OPEN_CLAIM_NULL;
struct iattr *sattr = c->sattr;
struct nfs4_label *label = c->label;
- struct nfs4_label *olabel = NULL;
int status;
/* Protect against reboot recovery conflicts */
if (opendata == NULL)
goto err_put_state_owner;
- if (label) {
- olabel = nfs4_label_alloc(server, GFP_KERNEL);
- if (IS_ERR(olabel)) {
- status = PTR_ERR(olabel);
- goto err_opendata_put;
- }
- }
-
if (server->attr_bitmask[2] & FATTR4_WORD2_MDSTHRESHOLD) {
if (!opendata->f_attr.mdsthreshold) {
opendata->f_attr.mdsthreshold = pnfs_mdsthreshold_alloc();
if (!opendata->f_attr.mdsthreshold)
- goto err_free_label;
+ goto err_opendata_put;
}
opendata->o_arg.open_bitmap = &nfs4_pnfs_open_bitmap[0];
}
status = _nfs4_open_and_get_state(opendata, flags, ctx);
if (status != 0)
- goto err_free_label;
+ goto err_opendata_put;
state = ctx->state;
if ((opendata->o_arg.open_flags & (O_CREAT|O_EXCL)) == (O_CREAT|O_EXCL) &&
nfs_fattr_init(opendata->o_res.f_attr);
status = nfs4_do_setattr(state->inode, cred,
opendata->o_res.f_attr, sattr,
- ctx, label, olabel);
+ ctx, label);
if (status == 0) {
nfs_setattr_update_inode(state->inode, sattr,
opendata->o_res.f_attr);
- nfs_setsecurity(state->inode, opendata->o_res.f_attr, olabel);
+ nfs_setsecurity(state->inode, opendata->o_res.f_attr);
}
sattr->ia_valid = ia_old;
}
opendata->f_attr.mdsthreshold = NULL;
}
- nfs4_label_free(olabel);
-
nfs4_opendata_put(opendata);
nfs4_put_state_owner(sp);
return 0;
-err_free_label:
- nfs4_label_free(olabel);
err_opendata_put:
nfs4_opendata_put(opendata);
err_put_state_owner:
static int nfs4_do_setattr(struct inode *inode, const struct cred *cred,
struct nfs_fattr *fattr, struct iattr *sattr,
- struct nfs_open_context *ctx, struct nfs4_label *ilabel,
- struct nfs4_label *olabel)
+ struct nfs_open_context *ctx, struct nfs4_label *ilabel)
{
struct nfs_server *server = NFS_SERVER(inode);
__u32 bitmask[NFS4_BITMASK_SZ];
};
struct nfs_setattrres res = {
.fattr = fattr,
- .label = olabel,
.server = server,
};
struct nfs4_exception exception = {
adjust_flags |= NFS_INO_INVALID_OTHER;
do {
- nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, olabel),
+ nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, fattr->label),
inode, adjust_flags);
err = _nfs4_do_setattr(inode, &arg, &res, cred, ctx);
.stateid = &calldata->arg.stateid,
};
- dprintk("%s: begin!\n", __func__);
if (!nfs4_sequence_done(task, &calldata->res.seq_res))
return;
trace_nfs4_close(state, &calldata->arg, &calldata->res, task->tk_status);
task->tk_status = 0;
nfs_release_seqid(calldata->arg.seqid);
nfs_refresh_inode(calldata->inode, &calldata->fattr);
- dprintk("%s: done, ret = %d!\n", __func__, task->tk_status);
+ dprintk("%s: ret = %d\n", __func__, task->tk_status);
return;
out_restart:
task->tk_status = 0;
bool is_rdonly, is_wronly, is_rdwr;
int call_close = 0;
- dprintk("%s: begin!\n", __func__);
if (nfs_wait_on_sequence(calldata->arg.seqid, task) != 0)
goto out_wait;
&calldata->res.seq_res,
task) != 0)
nfs_release_seqid(calldata->arg.seqid);
- dprintk("%s: done!\n", __func__);
return;
out_no_action:
task->tk_action = NULL;
.interruptible = true,
};
int err;
+
+ nfs4_server_set_init_caps(server);
do {
err = nfs4_handle_exception(server,
_nfs4_server_capabilities(server, fhandle),
{
int error;
struct nfs_fattr *fattr = info->fattr;
- struct nfs4_label *label = fattr->label;
error = nfs4_server_capabilities(server, mntfh);
if (error < 0) {
return error;
}
- error = nfs4_proc_getattr(server, mntfh, fattr, label, NULL);
+ error = nfs4_proc_getattr(server, mntfh, fattr, NULL);
if (error < 0) {
dprintk("nfs4_get_root: getattr error = %d\n", -error);
goto out;
}
static int _nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label,
- struct inode *inode)
+ struct nfs_fattr *fattr, struct inode *inode)
{
__u32 bitmask[NFS4_BITMASK_SZ];
struct nfs4_getattr_arg args = {
};
struct nfs4_getattr_res res = {
.fattr = fattr,
- .label = label,
.server = server,
};
struct rpc_message msg = {
if (inode && (server->flags & NFS_MOUNT_SOFTREVAL))
task_flags |= RPC_TASK_TIMEOUT;
- nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, label), inode, 0);
+ nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, fattr->label), inode, 0);
nfs_fattr_init(fattr);
nfs4_init_sequence(&args.seq_args, &res.seq_res, 0, 0);
return nfs4_do_call_sync(server->client, server, &msg,
}
int nfs4_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label,
- struct inode *inode)
+ struct nfs_fattr *fattr, struct inode *inode)
{
struct nfs4_exception exception = {
.interruptible = true,
};
int err;
do {
- err = _nfs4_proc_getattr(server, fhandle, fattr, label, inode);
+ err = _nfs4_proc_getattr(server, fhandle, fattr, inode);
trace_nfs4_getattr(server, fhandle, fattr, err);
err = nfs4_handle_exception(server, err,
&exception);
struct inode *inode = d_inode(dentry);
const struct cred *cred = NULL;
struct nfs_open_context *ctx = NULL;
- struct nfs4_label *label = NULL;
int status;
if (pnfs_ld_layoutret_on_setattr(inode) &&
cred = ctx->cred;
}
- label = nfs4_label_alloc(NFS_SERVER(inode), GFP_KERNEL);
- if (IS_ERR(label))
- return PTR_ERR(label);
-
/* Return any delegations if we're going to change ACLs */
if ((sattr->ia_valid & (ATTR_MODE|ATTR_UID|ATTR_GID)) != 0)
nfs4_inode_make_writeable(inode);
- status = nfs4_do_setattr(inode, cred, fattr, sattr, ctx, NULL, label);
+ status = nfs4_do_setattr(inode, cred, fattr, sattr, ctx, NULL);
if (status == 0) {
nfs_setattr_update_inode(inode, sattr, fattr);
- nfs_setsecurity(inode, fattr, label);
+ nfs_setsecurity(inode, fattr);
}
- nfs4_label_free(label);
return status;
}
static int _nfs4_proc_lookup(struct rpc_clnt *clnt, struct inode *dir,
struct dentry *dentry, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
struct nfs_server *server = NFS_SERVER(dir);
int status;
struct nfs4_lookup_res res = {
.server = server,
.fattr = fattr,
- .label = label,
.fh = fhandle,
};
struct rpc_message msg = {
if (nfs_lookup_is_soft_revalidate(dentry))
task_flags |= RPC_TASK_TIMEOUT;
- args.bitmask = nfs4_bitmask(server, label);
+ args.bitmask = nfs4_bitmask(server, fattr->label);
nfs_fattr_init(fattr);
static int nfs4_proc_lookup_common(struct rpc_clnt **clnt, struct inode *dir,
struct dentry *dentry, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
struct nfs4_exception exception = {
.interruptible = true,
const struct qstr *name = &dentry->d_name;
int err;
do {
- err = _nfs4_proc_lookup(client, dir, dentry, fhandle, fattr, label);
+ err = _nfs4_proc_lookup(client, dir, dentry, fhandle, fattr);
trace_nfs4_lookup(dir, name, err);
switch (err) {
case -NFS4ERR_BADNAME:
}
static int nfs4_proc_lookup(struct inode *dir, struct dentry *dentry,
- struct nfs_fh *fhandle, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fh *fhandle, struct nfs_fattr *fattr)
{
int status;
struct rpc_clnt *client = NFS_CLIENT(dir);
- status = nfs4_proc_lookup_common(&client, dir, dentry, fhandle, fattr, label);
+ status = nfs4_proc_lookup_common(&client, dir, dentry, fhandle, fattr);
if (client != NFS_CLIENT(dir)) {
rpc_shutdown_client(client);
nfs_fixup_secinfo_attributes(fattr);
struct rpc_clnt *client = NFS_CLIENT(dir);
int status;
- status = nfs4_proc_lookup_common(&client, dir, dentry, fhandle, fattr, NULL);
+ status = nfs4_proc_lookup_common(&client, dir, dentry, fhandle, fattr);
if (status < 0)
return ERR_PTR(status);
return (client == NFS_CLIENT(dir)) ? rpc_clone_client(client) : client;
}
static int _nfs4_proc_lookupp(struct inode *inode,
- struct nfs_fh *fhandle, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fh *fhandle, struct nfs_fattr *fattr)
{
struct rpc_clnt *clnt = NFS_CLIENT(inode);
struct nfs_server *server = NFS_SERVER(inode);
struct nfs4_lookupp_res res = {
.server = server,
.fattr = fattr,
- .label = label,
.fh = fhandle,
};
struct rpc_message msg = {
if (NFS_SERVER(inode)->flags & NFS_MOUNT_SOFTREVAL)
task_flags |= RPC_TASK_TIMEOUT;
- args.bitmask = nfs4_bitmask(server, label);
+ args.bitmask = nfs4_bitmask(server, fattr->label);
nfs_fattr_init(fattr);
}
static int nfs4_proc_lookupp(struct inode *inode, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label)
+ struct nfs_fattr *fattr)
{
struct nfs4_exception exception = {
.interruptible = true,
};
int err;
do {
- err = _nfs4_proc_lookupp(inode, fhandle, fattr, label);
+ err = _nfs4_proc_lookupp(inode, fhandle, fattr);
trace_nfs4_lookupp(inode, err);
err = nfs4_handle_exception(NFS_SERVER(inode), err,
&exception);
};
struct nfs4_link_res res = {
.server = server,
- .label = NULL,
};
struct rpc_message msg = {
.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_LINK],
};
int status = -ENOMEM;
- res.fattr = nfs_alloc_fattr();
+ res.fattr = nfs_alloc_fattr_with_label(server);
if (res.fattr == NULL)
goto out;
- res.label = nfs4_label_alloc(server, GFP_KERNEL);
- if (IS_ERR(res.label)) {
- status = PTR_ERR(res.label);
- goto out;
- }
-
nfs4_inode_make_writeable(inode);
- nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, res.label), inode,
+ nfs4_bitmap_copy_adjust(bitmask, nfs4_bitmask(server, res.fattr->label), inode,
NFS_INO_INVALID_CHANGE);
status = nfs4_call_sync(server->client, server, &msg, &arg.seq_args, &res.seq_res, 1);
if (!status) {
nfs4_inc_nlink(inode);
status = nfs_post_op_update_inode(inode, res.fattr);
if (!status)
- nfs_setsecurity(inode, res.fattr, res.label);
+ nfs_setsecurity(inode, res.fattr);
}
-
- nfs4_label_free(res.label);
-
out:
nfs_free_fattr(res.fattr);
return status;
struct nfs4_create_res res;
struct nfs_fh fh;
struct nfs_fattr fattr;
- struct nfs4_label *label;
};
static struct nfs4_createdata *nfs4_alloc_createdata(struct inode *dir,
if (data != NULL) {
struct nfs_server *server = NFS_SERVER(dir);
- data->label = nfs4_label_alloc(server, GFP_KERNEL);
- if (IS_ERR(data->label))
+ data->fattr.label = nfs4_label_alloc(server, GFP_KERNEL);
+ if (IS_ERR(data->fattr.label))
goto out_free;
data->msg.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_CREATE];
data->arg.name = name;
data->arg.attrs = sattr;
data->arg.ftype = ftype;
- data->arg.bitmask = nfs4_bitmask(server, data->label);
+ data->arg.bitmask = nfs4_bitmask(server, data->fattr.label);
data->arg.umask = current_umask();
data->res.server = server;
data->res.fh = &data->fh;
data->res.fattr = &data->fattr;
- data->res.label = data->label;
nfs_fattr_init(data->res.fattr);
}
return data;
data->res.fattr->time_start,
NFS_INO_INVALID_DATA);
spin_unlock(&dir->i_lock);
- status = nfs_instantiate(dentry, data->res.fh, data->res.fattr, data->res.label);
+ status = nfs_instantiate(dentry, data->res.fh, data->res.fattr);
}
return status;
}
static void nfs4_free_createdata(struct nfs4_createdata *data)
{
- nfs4_label_free(data->label);
+ nfs4_label_free(data->fattr.label);
kfree(data);
}
static int nfs4_read_done(struct rpc_task *task, struct nfs_pgio_header *hdr)
{
- dprintk("--> %s\n", __func__);
-
if (!nfs4_sequence_done(task, &hdr->res.seq_res))
return -EAGAIN;
if (nfs4_read_stateid_changed(task, &hdr->args))
size_t buflen)
{
struct nfs_server *server = NFS_SERVER(inode);
- struct nfs_fattr fattr;
struct nfs4_label label = {0, 0, buflen, buf};
u32 bitmask[3] = { 0, 0, FATTR4_WORD2_SECURITY_LABEL };
+ struct nfs_fattr fattr = {
+ .label = &label,
+ };
struct nfs4_getattr_arg arg = {
.fh = NFS_FH(inode),
.bitmask = bitmask,
};
struct nfs4_getattr_res res = {
.fattr = &fattr,
- .label = &label,
.server = server,
};
struct rpc_message msg = {
static int _nfs4_do_set_security_label(struct inode *inode,
struct nfs4_label *ilabel,
- struct nfs_fattr *fattr,
- struct nfs4_label *olabel)
+ struct nfs_fattr *fattr)
{
struct iattr sattr = {0};
};
struct nfs_setattrres res = {
.fattr = fattr,
- .label = olabel,
.server = server,
};
struct rpc_message msg = {
static int nfs4_do_set_security_label(struct inode *inode,
struct nfs4_label *ilabel,
- struct nfs_fattr *fattr,
- struct nfs4_label *olabel)
+ struct nfs_fattr *fattr)
{
struct nfs4_exception exception = { };
int err;
do {
- err = _nfs4_do_set_security_label(inode, ilabel,
- fattr, olabel);
+ err = _nfs4_do_set_security_label(inode, ilabel, fattr);
trace_nfs4_set_security_label(inode, err);
err = nfs4_handle_exception(NFS_SERVER(inode), err,
&exception);
static int
nfs4_set_security_label(struct inode *inode, const void *buf, size_t buflen)
{
- struct nfs4_label ilabel, *olabel = NULL;
- struct nfs_fattr fattr;
+ struct nfs4_label ilabel = {0, 0, buflen, (char *)buf };
+ struct nfs_fattr *fattr;
int status;
if (!nfs_server_capable(inode, NFS_CAP_SECURITY_LABEL))
return -EOPNOTSUPP;
- nfs_fattr_init(&fattr);
-
- ilabel.pi = 0;
- ilabel.lfs = 0;
- ilabel.label = (char *)buf;
- ilabel.len = buflen;
-
- olabel = nfs4_label_alloc(NFS_SERVER(inode), GFP_KERNEL);
- if (IS_ERR(olabel)) {
- status = -PTR_ERR(olabel);
- goto out;
- }
+ fattr = nfs_alloc_fattr_with_label(NFS_SERVER(inode));
+ if (fattr == NULL)
+ return -ENOMEM;
- status = nfs4_do_set_security_label(inode, &ilabel, &fattr, olabel);
+ status = nfs4_do_set_security_label(inode, &ilabel, fattr);
if (status == 0)
- nfs_setsecurity(inode, &fattr, olabel);
+ nfs_setsecurity(inode, fattr);
- nfs4_label_free(olabel);
-out:
return status;
}
#endif /* CONFIG_NFS_V4_SECURITY_LABEL */
struct nfs4_lockdata *data = calldata;
struct nfs4_state *state = data->lsp->ls_state;
- dprintk("%s: begin!\n", __func__);
if (nfs_wait_on_sequence(data->arg.lock_seqid, task) != 0)
goto out_wait;
/* Do we need to do an open_to_lock_owner? */
nfs_release_seqid(data->arg.lock_seqid);
out_wait:
nfs4_sequence_done(task, &data->res.seq_res);
- dprintk("%s: done!, ret = %d\n", __func__, data->rpc_status);
+ dprintk("%s: ret = %d\n", __func__, data->rpc_status);
}
static void nfs4_lock_done(struct rpc_task *task, void *calldata)
struct nfs4_lockdata *data = calldata;
struct nfs4_lock_state *lsp = data->lsp;
- dprintk("%s: begin!\n", __func__);
-
if (!nfs4_sequence_done(task, &data->res.seq_res))
return;
goto out_restart;
}
out_done:
- dprintk("%s: done, ret = %d!\n", __func__, data->rpc_status);
+ dprintk("%s: ret = %d!\n", __func__, data->rpc_status);
return;
out_restart:
if (!data->cancelled)
{
struct nfs4_lockdata *data = calldata;
- dprintk("%s: begin!\n", __func__);
nfs_free_seqid(data->arg.open_seqid);
if (data->cancelled && data->rpc_status == 0) {
struct rpc_task *task;
nfs4_put_lock_state(data->lsp);
put_nfs_open_context(data->ctx);
kfree(data);
- dprintk("%s: done!\n", __func__);
}
static const struct rpc_call_ops nfs4_lock_ops = {
if (client->cl_minorversion)
task_setup_data.flags |= RPC_TASK_MOVEABLE;
- dprintk("%s: begin!\n", __func__);
data = nfs4_alloc_lockdata(fl, nfs_file_open_context(fl->fl_file),
fl->fl_u.nfs4_fl.owner,
recovery_type == NFS_LOCK_NEW ? GFP_KERNEL : GFP_NOFS);
data->cancelled = true;
trace_nfs4_set_lock(fl, state, &data->res.stateid, cmd, ret);
rpc_put_task(task);
- dprintk("%s: done, ret = %d!\n", __func__, ret);
+ dprintk("%s: ret = %d\n", __func__, ret);
return ret;
}
struct nfs4_get_lease_time_data *data =
(struct nfs4_get_lease_time_data *)calldata;
- dprintk("--> %s\n", __func__);
/* just setup sequence, do not trigger session recovery
since we're invoked within one */
nfs4_setup_sequence(data->clp,
&data->args->la_seq_args,
&data->res->lr_seq_res,
task);
- dprintk("<-- %s\n", __func__);
}
/*
struct nfs4_get_lease_time_data *data =
(struct nfs4_get_lease_time_data *)calldata;
- dprintk("--> %s\n", __func__);
if (!nfs4_sequence_done(task, &data->res->lr_seq_res))
return;
switch (task->tk_status) {
case -NFS4ERR_DELAY:
case -NFS4ERR_GRACE:
- dprintk("%s Retry: tk_status %d\n", __func__, task->tk_status);
rpc_delay(task, NFS4_POLL_RETRY_MIN);
task->tk_status = 0;
fallthrough;
rpc_restart_call_prepare(task);
return;
}
- dprintk("<-- %s\n", __func__);
}
static const struct rpc_call_ops nfs4_get_lease_time_ops = {
dprintk("%s client>seqid %d sessionid %u:%u:%u:%u\n", __func__,
clp->cl_seqid, ptr[0], ptr[1], ptr[2], ptr[3]);
out:
- dprintk("<-- %s\n", __func__);
return status;
}
};
int status = 0;
- dprintk("--> nfs4_proc_destroy_session\n");
-
/* session is still being setup */
if (!test_and_clear_bit(NFS4_SESSION_ESTABLISHED, &session->session_state))
return 0;
if (status)
dprintk("NFS: Got error %d from the server on DESTROY_SESSION. "
"Session has been destroyed regardless...\n", status);
-
- dprintk("<-- nfs4_proc_destroy_session\n");
return status;
}
if (task->tk_status < 0) {
dprintk("%s ERROR %d\n", __func__, task->tk_status);
if (refcount_read(&clp->cl_count) == 1)
- goto out;
+ return;
if (nfs41_sequence_handle_errors(task, clp) == -EAGAIN) {
rpc_restart_call_prepare(task);
}
}
dprintk("%s rpc_cred %p\n", __func__, task->tk_msg.rpc_cred);
-out:
- dprintk("<-- %s\n", __func__);
}
static void nfs41_sequence_prepare(struct rpc_task *task, void *data)
struct nfs_client *clp = calldata->clp;
struct nfs4_sequence_res *res = &calldata->res.seq_res;
- dprintk("--> %s\n", __func__);
if (!nfs41_sequence_done(task, res))
return;
rpc_restart_call_prepare(task);
return;
}
- dprintk("<-- %s\n", __func__);
}
static void nfs4_free_reclaim_complete_data(void *data)
};
int status = -ENOMEM;
- dprintk("--> %s\n", __func__);
calldata = kzalloc(sizeof(*calldata), GFP_NOFS);
if (calldata == NULL)
goto out;
struct nfs4_layoutget *lgp = calldata;
struct nfs_server *server = NFS_SERVER(lgp->args.inode);
- dprintk("--> %s\n", __func__);
nfs4_setup_sequence(server->nfs_client, &lgp->args.seq_args,
&lgp->res.seq_res, task);
- dprintk("<-- %s\n", __func__);
}
static void nfs4_layoutget_done(struct rpc_task *task, void *calldata)
{
struct nfs4_layoutget *lgp = calldata;
- dprintk("--> %s\n", __func__);
nfs41_sequence_process(task, &lgp->res.seq_res);
- dprintk("<-- %s\n", __func__);
}
static int
status = err;
}
out:
- dprintk("<-- %s\n", __func__);
return status;
}
{
struct nfs4_layoutget *lgp = calldata;
- dprintk("--> %s\n", __func__);
nfs4_sequence_free_slot(&lgp->res.seq_res);
pnfs_layoutget_free(lgp);
- dprintk("<-- %s\n", __func__);
}
static const struct rpc_call_ops nfs4_layoutget_call_ops = {
};
int status = 0;
- dprintk("--> %s\n", __func__);
-
nfs4_init_sequence(&lgp->args.seq_args, &lgp->res.seq_res, 0, 0);
task = rpc_run_task(&task_setup_data);
{
struct nfs4_layoutreturn *lrp = calldata;
- dprintk("--> %s\n", __func__);
nfs4_setup_sequence(lrp->clp,
&lrp->args.seq_args,
&lrp->res.seq_res,
struct nfs4_layoutreturn *lrp = calldata;
struct nfs_server *server;
- dprintk("--> %s\n", __func__);
-
if (!nfs41_sequence_process(task, &lrp->res.seq_res))
return;
break;
goto out_restart;
}
- dprintk("<-- %s\n", __func__);
return;
out_restart:
task->tk_status = 0;
struct nfs4_layoutreturn *lrp = calldata;
struct pnfs_layout_hdr *lo = lrp->args.layout;
- dprintk("--> %s\n", __func__);
pnfs_layoutreturn_free_lsegs(lo, &lrp->args.stateid, &lrp->args.range,
lrp->res.lrs_present ? &lrp->res.stateid : NULL);
nfs4_sequence_free_slot(&lrp->res.seq_res);
nfs_iput_and_deactive(lrp->inode);
put_cred(lrp->cred);
kfree(calldata);
- dprintk("<-- %s\n", __func__);
}
static const struct rpc_call_ops nfs4_layoutreturn_call_ops = {
NFS_SP4_MACH_CRED_PNFS_CLEANUP,
&task_setup_data.rpc_client, &msg);
- dprintk("--> %s\n", __func__);
lrp->inode = nfs_igrab_and_active(lrp->args.inode);
if (!sync) {
if (!lrp->inode) {
};
int status;
- dprintk("--> %s\n", __func__);
status = nfs4_call_sync(server->client, server, &msg, &args.seq_args, &res.seq_res, 0);
if (res.notification & ~args.notify_types)
dprintk("%s: unsupported notification\n", __func__);
msg.rpc_cred = cred;
}
- dprintk("--> %s\n", __func__);
nfs4_init_sequence(&args.seq_args, &res.seq_res, 0, 0);
status = nfs4_call_sync_custom(&task_setup);
dprintk("<-- %s status=%d\n", __func__, status);
static void nfs41_free_stateid_release(void *calldata)
{
+ struct nfs_free_stateid_data *data = calldata;
+ struct nfs_client *clp = data->server->nfs_client;
+
+ nfs_put_client(clp);
kfree(calldata);
}
};
struct nfs_free_stateid_data *data;
struct rpc_task *task;
+ struct nfs_client *clp = server->nfs_client;
+
+ if (!refcount_inc_not_zero(&clp->cl_count))
+ return -EIO;
nfs4_state_protect(server->nfs_client, NFS_SP4_MACH_CRED_STATEID,
&task_setup.rpc_client, &msg);
struct nfs4_slot *slot,
struct nfs4_sequence_res *res)
{
+ u32 target_highest_slotid = min(res->sr_target_highest_slotid,
+ NFS4_MAX_SLOTID);
+ u32 highest_slotid = min(res->sr_highest_slotid, NFS4_MAX_SLOTID);
+
spin_lock(&tbl->slot_tbl_lock);
- if (!nfs41_is_outlier_target_slotid(tbl, res->sr_target_highest_slotid))
- nfs41_set_target_slotid_locked(tbl, res->sr_target_highest_slotid);
+ if (!nfs41_is_outlier_target_slotid(tbl, target_highest_slotid))
+ nfs41_set_target_slotid_locked(tbl, target_highest_slotid);
if (tbl->generation == slot->generation)
- nfs41_set_server_slotid_locked(tbl, res->sr_highest_slotid);
- nfs41_set_max_slotid_locked(tbl, res->sr_target_highest_slotid);
+ nfs41_set_server_slotid_locked(tbl, highest_slotid);
+ nfs41_set_max_slotid_locked(tbl, target_highest_slotid);
spin_unlock(&tbl->slot_tbl_lock);
}
#define NFS4_DEF_SLOT_TABLE_SIZE (64U)
#define NFS4_DEF_CB_SLOT_TABLE_SIZE (16U)
#define NFS4_MAX_SLOT_TABLE (1024U)
+#define NFS4_MAX_SLOTID (NFS4_MAX_SLOT_TABLE - 1U)
#define NFS4_NO_SLOT ((u32)-1)
#if IS_ENABLED(CONFIG_NFS_V4)
static void nfs4_clear_state_manager_bit(struct nfs_client *clp)
{
- smp_mb__before_atomic();
- clear_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state);
- smp_mb__after_atomic();
- wake_up_bit(&clp->cl_state, NFS4CLNT_MANAGER_RUNNING);
+ clear_and_wake_up_bit(NFS4CLNT_MANAGER_RUNNING, &clp->cl_state);
rpc_wake_up(&clp->cl_rpcwaitq);
}
#define _TRACE_NFS4_H
#include <linux/tracepoint.h>
+#include <trace/events/sunrpc_base.h>
-TRACE_DEFINE_ENUM(EPERM);
-TRACE_DEFINE_ENUM(ENOENT);
-TRACE_DEFINE_ENUM(EIO);
-TRACE_DEFINE_ENUM(ENXIO);
-TRACE_DEFINE_ENUM(EACCES);
-TRACE_DEFINE_ENUM(EEXIST);
-TRACE_DEFINE_ENUM(EXDEV);
-TRACE_DEFINE_ENUM(ENOTDIR);
-TRACE_DEFINE_ENUM(EISDIR);
-TRACE_DEFINE_ENUM(EFBIG);
-TRACE_DEFINE_ENUM(ENOSPC);
-TRACE_DEFINE_ENUM(EROFS);
-TRACE_DEFINE_ENUM(EMLINK);
-TRACE_DEFINE_ENUM(ENAMETOOLONG);
-TRACE_DEFINE_ENUM(ENOTEMPTY);
-TRACE_DEFINE_ENUM(EDQUOT);
-TRACE_DEFINE_ENUM(ESTALE);
-TRACE_DEFINE_ENUM(EBADHANDLE);
-TRACE_DEFINE_ENUM(EBADCOOKIE);
-TRACE_DEFINE_ENUM(ENOTSUPP);
-TRACE_DEFINE_ENUM(ETOOSMALL);
-TRACE_DEFINE_ENUM(EREMOTEIO);
-TRACE_DEFINE_ENUM(EBADTYPE);
-TRACE_DEFINE_ENUM(EAGAIN);
-TRACE_DEFINE_ENUM(ELOOP);
-TRACE_DEFINE_ENUM(EOPNOTSUPP);
-TRACE_DEFINE_ENUM(EDEADLK);
-TRACE_DEFINE_ENUM(ENOMEM);
-TRACE_DEFINE_ENUM(EKEYEXPIRED);
-TRACE_DEFINE_ENUM(ETIMEDOUT);
-TRACE_DEFINE_ENUM(ERESTARTSYS);
-TRACE_DEFINE_ENUM(ECONNREFUSED);
-TRACE_DEFINE_ENUM(ECONNRESET);
-TRACE_DEFINE_ENUM(ENETUNREACH);
-TRACE_DEFINE_ENUM(EHOSTUNREACH);
-TRACE_DEFINE_ENUM(EHOSTDOWN);
-TRACE_DEFINE_ENUM(EPIPE);
-TRACE_DEFINE_ENUM(EPFNOSUPPORT);
-TRACE_DEFINE_ENUM(EPROTONOSUPPORT);
-
-TRACE_DEFINE_ENUM(NFS4_OK);
-TRACE_DEFINE_ENUM(NFS4ERR_ACCESS);
-TRACE_DEFINE_ENUM(NFS4ERR_ATTRNOTSUPP);
-TRACE_DEFINE_ENUM(NFS4ERR_ADMIN_REVOKED);
-TRACE_DEFINE_ENUM(NFS4ERR_BACK_CHAN_BUSY);
-TRACE_DEFINE_ENUM(NFS4ERR_BADCHAR);
-TRACE_DEFINE_ENUM(NFS4ERR_BADHANDLE);
-TRACE_DEFINE_ENUM(NFS4ERR_BADIOMODE);
-TRACE_DEFINE_ENUM(NFS4ERR_BADLAYOUT);
-TRACE_DEFINE_ENUM(NFS4ERR_BADLABEL);
-TRACE_DEFINE_ENUM(NFS4ERR_BADNAME);
-TRACE_DEFINE_ENUM(NFS4ERR_BADOWNER);
-TRACE_DEFINE_ENUM(NFS4ERR_BADSESSION);
-TRACE_DEFINE_ENUM(NFS4ERR_BADSLOT);
-TRACE_DEFINE_ENUM(NFS4ERR_BADTYPE);
-TRACE_DEFINE_ENUM(NFS4ERR_BADXDR);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_COOKIE);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_HIGH_SLOT);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_RANGE);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_SEQID);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_SESSION_DIGEST);
-TRACE_DEFINE_ENUM(NFS4ERR_BAD_STATEID);
-TRACE_DEFINE_ENUM(NFS4ERR_CB_PATH_DOWN);
-TRACE_DEFINE_ENUM(NFS4ERR_CLID_INUSE);
-TRACE_DEFINE_ENUM(NFS4ERR_CLIENTID_BUSY);
-TRACE_DEFINE_ENUM(NFS4ERR_COMPLETE_ALREADY);
-TRACE_DEFINE_ENUM(NFS4ERR_CONN_NOT_BOUND_TO_SESSION);
-TRACE_DEFINE_ENUM(NFS4ERR_DEADLOCK);
-TRACE_DEFINE_ENUM(NFS4ERR_DEADSESSION);
-TRACE_DEFINE_ENUM(NFS4ERR_DELAY);
-TRACE_DEFINE_ENUM(NFS4ERR_DELEG_ALREADY_WANTED);
-TRACE_DEFINE_ENUM(NFS4ERR_DELEG_REVOKED);
-TRACE_DEFINE_ENUM(NFS4ERR_DENIED);
-TRACE_DEFINE_ENUM(NFS4ERR_DIRDELEG_UNAVAIL);
-TRACE_DEFINE_ENUM(NFS4ERR_DQUOT);
-TRACE_DEFINE_ENUM(NFS4ERR_ENCR_ALG_UNSUPP);
-TRACE_DEFINE_ENUM(NFS4ERR_EXIST);
-TRACE_DEFINE_ENUM(NFS4ERR_EXPIRED);
-TRACE_DEFINE_ENUM(NFS4ERR_FBIG);
-TRACE_DEFINE_ENUM(NFS4ERR_FHEXPIRED);
-TRACE_DEFINE_ENUM(NFS4ERR_FILE_OPEN);
-TRACE_DEFINE_ENUM(NFS4ERR_GRACE);
-TRACE_DEFINE_ENUM(NFS4ERR_HASH_ALG_UNSUPP);
-TRACE_DEFINE_ENUM(NFS4ERR_INVAL);
-TRACE_DEFINE_ENUM(NFS4ERR_IO);
-TRACE_DEFINE_ENUM(NFS4ERR_ISDIR);
-TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTTRYLATER);
-TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTUNAVAILABLE);
-TRACE_DEFINE_ENUM(NFS4ERR_LEASE_MOVED);
-TRACE_DEFINE_ENUM(NFS4ERR_LOCKED);
-TRACE_DEFINE_ENUM(NFS4ERR_LOCKS_HELD);
-TRACE_DEFINE_ENUM(NFS4ERR_LOCK_RANGE);
-TRACE_DEFINE_ENUM(NFS4ERR_MINOR_VERS_MISMATCH);
-TRACE_DEFINE_ENUM(NFS4ERR_MLINK);
-TRACE_DEFINE_ENUM(NFS4ERR_MOVED);
-TRACE_DEFINE_ENUM(NFS4ERR_NAMETOOLONG);
-TRACE_DEFINE_ENUM(NFS4ERR_NOENT);
-TRACE_DEFINE_ENUM(NFS4ERR_NOFILEHANDLE);
-TRACE_DEFINE_ENUM(NFS4ERR_NOMATCHING_LAYOUT);
-TRACE_DEFINE_ENUM(NFS4ERR_NOSPC);
-TRACE_DEFINE_ENUM(NFS4ERR_NOTDIR);
-TRACE_DEFINE_ENUM(NFS4ERR_NOTEMPTY);
-TRACE_DEFINE_ENUM(NFS4ERR_NOTSUPP);
-TRACE_DEFINE_ENUM(NFS4ERR_NOT_ONLY_OP);
-TRACE_DEFINE_ENUM(NFS4ERR_NOT_SAME);
-TRACE_DEFINE_ENUM(NFS4ERR_NO_GRACE);
-TRACE_DEFINE_ENUM(NFS4ERR_NXIO);
-TRACE_DEFINE_ENUM(NFS4ERR_OLD_STATEID);
-TRACE_DEFINE_ENUM(NFS4ERR_OPENMODE);
-TRACE_DEFINE_ENUM(NFS4ERR_OP_ILLEGAL);
-TRACE_DEFINE_ENUM(NFS4ERR_OP_NOT_IN_SESSION);
-TRACE_DEFINE_ENUM(NFS4ERR_PERM);
-TRACE_DEFINE_ENUM(NFS4ERR_PNFS_IO_HOLE);
-TRACE_DEFINE_ENUM(NFS4ERR_PNFS_NO_LAYOUT);
-TRACE_DEFINE_ENUM(NFS4ERR_RECALLCONFLICT);
-TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_BAD);
-TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_CONFLICT);
-TRACE_DEFINE_ENUM(NFS4ERR_REJECT_DELEG);
-TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG);
-TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG_TO_CACHE);
-TRACE_DEFINE_ENUM(NFS4ERR_REQ_TOO_BIG);
-TRACE_DEFINE_ENUM(NFS4ERR_RESOURCE);
-TRACE_DEFINE_ENUM(NFS4ERR_RESTOREFH);
-TRACE_DEFINE_ENUM(NFS4ERR_RETRY_UNCACHED_REP);
-TRACE_DEFINE_ENUM(NFS4ERR_RETURNCONFLICT);
-TRACE_DEFINE_ENUM(NFS4ERR_ROFS);
-TRACE_DEFINE_ENUM(NFS4ERR_SAME);
-TRACE_DEFINE_ENUM(NFS4ERR_SHARE_DENIED);
-TRACE_DEFINE_ENUM(NFS4ERR_SEQUENCE_POS);
-TRACE_DEFINE_ENUM(NFS4ERR_SEQ_FALSE_RETRY);
-TRACE_DEFINE_ENUM(NFS4ERR_SEQ_MISORDERED);
-TRACE_DEFINE_ENUM(NFS4ERR_SERVERFAULT);
-TRACE_DEFINE_ENUM(NFS4ERR_STALE);
-TRACE_DEFINE_ENUM(NFS4ERR_STALE_CLIENTID);
-TRACE_DEFINE_ENUM(NFS4ERR_STALE_STATEID);
-TRACE_DEFINE_ENUM(NFS4ERR_SYMLINK);
-TRACE_DEFINE_ENUM(NFS4ERR_TOOSMALL);
-TRACE_DEFINE_ENUM(NFS4ERR_TOO_MANY_OPS);
-TRACE_DEFINE_ENUM(NFS4ERR_UNKNOWN_LAYOUTTYPE);
-TRACE_DEFINE_ENUM(NFS4ERR_UNSAFE_COMPOUND);
-TRACE_DEFINE_ENUM(NFS4ERR_WRONGSEC);
-TRACE_DEFINE_ENUM(NFS4ERR_WRONG_CRED);
-TRACE_DEFINE_ENUM(NFS4ERR_WRONG_TYPE);
-TRACE_DEFINE_ENUM(NFS4ERR_XDEV);
-
-TRACE_DEFINE_ENUM(NFS4ERR_RESET_TO_MDS);
-TRACE_DEFINE_ENUM(NFS4ERR_RESET_TO_PNFS);
-
-#define show_nfsv4_errors(error) \
- __print_symbolic(error, \
- { NFS4_OK, "OK" }, \
- /* Mapped by nfs4_stat_to_errno() */ \
- { EPERM, "EPERM" }, \
- { ENOENT, "ENOENT" }, \
- { EIO, "EIO" }, \
- { ENXIO, "ENXIO" }, \
- { EACCES, "EACCES" }, \
- { EEXIST, "EEXIST" }, \
- { EXDEV, "EXDEV" }, \
- { ENOTDIR, "ENOTDIR" }, \
- { EISDIR, "EISDIR" }, \
- { EFBIG, "EFBIG" }, \
- { ENOSPC, "ENOSPC" }, \
- { EROFS, "EROFS" }, \
- { EMLINK, "EMLINK" }, \
- { ENAMETOOLONG, "ENAMETOOLONG" }, \
- { ENOTEMPTY, "ENOTEMPTY" }, \
- { EDQUOT, "EDQUOT" }, \
- { ESTALE, "ESTALE" }, \
- { EBADHANDLE, "EBADHANDLE" }, \
- { EBADCOOKIE, "EBADCOOKIE" }, \
- { ENOTSUPP, "ENOTSUPP" }, \
- { ETOOSMALL, "ETOOSMALL" }, \
- { EREMOTEIO, "EREMOTEIO" }, \
- { EBADTYPE, "EBADTYPE" }, \
- { EAGAIN, "EAGAIN" }, \
- { ELOOP, "ELOOP" }, \
- { EOPNOTSUPP, "EOPNOTSUPP" }, \
- { EDEADLK, "EDEADLK" }, \
- /* RPC errors */ \
- { ENOMEM, "ENOMEM" }, \
- { EKEYEXPIRED, "EKEYEXPIRED" }, \
- { ETIMEDOUT, "ETIMEDOUT" }, \
- { ERESTARTSYS, "ERESTARTSYS" }, \
- { ECONNREFUSED, "ECONNREFUSED" }, \
- { ECONNRESET, "ECONNRESET" }, \
- { ENETUNREACH, "ENETUNREACH" }, \
- { EHOSTUNREACH, "EHOSTUNREACH" }, \
- { EHOSTDOWN, "EHOSTDOWN" }, \
- { EPIPE, "EPIPE" }, \
- { EPFNOSUPPORT, "EPFNOSUPPORT" }, \
- { EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \
- /* NFSv4 native errors */ \
- { NFS4ERR_ACCESS, "ACCESS" }, \
- { NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \
- { NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \
- { NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \
- { NFS4ERR_BADCHAR, "BADCHAR" }, \
- { NFS4ERR_BADHANDLE, "BADHANDLE" }, \
- { NFS4ERR_BADIOMODE, "BADIOMODE" }, \
- { NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \
- { NFS4ERR_BADLABEL, "BADLABEL" }, \
- { NFS4ERR_BADNAME, "BADNAME" }, \
- { NFS4ERR_BADOWNER, "BADOWNER" }, \
- { NFS4ERR_BADSESSION, "BADSESSION" }, \
- { NFS4ERR_BADSLOT, "BADSLOT" }, \
- { NFS4ERR_BADTYPE, "BADTYPE" }, \
- { NFS4ERR_BADXDR, "BADXDR" }, \
- { NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \
- { NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \
- { NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \
- { NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \
- { NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \
- { NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \
- { NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \
- { NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \
- { NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \
- { NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \
- { NFS4ERR_CONN_NOT_BOUND_TO_SESSION, \
- "CONN_NOT_BOUND_TO_SESSION" }, \
- { NFS4ERR_DEADLOCK, "DEADLOCK" }, \
- { NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \
- { NFS4ERR_DELAY, "DELAY" }, \
- { NFS4ERR_DELEG_ALREADY_WANTED, \
- "DELEG_ALREADY_WANTED" }, \
- { NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \
- { NFS4ERR_DENIED, "DENIED" }, \
- { NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \
- { NFS4ERR_DQUOT, "DQUOT" }, \
- { NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \
- { NFS4ERR_EXIST, "EXIST" }, \
- { NFS4ERR_EXPIRED, "EXPIRED" }, \
- { NFS4ERR_FBIG, "FBIG" }, \
- { NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \
- { NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \
- { NFS4ERR_GRACE, "GRACE" }, \
- { NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \
- { NFS4ERR_INVAL, "INVAL" }, \
- { NFS4ERR_IO, "IO" }, \
- { NFS4ERR_ISDIR, "ISDIR" }, \
- { NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \
- { NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \
- { NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \
- { NFS4ERR_LOCKED, "LOCKED" }, \
- { NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \
- { NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \
- { NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \
- { NFS4ERR_MLINK, "MLINK" }, \
- { NFS4ERR_MOVED, "MOVED" }, \
- { NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \
- { NFS4ERR_NOENT, "NOENT" }, \
- { NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \
- { NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \
- { NFS4ERR_NOSPC, "NOSPC" }, \
- { NFS4ERR_NOTDIR, "NOTDIR" }, \
- { NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \
- { NFS4ERR_NOTSUPP, "NOTSUPP" }, \
- { NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \
- { NFS4ERR_NOT_SAME, "NOT_SAME" }, \
- { NFS4ERR_NO_GRACE, "NO_GRACE" }, \
- { NFS4ERR_NXIO, "NXIO" }, \
- { NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \
- { NFS4ERR_OPENMODE, "OPENMODE" }, \
- { NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \
- { NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \
- { NFS4ERR_PERM, "PERM" }, \
- { NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \
- { NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \
- { NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \
- { NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \
- { NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \
- { NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \
- { NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \
- { NFS4ERR_REP_TOO_BIG_TO_CACHE, \
- "REP_TOO_BIG_TO_CACHE" }, \
- { NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \
- { NFS4ERR_RESOURCE, "RESOURCE" }, \
- { NFS4ERR_RESTOREFH, "RESTOREFH" }, \
- { NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \
- { NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \
- { NFS4ERR_ROFS, "ROFS" }, \
- { NFS4ERR_SAME, "SAME" }, \
- { NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \
- { NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \
- { NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \
- { NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \
- { NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \
- { NFS4ERR_STALE, "STALE" }, \
- { NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \
- { NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \
- { NFS4ERR_SYMLINK, "SYMLINK" }, \
- { NFS4ERR_TOOSMALL, "TOOSMALL" }, \
- { NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \
- { NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \
- { NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \
- { NFS4ERR_WRONGSEC, "WRONGSEC" }, \
- { NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \
- { NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \
- { NFS4ERR_XDEV, "XDEV" }, \
- /* ***** Internal to Linux NFS client ***** */ \
- { NFS4ERR_RESET_TO_MDS, "RESET_TO_MDS" }, \
- { NFS4ERR_RESET_TO_PNFS, "RESET_TO_PNFS" })
-
-#define show_open_flags(flags) \
- __print_flags(flags, "|", \
- { O_CREAT, "O_CREAT" }, \
- { O_EXCL, "O_EXCL" }, \
- { O_TRUNC, "O_TRUNC" }, \
- { O_DIRECT, "O_DIRECT" })
-
-#define show_fmode_flags(mode) \
- __print_flags(mode, "|", \
- { ((__force unsigned long)FMODE_READ), "READ" }, \
- { ((__force unsigned long)FMODE_WRITE), "WRITE" }, \
- { ((__force unsigned long)FMODE_EXEC), "EXEC" })
+#include <trace/events/fs.h>
+#include <trace/events/nfs.h>
#define show_nfs_fattr_flags(valid) \
__print_flags((unsigned long)valid, "|", \
TP_printk(
"error=%ld (%s) dstaddr=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
__get_str(dstaddr)
)
);
DEFINE_NFS4_CLIENTID_EVENT(nfs4_sequence);
DEFINE_NFS4_CLIENTID_EVENT(nfs4_reclaim_complete);
-#define show_nfs4_sequence_status_flags(status) \
- __print_flags((unsigned long)status, "|", \
- { SEQ4_STATUS_CB_PATH_DOWN, "CB_PATH_DOWN" }, \
- { SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING, \
- "CB_GSS_CONTEXTS_EXPIRING" }, \
- { SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED, \
- "CB_GSS_CONTEXTS_EXPIRED" }, \
- { SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, \
- "EXPIRED_ALL_STATE_REVOKED" }, \
- { SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, \
- "EXPIRED_SOME_STATE_REVOKED" }, \
- { SEQ4_STATUS_ADMIN_STATE_REVOKED, \
- "ADMIN_STATE_REVOKED" }, \
- { SEQ4_STATUS_RECALLABLE_STATE_REVOKED, \
- "RECALLABLE_STATE_REVOKED" }, \
- { SEQ4_STATUS_LEASE_MOVED, "LEASE_MOVED" }, \
- { SEQ4_STATUS_RESTART_RECLAIM_NEEDED, \
- "RESTART_RECLAIM_NEEDED" }, \
- { SEQ4_STATUS_CB_PATH_DOWN_SESSION, \
- "CB_PATH_DOWN_SESSION" }, \
- { SEQ4_STATUS_BACKCHANNEL_FAULT, \
- "BACKCHANNEL_FAULT" })
-
TRACE_EVENT(nfs4_sequence_done,
TP_PROTO(
const struct nfs4_session *session,
__field(unsigned int, seq_nr)
__field(unsigned int, highest_slotid)
__field(unsigned int, target_highest_slotid)
- __field(unsigned int, status_flags)
+ __field(unsigned long, status_flags)
__field(unsigned long, error)
),
TP_printk(
"error=%ld (%s) session=0x%08x slot_nr=%u seq_nr=%u "
"highest_slotid=%u target_highest_slotid=%u "
- "status_flags=%u (%s)",
+ "status_flags=0x%lx (%s)",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
__entry->session,
__entry->slot_nr,
__entry->seq_nr,
__entry->highest_slotid,
__entry->target_highest_slotid,
__entry->status_flags,
- show_nfs4_sequence_status_flags(__entry->status_flags)
+ show_nfs4_seq4_status(__entry->status_flags)
)
);
"error=%ld (%s) session=0x%08x slot_nr=%u seq_nr=%u "
"highest_slotid=%u",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
__entry->session,
__entry->slot_nr,
__entry->seq_nr,
"error=%ld (%s) session=0x%08x slot_nr=%u seq_nr=%u "
"highest_slotid=%u",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
__entry->session,
__entry->slot_nr,
__entry->seq_nr,
)
);
+TRACE_EVENT(nfs4_cb_offload,
+ TP_PROTO(
+ const struct nfs_fh *cb_fh,
+ const nfs4_stateid *cb_stateid,
+ uint64_t cb_count,
+ int cb_error,
+ int cb_how_stable
+ ),
+
+ TP_ARGS(cb_fh, cb_stateid, cb_count, cb_error,
+ cb_how_stable),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, fhandle)
+ __field(loff_t, cb_count)
+ __field(int, cb_how)
+ __field(int, cb_stateid_seq)
+ __field(u32, cb_stateid_hash)
+ ),
+
+ TP_fast_assign(
+ __entry->error = cb_error < 0 ? -cb_error : 0;
+ __entry->fhandle = nfs_fhandle_hash(cb_fh);
+ __entry->cb_stateid_seq =
+ be32_to_cpu(cb_stateid->seqid);
+ __entry->cb_stateid_hash =
+ nfs_stateid_hash(cb_stateid);
+ __entry->cb_count = cb_count;
+ __entry->cb_how = cb_how_stable;
+ ),
+
+ TP_printk(
+ "error=%ld (%s) fhandle=0x%08x cb_stateid=%d:0x%08x "
+ "cb_count=%llu cb_how=%s",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ __entry->fhandle,
+ __entry->cb_stateid_seq, __entry->cb_stateid_hash,
+ __entry->cb_count,
+ show_nfs_stable_how(__entry->cb_how)
+ )
+);
#endif /* CONFIG_NFS_V4_1 */
TRACE_EVENT(nfs4_setup_sequence,
"hostname=%s clp state=%s error=%ld (%s) section=%s",
__get_str(hostname),
show_nfs4_clp_state(__entry->state), -__entry->error,
- show_nfsv4_errors(__entry->error), __get_str(section)
+ show_nfs4_status(__entry->error), __get_str(section)
)
)
__entry->expected = expected;
),
- TP_printk(
- "task:%u@%d xid=0x%08x operation=%u, expected=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x operation=%u, expected=%u",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->op, __entry->expected
)
__entry->error = error;
),
- TP_printk(
- "task:%u@%d xid=0x%08x error=%ld (%s) operation=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x error=%ld (%s) operation=%u",
__entry->task_id, __entry->client_id, __entry->xid,
- -__entry->error, show_nfsv4_errors(__entry->error),
+ -__entry->error, show_nfs4_status(__entry->error),
__entry->op
)
);
TP_STRUCT__entry(
__field(unsigned long, error)
- __field(unsigned int, flags)
- __field(unsigned int, fmode)
+ __field(unsigned long, flags)
+ __field(unsigned long, fmode)
__field(dev_t, dev)
__field(u32, fhandle)
__field(u64, fileid)
__entry->error = -error;
__entry->flags = flags;
- __entry->fmode = (__force unsigned int)ctx->mode;
+ __entry->fmode = (__force unsigned long)ctx->mode;
__entry->dev = ctx->dentry->d_sb->s_dev;
if (!IS_ERR_OR_NULL(state)) {
inode = state->inode;
),
TP_printk(
- "error=%ld (%s) flags=%d (%s) fmode=%s "
+ "error=%ld (%s) flags=%lu (%s) fmode=%s "
"fileid=%02x:%02x:%llu fhandle=0x%08x "
"name=%02x:%02x:%llu/%s stateid=%d:0x%08x "
"openstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
__entry->flags,
- show_open_flags(__entry->flags),
- show_fmode_flags(__entry->fmode),
+ show_fs_fcntl_open_flags(__entry->flags),
+ show_fs_fmode_flags(__entry->fmode),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
TP_printk(
"fmode=%s fileid=%02x:%02x:%llu "
"fhandle=0x%08x stateid=%d:0x%08x",
- __entry->fmode ? show_fmode_flags(__entry->fmode) :
+ __entry->fmode ? show_fs_fmode_flags(__entry->fmode) :
"closed",
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
"error=%ld (%s) fmode=%s fileid=%02x:%02x:%llu "
"fhandle=0x%08x openstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
- __entry->fmode ? show_fmode_flags(__entry->fmode) :
+ show_nfs4_status(__entry->error),
+ __entry->fmode ? show_fs_fmode_flags(__entry->fmode) :
"closed",
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
)
);
-TRACE_DEFINE_ENUM(F_GETLK);
-TRACE_DEFINE_ENUM(F_SETLK);
-TRACE_DEFINE_ENUM(F_SETLKW);
-TRACE_DEFINE_ENUM(F_RDLCK);
-TRACE_DEFINE_ENUM(F_WRLCK);
-TRACE_DEFINE_ENUM(F_UNLCK);
-
-#define show_lock_cmd(type) \
- __print_symbolic((int)type, \
- { F_GETLK, "GETLK" }, \
- { F_SETLK, "SETLK" }, \
- { F_SETLKW, "SETLKW" })
-#define show_lock_type(type) \
- __print_symbolic((int)type, \
- { F_RDLCK, "RDLCK" }, \
- { F_WRLCK, "WRLCK" }, \
- { F_UNLCK, "UNLCK" })
-
DECLARE_EVENT_CLASS(nfs4_lock_event,
TP_PROTO(
const struct file_lock *request,
TP_STRUCT__entry(
__field(unsigned long, error)
- __field(int, cmd)
- __field(char, type)
+ __field(unsigned long, cmd)
+ __field(unsigned long, type)
__field(loff_t, start)
__field(loff_t, end)
__field(dev_t, dev)
"fileid=%02x:%02x:%llu fhandle=0x%08x "
"stateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
- show_lock_cmd(__entry->cmd),
- show_lock_type(__entry->type),
+ show_nfs4_status(__entry->error),
+ show_fs_fcntl_cmd(__entry->cmd),
+ show_fs_fcntl_lock_type(__entry->type),
(long long)__entry->start,
(long long)__entry->end,
MAJOR(__entry->dev), MINOR(__entry->dev),
TP_STRUCT__entry(
__field(unsigned long, error)
- __field(int, cmd)
- __field(char, type)
+ __field(unsigned long, cmd)
+ __field(unsigned long, type)
__field(loff_t, start)
__field(loff_t, end)
__field(dev_t, dev)
"fileid=%02x:%02x:%llu fhandle=0x%08x "
"stateid=%d:0x%08x lockstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
- show_lock_cmd(__entry->cmd),
- show_lock_type(__entry->type),
+ show_nfs4_status(__entry->error),
+ show_fs_fcntl_cmd(__entry->cmd),
+ show_fs_fcntl_lock_type(__entry->type),
(long long)__entry->start,
(long long)__entry->end,
MAJOR(__entry->dev), MINOR(__entry->dev),
TP_printk(
"fmode=%s fileid=%02x:%02x:%llu fhandle=0x%08x",
- show_fmode_flags(__entry->fmode),
+ show_fs_fmode_flags(__entry->fmode),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle
"error=%ld (%s) dev=%02x:%02x fhandle=0x%08x "
"stateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
__entry->fhandle,
__entry->stateid_seq, __entry->stateid_hash
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"stateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
TP_printk(
"error=%ld (%s) name=%02x:%02x:%llu/%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"error=%ld (%s) inode=%02x:%02x:%llu",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->ino
)
"error=%ld (%s) oldname=%02x:%02x:%llu/%s "
"newname=%02x:%02x:%llu/%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->olddir,
__get_str(oldname),
TP_printk(
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"stateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"valid=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"dstaddr=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"stateid=%d:0x%08x dstaddr=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
TP_printk(
"error=%ld (%s) id=%u name=%s",
- -__entry->error, show_nfsv4_errors(__entry->error),
+ -__entry->error, show_nfs4_status(__entry->error),
__entry->id,
__get_str(name)
)
"offset=%lld count=%u res=%u stateid=%d:0x%08x "
"layoutstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"offset=%lld count=%u res=%u stateid=%d:0x%08x "
"layoutstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"offset=%lld count=%u layoutstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
#ifdef CONFIG_NFS_V4_1
DEFINE_NFS4_COMMIT_EVENT(nfs4_pnfs_commit_ds);
-TRACE_DEFINE_ENUM(IOMODE_READ);
-TRACE_DEFINE_ENUM(IOMODE_RW);
-TRACE_DEFINE_ENUM(IOMODE_ANY);
-
-#define show_pnfs_iomode(iomode) \
- __print_symbolic(iomode, \
- { IOMODE_READ, "READ" }, \
- { IOMODE_RW, "RW" }, \
- { IOMODE_ANY, "ANY" })
-
TRACE_EVENT(nfs4_layoutget,
TP_PROTO(
const struct nfs_open_context *ctx,
"iomode=%s offset=%llu count=%llu stateid=%d:0x%08x "
"layoutstateid=%d:0x%08x",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
- show_pnfs_iomode(__entry->iomode),
+ show_pnfs_layout_iomode(__entry->iomode),
(unsigned long long)__entry->offset,
(unsigned long long)__entry->count,
__entry->stateid_seq, __entry->stateid_hash,
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
- show_pnfs_iomode(__entry->iomode),
+ show_pnfs_layout_iomode(__entry->iomode),
(unsigned long long)__entry->pos,
(unsigned long long)__entry->count,
__entry->layoutstateid_seq, __entry->layoutstateid_hash,
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
- show_pnfs_iomode(__entry->iomode),
+ show_pnfs_layout_iomode(__entry->iomode),
(unsigned long long)__entry->pos,
(unsigned long long)__entry->count,
__entry->layoutstateid_seq, __entry->layoutstateid_hash,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"offset=%llu count=%u stateid=%d:0x%08x dstaddr=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"offset=%llu count=%u dstaddr=%s",
-__entry->error,
- show_nfsv4_errors(__entry->error),
+ show_nfs4_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
)
);
+TRACE_DEFINE_ENUM(NFS4_CONTENT_DATA);
+TRACE_DEFINE_ENUM(NFS4_CONTENT_HOLE);
+
+#define show_llseek_mode(what) \
+ __print_symbolic(what, \
+ { NFS4_CONTENT_DATA, "DATA" }, \
+ { NFS4_CONTENT_HOLE, "HOLE" })
+
+#ifdef CONFIG_NFS_V4_2
+TRACE_EVENT(nfs4_llseek,
+ TP_PROTO(
+ const struct inode *inode,
+ const struct nfs42_seek_args *args,
+ const struct nfs42_seek_res *res,
+ int error
+ ),
+
+ TP_ARGS(inode, args, res, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, fhandle)
+ __field(u32, fileid)
+ __field(dev_t, dev)
+ __field(int, stateid_seq)
+ __field(u32, stateid_hash)
+ __field(loff_t, offset_s)
+ __field(u32, what)
+ __field(loff_t, offset_r)
+ __field(u32, eof)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+ const struct nfs_fh *fh = args->sa_fh;
+
+ __entry->fileid = nfsi->fileid;
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fhandle = nfs_fhandle_hash(fh);
+ __entry->offset_s = args->sa_offset;
+ __entry->stateid_seq =
+ be32_to_cpu(args->sa_stateid.seqid);
+ __entry->stateid_hash =
+ nfs_stateid_hash(&args->sa_stateid);
+ __entry->what = args->sa_what;
+ if (error) {
+ __entry->error = -error;
+ __entry->offset_r = 0;
+ __entry->eof = 0;
+ } else {
+ __entry->error = 0;
+ __entry->offset_r = res->sr_offset;
+ __entry->eof = res->sr_eof;
+ }
+ ),
+
+ TP_printk(
+ "error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
+ "stateid=%d:0x%08x offset_s=%llu what=%s "
+ "offset_r=%llu eof=%u",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle,
+ __entry->stateid_seq, __entry->stateid_hash,
+ __entry->offset_s,
+ show_llseek_mode(__entry->what),
+ __entry->offset_r,
+ __entry->eof
+ )
+);
+
+DECLARE_EVENT_CLASS(nfs4_sparse_event,
+ TP_PROTO(
+ const struct inode *inode,
+ const struct nfs42_falloc_args *args,
+ int error
+ ),
+
+ TP_ARGS(inode, args, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(loff_t, offset)
+ __field(loff_t, len)
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(u64, fileid)
+ __field(int, stateid_seq)
+ __field(u32, stateid_hash)
+ ),
+
+ TP_fast_assign(
+ __entry->error = error < 0 ? -error : 0;
+ __entry->offset = args->falloc_offset;
+ __entry->len = args->falloc_length;
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fileid = NFS_FILEID(inode);
+ __entry->fhandle = nfs_fhandle_hash(NFS_FH(inode));
+ __entry->stateid_seq =
+ be32_to_cpu(args->falloc_stateid.seqid);
+ __entry->stateid_hash =
+ nfs_stateid_hash(&args->falloc_stateid);
+ ),
+
+ TP_printk(
+ "error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
+ "stateid=%d:0x%08x offset=%llu len=%llu",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle,
+ __entry->stateid_seq, __entry->stateid_hash,
+ (long long)__entry->offset,
+ (long long)__entry->len
+ )
+);
+#define DEFINE_NFS4_SPARSE_EVENT(name) \
+ DEFINE_EVENT(nfs4_sparse_event, name, \
+ TP_PROTO( \
+ const struct inode *inode, \
+ const struct nfs42_falloc_args *args, \
+ int error \
+ ), \
+ TP_ARGS(inode, args, error))
+DEFINE_NFS4_SPARSE_EVENT(nfs4_fallocate);
+DEFINE_NFS4_SPARSE_EVENT(nfs4_deallocate);
+
+TRACE_EVENT(nfs4_copy,
+ TP_PROTO(
+ const struct inode *src_inode,
+ const struct inode *dst_inode,
+ const struct nfs42_copy_args *args,
+ const struct nfs42_copy_res *res,
+ const struct nl4_server *nss,
+ int error
+ ),
+
+ TP_ARGS(src_inode, dst_inode, args, res, nss, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, src_fhandle)
+ __field(u32, src_fileid)
+ __field(u32, dst_fhandle)
+ __field(u32, dst_fileid)
+ __field(dev_t, src_dev)
+ __field(dev_t, dst_dev)
+ __field(int, src_stateid_seq)
+ __field(u32, src_stateid_hash)
+ __field(int, dst_stateid_seq)
+ __field(u32, dst_stateid_hash)
+ __field(loff_t, src_offset)
+ __field(loff_t, dst_offset)
+ __field(bool, sync)
+ __field(loff_t, len)
+ __field(int, res_stateid_seq)
+ __field(u32, res_stateid_hash)
+ __field(loff_t, res_count)
+ __field(bool, res_sync)
+ __field(bool, res_cons)
+ __field(bool, intra)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *src_nfsi = NFS_I(src_inode);
+ const struct nfs_inode *dst_nfsi = NFS_I(dst_inode);
+
+ __entry->src_fileid = src_nfsi->fileid;
+ __entry->src_dev = src_inode->i_sb->s_dev;
+ __entry->src_fhandle = nfs_fhandle_hash(args->src_fh);
+ __entry->src_offset = args->src_pos;
+ __entry->dst_fileid = dst_nfsi->fileid;
+ __entry->dst_dev = dst_inode->i_sb->s_dev;
+ __entry->dst_fhandle = nfs_fhandle_hash(args->dst_fh);
+ __entry->dst_offset = args->dst_pos;
+ __entry->len = args->count;
+ __entry->sync = args->sync;
+ __entry->src_stateid_seq =
+ be32_to_cpu(args->src_stateid.seqid);
+ __entry->src_stateid_hash =
+ nfs_stateid_hash(&args->src_stateid);
+ __entry->dst_stateid_seq =
+ be32_to_cpu(args->dst_stateid.seqid);
+ __entry->dst_stateid_hash =
+ nfs_stateid_hash(&args->dst_stateid);
+ __entry->intra = nss ? 0 : 1;
+ if (error) {
+ __entry->error = -error;
+ __entry->res_stateid_seq = 0;
+ __entry->res_stateid_hash = 0;
+ __entry->res_count = 0;
+ __entry->res_sync = 0;
+ __entry->res_cons = 0;
+ } else {
+ __entry->error = 0;
+ __entry->res_stateid_seq =
+ be32_to_cpu(res->write_res.stateid.seqid);
+ __entry->res_stateid_hash =
+ nfs_stateid_hash(&res->write_res.stateid);
+ __entry->res_count = res->write_res.count;
+ __entry->res_sync = res->synchronous;
+ __entry->res_cons = res->consecutive;
+ }
+ ),
+
+ TP_printk(
+ "error=%ld (%s) intra=%d src_fileid=%02x:%02x:%llu "
+ "src_fhandle=0x%08x dst_fileid=%02x:%02x:%llu "
+ "dst_fhandle=0x%08x src_stateid=%d:0x%08x "
+ "dst_stateid=%d:0x%08x src_offset=%llu dst_offset=%llu "
+ "len=%llu sync=%d cb_stateid=%d:0x%08x res_sync=%d "
+ "res_cons=%d res_count=%llu",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ __entry->intra,
+ MAJOR(__entry->src_dev), MINOR(__entry->src_dev),
+ (unsigned long long)__entry->src_fileid,
+ __entry->src_fhandle,
+ MAJOR(__entry->dst_dev), MINOR(__entry->dst_dev),
+ (unsigned long long)__entry->dst_fileid,
+ __entry->dst_fhandle,
+ __entry->src_stateid_seq, __entry->src_stateid_hash,
+ __entry->dst_stateid_seq, __entry->dst_stateid_hash,
+ __entry->src_offset,
+ __entry->dst_offset,
+ __entry->len,
+ __entry->sync,
+ __entry->res_stateid_seq, __entry->res_stateid_hash,
+ __entry->res_sync,
+ __entry->res_cons,
+ __entry->res_count
+ )
+);
+
+TRACE_EVENT(nfs4_clone,
+ TP_PROTO(
+ const struct inode *src_inode,
+ const struct inode *dst_inode,
+ const struct nfs42_clone_args *args,
+ int error
+ ),
+
+ TP_ARGS(src_inode, dst_inode, args, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, src_fhandle)
+ __field(u32, src_fileid)
+ __field(u32, dst_fhandle)
+ __field(u32, dst_fileid)
+ __field(dev_t, src_dev)
+ __field(dev_t, dst_dev)
+ __field(loff_t, src_offset)
+ __field(loff_t, dst_offset)
+ __field(int, src_stateid_seq)
+ __field(u32, src_stateid_hash)
+ __field(int, dst_stateid_seq)
+ __field(u32, dst_stateid_hash)
+ __field(loff_t, len)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *src_nfsi = NFS_I(src_inode);
+ const struct nfs_inode *dst_nfsi = NFS_I(dst_inode);
+
+ __entry->src_fileid = src_nfsi->fileid;
+ __entry->src_dev = src_inode->i_sb->s_dev;
+ __entry->src_fhandle = nfs_fhandle_hash(args->src_fh);
+ __entry->src_offset = args->src_offset;
+ __entry->dst_fileid = dst_nfsi->fileid;
+ __entry->dst_dev = dst_inode->i_sb->s_dev;
+ __entry->dst_fhandle = nfs_fhandle_hash(args->dst_fh);
+ __entry->dst_offset = args->dst_offset;
+ __entry->len = args->count;
+ __entry->error = error < 0 ? -error : 0;
+ __entry->src_stateid_seq =
+ be32_to_cpu(args->src_stateid.seqid);
+ __entry->src_stateid_hash =
+ nfs_stateid_hash(&args->src_stateid);
+ __entry->dst_stateid_seq =
+ be32_to_cpu(args->dst_stateid.seqid);
+ __entry->dst_stateid_hash =
+ nfs_stateid_hash(&args->dst_stateid);
+ ),
+
+ TP_printk(
+ "error=%ld (%s) src_fileid=%02x:%02x:%llu "
+ "src_fhandle=0x%08x dst_fileid=%02x:%02x:%llu "
+ "dst_fhandle=0x%08x src_stateid=%d:0x%08x "
+ "dst_stateid=%d:0x%08x src_offset=%llu "
+ "dst_offset=%llu len=%llu",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ MAJOR(__entry->src_dev), MINOR(__entry->src_dev),
+ (unsigned long long)__entry->src_fileid,
+ __entry->src_fhandle,
+ MAJOR(__entry->dst_dev), MINOR(__entry->dst_dev),
+ (unsigned long long)__entry->dst_fileid,
+ __entry->dst_fhandle,
+ __entry->src_stateid_seq, __entry->src_stateid_hash,
+ __entry->dst_stateid_seq, __entry->dst_stateid_hash,
+ __entry->src_offset,
+ __entry->dst_offset,
+ __entry->len
+ )
+);
+
+TRACE_EVENT(nfs4_copy_notify,
+ TP_PROTO(
+ const struct inode *inode,
+ const struct nfs42_copy_notify_args *args,
+ const struct nfs42_copy_notify_res *res,
+ int error
+ ),
+
+ TP_ARGS(inode, args, res, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, fhandle)
+ __field(u32, fileid)
+ __field(dev_t, dev)
+ __field(int, stateid_seq)
+ __field(u32, stateid_hash)
+ __field(int, res_stateid_seq)
+ __field(u32, res_stateid_hash)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->fileid = nfsi->fileid;
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fhandle = nfs_fhandle_hash(args->cna_src_fh);
+ __entry->stateid_seq =
+ be32_to_cpu(args->cna_src_stateid.seqid);
+ __entry->stateid_hash =
+ nfs_stateid_hash(&args->cna_src_stateid);
+ if (error) {
+ __entry->error = -error;
+ __entry->res_stateid_seq = 0;
+ __entry->res_stateid_hash = 0;
+ } else {
+ __entry->error = 0;
+ __entry->res_stateid_seq =
+ be32_to_cpu(res->cnr_stateid.seqid);
+ __entry->res_stateid_hash =
+ nfs_stateid_hash(&res->cnr_stateid);
+ }
+ ),
+
+ TP_printk(
+ "error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
+ "stateid=%d:0x%08x res_stateid=%d:0x%08x",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle,
+ __entry->stateid_seq, __entry->stateid_hash,
+ __entry->res_stateid_seq, __entry->res_stateid_hash
+ )
+);
+
+TRACE_EVENT(nfs4_offload_cancel,
+ TP_PROTO(
+ const struct nfs42_offload_status_args *args,
+ int error
+ ),
+
+ TP_ARGS(args, error),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, error)
+ __field(u32, fhandle)
+ __field(int, stateid_seq)
+ __field(u32, stateid_hash)
+ ),
+
+ TP_fast_assign(
+ __entry->fhandle = nfs_fhandle_hash(args->osa_src_fh);
+ __entry->error = error < 0 ? -error : 0;
+ __entry->stateid_seq =
+ be32_to_cpu(args->osa_stateid.seqid);
+ __entry->stateid_hash =
+ nfs_stateid_hash(&args->osa_stateid);
+ ),
+
+ TP_printk(
+ "error=%ld (%s) fhandle=0x%08x stateid=%d:0x%08x",
+ -__entry->error,
+ show_nfs4_status(__entry->error),
+ __entry->fhandle,
+ __entry->stateid_seq, __entry->stateid_hash
+ )
+);
+#endif /* CONFIG_NFS_V4_2 */
#endif /* CONFIG_NFS_V4_1 */
static int decode_compound_hdr(struct xdr_stream *xdr, struct compound_hdr *hdr)
{
- __be32 *p;
+ ssize_t ret;
+ void *ptr;
+ u32 tmp;
- p = xdr_inline_decode(xdr, 8);
- if (unlikely(!p))
+ if (xdr_stream_decode_u32(xdr, &tmp) < 0)
return -EIO;
- hdr->status = be32_to_cpup(p++);
- hdr->taglen = be32_to_cpup(p);
+ hdr->status = tmp;
- p = xdr_inline_decode(xdr, hdr->taglen + 4);
- if (unlikely(!p))
+ ret = xdr_stream_decode_opaque_inline(xdr, &ptr, NFS4_OPAQUE_LIMIT);
+ if (ret < 0)
+ return -EIO;
+ hdr->taglen = ret;
+ hdr->tag = ptr;
+
+ if (xdr_stream_decode_u32(xdr, &tmp) < 0)
return -EIO;
- hdr->tag = (char *)p;
- p += XDR_QUADLEN(hdr->taglen);
- hdr->nops = be32_to_cpup(p);
+ hdr->nops = tmp;
if (unlikely(hdr->nops < 1))
return nfs4_stat_to_errno(hdr->status);
return 0;
static int decode_getfattr_attrs(struct xdr_stream *xdr, uint32_t *bitmap,
struct nfs_fattr *fattr, struct nfs_fh *fh,
- struct nfs4_fs_locations *fs_loc, struct nfs4_label *label,
- const struct nfs_server *server)
+ struct nfs4_fs_locations *fs_loc, const struct nfs_server *server)
{
int status;
umode_t fmode = 0;
if (status < 0)
goto xdr_error;
- if (label) {
- status = decode_attr_security_label(xdr, bitmap, label);
+ if (fattr->label) {
+ status = decode_attr_security_label(xdr, bitmap, fattr->label);
if (status < 0)
goto xdr_error;
fattr->valid |= status;
static int decode_getfattr_generic(struct xdr_stream *xdr, struct nfs_fattr *fattr,
struct nfs_fh *fh, struct nfs4_fs_locations *fs_loc,
- struct nfs4_label *label, const struct nfs_server *server)
+ const struct nfs_server *server)
{
unsigned int savep;
uint32_t attrlen,
if (status < 0)
goto xdr_error;
- status = decode_getfattr_attrs(xdr, bitmap, fattr, fh, fs_loc,
- label, server);
+ status = decode_getfattr_attrs(xdr, bitmap, fattr, fh, fs_loc, server);
if (status < 0)
goto xdr_error;
return status;
}
-static int decode_getfattr_label(struct xdr_stream *xdr, struct nfs_fattr *fattr,
- struct nfs4_label *label, const struct nfs_server *server)
-{
- return decode_getfattr_generic(xdr, fattr, NULL, NULL, label, server);
-}
-
static int decode_getfattr(struct xdr_stream *xdr, struct nfs_fattr *fattr,
const struct nfs_server *server)
{
- return decode_getfattr_generic(xdr, fattr, NULL, NULL, NULL, server);
+ return decode_getfattr_generic(xdr, fattr, NULL, NULL, server);
}
/*
static int decode_op_map(struct xdr_stream *xdr, struct nfs4_op_map *op_map)
{
- __be32 *p;
- uint32_t bitmap_words;
- unsigned int i;
-
- p = xdr_inline_decode(xdr, 4);
- if (!p)
- return -EIO;
- bitmap_words = be32_to_cpup(p++);
- if (bitmap_words > NFS4_OP_MAP_NUM_WORDS)
+ if (xdr_stream_decode_uint32_array(xdr, op_map->u.words,
+ ARRAY_SIZE(op_map->u.words)) < 0)
return -EIO;
- p = xdr_inline_decode(xdr, 4 * bitmap_words);
- for (i = 0; i < bitmap_words; i++)
- op_map->u.words[i] = be32_to_cpup(p++);
-
return 0;
}
status = decode_getfh(xdr, res->fh);
if (status)
goto out;
- status = decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ status = decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
status = decode_getfh(xdr, res->fh);
if (status)
goto out;
- status = decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ status = decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
goto out;
status = decode_getfh(xdr, res->fh);
if (status == 0)
- status = decode_getfattr_label(xdr, res->fattr,
- res->label, res->server);
+ status = decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
status = decode_restorefh(xdr);
if (status)
goto out;
- decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
status = decode_getfh(xdr, res->fh);
if (status)
goto out;
- decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
status = decode_putfh(xdr);
if (status)
goto out;
- status = decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ status = decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
goto out;
if (res->access_request)
decode_access(xdr, &res->access_supported, &res->access_result);
- decode_getfattr_label(xdr, res->f_attr, res->f_label, res->server);
+ decode_getfattr(xdr, res->f_attr, res->server);
if (res->lg_res)
decode_layoutget(xdr, rqstp, res->lg_res);
out:
status = decode_setattr(xdr);
if (status)
goto out;
- decode_getfattr_label(xdr, res->fattr, res->label, res->server);
+ decode_getfattr(xdr, res->fattr, res->server);
out:
return status;
}
status = decode_getfattr_generic(xdr,
&res->fs_locations->fattr,
NULL, res->fs_locations,
- NULL, res->fs_locations->server);
+ res->fs_locations->server);
if (status)
goto out;
if (res->renew)
status = decode_getfattr_generic(xdr,
&res->fs_locations->fattr,
NULL, res->fs_locations,
- NULL, res->fs_locations->server);
+ res->fs_locations->server);
}
out:
return status;
return -EAGAIN;
if (decode_getfattr_attrs(xdr, bitmap, entry->fattr, entry->fh,
- NULL, entry->label, entry->server) < 0)
+ NULL, entry->server) < 0)
return -EAGAIN;
if (entry->fattr->valid & NFS_ATTR_FATTR_MOUNTED_ON_FILEID)
entry->ino = entry->fattr->mounted_on_fileid;
#include <linux/tracepoint.h>
#include <linux/iversion.h>
-TRACE_DEFINE_ENUM(DT_UNKNOWN);
-TRACE_DEFINE_ENUM(DT_FIFO);
-TRACE_DEFINE_ENUM(DT_CHR);
-TRACE_DEFINE_ENUM(DT_DIR);
-TRACE_DEFINE_ENUM(DT_BLK);
-TRACE_DEFINE_ENUM(DT_REG);
-TRACE_DEFINE_ENUM(DT_LNK);
-TRACE_DEFINE_ENUM(DT_SOCK);
-TRACE_DEFINE_ENUM(DT_WHT);
-
-#define nfs_show_file_type(ftype) \
- __print_symbolic(ftype, \
- { DT_UNKNOWN, "UNKNOWN" }, \
- { DT_FIFO, "FIFO" }, \
- { DT_CHR, "CHR" }, \
- { DT_DIR, "DIR" }, \
- { DT_BLK, "BLK" }, \
- { DT_REG, "REG" }, \
- { DT_LNK, "LNK" }, \
- { DT_SOCK, "SOCK" }, \
- { DT_WHT, "WHT" })
-
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_DATA);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_ATIME);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_ACCESS);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_ACL);
-TRACE_DEFINE_ENUM(NFS_INO_REVAL_PAGECACHE);
-TRACE_DEFINE_ENUM(NFS_INO_REVAL_FORCED);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_LABEL);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_CHANGE);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_CTIME);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_MTIME);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_SIZE);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_OTHER);
-TRACE_DEFINE_ENUM(NFS_INO_DATA_INVAL_DEFER);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_BLOCKS);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_XATTR);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_NLINK);
-TRACE_DEFINE_ENUM(NFS_INO_INVALID_MODE);
+#include <trace/events/fs.h>
+#include <trace/events/nfs.h>
+#include <trace/events/sunrpc_base.h>
#define nfs_show_cache_validity(v) \
__print_flags(v, "|", \
{ NFS_INO_INVALID_NLINK, "INVALID_NLINK" }, \
{ NFS_INO_INVALID_MODE, "INVALID_MODE" })
-TRACE_DEFINE_ENUM(NFS_INO_ADVISE_RDPLUS);
-TRACE_DEFINE_ENUM(NFS_INO_STALE);
-TRACE_DEFINE_ENUM(NFS_INO_ACL_LRU_SET);
-TRACE_DEFINE_ENUM(NFS_INO_INVALIDATING);
-TRACE_DEFINE_ENUM(NFS_INO_FSCACHE);
-TRACE_DEFINE_ENUM(NFS_INO_FSCACHE_LOCK);
-TRACE_DEFINE_ENUM(NFS_INO_LAYOUTCOMMIT);
-TRACE_DEFINE_ENUM(NFS_INO_LAYOUTCOMMITTING);
-TRACE_DEFINE_ENUM(NFS_INO_LAYOUTSTATS);
-TRACE_DEFINE_ENUM(NFS_INO_ODIRECT);
-
#define nfs_show_nfsi_flags(v) \
__print_flags(v, "|", \
{ BIT(NFS_INO_ADVISE_RDPLUS), "ADVISE_RDPLUS" }, \
"error=%ld (%s) fileid=%02x:%02x:%llu fhandle=0x%08x "
"type=%u (%s) version=%llu size=%lld "
"cache_validity=0x%lx (%s) nfs_flags=0x%lx (%s)",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
__entry->type,
- nfs_show_file_type(__entry->type),
+ show_fs_dirent_type(__entry->type),
(unsigned long long)__entry->version,
(long long)__entry->size,
__entry->cache_validity,
"type=%u (%s) version=%llu size=%lld "
"cache_validity=0x%lx (%s) nfs_flags=0x%lx (%s) "
"mask=0x%x permitted=0x%x",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->fileid,
__entry->fhandle,
__entry->type,
- nfs_show_file_type(__entry->type),
+ show_fs_dirent_type(__entry->type),
(unsigned long long)__entry->version,
(long long)__entry->size,
__entry->cache_validity,
)
);
-TRACE_DEFINE_ENUM(LOOKUP_FOLLOW);
-TRACE_DEFINE_ENUM(LOOKUP_DIRECTORY);
-TRACE_DEFINE_ENUM(LOOKUP_AUTOMOUNT);
-TRACE_DEFINE_ENUM(LOOKUP_PARENT);
-TRACE_DEFINE_ENUM(LOOKUP_REVAL);
-TRACE_DEFINE_ENUM(LOOKUP_RCU);
-TRACE_DEFINE_ENUM(LOOKUP_OPEN);
-TRACE_DEFINE_ENUM(LOOKUP_CREATE);
-TRACE_DEFINE_ENUM(LOOKUP_EXCL);
-TRACE_DEFINE_ENUM(LOOKUP_RENAME_TARGET);
-TRACE_DEFINE_ENUM(LOOKUP_EMPTY);
-TRACE_DEFINE_ENUM(LOOKUP_DOWN);
-
-#define show_lookup_flags(flags) \
- __print_flags(flags, "|", \
- { LOOKUP_FOLLOW, "FOLLOW" }, \
- { LOOKUP_DIRECTORY, "DIRECTORY" }, \
- { LOOKUP_AUTOMOUNT, "AUTOMOUNT" }, \
- { LOOKUP_PARENT, "PARENT" }, \
- { LOOKUP_REVAL, "REVAL" }, \
- { LOOKUP_RCU, "RCU" }, \
- { LOOKUP_OPEN, "OPEN" }, \
- { LOOKUP_CREATE, "CREATE" }, \
- { LOOKUP_EXCL, "EXCL" }, \
- { LOOKUP_RENAME_TARGET, "RENAME_TARGET" }, \
- { LOOKUP_EMPTY, "EMPTY" }, \
- { LOOKUP_DOWN, "DOWN" })
+DECLARE_EVENT_CLASS(nfs_update_size_class,
+ TP_PROTO(
+ const struct inode *inode,
+ loff_t new_size
+ ),
+
+ TP_ARGS(inode, new_size),
+
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(u64, fileid)
+ __field(u64, version)
+ __field(loff_t, cur_size)
+ __field(loff_t, new_size)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fhandle = nfs_fhandle_hash(&nfsi->fh);
+ __entry->fileid = nfsi->fileid;
+ __entry->version = inode_peek_iversion_raw(inode);
+ __entry->cur_size = i_size_read(inode);
+ __entry->new_size = new_size;
+ ),
+
+ TP_printk(
+ "fileid=%02x:%02x:%llu fhandle=0x%08x version=%llu cursize=%lld newsize=%lld",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle, __entry->version,
+ __entry->cur_size, __entry->new_size
+ )
+);
+
+#define DEFINE_NFS_UPDATE_SIZE_EVENT(name) \
+ DEFINE_EVENT(nfs_update_size_class, nfs_size_##name, \
+ TP_PROTO( \
+ const struct inode *inode, \
+ loff_t new_size \
+ ), \
+ TP_ARGS(inode, new_size))
+
+DEFINE_NFS_UPDATE_SIZE_EVENT(truncate);
+DEFINE_NFS_UPDATE_SIZE_EVENT(wcc);
+DEFINE_NFS_UPDATE_SIZE_EVENT(update);
+DEFINE_NFS_UPDATE_SIZE_EVENT(grow);
DECLARE_EVENT_CLASS(nfs_lookup_event,
TP_PROTO(
TP_printk(
"flags=0x%lx (%s) name=%02x:%02x:%llu/%s",
__entry->flags,
- show_lookup_flags(__entry->flags),
+ show_fs_lookup_flags(__entry->flags),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"error=%ld (%s) flags=0x%lx (%s) name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
__entry->flags,
- show_lookup_flags(__entry->flags),
+ show_fs_lookup_flags(__entry->flags),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
DEFINE_NFS_LOOKUP_EVENT(nfs_lookup_revalidate_enter);
DEFINE_NFS_LOOKUP_EVENT_DONE(nfs_lookup_revalidate_exit);
-TRACE_DEFINE_ENUM(O_WRONLY);
-TRACE_DEFINE_ENUM(O_RDWR);
-TRACE_DEFINE_ENUM(O_CREAT);
-TRACE_DEFINE_ENUM(O_EXCL);
-TRACE_DEFINE_ENUM(O_NOCTTY);
-TRACE_DEFINE_ENUM(O_TRUNC);
-TRACE_DEFINE_ENUM(O_APPEND);
-TRACE_DEFINE_ENUM(O_NONBLOCK);
-TRACE_DEFINE_ENUM(O_DSYNC);
-TRACE_DEFINE_ENUM(O_DIRECT);
-TRACE_DEFINE_ENUM(O_LARGEFILE);
-TRACE_DEFINE_ENUM(O_DIRECTORY);
-TRACE_DEFINE_ENUM(O_NOFOLLOW);
-TRACE_DEFINE_ENUM(O_NOATIME);
-TRACE_DEFINE_ENUM(O_CLOEXEC);
-
-#define show_open_flags(flags) \
- __print_flags(flags, "|", \
- { O_WRONLY, "O_WRONLY" }, \
- { O_RDWR, "O_RDWR" }, \
- { O_CREAT, "O_CREAT" }, \
- { O_EXCL, "O_EXCL" }, \
- { O_NOCTTY, "O_NOCTTY" }, \
- { O_TRUNC, "O_TRUNC" }, \
- { O_APPEND, "O_APPEND" }, \
- { O_NONBLOCK, "O_NONBLOCK" }, \
- { O_DSYNC, "O_DSYNC" }, \
- { O_DIRECT, "O_DIRECT" }, \
- { O_LARGEFILE, "O_LARGEFILE" }, \
- { O_DIRECTORY, "O_DIRECTORY" }, \
- { O_NOFOLLOW, "O_NOFOLLOW" }, \
- { O_NOATIME, "O_NOATIME" }, \
- { O_CLOEXEC, "O_CLOEXEC" })
-
-#define show_fmode_flags(mode) \
- __print_flags(mode, "|", \
- { ((__force unsigned long)FMODE_READ), "READ" }, \
- { ((__force unsigned long)FMODE_WRITE), "WRITE" }, \
- { ((__force unsigned long)FMODE_EXEC), "EXEC" })
-
TRACE_EVENT(nfs_atomic_open_enter,
TP_PROTO(
const struct inode *dir,
TP_STRUCT__entry(
__field(unsigned long, flags)
- __field(unsigned int, fmode)
+ __field(unsigned long, fmode)
__field(dev_t, dev)
__field(u64, dir)
__string(name, ctx->dentry->d_name.name)
__entry->dev = dir->i_sb->s_dev;
__entry->dir = NFS_FILEID(dir);
__entry->flags = flags;
- __entry->fmode = (__force unsigned int)ctx->mode;
+ __entry->fmode = (__force unsigned long)ctx->mode;
__assign_str(name, ctx->dentry->d_name.name);
),
TP_printk(
"flags=0x%lx (%s) fmode=%s name=%02x:%02x:%llu/%s",
__entry->flags,
- show_open_flags(__entry->flags),
- show_fmode_flags(__entry->fmode),
+ show_fs_fcntl_open_flags(__entry->flags),
+ show_fs_fmode_flags(__entry->fmode),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_STRUCT__entry(
__field(unsigned long, error)
__field(unsigned long, flags)
- __field(unsigned int, fmode)
+ __field(unsigned long, fmode)
__field(dev_t, dev)
__field(u64, dir)
__string(name, ctx->dentry->d_name.name)
__entry->dev = dir->i_sb->s_dev;
__entry->dir = NFS_FILEID(dir);
__entry->flags = flags;
- __entry->fmode = (__force unsigned int)ctx->mode;
+ __entry->fmode = (__force unsigned long)ctx->mode;
__assign_str(name, ctx->dentry->d_name.name);
),
TP_printk(
"error=%ld (%s) flags=0x%lx (%s) fmode=%s "
"name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
__entry->flags,
- show_open_flags(__entry->flags),
- show_fmode_flags(__entry->fmode),
+ show_fs_fcntl_open_flags(__entry->flags),
+ show_fs_fmode_flags(__entry->fmode),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"flags=0x%lx (%s) name=%02x:%02x:%llu/%s",
__entry->flags,
- show_open_flags(__entry->flags),
+ show_fs_fcntl_open_flags(__entry->flags),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"error=%ld (%s) flags=0x%lx (%s) name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
__entry->flags,
- show_open_flags(__entry->flags),
+ show_fs_fcntl_open_flags(__entry->flags),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"error=%ld (%s) name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
TP_printk(
"error=%ld (%s) fileid=%02x:%02x:%llu name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
__entry->fileid,
MAJOR(__entry->dev), MINOR(__entry->dev),
TP_printk(
"error=%ld (%s) old_name=%02x:%02x:%llu/%s "
"new_name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->old_dir,
__get_str(old_name),
TP_printk(
"error=%ld (%s) name=%02x:%02x:%llu/%s",
- -__entry->error, nfs_show_status(__entry->error),
+ -__entry->error, show_nfs_status(__entry->error),
MAJOR(__entry->dev), MINOR(__entry->dev),
(unsigned long long)__entry->dir,
__get_str(name)
)
);
+TRACE_EVENT(nfs_aop_readpage,
+ TP_PROTO(
+ const struct inode *inode,
+ struct page *page
+ ),
+
+ TP_ARGS(inode, page),
+
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(u64, fileid)
+ __field(u64, version)
+ __field(loff_t, offset)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fileid = nfsi->fileid;
+ __entry->fhandle = nfs_fhandle_hash(&nfsi->fh);
+ __entry->version = inode_peek_iversion_raw(inode);
+ __entry->offset = page_index(page) << PAGE_SHIFT;
+ ),
+
+ TP_printk(
+ "fileid=%02x:%02x:%llu fhandle=0x%08x version=%llu offset=%lld",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle, __entry->version,
+ __entry->offset
+ )
+);
+
+TRACE_EVENT(nfs_aop_readpage_done,
+ TP_PROTO(
+ const struct inode *inode,
+ struct page *page,
+ int ret
+ ),
+
+ TP_ARGS(inode, page, ret),
+
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(int, ret)
+ __field(u64, fileid)
+ __field(u64, version)
+ __field(loff_t, offset)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fileid = nfsi->fileid;
+ __entry->fhandle = nfs_fhandle_hash(&nfsi->fh);
+ __entry->version = inode_peek_iversion_raw(inode);
+ __entry->offset = page_index(page) << PAGE_SHIFT;
+ __entry->ret = ret;
+ ),
+
+ TP_printk(
+ "fileid=%02x:%02x:%llu fhandle=0x%08x version=%llu offset=%lld ret=%d",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle, __entry->version,
+ __entry->offset, __entry->ret
+ )
+);
+
+TRACE_EVENT(nfs_aop_readahead,
+ TP_PROTO(
+ const struct inode *inode,
+ struct page *page,
+ unsigned int nr_pages
+ ),
+
+ TP_ARGS(inode, page, nr_pages),
+
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(u64, fileid)
+ __field(u64, version)
+ __field(loff_t, offset)
+ __field(unsigned int, nr_pages)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fileid = nfsi->fileid;
+ __entry->fhandle = nfs_fhandle_hash(&nfsi->fh);
+ __entry->version = inode_peek_iversion_raw(inode);
+ __entry->offset = page_index(page) << PAGE_SHIFT;
+ __entry->nr_pages = nr_pages;
+ ),
+
+ TP_printk(
+ "fileid=%02x:%02x:%llu fhandle=0x%08x version=%llu offset=%lld nr_pages=%u",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle, __entry->version,
+ __entry->offset, __entry->nr_pages
+ )
+);
+
+TRACE_EVENT(nfs_aop_readahead_done,
+ TP_PROTO(
+ const struct inode *inode,
+ unsigned int nr_pages,
+ int ret
+ ),
+
+ TP_ARGS(inode, nr_pages, ret),
+
+ TP_STRUCT__entry(
+ __field(dev_t, dev)
+ __field(u32, fhandle)
+ __field(int, ret)
+ __field(u64, fileid)
+ __field(u64, version)
+ __field(loff_t, offset)
+ __field(unsigned int, nr_pages)
+ ),
+
+ TP_fast_assign(
+ const struct nfs_inode *nfsi = NFS_I(inode);
+
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->fileid = nfsi->fileid;
+ __entry->fhandle = nfs_fhandle_hash(&nfsi->fh);
+ __entry->version = inode_peek_iversion_raw(inode);
+ __entry->nr_pages = nr_pages;
+ __entry->ret = ret;
+ ),
+
+ TP_printk(
+ "fileid=%02x:%02x:%llu fhandle=0x%08x version=%llu nr_pages=%u ret=%d",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long long)__entry->fileid,
+ __entry->fhandle, __entry->version,
+ __entry->nr_pages, __entry->ret
+ )
+);
+
TRACE_EVENT(nfs_initiate_read,
TP_PROTO(
const struct nfs_pgio_header *hdr
)
);
-TRACE_DEFINE_ENUM(NFS_UNSTABLE);
-TRACE_DEFINE_ENUM(NFS_DATA_SYNC);
-TRACE_DEFINE_ENUM(NFS_FILE_SYNC);
-
-#define nfs_show_stable(stable) \
- __print_symbolic(stable, \
- { NFS_UNSTABLE, "UNSTABLE" }, \
- { NFS_DATA_SYNC, "DATA_SYNC" }, \
- { NFS_FILE_SYNC, "FILE_SYNC" })
-
TRACE_EVENT(nfs_initiate_write,
TP_PROTO(
const struct nfs_pgio_header *hdr
__field(u64, fileid)
__field(loff_t, offset)
__field(u32, count)
- __field(enum nfs3_stable_how, stable)
+ __field(unsigned long, stable)
),
TP_fast_assign(
(unsigned long long)__entry->fileid,
__entry->fhandle,
(long long)__entry->offset, __entry->count,
- nfs_show_stable(__entry->stable)
+ show_nfs_stable_how(__entry->stable)
)
);
__field(u32, arg_count)
__field(u32, res_count)
__field(int, status)
- __field(enum nfs3_stable_how, stable)
+ __field(unsigned long, stable)
__array(char, verifier, NFS4_VERIFIER_SIZE)
),
__entry->fhandle,
(long long)__entry->offset, __entry->arg_count,
__entry->res_count, __entry->status,
- nfs_show_stable(__entry->stable),
- __print_hex_str(__entry->verifier, NFS4_VERIFIER_SIZE)
+ show_nfs_stable_how(__entry->stable),
+ show_nfs4_verifier(__entry->verifier)
)
);
__field(u64, fileid)
__field(loff_t, offset)
__field(int, status)
- __field(enum nfs3_stable_how, stable)
+ __field(unsigned long, stable)
__array(char, verifier, NFS4_VERIFIER_SIZE)
),
(unsigned long long)__entry->fileid,
__entry->fhandle,
(long long)__entry->offset, __entry->status,
- nfs_show_stable(__entry->stable),
- __print_hex_str(__entry->verifier, NFS4_VERIFIER_SIZE)
+ show_nfs_stable_how(__entry->stable),
+ show_nfs4_verifier(__entry->verifier)
)
);
)
);
-TRACE_DEFINE_ENUM(NFS_OK);
-TRACE_DEFINE_ENUM(NFSERR_PERM);
-TRACE_DEFINE_ENUM(NFSERR_NOENT);
-TRACE_DEFINE_ENUM(NFSERR_IO);
-TRACE_DEFINE_ENUM(NFSERR_NXIO);
-TRACE_DEFINE_ENUM(ECHILD);
-TRACE_DEFINE_ENUM(NFSERR_EAGAIN);
-TRACE_DEFINE_ENUM(NFSERR_ACCES);
-TRACE_DEFINE_ENUM(NFSERR_EXIST);
-TRACE_DEFINE_ENUM(NFSERR_XDEV);
-TRACE_DEFINE_ENUM(NFSERR_NODEV);
-TRACE_DEFINE_ENUM(NFSERR_NOTDIR);
-TRACE_DEFINE_ENUM(NFSERR_ISDIR);
-TRACE_DEFINE_ENUM(NFSERR_INVAL);
-TRACE_DEFINE_ENUM(NFSERR_FBIG);
-TRACE_DEFINE_ENUM(NFSERR_NOSPC);
-TRACE_DEFINE_ENUM(NFSERR_ROFS);
-TRACE_DEFINE_ENUM(NFSERR_MLINK);
-TRACE_DEFINE_ENUM(NFSERR_OPNOTSUPP);
-TRACE_DEFINE_ENUM(NFSERR_NAMETOOLONG);
-TRACE_DEFINE_ENUM(NFSERR_NOTEMPTY);
-TRACE_DEFINE_ENUM(NFSERR_DQUOT);
-TRACE_DEFINE_ENUM(NFSERR_STALE);
-TRACE_DEFINE_ENUM(NFSERR_REMOTE);
-TRACE_DEFINE_ENUM(NFSERR_WFLUSH);
-TRACE_DEFINE_ENUM(NFSERR_BADHANDLE);
-TRACE_DEFINE_ENUM(NFSERR_NOT_SYNC);
-TRACE_DEFINE_ENUM(NFSERR_BAD_COOKIE);
-TRACE_DEFINE_ENUM(NFSERR_NOTSUPP);
-TRACE_DEFINE_ENUM(NFSERR_TOOSMALL);
-TRACE_DEFINE_ENUM(NFSERR_SERVERFAULT);
-TRACE_DEFINE_ENUM(NFSERR_BADTYPE);
-TRACE_DEFINE_ENUM(NFSERR_JUKEBOX);
-
-#define nfs_show_status(x) \
- __print_symbolic(x, \
- { NFS_OK, "OK" }, \
- { NFSERR_PERM, "PERM" }, \
- { NFSERR_NOENT, "NOENT" }, \
- { NFSERR_IO, "IO" }, \
- { NFSERR_NXIO, "NXIO" }, \
- { ECHILD, "CHILD" }, \
- { NFSERR_EAGAIN, "AGAIN" }, \
- { NFSERR_ACCES, "ACCES" }, \
- { NFSERR_EXIST, "EXIST" }, \
- { NFSERR_XDEV, "XDEV" }, \
- { NFSERR_NODEV, "NODEV" }, \
- { NFSERR_NOTDIR, "NOTDIR" }, \
- { NFSERR_ISDIR, "ISDIR" }, \
- { NFSERR_INVAL, "INVAL" }, \
- { NFSERR_FBIG, "FBIG" }, \
- { NFSERR_NOSPC, "NOSPC" }, \
- { NFSERR_ROFS, "ROFS" }, \
- { NFSERR_MLINK, "MLINK" }, \
- { NFSERR_OPNOTSUPP, "OPNOTSUPP" }, \
- { NFSERR_NAMETOOLONG, "NAMETOOLONG" }, \
- { NFSERR_NOTEMPTY, "NOTEMPTY" }, \
- { NFSERR_DQUOT, "DQUOT" }, \
- { NFSERR_STALE, "STALE" }, \
- { NFSERR_REMOTE, "REMOTE" }, \
- { NFSERR_WFLUSH, "WFLUSH" }, \
- { NFSERR_BADHANDLE, "BADHANDLE" }, \
- { NFSERR_NOT_SYNC, "NOTSYNC" }, \
- { NFSERR_BAD_COOKIE, "BADCOOKIE" }, \
- { NFSERR_NOTSUPP, "NOTSUPP" }, \
- { NFSERR_TOOSMALL, "TOOSMALL" }, \
- { NFSERR_SERVERFAULT, "REMOTEIO" }, \
- { NFSERR_BADTYPE, "BADTYPE" }, \
- { NFSERR_JUKEBOX, "JUKEBOX" })
-
DECLARE_EVENT_CLASS(nfs_xdr_event,
TP_PROTO(
const struct xdr_stream *xdr,
__assign_str(procedure, task->tk_msg.rpc_proc->p_name);
),
- TP_printk(
- "task:%u@%d xid=0x%08x %sv%d %s error=%ld (%s)",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x %sv%d %s error=%ld (%s)",
__entry->task_id, __entry->client_id, __entry->xid,
__get_str(program), __entry->version,
__get_str(procedure), -__entry->error,
- nfs_show_status(__entry->error)
+ show_nfs_status(__entry->error)
)
);
#define DEFINE_NFS_XDR_EVENT(name) \
void
nfs_page_clear_headlock(struct nfs_page *req)
{
- smp_mb__before_atomic();
- clear_bit(PG_HEADLOCK, &req->wb_flags);
+ clear_bit_unlock(PG_HEADLOCK, &req->wb_flags);
smp_mb__after_atomic();
if (!test_bit(PG_CONTENDED1, &req->wb_flags))
return;
*/
void nfs_unlock_request(struct nfs_page *req)
{
- if (!NFS_WBACK_BUSY(req)) {
- printk(KERN_ERR "NFS: Invalid unlock attempted\n");
- BUG();
- }
- smp_mb__before_atomic();
- clear_bit(PG_BUSY, &req->wb_flags);
+ clear_bit_unlock(PG_BUSY, &req->wb_flags);
smp_mb__after_atomic();
if (!test_bit(PG_CONTENDED2, &req->wb_flags))
return;
struct nfs_pgio_header *hdr = calldata;
struct inode *inode = hdr->inode;
- dprintk("NFS: %s: %5u, (status %d)\n", __func__,
- task->tk_pid, task->tk_status);
-
if (hdr->rw_ops->rw_done(task, hdr, inode) != 0)
return;
if (task->tk_status < 0)
PNFS_TRY_AGAIN = 2,
};
-/* error codes for internal use */
-#define NFS4ERR_RESET_TO_MDS 12001
-#define NFS4ERR_RESET_TO_PNFS 12002
-
#ifdef CONFIG_NFS_V4_1
#define LAYOUT_NFSV4_1_MODULE_PREFIX "nfs-layouttype4"
{
struct pnfs_ds_commit_info *fl_cinfo = cinfo->ds;
- if (!lseg || !fl_cinfo->ops->mark_request_commit)
+ if (!lseg || !fl_cinfo->ops || !fl_cinfo->ops->mark_request_commit)
return false;
fl_cinfo->ops->mark_request_commit(req, lseg, cinfo, ds_commit_idx);
return true;
goto out_error;
data->ds_commit_index = i;
list_add_tail(&data->list, list);
- atomic_inc(&cinfo->mds->rpcs_out);
nreq++;
}
mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
data->ds_commit_index = -1;
list_splice_init(mds_pages, &data->pages);
list_add_tail(&data->list, &list);
- atomic_inc(&cinfo->mds->rpcs_out);
nreq++;
}
}
smp_wmb();
- ds->ds_clp = clp;
+ WRITE_ONCE(ds->ds_clp, clp);
dprintk("%s [new] addr: %s\n", __func__, ds->ds_remotestr);
out:
return status;
}
smp_wmb();
- ds->ds_clp = clp;
+ WRITE_ONCE(ds->ds_clp, clp);
dprintk("%s [new] addr: %s\n", __func__, ds->ds_remotestr);
out:
return status;
info->dtpref = fsinfo.tsize;
info->maxfilesize = 0x7FFFFFFF;
info->lease_time = 0;
- info->change_attr_type = NFS4_CHANGE_TYPE_IS_TIME_METADATA;
+ info->change_attr_type = NFS4_CHANGE_TYPE_IS_UNDEFINED;
return 0;
}
*/
static int
nfs_proc_getattr(struct nfs_server *server, struct nfs_fh *fhandle,
- struct nfs_fattr *fattr, struct nfs4_label *label,
- struct inode *inode)
+ struct nfs_fattr *fattr, struct inode *inode)
{
struct rpc_message msg = {
.rpc_proc = &nfs_procedures[NFSPROC_GETATTR],
static int
nfs_proc_lookup(struct inode *dir, struct dentry *dentry,
- struct nfs_fh *fhandle, struct nfs_fattr *fattr,
- struct nfs4_label *label)
+ struct nfs_fh *fhandle, struct nfs_fattr *fattr)
{
struct nfs_diropargs arg = {
.fh = NFS_FH(dir),
status = rpc_call_sync(NFS_CLIENT(dir), &msg, 0);
nfs_mark_for_revalidate(dir);
if (status == 0)
- status = nfs_instantiate(dentry, data->res.fh, data->res.fattr, NULL);
+ status = nfs_instantiate(dentry, data->res.fh, data->res.fattr);
nfs_free_createdata(data);
out:
dprintk("NFS reply create: %d\n", status);
status = rpc_call_sync(NFS_CLIENT(dir), &msg, 0);
}
if (status == 0)
- status = nfs_instantiate(dentry, data->res.fh, data->res.fattr, NULL);
+ status = nfs_instantiate(dentry, data->res.fh, data->res.fattr);
nfs_free_createdata(data);
out:
dprintk("NFS reply mknod: %d\n", status);
* should fill in the data with a LOOKUP call on the wire.
*/
if (status == 0)
- status = nfs_instantiate(dentry, fh, fattr, NULL);
+ status = nfs_instantiate(dentry, fh, fattr);
out_free:
nfs_free_fattr(fattr);
status = rpc_call_sync(NFS_CLIENT(dir), &msg, 0);
nfs_mark_for_revalidate(dir);
if (status == 0)
- status = nfs_instantiate(dentry, data->res.fh, data->res.fattr, NULL);
+ status = nfs_instantiate(dentry, data->res.fh, data->res.fattr);
nfs_free_createdata(data);
out:
dprintk("NFS reply mkdir: %d\n", status);
struct inode *inode = page_file_mapping(page)->host;
int ret;
- dprintk("NFS: nfs_readpage (%p %ld@%lu)\n",
- page, PAGE_SIZE, page_index(page));
+ trace_nfs_aop_readpage(inode, page);
nfs_inc_stats(inode, NFSIOS_VFSREADPAGE);
/*
}
out:
put_nfs_open_context(desc.ctx);
+ trace_nfs_aop_readpage_done(inode, page, ret);
return ret;
out_unlock:
unlock_page(page);
+ trace_nfs_aop_readpage_done(inode, page, ret);
return ret;
}
struct inode *inode = mapping->host;
int ret;
- dprintk("NFS: nfs_readpages (%s/%Lu %d)\n",
- inode->i_sb->s_id,
- (unsigned long long)NFS_FILEID(inode),
- nr_pages);
+ trace_nfs_aop_readahead(inode, lru_to_page(pages), nr_pages);
nfs_inc_stats(inode, NFSIOS_VFSREADPAGES);
ret = -ESTALE;
read_complete:
put_nfs_open_context(desc.ctx);
out:
+ trace_nfs_aop_readahead_done(inode, nr_pages, ret);
return ret;
}
struct nfs_fs_context *ctx = nfs_fc2context(fc);
struct super_block *sb = fc->root->d_sb;
struct nfs_server *nfss = sb->s_fs_info;
+ int ret;
sync_filesystem(sb);
}
/* compare new mount options with old ones */
- return nfs_compare_remount_data(nfss, ctx);
+ ret = nfs_compare_remount_data(nfss, ctx);
+ if (ret)
+ return ret;
+
+ return nfs_probe_server(nfss, NFS_FH(d_inode(fc->root)));
}
EXPORT_SYMBOL_GPL(nfs_reconfigure);
end = page_file_offset(page) + ((loff_t)offset+count);
if (i_size >= end)
goto out;
+ trace_nfs_size_grow(inode, end);
i_size_write(inode, end);
NFS_I(inode)->cache_validity &= ~NFS_INO_INVALID_SIZE;
nfs_inc_stats(inode, NFSIOS_EXTENDWRITE);
struct nfs_page *req, *tmp;
int ret = 0;
-restart:
list_for_each_entry_safe(req, tmp, src, wb_list) {
kref_get(&req->wb_kref);
if (!nfs_lock_request(req)) {
- int status;
-
- /* Prevent deadlock with nfs_lock_and_join_requests */
- if (!list_empty(dst)) {
- nfs_release_request(req);
- continue;
- }
- /* Ensure we make progress to prevent livelock */
- mutex_unlock(&NFS_I(cinfo->inode)->commit_mutex);
- status = nfs_wait_on_request(req);
nfs_release_request(req);
- mutex_lock(&NFS_I(cinfo->inode)->commit_mutex);
- if (status < 0)
- break;
- goto restart;
+ continue;
}
nfs_request_remove_commit_list(req, cinfo);
clear_bit(PG_COMMIT_TO_DS, &req->wb_flags);
struct nfs_open_context *ctx = nfs_file_open_context(filp);
if (nfs_ctx_key_to_expire(ctx, inode) &&
- !ctx->ll_cred)
+ !rcu_access_pointer(ctx->ll_cred))
/* Already expired! */
return -EACCES;
return 0;
bool nfs_ctx_key_to_expire(struct nfs_open_context *ctx, struct inode *inode)
{
struct rpc_auth *auth = NFS_SERVER(inode)->client->cl_auth;
- struct rpc_cred *cred = ctx->ll_cred;
+ struct rpc_cred *cred, *new, *old = NULL;
struct auth_cred acred = {
.cred = ctx->cred,
};
+ bool ret = false;
- if (cred && !cred->cr_ops->crmatch(&acred, cred, 0)) {
- put_rpccred(cred);
- ctx->ll_cred = NULL;
- cred = NULL;
- }
- if (!cred)
- cred = auth->au_ops->lookup_cred(auth, &acred, 0);
- if (!cred || IS_ERR(cred))
+ rcu_read_lock();
+ cred = rcu_dereference(ctx->ll_cred);
+ if (cred && !(cred->cr_ops->crkey_timeout &&
+ cred->cr_ops->crkey_timeout(cred)))
+ goto out;
+ rcu_read_unlock();
+
+ new = auth->au_ops->lookup_cred(auth, &acred, 0);
+ if (new == cred) {
+ put_rpccred(new);
return true;
- ctx->ll_cred = cred;
- return !!(cred->cr_ops->crkey_timeout &&
- cred->cr_ops->crkey_timeout(cred));
+ }
+ if (IS_ERR_OR_NULL(new)) {
+ new = NULL;
+ ret = true;
+ } else if (new->cr_ops->crkey_timeout &&
+ new->cr_ops->crkey_timeout(new))
+ ret = true;
+
+ rcu_read_lock();
+ old = rcu_dereference_protected(xchg(&ctx->ll_cred,
+ RCU_INITIALIZER(new)), 1);
+out:
+ rcu_read_unlock();
+ put_rpccred(old);
+ return ret;
}
/*
status = nfs_writepage_setup(ctx, page, offset, count);
if (status < 0)
nfs_set_pageerror(mapping);
- else
- __set_page_dirty_nobuffers(page);
out:
dprintk("NFS: nfs_updatepage returns %d (isize %lld)\n",
status, (long long)i_size_read(inode));
atomic_inc(&cinfo->rpcs_out);
}
-static void nfs_commit_end(struct nfs_mds_commit_info *cinfo)
+bool nfs_commit_end(struct nfs_mds_commit_info *cinfo)
{
- if (atomic_dec_and_test(&cinfo->rpcs_out))
+ if (atomic_dec_and_test(&cinfo->rpcs_out)) {
wake_up_var(&cinfo->rpcs_out);
+ return true;
+ }
+ return false;
}
void nfs_commitdata_release(struct nfs_commit_data *data)
data->res.fattr = &data->fattr;
data->res.verf = &data->verf;
nfs_fattr_init(&data->fattr);
+ nfs_commit_begin(cinfo->mds);
}
EXPORT_SYMBOL_GPL(nfs_init_commit);
/* Set up the argument struct */
nfs_init_commit(data, head, NULL, cinfo);
- atomic_inc(&cinfo->mds->rpcs_out);
if (NFS_SERVER(inode)->nfs_client->cl_minorversion)
task_flags = RPC_TASK_MOVEABLE;
return nfs_initiate_commit(NFS_CLIENT(inode), data, NFS_PROTO(inode),
{
struct nfs_commit_data *data = calldata;
- dprintk("NFS: %5u nfs_commit_done (status %d)\n",
- task->tk_pid, task->tk_status);
-
/* Call the NFS version-specific code */
NFS_PROTO(data->inode)->commit_done(task, data);
trace_nfs_commit_done(task, data);
int may_wait = how & FLUSH_SYNC;
int ret, nscan;
+ how &= ~FLUSH_SYNC;
nfs_init_cinfo_from_inode(&cinfo, inode);
nfs_commit_begin(cinfo.mds);
for (;;) {
struct inode *inode, struct inode *dir,
const struct qstr *name, u32 cookie)
{
+ if (WARN_ON_ONCE(!inode))
+ return 0;
+
trace_nfsd_file_fsnotify_handle_event(inode, mask);
/* Should be no marks on non-regular files */
goto out_error;
fl->fh.size = fhp->fh_handle.fh_size;
- memcpy(fl->fh.data, &fhp->fh_handle.fh_base, fl->fh.size);
+ memcpy(fl->fh.data, &fhp->fh_handle.fh_raw, fl->fh.size);
/* Give whole file layout segments */
seg->offset = 0;
/* must initialize before using! but maxsize doesn't matter */
fh_init(&fh,0);
fh.fh_handle.fh_size = f->size;
- memcpy((char*)&fh.fh_handle.fh_base, f->data, f->size);
+ memcpy(&fh.fh_handle.fh_raw, f->data, f->size);
fh.fh_export = NULL;
access = (mode == O_WRONLY) ? NFSD_MAY_WRITE : NFSD_MAY_READ;
* XDR decode functions
*/
-static int nfsaclsvc_decode_getaclargs(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfsaclsvc_decode_getaclargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_getaclargs *argp = rqstp->rq_argp;
if (!svcxdr_decode_fhandle(xdr, &argp->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->mask) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-static int nfsaclsvc_decode_setaclargs(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfsaclsvc_decode_setaclargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_setaclargs *argp = rqstp->rq_argp;
if (!svcxdr_decode_fhandle(xdr, &argp->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->mask) < 0)
- return 0;
+ return false;
if (argp->mask & ~NFS_ACL_MASK)
- return 0;
+ return false;
if (!nfs_stream_decode_acl(xdr, NULL, (argp->mask & NFS_ACL) ?
&argp->acl_access : NULL))
- return 0;
+ return false;
if (!nfs_stream_decode_acl(xdr, NULL, (argp->mask & NFS_DFACL) ?
&argp->acl_default : NULL))
- return 0;
+ return false;
- return 1;
+ return true;
}
-static int nfsaclsvc_decode_accessargs(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfsaclsvc_decode_accessargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_accessargs *args = rqstp->rq_argp;
if (!svcxdr_decode_fhandle(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->access) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
/*
*/
/* GETACL */
-static int nfsaclsvc_encode_getaclres(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfsaclsvc_encode_getaclres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_getaclres *resp = rqstp->rq_resp;
struct dentry *dentry = resp->fh.fh_dentry;
struct inode *inode;
}
/* ACCESS */
-static int nfsaclsvc_encode_accessres(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfsaclsvc_encode_accessres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_accessres *resp = rqstp->rq_resp;
if (!svcxdr_encode_stat(xdr, resp->status))
* XDR decode functions
*/
-static int nfs3svc_decode_getaclargs(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfs3svc_decode_getaclargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_getaclargs *args = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->mask) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-static int nfs3svc_decode_setaclargs(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfs3svc_decode_setaclargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_setaclargs *argp = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &argp->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &argp->mask) < 0)
- return 0;
+ return false;
if (argp->mask & ~NFS_ACL_MASK)
- return 0;
+ return false;
if (!nfs_stream_decode_acl(xdr, NULL, (argp->mask & NFS_ACL) ?
&argp->acl_access : NULL))
- return 0;
+ return false;
if (!nfs_stream_decode_acl(xdr, NULL, (argp->mask & NFS_DFACL) ?
&argp->acl_default : NULL))
- return 0;
+ return false;
- return 1;
+ return true;
}
/*
*/
/* GETACL */
-static int nfs3svc_encode_getaclres(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfs3svc_encode_getaclres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_getaclres *resp = rqstp->rq_resp;
struct dentry *dentry = resp->fh.fh_dentry;
struct kvec *head = rqstp->rq_res.head;
int w;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
inode = d_inode(dentry);
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->mask) < 0)
- return 0;
+ return false;
base = (char *)xdr->p - (char *)head->iov_base;
(resp->mask & NFS_DFACL) ? resp->acl_default : NULL);
while (w > 0) {
if (!*(rqstp->rq_next_page++))
- return 0;
+ return false;
w -= PAGE_SIZE;
}
resp->mask & NFS_DFACL,
NFS_ACL_DEFAULT);
if (n <= 0)
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* SETACL */
-static int nfs3svc_encode_setaclres(struct svc_rqst *rqstp, __be32 *p)
+static bool
+nfs3svc_encode_setaclres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_attrstat *resp = rqstp->rq_resp;
return svcxdr_encode_nfsstat3(xdr, resp->status) &&
fh_copy(&resp->fh, &argp->fh);
resp->committed = argp->stable;
- nvecs = svc_fill_write_vector(rqstp, rqstp->rq_arg.pages,
- &argp->first, cnt);
+ nvecs = svc_fill_write_vector(rqstp, &argp->payload);
if (!nvecs) {
resp->status = nfserr_io;
goto out;
return false;
fh_init(fhp, NFS3_FHSIZE);
fhp->fh_handle.fh_size = size;
- memcpy(&fhp->fh_handle.fh_base, p, size);
+ memcpy(&fhp->fh_handle.fh_raw, p, size);
return true;
}
*p++ = cpu_to_be32(size);
if (size)
p[XDR_QUADLEN(size) - 1] = 0;
- memcpy(p, &fhp->fh_handle.fh_base, size);
+ memcpy(p, &fhp->fh_handle.fh_raw, size);
return true;
}
* XDR decode functions
*/
-int
-nfs3svc_decode_fhandleargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_fhandleargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_fhandle *args = rqstp->rq_argp;
return svcxdr_decode_nfs_fh3(xdr, &args->fh);
}
-int
-nfs3svc_decode_sattrargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_sattrargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_sattrargs *args = rqstp->rq_argp;
return svcxdr_decode_nfs_fh3(xdr, &args->fh) &&
svcxdr_decode_sattrguard3(xdr, args);
}
-int
-nfs3svc_decode_diropargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_diropargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_diropargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs3(xdr, &args->fh, &args->name, &args->len);
}
-int
-nfs3svc_decode_accessargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_accessargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_accessargs *args = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->access) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nfs3svc_decode_readargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_readargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_readargs *args = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u64(xdr, &args->offset) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nfs3svc_decode_writeargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_writeargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_writeargs *args = rqstp->rq_argp;
u32 max_blocksize = svc_max_payload(rqstp);
- struct kvec *head = rqstp->rq_arg.head;
- struct kvec *tail = rqstp->rq_arg.tail;
- size_t remaining;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u64(xdr, &args->offset) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->stable) < 0)
- return 0;
+ return false;
/* opaque data */
if (xdr_stream_decode_u32(xdr, &args->len) < 0)
- return 0;
+ return false;
/* request sanity */
if (args->count != args->len)
- return 0;
- remaining = head->iov_len + rqstp->rq_arg.page_len + tail->iov_len;
- remaining -= xdr_stream_pos(xdr);
- if (remaining < xdr_align_size(args->len))
- return 0;
+ return false;
if (args->count > max_blocksize) {
args->count = max_blocksize;
args->len = max_blocksize;
}
+ if (!xdr_stream_subsegment(xdr, &args->payload, args->count))
+ return false;
- args->first.iov_base = xdr->p;
- args->first.iov_len = head->iov_len - xdr_stream_pos(xdr);
-
- return 1;
+ return true;
}
-int
-nfs3svc_decode_createargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_createargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_createargs *args = rqstp->rq_argp;
if (!svcxdr_decode_diropargs3(xdr, &args->fh, &args->name, &args->len))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->createmode) < 0)
- return 0;
+ return false;
switch (args->createmode) {
case NFS3_CREATE_UNCHECKED:
case NFS3_CREATE_GUARDED:
case NFS3_CREATE_EXCLUSIVE:
args->verf = xdr_inline_decode(xdr, NFS3_CREATEVERFSIZE);
if (!args->verf)
- return 0;
+ return false;
break;
default:
- return 0;
+ return false;
}
- return 1;
+ return true;
}
-int
-nfs3svc_decode_mkdirargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_mkdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_createargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs3(xdr, &args->fh,
svcxdr_decode_sattr3(rqstp, xdr, &args->attrs);
}
-int
-nfs3svc_decode_symlinkargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_symlinkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_symlinkargs *args = rqstp->rq_argp;
struct kvec *head = rqstp->rq_arg.head;
struct kvec *tail = rqstp->rq_arg.tail;
size_t remaining;
if (!svcxdr_decode_diropargs3(xdr, &args->ffh, &args->fname, &args->flen))
- return 0;
+ return false;
if (!svcxdr_decode_sattr3(rqstp, xdr, &args->attrs))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->tlen) < 0)
- return 0;
+ return false;
/* request sanity */
remaining = head->iov_len + rqstp->rq_arg.page_len + tail->iov_len;
remaining -= xdr_stream_pos(xdr);
if (remaining < xdr_align_size(args->tlen))
- return 0;
+ return false;
args->first.iov_base = xdr->p;
args->first.iov_len = head->iov_len - xdr_stream_pos(xdr);
- return 1;
+ return true;
}
-int
-nfs3svc_decode_mknodargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_mknodargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_mknodargs *args = rqstp->rq_argp;
if (!svcxdr_decode_diropargs3(xdr, &args->fh, &args->name, &args->len))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->ftype) < 0)
- return 0;
+ return false;
switch (args->ftype) {
case NF3CHR:
case NF3BLK:
/* Valid XDR but illegal file types */
break;
default:
- return 0;
+ return false;
}
- return 1;
+ return true;
}
-int
-nfs3svc_decode_renameargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_renameargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_renameargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs3(xdr, &args->ffh,
&args->tname, &args->tlen);
}
-int
-nfs3svc_decode_linkargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_linkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_linkargs *args = rqstp->rq_argp;
return svcxdr_decode_nfs_fh3(xdr, &args->ffh) &&
&args->tname, &args->tlen);
}
-int
-nfs3svc_decode_readdirargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_readdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_readdirargs *args = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u64(xdr, &args->cookie) < 0)
- return 0;
+ return false;
args->verf = xdr_inline_decode(xdr, NFS3_COOKIEVERFSIZE);
if (!args->verf)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nfs3svc_decode_readdirplusargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_readdirplusargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_readdirargs *args = rqstp->rq_argp;
u32 dircount;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u64(xdr, &args->cookie) < 0)
- return 0;
+ return false;
args->verf = xdr_inline_decode(xdr, NFS3_COOKIEVERFSIZE);
if (!args->verf)
- return 0;
+ return false;
/* dircount is ignored */
if (xdr_stream_decode_u32(xdr, &dircount) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nfs3svc_decode_commitargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_decode_commitargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd3_commitargs *args = rqstp->rq_argp;
if (!svcxdr_decode_nfs_fh3(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u64(xdr, &args->offset) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
/*
*/
/* GETATTR */
-int
-nfs3svc_encode_getattrres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_getattrres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_attrstat *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
lease_get_mtime(d_inode(resp->fh.fh_dentry), &resp->stat.mtime);
if (!svcxdr_encode_fattr3(rqstp, xdr, &resp->fh, &resp->stat))
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
/* SETATTR, REMOVE, RMDIR */
-int
-nfs3svc_encode_wccstat(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_wccstat(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_attrstat *resp = rqstp->rq_resp;
return svcxdr_encode_nfsstat3(xdr, resp->status) &&
}
/* LOOKUP */
-int nfs3svc_encode_lookupres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_lookupres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_diropres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_nfs_fh3(xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->dirfh))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->dirfh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* ACCESS */
-int
-nfs3svc_encode_accessres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_accessres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_accessres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->access) < 0)
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* READLINK */
-int
-nfs3svc_encode_readlinkres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_readlinkres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_readlinkres *resp = rqstp->rq_resp;
struct kvec *head = rqstp->rq_res.head;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->len) < 0)
- return 0;
+ return false;
xdr_write_pages(xdr, resp->pages, 0, resp->len);
if (svc_encode_result_payload(rqstp, head->iov_len, resp->len) < 0)
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* READ */
-int
-nfs3svc_encode_readres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_readres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_readres *resp = rqstp->rq_resp;
struct kvec *head = rqstp->rq_res.head;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->count) < 0)
- return 0;
+ return false;
if (xdr_stream_encode_bool(xdr, resp->eof) < 0)
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->count) < 0)
- return 0;
+ return false;
xdr_write_pages(xdr, resp->pages, rqstp->rq_res.page_base,
resp->count);
if (svc_encode_result_payload(rqstp, head->iov_len, resp->count) < 0)
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* WRITE */
-int
-nfs3svc_encode_writeres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_writeres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_writeres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->count) < 0)
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->committed) < 0)
- return 0;
+ return false;
if (!svcxdr_encode_writeverf3(xdr, resp->verf))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* CREATE, MKDIR, SYMLINK, MKNOD */
-int
-nfs3svc_encode_createres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_createres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_diropres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_fh3(xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->dirfh))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->dirfh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* RENAME */
-int
-nfs3svc_encode_renameres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_renameres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_renameres *resp = rqstp->rq_resp;
return svcxdr_encode_nfsstat3(xdr, resp->status) &&
}
/* LINK */
-int
-nfs3svc_encode_linkres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_linkres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_linkres *resp = rqstp->rq_resp;
return svcxdr_encode_nfsstat3(xdr, resp->status) &&
}
/* READDIR */
-int
-nfs3svc_encode_readdirres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_readdirres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_readdirres *resp = rqstp->rq_resp;
struct xdr_buf *dirlist = &resp->dirlist;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_cookieverf3(xdr, resp->verf))
- return 0;
+ return false;
xdr_write_pages(xdr, dirlist->pages, 0, dirlist->len);
/* no more entries */
if (xdr_stream_encode_item_absent(xdr) < 0)
- return 0;
+ return false;
if (xdr_stream_encode_bool(xdr, resp->common.err == nfserr_eof) < 0)
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
static __be32
}
/* FSSTAT */
-int
-nfs3svc_encode_fsstatres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_fsstatres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_fsstatres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
if (!svcxdr_encode_fsstat3resok(xdr, resp))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
static bool
}
/* FSINFO */
-int
-nfs3svc_encode_fsinfores(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_fsinfores(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_fsinfores *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
if (!svcxdr_encode_fsinfo3resok(xdr, resp))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
static bool
}
/* PATHCONF */
-int
-nfs3svc_encode_pathconfres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_pathconfres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_pathconfres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
if (!svcxdr_encode_pathconf3resok(xdr, resp))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_post_op_attr(rqstp, xdr, &nfs3svc_null_fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/* COMMIT */
-int
-nfs3svc_encode_commitres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs3svc_encode_commitres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd3_commitres *resp = rqstp->rq_resp;
if (!svcxdr_encode_nfsstat3(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_writeverf3(xdr, resp->verf))
- return 0;
+ return false;
break;
default:
if (!svcxdr_encode_wcc_data(rqstp, xdr, &resp->fh))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
/*
BUG_ON(length > NFS4_FHSIZE);
p = xdr_reserve_space(xdr, 4 + length);
- xdr_encode_opaque(p, &fh->fh_base, length);
+ xdr_encode_opaque(p, &fh->fh_raw, length);
}
/*
fh_put(&cstate->current_fh);
cstate->current_fh.fh_handle.fh_size = putfh->pf_fhlen;
- memcpy(&cstate->current_fh.fh_handle.fh_base, putfh->pf_fhval,
+ memcpy(&cstate->current_fh.fh_handle.fh_raw, putfh->pf_fhval,
putfh->pf_fhlen);
ret = fh_verify(rqstp, &cstate->current_fh, 0, NFSD_MAY_BYPASS_GSS);
#ifdef CONFIG_NFSD_V4_2_INTER_SSC
write->wr_how_written = write->wr_stable_how;
- nvecs = svc_fill_write_vector(rqstp, write->wr_payload.pages,
- write->wr_payload.head, write->wr_buflen);
+ nvecs = svc_fill_write_vector(rqstp, &write->wr_payload);
WARN_ON_ONCE(nvecs > ARRAY_SIZE(rqstp->rq_vec));
status = nfsd_vfs_write(rqstp, &cstate->current_fh, nf,
static __be32 nfsd4_ssc_setup_dul(struct nfsd_net *nn, char *ipaddr,
struct nfsd4_ssc_umount_item **retwork, struct vfsmount **ss_mnt)
{
- struct nfsd4_ssc_umount_item *ni = 0;
+ struct nfsd4_ssc_umount_item *ni = NULL;
struct nfsd4_ssc_umount_item *work = NULL;
struct nfsd4_ssc_umount_item *tmp;
DEFINE_WAIT(wait);
s_fh = &cstate->save_fh;
copy->c_fh.size = s_fh->fh_handle.fh_size;
- memcpy(copy->c_fh.data, &s_fh->fh_handle.fh_base, copy->c_fh.size);
+ memcpy(copy->c_fh.data, &s_fh->fh_handle.fh_raw, copy->c_fh.size);
copy->stateid.seqid = cpu_to_be32(s_stid->si_generation);
memcpy(copy->stateid.other, (void *)&s_stid->si_opaque,
sizeof(stateid_opaque_t));
__be32 status;
resp->xdr = &rqstp->rq_res_stream;
+ resp->statusp = resp->xdr->p;
/* reserve space for: NFS status code */
xdr_reserve_space(resp->xdr, XDR_UNIT);
- resp->tagp = resp->xdr->p;
/* reserve space for: taglen, tag, and opcnt */
xdr_reserve_space(resp->xdr, XDR_UNIT * 2 + args->taglen);
resp->taglen = args->taglen;
}
spin_unlock(&blocked_delegations_lock);
}
- hash = jhash(&fh->fh_base, fh->fh_size, 0);
+ hash = jhash(&fh->fh_raw, fh->fh_size, 0);
if (test_bit(hash&255, bd->set[0]) &&
test_bit((hash>>8)&255, bd->set[0]) &&
test_bit((hash>>16)&255, bd->set[0]))
u32 hash;
struct bloom_pair *bd = &blocked_delegations;
- hash = jhash(&fh->fh_base, fh->fh_size, 0);
+ hash = jhash(&fh->fh_raw, fh->fh_size, 0);
spin_lock(&blocked_delegations_lock);
__set_bit(hash&255, bd->set[bd->new]);
static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
{
bool do_wakeup = false;
- struct nfsd4_ssc_umount_item *ni = 0;
+ struct nfsd4_ssc_umount_item *ni = NULL;
struct nfsd4_ssc_umount_item *tmp;
spin_lock(&nn->nfsd_ssc_lock);
p = xdr_inline_decode(argp->xdr, count << 2);
if (!p)
return nfserr_bad_xdr;
- i = 0;
- while (i < count)
- bmval[i++] = be32_to_cpup(p++);
- while (i < bmlen)
- bmval[i++] = 0;
+ for (i = 0; i < bmlen; i++)
+ bmval[i] = (i < count) ? be32_to_cpup(p++) : 0;
return nfs_ok;
}
return true;
}
-static int
+static bool
nfsd4_decode_compound(struct nfsd4_compoundargs *argp)
{
struct nfsd4_op *op;
int i;
if (xdr_stream_decode_u32(argp->xdr, &argp->taglen) < 0)
- return 0;
+ return false;
max_reply += XDR_UNIT;
argp->tag = NULL;
if (unlikely(argp->taglen)) {
if (argp->taglen > NFSD4_MAX_TAGLEN)
- return 0;
+ return false;
p = xdr_inline_decode(argp->xdr, argp->taglen);
if (!p)
- return 0;
+ return false;
argp->tag = svcxdr_savemem(argp, p, argp->taglen);
if (!argp->tag)
- return 0;
+ return false;
max_reply += xdr_align_size(argp->taglen);
}
if (xdr_stream_decode_u32(argp->xdr, &argp->minorversion) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(argp->xdr, &argp->opcnt) < 0)
- return 0;
+ return false;
/*
* NFS4ERR_RESOURCE is a more helpful error than GARBAGE_ARGS
* nfsd4_proc can handle this is an NFS-level error.
*/
if (argp->opcnt > NFSD_MAX_OPS_PER_COMPOUND)
- return 1;
+ return true;
if (argp->opcnt > ARRAY_SIZE(argp->iops)) {
argp->ops = kzalloc(argp->opcnt * sizeof(*argp->ops), GFP_KERNEL);
if (!argp->ops) {
argp->ops = argp->iops;
dprintk("nfsd: couldn't allocate room for COMPOUND\n");
- return 0;
+ return false;
}
}
op->replay = NULL;
if (xdr_stream_decode_u32(argp->xdr, &op->opnum) < 0)
- return 0;
+ return false;
if (nfsd4_opnum_in_range(argp, op)) {
op->status = nfsd4_dec_ops[op->opnum](argp, &op->u);
if (op->status != nfs_ok)
if (readcount > 1 || max_reply > PAGE_SIZE - auth_slack)
clear_bit(RQ_SPLICE_OK, &argp->rqstp->rq_flags);
- return 1;
+ return true;
}
static __be32 *encode_change(__be32 *p, struct kstat *stat, struct inode *inode,
p = xdr_reserve_space(xdr, fhp->fh_handle.fh_size + 4);
if (!p)
goto out_resource;
- p = xdr_encode_opaque(p, &fhp->fh_handle.fh_base,
+ p = xdr_encode_opaque(p, &fhp->fh_handle.fh_raw,
fhp->fh_handle.fh_size);
}
if (bmval0 & FATTR4_WORD0_FILEID) {
p = xdr_reserve_space(xdr, len + 4);
if (!p)
return nfserr_resource;
- p = xdr_encode_opaque(p, &fhp->fh_handle.fh_base, len);
+ p = xdr_encode_opaque(p, &fhp->fh_handle.fh_raw, len);
return 0;
}
}
}
-int
-nfs4svc_decode_compoundargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs4svc_decode_compoundargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
struct nfsd4_compoundargs *args = rqstp->rq_argp;
/* svcxdr_tmp_alloc */
args->to_free = NULL;
- args->xdr = &rqstp->rq_arg_stream;
+ args->xdr = xdr;
args->ops = args->iops;
args->rqstp = rqstp;
return nfsd4_decode_compound(args);
}
-int
-nfs4svc_encode_compoundres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfs4svc_encode_compoundres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
struct nfsd4_compoundres *resp = rqstp->rq_resp;
- struct xdr_buf *buf = resp->xdr->buf;
+ struct xdr_buf *buf = xdr->buf;
+ __be32 *p;
WARN_ON_ONCE(buf->len != buf->head[0].iov_len + buf->page_len +
buf->tail[0].iov_len);
- *p = resp->cstate.status;
+ /*
+ * Send buffer space for the following items is reserved
+ * at the top of nfsd4_proc_compound().
+ */
+ p = resp->statusp;
+
+ *p++ = resp->cstate.status;
- rqstp->rq_next_page = resp->xdr->page_ptr + 1;
+ rqstp->rq_next_page = xdr->page_ptr + 1;
- p = resp->tagp;
*p++ = htonl(resp->taglen);
memcpy(p, resp->tag, resp->taglen);
p += XDR_QUADLEN(resp->taglen);
*p++ = htonl(resp->opcnt);
nfsd4_sequence_done(resp);
- return 1;
+ return true;
}
list_move_tail(&rp->c_lru, &b->lru_head);
}
-static long
-prune_bucket(struct nfsd_drc_bucket *b, struct nfsd_net *nn)
+static long prune_bucket(struct nfsd_drc_bucket *b, struct nfsd_net *nn,
+ unsigned int max)
{
struct svc_cacherep *rp, *tmp;
long freed = 0;
time_before(jiffies, rp->c_timestamp + RC_EXPIRE))
break;
nfsd_reply_cache_free_locked(b, rp, nn);
- freed++;
+ if (max && freed++ > max)
+ break;
}
return freed;
}
+static long nfsd_prune_bucket(struct nfsd_drc_bucket *b, struct nfsd_net *nn)
+{
+ return prune_bucket(b, nn, 3);
+}
+
/*
* Walk the LRU list and prune off entries that are older than RC_EXPIRE.
* Also prune the oldest ones when the total exceeds the max number of entries.
if (list_empty(&b->lru_head))
continue;
spin_lock(&b->cache_lock);
- freed += prune_bucket(b, nn);
+ freed += prune_bucket(b, nn, 0);
spin_unlock(&b->cache_lock);
}
return freed;
atomic_inc(&nn->num_drc_entries);
nfsd_stats_drc_mem_usage_add(nn, sizeof(*rp));
- /* go ahead and prune the cache */
- prune_bucket(b, nn);
+ nfsd_prune_bucket(b, nn);
out_unlock:
spin_unlock(&b->cache_lock);
auth_domain_put(dom);
if (len)
return len;
-
+
mesg = buf;
len = SIMPLE_TRANSACTION_LIMIT;
- qword_addhex(&mesg, &len, (char*)&fh.fh_base, fh.fh_size);
+ qword_addhex(&mesg, &len, fh.fh_raw, fh.fh_size);
mesg[-1] = '\n';
- return mesg - buf;
+ return mesg - buf;
}
/*
*/
struct nfsd_voidargs { };
struct nfsd_voidres { };
-int nfssvc_decode_voidarg(struct svc_rqst *rqstp, __be32 *p);
-int nfssvc_encode_voidres(struct svc_rqst *rqstp, __be32 *p);
+bool nfssvc_decode_voidarg(struct svc_rqst *rqstp,
+ struct xdr_stream *xdr);
+bool nfssvc_encode_voidres(struct svc_rqst *rqstp,
+ struct xdr_stream *xdr);
/*
* Function prototypes.
static __be32 nfsd_set_fh_dentry(struct svc_rqst *rqstp, struct svc_fh *fhp)
{
struct knfsd_fh *fh = &fhp->fh_handle;
- struct fid *fid = NULL, sfid;
+ struct fid *fid = NULL;
struct svc_export *exp;
struct dentry *dentry;
int fileid_type;
int data_left = fh->fh_size/4;
+ int len;
__be32 error;
error = nfserr_stale;
if (rqstp->rq_vers == 4 && fh->fh_size == 0)
return nfserr_nofilehandle;
- if (fh->fh_version == 1) {
- int len;
-
- if (--data_left < 0)
- return error;
- if (fh->fh_auth_type != 0)
- return error;
- len = key_len(fh->fh_fsid_type) / 4;
- if (len == 0)
- return error;
- if (fh->fh_fsid_type == FSID_MAJOR_MINOR) {
- /* deprecated, convert to type 3 */
- len = key_len(FSID_ENCODE_DEV)/4;
- fh->fh_fsid_type = FSID_ENCODE_DEV;
- /*
- * struct knfsd_fh uses host-endian fields, which are
- * sometimes used to hold net-endian values. This
- * confuses sparse, so we must use __force here to
- * keep it from complaining.
- */
- fh->fh_fsid[0] = new_encode_dev(MKDEV(ntohl((__force __be32)fh->fh_fsid[0]),
- ntohl((__force __be32)fh->fh_fsid[1])));
- fh->fh_fsid[1] = fh->fh_fsid[2];
- }
- data_left -= len;
- if (data_left < 0)
- return error;
- exp = rqst_exp_find(rqstp, fh->fh_fsid_type, fh->fh_fsid);
- fid = (struct fid *)(fh->fh_fsid + len);
- } else {
- __u32 tfh[2];
- dev_t xdev;
- ino_t xino;
-
- if (fh->fh_size != NFS_FHSIZE)
- return error;
- /* assume old filehandle format */
- xdev = old_decode_dev(fh->ofh_xdev);
- xino = u32_to_ino_t(fh->ofh_xino);
- mk_fsid(FSID_DEV, tfh, xdev, xino, 0, NULL);
- exp = rqst_exp_find(rqstp, FSID_DEV, tfh);
+ if (fh->fh_version != 1)
+ return error;
+
+ if (--data_left < 0)
+ return error;
+ if (fh->fh_auth_type != 0)
+ return error;
+ len = key_len(fh->fh_fsid_type) / 4;
+ if (len == 0)
+ return error;
+ if (fh->fh_fsid_type == FSID_MAJOR_MINOR) {
+ /* deprecated, convert to type 3 */
+ len = key_len(FSID_ENCODE_DEV)/4;
+ fh->fh_fsid_type = FSID_ENCODE_DEV;
+ /*
+ * struct knfsd_fh uses host-endian fields, which are
+ * sometimes used to hold net-endian values. This
+ * confuses sparse, so we must use __force here to
+ * keep it from complaining.
+ */
+ fh->fh_fsid[0] = new_encode_dev(MKDEV(ntohl((__force __be32)fh->fh_fsid[0]),
+ ntohl((__force __be32)fh->fh_fsid[1])));
+ fh->fh_fsid[1] = fh->fh_fsid[2];
}
+ data_left -= len;
+ if (data_left < 0)
+ return error;
+ exp = rqst_exp_find(rqstp, fh->fh_fsid_type, fh->fh_fsid);
+ fid = (struct fid *)(fh->fh_fsid + len);
error = nfserr_stale;
if (IS_ERR(exp)) {
if (rqstp->rq_vers > 2)
error = nfserr_badhandle;
- if (fh->fh_version != 1) {
- sfid.i32.ino = fh->ofh_ino;
- sfid.i32.gen = fh->ofh_generation;
- sfid.i32.parent_ino = fh->ofh_dirino;
- fid = &sfid;
- data_left = 3;
- if (fh->ofh_dirino == 0)
- fileid_type = FILEID_INO32_GEN;
- else
- fileid_type = FILEID_INO32_GEN_PARENT;
- } else
- fileid_type = fh->fh_fileid_type;
+ fileid_type = fh->fh_fileid_type;
if (fileid_type == FILEID_ROOT)
dentry = dget(exp->ex_path.dentry);
}
}
-/*
- * for composing old style file handles
- */
-static inline void _fh_update_old(struct dentry *dentry,
- struct svc_export *exp,
- struct knfsd_fh *fh)
-{
- fh->ofh_ino = ino_t_to_u32(d_inode(dentry)->i_ino);
- fh->ofh_generation = d_inode(dentry)->i_generation;
- if (d_is_dir(dentry) ||
- (exp->ex_flags & NFSEXP_NOSUBTREECHECK))
- fh->ofh_dirino = 0;
-}
-
static bool is_root_export(struct svc_export *exp)
{
return exp->ex_path.dentry == exp->ex_path.dentry->d_sb->s_root;
/* ref_fh is a reference file handle.
* if it is non-null and for the same filesystem, then we should compose
* a filehandle which is of the same version, where possible.
- * Currently, that means that if ref_fh->fh_handle.fh_version == 0xca
- * Then create a 32byte filehandle using nfs_fhbase_old
- *
*/
struct inode * inode = d_inode(dentry);
fhp->fh_dentry = dget(dentry); /* our internal copy */
fhp->fh_export = exp_get(exp);
- if (fhp->fh_handle.fh_version == 0xca) {
- /* old style filehandle please */
- memset(&fhp->fh_handle.fh_base, 0, NFS_FHSIZE);
- fhp->fh_handle.fh_size = NFS_FHSIZE;
- fhp->fh_handle.ofh_dcookie = 0xfeebbaca;
- fhp->fh_handle.ofh_dev = old_encode_dev(ex_dev);
- fhp->fh_handle.ofh_xdev = fhp->fh_handle.ofh_dev;
- fhp->fh_handle.ofh_xino =
- ino_t_to_u32(d_inode(exp->ex_path.dentry)->i_ino);
- fhp->fh_handle.ofh_dirino = ino_t_to_u32(parent_ino(dentry));
- if (inode)
- _fh_update_old(dentry, exp, &fhp->fh_handle);
- } else {
- fhp->fh_handle.fh_size =
- key_len(fhp->fh_handle.fh_fsid_type) + 4;
- fhp->fh_handle.fh_auth_type = 0;
-
- mk_fsid(fhp->fh_handle.fh_fsid_type,
- fhp->fh_handle.fh_fsid,
- ex_dev,
- d_inode(exp->ex_path.dentry)->i_ino,
- exp->ex_fsid, exp->ex_uuid);
-
- if (inode)
- _fh_update(fhp, exp, dentry);
- if (fhp->fh_handle.fh_fileid_type == FILEID_INVALID) {
- fh_put(fhp);
- return nfserr_opnotsupp;
- }
+ fhp->fh_handle.fh_size =
+ key_len(fhp->fh_handle.fh_fsid_type) + 4;
+ fhp->fh_handle.fh_auth_type = 0;
+
+ mk_fsid(fhp->fh_handle.fh_fsid_type,
+ fhp->fh_handle.fh_fsid,
+ ex_dev,
+ d_inode(exp->ex_path.dentry)->i_ino,
+ exp->ex_fsid, exp->ex_uuid);
+
+ if (inode)
+ _fh_update(fhp, exp, dentry);
+ if (fhp->fh_handle.fh_fileid_type == FILEID_INVALID) {
+ fh_put(fhp);
+ return nfserr_opnotsupp;
}
return 0;
dentry = fhp->fh_dentry;
if (d_really_is_negative(dentry))
goto out_negative;
- if (fhp->fh_handle.fh_version != 1) {
- _fh_update_old(dentry, fhp->fh_export, &fhp->fh_handle);
- } else {
- if (fhp->fh_handle.fh_fileid_type != FILEID_ROOT)
- return 0;
+ if (fhp->fh_handle.fh_fileid_type != FILEID_ROOT)
+ return 0;
- _fh_update(fhp, fhp->fh_export, dentry);
- if (fhp->fh_handle.fh_fileid_type == FILEID_INVALID)
- return nfserr_opnotsupp;
- }
+ _fh_update(fhp, fhp->fh_export, dentry);
+ if (fhp->fh_handle.fh_fileid_type == FILEID_INVALID)
+ return nfserr_opnotsupp;
return 0;
out_bad:
printk(KERN_ERR "fh_update: fh not verified!\n");
char * SVCFH_fmt(struct svc_fh *fhp)
{
struct knfsd_fh *fh = &fhp->fh_handle;
+ static char buf[2+1+1+64*3+1];
- static char buf[80];
- sprintf(buf, "%d: %08x %08x %08x %08x %08x %08x",
- fh->fh_size,
- fh->fh_base.fh_pad[0],
- fh->fh_base.fh_pad[1],
- fh->fh_base.fh_pad[2],
- fh->fh_base.fh_pad[3],
- fh->fh_base.fh_pad[4],
- fh->fh_base.fh_pad[5]);
+ if (fh->fh_size < 0 || fh->fh_size> 64)
+ return "bad-fh";
+ sprintf(buf, "%d: %*ph", fh->fh_size, fh->fh_size, fh->fh_raw);
return buf;
}
#include <linux/crc32.h>
#include <linux/sunrpc/svc.h>
-#include <uapi/linux/nfsd/nfsfh.h>
#include <linux/iversion.h>
#include <linux/exportfs.h>
+#include <linux/nfs4.h>
+
+/*
+ * The file handle starts with a sequence of four-byte words.
+ * The first word contains a version number (1) and three descriptor bytes
+ * that tell how the remaining 3 variable length fields should be handled.
+ * These three bytes are auth_type, fsid_type and fileid_type.
+ *
+ * All four-byte values are in host-byte-order.
+ *
+ * The auth_type field is deprecated and must be set to 0.
+ *
+ * The fsid_type identifies how the filesystem (or export point) is
+ * encoded.
+ * Current values:
+ * 0 - 4 byte device id (ms-2-bytes major, ls-2-bytes minor), 4byte inode number
+ * NOTE: we cannot use the kdev_t device id value, because kdev_t.h
+ * says we mustn't. We must break it up and reassemble.
+ * 1 - 4 byte user specified identifier
+ * 2 - 4 byte major, 4 byte minor, 4 byte inode number - DEPRECATED
+ * 3 - 4 byte device id, encoded for user-space, 4 byte inode number
+ * 4 - 4 byte inode number and 4 byte uuid
+ * 5 - 8 byte uuid
+ * 6 - 16 byte uuid
+ * 7 - 8 byte inode number and 16 byte uuid
+ *
+ * The fileid_type identifies how the file within the filesystem is encoded.
+ * The values for this field are filesystem specific, exccept that
+ * filesystems must not use the values '0' or '0xff'. 'See enum fid_type'
+ * in include/linux/exportfs.h for currently registered values.
+ */
+
+struct knfsd_fh {
+ unsigned int fh_size; /*
+ * Points to the current size while
+ * building a new file handle.
+ */
+ union {
+ char fh_raw[NFS4_FHSIZE];
+ struct {
+ u8 fh_version; /* == 1 */
+ u8 fh_auth_type; /* deprecated */
+ u8 fh_fsid_type;
+ u8 fh_fileid_type;
+ u32 fh_fsid[]; /* flexible-array member */
+ };
+ };
+};
static inline __u32 ino_t_to_u32(ino_t ino)
{
fh_copy_shallow(struct knfsd_fh *dst, struct knfsd_fh *src)
{
dst->fh_size = src->fh_size;
- memcpy(&dst->fh_base, &src->fh_base, src->fh_size);
+ memcpy(&dst->fh_raw, &src->fh_raw, src->fh_size);
}
static __inline__ struct svc_fh *
{
if (fh1->fh_size != fh2->fh_size)
return false;
- if (memcmp(fh1->fh_base.fh_pad, fh2->fh_base.fh_pad, fh1->fh_size) != 0)
+ if (memcmp(fh1->fh_raw, fh2->fh_raw, fh1->fh_size) != 0)
return false;
return true;
}
*/
static inline u32 knfsd_fh_hash(const struct knfsd_fh *fh)
{
- return ~crc32_le(0xFFFFFFFF, (unsigned char *)&fh->fh_base, fh->fh_size);
+ return ~crc32_le(0xFFFFFFFF, fh->fh_raw, fh->fh_size);
}
#else
static inline u32 knfsd_fh_hash(const struct knfsd_fh *fh)
SVCFH_fmt(&argp->fh),
argp->len, argp->offset);
- nvecs = svc_fill_write_vector(rqstp, rqstp->rq_arg.pages,
- &argp->first, cnt);
+ nvecs = svc_fill_write_vector(rqstp, &argp->payload);
if (!nvecs) {
resp->status = nfserr_io;
goto out;
int nfsd_dispatch(struct svc_rqst *rqstp, __be32 *statp)
{
const struct svc_procedure *proc = rqstp->rq_procinfo;
- struct kvec *argv = &rqstp->rq_arg.head[0];
- struct kvec *resv = &rqstp->rq_res.head[0];
- __be32 *p;
/*
* Give the xdr decoder a chance to change this if it wants
rqstp->rq_cachetype = proc->pc_cachetype;
svcxdr_init_decode(rqstp);
- if (!proc->pc_decode(rqstp, argv->iov_base))
+ if (!proc->pc_decode(rqstp, &rqstp->rq_arg_stream))
goto out_decode_err;
switch (nfsd_cache_lookup(rqstp)) {
* Need to grab the location to store the status, as
* NFSv4 does some encoding while processing
*/
- p = resv->iov_base + resv->iov_len;
svcxdr_init_encode(rqstp);
*statp = proc->pc_func(rqstp);
if (*statp == rpc_drop_reply || test_bit(RQ_DROPME, &rqstp->rq_flags))
goto out_update_drop;
- if (!proc->pc_encode(rqstp, p))
+ if (!proc->pc_encode(rqstp, &rqstp->rq_res_stream))
goto out_encode_err;
nfsd_cache_update(rqstp, rqstp->rq_cachetype, statp + 1);
/**
* nfssvc_decode_voidarg - Decode void arguments
* @rqstp: Server RPC transaction context
- * @p: buffer containing arguments to decode
+ * @xdr: XDR stream positioned at arguments to decode
*
* Return values:
- * %0: Arguments were not valid
- * %1: Decoding was successful
+ * %false: Arguments were not valid
+ * %true: Decoding was successful
*/
-int nfssvc_decode_voidarg(struct svc_rqst *rqstp, __be32 *p)
+bool nfssvc_decode_voidarg(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
/**
* nfssvc_encode_voidres - Encode void results
* @rqstp: Server RPC transaction context
- * @p: buffer in which to encode results
+ * @xdr: XDR stream into which to encode results
*
* Return values:
- * %0: Local error while encoding
- * %1: Encoding was successful
+ * %false: Local error while encoding
+ * %true: Encoding was successful
*/
-int nfssvc_encode_voidres(struct svc_rqst *rqstp, __be32 *p)
+bool nfssvc_encode_voidres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- return 1;
+ return true;
}
int nfsd_pool_stats_open(struct inode *inode, struct file *file)
if (!p)
return false;
fh_init(fhp, NFS_FHSIZE);
- memcpy(&fhp->fh_handle.fh_base, p, NFS_FHSIZE);
+ memcpy(&fhp->fh_handle.fh_raw, p, NFS_FHSIZE);
fhp->fh_handle.fh_size = NFS_FHSIZE;
return true;
p = xdr_reserve_space(xdr, NFS_FHSIZE);
if (!p)
return false;
- memcpy(p, &fhp->fh_handle.fh_base, NFS_FHSIZE);
+ memcpy(p, &fhp->fh_handle.fh_raw, NFS_FHSIZE);
return true;
}
* XDR decode functions
*/
-int
-nfssvc_decode_fhandleargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_fhandleargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_fhandle *args = rqstp->rq_argp;
return svcxdr_decode_fhandle(xdr, &args->fh);
}
-int
-nfssvc_decode_sattrargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_sattrargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_sattrargs *args = rqstp->rq_argp;
return svcxdr_decode_fhandle(xdr, &args->fh) &&
svcxdr_decode_sattr(rqstp, xdr, &args->attrs);
}
-int
-nfssvc_decode_diropargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_diropargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_diropargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs(xdr, &args->fh, &args->name, &args->len);
}
-int
-nfssvc_decode_readargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_readargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_readargs *args = rqstp->rq_argp;
u32 totalcount;
if (!svcxdr_decode_fhandle(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->offset) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
/* totalcount is ignored */
if (xdr_stream_decode_u32(xdr, &totalcount) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
-int
-nfssvc_decode_writeargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_writeargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_writeargs *args = rqstp->rq_argp;
- struct kvec *head = rqstp->rq_arg.head;
- struct kvec *tail = rqstp->rq_arg.tail;
u32 beginoffset, totalcount;
- size_t remaining;
if (!svcxdr_decode_fhandle(xdr, &args->fh))
- return 0;
+ return false;
/* beginoffset is ignored */
if (xdr_stream_decode_u32(xdr, &beginoffset) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->offset) < 0)
- return 0;
+ return false;
/* totalcount is ignored */
if (xdr_stream_decode_u32(xdr, &totalcount) < 0)
- return 0;
+ return false;
/* opaque data */
if (xdr_stream_decode_u32(xdr, &args->len) < 0)
- return 0;
+ return false;
if (args->len > NFSSVC_MAXBLKSIZE_V2)
- return 0;
- remaining = head->iov_len + rqstp->rq_arg.page_len + tail->iov_len;
- remaining -= xdr_stream_pos(xdr);
- if (remaining < xdr_align_size(args->len))
- return 0;
- args->first.iov_base = xdr->p;
- args->first.iov_len = head->iov_len - xdr_stream_pos(xdr);
+ return false;
+ if (!xdr_stream_subsegment(xdr, &args->payload, args->len))
+ return false;
- return 1;
+ return true;
}
-int
-nfssvc_decode_createargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_createargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_createargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs(xdr, &args->fh,
svcxdr_decode_sattr(rqstp, xdr, &args->attrs);
}
-int
-nfssvc_decode_renameargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_renameargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_renameargs *args = rqstp->rq_argp;
return svcxdr_decode_diropargs(xdr, &args->ffh,
&args->tname, &args->tlen);
}
-int
-nfssvc_decode_linkargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_linkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_linkargs *args = rqstp->rq_argp;
return svcxdr_decode_fhandle(xdr, &args->ffh) &&
&args->tname, &args->tlen);
}
-int
-nfssvc_decode_symlinkargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_symlinkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_symlinkargs *args = rqstp->rq_argp;
struct kvec *head = rqstp->rq_arg.head;
if (!svcxdr_decode_diropargs(xdr, &args->ffh, &args->fname, &args->flen))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->tlen) < 0)
- return 0;
+ return false;
if (args->tlen == 0)
- return 0;
+ return false;
args->first.iov_len = head->iov_len - xdr_stream_pos(xdr);
args->first.iov_base = xdr_inline_decode(xdr, args->tlen);
if (!args->first.iov_base)
- return 0;
+ return false;
return svcxdr_decode_sattr(rqstp, xdr, &args->attrs);
}
-int
-nfssvc_decode_readdirargs(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_decode_readdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_arg_stream;
struct nfsd_readdirargs *args = rqstp->rq_argp;
if (!svcxdr_decode_fhandle(xdr, &args->fh))
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->cookie) < 0)
- return 0;
+ return false;
if (xdr_stream_decode_u32(xdr, &args->count) < 0)
- return 0;
+ return false;
- return 1;
+ return true;
}
/*
* XDR encode functions
*/
-int
-nfssvc_encode_statres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_statres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_stat *resp = rqstp->rq_resp;
return svcxdr_encode_stat(xdr, resp->status);
}
-int
-nfssvc_encode_attrstatres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_attrstatres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_attrstat *resp = rqstp->rq_resp;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_fattr(rqstp, xdr, &resp->fh, &resp->stat))
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
-int
-nfssvc_encode_diropres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_diropres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_diropres *resp = rqstp->rq_resp;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_fhandle(xdr, &resp->fh))
- return 0;
+ return false;
if (!svcxdr_encode_fattr(rqstp, xdr, &resp->fh, &resp->stat))
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
-int
-nfssvc_encode_readlinkres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_readlinkres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_readlinkres *resp = rqstp->rq_resp;
struct kvec *head = rqstp->rq_res.head;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (xdr_stream_encode_u32(xdr, resp->len) < 0)
- return 0;
+ return false;
xdr_write_pages(xdr, &resp->page, 0, resp->len);
if (svc_encode_result_payload(rqstp, head->iov_len, resp->len) < 0)
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
-int
-nfssvc_encode_readres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_readres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_readres *resp = rqstp->rq_resp;
struct kvec *head = rqstp->rq_res.head;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
if (!svcxdr_encode_fattr(rqstp, xdr, &resp->fh, &resp->stat))
- return 0;
+ return false;
if (xdr_stream_encode_u32(xdr, resp->count) < 0)
- return 0;
+ return false;
xdr_write_pages(xdr, resp->pages, rqstp->rq_res.page_base,
resp->count);
if (svc_encode_result_payload(rqstp, head->iov_len, resp->count) < 0)
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
-int
-nfssvc_encode_readdirres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_readdirres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_readdirres *resp = rqstp->rq_resp;
struct xdr_buf *dirlist = &resp->dirlist;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
xdr_write_pages(xdr, dirlist->pages, 0, dirlist->len);
/* no more entries */
if (xdr_stream_encode_item_absent(xdr) < 0)
- return 0;
+ return false;
if (xdr_stream_encode_bool(xdr, resp->common.err == nfserr_eof) < 0)
- return 0;
+ return false;
break;
}
- return 1;
+ return true;
}
-int
-nfssvc_encode_statfsres(struct svc_rqst *rqstp, __be32 *p)
+bool
+nfssvc_encode_statfsres(struct svc_rqst *rqstp, struct xdr_stream *xdr)
{
- struct xdr_stream *xdr = &rqstp->rq_res_stream;
struct nfsd_statfsres *resp = rqstp->rq_resp;
struct kstatfs *stat = &resp->stats;
+ __be32 *p;
if (!svcxdr_encode_stat(xdr, resp->status))
- return 0;
+ return false;
switch (resp->status) {
case nfs_ok:
p = xdr_reserve_space(xdr, XDR_UNIT * 5);
if (!p)
- return 0;
+ return false;
*p++ = cpu_to_be32(NFSSVC_MAXBLKSIZE_V2);
*p++ = cpu_to_be32(stat->f_bsize);
*p++ = cpu_to_be32(stat->f_blocks);
break;
}
- return 1;
+ return true;
}
/**
#define _NFSD_TRACE_H
#include <linux/tracepoint.h>
+
#include "export.h"
#include "nfsfh.h"
* returned. Otherwise the covered directory is returned.
* NOTE: this mountpoint crossing is not supported properly by all
* clients and is explicitly disallowed for NFSv3
+ * NeilBrown <neilb@cse.unsw.edu.au>
*/
__be32
nfsd_lookup(struct svc_rqst *rqstp, struct svc_fh *fhp, const char *name,
path.dentry = fhp->fh_dentry;
inode = d_inode(path.dentry);
- /* Disallow write access to files with the append-only bit set
- * or any access when mandatory locking enabled
- */
err = nfserr_perm;
if (IS_APPEND(inode) && (may_flags & NFSD_MAY_WRITE))
goto out;
if (nfsd_create_is_exclusive(createmode)) {
/* solaris7 gets confused (bugid 4218508) if these have
- * the high bit set, so just clear the high bits. If this is
+ * the high bit set, as do xfs filesystems without the
+ * "bigtime" feature. So just clear the high bits. If this is
* ever changed to use different attrs for storing the
* verifier, then do_open_lookup() will also need to be fixed
* accordingly.
svc_fh fh;
__u32 offset;
int len;
- struct kvec first;
+ struct xdr_buf payload;
};
struct nfsd_createargs {
#define NFS2_SVC_XDRSIZE sizeof(union nfsd_xdrstore)
-int nfssvc_decode_fhandleargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_sattrargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_diropargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_readargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_writeargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_createargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_renameargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_linkargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_symlinkargs(struct svc_rqst *, __be32 *);
-int nfssvc_decode_readdirargs(struct svc_rqst *, __be32 *);
-int nfssvc_encode_statres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_attrstatres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_diropres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_readlinkres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_readres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_statfsres(struct svc_rqst *, __be32 *);
-int nfssvc_encode_readdirres(struct svc_rqst *, __be32 *);
+bool nfssvc_decode_fhandleargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_sattrargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_diropargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_readargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_writeargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_createargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_renameargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_linkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_symlinkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_decode_readdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+
+bool nfssvc_encode_statres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_attrstatres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_diropres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_readlinkres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_readres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_statfsres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfssvc_encode_readdirres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
void nfssvc_encode_nfscookie(struct nfsd_readdirres *resp, u32 offset);
int nfssvc_encode_entry(void *data, const char *name, int namlen,
__u32 count;
int stable;
__u32 len;
- struct kvec first;
+ struct xdr_buf payload;
};
struct nfsd3_createargs {
#define NFS3_SVC_XDRSIZE sizeof(union nfsd3_xdrstore)
-int nfs3svc_decode_fhandleargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_sattrargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_diropargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_accessargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_readargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_writeargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_createargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_mkdirargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_mknodargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_renameargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_linkargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_symlinkargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_readdirargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_readdirplusargs(struct svc_rqst *, __be32 *);
-int nfs3svc_decode_commitargs(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_getattrres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_wccstat(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_lookupres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_accessres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_readlinkres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_readres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_writeres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_createres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_renameres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_linkres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_readdirres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_fsstatres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_fsinfores(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_pathconfres(struct svc_rqst *, __be32 *);
-int nfs3svc_encode_commitres(struct svc_rqst *, __be32 *);
+bool nfs3svc_decode_fhandleargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_sattrargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_diropargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_accessargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_readargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_writeargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_createargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_mkdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_mknodargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_renameargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_linkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_symlinkargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_readdirargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_readdirplusargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_decode_commitargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+
+bool nfs3svc_encode_getattrres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_wccstat(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_lookupres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_accessres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_readlinkres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_readres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_writeres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_createres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_renameres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_linkres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_readdirres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_fsstatres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_fsinfores(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_pathconfres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs3svc_encode_commitres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
void nfs3svc_release_fhandle(struct svc_rqst *);
void nfs3svc_release_fhandle2(struct svc_rqst *);
struct xdr_stream *xdr;
struct svc_rqst * rqstp;
+ __be32 *statusp;
u32 taglen;
char * tag;
u32 opcnt;
- __be32 * tagp; /* tag, opcount encode location */
+
struct nfsd4_compound_state cstate;
};
bool nfsd4_mach_creds_match(struct nfs4_client *cl, struct svc_rqst *rqstp);
-int nfs4svc_decode_compoundargs(struct svc_rqst *, __be32 *);
-int nfs4svc_encode_compoundres(struct svc_rqst *, __be32 *);
+bool nfs4svc_decode_compoundargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nfs4svc_encode_compoundres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
__be32 nfsd4_check_resp_size(struct nfsd4_compoundres *, u32);
void nfsd4_encode_operation(struct nfsd4_compoundres *, struct nfsd4_op *);
void nfsd4_encode_replay(struct xdr_stream *xdr, struct nfsd4_op *op);
// SPDX-License-Identifier: GPL-2.0+
/*
- * alloc.c - NILFS dat/inode allocator
+ * NILFS dat/inode allocator
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * alloc.h - persistent object (dat entry/disk inode) allocator/deallocator
+ * Persistent object (dat entry/disk inode) allocator/deallocator
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * bmap.c - NILFS block mapping.
+ * NILFS block mapping.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * bmap.h - NILFS block mapping.
+ * NILFS block mapping.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * btnode.c - NILFS B-tree node cache
+ * NILFS B-tree node cache
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * btnode.h - NILFS B-tree node cache
+ * NILFS B-tree node cache
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * btree.c - NILFS B-tree.
+ * NILFS B-tree.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * btree.h - NILFS B-tree.
+ * NILFS B-tree.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * cpfile.c - NILFS checkpoint file.
+ * NILFS checkpoint file.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * cpfile.h - NILFS checkpoint file.
+ * NILFS checkpoint file.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * dat.c - NILFS disk address translation.
+ * NILFS disk address translation.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * dat.h - NILFS disk address translation.
+ * NILFS disk address translation.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * dir.c - NILFS directory entry operations
+ * NILFS directory entry operations
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * direct.c - NILFS direct block pointer.
+ * NILFS direct block pointer.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * direct.h - NILFS direct block pointer.
+ * NILFS direct block pointer.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * file.c - NILFS regular file handling primitives including fsync().
+ * NILFS regular file handling primitives including fsync().
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * gcinode.c - dummy inodes to buffer blocks for garbage collection
+ * Dummy inodes to buffer blocks for garbage collection
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * ifile.c - NILFS inode file
+ * NILFS inode file
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * ifile.h - NILFS inode file
+ * NILFS inode file
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * inode.c - NILFS inode operations.
+ * NILFS inode operations.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * ioctl.c - NILFS ioctl operations.
+ * NILFS ioctl operations.
*
* Copyright (C) 2007, 2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * mdt.c - meta data file for NILFS
+ * Meta data file for NILFS
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * mdt.h - NILFS meta data file prototype and definitions
+ * NILFS meta data file prototype and definitions
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * namei.c - NILFS pathname lookup operations.
+ * NILFS pathname lookup operations.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * nilfs.h - NILFS local header file.
+ * NILFS local header file.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * page.c - buffer/page management specific to NILFS
+ * Buffer/page management specific to NILFS
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * page.h - buffer/page management specific to NILFS
+ * Buffer/page management specific to NILFS
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * recovery.c - NILFS recovery logic
+ * NILFS recovery logic
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * segbuf.c - NILFS segment buffer
+ * NILFS segment buffer
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * segbuf.h - NILFS Segment buffer prototypes and definitions
+ * NILFS Segment buffer prototypes and definitions
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * segment.c - NILFS segment constructor.
+ * NILFS segment constructor.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * segment.h - NILFS Segment constructor prototypes and definitions
+ * NILFS Segment constructor prototypes and definitions
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * sufile.c - NILFS segment usage file.
+ * NILFS segment usage file.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * sufile.h - NILFS segment usage file.
+ * NILFS segment usage file.
*
* Copyright (C) 2006-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * super.c - NILFS module and super block management.
+ * NILFS module and super block management.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
// SPDX-License-Identifier: GPL-2.0+
/*
- * sysfs.c - sysfs support implementation.
+ * Sysfs support implementation.
*
* Copyright (C) 2005-2014 Nippon Telegraph and Telephone Corporation.
* Copyright (C) 2014 HGST, Inc., a Western Digital Company.
nilfs_snapshot_inodes_count_show(struct nilfs_snapshot_attr *attr,
struct nilfs_root *root, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)atomic64_read(&root->inodes_count));
}
nilfs_snapshot_blocks_count_show(struct nilfs_snapshot_attr *attr,
struct nilfs_root *root, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)atomic64_read(&root->blocks_count));
}
nilfs_snapshot_README_show(struct nilfs_snapshot_attr *attr,
struct nilfs_root *root, char *buf)
{
- return snprintf(buf, PAGE_SIZE, snapshot_readme_str);
+ return sysfs_emit(buf, snapshot_readme_str);
}
NILFS_SNAPSHOT_RO_ATTR(inodes_count);
nilfs_mounted_snapshots_README_show(struct nilfs_mounted_snapshots_attr *attr,
struct the_nilfs *nilfs, char *buf)
{
- return snprintf(buf, PAGE_SIZE, mounted_snapshots_readme_str);
+ return sysfs_emit(buf, mounted_snapshots_readme_str);
}
NILFS_MOUNTED_SNAPSHOTS_RO_ATTR(README);
ncheckpoints = cpstat.cs_ncps;
- return snprintf(buf, PAGE_SIZE, "%llu\n", ncheckpoints);
+ return sysfs_emit(buf, "%llu\n", ncheckpoints);
}
static ssize_t
nsnapshots = cpstat.cs_nsss;
- return snprintf(buf, PAGE_SIZE, "%llu\n", nsnapshots);
+ return sysfs_emit(buf, "%llu\n", nsnapshots);
}
static ssize_t
last_cno = nilfs->ns_last_cno;
spin_unlock(&nilfs->ns_last_segment_lock);
- return snprintf(buf, PAGE_SIZE, "%llu\n", last_cno);
+ return sysfs_emit(buf, "%llu\n", last_cno);
}
static ssize_t
cno = nilfs->ns_cno;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", cno);
+ return sysfs_emit(buf, "%llu\n", cno);
}
static const char checkpoints_readme_str[] =
nilfs_checkpoints_README_show(struct nilfs_checkpoints_attr *attr,
struct the_nilfs *nilfs, char *buf)
{
- return snprintf(buf, PAGE_SIZE, checkpoints_readme_str);
+ return sysfs_emit(buf, checkpoints_readme_str);
}
NILFS_CHECKPOINTS_RO_ATTR(checkpoints_number);
struct the_nilfs *nilfs,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%lu\n", nilfs->ns_nsegments);
+ return sysfs_emit(buf, "%lu\n", nilfs->ns_nsegments);
}
static ssize_t
struct the_nilfs *nilfs,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%lu\n", nilfs->ns_blocks_per_segment);
+ return sysfs_emit(buf, "%lu\n", nilfs->ns_blocks_per_segment);
}
static ssize_t
ncleansegs = nilfs_sufile_get_ncleansegs(nilfs->ns_sufile);
up_read(&NILFS_MDT(nilfs->ns_dat)->mi_sem);
- return snprintf(buf, PAGE_SIZE, "%lu\n", ncleansegs);
+ return sysfs_emit(buf, "%lu\n", ncleansegs);
}
static ssize_t
return err;
}
- return snprintf(buf, PAGE_SIZE, "%llu\n", sustat.ss_ndirtysegs);
+ return sysfs_emit(buf, "%llu\n", sustat.ss_ndirtysegs);
}
static const char segments_readme_str[] =
struct the_nilfs *nilfs,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, segments_readme_str);
+ return sysfs_emit(buf, segments_readme_str);
}
NILFS_SEGMENTS_RO_ATTR(segments_number);
last_pseg = nilfs->ns_last_pseg;
spin_unlock(&nilfs->ns_last_segment_lock);
- return snprintf(buf, PAGE_SIZE, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)last_pseg);
}
last_seq = nilfs->ns_last_seq;
spin_unlock(&nilfs->ns_last_segment_lock);
- return snprintf(buf, PAGE_SIZE, "%llu\n", last_seq);
+ return sysfs_emit(buf, "%llu\n", last_seq);
}
static ssize_t
last_cno = nilfs->ns_last_cno;
spin_unlock(&nilfs->ns_last_segment_lock);
- return snprintf(buf, PAGE_SIZE, "%llu\n", last_cno);
+ return sysfs_emit(buf, "%llu\n", last_cno);
}
static ssize_t
seg_seq = nilfs->ns_seg_seq;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", seg_seq);
+ return sysfs_emit(buf, "%llu\n", seg_seq);
}
static ssize_t
segnum = nilfs->ns_segnum;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", segnum);
+ return sysfs_emit(buf, "%llu\n", segnum);
}
static ssize_t
nextnum = nilfs->ns_nextnum;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", nextnum);
+ return sysfs_emit(buf, "%llu\n", nextnum);
}
static ssize_t
pseg_offset = nilfs->ns_pseg_offset;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%lu\n", pseg_offset);
+ return sysfs_emit(buf, "%lu\n", pseg_offset);
}
static ssize_t
cno = nilfs->ns_cno;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", cno);
+ return sysfs_emit(buf, "%llu\n", cno);
}
static ssize_t
ctime = nilfs->ns_ctime;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", ctime);
+ return sysfs_emit(buf, "%llu\n", ctime);
}
static ssize_t
nongc_ctime = nilfs->ns_nongc_ctime;
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", nongc_ctime);
+ return sysfs_emit(buf, "%llu\n", nongc_ctime);
}
static ssize_t
ndirtyblks = atomic_read(&nilfs->ns_ndirtyblks);
up_read(&nilfs->ns_segctor_sem);
- return snprintf(buf, PAGE_SIZE, "%u\n", ndirtyblks);
+ return sysfs_emit(buf, "%u\n", ndirtyblks);
}
static const char segctor_readme_str[] =
nilfs_segctor_README_show(struct nilfs_segctor_attr *attr,
struct the_nilfs *nilfs, char *buf)
{
- return snprintf(buf, PAGE_SIZE, segctor_readme_str);
+ return sysfs_emit(buf, segctor_readme_str);
}
NILFS_SEGCTOR_RO_ATTR(last_pseg_block);
sbwtime = nilfs->ns_sbwtime;
up_read(&nilfs->ns_sem);
- return snprintf(buf, PAGE_SIZE, "%llu\n", sbwtime);
+ return sysfs_emit(buf, "%llu\n", sbwtime);
}
static ssize_t
sbwcount = nilfs->ns_sbwcount;
up_read(&nilfs->ns_sem);
- return snprintf(buf, PAGE_SIZE, "%u\n", sbwcount);
+ return sysfs_emit(buf, "%u\n", sbwcount);
}
static ssize_t
sb_update_freq = nilfs->ns_sb_update_freq;
up_read(&nilfs->ns_sem);
- return snprintf(buf, PAGE_SIZE, "%u\n", sb_update_freq);
+ return sysfs_emit(buf, "%u\n", sb_update_freq);
}
static ssize_t
nilfs_superblock_README_show(struct nilfs_superblock_attr *attr,
struct the_nilfs *nilfs, char *buf)
{
- return snprintf(buf, PAGE_SIZE, sb_readme_str);
+ return sysfs_emit(buf, sb_readme_str);
}
NILFS_SUPERBLOCK_RO_ATTR(sb_write_time);
u32 major = le32_to_cpu(sbp[0]->s_rev_level);
u16 minor = le16_to_cpu(sbp[0]->s_minor_rev_level);
- return snprintf(buf, PAGE_SIZE, "%d.%d\n", major, minor);
+ return sysfs_emit(buf, "%d.%d\n", major, minor);
}
static
struct the_nilfs *nilfs,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%u\n", nilfs->ns_blocksize);
+ return sysfs_emit(buf, "%u\n", nilfs->ns_blocksize);
}
static
struct nilfs_super_block **sbp = nilfs->ns_sbp;
u64 dev_size = le64_to_cpu(sbp[0]->s_dev_size);
- return snprintf(buf, PAGE_SIZE, "%llu\n", dev_size);
+ return sysfs_emit(buf, "%llu\n", dev_size);
}
static
sector_t free_blocks = 0;
nilfs_count_free_blocks(nilfs, &free_blocks);
- return snprintf(buf, PAGE_SIZE, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)free_blocks);
}
{
struct nilfs_super_block **sbp = nilfs->ns_sbp;
- return snprintf(buf, PAGE_SIZE, "%pUb\n", sbp[0]->s_uuid);
+ return sysfs_emit(buf, "%pUb\n", sbp[0]->s_uuid);
}
static
struct the_nilfs *nilfs,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, dev_readme_str);
+ return sysfs_emit(buf, dev_readme_str);
}
NILFS_DEV_RO_ATTR(revision);
static ssize_t nilfs_feature_revision_show(struct kobject *kobj,
struct attribute *attr, char *buf)
{
- return snprintf(buf, PAGE_SIZE, "%d.%d\n",
+ return sysfs_emit(buf, "%d.%d\n",
NILFS_CURRENT_REV, NILFS_MINOR_REV);
}
struct attribute *attr,
char *buf)
{
- return snprintf(buf, PAGE_SIZE, features_readme_str);
+ return sysfs_emit(buf, features_readme_str);
}
NILFS_FEATURE_RO_ATTR(revision);
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * sysfs.h - sysfs support declarations.
+ * Sysfs support declarations.
*
* Copyright (C) 2005-2014 Nippon Telegraph and Telephone Corporation.
* Copyright (C) 2014 HGST, Inc., a Western Digital Company.
// SPDX-License-Identifier: GPL-2.0+
/*
- * the_nilfs.c - the_nilfs shared structure.
+ * the_nilfs shared structure.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
/* SPDX-License-Identifier: GPL-2.0+ */
/*
- * the_nilfs.h - the_nilfs shared structure.
+ * the_nilfs shared structure.
*
* Copyright (C) 2005-2008 Nippon Telegraph and Telephone Corporation.
*
return fanotify_info_equal(info1, info2);
}
+static bool fanotify_error_event_equal(struct fanotify_error_event *fee1,
+ struct fanotify_error_event *fee2)
+{
+ /* Error events against the same file system are always merged. */
+ if (!fanotify_fsid_equal(&fee1->fsid, &fee2->fsid))
+ return false;
+
+ return true;
+}
+
static bool fanotify_should_merge(struct fanotify_event *old,
struct fanotify_event *new)
{
case FANOTIFY_EVENT_TYPE_FID_NAME:
return fanotify_name_event_equal(FANOTIFY_NE(old),
FANOTIFY_NE(new));
+ case FANOTIFY_EVENT_TYPE_FS_ERROR:
+ return fanotify_error_event_equal(FANOTIFY_EE(old),
+ FANOTIFY_EE(new));
default:
WARN_ON_ONCE(1);
}
break;
if (fanotify_should_merge(old, new)) {
old->mask |= new->mask;
+
+ if (fanotify_is_error_event(old->mask))
+ FANOTIFY_EE(old)->err_count++;
+
return 1;
}
}
static int fanotify_encode_fh_len(struct inode *inode)
{
int dwords = 0;
+ int fh_len;
if (!inode)
return 0;
exportfs_encode_inode_fh(inode, NULL, &dwords, NULL);
+ fh_len = dwords << 2;
+
+ /*
+ * struct fanotify_error_event might be preallocated and is
+ * limited to MAX_HANDLE_SZ. This should never happen, but
+ * safeguard by forcing an invalid file handle.
+ */
+ if (WARN_ON_ONCE(fh_len > MAX_HANDLE_SZ))
+ return 0;
- return dwords << 2;
+ return fh_len;
}
/*
fh->type = FILEID_ROOT;
fh->len = 0;
fh->flags = 0;
+
+ /*
+ * Invalid FHs are used by FAN_FS_ERROR for errors not
+ * linked to any inode. The f_handle won't be reported
+ * back to userspace.
+ */
if (!inode)
- return 0;
+ goto out;
/*
* !gpf means preallocated variable size fh, but fh_len could
fh->type = type;
fh->len = fh_len;
- /* Mix fh into event merge key */
- *hash ^= fanotify_hash_fh(fh);
+out:
+ /*
+ * Mix fh into event merge key. Hash might be NULL in case of
+ * unhashed FID events (i.e. FAN_FS_ERROR).
+ */
+ if (hash)
+ *hash ^= fanotify_hash_fh(fh);
return FANOTIFY_FH_HDR_LEN + fh_len;
if (event_mask & ALL_FSNOTIFY_DIRENT_EVENTS)
return dir;
- if (S_ISDIR(inode->i_mode))
+ if (inode && S_ISDIR(inode->i_mode))
return inode;
return dir;
return &fne->fae;
}
+static struct fanotify_event *fanotify_alloc_error_event(
+ struct fsnotify_group *group,
+ __kernel_fsid_t *fsid,
+ const void *data, int data_type,
+ unsigned int *hash)
+{
+ struct fs_error_report *report =
+ fsnotify_data_error_report(data, data_type);
+ struct inode *inode;
+ struct fanotify_error_event *fee;
+ int fh_len;
+
+ if (WARN_ON_ONCE(!report))
+ return NULL;
+
+ fee = mempool_alloc(&group->fanotify_data.error_events_pool, GFP_NOFS);
+ if (!fee)
+ return NULL;
+
+ fee->fae.type = FANOTIFY_EVENT_TYPE_FS_ERROR;
+ fee->error = report->error;
+ fee->err_count = 1;
+ fee->fsid = *fsid;
+
+ inode = report->inode;
+ fh_len = fanotify_encode_fh_len(inode);
+
+ /* Bad fh_len. Fallback to using an invalid fh. Should never happen. */
+ if (!fh_len && inode)
+ inode = NULL;
+
+ fanotify_encode_fh(&fee->object_fh, inode, fh_len, NULL, 0);
+
+ *hash ^= fanotify_hash_fsid(fsid);
+
+ return &fee->fae;
+}
+
static struct fanotify_event *fanotify_alloc_event(struct fsnotify_group *group,
u32 mask, const void *data,
int data_type, struct inode *dir,
if (fanotify_is_perm_event(mask)) {
event = fanotify_alloc_perm_event(path, gfp);
+ } else if (fanotify_is_error_event(mask)) {
+ event = fanotify_alloc_error_event(group, fsid, data,
+ data_type, &hash);
} else if (name_event && (file_name || child)) {
event = fanotify_alloc_name_event(id, fsid, file_name, child,
&hash, gfp);
assert_spin_locked(&group->notification_lock);
+ if (!fanotify_is_hashed_event(event->mask))
+ return;
+
pr_debug("%s: group=%p event=%p bucket=%u\n", __func__,
group, event, bucket);
BUILD_BUG_ON(FAN_ONDIR != FS_ISDIR);
BUILD_BUG_ON(FAN_OPEN_EXEC != FS_OPEN_EXEC);
BUILD_BUG_ON(FAN_OPEN_EXEC_PERM != FS_OPEN_EXEC_PERM);
+ BUILD_BUG_ON(FAN_FS_ERROR != FS_ERROR);
- BUILD_BUG_ON(HWEIGHT32(ALL_FANOTIFY_EVENT_BITS) != 19);
+ BUILD_BUG_ON(HWEIGHT32(ALL_FANOTIFY_EVENT_BITS) != 20);
mask = fanotify_group_event_mask(group, iter_info, mask, data,
data_type, dir);
}
fsn_event = &event->fse;
- ret = fsnotify_add_event(group, fsn_event, fanotify_merge,
- fanotify_is_hashed_event(mask) ?
- fanotify_insert_event : NULL);
+ ret = fsnotify_insert_event(group, fsn_event, fanotify_merge,
+ fanotify_insert_event);
if (ret) {
/* Permission events shouldn't be merged */
BUG_ON(ret == 1 && mask & FANOTIFY_PERM_EVENTS);
if (group->fanotify_data.ucounts)
dec_ucount(group->fanotify_data.ucounts,
UCOUNT_FANOTIFY_GROUPS);
+
+ if (mempool_initialized(&group->fanotify_data.error_events_pool))
+ mempool_exit(&group->fanotify_data.error_events_pool);
}
static void fanotify_free_path_event(struct fanotify_event *event)
kfree(FANOTIFY_NE(event));
}
-static void fanotify_free_event(struct fsnotify_event *fsn_event)
+static void fanotify_free_error_event(struct fsnotify_group *group,
+ struct fanotify_event *event)
+{
+ struct fanotify_error_event *fee = FANOTIFY_EE(event);
+
+ mempool_free(fee, &group->fanotify_data.error_events_pool);
+}
+
+static void fanotify_free_event(struct fsnotify_group *group,
+ struct fsnotify_event *fsn_event)
{
struct fanotify_event *event;
case FANOTIFY_EVENT_TYPE_OVERFLOW:
kfree(event);
break;
+ case FANOTIFY_EVENT_TYPE_FS_ERROR:
+ fanotify_free_error_event(group, event);
+ break;
default:
WARN_ON_ONCE(1);
}
FANOTIFY_EVENT_TYPE_PATH,
FANOTIFY_EVENT_TYPE_PATH_PERM,
FANOTIFY_EVENT_TYPE_OVERFLOW, /* struct fanotify_event */
+ FANOTIFY_EVENT_TYPE_FS_ERROR, /* struct fanotify_error_event */
__FANOTIFY_EVENT_TYPE_NUM
};
event->pid = NULL;
}
+#define FANOTIFY_INLINE_FH(name, size) \
+struct { \
+ struct fanotify_fh (name); \
+ /* Space for object_fh.buf[] - access with fanotify_fh_buf() */ \
+ unsigned char _inline_fh_buf[(size)]; \
+}
+
struct fanotify_fid_event {
struct fanotify_event fae;
__kernel_fsid_t fsid;
- struct fanotify_fh object_fh;
- /* Reserve space in object_fh.buf[] - access with fanotify_fh_buf() */
- unsigned char _inline_fh_buf[FANOTIFY_INLINE_FH_LEN];
+
+ FANOTIFY_INLINE_FH(object_fh, FANOTIFY_INLINE_FH_LEN);
};
static inline struct fanotify_fid_event *
return container_of(event, struct fanotify_name_event, fae);
}
+struct fanotify_error_event {
+ struct fanotify_event fae;
+ s32 error; /* Error reported by the Filesystem. */
+ u32 err_count; /* Suppressed errors count */
+
+ __kernel_fsid_t fsid; /* FSID this error refers to. */
+
+ FANOTIFY_INLINE_FH(object_fh, MAX_HANDLE_SZ);
+};
+
+static inline struct fanotify_error_event *
+FANOTIFY_EE(struct fanotify_event *event)
+{
+ return container_of(event, struct fanotify_error_event, fae);
+}
+
static inline __kernel_fsid_t *fanotify_event_fsid(struct fanotify_event *event)
{
if (event->type == FANOTIFY_EVENT_TYPE_FID)
return &FANOTIFY_FE(event)->fsid;
else if (event->type == FANOTIFY_EVENT_TYPE_FID_NAME)
return &FANOTIFY_NE(event)->fsid;
+ else if (event->type == FANOTIFY_EVENT_TYPE_FS_ERROR)
+ return &FANOTIFY_EE(event)->fsid;
else
return NULL;
}
return &FANOTIFY_FE(event)->object_fh;
else if (event->type == FANOTIFY_EVENT_TYPE_FID_NAME)
return fanotify_info_file_fh(&FANOTIFY_NE(event)->info);
+ else if (event->type == FANOTIFY_EVENT_TYPE_FS_ERROR)
+ return &FANOTIFY_EE(event)->object_fh;
else
return NULL;
}
return info ? fanotify_info_dir_fh_len(info) : 0;
}
+static inline bool fanotify_event_has_object_fh(struct fanotify_event *event)
+{
+ /* For error events, even zeroed fh are reported. */
+ if (event->type == FANOTIFY_EVENT_TYPE_FS_ERROR)
+ return true;
+ return fanotify_event_object_fh_len(event) > 0;
+}
+
+static inline bool fanotify_event_has_dir_fh(struct fanotify_event *event)
+{
+ return fanotify_event_dir_fh_len(event) > 0;
+}
+
struct fanotify_path_event {
struct fanotify_event fae;
struct path path;
return container_of(fse, struct fanotify_event, fse);
}
+static inline bool fanotify_is_error_event(u32 mask)
+{
+ return mask & FAN_FS_ERROR;
+}
+
static inline bool fanotify_event_has_path(struct fanotify_event *event)
{
return event->type == FANOTIFY_EVENT_TYPE_PATH ||
*/
static inline bool fanotify_is_hashed_event(u32 mask)
{
- return !fanotify_is_perm_event(mask) && !(mask & FS_Q_OVERFLOW);
+ return !(fanotify_is_perm_event(mask) ||
+ fsnotify_is_overflow_event(mask));
}
static inline unsigned int fanotify_event_hash_bucket(
#define FANOTIFY_DEFAULT_MAX_EVENTS 16384
#define FANOTIFY_OLD_DEFAULT_MAX_MARKS 8192
#define FANOTIFY_DEFAULT_MAX_GROUPS 128
+#define FANOTIFY_DEFAULT_FEE_POOL_SIZE 32
/*
* Legacy fanotify marks limits (8192) is per group and we introduced a tunable
(sizeof(struct fanotify_event_info_fid) + sizeof(struct file_handle))
#define FANOTIFY_PIDFD_INFO_HDR_LEN \
sizeof(struct fanotify_event_info_pidfd)
+#define FANOTIFY_ERROR_INFO_LEN \
+ (sizeof(struct fanotify_event_info_error))
static int fanotify_fid_info_len(int fh_len, int name_len)
{
FANOTIFY_EVENT_ALIGN);
}
-static int fanotify_event_info_len(unsigned int info_mode,
- struct fanotify_event *event)
+static size_t fanotify_event_len(unsigned int info_mode,
+ struct fanotify_event *event)
{
- struct fanotify_info *info = fanotify_event_info(event);
- int dir_fh_len = fanotify_event_dir_fh_len(event);
- int fh_len = fanotify_event_object_fh_len(event);
- int info_len = 0;
+ size_t event_len = FAN_EVENT_METADATA_LEN;
+ struct fanotify_info *info;
+ int dir_fh_len;
+ int fh_len;
int dot_len = 0;
- if (dir_fh_len) {
- info_len += fanotify_fid_info_len(dir_fh_len, info->name_len);
+ if (!info_mode)
+ return event_len;
+
+ if (fanotify_is_error_event(event->mask))
+ event_len += FANOTIFY_ERROR_INFO_LEN;
+
+ info = fanotify_event_info(event);
+
+ if (fanotify_event_has_dir_fh(event)) {
+ dir_fh_len = fanotify_event_dir_fh_len(event);
+ event_len += fanotify_fid_info_len(dir_fh_len, info->name_len);
} else if ((info_mode & FAN_REPORT_NAME) &&
(event->mask & FAN_ONDIR)) {
/*
}
if (info_mode & FAN_REPORT_PIDFD)
- info_len += FANOTIFY_PIDFD_INFO_HDR_LEN;
+ event_len += FANOTIFY_PIDFD_INFO_HDR_LEN;
- if (fh_len)
- info_len += fanotify_fid_info_len(fh_len, dot_len);
+ if (fanotify_event_has_object_fh(event)) {
+ fh_len = fanotify_event_object_fh_len(event);
+ event_len += fanotify_fid_info_len(fh_len, dot_len);
+ }
- return info_len;
+ return event_len;
}
/*
static struct fanotify_event *get_one_event(struct fsnotify_group *group,
size_t count)
{
- size_t event_size = FAN_EVENT_METADATA_LEN;
+ size_t event_size;
struct fanotify_event *event = NULL;
struct fsnotify_event *fsn_event;
unsigned int info_mode = FAN_GROUP_FLAG(group, FANOTIFY_INFO_MODES);
goto out;
event = FANOTIFY_E(fsn_event);
- if (info_mode)
- event_size += fanotify_event_info_len(info_mode, event);
+ event_size = fanotify_event_len(info_mode, event);
if (event_size > count) {
event = ERR_PTR(-EINVAL);
return -ENOENT;
}
+static size_t copy_error_info_to_user(struct fanotify_event *event,
+ char __user *buf, int count)
+{
+ struct fanotify_event_info_error info;
+ struct fanotify_error_event *fee = FANOTIFY_EE(event);
+
+ info.hdr.info_type = FAN_EVENT_INFO_TYPE_ERROR;
+ info.hdr.pad = 0;
+ info.hdr.len = FANOTIFY_ERROR_INFO_LEN;
+
+ if (WARN_ON(count < info.hdr.len))
+ return -EFAULT;
+
+ info.error = fee->error;
+ info.error_count = fee->err_count;
+
+ if (copy_to_user(buf, &info, sizeof(info)))
+ return -EFAULT;
+
+ return info.hdr.len;
+}
+
static int copy_fid_info_to_user(__kernel_fsid_t *fsid, struct fanotify_fh *fh,
int info_type, const char *name,
size_t name_len,
pr_debug("%s: fh_len=%zu name_len=%zu, info_len=%zu, count=%zu\n",
__func__, fh_len, name_len, info_len, count);
- if (!fh_len)
- return 0;
-
if (WARN_ON_ONCE(len < sizeof(info) || len > count))
return -EFAULT;
handle.handle_type = fh->type;
handle.handle_bytes = fh_len;
+
+ /* Mangle handle_type for bad file_handle */
+ if (!fh_len)
+ handle.handle_type = FILEID_INVALID;
+
if (copy_to_user(buf, &handle, sizeof(handle)))
return -EFAULT;
/*
* Event info records order is as follows: dir fid + name, child fid.
*/
- if (fanotify_event_dir_fh_len(event)) {
+ if (fanotify_event_has_dir_fh(event)) {
info_type = info->name_len ? FAN_EVENT_INFO_TYPE_DFID_NAME :
FAN_EVENT_INFO_TYPE_DFID;
ret = copy_fid_info_to_user(fanotify_event_fsid(event),
total_bytes += ret;
}
- if (fanotify_event_object_fh_len(event)) {
+ if (fanotify_event_has_object_fh(event)) {
const char *dot = NULL;
int dot_len = 0;
total_bytes += ret;
}
+ if (fanotify_is_error_event(event->mask)) {
+ ret = copy_error_info_to_user(event, buf, count);
+ if (ret < 0)
+ return ret;
+ buf += ret;
+ count -= ret;
+ total_bytes += ret;
+ }
+
return total_bytes;
}
pr_debug("%s: group=%p event=%p\n", __func__, group, event);
- metadata.event_len = FAN_EVENT_METADATA_LEN +
- fanotify_event_info_len(info_mode, event);
+ metadata.event_len = fanotify_event_len(info_mode, event);
metadata.metadata_len = FAN_EVENT_METADATA_LEN;
metadata.vers = FANOTIFY_METADATA_VERSION;
metadata.reserved = 0;
return ERR_PTR(ret);
}
+static int fanotify_group_init_error_pool(struct fsnotify_group *group)
+{
+ if (mempool_initialized(&group->fanotify_data.error_events_pool))
+ return 0;
+
+ return mempool_init_kmalloc_pool(&group->fanotify_data.error_events_pool,
+ FANOTIFY_DEFAULT_FEE_POOL_SIZE,
+ sizeof(struct fanotify_error_event));
+}
static int fanotify_add_mark(struct fsnotify_group *group,
fsnotify_connp_t *connp, unsigned int type,
{
struct fsnotify_mark *fsn_mark;
__u32 added;
+ int ret = 0;
mutex_lock(&group->mark_mutex);
fsn_mark = fsnotify_find_mark(connp, group);
return PTR_ERR(fsn_mark);
}
}
+
+ /*
+ * Error events are pre-allocated per group, only if strictly
+ * needed (i.e. FAN_FS_ERROR was requested).
+ */
+ if (!(flags & FAN_MARK_IGNORED_MASK) && (mask & FAN_FS_ERROR)) {
+ ret = fanotify_group_init_error_pool(group);
+ if (ret)
+ goto out;
+ }
+
added = fanotify_mark_add_to_mask(fsn_mark, mask, flags);
if (added & ~fsnotify_conn_mask(fsn_mark->connector))
fsnotify_recalc_mask(fsn_mark->connector);
+
+out:
mutex_unlock(&group->mark_mutex);
fsnotify_put_mark(fsn_mark);
- return 0;
+ return ret;
}
static int fanotify_add_vfsmount_mark(struct fsnotify_group *group,
return fd;
}
-/* Check if filesystem can encode a unique fid */
-static int fanotify_test_fid(struct path *path, __kernel_fsid_t *fsid)
+static int fanotify_test_fsid(struct dentry *dentry, __kernel_fsid_t *fsid)
{
__kernel_fsid_t root_fsid;
int err;
/*
- * Make sure path is not in filesystem with zero fsid (e.g. tmpfs).
+ * Make sure dentry is not of a filesystem with zero fsid (e.g. fuse).
*/
- err = vfs_get_fsid(path->dentry, fsid);
+ err = vfs_get_fsid(dentry, fsid);
if (err)
return err;
return -ENODEV;
/*
- * Make sure path is not inside a filesystem subvolume (e.g. btrfs)
+ * Make sure dentry is not of a filesystem subvolume (e.g. btrfs)
* which uses a different fsid than sb root.
*/
- err = vfs_get_fsid(path->dentry->d_sb->s_root, &root_fsid);
+ err = vfs_get_fsid(dentry->d_sb->s_root, &root_fsid);
if (err)
return err;
root_fsid.val[1] != fsid->val[1])
return -EXDEV;
+ return 0;
+}
+
+/* Check if filesystem can encode a unique fid */
+static int fanotify_test_fid(struct dentry *dentry)
+{
/*
* We need to make sure that the file system supports at least
* encoding a file handle so user can use name_to_handle_at() to
* objects. However, name_to_handle_at() requires that the
* filesystem also supports decoding file handles.
*/
- if (!path->dentry->d_sb->s_export_op ||
- !path->dentry->d_sb->s_export_op->fh_to_dentry)
+ if (!dentry->d_sb->s_export_op ||
+ !dentry->d_sb->s_export_op->fh_to_dentry)
return -EOPNOTSUPP;
return 0;
group->priority == FS_PRIO_0)
goto fput_and_out;
+ if (mask & FAN_FS_ERROR &&
+ mark_type != FAN_MARK_FILESYSTEM)
+ goto fput_and_out;
+
/*
- * Events with data type inode do not carry enough information to report
- * event->fd, so we do not allow setting a mask for inode events unless
- * group supports reporting fid.
- * inode events are not supported on a mount mark, because they do not
- * carry enough information (i.e. path) to be filtered by mount point.
+ * Events that do not carry enough information to report
+ * event->fd require a group that supports reporting fid. Those
+ * events are not supported on a mount mark, because they do not
+ * carry enough information (i.e. path) to be filtered by mount
+ * point.
*/
fid_mode = FAN_GROUP_FLAG(group, FANOTIFY_FID_BITS);
- if (mask & FANOTIFY_INODE_EVENTS &&
+ if (mask & ~(FANOTIFY_FD_EVENTS|FANOTIFY_EVENT_FLAGS) &&
(!fid_mode || mark_type == FAN_MARK_MOUNT))
goto fput_and_out;
}
if (fid_mode) {
- ret = fanotify_test_fid(&path, &__fsid);
+ ret = fanotify_test_fsid(path.dentry, &__fsid);
+ if (ret)
+ goto path_put_and_out;
+
+ ret = fanotify_test_fid(path.dentry);
if (ret)
goto path_put_and_out;
if (WARN_ON_ONCE(!ops->handle_inode_event))
return 0;
+ if (WARN_ON_ONCE(!inode && !dir))
+ return 0;
+
if ((inode_mark->mask & FS_EXCL_UNLINK) &&
path && d_unlinked(path->dentry))
return 0;
* @file_name is relative to
* @file_name: optional file name associated with event
* @inode: optional inode associated with event -
- * either @dir or @inode must be non-NULL.
- * if both are non-NULL event may be reported to both.
+ * If @dir and @inode are both non-NULL, event may be
+ * reported to both.
* @cookie: inotify rename cookie
*/
int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
const struct qstr *file_name, struct inode *inode, u32 cookie)
{
const struct path *path = fsnotify_data_path(data, data_type);
+ struct super_block *sb = fsnotify_data_sb(data, data_type);
struct fsnotify_iter_info iter_info = {};
- struct super_block *sb;
struct mount *mnt = NULL;
struct inode *parent = NULL;
int ret = 0;
*/
parent = dir;
}
- sb = inode->i_sb;
/*
* Optimization: srcu_read_lock() has a memory barrier which can
* that deliberately ignores overflow events.
*/
if (group->overflow_event)
- group->ops->free_event(group->overflow_event);
+ group->ops->free_event(group, group->overflow_event);
fsnotify_put_group(group);
}
if (len)
strcpy(event->name, name->name);
- ret = fsnotify_add_event(group, fsn_event, inotify_merge, NULL);
+ ret = fsnotify_add_event(group, fsn_event, inotify_merge);
if (ret) {
/* Our event wasn't used in the end. Free it. */
fsnotify_destroy_event(group, fsn_event);
dec_inotify_instances(group->inotify_data.ucounts);
}
-static void inotify_free_event(struct fsnotify_event *fsn_event)
+static void inotify_free_event(struct fsnotify_group *group,
+ struct fsnotify_event *fsn_event)
{
kfree(INOTIFY_E(fsn_event));
}
__u32 mask;
/*
- * Everything should accept their own ignored and should receive events
- * when the inode is unmounted. All directories care about children.
+ * Everything should receive events when the inode is unmounted.
+ * All directories care about children.
*/
- mask = (FS_IN_IGNORED | FS_UNMOUNT);
+ mask = (FS_UNMOUNT);
if (S_ISDIR(inode->i_mode))
mask |= FS_EVENT_ON_CHILD;
WARN_ON(!list_empty(&event->list));
spin_unlock(&group->notification_lock);
}
- group->ops->free_event(event);
+ group->ops->free_event(group, event);
}
/*
* 2 if the event was not queued - either the queue of events has overflown
* or the group is shutting down.
*/
-int fsnotify_add_event(struct fsnotify_group *group,
- struct fsnotify_event *event,
- int (*merge)(struct fsnotify_group *,
- struct fsnotify_event *),
- void (*insert)(struct fsnotify_group *,
- struct fsnotify_event *))
+int fsnotify_insert_event(struct fsnotify_group *group,
+ struct fsnotify_event *event,
+ int (*merge)(struct fsnotify_group *,
+ struct fsnotify_event *),
+ void (*insert)(struct fsnotify_group *,
+ struct fsnotify_event *))
{
int ret = 0;
struct list_head *list = &group->notification_list;
status = ocfs2_journal_access_di(handle, INODE_CACHE(tl_inode), tl_bh,
OCFS2_JOURNAL_ACCESS_WRITE);
if (status < 0) {
+ ocfs2_commit_trans(osb, handle);
mlog_errno(status);
goto bail;
}
data_alloc_bh, start_blk,
num_clusters);
if (status < 0) {
+ ocfs2_commit_trans(osb, handle);
mlog_errno(status);
goto bail;
}
}
/*
- * Zero the area past i_size but still within an allocated
- * cluster. This avoids exposing nonzero data on subsequent file
- * extends.
+ * Zero partial cluster for a hole punch or truncate. This avoids exposing
+ * nonzero data on subsequent file extends.
*
* We need to call this before i_size is updated on the inode because
* otherwise block_write_full_page() will skip writeout of pages past
- * i_size. The new_i_size parameter is passed for this reason.
+ * i_size.
*/
int ocfs2_zero_range_for_truncate(struct inode *inode, handle_t *handle,
u64 range_start, u64 range_end)
if (!ocfs2_sparse_alloc(OCFS2_SB(sb)))
return 0;
+ /*
+ * Avoid zeroing pages fully beyond current i_size. It is pointless as
+ * underlying blocks of those pages should be already zeroed out and
+ * page writeback will skip them anyway.
+ */
+ range_end = min_t(u64, range_end, i_size_read(inode));
+ if (range_start >= range_end)
+ return 0;
+
pages = kcalloc(ocfs2_pages_per_cluster(sb),
sizeof(struct page *), GFP_NOFS);
if (pages == NULL) {
goto out;
}
- if (range_start == range_end)
- goto out;
-
ret = ocfs2_extent_map_get_blocks(inode,
range_start >> sb->s_blocksize_bits,
&phys, NULL, &ext_flags);
continue;
}
retry:
- ret = -EINVAL;
mlog(0, "attempting to send begin reco msg to %d\n",
nodenum);
ret = o2net_send_message(DLM_BEGIN_RECO_MSG, dlm->key,
* greater than page size, so we have to truncate them
* anyway.
*/
- unmap_mapping_range(inode->i_mapping, new_i_size + PAGE_SIZE - 1, 0, 1);
- truncate_inode_pages(inode->i_mapping, new_i_size);
if (OCFS2_I(inode)->ip_dyn_features & OCFS2_INLINE_DATA_FL) {
+ unmap_mapping_range(inode->i_mapping,
+ new_i_size + PAGE_SIZE - 1, 0, 1);
+ truncate_inode_pages(inode->i_mapping, new_i_size);
status = ocfs2_truncate_inline(inode, di_bh, new_i_size,
i_size_read(inode), 1);
if (status)
goto bail_unlock_sem;
}
+ unmap_mapping_range(inode->i_mapping, new_i_size + PAGE_SIZE - 1, 0, 1);
+ truncate_inode_pages(inode->i_mapping, new_i_size);
+
status = ocfs2_commit_truncate(osb, inode, di_bh);
if (status < 0) {
mlog_errno(status);
struct inode *inode = NULL;
struct super_block *sb = osb->sb;
struct ocfs2_find_inode_args args;
- journal_t *journal = OCFS2_SB(sb)->journal->j_journal;
trace_ocfs2_iget_begin((unsigned long long)blkno, flags,
sysfile_type);
* part of the transaction - the inode could have been reclaimed and
* now it is reread from disk.
*/
- if (journal) {
+ if (osb->journal) {
transaction_t *transaction;
tid_t tid;
struct ocfs2_inode_info *oi = OCFS2_I(inode);
+ journal_t *journal = osb->journal->j_journal;
read_lock(&journal->j_state_lock);
if (journal->j_running_transaction)
write_unlock(&journal->j_state_lock);
}
-int ocfs2_journal_init(struct ocfs2_journal *journal, int *dirty)
+int ocfs2_journal_init(struct ocfs2_super *osb, int *dirty)
{
int status = -1;
struct inode *inode = NULL; /* the journal inode */
journal_t *j_journal = NULL;
+ struct ocfs2_journal *journal = NULL;
struct ocfs2_dinode *di = NULL;
struct buffer_head *bh = NULL;
- struct ocfs2_super *osb;
int inode_lock = 0;
- BUG_ON(!journal);
+ /* initialize our journal structure */
+ journal = kzalloc(sizeof(struct ocfs2_journal), GFP_KERNEL);
+ if (!journal) {
+ mlog(ML_ERROR, "unable to alloc journal\n");
+ status = -ENOMEM;
+ goto done;
+ }
+ osb->journal = journal;
+ journal->j_osb = osb;
- osb = journal->j_osb;
+ atomic_set(&journal->j_num_trans, 0);
+ init_rwsem(&journal->j_trans_barrier);
+ init_waitqueue_head(&journal->j_checkpointed);
+ spin_lock_init(&journal->j_lock);
+ journal->j_trans_id = 1UL;
+ INIT_LIST_HEAD(&journal->j_la_cleanups);
+ INIT_WORK(&journal->j_recovery_work, ocfs2_complete_recovery);
+ journal->j_state = OCFS2_JOURNAL_FREE;
/* already have the inode for our journal */
inode = ocfs2_get_system_file_inode(osb, JOURNAL_SYSTEM_INODE,
journal->j_state = OCFS2_JOURNAL_FREE;
-// up_write(&journal->j_trans_barrier);
done:
iput(inode);
+ kfree(journal);
+ osb->journal = NULL;
}
static void ocfs2_clear_journal_error(struct super_block *sb,
if (quota_enabled)
kfree(rm_quota);
- /* no one is callint kthread_stop() for us so the kthread() api
- * requires that we call do_exit(). And it isn't exported, but
- * complete_and_exit() seems to be a minimal wrapper around it. */
- complete_and_exit(NULL, status);
+ return status;
}
void ocfs2_recovery_thread(struct ocfs2_super *osb, int node_num)
* ocfs2_start_checkpoint - Kick the commit thread to do a checkpoint.
*/
void ocfs2_set_journal_params(struct ocfs2_super *osb);
-int ocfs2_journal_init(struct ocfs2_journal *journal,
- int *dirty);
+int ocfs2_journal_init(struct ocfs2_super *osb, int *dirty);
void ocfs2_journal_shutdown(struct ocfs2_super *osb);
int ocfs2_journal_wipe(struct ocfs2_journal *journal,
int full);
/* This will disable recovery and flush any recovery work. */
ocfs2_recovery_exit(osb);
- ocfs2_journal_shutdown(osb);
-
ocfs2_sync_blockdev(sb);
ocfs2_purge_refcount_trees(osb);
ocfs2_release_system_inodes(osb);
+ ocfs2_journal_shutdown(osb);
+
/*
* If we're dismounting due to mount error, mount.ocfs2 will clean
* up heartbeat. If we're a local mount, there is no heartbeat.
int i, cbits, bbits;
struct ocfs2_dinode *di = (struct ocfs2_dinode *)bh->b_data;
struct inode *inode = NULL;
- struct ocfs2_journal *journal;
struct ocfs2_super *osb;
u64 total_blocks;
get_random_bytes(&osb->s_next_generation, sizeof(u32));
- /* FIXME
- * This should be done in ocfs2_journal_init(), but unknown
- * ordering issues will cause the filesystem to crash.
- * If anyone wants to figure out what part of the code
- * refers to osb->journal before ocfs2_journal_init() is run,
- * be my guest.
- */
- /* initialize our journal structure */
-
- journal = kzalloc(sizeof(struct ocfs2_journal), GFP_KERNEL);
- if (!journal) {
- mlog(ML_ERROR, "unable to alloc journal\n");
- status = -ENOMEM;
- goto bail;
- }
- osb->journal = journal;
- journal->j_osb = osb;
-
- atomic_set(&journal->j_num_trans, 0);
- init_rwsem(&journal->j_trans_barrier);
- init_waitqueue_head(&journal->j_checkpointed);
- spin_lock_init(&journal->j_lock);
- journal->j_trans_id = (unsigned long) 1;
- INIT_LIST_HEAD(&journal->j_la_cleanups);
- INIT_WORK(&journal->j_recovery_work, ocfs2_complete_recovery);
- journal->j_state = OCFS2_JOURNAL_FREE;
-
INIT_WORK(&osb->dquot_drop_work, ocfs2_drop_dquot_refs);
init_llist_head(&osb->dquot_drop_list);
* ourselves. */
/* Init our journal object. */
- status = ocfs2_journal_init(osb->journal, &dirty);
+ status = ocfs2_journal_init(osb, &dirty);
if (status < 0) {
mlog(ML_ERROR, "Could not initialize journal!\n");
goto finally;
kfree(osb->osb_orphan_wipes);
kfree(osb->slot_recovery_generations);
- /* FIXME
- * This belongs in journal shutdown, but because we have to
- * allocate osb->journal at the start of ocfs2_initialize_osb(),
- * we free it here.
- */
- kfree(osb->journal);
kfree(osb->local_alloc_copy);
kfree(osb->uuid_str);
kfree(osb->vol_label);
* of THPs into the page cache will fail.
*/
smp_mb();
- if (filemap_nr_thps(inode->i_mapping))
- truncate_pagecache(inode, 0);
+ if (filemap_nr_thps(inode->i_mapping)) {
+ struct address_space *mapping = inode->i_mapping;
+
+ filemap_invalidate_lock(inode->i_mapping);
+ /*
+ * unmap_mapping_range just need to be called once
+ * here, because the private pages is not need to be
+ * unmapped mapping (e.g. data segment of dynamic
+ * shared libraries here).
+ */
+ unmap_mapping_range(mapping, 0, 0, 0);
+ truncate_inode_pages(mapping, 0);
+ filemap_invalidate_unlock(inode->i_mapping);
+ }
}
return 0;
gossip_debug(GOSSIP_DCACHE_DEBUG, "%s: attempting lookup.\n", __func__);
new_op = op_alloc(ORANGEFS_VFS_OP_LOOKUP);
- if (!new_op)
+ if (!new_op) {
+ ret = -ENOMEM;
goto out_put_parent;
+ }
new_op->upcall.req.lookup.sym_follow = ORANGEFS_LOOKUP_LINK_NO_FOLLOW;
new_op->upcall.req.lookup.parent_refn = parent->refn;
const char *devname,
void *data)
{
- int ret = -EINVAL;
+ int ret;
struct super_block *sb = ERR_PTR(-EINVAL);
struct orangefs_kernel_op_s *new_op;
struct dentry *d = ERR_PTR(-EINVAL);
sb->s_fs_info = kzalloc(sizeof(struct orangefs_sb_info_s), GFP_KERNEL);
if (!ORANGEFS_SB(sb)) {
d = ERR_PTR(-ENOMEM);
- goto free_op;
+ goto free_sb_and_op;
}
ret = orangefs_fill_sb(sb,
int err;
err = ovl_real_fileattr_get(old, &oldfa);
- if (err)
- return err;
-
- err = ovl_real_fileattr_get(new, &newfa);
- if (err)
+ if (err) {
+ /* Ntfs-3g returns -EINVAL for "no fileattr support" */
+ if (err == -ENOTTY || err == -EINVAL)
+ return 0;
+ pr_warn("failed to retrieve lower fileattr (%pd2, err=%i)\n",
+ old, err);
return err;
+ }
/*
* We cannot set immutable and append-only flags on upper inode,
return err;
}
+ /* Don't bother copying flags if none are set */
+ if (!(oldfa.flags & OVL_COPY_FS_FLAGS_MASK))
+ return 0;
+
+ err = ovl_real_fileattr_get(new, &newfa);
+ if (err) {
+ pr_warn("failed to retrieve upper fileattr (%pd2, err=%i)\n",
+ new, err);
+ return err;
+ }
+
BUILD_BUG_ON(OVL_COPY_FS_FLAGS_MASK & ~FS_COMMON_FL);
newfa.flags &= ~OVL_COPY_FS_FLAGS_MASK;
newfa.flags |= (oldfa.flags & OVL_COPY_FS_FLAGS_MASK);
goto out;
}
-static int ovl_mkdir_real(struct inode *dir, struct dentry **newdentry,
- umode_t mode)
+int ovl_mkdir_real(struct inode *dir, struct dentry **newdentry, umode_t mode)
{
int err;
struct dentry *d, *dentry = *newdentry;
struct ovl_aio_req {
struct kiocb iocb;
+ refcount_t ref;
struct kiocb *orig_iocb;
struct fd fd;
};
return flags;
}
+static inline void ovl_aio_put(struct ovl_aio_req *aio_req)
+{
+ if (refcount_dec_and_test(&aio_req->ref)) {
+ fdput(aio_req->fd);
+ kmem_cache_free(ovl_aio_request_cachep, aio_req);
+ }
+}
+
static void ovl_aio_cleanup_handler(struct ovl_aio_req *aio_req)
{
struct kiocb *iocb = &aio_req->iocb;
}
orig_iocb->ki_pos = iocb->ki_pos;
- fdput(aio_req->fd);
- kmem_cache_free(ovl_aio_request_cachep, aio_req);
+ ovl_aio_put(aio_req);
}
static void ovl_aio_rw_complete(struct kiocb *iocb, long res)
aio_req->orig_iocb = iocb;
kiocb_clone(&aio_req->iocb, iocb, real.file);
aio_req->iocb.ki_complete = ovl_aio_rw_complete;
+ refcount_set(&aio_req->ref, 2);
ret = vfs_iocb_iter_read(real.file, &aio_req->iocb, iter);
+ ovl_aio_put(aio_req);
if (ret != -EIOCBQUEUED)
ovl_aio_cleanup_handler(aio_req);
}
kiocb_clone(&aio_req->iocb, iocb, real.file);
aio_req->iocb.ki_flags = ifl;
aio_req->iocb.ki_complete = ovl_aio_rw_complete;
+ refcount_set(&aio_req->ref, 2);
ret = vfs_iocb_iter_write(real.file, &aio_req->iocb, iter);
+ ovl_aio_put(aio_req);
if (ret != -EIOCBQUEUED)
ovl_aio_cleanup_handler(aio_req);
}
if (err)
return err;
- return vfs_fileattr_get(realpath->dentry, fa);
+ err = vfs_fileattr_get(realpath->dentry, fa);
+ if (err == -ENOIOCTLCMD)
+ err = -ENOTTY;
+ return err;
}
int ovl_fileattr_get(struct dentry *dentry, struct fileattr *fa)
#define OVL_CATTR(m) (&(struct ovl_cattr) { .mode = (m) })
+int ovl_mkdir_real(struct inode *dir, struct dentry **newdentry, umode_t mode);
struct dentry *ovl_create_real(struct inode *dir, struct dentry *newdentry,
struct ovl_cattr *attr);
int ovl_cleanup(struct inode *dir, struct dentry *dentry);
goto retry;
}
- work = ovl_create_real(dir, work, OVL_CATTR(attr.ia_mode));
- err = PTR_ERR(work);
- if (IS_ERR(work))
- goto out_err;
+ err = ovl_mkdir_real(dir, &work, attr.ia_mode);
+ if (err)
+ goto out_dput;
+
+ /* Weird filesystem returning with hashed negative (kernfs)? */
+ err = -EINVAL;
+ if (d_really_is_negative(work))
+ goto out_dput;
/*
* Try to remove POSIX ACL xattrs from workdir. We are good if:
* to just call ->get_acl to fetch the ACL ourself. (This is going to
* be an unlikely race.)
*/
- if (cmpxchg(p, ACL_NOT_CACHED, sentinel) != ACL_NOT_CACHED)
- /* fall through */ ;
+ cmpxchg(p, ACL_NOT_CACHED, sentinel);
/*
* Normally, the ACL returned by ->get_acl will be cached.
{
struct inode *inode;
struct task_struct *task;
+ int ret = 0;
- if (flags & LOOKUP_RCU)
- return -ECHILD;
-
- inode = d_inode(dentry);
- task = get_proc_task(inode);
+ rcu_read_lock();
+ inode = d_inode_rcu(dentry);
+ if (!inode)
+ goto out;
+ task = pid_task(proc_pid(inode), PIDTYPE_PID);
if (task) {
pid_update_inode(task, inode);
- put_task_struct(task);
- return 1;
+ ret = 1;
}
- return 0;
+out:
+ rcu_read_unlock();
+ return ret;
}
static inline bool proc_inode_is_dead(struct inode *inode)
task = next_tid(task), ctx->pos++) {
char name[10 + 1];
unsigned int len;
+
tid = task_pid_nr_ns(task, ns);
+ if (!tid)
+ continue; /* The task has just exited. */
len = snprintf(name, sizeof(name), "%u", tid);
if (!proc_fill_cache(file, ctx, name, len,
proc_task_instantiate, task, NULL)) {
u64 pss_shmem;
u64 pss_locked;
u64 swap_pss;
- bool check_shmem_swap;
};
static void smaps_page_accumulate(struct mem_size_stats *mss,
__always_unused int depth, struct mm_walk *walk)
{
struct mem_size_stats *mss = walk->private;
+ struct vm_area_struct *vma = walk->vma;
- mss->swap += shmem_partial_swap_usage(
- walk->vma->vm_file->f_mapping, addr, end);
+ mss->swap += shmem_partial_swap_usage(walk->vma->vm_file->f_mapping,
+ linear_page_index(vma, addr),
+ linear_page_index(vma, end));
return 0;
}
#define smaps_pte_hole NULL
#endif /* CONFIG_SHMEM */
+static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)
+{
+#ifdef CONFIG_SHMEM
+ if (walk->ops->pte_hole) {
+ /* depth is not used */
+ smaps_pte_hole(addr, addr + PAGE_SIZE, 0, walk);
+ }
+#endif
+}
+
static void smaps_pte_entry(pte_t *pte, unsigned long addr,
struct mm_walk *walk)
{
}
} else if (is_pfn_swap_entry(swpent))
page = pfn_swap_entry_to_page(swpent);
- } else if (unlikely(IS_ENABLED(CONFIG_SHMEM) && mss->check_shmem_swap
- && pte_none(*pte))) {
- page = xa_load(&vma->vm_file->f_mapping->i_pages,
- linear_page_index(vma, addr));
- if (xa_is_value(page))
- mss->swap += PAGE_SIZE;
+ } else {
+ smaps_pte_hole_lookup(addr, walk);
return;
}
return;
#ifdef CONFIG_SHMEM
- /* In case of smaps_rollup, reset the value from previous vma */
- mss->check_shmem_swap = false;
if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {
/*
* For shared or readonly shmem mappings we know that all
!(vma->vm_flags & VM_WRITE))) {
mss->swap += shmem_swapped;
} else {
- mss->check_shmem_swap = true;
ops = &smaps_shmem_walk_ops;
}
}
/* Device Dump Size */
static size_t vmcoredd_orig_sz;
-/*
- * Returns > 0 for RAM pages, 0 for non-RAM pages, < 0 on error
- * The called function has to take care of module refcounting.
- */
-static int (*oldmem_pfn_is_ram)(unsigned long pfn);
-
-int register_oldmem_pfn_is_ram(int (*fn)(unsigned long pfn))
+static DECLARE_RWSEM(vmcore_cb_rwsem);
+/* List of registered vmcore callbacks. */
+static LIST_HEAD(vmcore_cb_list);
+/* Whether we had a surprise unregistration of a callback. */
+static bool vmcore_cb_unstable;
+/* Whether the vmcore has been opened once. */
+static bool vmcore_opened;
+
+void register_vmcore_cb(struct vmcore_cb *cb)
{
- if (oldmem_pfn_is_ram)
- return -EBUSY;
- oldmem_pfn_is_ram = fn;
- return 0;
+ down_write(&vmcore_cb_rwsem);
+ INIT_LIST_HEAD(&cb->next);
+ list_add_tail(&cb->next, &vmcore_cb_list);
+ /*
+ * Registering a vmcore callback after the vmcore was opened is
+ * very unusual (e.g., manual driver loading).
+ */
+ if (vmcore_opened)
+ pr_warn_once("Unexpected vmcore callback registration\n");
+ up_write(&vmcore_cb_rwsem);
}
-EXPORT_SYMBOL_GPL(register_oldmem_pfn_is_ram);
+EXPORT_SYMBOL_GPL(register_vmcore_cb);
-void unregister_oldmem_pfn_is_ram(void)
+void unregister_vmcore_cb(struct vmcore_cb *cb)
{
- oldmem_pfn_is_ram = NULL;
- wmb();
+ down_write(&vmcore_cb_rwsem);
+ list_del(&cb->next);
+ /*
+ * Unregistering a vmcore callback after the vmcore was opened is
+ * very unusual (e.g., forced driver removal), but we cannot stop
+ * unregistering.
+ */
+ if (vmcore_opened) {
+ pr_warn_once("Unexpected vmcore callback unregistration\n");
+ vmcore_cb_unstable = true;
+ }
+ up_write(&vmcore_cb_rwsem);
}
-EXPORT_SYMBOL_GPL(unregister_oldmem_pfn_is_ram);
+EXPORT_SYMBOL_GPL(unregister_vmcore_cb);
-static int pfn_is_ram(unsigned long pfn)
+static bool pfn_is_ram(unsigned long pfn)
{
- int (*fn)(unsigned long pfn);
- /* pfn is ram unless fn() checks pagetype */
- int ret = 1;
+ struct vmcore_cb *cb;
+ bool ret = true;
- /*
- * Ask hypervisor if the pfn is really ram.
- * A ballooned page contains no data and reading from such a page
- * will cause high load in the hypervisor.
- */
- fn = oldmem_pfn_is_ram;
- if (fn)
- ret = fn(pfn);
+ lockdep_assert_held_read(&vmcore_cb_rwsem);
+ if (unlikely(vmcore_cb_unstable))
+ return false;
+
+ list_for_each_entry(cb, &vmcore_cb_list, next) {
+ if (unlikely(!cb->pfn_is_ram))
+ continue;
+ ret = cb->pfn_is_ram(cb, pfn);
+ if (!ret)
+ break;
+ }
return ret;
}
+static int open_vmcore(struct inode *inode, struct file *file)
+{
+ down_read(&vmcore_cb_rwsem);
+ vmcore_opened = true;
+ up_read(&vmcore_cb_rwsem);
+
+ return 0;
+}
+
/* Reads a page from the oldmem device from given offset. */
ssize_t read_from_oldmem(char *buf, size_t count,
u64 *ppos, int userbuf,
offset = (unsigned long)(*ppos % PAGE_SIZE);
pfn = (unsigned long)(*ppos / PAGE_SIZE);
+ down_read(&vmcore_cb_rwsem);
do {
if (count > (PAGE_SIZE - offset))
nr_bytes = PAGE_SIZE - offset;
nr_bytes = count;
/* If pfn is not ram, return zeros for sparse dump files */
- if (pfn_is_ram(pfn) == 0)
- memset(buf, 0, nr_bytes);
- else {
+ if (!pfn_is_ram(pfn)) {
+ tmp = 0;
+ if (!userbuf)
+ memset(buf, 0, nr_bytes);
+ else if (clear_user(buf, nr_bytes))
+ tmp = -EFAULT;
+ } else {
if (encrypted)
tmp = copy_oldmem_page_encrypted(pfn, buf,
nr_bytes,
else
tmp = copy_oldmem_page(pfn, buf, nr_bytes,
offset, userbuf);
-
- if (tmp < 0)
- return tmp;
}
+ if (tmp < 0) {
+ up_read(&vmcore_cb_rwsem);
+ return tmp;
+ }
+
*ppos += nr_bytes;
count -= nr_bytes;
buf += nr_bytes;
offset = 0;
} while (count);
+ up_read(&vmcore_cb_rwsem);
return read;
}
unsigned long from, unsigned long pfn,
unsigned long size, pgprot_t prot)
{
+ int ret;
+
/*
* Check if oldmem_pfn_is_ram was registered to avoid
* looping over all pages without a reason.
*/
- if (oldmem_pfn_is_ram)
- return remap_oldmem_pfn_checked(vma, from, pfn, size, prot);
+ down_read(&vmcore_cb_rwsem);
+ if (!list_empty(&vmcore_cb_list) || vmcore_cb_unstable)
+ ret = remap_oldmem_pfn_checked(vma, from, pfn, size, prot);
else
- return remap_oldmem_pfn_range(vma, from, pfn, size, prot);
+ ret = remap_oldmem_pfn_range(vma, from, pfn, size, prot);
+ up_read(&vmcore_cb_rwsem);
+ return ret;
}
static int mmap_vmcore(struct file *file, struct vm_area_struct *vma)
#endif
static const struct proc_ops vmcore_proc_ops = {
+ .proc_open = open_vmcore,
.proc_read = read_vmcore,
.proc_lseek = default_llseek,
.proc_mmap = mmap_vmcore,
tristate "Log panic/oops to a block device"
depends on PSTORE
depends on BLOCK
- depends on BROKEN
select PSTORE_ZONE
default n
help
if (ret)
kfree(best_effort_dev);
else
- pr_info("attached %s (%zu) (no dedicated panic_write!)\n",
+ pr_info("attached %s (%lu) (no dedicated panic_write!)\n",
blkdev, best_effort_dev->zone.total_size);
return ret;
#if IS_ENABLED(CONFIG_PSTORE_ZSTD_COMPRESS)
static int zbufsize_zstd(size_t size)
{
- return ZSTD_compressBound(size);
+ return zstd_compress_bound(size);
}
#endif
quota_error(dquot->dq_sb, "Quota structure has offset to "
"other block (%u) than it should (%u)", blk,
(uint)(dquot->dq_off >> info->dqi_blocksize_bits));
+ ret = -EIO;
goto out_buf;
}
ret = read_blk(info, blk, buf);
goto out_buf;
}
newblk = le32_to_cpu(ref[get_index(info, dquot->dq_id, depth)]);
+ if (newblk < QT_TREEOFF || newblk >= info->dqi_blocks) {
+ quota_error(dquot->dq_sb, "Getting block too big (%u >= %u)",
+ newblk, info->dqi_blocks);
+ ret = -EUCLEAN;
+ goto out_buf;
+ }
+
if (depth == info->dqi_qtree_depth - 1) {
ret = free_dqentry(info, dquot, newblk);
newblk = 0;
blk = le32_to_cpu(ref[get_index(info, dquot->dq_id, depth)]);
if (!blk) /* No reference? */
goto out_buf;
+ if (blk < QT_TREEOFF || blk >= info->dqi_blocks) {
+ quota_error(dquot->dq_sb, "Getting block too big (%u >= %u)",
+ blk, info->dqi_blocks);
+ ret = -EUCLEAN;
+ goto out_buf;
+ }
+
if (depth < info->dqi_qtree_depth - 1)
ret = find_tree_dqentry(info, dquot, blk, depth+1);
else
int opt;
opt = fs_parse(fc, ramfs_fs_parameters, param, &result);
- if (opt < 0) {
+ if (opt == -ENOPARAM) {
+ opt = vfs_parse_fs_param_source(fc, param);
+ if (opt != -ENOPARAM)
+ return opt;
/*
* We might like to report bad mount options here;
* but traditionally ramfs has ignored all mount options,
* and as it is used as a !CONFIG_SHMEM simple substitute
* for tmpfs, better continue to ignore other mount options.
*/
- if (opt == -ENOPARAM)
- opt = 0;
- return opt;
+ return 0;
}
+ if (opt < 0)
+ return opt;
switch (opt) {
case Opt_mode:
unsigned long safe_mask = 0;
unsigned int commit_max_age = (unsigned int)-1;
struct reiserfs_journal *journal = SB_JOURNAL(s);
- char *new_opts;
int err;
char *qf_names[REISERFS_MAXQUOTAS];
unsigned int qfmt = 0;
int i;
#endif
- new_opts = kstrdup(arg, GFP_KERNEL);
- if (arg && !new_opts)
- return -ENOMEM;
-
sync_filesystem(s);
reiserfs_write_lock(s);
out_err_unlock:
reiserfs_write_unlock(s);
out_err:
- kfree(new_opts);
return err;
}
}
EXPORT_SYMBOL(seq_escape_mem);
-/**
- * seq_escape - print string into buffer, escaping some characters
- * @m: target buffer
- * @s: string
- * @esc: set of characters that need escaping
- *
- * Puts string into buffer, replacing each occurrence of character from
- * @esc with usual octal escape.
- * Use seq_has_overflowed() to check for errors.
- */
-void seq_escape(struct seq_file *m, const char *s, const char *esc)
-{
- seq_escape_str(m, s, ESCAPE_OCTAL, esc);
-}
-EXPORT_SYMBOL(seq_escape);
-
void seq_vprintf(struct seq_file *m, const char *f, va_list args)
{
int len;
--- /dev/null
+/* SPDX-License-Identifier: LGPL-2.1 */
+#ifndef _COMMON_SMB2PDU_H
+#define _COMMON_SMB2PDU_H
+
+/*
+ * Note that, due to trying to use names similar to the protocol specifications,
+ * there are many mixed case field names in the structures below. Although
+ * this does not match typical Linux kernel style, it is necessary to be
+ * able to match against the protocol specfication.
+ *
+ * SMB2 commands
+ * Some commands have minimal (wct=0,bcc=0), or uninteresting, responses
+ * (ie no useful data other than the SMB error code itself) and are marked such.
+ * Knowing this helps avoid response buffer allocations and copy in some cases.
+ */
+
+/* List of commands in host endian */
+#define SMB2_NEGOTIATE_HE 0x0000
+#define SMB2_SESSION_SETUP_HE 0x0001
+#define SMB2_LOGOFF_HE 0x0002 /* trivial request/resp */
+#define SMB2_TREE_CONNECT_HE 0x0003
+#define SMB2_TREE_DISCONNECT_HE 0x0004 /* trivial req/resp */
+#define SMB2_CREATE_HE 0x0005
+#define SMB2_CLOSE_HE 0x0006
+#define SMB2_FLUSH_HE 0x0007 /* trivial resp */
+#define SMB2_READ_HE 0x0008
+#define SMB2_WRITE_HE 0x0009
+#define SMB2_LOCK_HE 0x000A
+#define SMB2_IOCTL_HE 0x000B
+#define SMB2_CANCEL_HE 0x000C
+#define SMB2_ECHO_HE 0x000D
+#define SMB2_QUERY_DIRECTORY_HE 0x000E
+#define SMB2_CHANGE_NOTIFY_HE 0x000F
+#define SMB2_QUERY_INFO_HE 0x0010
+#define SMB2_SET_INFO_HE 0x0011
+#define SMB2_OPLOCK_BREAK_HE 0x0012
+
+/* The same list in little endian */
+#define SMB2_NEGOTIATE cpu_to_le16(SMB2_NEGOTIATE_HE)
+#define SMB2_SESSION_SETUP cpu_to_le16(SMB2_SESSION_SETUP_HE)
+#define SMB2_LOGOFF cpu_to_le16(SMB2_LOGOFF_HE)
+#define SMB2_TREE_CONNECT cpu_to_le16(SMB2_TREE_CONNECT_HE)
+#define SMB2_TREE_DISCONNECT cpu_to_le16(SMB2_TREE_DISCONNECT_HE)
+#define SMB2_CREATE cpu_to_le16(SMB2_CREATE_HE)
+#define SMB2_CLOSE cpu_to_le16(SMB2_CLOSE_HE)
+#define SMB2_FLUSH cpu_to_le16(SMB2_FLUSH_HE)
+#define SMB2_READ cpu_to_le16(SMB2_READ_HE)
+#define SMB2_WRITE cpu_to_le16(SMB2_WRITE_HE)
+#define SMB2_LOCK cpu_to_le16(SMB2_LOCK_HE)
+#define SMB2_IOCTL cpu_to_le16(SMB2_IOCTL_HE)
+#define SMB2_CANCEL cpu_to_le16(SMB2_CANCEL_HE)
+#define SMB2_ECHO cpu_to_le16(SMB2_ECHO_HE)
+#define SMB2_QUERY_DIRECTORY cpu_to_le16(SMB2_QUERY_DIRECTORY_HE)
+#define SMB2_CHANGE_NOTIFY cpu_to_le16(SMB2_CHANGE_NOTIFY_HE)
+#define SMB2_QUERY_INFO cpu_to_le16(SMB2_QUERY_INFO_HE)
+#define SMB2_SET_INFO cpu_to_le16(SMB2_SET_INFO_HE)
+#define SMB2_OPLOCK_BREAK cpu_to_le16(SMB2_OPLOCK_BREAK_HE)
+
+#define SMB2_INTERNAL_CMD cpu_to_le16(0xFFFF)
+
+#define NUMBER_OF_SMB2_COMMANDS 0x0013
+
+/*
+ * SMB2 Header Definition
+ *
+ * "MBZ" : Must be Zero
+ * "BB" : BugBug, Something to check/review/analyze later
+ * "PDU" : "Protocol Data Unit" (ie a network "frame")
+ *
+ */
+
+#define __SMB2_HEADER_STRUCTURE_SIZE 64
+#define SMB2_HEADER_STRUCTURE_SIZE \
+ cpu_to_le16(__SMB2_HEADER_STRUCTURE_SIZE)
+
+#define SMB2_PROTO_NUMBER cpu_to_le32(0x424d53fe)
+#define SMB2_TRANSFORM_PROTO_NUM cpu_to_le32(0x424d53fd)
+#define SMB2_COMPRESSION_TRANSFORM_ID cpu_to_le32(0x424d53fc)
+
+/*
+ * SMB2 flag definitions
+ */
+#define SMB2_FLAGS_SERVER_TO_REDIR cpu_to_le32(0x00000001)
+#define SMB2_FLAGS_ASYNC_COMMAND cpu_to_le32(0x00000002)
+#define SMB2_FLAGS_RELATED_OPERATIONS cpu_to_le32(0x00000004)
+#define SMB2_FLAGS_SIGNED cpu_to_le32(0x00000008)
+#define SMB2_FLAGS_PRIORITY_MASK cpu_to_le32(0x00000070) /* SMB3.1.1 */
+#define SMB2_FLAGS_DFS_OPERATIONS cpu_to_le32(0x10000000)
+#define SMB2_FLAGS_REPLAY_OPERATION cpu_to_le32(0x20000000) /* SMB3 & up */
+
+/* See MS-SMB2 section 2.2.1 */
+struct smb2_hdr {
+ __le32 ProtocolId; /* 0xFE 'S' 'M' 'B' */
+ __le16 StructureSize; /* 64 */
+ __le16 CreditCharge; /* MBZ */
+ __le32 Status; /* Error from server */
+ __le16 Command;
+ __le16 CreditRequest; /* CreditResponse */
+ __le32 Flags;
+ __le32 NextCommand;
+ __le64 MessageId;
+ union {
+ struct {
+ __le32 ProcessId;
+ __le32 TreeId;
+ } __packed SyncId;
+ __le64 AsyncId;
+ } __packed Id;
+ __le64 SessionId;
+ __u8 Signature[16];
+} __packed;
+
+struct smb2_pdu {
+ struct smb2_hdr hdr;
+ __le16 StructureSize2; /* size of wct area (varies, request specific) */
+} __packed;
+
+#define SMB3_AES_CCM_NONCE 11
+#define SMB3_AES_GCM_NONCE 12
+
+/* Transform flags (for 3.0 dialect this flag indicates CCM */
+#define TRANSFORM_FLAG_ENCRYPTED 0x0001
+struct smb2_transform_hdr {
+ __le32 ProtocolId; /* 0xFD 'S' 'M' 'B' */
+ __u8 Signature[16];
+ __u8 Nonce[16];
+ __le32 OriginalMessageSize;
+ __u16 Reserved1;
+ __le16 Flags; /* EncryptionAlgorithm for 3.0, enc enabled for 3.1.1 */
+ __le64 SessionId;
+} __packed;
+
+
+/* See MS-SMB2 2.2.42 */
+struct smb2_compression_transform_hdr_unchained {
+ __le32 ProtocolId; /* 0xFC 'S' 'M' 'B' */
+ __le32 OriginalCompressedSegmentSize;
+ __le16 CompressionAlgorithm;
+ __le16 Flags;
+ __le16 Length; /* if chained it is length, else offset */
+} __packed;
+
+/* See MS-SMB2 2.2.42.1 */
+#define SMB2_COMPRESSION_FLAG_NONE 0x0000
+#define SMB2_COMPRESSION_FLAG_CHAINED 0x0001
+
+struct compression_payload_header {
+ __le16 CompressionAlgorithm;
+ __le16 Flags;
+ __le32 Length; /* length of compressed playload including field below if present */
+ /* __le32 OriginalPayloadSize; */ /* optional, present when LZNT1, LZ77, LZ77+Huffman */
+} __packed;
+
+/* See MS-SMB2 2.2.42.2 */
+struct smb2_compression_transform_hdr_chained {
+ __le32 ProtocolId; /* 0xFC 'S' 'M' 'B' */
+ __le32 OriginalCompressedSegmentSize;
+ /* struct compression_payload_header[] */
+} __packed;
+
+/* See MS-SMB2 2.2.42.2.2 */
+struct compression_pattern_payload_v1 {
+ __le16 Pattern;
+ __le16 Reserved1;
+ __le16 Reserved2;
+ __le32 Repetitions;
+} __packed;
+
+/* See MS-SMB2 section 2.2.9.2 */
+/* Context Types */
+#define SMB2_RESERVED_TREE_CONNECT_CONTEXT_ID 0x0000
+#define SMB2_REMOTED_IDENTITY_TREE_CONNECT_CONTEXT_ID cpu_to_le16(0x0001)
+
+struct tree_connect_contexts {
+ __le16 ContextType;
+ __le16 DataLength;
+ __le32 Reserved;
+ __u8 Data[];
+} __packed;
+
+/* Remoted identity tree connect context structures - see MS-SMB2 2.2.9.2.1 */
+struct smb3_blob_data {
+ __le16 BlobSize;
+ __u8 BlobData[];
+} __packed;
+
+/* Valid values for Attr */
+#define SE_GROUP_MANDATORY 0x00000001
+#define SE_GROUP_ENABLED_BY_DEFAULT 0x00000002
+#define SE_GROUP_ENABLED 0x00000004
+#define SE_GROUP_OWNER 0x00000008
+#define SE_GROUP_USE_FOR_DENY_ONLY 0x00000010
+#define SE_GROUP_INTEGRITY 0x00000020
+#define SE_GROUP_INTEGRITY_ENABLED 0x00000040
+#define SE_GROUP_RESOURCE 0x20000000
+#define SE_GROUP_LOGON_ID 0xC0000000
+
+/* struct sid_attr_data is SidData array in BlobData format then le32 Attr */
+
+struct sid_array_data {
+ __le16 SidAttrCount;
+ /* SidAttrList - array of sid_attr_data structs */
+} __packed;
+
+struct luid_attr_data {
+
+} __packed;
+
+/*
+ * struct privilege_data is the same as BLOB_DATA - see MS-SMB2 2.2.9.2.1.5
+ * but with size of LUID_ATTR_DATA struct and BlobData set to LUID_ATTR DATA
+ */
+
+struct privilege_array_data {
+ __le16 PrivilegeCount;
+ /* array of privilege_data structs */
+} __packed;
+
+struct remoted_identity_tcon_context {
+ __le16 TicketType; /* must be 0x0001 */
+ __le16 TicketSize; /* total size of this struct */
+ __le16 User; /* offset to SID_ATTR_DATA struct with user info */
+ __le16 UserName; /* offset to null terminated Unicode username string */
+ __le16 Domain; /* offset to null terminated Unicode domain name */
+ __le16 Groups; /* offset to SID_ARRAY_DATA struct with group info */
+ __le16 RestrictedGroups; /* similar to above */
+ __le16 Privileges; /* offset to PRIVILEGE_ARRAY_DATA struct */
+ __le16 PrimaryGroup; /* offset to SID_ARRAY_DATA struct */
+ __le16 Owner; /* offset to BLOB_DATA struct */
+ __le16 DefaultDacl; /* offset to BLOB_DATA struct */
+ __le16 DeviceGroups; /* offset to SID_ARRAY_DATA struct */
+ __le16 UserClaims; /* offset to BLOB_DATA struct */
+ __le16 DeviceClaims; /* offset to BLOB_DATA struct */
+ __u8 TicketInfo[]; /* variable length buf - remoted identity data */
+} __packed;
+
+struct smb2_tree_connect_req_extension {
+ __le32 TreeConnectContextOffset;
+ __le16 TreeConnectContextCount;
+ __u8 Reserved[10];
+ __u8 PathName[]; /* variable sized array */
+ /* followed by array of TreeConnectContexts */
+} __packed;
+
+/* Flags/Reserved for SMB3.1.1 */
+#define SMB2_TREE_CONNECT_FLAG_CLUSTER_RECONNECT cpu_to_le16(0x0001)
+#define SMB2_TREE_CONNECT_FLAG_REDIRECT_TO_OWNER cpu_to_le16(0x0002)
+#define SMB2_TREE_CONNECT_FLAG_EXTENSION_PRESENT cpu_to_le16(0x0004)
+
+struct smb2_tree_connect_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 9 */
+ __le16 Flags; /* Flags in SMB3.1.1 */
+ __le16 PathOffset;
+ __le16 PathLength;
+ __u8 Buffer[1]; /* variable length */
+} __packed;
+
+/* Possible ShareType values */
+#define SMB2_SHARE_TYPE_DISK 0x01
+#define SMB2_SHARE_TYPE_PIPE 0x02
+#define SMB2_SHARE_TYPE_PRINT 0x03
+
+/*
+ * Possible ShareFlags - exactly one and only one of the first 4 caching flags
+ * must be set (any of the remaining, SHI1005, flags may be set individually
+ * or in combination.
+ */
+#define SMB2_SHAREFLAG_MANUAL_CACHING 0x00000000
+#define SMB2_SHAREFLAG_AUTO_CACHING 0x00000010
+#define SMB2_SHAREFLAG_VDO_CACHING 0x00000020
+#define SMB2_SHAREFLAG_NO_CACHING 0x00000030
+#define SHI1005_FLAGS_DFS 0x00000001
+#define SHI1005_FLAGS_DFS_ROOT 0x00000002
+#define SHI1005_FLAGS_RESTRICT_EXCLUSIVE_OPENS 0x00000100
+#define SHI1005_FLAGS_FORCE_SHARED_DELETE 0x00000200
+#define SHI1005_FLAGS_ALLOW_NAMESPACE_CACHING 0x00000400
+#define SHI1005_FLAGS_ACCESS_BASED_DIRECTORY_ENUM 0x00000800
+#define SHI1005_FLAGS_FORCE_LEVELII_OPLOCK 0x00001000
+#define SHI1005_FLAGS_ENABLE_HASH_V1 0x00002000
+#define SHI1005_FLAGS_ENABLE_HASH_V2 0x00004000
+#define SHI1005_FLAGS_ENCRYPT_DATA 0x00008000
+#define SMB2_SHAREFLAG_IDENTITY_REMOTING 0x00040000 /* 3.1.1 */
+#define SMB2_SHAREFLAG_COMPRESS_DATA 0x00100000 /* 3.1.1 */
+#define SHI1005_FLAGS_ALL 0x0014FF33
+
+/* Possible share capabilities */
+#define SMB2_SHARE_CAP_DFS cpu_to_le32(0x00000008) /* all dialects */
+#define SMB2_SHARE_CAP_CONTINUOUS_AVAILABILITY cpu_to_le32(0x00000010) /* 3.0 */
+#define SMB2_SHARE_CAP_SCALEOUT cpu_to_le32(0x00000020) /* 3.0 */
+#define SMB2_SHARE_CAP_CLUSTER cpu_to_le32(0x00000040) /* 3.0 */
+#define SMB2_SHARE_CAP_ASYMMETRIC cpu_to_le32(0x00000080) /* 3.02 */
+#define SMB2_SHARE_CAP_REDIRECT_TO_OWNER cpu_to_le32(0x00000100) /* 3.1.1 */
+
+struct smb2_tree_connect_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 16 */
+ __u8 ShareType; /* see below */
+ __u8 Reserved;
+ __le32 ShareFlags; /* see below */
+ __le32 Capabilities; /* see below */
+ __le32 MaximalAccess;
+} __packed;
+
+struct smb2_tree_disconnect_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 4 */
+ __le16 Reserved;
+} __packed;
+
+struct smb2_tree_disconnect_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 4 */
+ __le16 Reserved;
+} __packed;
+
+
+/*
+ * SMB2_NEGOTIATE_PROTOCOL See MS-SMB2 section 2.2.3
+ */
+/* SecurityMode flags */
+#define SMB2_NEGOTIATE_SIGNING_ENABLED 0x0001
+#define SMB2_NEGOTIATE_SIGNING_ENABLED_LE cpu_to_le16(0x0001)
+#define SMB2_NEGOTIATE_SIGNING_REQUIRED 0x0002
+#define SMB2_NEGOTIATE_SIGNING_REQUIRED_LE cpu_to_le16(0x0002)
+#define SMB2_SEC_MODE_FLAGS_ALL 0x0003
+
+/* Capabilities flags */
+#define SMB2_GLOBAL_CAP_DFS 0x00000001
+#define SMB2_GLOBAL_CAP_LEASING 0x00000002 /* Resp only New to SMB2.1 */
+#define SMB2_GLOBAL_CAP_LARGE_MTU 0X00000004 /* Resp only New to SMB2.1 */
+#define SMB2_GLOBAL_CAP_MULTI_CHANNEL 0x00000008 /* New to SMB3 */
+#define SMB2_GLOBAL_CAP_PERSISTENT_HANDLES 0x00000010 /* New to SMB3 */
+#define SMB2_GLOBAL_CAP_DIRECTORY_LEASING 0x00000020 /* New to SMB3 */
+#define SMB2_GLOBAL_CAP_ENCRYPTION 0x00000040 /* New to SMB3 */
+/* Internal types */
+#define SMB2_NT_FIND 0x00100000
+#define SMB2_LARGE_FILES 0x00200000
+
+#define SMB2_CLIENT_GUID_SIZE 16
+#define SMB2_CREATE_GUID_SIZE 16
+
+/* Dialects */
+#define SMB10_PROT_ID 0x0000 /* local only, not sent on wire w/CIFS negprot */
+#define SMB20_PROT_ID 0x0202
+#define SMB21_PROT_ID 0x0210
+#define SMB2X_PROT_ID 0x02FF
+#define SMB30_PROT_ID 0x0300
+#define SMB302_PROT_ID 0x0302
+#define SMB311_PROT_ID 0x0311
+#define BAD_PROT_ID 0xFFFF
+
+#define SMB311_SALT_SIZE 32
+/* Hash Algorithm Types */
+#define SMB2_PREAUTH_INTEGRITY_SHA512 cpu_to_le16(0x0001)
+#define SMB2_PREAUTH_HASH_SIZE 64
+
+/* Negotiate Contexts - ContextTypes. See MS-SMB2 section 2.2.3.1 for details */
+#define SMB2_PREAUTH_INTEGRITY_CAPABILITIES cpu_to_le16(1)
+#define SMB2_ENCRYPTION_CAPABILITIES cpu_to_le16(2)
+#define SMB2_COMPRESSION_CAPABILITIES cpu_to_le16(3)
+#define SMB2_NETNAME_NEGOTIATE_CONTEXT_ID cpu_to_le16(5)
+#define SMB2_TRANSPORT_CAPABILITIES cpu_to_le16(6)
+#define SMB2_RDMA_TRANSFORM_CAPABILITIES cpu_to_le16(7)
+#define SMB2_SIGNING_CAPABILITIES cpu_to_le16(8)
+#define SMB2_POSIX_EXTENSIONS_AVAILABLE cpu_to_le16(0x100)
+
+struct smb2_neg_context {
+ __le16 ContextType;
+ __le16 DataLength;
+ __le32 Reserved;
+ /* Followed by array of data. NOTE: some servers require padding to 8 byte boundary */
+} __packed;
+
+/*
+ * SaltLength that the server send can be zero, so the only three required
+ * fields (all __le16) end up six bytes total, so the minimum context data len
+ * in the response is six bytes which accounts for
+ *
+ * HashAlgorithmCount, SaltLength, and 1 HashAlgorithm.
+ */
+#define MIN_PREAUTH_CTXT_DATA_LEN 6
+
+struct smb2_preauth_neg_context {
+ __le16 ContextType; /* 1 */
+ __le16 DataLength;
+ __le32 Reserved;
+ __le16 HashAlgorithmCount; /* 1 */
+ __le16 SaltLength;
+ __le16 HashAlgorithms; /* HashAlgorithms[0] since only one defined */
+ __u8 Salt[SMB311_SALT_SIZE];
+} __packed;
+
+/* Encryption Algorithms Ciphers */
+#define SMB2_ENCRYPTION_AES128_CCM cpu_to_le16(0x0001)
+#define SMB2_ENCRYPTION_AES128_GCM cpu_to_le16(0x0002)
+#define SMB2_ENCRYPTION_AES256_CCM cpu_to_le16(0x0003)
+#define SMB2_ENCRYPTION_AES256_GCM cpu_to_le16(0x0004)
+
+/* Min encrypt context data is one cipher so 2 bytes + 2 byte count field */
+#define MIN_ENCRYPT_CTXT_DATA_LEN 4
+struct smb2_encryption_neg_context {
+ __le16 ContextType; /* 2 */
+ __le16 DataLength;
+ __le32 Reserved;
+ /* CipherCount usally 2, but can be 3 when AES256-GCM enabled */
+ __le16 CipherCount; /* AES128-GCM and AES128-CCM by default */
+ __le16 Ciphers[];
+} __packed;
+
+/* See MS-SMB2 2.2.3.1.3 */
+#define SMB3_COMPRESS_NONE cpu_to_le16(0x0000)
+#define SMB3_COMPRESS_LZNT1 cpu_to_le16(0x0001)
+#define SMB3_COMPRESS_LZ77 cpu_to_le16(0x0002)
+#define SMB3_COMPRESS_LZ77_HUFF cpu_to_le16(0x0003)
+/* Pattern scanning algorithm See MS-SMB2 3.1.4.4.1 */
+#define SMB3_COMPRESS_PATTERN cpu_to_le16(0x0004) /* Pattern_V1 */
+
+/* Compression Flags */
+#define SMB2_COMPRESSION_CAPABILITIES_FLAG_NONE cpu_to_le32(0x00000000)
+#define SMB2_COMPRESSION_CAPABILITIES_FLAG_CHAINED cpu_to_le32(0x00000001)
+
+struct smb2_compression_capabilities_context {
+ __le16 ContextType; /* 3 */
+ __le16 DataLength;
+ __le32 Reserved;
+ __le16 CompressionAlgorithmCount;
+ __le16 Padding;
+ __le32 Flags;
+ __le16 CompressionAlgorithms[3];
+ __u16 Pad; /* Some servers require pad to DataLen multiple of 8 */
+ /* Check if pad needed */
+} __packed;
+
+/*
+ * For smb2_netname_negotiate_context_id See MS-SMB2 2.2.3.1.4.
+ * Its struct simply contains NetName, an array of Unicode characters
+ */
+struct smb2_netname_neg_context {
+ __le16 ContextType; /* 5 */
+ __le16 DataLength;
+ __le32 Reserved;
+ __le16 NetName[]; /* hostname of target converted to UCS-2 */
+} __packed;
+
+/*
+ * For smb2_transport_capabilities context see MS-SMB2 2.2.3.1.5
+ * and 2.2.4.1.5
+ */
+
+/* Flags */
+#define SMB2_ACCEPT_TRANSFORM_LEVEL_SECURITY 0x00000001
+
+struct smb2_transport_capabilities_context {
+ __le16 ContextType; /* 6 */
+ __le16 DataLength;
+ __u32 Reserved;
+ __le32 Flags;
+ __u32 Pad;
+} __packed;
+
+/*
+ * For rdma transform capabilities context see MS-SMB2 2.2.3.1.6
+ * and 2.2.4.1.6
+ */
+
+/* RDMA Transform IDs */
+#define SMB2_RDMA_TRANSFORM_NONE 0x0000
+#define SMB2_RDMA_TRANSFORM_ENCRYPTION 0x0001
+#define SMB2_RDMA_TRANSFORM_SIGNING 0x0002
+
+struct smb2_rdma_transform_capabilities_context {
+ __le16 ContextType; /* 7 */
+ __le16 DataLength;
+ __u32 Reserved;
+ __le16 TransformCount;
+ __u16 Reserved1;
+ __u32 Reserved2;
+ __le16 RDMATransformIds[];
+} __packed;
+
+/*
+ * For signing capabilities context see MS-SMB2 2.2.3.1.7
+ * and 2.2.4.1.7
+ */
+
+/* Signing algorithms */
+#define SIGNING_ALG_HMAC_SHA256 0
+#define SIGNING_ALG_HMAC_SHA256_LE cpu_to_le16(0)
+#define SIGNING_ALG_AES_CMAC 1
+#define SIGNING_ALG_AES_CMAC_LE cpu_to_le16(1)
+#define SIGNING_ALG_AES_GMAC 2
+#define SIGNING_ALG_AES_GMAC_LE cpu_to_le16(2)
+
+struct smb2_signing_capabilities {
+ __le16 ContextType; /* 8 */
+ __le16 DataLength;
+ __le32 Reserved;
+ __le16 SigningAlgorithmCount;
+ __le16 SigningAlgorithms[];
+ /* Followed by padding to 8 byte boundary (required by some servers) */
+} __packed;
+
+#define POSIX_CTXT_DATA_LEN 16
+struct smb2_posix_neg_context {
+ __le16 ContextType; /* 0x100 */
+ __le16 DataLength;
+ __le32 Reserved;
+ __u8 Name[16]; /* POSIX ctxt GUID 93AD25509CB411E7B42383DE968BCD7C */
+} __packed;
+
+struct smb2_negotiate_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 36 */
+ __le16 DialectCount;
+ __le16 SecurityMode;
+ __le16 Reserved; /* MBZ */
+ __le32 Capabilities;
+ __u8 ClientGUID[SMB2_CLIENT_GUID_SIZE];
+ /* In SMB3.02 and earlier next three were MBZ le64 ClientStartTime */
+ __le32 NegotiateContextOffset; /* SMB3.1.1 only. MBZ earlier */
+ __le16 NegotiateContextCount; /* SMB3.1.1 only. MBZ earlier */
+ __le16 Reserved2;
+ __le16 Dialects[];
+} __packed;
+
+struct smb2_negotiate_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 65 */
+ __le16 SecurityMode;
+ __le16 DialectRevision;
+ __le16 NegotiateContextCount; /* Prior to SMB3.1.1 was Reserved & MBZ */
+ __u8 ServerGUID[16];
+ __le32 Capabilities;
+ __le32 MaxTransactSize;
+ __le32 MaxReadSize;
+ __le32 MaxWriteSize;
+ __le64 SystemTime; /* MBZ */
+ __le64 ServerStartTime;
+ __le16 SecurityBufferOffset;
+ __le16 SecurityBufferLength;
+ __le32 NegotiateContextOffset; /* Pre:SMB3.1.1 was reserved/ignored */
+ __u8 Buffer[1]; /* variable length GSS security buffer */
+} __packed;
+
+
+/*
+ * SMB2_SESSION_SETUP See MS-SMB2 section 2.2.5
+ */
+/* Flags */
+#define SMB2_SESSION_REQ_FLAG_BINDING 0x01
+#define SMB2_SESSION_REQ_FLAG_ENCRYPT_DATA 0x04
+
+struct smb2_sess_setup_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 25 */
+ __u8 Flags;
+ __u8 SecurityMode;
+ __le32 Capabilities;
+ __le32 Channel;
+ __le16 SecurityBufferOffset;
+ __le16 SecurityBufferLength;
+ __le64 PreviousSessionId;
+ __u8 Buffer[1]; /* variable length GSS security buffer */
+} __packed;
+
+/* Currently defined SessionFlags */
+#define SMB2_SESSION_FLAG_IS_GUEST 0x0001
+#define SMB2_SESSION_FLAG_IS_GUEST_LE cpu_to_le16(0x0001)
+#define SMB2_SESSION_FLAG_IS_NULL 0x0002
+#define SMB2_SESSION_FLAG_IS_NULL_LE cpu_to_le16(0x0002)
+#define SMB2_SESSION_FLAG_ENCRYPT_DATA 0x0004
+#define SMB2_SESSION_FLAG_ENCRYPT_DATA_LE cpu_to_le16(0x0004)
+
+struct smb2_sess_setup_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 9 */
+ __le16 SessionFlags;
+ __le16 SecurityBufferOffset;
+ __le16 SecurityBufferLength;
+ __u8 Buffer[1]; /* variable length GSS security buffer */
+} __packed;
+
+
+/*
+ * SMB2_LOGOFF See MS-SMB2 section 2.2.7
+ */
+struct smb2_logoff_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 4 */
+ __le16 Reserved;
+} __packed;
+
+struct smb2_logoff_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 4 */
+ __le16 Reserved;
+} __packed;
+
+
+/*
+ * SMB2_CLOSE See MS-SMB2 section 2.2.15
+ */
+/* Currently defined values for close flags */
+#define SMB2_CLOSE_FLAG_POSTQUERY_ATTRIB cpu_to_le16(0x0001)
+struct smb2_close_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 24 */
+ __le16 Flags;
+ __le32 Reserved;
+ __le64 PersistentFileId; /* opaque endianness */
+ __le64 VolatileFileId; /* opaque endianness */
+} __packed;
+
+/*
+ * Maximum size of a SMB2_CLOSE response is 64 (smb2 header) + 60 (data)
+ */
+#define MAX_SMB2_CLOSE_RESPONSE_SIZE 124
+
+struct smb2_close_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* 60 */
+ __le16 Flags;
+ __le32 Reserved;
+ __le64 CreationTime;
+ __le64 LastAccessTime;
+ __le64 LastWriteTime;
+ __le64 ChangeTime;
+ __le64 AllocationSize; /* Beginning of FILE_STANDARD_INFO equivalent */
+ __le64 EndOfFile;
+ __le32 Attributes;
+} __packed;
+
+
+/*
+ * SMB2_READ See MS-SMB2 section 2.2.19
+ */
+/* For read request Flags field below, following flag is defined for SMB3.02 */
+#define SMB2_READFLAG_READ_UNBUFFERED 0x01
+#define SMB2_READFLAG_REQUEST_COMPRESSED 0x02 /* See MS-SMB2 2.2.19 */
+
+/* Channel field for read and write: exactly one of following flags can be set*/
+#define SMB2_CHANNEL_NONE cpu_to_le32(0x00000000)
+#define SMB2_CHANNEL_RDMA_V1 cpu_to_le32(0x00000001)
+#define SMB2_CHANNEL_RDMA_V1_INVALIDATE cpu_to_le32(0x00000002)
+#define SMB2_CHANNEL_RDMA_TRANSFORM cpu_to_le32(0x00000003)
+
+/* SMB2 read request without RFC1001 length at the beginning */
+struct smb2_read_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 49 */
+ __u8 Padding; /* offset from start of SMB2 header to place read */
+ __u8 Flags; /* MBZ unless SMB3.02 or later */
+ __le32 Length;
+ __le64 Offset;
+ __le64 PersistentFileId;
+ __le64 VolatileFileId;
+ __le32 MinimumCount;
+ __le32 Channel; /* MBZ except for SMB3 or later */
+ __le32 RemainingBytes;
+ __le16 ReadChannelInfoOffset;
+ __le16 ReadChannelInfoLength;
+ __u8 Buffer[1];
+} __packed;
+
+/* Read flags */
+#define SMB2_READFLAG_RESPONSE_NONE cpu_to_le32(0x00000000)
+#define SMB2_READFLAG_RESPONSE_RDMA_TRANSFORM cpu_to_le32(0x00000001)
+
+struct smb2_read_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 17 */
+ __u8 DataOffset;
+ __u8 Reserved;
+ __le32 DataLength;
+ __le32 DataRemaining;
+ __le32 Flags;
+ __u8 Buffer[1];
+} __packed;
+
+
+/*
+ * SMB2_WRITE See MS-SMB2 section 2.2.21
+ */
+/* For write request Flags field below the following flags are defined: */
+#define SMB2_WRITEFLAG_WRITE_THROUGH 0x00000001 /* SMB2.1 or later */
+#define SMB2_WRITEFLAG_WRITE_UNBUFFERED 0x00000002 /* SMB3.02 or later */
+
+struct smb2_write_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 49 */
+ __le16 DataOffset; /* offset from start of SMB2 header to write data */
+ __le32 Length;
+ __le64 Offset;
+ __le64 PersistentFileId; /* opaque endianness */
+ __le64 VolatileFileId; /* opaque endianness */
+ __le32 Channel; /* MBZ unless SMB3.02 or later */
+ __le32 RemainingBytes;
+ __le16 WriteChannelInfoOffset;
+ __le16 WriteChannelInfoLength;
+ __le32 Flags;
+ __u8 Buffer[1];
+} __packed;
+
+struct smb2_write_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 17 */
+ __u8 DataOffset;
+ __u8 Reserved;
+ __le32 DataLength;
+ __le32 DataRemaining;
+ __u32 Reserved2;
+ __u8 Buffer[1];
+} __packed;
+
+
+/*
+ * SMB2_FLUSH See MS-SMB2 section 2.2.17
+ */
+struct smb2_flush_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 24 */
+ __le16 Reserved1;
+ __le32 Reserved2;
+ __le64 PersistentFileId;
+ __le64 VolatileFileId;
+} __packed;
+
+struct smb2_flush_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize;
+ __le16 Reserved;
+} __packed;
+
+
+/*
+ * SMB2_NOTIFY See MS-SMB2 section 2.2.35
+ */
+/* notify flags */
+#define SMB2_WATCH_TREE 0x0001
+
+/* notify completion filter flags. See MS-FSCC 2.6 and MS-SMB2 2.2.35 */
+#define FILE_NOTIFY_CHANGE_FILE_NAME 0x00000001
+#define FILE_NOTIFY_CHANGE_DIR_NAME 0x00000002
+#define FILE_NOTIFY_CHANGE_ATTRIBUTES 0x00000004
+#define FILE_NOTIFY_CHANGE_SIZE 0x00000008
+#define FILE_NOTIFY_CHANGE_LAST_WRITE 0x00000010
+#define FILE_NOTIFY_CHANGE_LAST_ACCESS 0x00000020
+#define FILE_NOTIFY_CHANGE_CREATION 0x00000040
+#define FILE_NOTIFY_CHANGE_EA 0x00000080
+#define FILE_NOTIFY_CHANGE_SECURITY 0x00000100
+#define FILE_NOTIFY_CHANGE_STREAM_NAME 0x00000200
+#define FILE_NOTIFY_CHANGE_STREAM_SIZE 0x00000400
+#define FILE_NOTIFY_CHANGE_STREAM_WRITE 0x00000800
+
+/* SMB2 Notify Action Flags */
+#define FILE_ACTION_ADDED 0x00000001
+#define FILE_ACTION_REMOVED 0x00000002
+#define FILE_ACTION_MODIFIED 0x00000003
+#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
+#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
+#define FILE_ACTION_ADDED_STREAM 0x00000006
+#define FILE_ACTION_REMOVED_STREAM 0x00000007
+#define FILE_ACTION_MODIFIED_STREAM 0x00000008
+#define FILE_ACTION_REMOVED_BY_DELETE 0x00000009
+
+struct smb2_change_notify_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize;
+ __le16 Flags;
+ __le32 OutputBufferLength;
+ __le64 PersistentFileId; /* opaque endianness */
+ __le64 VolatileFileId; /* opaque endianness */
+ __le32 CompletionFilter;
+ __u32 Reserved;
+} __packed;
+
+struct smb2_change_notify_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 9 */
+ __le16 OutputBufferOffset;
+ __le32 OutputBufferLength;
+ __u8 Buffer[1]; /* array of file notify structs */
+} __packed;
+
+
+/*
+ * SMB2_CREATE See MS-SMB2 section 2.2.13
+ */
+/* Oplock levels */
+#define SMB2_OPLOCK_LEVEL_NONE 0x00
+#define SMB2_OPLOCK_LEVEL_II 0x01
+#define SMB2_OPLOCK_LEVEL_EXCLUSIVE 0x08
+#define SMB2_OPLOCK_LEVEL_BATCH 0x09
+#define SMB2_OPLOCK_LEVEL_LEASE 0xFF
+/* Non-spec internal type */
+#define SMB2_OPLOCK_LEVEL_NOCHANGE 0x99
+
+/* Impersonation Levels. See MS-WPO section 9.7 and MSDN-IMPERS */
+#define IL_ANONYMOUS cpu_to_le32(0x00000000)
+#define IL_IDENTIFICATION cpu_to_le32(0x00000001)
+#define IL_IMPERSONATION cpu_to_le32(0x00000002)
+#define IL_DELEGATE cpu_to_le32(0x00000003)
+
+/* File Attrubutes */
+#define FILE_ATTRIBUTE_READONLY 0x00000001
+#define FILE_ATTRIBUTE_HIDDEN 0x00000002
+#define FILE_ATTRIBUTE_SYSTEM 0x00000004
+#define FILE_ATTRIBUTE_DIRECTORY 0x00000010
+#define FILE_ATTRIBUTE_ARCHIVE 0x00000020
+#define FILE_ATTRIBUTE_NORMAL 0x00000080
+#define FILE_ATTRIBUTE_TEMPORARY 0x00000100
+#define FILE_ATTRIBUTE_SPARSE_FILE 0x00000200
+#define FILE_ATTRIBUTE_REPARSE_POINT 0x00000400
+#define FILE_ATTRIBUTE_COMPRESSED 0x00000800
+#define FILE_ATTRIBUTE_OFFLINE 0x00001000
+#define FILE_ATTRIBUTE_NOT_CONTENT_INDEXED 0x00002000
+#define FILE_ATTRIBUTE_ENCRYPTED 0x00004000
+#define FILE_ATTRIBUTE_INTEGRITY_STREAM 0x00008000
+#define FILE_ATTRIBUTE_NO_SCRUB_DATA 0x00020000
+#define FILE_ATTRIBUTE__MASK 0x00007FB7
+
+#define FILE_ATTRIBUTE_READONLY_LE cpu_to_le32(0x00000001)
+#define FILE_ATTRIBUTE_HIDDEN_LE cpu_to_le32(0x00000002)
+#define FILE_ATTRIBUTE_SYSTEM_LE cpu_to_le32(0x00000004)
+#define FILE_ATTRIBUTE_DIRECTORY_LE cpu_to_le32(0x00000010)
+#define FILE_ATTRIBUTE_ARCHIVE_LE cpu_to_le32(0x00000020)
+#define FILE_ATTRIBUTE_NORMAL_LE cpu_to_le32(0x00000080)
+#define FILE_ATTRIBUTE_TEMPORARY_LE cpu_to_le32(0x00000100)
+#define FILE_ATTRIBUTE_SPARSE_FILE_LE cpu_to_le32(0x00000200)
+#define FILE_ATTRIBUTE_REPARSE_POINT_LE cpu_to_le32(0x00000400)
+#define FILE_ATTRIBUTE_COMPRESSED_LE cpu_to_le32(0x00000800)
+#define FILE_ATTRIBUTE_OFFLINE_LE cpu_to_le32(0x00001000)
+#define FILE_ATTRIBUTE_NOT_CONTENT_INDEXED_LE cpu_to_le32(0x00002000)
+#define FILE_ATTRIBUTE_ENCRYPTED_LE cpu_to_le32(0x00004000)
+#define FILE_ATTRIBUTE_INTEGRITY_STREAM_LE cpu_to_le32(0x00008000)
+#define FILE_ATTRIBUTE_NO_SCRUB_DATA_LE cpu_to_le32(0x00020000)
+#define FILE_ATTRIBUTE_MASK_LE cpu_to_le32(0x00007FB7)
+
+/* Desired Access Flags */
+#define FILE_READ_DATA_LE cpu_to_le32(0x00000001)
+#define FILE_LIST_DIRECTORY_LE cpu_to_le32(0x00000001)
+#define FILE_WRITE_DATA_LE cpu_to_le32(0x00000002)
+#define FILE_APPEND_DATA_LE cpu_to_le32(0x00000004)
+#define FILE_ADD_SUBDIRECTORY_LE cpu_to_le32(0x00000004)
+#define FILE_READ_EA_LE cpu_to_le32(0x00000008)
+#define FILE_WRITE_EA_LE cpu_to_le32(0x00000010)
+#define FILE_EXECUTE_LE cpu_to_le32(0x00000020)
+#define FILE_DELETE_CHILD_LE cpu_to_le32(0x00000040)
+#define FILE_READ_ATTRIBUTES_LE cpu_to_le32(0x00000080)
+#define FILE_WRITE_ATTRIBUTES_LE cpu_to_le32(0x00000100)
+#define FILE_DELETE_LE cpu_to_le32(0x00010000)
+#define FILE_READ_CONTROL_LE cpu_to_le32(0x00020000)
+#define FILE_WRITE_DAC_LE cpu_to_le32(0x00040000)
+#define FILE_WRITE_OWNER_LE cpu_to_le32(0x00080000)
+#define FILE_SYNCHRONIZE_LE cpu_to_le32(0x00100000)
+#define FILE_ACCESS_SYSTEM_SECURITY_LE cpu_to_le32(0x01000000)
+#define FILE_MAXIMAL_ACCESS_LE cpu_to_le32(0x02000000)
+#define FILE_GENERIC_ALL_LE cpu_to_le32(0x10000000)
+#define FILE_GENERIC_EXECUTE_LE cpu_to_le32(0x20000000)
+#define FILE_GENERIC_WRITE_LE cpu_to_le32(0x40000000)
+#define FILE_GENERIC_READ_LE cpu_to_le32(0x80000000)
+#define DESIRED_ACCESS_MASK cpu_to_le32(0xF21F01FF)
+
+
+#define FILE_READ_DESIRED_ACCESS_LE (FILE_READ_DATA_LE | \
+ FILE_READ_EA_LE | \
+ FILE_GENERIC_READ_LE)
+#define FILE_WRITE_DESIRE_ACCESS_LE (FILE_WRITE_DATA_LE | \
+ FILE_APPEND_DATA_LE | \
+ FILE_WRITE_EA_LE | \
+ FILE_WRITE_ATTRIBUTES_LE | \
+ FILE_GENERIC_WRITE_LE)
+
+/* ShareAccess Flags */
+#define FILE_SHARE_READ_LE cpu_to_le32(0x00000001)
+#define FILE_SHARE_WRITE_LE cpu_to_le32(0x00000002)
+#define FILE_SHARE_DELETE_LE cpu_to_le32(0x00000004)
+#define FILE_SHARE_ALL_LE cpu_to_le32(0x00000007)
+
+/* CreateDisposition Flags */
+#define FILE_SUPERSEDE_LE cpu_to_le32(0x00000000)
+#define FILE_OPEN_LE cpu_to_le32(0x00000001)
+#define FILE_CREATE_LE cpu_to_le32(0x00000002)
+#define FILE_OPEN_IF_LE cpu_to_le32(0x00000003)
+#define FILE_OVERWRITE_LE cpu_to_le32(0x00000004)
+#define FILE_OVERWRITE_IF_LE cpu_to_le32(0x00000005)
+#define FILE_CREATE_MASK_LE cpu_to_le32(0x00000007)
+
+#define FILE_READ_RIGHTS (FILE_READ_DATA | FILE_READ_EA \
+ | FILE_READ_ATTRIBUTES)
+#define FILE_WRITE_RIGHTS (FILE_WRITE_DATA | FILE_APPEND_DATA \
+ | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES)
+#define FILE_EXEC_RIGHTS (FILE_EXECUTE)
+
+/* CreateOptions Flags */
+#define FILE_DIRECTORY_FILE_LE cpu_to_le32(0x00000001)
+/* same as #define CREATE_NOT_FILE_LE cpu_to_le32(0x00000001) */
+#define FILE_WRITE_THROUGH_LE cpu_to_le32(0x00000002)
+#define FILE_SEQUENTIAL_ONLY_LE cpu_to_le32(0x00000004)
+#define FILE_NO_INTERMEDIATE_BUFFERING_LE cpu_to_le32(0x00000008)
+#define FILE_NON_DIRECTORY_FILE_LE cpu_to_le32(0x00000040)
+#define FILE_COMPLETE_IF_OPLOCKED_LE cpu_to_le32(0x00000100)
+#define FILE_NO_EA_KNOWLEDGE_LE cpu_to_le32(0x00000200)
+#define FILE_RANDOM_ACCESS_LE cpu_to_le32(0x00000800)
+#define FILE_DELETE_ON_CLOSE_LE cpu_to_le32(0x00001000)
+#define FILE_OPEN_BY_FILE_ID_LE cpu_to_le32(0x00002000)
+#define FILE_OPEN_FOR_BACKUP_INTENT_LE cpu_to_le32(0x00004000)
+#define FILE_NO_COMPRESSION_LE cpu_to_le32(0x00008000)
+#define FILE_OPEN_REPARSE_POINT_LE cpu_to_le32(0x00200000)
+#define FILE_OPEN_NO_RECALL_LE cpu_to_le32(0x00400000)
+#define CREATE_OPTIONS_MASK_LE cpu_to_le32(0x00FFFFFF)
+
+#define FILE_READ_RIGHTS_LE (FILE_READ_DATA_LE | FILE_READ_EA_LE \
+ | FILE_READ_ATTRIBUTES_LE)
+#define FILE_WRITE_RIGHTS_LE (FILE_WRITE_DATA_LE | FILE_APPEND_DATA_LE \
+ | FILE_WRITE_EA_LE | FILE_WRITE_ATTRIBUTES_LE)
+#define FILE_EXEC_RIGHTS_LE (FILE_EXECUTE_LE)
+
+/* Create Context Values */
+#define SMB2_CREATE_EA_BUFFER "ExtA" /* extended attributes */
+#define SMB2_CREATE_SD_BUFFER "SecD" /* security descriptor */
+#define SMB2_CREATE_DURABLE_HANDLE_REQUEST "DHnQ"
+#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT "DHnC"
+#define SMB2_CREATE_ALLOCATION_SIZE "AISi"
+#define SMB2_CREATE_QUERY_MAXIMAL_ACCESS_REQUEST "MxAc"
+#define SMB2_CREATE_TIMEWARP_REQUEST "TWrp"
+#define SMB2_CREATE_QUERY_ON_DISK_ID "QFid"
+#define SMB2_CREATE_REQUEST_LEASE "RqLs"
+#define SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 "DH2Q"
+#define SMB2_CREATE_DURABLE_HANDLE_RECONNECT_V2 "DH2C"
+#define SMB2_CREATE_TAG_POSIX "\x93\xAD\x25\x50\x9C\xB4\x11\xE7\xB4\x23\x83\xDE\x96\x8B\xCD\x7C"
+
+/* Flag (SMB3 open response) values */
+#define SMB2_CREATE_FLAG_REPARSEPOINT 0x01
+
+struct create_context {
+ __le32 Next;
+ __le16 NameOffset;
+ __le16 NameLength;
+ __le16 Reserved;
+ __le16 DataOffset;
+ __le32 DataLength;
+ __u8 Buffer[];
+} __packed;
+
+struct smb2_create_req {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 57 */
+ __u8 SecurityFlags;
+ __u8 RequestedOplockLevel;
+ __le32 ImpersonationLevel;
+ __le64 SmbCreateFlags;
+ __le64 Reserved;
+ __le32 DesiredAccess;
+ __le32 FileAttributes;
+ __le32 ShareAccess;
+ __le32 CreateDisposition;
+ __le32 CreateOptions;
+ __le16 NameOffset;
+ __le16 NameLength;
+ __le32 CreateContextsOffset;
+ __le32 CreateContextsLength;
+ __u8 Buffer[];
+} __packed;
+
+struct smb2_create_rsp {
+ struct smb2_hdr hdr;
+ __le16 StructureSize; /* Must be 89 */
+ __u8 OplockLevel;
+ __u8 Flags; /* 0x01 if reparse point */
+ __le32 CreateAction;
+ __le64 CreationTime;
+ __le64 LastAccessTime;
+ __le64 LastWriteTime;
+ __le64 ChangeTime;
+ __le64 AllocationSize;
+ __le64 EndofFile;
+ __le32 FileAttributes;
+ __le32 Reserved2;
+ __le64 PersistentFileId;
+ __le64 VolatileFileId;
+ __le32 CreateContextsOffset;
+ __le32 CreateContextsLength;
+ __u8 Buffer[1];
+} __packed;
+
+
+#endif /* _COMMON_SMB2PDU_H */
goto failed;
wksp->window_size = max_t(size_t,
msblk->block_size, SQUASHFS_METADATA_SIZE);
- wksp->mem_size = ZSTD_DStreamWorkspaceBound(wksp->window_size);
+ wksp->mem_size = zstd_dstream_workspace_bound(wksp->window_size);
wksp->mem = vmalloc(wksp->mem_size);
if (wksp->mem == NULL)
goto failed;
struct squashfs_page_actor *output)
{
struct workspace *wksp = strm;
- ZSTD_DStream *stream;
+ zstd_dstream *stream;
size_t total_out = 0;
int error = 0;
- ZSTD_inBuffer in_buf = { NULL, 0, 0 };
- ZSTD_outBuffer out_buf = { NULL, 0, 0 };
+ zstd_in_buffer in_buf = { NULL, 0, 0 };
+ zstd_out_buffer out_buf = { NULL, 0, 0 };
struct bvec_iter_all iter_all = {};
struct bio_vec *bvec = bvec_init_iter_all(&iter_all);
- stream = ZSTD_initDStream(wksp->window_size, wksp->mem, wksp->mem_size);
+ stream = zstd_init_dstream(wksp->window_size, wksp->mem, wksp->mem_size);
if (!stream) {
ERROR("Failed to initialize zstd decompressor\n");
}
total_out -= out_buf.pos;
- zstd_err = ZSTD_decompressStream(stream, &out_buf, &in_buf);
+ zstd_err = zstd_decompress_stream(stream, &out_buf, &in_buf);
total_out += out_buf.pos; /* add the additional data produced */
if (zstd_err == 0)
break;
- if (ZSTD_isError(zstd_err)) {
+ if (zstd_is_error(zstd_err)) {
ERROR("zstd decompression error: %d\n",
- (int)ZSTD_getErrorCode(zstd_err));
+ (int)zstd_get_error_code(zstd_err));
error = -EIO;
break;
}
spin_unlock(&sb_lock);
up_write(&sb->s_umount);
if (sb->s_bdi != &noop_backing_dev_info) {
+ if (sb->s_iflags & SB_I_PERSB_BDI)
+ bdi_unregister(sb->s_bdi);
bdi_put(sb->s_bdi);
sb->s_bdi = &noop_backing_dev_info;
}
}
WARN_ON(sb->s_bdi != &noop_backing_dev_info);
sb->s_bdi = bdi;
+ sb->s_iflags |= SB_I_PERSB_BDI;
return 0;
}
struct sysv_sb_info *sbi;
struct buffer_head *bh;
- if (440 != sizeof (struct v7_super_block))
- panic("V7 FS: bad super-block size");
- if (64 != sizeof (struct sysv_inode))
- panic("sysv fs: bad i-node size");
+ BUILD_BUG_ON(sizeof(struct v7_super_block) != 440);
+ BUILD_BUG_ON(sizeof(struct sysv_inode) != 64);
sbi = kzalloc(sizeof(struct sysv_sb_info), GFP_KERNEL);
if (!sbi)
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/bio.h>
+#include <linux/iversion.h>
#include "udf_i.h"
#include "udf_sb.h"
struct fileIdentDesc *fi = NULL;
struct fileIdentDesc cfi;
udf_pblk_t block, iblock;
- loff_t nf_pos;
+ loff_t nf_pos, emit_pos = 0;
int flen;
unsigned char *fname = NULL, *copy_name = NULL;
unsigned char *nameptr;
int i, num, ret = 0;
struct extent_position epos = { NULL, 0, {0, 0} };
struct super_block *sb = dir->i_sb;
+ bool pos_valid = false;
if (ctx->pos == 0) {
if (!dir_emit_dot(file, ctx))
if (nf_pos >= size)
goto out;
+ /*
+ * Something changed since last readdir (either lseek was called or dir
+ * changed)? We need to verify the position correctly points at the
+ * beginning of some dir entry so that the directory parsing code does
+ * not get confused. Since UDF does not have any reliable way of
+ * identifying beginning of dir entry (names are under user control),
+ * we need to scan the directory from the beginning.
+ */
+ if (!inode_eq_iversion(dir, file->f_version)) {
+ emit_pos = nf_pos;
+ nf_pos = 0;
+ } else {
+ pos_valid = true;
+ }
+
fname = kmalloc(UDF_NAME_LEN, GFP_NOFS);
if (!fname) {
ret = -ENOMEM;
while (nf_pos < size) {
struct kernel_lb_addr tloc;
+ loff_t cur_pos = nf_pos;
- ctx->pos = (nf_pos >> 2) + 1;
+ /* Update file position only if we got past the current one */
+ if (nf_pos >= emit_pos) {
+ ctx->pos = (nf_pos >> 2) + 1;
+ pos_valid = true;
+ }
fi = udf_fileident_read(dir, &nf_pos, &fibh, &cfi, &epos, &eloc,
&elen, &offset);
if (!fi)
goto out;
+ /* Still not at offset where user asked us to read from? */
+ if (cur_pos < emit_pos)
+ continue;
liu = le16_to_cpu(cfi.lengthOfImpUse);
lfi = cfi.lengthFileIdent;
} /* end while */
ctx->pos = (nf_pos >> 2) + 1;
+ pos_valid = true;
out:
+ if (pos_valid)
+ file->f_version = inode_query_iversion(dir);
if (fibh.sbh != fibh.ebh)
brelse(fibh.ebh);
brelse(fibh.sbh);
#include <linux/sched.h>
#include <linux/crc-itu-t.h>
#include <linux/exportfs.h>
+#include <linux/iversion.h>
static inline int udf_match(int len1, const unsigned char *name1, int len2,
const unsigned char *name2)
mark_buffer_dirty_inode(fibh->ebh, inode);
mark_buffer_dirty_inode(fibh->sbh, inode);
}
+ inode_inc_iversion(inode);
+
return 0;
}
#include <linux/crc-itu-t.h>
#include <linux/log2.h>
#include <asm/byteorder.h>
+#include <linux/iversion.h>
#include "udf_sb.h"
#include "udf_i.h"
init_rwsem(&ei->i_data_sem);
ei->cached_extent.lstart = -1;
spin_lock_init(&ei->i_extent_cache_lock);
+ inode_set_iversion(&ei->vfs_inode, 1);
return &ei->vfs_inode;
}
spin_unlock(&mp->m_perag_lock);
radix_tree_preload_end();
+#ifdef __KERNEL__
/* Place kernel structure only init below this point. */
spin_lock_init(&pag->pag_ici_lock);
spin_lock_init(&pag->pagb_lock);
init_waitqueue_head(&pag->pagb_wait);
pag->pagb_count = 0;
pag->pagb_tree = RB_ROOT;
+#endif /* __KERNEL__ */
error = xfs_buf_hash_init(pag);
if (error)
/* Blocks reserved for the reverse mapping btree. */
struct xfs_ag_resv pag_rmapbt_resv;
+ /* for rcu-safe freeing */
+ struct rcu_head rcu_head;
+
+#ifdef __KERNEL__
/* -- kernel only structures below this line -- */
/*
spinlock_t pag_buf_lock; /* lock for pag_buf_hash */
struct rhashtable pag_buf_hash;
- /* for rcu-safe freeing */
- struct rcu_head rcu_head;
-
/* background prealloc block trimming */
struct delayed_work pag_blockgc_work;
* or have some other means to control concurrency.
*/
struct rhashtable pagi_unlinked_hash;
+#endif /* __KERNEL__ */
};
int xfs_initialize_perag(struct xfs_mount *mp, xfs_agnumber_t agcount,
return error;
}
+#ifdef __KERNEL__
struct xfs_btree_split_args {
struct xfs_btree_cur *cur;
int level;
destroy_work_on_stack(&args.work);
return args.result;
}
+#else
+#define xfs_btree_split __xfs_btree_split
+#endif /* __KERNEL__ */
/*
{
struct xfs_da_intnode *node1;
struct xfs_da_intnode *node2;
- struct xfs_da_intnode *tmpnode;
struct xfs_da_node_entry *btree1;
struct xfs_da_node_entry *btree2;
struct xfs_da_node_entry *btree_s;
((be32_to_cpu(btree2[0].hashval) < be32_to_cpu(btree1[0].hashval)) ||
(be32_to_cpu(btree2[nodehdr2.count - 1].hashval) <
be32_to_cpu(btree1[nodehdr1.count - 1].hashval)))) {
- tmpnode = node1;
- node1 = node2;
- node2 = tmpnode;
+ swap(node1, node2);
xfs_da3_node_hdr_from_disk(dp->i_mount, &nodehdr1, node1);
xfs_da3_node_hdr_from_disk(dp->i_mount, &nodehdr2, node2);
btree1 = nodehdr1.btree;
int state; /* Current state */
struct acpi_device_power_flags flags;
struct acpi_device_power_state states[ACPI_D_STATE_COUNT]; /* Power states (D0-D3Cold) */
+ u8 state_for_enumeration; /* Deepest power state for enumeration */
};
struct acpi_dep_data {
#ifndef __ACGCC_H__
#define __ACGCC_H__
-/*
- * Use compiler specific <stdarg.h> is a good practice for even when
- * -nostdinc is specified (i.e., ACPI_USE_STANDARD_HEADERS undefined.
- */
#ifndef va_arg
-#ifdef ACPI_USE_BUILTIN_STDARG
-typedef __builtin_va_list va_list;
-#define va_start(v, l) __builtin_va_start(v, l)
-#define va_end(v) __builtin_va_end(v)
-#define va_arg(v, l) __builtin_va_arg(v, l)
-#define va_copy(d, s) __builtin_va_copy(d, s)
-#else
#ifdef __KERNEL__
#include <linux/stdarg.h>
#else
-/* Used to build acpi tools */
#include <stdarg.h>
#endif /* __KERNEL__ */
-#endif /* ACPI_USE_BUILTIN_STDARG */
#endif /* ! va_arg */
#define ACPI_INLINE __inline__
#define dereference_kernel_function_descriptor(p) ((void *)(p))
#endif
-/* random extra sections (if any). Override
- * in asm/sections.h */
-#ifndef arch_is_kernel_text
-static inline int arch_is_kernel_text(unsigned long addr)
-{
- return 0;
-}
-#endif
-
-#ifndef arch_is_kernel_data
-static inline int arch_is_kernel_data(unsigned long addr)
-{
- return 0;
-}
-#endif
-
-/*
- * Check if an address is part of freed initmem. This is needed on architectures
- * with virt == phys kernel mapping, for code that wants to check if an address
- * is part of a static object within [_stext, _end]. After initmem is freed,
- * memory can be allocated from it, and such allocations would then have
- * addresses within the range [_stext, _end].
- */
-#ifndef arch_is_kernel_initmem_freed
-static inline int arch_is_kernel_initmem_freed(unsigned long addr)
-{
- return 0;
-}
-#endif
-
/**
* memory_contains - checks if an object is contained within a memory region
* @begin: virtual address of the beginning of the memory region
}
/**
+ * is_kernel_core_data - checks if the pointer address is located in the
+ * .data section
+ *
+ * @addr: address to check
+ *
+ * Returns: true if the address is located in .data, false otherwise.
+ * Note: On some archs it may return true for core RODATA, and false
+ * for others. But will always be true for core RW data.
+ */
+static inline bool is_kernel_core_data(unsigned long addr)
+{
+ return addr >= (unsigned long)_sdata &&
+ addr < (unsigned long)_edata;
+}
+
+/**
* is_kernel_rodata - checks if the pointer address is located in the
* .rodata section
*
addr < (unsigned long)__end_rodata;
}
+/**
+ * is_kernel_inittext - checks if the pointer address is located in the
+ * .init.text section
+ *
+ * @addr: address to check
+ *
+ * Returns: true if the address is located in .init.text, false otherwise.
+ */
+static inline bool is_kernel_inittext(unsigned long addr)
+{
+ return addr >= (unsigned long)_sinittext &&
+ addr < (unsigned long)_einittext;
+}
+
+/**
+ * __is_kernel_text - checks if the pointer address is located in the
+ * .text section
+ *
+ * @addr: address to check
+ *
+ * Returns: true if the address is located in .text, false otherwise.
+ * Note: an internal helper, only check the range of _stext to _etext.
+ */
+static inline bool __is_kernel_text(unsigned long addr)
+{
+ return addr >= (unsigned long)_stext &&
+ addr < (unsigned long)_etext;
+}
+
+/**
+ * __is_kernel - checks if the pointer address is located in the kernel range
+ *
+ * @addr: address to check
+ *
+ * Returns: true if the address is located in the kernel range, false otherwise.
+ * Note: an internal helper, only check the range of _stext to _end.
+ */
+static inline bool __is_kernel(unsigned long addr)
+{
+ return addr >= (unsigned long)_stext &&
+ addr < (unsigned long)_end;
+}
+
#endif /* _ASM_GENERIC_SECTIONS_H_ */
unsigned long *args);
/**
- * syscall_set_arguments - change system call parameter value
- * @task: task of interest, must be in system call entry tracing
- * @regs: task_pt_regs() of @task
- * @args: array of argument values to store
- *
- * Changes 6 arguments to the system call.
- * The first argument gets value @args[0], and so on.
- *
- * It's only valid to call this when @task is stopped for tracing on
- * entry to a system call, due to %SYSCALL_WORK_SYSCALL_TRACE or
- * %SYSCALL_WORK_SYSCALL_AUDIT.
- */
-void syscall_set_arguments(struct task_struct *task, struct pt_regs *regs,
- const unsigned long *args);
-
-/**
* syscall_get_arch - return the AUDIT_ARCH for the current system call
* @task: task of interest, must be blocked
*
#ifndef DRM_MODESET_LOCK_H_
#define DRM_MODESET_LOCK_H_
+#include <linux/types.h> /* stackdepot.h is not self-contained */
+#include <linux/stackdepot.h>
#include <linux/ww_mutex.h>
struct drm_modeset_lock;
struct drm_modeset_lock *contended;
/*
+ * Stack depot for debugging when a contended lock was not backed off
+ * from.
+ */
+ depot_stack_handle_t stack_depot;
+
+ /*
* list of held locks (drm_modeset_lock)
*/
struct list_head locked;
* @bo: Pointer to a ttm_buffer_object to be initialized.
* @size: Requested size of buffer object.
* @type: Requested type of buffer object.
- * @flags: Initial placement flags.
+ * @placement: Initial placement for buffer object.
* @page_alignment: Data alignment in pages.
* @ctx: TTM operation context for memory allocation.
+ * @sg: Scatter-gather table.
* @resv: Pointer to a dma_resv, or NULL to let ttm allocate one.
* @destroy: Destroy function. Use NULL for kfree().
*
* @bo: Pointer to a ttm_buffer_object to be initialized.
* @size: Requested size of buffer object.
* @type: Requested type of buffer object.
- * @flags: Initial placement flags.
+ * @placement: Initial placement for buffer object.
* @page_alignment: Data alignment in pages.
* @interruptible: If needing to sleep to wait for GPU resources,
* sleep interruptible.
* holds a pointer to a persistent shmem object. Typically, this would
* point to the shmem object backing a GEM object if TTM is used to back a
* GEM user interface.
+ * @sg: Scatter-gather table.
* @resv: Pointer to a dma_resv, or NULL to let ttm allocate one.
* @destroy: Destroy function. Use NULL for kfree().
*
vm_fault_t ttm_bo_vm_fault_reserved(struct vm_fault *vmf,
pgprot_t prot,
- pgoff_t num_prefault,
- pgoff_t fault_page_size);
+ pgoff_t num_prefault);
vm_fault_t ttm_bo_vm_fault(struct vm_fault *vmf);
#define AM4_L3S_VPFE0_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x68)
#define AM4_L3S_VPFE1_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x70)
#define AM4_L3S_GPMC_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x220)
+#define AM4_L3S_ADC1_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x230)
#define AM4_L3S_MCASP0_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x238)
#define AM4_L3S_MCASP1_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x240)
#define AM4_L3S_MMC3_CLKCTRL AM4_L3S_CLKCTRL_INDEX(0x248)
#include <kunit/assert.h>
#include <kunit/try-catch.h>
-#include <linux/kernel.h>
+
+#include <linux/container_of.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/kconfig.h>
+#include <linux/kref.h>
+#include <linux/list.h>
#include <linux/module.h>
#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/string.h>
#include <linux/types.h>
-#include <linux/kref.h>
+
+#include <asm/rwonce.h>
struct kunit_resource;
#define OSC_PCI_MSI_SUPPORT 0x00000010
#define OSC_PCI_EDR_SUPPORT 0x00000080
#define OSC_PCI_HPX_TYPE_3_SUPPORT 0x00000100
-#define OSC_PCI_SUPPORT_MASKS 0x0000019f
/* PCI Host Bridge _OSC: Capabilities DWORD 3: Control Field */
#define OSC_PCI_EXPRESS_NATIVE_HP_CONTROL 0x00000001
#define OSC_PCI_EXPRESS_CAPABILITY_CONTROL 0x00000010
#define OSC_PCI_EXPRESS_LTR_CONTROL 0x00000020
#define OSC_PCI_EXPRESS_DPC_CONTROL 0x00000080
-#define OSC_PCI_CONTROL_MASKS 0x000000bf
#define ACPI_GSB_ACCESS_ATTRIB_QUICK 0x00000002
#define ACPI_GSB_ACCESS_ATTRIB_SEND_RCV 0x00000004
return -ENODEV;
}
+static inline int acpi_register_wakeup_handler(int wake_irq,
+ bool (*wakeup)(void *context), void *context)
+{
+ return -ENXIO;
+}
+
+static inline void acpi_unregister_wakeup_handler(
+ bool (*wakeup)(void *context), void *context) { }
+
#endif /* !CONFIG_ACPI */
#ifdef CONFIG_ACPI_HOTPLUG_IOAPIC
int acpi_subsys_runtime_resume(struct device *dev);
int acpi_dev_pm_attach(struct device *dev, bool power_on);
bool acpi_storage_d3(struct device *dev);
+bool acpi_dev_state_d0(struct device *dev);
#else
static inline int acpi_subsys_runtime_suspend(struct device *dev) { return 0; }
static inline int acpi_subsys_runtime_resume(struct device *dev) { return 0; }
{
return false;
}
+static inline bool acpi_dev_state_d0(struct device *dev)
+{
+ return true;
+}
#endif
#if defined(CONFIG_ACPI) && defined(CONFIG_PM_SLEEP)
* change as blkcg is disabled and enabled higher up in the hierarchy, a wb
* is tested for blkcg after lookup and removed from index on mismatch so
* that a new wb for the combination can be created.
+ *
+ * Each bdi_writeback that is not embedded into the backing_dev_info must hold
+ * a reference to the parent backing_dev_info. See cgwb_create() for details.
*/
struct bdi_writeback {
struct backing_dev_info *bdi; /* our parent bdi */
}
long congestion_wait(int sync, long timeout);
-long wait_iff_congested(int sync, long timeout);
static inline bool mapping_can_writeback(struct address_space *mapping)
{
void blk_mq_start_stopped_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async);
void blk_mq_quiesce_queue(struct request_queue *q);
+void blk_mq_wait_quiesce_done(struct request_queue *q);
void blk_mq_unquiesce_queue(struct request_queue *q);
void blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, unsigned long msecs);
void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async);
#ifndef _LINUX_BH_H
#define _LINUX_BH_H
+#include <linux/instruction_pointer.h>
#include <linux/preempt.h>
#if defined(CONFIG_PREEMPT_RT) || defined(CONFIG_TRACE_IRQFLAGS)
atomic64_t usercnt;
struct work_struct work;
struct mutex freeze_mutex;
- u64 writecnt; /* writable mmap cnt; protected by freeze_mutex */
+ atomic64_t writecnt;
};
static inline bool map_value_has_spin_lock(const struct bpf_map *map)
aux->ctx_field_size = size;
}
+static inline bool bpf_pseudo_func(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
+ insn->src_reg == BPF_PSEUDO_FUNC;
+}
+
struct bpf_prog_ops {
int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
union bpf_attr __user *uattr);
void *bpf_map_area_alloc(u64 size, int numa_node);
void *bpf_map_area_mmapable_alloc(u64 size, int numa_node);
void bpf_map_area_free(void *base);
+bool bpf_map_write_active(const struct bpf_map *map);
void bpf_map_init_from_attr(struct bpf_map *map, union bpf_attr *attr);
int generic_map_lookup_batch(struct bpf_map *map,
const union bpf_attr *attr,
CEPH_SESSION_REQUEST_FLUSH_MDLOG,
};
+#define CEPH_SESSION_BLOCKLISTED (1 << 0) /* session blocklisted */
+
extern const char *ceph_session_op_name(int op);
struct ceph_mds_session_head {
u64 expected_object_size,
u64 expected_write_size,
u32 flags);
+extern int osd_req_op_copy_from_init(struct ceph_osd_request *req,
+ u64 src_snapid, u64 src_version,
+ struct ceph_object_id *src_oid,
+ struct ceph_object_locator *src_oloc,
+ u32 src_fadvise_flags,
+ u32 dst_fadvise_flags,
+ u32 truncate_seq, u64 truncate_size,
+ u8 copy_from_flags);
extern struct ceph_osd_request *ceph_osdc_alloc_request(struct ceph_osd_client *osdc,
struct ceph_snap_context *snapc,
struct page *req_page, size_t req_len,
struct page **resp_pages, size_t *resp_len);
-int ceph_osdc_copy_from(struct ceph_osd_client *osdc,
- u64 src_snapid, u64 src_version,
- struct ceph_object_id *src_oid,
- struct ceph_object_locator *src_oloc,
- u32 src_fadvise_flags,
- struct ceph_object_id *dst_oid,
- struct ceph_object_locator *dst_oloc,
- u32 dst_fadvise_flags,
- u32 truncate_seq, u64 truncate_size,
- u8 copy_from_flags);
-
/* watch/notify */
struct ceph_osd_linger_request *
ceph_osdc_watch(struct ceph_osd_client *osdc,
struct cma **res_cma);
extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align,
bool no_warn);
+extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count);
extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count);
extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data);
#else
#define __diag_GCC_8(s)
#endif
+
+/*
+ * Prior to 9.1, -Wno-alloc-size-larger-than (and therefore the "alloc_size"
+ * attribute) do not work, and must be disabled.
+ */
+#if GCC_VERSION < 90100
+#undef __alloc_size__
+#endif
#define __aligned_largest __attribute__((__aligned__))
/*
+ * Note: do not use this directly. Instead, use __alloc_size() since it is conditionally
+ * available and includes other attributes.
+ *
+ * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-alloc_005fsize-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#alloc-size
+ */
+#define __alloc_size__(x, ...) __attribute__((__alloc_size__(x, ## __VA_ARGS__)))
+
+/*
* Note: users of __always_inline currently do not write "inline" themselves,
* which seems to be required by gcc to apply the attribute according
* to its docs (and also "warning: always_inline function might not be
#define __deprecated
/*
- * Optional: only supported since gcc >= 5.1
* Optional: not supported by clang
* Optional: not supported by icc
*
/*
* gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-malloc-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#malloc
*/
#define __malloc __attribute__((__malloc__))
# define __cficanonical
#endif
+/*
+ * Any place that could be marked with the "alloc_size" attribute is also
+ * a place to be marked with the "malloc" attribute. Do this as part of the
+ * __alloc_size macro to avoid redundant attributes and to avoid missing a
+ * __malloc marking.
+ */
+#ifdef __alloc_size__
+# define __alloc_size(x, ...) __alloc_size__(x, ## __VA_ARGS__) __malloc
+#else
+# define __alloc_size(x, ...) __malloc
+#endif
+
#ifndef asm_volatile_goto
#define asm_volatile_goto(x...) asm goto(x)
#endif
#ifdef __OPTIMIZE__
# define __compiletime_assert(condition, msg, prefix, suffix) \
do { \
- extern void prefix ## suffix(void) __compiletime_error(msg); \
+ /* \
+ * __noreturn is needed to give the compiler enough \
+ * information to avoid certain possibly-uninitialized \
+ * warnings (regardless of the build failing). \
+ */ \
+ __noreturn extern void prefix ## suffix(void) \
+ __compiletime_error(msg); \
if (!(condition)) \
prefix ## suffix(); \
} while (0)
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_CONTAINER_OF_H
+#define _LINUX_CONTAINER_OF_H
+
+#include <linux/build_bug.h>
+#include <linux/err.h>
+
+#define typeof_member(T, m) typeof(((T*)0)->m)
+
+/**
+ * container_of - cast a member of a structure out to the containing structure
+ * @ptr: the pointer to the member.
+ * @type: the type of the container struct this is embedded in.
+ * @member: the name of the member within the struct.
+ *
+ */
+#define container_of(ptr, type, member) ({ \
+ void *__mptr = (void *)(ptr); \
+ static_assert(__same_type(*(ptr), ((type *)0)->member) || \
+ __same_type(*(ptr), void), \
+ "pointer type mismatch in container_of()"); \
+ ((type *)(__mptr - offsetof(type, member))); })
+
+/**
+ * container_of_safe - cast a member of a structure out to the containing structure
+ * @ptr: the pointer to the member.
+ * @type: the type of the container struct this is embedded in.
+ * @member: the name of the member within the struct.
+ *
+ * If IS_ERR_OR_NULL(ptr), ptr is returned unchanged.
+ */
+#define container_of_safe(ptr, type, member) ({ \
+ void *__mptr = (void *)(ptr); \
+ static_assert(__same_type(*(ptr), ((type *)0)->member) || \
+ __same_type(*(ptr), void), \
+ "pointer type mismatch in container_of_safe()"); \
+ IS_ERR_OR_NULL(__mptr) ? ERR_CAST(__mptr) : \
+ ((type *)(__mptr - offsetof(type, member))); })
+
+#endif /* _LINUX_CONTAINER_OF_H */
*/
extern struct static_key_false cpusets_pre_enable_key;
extern struct static_key_false cpusets_enabled_key;
+extern struct static_key_false cpusets_insane_config_key;
+
static inline bool cpusets_enabled(void)
{
return static_branch_unlikely(&cpusets_enabled_key);
static_branch_dec_cpuslocked(&cpusets_pre_enable_key);
}
+/*
+ * This will get enabled whenever a cpuset configuration is considered
+ * unsupportable in general. E.g. movable only node which cannot satisfy
+ * any non movable allocations (see update_nodemask). Page allocator
+ * needs to make additional checks for those configurations and this
+ * check is meant to guard those checks without any overhead for sane
+ * configurations.
+ */
+static inline bool cpusets_insane_config(void)
+{
+ return static_branch_unlikely(&cpusets_insane_config_key);
+}
+
extern int cpuset_init(void);
extern void cpuset_init_smp(void);
extern void cpuset_force_rebuild(void);
static inline bool cpusets_enabled(void) { return false; }
+static inline bool cpusets_insane_config(void) { return false; }
+
static inline int cpuset_init(void) { return 0; }
static inline void cpuset_init_smp(void) {}
#include <linux/pgtable.h>
#include <uapi/linux/vmcore.h>
-#include <linux/pgtable.h> /* for pgprot_t */
-
/* For IS_ENABLED(CONFIG_CRASH_DUMP) */
#define ELFCORE_ADDR_MAX (-1ULL)
#define ELFCORE_ADDR_ERR (-2ULL)
elfcorehdr_addr = ELFCORE_ADDR_ERR;
}
-#define HAVE_OLDMEM_PFN_IS_RAM 1
-extern int register_oldmem_pfn_is_ram(int (*fn)(unsigned long pfn));
-extern void unregister_oldmem_pfn_is_ram(void);
+/**
+ * struct vmcore_cb - driver callbacks for /proc/vmcore handling
+ * @pfn_is_ram: check whether a PFN really is RAM and should be accessed when
+ * reading the vmcore. Will return "true" if it is RAM or if the
+ * callback cannot tell. If any callback returns "false", it's not
+ * RAM and the page must not be accessed; zeroes should be
+ * indicated in the vmcore instead. For example, a ballooned page
+ * contains no data and reading from such a page will cause high
+ * load in the hypervisor.
+ * @next: List head to manage registered callbacks internally; initialized by
+ * register_vmcore_cb().
+ *
+ * vmcore callbacks allow drivers managing physical memory ranges to
+ * coordinate with vmcore handling code, for example, to prevent accessing
+ * physical memory ranges that should not be accessed when reading the vmcore,
+ * although included in the vmcore header as memory ranges to dump.
+ */
+struct vmcore_cb {
+ bool (*pfn_is_ram)(struct vmcore_cb *cb, unsigned long pfn);
+ struct list_head next;
+};
+extern void register_vmcore_cb(struct vmcore_cb *cb);
+extern void unregister_vmcore_cb(struct vmcore_cb *cb);
#else /* !CONFIG_CRASH_DUMP */
-static inline bool is_kdump_kernel(void) { return 0; }
+static inline bool is_kdump_kernel(void) { return false; }
#endif /* CONFIG_CRASH_DUMP */
/* Device Dump information to be filled by drivers */
/* Minimal region size. Every damon_region is aligned by this. */
#define DAMON_MIN_REGION PAGE_SIZE
+/* Max priority score for DAMON-based operation schemes */
+#define DAMOS_MAX_SCORE (99)
/**
* struct damon_addr_range - Represents an address region of [@start, @end).
* @sampling_addr: Address of the sample for the next access check.
* @nr_accesses: Access frequency of this region.
* @list: List head for siblings.
+ * @age: Age of this region.
+ *
+ * @age is initially zero, increased for each aggregation interval, and reset
+ * to zero again if the access frequency is significantly changed. If two
+ * regions are merged into a new region, both @nr_accesses and @age of the new
+ * region are set as region size-weighted average of those of the two regions.
*/
struct damon_region {
struct damon_addr_range ar;
unsigned long sampling_addr;
unsigned int nr_accesses;
struct list_head list;
+
+ unsigned int age;
+/* private: Internal value for age calculation. */
+ unsigned int last_nr_accesses;
};
/**
struct list_head list;
};
+/**
+ * enum damos_action - Represents an action of a Data Access Monitoring-based
+ * Operation Scheme.
+ *
+ * @DAMOS_WILLNEED: Call ``madvise()`` for the region with MADV_WILLNEED.
+ * @DAMOS_COLD: Call ``madvise()`` for the region with MADV_COLD.
+ * @DAMOS_PAGEOUT: Call ``madvise()`` for the region with MADV_PAGEOUT.
+ * @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
+ * @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
+ * @DAMOS_STAT: Do nothing but count the stat.
+ */
+enum damos_action {
+ DAMOS_WILLNEED,
+ DAMOS_COLD,
+ DAMOS_PAGEOUT,
+ DAMOS_HUGEPAGE,
+ DAMOS_NOHUGEPAGE,
+ DAMOS_STAT, /* Do nothing but only record the stat */
+};
+
+/**
+ * struct damos_quota - Controls the aggressiveness of the given scheme.
+ * @ms: Maximum milliseconds that the scheme can use.
+ * @sz: Maximum bytes of memory that the action can be applied.
+ * @reset_interval: Charge reset interval in milliseconds.
+ *
+ * @weight_sz: Weight of the region's size for prioritization.
+ * @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.
+ * @weight_age: Weight of the region's age for prioritization.
+ *
+ * To avoid consuming too much CPU time or IO resources for applying the
+ * &struct damos->action to large memory, DAMON allows users to set time and/or
+ * size quotas. The quotas can be set by writing non-zero values to &ms and
+ * &sz, respectively. If the time quota is set, DAMON tries to use only up to
+ * &ms milliseconds within &reset_interval for applying the action. If the
+ * size quota is set, DAMON tries to apply the action only up to &sz bytes
+ * within &reset_interval.
+ *
+ * Internally, the time quota is transformed to a size quota using estimated
+ * throughput of the scheme's action. DAMON then compares it against &sz and
+ * uses smaller one as the effective quota.
+ *
+ * For selecting regions within the quota, DAMON prioritizes current scheme's
+ * target memory regions using the &struct damon_primitive->get_scheme_score.
+ * You could customize the prioritization logic by setting &weight_sz,
+ * &weight_nr_accesses, and &weight_age, because monitoring primitives are
+ * encouraged to respect those.
+ */
+struct damos_quota {
+ unsigned long ms;
+ unsigned long sz;
+ unsigned long reset_interval;
+
+ unsigned int weight_sz;
+ unsigned int weight_nr_accesses;
+ unsigned int weight_age;
+
+/* private: */
+ /* For throughput estimation */
+ unsigned long total_charged_sz;
+ unsigned long total_charged_ns;
+
+ unsigned long esz; /* Effective size quota in bytes */
+
+ /* For charging the quota */
+ unsigned long charged_sz;
+ unsigned long charged_from;
+ struct damon_target *charge_target_from;
+ unsigned long charge_addr_from;
+
+ /* For prioritization */
+ unsigned long histogram[DAMOS_MAX_SCORE + 1];
+ unsigned int min_score;
+};
+
+/**
+ * enum damos_wmark_metric - Represents the watermark metric.
+ *
+ * @DAMOS_WMARK_NONE: Ignore the watermarks of the given scheme.
+ * @DAMOS_WMARK_FREE_MEM_RATE: Free memory rate of the system in [0,1000].
+ */
+enum damos_wmark_metric {
+ DAMOS_WMARK_NONE,
+ DAMOS_WMARK_FREE_MEM_RATE,
+};
+
+/**
+ * struct damos_watermarks - Controls when a given scheme should be activated.
+ * @metric: Metric for the watermarks.
+ * @interval: Watermarks check time interval in microseconds.
+ * @high: High watermark.
+ * @mid: Middle watermark.
+ * @low: Low watermark.
+ *
+ * If &metric is &DAMOS_WMARK_NONE, the scheme is always active. Being active
+ * means DAMON does monitoring and applying the action of the scheme to
+ * appropriate memory regions. Else, DAMON checks &metric of the system for at
+ * least every &interval microseconds and works as below.
+ *
+ * If &metric is higher than &high, the scheme is inactivated. If &metric is
+ * between &mid and &low, the scheme is activated. If &metric is lower than
+ * &low, the scheme is inactivated.
+ */
+struct damos_watermarks {
+ enum damos_wmark_metric metric;
+ unsigned long interval;
+ unsigned long high;
+ unsigned long mid;
+ unsigned long low;
+
+/* private: */
+ bool activated;
+};
+
+/**
+ * struct damos - Represents a Data Access Monitoring-based Operation Scheme.
+ * @min_sz_region: Minimum size of target regions.
+ * @max_sz_region: Maximum size of target regions.
+ * @min_nr_accesses: Minimum ``->nr_accesses`` of target regions.
+ * @max_nr_accesses: Maximum ``->nr_accesses`` of target regions.
+ * @min_age_region: Minimum age of target regions.
+ * @max_age_region: Maximum age of target regions.
+ * @action: &damo_action to be applied to the target regions.
+ * @quota: Control the aggressiveness of this scheme.
+ * @wmarks: Watermarks for automated (in)activation of this scheme.
+ * @stat_count: Total number of regions that this scheme is applied.
+ * @stat_sz: Total size of regions that this scheme is applied.
+ * @list: List head for siblings.
+ *
+ * For each aggregation interval, DAMON finds regions which fit in the
+ * condition (&min_sz_region, &max_sz_region, &min_nr_accesses,
+ * &max_nr_accesses, &min_age_region, &max_age_region) and applies &action to
+ * those. To avoid consuming too much CPU time or IO resources for the
+ * &action, "a is used.
+ *
+ * To do the work only when needed, schemes can be activated for specific
+ * system situations using &wmarks. If all schemes that registered to the
+ * monitoring context are inactive, DAMON stops monitoring either, and just
+ * repeatedly checks the watermarks.
+ *
+ * If all schemes that registered to a &struct damon_ctx are inactive, DAMON
+ * stops monitoring and just repeatedly checks the watermarks.
+ *
+ * After applying the &action to each region, &stat_count and &stat_sz is
+ * updated to reflect the number of regions and total size of regions that the
+ * &action is applied.
+ */
+struct damos {
+ unsigned long min_sz_region;
+ unsigned long max_sz_region;
+ unsigned int min_nr_accesses;
+ unsigned int max_nr_accesses;
+ unsigned int min_age_region;
+ unsigned int max_age_region;
+ enum damos_action action;
+ struct damos_quota quota;
+ struct damos_watermarks wmarks;
+ unsigned long stat_count;
+ unsigned long stat_sz;
+ struct list_head list;
+};
+
struct damon_ctx;
/**
- * struct damon_primitive Monitoring primitives for given use cases.
+ * struct damon_primitive - Monitoring primitives for given use cases.
*
* @init: Initialize primitive-internal data structures.
* @update: Update primitive-internal data structures.
* @prepare_access_checks: Prepare next access check of target regions.
* @check_accesses: Check the accesses to target regions.
* @reset_aggregated: Reset aggregated accesses monitoring results.
+ * @get_scheme_score: Get the score of a region for a scheme.
+ * @apply_scheme: Apply a DAMON-based operation scheme.
* @target_valid: Determine if the target is valid.
* @cleanup: Clean up the context.
*
* of its update. The value will be used for regions adjustment threshold.
* @reset_aggregated should reset the access monitoring results that aggregated
* by @check_accesses.
+ * @get_scheme_score should return the priority score of a region for a scheme
+ * as an integer in [0, &DAMOS_MAX_SCORE].
+ * @apply_scheme is called from @kdamond when a region for user provided
+ * DAMON-based operation scheme is found. It should apply the scheme's action
+ * to the region. This is not used for &DAMON_ARBITRARY_TARGET case.
* @target_valid should check whether the target is still valid for the
* monitoring.
* @cleanup is called from @kdamond just before its termination.
void (*prepare_access_checks)(struct damon_ctx *context);
unsigned int (*check_accesses)(struct damon_ctx *context);
void (*reset_aggregated)(struct damon_ctx *context);
+ int (*get_scheme_score)(struct damon_ctx *context,
+ struct damon_target *t, struct damon_region *r,
+ struct damos *scheme);
+ int (*apply_scheme)(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme);
bool (*target_valid)(void *target);
void (*cleanup)(struct damon_ctx *context);
};
-/*
- * struct damon_callback Monitoring events notification callbacks.
+/**
+ * struct damon_callback - Monitoring events notification callbacks.
*
* @before_start: Called before starting the monitoring.
* @after_sampling: Called after each sampling.
int (*before_start)(struct damon_ctx *context);
int (*after_sampling)(struct damon_ctx *context);
int (*after_aggregation)(struct damon_ctx *context);
- int (*before_terminate)(struct damon_ctx *context);
+ void (*before_terminate)(struct damon_ctx *context);
};
/**
* @min_nr_regions: The minimum number of adaptive monitoring regions.
* @max_nr_regions: The maximum number of adaptive monitoring regions.
* @adaptive_targets: Head of monitoring targets (&damon_target) list.
+ * @schemes: Head of schemes (&damos) list.
*/
struct damon_ctx {
unsigned long sample_interval;
/* public: */
struct task_struct *kdamond;
- bool kdamond_stop;
struct mutex kdamond_lock;
struct damon_primitive primitive;
unsigned long min_nr_regions;
unsigned long max_nr_regions;
struct list_head adaptive_targets;
+ struct list_head schemes;
};
#define damon_next_region(r) \
#define damon_prev_region(r) \
(container_of(r->list.prev, struct damon_region, list))
+#define damon_last_region(t) \
+ (list_last_entry(&t->regions_list, struct damon_region, list))
+
#define damon_for_each_region(r, t) \
list_for_each_entry(r, &t->regions_list, list)
#define damon_for_each_target_safe(t, next, ctx) \
list_for_each_entry_safe(t, next, &(ctx)->adaptive_targets, list)
+#define damon_for_each_scheme(s, ctx) \
+ list_for_each_entry(s, &(ctx)->schemes, list)
+
+#define damon_for_each_scheme_safe(s, next, ctx) \
+ list_for_each_entry_safe(s, next, &(ctx)->schemes, list)
+
#ifdef CONFIG_DAMON
struct damon_region *damon_new_region(unsigned long start, unsigned long end);
void damon_add_region(struct damon_region *r, struct damon_target *t);
void damon_destroy_region(struct damon_region *r, struct damon_target *t);
+struct damos *damon_new_scheme(
+ unsigned long min_sz_region, unsigned long max_sz_region,
+ unsigned int min_nr_accesses, unsigned int max_nr_accesses,
+ unsigned int min_age_region, unsigned int max_age_region,
+ enum damos_action action, struct damos_quota *quota,
+ struct damos_watermarks *wmarks);
+void damon_add_scheme(struct damon_ctx *ctx, struct damos *s);
+void damon_destroy_scheme(struct damos *s);
+
struct damon_target *damon_new_target(unsigned long id);
void damon_add_target(struct damon_ctx *ctx, struct damon_target *t);
+bool damon_targets_empty(struct damon_ctx *ctx);
void damon_free_target(struct damon_target *t);
void damon_destroy_target(struct damon_target *t);
unsigned int damon_nr_regions(struct damon_target *t);
int damon_set_attrs(struct damon_ctx *ctx, unsigned long sample_int,
unsigned long aggr_int, unsigned long primitive_upd_int,
unsigned long min_nr_reg, unsigned long max_nr_reg);
+int damon_set_schemes(struct damon_ctx *ctx,
+ struct damos **schemes, ssize_t nr_schemes);
int damon_nr_running_ctxs(void);
int damon_start(struct damon_ctx **ctxs, int nr_ctxs);
unsigned int damon_va_check_accesses(struct damon_ctx *ctx);
bool damon_va_target_valid(void *t);
void damon_va_cleanup(struct damon_ctx *ctx);
+int damon_va_apply_scheme(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme);
+int damon_va_scheme_score(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme);
void damon_va_set_primitives(struct damon_ctx *ctx);
#endif /* CONFIG_DAMON_VADDR */
+#ifdef CONFIG_DAMON_PADDR
+
+/* Monitoring primitives for the physical memory address space */
+void damon_pa_prepare_access_checks(struct damon_ctx *ctx);
+unsigned int damon_pa_check_accesses(struct damon_ctx *ctx);
+bool damon_pa_target_valid(void *t);
+int damon_pa_apply_scheme(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme);
+int damon_pa_scheme_score(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme);
+void damon_pa_set_primitives(struct damon_ctx *ctx);
+
+#endif /* CONFIG_DAMON_PADDR */
+
#endif /* _DAMON_H */
int (*zero_page_range)(struct dax_device *, pgoff_t, size_t);
};
-extern struct attribute_group dax_attribute_group;
-
#if IS_ENABLED(CONFIG_DAX)
struct dax_device *alloc_dax(void *private, const char *host,
const struct dax_operations *ops, unsigned long flags);
* https://lists.openwall.net/linux-kernel/2011/01/09/56
*/
-#include <linux/kernel.h>
+#include <linux/math.h>
extern unsigned long loops_per_jiffy;
/** @index: index into the shared fences */
unsigned int index;
- /** @fences: the shared fences */
+ /** @fences: the shared fences; private, *MUST* not dereference */
struct dma_resv_list *fences;
+ /** @shared_count: number of shared fences */
+ unsigned int shared_count;
+
/** @is_restarted: true if this is the first returned fence */
bool is_restarted;
};
struct dma_fence *dma_resv_iter_first_unlocked(struct dma_resv_iter *cursor);
struct dma_fence *dma_resv_iter_next_unlocked(struct dma_resv_iter *cursor);
+struct dma_fence *dma_resv_iter_first(struct dma_resv_iter *cursor);
+struct dma_fence *dma_resv_iter_next(struct dma_resv_iter *cursor);
/**
* dma_resv_iter_begin - initialize a dma_resv_iter object
for (fence = dma_resv_iter_first_unlocked(cursor); \
fence; fence = dma_resv_iter_next_unlocked(cursor))
+/**
+ * dma_resv_for_each_fence - fence iterator
+ * @cursor: a struct dma_resv_iter pointer
+ * @obj: a dma_resv object pointer
+ * @all_fences: true if all fences should be returned
+ * @fence: the current fence
+ *
+ * Iterate over the fences in a struct dma_resv object while holding the
+ * &dma_resv.lock. @all_fences controls if the shared fences are returned as
+ * well. The cursor initialisation is part of the iterator and the fence stays
+ * valid as long as the lock is held and so no extra reference to the fence is
+ * taken.
+ */
+#define dma_resv_for_each_fence(cursor, obj, all_fences, fence) \
+ for (dma_resv_iter_begin(cursor, obj, all_fences), \
+ fence = dma_resv_iter_first(cursor); fence; \
+ fence = dma_resv_iter_next(cursor))
+
#define dma_resv_held(obj) lockdep_is_held(&(obj)->lock.base)
#define dma_resv_assert_held(obj) lockdep_assert_held(&(obj)->lock.base)
void (*device_issue_pending)(struct dma_chan *chan);
void (*device_release)(struct dma_device *dev);
/* debugfs support */
-#ifdef CONFIG_DEBUG_FS
void (*dbg_summary_show)(struct seq_file *s, struct dma_device *dev);
struct dentry *dbg_dev_root;
-#endif
};
static inline int dmaengine_slave_config(struct dma_chan *chan,
struct ocelot_skb_cb {
struct sk_buff *clone;
unsigned int ptp_class; /* valid only for clones */
+ u32 tstamp_lo;
u8 ptp_cmd;
u8 ts_id;
};
/* OEM GUIDs */
#define DELLEMC_EFI_RCI2_TABLE_GUID EFI_GUID(0x2d9f28a2, 0xa886, 0x456a, 0x97, 0xa8, 0xf1, 0x1e, 0xf2, 0x4f, 0xf4, 0x55)
+#define AMD_SEV_MEM_ENCRYPT_GUID EFI_GUID(0x0cf29b71, 0x9e51, 0x433a, 0xa3, 0xb7, 0x81, 0xf3, 0xab, 0x16, 0xb8, 0x75)
typedef struct {
efi_guid_t guid;
#define __ETHTOOL_LINK_MODE_MASK_NWORDS \
DIV_ROUND_UP(__ETHTOOL_LINK_MODE_MASK_NBITS, 32)
+#define ETHTOOL_PAUSE_STAT_CNT (__ETHTOOL_A_PAUSE_STAT_CNT - \
+ ETHTOOL_A_PAUSE_STAT_TX_FRAMES)
+
enum ethtool_multicast_groups {
ETHNL_MCGRP_MONITOR,
};
*/
#define FANOTIFY_DIRENT_EVENTS (FAN_MOVE | FAN_CREATE | FAN_DELETE)
+/* Events that can be reported with event->fd */
+#define FANOTIFY_FD_EVENTS (FANOTIFY_PATH_EVENTS | FANOTIFY_PERM_EVENTS)
+
/* Events that can only be reported with data type FSNOTIFY_EVENT_INODE */
#define FANOTIFY_INODE_EVENTS (FANOTIFY_DIRENT_EVENTS | \
FAN_ATTRIB | FAN_MOVE_SELF | FAN_DELETE_SELF)
+/* Events that can only be reported with data type FSNOTIFY_EVENT_ERROR */
+#define FANOTIFY_ERROR_EVENTS (FAN_FS_ERROR)
+
/* Events that user can request to be notified on */
#define FANOTIFY_EVENTS (FANOTIFY_PATH_EVENTS | \
- FANOTIFY_INODE_EVENTS)
+ FANOTIFY_INODE_EVENTS | \
+ FANOTIFY_ERROR_EVENTS)
/* Events that require a permission response from user */
#define FANOTIFY_PERM_EVENTS (FAN_OPEN_PERM | FAN_ACCESS_PERM | \
/* Draws a rectangle */
void (*fb_fillrect) (struct fb_info *info, const struct fb_fillrect *rect);
- /* Copy data from area to another */
+ /* Copy data from area to another. Obsolete. */
void (*fb_copyarea) (struct fb_info *info, const struct fb_copyarea *region);
/* Draws a image to the display */
void (*fb_imageblit) (struct fb_info *info, const struct fb_image *image);
PM_SET_REQUIREMENT = 15,
PM_RESET_ASSERT = 17,
PM_RESET_GET_STATUS = 18,
+ PM_MMIO_WRITE = 19,
+ PM_MMIO_READ = 20,
PM_PM_INIT_FINALIZE = 21,
PM_FPGA_LOAD = 22,
PM_FPGA_GET_STATUS = 23,
int zynqmp_pm_reset_assert(const enum zynqmp_pm_reset reset,
const enum zynqmp_pm_reset_action assert_flag);
int zynqmp_pm_reset_get_status(const enum zynqmp_pm_reset reset, u32 *status);
+unsigned int zynqmp_pm_bootmode_read(u32 *ps_mode);
+int zynqmp_pm_bootmode_write(u32 ps_mode);
int zynqmp_pm_init_finalize(void);
int zynqmp_pm_set_suspend_mode(u32 mode);
int zynqmp_pm_request_node(const u32 node, const u32 capabilities,
return -ENODEV;
}
+static inline unsigned int zynqmp_pm_bootmode_read(u32 *ps_mode)
+{
+ return -ENODEV;
+}
+
+static inline int zynqmp_pm_bootmode_write(u32 ps_mode)
+{
+ return -ENODEV;
+}
+
static inline int zynqmp_pm_init_finalize(void)
{
return -ENODEV;
#define SB_I_UNTRUSTED_MOUNTER 0x00000040
#define SB_I_SKIP_SYNC 0x00000100 /* Skip superblock at global sync */
+#define SB_I_PERSB_BDI 0x00000200 /* has a per-sb bdi */
/* Possible states of 'frozen' field */
enum {
}
extern void inode_sb_list_add(struct inode *inode);
+extern void inode_add_lru(struct inode *inode);
extern int sb_set_blocksize(struct super_block *, int);
extern int sb_min_blocksize(struct super_block *, int);
extern int simple_link(struct dentry *, struct inode *, struct dentry *);
extern int simple_unlink(struct inode *, struct dentry *);
extern int simple_rmdir(struct inode *, struct dentry *);
+extern int simple_rename_exchange(struct inode *old_dir, struct dentry *old_dentry,
+ struct inode *new_dir, struct dentry *new_dentry);
extern int simple_rename(struct user_namespace *, struct inode *,
struct dentry *, struct inode *, struct dentry *,
unsigned int);
static inline bool fscache_cookie_enabled(struct fscache_cookie *cookie)
{
- return test_bit(FSCACHE_COOKIE_ENABLED, &cookie->flags);
+ return fscache_cookie_valid(cookie) && test_bit(FSCACHE_COOKIE_ENABLED, &cookie->flags);
}
/*
* FS_EVENT_ON_CHILD mask on the parent inode and will not be reported if only
* the child is interested and not the parent.
*/
-static inline void fsnotify_name(struct inode *dir, __u32 mask,
- struct inode *child,
- const struct qstr *name, u32 cookie)
+static inline int fsnotify_name(__u32 mask, const void *data, int data_type,
+ struct inode *dir, const struct qstr *name,
+ u32 cookie)
{
if (atomic_long_read(&dir->i_sb->s_fsnotify_connectors) == 0)
- return;
+ return 0;
- fsnotify(mask, child, FSNOTIFY_EVENT_INODE, dir, name, NULL, cookie);
+ return fsnotify(mask, data, data_type, dir, name, NULL, cookie);
}
static inline void fsnotify_dirent(struct inode *dir, struct dentry *dentry,
__u32 mask)
{
- fsnotify_name(dir, mask, d_inode(dentry), &dentry->d_name, 0);
+ fsnotify_name(mask, dentry, FSNOTIFY_EVENT_DENTRY, dir, &dentry->d_name, 0);
}
static inline void fsnotify_inode(struct inode *inode, __u32 mask)
*/
static inline void fsnotify_dentry(struct dentry *dentry, __u32 mask)
{
- fsnotify_parent(dentry, mask, d_inode(dentry), FSNOTIFY_EVENT_INODE);
+ fsnotify_parent(dentry, mask, dentry, FSNOTIFY_EVENT_DENTRY);
}
static inline int fsnotify_file(struct file *file, __u32 mask)
new_dir_mask |= FS_ISDIR;
}
- fsnotify_name(old_dir, old_dir_mask, source, old_name, fs_cookie);
- fsnotify_name(new_dir, new_dir_mask, source, new_name, fs_cookie);
+ fsnotify_name(old_dir_mask, source, FSNOTIFY_EVENT_INODE,
+ old_dir, old_name, fs_cookie);
+ fsnotify_name(new_dir_mask, source, FSNOTIFY_EVENT_INODE,
+ new_dir, new_name, fs_cookie);
if (target)
fsnotify_link_count(target);
/*
* fsnotify_create - 'name' was linked in
+ *
+ * Caller must make sure that dentry->d_name is stable.
+ * Note: some filesystems (e.g. kernfs) leave @dentry negative and instantiate
+ * ->d_inode later
*/
-static inline void fsnotify_create(struct inode *inode, struct dentry *dentry)
+static inline void fsnotify_create(struct inode *dir, struct dentry *dentry)
{
- audit_inode_child(inode, dentry, AUDIT_TYPE_CHILD_CREATE);
+ audit_inode_child(dir, dentry, AUDIT_TYPE_CHILD_CREATE);
- fsnotify_dirent(inode, dentry, FS_CREATE);
+ fsnotify_dirent(dir, dentry, FS_CREATE);
}
/*
* fsnotify_link - new hardlink in 'inode' directory
+ *
+ * Caller must make sure that new_dentry->d_name is stable.
* Note: We have to pass also the linked inode ptr as some filesystems leave
* new_dentry->d_inode NULL and instantiate inode pointer later
*/
fsnotify_link_count(inode);
audit_inode_child(dir, new_dentry, AUDIT_TYPE_CHILD_CREATE);
- fsnotify_name(dir, FS_CREATE, inode, &new_dentry->d_name, 0);
+ fsnotify_name(FS_CREATE, inode, FSNOTIFY_EVENT_INODE,
+ dir, &new_dentry->d_name, 0);
}
/*
/*
* fsnotify_mkdir - directory 'name' was created
+ *
+ * Caller must make sure that dentry->d_name is stable.
+ * Note: some filesystems (e.g. kernfs) leave @dentry negative and instantiate
+ * ->d_inode later
*/
-static inline void fsnotify_mkdir(struct inode *inode, struct dentry *dentry)
+static inline void fsnotify_mkdir(struct inode *dir, struct dentry *dentry)
{
- audit_inode_child(inode, dentry, AUDIT_TYPE_CHILD_CREATE);
+ audit_inode_child(dir, dentry, AUDIT_TYPE_CHILD_CREATE);
- fsnotify_dirent(inode, dentry, FS_CREATE | FS_ISDIR);
+ fsnotify_dirent(dir, dentry, FS_CREATE | FS_ISDIR);
}
/*
fsnotify_dentry(dentry, mask);
}
+static inline int fsnotify_sb_error(struct super_block *sb, struct inode *inode,
+ int error)
+{
+ struct fs_error_report report = {
+ .error = error,
+ .inode = inode,
+ .sb = sb,
+ };
+
+ return fsnotify(FS_ERROR, &report, FSNOTIFY_EVENT_ERROR,
+ NULL, NULL, NULL, 0);
+}
+
#endif /* _LINUX_FS_NOTIFY_H */
#include <linux/atomic.h>
#include <linux/user_namespace.h>
#include <linux/refcount.h>
+#include <linux/mempool.h>
/*
* IN_* from inotfy.h lines up EXACTLY with FS_*, this is so we can easily
#define FS_UNMOUNT 0x00002000 /* inode on umount fs */
#define FS_Q_OVERFLOW 0x00004000 /* Event queued overflowed */
+#define FS_ERROR 0x00008000 /* Filesystem Error (fanotify) */
+
+/*
+ * FS_IN_IGNORED overloads FS_ERROR. It is only used internally by inotify
+ * which does not support FS_ERROR.
+ */
#define FS_IN_IGNORED 0x00008000 /* last inotify event here */
#define FS_OPEN_PERM 0x00010000 /* open event in an permission hook */
#define ALL_FSNOTIFY_EVENTS (ALL_FSNOTIFY_DIRENT_EVENTS | \
FS_EVENTS_POSS_ON_CHILD | \
FS_DELETE_SELF | FS_MOVE_SELF | FS_DN_RENAME | \
- FS_UNMOUNT | FS_Q_OVERFLOW | FS_IN_IGNORED)
+ FS_UNMOUNT | FS_Q_OVERFLOW | FS_IN_IGNORED | \
+ FS_ERROR)
/* Extra flags that may be reported with event or control handling of events */
#define ALL_FSNOTIFY_FLAGS (FS_EXCL_UNLINK | FS_ISDIR | FS_IN_ONESHOT | \
* @dir: optional directory associated with event -
* if @file_name is not NULL, this is the directory that
* @file_name is relative to.
+ * Either @inode or @dir must be non-NULL.
* @file_name: optional file name associated with event
* @cookie: inotify rename cookie
*
const struct qstr *file_name, u32 cookie);
void (*free_group_priv)(struct fsnotify_group *group);
void (*freeing_mark)(struct fsnotify_mark *mark, struct fsnotify_group *group);
- void (*free_event)(struct fsnotify_event *event);
+ void (*free_event)(struct fsnotify_group *group, struct fsnotify_event *event);
/* called on final put+free to free memory */
void (*free_mark)(struct fsnotify_mark *mark);
};
int flags; /* flags from fanotify_init() */
int f_flags; /* event_f_flags from fanotify_init() */
struct ucounts *ucounts;
+ mempool_t error_events_pool;
} fanotify_data;
#endif /* CONFIG_FANOTIFY */
};
FSNOTIFY_EVENT_NONE,
FSNOTIFY_EVENT_PATH,
FSNOTIFY_EVENT_INODE,
+ FSNOTIFY_EVENT_DENTRY,
+ FSNOTIFY_EVENT_ERROR,
+};
+
+struct fs_error_report {
+ int error;
+ struct inode *inode;
+ struct super_block *sb;
};
static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
switch (data_type) {
case FSNOTIFY_EVENT_INODE:
return (struct inode *)data;
+ case FSNOTIFY_EVENT_DENTRY:
+ return d_inode(data);
case FSNOTIFY_EVENT_PATH:
return d_inode(((const struct path *)data)->dentry);
+ case FSNOTIFY_EVENT_ERROR:
+ return ((struct fs_error_report *)data)->inode;
+ default:
+ return NULL;
+ }
+}
+
+static inline struct dentry *fsnotify_data_dentry(const void *data, int data_type)
+{
+ switch (data_type) {
+ case FSNOTIFY_EVENT_DENTRY:
+ /* Non const is needed for dget() */
+ return (struct dentry *)data;
+ case FSNOTIFY_EVENT_PATH:
+ return ((const struct path *)data)->dentry;
default:
return NULL;
}
}
}
+static inline struct super_block *fsnotify_data_sb(const void *data,
+ int data_type)
+{
+ switch (data_type) {
+ case FSNOTIFY_EVENT_INODE:
+ return ((struct inode *)data)->i_sb;
+ case FSNOTIFY_EVENT_DENTRY:
+ return ((struct dentry *)data)->d_sb;
+ case FSNOTIFY_EVENT_PATH:
+ return ((const struct path *)data)->dentry->d_sb;
+ case FSNOTIFY_EVENT_ERROR:
+ return ((struct fs_error_report *) data)->sb;
+ default:
+ return NULL;
+ }
+}
+
+static inline struct fs_error_report *fsnotify_data_error_report(
+ const void *data,
+ int data_type)
+{
+ switch (data_type) {
+ case FSNOTIFY_EVENT_ERROR:
+ return (struct fs_error_report *) data;
+ default:
+ return NULL;
+ }
+}
+
enum fsnotify_obj_type {
FSNOTIFY_OBJ_TYPE_INODE,
FSNOTIFY_OBJ_TYPE_PARENT,
extern void fsnotify_destroy_event(struct fsnotify_group *group,
struct fsnotify_event *event);
/* attach the event to the group notification queue */
-extern int fsnotify_add_event(struct fsnotify_group *group,
- struct fsnotify_event *event,
- int (*merge)(struct fsnotify_group *,
- struct fsnotify_event *),
- void (*insert)(struct fsnotify_group *,
- struct fsnotify_event *));
+extern int fsnotify_insert_event(struct fsnotify_group *group,
+ struct fsnotify_event *event,
+ int (*merge)(struct fsnotify_group *,
+ struct fsnotify_event *),
+ void (*insert)(struct fsnotify_group *,
+ struct fsnotify_event *));
+
+static inline int fsnotify_add_event(struct fsnotify_group *group,
+ struct fsnotify_event *event,
+ int (*merge)(struct fsnotify_group *,
+ struct fsnotify_event *))
+{
+ return fsnotify_insert_event(group, event, merge, NULL);
+}
+
/* Queue overflow event to a notification group */
static inline void fsnotify_queue_overflow(struct fsnotify_group *group)
{
- fsnotify_add_event(group, group->overflow_event, NULL, NULL);
+ fsnotify_add_event(group, group->overflow_event, NULL);
+}
+
+static inline bool fsnotify_is_overflow_event(u32 mask)
+{
+ return mask & FS_Q_OVERFLOW;
}
static inline bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group)
#include <asm/page.h>
#include <linux/bug.h>
-#include <linux/kernel.h>
#include <linux/log2.h>
+#include <linux/math.h>
+#include <linux/types.h>
struct genradix_root;
void disk_uevent(struct gendisk *disk, enum kobject_action action);
/* block/genhd.c */
-int device_add_disk(struct device *parent, struct gendisk *disk,
- const struct attribute_group **groups);
-static inline int add_disk(struct gendisk *disk)
+int __must_check device_add_disk(struct device *parent, struct gendisk *disk,
+ const struct attribute_group **groups);
+static inline int __must_check add_disk(struct gendisk *disk)
{
return device_add_disk(NULL, disk, NULL);
}
static inline loff_t bdev_nr_bytes(struct block_device *bdev)
{
- return bdev_nr_sectors(bdev) << SECTOR_SHIFT;
+ return (loff_t)bdev_nr_sectors(bdev) << SECTOR_SHIFT;
}
static inline sector_t get_capacity(struct gendisk *disk)
struct list_head *page_list,
struct page **page_array);
+unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
+ unsigned long nr_pages,
+ struct page **page_array);
+
/* Bulk allocate order-0 pages */
static inline unsigned long
alloc_pages_bulk_list(gfp_t gfp, unsigned long nr_pages, struct list_head *list)
extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order);
extern unsigned long get_zeroed_page(gfp_t gfp_mask);
-void *alloc_pages_exact(size_t size, gfp_t gfp_mask);
+void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1);
void free_pages_exact(void *virt, size_t size);
-void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask);
+__meminit void *alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask) __alloc_size(1);
#define __get_free_page(gfp_mask) \
__get_free_pages((gfp_mask), 0)
#define HID_DG_TOUCH 0x000d0033
#define HID_DG_UNTOUCH 0x000d0034
#define HID_DG_TAP 0x000d0035
+#define HID_DG_TRANSDUCER_INDEX 0x000d0038
#define HID_DG_TABLETFUNCTIONKEY 0x000d0039
#define HID_DG_PROGRAMCHANGEKEY 0x000d003a
#define HID_DG_BATTERYSTRENGTH 0x000d003b
#define HID_DG_BARRELSWITCH 0x000d0044
#define HID_DG_ERASER 0x000d0045
#define HID_DG_TABLETPICK 0x000d0046
+#define HID_DG_PEN_COLOR 0x000d005c
+#define HID_DG_PEN_LINE_WIDTH 0x000d005e
+#define HID_DG_PEN_LINE_STYLE 0x000d0070
+#define HID_DG_PEN_LINE_STYLE_INK 0x000d0072
+#define HID_DG_PEN_LINE_STYLE_PENCIL 0x000d0073
+#define HID_DG_PEN_LINE_STYLE_HIGHLIGHTER 0x000d0074
+#define HID_DG_PEN_LINE_STYLE_CHISEL_MARKER 0x000d0075
+#define HID_DG_PEN_LINE_STYLE_BRUSH 0x000d0076
+#define HID_DG_PEN_LINE_STYLE_NO_PREFERENCE 0x000d0077
#define HID_CP_CONSUMERCONTROL 0x000c0001
#define HID_CP_NUMERICKEYPAD 0x000c0002
/* BIT(9) reserved for backward compatibility, was NO_INIT_INPUT_REPORTS */
#define HID_QUIRK_ALWAYS_POLL BIT(10)
#define HID_QUIRK_INPUT_PER_APP BIT(11)
+#define HID_QUIRK_X_INVERT BIT(12)
+#define HID_QUIRK_Y_INVERT BIT(13)
#define HID_QUIRK_SKIP_OUTPUT_REPORTS BIT(16)
#define HID_QUIRK_SKIP_OUTPUT_REPORT_ID BIT(17)
#define HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP BIT(18)
container_of(pdrv, struct hid_driver, driver)
/**
- * hid_ll_driver - low level driver callbacks
+ * struct hid_ll_driver - low level driver callbacks
* @start: called on probe to start the device
* @stop: called on remove
* @open: called by input layer on open
return hdev->ll_driver == driver;
}
+static inline bool hid_is_usb(struct hid_device *hdev)
+{
+ return hid_is_using_ll_driver(hdev, &usb_hid_driver);
+}
+
#define PM_HINT_FULLON 1<<5
#define PM_HINT_NORMAL 1<<1
int hid_set_field(struct hid_field *, unsigned, __s32);
int hid_input_report(struct hid_device *, int type, u8 *, u32, int);
-int hidinput_find_field(struct hid_device *hid, unsigned int type, unsigned int code, struct hid_field **field);
struct hid_field *hidinput_get_led_field(struct hid_device *hid);
unsigned int hidinput_count_leds(struct hid_device *hid);
__s32 hidinput_calc_abs_res(const struct hid_field *field, __u16 code);
__u32 hid_field_extract(const struct hid_device *hid, __u8 *report,
unsigned offset, unsigned n);
+#ifdef CONFIG_PM
+int hid_driver_suspend(struct hid_device *hdev, pm_message_t state);
+int hid_driver_reset_resume(struct hid_device *hdev);
+int hid_driver_resume(struct hid_device *hdev);
+#else
+static inline int hid_driver_suspend(struct hid_device *hdev, pm_message_t state) { return 0; }
+static inline int hid_driver_reset_resume(struct hid_device *hdev) { return 0; }
+static inline int hid_driver_resume(struct hid_device *hdev) { return 0; }
+#endif
+
/**
* hid_device_io_start - enable HID input during probe, remove
*
bmap = input->ledbit;
limit = LED_MAX;
break;
+ case EV_MSC:
+ bmap = input->mscbit;
+ limit = MSC_MAX;
+ break;
}
if (unlikely(c > limit || !bmap)) {
void hid_hw_stop(struct hid_device *hdev);
int __must_check hid_hw_open(struct hid_device *hdev);
void hid_hw_close(struct hid_device *hdev);
+void hid_hw_request(struct hid_device *hdev,
+ struct hid_report *report, int reqtype);
+int hid_hw_raw_request(struct hid_device *hdev,
+ unsigned char reportnum, __u8 *buf,
+ size_t len, unsigned char rtype, int reqtype);
+int hid_hw_output_report(struct hid_device *hdev, __u8 *buf, size_t len);
/**
* hid_hw_power - requests underlying HW to go into given power mode
/**
- * hid_hw_request - send report request to device
- *
- * @hdev: hid device
- * @report: report to send
- * @reqtype: hid request type
- */
-static inline void hid_hw_request(struct hid_device *hdev,
- struct hid_report *report, int reqtype)
-{
- if (hdev->ll_driver->request)
- return hdev->ll_driver->request(hdev, report, reqtype);
-
- __hid_request(hdev, report, reqtype);
-}
-
-/**
- * hid_hw_raw_request - send report request to device
- *
- * @hdev: hid device
- * @reportnum: report ID
- * @buf: in/out data to transfer
- * @len: length of buf
- * @rtype: HID report type
- * @reqtype: HID_REQ_GET_REPORT or HID_REQ_SET_REPORT
- *
- * Return: count of data transferred, negative if error
- *
- * Same behavior as hid_hw_request, but with raw buffers instead.
- */
-static inline int hid_hw_raw_request(struct hid_device *hdev,
- unsigned char reportnum, __u8 *buf,
- size_t len, unsigned char rtype, int reqtype)
-{
- if (len < 1 || len > HID_MAX_BUFFER_SIZE || !buf)
- return -EINVAL;
-
- return hdev->ll_driver->raw_request(hdev, reportnum, buf, len,
- rtype, reqtype);
-}
-
-/**
- * hid_hw_output_report - send output report to device
- *
- * @hdev: hid device
- * @buf: raw data to transfer
- * @len: length of buf
- *
- * Return: count of data transferred, negative if error
- */
-static inline int hid_hw_output_report(struct hid_device *hdev, __u8 *buf,
- size_t len)
-{
- if (len < 1 || len > HID_MAX_BUFFER_SIZE || !buf)
- return -EINVAL;
-
- if (hdev->ll_driver->output_report)
- return hdev->ll_driver->output_report(hdev, buf, len);
-
- return -ENOSYS;
-}
-
-/**
* hid_hw_idle - send idle request to device
*
* @hdev: hid device
}
/**
- * hid_may_wakeup - return if the hid device may act as a wakeup source during system-suspend
+ * hid_hw_may_wakeup - return if the hid device may act as a wakeup source during system-suspend
*
* @hdev: hid device
*/
#ifndef clear_user_highpage
static inline void clear_user_highpage(struct page *page, unsigned long vaddr)
{
- void *addr = kmap_atomic(page);
+ void *addr = kmap_local_page(page);
clear_user_page(addr, vaddr, page);
- kunmap_atomic(addr);
+ kunmap_local(addr);
}
#endif
static inline void clear_highpage(struct page *page)
{
- void *kaddr = kmap_atomic(page);
+ void *kaddr = kmap_local_page(page);
clear_page(kaddr);
- kunmap_atomic(kaddr);
+ kunmap_local(kaddr);
}
#ifndef __HAVE_ARCH_TAG_CLEAR_HIGHPAGE
unsigned start1, unsigned end1,
unsigned start2, unsigned end2)
{
- void *kaddr = kmap_atomic(page);
+ void *kaddr = kmap_local_page(page);
unsigned int i;
BUG_ON(end1 > page_size(page) || end2 > page_size(page));
if (end2 > start2)
memset(kaddr + start2, 0, end2 - start2);
- kunmap_atomic(kaddr);
+ kunmap_local(kaddr);
for (i = 0; i < compound_nr(page); i++)
flush_dcache_page(page + i);
}
{
char *vfrom, *vto;
- vfrom = kmap_atomic(from);
- vto = kmap_atomic(to);
+ vfrom = kmap_local_page(from);
+ vto = kmap_local_page(to);
copy_user_page(vto, vfrom, vaddr, to);
- kunmap_atomic(vto);
- kunmap_atomic(vfrom);
+ kunmap_local(vto);
+ kunmap_local(vfrom);
}
#endif
{
char *vfrom, *vto;
- vfrom = kmap_atomic(from);
- vto = kmap_atomic(to);
+ vfrom = kmap_local_page(from);
+ vto = kmap_local_page(to);
copy_page(vto, vfrom);
- kunmap_atomic(vto);
- kunmap_atomic(vfrom);
+ kunmap_local(vto);
+ kunmap_local(vfrom);
}
#endif
void hugepage_put_subpool(struct hugepage_subpool *spool);
void reset_vma_resv_huge_pages(struct vm_area_struct *vma);
+void clear_vma_resv_huge_pages(struct vm_area_struct *vma);
int hugetlb_sysctl_handler(struct ctl_table *, int, void *, size_t *, loff_t *);
int hugetlb_overcommit_handler(struct ctl_table *, int, void *, size_t *,
loff_t *);
int hugetlb_mempolicy_sysctl_handler(struct ctl_table *, int, void *, size_t *,
loff_t *);
+int move_hugetlb_page_tables(struct vm_area_struct *vma,
+ struct vm_area_struct *new_vma,
+ unsigned long old_addr, unsigned long new_addr,
+ unsigned long len);
int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *);
long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *,
struct page **, struct vm_area_struct **,
struct vm_area_struct *vma,
unsigned long start, unsigned long end,
struct page *ref_page);
-void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long start, unsigned long end,
- struct page *ref_page);
void hugetlb_report_meminfo(struct seq_file *);
int hugetlb_report_node_meminfo(char *buf, int len, int nid);
void hugetlb_show_meminfo(void);
{
}
+static inline void clear_vma_resv_huge_pages(struct vm_area_struct *vma)
+{
+}
+
static inline unsigned long hugetlb_total_pages(void)
{
return 0;
return 0;
}
+static inline int move_hugetlb_page_tables(struct vm_area_struct *vma,
+ struct vm_area_struct *new_vma,
+ unsigned long old_addr,
+ unsigned long new_addr,
+ unsigned long len)
+{
+ BUG();
+ return 0;
+}
+
static inline void hugetlb_report_meminfo(struct seq_file *m)
{
}
BUG();
}
-static inline void __unmap_hugepage_range(struct mmu_gather *tlb,
- struct vm_area_struct *vma, unsigned long start,
- unsigned long end, struct page *ref_page)
-{
- BUG();
-}
-
static inline vm_fault_t hugetlb_fault(struct mm_struct *mm,
struct vm_area_struct *vma, unsigned long address,
unsigned int flags)
extern const struct file_operations hugetlbfs_file_operations;
extern const struct vm_operations_struct hugetlb_vm_ops;
struct file *hugetlb_file_setup(const char *name, size_t size, vm_flags_t acct,
- struct ucounts **ucounts, int creat_flags,
- int page_size_log);
+ int creat_flags, int page_size_log);
static inline bool is_file_hugepages(struct file *file)
{
#define is_file_hugepages(file) false
static inline struct file *
hugetlb_file_setup(const char *name, size_t size, vm_flags_t acctflag,
- struct ucounts **ucounts, int creat_flags,
- int page_size_log)
+ int creat_flags, int page_size_log)
{
return ERR_PTR(-ENOSYS);
}
int next_nid_to_alloc;
int next_nid_to_free;
unsigned int order;
+ unsigned int demote_order;
unsigned long mask;
unsigned long max_huge_pages;
unsigned long nr_huge_pages;
unsigned long nr_overcommit_huge_pages;
struct list_head hugepage_activelist;
struct list_head hugepage_freelists[MAX_NUMNODES];
+ unsigned int max_huge_pages_node[MAX_NUMNODES];
unsigned int nr_huge_pages_node[MAX_NUMNODES];
unsigned int free_huge_pages_node[MAX_NUMNODES];
unsigned int surplus_huge_pages_node[MAX_NUMNODES];
unsigned long address, struct page *page);
/* arch callback */
-int __init __alloc_bootmem_huge_page(struct hstate *h);
-int __init alloc_bootmem_huge_page(struct hstate *h);
+int __init __alloc_bootmem_huge_page(struct hstate *h, int nid);
+int __init alloc_bootmem_huge_page(struct hstate *h, int nid);
+bool __init hugetlb_node_alloc_supported(void);
void __init hugetlb_add_hstate(unsigned order);
bool __init arch_hugetlb_valid_size(unsigned long size);
css_get(resv_map->css);
}
+static inline void resv_map_put_hugetlb_cgroup_uncharge_info(
+ struct resv_map *resv_map)
+{
+ if (resv_map->css)
+ css_put(resv_map->css);
+}
+
extern int hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages,
struct hugetlb_cgroup **ptr);
extern int hugetlb_cgroup_charge_cgroup_rsvd(int idx, unsigned long nr_pages,
{
}
+static inline void resv_map_put_hugetlb_cgroup_uncharge_info(
+ struct resv_map *resv_map)
+{
+}
+
static inline int hugetlb_cgroup_charge_cgroup(int idx, unsigned long nr_pages,
struct hugetlb_cgroup **ptr)
{
#define _LINUX_I2C_H
#include <linux/acpi.h> /* for acpi_handle */
+#include <linux/bits.h>
#include <linux/mod_devicetable.h>
#include <linux/device.h> /* for struct device */
#include <linux/sched.h> /* for completion */
};
/**
+ * enum i2c_driver_flags - Flags for an I2C device driver
+ *
+ * @I2C_DRV_ACPI_WAIVE_D0_PROBE: Don't put the device in D0 state for probe
+ */
+enum i2c_driver_flags {
+ I2C_DRV_ACPI_WAIVE_D0_PROBE = BIT(0),
+};
+
+/**
* struct i2c_driver - represent an I2C device driver
* @class: What kind of i2c device we instantiate (for detect)
* @probe: Callback for device binding - soon to be deprecated
* @detect: Callback for device detection
* @address_list: The I2C addresses to probe (for detect)
* @clients: List of detected clients we created (for i2c-core use only)
+ * @flags: A bitmask of flags defined in &enum i2c_driver_flags
*
* The driver.owner field should be set to the module owner of this driver.
* The driver.name field should be set to the name of this driver.
int (*detect)(struct i2c_client *client, struct i2c_board_info *info);
const unsigned short *address_list;
struct list_head clients;
+
+ u32 flags;
};
#define to_i2c_driver(d) container_of(d, struct i2c_driver, driver)
struct i2c_client *i2c_acpi_new_device(struct device *dev, int index,
struct i2c_board_info *info);
struct i2c_adapter *i2c_acpi_find_adapter_by_handle(acpi_handle handle);
+bool i2c_acpi_waive_d0_probe(struct device *dev);
#else
static inline bool i2c_acpi_get_i2c_resource(struct acpi_resource *ares,
struct acpi_resource_i2c_serialbus **i2c)
{
return NULL;
}
+static inline bool i2c_acpi_waive_d0_probe(struct device *dev)
+{
+ return false;
+}
#endif /* CONFIG_ACPI */
#endif /* _LINUX_I2C_H */
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LINUX_CY8CTMG110_PDATA_H
-#define _LINUX_CY8CTMG110_PDATA_H
-
-struct cy8ctmg110_pdata
-{
- int reset_pin; /* Reset pin is wired to this GPIO (optional) */
-};
-
-#endif
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_INSTRUCTION_POINTER_H
+#define _LINUX_INSTRUCTION_POINTER_H
+
+#define _RET_IP_ (unsigned long)__builtin_return_address(0)
+#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
+
+#endif /* _LINUX_INSTRUCTION_POINTER_H */
#define _INTEL_ISH_CLIENT_IF_H_
#include <linux/device.h>
-#include <linux/uuid.h>
+#include <linux/mod_devicetable.h>
struct ishtp_cl_device;
struct ishtp_device;
struct ishtp_cl_driver {
struct device_driver driver;
const char *name;
- const guid_t *guid;
+ const struct ishtp_device_id *id;
int (*probe)(struct ishtp_cl_device *dev);
void (*remove)(struct ishtp_cl_device *dev);
int (*reset)(struct ishtp_cl_device *dev);
iomap->base = base;
iomap->size = size;
-#if defined(pgprot_noncached_wc) /* archs can't agree on a name ... */
- iomap->prot = pgprot_noncached_wc(PAGE_KERNEL);
-#elif defined(pgprot_writecombine)
iomap->prot = pgprot_writecombine(PAGE_KERNEL);
-#else
- iomap->prot = pgprot_noncached(PAGE_KERNEL);
-#endif
return iomap;
}
return ns;
}
+static inline struct ipc_namespace *get_ipc_ns_not_zero(struct ipc_namespace *ns)
+{
+ if (ns) {
+ if (refcount_inc_not_zero(&ns->ns.count))
+ return ns;
+ }
+
+ return NULL;
+}
+
extern void put_ipc_ns(struct ipc_namespace *ns);
#else
static inline struct ipc_namespace *copy_ipcs(unsigned long flags,
return ns;
}
+static inline struct ipc_namespace *get_ipc_ns_not_zero(struct ipc_namespace *ns)
+{
+ return ns;
+}
+
static inline void put_ipc_ns(struct ipc_namespace *ns)
{
}
u32 param[IRQ_DOMAIN_IRQ_SPEC_PARAMS];
};
+/* Conversion function from of_phandle_args fields to fwspec */
+void of_phandle_args_to_fwspec(struct device_node *np, const u32 *args,
+ unsigned int count, struct irq_fwspec *fwspec);
+
/*
* Should several domains have the same device node, but serve
* different purposes (for example one domain is for PCI/MSI, and the
struct cred;
struct module;
-static inline int is_kernel_inittext(unsigned long addr)
-{
- if (addr >= (unsigned long)_sinittext
- && addr <= (unsigned long)_einittext)
- return 1;
- return 0;
-}
-
static inline int is_kernel_text(unsigned long addr)
{
- if ((addr >= (unsigned long)_stext && addr <= (unsigned long)_etext) ||
- arch_is_kernel_text(addr))
+ if (__is_kernel_text(addr))
return 1;
return in_gate_area_no_mm(addr);
}
static inline int is_kernel(unsigned long addr)
{
- if (addr >= (unsigned long)_stext && addr <= (unsigned long)_end)
+ if (__is_kernel(addr))
return 1;
return in_gate_area_no_mm(addr);
}
void kasan_cache_shrink(struct kmem_cache *cache);
void kasan_cache_shutdown(struct kmem_cache *cache);
void kasan_record_aux_stack(void *ptr);
+void kasan_record_aux_stack_noalloc(void *ptr);
#else /* CONFIG_KASAN_GENERIC */
static inline void kasan_cache_shrink(struct kmem_cache *cache) {}
static inline void kasan_cache_shutdown(struct kmem_cache *cache) {}
static inline void kasan_record_aux_stack(void *ptr) {}
+static inline void kasan_record_aux_stack_noalloc(void *ptr) {}
#endif /* CONFIG_KASAN_GENERIC */
unsigned long free_region_start,
unsigned long free_region_end);
+void kasan_populate_early_vm_area_shadow(void *start, unsigned long size);
+
#else /* CONFIG_KASAN_VMALLOC */
static inline int kasan_populate_vmalloc(unsigned long start,
unsigned long free_region_start,
unsigned long free_region_end) {}
+static inline void kasan_populate_early_vm_area_shadow(void *start,
+ unsigned long size)
+{ }
+
#endif /* CONFIG_KASAN_VMALLOC */
#if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
/* Scoped access information. */
struct kcsan_scoped_access {
struct list_head list;
+ /* Access information. */
const volatile void *ptr;
size_t size;
int type;
+ /* Location where scoped access was set up. */
+ unsigned long ip;
};
/*
* Automatically call kcsan_end_scoped_access() when kcsan_scoped_access goes
#include <linux/stddef.h>
#include <linux/types.h>
#include <linux/compiler.h>
+#include <linux/container_of.h>
#include <linux/bitops.h>
#include <linux/kstrtox.h>
#include <linux/log2.h>
#include <linux/printk.h>
#include <linux/build_bug.h>
#include <linux/static_call_types.h>
+#include <linux/instruction_pointer.h>
#include <asm/byteorder.h>
#include <uapi/linux/kernel.h>
} \
)
-#define typeof_member(T, m) typeof(((T*)0)->m)
-
-#define _RET_IP_ (unsigned long)__builtin_return_address(0)
-#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
-
/**
* upper_32_bits - return bits 32-63 of a number
* @n: the number we're accessing
struct completion;
struct user;
-#ifdef CONFIG_PREEMPT_VOLUNTARY
+#ifdef CONFIG_PREEMPT_VOLUNTARY_BUILD
extern int __cond_resched(void);
# define might_resched() __cond_resched()
extern char *next_arg(char *args, char **param, char **val);
extern int core_kernel_text(unsigned long addr);
-extern int init_kernel_text(unsigned long addr);
-extern int core_kernel_data(unsigned long addr);
extern int __kernel_text_address(unsigned long addr);
extern int kernel_text_address(unsigned long addr);
extern int func_ptr_is_kernel_text(void *ptr);
extern enum system_states {
SYSTEM_BOOTING,
SYSTEM_SCHEDULING,
+ SYSTEM_FREEING_INITMEM,
SYSTEM_RUNNING,
SYSTEM_HALT,
SYSTEM_POWER_OFF,
#define __CONCAT(a, b) a ## b
#define CONCATENATE(a, b) __CONCAT(a, b)
-/**
- * container_of - cast a member of a structure out to the containing structure
- * @ptr: the pointer to the member.
- * @type: the type of the container struct this is embedded in.
- * @member: the name of the member within the struct.
- *
- */
-#define container_of(ptr, type, member) ({ \
- void *__mptr = (void *)(ptr); \
- BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
- !__same_type(*(ptr), void), \
- "pointer type mismatch in container_of()"); \
- ((type *)(__mptr - offsetof(type, member))); })
-
-/**
- * container_of_safe - cast a member of a structure out to the containing structure
- * @ptr: the pointer to the member.
- * @type: the type of the container struct this is embedded in.
- * @member: the name of the member within the struct.
- *
- * If IS_ERR_OR_NULL(ptr), ptr is returned unchanged.
- */
-#define container_of_safe(ptr, type, member) ({ \
- void *__mptr = (void *)(ptr); \
- BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
- !__same_type(*(ptr), void), \
- "pointer type mismatch in container_of()"); \
- IS_ERR_OR_NULL(__mptr) ? ERR_CAST(__mptr) : \
- ((type *)(__mptr - offsetof(type, member))); })
-
/* Rebuild everything on CONFIG_FTRACE_MCOUNT_RECORD */
#ifdef CONFIG_FTRACE_MCOUNT_RECORD
# define REBUILD_DUE_TO_FTRACE_MCOUNT_RECORD
#ifdef CONFIG_KFENCE
+#include <linux/atomic.h>
+#include <linux/static_key.h>
+
/*
* We allocate an even number of pages, as it simplifies calculations to map
* address to metadata indices; effectively, the very first page serves as an
#define KFENCE_POOL_SIZE ((CONFIG_KFENCE_NUM_OBJECTS + 1) * 2 * PAGE_SIZE)
extern char *__kfence_pool;
-#ifdef CONFIG_KFENCE_STATIC_KEYS
-#include <linux/static_key.h>
DECLARE_STATIC_KEY_FALSE(kfence_allocation_key);
-#else
-#include <linux/atomic.h>
extern atomic_t kfence_allocation_gate;
-#endif
/**
* is_kfence_address() - check if an address belongs to KFENCE pool
*/
static __always_inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags)
{
-#ifdef CONFIG_KFENCE_STATIC_KEYS
- if (static_branch_unlikely(&kfence_allocation_key))
+#if defined(CONFIG_KFENCE_STATIC_KEYS) || CONFIG_KFENCE_SAMPLE_INTERVAL == 0
+ if (!static_branch_unlikely(&kfence_allocation_key))
+ return NULL;
#else
- if (unlikely(!atomic_read(&kfence_allocation_gate)))
+ if (!static_branch_likely(&kfence_allocation_key))
+ return NULL;
#endif
- return __kfence_alloc(s, size, flags);
- return NULL;
+ if (likely(atomic_read(&kfence_allocation_gate)))
+ return NULL;
+ return __kfence_alloc(s, size, flags);
}
/**
#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_UNBLOCK 2
#define KVM_REQ_UNHALT 3
-#define KVM_REQ_VM_BUGGED (4 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_VM_DEAD (4 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQUEST_ARCH_BASE 8
#define KVM_ARCH_REQ_FLAGS(nr, flags) ({ \
unsigned int max_halt_poll_ns;
u32 dirty_ring_size;
bool vm_bugged;
+ bool vm_dead;
#ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
struct notifier_block pm_notifier;
#define vcpu_err(vcpu, fmt, ...) \
kvm_err("vcpu%i " fmt, (vcpu)->vcpu_id, ## __VA_ARGS__)
+static inline void kvm_vm_dead(struct kvm *kvm)
+{
+ kvm->vm_dead = true;
+ kvm_make_all_cpus_request(kvm, KVM_REQ_VM_DEAD);
+}
+
static inline void kvm_vm_bugged(struct kvm *kvm)
{
kvm->vm_bugged = true;
- kvm_make_all_cpus_request(kvm, KVM_REQ_VM_BUGGED);
+ kvm_vm_dead(kvm);
}
+
#define KVM_BUG(cond, kvm, fmt...) \
({ \
int __ret = (cond); \
void kvm_set_pfn_dirty(kvm_pfn_t pfn);
void kvm_set_pfn_accessed(kvm_pfn_t pfn);
-void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache);
+void kvm_release_pfn(kvm_pfn_t pfn, bool dirty);
int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
int len);
int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len);
kvm_pfn_t kvm_vcpu_gfn_to_pfn_atomic(struct kvm_vcpu *vcpu, gfn_t gfn);
kvm_pfn_t kvm_vcpu_gfn_to_pfn(struct kvm_vcpu *vcpu, gfn_t gfn);
int kvm_vcpu_map(struct kvm_vcpu *vcpu, gpa_t gpa, struct kvm_host_map *map);
-int kvm_map_gfn(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache, bool atomic);
struct page *kvm_vcpu_gfn_to_page(struct kvm_vcpu *vcpu, gfn_t gfn);
void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map, bool dirty);
-int kvm_unmap_gfn(struct kvm_vcpu *vcpu, struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache, bool dirty, bool atomic);
unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn);
unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable);
int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, int offset,
struct kvm_memory_slot *memslot;
};
-struct gfn_to_pfn_cache {
- u64 generation;
- gfn_t gfn;
- kvm_pfn_t pfn;
- bool dirty;
-};
-
#ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE
/*
* Memory caches are used to preallocate memory ahead of various MMU flows,
/* This should match the actual table size of
* ata_eh_cmd_timeout_table in libata-eh.c.
*/
- ATA_EH_CMD_TIMEOUT_TABLE_SIZE = 6,
+ ATA_EH_CMD_TIMEOUT_TABLE_SIZE = 7,
/* Horkage types. May be set by libata or controller on drives
(some horkage may be drive/controller pair dependent */
ATA_HORKAGE_MAX_SEC_1024 = (1 << 25), /* Limit max sects to 1024 */
ATA_HORKAGE_MAX_TRIM_128M = (1 << 26), /* Limit max trim size to 128M */
ATA_HORKAGE_NO_NCQ_ON_ATI = (1 << 27), /* Disable NCQ on ATI chipset */
+ ATA_HORKAGE_NO_ID_DEV_LOG = (1 << 28), /* Identify device log missing */
/* DMA mask for user DMA control: User visible values; DO NOT
renumber */
*/
extern const struct ata_port_operations ata_base_port_ops;
extern const struct ata_port_operations sata_port_ops;
-extern struct device_attribute *ata_common_sdev_attrs[];
+extern const struct attribute_group *ata_common_sdev_groups[];
/*
* All sht initializers (BASE, PIO, BMDMA, NCQ) must be instantiated
#define ATA_BASE_SHT(drv_name) \
ATA_SUBBASE_SHT(drv_name), \
- .sdev_attrs = ata_common_sdev_attrs
+ .sdev_groups = ata_common_sdev_groups
#ifdef CONFIG_SATA_HOST
-extern struct device_attribute *ata_ncq_sdev_attrs[];
+extern const struct attribute_group *ata_ncq_sdev_groups[];
#define ATA_NCQ_SHT(drv_name) \
ATA_SUBBASE_SHT(drv_name), \
- .sdev_attrs = ata_ncq_sdev_attrs, \
+ .sdev_groups = ata_ncq_sdev_groups, \
.change_queue_depth = ata_scsi_change_queue_depth
#endif
#ifndef _LINUX_LIST_H
#define _LINUX_LIST_H
+#include <linux/container_of.h>
#include <linux/types.h>
#include <linux/stddef.h>
#include <linux/poison.h>
#include <linux/const.h>
-#include <linux/kernel.h>
+
+#include <asm/barrier.h>
/*
* Circular doubly linked list implementation.
*/
#include <linux/atomic.h>
-#include <linux/kernel.h>
+#include <linux/container_of.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
struct llist_head {
struct llist_node *first;
*/
#define NLMSVC_XDRSIZE sizeof(struct nlm_args)
-int nlmsvc_decode_testargs(struct svc_rqst *, __be32 *);
-int nlmsvc_encode_testres(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_lockargs(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_cancargs(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_unlockargs(struct svc_rqst *, __be32 *);
-int nlmsvc_encode_res(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_res(struct svc_rqst *, __be32 *);
-int nlmsvc_encode_void(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_void(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_shareargs(struct svc_rqst *, __be32 *);
-int nlmsvc_encode_shareres(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_notify(struct svc_rqst *, __be32 *);
-int nlmsvc_decode_reboot(struct svc_rqst *, __be32 *);
+bool nlmsvc_decode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_reboot(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_shareargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_decode_notify(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+
+bool nlmsvc_encode_testres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_encode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_encode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlmsvc_encode_shareres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
#endif /* LOCKD_XDR_H */
#define nlm4_fbig cpu_to_be32(NLM_FBIG)
#define nlm4_failed cpu_to_be32(NLM_FAILED)
-
-
-int nlm4svc_decode_testargs(struct svc_rqst *, __be32 *);
-int nlm4svc_encode_testres(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_lockargs(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_cancargs(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_unlockargs(struct svc_rqst *, __be32 *);
-int nlm4svc_encode_res(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_res(struct svc_rqst *, __be32 *);
-int nlm4svc_encode_void(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_void(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_shareargs(struct svc_rqst *, __be32 *);
-int nlm4svc_encode_shareres(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_notify(struct svc_rqst *, __be32 *);
-int nlm4svc_decode_reboot(struct svc_rqst *, __be32 *);
+bool nlm4svc_decode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_reboot(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_shareargs(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_decode_notify(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+
+bool nlm4svc_encode_testres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_encode_res(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_encode_void(struct svc_rqst *rqstp, struct xdr_stream *xdr);
+bool nlm4svc_encode_shareres(struct svc_rqst *rqstp, struct xdr_stream *xdr);
extern const struct rpc_version nlm_version4;
LSM_HOOK(int, 0, tun_dev_attach_queue, void *security)
LSM_HOOK(int, 0, tun_dev_attach, struct sock *sk, void *security)
LSM_HOOK(int, 0, tun_dev_open, void *security)
-LSM_HOOK(int, 0, sctp_assoc_request, struct sctp_endpoint *ep,
+LSM_HOOK(int, 0, sctp_assoc_request, struct sctp_association *asoc,
struct sk_buff *skb)
LSM_HOOK(int, 0, sctp_bind_connect, struct sock *sk, int optname,
struct sockaddr *address, int addrlen)
-LSM_HOOK(void, LSM_RET_VOID, sctp_sk_clone, struct sctp_endpoint *ep,
+LSM_HOOK(void, LSM_RET_VOID, sctp_sk_clone, struct sctp_association *asoc,
struct sock *sk, struct sock *newsk)
#endif /* CONFIG_SECURITY_NETWORK */
* Security hooks for SCTP
*
* @sctp_assoc_request:
- * Passes the @ep and @chunk->skb of the association INIT packet to
+ * Passes the @asoc and @chunk->skb of the association INIT packet to
* the security module.
- * @ep pointer to sctp endpoint structure.
+ * @asoc pointer to sctp association structure.
* @skb pointer to skbuff of association packet.
* Return 0 on success, error on failure.
* @sctp_bind_connect:
* Called whenever a new socket is created by accept(2) (i.e. a TCP
* style socket) or when a socket is 'peeled off' e.g userspace
* calls sctp_peeloff(3).
- * @ep pointer to current sctp endpoint structure.
+ * @asoc pointer to current sctp association structure.
* @sk pointer to current sock structure.
- * @sk pointer to new sock structure.
+ * @newsk pointer to new sock structure.
*
* Security hooks for Infiniband
*
/**
* enum memblock_flags - definition of memory region attributes
* @MEMBLOCK_NONE: no special request
- * @MEMBLOCK_HOTPLUG: hotpluggable region
+ * @MEMBLOCK_HOTPLUG: memory region indicated in the firmware-provided memory
+ * map during early boot as hot(un)pluggable system RAM (e.g., memory range
+ * that might get hotunplugged later). With "movable_node" set on the kernel
+ * commandline, try keeping this memory region hotunpluggable. Does not apply
+ * to memblocks added ("hotplugged") after early boot.
* @MEMBLOCK_MIRROR: mirrored region
* @MEMBLOCK_NOMAP: don't add to kernel direct mapping and treat as
* reserved in the memory map; refer to memblock_mark_nomap() description
* for further details
+ * @MEMBLOCK_DRIVER_MANAGED: memory region that is always detected and added
+ * via a driver, and never indicated in the firmware-provided memory map as
+ * system RAM. This corresponds to IORESOURCE_SYSRAM_DRIVER_MANAGED in the
+ * kernel resource tree.
*/
enum memblock_flags {
MEMBLOCK_NONE = 0x0, /* No special request */
MEMBLOCK_HOTPLUG = 0x1, /* hotpluggable region */
MEMBLOCK_MIRROR = 0x2, /* mirrored region */
MEMBLOCK_NOMAP = 0x4, /* don't add to kernel direct mapping */
+ MEMBLOCK_DRIVER_MANAGED = 0x8, /* always detected via a driver */
};
/**
#endif
void memblock_allow_resize(void);
-int memblock_add_node(phys_addr_t base, phys_addr_t size, int nid);
+int memblock_add_node(phys_addr_t base, phys_addr_t size, int nid,
+ enum memblock_flags flags);
int memblock_add(phys_addr_t base, phys_addr_t size);
int memblock_remove(phys_addr_t base, phys_addr_t size);
-int memblock_free(phys_addr_t base, phys_addr_t size);
+int memblock_phys_free(phys_addr_t base, phys_addr_t size);
int memblock_reserve(phys_addr_t base, phys_addr_t size);
#ifdef CONFIG_HAVE_MEMBLOCK_PHYS_MAP
int memblock_physmem_add(phys_addr_t base, phys_addr_t size);
int memblock_clear_nomap(phys_addr_t base, phys_addr_t size);
void memblock_free_all(void);
-void memblock_free_ptr(void *ptr, size_t size);
+void memblock_free(void *ptr, size_t size);
void reset_node_managed_pages(pg_data_t *pgdat);
void reset_all_zones_managed_pages(void);
struct memblock_type *type_b, phys_addr_t *out_start,
phys_addr_t *out_end, int *out_nid);
-void __memblock_free_late(phys_addr_t base, phys_addr_t size);
+void memblock_free_late(phys_addr_t base, phys_addr_t size);
#ifdef CONFIG_HAVE_MEMBLOCK_PHYS_MAP
static inline void __next_physmem_range(u64 *idx, struct memblock_type *type,
*/
#define for_each_mem_range(i, p_start, p_end) \
__for_each_mem_range(i, &memblock.memory, NULL, NUMA_NO_NODE, \
- MEMBLOCK_HOTPLUG, p_start, p_end, NULL)
+ MEMBLOCK_HOTPLUG | MEMBLOCK_DRIVER_MANAGED, \
+ p_start, p_end, NULL)
/**
* for_each_mem_range_rev - reverse iterate through memblock areas from
*/
#define for_each_mem_range_rev(i, p_start, p_end) \
__for_each_mem_range_rev(i, &memblock.memory, NULL, NUMA_NO_NODE, \
- MEMBLOCK_HOTPLUG, p_start, p_end, NULL)
+ MEMBLOCK_HOTPLUG | MEMBLOCK_DRIVER_MANAGED,\
+ p_start, p_end, NULL)
/**
* for_each_reserved_mem_range - iterate over all reserved memblock areas
return m->flags & MEMBLOCK_NOMAP;
}
+static inline bool memblock_is_driver_managed(struct memblock_region *m)
+{
+ return m->flags & MEMBLOCK_DRIVER_MANAGED;
+}
+
int memblock_search_pfn_nid(unsigned long pfn, unsigned long *start_pfn,
unsigned long *end_pfn);
void __next_mem_pfn_range(int *idx, int nid, unsigned long *out_start_pfn,
/* Flags for memblock allocation APIs */
#define MEMBLOCK_ALLOC_ANYWHERE (~(phys_addr_t)0)
#define MEMBLOCK_ALLOC_ACCESSIBLE 0
-#define MEMBLOCK_ALLOC_KASAN 1
+#define MEMBLOCK_ALLOC_NOLEAKTRACE 1
/* We are using top down, so it is safe to use 0 here */
#define MEMBLOCK_LOW_LIMIT 0
MEMBLOCK_ALLOC_ACCESSIBLE, nid);
}
-static inline void memblock_free_early(phys_addr_t base,
- phys_addr_t size)
-{
- memblock_free(base, size);
-}
-
-static inline void memblock_free_early_nid(phys_addr_t base,
- phys_addr_t size, int nid)
-{
- memblock_free(base, size);
-}
-
-static inline void memblock_free_late(phys_addr_t base, phys_addr_t size)
-{
- __memblock_free_late(base, size);
-}
-
/*
* Set the allocation direction to bottom-up or top-down.
*/
struct mem_cgroup_threshold_ary *spare;
};
-enum memcg_kmem_state {
- KMEM_NONE,
- KMEM_ALLOCATED,
- KMEM_ONLINE,
-};
-
#if defined(CONFIG_SMP)
struct memcg_padding {
char x[0];
#ifdef CONFIG_MEMCG_KMEM
int kmemcg_id;
- enum memcg_kmem_state kmem_state;
struct obj_cgroup __rcu *objcg;
struct list_head objcg_list; /* list of inherited objcgs */
#endif
if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->tcpmem_pressure)
return true;
do {
- if (time_before(jiffies, memcg->socket_pressure))
+ if (time_before(jiffies, READ_ONCE(memcg->socket_pressure)))
return true;
} while ((memcg = parent_mem_cgroup(memcg)));
return false;
unsigned long start_pfn;
unsigned long nr_pages;
int status_change_nid_normal;
- int status_change_nid_high;
int status_change_nid;
};
#define SLAB_CALLBACK_PRI 1
#define IPC_CALLBACK_PRI 10
-#ifndef CONFIG_MEMORY_HOTPLUG_SPARSE
+#ifndef CONFIG_MEMORY_HOTPLUG
static inline void memory_dev_init(void)
{
return;
{
return 0;
}
-#else
+static inline int hotplug_memory_notifier(notifier_fn_t fn, int pri)
+{
+ return 0;
+}
+/* These aren't inline functions due to a GCC bug. */
+#define register_hotmemory_notifier(nb) ({ (void)(nb); 0; })
+#define unregister_hotmemory_notifier(nb) ({ (void)(nb); })
+#else /* CONFIG_MEMORY_HOTPLUG */
extern int register_memory_notifier(struct notifier_block *nb);
extern void unregister_memory_notifier(struct notifier_block *nb);
int create_memory_block_devices(unsigned long start, unsigned long size,
extern int walk_memory_blocks(unsigned long start, unsigned long size,
void *arg, walk_memory_blocks_func_t func);
extern int for_each_memory_block(void *arg, walk_memory_blocks_func_t func);
-#define CONFIG_MEM_BLOCK_SIZE (PAGES_PER_SECTION<<PAGE_SHIFT)
extern int memory_group_register_static(int nid, unsigned long max_pages);
extern int memory_group_register_dynamic(int nid, unsigned long unit_pages);
typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);
int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
struct memory_group *excluded, void *arg);
-#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
-
-#ifdef CONFIG_MEMORY_HOTPLUG
#define hotplug_memory_notifier(fn, pri) ({ \
static __meminitdata struct notifier_block fn##_mem_nb =\
{ .notifier_call = fn, .priority = pri };\
})
#define register_hotmemory_notifier(nb) register_memory_notifier(nb)
#define unregister_hotmemory_notifier(nb) unregister_memory_notifier(nb)
-#else
-static inline int hotplug_memory_notifier(notifier_fn_t fn, int pri)
-{
- return 0;
-}
-/* These aren't inline functions due to a GCC bug. */
-#define register_hotmemory_notifier(nb) ({ (void)(nb); 0; })
-#define unregister_hotmemory_notifier(nb) ({ (void)(nb); })
-#endif
+#endif /* CONFIG_MEMORY_HOTPLUG */
/*
* Kernel text modification mutex, used for code patching. Users of this lock
{
seqlock_init(&zone->span_seqlock);
}
-extern int zone_grow_free_lists(struct zone *zone, unsigned long new_nr_pages);
-extern int zone_grow_waitqueues(struct zone *zone, unsigned long nr_pages);
-extern int add_one_highpage(struct page *page, int pfn, int bad_ppro);
extern void adjust_present_page_count(struct page *page,
struct memory_group *group,
long nr_pages);
#include <linux/sched.h>
#include <linux/mmzone.h>
-#include <linux/dax.h>
#include <linux/slab.h>
#include <linux/rbtree.h>
#include <linux/spinlock.h>
extern int mpol_misplaced(struct page *, struct vm_area_struct *, unsigned long);
extern void mpol_put_task_policy(struct task_struct *);
-extern bool numa_demotion_enabled;
-
static inline bool mpol_is_preferred_many(struct mempolicy *pol)
{
return (pol->mode == MPOL_PREFERRED_MANY);
return NULL;
}
-#define numa_demotion_enabled false
-
static inline bool mpol_is_preferred_many(struct mempolicy *pol)
{
return false;
PMIC_DA9063_BB = 0x5,
PMIC_DA9063_CA = 0x6,
PMIC_DA9063_DA = 0x7,
+ PMIC_DA9063_EA = 0x8,
};
/* Interrupts */
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Header file for device driver Hi6421 PMIC
- *
- * Copyright (c) 2013 Linaro Ltd.
- * Copyright (C) 2011 Hisilicon.
- * Copyright (c) 2020-2021 Huawei Technologies Co., Ltd
- *
- * Guodong Xu <guodong.xu@linaro.org>
- */
-
-#ifndef __HISI_PMIC_H
-#define __HISI_PMIC_H
-
-#include <linux/irqdomain.h>
-#include <linux/regmap.h>
-
-struct hi6421_spmi_pmic {
- struct resource *res;
- struct device *dev;
- void __iomem *regs;
- struct regmap *regmap;
-};
-
-#endif /* __HISI_PMIC_H */
/* Reserved: 0x7A-0x7D */
MAX77686_REG_BBAT_CHG = 0x7E,
- MAX77686_REG_32KHZ = 0x7F,
+ MAX77686_REG_32KHZ = 0x7F,
MAX77686_REG_PMIC_END = 0x80,
};
enum max77686_rtc_reg {
- MAX77686_RTC_INT = 0x00,
- MAX77686_RTC_INTM = 0x01,
+ MAX77686_RTC_INT = 0x00,
+ MAX77686_RTC_INTM = 0x01,
MAX77686_RTC_CONTROLM = 0x02,
MAX77686_RTC_CONTROL = 0x03,
MAX77686_RTC_UPDATE0 = 0x04,
/* Reserved: 0x5 */
MAX77686_WTSR_SMPL_CNTL = 0x06,
- MAX77686_RTC_SEC = 0x07,
- MAX77686_RTC_MIN = 0x08,
- MAX77686_RTC_HOUR = 0x09,
+ MAX77686_RTC_SEC = 0x07,
+ MAX77686_RTC_MIN = 0x08,
+ MAX77686_RTC_HOUR = 0x09,
MAX77686_RTC_WEEKDAY = 0x0A,
- MAX77686_RTC_MONTH = 0x0B,
- MAX77686_RTC_YEAR = 0x0C,
- MAX77686_RTC_DATE = 0x0D,
- MAX77686_ALARM1_SEC = 0x0E,
- MAX77686_ALARM1_MIN = 0x0F,
+ MAX77686_RTC_MONTH = 0x0B,
+ MAX77686_RTC_YEAR = 0x0C,
+ MAX77686_RTC_DATE = 0x0D,
+ MAX77686_ALARM1_SEC = 0x0E,
+ MAX77686_ALARM1_MIN = 0x0F,
MAX77686_ALARM1_HOUR = 0x10,
MAX77686_ALARM1_WEEKDAY = 0x11,
MAX77686_ALARM1_MONTH = 0x12,
MAX77686_ALARM1_YEAR = 0x13,
MAX77686_ALARM1_DATE = 0x14,
- MAX77686_ALARM2_SEC = 0x15,
- MAX77686_ALARM2_MIN = 0x16,
+ MAX77686_ALARM2_SEC = 0x15,
+ MAX77686_ALARM2_MIN = 0x16,
MAX77686_ALARM2_HOUR = 0x17,
MAX77686_ALARM2_WEEKDAY = 0x18,
MAX77686_ALARM2_MONTH = 0x19,
-#ifndef __LINUX_TI_AM335X_TSCADC_MFD_H
-#define __LINUX_TI_AM335X_TSCADC_MFD_H
-
+/* SPDX-License-Identifier: GPL-2.0-only */
/*
* TI Touch Screen / ADC MFD driver
*
* Copyright (C) 2012 Texas Instruments Incorporated - https://www.ti.com/
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed "as is" WITHOUT ANY WARRANTY of any
- * kind, whether express or implied; without even the implied warranty
- * of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
*/
+#ifndef __LINUX_TI_AM335X_TSCADC_MFD_H
+#define __LINUX_TI_AM335X_TSCADC_MFD_H
+
+#include <linux/bitfield.h>
#include <linux/mfd/core.h>
+#include <linux/units.h>
#define REG_RAWIRQSTATUS 0x024
#define REG_IRQSTATUS 0x028
/* IRQ wakeup enable */
#define IRQWKUP_ENB BIT(0)
-/* Step Enable */
-#define STEPENB_MASK (0x1FFFF << 0)
-#define STEPENB(val) ((val) << 0)
-#define ENB(val) (1 << (val))
-#define STPENB_STEPENB STEPENB(0x1FFFF)
-#define STPENB_STEPENB_TC STEPENB(0x1FFF)
-
/* IRQ enable */
#define IRQENB_HW_PEN BIT(0)
#define IRQENB_EOS BIT(1)
#define IRQENB_PENUP BIT(9)
/* Step Configuration */
-#define STEPCONFIG_MODE_MASK (3 << 0)
-#define STEPCONFIG_MODE(val) ((val) << 0)
+#define STEPCONFIG_MODE(val) FIELD_PREP(GENMASK(1, 0), (val))
#define STEPCONFIG_MODE_SWCNT STEPCONFIG_MODE(1)
#define STEPCONFIG_MODE_HWSYNC STEPCONFIG_MODE(2)
-#define STEPCONFIG_AVG_MASK (7 << 2)
-#define STEPCONFIG_AVG(val) ((val) << 2)
+#define STEPCONFIG_AVG(val) FIELD_PREP(GENMASK(4, 2), (val))
#define STEPCONFIG_AVG_16 STEPCONFIG_AVG(4)
#define STEPCONFIG_XPP BIT(5)
#define STEPCONFIG_XNN BIT(6)
#define STEPCONFIG_YNN BIT(8)
#define STEPCONFIG_XNP BIT(9)
#define STEPCONFIG_YPN BIT(10)
-#define STEPCONFIG_RFP(val) ((val) << 12)
-#define STEPCONFIG_RFP_VREFP (0x3 << 12)
-#define STEPCONFIG_INM_MASK (0xF << 15)
-#define STEPCONFIG_INM(val) ((val) << 15)
+#define STEPCONFIG_RFP(val) FIELD_PREP(GENMASK(13, 12), (val))
+#define STEPCONFIG_RFP_VREFP STEPCONFIG_RFP(3)
+#define STEPCONFIG_INM(val) FIELD_PREP(GENMASK(18, 15), (val))
#define STEPCONFIG_INM_ADCREFM STEPCONFIG_INM(8)
-#define STEPCONFIG_INP_MASK (0xF << 19)
-#define STEPCONFIG_INP(val) ((val) << 19)
+#define STEPCONFIG_INP(val) FIELD_PREP(GENMASK(22, 19), (val))
#define STEPCONFIG_INP_AN4 STEPCONFIG_INP(4)
#define STEPCONFIG_INP_ADCREFM STEPCONFIG_INP(8)
#define STEPCONFIG_FIFO1 BIT(26)
-#define STEPCONFIG_RFM(val) ((val) << 23)
-#define STEPCONFIG_RFM_VREFN (0x3 << 23)
+#define STEPCONFIG_RFM(val) FIELD_PREP(GENMASK(24, 23), (val))
+#define STEPCONFIG_RFM_VREFN STEPCONFIG_RFM(3)
/* Delay register */
-#define STEPDELAY_OPEN_MASK (0x3FFFF << 0)
-#define STEPDELAY_OPEN(val) ((val) << 0)
+#define STEPDELAY_OPEN(val) FIELD_PREP(GENMASK(17, 0), (val))
#define STEPCONFIG_OPENDLY STEPDELAY_OPEN(0x098)
-#define STEPDELAY_SAMPLE_MASK (0xFF << 24)
-#define STEPDELAY_SAMPLE(val) ((val) << 24)
+#define STEPCONFIG_MAX_OPENDLY GENMASK(17, 0)
+#define STEPDELAY_SAMPLE(val) FIELD_PREP(GENMASK(31, 24), (val))
#define STEPCONFIG_SAMPLEDLY STEPDELAY_SAMPLE(0)
+#define STEPCONFIG_MAX_SAMPLE GENMASK(7, 0)
/* Charge Config */
-#define STEPCHARGE_RFP_MASK (7 << 12)
-#define STEPCHARGE_RFP(val) ((val) << 12)
+#define STEPCHARGE_RFP(val) FIELD_PREP(GENMASK(14, 12), (val))
#define STEPCHARGE_RFP_XPUL STEPCHARGE_RFP(1)
-#define STEPCHARGE_INM_MASK (0xF << 15)
-#define STEPCHARGE_INM(val) ((val) << 15)
+#define STEPCHARGE_INM(val) FIELD_PREP(GENMASK(18, 15), (val))
#define STEPCHARGE_INM_AN1 STEPCHARGE_INM(1)
-#define STEPCHARGE_INP_MASK (0xF << 19)
-#define STEPCHARGE_INP(val) ((val) << 19)
-#define STEPCHARGE_RFM_MASK (3 << 23)
-#define STEPCHARGE_RFM(val) ((val) << 23)
+#define STEPCHARGE_INP(val) FIELD_PREP(GENMASK(22, 19), (val))
+#define STEPCHARGE_RFM(val) FIELD_PREP(GENMASK(24, 23), (val))
#define STEPCHARGE_RFM_XNUR STEPCHARGE_RFM(1)
/* Charge delay */
-#define CHARGEDLY_OPEN_MASK (0x3FFFF << 0)
-#define CHARGEDLY_OPEN(val) ((val) << 0)
+#define CHARGEDLY_OPEN(val) FIELD_PREP(GENMASK(17, 0), (val))
#define CHARGEDLY_OPENDLY CHARGEDLY_OPEN(0x400)
/* Control register */
-#define CNTRLREG_TSCSSENB BIT(0)
+#define CNTRLREG_SSENB BIT(0)
#define CNTRLREG_STEPID BIT(1)
-#define CNTRLREG_STEPCONFIGWRT BIT(2)
+#define CNTRLREG_TSC_STEPCONFIGWRT BIT(2)
#define CNTRLREG_POWERDOWN BIT(4)
-#define CNTRLREG_AFE_CTRL_MASK (3 << 5)
-#define CNTRLREG_AFE_CTRL(val) ((val) << 5)
-#define CNTRLREG_4WIRE CNTRLREG_AFE_CTRL(1)
-#define CNTRLREG_5WIRE CNTRLREG_AFE_CTRL(2)
-#define CNTRLREG_8WIRE CNTRLREG_AFE_CTRL(3)
-#define CNTRLREG_TSCENB BIT(7)
+#define CNTRLREG_TSC_AFE_CTRL(val) FIELD_PREP(GENMASK(6, 5), (val))
+#define CNTRLREG_TSC_4WIRE CNTRLREG_TSC_AFE_CTRL(1)
+#define CNTRLREG_TSC_5WIRE CNTRLREG_TSC_AFE_CTRL(2)
+#define CNTRLREG_TSC_8WIRE CNTRLREG_TSC_AFE_CTRL(3)
+#define CNTRLREG_TSC_ENB BIT(7)
+
+/*Control registers bitfields for MAGADC IP */
+#define CNTRLREG_MAGADCENB BIT(0)
+#define CNTRLREG_MAG_PREAMP_PWRDOWN BIT(5)
+#define CNTRLREG_MAG_PREAMP_BYPASS BIT(6)
/* FIFO READ Register */
-#define FIFOREAD_DATA_MASK (0xfff << 0)
-#define FIFOREAD_CHNLID_MASK (0xf << 16)
+#define FIFOREAD_DATA_MASK GENMASK(11, 0)
+#define FIFOREAD_CHNLID_MASK GENMASK(19, 16)
/* DMA ENABLE/CLEAR Register */
#define DMA_FIFO0 BIT(0)
#define DMA_FIFO1 BIT(1)
/* Sequencer Status */
-#define SEQ_STATUS BIT(5)
+#define SEQ_STATUS BIT(5)
#define CHARGE_STEP 0x11
-#define ADC_CLK 3000000
+#define TSC_ADC_CLK (3 * HZ_PER_MHZ)
+#define MAG_ADC_CLK (13 * HZ_PER_MHZ)
#define TOTAL_STEPS 16
#define TOTAL_CHANNELS 8
#define FIFO1_THRESHOLD 19
*
* max processing time: 266431 * 308ns = 83ms(approx)
*/
-#define IDLE_TIMEOUT 83 /* milliseconds */
+#define IDLE_TIMEOUT_MS 83 /* milliseconds */
#define TSCADC_CELLS 2
+struct ti_tscadc_data {
+ char *adc_feature_name;
+ char *adc_feature_compatible;
+ char *secondary_feature_name;
+ char *secondary_feature_compatible;
+ unsigned int target_clk_rate;
+};
+
struct ti_tscadc_dev {
struct device *dev;
struct regmap *regmap;
void __iomem *tscadc_base;
phys_addr_t tscadc_phys_base;
+ const struct ti_tscadc_data *data;
int irq;
- int used_cells; /* 1-2 */
- int tsc_wires;
- int tsc_cell; /* -1 if not used */
- int adc_cell; /* -1 if not used */
struct mfd_cell cells[TSCADC_CELLS];
+ u32 ctrl;
u32 reg_se_cache;
bool adc_waiting;
bool adc_in_use;
return *tscadc_dev;
}
+static inline bool ti_adc_with_touchscreen(struct ti_tscadc_dev *tscadc)
+{
+ return of_device_is_compatible(tscadc->dev->of_node,
+ "ti,am3359-tscadc");
+}
+
void am335x_tsc_se_set_cache(struct ti_tscadc_dev *tsadc, u32 val);
void am335x_tsc_se_set_once(struct ti_tscadc_dev *tsadc, u32 val);
void am335x_tsc_se_clr(struct ti_tscadc_dev *tsadc, u32 val);
extern const struct regmap_config tps65912_regmap_config;
int tps65912_device_init(struct tps65912 *tps);
-int tps65912_device_exit(struct tps65912 *tps);
+void tps65912_device_exit(struct tps65912 *tps);
#endif /* __LINUX_MFD_TPS65912_H */
+++ /dev/null
-/*
- * tps80031.h -- TI TPS80031 and TI TPS80032 PMIC driver.
- *
- * Copyright (c) 2012, NVIDIA Corporation.
- *
- * Author: Laxman Dewangan <ldewangan@nvidia.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation version 2.
- *
- * This program is distributed "as is" WITHOUT ANY WARRANTY of any kind,
- * whether express or implied; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
- * 02111-1307, USA
- */
-
-#ifndef __LINUX_MFD_TPS80031_H
-#define __LINUX_MFD_TPS80031_H
-
-#include <linux/device.h>
-#include <linux/regmap.h>
-
-/* Pull-ups/Pull-downs */
-#define TPS80031_CFG_INPUT_PUPD1 0xF0
-#define TPS80031_CFG_INPUT_PUPD2 0xF1
-#define TPS80031_CFG_INPUT_PUPD3 0xF2
-#define TPS80031_CFG_INPUT_PUPD4 0xF3
-#define TPS80031_CFG_LDO_PD1 0xF4
-#define TPS80031_CFG_LDO_PD2 0xF5
-#define TPS80031_CFG_SMPS_PD 0xF6
-
-/* Real Time Clock */
-#define TPS80031_SECONDS_REG 0x00
-#define TPS80031_MINUTES_REG 0x01
-#define TPS80031_HOURS_REG 0x02
-#define TPS80031_DAYS_REG 0x03
-#define TPS80031_MONTHS_REG 0x04
-#define TPS80031_YEARS_REG 0x05
-#define TPS80031_WEEKS_REG 0x06
-#define TPS80031_ALARM_SECONDS_REG 0x08
-#define TPS80031_ALARM_MINUTES_REG 0x09
-#define TPS80031_ALARM_HOURS_REG 0x0A
-#define TPS80031_ALARM_DAYS_REG 0x0B
-#define TPS80031_ALARM_MONTHS_REG 0x0C
-#define TPS80031_ALARM_YEARS_REG 0x0D
-#define TPS80031_RTC_CTRL_REG 0x10
-#define TPS80031_RTC_STATUS_REG 0x11
-#define TPS80031_RTC_INTERRUPTS_REG 0x12
-#define TPS80031_RTC_COMP_LSB_REG 0x13
-#define TPS80031_RTC_COMP_MSB_REG 0x14
-#define TPS80031_RTC_RESET_STATUS_REG 0x16
-
-/*PMC Master Module */
-#define TPS80031_PHOENIX_START_CONDITION 0x1F
-#define TPS80031_PHOENIX_MSK_TRANSITION 0x20
-#define TPS80031_STS_HW_CONDITIONS 0x21
-#define TPS80031_PHOENIX_LAST_TURNOFF_STS 0x22
-#define TPS80031_VSYSMIN_LO_THRESHOLD 0x23
-#define TPS80031_VSYSMIN_HI_THRESHOLD 0x24
-#define TPS80031_PHOENIX_DEV_ON 0x25
-#define TPS80031_STS_PWR_GRP_STATE 0x27
-#define TPS80031_PH_CFG_VSYSLOW 0x28
-#define TPS80031_PH_STS_BOOT 0x29
-#define TPS80031_PHOENIX_SENS_TRANSITION 0x2A
-#define TPS80031_PHOENIX_SEQ_CFG 0x2B
-#define TPS80031_PRIMARY_WATCHDOG_CFG 0X2C
-#define TPS80031_KEY_PRESS_DUR_CFG 0X2D
-#define TPS80031_SMPS_LDO_SHORT_STS 0x2E
-
-/* PMC Slave Module - Broadcast */
-#define TPS80031_BROADCAST_ADDR_ALL 0x31
-#define TPS80031_BROADCAST_ADDR_REF 0x32
-#define TPS80031_BROADCAST_ADDR_PROV 0x33
-#define TPS80031_BROADCAST_ADDR_CLK_RST 0x34
-
-/* PMC Slave Module SMPS Regulators */
-#define TPS80031_SMPS4_CFG_TRANS 0x41
-#define TPS80031_SMPS4_CFG_STATE 0x42
-#define TPS80031_SMPS4_CFG_VOLTAGE 0x44
-#define TPS80031_VIO_CFG_TRANS 0x47
-#define TPS80031_VIO_CFG_STATE 0x48
-#define TPS80031_VIO_CFG_FORCE 0x49
-#define TPS80031_VIO_CFG_VOLTAGE 0x4A
-#define TPS80031_VIO_CFG_STEP 0x48
-#define TPS80031_SMPS1_CFG_TRANS 0x53
-#define TPS80031_SMPS1_CFG_STATE 0x54
-#define TPS80031_SMPS1_CFG_FORCE 0x55
-#define TPS80031_SMPS1_CFG_VOLTAGE 0x56
-#define TPS80031_SMPS1_CFG_STEP 0x57
-#define TPS80031_SMPS2_CFG_TRANS 0x59
-#define TPS80031_SMPS2_CFG_STATE 0x5A
-#define TPS80031_SMPS2_CFG_FORCE 0x5B
-#define TPS80031_SMPS2_CFG_VOLTAGE 0x5C
-#define TPS80031_SMPS2_CFG_STEP 0x5D
-#define TPS80031_SMPS3_CFG_TRANS 0x65
-#define TPS80031_SMPS3_CFG_STATE 0x66
-#define TPS80031_SMPS3_CFG_VOLTAGE 0x68
-
-/* PMC Slave Module LDO Regulators */
-#define TPS80031_VANA_CFG_TRANS 0x81
-#define TPS80031_VANA_CFG_STATE 0x82
-#define TPS80031_VANA_CFG_VOLTAGE 0x83
-#define TPS80031_LDO2_CFG_TRANS 0x85
-#define TPS80031_LDO2_CFG_STATE 0x86
-#define TPS80031_LDO2_CFG_VOLTAGE 0x87
-#define TPS80031_LDO4_CFG_TRANS 0x89
-#define TPS80031_LDO4_CFG_STATE 0x8A
-#define TPS80031_LDO4_CFG_VOLTAGE 0x8B
-#define TPS80031_LDO3_CFG_TRANS 0x8D
-#define TPS80031_LDO3_CFG_STATE 0x8E
-#define TPS80031_LDO3_CFG_VOLTAGE 0x8F
-#define TPS80031_LDO6_CFG_TRANS 0x91
-#define TPS80031_LDO6_CFG_STATE 0x92
-#define TPS80031_LDO6_CFG_VOLTAGE 0x93
-#define TPS80031_LDOLN_CFG_TRANS 0x95
-#define TPS80031_LDOLN_CFG_STATE 0x96
-#define TPS80031_LDOLN_CFG_VOLTAGE 0x97
-#define TPS80031_LDO5_CFG_TRANS 0x99
-#define TPS80031_LDO5_CFG_STATE 0x9A
-#define TPS80031_LDO5_CFG_VOLTAGE 0x9B
-#define TPS80031_LDO1_CFG_TRANS 0x9D
-#define TPS80031_LDO1_CFG_STATE 0x9E
-#define TPS80031_LDO1_CFG_VOLTAGE 0x9F
-#define TPS80031_LDOUSB_CFG_TRANS 0xA1
-#define TPS80031_LDOUSB_CFG_STATE 0xA2
-#define TPS80031_LDOUSB_CFG_VOLTAGE 0xA3
-#define TPS80031_LDO7_CFG_TRANS 0xA5
-#define TPS80031_LDO7_CFG_STATE 0xA6
-#define TPS80031_LDO7_CFG_VOLTAGE 0xA7
-
-/* PMC Slave Module External Control */
-#define TPS80031_REGEN1_CFG_TRANS 0xAE
-#define TPS80031_REGEN1_CFG_STATE 0xAF
-#define TPS80031_REGEN2_CFG_TRANS 0xB1
-#define TPS80031_REGEN2_CFG_STATE 0xB2
-#define TPS80031_SYSEN_CFG_TRANS 0xB4
-#define TPS80031_SYSEN_CFG_STATE 0xB5
-
-/* PMC Slave Module Internal Control */
-#define TPS80031_NRESPWRON_CFG_TRANS 0xB7
-#define TPS80031_NRESPWRON_CFG_STATE 0xB8
-#define TPS80031_CLK32KAO_CFG_TRANS 0xBA
-#define TPS80031_CLK32KAO_CFG_STATE 0xBB
-#define TPS80031_CLK32KG_CFG_TRANS 0xBD
-#define TPS80031_CLK32KG_CFG_STATE 0xBE
-#define TPS80031_CLK32KAUDIO_CFG_TRANS 0xC0
-#define TPS80031_CLK32KAUDIO_CFG_STATE 0xC1
-#define TPS80031_VRTC_CFG_TRANS 0xC3
-#define TPS80031_VRTC_CFG_STATE 0xC4
-#define TPS80031_BIAS_CFG_TRANS 0xC6
-#define TPS80031_BIAS_CFG_STATE 0xC7
-#define TPS80031_VSYSMIN_HI_CFG_TRANS 0xC9
-#define TPS80031_VSYSMIN_HI_CFG_STATE 0xCA
-#define TPS80031_RC6MHZ_CFG_TRANS 0xCC
-#define TPS80031_RC6MHZ_CFG_STATE 0xCD
-#define TPS80031_TMP_CFG_TRANS 0xCF
-#define TPS80031_TMP_CFG_STATE 0xD0
-
-/* PMC Slave Module resources assignment */
-#define TPS80031_PREQ1_RES_ASS_A 0xD7
-#define TPS80031_PREQ1_RES_ASS_B 0xD8
-#define TPS80031_PREQ1_RES_ASS_C 0xD9
-#define TPS80031_PREQ2_RES_ASS_A 0xDA
-#define TPS80031_PREQ2_RES_ASS_B 0xDB
-#define TPS80031_PREQ2_RES_ASS_C 0xDC
-#define TPS80031_PREQ3_RES_ASS_A 0xDD
-#define TPS80031_PREQ3_RES_ASS_B 0xDE
-#define TPS80031_PREQ3_RES_ASS_C 0xDF
-
-/* PMC Slave Module Miscellaneous */
-#define TPS80031_SMPS_OFFSET 0xE0
-#define TPS80031_SMPS_MULT 0xE3
-#define TPS80031_MISC1 0xE4
-#define TPS80031_MISC2 0xE5
-#define TPS80031_BBSPOR_CFG 0xE6
-#define TPS80031_TMP_CFG 0xE7
-
-/* Battery Charging Controller and Indicator LED */
-#define TPS80031_CONTROLLER_CTRL2 0xDA
-#define TPS80031_CONTROLLER_VSEL_COMP 0xDB
-#define TPS80031_CHARGERUSB_VSYSREG 0xDC
-#define TPS80031_CHARGERUSB_VICHRG_PC 0xDD
-#define TPS80031_LINEAR_CHRG_STS 0xDE
-#define TPS80031_CONTROLLER_INT_MASK 0xE0
-#define TPS80031_CONTROLLER_CTRL1 0xE1
-#define TPS80031_CONTROLLER_WDG 0xE2
-#define TPS80031_CONTROLLER_STAT1 0xE3
-#define TPS80031_CHARGERUSB_INT_STATUS 0xE4
-#define TPS80031_CHARGERUSB_INT_MASK 0xE5
-#define TPS80031_CHARGERUSB_STATUS_INT1 0xE6
-#define TPS80031_CHARGERUSB_STATUS_INT2 0xE7
-#define TPS80031_CHARGERUSB_CTRL1 0xE8
-#define TPS80031_CHARGERUSB_CTRL2 0xE9
-#define TPS80031_CHARGERUSB_CTRL3 0xEA
-#define TPS80031_CHARGERUSB_STAT1 0xEB
-#define TPS80031_CHARGERUSB_VOREG 0xEC
-#define TPS80031_CHARGERUSB_VICHRG 0xED
-#define TPS80031_CHARGERUSB_CINLIMIT 0xEE
-#define TPS80031_CHARGERUSB_CTRLLIMIT1 0xEF
-#define TPS80031_CHARGERUSB_CTRLLIMIT2 0xF0
-#define TPS80031_LED_PWM_CTRL1 0xF4
-#define TPS80031_LED_PWM_CTRL2 0xF5
-
-/* USB On-The-Go */
-#define TPS80031_BACKUP_REG 0xFA
-#define TPS80031_USB_VENDOR_ID_LSB 0x00
-#define TPS80031_USB_VENDOR_ID_MSB 0x01
-#define TPS80031_USB_PRODUCT_ID_LSB 0x02
-#define TPS80031_USB_PRODUCT_ID_MSB 0x03
-#define TPS80031_USB_VBUS_CTRL_SET 0x04
-#define TPS80031_USB_VBUS_CTRL_CLR 0x05
-#define TPS80031_USB_ID_CTRL_SET 0x06
-#define TPS80031_USB_ID_CTRL_CLR 0x07
-#define TPS80031_USB_VBUS_INT_SRC 0x08
-#define TPS80031_USB_VBUS_INT_LATCH_SET 0x09
-#define TPS80031_USB_VBUS_INT_LATCH_CLR 0x0A
-#define TPS80031_USB_VBUS_INT_EN_LO_SET 0x0B
-#define TPS80031_USB_VBUS_INT_EN_LO_CLR 0x0C
-#define TPS80031_USB_VBUS_INT_EN_HI_SET 0x0D
-#define TPS80031_USB_VBUS_INT_EN_HI_CLR 0x0E
-#define TPS80031_USB_ID_INT_SRC 0x0F
-#define TPS80031_USB_ID_INT_LATCH_SET 0x10
-#define TPS80031_USB_ID_INT_LATCH_CLR 0x11
-#define TPS80031_USB_ID_INT_EN_LO_SET 0x12
-#define TPS80031_USB_ID_INT_EN_LO_CLR 0x13
-#define TPS80031_USB_ID_INT_EN_HI_SET 0x14
-#define TPS80031_USB_ID_INT_EN_HI_CLR 0x15
-#define TPS80031_USB_OTG_ADP_CTRL 0x16
-#define TPS80031_USB_OTG_ADP_HIGH 0x17
-#define TPS80031_USB_OTG_ADP_LOW 0x18
-#define TPS80031_USB_OTG_ADP_RISE 0x19
-#define TPS80031_USB_OTG_REVISION 0x1A
-
-/* Gas Gauge */
-#define TPS80031_FG_REG_00 0xC0
-#define TPS80031_FG_REG_01 0xC1
-#define TPS80031_FG_REG_02 0xC2
-#define TPS80031_FG_REG_03 0xC3
-#define TPS80031_FG_REG_04 0xC4
-#define TPS80031_FG_REG_05 0xC5
-#define TPS80031_FG_REG_06 0xC6
-#define TPS80031_FG_REG_07 0xC7
-#define TPS80031_FG_REG_08 0xC8
-#define TPS80031_FG_REG_09 0xC9
-#define TPS80031_FG_REG_10 0xCA
-#define TPS80031_FG_REG_11 0xCB
-
-/* General Purpose ADC */
-#define TPS80031_GPADC_CTRL 0x2E
-#define TPS80031_GPADC_CTRL2 0x2F
-#define TPS80031_RTSELECT_LSB 0x32
-#define TPS80031_RTSELECT_ISB 0x33
-#define TPS80031_RTSELECT_MSB 0x34
-#define TPS80031_GPSELECT_ISB 0x35
-#define TPS80031_CTRL_P1 0x36
-#define TPS80031_RTCH0_LSB 0x37
-#define TPS80031_RTCH0_MSB 0x38
-#define TPS80031_RTCH1_LSB 0x39
-#define TPS80031_RTCH1_MSB 0x3A
-#define TPS80031_GPCH0_LSB 0x3B
-#define TPS80031_GPCH0_MSB 0x3C
-
-/* SIM, MMC and Battery Detection */
-#define TPS80031_SIMDEBOUNCING 0xEB
-#define TPS80031_SIMCTRL 0xEC
-#define TPS80031_MMCDEBOUNCING 0xED
-#define TPS80031_MMCCTRL 0xEE
-#define TPS80031_BATDEBOUNCING 0xEF
-
-/* Vibrator Driver and PWMs */
-#define TPS80031_VIBCTRL 0x9B
-#define TPS80031_VIBMODE 0x9C
-#define TPS80031_PWM1ON 0xBA
-#define TPS80031_PWM1OFF 0xBB
-#define TPS80031_PWM2ON 0xBD
-#define TPS80031_PWM2OFF 0xBE
-
-/* Control Interface */
-#define TPS80031_INT_STS_A 0xD0
-#define TPS80031_INT_STS_B 0xD1
-#define TPS80031_INT_STS_C 0xD2
-#define TPS80031_INT_MSK_LINE_A 0xD3
-#define TPS80031_INT_MSK_LINE_B 0xD4
-#define TPS80031_INT_MSK_LINE_C 0xD5
-#define TPS80031_INT_MSK_STS_A 0xD6
-#define TPS80031_INT_MSK_STS_B 0xD7
-#define TPS80031_INT_MSK_STS_C 0xD8
-#define TPS80031_TOGGLE1 0x90
-#define TPS80031_TOGGLE2 0x91
-#define TPS80031_TOGGLE3 0x92
-#define TPS80031_PWDNSTATUS1 0x93
-#define TPS80031_PWDNSTATUS2 0x94
-#define TPS80031_VALIDITY0 0x17
-#define TPS80031_VALIDITY1 0x18
-#define TPS80031_VALIDITY2 0x19
-#define TPS80031_VALIDITY3 0x1A
-#define TPS80031_VALIDITY4 0x1B
-#define TPS80031_VALIDITY5 0x1C
-#define TPS80031_VALIDITY6 0x1D
-#define TPS80031_VALIDITY7 0x1E
-
-/* Version number related register */
-#define TPS80031_JTAGVERNUM 0x87
-#define TPS80031_EPROM_REV 0xDF
-
-/* GPADC Trimming Bits. */
-#define TPS80031_GPADC_TRIM0 0xCC
-#define TPS80031_GPADC_TRIM1 0xCD
-#define TPS80031_GPADC_TRIM2 0xCE
-#define TPS80031_GPADC_TRIM3 0xCF
-#define TPS80031_GPADC_TRIM4 0xD0
-#define TPS80031_GPADC_TRIM5 0xD1
-#define TPS80031_GPADC_TRIM6 0xD2
-#define TPS80031_GPADC_TRIM7 0xD3
-#define TPS80031_GPADC_TRIM8 0xD4
-#define TPS80031_GPADC_TRIM9 0xD5
-#define TPS80031_GPADC_TRIM10 0xD6
-#define TPS80031_GPADC_TRIM11 0xD7
-#define TPS80031_GPADC_TRIM12 0xD8
-#define TPS80031_GPADC_TRIM13 0xD9
-#define TPS80031_GPADC_TRIM14 0xDA
-#define TPS80031_GPADC_TRIM15 0xDB
-#define TPS80031_GPADC_TRIM16 0xDC
-#define TPS80031_GPADC_TRIM17 0xDD
-#define TPS80031_GPADC_TRIM18 0xDE
-
-/* TPS80031_CONTROLLER_STAT1 bit fields */
-#define TPS80031_CONTROLLER_STAT1_BAT_TEMP 0
-#define TPS80031_CONTROLLER_STAT1_BAT_REMOVED 1
-#define TPS80031_CONTROLLER_STAT1_VBUS_DET 2
-#define TPS80031_CONTROLLER_STAT1_VAC_DET 3
-#define TPS80031_CONTROLLER_STAT1_FAULT_WDG 4
-#define TPS80031_CONTROLLER_STAT1_LINCH_GATED 6
-/* TPS80031_CONTROLLER_INT_MASK bit filed */
-#define TPS80031_CONTROLLER_INT_MASK_MVAC_DET 0
-#define TPS80031_CONTROLLER_INT_MASK_MVBUS_DET 1
-#define TPS80031_CONTROLLER_INT_MASK_MBAT_TEMP 2
-#define TPS80031_CONTROLLER_INT_MASK_MFAULT_WDG 3
-#define TPS80031_CONTROLLER_INT_MASK_MBAT_REMOVED 4
-#define TPS80031_CONTROLLER_INT_MASK_MLINCH_GATED 5
-
-#define TPS80031_CHARGE_CONTROL_SUB_INT_MASK 0x3F
-
-/* TPS80031_PHOENIX_DEV_ON bit field */
-#define TPS80031_DEVOFF 0x1
-
-#define TPS80031_EXT_CONTROL_CFG_TRANS 0
-#define TPS80031_EXT_CONTROL_CFG_STATE 1
-
-/* State register field */
-#define TPS80031_STATE_OFF 0x00
-#define TPS80031_STATE_ON 0x01
-#define TPS80031_STATE_MASK 0x03
-
-/* Trans register field */
-#define TPS80031_TRANS_ACTIVE_OFF 0x00
-#define TPS80031_TRANS_ACTIVE_ON 0x01
-#define TPS80031_TRANS_ACTIVE_MASK 0x03
-#define TPS80031_TRANS_SLEEP_OFF 0x00
-#define TPS80031_TRANS_SLEEP_ON 0x04
-#define TPS80031_TRANS_SLEEP_MASK 0x0C
-#define TPS80031_TRANS_OFF_OFF 0x00
-#define TPS80031_TRANS_OFF_ACTIVE 0x10
-#define TPS80031_TRANS_OFF_MASK 0x30
-
-#define TPS80031_EXT_PWR_REQ (TPS80031_PWR_REQ_INPUT_PREQ1 | \
- TPS80031_PWR_REQ_INPUT_PREQ2 | \
- TPS80031_PWR_REQ_INPUT_PREQ3)
-
-/* TPS80031_BBSPOR_CFG bit field */
-#define TPS80031_BBSPOR_CHG_EN 0x8
-#define TPS80031_MAX_REGISTER 0xFF
-
-struct i2c_client;
-
-/* Supported chips */
-enum chips {
- TPS80031 = 0x00000001,
- TPS80032 = 0x00000002,
-};
-
-enum {
- TPS80031_INT_PWRON,
- TPS80031_INT_RPWRON,
- TPS80031_INT_SYS_VLOW,
- TPS80031_INT_RTC_ALARM,
- TPS80031_INT_RTC_PERIOD,
- TPS80031_INT_HOT_DIE,
- TPS80031_INT_VXX_SHORT,
- TPS80031_INT_SPDURATION,
- TPS80031_INT_WATCHDOG,
- TPS80031_INT_BAT,
- TPS80031_INT_SIM,
- TPS80031_INT_MMC,
- TPS80031_INT_RES,
- TPS80031_INT_GPADC_RT,
- TPS80031_INT_GPADC_SW2_EOC,
- TPS80031_INT_CC_AUTOCAL,
- TPS80031_INT_ID_WKUP,
- TPS80031_INT_VBUSS_WKUP,
- TPS80031_INT_ID,
- TPS80031_INT_VBUS,
- TPS80031_INT_CHRG_CTRL,
- TPS80031_INT_EXT_CHRG,
- TPS80031_INT_INT_CHRG,
- TPS80031_INT_RES2,
- TPS80031_INT_BAT_TEMP_OVRANGE,
- TPS80031_INT_BAT_REMOVED,
- TPS80031_INT_VBUS_DET,
- TPS80031_INT_VAC_DET,
- TPS80031_INT_FAULT_WDG,
- TPS80031_INT_LINCH_GATED,
-
- /* Last interrupt id to get the end number */
- TPS80031_INT_NR,
-};
-
-/* TPS80031 Slave IDs */
-#define TPS80031_NUM_SLAVES 4
-#define TPS80031_SLAVE_ID0 0
-#define TPS80031_SLAVE_ID1 1
-#define TPS80031_SLAVE_ID2 2
-#define TPS80031_SLAVE_ID3 3
-
-/* TPS80031 I2C addresses */
-#define TPS80031_I2C_ID0_ADDR 0x12
-#define TPS80031_I2C_ID1_ADDR 0x48
-#define TPS80031_I2C_ID2_ADDR 0x49
-#define TPS80031_I2C_ID3_ADDR 0x4A
-
-enum {
- TPS80031_REGULATOR_VIO,
- TPS80031_REGULATOR_SMPS1,
- TPS80031_REGULATOR_SMPS2,
- TPS80031_REGULATOR_SMPS3,
- TPS80031_REGULATOR_SMPS4,
- TPS80031_REGULATOR_VANA,
- TPS80031_REGULATOR_LDO1,
- TPS80031_REGULATOR_LDO2,
- TPS80031_REGULATOR_LDO3,
- TPS80031_REGULATOR_LDO4,
- TPS80031_REGULATOR_LDO5,
- TPS80031_REGULATOR_LDO6,
- TPS80031_REGULATOR_LDO7,
- TPS80031_REGULATOR_LDOLN,
- TPS80031_REGULATOR_LDOUSB,
- TPS80031_REGULATOR_VBUS,
- TPS80031_REGULATOR_REGEN1,
- TPS80031_REGULATOR_REGEN2,
- TPS80031_REGULATOR_SYSEN,
- TPS80031_REGULATOR_MAX,
-};
-
-/* Different configurations for the rails */
-enum {
- /* USBLDO input selection */
- TPS80031_USBLDO_INPUT_VSYS = 0x00000001,
- TPS80031_USBLDO_INPUT_PMID = 0x00000002,
-
- /* LDO3 output mode */
- TPS80031_LDO3_OUTPUT_VIB = 0x00000004,
-
- /* VBUS configuration */
- TPS80031_VBUS_DISCHRG_EN_PDN = 0x00000004,
- TPS80031_VBUS_SW_ONLY = 0x00000008,
- TPS80031_VBUS_SW_N_ID = 0x00000010,
-};
-
-/* External controls requests */
-enum tps80031_ext_control {
- TPS80031_PWR_REQ_INPUT_NONE = 0x00000000,
- TPS80031_PWR_REQ_INPUT_PREQ1 = 0x00000001,
- TPS80031_PWR_REQ_INPUT_PREQ2 = 0x00000002,
- TPS80031_PWR_REQ_INPUT_PREQ3 = 0x00000004,
- TPS80031_PWR_OFF_ON_SLEEP = 0x00000008,
- TPS80031_PWR_ON_ON_SLEEP = 0x00000010,
-};
-
-enum tps80031_pupd_pins {
- TPS80031_PREQ1 = 0,
- TPS80031_PREQ2A,
- TPS80031_PREQ2B,
- TPS80031_PREQ2C,
- TPS80031_PREQ3,
- TPS80031_NRES_WARM,
- TPS80031_PWM_FORCE,
- TPS80031_CHRG_EXT_CHRG_STATZ,
- TPS80031_SIM,
- TPS80031_MMC,
- TPS80031_GPADC_START,
- TPS80031_DVSI2C_SCL,
- TPS80031_DVSI2C_SDA,
- TPS80031_CTLI2C_SCL,
- TPS80031_CTLI2C_SDA,
-};
-
-enum tps80031_pupd_settings {
- TPS80031_PUPD_NORMAL,
- TPS80031_PUPD_PULLDOWN,
- TPS80031_PUPD_PULLUP,
-};
-
-struct tps80031 {
- struct device *dev;
- unsigned long chip_info;
- int es_version;
- struct i2c_client *clients[TPS80031_NUM_SLAVES];
- struct regmap *regmap[TPS80031_NUM_SLAVES];
- struct regmap_irq_chip_data *irq_data;
-};
-
-struct tps80031_pupd_init_data {
- int input_pin;
- int setting;
-};
-
-/*
- * struct tps80031_regulator_platform_data - tps80031 regulator platform data.
- *
- * @reg_init_data: The regulator init data.
- * @ext_ctrl_flag: External control flag for sleep/power request control.
- * @config_flags: Configuration flag to configure the rails.
- * It should be ORed of config enums.
- */
-
-struct tps80031_regulator_platform_data {
- struct regulator_init_data *reg_init_data;
- unsigned int ext_ctrl_flag;
- unsigned int config_flags;
-};
-
-struct tps80031_platform_data {
- int irq_base;
- bool use_power_off;
- struct tps80031_pupd_init_data *pupd_init_data;
- int pupd_init_data_size;
- struct tps80031_regulator_platform_data
- *regulator_pdata[TPS80031_REGULATOR_MAX];
-};
-
-static inline int tps80031_write(struct device *dev, int sid,
- int reg, uint8_t val)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_write(tps80031->regmap[sid], reg, val);
-}
-
-static inline int tps80031_writes(struct device *dev, int sid, int reg,
- int len, uint8_t *val)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_bulk_write(tps80031->regmap[sid], reg, val, len);
-}
-
-static inline int tps80031_read(struct device *dev, int sid,
- int reg, uint8_t *val)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
- unsigned int ival;
- int ret;
-
- ret = regmap_read(tps80031->regmap[sid], reg, &ival);
- if (ret < 0) {
- dev_err(dev, "failed reading from reg 0x%02x\n", reg);
- return ret;
- }
-
- *val = ival;
- return ret;
-}
-
-static inline int tps80031_reads(struct device *dev, int sid,
- int reg, int len, uint8_t *val)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_bulk_read(tps80031->regmap[sid], reg, val, len);
-}
-
-static inline int tps80031_set_bits(struct device *dev, int sid,
- int reg, uint8_t bit_mask)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_update_bits(tps80031->regmap[sid], reg,
- bit_mask, bit_mask);
-}
-
-static inline int tps80031_clr_bits(struct device *dev, int sid,
- int reg, uint8_t bit_mask)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_update_bits(tps80031->regmap[sid], reg, bit_mask, 0);
-}
-
-static inline int tps80031_update(struct device *dev, int sid,
- int reg, uint8_t val, uint8_t mask)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_update_bits(tps80031->regmap[sid], reg, mask, val);
-}
-
-static inline unsigned long tps80031_get_chip_info(struct device *dev)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return tps80031->chip_info;
-}
-
-static inline int tps80031_get_pmu_version(struct device *dev)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return tps80031->es_version;
-}
-
-static inline int tps80031_irq_get_virq(struct device *dev, int irq)
-{
- struct tps80031 *tps80031 = dev_get_drvdata(dev);
-
- return regmap_irq_get_virq(tps80031->irq_data, irq);
-}
-
-extern int tps80031_ext_power_req_config(struct device *dev,
- unsigned long ext_ctrl_flag, int preq_bit,
- int state_reg_add, int trans_reg_add);
-#endif /*__LINUX_MFD_TPS80031_H */
*/
#define MIGRATEPAGE_SUCCESS 0
-/*
- * Keep sync with:
- * - macro MIGRATE_REASON in include/trace/events/migrate.h
- * - migrate_reason_names[MR_TYPES] in mm/debug.c
- */
-enum migrate_reason {
- MR_COMPACTION,
- MR_MEMORY_FAILURE,
- MR_MEMORY_HOTPLUG,
- MR_SYSCALL, /* also applies to cpusets */
- MR_MEMPOLICY_MBIND,
- MR_NUMA_MISPLACED,
- MR_CONTIG_RANGE,
- MR_LONGTERM_PIN,
- MR_DEMOTION,
- MR_TYPES
-};
-
+/* Defined in mm/debug.c: */
extern const char *migrate_reason_names[MR_TYPES];
#ifdef CONFIG_MIGRATION
void folio_migrate_copy(struct folio *newfolio, struct folio *folio);
int folio_migrate_mapping(struct address_space *mapping,
struct folio *newfolio, struct folio *folio, int extra_count);
+
+extern bool numa_demotion_enabled;
#else
static inline void putback_movable_pages(struct list_head *l) {}
{
return -ENOSYS;
}
+
+#define numa_demotion_enabled false
#endif /* CONFIG_MIGRATION */
#ifdef CONFIG_COMPACTION
*/
#define MIGRATE_PFN_VALID (1UL << 0)
#define MIGRATE_PFN_MIGRATE (1UL << 1)
-#define MIGRATE_PFN_LOCKED (1UL << 2)
#define MIGRATE_PFN_WRITE (1UL << 3)
#define MIGRATE_PFN_SHIFT 6
MIGRATE_SYNC_NO_COPY,
};
+enum migrate_reason {
+ MR_COMPACTION,
+ MR_MEMORY_FAILURE,
+ MR_MEMORY_HOTPLUG,
+ MR_SYSCALL, /* also applies to cpusets */
+ MR_MEMPOLICY_MBIND,
+ MR_NUMA_MISPLACED,
+ MR_CONTIG_RANGE,
+ MR_LONGTERM_PIN,
+ MR_DEMOTION,
+ MR_TYPES
+};
+
#endif /* MIGRATE_MODE_H_INCLUDED */
GENMASK(31 - ESW_TUN_ID_BITS - ESW_RESERVED_BITS, \
ESW_TUN_OPTS_OFFSET + 1)
-u8 mlx5_eswitch_mode(struct mlx5_core_dev *dev);
+u8 mlx5_eswitch_mode(const struct mlx5_core_dev *dev);
u16 mlx5_eswitch_get_total_vports(const struct mlx5_core_dev *dev);
struct mlx5_core_dev *mlx5_eswitch_get_core_dev(struct mlx5_eswitch *esw);
#else /* CONFIG_MLX5_ESWITCH */
-static inline u8 mlx5_eswitch_mode(struct mlx5_core_dev *dev)
+static inline u8 mlx5_eswitch_mode(const struct mlx5_core_dev *dev)
{
return MLX5_ESWITCH_NONE;
}
}
#endif
-extern void *kvmalloc_node(size_t size, gfp_t flags, int node);
-static inline void *kvmalloc(size_t size, gfp_t flags)
-{
- return kvmalloc_node(size, flags, NUMA_NO_NODE);
-}
-static inline void *kvzalloc_node(size_t size, gfp_t flags, int node)
-{
- return kvmalloc_node(size, flags | __GFP_ZERO, node);
-}
-static inline void *kvzalloc(size_t size, gfp_t flags)
-{
- return kvmalloc(size, flags | __GFP_ZERO);
-}
-
-static inline void *kvmalloc_array(size_t n, size_t size, gfp_t flags)
-{
- size_t bytes;
-
- if (unlikely(check_mul_overflow(n, size, &bytes)))
- return NULL;
-
- return kvmalloc(bytes, flags);
-}
-
-static inline void *kvcalloc(size_t n, size_t size, gfp_t flags)
-{
- return kvmalloc_array(n, size, flags | __GFP_ZERO);
-}
-
-extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize,
- gfp_t flags);
-extern void kvfree(const void *addr);
-extern void kvfree_sensitive(const void *addr, size_t len);
-
static inline int head_compound_mapcount(struct page *head)
{
return atomic_read(compound_mapcount_ptr(head)) + 1;
void split_page(struct page *page, unsigned int order);
void folio_copy(struct folio *dst, struct folio *src);
+unsigned long nr_free_buffer_pages(void);
+
/*
* Compound pages have a destructor function. Provide a
* prototype for that function and accessor functions.
* Parameter block passed down to zap_pte_range in exceptional cases.
*/
struct zap_details {
- struct address_space *check_mapping; /* Check page->mapping if set */
- pgoff_t first_index; /* Lowest page->index to unmap */
- pgoff_t last_index; /* Highest page->index to unmap */
+ struct address_space *zap_mapping; /* Check page->mapping if set */
struct page *single_page; /* Locked page to be unmapped */
};
+/*
+ * We set details->zap_mappings when we want to unmap shared but keep private
+ * pages. Return true if skip zapping this page, false otherwise.
+ */
+static inline bool
+zap_skip_check_mapping(struct zap_details *details, struct page *page)
+{
+ if (!details || !page)
+ return false;
+
+ return details->zap_mapping &&
+ (details->zap_mapping != page_rmapping(page));
+}
+
struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr,
pte_t pte);
struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr,
* unsigned long max_zone_pfns[MAX_NR_ZONES] = {max_dma, max_normal_pfn,
* max_highmem_pfn};
* for_each_valid_physical_page_range()
- * memblock_add_node(base, size, nid)
+ * memblock_add_node(base, size, nid, MEMBLOCK_NONE)
* free_area_init(max_zone_pfns);
*/
void free_area_init(unsigned long *max_zone_pfn);
unsigned long, unsigned long, enum meminit_context,
struct vmem_altmap *, int migratetype);
extern void setup_per_zone_wmarks(void);
+extern void calculate_min_free_kbytes(void);
extern int __meminit init_per_zone_wmark_min(void);
extern void mem_init(void);
extern void __init mmap_init(void);
struct page_pool *pp;
unsigned long _pp_mapping_pad;
unsigned long dma_addr;
- atomic_long_t pp_frag_count;
+ union {
+ /**
+ * dma_addr_upper: might require a 64-bit
+ * value on 32-bit architectures.
+ */
+ unsigned long dma_addr_upper;
+ /**
+ * For frag page support, not supported in
+ * 32-bit architectures with 64-bit DMA.
+ */
+ atomic_long_t pp_frag_count;
+ };
};
struct { /* slab, slob and slub */
union {
struct page *next;
#ifdef CONFIG_64BIT
int pages; /* Nr of pages left */
- int pobjects; /* Approximate count */
#else
short int pages;
- short int pobjects;
#endif
};
};
NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */
NR_DIRTIED, /* page dirtyings since bootup */
NR_WRITTEN, /* page writings since bootup */
+ NR_THROTTLED_WRITTEN, /* NR_WRITTEN while reclaim throttled */
NR_KERNEL_MISC_RECLAIMABLE, /* reclaimable non-slab kernel pages */
NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */
NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */
NR_LRU_LISTS
};
+enum vmscan_throttle_state {
+ VMSCAN_THROTTLE_WRITEBACK,
+ VMSCAN_THROTTLE_ISOLATED,
+ VMSCAN_THROTTLE_NOPROGRESS,
+ NR_VMSCAN_THROTTLE,
+};
+
#define for_each_lru(lru) for (lru = 0; lru < NR_LRU_LISTS; lru++)
#define for_each_evictable_lru(lru) for (lru = 0; lru <= LRU_ACTIVE_FILE; lru++)
int node_id;
wait_queue_head_t kswapd_wait;
wait_queue_head_t pfmemalloc_wait;
+
+ /* workqueues for throttling reclaim for different reasons. */
+ wait_queue_head_t reclaim_wait[NR_VMSCAN_THROTTLE];
+
+ atomic_t nr_writeback_throttled;/* nr of writeback-throttled tasks */
+ unsigned long nr_reclaim_start; /* nr pages written while throttled
+ * when throttling started. */
struct task_struct *kswapd; /* Protected by
mem_hotplug_begin/end() */
int kswapd_order;
#define for_each_zone_zonelist(zone, z, zlist, highidx) \
for_each_zone_zonelist_nodemask(zone, z, zlist, highidx, NULL)
+/* Whether the 'nodes' are all movable nodes */
+static inline bool movable_only_nodes(nodemask_t *nodes)
+{
+ struct zonelist *zonelist;
+ struct zoneref *z;
+ int nid;
+
+ if (nodes_empty(*nodes))
+ return false;
+
+ /*
+ * We can chose arbitrary node from the nodemask to get a
+ * zonelist as they are interlinked. We just need to find
+ * at least one zone that can satisfy kernel allocations.
+ */
+ nid = first_node(*nodes);
+ zonelist = &NODE_DATA(nid)->node_zonelists[ZONELIST_FALLBACK];
+ z = first_zones_zonelist(zonelist, ZONE_NORMAL, nodes);
+ return (!z->zone) ? true : false;
+}
+
+
#ifdef CONFIG_SPARSEMEM
#include <asm/sparsemem.h>
#endif
if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS)
return 0;
- ms = __nr_to_section(pfn_to_section_nr(pfn));
+ ms = __pfn_to_section(pfn);
if (!valid_section(ms))
return 0;
/*
{
if (pfn_to_section_nr(pfn) >= NR_MEM_SECTIONS)
return 0;
- return present_section(__nr_to_section(pfn_to_section_nr(pfn)));
+ return present_section(__pfn_to_section(pfn));
}
static inline unsigned long next_present_section_nr(unsigned long section_nr)
kernel_ulong_t driver_data;
};
+/* ISHTP (Integrated Sensor Hub Transport Protocol) */
+
+#define ISHTP_MODULE_PREFIX "ishtp:"
+
+/**
+ * struct ishtp_device_id - ISHTP device identifier
+ * @guid: GUID of the device.
+ * @driver_data: pointer to driver specific data
+ */
+struct ishtp_device_id {
+ guid_t guid;
+ kernel_ulong_t driver_data;
+};
+
#endif /* LINUX_MOD_DEVICETABLE_H */
u8 is_msix : 1;
u8 multiple : 3;
u8 multi_cap : 3;
- u8 maskbit : 1;
+ u8 can_mask : 1;
u8 is_64 : 1;
u8 is_virtual : 1;
u16 entry_nr;
uint8_t *oobbuf;
};
-#define MTD_MAX_OOBFREE_ENTRIES_LARGE 32
-#define MTD_MAX_ECCPOS_ENTRIES_LARGE 640
/**
* struct mtd_oob_region - oob region definition
* @offset: region offset
struct nd_namespace_io nsio;
unsigned long lbasize;
char *alt_name;
- u8 *uuid;
+ uuid_t *uuid;
int id;
};
struct nd_namespace_blk {
struct nd_namespace_common common;
char *alt_name;
- u8 *uuid;
+ uuid_t *uuid;
int id;
unsigned long lbasize;
resource_size_t size;
short error; /* 0 or error that occurred */
loff_t i_size; /* Size of the file */
loff_t start; /* Start position */
- pgoff_t no_unlock_page; /* Don't unlock this page after read */
+ pgoff_t no_unlock_folio; /* Don't unlock this folio after read */
refcount_t usage;
unsigned long flags;
#define NETFS_RREQ_INCOMPLETE_IO 0 /* Some ioreqs terminated short or with error */
#define NETFS_RREQ_WRITE_TO_CACHE 1 /* Need to write to the cache */
-#define NETFS_RREQ_NO_UNLOCK_PAGE 2 /* Don't unlock no_unlock_page on completion */
-#define NETFS_RREQ_DONT_UNLOCK_PAGES 3 /* Don't unlock the pages on completion */
+#define NETFS_RREQ_NO_UNLOCK_FOLIO 2 /* Don't unlock no_unlock_folio on completion */
+#define NETFS_RREQ_DONT_UNLOCK_FOLIOS 3 /* Don't unlock the folios on completion */
#define NETFS_RREQ_FAILED 4 /* The request failed */
#define NETFS_RREQ_IN_PROGRESS 5 /* Unlocked when the request completes */
const struct netfs_read_request_ops *netfs_ops;
void (*issue_op)(struct netfs_read_subrequest *subreq);
bool (*is_still_valid)(struct netfs_read_request *rreq);
int (*check_write_begin)(struct file *file, loff_t pos, unsigned len,
- struct page *page, void **_fsdata);
+ struct folio *folio, void **_fsdata);
void (*done)(struct netfs_read_request *rreq);
void (*cleanup)(struct address_space *mapping, void *netfs_priv);
};
const struct netfs_read_request_ops *,
void *);
extern int netfs_readpage(struct file *,
- struct page *,
+ struct folio *,
const struct netfs_read_request_ops *,
void *);
extern int netfs_write_begin(struct file *, struct address_space *,
- loff_t, unsigned int, unsigned int, struct page **,
+ loff_t, unsigned int, unsigned int, struct folio **,
void **,
const struct netfs_read_request_ops *,
void *);
NFS4ERR_XATTR2BIG = 10096,
};
+/* error codes for internal client use */
+#define NFS4ERR_RESET_TO_MDS 12001
+#define NFS4ERR_RESET_TO_PNFS 12002
+
static inline bool seqid_mutating_err(u32 err)
{
/* See RFC 7530, section 9.1.7 */
fl_owner_t flock_owner;
struct dentry *dentry;
const struct cred *cred;
- struct rpc_cred *ll_cred; /* low-level cred - use to check for expiry */
+ struct rpc_cred __rcu *ll_cred; /* low-level cred - use to check for expiry */
struct nfs4_state *state;
fmode_t mode;
__be32 verf[NFS_DIR_VERIFIER_SIZE];
__u64 dir_cookie;
__u64 dup_cookie;
+ pgoff_t page_index;
signed char duped;
};
unsigned long attrtimeo_timestamp;
unsigned long attr_gencount;
- /* "Generation counter" for the attribute cache. This is
- * bumped whenever we update the metadata on the
- * server.
- */
- unsigned long cache_change_attribute;
struct rb_root access_cache;
struct list_head access_cache_entry_lru;
struct list_head access_cache_inode_lru;
- /*
- * This is the cookie verifier used for NFSv3 readdir
- * operations
- */
- __be32 cookieverf[NFS_DIR_VERIFIER_SIZE];
-
- atomic_long_t nrequests;
- struct nfs_mds_commit_info commit_info;
+ union {
+ /* Directory */
+ struct {
+ /* "Generation counter" for the attribute cache.
+ * This is bumped whenever we update the metadata
+ * on the server.
+ */
+ unsigned long cache_change_attribute;
+ /*
+ * This is the cookie verifier used for NFSv3 readdir
+ * operations
+ */
+ __be32 cookieverf[NFS_DIR_VERIFIER_SIZE];
+ /* Readers: in-flight sillydelete RPC calls */
+ /* Writers: rmdir */
+ struct rw_semaphore rmdir_sem;
+ };
+ /* Regular file */
+ struct {
+ atomic_long_t nrequests;
+ struct nfs_mds_commit_info commit_info;
+ struct mutex commit_mutex;
+ };
+ };
/* Open contexts for shared mmap writes */
struct list_head open_files;
- /* Readers: in-flight sillydelete RPC calls */
- /* Writers: rmdir */
- struct rw_semaphore rmdir_sem;
- struct mutex commit_mutex;
-
- /* track last access to cached pages */
- unsigned long page_index;
-
#if IS_ENABLED(CONFIG_NFS_V4)
struct nfs4_cached_acl *nfs4_acl;
/* NFSv4 state */
#define NFS_INO_INVALIDATING (3) /* inode is being invalidated */
#define NFS_INO_FSCACHE (5) /* inode can be cached by FS-Cache */
#define NFS_INO_FSCACHE_LOCK (6) /* FS-Cache cookie management lock */
+#define NFS_INO_FORCE_READDIR (7) /* force readdirplus */
#define NFS_INO_LAYOUTCOMMIT (9) /* layoutcommit required */
#define NFS_INO_LAYOUTCOMMITTING (10) /* layoutcommit inflight */
#define NFS_INO_LAYOUTSTATS (11) /* layoutstats inflight */
extern void nfs_set_inode_stale(struct inode *inode);
extern void nfs_invalidate_atime(struct inode *);
extern struct inode *nfs_fhget(struct super_block *, struct nfs_fh *,
- struct nfs_fattr *, struct nfs4_label *);
+ struct nfs_fattr *);
struct inode *nfs_ilookup(struct super_block *sb, struct nfs_fattr *, struct nfs_fh *);
extern int nfs_refresh_inode(struct inode *, struct nfs_fattr *);
extern int nfs_post_op_update_inode(struct inode *inode, struct nfs_fattr *fattr);
extern int nfs_revalidate_mapping_rcu(struct inode *inode);
extern int nfs_setattr(struct user_namespace *, struct dentry *, struct iattr *);
extern void nfs_setattr_update_inode(struct inode *inode, struct iattr *attr, struct nfs_fattr *);
-extern void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr,
- struct nfs4_label *label);
+extern void nfs_setsecurity(struct inode *inode, struct nfs_fattr *fattr);
extern struct nfs_open_context *get_nfs_open_context(struct nfs_open_context *ctx);
extern void put_nfs_open_context(struct nfs_open_context *ctx);
extern struct nfs_open_context *nfs_find_open_context(struct inode *inode, const struct cred *cred, fmode_t mode);
extern unsigned long nfs_inc_attr_generation_counter(void);
extern struct nfs_fattr *nfs_alloc_fattr(void);
+extern struct nfs_fattr *nfs_alloc_fattr_with_label(struct nfs_server *server);
+
+static inline void nfs4_label_free(struct nfs4_label *label)
+{
+#ifdef CONFIG_NFS_V4_SECURITY_LABEL
+ if (label) {
+ kfree(label->label);
+ kfree(label);
+ }
+#endif
+}
static inline void nfs_free_fattr(const struct nfs_fattr *fattr)
{
+ if (fattr)
+ nfs4_label_free(fattr->label);
kfree(fattr);
}
extern void nfs_clear_verifier_delegated(struct inode *inode);
#endif /* IS_ENABLED(CONFIG_NFS_V4) */
extern struct dentry *nfs_add_or_obtain(struct dentry *dentry,
- struct nfs_fh *fh, struct nfs_fattr *fattr,
- struct nfs4_label *label);
+ struct nfs_fh *fh, struct nfs_fattr *fattr);
extern int nfs_instantiate(struct dentry *dentry, struct nfs_fh *fh,
- struct nfs_fattr *fattr, struct nfs4_label *label);
+ struct nfs_fattr *fattr);
extern int nfs_may_open(struct inode *inode, const struct cred *cred, int openflags);
extern void nfs_access_zap_cache(struct inode *inode);
extern int nfs_access_get_cached(struct inode *inode, const struct cred *cred, struct nfs_access_entry *res,
extern int nfs_commit_inode(struct inode *, int);
extern struct nfs_commit_data *nfs_commitdata_alloc(bool never_fail);
extern void nfs_commit_free(struct nfs_commit_data *data);
+bool nfs_commit_end(struct nfs_mds_commit_info *cinfo);
static inline int
nfs_have_writebacks(struct inode *inode)
{
- return atomic_long_read(&NFS_I(inode)->nrequests) != 0;
+ if (S_ISREG(inode->i_mode))
+ return atomic_long_read(&NFS_I(inode)->nrequests) != 0;
+ return 0;
}
/*
struct nfs4_change_info cinfo;
__u32 rflags;
struct nfs_fattr * f_attr;
- struct nfs4_label *f_label;
struct nfs_seqid * seqid;
const struct nfs_server *server;
fmode_t delegation_type;
int eof;
struct nfs_fh * fh;
struct nfs_fattr * fattr;
- struct nfs4_label *label;
unsigned char d_type;
struct nfs_server * server;
};
struct nfs_setattrres {
struct nfs4_sequence_res seq_res;
struct nfs_fattr * fattr;
- struct nfs4_label *label;
const struct nfs_server * server;
};
const struct nfs_server * server;
struct nfs_fh * fh;
struct nfs_fattr * fattr;
- struct nfs4_label *label;
struct nfs4_change_info dir_cinfo;
};
struct nfs4_sequence_res seq_res;
const struct nfs_server * server;
struct nfs_fattr * fattr;
- struct nfs4_label *label;
};
struct nfs4_link_arg {
struct nfs4_sequence_res seq_res;
const struct nfs_server * server;
struct nfs_fattr * fattr;
- struct nfs4_label *label;
struct nfs4_change_info cinfo;
struct nfs_fattr * dir_attr;
};
const struct nfs_server * server;
struct nfs_fattr * fattr;
struct nfs_fh * fh;
- struct nfs4_label *label;
};
struct nfs4_lookupp_arg {
const struct nfs_server *server;
struct nfs_fattr *fattr;
struct nfs_fh *fh;
- struct nfs4_label *label;
};
struct nfs4_lookup_root_arg {
int (*submount) (struct fs_context *, struct nfs_server *);
int (*try_get_tree) (struct fs_context *);
int (*getattr) (struct nfs_server *, struct nfs_fh *,
- struct nfs_fattr *, struct nfs4_label *,
- struct inode *);
+ struct nfs_fattr *, struct inode *);
int (*setattr) (struct dentry *, struct nfs_fattr *,
struct iattr *);
int (*lookup) (struct inode *, struct dentry *,
- struct nfs_fh *, struct nfs_fattr *,
- struct nfs4_label *);
+ struct nfs_fh *, struct nfs_fattr *);
int (*lookupp) (struct inode *, struct nfs_fh *,
- struct nfs_fattr *, struct nfs4_label *);
+ struct nfs_fattr *);
int (*access) (struct inode *, struct nfs_access_entry *);
int (*readlink)(struct inode *, struct page *, unsigned int,
unsigned int);
struct device dev;
struct list_head access_list;
-#if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_HUGETLBFS)
+#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_HUGETLBFS)
struct work_struct node_work;
#endif
#ifdef CONFIG_HMEM_REPORTING
extern struct node *node_devices[];
typedef void (*node_registration_func_t)(struct node *);
-#if defined(CONFIG_MEMORY_HOTPLUG_SPARSE) && defined(CONFIG_NUMA)
+#if defined(CONFIG_MEMORY_HOTPLUG) && defined(CONFIG_NUMA)
void link_mem_sections(int nid, unsigned long start_pfn,
unsigned long end_pfn,
enum meminit_context context);
#define PAGE_POISON_PATTERN -1l
static inline int PagePoisoned(const struct page *page)
{
- return page->flags == PAGE_POISON_PATTERN;
+ return READ_ONCE(page->flags) == PAGE_POISON_PATTERN;
}
#ifdef CONFIG_DEBUG_VM
extern struct static_key_false page_owner_inited;
extern struct page_ext_operations page_owner_ops;
-extern void __reset_page_owner(struct page *page, unsigned int order);
+extern void __reset_page_owner(struct page *page, unsigned short order);
extern void __set_page_owner(struct page *page,
- unsigned int order, gfp_t gfp_mask);
+ unsigned short order, gfp_t gfp_mask);
extern void __split_page_owner(struct page *page, unsigned int nr);
extern void __folio_copy_owner(struct folio *newfolio, struct folio *old);
extern void __set_page_owner_migrate_reason(struct page *page, int reason);
extern void pagetypeinfo_showmixedcount_print(struct seq_file *m,
pg_data_t *pgdat, struct zone *zone);
-static inline void reset_page_owner(struct page *page, unsigned int order)
+static inline void reset_page_owner(struct page *page, unsigned short order)
{
if (static_branch_unlikely(&page_owner_inited))
__reset_page_owner(page, order);
}
static inline void set_page_owner(struct page *page,
- unsigned int order, gfp_t gfp_mask)
+ unsigned short order, gfp_t gfp_mask)
{
if (static_branch_unlikely(&page_owner_inited))
__set_page_owner(page, order, gfp_mask);
__dump_page_owner(page);
}
#else
-static inline void reset_page_owner(struct page *page, unsigned int order)
+static inline void reset_page_owner(struct page *page, unsigned short order)
{
}
static inline void set_page_owner(struct page *page,
{
}
static inline void split_page_owner(struct page *page,
- unsigned int order)
+ unsigned short order)
{
}
static inline void folio_copy_owner(struct folio *newfolio, struct folio *folio)
}
/*
+ * mapping_shrinkable - test if page cache state allows inode reclaim
+ * @mapping: the page cache mapping
+ *
+ * This checks the mapping's cache state for the pupose of inode
+ * reclaim and LRU management.
+ *
+ * The caller is expected to hold the i_lock, but is not required to
+ * hold the i_pages lock, which usually protects cache state. That's
+ * because the i_lock and the list_lru lock that protect the inode and
+ * its LRU state don't nest inside the irq-safe i_pages lock.
+ *
+ * Cache deletions are performed under the i_lock, which ensures that
+ * when an inode goes empty, it will reliably get queued on the LRU.
+ *
+ * Cache additions do not acquire the i_lock and may race with this
+ * check, in which case we'll report the inode as shrinkable when it
+ * has cache pages. This is okay: the shrinker also checks the
+ * refcount and the referenced bit, which will be elevated or set in
+ * the process of adding new cache pages to an inode.
+ */
+static inline bool mapping_shrinkable(struct address_space *mapping)
+{
+ void *head;
+
+ /*
+ * On highmem systems, there could be lowmem pressure from the
+ * inodes before there is highmem pressure from the page
+ * cache. Make inodes shrinkable regardless of cache state.
+ */
+ if (IS_ENABLED(CONFIG_HIGHMEM))
+ return true;
+
+ /* Cache completely empty? Shrink away. */
+ head = rcu_access_pointer(mapping->i_pages.xa_head);
+ if (!head)
+ return true;
+
+ /*
+ * The xarray stores single offset-0 entries directly in the
+ * head pointer, which allows non-resident page cache entries
+ * to escape the shadow shrinker's list of xarray nodes. The
+ * inode shrinker needs to pick them up under memory pressure.
+ */
+ if (!xa_is_node(head) && xa_is_value(head))
+ return true;
+
+ return false;
+}
+
+/*
* Bits in mapping->flags.
*/
enum mapping_flags {
return folio_mapping(folio);
}
+/**
+ * folio_inode - Get the host inode for this folio.
+ * @folio: The folio.
+ *
+ * For folios which are in the page cache, return the inode that this folio
+ * belongs to.
+ *
+ * Do not call this for folios which aren't in the page cache.
+ */
+static inline struct inode *folio_inode(struct folio *folio)
+{
+ return folio->mapping->host;
+}
+
static inline bool page_cache_add_speculative(struct page *page, int count)
{
VM_BUG_ON_PAGE(PageTail(page), page);
}
/**
+ * folio_change_private - Change private data on a folio.
+ * @folio: Folio to change the data on.
+ * @data: Data to set on the folio.
+ *
+ * Change the private data attached to a folio and return the old
+ * data. The page must previously have had data attached and the data
+ * must be detached before the folio will be freed.
+ *
+ * Return: Data that was previously attached to the folio.
+ */
+static inline void *folio_change_private(struct folio *folio, void *data)
+{
+ void *old = folio_get_private(folio);
+
+ folio->private = data;
+ return old;
+}
+
+/**
* folio_detach_private - Detach private data from a folio.
* @folio: Folio to detach data from.
*
PCI_DEV_FLAGS_NO_FLR_RESET = (__force pci_dev_flags_t) (1 << 10),
/* Don't use Relaxed Ordering for TLPs directed at this device */
PCI_DEV_FLAGS_NO_RELAXED_ORDERING = (__force pci_dev_flags_t) (1 << 11),
+ /* Device does honor MSI masking despite saying otherwise */
+ PCI_DEV_FLAGS_HAS_MSI_MASKING = (__force pci_dev_flags_t) (1 << 12),
};
enum pci_irq_reroute_variant {
struct pci_dynids dynids;
};
-#define to_pci_driver(drv) container_of(drv, struct pci_driver, driver)
+static inline struct pci_driver *to_pci_driver(struct device_driver *drv)
+{
+ return drv ? container_of(drv, struct pci_driver, driver) : NULL;
+}
/**
* PCI_DEVICE - macro used to describe a specific PCI device
u16 pci_find_next_ext_capability(struct pci_dev *dev, u16 pos, int cap);
struct pci_bus *pci_find_next_bus(const struct pci_bus *from);
u16 pci_find_vsec_capability(struct pci_dev *dev, u16 vendor, int cap);
+u16 pci_find_dvsec_capability(struct pci_dev *dev, u16 vendor, u16 dvsec);
u64 pci_get_dsn(struct pci_dev *dev);
/* Vital Product Data routines */
ssize_t pci_read_vpd(struct pci_dev *dev, loff_t pos, size_t count, void *buf);
ssize_t pci_write_vpd(struct pci_dev *dev, loff_t pos, size_t count, const void *buf);
+ssize_t pci_read_vpd_any(struct pci_dev *dev, loff_t pos, size_t count, void *buf);
+ssize_t pci_write_vpd_any(struct pci_dev *dev, loff_t pos, size_t count, const void *buf);
/* Helper functions for low-level code (drivers/pci/setup-[bus,res].c) */
resource_size_t pcibios_retrieve_fw_addr(struct pci_dev *dev, int idx);
#define PCI_IRQ_ALL_TYPES \
(PCI_IRQ_LEGACY | PCI_IRQ_MSI | PCI_IRQ_MSIX)
-/* kmem_cache style wrapper around pci_alloc_consistent() */
-
#include <linux/dmapool.h>
-#define pci_pool dma_pool
-#define pci_pool_create(name, pdev, size, align, allocation) \
- dma_pool_create(name, &pdev->dev, size, align, allocation)
-#define pci_pool_destroy(pool) dma_pool_destroy(pool)
-#define pci_pool_alloc(pool, flags, handle) dma_pool_alloc(pool, flags, handle)
-#define pci_pool_zalloc(pool, flags, handle) \
- dma_pool_zalloc(pool, flags, handle)
-#define pci_pool_free(pool, vaddr, addr) dma_pool_free(pool, vaddr, addr)
-
struct msix_entry {
u32 vector; /* Kernel uses to write allocated vector */
u16 entry; /* Driver uses to specify entry, OS writes */
bool pci_cfg_access_trylock(struct pci_dev *dev);
void pci_cfg_access_unlock(struct pci_dev *dev);
+void pci_dev_lock(struct pci_dev *dev);
int pci_dev_trylock(struct pci_dev *dev);
void pci_dev_unlock(struct pci_dev *dev);
void pcibios_set_master(struct pci_dev *dev);
int pcibios_set_pcie_reset_state(struct pci_dev *dev,
enum pcie_reset_state state);
-int pcibios_add_device(struct pci_dev *dev);
+int pcibios_device_add(struct pci_dev *dev);
void pcibios_release_device(struct pci_dev *dev);
#ifdef CONFIG_PCI
void pcibios_penalize_isa_irq(int irq, int active);
#include <linux/preempt.h>
#include <linux/smp.h>
#include <linux/cpumask.h>
-#include <linux/printk.h>
#include <linux/pfn.h>
#include <linux/init.h>
pcpu_fc_populate_pte_fn_t populate_pte_fn);
#endif
-extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align);
+extern void __percpu *__alloc_reserved_percpu(size_t size, size_t align) __alloc_size(1);
extern bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr);
extern bool is_kernel_percpu_address(unsigned long addr);
extern void __init setup_per_cpu_areas(void);
#endif
-extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp);
-extern void __percpu *__alloc_percpu(size_t size, size_t align);
+extern void __percpu *__alloc_percpu_gfp(size_t size, size_t align, gfp_t gfp) __alloc_size(1);
+extern void __percpu *__alloc_percpu(size_t size, size_t align) __alloc_size(1);
extern void free_percpu(void __percpu *__pdata);
extern phys_addr_t per_cpu_ptr_to_phys(void *addr);
extern struct pid *pidfd_pid(const struct file *file);
struct pid *pidfd_get_pid(unsigned int fd, unsigned int *flags);
+struct task_struct *pidfd_get_task(int pidfd, unsigned int *flags);
int pidfd_create(struct pid *pid, unsigned int flags);
static inline struct pid *get_pid(struct pid *pid)
struct cros_ec_debugfs *debug_info;
bool has_kb_wake_angle;
u16 cmd_offset;
- u32 features[2];
+ struct ec_response_get_features features;
};
#define to_cros_ec_dev(dev) container_of(dev, struct cros_ec_dev, class_dev)
u32 cros_ec_get_host_event(struct cros_ec_device *ec_dev);
-int cros_ec_check_features(struct cros_ec_dev *ec, int feature);
+bool cros_ec_check_features(struct cros_ec_dev *ec, int feature);
int cros_ec_get_sensor_count(struct cros_ec_dev *ec);
+int cros_ec_command(struct cros_ec_device *ec_dev, unsigned int version, int command, void *outdata,
+ int outsize, void *indata, int insize);
+
/**
* cros_ec_get_time_ns() - Return time in ns.
*
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright (C) ST Ericsson SA 2011
- *
- * STE Ux500 Watchdog platform data
- */
-#ifndef __UX500_WDT_H
-#define __UX500_WDT_H
-
-/**
- * struct ux500_wdt_data
- */
-struct ux500_wdt_data {
- unsigned int timeout;
- bool has_28_bits_resolution;
-};
-
-#endif /* __UX500_WDT_H */
#ifndef _LINUX_PLIST_H_
#define _LINUX_PLIST_H_
-#include <linux/kernel.h>
+#include <linux/container_of.h>
#include <linux/list.h>
+#include <linux/types.h>
+
+#include <asm/bug.h>
struct plist_head {
struct list_head node_list;
struct opp_table *dev_pm_opp_register_set_opp_helper(struct device *dev, int (*set_opp)(struct dev_pm_set_opp_data *data));
void dev_pm_opp_unregister_set_opp_helper(struct opp_table *opp_table);
int devm_pm_opp_register_set_opp_helper(struct device *dev, int (*set_opp)(struct dev_pm_set_opp_data *data));
-struct opp_table *dev_pm_opp_attach_genpd(struct device *dev, const char **names, struct device ***virt_devs);
+struct opp_table *dev_pm_opp_attach_genpd(struct device *dev, const char * const *names, struct device ***virt_devs);
void dev_pm_opp_detach_genpd(struct opp_table *opp_table);
-int devm_pm_opp_attach_genpd(struct device *dev, const char **names, struct device ***virt_devs);
+int devm_pm_opp_attach_genpd(struct device *dev, const char * const *names, struct device ***virt_devs);
struct dev_pm_opp *dev_pm_opp_xlate_required_opp(struct opp_table *src_table, struct opp_table *dst_table, struct dev_pm_opp *src_opp);
int dev_pm_opp_xlate_performance_state(struct opp_table *src_table, struct opp_table *dst_table, unsigned int pstate);
int dev_pm_opp_set_rate(struct device *dev, unsigned long target_freq);
return -EOPNOTSUPP;
}
-static inline struct opp_table *dev_pm_opp_attach_genpd(struct device *dev, const char **names, struct device ***virt_devs)
+static inline struct opp_table *dev_pm_opp_attach_genpd(struct device *dev, const char * const *names, struct device ***virt_devs)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void dev_pm_opp_detach_genpd(struct opp_table *opp_table) {}
static inline int devm_pm_opp_attach_genpd(struct device *dev,
- const char **names,
+ const char * const *names,
struct device ***virt_devs)
{
return -EOPNOTSUPP;
#if defined(CONFIG_PM_OPP) && defined(CONFIG_OF)
int dev_pm_opp_of_add_table(struct device *dev);
int dev_pm_opp_of_add_table_indexed(struct device *dev, int index);
+int devm_pm_opp_of_add_table_indexed(struct device *dev, int index);
int dev_pm_opp_of_add_table_noclk(struct device *dev, int index);
+int devm_pm_opp_of_add_table_noclk(struct device *dev, int index);
void dev_pm_opp_of_remove_table(struct device *dev);
int devm_pm_opp_of_add_table(struct device *dev);
int dev_pm_opp_of_cpumask_add_table(const struct cpumask *cpumask);
return -EOPNOTSUPP;
}
+static inline int devm_pm_opp_of_add_table_indexed(struct device *dev, int index)
+{
+ return -EOPNOTSUPP;
+}
+
static inline int dev_pm_opp_of_add_table_noclk(struct device *dev, int index)
{
return -EOPNOTSUPP;
}
+static inline int devm_pm_opp_of_add_table_noclk(struct device *dev, int index)
+{
+ return -EOPNOTSUPP;
+}
+
static inline void dev_pm_opp_of_remove_table(struct device *dev)
{
}
+++ /dev/null
-/*
- * pNFS-osd on-the-wire data structures
- *
- * Copyright (C) 2007 Panasas Inc. [year of first publication]
- * All rights reserved.
- *
- * Benny Halevy <bhalevy@panasas.com>
- * Boaz Harrosh <ooo@electrozaur.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2
- * See the file COPYING included with this distribution for more details.
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- *
- * 1. Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- * notice, this list of conditions and the following disclaimer in the
- * documentation and/or other materials provided with the distribution.
- * 3. Neither the name of the Panasas company nor the names of its
- * contributors may be used to endorse or promote products derived
- * from this software without specific prior written permission.
- *
- * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED
- * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
- * DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
- * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
- * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
- * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- */
-#ifndef __PNFS_OSD_XDR_H__
-#define __PNFS_OSD_XDR_H__
-
-#include <linux/nfs_fs.h>
-
-/*
- * draft-ietf-nfsv4-minorversion-22
- * draft-ietf-nfsv4-pnfs-obj-12
- */
-
-/* Layout Structure */
-
-enum pnfs_osd_raid_algorithm4 {
- PNFS_OSD_RAID_0 = 1,
- PNFS_OSD_RAID_4 = 2,
- PNFS_OSD_RAID_5 = 3,
- PNFS_OSD_RAID_PQ = 4 /* Reed-Solomon P+Q */
-};
-
-/* struct pnfs_osd_data_map4 {
- * uint32_t odm_num_comps;
- * length4 odm_stripe_unit;
- * uint32_t odm_group_width;
- * uint32_t odm_group_depth;
- * uint32_t odm_mirror_cnt;
- * pnfs_osd_raid_algorithm4 odm_raid_algorithm;
- * };
- */
-struct pnfs_osd_data_map {
- u32 odm_num_comps;
- u64 odm_stripe_unit;
- u32 odm_group_width;
- u32 odm_group_depth;
- u32 odm_mirror_cnt;
- u32 odm_raid_algorithm;
-};
-
-/* struct pnfs_osd_objid4 {
- * deviceid4 oid_device_id;
- * uint64_t oid_partition_id;
- * uint64_t oid_object_id;
- * };
- */
-struct pnfs_osd_objid {
- struct nfs4_deviceid oid_device_id;
- u64 oid_partition_id;
- u64 oid_object_id;
-};
-
-/* For printout. I use:
- * kprint("dev(%llx:%llx)", _DEVID_LO(pointer), _DEVID_HI(pointer));
- * BE style
- */
-#define _DEVID_LO(oid_device_id) \
- (unsigned long long)be64_to_cpup((__be64 *)(oid_device_id)->data)
-
-#define _DEVID_HI(oid_device_id) \
- (unsigned long long)be64_to_cpup(((__be64 *)(oid_device_id)->data) + 1)
-
-enum pnfs_osd_version {
- PNFS_OSD_MISSING = 0,
- PNFS_OSD_VERSION_1 = 1,
- PNFS_OSD_VERSION_2 = 2
-};
-
-struct pnfs_osd_opaque_cred {
- u32 cred_len;
- void *cred;
-};
-
-enum pnfs_osd_cap_key_sec {
- PNFS_OSD_CAP_KEY_SEC_NONE = 0,
- PNFS_OSD_CAP_KEY_SEC_SSV = 1,
-};
-
-/* struct pnfs_osd_object_cred4 {
- * pnfs_osd_objid4 oc_object_id;
- * pnfs_osd_version4 oc_osd_version;
- * pnfs_osd_cap_key_sec4 oc_cap_key_sec;
- * opaque oc_capability_key<>;
- * opaque oc_capability<>;
- * };
- */
-struct pnfs_osd_object_cred {
- struct pnfs_osd_objid oc_object_id;
- u32 oc_osd_version;
- u32 oc_cap_key_sec;
- struct pnfs_osd_opaque_cred oc_cap_key;
- struct pnfs_osd_opaque_cred oc_cap;
-};
-
-/* struct pnfs_osd_layout4 {
- * pnfs_osd_data_map4 olo_map;
- * uint32_t olo_comps_index;
- * pnfs_osd_object_cred4 olo_components<>;
- * };
- */
-struct pnfs_osd_layout {
- struct pnfs_osd_data_map olo_map;
- u32 olo_comps_index;
- u32 olo_num_comps;
- struct pnfs_osd_object_cred *olo_comps;
-};
-
-/* Device Address */
-enum pnfs_osd_targetid_type {
- OBJ_TARGET_ANON = 1,
- OBJ_TARGET_SCSI_NAME = 2,
- OBJ_TARGET_SCSI_DEVICE_ID = 3,
-};
-
-/* union pnfs_osd_targetid4 switch (pnfs_osd_targetid_type4 oti_type) {
- * case OBJ_TARGET_SCSI_NAME:
- * string oti_scsi_name<>;
- *
- * case OBJ_TARGET_SCSI_DEVICE_ID:
- * opaque oti_scsi_device_id<>;
- *
- * default:
- * void;
- * };
- *
- * union pnfs_osd_targetaddr4 switch (bool ota_available) {
- * case TRUE:
- * netaddr4 ota_netaddr;
- * case FALSE:
- * void;
- * };
- *
- * struct pnfs_osd_deviceaddr4 {
- * pnfs_osd_targetid4 oda_targetid;
- * pnfs_osd_targetaddr4 oda_targetaddr;
- * uint64_t oda_lun;
- * opaque oda_systemid<>;
- * pnfs_osd_object_cred4 oda_root_obj_cred;
- * opaque oda_osdname<>;
- * };
- */
-struct pnfs_osd_targetid {
- u32 oti_type;
- struct nfs4_string oti_scsi_device_id;
-};
-
-/* struct netaddr4 {
- * // see struct rpcb in RFC1833
- * string r_netid<>; // network id
- * string r_addr<>; // universal address
- * };
- */
-struct pnfs_osd_net_addr {
- struct nfs4_string r_netid;
- struct nfs4_string r_addr;
-};
-
-struct pnfs_osd_targetaddr {
- u32 ota_available;
- struct pnfs_osd_net_addr ota_netaddr;
-};
-
-struct pnfs_osd_deviceaddr {
- struct pnfs_osd_targetid oda_targetid;
- struct pnfs_osd_targetaddr oda_targetaddr;
- u8 oda_lun[8];
- struct nfs4_string oda_systemid;
- struct pnfs_osd_object_cred oda_root_obj_cred;
- struct nfs4_string oda_osdname;
-};
-
-/* LAYOUTCOMMIT: layoutupdate */
-
-/* union pnfs_osd_deltaspaceused4 switch (bool dsu_valid) {
- * case TRUE:
- * int64_t dsu_delta;
- * case FALSE:
- * void;
- * };
- *
- * struct pnfs_osd_layoutupdate4 {
- * pnfs_osd_deltaspaceused4 olu_delta_space_used;
- * bool olu_ioerr_flag;
- * };
- */
-struct pnfs_osd_layoutupdate {
- u32 dsu_valid;
- s64 dsu_delta;
- u32 olu_ioerr_flag;
-};
-
-/* LAYOUTRETURN: I/O Rrror Report */
-
-enum pnfs_osd_errno {
- PNFS_OSD_ERR_EIO = 1,
- PNFS_OSD_ERR_NOT_FOUND = 2,
- PNFS_OSD_ERR_NO_SPACE = 3,
- PNFS_OSD_ERR_BAD_CRED = 4,
- PNFS_OSD_ERR_NO_ACCESS = 5,
- PNFS_OSD_ERR_UNREACHABLE = 6,
- PNFS_OSD_ERR_RESOURCE = 7
-};
-
-/* struct pnfs_osd_ioerr4 {
- * pnfs_osd_objid4 oer_component;
- * length4 oer_comp_offset;
- * length4 oer_comp_length;
- * bool oer_iswrite;
- * pnfs_osd_errno4 oer_errno;
- * };
- */
-struct pnfs_osd_ioerr {
- struct pnfs_osd_objid oer_component;
- u64 oer_comp_offset;
- u64 oer_comp_length;
- u32 oer_iswrite;
- u32 oer_errno;
-};
-
-/* OSD XDR Client API */
-/* Layout helpers */
-/* Layout decoding is done in two parts:
- * 1. First Call pnfs_osd_xdr_decode_layout_map to read in only the header part
- * of the layout. @iter members need not be initialized.
- * Returned:
- * @layout members are set. (@layout->olo_comps set to NULL).
- *
- * Zero on success, or negative error if passed xdr is broken.
- *
- * 2. 2nd Call pnfs_osd_xdr_decode_layout_comp() in a loop until it returns
- * false, to decode the next component.
- * Returned:
- * true if there is more to decode or false if we are done or error.
- *
- * Example:
- * struct pnfs_osd_xdr_decode_layout_iter iter;
- * struct pnfs_osd_layout layout;
- * struct pnfs_osd_object_cred comp;
- * int status;
- *
- * status = pnfs_osd_xdr_decode_layout_map(&layout, &iter, xdr);
- * if (unlikely(status))
- * goto err;
- * while(pnfs_osd_xdr_decode_layout_comp(&comp, &iter, xdr, &status)) {
- * // All of @comp strings point to inside the xdr_buffer
- * // or scrach buffer. Copy them out to user memory eg.
- * copy_single_comp(dest_comp++, &comp);
- * }
- * if (unlikely(status))
- * goto err;
- */
-
-struct pnfs_osd_xdr_decode_layout_iter {
- unsigned total_comps;
- unsigned decoded_comps;
-};
-
-extern int pnfs_osd_xdr_decode_layout_map(struct pnfs_osd_layout *layout,
- struct pnfs_osd_xdr_decode_layout_iter *iter, struct xdr_stream *xdr);
-
-extern bool pnfs_osd_xdr_decode_layout_comp(struct pnfs_osd_object_cred *comp,
- struct pnfs_osd_xdr_decode_layout_iter *iter, struct xdr_stream *xdr,
- int *err);
-
-/* Device Info helpers */
-
-/* Note: All strings inside @deviceaddr point to space inside @p.
- * @p should stay valid while @deviceaddr is in use.
- */
-extern void pnfs_osd_xdr_decode_deviceaddr(
- struct pnfs_osd_deviceaddr *deviceaddr, __be32 *p);
-
-/* layoutupdate (layout_commit) xdr helpers */
-extern int
-pnfs_osd_xdr_encode_layoutupdate(struct xdr_stream *xdr,
- struct pnfs_osd_layoutupdate *lou);
-
-/* osd_ioerror encoding (layout_return) */
-extern __be32 *pnfs_osd_xdr_ioerr_reserve_space(struct xdr_stream *xdr);
-extern void pnfs_osd_xdr_encode_ioerr(__be32 *p, struct pnfs_osd_ioerr *ioerr);
-
-#endif /* __PNFS_OSD_XDR_H__ */
#endif
#ifdef CONFIG_POSIX_CPU_TIMERS_TASK_WORK
+void clear_posix_cputimers_work(struct task_struct *p);
void posix_cputimers_init_work(void);
#else
+static inline void clear_posix_cputimers_work(struct task_struct *p) { }
static inline void posix_cputimers_init_work(void) { }
#endif
void show_regs_print_info(const char *log_lvl);
extern asmlinkage void dump_stack_lvl(const char *log_lvl) __cold;
extern asmlinkage void dump_stack(void) __cold;
+void printk_trigger_flush(void);
#else
static inline __printf(1, 0)
int vprintk(const char *s, va_list args)
static inline void dump_stack(void)
{
}
+static inline void printk_trigger_flush(void)
+{
+}
#endif
#ifdef CONFIG_SMP
#else
static inline struct pwm_device *pwm_request(int pwm_id, const char *label)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
static inline void pwm_free(struct pwm_device *pwm)
{
+ might_sleep();
}
static inline int pwm_apply_state(struct pwm_device *pwm,
const struct pwm_state *state)
{
+ might_sleep();
return -ENOTSUPP;
}
static inline int pwm_config(struct pwm_device *pwm, int duty_ns,
int period_ns)
{
+ might_sleep();
return -EINVAL;
}
static inline int pwm_enable(struct pwm_device *pwm)
{
+ might_sleep();
return -EINVAL;
}
static inline void pwm_disable(struct pwm_device *pwm)
{
+ might_sleep();
}
static inline int pwm_set_chip_data(struct pwm_device *pwm, void *data)
unsigned int index,
const char *label)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
static inline struct pwm_device *pwm_get(struct device *dev,
const char *consumer)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
struct device_node *np,
const char *con_id)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
static inline void pwm_put(struct pwm_device *pwm)
{
+ might_sleep();
}
static inline struct pwm_device *devm_pwm_get(struct device *dev,
const char *consumer)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
struct device_node *np,
const char *con_id)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
devm_fwnode_pwm_get(struct device *dev, struct fwnode_handle *fwnode,
const char *con_id)
{
+ might_sleep();
return ERR_PTR(-ENODEV);
}
#endif
#define _LINUX_RADIX_TREE_H
#include <linux/bitops.h>
-#include <linux/kernel.h>
+#include <linux/gfp.h>
#include <linux/list.h>
+#include <linux/lockdep.h>
+#include <linux/math.h>
#include <linux/percpu.h>
#include <linux/preempt.h>
#include <linux/rcupdate.h>
void *priv);
int rproc_coredump_set_elf_info(struct rproc *rproc, u8 class, u16 machine);
-static inline struct rproc_vdev *vdev_to_rvdev(struct virtio_device *vdev)
-{
- return container_of(vdev->dev.parent, struct rproc_vdev, dev);
-}
-
-static inline struct rproc *vdev_to_rproc(struct virtio_device *vdev)
-{
- struct rproc_vdev *rvdev = vdev_to_rvdev(vdev);
-
- return rvdev->rproc;
-}
-
void rproc_add_subdev(struct rproc *rproc, struct rproc_subdev *subdev);
void rproc_remove_subdev(struct rproc *rproc, struct rproc_subdev *subdev);
/* This shouldn't be possible */
WARN_ON(1);
- return ERR_PTR(-ENXIO);
+ return NULL;
}
static inline int rpmsg_send(struct rpmsg_endpoint *ept, void *data, int len)
int (*alarm_irq_enable)(struct device *, unsigned int enabled);
int (*read_offset)(struct device *, long *offset);
int (*set_offset)(struct device *, long offset);
+ int (*param_get)(struct device *, struct rtc_param *param);
+ int (*param_set)(struct device *, struct rtc_param *param);
};
struct rtc_device;
/* flags */
#define RTC_DEV_BUSY 0
+#define RTC_NO_CDEV 1
struct rtc_device {
struct device dev;
#include <linux/linkage.h>
#include <linux/types.h>
-#include <linux/kernel.h>
#include <linux/list.h>
#include <linux/spinlock.h>
#include <linux/atomic.h>
#ifndef __LINUX_SCALE_BITMAP_H
#define __LINUX_SCALE_BITMAP_H
-#include <linux/kernel.h>
+#include <linux/atomic.h>
+#include <linux/bitops.h>
+#include <linux/cache.h>
+#include <linux/list.h>
+#include <linux/log2.h>
+#include <linux/minmax.h>
+#include <linux/percpu.h>
#include <linux/slab.h>
+#include <linux/smp.h>
+#include <linux/types.h>
+#include <linux/wait.h>
struct seq_file;
extern __must_check bool do_notify_parent(struct task_struct *, int);
extern void __wake_up_parent(struct task_struct *p, struct task_struct *parent);
extern void force_sig(int);
+extern void force_fatal_sig(int);
+extern void force_exit_sig(int);
extern int send_sig(int, struct task_struct *, int);
extern int zap_other_threads(struct task_struct *p);
extern struct sigqueue *sigqueue_alloc(void);
* Protects ->fs, ->files, ->mm, ->group_info, ->comm, keyring
* subscriptions and synchronises with wait4(). Also used in procfs. Also
* pins the final release of task.io_context. Also protects ->cpuset and
- * ->cgroup.subsys[]. And ->vfork_done.
+ * ->cgroup.subsys[]. And ->vfork_done. And ->sysvshm.shm_clist.
*
* Nests both inside and outside of read_lock(&tasklist_lock).
* It must not be nested with write_lock_irq(&tasklist_lock),
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * This is the official version 1.1 of sdb.h
- */
-#ifndef __SDB_H__
-#define __SDB_H__
-#ifdef __KERNEL__
-#include <linux/types.h>
-#else
-#include <stdint.h>
-#endif
-
-/*
- * All structures are 64 bytes long and are expected
- * to live in an array, one for each interconnect.
- * Most fields of the structures are shared among the
- * various types, and most-specific fields are at the
- * beginning (for alignment reasons, and to keep the
- * magic number at the head of the interconnect record
- */
-
-/* Product, 40 bytes at offset 24, 8-byte aligned
- *
- * device_id is vendor-assigned; version is device-specific,
- * date is hex (e.g 0x20120501), name is UTF-8, blank-filled
- * and not terminated with a 0 byte.
- */
-struct sdb_product {
- uint64_t vendor_id; /* 0x18..0x1f */
- uint32_t device_id; /* 0x20..0x23 */
- uint32_t version; /* 0x24..0x27 */
- uint32_t date; /* 0x28..0x2b */
- uint8_t name[19]; /* 0x2c..0x3e */
- uint8_t record_type; /* 0x3f */
-};
-
-/*
- * Component, 56 bytes at offset 8, 8-byte aligned
- *
- * The address range is first to last, inclusive
- * (for example 0x100000 - 0x10ffff)
- */
-struct sdb_component {
- uint64_t addr_first; /* 0x08..0x0f */
- uint64_t addr_last; /* 0x10..0x17 */
- struct sdb_product product; /* 0x18..0x3f */
-};
-
-/* Type of the SDB record */
-enum sdb_record_type {
- sdb_type_interconnect = 0x00,
- sdb_type_device = 0x01,
- sdb_type_bridge = 0x02,
- sdb_type_integration = 0x80,
- sdb_type_repo_url = 0x81,
- sdb_type_synthesis = 0x82,
- sdb_type_empty = 0xFF,
-};
-
-/* Type 0: interconnect (first of the array)
- *
- * sdb_records is the length of the table including this first
- * record, version is 1. The bus type is enumerated later.
- */
-#define SDB_MAGIC 0x5344422d /* "SDB-" */
-struct sdb_interconnect {
- uint32_t sdb_magic; /* 0x00-0x03 */
- uint16_t sdb_records; /* 0x04-0x05 */
- uint8_t sdb_version; /* 0x06 */
- uint8_t sdb_bus_type; /* 0x07 */
- struct sdb_component sdb_component; /* 0x08-0x3f */
-};
-
-/* Type 1: device
- *
- * class is 0 for "custom device", other values are
- * to be standardized; ABI version is for the driver,
- * bus-specific bits are defined by each bus (see below)
- */
-struct sdb_device {
- uint16_t abi_class; /* 0x00-0x01 */
- uint8_t abi_ver_major; /* 0x02 */
- uint8_t abi_ver_minor; /* 0x03 */
- uint32_t bus_specific; /* 0x04-0x07 */
- struct sdb_component sdb_component; /* 0x08-0x3f */
-};
-
-/* Type 2: bridge
- *
- * child is the address of the nested SDB table
- */
-struct sdb_bridge {
- uint64_t sdb_child; /* 0x00-0x07 */
- struct sdb_component sdb_component; /* 0x08-0x3f */
-};
-
-/* Type 0x80: integration
- *
- * all types with bit 7 set are meta-information, so
- * software can ignore the types it doesn't know. Here we
- * just provide product information for an aggregate device
- */
-struct sdb_integration {
- uint8_t reserved[24]; /* 0x00-0x17 */
- struct sdb_product product; /* 0x08-0x3f */
-};
-
-/* Type 0x81: Top module repository url
- *
- * again, an informative field that software can ignore
- */
-struct sdb_repo_url {
- uint8_t repo_url[63]; /* 0x00-0x3e */
- uint8_t record_type; /* 0x3f */
-};
-
-/* Type 0x82: Synthesis tool information
- *
- * this informative record
- */
-struct sdb_synthesis {
- uint8_t syn_name[16]; /* 0x00-0x0f */
- uint8_t commit_id[16]; /* 0x10-0x1f */
- uint8_t tool_name[8]; /* 0x20-0x27 */
- uint32_t tool_version; /* 0x28-0x2b */
- uint32_t date; /* 0x2c-0x2f */
- uint8_t user_name[15]; /* 0x30-0x3e */
- uint8_t record_type; /* 0x3f */
-};
-
-/* Type 0xff: empty
- *
- * this allows keeping empty slots during development,
- * so they can be filled later with minimal efforts and
- * no misleading description is ever shipped -- hopefully.
- * It can also be used to pad a table to a desired length.
- */
-struct sdb_empty {
- uint8_t reserved[63]; /* 0x00-0x3e */
- uint8_t record_type; /* 0x3f */
-};
-
-/* The type of bus, for bus-specific flags */
-enum sdb_bus_type {
- sdb_wishbone = 0x00,
- sdb_data = 0x01,
-};
-
-#define SDB_WB_WIDTH_MASK 0x0f
-#define SDB_WB_ACCESS8 0x01
-#define SDB_WB_ACCESS16 0x02
-#define SDB_WB_ACCESS32 0x04
-#define SDB_WB_ACCESS64 0x08
-#define SDB_WB_LITTLE_ENDIAN 0x80
-
-#define SDB_DATA_READ 0x04
-#define SDB_DATA_WRITE 0x02
-#define SDB_DATA_EXEC 0x01
-
-#endif /* __SDB_H__ */
struct xfrm_state;
struct xfrm_user_sec_ctx;
struct seq_file;
-struct sctp_endpoint;
+struct sctp_association;
#ifdef CONFIG_MMU
extern unsigned long mmap_min_addr;
int security_tun_dev_attach_queue(void *security);
int security_tun_dev_attach(struct sock *sk, void *security);
int security_tun_dev_open(void *security);
-int security_sctp_assoc_request(struct sctp_endpoint *ep, struct sk_buff *skb);
+int security_sctp_assoc_request(struct sctp_association *asoc, struct sk_buff *skb);
int security_sctp_bind_connect(struct sock *sk, int optname,
struct sockaddr *address, int addrlen);
-void security_sctp_sk_clone(struct sctp_endpoint *ep, struct sock *sk,
+void security_sctp_sk_clone(struct sctp_association *asoc, struct sock *sk,
struct sock *newsk);
#else /* CONFIG_SECURITY_NETWORK */
return 0;
}
-static inline int security_sctp_assoc_request(struct sctp_endpoint *ep,
+static inline int security_sctp_assoc_request(struct sctp_association *asoc,
struct sk_buff *skb)
{
return 0;
return 0;
}
-static inline void security_sctp_sk_clone(struct sctp_endpoint *ep,
+static inline void security_sctp_sk_clone(struct sctp_association *asoc,
struct sock *sk,
struct sock *newsk)
{
#include <linux/types.h>
#include <linux/string.h>
+#include <linux/string_helpers.h>
#include <linux/bug.h>
#include <linux/mutex.h>
#include <linux/cpumask.h>
seq_escape_mem(m, src, strlen(src), flags, esc);
}
-void seq_escape(struct seq_file *m, const char *s, const char *esc);
+/**
+ * seq_escape - print string into buffer, escaping some characters
+ * @m: target buffer
+ * @s: NULL-terminated string
+ * @esc: set of characters that need escaping
+ *
+ * Puts string into buffer, replacing each occurrence of character from
+ * @esc with usual octal escape.
+ *
+ * Use seq_has_overflowed() to check for errors.
+ */
+static inline void seq_escape(struct seq_file *m, const char *s, const char *esc)
+{
+ seq_escape_str(m, s, ESCAPE_OCTAL, esc);
+}
void seq_hex_dump(struct seq_file *m, const char *prefix_str, int prefix_type,
int rowsize, int groupsize, const void *buf, size_t len,
#define DEFINE_PROC_SHOW_ATTRIBUTE(__name) \
static int __name ## _open(struct inode *inode, struct file *file) \
{ \
- return single_open(file, __name ## _show, inode->i_private); \
+ return single_open(file, __name ## _show, PDE_DATA(inode)); \
} \
\
static const struct proc_ops __name ## _proc_ops = { \
#define sigmask(sig) (1UL << ((sig) - 1))
#ifndef __HAVE_ARCH_SIG_SETOPS
-#include <linux/string.h>
#define _SIG_SET_BINOP(name, op) \
static inline void name(sigset_t *r, const sigset_t *a, const sigset_t *b) \
int sig;
};
+/* Used to kill the race between sigaction and forced signals */
+#define SA_IMMUTABLE 0x00800000
+
#ifndef __ARCH_UAPI_SA_FLAGS
#ifdef SA_RESTORER
#define __ARCH_UAPI_SA_FLAGS SA_RESTORER
* all frags to avoid possible bad checksum
*/
SKBFL_SHARED_FRAG = BIT(1),
+
+ /* segment contains only zerocopy data and should not be
+ * charged to the kernel memory.
+ */
+ SKBFL_PURE_ZEROCOPY = BIT(2),
};
#define SKBFL_ZEROCOPY_FRAG (SKBFL_ZEROCOPY_ENABLE | SKBFL_SHARED_FRAG)
+#define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY)
/*
* The callback notifies userspace to release buffers when skb DMA is done in
return is_zcopy ? skb_uarg(skb) : NULL;
}
+static inline bool skb_zcopy_pure(const struct sk_buff *skb)
+{
+ return skb_shinfo(skb)->flags & SKBFL_PURE_ZEROCOPY;
+}
+
+static inline bool skb_pure_zcopy_same(const struct sk_buff *skb1,
+ const struct sk_buff *skb2)
+{
+ return skb_zcopy_pure(skb1) == skb_zcopy_pure(skb2);
+}
+
static inline void net_zcopy_get(struct ubuf_info *uarg)
{
refcount_inc(&uarg->refcnt);
if (!skb_zcopy_is_nouarg(skb))
uarg->callback(skb, uarg, zerocopy_success);
- skb_shinfo(skb)->flags &= ~SKBFL_ZEROCOPY_FRAG;
+ skb_shinfo(skb)->flags &= ~SKBFL_ALL_ZEROCOPY;
}
}
return 0;
}
+/* This variant of skb_unclone() makes sure skb->truesize is not changed */
+static inline int skb_unclone_keeptruesize(struct sk_buff *skb, gfp_t pri)
+{
+ might_sleep_if(gfpflags_allow_blocking(pri));
+
+ if (skb_cloned(skb)) {
+ unsigned int save = skb->truesize;
+ int res;
+
+ res = pskb_expand_head(skb, 0, 0, pri);
+ skb->truesize = save;
+ return res;
+ }
+ return 0;
+}
+
/**
* skb_header_cloned - is the header a clone
* @skb: buffer to check
return;
}
- if (unlikely(skb->ip_summed != CHECKSUM_COMPLETE)) {
+ if (unlikely(skb->ip_summed != CHECKSUM_COMPLETE)) {
__skb_checksum_complete(skb);
skb_postpull_rcsum(skb, skb->data, ptr - (void *)skb->data);
}
return !!psock->saved_data_ready;
}
+static inline bool sk_is_tcp(const struct sock *sk)
+{
+ return sk->sk_type == SOCK_STREAM &&
+ sk->sk_protocol == IPPROTO_TCP;
+}
+
+static inline bool sk_is_udp(const struct sock *sk)
+{
+ return sk->sk_type == SOCK_DGRAM &&
+ sk->sk_protocol == IPPROTO_UDP;
+}
+
#if IS_ENABLED(CONFIG_NET_SOCK_MSG)
#define BPF_F_STRPARSER (1UL << 1)
void __init kmem_cache_init(void);
bool slab_is_available(void);
-extern bool usercopy_fallback;
-
struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
unsigned int align, slab_flags_t flags,
void (*ctor)(void *));
slab_flags_t flags,
unsigned int useroffset, unsigned int usersize,
void (*ctor)(void *));
-void kmem_cache_destroy(struct kmem_cache *);
-int kmem_cache_shrink(struct kmem_cache *);
+void kmem_cache_destroy(struct kmem_cache *s);
+int kmem_cache_shrink(struct kmem_cache *s);
/*
* Please use this macro to create slab caches. Simply specify the
/*
* Common kmalloc functions provided by all allocators
*/
-void * __must_check krealloc(const void *, size_t, gfp_t);
-void kfree(const void *);
-void kfree_sensitive(const void *);
-size_t __ksize(const void *);
-size_t ksize(const void *);
+void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __alloc_size(2);
+void kfree(const void *objp);
+void kfree_sensitive(const void *objp);
+size_t __ksize(const void *objp);
+size_t ksize(const void *objp);
#ifdef CONFIG_PRINTK
bool kmem_valid_obj(void *object);
void kmem_dump_obj(void *object);
#define kmalloc_index(s) __kmalloc_index(s, true)
#endif /* !CONFIG_SLOB */
-void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __malloc;
-void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags) __assume_slab_alignment __malloc;
-void kmem_cache_free(struct kmem_cache *, void *);
+void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1);
+void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
+void kmem_cache_free(struct kmem_cache *s, void *objp);
/*
* Bulk allocation and freeing operations. These are accelerated in an
*
* Note that interrupts must be enabled when calling these functions.
*/
-void kmem_cache_free_bulk(struct kmem_cache *, size_t, void **);
-int kmem_cache_alloc_bulk(struct kmem_cache *, gfp_t, size_t, void **);
+void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p);
+int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p);
/*
* Caller must not use kfree_bulk() on memory not originally allocated
}
#ifdef CONFIG_NUMA
-void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment __malloc;
-void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node) __assume_slab_alignment __malloc;
+void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment
+ __alloc_size(1);
+void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment
+ __malloc;
#else
-static __always_inline void *__kmalloc_node(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *__kmalloc_node(size_t size, gfp_t flags, int node)
{
return __kmalloc(size, flags);
}
#endif
#ifdef CONFIG_TRACING
-extern void *kmem_cache_alloc_trace(struct kmem_cache *, gfp_t, size_t) __assume_slab_alignment __malloc;
+extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size)
+ __assume_slab_alignment __alloc_size(3);
#ifdef CONFIG_NUMA
-extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
- gfp_t gfpflags,
- int node, size_t size) __assume_slab_alignment __malloc;
+extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
+ int node, size_t size) __assume_slab_alignment
+ __alloc_size(4);
#else
-static __always_inline void *
-kmem_cache_alloc_node_trace(struct kmem_cache *s,
- gfp_t gfpflags,
- int node, size_t size)
+static __always_inline __alloc_size(4) void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
+ gfp_t gfpflags, int node, size_t size)
{
return kmem_cache_alloc_trace(s, gfpflags, size);
}
#endif /* CONFIG_NUMA */
#else /* CONFIG_TRACING */
-static __always_inline void *kmem_cache_alloc_trace(struct kmem_cache *s,
- gfp_t flags, size_t size)
+static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s,
+ gfp_t flags, size_t size)
{
void *ret = kmem_cache_alloc(s, flags);
return ret;
}
-static __always_inline void *
-kmem_cache_alloc_node_trace(struct kmem_cache *s,
- gfp_t gfpflags,
- int node, size_t size)
+static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
+ int node, size_t size)
{
void *ret = kmem_cache_alloc_node(s, gfpflags, node);
}
#endif /* CONFIG_TRACING */
-extern void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment __malloc;
+extern void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment
+ __alloc_size(1);
#ifdef CONFIG_TRACING
-extern void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment __malloc;
+extern void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
+ __assume_page_alignment __alloc_size(1);
#else
-static __always_inline void *
-kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
+static __always_inline __alloc_size(1) void *kmalloc_order_trace(size_t size, gfp_t flags,
+ unsigned int order)
{
return kmalloc_order(size, flags, order);
}
#endif
-static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t flags)
{
unsigned int order = get_order(size);
return kmalloc_order_trace(size, flags, order);
* Try really hard to succeed the allocation but fail
* eventually.
*/
-static __always_inline void *kmalloc(size_t size, gfp_t flags)
+static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
{
if (__builtin_constant_p(size)) {
#ifndef CONFIG_SLOB
return __kmalloc(size, flags);
}
-static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
+static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
{
#ifndef CONFIG_SLOB
if (__builtin_constant_p(size) &&
* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline void *kmalloc_array(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *kmalloc_array(size_t n, size_t size, gfp_t flags)
{
size_t bytes;
* @new_size: new size of a single member of the array
* @flags: the type of memory to allocate (see kmalloc)
*/
-static __must_check inline void *
-krealloc_array(void *p, size_t new_n, size_t new_size, gfp_t flags)
+static inline __alloc_size(2, 3) void * __must_check krealloc_array(void *p,
+ size_t new_n,
+ size_t new_size,
+ gfp_t flags)
{
size_t bytes;
* @size: element size.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline void *kcalloc(size_t n, size_t size, gfp_t flags)
+static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flags)
{
return kmalloc_array(n, size, flags | __GFP_ZERO);
}
* allocator where we care about the real place the memory allocation
* request comes from.
*/
-extern void *__kmalloc_track_caller(size_t, gfp_t, unsigned long);
+extern void *__kmalloc_track_caller(size_t size, gfp_t flags, unsigned long caller)
+ __alloc_size(1);
#define kmalloc_track_caller(size, flags) \
__kmalloc_track_caller(size, flags, _RET_IP_)
-static inline void *kmalloc_array_node(size_t n, size_t size, gfp_t flags,
- int node)
+static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags,
+ int node)
{
size_t bytes;
return __kmalloc_node(bytes, flags, node);
}
-static inline void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node)
+static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t flags, int node)
{
return kmalloc_array_node(n, size, flags | __GFP_ZERO, node);
}
#ifdef CONFIG_NUMA
-extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long);
+extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
+ unsigned long caller) __alloc_size(1);
#define kmalloc_node_track_caller(size, flags, node) \
__kmalloc_node_track_caller(size, flags, node, \
_RET_IP_)
* @size: how many bytes of memory are required.
* @flags: the type of memory to allocate (see kmalloc).
*/
-static inline void *kzalloc(size_t size, gfp_t flags)
+static inline __alloc_size(1) void *kzalloc(size_t size, gfp_t flags)
{
return kmalloc(size, flags | __GFP_ZERO);
}
* @flags: the type of memory to allocate (see kmalloc).
* @node: memory node from which to allocate
*/
-static inline void *kzalloc_node(size_t size, gfp_t flags, int node)
+static inline __alloc_size(1) void *kzalloc_node(size_t size, gfp_t flags, int node)
{
return kmalloc_node(size, flags | __GFP_ZERO, node);
}
+extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_size(1);
+static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags)
+{
+ return kvmalloc_node(size, flags, NUMA_NO_NODE);
+}
+static inline __alloc_size(1) void *kvzalloc_node(size_t size, gfp_t flags, int node)
+{
+ return kvmalloc_node(size, flags | __GFP_ZERO, node);
+}
+static inline __alloc_size(1) void *kvzalloc(size_t size, gfp_t flags)
+{
+ return kvmalloc(size, flags | __GFP_ZERO);
+}
+
+static inline __alloc_size(1, 2) void *kvmalloc_array(size_t n, size_t size, gfp_t flags)
+{
+ size_t bytes;
+
+ if (unlikely(check_mul_overflow(n, size, &bytes)))
+ return NULL;
+
+ return kvmalloc(bytes, flags);
+}
+
+static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t flags)
+{
+ return kvmalloc_array(n, size, flags | __GFP_ZERO);
+}
+
+extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags)
+ __alloc_size(3);
+extern void kvfree(const void *addr);
+extern void kvfree_sensitive(const void *addr, size_t len);
+
unsigned int kmem_cache_size(struct kmem_cache *s);
void __init kmem_cache_init_late(void);
#ifdef CONFIG_SLUB_CPU_PARTIAL
/* Number of per cpu partial objects to keep around */
unsigned int cpu_partial;
+ /* Number of per cpu partial pages to keep around */
+ unsigned int cpu_partial_pages;
#endif
struct kmem_cache_order_objects oo;
struct kmem_cache_node *node[MAX_NUMNODES];
};
-#ifdef CONFIG_SLUB_CPU_PARTIAL
-#define slub_cpu_partial(s) ((s)->cpu_partial)
-#define slub_set_cpu_partial(s, n) \
-({ \
- slub_cpu_partial(s) = (n); \
-})
-#else
-#define slub_cpu_partial(s) (0)
-#define slub_set_cpu_partial(s, n)
-#endif /* CONFIG_SLUB_CPU_PARTIAL */
-
#ifdef CONFIG_SYSFS
#define SLAB_SUPPORTS_SYSFS
void sysfs_slab_unlink(struct kmem_cache *);
#ifdef CONFIG_SMP
#include <linux/preempt.h>
-#include <linux/kernel.h>
#include <linux/compiler.h>
#include <linux/thread_info.h>
#include <asm/smp.h>
/* SPDX-License-Identifier: GPL-2.0 */
/* linux/spi/ads7846.h */
-/* Touchscreen characteristics vary between boards and models. The
- * platform_data for the device's "struct device" holds this information.
- *
- * It's OK if the min/max values are zero.
- */
-enum ads7846_filter {
- ADS7846_FILTER_OK,
- ADS7846_FILTER_REPEAT,
- ADS7846_FILTER_IGNORE,
-};
-
struct ads7846_platform_data {
u16 model; /* 7843, 7845, 7846, 7873. */
u16 vref_delay_usecs; /* 0 for external vref; etc */
int gpio_pendown_debounce; /* platform specific debounce time for
* the gpio_pendown */
int (*get_pendown_state)(void);
- int (*filter_init) (const struct ads7846_platform_data *pdata,
- void **filter_data);
- int (*filter) (void *filter_data, int data_idx, int *val);
- void (*filter_cleanup)(void *filter_data);
void (*wait_for_sync)(void);
bool wakeup;
unsigned long irq_flags;
u32 input_pullup_active;
};
-extern int __max730x_remove(struct device *dev);
+extern void __max730x_remove(struct device *dev);
extern int __max730x_probe(struct max7301 *ts);
#endif
#include <linux/compiler.h>
#include <linux/irqflags.h>
#include <linux/thread_info.h>
-#include <linux/kernel.h>
#include <linux/stringify.h>
#include <linux/bottom_half.h>
#include <linux/lockdep.h>
#ifndef _LINUX_STACKDEPOT_H
#define _LINUX_STACKDEPOT_H
+#include <linux/gfp.h>
+
typedef u32 depot_stack_handle_t;
+depot_stack_handle_t __stack_depot_save(unsigned long *entries,
+ unsigned int nr_entries,
+ gfp_t gfp_flags, bool can_alloc);
+
depot_stack_handle_t stack_depot_save(unsigned long *entries,
unsigned int nr_entries, gfp_t gfp_flags);
unsigned int stack_depot_fetch(depot_stack_handle_t handle,
unsigned long **entries);
-unsigned int filter_irq_stacks(unsigned long *entries, unsigned int nr_entries);
+int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
+ int spaces);
+
+void stack_depot_print(depot_stack_handle_t stack);
#ifdef CONFIG_STACKDEPOT
int stack_depot_init(void);
unsigned int stack_trace_save_regs(struct pt_regs *regs, unsigned long *store,
unsigned int size, unsigned int skipnr);
unsigned int stack_trace_save_user(unsigned long *store, unsigned int size);
+unsigned int filter_irq_stacks(unsigned long *entries, unsigned int nr_entries);
/* Internal interfaces. Do not use in generic code */
#ifdef CONFIG_ARCH_STACKWALK
#include <linux/bits.h>
#include <linux/ctype.h>
+#include <linux/string.h>
#include <linux/types.h>
struct file;
unsigned int cl_clid; /* client id */
struct list_head cl_clients; /* Global list of clients */
struct list_head cl_tasks; /* List of tasks */
+ atomic_t cl_pid; /* task PID counter */
spinlock_t cl_lock; /* spinlock */
struct rpc_xprt __rcu * cl_xprt; /* transport */
const struct rpc_procinfo *cl_procinfo; /* procedure info */
#define RPC_TASK_MSG_PIN_WAIT 5
#define RPC_TASK_SIGNALLED 6
-#define RPC_IS_RUNNING(t) test_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
-#define rpc_set_running(t) set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
#define rpc_test_and_set_running(t) \
test_and_set_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
-#define rpc_clear_running(t) \
- do { \
- smp_mb__before_atomic(); \
- clear_bit(RPC_TASK_RUNNING, &(t)->tk_runstate); \
- smp_mb__after_atomic(); \
- } while (0)
+#define rpc_clear_running(t) clear_bit(RPC_TASK_RUNNING, &(t)->tk_runstate)
#define RPC_IS_QUEUED(t) test_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)
#define rpc_set_queued(t) set_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)
-#define rpc_clear_queued(t) \
- do { \
- smp_mb__before_atomic(); \
- clear_bit(RPC_TASK_QUEUED, &(t)->tk_runstate); \
- smp_mb__after_atomic(); \
- } while (0)
+#define rpc_clear_queued(t) clear_bit(RPC_TASK_QUEUED, &(t)->tk_runstate)
#define RPC_IS_ACTIVATED(t) test_bit(RPC_TASK_ACTIVE, &(t)->tk_runstate)
/* Need xprt with congestion control */
bool vs_need_cong_ctrl;
- /* Override dispatch function (e.g. when caching replies).
- * A return value of 0 means drop the request.
- * vs_dispatch == NULL means use default dispatcher.
- */
+ /* Dispatch function */
int (*vs_dispatch)(struct svc_rqst *, __be32 *);
};
/* process the request: */
__be32 (*pc_func)(struct svc_rqst *);
/* XDR decode args: */
- int (*pc_decode)(struct svc_rqst *, __be32 *data);
+ bool (*pc_decode)(struct svc_rqst *rqstp,
+ struct xdr_stream *xdr);
/* XDR encode result: */
- int (*pc_encode)(struct svc_rqst *, __be32 *data);
+ bool (*pc_encode)(struct svc_rqst *rqstp,
+ struct xdr_stream *xdr);
/* XDR free result: */
void (*pc_release)(struct svc_rqst *);
unsigned int pc_argsize; /* argument struct size */
unsigned int offset,
unsigned int length);
unsigned int svc_fill_write_vector(struct svc_rqst *rqstp,
- struct page **pages,
- struct kvec *first, size_t total);
+ struct xdr_buf *payload);
char *svc_fill_symlink_pathname(struct svc_rqst *rqstp,
struct kvec *first, void *p,
size_t total);
/* linux/mm/page_alloc.c */
extern unsigned long totalreserve_pages;
-extern unsigned long nr_free_buffer_pages(void);
/* Definition of global_zone_page_state not available yet */
#define nr_free_pages() global_zone_page_state(NR_FREE_PAGES)
#define SWITCHTEC_EVENT_EN_CLI BIT(2)
#define SWITCHTEC_EVENT_EN_IRQ BIT(3)
#define SWITCHTEC_EVENT_FATAL BIT(4)
+#define SWITCHTEC_EVENT_NOT_SUPP BIT(31)
#define SWITCHTEC_DMA_MRPC_EN BIT(0)
#define PERF_MAX_TRACE_SIZE 8192
-#define MAX_FILTER_STR_VAL 256 /* Should handle KSYM_SYMBOL_LEN */
+#define MAX_FILTER_STR_VAL 256U /* Should handle KSYM_SYMBOL_LEN */
enum event_trigger_type {
ETT_NONE = (0),
#define HCD_FLAG_RH_RUNNING 5 /* root hub is running? */
#define HCD_FLAG_DEAD 6 /* controller has died? */
#define HCD_FLAG_INTF_AUTHORIZED 7 /* authorize interfaces? */
-#define HCD_FLAG_DEFER_RH_REGISTER 8 /* Defer roothub registration */
/* The flags can be tested using these macros; they are likely to
* be slightly faster than test_bit().
#define HCD_WAKEUP_PENDING(hcd) ((hcd)->flags & (1U << HCD_FLAG_WAKEUP_PENDING))
#define HCD_RH_RUNNING(hcd) ((hcd)->flags & (1U << HCD_FLAG_RH_RUNNING))
#define HCD_DEAD(hcd) ((hcd)->flags & (1U << HCD_FLAG_DEAD))
-#define HCD_DEFER_RH_REGISTER(hcd) ((hcd)->flags & (1U << HCD_FLAG_DEFER_RH_REGISTER))
/*
* Specifies if interfaces are authorized by default
#else
#define MODULE_VERMAGIC_SMP ""
#endif
-#ifdef CONFIG_PREEMPT
+#ifdef CONFIG_PREEMPT_BUILD
#define MODULE_VERMAGIC_PREEMPT "preempt "
#elif defined(CONFIG_PREEMPT_RT)
#define MODULE_VERMAGIC_PREEMPT "preempt_rt "
if (hdr->gso_type != VIRTIO_NET_HDR_GSO_NONE) {
u16 gso_size = __virtio16_to_cpu(little_endian, hdr->gso_size);
+ unsigned int nh_off = p_off;
struct skb_shared_info *shinfo = skb_shinfo(skb);
+ /* UFO may not include transport header in gso_size. */
+ if (gso_type & SKB_GSO_UDP)
+ nh_off -= thlen;
+
/* Too small packets are not really GSO ones. */
- if (skb->len - p_off > gso_size) {
+ if (skb->len - nh_off > gso_size) {
shinfo->gso_size = gso_size;
shinfo->gso_type = gso_type;
#define VM_USERMAP 0x00000008 /* suitable for remap_vmalloc_range */
#define VM_DMA_COHERENT 0x00000010 /* dma_alloc_coherent */
#define VM_UNINITIALIZED 0x00000020 /* vm_struct is not fully initialized */
-#define VM_NO_GUARD 0x00000040 /* don't add guard page */
+#define VM_NO_GUARD 0x00000040 /* ***DANGEROUS*** don't add guard page */
#define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */
#define VM_FLUSH_RESET_PERMS 0x00000100 /* reset direct map and flush TLB on unmap, can't be freed in atomic context */
#define VM_MAP_PUT_PAGES 0x00000200 /* put pages and free array in vfree */
static inline unsigned long vmalloc_nr_pages(void) { return 0; }
#endif
-extern void *vmalloc(unsigned long size);
-extern void *vzalloc(unsigned long size);
-extern void *vmalloc_user(unsigned long size);
-extern void *vmalloc_node(unsigned long size, int node);
-extern void *vzalloc_node(unsigned long size, int node);
-extern void *vmalloc_32(unsigned long size);
-extern void *vmalloc_32_user(unsigned long size);
-extern void *__vmalloc(unsigned long size, gfp_t gfp_mask);
+extern void *vmalloc(unsigned long size) __alloc_size(1);
+extern void *vzalloc(unsigned long size) __alloc_size(1);
+extern void *vmalloc_user(unsigned long size) __alloc_size(1);
+extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1);
+extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1);
+extern void *vmalloc_32(unsigned long size) __alloc_size(1);
+extern void *vmalloc_32_user(unsigned long size) __alloc_size(1);
+extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1);
extern void *__vmalloc_node_range(unsigned long size, unsigned long align,
unsigned long start, unsigned long end, gfp_t gfp_mask,
pgprot_t prot, unsigned long vm_flags, int node,
- const void *caller);
+ const void *caller) __alloc_size(1);
void *__vmalloc_node(unsigned long size, unsigned long align, gfp_t gfp_mask,
- int node, const void *caller);
-void *vmalloc_no_huge(unsigned long size);
+ int node, const void *caller) __alloc_size(1);
+void *vmalloc_no_huge(unsigned long size) __alloc_size(1);
extern void vfree(const void *addr);
extern void vfree_atomic(const void *addr);
+/* SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause */
/*
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * Copyright (c) Yann Collet, Facebook, Inc.
* All rights reserved.
*
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd) and
+ * the GPLv2 (found in the COPYING file in the root directory of
+ * https://github.com/facebook/zstd). You may select, at your option, one of the
+ * above-listed licenses.
*/
-#ifndef ZSTD_H
-#define ZSTD_H
+#ifndef LINUX_ZSTD_H
+#define LINUX_ZSTD_H
-/* ====== Dependency ======*/
-#include <linux/types.h> /* size_t */
+/**
+ * This is a kernel-style API that wraps the upstream zstd API, which cannot be
+ * used directly because the symbols aren't exported. It exposes the minimal
+ * functionality which is currently required by users of zstd in the kernel.
+ * Expose extra functions from lib/zstd/zstd.h as needed.
+ */
+/* ====== Dependency ====== */
+#include <linux/types.h>
+#include <linux/zstd_errors.h>
+#include <linux/zstd_lib.h>
-/*-*****************************************************************************
- * Introduction
+/* ====== Helper Functions ====== */
+/**
+ * zstd_compress_bound() - maximum compressed size in worst case scenario
+ * @src_size: The size of the data to compress.
*
- * zstd, short for Zstandard, is a fast lossless compression algorithm,
- * targeting real-time compression scenarios at zlib-level and better
- * compression ratios. The zstd compression library provides in-memory
- * compression and decompression functions. The library supports compression
- * levels from 1 up to ZSTD_maxCLevel() which is 22. Levels >= 20, labeled
- * ultra, should be used with caution, as they require more memory.
- * Compression can be done in:
- * - a single step, reusing a context (described as Explicit memory management)
- * - unbounded multiple steps (described as Streaming compression)
- * The compression ratio achievable on small data can be highly improved using
- * compression with a dictionary in:
- * - a single step (described as Simple dictionary API)
- * - a single step, reusing a dictionary (described as Fast dictionary API)
- ******************************************************************************/
-
-/*====== Helper functions ======*/
+ * Return: The maximum compressed size in the worst case scenario.
+ */
+size_t zstd_compress_bound(size_t src_size);
/**
- * enum ZSTD_ErrorCode - zstd error codes
+ * zstd_is_error() - tells if a size_t function result is an error code
+ * @code: The function result to check for error.
*
- * Functions that return size_t can be checked for errors using ZSTD_isError()
- * and the ZSTD_ErrorCode can be extracted using ZSTD_getErrorCode().
+ * Return: Non-zero iff the code is an error.
+ */
+unsigned int zstd_is_error(size_t code);
+
+/**
+ * enum zstd_error_code - zstd error codes
*/
-typedef enum {
- ZSTD_error_no_error,
- ZSTD_error_GENERIC,
- ZSTD_error_prefix_unknown,
- ZSTD_error_version_unsupported,
- ZSTD_error_parameter_unknown,
- ZSTD_error_frameParameter_unsupported,
- ZSTD_error_frameParameter_unsupportedBy32bits,
- ZSTD_error_frameParameter_windowTooLarge,
- ZSTD_error_compressionParameter_unsupported,
- ZSTD_error_init_missing,
- ZSTD_error_memory_allocation,
- ZSTD_error_stage_wrong,
- ZSTD_error_dstSize_tooSmall,
- ZSTD_error_srcSize_wrong,
- ZSTD_error_corruption_detected,
- ZSTD_error_checksum_wrong,
- ZSTD_error_tableLog_tooLarge,
- ZSTD_error_maxSymbolValue_tooLarge,
- ZSTD_error_maxSymbolValue_tooSmall,
- ZSTD_error_dictionary_corrupted,
- ZSTD_error_dictionary_wrong,
- ZSTD_error_dictionaryCreation_failed,
- ZSTD_error_maxCode
-} ZSTD_ErrorCode;
+typedef ZSTD_ErrorCode zstd_error_code;
/**
- * ZSTD_maxCLevel() - maximum compression level available
+ * zstd_get_error_code() - translates an error function result to an error code
+ * @code: The function result for which zstd_is_error(code) is true.
*
- * Return: Maximum compression level available.
+ * Return: A unique error code for this error.
*/
-int ZSTD_maxCLevel(void);
+zstd_error_code zstd_get_error_code(size_t code);
+
/**
- * ZSTD_compressBound() - maximum compressed size in worst case scenario
- * @srcSize: The size of the data to compress.
+ * zstd_get_error_name() - translates an error function result to a string
+ * @code: The function result for which zstd_is_error(code) is true.
*
- * Return: The maximum compressed size in the worst case scenario.
+ * Return: An error string corresponding to the error code.
*/
-size_t ZSTD_compressBound(size_t srcSize);
+const char *zstd_get_error_name(size_t code);
+
/**
- * ZSTD_isError() - tells if a size_t function result is an error code
- * @code: The function result to check for error.
+ * zstd_min_clevel() - minimum allowed compression level
*
- * Return: Non-zero iff the code is an error.
+ * Return: The minimum allowed compression level.
*/
-static __attribute__((unused)) unsigned int ZSTD_isError(size_t code)
-{
- return code > (size_t)-ZSTD_error_maxCode;
-}
+int zstd_min_clevel(void);
+
/**
- * ZSTD_getErrorCode() - translates an error function result to a ZSTD_ErrorCode
- * @functionResult: The result of a function for which ZSTD_isError() is true.
+ * zstd_max_clevel() - maximum allowed compression level
*
- * Return: The ZSTD_ErrorCode corresponding to the functionResult or 0
- * if the functionResult isn't an error.
+ * Return: The maximum allowed compression level.
*/
-static __attribute__((unused)) ZSTD_ErrorCode ZSTD_getErrorCode(
- size_t functionResult)
-{
- if (!ZSTD_isError(functionResult))
- return (ZSTD_ErrorCode)0;
- return (ZSTD_ErrorCode)(0 - functionResult);
-}
+int zstd_max_clevel(void);
+
+/* ====== Parameter Selection ====== */
/**
- * enum ZSTD_strategy - zstd compression search strategy
+ * enum zstd_strategy - zstd compression search strategy
*
- * From faster to stronger.
+ * From faster to stronger. See zstd_lib.h.
*/
-typedef enum {
- ZSTD_fast,
- ZSTD_dfast,
- ZSTD_greedy,
- ZSTD_lazy,
- ZSTD_lazy2,
- ZSTD_btlazy2,
- ZSTD_btopt,
- ZSTD_btopt2
-} ZSTD_strategy;
+typedef ZSTD_strategy zstd_strategy;
/**
- * struct ZSTD_compressionParameters - zstd compression parameters
+ * struct zstd_compression_parameters - zstd compression parameters
* @windowLog: Log of the largest match distance. Larger means more
* compression, and more memory needed during decompression.
- * @chainLog: Fully searched segment. Larger means more compression, slower,
- * and more memory (useless for fast).
+ * @chainLog: Fully searched segment. Larger means more compression,
+ * slower, and more memory (useless for fast).
* @hashLog: Dispatch table. Larger means more compression,
* slower, and more memory.
* @searchLog: Number of searches. Larger means more compression and slower.
* @targetLength: Acceptable match size for optimal parser (only). Larger means
* more compression, and slower.
* @strategy: The zstd compression strategy.
+ *
+ * See zstd_lib.h.
*/
-typedef struct {
- unsigned int windowLog;
- unsigned int chainLog;
- unsigned int hashLog;
- unsigned int searchLog;
- unsigned int searchLength;
- unsigned int targetLength;
- ZSTD_strategy strategy;
-} ZSTD_compressionParameters;
+typedef ZSTD_compressionParameters zstd_compression_parameters;
/**
- * struct ZSTD_frameParameters - zstd frame parameters
- * @contentSizeFlag: Controls whether content size will be present in the frame
- * header (when known).
- * @checksumFlag: Controls whether a 32-bit checksum is generated at the end
- * of the frame for error detection.
- * @noDictIDFlag: Controls whether dictID will be saved into the frame header
- * when using dictionary compression.
+ * struct zstd_frame_parameters - zstd frame parameters
+ * @contentSizeFlag: Controls whether content size will be present in the
+ * frame header (when known).
+ * @checksumFlag: Controls whether a 32-bit checksum is generated at the
+ * end of the frame for error detection.
+ * @noDictIDFlag: Controls whether dictID will be saved into the frame
+ * header when using dictionary compression.
*
- * The default value is all fields set to 0.
+ * The default value is all fields set to 0. See zstd_lib.h.
*/
-typedef struct {
- unsigned int contentSizeFlag;
- unsigned int checksumFlag;
- unsigned int noDictIDFlag;
-} ZSTD_frameParameters;
+typedef ZSTD_frameParameters zstd_frame_parameters;
/**
- * struct ZSTD_parameters - zstd parameters
+ * struct zstd_parameters - zstd parameters
* @cParams: The compression parameters.
* @fParams: The frame parameters.
*/
-typedef struct {
- ZSTD_compressionParameters cParams;
- ZSTD_frameParameters fParams;
-} ZSTD_parameters;
+typedef ZSTD_parameters zstd_parameters;
/**
- * ZSTD_getCParams() - returns ZSTD_compressionParameters for selected level
- * @compressionLevel: The compression level from 1 to ZSTD_maxCLevel().
- * @estimatedSrcSize: The estimated source size to compress or 0 if unknown.
- * @dictSize: The dictionary size or 0 if a dictionary isn't being used.
+ * zstd_get_params() - returns zstd_parameters for selected level
+ * @level: The compression level
+ * @estimated_src_size: The estimated source size to compress or 0
+ * if unknown.
*
- * Return: The selected ZSTD_compressionParameters.
+ * Return: The selected zstd_parameters.
*/
-ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel,
- unsigned long long estimatedSrcSize, size_t dictSize);
+zstd_parameters zstd_get_params(int level,
+ unsigned long long estimated_src_size);
-/**
- * ZSTD_getParams() - returns ZSTD_parameters for selected level
- * @compressionLevel: The compression level from 1 to ZSTD_maxCLevel().
- * @estimatedSrcSize: The estimated source size to compress or 0 if unknown.
- * @dictSize: The dictionary size or 0 if a dictionary isn't being used.
- *
- * The same as ZSTD_getCParams() except also selects the default frame
- * parameters (all zero).
- *
- * Return: The selected ZSTD_parameters.
- */
-ZSTD_parameters ZSTD_getParams(int compressionLevel,
- unsigned long long estimatedSrcSize, size_t dictSize);
+/* ====== Single-pass Compression ====== */
-/*-*************************************
- * Explicit memory management
- **************************************/
+typedef ZSTD_CCtx zstd_cctx;
/**
- * ZSTD_CCtxWorkspaceBound() - amount of memory needed to initialize a ZSTD_CCtx
- * @cParams: The compression parameters to be used for compression.
+ * zstd_cctx_workspace_bound() - max memory needed to initialize a zstd_cctx
+ * @parameters: The compression parameters to be used.
*
* If multiple compression parameters might be used, the caller must call
- * ZSTD_CCtxWorkspaceBound() for each set of parameters and use the maximum
+ * zstd_cctx_workspace_bound() for each set of parameters and use the maximum
* size.
*
- * Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initCCtx().
+ * Return: A lower bound on the size of the workspace that is passed to
+ * zstd_init_cctx().
*/
-size_t ZSTD_CCtxWorkspaceBound(ZSTD_compressionParameters cParams);
+size_t zstd_cctx_workspace_bound(const zstd_compression_parameters *parameters);
/**
- * struct ZSTD_CCtx - the zstd compression context
- *
- * When compressing many times it is recommended to allocate a context just once
- * and reuse it for each successive compression operation.
- */
-typedef struct ZSTD_CCtx_s ZSTD_CCtx;
-/**
- * ZSTD_initCCtx() - initialize a zstd compression context
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace. Use ZSTD_CCtxWorkspaceBound() to
- * determine how large the workspace must be.
- *
- * Return: A compression context emplaced into workspace.
- */
-ZSTD_CCtx *ZSTD_initCCtx(void *workspace, size_t workspaceSize);
-
-/**
- * ZSTD_compressCCtx() - compress src into dst
- * @ctx: The context. Must have been initialized with a workspace at
- * least as large as ZSTD_CCtxWorkspaceBound(params.cParams).
- * @dst: The buffer to compress src into.
- * @dstCapacity: The size of the destination buffer. May be any size, but
- * ZSTD_compressBound(srcSize) is guaranteed to be large enough.
- * @src: The data to compress.
- * @srcSize: The size of the data to compress.
- * @params: The parameters to use for compression. See ZSTD_getParams().
- *
- * Return: The compressed size or an error, which can be checked using
- * ZSTD_isError().
- */
-size_t ZSTD_compressCCtx(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize, ZSTD_parameters params);
-
-/**
- * ZSTD_DCtxWorkspaceBound() - amount of memory needed to initialize a ZSTD_DCtx
- *
- * Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initDCtx().
- */
-size_t ZSTD_DCtxWorkspaceBound(void);
-
-/**
- * struct ZSTD_DCtx - the zstd decompression context
- *
- * When decompressing many times it is recommended to allocate a context just
- * once and reuse it for each successive decompression operation.
- */
-typedef struct ZSTD_DCtx_s ZSTD_DCtx;
-/**
- * ZSTD_initDCtx() - initialize a zstd decompression context
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace. Use ZSTD_DCtxWorkspaceBound() to
- * determine how large the workspace must be.
- *
- * Return: A decompression context emplaced into workspace.
- */
-ZSTD_DCtx *ZSTD_initDCtx(void *workspace, size_t workspaceSize);
-
-/**
- * ZSTD_decompressDCtx() - decompress zstd compressed src into dst
- * @ctx: The decompression context.
- * @dst: The buffer to decompress src into.
- * @dstCapacity: The size of the destination buffer. Must be at least as large
- * as the decompressed size. If the caller cannot upper bound the
- * decompressed size, then it's better to use the streaming API.
- * @src: The zstd compressed data to decompress. Multiple concatenated
- * frames and skippable frames are allowed.
- * @srcSize: The exact size of the data to decompress.
- *
- * Return: The decompressed size or an error, which can be checked using
- * ZSTD_isError().
- */
-size_t ZSTD_decompressDCtx(ZSTD_DCtx *ctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-
-/*-************************
- * Simple dictionary API
- **************************/
-
-/**
- * ZSTD_compress_usingDict() - compress src into dst using a dictionary
- * @ctx: The context. Must have been initialized with a workspace at
- * least as large as ZSTD_CCtxWorkspaceBound(params.cParams).
- * @dst: The buffer to compress src into.
- * @dstCapacity: The size of the destination buffer. May be any size, but
- * ZSTD_compressBound(srcSize) is guaranteed to be large enough.
- * @src: The data to compress.
- * @srcSize: The size of the data to compress.
- * @dict: The dictionary to use for compression.
- * @dictSize: The size of the dictionary.
- * @params: The parameters to use for compression. See ZSTD_getParams().
- *
- * Compression using a predefined dictionary. The same dictionary must be used
- * during decompression.
- *
- * Return: The compressed size or an error, which can be checked using
- * ZSTD_isError().
- */
-size_t ZSTD_compress_usingDict(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize, const void *dict, size_t dictSize,
- ZSTD_parameters params);
-
-/**
- * ZSTD_decompress_usingDict() - decompress src into dst using a dictionary
- * @ctx: The decompression context.
- * @dst: The buffer to decompress src into.
- * @dstCapacity: The size of the destination buffer. Must be at least as large
- * as the decompressed size. If the caller cannot upper bound the
- * decompressed size, then it's better to use the streaming API.
- * @src: The zstd compressed data to decompress. Multiple concatenated
- * frames and skippable frames are allowed.
- * @srcSize: The exact size of the data to decompress.
- * @dict: The dictionary to use for decompression. The same dictionary
- * must've been used to compress the data.
- * @dictSize: The size of the dictionary.
- *
- * Return: The decompressed size or an error, which can be checked using
- * ZSTD_isError().
- */
-size_t ZSTD_decompress_usingDict(ZSTD_DCtx *ctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize, const void *dict, size_t dictSize);
-
-/*-**************************
- * Fast dictionary API
- ***************************/
-
-/**
- * ZSTD_CDictWorkspaceBound() - memory needed to initialize a ZSTD_CDict
- * @cParams: The compression parameters to be used for compression.
+ * zstd_init_cctx() - initialize a zstd compression context
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspace_size: The size of workspace. Use zstd_cctx_workspace_bound() to
+ * determine how large the workspace must be.
*
- * Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initCDict().
- */
-size_t ZSTD_CDictWorkspaceBound(ZSTD_compressionParameters cParams);
-
-/**
- * struct ZSTD_CDict - a digested dictionary to be used for compression
+ * Return: A zstd compression context or NULL on error.
*/
-typedef struct ZSTD_CDict_s ZSTD_CDict;
+zstd_cctx *zstd_init_cctx(void *workspace, size_t workspace_size);
/**
- * ZSTD_initCDict() - initialize a digested dictionary for compression
- * @dictBuffer: The dictionary to digest. The buffer is referenced by the
- * ZSTD_CDict so it must outlive the returned ZSTD_CDict.
- * @dictSize: The size of the dictionary.
- * @params: The parameters to use for compression. See ZSTD_getParams().
- * @workspace: The workspace. It must outlive the returned ZSTD_CDict.
- * @workspaceSize: The workspace size. Must be at least
- * ZSTD_CDictWorkspaceBound(params.cParams).
+ * zstd_compress_cctx() - compress src into dst with the initialized parameters
+ * @cctx: The context. Must have been initialized with zstd_init_cctx().
+ * @dst: The buffer to compress src into.
+ * @dst_capacity: The size of the destination buffer. May be any size, but
+ * ZSTD_compressBound(srcSize) is guaranteed to be large enough.
+ * @src: The data to compress.
+ * @src_size: The size of the data to compress.
+ * @parameters: The compression parameters to be used.
*
- * When compressing multiple messages / blocks with the same dictionary it is
- * recommended to load it just once. The ZSTD_CDict merely references the
- * dictBuffer, so it must outlive the returned ZSTD_CDict.
- *
- * Return: The digested dictionary emplaced into workspace.
+ * Return: The compressed size or an error, which can be checked using
+ * zstd_is_error().
*/
-ZSTD_CDict *ZSTD_initCDict(const void *dictBuffer, size_t dictSize,
- ZSTD_parameters params, void *workspace, size_t workspaceSize);
+size_t zstd_compress_cctx(zstd_cctx *cctx, void *dst, size_t dst_capacity,
+ const void *src, size_t src_size, const zstd_parameters *parameters);
-/**
- * ZSTD_compress_usingCDict() - compress src into dst using a ZSTD_CDict
- * @ctx: The context. Must have been initialized with a workspace at
- * least as large as ZSTD_CCtxWorkspaceBound(cParams) where
- * cParams are the compression parameters used to initialize the
- * cdict.
- * @dst: The buffer to compress src into.
- * @dstCapacity: The size of the destination buffer. May be any size, but
- * ZSTD_compressBound(srcSize) is guaranteed to be large enough.
- * @src: The data to compress.
- * @srcSize: The size of the data to compress.
- * @cdict: The digested dictionary to use for compression.
- * @params: The parameters to use for compression. See ZSTD_getParams().
- *
- * Compression using a digested dictionary. The same dictionary must be used
- * during decompression.
- *
- * Return: The compressed size or an error, which can be checked using
- * ZSTD_isError().
- */
-size_t ZSTD_compress_usingCDict(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize, const ZSTD_CDict *cdict);
+/* ====== Single-pass Decompression ====== */
+typedef ZSTD_DCtx zstd_dctx;
/**
- * ZSTD_DDictWorkspaceBound() - memory needed to initialize a ZSTD_DDict
+ * zstd_dctx_workspace_bound() - max memory needed to initialize a zstd_dctx
*
- * Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initDDict().
- */
-size_t ZSTD_DDictWorkspaceBound(void);
-
-/**
- * struct ZSTD_DDict - a digested dictionary to be used for decompression
+ * Return: A lower bound on the size of the workspace that is passed to
+ * zstd_init_dctx().
*/
-typedef struct ZSTD_DDict_s ZSTD_DDict;
+size_t zstd_dctx_workspace_bound(void);
/**
- * ZSTD_initDDict() - initialize a digested dictionary for decompression
- * @dictBuffer: The dictionary to digest. The buffer is referenced by the
- * ZSTD_DDict so it must outlive the returned ZSTD_DDict.
- * @dictSize: The size of the dictionary.
- * @workspace: The workspace. It must outlive the returned ZSTD_DDict.
- * @workspaceSize: The workspace size. Must be at least
- * ZSTD_DDictWorkspaceBound().
- *
- * When decompressing multiple messages / blocks with the same dictionary it is
- * recommended to load it just once. The ZSTD_DDict merely references the
- * dictBuffer, so it must outlive the returned ZSTD_DDict.
+ * zstd_init_dctx() - initialize a zstd decompression context
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspace_size: The size of workspace. Use zstd_dctx_workspace_bound() to
+ * determine how large the workspace must be.
*
- * Return: The digested dictionary emplaced into workspace.
+ * Return: A zstd decompression context or NULL on error.
*/
-ZSTD_DDict *ZSTD_initDDict(const void *dictBuffer, size_t dictSize,
- void *workspace, size_t workspaceSize);
+zstd_dctx *zstd_init_dctx(void *workspace, size_t workspace_size);
/**
- * ZSTD_decompress_usingDDict() - decompress src into dst using a ZSTD_DDict
- * @ctx: The decompression context.
- * @dst: The buffer to decompress src into.
- * @dstCapacity: The size of the destination buffer. Must be at least as large
- * as the decompressed size. If the caller cannot upper bound the
- * decompressed size, then it's better to use the streaming API.
- * @src: The zstd compressed data to decompress. Multiple concatenated
- * frames and skippable frames are allowed.
- * @srcSize: The exact size of the data to decompress.
- * @ddict: The digested dictionary to use for decompression. The same
- * dictionary must've been used to compress the data.
+ * zstd_decompress_dctx() - decompress zstd compressed src into dst
+ * @dctx: The decompression context.
+ * @dst: The buffer to decompress src into.
+ * @dst_capacity: The size of the destination buffer. Must be at least as large
+ * as the decompressed size. If the caller cannot upper bound the
+ * decompressed size, then it's better to use the streaming API.
+ * @src: The zstd compressed data to decompress. Multiple concatenated
+ * frames and skippable frames are allowed.
+ * @src_size: The exact size of the data to decompress.
*
- * Return: The decompressed size or an error, which can be checked using
- * ZSTD_isError().
+ * Return: The decompressed size or an error, which can be checked using
+ * zstd_is_error().
*/
-size_t ZSTD_decompress_usingDDict(ZSTD_DCtx *dctx, void *dst,
- size_t dstCapacity, const void *src, size_t srcSize,
- const ZSTD_DDict *ddict);
+size_t zstd_decompress_dctx(zstd_dctx *dctx, void *dst, size_t dst_capacity,
+ const void *src, size_t src_size);
-
-/*-**************************
- * Streaming
- ***************************/
+/* ====== Streaming Buffers ====== */
/**
- * struct ZSTD_inBuffer - input buffer for streaming
+ * struct zstd_in_buffer - input buffer for streaming
* @src: Start of the input buffer.
* @size: Size of the input buffer.
* @pos: Position where reading stopped. Will be updated.
* Necessarily 0 <= pos <= size.
+ *
+ * See zstd_lib.h.
*/
-typedef struct ZSTD_inBuffer_s {
- const void *src;
- size_t size;
- size_t pos;
-} ZSTD_inBuffer;
+typedef ZSTD_inBuffer zstd_in_buffer;
/**
- * struct ZSTD_outBuffer - output buffer for streaming
+ * struct zstd_out_buffer - output buffer for streaming
* @dst: Start of the output buffer.
* @size: Size of the output buffer.
* @pos: Position where writing stopped. Will be updated.
* Necessarily 0 <= pos <= size.
+ *
+ * See zstd_lib.h.
*/
-typedef struct ZSTD_outBuffer_s {
- void *dst;
- size_t size;
- size_t pos;
-} ZSTD_outBuffer;
+typedef ZSTD_outBuffer zstd_out_buffer;
+/* ====== Streaming Compression ====== */
-
-/*-*****************************************************************************
- * Streaming compression - HowTo
- *
- * A ZSTD_CStream object is required to track streaming operation.
- * Use ZSTD_initCStream() to initialize a ZSTD_CStream object.
- * ZSTD_CStream objects can be reused multiple times on consecutive compression
- * operations. It is recommended to re-use ZSTD_CStream in situations where many
- * streaming operations will be achieved consecutively. Use one separate
- * ZSTD_CStream per thread for parallel execution.
- *
- * Use ZSTD_compressStream() repetitively to consume input stream.
- * The function will automatically update both `pos` fields.
- * Note that it may not consume the entire input, in which case `pos < size`,
- * and it's up to the caller to present again remaining data.
- * It returns a hint for the preferred number of bytes to use as an input for
- * the next function call.
- *
- * At any moment, it's possible to flush whatever data remains within internal
- * buffer, using ZSTD_flushStream(). `output->pos` will be updated. There might
- * still be some content left within the internal buffer if `output->size` is
- * too small. It returns the number of bytes left in the internal buffer and
- * must be called until it returns 0.
- *
- * ZSTD_endStream() instructs to finish a frame. It will perform a flush and
- * write frame epilogue. The epilogue is required for decoders to consider a
- * frame completed. Similar to ZSTD_flushStream(), it may not be able to flush
- * the full content if `output->size` is too small. In which case, call again
- * ZSTD_endStream() to complete the flush. It returns the number of bytes left
- * in the internal buffer and must be called until it returns 0.
- ******************************************************************************/
+typedef ZSTD_CStream zstd_cstream;
/**
- * ZSTD_CStreamWorkspaceBound() - memory needed to initialize a ZSTD_CStream
- * @cParams: The compression parameters to be used for compression.
+ * zstd_cstream_workspace_bound() - memory needed to initialize a zstd_cstream
+ * @cparams: The compression parameters to be used for compression.
*
* Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initCStream() and ZSTD_initCStream_usingCDict().
- */
-size_t ZSTD_CStreamWorkspaceBound(ZSTD_compressionParameters cParams);
-
-/**
- * struct ZSTD_CStream - the zstd streaming compression context
- */
-typedef struct ZSTD_CStream_s ZSTD_CStream;
-
-/*===== ZSTD_CStream management functions =====*/
-/**
- * ZSTD_initCStream() - initialize a zstd streaming compression context
- * @params: The zstd compression parameters.
- * @pledgedSrcSize: If params.fParams.contentSizeFlag == 1 then the caller must
- * pass the source size (zero means empty source). Otherwise,
- * the caller may optionally pass the source size, or zero if
- * unknown.
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace.
- * Use ZSTD_CStreamWorkspaceBound(params.cParams) to determine
- * how large the workspace must be.
- *
- * Return: The zstd streaming compression context.
+ * zstd_init_cstream().
*/
-ZSTD_CStream *ZSTD_initCStream(ZSTD_parameters params,
- unsigned long long pledgedSrcSize, void *workspace,
- size_t workspaceSize);
+size_t zstd_cstream_workspace_bound(const zstd_compression_parameters *cparams);
/**
- * ZSTD_initCStream_usingCDict() - initialize a streaming compression context
- * @cdict: The digested dictionary to use for compression.
- * @pledgedSrcSize: Optionally the source size, or zero if unknown.
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace. Call ZSTD_CStreamWorkspaceBound()
- * with the cParams used to initialize the cdict to determine
- * how large the workspace must be.
+ * zstd_init_cstream() - initialize a zstd streaming compression context
+ * @parameters The zstd parameters to use for compression.
+ * @pledged_src_size: If params.fParams.contentSizeFlag == 1 then the caller
+ * must pass the source size (zero means empty source).
+ * Otherwise, the caller may optionally pass the source
+ * size, or zero if unknown.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspace_size: The size of workspace.
+ * Use zstd_cstream_workspace_bound(params->cparams) to
+ * determine how large the workspace must be.
*
- * Return: The zstd streaming compression context.
+ * Return: The zstd streaming compression context or NULL on error.
*/
-ZSTD_CStream *ZSTD_initCStream_usingCDict(const ZSTD_CDict *cdict,
- unsigned long long pledgedSrcSize, void *workspace,
- size_t workspaceSize);
+zstd_cstream *zstd_init_cstream(const zstd_parameters *parameters,
+ unsigned long long pledged_src_size, void *workspace, size_t workspace_size);
-/*===== Streaming compression functions =====*/
/**
- * ZSTD_resetCStream() - reset the context using parameters from creation
- * @zcs: The zstd streaming compression context to reset.
- * @pledgedSrcSize: Optionally the source size, or zero if unknown.
+ * zstd_reset_cstream() - reset the context using parameters from creation
+ * @cstream: The zstd streaming compression context to reset.
+ * @pledged_src_size: Optionally the source size, or zero if unknown.
*
* Resets the context using the parameters from creation. Skips dictionary
- * loading, since it can be reused. If `pledgedSrcSize` is non-zero the frame
+ * loading, since it can be reused. If `pledged_src_size` is non-zero the frame
* content size is always written into the frame header.
*
- * Return: Zero or an error, which can be checked using ZSTD_isError().
+ * Return: Zero or an error, which can be checked using
+ * zstd_is_error().
*/
-size_t ZSTD_resetCStream(ZSTD_CStream *zcs, unsigned long long pledgedSrcSize);
+size_t zstd_reset_cstream(zstd_cstream *cstream,
+ unsigned long long pledged_src_size);
+
/**
- * ZSTD_compressStream() - streaming compress some of input into output
- * @zcs: The zstd streaming compression context.
- * @output: Destination buffer. `output->pos` is updated to indicate how much
- * compressed data was written.
- * @input: Source buffer. `input->pos` is updated to indicate how much data was
- * read. Note that it may not consume the entire input, in which case
- * `input->pos < input->size`, and it's up to the caller to present
- * remaining data again.
+ * zstd_compress_stream() - streaming compress some of input into output
+ * @cstream: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
+ * @input: Source buffer. `input->pos` is updated to indicate how much data
+ * was read. Note that it may not consume the entire input, in which
+ * case `input->pos < input->size`, and it's up to the caller to
+ * present remaining data again.
*
* The `input` and `output` buffers may be any size. Guaranteed to make some
* forward progress if `input` and `output` are not empty.
*
- * Return: A hint for the number of bytes to use as the input for the next
- * function call or an error, which can be checked using
- * ZSTD_isError().
+ * Return: A hint for the number of bytes to use as the input for the next
+ * function call or an error, which can be checked using
+ * zstd_is_error().
*/
-size_t ZSTD_compressStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output,
- ZSTD_inBuffer *input);
+size_t zstd_compress_stream(zstd_cstream *cstream, zstd_out_buffer *output,
+ zstd_in_buffer *input);
+
/**
- * ZSTD_flushStream() - flush internal buffers into output
- * @zcs: The zstd streaming compression context.
- * @output: Destination buffer. `output->pos` is updated to indicate how much
- * compressed data was written.
+ * zstd_flush_stream() - flush internal buffers into output
+ * @cstream: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
*
- * ZSTD_flushStream() must be called until it returns 0, meaning all the data
- * has been flushed. Since ZSTD_flushStream() causes a block to be ended,
+ * zstd_flush_stream() must be called until it returns 0, meaning all the data
+ * has been flushed. Since zstd_flush_stream() causes a block to be ended,
* calling it too often will degrade the compression ratio.
*
- * Return: The number of bytes still present within internal buffers or an
- * error, which can be checked using ZSTD_isError().
+ * Return: The number of bytes still present within internal buffers or an
+ * error, which can be checked using zstd_is_error().
*/
-size_t ZSTD_flushStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
-/**
- * ZSTD_endStream() - flush internal buffers into output and end the frame
- * @zcs: The zstd streaming compression context.
- * @output: Destination buffer. `output->pos` is updated to indicate how much
- * compressed data was written.
- *
- * ZSTD_endStream() must be called until it returns 0, meaning all the data has
- * been flushed and the frame epilogue has been written.
- *
- * Return: The number of bytes still present within internal buffers or an
- * error, which can be checked using ZSTD_isError().
- */
-size_t ZSTD_endStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output);
+size_t zstd_flush_stream(zstd_cstream *cstream, zstd_out_buffer *output);
/**
- * ZSTD_CStreamInSize() - recommended size for the input buffer
- *
- * Return: The recommended size for the input buffer.
- */
-size_t ZSTD_CStreamInSize(void);
-/**
- * ZSTD_CStreamOutSize() - recommended size for the output buffer
+ * zstd_end_stream() - flush internal buffers into output and end the frame
+ * @cstream: The zstd streaming compression context.
+ * @output: Destination buffer. `output->pos` is updated to indicate how much
+ * compressed data was written.
*
- * When the output buffer is at least this large, it is guaranteed to be large
- * enough to flush at least one complete compressed block.
+ * zstd_end_stream() must be called until it returns 0, meaning all the data has
+ * been flushed and the frame epilogue has been written.
*
- * Return: The recommended size for the output buffer.
+ * Return: The number of bytes still present within internal buffers or an
+ * error, which can be checked using zstd_is_error().
*/
-size_t ZSTD_CStreamOutSize(void);
+size_t zstd_end_stream(zstd_cstream *cstream, zstd_out_buffer *output);
+/* ====== Streaming Decompression ====== */
-
-/*-*****************************************************************************
- * Streaming decompression - HowTo
- *
- * A ZSTD_DStream object is required to track streaming operations.
- * Use ZSTD_initDStream() to initialize a ZSTD_DStream object.
- * ZSTD_DStream objects can be re-used multiple times.
- *
- * Use ZSTD_decompressStream() repetitively to consume your input.
- * The function will update both `pos` fields.
- * If `input->pos < input->size`, some input has not been consumed.
- * It's up to the caller to present again remaining data.
- * If `output->pos < output->size`, decoder has flushed everything it could.
- * Returns 0 iff a frame is completely decoded and fully flushed.
- * Otherwise it returns a suggested next input size that will never load more
- * than the current frame.
- ******************************************************************************/
+typedef ZSTD_DStream zstd_dstream;
/**
- * ZSTD_DStreamWorkspaceBound() - memory needed to initialize a ZSTD_DStream
- * @maxWindowSize: The maximum window size allowed for compressed frames.
+ * zstd_dstream_workspace_bound() - memory needed to initialize a zstd_dstream
+ * @max_window_size: The maximum window size allowed for compressed frames.
*
- * Return: A lower bound on the size of the workspace that is passed to
- * ZSTD_initDStream() and ZSTD_initDStream_usingDDict().
+ * Return: A lower bound on the size of the workspace that is passed
+ * to zstd_init_dstream().
*/
-size_t ZSTD_DStreamWorkspaceBound(size_t maxWindowSize);
+size_t zstd_dstream_workspace_bound(size_t max_window_size);
/**
- * struct ZSTD_DStream - the zstd streaming decompression context
- */
-typedef struct ZSTD_DStream_s ZSTD_DStream;
-/*===== ZSTD_DStream management functions =====*/
-/**
- * ZSTD_initDStream() - initialize a zstd streaming decompression context
- * @maxWindowSize: The maximum window size allowed for compressed frames.
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace.
- * Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
- * how large the workspace must be.
- *
- * Return: The zstd streaming decompression context.
- */
-ZSTD_DStream *ZSTD_initDStream(size_t maxWindowSize, void *workspace,
- size_t workspaceSize);
-/**
- * ZSTD_initDStream_usingDDict() - initialize streaming decompression context
- * @maxWindowSize: The maximum window size allowed for compressed frames.
- * @ddict: The digested dictionary to use for decompression.
- * @workspace: The workspace to emplace the context into. It must outlive
- * the returned context.
- * @workspaceSize: The size of workspace.
- * Use ZSTD_DStreamWorkspaceBound(maxWindowSize) to determine
- * how large the workspace must be.
+ * zstd_init_dstream() - initialize a zstd streaming decompression context
+ * @max_window_size: The maximum window size allowed for compressed frames.
+ * @workspace: The workspace to emplace the context into. It must outlive
+ * the returned context.
+ * @workspaceSize: The size of workspace.
+ * Use zstd_dstream_workspace_bound(max_window_size) to
+ * determine how large the workspace must be.
*
- * Return: The zstd streaming decompression context.
+ * Return: The zstd streaming decompression context.
*/
-ZSTD_DStream *ZSTD_initDStream_usingDDict(size_t maxWindowSize,
- const ZSTD_DDict *ddict, void *workspace, size_t workspaceSize);
+zstd_dstream *zstd_init_dstream(size_t max_window_size, void *workspace,
+ size_t workspace_size);
-/*===== Streaming decompression functions =====*/
/**
- * ZSTD_resetDStream() - reset the context using parameters from creation
- * @zds: The zstd streaming decompression context to reset.
+ * zstd_reset_dstream() - reset the context using parameters from creation
+ * @dstream: The zstd streaming decompression context to reset.
*
* Resets the context using the parameters from creation. Skips dictionary
* loading, since it can be reused.
*
- * Return: Zero or an error, which can be checked using ZSTD_isError().
+ * Return: Zero or an error, which can be checked using zstd_is_error().
*/
-size_t ZSTD_resetDStream(ZSTD_DStream *zds);
+size_t zstd_reset_dstream(zstd_dstream *dstream);
+
/**
- * ZSTD_decompressStream() - streaming decompress some of input into output
- * @zds: The zstd streaming decompression context.
- * @output: Destination buffer. `output.pos` is updated to indicate how much
- * decompressed data was written.
- * @input: Source buffer. `input.pos` is updated to indicate how much data was
- * read. Note that it may not consume the entire input, in which case
- * `input.pos < input.size`, and it's up to the caller to present
- * remaining data again.
+ * zstd_decompress_stream() - streaming decompress some of input into output
+ * @dstream: The zstd streaming decompression context.
+ * @output: Destination buffer. `output.pos` is updated to indicate how much
+ * decompressed data was written.
+ * @input: Source buffer. `input.pos` is updated to indicate how much data was
+ * read. Note that it may not consume the entire input, in which case
+ * `input.pos < input.size`, and it's up to the caller to present
+ * remaining data again.
*
* The `input` and `output` buffers may be any size. Guaranteed to make some
* forward progress if `input` and `output` are not empty.
- * ZSTD_decompressStream() will not consume the last byte of the frame until
+ * zstd_decompress_stream() will not consume the last byte of the frame until
* the entire frame is flushed.
*
- * Return: Returns 0 iff a frame is completely decoded and fully flushed.
- * Otherwise returns a hint for the number of bytes to use as the input
- * for the next function call or an error, which can be checked using
- * ZSTD_isError(). The size hint will never load more than the frame.
+ * Return: Returns 0 iff a frame is completely decoded and fully flushed.
+ * Otherwise returns a hint for the number of bytes to use as the
+ * input for the next function call or an error, which can be checked
+ * using zstd_is_error(). The size hint will never load more than the
+ * frame.
*/
-size_t ZSTD_decompressStream(ZSTD_DStream *zds, ZSTD_outBuffer *output,
- ZSTD_inBuffer *input);
+size_t zstd_decompress_stream(zstd_dstream *dstream, zstd_out_buffer *output,
+ zstd_in_buffer *input);
-/**
- * ZSTD_DStreamInSize() - recommended size for the input buffer
- *
- * Return: The recommended size for the input buffer.
- */
-size_t ZSTD_DStreamInSize(void);
-/**
- * ZSTD_DStreamOutSize() - recommended size for the output buffer
- *
- * When the output buffer is at least this large, it is guaranteed to be large
- * enough to flush at least one complete decompressed block.
- *
- * Return: The recommended size for the output buffer.
- */
-size_t ZSTD_DStreamOutSize(void);
-
-
-/* --- Constants ---*/
-#define ZSTD_MAGICNUMBER 0xFD2FB528 /* >= v0.8.0 */
-#define ZSTD_MAGIC_SKIPPABLE_START 0x184D2A50U
-
-#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
-#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
-
-#define ZSTD_WINDOWLOG_MAX_32 27
-#define ZSTD_WINDOWLOG_MAX_64 27
-#define ZSTD_WINDOWLOG_MAX \
- ((unsigned int)(sizeof(size_t) == 4 \
- ? ZSTD_WINDOWLOG_MAX_32 \
- : ZSTD_WINDOWLOG_MAX_64))
-#define ZSTD_WINDOWLOG_MIN 10
-#define ZSTD_HASHLOG_MAX ZSTD_WINDOWLOG_MAX
-#define ZSTD_HASHLOG_MIN 6
-#define ZSTD_CHAINLOG_MAX (ZSTD_WINDOWLOG_MAX+1)
-#define ZSTD_CHAINLOG_MIN ZSTD_HASHLOG_MIN
-#define ZSTD_HASHLOG3_MAX 17
-#define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
-#define ZSTD_SEARCHLOG_MIN 1
-/* only for ZSTD_fast, other strategies are limited to 6 */
-#define ZSTD_SEARCHLENGTH_MAX 7
-/* only for ZSTD_btopt, other strategies are limited to 4 */
-#define ZSTD_SEARCHLENGTH_MIN 3
-#define ZSTD_TARGETLENGTH_MIN 4
-#define ZSTD_TARGETLENGTH_MAX 999
-
-/* for static allocation */
-#define ZSTD_FRAMEHEADERSIZE_MAX 18
-#define ZSTD_FRAMEHEADERSIZE_MIN 6
-#define ZSTD_frameHeaderSize_prefix 5
-#define ZSTD_frameHeaderSize_min ZSTD_FRAMEHEADERSIZE_MIN
-#define ZSTD_frameHeaderSize_max ZSTD_FRAMEHEADERSIZE_MAX
-/* magic number + skippable frame length */
-#define ZSTD_skippableHeaderSize 8
-
-
-/*-*************************************
- * Compressed size functions
- **************************************/
-
-/**
- * ZSTD_findFrameCompressedSize() - returns the size of a compressed frame
- * @src: Source buffer. It should point to the start of a zstd encoded frame
- * or a skippable frame.
- * @srcSize: The size of the source buffer. It must be at least as large as the
- * size of the frame.
- *
- * Return: The compressed size of the frame pointed to by `src` or an error,
- * which can be check with ZSTD_isError().
- * Suitable to pass to ZSTD_decompress() or similar functions.
- */
-size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize);
-
-/*-*************************************
- * Decompressed size functions
- **************************************/
-/**
- * ZSTD_getFrameContentSize() - returns the content size in a zstd frame header
- * @src: It should point to the start of a zstd encoded frame.
- * @srcSize: The size of the source buffer. It must be at least as large as the
- * frame header. `ZSTD_frameHeaderSize_max` is always large enough.
- *
- * Return: The frame content size stored in the frame header if known.
- * `ZSTD_CONTENTSIZE_UNKNOWN` if the content size isn't stored in the
- * frame header. `ZSTD_CONTENTSIZE_ERROR` on invalid input.
- */
-unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
+/* ====== Frame Inspection Functions ====== */
/**
- * ZSTD_findDecompressedSize() - returns decompressed size of a series of frames
- * @src: It should point to the start of a series of zstd encoded and/or
- * skippable frames.
- * @srcSize: The exact size of the series of frames.
+ * zstd_find_frame_compressed_size() - returns the size of a compressed frame
+ * @src: Source buffer. It should point to the start of a zstd encoded
+ * frame or a skippable frame.
+ * @src_size: The size of the source buffer. It must be at least as large as the
+ * size of the frame.
*
- * If any zstd encoded frame in the series doesn't have the frame content size
- * set, `ZSTD_CONTENTSIZE_UNKNOWN` is returned. But frame content size is always
- * set when using ZSTD_compress(). The decompressed size can be very large.
- * If the source is untrusted, the decompressed size could be wrong or
- * intentionally modified. Always ensure the result fits within the
- * application's authorized limits. ZSTD_findDecompressedSize() handles multiple
- * frames, and so it must traverse the input to read each frame header. This is
- * efficient as most of the data is skipped, however it does mean that all frame
- * data must be present and valid.
- *
- * Return: Decompressed size of all the data contained in the frames if known.
- * `ZSTD_CONTENTSIZE_UNKNOWN` if the decompressed size is unknown.
- * `ZSTD_CONTENTSIZE_ERROR` if an error occurred.
- */
-unsigned long long ZSTD_findDecompressedSize(const void *src, size_t srcSize);
-
-/*-*************************************
- * Advanced compression functions
- **************************************/
-/**
- * ZSTD_checkCParams() - ensure parameter values remain within authorized range
- * @cParams: The zstd compression parameters.
- *
- * Return: Zero or an error, which can be checked using ZSTD_isError().
+ * Return: The compressed size of the frame pointed to by `src` or an error,
+ * which can be check with zstd_is_error().
+ * Suitable to pass to ZSTD_decompress() or similar functions.
*/
-size_t ZSTD_checkCParams(ZSTD_compressionParameters cParams);
+size_t zstd_find_frame_compressed_size(const void *src, size_t src_size);
/**
- * ZSTD_adjustCParams() - optimize parameters for a given srcSize and dictSize
- * @srcSize: Optionally the estimated source size, or zero if unknown.
- * @dictSize: Optionally the estimated dictionary size, or zero if unknown.
- *
- * Return: The optimized parameters.
- */
-ZSTD_compressionParameters ZSTD_adjustCParams(
- ZSTD_compressionParameters cParams, unsigned long long srcSize,
- size_t dictSize);
-
-/*--- Advanced decompression functions ---*/
-
-/**
- * ZSTD_isFrame() - returns true iff the buffer starts with a valid frame
- * @buffer: The source buffer to check.
- * @size: The size of the source buffer, must be at least 4 bytes.
- *
- * Return: True iff the buffer starts with a zstd or skippable frame identifier.
- */
-unsigned int ZSTD_isFrame(const void *buffer, size_t size);
-
-/**
- * ZSTD_getDictID_fromDict() - returns the dictionary id stored in a dictionary
- * @dict: The dictionary buffer.
- * @dictSize: The size of the dictionary buffer.
- *
- * Return: The dictionary id stored within the dictionary or 0 if the
- * dictionary is not a zstd dictionary. If it returns 0 the
- * dictionary can still be loaded as a content-only dictionary.
- */
-unsigned int ZSTD_getDictID_fromDict(const void *dict, size_t dictSize);
-
-/**
- * ZSTD_getDictID_fromDDict() - returns the dictionary id stored in a ZSTD_DDict
- * @ddict: The ddict to find the id of.
- *
- * Return: The dictionary id stored within `ddict` or 0 if the dictionary is not
- * a zstd dictionary. If it returns 0 `ddict` will be loaded as a
- * content-only dictionary.
- */
-unsigned int ZSTD_getDictID_fromDDict(const ZSTD_DDict *ddict);
-
-/**
- * ZSTD_getDictID_fromFrame() - returns the dictionary id stored in a zstd frame
- * @src: Source buffer. It must be a zstd encoded frame.
- * @srcSize: The size of the source buffer. It must be at least as large as the
- * frame header. `ZSTD_frameHeaderSize_max` is always large enough.
- *
- * Return: The dictionary id required to decompress the frame stored within
- * `src` or 0 if the dictionary id could not be decoded. It can return
- * 0 if the frame does not require a dictionary, the dictionary id
- * wasn't stored in the frame, `src` is not a zstd frame, or `srcSize`
- * is too small.
- */
-unsigned int ZSTD_getDictID_fromFrame(const void *src, size_t srcSize);
-
-/**
- * struct ZSTD_frameParams - zstd frame parameters stored in the frame header
- * @frameContentSize: The frame content size, or 0 if not present.
+ * struct zstd_frame_params - zstd frame parameters stored in the frame header
+ * @frameContentSize: The frame content size, or ZSTD_CONTENTSIZE_UNKNOWN if not
+ * present.
* @windowSize: The window size, or 0 if the frame is a skippable frame.
+ * @blockSizeMax: The maximum block size.
+ * @frameType: The frame type (zstd or skippable)
+ * @headerSize: The size of the frame header.
* @dictID: The dictionary id, or 0 if not present.
* @checksumFlag: Whether a checksum was used.
+ *
+ * See zstd_lib.h.
*/
-typedef struct {
- unsigned long long frameContentSize;
- unsigned int windowSize;
- unsigned int dictID;
- unsigned int checksumFlag;
-} ZSTD_frameParams;
+typedef ZSTD_frameHeader zstd_frame_header;
/**
- * ZSTD_getFrameParams() - extracts parameters from a zstd or skippable frame
- * @fparamsPtr: On success the frame parameters are written here.
- * @src: The source buffer. It must point to a zstd or skippable frame.
- * @srcSize: The size of the source buffer. `ZSTD_frameHeaderSize_max` is
- * always large enough to succeed.
+ * zstd_get_frame_header() - extracts parameters from a zstd or skippable frame
+ * @params: On success the frame parameters are written here.
+ * @src: The source buffer. It must point to a zstd or skippable frame.
+ * @src_size: The size of the source buffer.
*
- * Return: 0 on success. If more data is required it returns how many bytes
- * must be provided to make forward progress. Otherwise it returns
- * an error, which can be checked using ZSTD_isError().
+ * Return: 0 on success. If more data is required it returns how many bytes
+ * must be provided to make forward progress. Otherwise it returns
+ * an error, which can be checked using zstd_is_error().
*/
-size_t ZSTD_getFrameParams(ZSTD_frameParams *fparamsPtr, const void *src,
- size_t srcSize);
-
-/*-*****************************************************************************
- * Buffer-less and synchronous inner streaming functions
- *
- * This is an advanced API, giving full control over buffer management, for
- * users which need direct control over memory.
- * But it's also a complex one, with many restrictions (documented below).
- * Prefer using normal streaming API for an easier experience
- ******************************************************************************/
-
-/*-*****************************************************************************
- * Buffer-less streaming compression (synchronous mode)
- *
- * A ZSTD_CCtx object is required to track streaming operations.
- * Use ZSTD_initCCtx() to initialize a context.
- * ZSTD_CCtx object can be re-used multiple times within successive compression
- * operations.
- *
- * Start by initializing a context.
- * Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary
- * compression,
- * or ZSTD_compressBegin_advanced(), for finer parameter control.
- * It's also possible to duplicate a reference context which has already been
- * initialized, using ZSTD_copyCCtx()
- *
- * Then, consume your input using ZSTD_compressContinue().
- * There are some important considerations to keep in mind when using this
- * advanced function :
- * - ZSTD_compressContinue() has no internal buffer. It uses externally provided
- * buffer only.
- * - Interface is synchronous : input is consumed entirely and produce 1+
- * (or more) compressed blocks.
- * - Caller must ensure there is enough space in `dst` to store compressed data
- * under worst case scenario. Worst case evaluation is provided by
- * ZSTD_compressBound().
- * ZSTD_compressContinue() doesn't guarantee recover after a failed
- * compression.
- * - ZSTD_compressContinue() presumes prior input ***is still accessible and
- * unmodified*** (up to maximum distance size, see WindowLog).
- * It remembers all previous contiguous blocks, plus one separated memory
- * segment (which can itself consists of multiple contiguous blocks)
- * - ZSTD_compressContinue() detects that prior input has been overwritten when
- * `src` buffer overlaps. In which case, it will "discard" the relevant memory
- * section from its history.
- *
- * Finish a frame with ZSTD_compressEnd(), which will write the last block(s)
- * and optional checksum. It's possible to use srcSize==0, in which case, it
- * will write a final empty block to end the frame. Without last block mark,
- * frames will be considered unfinished (corrupted) by decoders.
- *
- * `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress some new
- * frame.
- ******************************************************************************/
-
-/*===== Buffer-less streaming compression functions =====*/
-size_t ZSTD_compressBegin(ZSTD_CCtx *cctx, int compressionLevel);
-size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx *cctx, const void *dict,
- size_t dictSize, int compressionLevel);
-size_t ZSTD_compressBegin_advanced(ZSTD_CCtx *cctx, const void *dict,
- size_t dictSize, ZSTD_parameters params,
- unsigned long long pledgedSrcSize);
-size_t ZSTD_copyCCtx(ZSTD_CCtx *cctx, const ZSTD_CCtx *preparedCCtx,
- unsigned long long pledgedSrcSize);
-size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx *cctx, const ZSTD_CDict *cdict,
- unsigned long long pledgedSrcSize);
-size_t ZSTD_compressContinue(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-size_t ZSTD_compressEnd(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-
-
-
-/*-*****************************************************************************
- * Buffer-less streaming decompression (synchronous mode)
- *
- * A ZSTD_DCtx object is required to track streaming operations.
- * Use ZSTD_initDCtx() to initialize a context.
- * A ZSTD_DCtx object can be re-used multiple times.
- *
- * First typical operation is to retrieve frame parameters, using
- * ZSTD_getFrameParams(). It fills a ZSTD_frameParams structure which provide
- * important information to correctly decode the frame, such as the minimum
- * rolling buffer size to allocate to decompress data (`windowSize`), and the
- * dictionary ID used.
- * Note: content size is optional, it may not be present. 0 means unknown.
- * Note that these values could be wrong, either because of data malformation,
- * or because an attacker is spoofing deliberate false information. As a
- * consequence, check that values remain within valid application range,
- * especially `windowSize`, before allocation. Each application can set its own
- * limit, depending on local restrictions. For extended interoperability, it is
- * recommended to support at least 8 MB.
- * Frame parameters are extracted from the beginning of the compressed frame.
- * Data fragment must be large enough to ensure successful decoding, typically
- * `ZSTD_frameHeaderSize_max` bytes.
- * Result: 0: successful decoding, the `ZSTD_frameParams` structure is filled.
- * >0: `srcSize` is too small, provide at least this many bytes.
- * errorCode, which can be tested using ZSTD_isError().
- *
- * Start decompression, with ZSTD_decompressBegin() or
- * ZSTD_decompressBegin_usingDict(). Alternatively, you can copy a prepared
- * context, using ZSTD_copyDCtx().
- *
- * Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue()
- * alternatively.
- * ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize'
- * to ZSTD_decompressContinue().
- * ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will
- * fail.
- *
- * The result of ZSTD_decompressContinue() is the number of bytes regenerated
- * within 'dst' (necessarily <= dstCapacity). It can be zero, which is not an
- * error; it just means ZSTD_decompressContinue() has decoded some metadata
- * item. It can also be an error code, which can be tested with ZSTD_isError().
- *
- * ZSTD_decompressContinue() needs previous data blocks during decompression, up
- * to `windowSize`. They should preferably be located contiguously, prior to
- * current block. Alternatively, a round buffer of sufficient size is also
- * possible. Sufficient size is determined by frame parameters.
- * ZSTD_decompressContinue() is very sensitive to contiguity, if 2 blocks don't
- * follow each other, make sure that either the compressor breaks contiguity at
- * the same place, or that previous contiguous segment is large enough to
- * properly handle maximum back-reference.
- *
- * A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
- * Context can then be reset to start a new decompression.
- *
- * Note: it's possible to know if next input to present is a header or a block,
- * using ZSTD_nextInputType(). This information is not required to properly
- * decode a frame.
- *
- * == Special case: skippable frames ==
- *
- * Skippable frames allow integration of user-defined data into a flow of
- * concatenated frames. Skippable frames will be ignored (skipped) by a
- * decompressor. The format of skippable frames is as follows:
- * a) Skippable frame ID - 4 Bytes, Little endian format, any value from
- * 0x184D2A50 to 0x184D2A5F
- * b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
- * c) Frame Content - any content (User Data) of length equal to Frame Size
- * For skippable frames ZSTD_decompressContinue() always returns 0.
- * For skippable frames ZSTD_getFrameParams() returns fparamsPtr->windowLog==0
- * what means that a frame is skippable.
- * Note: If fparamsPtr->frameContentSize==0, it is ambiguous: the frame might
- * actually be a zstd encoded frame with no content. For purposes of
- * decompression, it is valid in both cases to skip the frame using
- * ZSTD_findFrameCompressedSize() to find its size in bytes.
- * It also returns frame size as fparamsPtr->frameContentSize.
- ******************************************************************************/
-
-/*===== Buffer-less streaming decompression functions =====*/
-size_t ZSTD_decompressBegin(ZSTD_DCtx *dctx);
-size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx *dctx, const void *dict,
- size_t dictSize);
-void ZSTD_copyDCtx(ZSTD_DCtx *dctx, const ZSTD_DCtx *preparedDCtx);
-size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx *dctx);
-size_t ZSTD_decompressContinue(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-typedef enum {
- ZSTDnit_frameHeader,
- ZSTDnit_blockHeader,
- ZSTDnit_block,
- ZSTDnit_lastBlock,
- ZSTDnit_checksum,
- ZSTDnit_skippableFrame
-} ZSTD_nextInputType_e;
-ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx *dctx);
-
-/*-*****************************************************************************
- * Block functions
- *
- * Block functions produce and decode raw zstd blocks, without frame metadata.
- * Frame metadata cost is typically ~18 bytes, which can be non-negligible for
- * very small blocks (< 100 bytes). User will have to take in charge required
- * information to regenerate data, such as compressed and content sizes.
- *
- * A few rules to respect:
- * - Compressing and decompressing require a context structure
- * + Use ZSTD_initCCtx() and ZSTD_initDCtx()
- * - It is necessary to init context before starting
- * + compression : ZSTD_compressBegin()
- * + decompression : ZSTD_decompressBegin()
- * + variants _usingDict() are also allowed
- * + copyCCtx() and copyDCtx() work too
- * - Block size is limited, it must be <= ZSTD_getBlockSizeMax()
- * + If you need to compress more, cut data into multiple blocks
- * + Consider using the regular ZSTD_compress() instead, as frame metadata
- * costs become negligible when source size is large.
- * - When a block is considered not compressible enough, ZSTD_compressBlock()
- * result will be zero. In which case, nothing is produced into `dst`.
- * + User must test for such outcome and deal directly with uncompressed data
- * + ZSTD_decompressBlock() doesn't accept uncompressed data as input!!!
- * + In case of multiple successive blocks, decoder must be informed of
- * uncompressed block existence to follow proper history. Use
- * ZSTD_insertBlock() in such a case.
- ******************************************************************************/
-
-/* Define for static allocation */
-#define ZSTD_BLOCKSIZE_ABSOLUTEMAX (128 * 1024)
-/*===== Raw zstd block functions =====*/
-size_t ZSTD_getBlockSizeMax(ZSTD_CCtx *cctx);
-size_t ZSTD_compressBlock(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-size_t ZSTD_decompressBlock(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity,
- const void *src, size_t srcSize);
-size_t ZSTD_insertBlock(ZSTD_DCtx *dctx, const void *blockStart,
- size_t blockSize);
+size_t zstd_get_frame_header(zstd_frame_header *params, const void *src,
+ size_t src_size);
-#endif /* ZSTD_H */
+#endif /* LINUX_ZSTD_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_ERRORS_H_398273423
+#define ZSTD_ERRORS_H_398273423
+
+
+/*===== dependency =====*/
+#include <linux/types.h> /* size_t */
+
+
+/* ===== ZSTDERRORLIB_API : control library symbols visibility ===== */
+#define ZSTDERRORLIB_VISIBILITY
+#define ZSTDERRORLIB_API ZSTDERRORLIB_VISIBILITY
+
+/*-*********************************************
+ * Error codes list
+ *-*********************************************
+ * Error codes _values_ are pinned down since v1.3.1 only.
+ * Therefore, don't rely on values if you may link to any version < v1.3.1.
+ *
+ * Only values < 100 are considered stable.
+ *
+ * note 1 : this API shall be used with static linking only.
+ * dynamic linking is not yet officially supported.
+ * note 2 : Prefer relying on the enum than on its value whenever possible
+ * This is the only supported way to use the error list < v1.3.1
+ * note 3 : ZSTD_isError() is always correct, whatever the library version.
+ **********************************************/
+typedef enum {
+ ZSTD_error_no_error = 0,
+ ZSTD_error_GENERIC = 1,
+ ZSTD_error_prefix_unknown = 10,
+ ZSTD_error_version_unsupported = 12,
+ ZSTD_error_frameParameter_unsupported = 14,
+ ZSTD_error_frameParameter_windowTooLarge = 16,
+ ZSTD_error_corruption_detected = 20,
+ ZSTD_error_checksum_wrong = 22,
+ ZSTD_error_dictionary_corrupted = 30,
+ ZSTD_error_dictionary_wrong = 32,
+ ZSTD_error_dictionaryCreation_failed = 34,
+ ZSTD_error_parameter_unsupported = 40,
+ ZSTD_error_parameter_outOfBound = 42,
+ ZSTD_error_tableLog_tooLarge = 44,
+ ZSTD_error_maxSymbolValue_tooLarge = 46,
+ ZSTD_error_maxSymbolValue_tooSmall = 48,
+ ZSTD_error_stage_wrong = 60,
+ ZSTD_error_init_missing = 62,
+ ZSTD_error_memory_allocation = 64,
+ ZSTD_error_workSpace_tooSmall= 66,
+ ZSTD_error_dstSize_tooSmall = 70,
+ ZSTD_error_srcSize_wrong = 72,
+ ZSTD_error_dstBuffer_null = 74,
+ /* following error codes are __NOT STABLE__, they can be removed or changed in future versions */
+ ZSTD_error_frameIndex_tooLarge = 100,
+ ZSTD_error_seekableIO = 102,
+ ZSTD_error_dstBuffer_wrong = 104,
+ ZSTD_error_srcBuffer_wrong = 105,
+ ZSTD_error_maxCode = 120 /* never EVER use this value directly, it can change in future versions! Use ZSTD_isError() instead */
+} ZSTD_ErrorCode;
+
+/*! ZSTD_getErrorCode() :
+ convert a `size_t` function result into a `ZSTD_ErrorCode` enum type,
+ which can be used to compare with enum list published above */
+ZSTDERRORLIB_API ZSTD_ErrorCode ZSTD_getErrorCode(size_t functionResult);
+ZSTDERRORLIB_API const char* ZSTD_getErrorString(ZSTD_ErrorCode code); /*< Same as ZSTD_getErrorName, but using a `ZSTD_ErrorCode` enum argument */
+
+
+
+#endif /* ZSTD_ERRORS_H_398273423 */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_H_235446
+#define ZSTD_H_235446
+
+/* ====== Dependency ======*/
+#include <linux/limits.h> /* INT_MAX */
+#include <linux/types.h> /* size_t */
+
+
+/* ===== ZSTDLIB_API : control library symbols visibility ===== */
+#define ZSTDLIB_VISIBILITY
+#define ZSTDLIB_API ZSTDLIB_VISIBILITY
+
+
+/* *****************************************************************************
+ Introduction
+
+ zstd, short for Zstandard, is a fast lossless compression algorithm, targeting
+ real-time compression scenarios at zlib-level and better compression ratios.
+ The zstd compression library provides in-memory compression and decompression
+ functions.
+
+ The library supports regular compression levels from 1 up to ZSTD_maxCLevel(),
+ which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
+ caution, as they require more memory. The library also offers negative
+ compression levels, which extend the range of speed vs. ratio preferences.
+ The lower the level, the faster the speed (at the cost of compression).
+
+ Compression can be done in:
+ - a single step (described as Simple API)
+ - a single step, reusing a context (described as Explicit context)
+ - unbounded multiple steps (described as Streaming compression)
+
+ The compression ratio achievable on small data can be highly improved using
+ a dictionary. Dictionary compression can be performed in:
+ - a single step (described as Simple dictionary API)
+ - a single step, reusing a dictionary (described as Bulk-processing
+ dictionary API)
+
+ Advanced experimental functions can be accessed using
+ `#define ZSTD_STATIC_LINKING_ONLY` before including zstd.h.
+
+ Advanced experimental APIs should never be used with a dynamically-linked
+ library. They are not "stable"; their definitions or signatures may change in
+ the future. Only static linking is allowed.
+*******************************************************************************/
+
+/*------ Version ------*/
+#define ZSTD_VERSION_MAJOR 1
+#define ZSTD_VERSION_MINOR 4
+#define ZSTD_VERSION_RELEASE 10
+#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
+
+/*! ZSTD_versionNumber() :
+ * Return runtime library version, the value is (MAJOR*100*100 + MINOR*100 + RELEASE). */
+ZSTDLIB_API unsigned ZSTD_versionNumber(void);
+
+#define ZSTD_LIB_VERSION ZSTD_VERSION_MAJOR.ZSTD_VERSION_MINOR.ZSTD_VERSION_RELEASE
+#define ZSTD_QUOTE(str) #str
+#define ZSTD_EXPAND_AND_QUOTE(str) ZSTD_QUOTE(str)
+#define ZSTD_VERSION_STRING ZSTD_EXPAND_AND_QUOTE(ZSTD_LIB_VERSION)
+
+/*! ZSTD_versionString() :
+ * Return runtime library version, like "1.4.5". Requires v1.3.0+. */
+ZSTDLIB_API const char* ZSTD_versionString(void);
+
+/* *************************************
+ * Default constant
+ ***************************************/
+#ifndef ZSTD_CLEVEL_DEFAULT
+# define ZSTD_CLEVEL_DEFAULT 3
+#endif
+
+/* *************************************
+ * Constants
+ ***************************************/
+
+/* All magic numbers are supposed read/written to/from files/memory using little-endian convention */
+#define ZSTD_MAGICNUMBER 0xFD2FB528 /* valid since v0.8.0 */
+#define ZSTD_MAGIC_DICTIONARY 0xEC30A437 /* valid since v0.7.0 */
+#define ZSTD_MAGIC_SKIPPABLE_START 0x184D2A50 /* all 16 values, from 0x184D2A50 to 0x184D2A5F, signal the beginning of a skippable frame */
+#define ZSTD_MAGIC_SKIPPABLE_MASK 0xFFFFFFF0
+
+#define ZSTD_BLOCKSIZELOG_MAX 17
+#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX)
+
+
+
+/* *************************************
+* Simple API
+***************************************/
+/*! ZSTD_compress() :
+ * Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
+ * Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
+ * @return : compressed size written into `dst` (<= `dstCapacity),
+ * or an error code if it fails (which can be tested using ZSTD_isError()). */
+ZSTDLIB_API size_t ZSTD_compress( void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ int compressionLevel);
+
+/*! ZSTD_decompress() :
+ * `compressedSize` : must be the _exact_ size of some number of compressed and/or skippable frames.
+ * `dstCapacity` is an upper bound of originalSize to regenerate.
+ * If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data.
+ * @return : the number of bytes decompressed into `dst` (<= `dstCapacity`),
+ * or an errorCode if it fails (which can be tested using ZSTD_isError()). */
+ZSTDLIB_API size_t ZSTD_decompress( void* dst, size_t dstCapacity,
+ const void* src, size_t compressedSize);
+
+/*! ZSTD_getFrameContentSize() : requires v1.3.0+
+ * `src` should point to the start of a ZSTD encoded frame.
+ * `srcSize` must be at least as large as the frame header.
+ * hint : any size >= `ZSTD_frameHeaderSize_max` is large enough.
+ * @return : - decompressed size of `src` frame content, if known
+ * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
+ * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small)
+ * note 1 : a 0 return value means the frame is valid but "empty".
+ * note 2 : decompressed size is an optional field, it may not be present, typically in streaming mode.
+ * When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
+ * In which case, it's necessary to use streaming mode to decompress data.
+ * Optionally, application can rely on some implicit limit,
+ * as ZSTD_decompress() only needs an upper bound of decompressed size.
+ * (For example, data could be necessarily cut into blocks <= 16 KB).
+ * note 3 : decompressed size is always present when compression is completed using single-pass functions,
+ * such as ZSTD_compress(), ZSTD_compressCCtx() ZSTD_compress_usingDict() or ZSTD_compress_usingCDict().
+ * note 4 : decompressed size can be very large (64-bits value),
+ * potentially larger than what local system can handle as a single memory segment.
+ * In which case, it's necessary to use streaming mode to decompress data.
+ * note 5 : If source is untrusted, decompressed size could be wrong or intentionally modified.
+ * Always ensure return value fits within application's authorized limits.
+ * Each application can set its own limits.
+ * note 6 : This function replaces ZSTD_getDecompressedSize() */
+#define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
+#define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
+ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
+
+/*! ZSTD_getDecompressedSize() :
+ * NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize().
+ * Both functions work the same way, but ZSTD_getDecompressedSize() blends
+ * "empty", "unknown" and "error" results to the same return value (0),
+ * while ZSTD_getFrameContentSize() gives them separate return values.
+ * @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
+ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
+
+/*! ZSTD_findFrameCompressedSize() :
+ * `src` should point to the start of a ZSTD frame or skippable frame.
+ * `srcSize` must be >= first frame size
+ * @return : the compressed size of the first frame starting at `src`,
+ * suitable to pass as `srcSize` to `ZSTD_decompress` or similar,
+ * or an error code if input is invalid */
+ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
+
+
+/*====== Helper functions ======*/
+#define ZSTD_COMPRESSBOUND(srcSize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /* margin, from 64 to 0 */ : 0)) /* this formula ensures that bound(A) + bound(B) <= bound(A+B) as long as A and B >= 128 KB */
+ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
+ZSTDLIB_API unsigned ZSTD_isError(size_t code); /*!< tells if a `size_t` function result is an error code */
+ZSTDLIB_API const char* ZSTD_getErrorName(size_t code); /*!< provides readable string from an error code */
+ZSTDLIB_API int ZSTD_minCLevel(void); /*!< minimum negative compression level allowed */
+ZSTDLIB_API int ZSTD_maxCLevel(void); /*!< maximum compression level available */
+
+
+/* *************************************
+* Explicit context
+***************************************/
+/*= Compression context
+ * When compressing many times,
+ * it is recommended to allocate a context just once,
+ * and re-use it for each successive compression operation.
+ * This will make workload friendlier for system's memory.
+ * Note : re-using context is just a speed / resource optimization.
+ * It doesn't change the compression ratio, which remains identical.
+ * Note 2 : In multi-threaded environments,
+ * use one different context per thread for parallel execution.
+ */
+typedef struct ZSTD_CCtx_s ZSTD_CCtx;
+ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx(void);
+ZSTDLIB_API size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx); /* accept NULL pointer */
+
+/*! ZSTD_compressCCtx() :
+ * Same as ZSTD_compress(), using an explicit ZSTD_CCtx.
+ * Important : in order to behave similarly to `ZSTD_compress()`,
+ * this function compresses at requested compression level,
+ * __ignoring any other parameter__ .
+ * If any advanced parameter was set using the advanced API,
+ * they will all be reset. Only `compressionLevel` remains.
+ */
+ZSTDLIB_API size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ int compressionLevel);
+
+/*= Decompression context
+ * When decompressing many times,
+ * it is recommended to allocate a context only once,
+ * and re-use it for each successive compression operation.
+ * This will make workload friendlier for system's memory.
+ * Use one context per thread for parallel execution. */
+typedef struct ZSTD_DCtx_s ZSTD_DCtx;
+ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx(void);
+ZSTDLIB_API size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx); /* accept NULL pointer */
+
+/*! ZSTD_decompressDCtx() :
+ * Same as ZSTD_decompress(),
+ * requires an allocated ZSTD_DCtx.
+ * Compatible with sticky parameters.
+ */
+ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize);
+
+
+/* *************************************
+* Advanced compression API
+***************************************/
+
+/* API design :
+ * Parameters are pushed one by one into an existing context,
+ * using ZSTD_CCtx_set*() functions.
+ * Pushed parameters are sticky : they are valid for next compressed frame, and any subsequent frame.
+ * "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` !
+ * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
+ *
+ * It's possible to reset all parameters to "default" using ZSTD_CCtx_reset().
+ *
+ * This API supercedes all other "advanced" API entry points in the experimental section.
+ * In the future, we expect to remove from experimental API entry points which are redundant with this API.
+ */
+
+
+/* Compression strategies, listed from fastest to strongest */
+typedef enum { ZSTD_fast=1,
+ ZSTD_dfast=2,
+ ZSTD_greedy=3,
+ ZSTD_lazy=4,
+ ZSTD_lazy2=5,
+ ZSTD_btlazy2=6,
+ ZSTD_btopt=7,
+ ZSTD_btultra=8,
+ ZSTD_btultra2=9
+ /* note : new strategies _might_ be added in the future.
+ Only the order (from fast to strong) is guaranteed */
+} ZSTD_strategy;
+
+
+typedef enum {
+
+ /* compression parameters
+ * Note: When compressing with a ZSTD_CDict these parameters are superseded
+ * by the parameters used to construct the ZSTD_CDict.
+ * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
+ ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
+ * Note that exact compression parameters are dynamically determined,
+ * depending on both compression level and srcSize (when known).
+ * Default level is ZSTD_CLEVEL_DEFAULT==3.
+ * Special: value 0 means default, which is controlled by ZSTD_CLEVEL_DEFAULT.
+ * Note 1 : it's possible to pass a negative compression level.
+ * Note 2 : setting a level does not automatically set all other compression parameters
+ * to default. Setting this will however eventually dynamically impact the compression
+ * parameters which have not been manually set. The manually set
+ * ones will 'stick'. */
+ /* Advanced compression parameters :
+ * It's possible to pin down compression parameters to some specific values.
+ * In which case, these values are no longer dynamically selected by the compressor */
+ ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
+ * This will set a memory budget for streaming decompression,
+ * with larger values requiring more memory
+ * and typically compressing more.
+ * Must be clamped between ZSTD_WINDOWLOG_MIN and ZSTD_WINDOWLOG_MAX.
+ * Special: value 0 means "use default windowLog".
+ * Note: Using a windowLog greater than ZSTD_WINDOWLOG_LIMIT_DEFAULT
+ * requires explicitly allowing such size at streaming decompression stage. */
+ ZSTD_c_hashLog=102, /* Size of the initial probe table, as a power of 2.
+ * Resulting memory usage is (1 << (hashLog+2)).
+ * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX.
+ * Larger tables improve compression ratio of strategies <= dFast,
+ * and improve speed of strategies > dFast.
+ * Special: value 0 means "use default hashLog". */
+ ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
+ * Resulting memory usage is (1 << (chainLog+2)).
+ * Must be clamped between ZSTD_CHAINLOG_MIN and ZSTD_CHAINLOG_MAX.
+ * Larger tables result in better and slower compression.
+ * This parameter is useless for "fast" strategy.
+ * It's still useful when using "dfast" strategy,
+ * in which case it defines a secondary probe table.
+ * Special: value 0 means "use default chainLog". */
+ ZSTD_c_searchLog=104, /* Number of search attempts, as a power of 2.
+ * More attempts result in better and slower compression.
+ * This parameter is useless for "fast" and "dFast" strategies.
+ * Special: value 0 means "use default searchLog". */
+ ZSTD_c_minMatch=105, /* Minimum size of searched matches.
+ * Note that Zstandard can still find matches of smaller size,
+ * it just tweaks its search algorithm to look for this size and larger.
+ * Larger values increase compression and decompression speed, but decrease ratio.
+ * Must be clamped between ZSTD_MINMATCH_MIN and ZSTD_MINMATCH_MAX.
+ * Note that currently, for all strategies < btopt, effective minimum is 4.
+ * , for all strategies > fast, effective maximum is 6.
+ * Special: value 0 means "use default minMatchLength". */
+ ZSTD_c_targetLength=106, /* Impact of this field depends on strategy.
+ * For strategies btopt, btultra & btultra2:
+ * Length of Match considered "good enough" to stop search.
+ * Larger values make compression stronger, and slower.
+ * For strategy fast:
+ * Distance between match sampling.
+ * Larger values make compression faster, and weaker.
+ * Special: value 0 means "use default targetLength". */
+ ZSTD_c_strategy=107, /* See ZSTD_strategy enum definition.
+ * The higher the value of selected strategy, the more complex it is,
+ * resulting in stronger and slower compression.
+ * Special: value 0 means "use default strategy". */
+
+ /* LDM mode parameters */
+ ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching.
+ * This parameter is designed to improve compression ratio
+ * for large inputs, by finding large matches at long distance.
+ * It increases memory usage and window size.
+ * Note: enabling this parameter increases default ZSTD_c_windowLog to 128 MB
+ * except when expressly set to a different value.
+ * Note: will be enabled by default if ZSTD_c_windowLog >= 128 MB and
+ * compression strategy >= ZSTD_btopt (== compression level 16+) */
+ ZSTD_c_ldmHashLog=161, /* Size of the table for long distance matching, as a power of 2.
+ * Larger values increase memory usage and compression ratio,
+ * but decrease compression speed.
+ * Must be clamped between ZSTD_HASHLOG_MIN and ZSTD_HASHLOG_MAX
+ * default: windowlog - 7.
+ * Special: value 0 means "automatically determine hashlog". */
+ ZSTD_c_ldmMinMatch=162, /* Minimum match size for long distance matcher.
+ * Larger/too small values usually decrease compression ratio.
+ * Must be clamped between ZSTD_LDM_MINMATCH_MIN and ZSTD_LDM_MINMATCH_MAX.
+ * Special: value 0 means "use default value" (default: 64). */
+ ZSTD_c_ldmBucketSizeLog=163, /* Log size of each bucket in the LDM hash table for collision resolution.
+ * Larger values improve collision resolution but decrease compression speed.
+ * The maximum value is ZSTD_LDM_BUCKETSIZELOG_MAX.
+ * Special: value 0 means "use default value" (default: 3). */
+ ZSTD_c_ldmHashRateLog=164, /* Frequency of inserting/looking up entries into the LDM hash table.
+ * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
+ * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
+ * Larger values improve compression speed.
+ * Deviating far from default value will likely result in a compression ratio decrease.
+ * Special: value 0 means "automatically determine hashRateLog". */
+
+ /* frame parameters */
+ ZSTD_c_contentSizeFlag=200, /* Content size will be written into frame header _whenever known_ (default:1)
+ * Content size must be known at the beginning of compression.
+ * This is automatically the case when using ZSTD_compress2(),
+ * For streaming scenarios, content size must be provided with ZSTD_CCtx_setPledgedSrcSize() */
+ ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
+ ZSTD_c_dictIDFlag=202, /* When applicable, dictionary's ID is written into frame header (default:1) */
+
+ /* multi-threading parameters */
+ /* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_MULTITHREAD).
+ * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
+ * In a situation where it's unknown if the linked library supports multi-threading or not,
+ * setting ZSTD_c_nbWorkers to any value >= 1 and consulting the return value provides a quick way to check this property.
+ */
+ ZSTD_c_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel.
+ * When nbWorkers >= 1, triggers asynchronous mode when invoking ZSTD_compressStream*() :
+ * ZSTD_compressStream*() consumes input and flush output if possible, but immediately gives back control to caller,
+ * while compression is performed in parallel, within worker thread(s).
+ * (note : a strong exception to this rule is when first invocation of ZSTD_compressStream2() sets ZSTD_e_end :
+ * in which case, ZSTD_compressStream2() delegates to ZSTD_compress2(), which is always a blocking call).
+ * More workers improve speed, but also increase memory usage.
+ * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
+ * compression is performed inside Caller's thread, and all invocations are blocking */
+ ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
+ * Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
+ * 0 means default, which is dynamically determined based on compression parameters.
+ * Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
+ * The minimum size is automatically and transparently enforced. */
+ ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size.
+ * The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
+ * It helps preserve compression ratio, while each job is compressed in parallel.
+ * This value is enforced only when nbWorkers >= 1.
+ * Larger values increase compression ratio, but decrease speed.
+ * Possible values range from 0 to 9 :
+ * - 0 means "default" : value will be determined by the library, depending on strategy
+ * - 1 means "no overlap"
+ * - 9 means "full overlap", using a full window size.
+ * Each intermediate rank increases/decreases load size by a factor 2 :
+ * 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; 0:default
+ * default value varies between 6 and 9, depending on strategy */
+
+ /* note : additional experimental parameters are also available
+ * within the experimental section of the API.
+ * At the time of this writing, they include :
+ * ZSTD_c_rsyncable
+ * ZSTD_c_format
+ * ZSTD_c_forceMaxWindow
+ * ZSTD_c_forceAttachDict
+ * ZSTD_c_literalCompressionMode
+ * ZSTD_c_targetCBlockSize
+ * ZSTD_c_srcSizeHint
+ * ZSTD_c_enableDedicatedDictSearch
+ * ZSTD_c_stableInBuffer
+ * ZSTD_c_stableOutBuffer
+ * ZSTD_c_blockDelimiters
+ * ZSTD_c_validateSequences
+ * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
+ * note : never ever use experimentalParam? names directly;
+ * also, the enums values themselves are unstable and can still change.
+ */
+ ZSTD_c_experimentalParam1=500,
+ ZSTD_c_experimentalParam2=10,
+ ZSTD_c_experimentalParam3=1000,
+ ZSTD_c_experimentalParam4=1001,
+ ZSTD_c_experimentalParam5=1002,
+ ZSTD_c_experimentalParam6=1003,
+ ZSTD_c_experimentalParam7=1004,
+ ZSTD_c_experimentalParam8=1005,
+ ZSTD_c_experimentalParam9=1006,
+ ZSTD_c_experimentalParam10=1007,
+ ZSTD_c_experimentalParam11=1008,
+ ZSTD_c_experimentalParam12=1009
+} ZSTD_cParameter;
+
+typedef struct {
+ size_t error;
+ int lowerBound;
+ int upperBound;
+} ZSTD_bounds;
+
+/*! ZSTD_cParam_getBounds() :
+ * All parameters must belong to an interval with lower and upper bounds,
+ * otherwise they will either trigger an error or be automatically clamped.
+ * @return : a structure, ZSTD_bounds, which contains
+ * - an error status field, which must be tested using ZSTD_isError()
+ * - lower and upper bounds, both inclusive
+ */
+ZSTDLIB_API ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter cParam);
+
+/*! ZSTD_CCtx_setParameter() :
+ * Set one compression parameter, selected by enum ZSTD_cParameter.
+ * All parameters have valid bounds. Bounds can be queried using ZSTD_cParam_getBounds().
+ * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
+ * Setting a parameter is generally only possible during frame initialization (before starting compression).
+ * Exception : when using multi-threading mode (nbWorkers >= 1),
+ * the following parameters can be updated _during_ compression (within same frame):
+ * => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy.
+ * new parameters will be active for next job only (after a flush()).
+ * @return : an error code (which can be tested using ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value);
+
+/*! ZSTD_CCtx_setPledgedSrcSize() :
+ * Total input data size to be compressed as a single frame.
+ * Value will be written in frame header, unless if explicitly forbidden using ZSTD_c_contentSizeFlag.
+ * This value will also be controlled at end of frame, and trigger an error if not respected.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Note 1 : pledgedSrcSize==0 actually means zero, aka an empty frame.
+ * In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN.
+ * ZSTD_CONTENTSIZE_UNKNOWN is default value for any new frame.
+ * Note 2 : pledgedSrcSize is only valid once, for the next frame.
+ * It's discarded at the end of the frame, and replaced by ZSTD_CONTENTSIZE_UNKNOWN.
+ * Note 3 : Whenever all input data is provided and consumed in a single round,
+ * for example with ZSTD_compress2(),
+ * or invoking immediately ZSTD_compressStream2(,,,ZSTD_e_end),
+ * this value is automatically overridden by srcSize instead.
+ */
+ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize);
+
+typedef enum {
+ ZSTD_reset_session_only = 1,
+ ZSTD_reset_parameters = 2,
+ ZSTD_reset_session_and_parameters = 3
+} ZSTD_ResetDirective;
+
+/*! ZSTD_CCtx_reset() :
+ * There are 2 different things that can be reset, independently or jointly :
+ * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
+ * Useful after an error, or to interrupt any ongoing compression.
+ * Any internal data not yet flushed is cancelled.
+ * Compression parameters and dictionary remain unchanged.
+ * They will be used to compress next frame.
+ * Resetting session never fails.
+ * - The parameters : changes all parameters back to "default".
+ * This removes any reference to any dictionary too.
+ * Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing)
+ * otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError())
+ * - Both : similar to resetting the session, followed by resetting parameters.
+ */
+ZSTDLIB_API size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset);
+
+/*! ZSTD_compress2() :
+ * Behave the same as ZSTD_compressCCtx(), but compression parameters are set using the advanced API.
+ * ZSTD_compress2() always starts a new frame.
+ * Should cctx hold data from a previously unfinished frame, everything about it is forgotten.
+ * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
+ * - The function is always blocking, returns when compression is completed.
+ * Hint : compression runs faster if `dstCapacity` >= `ZSTD_compressBound(srcSize)`.
+ * @return : compressed size written into `dst` (<= `dstCapacity),
+ * or an error code if it fails (which can be tested using ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize);
+
+
+/* *************************************
+* Advanced decompression API
+***************************************/
+
+/* The advanced API pushes parameters one by one into an existing DCtx context.
+ * Parameters are sticky, and remain valid for all following frames
+ * using the same DCtx context.
+ * It's possible to reset parameters to default values using ZSTD_DCtx_reset().
+ * Note : This API is compatible with existing ZSTD_decompressDCtx() and ZSTD_decompressStream().
+ * Therefore, no new decompression function is necessary.
+ */
+
+typedef enum {
+
+ ZSTD_d_windowLogMax=100, /* Select a size limit (in power of 2) beyond which
+ * the streaming API will refuse to allocate memory buffer
+ * in order to protect the host from unreasonable memory requirements.
+ * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
+ * By default, a decompression context accepts window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT).
+ * Special: value 0 means "use default maximum windowLog". */
+
+ /* note : additional experimental parameters are also available
+ * within the experimental section of the API.
+ * At the time of this writing, they include :
+ * ZSTD_d_format
+ * ZSTD_d_stableOutBuffer
+ * ZSTD_d_forceIgnoreChecksum
+ * ZSTD_d_refMultipleDDicts
+ * Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
+ * note : never ever use experimentalParam? names directly
+ */
+ ZSTD_d_experimentalParam1=1000,
+ ZSTD_d_experimentalParam2=1001,
+ ZSTD_d_experimentalParam3=1002,
+ ZSTD_d_experimentalParam4=1003
+
+} ZSTD_dParameter;
+
+/*! ZSTD_dParam_getBounds() :
+ * All parameters must belong to an interval with lower and upper bounds,
+ * otherwise they will either trigger an error or be automatically clamped.
+ * @return : a structure, ZSTD_bounds, which contains
+ * - an error status field, which must be tested using ZSTD_isError()
+ * - both lower and upper bounds, inclusive
+ */
+ZSTDLIB_API ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam);
+
+/*! ZSTD_DCtx_setParameter() :
+ * Set one compression parameter, selected by enum ZSTD_dParameter.
+ * All parameters have valid bounds. Bounds can be queried using ZSTD_dParam_getBounds().
+ * Providing a value beyond bound will either clamp it, or trigger an error (depending on parameter).
+ * Setting a parameter is only possible during frame initialization (before starting decompression).
+ * @return : 0, or an error code (which can be tested using ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int value);
+
+/*! ZSTD_DCtx_reset() :
+ * Return a DCtx to clean state.
+ * Session and parameters can be reset jointly or separately.
+ * Parameters can only be reset when no active frame is being decompressed.
+ * @return : 0, or an error code, which can be tested with ZSTD_isError()
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset);
+
+
+/* **************************
+* Streaming
+****************************/
+
+typedef struct ZSTD_inBuffer_s {
+ const void* src; /*< start of input buffer */
+ size_t size; /*< size of input buffer */
+ size_t pos; /*< position where reading stopped. Will be updated. Necessarily 0 <= pos <= size */
+} ZSTD_inBuffer;
+
+typedef struct ZSTD_outBuffer_s {
+ void* dst; /*< start of output buffer */
+ size_t size; /*< size of output buffer */
+ size_t pos; /*< position where writing stopped. Will be updated. Necessarily 0 <= pos <= size */
+} ZSTD_outBuffer;
+
+
+
+/*-***********************************************************************
+* Streaming compression - HowTo
+*
+* A ZSTD_CStream object is required to track streaming operation.
+* Use ZSTD_createCStream() and ZSTD_freeCStream() to create/release resources.
+* ZSTD_CStream objects can be reused multiple times on consecutive compression operations.
+* It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-using already allocated memory.
+*
+* For parallel execution, use one separate ZSTD_CStream per thread.
+*
+* note : since v1.3.0, ZSTD_CStream and ZSTD_CCtx are the same thing.
+*
+* Parameters are sticky : when starting a new compression on the same context,
+* it will re-use the same sticky parameters as previous compression session.
+* When in doubt, it's recommended to fully initialize the context before usage.
+* Use ZSTD_CCtx_reset() to reset the context and ZSTD_CCtx_setParameter(),
+* ZSTD_CCtx_setPledgedSrcSize(), or ZSTD_CCtx_loadDictionary() and friends to
+* set more specific parameters, the pledged source size, or load a dictionary.
+*
+* Use ZSTD_compressStream2() with ZSTD_e_continue as many times as necessary to
+* consume input stream. The function will automatically update both `pos`
+* fields within `input` and `output`.
+* Note that the function may not consume the entire input, for example, because
+* the output buffer is already full, in which case `input.pos < input.size`.
+* The caller must check if input has been entirely consumed.
+* If not, the caller must make some room to receive more compressed data,
+* and then present again remaining input data.
+* note: ZSTD_e_continue is guaranteed to make some forward progress when called,
+* but doesn't guarantee maximal forward progress. This is especially relevant
+* when compressing with multiple threads. The call won't block if it can
+* consume some input, but if it can't it will wait for some, but not all,
+* output to be flushed.
+* @return : provides a minimum amount of data remaining to be flushed from internal buffers
+* or an error code, which can be tested using ZSTD_isError().
+*
+* At any moment, it's possible to flush whatever data might remain stuck within internal buffer,
+* using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
+* Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be enough (return code > 0).
+* In which case, make some room to receive more compressed data, and call again ZSTD_compressStream2() with ZSTD_e_flush.
+* You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the
+* operation.
+* note: ZSTD_e_flush will flush as much output as possible, meaning when compressing with multiple threads, it will
+* block until the flush is complete or the output buffer is full.
+* @return : 0 if internal buffers are entirely flushed,
+* >0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
+* or an error code, which can be tested using ZSTD_isError().
+*
+* Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
+* It will perform a flush and write frame epilogue.
+* The epilogue is required for decoders to consider a frame completed.
+* flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush.
+* You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to
+* start a new frame.
+* note: ZSTD_e_end will flush as much output as possible, meaning when compressing with multiple threads, it will
+* block until the flush is complete or the output buffer is full.
+* @return : 0 if frame fully completed and fully flushed,
+* >0 if some data still present within internal buffer (the value is minimal estimation of remaining size),
+* or an error code, which can be tested using ZSTD_isError().
+*
+* *******************************************************************/
+
+typedef ZSTD_CCtx ZSTD_CStream; /*< CCtx and CStream are now effectively same object (>= v1.3.0) */
+ /* Continue to distinguish them for compatibility with older versions <= v1.2.0 */
+/*===== ZSTD_CStream management functions =====*/
+ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream(void);
+ZSTDLIB_API size_t ZSTD_freeCStream(ZSTD_CStream* zcs); /* accept NULL pointer */
+
+/*===== Streaming compression functions =====*/
+typedef enum {
+ ZSTD_e_continue=0, /* collect more data, encoder decides when to output compressed result, for optimal compression ratio */
+ ZSTD_e_flush=1, /* flush any data provided so far,
+ * it creates (at least) one new block, that can be decoded immediately on reception;
+ * frame will continue: any future data can still reference previously compressed data, improving compression.
+ * note : multithreaded compression will block to flush as much output as possible. */
+ ZSTD_e_end=2 /* flush any remaining data _and_ close current frame.
+ * note that frame is only closed after compressed data is fully flushed (return value == 0).
+ * After that point, any additional data starts a new frame.
+ * note : each frame is independent (does not reference any content from previous frame).
+ : note : multithreaded compression will block to flush as much output as possible. */
+} ZSTD_EndDirective;
+
+/*! ZSTD_compressStream2() :
+ * Behaves about the same as ZSTD_compressStream, with additional control on end directive.
+ * - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
+ * - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
+ * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
+ * - output->pos and input->pos will be updated. They are guaranteed to remain below their respective limit.
+ * - endOp must be a valid directive
+ * - When nbWorkers==0 (default), function is blocking : it completes its job before returning to caller.
+ * - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs to internal worker threads, flush to output whatever is available,
+ * and then immediately returns, just indicating that there is some data remaining to be flushed.
+ * The function nonetheless guarantees forward progress : it will return only after it reads or write at least 1+ byte.
+ * - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity, the function delegates to ZSTD_compress2() which is always blocking.
+ * - @return provides a minimum amount of data remaining to be flushed from internal buffers
+ * or an error code, which can be tested using ZSTD_isError().
+ * if @return != 0, flush is not fully completed, there is still some data left within internal buffers.
+ * This is useful for ZSTD_e_flush, since in this case more flushes are necessary to empty all buffers.
+ * For ZSTD_e_end, @return == 0 when internal buffers are fully flushed and frame is completed.
+ * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
+ * only ZSTD_e_end or ZSTD_e_flush operations are allowed.
+ * Before starting a new compression job, or changing compression parameters,
+ * it is required to fully flush internal buffers.
+ */
+ZSTDLIB_API size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
+ ZSTD_outBuffer* output,
+ ZSTD_inBuffer* input,
+ ZSTD_EndDirective endOp);
+
+
+/* These buffer sizes are softly recommended.
+ * They are not required : ZSTD_compressStream*() happily accepts any buffer size, for both input and output.
+ * Respecting the recommended size just makes it a bit easier for ZSTD_compressStream*(),
+ * reducing the amount of memory shuffling and buffering, resulting in minor performance savings.
+ *
+ * However, note that these recommendations are from the perspective of a C caller program.
+ * If the streaming interface is invoked from some other language,
+ * especially managed ones such as Java or Go, through a foreign function interface such as jni or cgo,
+ * a major performance rule is to reduce crossing such interface to an absolute minimum.
+ * It's not rare that performance ends being spent more into the interface, rather than compression itself.
+ * In which cases, prefer using large buffers, as large as practical,
+ * for both input and output, to reduce the nb of roundtrips.
+ */
+ZSTDLIB_API size_t ZSTD_CStreamInSize(void); /*< recommended size for input buffer */
+ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /*< recommended size for output buffer. Guarantee to successfully flush at least one complete compressed block. */
+
+
+/* *****************************************************************************
+ * This following is a legacy streaming API.
+ * It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
+ * It is redundant, but remains fully supported.
+ * Advanced parameters and dictionary compression can only be used through the
+ * new API.
+ ******************************************************************************/
+
+/*!
+ * Equivalent to:
+ *
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
+ * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
+ */
+ZSTDLIB_API size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel);
+/*!
+ * Alternative for ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue).
+ * NOTE: The return value is different. ZSTD_compressStream() returns a hint for
+ * the next read size (if non-zero and not an error). ZSTD_compressStream2()
+ * returns the minimum nb of bytes left to flush (if non-zero and not an error).
+ */
+ZSTDLIB_API size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
+/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_flush). */
+ZSTDLIB_API size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
+/*! Equivalent to ZSTD_compressStream2(zcs, output, &emptyInput, ZSTD_e_end). */
+ZSTDLIB_API size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output);
+
+
+/*-***************************************************************************
+* Streaming decompression - HowTo
+*
+* A ZSTD_DStream object is required to track streaming operations.
+* Use ZSTD_createDStream() and ZSTD_freeDStream() to create/release resources.
+* ZSTD_DStream objects can be re-used multiple times.
+*
+* Use ZSTD_initDStream() to start a new decompression operation.
+* @return : recommended first input size
+* Alternatively, use advanced API to set specific properties.
+*
+* Use ZSTD_decompressStream() repetitively to consume your input.
+* The function will update both `pos` fields.
+* If `input.pos < input.size`, some input has not been consumed.
+* It's up to the caller to present again remaining data.
+* The function tries to flush all data decoded immediately, respecting output buffer size.
+* If `output.pos < output.size`, decoder has flushed everything it could.
+* But if `output.pos == output.size`, there might be some data left within internal buffers.,
+* In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
+* Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
+* @return : 0 when a frame is completely decoded and fully flushed,
+* or an error code, which can be tested using ZSTD_isError(),
+* or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
+* the return value is a suggested next input size (just a hint for better latency)
+* that will never request more than the remaining frame size.
+* *******************************************************************************/
+
+typedef ZSTD_DCtx ZSTD_DStream; /*< DCtx and DStream are now effectively same object (>= v1.3.0) */
+ /* For compatibility with versions <= v1.2.0, prefer differentiating them. */
+/*===== ZSTD_DStream management functions =====*/
+ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream(void);
+ZSTDLIB_API size_t ZSTD_freeDStream(ZSTD_DStream* zds); /* accept NULL pointer */
+
+/*===== Streaming decompression functions =====*/
+
+/* This function is redundant with the advanced API and equivalent to:
+ *
+ * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
+ * ZSTD_DCtx_refDDict(zds, NULL);
+ */
+ZSTDLIB_API size_t ZSTD_initDStream(ZSTD_DStream* zds);
+
+ZSTDLIB_API size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
+
+ZSTDLIB_API size_t ZSTD_DStreamInSize(void); /*!< recommended size for input buffer */
+ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /*!< recommended size for output buffer. Guarantee to successfully flush at least one complete block in all circumstances. */
+
+
+/* ************************
+* Simple dictionary API
+***************************/
+/*! ZSTD_compress_usingDict() :
+ * Compression at an explicit compression level using a Dictionary.
+ * A dictionary can be any arbitrary data segment (also called a prefix),
+ * or a buffer with specified information (see dictBuilder/zdict.h).
+ * Note : This function loads the dictionary, resulting in significant startup delay.
+ * It's intended for a dictionary used only once.
+ * Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. */
+ZSTDLIB_API size_t ZSTD_compress_usingDict(ZSTD_CCtx* ctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize,
+ int compressionLevel);
+
+/*! ZSTD_decompress_usingDict() :
+ * Decompression using a known Dictionary.
+ * Dictionary must be identical to the one used during compression.
+ * Note : This function loads the dictionary, resulting in significant startup delay.
+ * It's intended for a dictionary used only once.
+ * Note : When `dict == NULL || dictSize < 8` no dictionary is used. */
+ZSTDLIB_API size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize);
+
+
+/* *********************************
+ * Bulk processing dictionary API
+ **********************************/
+typedef struct ZSTD_CDict_s ZSTD_CDict;
+
+/*! ZSTD_createCDict() :
+ * When compressing multiple messages or blocks using the same dictionary,
+ * it's recommended to digest the dictionary only once, since it's a costly operation.
+ * ZSTD_createCDict() will create a state from digesting a dictionary.
+ * The resulting state can be used for future compression operations with very limited startup cost.
+ * ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only.
+ * @dictBuffer can be released after ZSTD_CDict creation, because its content is copied within CDict.
+ * Note 1 : Consider experimental function `ZSTD_createCDict_byReference()` if you prefer to not duplicate @dictBuffer content.
+ * Note 2 : A ZSTD_CDict can be created from an empty @dictBuffer,
+ * in which case the only thing that it transports is the @compressionLevel.
+ * This can be useful in a pipeline featuring ZSTD_compress_usingCDict() exclusively,
+ * expecting a ZSTD_CDict parameter with any data, including those without a known dictionary. */
+ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict(const void* dictBuffer, size_t dictSize,
+ int compressionLevel);
+
+/*! ZSTD_freeCDict() :
+ * Function frees memory allocated by ZSTD_createCDict().
+ * If a NULL pointer is passed, no operation is performed. */
+ZSTDLIB_API size_t ZSTD_freeCDict(ZSTD_CDict* CDict);
+
+/*! ZSTD_compress_usingCDict() :
+ * Compression using a digested Dictionary.
+ * Recommended when same dictionary is used multiple times.
+ * Note : compression level is _decided at dictionary creation time_,
+ * and frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no) */
+ZSTDLIB_API size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_CDict* cdict);
+
+
+typedef struct ZSTD_DDict_s ZSTD_DDict;
+
+/*! ZSTD_createDDict() :
+ * Create a digested dictionary, ready to start decompression operation without startup delay.
+ * dictBuffer can be released after DDict creation, as its content is copied inside DDict. */
+ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict(const void* dictBuffer, size_t dictSize);
+
+/*! ZSTD_freeDDict() :
+ * Function frees memory allocated with ZSTD_createDDict()
+ * If a NULL pointer is passed, no operation is performed. */
+ZSTDLIB_API size_t ZSTD_freeDDict(ZSTD_DDict* ddict);
+
+/*! ZSTD_decompress_usingDDict() :
+ * Decompression using a digested Dictionary.
+ * Recommended when same dictionary is used multiple times. */
+ZSTDLIB_API size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_DDict* ddict);
+
+
+/* ******************************
+ * Dictionary helper functions
+ *******************************/
+
+/*! ZSTD_getDictID_fromDict() :
+ * Provides the dictID stored within dictionary.
+ * if @return == 0, the dictionary is not conformant with Zstandard specification.
+ * It can still be loaded, but as a content-only dictionary. */
+ZSTDLIB_API unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize);
+
+/*! ZSTD_getDictID_fromDDict() :
+ * Provides the dictID of the dictionary loaded into `ddict`.
+ * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
+ * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
+ZSTDLIB_API unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict);
+
+/*! ZSTD_getDictID_fromFrame() :
+ * Provides the dictID required to decompressed the frame stored within `src`.
+ * If @return == 0, the dictID could not be decoded.
+ * This could for one of the following reasons :
+ * - The frame does not require a dictionary to be decoded (most common case).
+ * - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
+ * Note : this use case also happens when using a non-conformant dictionary.
+ * - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
+ * - This is not a Zstandard frame.
+ * When identifying the exact failure cause, it's possible to use ZSTD_getFrameHeader(), which will provide a more precise error code. */
+ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
+
+
+/* *****************************************************************************
+ * Advanced dictionary and prefix API
+ *
+ * This API allows dictionaries to be used with ZSTD_compress2(),
+ * ZSTD_compressStream2(), and ZSTD_decompress(). Dictionaries are sticky, and
+ * only reset with the context is reset with ZSTD_reset_parameters or
+ * ZSTD_reset_session_and_parameters. Prefixes are single-use.
+ ******************************************************************************/
+
+
+/*! ZSTD_CCtx_loadDictionary() :
+ * Create an internal CDict from `dict` buffer.
+ * Decompression will have to use same dictionary.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
+ * meaning "return to no-dictionary mode".
+ * Note 1 : Dictionary is sticky, it will be used for all future compressed frames.
+ * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
+ * Note 2 : Loading a dictionary involves building tables.
+ * It's also a CPU consuming operation, with non-negligible impact on latency.
+ * Tables are dependent on compression parameters, and for this reason,
+ * compression parameters can no longer be changed after loading a dictionary.
+ * Note 3 :`dict` content will be copied internally.
+ * Use experimental ZSTD_CCtx_loadDictionary_byReference() to reference content instead.
+ * In such a case, dictionary buffer must outlive its users.
+ * Note 4 : Use ZSTD_CCtx_loadDictionary_advanced()
+ * to precisely select how dictionary content must be interpreted. */
+ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
+
+/*! ZSTD_CCtx_refCDict() :
+ * Reference a prepared dictionary, to be used for all next compressed frames.
+ * Note that compression parameters are enforced from within CDict,
+ * and supersede any compression parameter previously set within CCtx.
+ * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
+ * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
+ * The dictionary will remain valid for future compressed frames using same CCtx.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Special : Referencing a NULL CDict means "return to no-dictionary mode".
+ * Note 1 : Currently, only one dictionary can be managed.
+ * Referencing a new dictionary effectively "discards" any previous one.
+ * Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. */
+ZSTDLIB_API size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);
+
+/*! ZSTD_CCtx_refPrefix() :
+ * Reference a prefix (single-usage dictionary) for next compressed frame.
+ * A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
+ * Decompression will need same prefix to properly regenerate data.
+ * Compressing with a prefix is similar in outcome as performing a diff and compressing it,
+ * but performs much faster, especially during decompression (compression speed is tunable with compression level).
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary
+ * Note 1 : Prefix buffer is referenced. It **must** outlive compression.
+ * Its content must remain unmodified during compression.
+ * Note 2 : If the intention is to diff some large src data blob with some prior version of itself,
+ * ensure that the window size is large enough to contain the entire source.
+ * See ZSTD_c_windowLog.
+ * Note 3 : Referencing a prefix involves building tables, which are dependent on compression parameters.
+ * It's a CPU consuming operation, with non-negligible impact on latency.
+ * If there is a need to use the same prefix multiple times, consider loadDictionary instead.
+ * Note 4 : By default, the prefix is interpreted as raw content (ZSTD_dct_rawContent).
+ * Use experimental ZSTD_CCtx_refPrefix_advanced() to alter dictionary interpretation. */
+ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
+ const void* prefix, size_t prefixSize);
+
+/*! ZSTD_DCtx_loadDictionary() :
+ * Create an internal DDict from dict buffer,
+ * to be used to decompress next frames.
+ * The dictionary remains valid for all future frames, until explicitly invalidated.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
+ * meaning "return to no-dictionary mode".
+ * Note 1 : Loading a dictionary involves building tables,
+ * which has a non-negligible impact on CPU usage and latency.
+ * It's recommended to "load once, use many times", to amortize the cost
+ * Note 2 :`dict` content will be copied internally, so `dict` can be released after loading.
+ * Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead.
+ * Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to take control of
+ * how dictionary content is loaded and interpreted.
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
+
+/*! ZSTD_DCtx_refDDict() :
+ * Reference a prepared dictionary, to be used to decompress next frames.
+ * The dictionary remains active for decompression of future frames using same DCtx.
+ *
+ * If called with ZSTD_d_refMultipleDDicts enabled, repeated calls of this function
+ * will store the DDict references in a table, and the DDict used for decompression
+ * will be determined at decompression time, as per the dict ID in the frame.
+ * The memory for the table is allocated on the first call to refDDict, and can be
+ * freed with ZSTD_freeDCtx().
+ *
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Note 1 : Currently, only one dictionary can be managed.
+ * Referencing a new dictionary effectively "discards" any previous one.
+ * Special: referencing a NULL DDict means "return to no-dictionary mode".
+ * Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
+
+/*! ZSTD_DCtx_refPrefix() :
+ * Reference a prefix (single-usage dictionary) to decompress next frame.
+ * This is the reverse operation of ZSTD_CCtx_refPrefix(),
+ * and must use the same prefix as the one used during compression.
+ * Prefix is **only used once**. Reference is discarded at end of frame.
+ * End of frame is reached when ZSTD_decompressStream() returns 0.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ * Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary
+ * Note 2 : Prefix buffer is referenced. It **must** outlive decompression.
+ * Prefix buffer must remain unmodified up to the end of frame,
+ * reached when ZSTD_decompressStream() returns 0.
+ * Note 3 : By default, the prefix is treated as raw content (ZSTD_dct_rawContent).
+ * Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode (Experimental section)
+ * Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost.
+ * A full dictionary is more costly, as it requires building tables.
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx,
+ const void* prefix, size_t prefixSize);
+
+/* === Memory management === */
+
+/*! ZSTD_sizeof_*() :
+ * These functions give the _current_ memory usage of selected object.
+ * Note that object memory usage can evolve (increase or decrease) over time. */
+ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
+ZSTDLIB_API size_t ZSTD_sizeof_DCtx(const ZSTD_DCtx* dctx);
+ZSTDLIB_API size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs);
+ZSTDLIB_API size_t ZSTD_sizeof_DStream(const ZSTD_DStream* zds);
+ZSTDLIB_API size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict);
+ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
+
+#endif /* ZSTD_H_235446 */
+
+
+/* **************************************************************************************
+ * ADVANCED AND EXPERIMENTAL FUNCTIONS
+ ****************************************************************************************
+ * The definitions in the following section are considered experimental.
+ * They are provided for advanced scenarios.
+ * They should never be used with a dynamic library, as prototypes may change in the future.
+ * Use them only in association with static linking.
+ * ***************************************************************************************/
+
+#if !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY)
+#define ZSTD_H_ZSTD_STATIC_LINKING_ONLY
+
+/* **************************************************************************************
+ * experimental API (static linking only)
+ ****************************************************************************************
+ * The following symbols and constants
+ * are not planned to join "stable API" status in the near future.
+ * They can still change in future versions.
+ * Some of them are planned to remain in the static_only section indefinitely.
+ * Some of them might be removed in the future (especially when redundant with existing stable functions)
+ * ***************************************************************************************/
+
+#define ZSTD_FRAMEHEADERSIZE_PREFIX(format) ((format) == ZSTD_f_zstd1 ? 5 : 1) /* minimum input size required to query frame header size */
+#define ZSTD_FRAMEHEADERSIZE_MIN(format) ((format) == ZSTD_f_zstd1 ? 6 : 2)
+#define ZSTD_FRAMEHEADERSIZE_MAX 18 /* can be useful for static allocation */
+#define ZSTD_SKIPPABLEHEADERSIZE 8
+
+/* compression parameter bounds */
+#define ZSTD_WINDOWLOG_MAX_32 30
+#define ZSTD_WINDOWLOG_MAX_64 31
+#define ZSTD_WINDOWLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_WINDOWLOG_MAX_32 : ZSTD_WINDOWLOG_MAX_64))
+#define ZSTD_WINDOWLOG_MIN 10
+#define ZSTD_HASHLOG_MAX ((ZSTD_WINDOWLOG_MAX < 30) ? ZSTD_WINDOWLOG_MAX : 30)
+#define ZSTD_HASHLOG_MIN 6
+#define ZSTD_CHAINLOG_MAX_32 29
+#define ZSTD_CHAINLOG_MAX_64 30
+#define ZSTD_CHAINLOG_MAX ((int)(sizeof(size_t) == 4 ? ZSTD_CHAINLOG_MAX_32 : ZSTD_CHAINLOG_MAX_64))
+#define ZSTD_CHAINLOG_MIN ZSTD_HASHLOG_MIN
+#define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
+#define ZSTD_SEARCHLOG_MIN 1
+#define ZSTD_MINMATCH_MAX 7 /* only for ZSTD_fast, other strategies are limited to 6 */
+#define ZSTD_MINMATCH_MIN 3 /* only for ZSTD_btopt+, faster strategies are limited to 4 */
+#define ZSTD_TARGETLENGTH_MAX ZSTD_BLOCKSIZE_MAX
+#define ZSTD_TARGETLENGTH_MIN 0 /* note : comparing this constant to an unsigned results in a tautological test */
+#define ZSTD_STRATEGY_MIN ZSTD_fast
+#define ZSTD_STRATEGY_MAX ZSTD_btultra2
+
+
+#define ZSTD_OVERLAPLOG_MIN 0
+#define ZSTD_OVERLAPLOG_MAX 9
+
+#define ZSTD_WINDOWLOG_LIMIT_DEFAULT 27 /* by default, the streaming decoder will refuse any frame
+ * requiring larger than (1<<ZSTD_WINDOWLOG_LIMIT_DEFAULT) window size,
+ * to preserve host's memory from unreasonable requirements.
+ * This limit can be overridden using ZSTD_DCtx_setParameter(,ZSTD_d_windowLogMax,).
+ * The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional memory is allocated */
+
+
+/* LDM parameter bounds */
+#define ZSTD_LDM_HASHLOG_MIN ZSTD_HASHLOG_MIN
+#define ZSTD_LDM_HASHLOG_MAX ZSTD_HASHLOG_MAX
+#define ZSTD_LDM_MINMATCH_MIN 4
+#define ZSTD_LDM_MINMATCH_MAX 4096
+#define ZSTD_LDM_BUCKETSIZELOG_MIN 1
+#define ZSTD_LDM_BUCKETSIZELOG_MAX 8
+#define ZSTD_LDM_HASHRATELOG_MIN 0
+#define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
+
+/* Advanced parameter bounds */
+#define ZSTD_TARGETCBLOCKSIZE_MIN 64
+#define ZSTD_TARGETCBLOCKSIZE_MAX ZSTD_BLOCKSIZE_MAX
+#define ZSTD_SRCSIZEHINT_MIN 0
+#define ZSTD_SRCSIZEHINT_MAX INT_MAX
+
+/* internal */
+#define ZSTD_HASHLOG3_MAX 17
+
+
+/* --- Advanced types --- */
+
+typedef struct ZSTD_CCtx_params_s ZSTD_CCtx_params;
+
+typedef struct {
+ unsigned int offset; /* The offset of the match. (NOT the same as the offset code)
+ * If offset == 0 and matchLength == 0, this sequence represents the last
+ * literals in the block of litLength size.
+ */
+
+ unsigned int litLength; /* Literal length of the sequence. */
+ unsigned int matchLength; /* Match length of the sequence. */
+
+ /* Note: Users of this API may provide a sequence with matchLength == litLength == offset == 0.
+ * In this case, we will treat the sequence as a marker for a block boundary.
+ */
+
+ unsigned int rep; /* Represents which repeat offset is represented by the field 'offset'.
+ * Ranges from [0, 3].
+ *
+ * Repeat offsets are essentially previous offsets from previous sequences sorted in
+ * recency order. For more detail, see doc/zstd_compression_format.md
+ *
+ * If rep == 0, then 'offset' does not contain a repeat offset.
+ * If rep > 0:
+ * If litLength != 0:
+ * rep == 1 --> offset == repeat_offset_1
+ * rep == 2 --> offset == repeat_offset_2
+ * rep == 3 --> offset == repeat_offset_3
+ * If litLength == 0:
+ * rep == 1 --> offset == repeat_offset_2
+ * rep == 2 --> offset == repeat_offset_3
+ * rep == 3 --> offset == repeat_offset_1 - 1
+ *
+ * Note: This field is optional. ZSTD_generateSequences() will calculate the value of
+ * 'rep', but repeat offsets do not necessarily need to be calculated from an external
+ * sequence provider's perspective. For example, ZSTD_compressSequences() does not
+ * use this 'rep' field at all (as of now).
+ */
+} ZSTD_Sequence;
+
+typedef struct {
+ unsigned windowLog; /*< largest match distance : larger == more compression, more memory needed during decompression */
+ unsigned chainLog; /*< fully searched segment : larger == more compression, slower, more memory (useless for fast) */
+ unsigned hashLog; /*< dispatch table : larger == faster, more memory */
+ unsigned searchLog; /*< nb of searches : larger == more compression, slower */
+ unsigned minMatch; /*< match length searched : larger == faster decompression, sometimes less compression */
+ unsigned targetLength; /*< acceptable match size for optimal parser (only) : larger == more compression, slower */
+ ZSTD_strategy strategy; /*< see ZSTD_strategy definition above */
+} ZSTD_compressionParameters;
+
+typedef struct {
+ int contentSizeFlag; /*< 1: content size will be in frame header (when known) */
+ int checksumFlag; /*< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, for error detection */
+ int noDictIDFlag; /*< 1: no dictID will be saved into frame header (dictID is only useful for dictionary compression) */
+} ZSTD_frameParameters;
+
+typedef struct {
+ ZSTD_compressionParameters cParams;
+ ZSTD_frameParameters fParams;
+} ZSTD_parameters;
+
+typedef enum {
+ ZSTD_dct_auto = 0, /* dictionary is "full" when starting with ZSTD_MAGIC_DICTIONARY, otherwise it is "rawContent" */
+ ZSTD_dct_rawContent = 1, /* ensures dictionary is always loaded as rawContent, even if it starts with ZSTD_MAGIC_DICTIONARY */
+ ZSTD_dct_fullDict = 2 /* refuses to load a dictionary if it does not respect Zstandard's specification, starting with ZSTD_MAGIC_DICTIONARY */
+} ZSTD_dictContentType_e;
+
+typedef enum {
+ ZSTD_dlm_byCopy = 0, /*< Copy dictionary content internally */
+ ZSTD_dlm_byRef = 1 /*< Reference dictionary content -- the dictionary buffer must outlive its users. */
+} ZSTD_dictLoadMethod_e;
+
+typedef enum {
+ ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default) */
+ ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
+ * Useful to save 4 bytes per generated frame.
+ * Decoder cannot recognise automatically this format, requiring this instruction. */
+} ZSTD_format_e;
+
+typedef enum {
+ /* Note: this enum controls ZSTD_d_forceIgnoreChecksum */
+ ZSTD_d_validateChecksum = 0,
+ ZSTD_d_ignoreChecksum = 1
+} ZSTD_forceIgnoreChecksum_e;
+
+typedef enum {
+ /* Note: this enum controls ZSTD_d_refMultipleDDicts */
+ ZSTD_rmd_refSingleDDict = 0,
+ ZSTD_rmd_refMultipleDDicts = 1
+} ZSTD_refMultipleDDicts_e;
+
+typedef enum {
+ /* Note: this enum and the behavior it controls are effectively internal
+ * implementation details of the compressor. They are expected to continue
+ * to evolve and should be considered only in the context of extremely
+ * advanced performance tuning.
+ *
+ * Zstd currently supports the use of a CDict in three ways:
+ *
+ * - The contents of the CDict can be copied into the working context. This
+ * means that the compression can search both the dictionary and input
+ * while operating on a single set of internal tables. This makes
+ * the compression faster per-byte of input. However, the initial copy of
+ * the CDict's tables incurs a fixed cost at the beginning of the
+ * compression. For small compressions (< 8 KB), that copy can dominate
+ * the cost of the compression.
+ *
+ * - The CDict's tables can be used in-place. In this model, compression is
+ * slower per input byte, because the compressor has to search two sets of
+ * tables. However, this model incurs no start-up cost (as long as the
+ * working context's tables can be reused). For small inputs, this can be
+ * faster than copying the CDict's tables.
+ *
+ * - The CDict's tables are not used at all, and instead we use the working
+ * context alone to reload the dictionary and use params based on the source
+ * size. See ZSTD_compress_insertDictionary() and ZSTD_compress_usingDict().
+ * This method is effective when the dictionary sizes are very small relative
+ * to the input size, and the input size is fairly large to begin with.
+ *
+ * Zstd has a simple internal heuristic that selects which strategy to use
+ * at the beginning of a compression. However, if experimentation shows that
+ * Zstd is making poor choices, it is possible to override that choice with
+ * this enum.
+ */
+ ZSTD_dictDefaultAttach = 0, /* Use the default heuristic. */
+ ZSTD_dictForceAttach = 1, /* Never copy the dictionary. */
+ ZSTD_dictForceCopy = 2, /* Always copy the dictionary. */
+ ZSTD_dictForceLoad = 3 /* Always reload the dictionary */
+} ZSTD_dictAttachPref_e;
+
+typedef enum {
+ ZSTD_lcm_auto = 0, /*< Automatically determine the compression mode based on the compression level.
+ * Negative compression levels will be uncompressed, and positive compression
+ * levels will be compressed. */
+ ZSTD_lcm_huffman = 1, /*< Always attempt Huffman compression. Uncompressed literals will still be
+ * emitted if Huffman compression is not profitable. */
+ ZSTD_lcm_uncompressed = 2 /*< Always emit uncompressed literals. */
+} ZSTD_literalCompressionMode_e;
+
+
+/* *************************************
+* Frame size functions
+***************************************/
+
+/*! ZSTD_findDecompressedSize() :
+ * `src` should point to the start of a series of ZSTD encoded and/or skippable frames
+ * `srcSize` must be the _exact_ size of this series
+ * (i.e. there should be a frame boundary at `src + srcSize`)
+ * @return : - decompressed size of all data in all successive frames
+ * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
+ * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
+ *
+ * note 1 : decompressed size is an optional field, that may not be present, especially in streaming mode.
+ * When `return==ZSTD_CONTENTSIZE_UNKNOWN`, data to decompress could be any size.
+ * In which case, it's necessary to use streaming mode to decompress data.
+ * note 2 : decompressed size is always present when compression is done with ZSTD_compress()
+ * note 3 : decompressed size can be very large (64-bits value),
+ * potentially larger than what local system can handle as a single memory segment.
+ * In which case, it's necessary to use streaming mode to decompress data.
+ * note 4 : If source is untrusted, decompressed size could be wrong or intentionally modified.
+ * Always ensure result fits within application's authorized limits.
+ * Each application can set its own limits.
+ * note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to
+ * read each contained frame header. This is fast as most of the data is skipped,
+ * however it does mean that all frame data must be present and valid. */
+ZSTDLIB_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);
+
+/*! ZSTD_decompressBound() :
+ * `src` should point to the start of a series of ZSTD encoded and/or skippable frames
+ * `srcSize` must be the _exact_ size of this series
+ * (i.e. there should be a frame boundary at `src + srcSize`)
+ * @return : - upper-bound for the decompressed size of all data in all successive frames
+ * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
+ *
+ * note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
+ * note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD encoded frame of `src`.
+ * in this case, `ZSTD_findDecompressedSize` and `ZSTD_decompressBound` return the same value.
+ * note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calculated by:
+ * upper-bound = # blocks * min(128 KB, Window_Size)
+ */
+ZSTDLIB_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
+
+/*! ZSTD_frameHeaderSize() :
+ * srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX.
+ * @return : size of the Frame Header,
+ * or an error code (if srcSize is too small) */
+ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
+
+typedef enum {
+ ZSTD_sf_noBlockDelimiters = 0, /* Representation of ZSTD_Sequence has no block delimiters, sequences only */
+ ZSTD_sf_explicitBlockDelimiters = 1 /* Representation of ZSTD_Sequence contains explicit block delimiters */
+} ZSTD_sequenceFormat_e;
+
+/*! ZSTD_generateSequences() :
+ * Generate sequences using ZSTD_compress2, given a source buffer.
+ *
+ * Each block will end with a dummy sequence
+ * with offset == 0, matchLength == 0, and litLength == length of last literals.
+ * litLength may be == 0, and if so, then the sequence of (of: 0 ml: 0 ll: 0)
+ * simply acts as a block delimiter.
+ *
+ * zc can be used to insert custom compression params.
+ * This function invokes ZSTD_compress2
+ *
+ * The output of this function can be fed into ZSTD_compressSequences() with CCtx
+ * setting of ZSTD_c_blockDelimiters as ZSTD_sf_explicitBlockDelimiters
+ * @return : number of sequences generated
+ */
+
+ZSTDLIB_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
+ size_t outSeqsSize, const void* src, size_t srcSize);
+
+/*! ZSTD_mergeBlockDelimiters() :
+ * Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals
+ * by merging them into into the literals of the next sequence.
+ *
+ * As such, the final generated result has no explicit representation of block boundaries,
+ * and the final last literals segment is not represented in the sequences.
+ *
+ * The output of this function can be fed into ZSTD_compressSequences() with CCtx
+ * setting of ZSTD_c_blockDelimiters as ZSTD_sf_noBlockDelimiters
+ * @return : number of sequences left after merging
+ */
+ZSTDLIB_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize);
+
+/*! ZSTD_compressSequences() :
+ * Compress an array of ZSTD_Sequence, generated from the original source buffer, into dst.
+ * If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.)
+ * The entire source is compressed into a single frame.
+ *
+ * The compression behavior changes based on cctx params. In particular:
+ * If ZSTD_c_blockDelimiters == ZSTD_sf_noBlockDelimiters, the array of ZSTD_Sequence is expected to contain
+ * no block delimiters (defined in ZSTD_Sequence). Block boundaries are roughly determined based on
+ * the block size derived from the cctx, and sequences may be split. This is the default setting.
+ *
+ * If ZSTD_c_blockDelimiters == ZSTD_sf_explicitBlockDelimiters, the array of ZSTD_Sequence is expected to contain
+ * block delimiters (defined in ZSTD_Sequence). Behavior is undefined if no block delimiters are provided.
+ *
+ * If ZSTD_c_validateSequences == 0, this function will blindly accept the sequences provided. Invalid sequences cause undefined
+ * behavior. If ZSTD_c_validateSequences == 1, then if sequence is invalid (see doc/zstd_compression_format.md for
+ * specifics regarding offset/matchlength requirements) then the function will bail out and return an error.
+ *
+ * In addition to the two adjustable experimental params, there are other important cctx params.
+ * - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the match finder. It has a minimum value of ZSTD_MINMATCH_MIN.
+ * - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would in typical compression.
+ * - ZSTD_c_windowLog affects offset validation: this function will return an error at higher debug levels if a provided offset
+ * is larger than what the spec allows for a given window log and dictionary (if present). See: doc/zstd_compression_format.md
+ *
+ * Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep is unused.
+ * Note 2: Once we integrate ability to ingest repcodes, the explicit block delims mode must respect those repcodes exactly,
+ * and cannot emit an RLE block that disagrees with the repcode history
+ * @return : final compressed size or a ZSTD error.
+ */
+ZSTDLIB_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize,
+ const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
+ const void* src, size_t srcSize);
+
+
+/*! ZSTD_writeSkippableFrame() :
+ * Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
+ *
+ * Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic number,
+ * ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15.
+ * As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so
+ * the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant.
+ *
+ * Returns an error if destination buffer is not large enough, if the source size is not representable
+ * with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore invalid).
+ *
+ * @return : number of bytes written or a ZSTD error.
+ */
+ZSTDLIB_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, unsigned magicVariant);
+
+
+/* *************************************
+* Memory management
+***************************************/
+
+/*! ZSTD_estimate*() :
+ * These functions make it possible to estimate memory usage
+ * of a future {D,C}Ctx, before its creation.
+ *
+ * ZSTD_estimateCCtxSize() will provide a memory budget large enough
+ * for any compression level up to selected one.
+ * Note : Unlike ZSTD_estimateCStreamSize*(), this estimate
+ * does not include space for a window buffer.
+ * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
+ * The estimate will assume the input may be arbitrarily large,
+ * which is the worst case.
+ *
+ * When srcSize can be bound by a known and rather "small" value,
+ * this fact can be used to provide a tighter estimation
+ * because the CCtx compression context will need less memory.
+ * This tighter estimation can be provided by more advanced functions
+ * ZSTD_estimateCCtxSize_usingCParams(), which can be used in tandem with ZSTD_getCParams(),
+ * and ZSTD_estimateCCtxSize_usingCCtxParams(), which can be used in tandem with ZSTD_CCtxParams_setParameter().
+ * Both can be used to estimate memory using custom compression parameters and arbitrary srcSize limits.
+ *
+ * Note 2 : only single-threaded compression is supported.
+ * ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
+ */
+ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
+ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
+ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
+ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
+
+/*! ZSTD_estimateCStreamSize() :
+ * ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
+ * It will also consider src size to be arbitrarily "large", which is worst case.
+ * If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation.
+ * ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel.
+ * ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_c_nbWorkers is >= 1.
+ * Note : CStream size estimation is only correct for single-threaded compression.
+ * ZSTD_DStream memory budget depends on window Size.
+ * This information can be passed manually, using ZSTD_estimateDStreamSize,
+ * or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame();
+ * Note : if streaming is init with function ZSTD_init?Stream_usingDict(),
+ * an internal ?Dict will be created, which additional size is not estimated here.
+ * In this case, get total size by adding ZSTD_estimate?DictSize */
+ZSTDLIB_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
+ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
+ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
+ZSTDLIB_API size_t ZSTD_estimateDStreamSize(size_t windowSize);
+ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
+
+/*! ZSTD_estimate?DictSize() :
+ * ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict().
+ * ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced().
+ * Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller.
+ */
+ZSTDLIB_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
+ZSTDLIB_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
+ZSTDLIB_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);
+
+/*! ZSTD_initStatic*() :
+ * Initialize an object using a pre-allocated fixed-size buffer.
+ * workspace: The memory area to emplace the object into.
+ * Provided pointer *must be 8-bytes aligned*.
+ * Buffer must outlive object.
+ * workspaceSize: Use ZSTD_estimate*Size() to determine
+ * how large workspace must be to support target scenario.
+ * @return : pointer to object (same address as workspace, just different type),
+ * or NULL if error (size too small, incorrect alignment, etc.)
+ * Note : zstd will never resize nor malloc() when using a static buffer.
+ * If the object requires more memory than available,
+ * zstd will just error out (typically ZSTD_error_memory_allocation).
+ * Note 2 : there is no corresponding "free" function.
+ * Since workspace is allocated externally, it must be freed externally too.
+ * Note 3 : cParams : use ZSTD_getCParams() to convert a compression level
+ * into its associated cParams.
+ * Limitation 1 : currently not compatible with internal dictionary creation, triggered by
+ * ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict().
+ * Limitation 2 : static cctx currently not compatible with multi-threading.
+ * Limitation 3 : static dctx is incompatible with legacy support.
+ */
+ZSTDLIB_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize);
+ZSTDLIB_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticCCtx() */
+
+ZSTDLIB_API ZSTD_DCtx* ZSTD_initStaticDCtx(void* workspace, size_t workspaceSize);
+ZSTDLIB_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticDCtx() */
+
+ZSTDLIB_API const ZSTD_CDict* ZSTD_initStaticCDict(
+ void* workspace, size_t workspaceSize,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_compressionParameters cParams);
+
+ZSTDLIB_API const ZSTD_DDict* ZSTD_initStaticDDict(
+ void* workspace, size_t workspaceSize,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType);
+
+
+/*! Custom memory allocation :
+ * These prototypes make it possible to pass your own allocation/free functions.
+ * ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below.
+ * All allocation/free operations will be completed using these custom variants instead of regular <stdlib.h> ones.
+ */
+typedef void* (*ZSTD_allocFunction) (void* opaque, size_t size);
+typedef void (*ZSTD_freeFunction) (void* opaque, void* address);
+typedef struct { ZSTD_allocFunction customAlloc; ZSTD_freeFunction customFree; void* opaque; } ZSTD_customMem;
+static
+__attribute__((__unused__))
+ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; /*< this constant defers to stdlib's functions */
+
+ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
+ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
+ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem);
+ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
+
+ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_compressionParameters cParams,
+ ZSTD_customMem customMem);
+
+/* ! Thread pool :
+ * These prototypes make it possible to share a thread pool among multiple compression contexts.
+ * This can limit resources for applications with multiple threads where each one uses
+ * a threaded compression mode (via ZSTD_c_nbWorkers parameter).
+ * ZSTD_createThreadPool creates a new thread pool with a given number of threads.
+ * Note that the lifetime of such pool must exist while being used.
+ * ZSTD_CCtx_refThreadPool assigns a thread pool to a context (use NULL argument value
+ * to use an internal thread pool).
+ * ZSTD_freeThreadPool frees a thread pool, accepts NULL pointer.
+ */
+typedef struct POOL_ctx_s ZSTD_threadPool;
+ZSTDLIB_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads);
+ZSTDLIB_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); /* accept NULL pointer */
+ZSTDLIB_API size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool);
+
+
+/*
+ * This API is temporary and is expected to change or disappear in the future!
+ */
+ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced2(
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_customMem customMem);
+
+ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_advanced(
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_customMem customMem);
+
+
+/* *************************************
+* Advanced compression functions
+***************************************/
+
+/*! ZSTD_createCDict_byReference() :
+ * Create a digested dictionary for compression
+ * Dictionary content is just referenced, not duplicated.
+ * As a consequence, `dictBuffer` **must** outlive CDict,
+ * and its content must remain unmodified throughout the lifetime of CDict.
+ * note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */
+ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);
+
+/*! ZSTD_getDictID_fromCDict() :
+ * Provides the dictID of the dictionary loaded into `cdict`.
+ * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
+ * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
+ZSTDLIB_API unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict);
+
+/*! ZSTD_getCParams() :
+ * @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize.
+ * `estimatedSrcSize` value is optional, select 0 if not known */
+ZSTDLIB_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
+
+/*! ZSTD_getParams() :
+ * same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`.
+ * All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 */
+ZSTDLIB_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
+
+/*! ZSTD_checkCParams() :
+ * Ensure param values remain within authorized range.
+ * @return 0 on success, or an error code (can be checked with ZSTD_isError()) */
+ZSTDLIB_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
+
+/*! ZSTD_adjustCParams() :
+ * optimize params for a given `srcSize` and `dictSize`.
+ * `srcSize` can be unknown, in which case use ZSTD_CONTENTSIZE_UNKNOWN.
+ * `dictSize` must be `0` when there is no dictionary.
+ * cPar can be invalid : all parameters will be clamped within valid range in the @return struct.
+ * This function never fails (wide contract) */
+ZSTDLIB_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
+
+/*! ZSTD_compress_advanced() :
+ * Note : this function is now DEPRECATED.
+ * It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
+ * This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x */
+ZSTDLIB_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize,
+ ZSTD_parameters params);
+
+/*! ZSTD_compress_usingCDict_advanced() :
+ * Note : this function is now REDUNDANT.
+ * It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
+ * This prototype will be marked as deprecated and generate compilation warning in some future version */
+ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_CDict* cdict,
+ ZSTD_frameParameters fParams);
+
+
+/*! ZSTD_CCtx_loadDictionary_byReference() :
+ * Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx.
+ * It saves some memory, but also requires that `dict` outlives its usage within `cctx` */
+ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
+
+/*! ZSTD_CCtx_loadDictionary_advanced() :
+ * Same as ZSTD_CCtx_loadDictionary(), but gives finer control over
+ * how to load the dictionary (by copy ? by reference ?)
+ * and how to interpret it (automatic ? force raw mode ? full mode only ?) */
+ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
+
+/*! ZSTD_CCtx_refPrefix_advanced() :
+ * Same as ZSTD_CCtx_refPrefix(), but gives finer control over
+ * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
+ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
+
+/* === experimental parameters === */
+/* these parameters can be used with ZSTD_setParameter()
+ * they are not guaranteed to remain supported in the future */
+
+ /* Enables rsyncable mode,
+ * which makes compressed files more rsync friendly
+ * by adding periodic synchronization points to the compressed data.
+ * The target average block size is ZSTD_c_jobSize / 2.
+ * It's possible to modify the job size to increase or decrease
+ * the granularity of the synchronization point.
+ * Once the jobSize is smaller than the window size,
+ * it will result in compression ratio degradation.
+ * NOTE 1: rsyncable mode only works when multithreading is enabled.
+ * NOTE 2: rsyncable performs poorly in combination with long range mode,
+ * since it will decrease the effectiveness of synchronization points,
+ * though mileage may vary.
+ * NOTE 3: Rsyncable mode limits maximum compression speed to ~400 MB/s.
+ * If the selected compression level is already running significantly slower,
+ * the overall speed won't be significantly impacted.
+ */
+ #define ZSTD_c_rsyncable ZSTD_c_experimentalParam1
+
+/* Select a compression format.
+ * The value must be of type ZSTD_format_e.
+ * See ZSTD_format_e enum definition for details */
+#define ZSTD_c_format ZSTD_c_experimentalParam2
+
+/* Force back-reference distances to remain < windowSize,
+ * even when referencing into Dictionary content (default:0) */
+#define ZSTD_c_forceMaxWindow ZSTD_c_experimentalParam3
+
+/* Controls whether the contents of a CDict
+ * are used in place, or copied into the working context.
+ * Accepts values from the ZSTD_dictAttachPref_e enum.
+ * See the comments on that enum for an explanation of the feature. */
+#define ZSTD_c_forceAttachDict ZSTD_c_experimentalParam4
+
+/* Controls how the literals are compressed (default is auto).
+ * The value must be of type ZSTD_literalCompressionMode_e.
+ * See ZSTD_literalCompressionMode_t enum definition for details.
+ */
+#define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5
+
+/* Tries to fit compressed block size to be around targetCBlockSize.
+ * No target when targetCBlockSize == 0.
+ * There is no guarantee on compressed block size (default:0) */
+#define ZSTD_c_targetCBlockSize ZSTD_c_experimentalParam6
+
+/* User's best guess of source size.
+ * Hint is not valid when srcSizeHint == 0.
+ * There is no guarantee that hint is close to actual source size,
+ * but compression ratio may regress significantly if guess considerably underestimates */
+#define ZSTD_c_srcSizeHint ZSTD_c_experimentalParam7
+
+/* Controls whether the new and experimental "dedicated dictionary search
+ * structure" can be used. This feature is still rough around the edges, be
+ * prepared for surprising behavior!
+ *
+ * How to use it:
+ *
+ * When using a CDict, whether to use this feature or not is controlled at
+ * CDict creation, and it must be set in a CCtxParams set passed into that
+ * construction (via ZSTD_createCDict_advanced2()). A compression will then
+ * use the feature or not based on how the CDict was constructed; the value of
+ * this param, set in the CCtx, will have no effect.
+ *
+ * However, when a dictionary buffer is passed into a CCtx, such as via
+ * ZSTD_CCtx_loadDictionary(), this param can be set on the CCtx to control
+ * whether the CDict that is created internally can use the feature or not.
+ *
+ * What it does:
+ *
+ * Normally, the internal data structures of the CDict are analogous to what
+ * would be stored in a CCtx after compressing the contents of a dictionary.
+ * To an approximation, a compression using a dictionary can then use those
+ * data structures to simply continue what is effectively a streaming
+ * compression where the simulated compression of the dictionary left off.
+ * Which is to say, the search structures in the CDict are normally the same
+ * format as in the CCtx.
+ *
+ * It is possible to do better, since the CDict is not like a CCtx: the search
+ * structures are written once during CDict creation, and then are only read
+ * after that, while the search structures in the CCtx are both read and
+ * written as the compression goes along. This means we can choose a search
+ * structure for the dictionary that is read-optimized.
+ *
+ * This feature enables the use of that different structure.
+ *
+ * Note that some of the members of the ZSTD_compressionParameters struct have
+ * different semantics and constraints in the dedicated search structure. It is
+ * highly recommended that you simply set a compression level in the CCtxParams
+ * you pass into the CDict creation call, and avoid messing with the cParams
+ * directly.
+ *
+ * Effects:
+ *
+ * This will only have any effect when the selected ZSTD_strategy
+ * implementation supports this feature. Currently, that's limited to
+ * ZSTD_greedy, ZSTD_lazy, and ZSTD_lazy2.
+ *
+ * Note that this means that the CDict tables can no longer be copied into the
+ * CCtx, so the dict attachment mode ZSTD_dictForceCopy will no longer be
+ * useable. The dictionary can only be attached or reloaded.
+ *
+ * In general, you should expect compression to be faster--sometimes very much
+ * so--and CDict creation to be slightly slower. Eventually, we will probably
+ * make this mode the default.
+ */
+#define ZSTD_c_enableDedicatedDictSearch ZSTD_c_experimentalParam8
+
+/* ZSTD_c_stableInBuffer
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable.
+ *
+ * Tells the compressor that the ZSTD_inBuffer will ALWAYS be the same
+ * between calls, except for the modifications that zstd makes to pos (the
+ * caller must not modify pos). This is checked by the compressor, and
+ * compression will fail if it ever changes. This means the only flush
+ * mode that makes sense is ZSTD_e_end, so zstd will error if ZSTD_e_end
+ * is not used. The data in the ZSTD_inBuffer in the range [src, src + pos)
+ * MUST not be modified during compression or you will get data corruption.
+ *
+ * When this flag is enabled zstd won't allocate an input window buffer,
+ * because the user guarantees it can reference the ZSTD_inBuffer until
+ * the frame is complete. But, it will still allocate an output buffer
+ * large enough to fit a block (see ZSTD_c_stableOutBuffer). This will also
+ * avoid the memcpy() from the input buffer to the input window buffer.
+ *
+ * NOTE: ZSTD_compressStream2() will error if ZSTD_e_end is not used.
+ * That means this flag cannot be used with ZSTD_compressStream().
+ *
+ * NOTE: So long as the ZSTD_inBuffer always points to valid memory, using
+ * this flag is ALWAYS memory safe, and will never access out-of-bounds
+ * memory. However, compression WILL fail if you violate the preconditions.
+ *
+ * WARNING: The data in the ZSTD_inBuffer in the range [dst, dst + pos) MUST
+ * not be modified during compression or you will get data corruption. This
+ * is because zstd needs to reference data in the ZSTD_inBuffer to find
+ * matches. Normally zstd maintains its own window buffer for this purpose,
+ * but passing this flag tells zstd to use the user provided buffer.
+ */
+#define ZSTD_c_stableInBuffer ZSTD_c_experimentalParam9
+
+/* ZSTD_c_stableOutBuffer
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable.
+ *
+ * Tells he compressor that the ZSTD_outBuffer will not be resized between
+ * calls. Specifically: (out.size - out.pos) will never grow. This gives the
+ * compressor the freedom to say: If the compressed data doesn't fit in the
+ * output buffer then return ZSTD_error_dstSizeTooSmall. This allows us to
+ * always decompress directly into the output buffer, instead of decompressing
+ * into an internal buffer and copying to the output buffer.
+ *
+ * When this flag is enabled zstd won't allocate an output buffer, because
+ * it can write directly to the ZSTD_outBuffer. It will still allocate the
+ * input window buffer (see ZSTD_c_stableInBuffer).
+ *
+ * Zstd will check that (out.size - out.pos) never grows and return an error
+ * if it does. While not strictly necessary, this should prevent surprises.
+ */
+#define ZSTD_c_stableOutBuffer ZSTD_c_experimentalParam10
+
+/* ZSTD_c_blockDelimiters
+ * Default is 0 == ZSTD_sf_noBlockDelimiters.
+ *
+ * For use with sequence compression API: ZSTD_compressSequences().
+ *
+ * Designates whether or not the given array of ZSTD_Sequence contains block delimiters
+ * and last literals, which are defined as sequences with offset == 0 and matchLength == 0.
+ * See the definition of ZSTD_Sequence for more specifics.
+ */
+#define ZSTD_c_blockDelimiters ZSTD_c_experimentalParam11
+
+/* ZSTD_c_validateSequences
+ * Default is 0 == disabled. Set to 1 to enable sequence validation.
+ *
+ * For use with sequence compression API: ZSTD_compressSequences().
+ * Designates whether or not we validate sequences provided to ZSTD_compressSequences()
+ * during function execution.
+ *
+ * Without validation, providing a sequence that does not conform to the zstd spec will cause
+ * undefined behavior, and may produce a corrupted block.
+ *
+ * With validation enabled, a if sequence is invalid (see doc/zstd_compression_format.md for
+ * specifics regarding offset/matchlength requirements) then the function will bail out and
+ * return an error.
+ *
+ */
+#define ZSTD_c_validateSequences ZSTD_c_experimentalParam12
+
+/*! ZSTD_CCtx_getParameter() :
+ * Get the requested compression parameter value, selected by enum ZSTD_cParameter,
+ * and store it into int* value.
+ * @return : 0, or an error code (which can be tested with ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value);
+
+
+/*! ZSTD_CCtx_params :
+ * Quick howto :
+ * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
+ * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
+ * an existing ZSTD_CCtx_params structure.
+ * This is similar to
+ * ZSTD_CCtx_setParameter().
+ * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
+ * an existing CCtx.
+ * These parameters will be applied to
+ * all subsequent frames.
+ * - ZSTD_compressStream2() : Do compression using the CCtx.
+ * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
+ *
+ * This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
+ * for static allocation of CCtx for single-threaded compression.
+ */
+ZSTDLIB_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
+ZSTDLIB_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); /* accept NULL pointer */
+
+/*! ZSTD_CCtxParams_reset() :
+ * Reset params to default values.
+ */
+ZSTDLIB_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params);
+
+/*! ZSTD_CCtxParams_init() :
+ * Initializes the compression parameters of cctxParams according to
+ * compression level. All other parameters are reset to their default values.
+ */
+ZSTDLIB_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel);
+
+/*! ZSTD_CCtxParams_init_advanced() :
+ * Initializes the compression and frame parameters of cctxParams according to
+ * params. All other parameters are reset to their default values.
+ */
+ZSTDLIB_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params);
+
+/*! ZSTD_CCtxParams_setParameter() :
+ * Similar to ZSTD_CCtx_setParameter.
+ * Set one compression parameter, selected by enum ZSTD_cParameter.
+ * Parameters must be applied to a ZSTD_CCtx using
+ * ZSTD_CCtx_setParametersUsingCCtxParams().
+ * @result : a code representing success or failure (which can be tested with
+ * ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value);
+
+/*! ZSTD_CCtxParams_getParameter() :
+ * Similar to ZSTD_CCtx_getParameter.
+ * Get the requested value of one compression parameter, selected by enum ZSTD_cParameter.
+ * @result : 0, or an error code (which can be tested with ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value);
+
+/*! ZSTD_CCtx_setParametersUsingCCtxParams() :
+ * Apply a set of ZSTD_CCtx_params to the compression context.
+ * This can be done even after compression is started,
+ * if nbWorkers==0, this will have no impact until a new compression is started.
+ * if nbWorkers>=1, new parameters will be picked up at next job,
+ * with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated).
+ */
+ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
+ ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
+
+/*! ZSTD_compressStream2_simpleArgs() :
+ * Same as ZSTD_compressStream2(),
+ * but using only integral types as arguments.
+ * This variant might be helpful for binders from dynamic languages
+ * which have troubles handling structures containing memory pointers.
+ */
+ZSTDLIB_API size_t ZSTD_compressStream2_simpleArgs (
+ ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity, size_t* dstPos,
+ const void* src, size_t srcSize, size_t* srcPos,
+ ZSTD_EndDirective endOp);
+
+
+/* *************************************
+* Advanced decompression functions
+***************************************/
+
+/*! ZSTD_isFrame() :
+ * Tells if the content of `buffer` starts with a valid Frame Identifier.
+ * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
+ * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
+ * Note 3 : Skippable Frame Identifiers are considered valid. */
+ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);
+
+/*! ZSTD_createDDict_byReference() :
+ * Create a digested dictionary, ready to start decompression operation without startup delay.
+ * Dictionary content is referenced, and therefore stays in dictBuffer.
+ * It is important that dictBuffer outlives DDict,
+ * it must remain read accessible throughout the lifetime of DDict */
+ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize);
+
+/*! ZSTD_DCtx_loadDictionary_byReference() :
+ * Same as ZSTD_DCtx_loadDictionary(),
+ * but references `dict` content instead of copying it into `dctx`.
+ * This saves memory if `dict` remains around.,
+ * However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. */
+ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
+
+/*! ZSTD_DCtx_loadDictionary_advanced() :
+ * Same as ZSTD_DCtx_loadDictionary(),
+ * but gives direct control over
+ * how to load the dictionary (by copy ? by reference ?)
+ * and how to interpret it (automatic ? force raw mode ? full mode only ?). */
+ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
+
+/*! ZSTD_DCtx_refPrefix_advanced() :
+ * Same as ZSTD_DCtx_refPrefix(), but gives finer control over
+ * how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
+ZSTDLIB_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
+
+/*! ZSTD_DCtx_setMaxWindowSize() :
+ * Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
+ * This protects a decoder context from reserving too much memory for itself (potential attack scenario).
+ * This parameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
+ * By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
+ * @return : 0, or an error code (which can be tested using ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
+
+/*! ZSTD_DCtx_getParameter() :
+ * Get the requested decompression parameter value, selected by enum ZSTD_dParameter,
+ * and store it into int* value.
+ * @return : 0, or an error code (which can be tested with ZSTD_isError()).
+ */
+ZSTDLIB_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value);
+
+/* ZSTD_d_format
+ * experimental parameter,
+ * allowing selection between ZSTD_format_e input compression formats
+ */
+#define ZSTD_d_format ZSTD_d_experimentalParam1
+/* ZSTD_d_stableOutBuffer
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable.
+ *
+ * Tells the decompressor that the ZSTD_outBuffer will ALWAYS be the same
+ * between calls, except for the modifications that zstd makes to pos (the
+ * caller must not modify pos). This is checked by the decompressor, and
+ * decompression will fail if it ever changes. Therefore the ZSTD_outBuffer
+ * MUST be large enough to fit the entire decompressed frame. This will be
+ * checked when the frame content size is known. The data in the ZSTD_outBuffer
+ * in the range [dst, dst + pos) MUST not be modified during decompression
+ * or you will get data corruption.
+ *
+ * When this flags is enabled zstd won't allocate an output buffer, because
+ * it can write directly to the ZSTD_outBuffer, but it will still allocate
+ * an input buffer large enough to fit any compressed block. This will also
+ * avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
+ * If you need to avoid the input buffer allocation use the buffer-less
+ * streaming API.
+ *
+ * NOTE: So long as the ZSTD_outBuffer always points to valid memory, using
+ * this flag is ALWAYS memory safe, and will never access out-of-bounds
+ * memory. However, decompression WILL fail if you violate the preconditions.
+ *
+ * WARNING: The data in the ZSTD_outBuffer in the range [dst, dst + pos) MUST
+ * not be modified during decompression or you will get data corruption. This
+ * is because zstd needs to reference data in the ZSTD_outBuffer to regenerate
+ * matches. Normally zstd maintains its own buffer for this purpose, but passing
+ * this flag tells zstd to use the user provided buffer.
+ */
+#define ZSTD_d_stableOutBuffer ZSTD_d_experimentalParam2
+
+/* ZSTD_d_forceIgnoreChecksum
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable
+ *
+ * Tells the decompressor to skip checksum validation during decompression, regardless
+ * of whether checksumming was specified during compression. This offers some
+ * slight performance benefits, and may be useful for debugging.
+ * Param has values of type ZSTD_forceIgnoreChecksum_e
+ */
+#define ZSTD_d_forceIgnoreChecksum ZSTD_d_experimentalParam3
+
+/* ZSTD_d_refMultipleDDicts
+ * Experimental parameter.
+ * Default is 0 == disabled. Set to 1 to enable
+ *
+ * If enabled and dctx is allocated on the heap, then additional memory will be allocated
+ * to store references to multiple ZSTD_DDict. That is, multiple calls of ZSTD_refDDict()
+ * using a given ZSTD_DCtx, rather than overwriting the previous DDict reference, will instead
+ * store all references. At decompression time, the appropriate dictID is selected
+ * from the set of DDicts based on the dictID in the frame.
+ *
+ * Usage is simply calling ZSTD_refDDict() on multiple dict buffers.
+ *
+ * Param has values of byte ZSTD_refMultipleDDicts_e
+ *
+ * WARNING: Enabling this parameter and calling ZSTD_DCtx_refDDict(), will trigger memory
+ * allocation for the hash table. ZSTD_freeDCtx() also frees this memory.
+ * Memory is allocated as per ZSTD_DCtx::customMem.
+ *
+ * Although this function allocates memory for the table, the user is still responsible for
+ * memory management of the underlying ZSTD_DDict* themselves.
+ */
+#define ZSTD_d_refMultipleDDicts ZSTD_d_experimentalParam4
+
+
+/*! ZSTD_DCtx_setFormat() :
+ * Instruct the decoder context about what kind of data to decode next.
+ * This instruction is mandatory to decode data without a fully-formed header,
+ * such ZSTD_f_zstd1_magicless for example.
+ * @return : 0, or an error code (which can be tested using ZSTD_isError()). */
+ZSTDLIB_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
+
+/*! ZSTD_decompressStream_simpleArgs() :
+ * Same as ZSTD_decompressStream(),
+ * but using only integral types as arguments.
+ * This can be helpful for binders from dynamic languages
+ * which have troubles handling structures containing memory pointers.
+ */
+ZSTDLIB_API size_t ZSTD_decompressStream_simpleArgs (
+ ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity, size_t* dstPos,
+ const void* src, size_t srcSize, size_t* srcPos);
+
+
+/* ******************************************************************
+* Advanced streaming functions
+* Warning : most of these functions are now redundant with the Advanced API.
+* Once Advanced API reaches "stable" status,
+* redundant functions will be deprecated, and then at some point removed.
+********************************************************************/
+
+/*===== Advanced Streaming compression functions =====*/
+
+/*! ZSTD_initCStream_srcSize() :
+ * This function is deprecated, and equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
+ * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
+ * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
+ *
+ * pledgedSrcSize must be correct. If it is not known at init time, use
+ * ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs,
+ * "0" also disables frame content size field. It may be enabled in the future.
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t
+ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
+ int compressionLevel,
+ unsigned long long pledgedSrcSize);
+
+/*! ZSTD_initCStream_usingDict() :
+ * This function is deprecated, and is equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
+ * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
+ *
+ * Creates of an internal CDict (incompatible with static CCtx), except if
+ * dict == NULL or dictSize < 8, in which case no dict is used.
+ * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
+ * it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t
+ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
+ const void* dict, size_t dictSize,
+ int compressionLevel);
+
+/*! ZSTD_initCStream_advanced() :
+ * This function is deprecated, and is approximately equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * // Pseudocode: Set each zstd parameter and leave the rest as-is.
+ * for ((param, value) : params) {
+ * ZSTD_CCtx_setParameter(zcs, param, value);
+ * }
+ * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
+ * ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
+ *
+ * dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy.
+ * pledgedSrcSize must be correct.
+ * If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t
+ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
+ const void* dict, size_t dictSize,
+ ZSTD_parameters params,
+ unsigned long long pledgedSrcSize);
+
+/*! ZSTD_initCStream_usingCDict() :
+ * This function is deprecated, and equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * ZSTD_CCtx_refCDict(zcs, cdict);
+ *
+ * note : cdict will just be referenced, and must outlive compression session
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
+
+/*! ZSTD_initCStream_usingCDict_advanced() :
+ * This function is DEPRECATED, and is approximately equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
+ * for ((fParam, value) : fParams) {
+ * ZSTD_CCtx_setParameter(zcs, fParam, value);
+ * }
+ * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
+ * ZSTD_CCtx_refCDict(zcs, cdict);
+ *
+ * same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
+ * pledgedSrcSize must be correct. If srcSize is not known at init time, use
+ * value ZSTD_CONTENTSIZE_UNKNOWN.
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t
+ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
+ const ZSTD_CDict* cdict,
+ ZSTD_frameParameters fParams,
+ unsigned long long pledgedSrcSize);
+
+/*! ZSTD_resetCStream() :
+ * This function is deprecated, and is equivalent to:
+ * ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ * ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
+ *
+ * start a new frame, using same parameters from previous frame.
+ * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
+ * Note that zcs must be init at least once before using ZSTD_resetCStream().
+ * If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN.
+ * If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end.
+ * For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs,
+ * but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead.
+ * @return : 0, or an error code (which can be tested using ZSTD_isError())
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
+
+
+typedef struct {
+ unsigned long long ingested; /* nb input bytes read and buffered */
+ unsigned long long consumed; /* nb input bytes actually compressed */
+ unsigned long long produced; /* nb of compressed bytes generated and buffered */
+ unsigned long long flushed; /* nb of compressed bytes flushed : not provided; can be tracked from caller side */
+ unsigned currentJobID; /* MT only : latest started job nb */
+ unsigned nbActiveWorkers; /* MT only : nb of workers actively compressing at probe time */
+} ZSTD_frameProgression;
+
+/* ZSTD_getFrameProgression() :
+ * tells how much data has been ingested (read from input)
+ * consumed (input actually compressed) and produced (output) for current frame.
+ * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
+ * Aggregates progression inside active worker threads.
+ */
+ZSTDLIB_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx);
+
+/*! ZSTD_toFlushNow() :
+ * Tell how many bytes are ready to be flushed immediately.
+ * Useful for multithreading scenarios (nbWorkers >= 1).
+ * Probe the oldest active job, defined as oldest job not yet entirely flushed,
+ * and check its output buffer.
+ * @return : amount of data stored in oldest job and ready to be flushed immediately.
+ * if @return == 0, it means either :
+ * + there is no active job (could be checked with ZSTD_frameProgression()), or
+ * + oldest job is still actively compressing data,
+ * but everything it has produced has also been flushed so far,
+ * therefore flush speed is limited by production speed of oldest job
+ * irrespective of the speed of concurrent (and newer) jobs.
+ */
+ZSTDLIB_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);
+
+
+/*===== Advanced Streaming decompression functions =====*/
+
+/*!
+ * This function is deprecated, and is equivalent to:
+ *
+ * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
+ * ZSTD_DCtx_loadDictionary(zds, dict, dictSize);
+ *
+ * note: no dictionary will be used if dict == NULL or dictSize < 8
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
+
+/*!
+ * This function is deprecated, and is equivalent to:
+ *
+ * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
+ * ZSTD_DCtx_refDDict(zds, ddict);
+ *
+ * note : ddict is referenced, it must outlive decompression session
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
+
+/*!
+ * This function is deprecated, and is equivalent to:
+ *
+ * ZSTD_DCtx_reset(zds, ZSTD_reset_session_only);
+ *
+ * re-use decompression parameters from previous init; saves dictionary loading
+ * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ */
+ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
+
+
+/* *******************************************************************
+* Buffer-less and synchronous inner streaming functions
+*
+* This is an advanced API, giving full control over buffer management, for users which need direct control over memory.
+* But it's also a complex one, with several restrictions, documented below.
+* Prefer normal streaming API for an easier experience.
+********************************************************************* */
+
+/*
+ Buffer-less streaming compression (synchronous mode)
+
+ A ZSTD_CCtx object is required to track streaming operations.
+ Use ZSTD_createCCtx() / ZSTD_freeCCtx() to manage resource.
+ ZSTD_CCtx object can be re-used multiple times within successive compression operations.
+
+ Start by initializing a context.
+ Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
+ or ZSTD_compressBegin_advanced(), for finer parameter control.
+ It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
+
+ Then, consume your input using ZSTD_compressContinue().
+ There are some important considerations to keep in mind when using this advanced function :
+ - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
+ - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
+ - Caller must ensure there is enough space in `dst` to store compressed data under worst case scenario.
+ Worst case evaluation is provided by ZSTD_compressBound().
+ ZSTD_compressContinue() doesn't guarantee recover after a failed compression.
+ - ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to maximum distance size, see WindowLog).
+ It remembers all previous contiguous blocks, plus one separated memory segment (which can itself consists of multiple contiguous blocks)
+ - ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
+ In which case, it will "discard" the relevant memory section from its history.
+
+ Finish a frame with ZSTD_compressEnd(), which will write the last block(s) and optional checksum.
+ It's possible to use srcSize==0, in which case, it will write a final empty block to end the frame.
+ Without last block mark, frames are considered unfinished (hence corrupted) by compliant decoders.
+
+ `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
+*/
+
+/*===== Buffer-less streaming compression functions =====*/
+ZSTDLIB_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
+ZSTDLIB_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
+ZSTDLIB_API size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /*< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */
+ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /*< note: fails if cdict==NULL */
+ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
+ZSTDLIB_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /*< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
+
+ZSTDLIB_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+
+
+/*
+ Buffer-less streaming decompression (synchronous mode)
+
+ A ZSTD_DCtx object is required to track streaming operations.
+ Use ZSTD_createDCtx() / ZSTD_freeDCtx() to manage it.
+ A ZSTD_DCtx object can be re-used multiple times.
+
+ First typical operation is to retrieve frame parameters, using ZSTD_getFrameHeader().
+ Frame header is extracted from the beginning of compressed frame, so providing only the frame's beginning is enough.
+ Data fragment must be large enough to ensure successful decoding.
+ `ZSTD_frameHeaderSize_max` bytes is guaranteed to always be large enough.
+ @result : 0 : successful decoding, the `ZSTD_frameHeader` structure is correctly filled.
+ >0 : `srcSize` is too small, please provide at least @result bytes on next attempt.
+ errorCode, which can be tested using ZSTD_isError().
+
+ It fills a ZSTD_frameHeader structure with important information to correctly decode the frame,
+ such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
+ Note that these values could be wrong, either because of data corruption, or because a 3rd party deliberately spoofs false information.
+ As a consequence, check that values remain within valid application range.
+ For example, do not allocate memory blindly, check that `windowSize` is within expectation.
+ Each application can set its own limits, depending on local restrictions.
+ For extended interoperability, it is recommended to support `windowSize` of at least 8 MB.
+
+ ZSTD_decompressContinue() needs previous data blocks during decompression, up to `windowSize` bytes.
+ ZSTD_decompressContinue() is very sensitive to contiguity,
+ if 2 blocks don't follow each other, make sure that either the compressor breaks contiguity at the same place,
+ or that previous contiguous segment is large enough to properly handle maximum back-reference distance.
+ There are multiple ways to guarantee this condition.
+
+ The most memory efficient way is to use a round buffer of sufficient size.
+ Sufficient size is determined by invoking ZSTD_decodingBufferSize_min(),
+ which can @return an error code if required value is too large for current system (in 32-bits mode).
+ In a round buffer methodology, ZSTD_decompressContinue() decompresses each block next to previous one,
+ up to the moment there is not enough room left in the buffer to guarantee decoding another full block,
+ which maximum size is provided in `ZSTD_frameHeader` structure, field `blockSizeMax`.
+ At which point, decoding can resume from the beginning of the buffer.
+ Note that already decoded data stored in the buffer should be flushed before being overwritten.
+
+ There are alternatives possible, for example using two or more buffers of size `windowSize` each, though they consume more memory.
+
+ Finally, if you control the compression process, you can also ignore all buffer size rules,
+ as long as the encoder and decoder progress in "lock-step",
+ aka use exactly the same buffer sizes, break contiguity at the same place, etc.
+
+ Once buffers are setup, start decompression, with ZSTD_decompressBegin().
+ If decompression requires a dictionary, use ZSTD_decompressBegin_usingDict() or ZSTD_decompressBegin_usingDDict().
+
+ Then use ZSTD_nextSrcSizeToDecompress() and ZSTD_decompressContinue() alternatively.
+ ZSTD_nextSrcSizeToDecompress() tells how many bytes to provide as 'srcSize' to ZSTD_decompressContinue().
+ ZSTD_decompressContinue() requires this _exact_ amount of bytes, or it will fail.
+
+ @result of ZSTD_decompressContinue() is the number of bytes regenerated within 'dst' (necessarily <= dstCapacity).
+ It can be zero : it just means ZSTD_decompressContinue() has decoded some metadata item.
+ It can also be an error code, which can be tested with ZSTD_isError().
+
+ A frame is fully decoded when ZSTD_nextSrcSizeToDecompress() returns zero.
+ Context can then be reset to start a new decompression.
+
+ Note : it's possible to know if next input to present is a header or a block, using ZSTD_nextInputType().
+ This information is not required to properly decode a frame.
+
+ == Special case : skippable frames ==
+
+ Skippable frames allow integration of user-defined data into a flow of concatenated frames.
+ Skippable frames will be ignored (skipped) by decompressor.
+ The format of skippable frames is as follows :
+ a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
+ b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
+ c) Frame Content - any content (User Data) of length equal to Frame Size
+ For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
+ For skippable frames ZSTD_decompressContinue() always returns 0 : it only skips the content.
+*/
+
+/*===== Buffer-less streaming decompression functions =====*/
+typedef enum { ZSTD_frame, ZSTD_skippableFrame } ZSTD_frameType_e;
+typedef struct {
+ unsigned long long frameContentSize; /* if == ZSTD_CONTENTSIZE_UNKNOWN, it means this field is not available. 0 means "empty" */
+ unsigned long long windowSize; /* can be very large, up to <= frameContentSize */
+ unsigned blockSizeMax;
+ ZSTD_frameType_e frameType; /* if == ZSTD_skippableFrame, frameContentSize is the size of skippable content */
+ unsigned headerSize;
+ unsigned dictID;
+ unsigned checksumFlag;
+} ZSTD_frameHeader;
+
+/*! ZSTD_getFrameHeader() :
+ * decode Frame Header, or requires larger `srcSize`.
+ * @return : 0, `zfhPtr` is correctly filled,
+ * >0, `srcSize` is too small, value is wanted `srcSize` amount,
+ * or an error code, which can be tested using ZSTD_isError() */
+ZSTDLIB_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /*< doesn't consume input */
+/*! ZSTD_getFrameHeader_advanced() :
+ * same as ZSTD_getFrameHeader(),
+ * with added capability to select a format (like ZSTD_f_zstd1_magicless) */
+ZSTDLIB_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format);
+ZSTDLIB_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); /*< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */
+
+ZSTDLIB_API size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
+ZSTDLIB_API size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
+ZSTDLIB_API size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
+
+ZSTDLIB_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
+ZSTDLIB_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+
+/* misc */
+ZSTDLIB_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
+typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
+ZSTDLIB_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
+
+
+
+
+/* ============================ */
+/* Block level API */
+/* ============================ */
+
+/*!
+ Block functions produce and decode raw zstd blocks, without frame metadata.
+ Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 100 bytes).
+ But users will have to take in charge needed metadata to regenerate data, such as compressed and content sizes.
+
+ A few rules to respect :
+ - Compressing and decompressing require a context structure
+ + Use ZSTD_createCCtx() and ZSTD_createDCtx()
+ - It is necessary to init context before starting
+ + compression : any ZSTD_compressBegin*() variant, including with dictionary
+ + decompression : any ZSTD_decompressBegin*() variant, including with dictionary
+ + copyCCtx() and copyDCtx() can be used too
+ - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
+ + If input is larger than a block size, it's necessary to split input data into multiple blocks
+ + For inputs larger than a single block, consider using regular ZSTD_compress() instead.
+ Frame metadata is not that costly, and quickly becomes negligible as source size grows larger than a block.
+ - When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero) !
+ ===> In which case, nothing is produced into `dst` !
+ + User __must__ test for such outcome and deal directly with uncompressed data
+ + A block cannot be declared incompressible if ZSTD_compressBlock() return value was != 0.
+ Doing so would mess up with statistics history, leading to potential data corruption.
+ + ZSTD_decompressBlock() _doesn't accept uncompressed data as input_ !!
+ + In case of multiple successive blocks, should some of them be uncompressed,
+ decoder must be informed of their existence in order to follow proper history.
+ Use ZSTD_insertBlock() for such a case.
+*/
+
+/*===== Raw zstd block functions =====*/
+ZSTDLIB_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
+ZSTDLIB_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */
+
+
+#endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */
+
#include <linux/bitmap.h>
#include <linux/bug.h>
+#include <linux/container_of.h>
#include <linux/fwnode.h>
-#include <linux/kernel.h>
#include <linux/list.h>
#include <linux/media.h>
+#include <linux/types.h>
/* Enums used internally at the media controller to represent graphs */
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * include/net/9p/9p.h
- *
* 9P protocol definitions.
*
* Copyright (C) 2005 by Latchesar Ionkov <lucho@ionkov.net>
*/
enum p9_debug_flags {
- P9_DEBUG_ERROR = (1<<0),
- P9_DEBUG_9P = (1<<2),
+ P9_DEBUG_ERROR = (1<<0),
+ P9_DEBUG_9P = (1<<2),
P9_DEBUG_VFS = (1<<3),
P9_DEBUG_CONV = (1<<4),
P9_DEBUG_MUX = (1<<5),
P9_DEBUG_TRANS = (1<<6),
- P9_DEBUG_SLABS = (1<<7),
+ P9_DEBUG_SLABS = (1<<7),
P9_DEBUG_FCALL = (1<<8),
P9_DEBUG_FID = (1<<9),
P9_DEBUG_PKT = (1<<10),
};
/* 9P Magic Numbers */
-#define P9_NOTAG (u16)(~0)
-#define P9_NOFID (u32)(~0)
+#define P9_NOTAG ((u16)(~0))
+#define P9_NOFID ((u32)(~0))
#define P9_MAXWELEM 16
/* Minimal header size: size[4] type[1] tag[2] */
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * include/net/9p/client.h
- *
* 9P Client Definitions
*
* Copyright (C) 2008 by Eric Van Hensbergen <ericvh@gmail.com>
* @p9_proto_2000L: 9P2000.L extension
*/
-enum p9_proto_versions{
+enum p9_proto_versions {
p9_proto_legacy,
p9_proto_2000u,
p9_proto_2000L,
u64 request_mask);
int p9_client_mknod_dotl(struct p9_fid *oldfid, const char *name, int mode,
- dev_t rdev, kgid_t gid, struct p9_qid *);
+ dev_t rdev, kgid_t gid, struct p9_qid *qid);
int p9_client_mkdir_dotl(struct p9_fid *fid, const char *name, int mode,
- kgid_t gid, struct p9_qid *);
+ kgid_t gid, struct p9_qid *qid);
int p9_client_lock_dotl(struct p9_fid *fid, struct p9_flock *flock, u8 *status);
int p9_client_getlock_dotl(struct p9_fid *fid, struct p9_getlock *fl);
void p9_fcall_fini(struct p9_fcall *fc);
-struct p9_req_t *p9_tag_lookup(struct p9_client *, u16);
+struct p9_req_t *p9_tag_lookup(struct p9_client *c, u16 tag);
static inline void p9_req_get(struct p9_req_t *r)
{
void p9_client_cb(struct p9_client *c, struct p9_req_t *req, int status);
-int p9_parse_header(struct p9_fcall *, int32_t *, int8_t *, int16_t *, int);
-int p9stat_read(struct p9_client *, char *, int, struct p9_wstat *);
-void p9stat_free(struct p9_wstat *);
+int p9_parse_header(struct p9_fcall *pdu, int32_t *size, int8_t *type,
+ int16_t *tag, int rewind);
+int p9stat_read(struct p9_client *clnt, char *buf, int len,
+ struct p9_wstat *st);
+void p9stat_free(struct p9_wstat *stbuf);
int p9_is_proto_dotu(struct p9_client *clnt);
int p9_is_proto_dotl(struct p9_client *clnt);
-struct p9_fid *p9_client_xattrwalk(struct p9_fid *, const char *, u64 *);
-int p9_client_xattrcreate(struct p9_fid *, const char *, u64, int);
+struct p9_fid *p9_client_xattrwalk(struct p9_fid *file_fid,
+ const char *attr_name, u64 *attr_size);
+int p9_client_xattrcreate(struct p9_fid *fid, const char *name,
+ u64 attr_size, int flags);
int p9_client_readlink(struct p9_fid *fid, char **target);
int p9_client_init(void);
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * include/net/9p/transport.h
- *
* Transport Definition
*
* Copyright (C) 2005 by Latchesar Ionkov <lucho@ionkov.net>
#ifndef NET_9P_TRANSPORT_H
#define NET_9P_TRANSPORT_H
+#include <linux/module.h>
+
#define P9_DEF_MIN_RESVPORT (665U)
#define P9_DEF_MAX_RESVPORT (1023U)
int maxsize; /* max message size of transport */
int def; /* this transport should be default */
struct module *owner;
- int (*create)(struct p9_client *, const char *, char *);
- void (*close) (struct p9_client *);
- int (*request) (struct p9_client *, struct p9_req_t *req);
- int (*cancel) (struct p9_client *, struct p9_req_t *req);
- int (*cancelled)(struct p9_client *, struct p9_req_t *req);
- int (*zc_request)(struct p9_client *, struct p9_req_t *,
- struct iov_iter *, struct iov_iter *, int , int, int);
- int (*show_options)(struct seq_file *, struct p9_client *);
+ int (*create)(struct p9_client *client,
+ const char *devname, char *args);
+ void (*close)(struct p9_client *client);
+ int (*request)(struct p9_client *client, struct p9_req_t *req);
+ int (*cancel)(struct p9_client *client, struct p9_req_t *req);
+ int (*cancelled)(struct p9_client *client, struct p9_req_t *req);
+ int (*zc_request)(struct p9_client *client, struct p9_req_t *req,
+ struct iov_iter *uidata, struct iov_iter *uodata,
+ int inlen, int outlen, int in_hdr_len);
+ int (*show_options)(struct seq_file *m, struct p9_client *client);
};
void v9fs_register_trans(struct p9_trans_module *m);
struct p9_trans_module *v9fs_get_trans_by_name(char *s);
struct p9_trans_module *v9fs_get_default_trans(void);
void v9fs_put_trans(struct p9_trans_module *m);
+
+#define MODULE_ALIAS_9P(transport) \
+ MODULE_ALIAS("9p-" transport)
+
#endif /* NET_9P_TRANSPORT_H */
static inline
struct hlist_head *llc_sk_dev_hash(struct llc_sap *sap, int ifindex)
{
- return &sap->sk_dev_hash[ifindex % LLC_SK_DEV_HASH_ENTRIES];
+ u32 bucket = hash_32(ifindex, LLC_SK_DEV_HASH_BITS);
+
+ return &sap->sk_dev_hash[bucket];
}
static inline
NCI_UP,
NCI_DATA_EXCHANGE,
NCI_DATA_EXCHANGE_TO,
+ NCI_UNREG,
};
/* NCI device states */
page_pool_put_full_page(pool, page, true);
}
+#define PAGE_POOL_DMA_USE_PP_FRAG_COUNT \
+ (sizeof(dma_addr_t) > sizeof(unsigned long))
+
static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
{
- return page->dma_addr;
+ dma_addr_t ret = page->dma_addr;
+
+ if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT)
+ ret |= (dma_addr_t)page->dma_addr_upper << 16 << 16;
+
+ return ret;
}
static inline void page_pool_set_dma_addr(struct page *page, dma_addr_t addr)
{
page->dma_addr = addr;
+ if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT)
+ page->dma_addr_upper = upper_32_bits(addr);
}
static inline void page_pool_set_frag_count(struct page *page, long nr)
reconf_enable:1;
__u8 strreset_enable;
-
- /* Security identifiers from incoming (INIT). These are set by
- * security_sctp_assoc_request(). These will only be used by
- * SCTP TCP type sockets and peeled off connections as they
- * cause a new socket to be generated. security_sctp_sk_clone()
- * will then plug these into the new socket.
- */
-
- u32 secid;
- u32 peer_secid;
};
/* Recover the outter endpoint structure. */
__u64 abandoned_unsent[SCTP_PR_INDEX(MAX) + 1];
__u64 abandoned_sent[SCTP_PR_INDEX(MAX) + 1];
+ /* Security identifiers from incoming (INIT). These are set by
+ * security_sctp_assoc_request(). These will only be used by
+ * SCTP TCP type sockets and peeled off connections as they
+ * cause a new socket to be generated. security_sctp_sk_clone()
+ * will then plug these into the new socket.
+ */
+
+ u32 secid;
+ u32 peer_secid;
+
struct rcu_head rcu;
};
int offset;
};
+struct _strp_msg {
+ /* Internal cb structure. struct strp_msg must be first for passing
+ * to upper layer.
+ */
+ struct strp_msg strp;
+ int accum_len;
+};
+
+struct sk_skb_cb {
+#define SK_SKB_CB_PRIV_LEN 20
+ unsigned char data[SK_SKB_CB_PRIV_LEN];
+ struct _strp_msg strp;
+ /* temp_reg is a temporary register used for bpf_convert_data_end_access
+ * when dst_reg == src_reg.
+ */
+ u64 temp_reg;
+};
+
static inline struct strp_msg *strp_msg(struct sk_buff *skb)
{
return (struct strp_msg *)((void *)skb->cb +
- offsetof(struct qdisc_skb_cb, data));
+ offsetof(struct sk_skb_cb, strp));
}
/* Structure for an attached lower socket */
static inline void tcp_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
{
sk_wmem_queued_add(sk, -skb->truesize);
- sk_mem_uncharge(sk, skb->truesize);
+ if (!skb_zcopy_pure(skb))
+ sk_mem_uncharge(sk, skb->truesize);
+ else
+ sk_mem_uncharge(sk, SKB_TRUESIZE(skb_end_offset(skb)));
__kfree_skb(skb);
}
const struct sk_buff *from)
{
return likely(tcp_skb_can_collapse_to(to) &&
- mptcp_skb_can_collapse(to, from));
+ mptcp_skb_can_collapse(to, from) &&
+ skb_pure_zcopy_same(to, from));
}
/* Events passed to congestion control interface */
* constant as well and the compiler checks they are the same.
*/
#define MODULE_ALIAS_RDMA_NETLINK(_index, _val) \
- static inline void __chk_##_index(void) \
+ static inline void __maybe_unused __chk_##_index(void) \
{ \
BUILD_BUG_ON(_index != _val); \
} \
int sas_set_phy_speed(struct sas_phy *phy, struct sas_phy_linkrates *rates);
int sas_phy_reset(struct sas_phy *phy, int hard_reset);
+int sas_phy_enable(struct sas_phy *phy, int enable);
extern int sas_queuecommand(struct Scsi_Host *, struct scsi_cmnd *);
extern int sas_target_alloc(struct scsi_target *);
extern int sas_slave_configure(struct scsi_device *);
#include <linux/timer.h>
#include <linux/scatterlist.h>
#include <scsi/scsi_device.h>
-#include <scsi/scsi_host.h>
#include <scsi/scsi_request.h>
struct Scsi_Host;
#define SCMD_STATE_COMPLETE 0
#define SCMD_STATE_INFLIGHT 1
+enum scsi_cmnd_submitter {
+ SUBMITTED_BY_BLOCK_LAYER = 0,
+ SUBMITTED_BY_SCSI_ERROR_HANDLER = 1,
+ SUBMITTED_BY_SCSI_RESET_IOCTL = 2,
+} __packed;
+
struct scsi_cmnd {
struct scsi_request req;
struct scsi_device *device;
- struct list_head eh_entry; /* entry for the host eh_cmd_q */
+ struct list_head eh_entry; /* entry for the host eh_abort_list/eh_cmd_q */
struct delayed_work abort_work;
struct rcu_head rcu;
unsigned char prot_op;
unsigned char prot_type;
unsigned char prot_flags;
+ enum scsi_cmnd_submitter submitter;
unsigned short cmd_len;
enum dma_data_direction sc_data_direction;
* command (auto-sense). Length must be
* SCSI_SENSE_BUFFERSIZE bytes. */
- /* Low-level done function - can be used by low-level driver to point
- * to completion function. Not used by mid/upper level code. */
- void (*scsi_done) (struct scsi_cmnd *);
-
/*
* The following fields can be written to by the host specific code.
* Everything else should be left alone.
return *(struct scsi_driver **)rq->rq_disk->private_data;
}
+void scsi_done(struct scsi_cmnd *cmd);
+
extern void scsi_finish_command(struct scsi_cmnd *cmd);
extern void *scsi_kmap_atomic_sg(struct scatterlist *sg, int sg_count,
* creation time */
unsigned ignore_media_change:1; /* Ignore MEDIA CHANGE on resume */
+ unsigned int queue_stopped; /* request queue is quiesced */
bool offline_already; /* Device offline message logged */
atomic_t disk_events_disable_depth; /* disable depth for disk events */
#define SCSI_DEFAULT_HOST_BLOCKED 7
/*
- * Pointer to the sysfs class properties for this host, NULL terminated.
+ * Pointer to the SCSI host sysfs attribute groups, NULL terminated.
*/
- struct device_attribute **shost_attrs;
-
- /*
- * Pointer to the SCSI device properties for this host, NULL terminated.
- */
- struct device_attribute **sdev_attrs;
+ const struct attribute_group **shost_groups;
/*
* Pointer to the SCSI device attribute groups for this host,
unsigned long irq_flags; \
int rc; \
spin_lock_irqsave(shost->host_lock, irq_flags); \
- rc = func_name##_lck (cmd, cmd->scsi_done); \
+ rc = func_name##_lck(cmd); \
spin_unlock_irqrestore(shost->host_lock, irq_flags); \
return rc; \
}
struct mutex scan_mutex;/* serialize scanning activity */
+ struct list_head eh_abort_list;
struct list_head eh_cmd_q;
struct task_struct * ehandler; /* Error recovery thread. */
struct completion * eh_action; /* Wait for specific actions on the
/* ldm bits */
struct device shost_gendev, shost_dev;
+ /*
+ * The array size 3 provides space for one attribute group defined by
+ * the SCSI core, one attribute group defined by the SCSI LLD and one
+ * terminating NULL pointer.
+ */
+ const struct attribute_group *shost_dev_attr_groups[3];
/*
* Points to the transport data (if any) which is allocated
struct class_container;
/*
- * These two functions are used to allocate and free a pseudo device
- * which will connect to the host adapter itself rather than any
- * physical device. You must deallocate when you are done with the
- * thing. This physical pseudo-device isn't real and won't be available
- * from any high-level drivers.
- */
-extern void scsi_free_host_dev(struct scsi_device *);
-extern struct scsi_device *scsi_get_host_dev(struct Scsi_Host *);
-
-/*
* DIF defines the exchange of protection information between
* initiator and SBC block device.
*
SAS_LINK_RATE_G2 = SAS_LINK_RATE_3_0_GBPS,
SAS_LINK_RATE_6_0_GBPS = 10,
SAS_LINK_RATE_12_0_GBPS = 11,
+ SAS_LINK_RATE_22_5_GBPS = 12,
/* These are virtual to the transport class and may never
* be signalled normally since the standard defined field
* is only 4 bits */
#define SNDRV_DMA_TYPE_CONTINUOUS 1 /* continuous no-DMA memory */
#define SNDRV_DMA_TYPE_DEV 2 /* generic device continuous */
#define SNDRV_DMA_TYPE_DEV_WC 5 /* continuous write-combined */
+#ifdef CONFIG_SND_DMA_SGBUF
+#define SNDRV_DMA_TYPE_DEV_SG 3 /* generic device SG-buffer */
+#define SNDRV_DMA_TYPE_DEV_WC_SG 6 /* SG write-combined */
+#else
+#define SNDRV_DMA_TYPE_DEV_SG SNDRV_DMA_TYPE_DEV /* no SG-buf support */
+#define SNDRV_DMA_TYPE_DEV_WC_SG SNDRV_DMA_TYPE_DEV_WC
+#endif
#ifdef CONFIG_GENERIC_ALLOCATOR
#define SNDRV_DMA_TYPE_DEV_IRAM 4 /* generic device iram-buffer */
#else
#define SNDRV_DMA_TYPE_VMALLOC 7 /* vmalloc'ed buffer */
#define SNDRV_DMA_TYPE_NONCONTIG 8 /* non-coherent SG buffer */
#define SNDRV_DMA_TYPE_NONCOHERENT 9 /* non-coherent buffer */
-#ifdef CONFIG_SND_DMA_SGBUF
-#define SNDRV_DMA_TYPE_DEV_SG SNDRV_DMA_TYPE_NONCONTIG
-#define SNDRV_DMA_TYPE_DEV_WC_SG 6 /* SG write-combined */
-#else
-#define SNDRV_DMA_TYPE_DEV_SG SNDRV_DMA_TYPE_DEV /* no SG-buf support */
-#define SNDRV_DMA_TYPE_DEV_WC_SG SNDRV_DMA_TYPE_DEV_WC
-#endif
/*
* info for buffer allocation
/* ALUA target port group linkage */
struct list_head lun_tg_pt_gp_link;
- struct t10_alua_tg_pt_gp *lun_tg_pt_gp;
+ struct t10_alua_tg_pt_gp __rcu *lun_tg_pt_gp;
spinlock_t lun_tg_pt_gp_lock;
struct se_portal_group *lun_tpg;
atomic_long_t read_bytes;
atomic_long_t write_bytes;
/* Active commands on this virtual SE device */
- atomic_t simple_cmds;
- atomic_t dev_ordered_sync;
+ atomic_t non_ordered;
+ bool ordered_sync_in_progress;
+ atomic_t delayed_cmd_count;
atomic_t dev_qf_count;
u32 export_count;
spinlock_t delayed_cmd_lock;
struct list_head dev_sep_list;
struct list_head dev_tmr_list;
struct work_struct qf_work_queue;
+ struct work_struct delayed_cmd_work;
struct list_head delayed_cmd_list;
struct list_head qf_cmd_list;
/* Pointer to associated SE HBA */
* Negative values can be used by fabric drivers for internal use TPGs.
*/
int proto_id;
+ bool enabled;
/* Used for PR SPEC_I_PT=1 and REGISTER_AND_MOVE */
atomic_t tpg_pr_ref_count;
/* Spinlock for adding/removing ACLed Nodes */
void (*add_wwn_groups)(struct se_wwn *);
struct se_portal_group *(*fabric_make_tpg)(struct se_wwn *,
const char *);
+ int (*fabric_enable_tpg)(struct se_portal_group *se_tpg, bool enable);
void (*fabric_drop_tpg)(struct se_portal_group *);
int (*fabric_post_link)(struct se_portal_group *,
struct se_lun *);
__entry->vnode, __entry->off, __entry->i_size)
);
-TRACE_EVENT(afs_page_dirty,
- TP_PROTO(struct afs_vnode *vnode, const char *where, struct page *page),
+TRACE_EVENT(afs_folio_dirty,
+ TP_PROTO(struct afs_vnode *vnode, const char *where, struct folio *folio),
- TP_ARGS(vnode, where, page),
+ TP_ARGS(vnode, where, folio),
TP_STRUCT__entry(
__field(struct afs_vnode *, vnode )
__field(const char *, where )
- __field(pgoff_t, page )
+ __field(pgoff_t, index )
__field(unsigned long, from )
__field(unsigned long, to )
),
TP_fast_assign(
+ unsigned long priv = (unsigned long)folio_get_private(folio);
__entry->vnode = vnode;
__entry->where = where;
- __entry->page = page->index;
- __entry->from = afs_page_dirty_from(page, page->private);
- __entry->to = afs_page_dirty_to(page, page->private);
- __entry->to |= (afs_is_page_dirty_mmapped(page->private) ?
- (1UL << (BITS_PER_LONG - 1)) : 0);
+ __entry->index = folio_index(folio);
+ __entry->from = afs_folio_dirty_from(folio, priv);
+ __entry->to = afs_folio_dirty_to(folio, priv);
+ __entry->to |= (afs_is_folio_dirty_mmapped(priv) ?
+ (1UL << (BITS_PER_LONG - 1)) : 0);
),
TP_printk("vn=%p %lx %s %lx-%lx%s",
- __entry->vnode, __entry->page, __entry->where,
+ __entry->vnode, __entry->index, __entry->where,
__entry->from,
__entry->to & ~(1UL << (BITS_PER_LONG - 1)),
__entry->to & (1UL << (BITS_PER_LONG - 1)) ? " M" : "")
);
TRACE_EVENT(f2fs_map_blocks,
- TP_PROTO(struct inode *inode, struct f2fs_map_blocks *map, int ret),
+ TP_PROTO(struct inode *inode, struct f2fs_map_blocks *map,
+ int create, int flag, int ret),
- TP_ARGS(inode, map, ret),
+ TP_ARGS(inode, map, create, flag, ret),
TP_STRUCT__entry(
__field(dev_t, dev)
__field(unsigned int, m_flags)
__field(int, m_seg_type)
__field(bool, m_may_create)
+ __field(bool, m_multidev_dio)
+ __field(int, create)
+ __field(int, flag)
__field(int, ret)
),
TP_fast_assign(
- __entry->dev = inode->i_sb->s_dev;
+ __entry->dev = map->m_bdev->bd_dev;
__entry->ino = inode->i_ino;
__entry->m_lblk = map->m_lblk;
__entry->m_pblk = map->m_pblk;
__entry->m_flags = map->m_flags;
__entry->m_seg_type = map->m_seg_type;
__entry->m_may_create = map->m_may_create;
+ __entry->m_multidev_dio = map->m_multidev_dio;
+ __entry->create = create;
+ __entry->flag = flag;
__entry->ret = ret;
),
TP_printk("dev = (%d,%d), ino = %lu, file offset = %llu, "
- "start blkaddr = 0x%llx, len = 0x%llx, flags = %u,"
- "seg_type = %d, may_create = %d, err = %d",
+ "start blkaddr = 0x%llx, len = 0x%llx, flags = %u, "
+ "seg_type = %d, may_create = %d, multidevice = %d, "
+ "create = %d, flag = %d, err = %d",
show_dev_ino(__entry),
(unsigned long long)__entry->m_lblk,
(unsigned long long)__entry->m_pblk,
__entry->m_flags,
__entry->m_seg_type,
__entry->m_may_create,
+ __entry->m_multidev_dio,
+ __entry->create,
+ __entry->flag,
__entry->ret)
);
TP_STRUCT__entry(
__field(dev_t, dev)
__field(ino_t, ino)
- __field(const char *, name)
+ __string(name, dentry->d_name.name)
__field(unsigned int, flags)
),
TP_fast_assign(
__entry->dev = dir->i_sb->s_dev;
__entry->ino = dir->i_ino;
- __entry->name = dentry->d_name.name;
+ __assign_str(name, dentry->d_name.name);
__entry->flags = flags;
),
TP_printk("dev = (%d,%d), pino = %lu, name:%s, flags:%u",
show_dev_ino(__entry),
- __entry->name,
+ __get_str(name),
__entry->flags)
);
TP_STRUCT__entry(
__field(dev_t, dev)
__field(ino_t, ino)
- __field(const char *, name)
+ __string(name, dentry->d_name.name)
__field(nid_t, cino)
__field(int, err)
),
TP_fast_assign(
__entry->dev = dir->i_sb->s_dev;
__entry->ino = dir->i_ino;
- __entry->name = dentry->d_name.name;
+ __assign_str(name, dentry->d_name.name);
__entry->cino = ino;
__entry->err = err;
),
TP_printk("dev = (%d,%d), pino = %lu, name:%s, ino:%u, err:%d",
show_dev_ino(__entry),
- __entry->name,
+ __get_str(name),
__entry->cino,
__entry->err)
);
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Display helpers for generic filesystem items
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2020, Oracle and/or its affiliates.
+ */
+
+#include <linux/fs.h>
+
+#define show_fs_dirent_type(x) \
+ __print_symbolic(x, \
+ { DT_UNKNOWN, "UNKNOWN" }, \
+ { DT_FIFO, "FIFO" }, \
+ { DT_CHR, "CHR" }, \
+ { DT_DIR, "DIR" }, \
+ { DT_BLK, "BLK" }, \
+ { DT_REG, "REG" }, \
+ { DT_LNK, "LNK" }, \
+ { DT_SOCK, "SOCK" }, \
+ { DT_WHT, "WHT" })
+
+#define show_fs_fcntl_open_flags(x) \
+ __print_flags(x, "|", \
+ { O_WRONLY, "O_WRONLY" }, \
+ { O_RDWR, "O_RDWR" }, \
+ { O_CREAT, "O_CREAT" }, \
+ { O_EXCL, "O_EXCL" }, \
+ { O_NOCTTY, "O_NOCTTY" }, \
+ { O_TRUNC, "O_TRUNC" }, \
+ { O_APPEND, "O_APPEND" }, \
+ { O_NONBLOCK, "O_NONBLOCK" }, \
+ { O_DSYNC, "O_DSYNC" }, \
+ { O_DIRECT, "O_DIRECT" }, \
+ { O_LARGEFILE, "O_LARGEFILE" }, \
+ { O_DIRECTORY, "O_DIRECTORY" }, \
+ { O_NOFOLLOW, "O_NOFOLLOW" }, \
+ { O_NOATIME, "O_NOATIME" }, \
+ { O_CLOEXEC, "O_CLOEXEC" })
+
+#define __fmode_flag(x) { (__force unsigned long)FMODE_##x, #x }
+#define show_fs_fmode_flags(x) \
+ __print_flags(x, "|", \
+ __fmode_flag(READ), \
+ __fmode_flag(WRITE), \
+ __fmode_flag(EXEC))
+
+#ifdef CONFIG_64BIT
+#define show_fs_fcntl_cmd(x) \
+ __print_symbolic(x, \
+ { F_DUPFD, "DUPFD" }, \
+ { F_GETFD, "GETFD" }, \
+ { F_SETFD, "SETFD" }, \
+ { F_GETFL, "GETFL" }, \
+ { F_SETFL, "SETFL" }, \
+ { F_GETLK, "GETLK" }, \
+ { F_SETLK, "SETLK" }, \
+ { F_SETLKW, "SETLKW" }, \
+ { F_SETOWN, "SETOWN" }, \
+ { F_GETOWN, "GETOWN" }, \
+ { F_SETSIG, "SETSIG" }, \
+ { F_GETSIG, "GETSIG" }, \
+ { F_SETOWN_EX, "SETOWN_EX" }, \
+ { F_GETOWN_EX, "GETOWN_EX" }, \
+ { F_GETOWNER_UIDS, "GETOWNER_UIDS" }, \
+ { F_OFD_GETLK, "OFD_GETLK" }, \
+ { F_OFD_SETLK, "OFD_SETLK" }, \
+ { F_OFD_SETLKW, "OFD_SETLKW" })
+#else /* CONFIG_64BIT */
+#define show_fs_fcntl_cmd(x) \
+ __print_symbolic(x, \
+ { F_DUPFD, "DUPFD" }, \
+ { F_GETFD, "GETFD" }, \
+ { F_SETFD, "SETFD" }, \
+ { F_GETFL, "GETFL" }, \
+ { F_SETFL, "SETFL" }, \
+ { F_GETLK, "GETLK" }, \
+ { F_SETLK, "SETLK" }, \
+ { F_SETLKW, "SETLKW" }, \
+ { F_SETOWN, "SETOWN" }, \
+ { F_GETOWN, "GETOWN" }, \
+ { F_SETSIG, "SETSIG" }, \
+ { F_GETSIG, "GETSIG" }, \
+ { F_GETLK64, "GETLK64" }, \
+ { F_SETLK64, "SETLK64" }, \
+ { F_SETLKW64, "SETLKW64" }, \
+ { F_SETOWN_EX, "SETOWN_EX" }, \
+ { F_GETOWN_EX, "GETOWN_EX" }, \
+ { F_GETOWNER_UIDS, "GETOWNER_UIDS" }, \
+ { F_OFD_GETLK, "OFD_GETLK" }, \
+ { F_OFD_SETLK, "OFD_SETLK" }, \
+ { F_OFD_SETLKW, "OFD_SETLKW" })
+#endif /* CONFIG_64BIT */
+
+#define show_fs_fcntl_lock_type(x) \
+ __print_symbolic(x, \
+ { F_RDLCK, "RDLCK" }, \
+ { F_WRLCK, "WRLCK" }, \
+ { F_UNLCK, "UNLCK" })
+
+#define show_fs_lookup_flags(flags) \
+ __print_flags(flags, "|", \
+ { LOOKUP_FOLLOW, "FOLLOW" }, \
+ { LOOKUP_DIRECTORY, "DIRECTORY" }, \
+ { LOOKUP_AUTOMOUNT, "AUTOMOUNT" }, \
+ { LOOKUP_EMPTY, "EMPTY" }, \
+ { LOOKUP_DOWN, "DOWN" }, \
+ { LOOKUP_MOUNTPOINT, "MOUNTPOINT" }, \
+ { LOOKUP_REVAL, "REVAL" }, \
+ { LOOKUP_RCU, "RCU" }, \
+ { LOOKUP_OPEN, "OPEN" }, \
+ { LOOKUP_CREATE, "CREATE" }, \
+ { LOOKUP_EXCL, "EXCL" }, \
+ { LOOKUP_RENAME_TARGET, "RENAME_TARGET" }, \
+ { LOOKUP_PARENT, "PARENT" }, \
+ { LOOKUP_NO_SYMLINKS, "NO_SYMLINKS" }, \
+ { LOOKUP_NO_MAGICLINKS, "NO_MAGICLINKS" }, \
+ { LOOKUP_NO_XDEV, "NO_XDEV" }, \
+ { LOOKUP_BENEATH, "BENEATH" }, \
+ { LOOKUP_IN_ROOT, "IN_ROOT" }, \
+ { LOOKUP_CACHED, "CACHED" })
extern int trace_mmap_lock_reg(void);
extern void trace_mmap_lock_unreg(void);
-TRACE_EVENT_FN(mmap_lock_start_locking,
+DECLARE_EVENT_CLASS(mmap_lock,
TP_PROTO(struct mm_struct *mm, const char *memcg_path, bool write),
),
TP_printk(
- "mm=%p memcg_path=%s write=%s\n",
+ "mm=%p memcg_path=%s write=%s",
__entry->mm,
__get_str(memcg_path),
__entry->write ? "true" : "false"
- ),
-
- trace_mmap_lock_reg, trace_mmap_lock_unreg
+ )
);
+#define DEFINE_MMAP_LOCK_EVENT(name) \
+ DEFINE_EVENT_FN(mmap_lock, name, \
+ TP_PROTO(struct mm_struct *mm, const char *memcg_path, \
+ bool write), \
+ TP_ARGS(mm, memcg_path, write), \
+ trace_mmap_lock_reg, trace_mmap_lock_unreg)
+
+DEFINE_MMAP_LOCK_EVENT(mmap_lock_start_locking);
+DEFINE_MMAP_LOCK_EVENT(mmap_lock_released);
+
TRACE_EVENT_FN(mmap_lock_acquire_returned,
TP_PROTO(struct mm_struct *mm, const char *memcg_path, bool write,
),
TP_printk(
- "mm=%p memcg_path=%s write=%s success=%s\n",
+ "mm=%p memcg_path=%s write=%s success=%s",
__entry->mm,
__get_str(memcg_path),
__entry->write ? "true" : "false",
trace_mmap_lock_reg, trace_mmap_lock_unreg
);
-TRACE_EVENT_FN(mmap_lock_released,
-
- TP_PROTO(struct mm_struct *mm, const char *memcg_path, bool write),
-
- TP_ARGS(mm, memcg_path, write),
-
- TP_STRUCT__entry(
- __field(struct mm_struct *, mm)
- __string(memcg_path, memcg_path)
- __field(bool, write)
- ),
-
- TP_fast_assign(
- __entry->mm = mm;
- __assign_str(memcg_path, memcg_path);
- __entry->write = write;
- ),
-
- TP_printk(
- "mm=%p memcg_path=%s write=%s\n",
- __entry->mm,
- __get_str(memcg_path),
- __entry->write ? "true" : "false"
- ),
-
- trace_mmap_lock_reg, trace_mmap_lock_unreg
-);
-
#endif /* _TRACE_MMAP_LOCK_H */
/* This part must be outside protection */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Display helpers for NFS protocol elements
+ *
+ * Author: Chuck Lever <chuck.lever@oracle.com>
+ *
+ * Copyright (c) 2020, Oracle and/or its affiliates.
+ */
+
+#include <linux/nfs.h>
+#include <linux/nfs4.h>
+#include <uapi/linux/nfs.h>
+
+TRACE_DEFINE_ENUM(NFS_OK);
+TRACE_DEFINE_ENUM(NFSERR_PERM);
+TRACE_DEFINE_ENUM(NFSERR_NOENT);
+TRACE_DEFINE_ENUM(NFSERR_IO);
+TRACE_DEFINE_ENUM(NFSERR_NXIO);
+TRACE_DEFINE_ENUM(NFSERR_EAGAIN);
+TRACE_DEFINE_ENUM(NFSERR_ACCES);
+TRACE_DEFINE_ENUM(NFSERR_EXIST);
+TRACE_DEFINE_ENUM(NFSERR_XDEV);
+TRACE_DEFINE_ENUM(NFSERR_NODEV);
+TRACE_DEFINE_ENUM(NFSERR_NOTDIR);
+TRACE_DEFINE_ENUM(NFSERR_ISDIR);
+TRACE_DEFINE_ENUM(NFSERR_INVAL);
+TRACE_DEFINE_ENUM(NFSERR_FBIG);
+TRACE_DEFINE_ENUM(NFSERR_NOSPC);
+TRACE_DEFINE_ENUM(NFSERR_ROFS);
+TRACE_DEFINE_ENUM(NFSERR_MLINK);
+TRACE_DEFINE_ENUM(NFSERR_OPNOTSUPP);
+TRACE_DEFINE_ENUM(NFSERR_NAMETOOLONG);
+TRACE_DEFINE_ENUM(NFSERR_NOTEMPTY);
+TRACE_DEFINE_ENUM(NFSERR_DQUOT);
+TRACE_DEFINE_ENUM(NFSERR_STALE);
+TRACE_DEFINE_ENUM(NFSERR_REMOTE);
+TRACE_DEFINE_ENUM(NFSERR_WFLUSH);
+TRACE_DEFINE_ENUM(NFSERR_BADHANDLE);
+TRACE_DEFINE_ENUM(NFSERR_NOT_SYNC);
+TRACE_DEFINE_ENUM(NFSERR_BAD_COOKIE);
+TRACE_DEFINE_ENUM(NFSERR_NOTSUPP);
+TRACE_DEFINE_ENUM(NFSERR_TOOSMALL);
+TRACE_DEFINE_ENUM(NFSERR_SERVERFAULT);
+TRACE_DEFINE_ENUM(NFSERR_BADTYPE);
+TRACE_DEFINE_ENUM(NFSERR_JUKEBOX);
+
+#define show_nfs_status(x) \
+ __print_symbolic(x, \
+ { NFS_OK, "OK" }, \
+ { NFSERR_PERM, "PERM" }, \
+ { NFSERR_NOENT, "NOENT" }, \
+ { NFSERR_IO, "IO" }, \
+ { NFSERR_NXIO, "NXIO" }, \
+ { ECHILD, "CHILD" }, \
+ { NFSERR_EAGAIN, "AGAIN" }, \
+ { NFSERR_ACCES, "ACCES" }, \
+ { NFSERR_EXIST, "EXIST" }, \
+ { NFSERR_XDEV, "XDEV" }, \
+ { NFSERR_NODEV, "NODEV" }, \
+ { NFSERR_NOTDIR, "NOTDIR" }, \
+ { NFSERR_ISDIR, "ISDIR" }, \
+ { NFSERR_INVAL, "INVAL" }, \
+ { NFSERR_FBIG, "FBIG" }, \
+ { NFSERR_NOSPC, "NOSPC" }, \
+ { NFSERR_ROFS, "ROFS" }, \
+ { NFSERR_MLINK, "MLINK" }, \
+ { NFSERR_OPNOTSUPP, "OPNOTSUPP" }, \
+ { NFSERR_NAMETOOLONG, "NAMETOOLONG" }, \
+ { NFSERR_NOTEMPTY, "NOTEMPTY" }, \
+ { NFSERR_DQUOT, "DQUOT" }, \
+ { NFSERR_STALE, "STALE" }, \
+ { NFSERR_REMOTE, "REMOTE" }, \
+ { NFSERR_WFLUSH, "WFLUSH" }, \
+ { NFSERR_BADHANDLE, "BADHANDLE" }, \
+ { NFSERR_NOT_SYNC, "NOTSYNC" }, \
+ { NFSERR_BAD_COOKIE, "BADCOOKIE" }, \
+ { NFSERR_NOTSUPP, "NOTSUPP" }, \
+ { NFSERR_TOOSMALL, "TOOSMALL" }, \
+ { NFSERR_SERVERFAULT, "REMOTEIO" }, \
+ { NFSERR_BADTYPE, "BADTYPE" }, \
+ { NFSERR_JUKEBOX, "JUKEBOX" })
+
+TRACE_DEFINE_ENUM(NFS_UNSTABLE);
+TRACE_DEFINE_ENUM(NFS_DATA_SYNC);
+TRACE_DEFINE_ENUM(NFS_FILE_SYNC);
+
+#define show_nfs_stable_how(x) \
+ __print_symbolic(x, \
+ { NFS_UNSTABLE, "UNSTABLE" }, \
+ { NFS_DATA_SYNC, "DATA_SYNC" }, \
+ { NFS_FILE_SYNC, "FILE_SYNC" })
+
+TRACE_DEFINE_ENUM(NFS4_OK);
+TRACE_DEFINE_ENUM(NFS4ERR_ACCESS);
+TRACE_DEFINE_ENUM(NFS4ERR_ATTRNOTSUPP);
+TRACE_DEFINE_ENUM(NFS4ERR_ADMIN_REVOKED);
+TRACE_DEFINE_ENUM(NFS4ERR_BACK_CHAN_BUSY);
+TRACE_DEFINE_ENUM(NFS4ERR_BADCHAR);
+TRACE_DEFINE_ENUM(NFS4ERR_BADHANDLE);
+TRACE_DEFINE_ENUM(NFS4ERR_BADIOMODE);
+TRACE_DEFINE_ENUM(NFS4ERR_BADLAYOUT);
+TRACE_DEFINE_ENUM(NFS4ERR_BADLABEL);
+TRACE_DEFINE_ENUM(NFS4ERR_BADNAME);
+TRACE_DEFINE_ENUM(NFS4ERR_BADOWNER);
+TRACE_DEFINE_ENUM(NFS4ERR_BADSESSION);
+TRACE_DEFINE_ENUM(NFS4ERR_BADSLOT);
+TRACE_DEFINE_ENUM(NFS4ERR_BADTYPE);
+TRACE_DEFINE_ENUM(NFS4ERR_BADXDR);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_COOKIE);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_HIGH_SLOT);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_RANGE);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_SEQID);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_SESSION_DIGEST);
+TRACE_DEFINE_ENUM(NFS4ERR_BAD_STATEID);
+TRACE_DEFINE_ENUM(NFS4ERR_CB_PATH_DOWN);
+TRACE_DEFINE_ENUM(NFS4ERR_CLID_INUSE);
+TRACE_DEFINE_ENUM(NFS4ERR_CLIENTID_BUSY);
+TRACE_DEFINE_ENUM(NFS4ERR_COMPLETE_ALREADY);
+TRACE_DEFINE_ENUM(NFS4ERR_CONN_NOT_BOUND_TO_SESSION);
+TRACE_DEFINE_ENUM(NFS4ERR_DEADLOCK);
+TRACE_DEFINE_ENUM(NFS4ERR_DEADSESSION);
+TRACE_DEFINE_ENUM(NFS4ERR_DELAY);
+TRACE_DEFINE_ENUM(NFS4ERR_DELEG_ALREADY_WANTED);
+TRACE_DEFINE_ENUM(NFS4ERR_DELEG_REVOKED);
+TRACE_DEFINE_ENUM(NFS4ERR_DENIED);
+TRACE_DEFINE_ENUM(NFS4ERR_DIRDELEG_UNAVAIL);
+TRACE_DEFINE_ENUM(NFS4ERR_DQUOT);
+TRACE_DEFINE_ENUM(NFS4ERR_ENCR_ALG_UNSUPP);
+TRACE_DEFINE_ENUM(NFS4ERR_EXIST);
+TRACE_DEFINE_ENUM(NFS4ERR_EXPIRED);
+TRACE_DEFINE_ENUM(NFS4ERR_FBIG);
+TRACE_DEFINE_ENUM(NFS4ERR_FHEXPIRED);
+TRACE_DEFINE_ENUM(NFS4ERR_FILE_OPEN);
+TRACE_DEFINE_ENUM(NFS4ERR_GRACE);
+TRACE_DEFINE_ENUM(NFS4ERR_HASH_ALG_UNSUPP);
+TRACE_DEFINE_ENUM(NFS4ERR_INVAL);
+TRACE_DEFINE_ENUM(NFS4ERR_IO);
+TRACE_DEFINE_ENUM(NFS4ERR_ISDIR);
+TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTTRYLATER);
+TRACE_DEFINE_ENUM(NFS4ERR_LAYOUTUNAVAILABLE);
+TRACE_DEFINE_ENUM(NFS4ERR_LEASE_MOVED);
+TRACE_DEFINE_ENUM(NFS4ERR_LOCKED);
+TRACE_DEFINE_ENUM(NFS4ERR_LOCKS_HELD);
+TRACE_DEFINE_ENUM(NFS4ERR_LOCK_RANGE);
+TRACE_DEFINE_ENUM(NFS4ERR_MINOR_VERS_MISMATCH);
+TRACE_DEFINE_ENUM(NFS4ERR_MLINK);
+TRACE_DEFINE_ENUM(NFS4ERR_MOVED);
+TRACE_DEFINE_ENUM(NFS4ERR_NAMETOOLONG);
+TRACE_DEFINE_ENUM(NFS4ERR_NOENT);
+TRACE_DEFINE_ENUM(NFS4ERR_NOFILEHANDLE);
+TRACE_DEFINE_ENUM(NFS4ERR_NOMATCHING_LAYOUT);
+TRACE_DEFINE_ENUM(NFS4ERR_NOSPC);
+TRACE_DEFINE_ENUM(NFS4ERR_NOTDIR);
+TRACE_DEFINE_ENUM(NFS4ERR_NOTEMPTY);
+TRACE_DEFINE_ENUM(NFS4ERR_NOTSUPP);
+TRACE_DEFINE_ENUM(NFS4ERR_NOT_ONLY_OP);
+TRACE_DEFINE_ENUM(NFS4ERR_NOT_SAME);
+TRACE_DEFINE_ENUM(NFS4ERR_NO_GRACE);
+TRACE_DEFINE_ENUM(NFS4ERR_NXIO);
+TRACE_DEFINE_ENUM(NFS4ERR_OLD_STATEID);
+TRACE_DEFINE_ENUM(NFS4ERR_OPENMODE);
+TRACE_DEFINE_ENUM(NFS4ERR_OP_ILLEGAL);
+TRACE_DEFINE_ENUM(NFS4ERR_OP_NOT_IN_SESSION);
+TRACE_DEFINE_ENUM(NFS4ERR_PERM);
+TRACE_DEFINE_ENUM(NFS4ERR_PNFS_IO_HOLE);
+TRACE_DEFINE_ENUM(NFS4ERR_PNFS_NO_LAYOUT);
+TRACE_DEFINE_ENUM(NFS4ERR_RECALLCONFLICT);
+TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_BAD);
+TRACE_DEFINE_ENUM(NFS4ERR_RECLAIM_CONFLICT);
+TRACE_DEFINE_ENUM(NFS4ERR_REJECT_DELEG);
+TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG);
+TRACE_DEFINE_ENUM(NFS4ERR_REP_TOO_BIG_TO_CACHE);
+TRACE_DEFINE_ENUM(NFS4ERR_REQ_TOO_BIG);
+TRACE_DEFINE_ENUM(NFS4ERR_RESOURCE);
+TRACE_DEFINE_ENUM(NFS4ERR_RESTOREFH);
+TRACE_DEFINE_ENUM(NFS4ERR_RETRY_UNCACHED_REP);
+TRACE_DEFINE_ENUM(NFS4ERR_RETURNCONFLICT);
+TRACE_DEFINE_ENUM(NFS4ERR_ROFS);
+TRACE_DEFINE_ENUM(NFS4ERR_SAME);
+TRACE_DEFINE_ENUM(NFS4ERR_SHARE_DENIED);
+TRACE_DEFINE_ENUM(NFS4ERR_SEQUENCE_POS);
+TRACE_DEFINE_ENUM(NFS4ERR_SEQ_FALSE_RETRY);
+TRACE_DEFINE_ENUM(NFS4ERR_SEQ_MISORDERED);
+TRACE_DEFINE_ENUM(NFS4ERR_SERVERFAULT);
+TRACE_DEFINE_ENUM(NFS4ERR_STALE);
+TRACE_DEFINE_ENUM(NFS4ERR_STALE_CLIENTID);
+TRACE_DEFINE_ENUM(NFS4ERR_STALE_STATEID);
+TRACE_DEFINE_ENUM(NFS4ERR_SYMLINK);
+TRACE_DEFINE_ENUM(NFS4ERR_TOOSMALL);
+TRACE_DEFINE_ENUM(NFS4ERR_TOO_MANY_OPS);
+TRACE_DEFINE_ENUM(NFS4ERR_UNKNOWN_LAYOUTTYPE);
+TRACE_DEFINE_ENUM(NFS4ERR_UNSAFE_COMPOUND);
+TRACE_DEFINE_ENUM(NFS4ERR_WRONGSEC);
+TRACE_DEFINE_ENUM(NFS4ERR_WRONG_CRED);
+TRACE_DEFINE_ENUM(NFS4ERR_WRONG_TYPE);
+TRACE_DEFINE_ENUM(NFS4ERR_XDEV);
+
+TRACE_DEFINE_ENUM(NFS4ERR_RESET_TO_MDS);
+TRACE_DEFINE_ENUM(NFS4ERR_RESET_TO_PNFS);
+
+#define show_nfs4_status(x) \
+ __print_symbolic(x, \
+ { NFS4_OK, "OK" }, \
+ { EPERM, "EPERM" }, \
+ { ENOENT, "ENOENT" }, \
+ { EIO, "EIO" }, \
+ { ENXIO, "ENXIO" }, \
+ { EACCES, "EACCES" }, \
+ { EEXIST, "EEXIST" }, \
+ { EXDEV, "EXDEV" }, \
+ { ENOTDIR, "ENOTDIR" }, \
+ { EISDIR, "EISDIR" }, \
+ { EFBIG, "EFBIG" }, \
+ { ENOSPC, "ENOSPC" }, \
+ { EROFS, "EROFS" }, \
+ { EMLINK, "EMLINK" }, \
+ { ENAMETOOLONG, "ENAMETOOLONG" }, \
+ { ENOTEMPTY, "ENOTEMPTY" }, \
+ { EDQUOT, "EDQUOT" }, \
+ { ESTALE, "ESTALE" }, \
+ { EBADHANDLE, "EBADHANDLE" }, \
+ { EBADCOOKIE, "EBADCOOKIE" }, \
+ { ENOTSUPP, "ENOTSUPP" }, \
+ { ETOOSMALL, "ETOOSMALL" }, \
+ { EREMOTEIO, "EREMOTEIO" }, \
+ { EBADTYPE, "EBADTYPE" }, \
+ { EAGAIN, "EAGAIN" }, \
+ { ELOOP, "ELOOP" }, \
+ { EOPNOTSUPP, "EOPNOTSUPP" }, \
+ { EDEADLK, "EDEADLK" }, \
+ { ENOMEM, "ENOMEM" }, \
+ { EKEYEXPIRED, "EKEYEXPIRED" }, \
+ { ETIMEDOUT, "ETIMEDOUT" }, \
+ { ERESTARTSYS, "ERESTARTSYS" }, \
+ { ECONNREFUSED, "ECONNREFUSED" }, \
+ { ECONNRESET, "ECONNRESET" }, \
+ { ENETUNREACH, "ENETUNREACH" }, \
+ { EHOSTUNREACH, "EHOSTUNREACH" }, \
+ { EHOSTDOWN, "EHOSTDOWN" }, \
+ { EPIPE, "EPIPE" }, \
+ { EPFNOSUPPORT, "EPFNOSUPPORT" }, \
+ { EPROTONOSUPPORT, "EPROTONOSUPPORT" }, \
+ { NFS4ERR_ACCESS, "ACCESS" }, \
+ { NFS4ERR_ATTRNOTSUPP, "ATTRNOTSUPP" }, \
+ { NFS4ERR_ADMIN_REVOKED, "ADMIN_REVOKED" }, \
+ { NFS4ERR_BACK_CHAN_BUSY, "BACK_CHAN_BUSY" }, \
+ { NFS4ERR_BADCHAR, "BADCHAR" }, \
+ { NFS4ERR_BADHANDLE, "BADHANDLE" }, \
+ { NFS4ERR_BADIOMODE, "BADIOMODE" }, \
+ { NFS4ERR_BADLAYOUT, "BADLAYOUT" }, \
+ { NFS4ERR_BADLABEL, "BADLABEL" }, \
+ { NFS4ERR_BADNAME, "BADNAME" }, \
+ { NFS4ERR_BADOWNER, "BADOWNER" }, \
+ { NFS4ERR_BADSESSION, "BADSESSION" }, \
+ { NFS4ERR_BADSLOT, "BADSLOT" }, \
+ { NFS4ERR_BADTYPE, "BADTYPE" }, \
+ { NFS4ERR_BADXDR, "BADXDR" }, \
+ { NFS4ERR_BAD_COOKIE, "BAD_COOKIE" }, \
+ { NFS4ERR_BAD_HIGH_SLOT, "BAD_HIGH_SLOT" }, \
+ { NFS4ERR_BAD_RANGE, "BAD_RANGE" }, \
+ { NFS4ERR_BAD_SEQID, "BAD_SEQID" }, \
+ { NFS4ERR_BAD_SESSION_DIGEST, "BAD_SESSION_DIGEST" }, \
+ { NFS4ERR_BAD_STATEID, "BAD_STATEID" }, \
+ { NFS4ERR_CB_PATH_DOWN, "CB_PATH_DOWN" }, \
+ { NFS4ERR_CLID_INUSE, "CLID_INUSE" }, \
+ { NFS4ERR_CLIENTID_BUSY, "CLIENTID_BUSY" }, \
+ { NFS4ERR_COMPLETE_ALREADY, "COMPLETE_ALREADY" }, \
+ { NFS4ERR_CONN_NOT_BOUND_TO_SESSION, "CONN_NOT_BOUND_TO_SESSION" }, \
+ { NFS4ERR_DEADLOCK, "DEADLOCK" }, \
+ { NFS4ERR_DEADSESSION, "DEAD_SESSION" }, \
+ { NFS4ERR_DELAY, "DELAY" }, \
+ { NFS4ERR_DELEG_ALREADY_WANTED, "DELEG_ALREADY_WANTED" }, \
+ { NFS4ERR_DELEG_REVOKED, "DELEG_REVOKED" }, \
+ { NFS4ERR_DENIED, "DENIED" }, \
+ { NFS4ERR_DIRDELEG_UNAVAIL, "DIRDELEG_UNAVAIL" }, \
+ { NFS4ERR_DQUOT, "DQUOT" }, \
+ { NFS4ERR_ENCR_ALG_UNSUPP, "ENCR_ALG_UNSUPP" }, \
+ { NFS4ERR_EXIST, "EXIST" }, \
+ { NFS4ERR_EXPIRED, "EXPIRED" }, \
+ { NFS4ERR_FBIG, "FBIG" }, \
+ { NFS4ERR_FHEXPIRED, "FHEXPIRED" }, \
+ { NFS4ERR_FILE_OPEN, "FILE_OPEN" }, \
+ { NFS4ERR_GRACE, "GRACE" }, \
+ { NFS4ERR_HASH_ALG_UNSUPP, "HASH_ALG_UNSUPP" }, \
+ { NFS4ERR_INVAL, "INVAL" }, \
+ { NFS4ERR_IO, "IO" }, \
+ { NFS4ERR_ISDIR, "ISDIR" }, \
+ { NFS4ERR_LAYOUTTRYLATER, "LAYOUTTRYLATER" }, \
+ { NFS4ERR_LAYOUTUNAVAILABLE, "LAYOUTUNAVAILABLE" }, \
+ { NFS4ERR_LEASE_MOVED, "LEASE_MOVED" }, \
+ { NFS4ERR_LOCKED, "LOCKED" }, \
+ { NFS4ERR_LOCKS_HELD, "LOCKS_HELD" }, \
+ { NFS4ERR_LOCK_RANGE, "LOCK_RANGE" }, \
+ { NFS4ERR_MINOR_VERS_MISMATCH, "MINOR_VERS_MISMATCH" }, \
+ { NFS4ERR_MLINK, "MLINK" }, \
+ { NFS4ERR_MOVED, "MOVED" }, \
+ { NFS4ERR_NAMETOOLONG, "NAMETOOLONG" }, \
+ { NFS4ERR_NOENT, "NOENT" }, \
+ { NFS4ERR_NOFILEHANDLE, "NOFILEHANDLE" }, \
+ { NFS4ERR_NOMATCHING_LAYOUT, "NOMATCHING_LAYOUT" }, \
+ { NFS4ERR_NOSPC, "NOSPC" }, \
+ { NFS4ERR_NOTDIR, "NOTDIR" }, \
+ { NFS4ERR_NOTEMPTY, "NOTEMPTY" }, \
+ { NFS4ERR_NOTSUPP, "NOTSUPP" }, \
+ { NFS4ERR_NOT_ONLY_OP, "NOT_ONLY_OP" }, \
+ { NFS4ERR_NOT_SAME, "NOT_SAME" }, \
+ { NFS4ERR_NO_GRACE, "NO_GRACE" }, \
+ { NFS4ERR_NXIO, "NXIO" }, \
+ { NFS4ERR_OLD_STATEID, "OLD_STATEID" }, \
+ { NFS4ERR_OPENMODE, "OPENMODE" }, \
+ { NFS4ERR_OP_ILLEGAL, "OP_ILLEGAL" }, \
+ { NFS4ERR_OP_NOT_IN_SESSION, "OP_NOT_IN_SESSION" }, \
+ { NFS4ERR_PERM, "PERM" }, \
+ { NFS4ERR_PNFS_IO_HOLE, "PNFS_IO_HOLE" }, \
+ { NFS4ERR_PNFS_NO_LAYOUT, "PNFS_NO_LAYOUT" }, \
+ { NFS4ERR_RECALLCONFLICT, "RECALLCONFLICT" }, \
+ { NFS4ERR_RECLAIM_BAD, "RECLAIM_BAD" }, \
+ { NFS4ERR_RECLAIM_CONFLICT, "RECLAIM_CONFLICT" }, \
+ { NFS4ERR_REJECT_DELEG, "REJECT_DELEG" }, \
+ { NFS4ERR_REP_TOO_BIG, "REP_TOO_BIG" }, \
+ { NFS4ERR_REP_TOO_BIG_TO_CACHE, "REP_TOO_BIG_TO_CACHE" }, \
+ { NFS4ERR_REQ_TOO_BIG, "REQ_TOO_BIG" }, \
+ { NFS4ERR_RESOURCE, "RESOURCE" }, \
+ { NFS4ERR_RESTOREFH, "RESTOREFH" }, \
+ { NFS4ERR_RETRY_UNCACHED_REP, "RETRY_UNCACHED_REP" }, \
+ { NFS4ERR_RETURNCONFLICT, "RETURNCONFLICT" }, \
+ { NFS4ERR_ROFS, "ROFS" }, \
+ { NFS4ERR_SAME, "SAME" }, \
+ { NFS4ERR_SHARE_DENIED, "SHARE_DENIED" }, \
+ { NFS4ERR_SEQUENCE_POS, "SEQUENCE_POS" }, \
+ { NFS4ERR_SEQ_FALSE_RETRY, "SEQ_FALSE_RETRY" }, \
+ { NFS4ERR_SEQ_MISORDERED, "SEQ_MISORDERED" }, \
+ { NFS4ERR_SERVERFAULT, "SERVERFAULT" }, \
+ { NFS4ERR_STALE, "STALE" }, \
+ { NFS4ERR_STALE_CLIENTID, "STALE_CLIENTID" }, \
+ { NFS4ERR_STALE_STATEID, "STALE_STATEID" }, \
+ { NFS4ERR_SYMLINK, "SYMLINK" }, \
+ { NFS4ERR_TOOSMALL, "TOOSMALL" }, \
+ { NFS4ERR_TOO_MANY_OPS, "TOO_MANY_OPS" }, \
+ { NFS4ERR_UNKNOWN_LAYOUTTYPE, "UNKNOWN_LAYOUTTYPE" }, \
+ { NFS4ERR_UNSAFE_COMPOUND, "UNSAFE_COMPOUND" }, \
+ { NFS4ERR_WRONGSEC, "WRONGSEC" }, \
+ { NFS4ERR_WRONG_CRED, "WRONG_CRED" }, \
+ { NFS4ERR_WRONG_TYPE, "WRONG_TYPE" }, \
+ { NFS4ERR_XDEV, "XDEV" }, \
+ /* ***** Internal to Linux NFS client ***** */ \
+ { NFS4ERR_RESET_TO_MDS, "RESET_TO_MDS" }, \
+ { NFS4ERR_RESET_TO_PNFS, "RESET_TO_PNFS" })
+
+#define show_nfs4_verifier(x) \
+ __print_hex_str(x, NFS4_VERIFIER_SIZE)
+
+TRACE_DEFINE_ENUM(IOMODE_READ);
+TRACE_DEFINE_ENUM(IOMODE_RW);
+TRACE_DEFINE_ENUM(IOMODE_ANY);
+
+#define show_pnfs_layout_iomode(x) \
+ __print_symbolic(x, \
+ { IOMODE_READ, "READ" }, \
+ { IOMODE_RW, "RW" }, \
+ { IOMODE_ANY, "ANY" })
+
+#define show_nfs4_seq4_status(x) \
+ __print_flags(x, "|", \
+ { SEQ4_STATUS_CB_PATH_DOWN, "CB_PATH_DOWN" }, \
+ { SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRING, "CB_GSS_CONTEXTS_EXPIRING" }, \
+ { SEQ4_STATUS_CB_GSS_CONTEXTS_EXPIRED, "CB_GSS_CONTEXTS_EXPIRED" }, \
+ { SEQ4_STATUS_EXPIRED_ALL_STATE_REVOKED, "EXPIRED_ALL_STATE_REVOKED" }, \
+ { SEQ4_STATUS_EXPIRED_SOME_STATE_REVOKED, "EXPIRED_SOME_STATE_REVOKED" }, \
+ { SEQ4_STATUS_ADMIN_STATE_REVOKED, "ADMIN_STATE_REVOKED" }, \
+ { SEQ4_STATUS_RECALLABLE_STATE_REVOKED, "RECALLABLE_STATE_REVOKED" }, \
+ { SEQ4_STATUS_LEASE_MOVED, "LEASE_MOVED" }, \
+ { SEQ4_STATUS_RESTART_RECLAIM_NEEDED, "RESTART_RECLAIM_NEEDED" }, \
+ { SEQ4_STATUS_CB_PATH_DOWN_SESSION, "CB_PATH_DOWN_SESSION" }, \
+ { SEQ4_STATUS_BACKCHANNEL_FAULT, "BACKCHANNEL_FAULT" })
#include <linux/tracepoint.h>
+#include <trace/events/sunrpc_base.h>
+
/**
** GSS-API related trace events
**/
__entry->maj_stat = maj_stat;
),
- TP_printk("task:%u@%u maj_stat=%s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " maj_stat=%s",
__entry->task_id, __entry->client_id,
__entry->maj_stat == 0 ?
"GSS_S_COMPLETE" : show_gss_status(__entry->maj_stat))
__entry->client_id = task->tk_client->cl_clid;
),
- TP_printk("task:%u@%u", __entry->task_id, __entry->client_id)
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER,
+ __entry->task_id, __entry->client_id)
);
TRACE_EVENT(rpcgss_bad_seqno,
__entry->received = received;
),
- TP_printk("task:%u@%u expected seqno %u, received seqno %u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " expected seqno %u, received seqno %u",
__entry->task_id, __entry->client_id,
__entry->expected, __entry->received)
);
__entry->seqno = rqst->rq_seqno;
),
- TP_printk("task:%u@%u xid=0x%08x seqno=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x seqno=%u",
__entry->task_id, __entry->client_id,
__entry->xid, __entry->seqno)
);
__entry->ret = ret;
),
- TP_printk("task:%u@%u xid=0x%08x rq_seqno=%u seq_xmit=%u reencode %sneeded",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x rq_seqno=%u seq_xmit=%u reencode %sneeded",
__entry->task_id, __entry->client_id,
__entry->xid, __entry->seqno, __entry->seq_xmit,
__entry->ret ? "" : "un")
__entry->verfsize = auth->au_verfsize;
),
- TP_printk("task:%u@%u xid=0x%08x auth=%p rslack=%u ralign=%u verfsize=%u\n",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x auth=%p rslack=%u ralign=%u verfsize=%u\n",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->auth, __entry->rslack, __entry->ralign,
__entry->verfsize)
#include <linux/sunrpc/rpc_rdma_cid.h>
#include <linux/tracepoint.h>
#include <rdma/ib_cm.h>
+
#include <trace/events/rdma.h>
+#include <trace/events/sunrpc_base.h>
/**
** Event classes
), \
TP_ARGS(wc, cid))
+DECLARE_EVENT_CLASS(rpcrdma_send_completion_class,
+ TP_PROTO(
+ const struct ib_wc *wc,
+ const struct rpc_rdma_cid *cid
+ ),
+
+ TP_ARGS(wc, cid),
+
+ TP_STRUCT__entry(
+ __field(u32, cq_id)
+ __field(int, completion_id)
+ ),
+
+ TP_fast_assign(
+ __entry->cq_id = cid->ci_queue_id;
+ __entry->completion_id = cid->ci_completion_id;
+ ),
+
+ TP_printk("cq.id=%u cid=%d",
+ __entry->cq_id, __entry->completion_id
+ )
+);
+
+#define DEFINE_SEND_COMPLETION_EVENT(name) \
+ DEFINE_EVENT(rpcrdma_send_completion_class, name, \
+ TP_PROTO( \
+ const struct ib_wc *wc, \
+ const struct rpc_rdma_cid *cid \
+ ), \
+ TP_ARGS(wc, cid))
+
+DECLARE_EVENT_CLASS(rpcrdma_send_flush_class,
+ TP_PROTO(
+ const struct ib_wc *wc,
+ const struct rpc_rdma_cid *cid
+ ),
+
+ TP_ARGS(wc, cid),
+
+ TP_STRUCT__entry(
+ __field(u32, cq_id)
+ __field(int, completion_id)
+ __field(unsigned long, status)
+ __field(unsigned int, vendor_err)
+ ),
+
+ TP_fast_assign(
+ __entry->cq_id = cid->ci_queue_id;
+ __entry->completion_id = cid->ci_completion_id;
+ __entry->status = wc->status;
+ __entry->vendor_err = wc->vendor_err;
+ ),
+
+ TP_printk("cq.id=%u cid=%d status=%s (%lu/0x%x)",
+ __entry->cq_id, __entry->completion_id,
+ rdma_show_wc_status(__entry->status),
+ __entry->status, __entry->vendor_err
+ )
+);
+
+#define DEFINE_SEND_FLUSH_EVENT(name) \
+ DEFINE_EVENT(rpcrdma_send_flush_class, name, \
+ TP_PROTO( \
+ const struct ib_wc *wc, \
+ const struct rpc_rdma_cid *cid \
+ ), \
+ TP_ARGS(wc, cid))
+
DECLARE_EVENT_CLASS(rpcrdma_mr_completion_class,
TP_PROTO(
const struct ib_wc *wc,
), \
TP_ARGS(wc, cid))
+DECLARE_EVENT_CLASS(rpcrdma_receive_success_class,
+ TP_PROTO(
+ const struct ib_wc *wc,
+ const struct rpc_rdma_cid *cid
+ ),
+
+ TP_ARGS(wc, cid),
+
+ TP_STRUCT__entry(
+ __field(u32, cq_id)
+ __field(int, completion_id)
+ __field(u32, received)
+ ),
+
+ TP_fast_assign(
+ __entry->cq_id = cid->ci_queue_id;
+ __entry->completion_id = cid->ci_completion_id;
+ __entry->received = wc->byte_len;
+ ),
+
+ TP_printk("cq.id=%u cid=%d received=%u",
+ __entry->cq_id, __entry->completion_id,
+ __entry->received
+ )
+);
+
+#define DEFINE_RECEIVE_SUCCESS_EVENT(name) \
+ DEFINE_EVENT(rpcrdma_receive_success_class, name, \
+ TP_PROTO( \
+ const struct ib_wc *wc, \
+ const struct rpc_rdma_cid *cid \
+ ), \
+ TP_ARGS(wc, cid))
+
+DECLARE_EVENT_CLASS(rpcrdma_receive_flush_class,
+ TP_PROTO(
+ const struct ib_wc *wc,
+ const struct rpc_rdma_cid *cid
+ ),
+
+ TP_ARGS(wc, cid),
+
+ TP_STRUCT__entry(
+ __field(u32, cq_id)
+ __field(int, completion_id)
+ __field(unsigned long, status)
+ __field(unsigned int, vendor_err)
+ ),
+
+ TP_fast_assign(
+ __entry->cq_id = cid->ci_queue_id;
+ __entry->completion_id = cid->ci_completion_id;
+ __entry->status = wc->status;
+ __entry->vendor_err = wc->vendor_err;
+ ),
+
+ TP_printk("cq.id=%u cid=%d status=%s (%lu/0x%x)",
+ __entry->cq_id, __entry->completion_id,
+ rdma_show_wc_status(__entry->status),
+ __entry->status, __entry->vendor_err
+ )
+);
+
+#define DEFINE_RECEIVE_FLUSH_EVENT(name) \
+ DEFINE_EVENT(rpcrdma_receive_flush_class, name, \
+ TP_PROTO( \
+ const struct ib_wc *wc, \
+ const struct rpc_rdma_cid *cid \
+ ), \
+ TP_ARGS(wc, cid))
+
DECLARE_EVENT_CLASS(xprtrdma_reply_class,
TP_PROTO(
const struct rpcrdma_rep *rep
__entry->nsegs = nsegs;
),
- TP_printk("task:%u@%u pos=%u %u@0x%016llx:0x%08x (%s)",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " pos=%u %u@0x%016llx:0x%08x (%s)",
__entry->task_id, __entry->client_id,
__entry->pos, __entry->length,
(unsigned long long)__entry->offset, __entry->handle,
__entry->nsegs = nsegs;
),
- TP_printk("task:%u@%u %u@0x%016llx:0x%08x (%s)",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " %u@0x%016llx:0x%08x (%s)",
__entry->task_id, __entry->client_id,
__entry->length, (unsigned long long)__entry->offset,
__entry->handle,
TP_fast_assign(
const struct rpcrdma_req *req = mr->mr_req;
- const struct rpc_task *task = req->rl_slot.rq_task;
- __entry->task_id = task->tk_pid;
- __entry->client_id = task->tk_client->cl_clid;
+ if (req) {
+ const struct rpc_task *task = req->rl_slot.rq_task;
+
+ __entry->task_id = task->tk_pid;
+ __entry->client_id = task->tk_client->cl_clid;
+ } else {
+ __entry->task_id = 0;
+ __entry->client_id = -1;
+ }
__entry->mr_id = mr->mr_ibmr->res.id;
__entry->nents = mr->mr_nents;
__entry->handle = mr->mr_handle;
__entry->dir = mr->mr_dir;
),
- TP_printk("task:%u@%u mr.id=%u nents=%d %u@0x%016llx:0x%08x (%s)",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " mr.id=%u nents=%d %u@0x%016llx:0x%08x (%s)",
__entry->task_id, __entry->client_id,
__entry->mr_id, __entry->nents, __entry->length,
(unsigned long long)__entry->offset, __entry->handle,
__assign_str(port, rpcrdma_portstr(r_xprt));
),
- TP_printk("peer=[%s]:%s task:%u@%u",
- __get_str(addr), __get_str(port),
- __entry->task_id, __entry->client_id
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " peer=[%s]:%s",
+ __entry->task_id, __entry->client_id,
+ __get_str(addr), __get_str(port)
)
);
DEFINE_RDCH_EVENT(read);
DEFINE_WRCH_EVENT(write);
DEFINE_WRCH_EVENT(reply);
+DEFINE_WRCH_EVENT(wp);
TRACE_DEFINE_ENUM(rpcrdma_noch);
TRACE_DEFINE_ENUM(rpcrdma_noch_pullup);
__entry->wtype = wtype;
),
- TP_printk("task:%u@%u xid=0x%08x: hdr=%u xdr=%u/%u/%u %s/%s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x hdr=%u xdr=%u/%u/%u %s/%s",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->hdrlen,
__entry->headlen, __entry->pagelen, __entry->taillen,
__entry->ret = ret;
),
- TP_printk("task:%u@%u xid=0x%08x: ret=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x ret=%d",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->ret
)
__entry->ret = ret;
),
- TP_printk("task:%u@%u xid=0x%08x: ret=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x ret=%d",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->ret
)
__entry->signaled = req->rl_wr.send_flags & IB_SEND_SIGNALED;
),
- TP_printk("task:%u@%u cq.id=%u cid=%d (%d SGE%s) %s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " cq.id=%u cid=%d (%d SGE%s) %s",
__entry->task_id, __entry->client_id,
__entry->cq_id, __entry->completion_id,
__entry->num_sge, (__entry->num_sge == 1 ? "" : "s"),
__entry->rc = rc;
),
- TP_printk("task:%u@%u cq.id=%u rc=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " cq.id=%u rc=%d",
__entry->task_id, __entry->client_id,
__entry->cq_id, __entry->rc
)
__entry->status = status;
),
- TP_printk("task:%u@%u status=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " status=%d",
__entry->task_id, __entry->client_id, __entry->status
)
);
__entry->credits = credits;
),
- TP_printk("task:%u@%u xid=0x%08x credits=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x credits=%u",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->credits
)
__entry->max = be32_to_cpup(max);
),
- TP_printk("task:%u@%u xid=0x%08x versions=[%u, %u]",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x versions=[%u, %u]",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->min, __entry->max
)
__entry->xid = be32_to_cpu(rqst->rq_xid);
),
- TP_printk("task:%u@%u xid=0x%08x",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x",
__entry->task_id, __entry->client_id, __entry->xid
)
);
__entry->procedure = be32_to_cpup(procedure);
),
- TP_printk("task:%u@%u xid=0x%08x procedure=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x procedure=%u",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->procedure
)
__entry->taillen = rqst->rq_rcv_buf.tail[0].iov_len;
),
- TP_printk("task:%u@%u fixup=%lu xdr=%zu/%u/%zu",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " fixup=%lu xdr=%zu/%u/%zu",
__entry->task_id, __entry->client_id, __entry->fixup,
__entry->headlen, __entry->pagelen, __entry->taillen
)
__entry->client_id = task->tk_client->cl_clid;
),
- TP_printk("task:%u@%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER,
__entry->task_id, __entry->client_id
)
);
)
);
-DEFINE_COMPLETION_EVENT(svcrdma_wc_send);
+DEFINE_SEND_COMPLETION_EVENT(svcrdma_wc_send);
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_send_flush);
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_send_err);
TRACE_EVENT(svcrdma_post_recv,
TP_PROTO(
)
);
-DEFINE_RECEIVE_COMPLETION_EVENT(svcrdma_wc_receive);
+DEFINE_RECEIVE_SUCCESS_EVENT(svcrdma_wc_recv);
+DEFINE_RECEIVE_FLUSH_EVENT(svcrdma_wc_recv_flush);
+DEFINE_RECEIVE_FLUSH_EVENT(svcrdma_wc_recv_err);
TRACE_EVENT(svcrdma_rq_post_err,
TP_PROTO(
DEFINE_POST_CHUNK_EVENT(write);
DEFINE_POST_CHUNK_EVENT(reply);
-DEFINE_COMPLETION_EVENT(svcrdma_wc_read);
-DEFINE_COMPLETION_EVENT(svcrdma_wc_write);
+TRACE_EVENT(svcrdma_wc_read,
+ TP_PROTO(
+ const struct ib_wc *wc,
+ const struct rpc_rdma_cid *cid,
+ unsigned int totalbytes,
+ const ktime_t posttime
+ ),
+
+ TP_ARGS(wc, cid, totalbytes, posttime),
+
+ TP_STRUCT__entry(
+ __field(u32, cq_id)
+ __field(int, completion_id)
+ __field(s64, read_latency)
+ __field(unsigned int, totalbytes)
+ ),
+
+ TP_fast_assign(
+ __entry->cq_id = cid->ci_queue_id;
+ __entry->completion_id = cid->ci_completion_id;
+ __entry->totalbytes = totalbytes;
+ __entry->read_latency = ktime_us_delta(ktime_get(), posttime);
+ ),
+
+ TP_printk("cq.id=%u cid=%d totalbytes=%u latency-us=%lld",
+ __entry->cq_id, __entry->completion_id,
+ __entry->totalbytes, __entry->read_latency
+ )
+);
+
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_read_flush);
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_read_err);
+
+DEFINE_SEND_COMPLETION_EVENT(svcrdma_wc_write);
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_write_flush);
+DEFINE_SEND_FLUSH_EVENT(svcrdma_wc_write_err);
TRACE_EVENT(svcrdma_qp_error,
TP_PROTO(
#include <linux/net.h>
#include <linux/tracepoint.h>
+#include <trace/events/sunrpc_base.h>
+
TRACE_DEFINE_ENUM(SOCK_STREAM);
TRACE_DEFINE_ENUM(SOCK_DGRAM);
TRACE_DEFINE_ENUM(SOCK_RAW);
__field(size_t, head_len)
__field(const void *, tail_base)
__field(size_t, tail_len)
+ __field(unsigned int, page_base)
__field(unsigned int, page_len)
__field(unsigned int, msg_len)
),
__entry->head_len = xdr->head[0].iov_len;
__entry->tail_base = xdr->tail[0].iov_base;
__entry->tail_len = xdr->tail[0].iov_len;
+ __entry->page_base = xdr->page_base;
__entry->page_len = xdr->page_len;
__entry->msg_len = xdr->len;
),
- TP_printk("task:%u@%u head=[%p,%zu] page=%u tail=[%p,%zu] len=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " head=[%p,%zu] page=%u(%u) tail=[%p,%zu] len=%u",
__entry->task_id, __entry->client_id,
- __entry->head_base, __entry->head_len, __entry->page_len,
- __entry->tail_base, __entry->tail_len, __entry->msg_len
+ __entry->head_base, __entry->head_len,
+ __entry->page_len, __entry->page_base,
+ __entry->tail_base, __entry->tail_len,
+ __entry->msg_len
)
);
__entry->client_id = clnt->cl_clid;
),
- TP_printk("clid=%u", __entry->client_id)
+ TP_printk("client=" SUNRPC_TRACE_CLID_SPECIFIER, __entry->client_id)
);
#define DEFINE_RPC_CLNT_EVENT(name) \
__assign_str(server, server);
),
- TP_printk("client=%u peer=[%s]:%s program=%s server=%s",
+ TP_printk("client=" SUNRPC_TRACE_CLID_SPECIFIER
+ " peer=[%s]:%s program=%s server=%s",
__entry->client_id, __get_str(addr), __get_str(port),
__get_str(program), __get_str(server))
);
__entry->error = error;
),
- TP_printk("client=%u error=%d", __entry->client_id, __entry->error)
+ TP_printk("client=" SUNRPC_TRACE_CLID_SPECIFIER " error=%d",
+ __entry->client_id, __entry->error)
);
__entry->status = task->tk_status;
),
- TP_printk("task:%u@%u status=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " status=%d",
__entry->task_id, __entry->client_id,
__entry->status)
);
__assign_str(procname, rpc_proc_name(task));
),
- TP_printk("task:%u@%u %sv%d %s (%ssync)",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " %sv%d %s (%ssync)",
__entry->task_id, __entry->client_id,
__get_str(progname), __entry->version,
__get_str(procname), __entry->async ? "a": ""
__entry->flags = task->tk_flags;
),
- TP_printk("task:%u@%d flags=%s runstate=%s status=%d action=%ps",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " flags=%s runstate=%s status=%d action=%ps",
__entry->task_id, __entry->client_id,
rpc_show_task_flags(__entry->flags),
rpc_show_runstate(__entry->runstate),
DEFINE_RPC_RUNNING_EVENT(timeout);
DEFINE_RPC_RUNNING_EVENT(signalled);
DEFINE_RPC_RUNNING_EVENT(end);
+DEFINE_RPC_RUNNING_EVENT(call_done);
DECLARE_EVENT_CLASS(rpc_task_queued,
__assign_str(q_name, rpc_qname(q));
),
- TP_printk("task:%u@%d flags=%s runstate=%s status=%d timeout=%lu queue=%s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " flags=%s runstate=%s status=%d timeout=%lu queue=%s",
__entry->task_id, __entry->client_id,
rpc_show_task_flags(__entry->flags),
rpc_show_runstate(__entry->runstate),
__entry->client_id = task->tk_client->cl_clid;
),
- TP_printk("task:%u@%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER,
__entry->task_id, __entry->client_id)
);
__assign_str(servername, task->tk_xprt->servername);
),
- TP_printk("task:%u@%d server=%s xid=0x%08x %sv%d %s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " server=%s xid=0x%08x %sv%d %s",
__entry->task_id, __entry->client_id, __get_str(servername),
__entry->xid, __get_str(progname), __entry->version,
__get_str(procname))
__entry->status = status;
),
- TP_printk("task:%u@%u callsize=%zu recvsize=%zu status=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " callsize=%zu recvsize=%zu status=%d",
__entry->task_id, __entry->client_id,
__entry->callsize, __entry->recvsize, __entry->status
)
__entry->rpc_status = rpc_status;
),
- TP_printk("task:%u@%u tk_status=%d rpc_status=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " tk_status=%d rpc_status=%d",
__entry->task_id, __entry->client_id,
__entry->tk_status, __entry->rpc_status)
);
__entry->execute = ktime_to_us(execute);
),
- TP_printk("task:%u@%d xid=0x%08x %sv%d %s backlog=%lu rtt=%lu execute=%lu",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x %sv%d %s backlog=%lu rtt=%lu execute=%lu",
__entry->task_id, __entry->client_id, __entry->xid,
__get_str(progname), __entry->version, __get_str(procname),
__entry->backlog, __entry->rtt, __entry->execute)
__entry->version = task->tk_client->cl_vers;
__assign_str(procedure, task->tk_msg.rpc_proc->p_name);
} else {
- __entry->task_id = 0;
- __entry->client_id = 0;
+ __entry->task_id = -1;
+ __entry->client_id = -1;
__assign_str(progname, "unknown");
__entry->version = 0;
__assign_str(procedure, "unknown");
__entry->len = xdr->buf->len;
),
- TP_printk(
- "task:%u@%u %sv%d %s requested=%zu p=%p end=%p xdr=[%p,%zu]/%u/[%p,%zu]/%u\n",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " %sv%d %s requested=%zu p=%p end=%p xdr=[%p,%zu]/%u/[%p,%zu]/%u\n",
__entry->task_id, __entry->client_id,
__get_str(progname), __entry->version, __get_str(procedure),
__entry->requested, __entry->p, __entry->end,
__entry->len = xdr->buf->len;
),
- TP_printk(
- "task:%u@%u %sv%d %s offset=%zu copied=%u xdr=[%p,%zu]/%u/[%p,%zu]/%u\n",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " %sv%d %s offset=%zu copied=%u xdr=[%p,%zu]/%u/[%p,%zu]/%u\n",
__entry->task_id, __entry->client_id,
__get_str(progname), __entry->version, __get_str(procedure),
__entry->offset, __entry->copied,
__entry->remaining = rqst->rq_slen - transport->xmit.offset;
),
- TP_printk("task:%u@%u total=%u remaining=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " total=%u remaining=%u",
__entry->task_id, __entry->client_id,
__entry->total, __entry->remaining
)
#define rpc_show_xprt_state(x) \
__print_flags(x, "|", \
- { (1UL << XPRT_LOCKED), "LOCKED"}, \
- { (1UL << XPRT_CONNECTED), "CONNECTED"}, \
- { (1UL << XPRT_CONNECTING), "CONNECTING"}, \
- { (1UL << XPRT_CLOSE_WAIT), "CLOSE_WAIT"}, \
- { (1UL << XPRT_BOUND), "BOUND"}, \
- { (1UL << XPRT_BINDING), "BINDING"}, \
- { (1UL << XPRT_CLOSING), "CLOSING"}, \
- { (1UL << XPRT_OFFLINE), "OFFLINE"}, \
- { (1UL << XPRT_REMOVE), "REMOVE"}, \
- { (1UL << XPRT_CONGESTED), "CONGESTED"}, \
- { (1UL << XPRT_CWND_WAIT), "CWND_WAIT"}, \
- { (1UL << XPRT_WRITE_SPACE), "WRITE_SPACE"})
+ { BIT(XPRT_LOCKED), "LOCKED" }, \
+ { BIT(XPRT_CONNECTED), "CONNECTED" }, \
+ { BIT(XPRT_CONNECTING), "CONNECTING" }, \
+ { BIT(XPRT_CLOSE_WAIT), "CLOSE_WAIT" }, \
+ { BIT(XPRT_BOUND), "BOUND" }, \
+ { BIT(XPRT_BINDING), "BINDING" }, \
+ { BIT(XPRT_CLOSING), "CLOSING" }, \
+ { BIT(XPRT_OFFLINE), "OFFLINE" }, \
+ { BIT(XPRT_REMOVE), "REMOVE" }, \
+ { BIT(XPRT_CONGESTED), "CONGESTED" }, \
+ { BIT(XPRT_CWND_WAIT), "CWND_WAIT" }, \
+ { BIT(XPRT_WRITE_SPACE), "WRITE_SPACE" })
DECLARE_EVENT_CLASS(rpc_xprt_lifetime_class,
TP_PROTO(
__entry->status = status;
),
- TP_printk(
- "task:%u@%u xid=0x%08x seqno=%u status=%d",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x seqno=%u status=%d",
__entry->task_id, __entry->client_id, __entry->xid,
__entry->seqno, __entry->status)
);
__assign_str(procname, rpc_proc_name(task));
),
- TP_printk(
- "task:%u@%u xid=0x%08x %sv%d %s ntrans=%d timeout=%lu",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " xid=0x%08x %sv%d %s ntrans=%d timeout=%lu",
__entry->task_id, __entry->client_id, __entry->xid,
__get_str(progname), __entry->version, __get_str(procname),
__entry->ntrans, __entry->timeout
xprt->snd_task->tk_pid : -1;
),
- TP_printk("task:%u@%u snd_task:%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " snd_task:" SUNRPC_TRACE_PID_SPECIFIER,
__entry->task_id, __entry->client_id,
__entry->snd_task_id)
);
__entry->wait = test_bit(XPRT_CWND_WAIT, &xprt->state);
),
- TP_printk("task:%u@%u snd_task:%u cong=%lu cwnd=%lu%s",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " snd_task:" SUNRPC_TRACE_PID_SPECIFIER
+ " cong=%lu cwnd=%lu%s",
__entry->task_id, __entry->client_id,
__entry->snd_task_id, __entry->cong, __entry->cwnd,
__entry->wait ? " (wait)" : "")
__entry->xid = be32_to_cpu(rqst->rq_xid);
),
- TP_printk("task:%u@%u xid=0x%08x",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " xid=0x%08x",
__entry->task_id, __entry->client_id, __entry->xid
)
);
__assign_str(servername, task->tk_xprt->servername);
),
- TP_printk("task:%u@%u server=%s program=%u version=%u protocol=%d bind_version=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER
+ " server=%s program=%u version=%u protocol=%d bind_version=%u",
__entry->task_id, __entry->client_id, __get_str(servername),
__entry->program, __entry->version, __entry->protocol,
__entry->bind_version
__entry->port = port;
),
- TP_printk("task:%u@%u status=%d port=%u",
+ TP_printk(SUNRPC_TRACE_TASK_SPECIFIER " status=%d port=%u",
__entry->task_id, __entry->client_id,
__entry->status, __entry->port
)
__field(size_t, head_len)
__field(const void *, tail_base)
__field(size_t, tail_len)
+ __field(unsigned int, page_base)
__field(unsigned int, page_len)
__field(unsigned int, msg_len)
),
__entry->head_len = xdr->head[0].iov_len;
__entry->tail_base = xdr->tail[0].iov_base;
__entry->tail_len = xdr->tail[0].iov_len;
+ __entry->page_base = xdr->page_base;
__entry->page_len = xdr->page_len;
__entry->msg_len = xdr->len;
),
- TP_printk("xid=0x%08x head=[%p,%zu] page=%u tail=[%p,%zu] len=%u",
+ TP_printk("xid=0x%08x head=[%p,%zu] page=%u(%u) tail=[%p,%zu] len=%u",
__entry->xid,
- __entry->head_base, __entry->head_len, __entry->page_len,
- __entry->tail_base, __entry->tail_len, __entry->msg_len
+ __entry->head_base, __entry->head_len,
+ __entry->page_len, __entry->page_base,
+ __entry->tail_base, __entry->tail_len,
+ __entry->msg_len
)
);
TP_printk("pid=%d", __entry->pid)
);
+TRACE_EVENT(svc_alloc_arg_err,
+ TP_PROTO(
+ unsigned int pages
+ ),
+
+ TP_ARGS(pages),
+
+ TP_STRUCT__entry(
+ __field(unsigned int, pages)
+ ),
+
+ TP_fast_assign(
+ __entry->pages = pages;
+ ),
+
+ TP_printk("pages=%u", __entry->pages)
+);
+
TRACE_EVENT(svc_handle_xprt,
TP_PROTO(struct svc_xprt *xprt, int len),
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2021 Oracle and/or its affiliates.
+ *
+ * Common types and format specifiers for sunrpc.
+ */
+
+#if !defined(_TRACE_SUNRPC_BASE_H)
+#define _TRACE_SUNRPC_BASE_H
+
+#include <linux/tracepoint.h>
+
+#define SUNRPC_TRACE_PID_SPECIFIER "%08x"
+#define SUNRPC_TRACE_CLID_SPECIFIER "%08x"
+#define SUNRPC_TRACE_TASK_SPECIFIER \
+ "task:" SUNRPC_TRACE_PID_SPECIFIER "@" SUNRPC_TRACE_CLID_SPECIFIER
+
+#endif /* _TRACE_SUNRPC_BASE_H */
{RECLAIM_WB_ASYNC, "RECLAIM_WB_ASYNC"} \
) : "RECLAIM_WB_NONE"
+#define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK)
+#define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED)
+#define _VMSCAN_THROTTLE_NOPROGRESS (1 << VMSCAN_THROTTLE_NOPROGRESS)
+
+#define show_throttle_flags(flags) \
+ (flags) ? __print_flags(flags, "|", \
+ {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \
+ {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"}, \
+ {_VMSCAN_THROTTLE_NOPROGRESS, "VMSCAN_THROTTLE_NOPROGRESS"} \
+ ) : "VMSCAN_THROTTLE_NONE"
+
+
#define trace_reclaim_flags(file) ( \
(file ? RECLAIM_WB_FILE : RECLAIM_WB_ANON) | \
(RECLAIM_WB_ASYNC) \
TP_ARGS(nr_reclaimed)
);
+TRACE_EVENT(mm_vmscan_throttled,
+
+ TP_PROTO(int nid, int usec_timeout, int usec_delayed, int reason),
+
+ TP_ARGS(nid, usec_timeout, usec_delayed, reason),
+
+ TP_STRUCT__entry(
+ __field(int, nid)
+ __field(int, usec_timeout)
+ __field(int, usec_delayed)
+ __field(int, reason)
+ ),
+
+ TP_fast_assign(
+ __entry->nid = nid;
+ __entry->usec_timeout = usec_timeout;
+ __entry->usec_delayed = usec_delayed;
+ __entry->reason = 1U << reason;
+ ),
+
+ TP_printk("nid=%d usec_timeout=%d usect_delayed=%d reason=%s",
+ __entry->nid,
+ __entry->usec_timeout,
+ __entry->usec_delayed,
+ show_throttle_flags(__entry->reason))
+);
#endif /* _TRACE_VMSCAN_H */
/* This part must be outside protection */
TP_ARGS(usec_timeout, usec_delayed)
);
-DEFINE_EVENT(writeback_congest_waited_template, writeback_wait_iff_congested,
-
- TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
-
- TP_ARGS(usec_timeout, usec_delayed)
-);
-
DECLARE_EVENT_CLASS(writeback_single_inode_template,
TP_PROTO(struct inode *inode,
#define SA_UNSUPPORTED 0x00000400
#define SA_EXPOSE_TAGBITS 0x00000800
/* 0x00010000 used on mips */
+/* 0x00800000 used for internal SA_IMMUTABLE */
/* 0x01000000 used on x86 */
/* 0x02000000 used on x86 */
/*
#define AUDIT_EVENT_LISTENER 1335 /* Task joined multicast read socket */
#define AUDIT_URINGOP 1336 /* io_uring operation */
#define AUDIT_OPENAT2 1337 /* Record showing openat2 how args */
+#define AUDIT_DM_CTRL 1338 /* Device Mapper target control */
+#define AUDIT_DM_EVENT 1339 /* Device Mapper events */
#define AUDIT_AVC 1400 /* SE Linux avc denial or grant */
#define AUDIT_SELINUX_ERR 1401 /* Internal SE Linux Errors */
ETHTOOL_A_PAUSE_STAT_TX_FRAMES,
ETHTOOL_A_PAUSE_STAT_RX_FRAMES,
- /* add new constants above here */
+ /* add new constants above here
+ * adjust ETHTOOL_PAUSE_STAT_CNT if adding non-stats!
+ */
__ETHTOOL_A_PAUSE_STAT_CNT,
ETHTOOL_A_PAUSE_STAT_MAX = (__ETHTOOL_A_PAUSE_STAT_CNT - 1)
};
#define FAN_OPEN_EXEC 0x00001000 /* File was opened for exec */
#define FAN_Q_OVERFLOW 0x00004000 /* Event queued overflowed */
+#define FAN_FS_ERROR 0x00008000 /* Filesystem error */
#define FAN_OPEN_PERM 0x00010000 /* File open in perm check */
#define FAN_ACCESS_PERM 0x00020000 /* File accessed in perm check */
#define FAN_EVENT_INFO_TYPE_DFID_NAME 2
#define FAN_EVENT_INFO_TYPE_DFID 3
#define FAN_EVENT_INFO_TYPE_PIDFD 4
+#define FAN_EVENT_INFO_TYPE_ERROR 5
/* Variable length info record following event metadata */
struct fanotify_event_info_header {
__s32 pidfd;
};
+struct fanotify_event_info_error {
+ struct fanotify_event_info_header hdr;
+ __s32 error;
+ __u32 error_count;
+};
+
struct fanotify_response {
__s32 fd;
__u32 response;
*
* 7.34
* - add FUSE_SYNCFS
+ *
+ * 7.35
+ * - add FOPEN_NOFLUSH
*/
#ifndef _LINUX_FUSE_H
#define FUSE_KERNEL_VERSION 7
/** Minor version number of this interface */
-#define FUSE_KERNEL_MINOR_VERSION 34
+#define FUSE_KERNEL_MINOR_VERSION 35
/** The node ID of the root inode */
#define FUSE_ROOT_ID 1
* FOPEN_NONSEEKABLE: the file is not seekable
* FOPEN_CACHE_DIR: allow caching this directory
* FOPEN_STREAM: the file is stream-like (no file position at all)
+ * FOPEN_NOFLUSH: don't flush data cache on close (unless FUSE_WRITEBACK_CACHE)
*/
#define FOPEN_DIRECT_IO (1 << 0)
#define FOPEN_KEEP_CACHE (1 << 1)
#define FOPEN_NONSEEKABLE (1 << 2)
#define FOPEN_CACHE_DIR (1 << 3)
#define FOPEN_STREAM (1 << 4)
+#define FOPEN_NOFLUSH (1 << 5)
/**
* INIT request/reply flags
#define KVM_CAP_BINARY_STATS_FD 203
#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204
#define KVM_CAP_ARM_MTE 205
+#define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206
#ifdef KVM_CAP_IRQ_ROUTING
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2021 Glider bv
+ *
+ * Based on include/uapi/linux/map_to_7segment.h:
+
+ * Copyright (c) 2005 Henk Vergonet <Henk.Vergonet@gmail.com>
+ */
+
+#ifndef MAP_TO_14SEGMENT_H
+#define MAP_TO_14SEGMENT_H
+
+/* This file provides translation primitives and tables for the conversion
+ * of (ASCII) characters to a 14-segments notation.
+ *
+ * The 14 segment's wikipedia notation below is used as standard.
+ * See: https://en.wikipedia.org/wiki/Fourteen-segment_display
+ *
+ * Notation: +---a---+
+ * |\ | /|
+ * f h i j b
+ * | \|/ |
+ * +-g1+-g2+
+ * | /|\ |
+ * e k l m c
+ * |/ | \|
+ * +---d---+
+ *
+ * Usage:
+ *
+ * Register a map variable, and fill it with a character set:
+ * static SEG14_DEFAULT_MAP(map_seg14);
+ *
+ *
+ * Then use for conversion:
+ * seg14 = map_to_seg14(&map_seg14, some_char);
+ * ...
+ *
+ * In device drivers it is recommended, if required, to make the char map
+ * accessible via the sysfs interface using the following scheme:
+ *
+ * static ssize_t map_seg14_show(struct device *dev,
+ * struct device_attribute *attr, char *buf)
+ * {
+ * memcpy(buf, &map_seg14, sizeof(map_seg14));
+ * return sizeof(map_seg14);
+ * }
+ * static ssize_t map_seg14_store(struct device *dev,
+ * struct device_attribute *attr,
+ * const char *buf, size_t cnt)
+ * {
+ * if (cnt != sizeof(map_seg14))
+ * return -EINVAL;
+ * memcpy(&map_seg14, buf, cnt);
+ * return cnt;
+ * }
+ * static DEVICE_ATTR_RW(map_seg14);
+ */
+#include <linux/errno.h>
+#include <linux/types.h>
+
+#include <asm/byteorder.h>
+
+#define BIT_SEG14_A 0
+#define BIT_SEG14_B 1
+#define BIT_SEG14_C 2
+#define BIT_SEG14_D 3
+#define BIT_SEG14_E 4
+#define BIT_SEG14_F 5
+#define BIT_SEG14_G1 6
+#define BIT_SEG14_G2 7
+#define BIT_SEG14_H 8
+#define BIT_SEG14_I 9
+#define BIT_SEG14_J 10
+#define BIT_SEG14_K 11
+#define BIT_SEG14_L 12
+#define BIT_SEG14_M 13
+#define BIT_SEG14_RESERVED1 14
+#define BIT_SEG14_RESERVED2 15
+
+struct seg14_conversion_map {
+ __be16 table[128];
+};
+
+static __inline__ int map_to_seg14(struct seg14_conversion_map *map, int c)
+{
+ if (c < 0 || c >= sizeof(map->table) / sizeof(map->table[0]))
+ return -EINVAL;
+
+ return __be16_to_cpu(map->table[c]);
+}
+
+#define SEG14_CONVERSION_MAP(_name, _map) \
+ struct seg14_conversion_map _name = { .table = { _map } }
+
+/*
+ * It is recommended to use a facility that allows user space to redefine
+ * custom character sets for LCD devices. Please use a sysfs interface
+ * as described above.
+ */
+#define MAP_TO_SEG14_SYSFS_FILE "map_seg14"
+
+/*******************************************************************************
+ * ASCII conversion table
+ ******************************************************************************/
+
+#define _SEG14(sym, a, b, c, d, e, f, g1, g2, h, j, k, l, m, n) \
+ __cpu_to_be16( a << BIT_SEG14_A | b << BIT_SEG14_B | \
+ c << BIT_SEG14_C | d << BIT_SEG14_D | \
+ e << BIT_SEG14_E | f << BIT_SEG14_F | \
+ g1 << BIT_SEG14_G1 | g2 << BIT_SEG14_G2 | \
+ h << BIT_SEG14_H | j << BIT_SEG14_I | \
+ k << BIT_SEG14_J | l << BIT_SEG14_K | \
+ m << BIT_SEG14_L | n << BIT_SEG14_M )
+
+#define _MAP_0_32_ASCII_SEG14_NON_PRINTABLE \
+ 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
+
+#define _MAP_33_47_ASCII_SEG14_SYMBOL \
+ _SEG14('!', 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('"', 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0), \
+ _SEG14('#', 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0), \
+ _SEG14('$', 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0), \
+ _SEG14('%', 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0), \
+ _SEG14('&', 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1), \
+ _SEG14('\'',0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0), \
+ _SEG14('(', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1), \
+ _SEG14(')', 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0), \
+ _SEG14('*', 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1), \
+ _SEG14('+', 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0), \
+ _SEG14(',', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0), \
+ _SEG14('-', 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('.', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1), \
+ _SEG14('/', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0),
+
+#define _MAP_48_57_ASCII_SEG14_NUMERIC \
+ _SEG14('0', 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0), \
+ _SEG14('1', 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0), \
+ _SEG14('2', 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('3', 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('4', 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('5', 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1), \
+ _SEG14('6', 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('7', 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0), \
+ _SEG14('8', 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('9', 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0),
+
+#define _MAP_58_64_ASCII_SEG14_SYMBOL \
+ _SEG14(':', 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0), \
+ _SEG14(';', 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0), \
+ _SEG14('<', 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1), \
+ _SEG14('=', 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('>', 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0), \
+ _SEG14('?', 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0), \
+ _SEG14('@', 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0),
+
+#define _MAP_65_90_ASCII_SEG14_ALPHA_UPPER \
+ _SEG14('A', 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('B', 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0), \
+ _SEG14('C', 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('D', 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0), \
+ _SEG14('E', 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('F', 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('G', 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('H', 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('I', 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0), \
+ _SEG14('J', 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('K', 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1), \
+ _SEG14('L', 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('M', 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0), \
+ _SEG14('N', 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1), \
+ _SEG14('O', 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('P', 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('Q', 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1), \
+ _SEG14('R', 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1), \
+ _SEG14('S', 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('T', 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0), \
+ _SEG14('U', 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('V', 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0), \
+ _SEG14('W', 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1), \
+ _SEG14('X', 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1), \
+ _SEG14('Y', 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0), \
+ _SEG14('Z', 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0),
+
+#define _MAP_91_96_ASCII_SEG14_SYMBOL \
+ _SEG14('[', 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('\\',0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1), \
+ _SEG14(']', 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('^', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1), \
+ _SEG14('_', 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('`', 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0),
+
+#define _MAP_97_122_ASCII_SEG14_ALPHA_LOWER \
+ _SEG14('a', 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0), \
+ _SEG14('b', 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1), \
+ _SEG14('c', 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('d', 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0), \
+ _SEG14('e', 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0), \
+ _SEG14('f', 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0), \
+ _SEG14('g', 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0), \
+ _SEG14('h', 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0), \
+ _SEG14('i', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0), \
+ _SEG14('j', 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0), \
+ _SEG14('k', 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1), \
+ _SEG14('l', 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('m', 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0), \
+ _SEG14('n', 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0), \
+ _SEG14('o', 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0), \
+ _SEG14('p', 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0), \
+ _SEG14('q', 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0), \
+ _SEG14('r', 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('s', 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1), \
+ _SEG14('t', 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('u', 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0), \
+ _SEG14('v', 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0), \
+ _SEG14('w', 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1), \
+ _SEG14('x', 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1), \
+ _SEG14('y', 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0), \
+ _SEG14('z', 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0),
+
+#define _MAP_123_126_ASCII_SEG14_SYMBOL \
+ _SEG14('{', 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0), \
+ _SEG14('|', 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0), \
+ _SEG14('}', 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1), \
+ _SEG14('~', 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0),
+
+/* Maps */
+#define MAP_ASCII14SEG_ALPHANUM \
+ _MAP_0_32_ASCII_SEG14_NON_PRINTABLE \
+ _MAP_33_47_ASCII_SEG14_SYMBOL \
+ _MAP_48_57_ASCII_SEG14_NUMERIC \
+ _MAP_58_64_ASCII_SEG14_SYMBOL \
+ _MAP_65_90_ASCII_SEG14_ALPHA_UPPER \
+ _MAP_91_96_ASCII_SEG14_SYMBOL \
+ _MAP_97_122_ASCII_SEG14_ALPHA_LOWER \
+ _MAP_123_126_ASCII_SEG14_SYMBOL
+
+#define SEG14_DEFAULT_MAP(_name) \
+ SEG14_CONVERSION_MAP(_name, MAP_ASCII14SEG_ALPHANUM)
+
+#endif /* MAP_TO_14SEGMENT_H */
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-/*
- * This file describes the layout of the file handles as passed
- * over the wire.
- *
- * Copyright (C) 1995, 1996, 1997 Olaf Kirch <okir@monad.swb.de>
- */
-
-#ifndef _UAPI_LINUX_NFSD_FH_H
-#define _UAPI_LINUX_NFSD_FH_H
-
-#include <linux/types.h>
-#include <linux/nfs.h>
-#include <linux/nfs2.h>
-#include <linux/nfs3.h>
-#include <linux/nfs4.h>
-
-/*
- * This is the old "dentry style" Linux NFSv2 file handle.
- *
- * The xino and xdev fields are currently used to transport the
- * ino/dev of the exported inode.
- */
-struct nfs_fhbase_old {
- __u32 fb_dcookie; /* dentry cookie - always 0xfeebbaca */
- __u32 fb_ino; /* our inode number */
- __u32 fb_dirino; /* dir inode number, 0 for directories */
- __u32 fb_dev; /* our device */
- __u32 fb_xdev;
- __u32 fb_xino;
- __u32 fb_generation;
-};
-
-/*
- * This is the new flexible, extensible style NFSv2/v3/v4 file handle.
- *
- * The file handle starts with a sequence of four-byte words.
- * The first word contains a version number (1) and three descriptor bytes
- * that tell how the remaining 3 variable length fields should be handled.
- * These three bytes are auth_type, fsid_type and fileid_type.
- *
- * All four-byte values are in host-byte-order.
- *
- * The auth_type field is deprecated and must be set to 0.
- *
- * The fsid_type identifies how the filesystem (or export point) is
- * encoded.
- * Current values:
- * 0 - 4 byte device id (ms-2-bytes major, ls-2-bytes minor), 4byte inode number
- * NOTE: we cannot use the kdev_t device id value, because kdev_t.h
- * says we mustn't. We must break it up and reassemble.
- * 1 - 4 byte user specified identifier
- * 2 - 4 byte major, 4 byte minor, 4 byte inode number - DEPRECATED
- * 3 - 4 byte device id, encoded for user-space, 4 byte inode number
- * 4 - 4 byte inode number and 4 byte uuid
- * 5 - 8 byte uuid
- * 6 - 16 byte uuid
- * 7 - 8 byte inode number and 16 byte uuid
- *
- * The fileid_type identified how the file within the filesystem is encoded.
- * The values for this field are filesystem specific, exccept that
- * filesystems must not use the values '0' or '0xff'. 'See enum fid_type'
- * in include/linux/exportfs.h for currently registered values.
- */
-struct nfs_fhbase_new {
- union {
- struct {
- __u8 fb_version_aux; /* == 1, even => nfs_fhbase_old */
- __u8 fb_auth_type_aux;
- __u8 fb_fsid_type_aux;
- __u8 fb_fileid_type_aux;
- __u32 fb_auth[1];
- /* __u32 fb_fsid[0]; floating */
- /* __u32 fb_fileid[0]; floating */
- };
- struct {
- __u8 fb_version; /* == 1, even => nfs_fhbase_old */
- __u8 fb_auth_type;
- __u8 fb_fsid_type;
- __u8 fb_fileid_type;
- __u32 fb_auth_flex[]; /* flexible-array member */
- };
- };
-};
-
-struct knfsd_fh {
- unsigned int fh_size; /* significant for NFSv3.
- * Points to the current size while building
- * a new file handle
- */
- union {
- struct nfs_fhbase_old fh_old;
- __u32 fh_pad[NFS4_FHSIZE/4];
- struct nfs_fhbase_new fh_new;
- } fh_base;
-};
-
-#define ofh_dcookie fh_base.fh_old.fb_dcookie
-#define ofh_ino fh_base.fh_old.fb_ino
-#define ofh_dirino fh_base.fh_old.fb_dirino
-#define ofh_dev fh_base.fh_old.fb_dev
-#define ofh_xdev fh_base.fh_old.fb_xdev
-#define ofh_xino fh_base.fh_old.fb_xino
-#define ofh_generation fh_base.fh_old.fb_generation
-
-#define fh_version fh_base.fh_new.fb_version
-#define fh_fsid_type fh_base.fh_new.fb_fsid_type
-#define fh_auth_type fh_base.fh_new.fb_auth_type
-#define fh_fileid_type fh_base.fh_new.fb_fileid_type
-#define fh_fsid fh_base.fh_new.fb_auth_flex
-
-/* Do not use, provided for userspace compatiblity. */
-#define fh_auth fh_base.fh_new.fb_auth
-
-#endif /* _UAPI_LINUX_NFSD_FH_H */
#define PCI_EXP_DEVCTL_URRE 0x0008 /* Unsupported Request Reporting En. */
#define PCI_EXP_DEVCTL_RELAX_EN 0x0010 /* Enable relaxed ordering */
#define PCI_EXP_DEVCTL_PAYLOAD 0x00e0 /* Max_Payload_Size */
+#define PCI_EXP_DEVCTL_PAYLOAD_128B 0x0000 /* 128 Bytes */
+#define PCI_EXP_DEVCTL_PAYLOAD_256B 0x0020 /* 256 Bytes */
+#define PCI_EXP_DEVCTL_PAYLOAD_512B 0x0040 /* 512 Bytes */
+#define PCI_EXP_DEVCTL_PAYLOAD_1024B 0x0060 /* 1024 Bytes */
+#define PCI_EXP_DEVCTL_PAYLOAD_2048B 0x0080 /* 2048 Bytes */
+#define PCI_EXP_DEVCTL_PAYLOAD_4096B 0x00a0 /* 4096 Bytes */
#define PCI_EXP_DEVCTL_EXT_TAG 0x0100 /* Extended Tag Field Enable */
#define PCI_EXP_DEVCTL_PHANTOM 0x0200 /* Phantom Functions Enable */
#define PCI_EXP_DEVCTL_AUX_PME 0x0400 /* Auxiliary Power PM Enable */
#define PR_GET_TAGGED_ADDR_CTRL 56
# define PR_TAGGED_ADDR_ENABLE (1UL << 0)
/* MTE tag check fault modes */
-# define PR_MTE_TCF_NONE 0
+# define PR_MTE_TCF_NONE 0UL
# define PR_MTE_TCF_SYNC (1UL << 1)
# define PR_MTE_TCF_ASYNC (1UL << 2)
# define PR_MTE_TCF_MASK (PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC)
# define PR_SCHED_CORE_SHARE_TO 2 /* push core_sched cookie to pid */
# define PR_SCHED_CORE_SHARE_FROM 3 /* pull core_sched cookie to pid */
# define PR_SCHED_CORE_MAX 4
+# define PR_SCHED_CORE_SCOPE_THREAD 0
+# define PR_SCHED_CORE_SCOPE_THREAD_GROUP 1
+# define PR_SCHED_CORE_SCOPE_PROCESS_GROUP 2
#endif /* _LINUX_PRCTL_H */
#include <linux/const.h>
#include <linux/ioctl.h>
+#include <linux/types.h>
/*
* The struct used to pass data via the following ioctl. Similar to the
long pll_clock; /* base PLL frequency */
};
+struct rtc_param {
+ __u64 param;
+ union {
+ __u64 uvalue;
+ __s64 svalue;
+ __u64 ptr;
+ };
+ __u32 index;
+ __u32 __pad;
+};
+
/*
* ioctl calls that are permitted to the /dev/rtc interface, if
* any of the RTC drivers are enabled.
#define RTC_PLL_GET _IOR('p', 0x11, struct rtc_pll_info) /* Get PLL correction */
#define RTC_PLL_SET _IOW('p', 0x12, struct rtc_pll_info) /* Set PLL correction */
+#define RTC_PARAM_GET _IOW('p', 0x13, struct rtc_param) /* Get parameter */
+#define RTC_PARAM_SET _IOW('p', 0x14, struct rtc_param) /* Set parameter */
+
#define RTC_VL_DATA_INVALID _BITUL(0) /* Voltage too low, RTC data is invalid */
#define RTC_VL_BACKUP_LOW _BITUL(1) /* Backup voltage is low */
#define RTC_VL_BACKUP_EMPTY _BITUL(2) /* Backup empty or not present */
#define RTC_FEATURE_ALARM 0
#define RTC_FEATURE_ALARM_RES_MINUTE 1
#define RTC_FEATURE_NEED_WEEK_DAY 2
-#define RTC_FEATURE_CNT 3
+#define RTC_FEATURE_ALARM_RES_2S 3
+#define RTC_FEATURE_UPDATE_INTERRUPT 4
+#define RTC_FEATURE_CORRECTION 5
+#define RTC_FEATURE_BACKUP_SWITCH_MODE 6
+#define RTC_FEATURE_CNT 7
+
+/* parameter list */
+#define RTC_PARAM_FEATURES 0
+#define RTC_PARAM_CORRECTION 1
+#define RTC_PARAM_BACKUP_SWITCH_MODE 2
+
+#define RTC_BSM_DISABLED 0
+#define RTC_BSM_DIRECT 1
+#define RTC_BSM_LEVEL 2
+#define RTC_BSM_STANDBY 3
#define RTC_MAX_FREQ 8192
#include <linux/types.h>
+/* Virtio GPIO Feature bits */
+#define VIRTIO_GPIO_F_IRQ 0
+
/* Virtio GPIO request types */
#define VIRTIO_GPIO_MSG_GET_NAMES 0x0001
#define VIRTIO_GPIO_MSG_GET_DIRECTION 0x0002
#define VIRTIO_GPIO_MSG_SET_DIRECTION 0x0003
#define VIRTIO_GPIO_MSG_GET_VALUE 0x0004
#define VIRTIO_GPIO_MSG_SET_VALUE 0x0005
+#define VIRTIO_GPIO_MSG_IRQ_TYPE 0x0006
/* Possible values of the status field */
#define VIRTIO_GPIO_STATUS_OK 0x0
#define VIRTIO_GPIO_DIRECTION_OUT 0x01
#define VIRTIO_GPIO_DIRECTION_IN 0x02
+/* Virtio GPIO IRQ types */
+#define VIRTIO_GPIO_IRQ_TYPE_NONE 0x00
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_RISING 0x01
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_FALLING 0x02
+#define VIRTIO_GPIO_IRQ_TYPE_EDGE_BOTH 0x03
+#define VIRTIO_GPIO_IRQ_TYPE_LEVEL_HIGH 0x04
+#define VIRTIO_GPIO_IRQ_TYPE_LEVEL_LOW 0x08
+
struct virtio_gpio_config {
__le16 ngpio;
__u8 padding[2];
__le32 gpio_names_size;
-} __packed;
+};
/* Virtio GPIO Request / Response */
struct virtio_gpio_request {
__u8 value[];
};
+/* Virtio GPIO IRQ Request / Response */
+struct virtio_gpio_irq_request {
+ __le16 gpio;
+};
+
+struct virtio_gpio_irq_response {
+ __u8 status;
+};
+
+/* Possible values of the interrupt status field */
+#define VIRTIO_GPIO_IRQ_STATUS_INVALID 0x0
+#define VIRTIO_GPIO_IRQ_STATUS_VALID 0x1
+
#endif /* _LINUX_VIRTIO_GPIO_H */
* explicitly triggered (VIRTIO_MEM_REQ_UNPLUG).
*
* There are no guarantees what will happen if unplugged memory is
- * read/written. Such memory should, in general, not be touched. E.g.,
- * even writing might succeed, but the values will simply be discarded at
- * random points in time.
+ * read/written. In general, unplugged memory should not be touched, because
+ * the resulting action is undefined. There is one exception: without
+ * VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE, unplugged memory inside the usable
+ * region can be read, to simplify creation of memory dumps.
*
* It can happen that the device cannot process a request, because it is
* busy. The device driver has to retry later.
/* node_id is an ACPI PXM and is valid */
#define VIRTIO_MEM_F_ACPI_PXM 0
+/* unplugged memory must not be accessed */
+#define VIRTIO_MEM_F_UNPLUGGED_INACCESSIBLE 1
/* --- virtio-mem: guest -> host requests --- */
int HYPERVISOR_memory_op(unsigned int cmd, void *arg);
int HYPERVISOR_physdev_op(int cmd, void *arg);
int HYPERVISOR_vcpu_op(int cmd, int vcpuid, void *extra_args);
-int HYPERVISOR_tmem_op(void *arg);
int HYPERVISOR_vm_assist(unsigned int cmd, unsigned int type);
int HYPERVISOR_dm_op(domid_t domid, unsigned int nr_bufs,
struct xen_dm_op_buf *bufs);
return HYPERVISOR_sched_op(SCHEDOP_shutdown, &r);
}
-static inline void
-MULTI_update_va_mapping(struct multicall_entry *mcl, unsigned long va,
- unsigned int new_val, unsigned long flags)
-{
- BUG();
-}
-
-static inline void
-MULTI_mmu_update(struct multicall_entry *mcl, struct mmu_update *req,
- int count, int *success_count, domid_t domid)
-{
- BUG();
-}
-
#endif /* _ASM_ARM_XEN_HYPERCALL_H */
void balloon_set_new_target(unsigned long target);
-int alloc_xenballooned_pages(int nr_pages, struct page **pages);
-void free_xenballooned_pages(int nr_pages, struct page **pages);
-
#ifdef CONFIG_XEN_BALLOON
void xen_balloon_init(void);
#else
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* callback.h
*
* Register guest OS callbacks with Xen.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2006, Ian Campbell
*/
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* elfnote.h
*
* Definitions used for the Xen ELF notes.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2006, Ian Campbell, XenSource Ltd.
*/
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* event_channel.h
*
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* features.h
*
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* grant_table.h
*
* Interface for granting foreign access to page frames, and receiving
* page-ownership transfers.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2004, K A Fraser
*/
+/* SPDX-License-Identifier: MIT */
/*
* Copyright (c) 2016, Citrix Systems Inc
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
*/
#ifndef __XEN_PUBLIC_HVM_DM_OP_H__
-/*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- */
+/* SPDX-License-Identifier: MIT */
#ifndef __XEN_PUBLIC_HVM_HVM_OP_H__
#define __XEN_PUBLIC_HVM_HVM_OP_H__
+/* SPDX-License-Identifier: MIT */
/*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2015, Roger Pau Monne <roger.pau@citrix.com>
*/
-/*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- */
+/* SPDX-License-Identifier: MIT */
#ifndef __XEN_PUBLIC_HVM_PARAMS_H__
#define __XEN_PUBLIC_HVM_PARAMS_H__
+/* SPDX-License-Identifier: MIT */
/*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2016, Citrix Systems, Inc.
*/
+/* SPDX-License-Identifier: MIT */
/*
* 9pfs.h -- Xen 9PFS transport
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (C) 2017 Stefano Stabellini <stefano@aporeto.com>
*/
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* blkif.h
*
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* console.h
*
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* displif.h
*
* Unified display device I/O interface for Xen guest OSes.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (C) 2016-2017 EPAM Systems Inc.
*
* Authors: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
+/* SPDX-License-Identifier: MIT */
/*
* fbif.h -- Xen virtual frame buffer device
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (C) 2005 Anthony Liguori <aliguori@us.ibm.com>
* Copyright (C) 2006 Red Hat, Inc., Markus Armbruster <armbru@redhat.com>
*/
+/* SPDX-License-Identifier: MIT */
/*
* kbdif.h -- Xen virtual keyboard/mouse
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (C) 2005 Anthony Liguori <aliguori@us.ibm.com>
* Copyright (C) 2006 Red Hat, Inc., Markus Armbruster <armbru@redhat.com>
*/
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* xen_netif.h
*
* Unified network-device I/O interface for Xen guest OSes.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2003-2004, Keir Fraser
*/
+/* SPDX-License-Identifier: MIT */
/*
* PCI Backend/Frontend Common Data Structures & Macros
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Author: Ryan Wilson <hap9@epoch.ncsc.mil>
*/
#ifndef __XEN_PCI_COMMON_H__
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
#ifndef __XEN_PROTOCOLS_H__
#define __XEN_PROTOCOLS_H__
+/* SPDX-License-Identifier: MIT */
+
#ifndef __XEN_PUBLIC_IO_XEN_PVCALLS_H__
#define __XEN_PUBLIC_IO_XEN_PVCALLS_H__
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* ring.h
*
* Shared producer-consumer ring macros.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Tim Deegan and Andrew Warfield November 2004.
*/
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* sndif.h
*
* Unified sound-device I/O interface for Xen guest OSes.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (C) 2013-2015 GlobalLogic Inc.
* Copyright (C) 2016-2017 EPAM Systems Inc.
*
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* vscsiif.h
*
* Based on the blkif.h code.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright(c) FUJITSU Limited 2008.
*/
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/*****************************************************************************
* xenbus.h
*
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/*
* Details of the "wire" protocol between Xen Store Daemon and client
* library or guest kernel.
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* memory.h
*
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* nmi.h
*
-/*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- */
+/* SPDX-License-Identifier: MIT */
#ifndef __XEN_PUBLIC_PHYSDEV_H__
#define __XEN_PUBLIC_PHYSDEV_H__
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* platform.h
*
* Hardware platform operations. Intended for use by domain-0 kernel.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2002-2006, K Fraser
*/
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* sched.h
*
* Scheduler state interactions
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2005, Keir Fraser <keir@xensource.com>
*/
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* vcpu.h
*
* VCPU initialisation, query, and hotplug.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2005, Keir Fraser <keir@xensource.com>
*/
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* version.h
*
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* arch-x86/mca.h
* Guest OS machine check interface to x86 Xen.
+/* SPDX-License-Identifier: MIT */
/******************************************************************************
* xen.h
*
* Guest OS interface to Xen.
*
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to
- * deal in the Software without restriction, including without limitation the
- * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
- * sell copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
- * DEALINGS IN THE SOFTWARE.
- *
* Copyright (c) 2004, K A Fraser
*/
-/* SPDX-License-Identifier: GPL-2.0 */
+/* SPDX-License-Identifier: MIT */
#ifndef __XEN_PUBLIC_XENPMU_H__
#define __XEN_PUBLIC_XENPMU_H__
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef __XEN_PCI_H__
+#define __XEN_PCI_H__
+
+#if defined(CONFIG_XEN_DOM0)
+int xen_find_device_domain_owner(struct pci_dev *dev);
+int xen_register_device_domain_owner(struct pci_dev *dev, uint16_t domain);
+int xen_unregister_device_domain_owner(struct pci_dev *dev);
+#else
+static inline int xen_find_device_domain_owner(struct pci_dev *dev)
+{
+ return -1;
+}
+
+static inline int xen_register_device_domain_owner(struct pci_dev *dev,
+ uint16_t domain)
+{
+ return -1;
+}
+
+static inline int xen_unregister_device_domain_owner(struct pci_dev *dev)
+{
+ return -1;
+}
+#endif
+
+#endif
extern u64 xen_saved_max_mem_size;
#endif
-#ifdef CONFIG_XEN_UNPOPULATED_ALLOC
int xen_alloc_unpopulated_pages(unsigned int nr_pages, struct page **pages);
void xen_free_unpopulated_pages(unsigned int nr_pages, struct page **pages);
-#else
-#define xen_alloc_unpopulated_pages alloc_xenballooned_pages
-#define xen_free_unpopulated_pages free_xenballooned_pages
-#include <xen/balloon.h>
-#endif
#endif /* _XEN_XEN_H */
config CC_HAS_INT128
def_bool !$(cc-option,$(m64-flag) -D__SIZEOF_INT128__=0) && 64BIT
+config CC_IMPLICIT_FALLTHROUGH
+ string
+ default "-Wimplicit-fallthrough=5" if CC_IS_GCC && $(cc-option,-Wimplicit-fallthrough=5)
+ default "-Wimplicit-fallthrough" if CC_IS_CLANG && $(cc-option,-Wunreachable-code-fallthrough)
+
#
# For architectures that know their GCC __int128 support is sound
#
bool "Memory placement aware NUMA scheduler"
depends on ARCH_SUPPORTS_NUMA_BALANCING
depends on !ARCH_WANT_NUMA_VARIABLE_LOCALITY
- depends on SMP && NUMA && MIGRATION
+ depends on SMP && NUMA && MIGRATION && !PREEMPT_RT
help
This option adds support for automatic NUMA aware memory/task placement.
The mechanism is quite primitive and is based on migrating memory when
config SLAB
bool "SLAB"
+ depends on !PREEMPT_RT
select HAVE_HARDENED_USERCOPY_ALLOCATOR
help
The regular slab allocator that is established and known to work
config SLOB
depends on EXPERT
bool "SLOB (Simple Allocator)"
+ depends on !PREEMPT_RT
help
SLOB replaces the stock allocator with a drastically simpler
allocator. SLOB is generally more space efficient but
quiet_cmd_compile.h = CHK $@
cmd_compile.h = \
$(CONFIG_SHELL) $(srctree)/scripts/mkcompile_h $@ \
- "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT)" \
+ "$(UTS_MACHINE)" "$(CONFIG_SMP)" "$(CONFIG_PREEMPT_BUILD)" \
"$(CONFIG_PREEMPT_RT)" $(CONFIG_CC_VERSION_TEXT) "$(LD)"
include/generated/compile.h: FORCE
unsigned long aligned_start = ALIGN_DOWN(start, PAGE_SIZE);
unsigned long aligned_end = ALIGN(end, PAGE_SIZE);
- memblock_free(__pa(aligned_start), aligned_end - aligned_start);
+ memblock_free((void *)aligned_start, aligned_end - aligned_start);
#endif
free_reserved_area((void *)start, (void *)end, POISON_FREE_INITMEM,
ret = xbc_snprint_cmdline(new_cmdline, len + 1, root);
if (ret < 0 || ret > len) {
pr_err("Failed to print extra kernel cmdline.\n");
- memblock_free_ptr(new_cmdline, len + 1);
+ memblock_free(new_cmdline, len + 1);
return NULL;
}
for (p = &envp_init[2]; *p; p++)
end += sprintf(end, " %s", *p);
- pr_notice("Unknown command line parameters:%s\n", unknown_options);
- memblock_free_ptr(unknown_options, len);
+ /* Start at unknown_options[1] to skip the initial space */
+ pr_notice("Unknown kernel command line parameters \"%s\", will be passed to user space.\n",
+ &unknown_options[1]);
+ memblock_free(unknown_options, len);
}
asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
kernel_init_freeable();
/* need to finish all async __init code before freeing the memory */
async_synchronize_full();
+
+ system_state = SYSTEM_FREEING_INITMEM;
kprobe_free_init_mem();
ftrace_free_init_mem();
kgdb_free_init_mem();
#include <linux/nsproxy.h>
#include <linux/sysctl.h>
#include <linux/uaccess.h>
+#include <linux/capability.h>
#include <linux/ipc_namespace.h>
#include <linux/msg.h>
#include "util.h"
return which;
}
-#ifdef CONFIG_PROC_SYSCTL
static int proc_ipc_dointvec(struct ctl_table *table, int write,
void *buffer, size_t *lenp, loff_t *ppos)
{
return ret;
}
-#else
-#define proc_ipc_doulongvec_minmax NULL
-#define proc_ipc_dointvec NULL
-#define proc_ipc_dointvec_minmax NULL
-#define proc_ipc_dointvec_minmax_orphans NULL
-#define proc_ipc_auto_msgmni NULL
-#define proc_ipc_sem_dointvec NULL
+#ifdef CONFIG_CHECKPOINT_RESTORE
+static int proc_ipc_dointvec_minmax_checkpoint_restore(struct ctl_table *table,
+ int write, void *buffer, size_t *lenp, loff_t *ppos)
+{
+ struct user_namespace *user_ns = current->nsproxy->ipc_ns->user_ns;
+
+ if (write && !checkpoint_restore_ns_capable(user_ns))
+ return -EPERM;
+
+ return proc_ipc_dointvec_minmax(table, write, buffer, lenp, ppos);
+}
#endif
int ipc_mni = IPCMNI;
.procname = "sem_next_id",
.data = &init_ipc_ns.ids[IPC_SEM_IDS].next_id,
.maxlen = sizeof(init_ipc_ns.ids[IPC_SEM_IDS].next_id),
- .mode = 0644,
- .proc_handler = proc_ipc_dointvec_minmax,
+ .mode = 0666,
+ .proc_handler = proc_ipc_dointvec_minmax_checkpoint_restore,
.extra1 = SYSCTL_ZERO,
.extra2 = SYSCTL_INT_MAX,
},
.procname = "msg_next_id",
.data = &init_ipc_ns.ids[IPC_MSG_IDS].next_id,
.maxlen = sizeof(init_ipc_ns.ids[IPC_MSG_IDS].next_id),
- .mode = 0644,
- .proc_handler = proc_ipc_dointvec_minmax,
+ .mode = 0666,
+ .proc_handler = proc_ipc_dointvec_minmax_checkpoint_restore,
.extra1 = SYSCTL_ZERO,
.extra2 = SYSCTL_INT_MAX,
},
.procname = "shm_next_id",
.data = &init_ipc_ns.ids[IPC_SHM_IDS].next_id,
.maxlen = sizeof(init_ipc_ns.ids[IPC_SHM_IDS].next_id),
- .mode = 0644,
- .proc_handler = proc_ipc_dointvec_minmax,
+ .mode = 0666,
+ .proc_handler = proc_ipc_dointvec_minmax_checkpoint_restore,
.extra1 = SYSCTL_ZERO,
.extra2 = SYSCTL_INT_MAX,
},
struct pid *shm_lprid;
struct ucounts *mlock_ucounts;
- /* The task created the shm object. NULL if the task is dead. */
+ /*
+ * The task created the shm object, for
+ * task_lock(shp->shm_creator)
+ */
struct task_struct *shm_creator;
- struct list_head shm_clist; /* list by creator */
+
+ /*
+ * List by creator. task_lock(->shm_creator) required for read/write.
+ * If list_empty(), then the creator is dead already.
+ */
+ struct list_head shm_clist;
+ struct ipc_namespace *ns;
} __randomize_layout;
/* shm_mode upper byte flags */
struct shmid_kernel *shp;
shp = container_of(ipcp, struct shmid_kernel, shm_perm);
+ WARN_ON(ns != shp->ns);
if (shp->shm_nattch) {
shp->shm_perm.mode |= SHM_DEST;
kfree(shp);
}
-static inline void shm_rmid(struct ipc_namespace *ns, struct shmid_kernel *s)
+/*
+ * It has to be called with shp locked.
+ * It must be called before ipc_rmid()
+ */
+static inline void shm_clist_rm(struct shmid_kernel *shp)
{
- list_del(&s->shm_clist);
- ipc_rmid(&shm_ids(ns), &s->shm_perm);
+ struct task_struct *creator;
+
+ /* ensure that shm_creator does not disappear */
+ rcu_read_lock();
+
+ /*
+ * A concurrent exit_shm may do a list_del_init() as well.
+ * Just do nothing if exit_shm already did the work
+ */
+ if (!list_empty(&shp->shm_clist)) {
+ /*
+ * shp->shm_creator is guaranteed to be valid *only*
+ * if shp->shm_clist is not empty.
+ */
+ creator = shp->shm_creator;
+
+ task_lock(creator);
+ /*
+ * list_del_init() is a nop if the entry was already removed
+ * from the list.
+ */
+ list_del_init(&shp->shm_clist);
+ task_unlock(creator);
+ }
+ rcu_read_unlock();
+}
+
+static inline void shm_rmid(struct shmid_kernel *s)
+{
+ shm_clist_rm(s);
+ ipc_rmid(&shm_ids(s->ns), &s->shm_perm);
}
shm_file = shp->shm_file;
shp->shm_file = NULL;
ns->shm_tot -= (shp->shm_segsz + PAGE_SIZE - 1) >> PAGE_SHIFT;
- shm_rmid(ns, shp);
+ shm_rmid(shp);
shm_unlock(shp);
if (!is_file_hugepages(shm_file))
shmem_lock(shm_file, 0, shp->mlock_ucounts);
- else if (shp->mlock_ucounts)
- user_shm_unlock(i_size_read(file_inode(shm_file)),
- shp->mlock_ucounts);
fput(shm_file);
ipc_update_pid(&shp->shm_cprid, NULL);
ipc_update_pid(&shp->shm_lprid, NULL);
*
* 2) sysctl kernel.shm_rmid_forced is set to 1.
*/
-static bool shm_may_destroy(struct ipc_namespace *ns, struct shmid_kernel *shp)
+static bool shm_may_destroy(struct shmid_kernel *shp)
{
return (shp->shm_nattch == 0) &&
- (ns->shm_rmid_forced ||
+ (shp->ns->shm_rmid_forced ||
(shp->shm_perm.mode & SHM_DEST));
}
ipc_update_pid(&shp->shm_lprid, task_tgid(current));
shp->shm_dtim = ktime_get_real_seconds();
shp->shm_nattch--;
- if (shm_may_destroy(ns, shp))
+ if (shm_may_destroy(shp))
shm_destroy(ns, shp);
else
shm_unlock(shp);
*
* As shp->* are changed under rwsem, it's safe to skip shp locking.
*/
- if (shp->shm_creator != NULL)
+ if (!list_empty(&shp->shm_clist))
return 0;
- if (shm_may_destroy(ns, shp)) {
+ if (shm_may_destroy(shp)) {
shm_lock_by_ptr(shp);
shm_destroy(ns, shp);
}
/* Locking assumes this will only be called with task == current */
void exit_shm(struct task_struct *task)
{
- struct ipc_namespace *ns = task->nsproxy->ipc_ns;
- struct shmid_kernel *shp, *n;
+ for (;;) {
+ struct shmid_kernel *shp;
+ struct ipc_namespace *ns;
- if (list_empty(&task->sysvshm.shm_clist))
- return;
+ task_lock(task);
+
+ if (list_empty(&task->sysvshm.shm_clist)) {
+ task_unlock(task);
+ break;
+ }
+
+ shp = list_first_entry(&task->sysvshm.shm_clist, struct shmid_kernel,
+ shm_clist);
- /*
- * If kernel.shm_rmid_forced is not set then only keep track of
- * which shmids are orphaned, so that a later set of the sysctl
- * can clean them up.
- */
- if (!ns->shm_rmid_forced) {
- down_read(&shm_ids(ns).rwsem);
- list_for_each_entry(shp, &task->sysvshm.shm_clist, shm_clist)
- shp->shm_creator = NULL;
/*
- * Only under read lock but we are only called on current
- * so no entry on the list will be shared.
+ * 1) Get pointer to the ipc namespace. It is worth to say
+ * that this pointer is guaranteed to be valid because
+ * shp lifetime is always shorter than namespace lifetime
+ * in which shp lives.
+ * We taken task_lock it means that shp won't be freed.
*/
- list_del(&task->sysvshm.shm_clist);
- up_read(&shm_ids(ns).rwsem);
- return;
- }
+ ns = shp->ns;
- /*
- * Destroy all already created segments, that were not yet mapped,
- * and mark any mapped as orphan to cover the sysctl toggling.
- * Destroy is skipped if shm_may_destroy() returns false.
- */
- down_write(&shm_ids(ns).rwsem);
- list_for_each_entry_safe(shp, n, &task->sysvshm.shm_clist, shm_clist) {
- shp->shm_creator = NULL;
+ /*
+ * 2) If kernel.shm_rmid_forced is not set then only keep track of
+ * which shmids are orphaned, so that a later set of the sysctl
+ * can clean them up.
+ */
+ if (!ns->shm_rmid_forced)
+ goto unlink_continue;
- if (shm_may_destroy(ns, shp)) {
- shm_lock_by_ptr(shp);
- shm_destroy(ns, shp);
+ /*
+ * 3) get a reference to the namespace.
+ * The refcount could be already 0. If it is 0, then
+ * the shm objects will be free by free_ipc_work().
+ */
+ ns = get_ipc_ns_not_zero(ns);
+ if (!ns) {
+unlink_continue:
+ list_del_init(&shp->shm_clist);
+ task_unlock(task);
+ continue;
}
- }
- /* Remove the list head from any segments still attached. */
- list_del(&task->sysvshm.shm_clist);
- up_write(&shm_ids(ns).rwsem);
+ /*
+ * 4) get a reference to shp.
+ * This cannot fail: shm_clist_rm() is called before
+ * ipc_rmid(), thus the refcount cannot be 0.
+ */
+ WARN_ON(!ipc_rcu_getref(&shp->shm_perm));
+
+ /*
+ * 5) unlink the shm segment from the list of segments
+ * created by current.
+ * This must be done last. After unlinking,
+ * only the refcounts obtained above prevent IPC_RMID
+ * from destroying the segment or the namespace.
+ */
+ list_del_init(&shp->shm_clist);
+
+ task_unlock(task);
+
+ /*
+ * 6) we have all references
+ * Thus lock & if needed destroy shp.
+ */
+ down_write(&shm_ids(ns).rwsem);
+ shm_lock_by_ptr(shp);
+ /*
+ * rcu_read_lock was implicitly taken in shm_lock_by_ptr, it's
+ * safe to call ipc_rcu_putref here
+ */
+ ipc_rcu_putref(&shp->shm_perm, shm_rcu_free);
+
+ if (ipc_valid_object(&shp->shm_perm)) {
+ if (shm_may_destroy(shp))
+ shm_destroy(ns, shp);
+ else
+ shm_unlock(shp);
+ } else {
+ /*
+ * Someone else deleted the shp from namespace
+ * idr/kht while we have waited.
+ * Just unlock and continue.
+ */
+ shm_unlock(shp);
+ }
+
+ up_write(&shm_ids(ns).rwsem);
+ put_ipc_ns(ns); /* paired with get_ipc_ns_not_zero */
+ }
}
static vm_fault_t shm_fault(struct vm_fault *vmf)
if (shmflg & SHM_NORESERVE)
acctflag = VM_NORESERVE;
file = hugetlb_file_setup(name, hugesize, acctflag,
- &shp->mlock_ucounts, HUGETLB_SHMFS_INODE,
- (shmflg >> SHM_HUGE_SHIFT) & SHM_HUGE_MASK);
+ HUGETLB_SHMFS_INODE, (shmflg >> SHM_HUGE_SHIFT) & SHM_HUGE_MASK);
} else {
/*
* Do not allow no accounting for OVERCOMMIT_NEVER, even
if (error < 0)
goto no_id;
+ shp->ns = ns;
+
+ task_lock(current);
list_add(&shp->shm_clist, ¤t->sysvshm.shm_clist);
+ task_unlock(current);
/*
* shmid gets reported as "inode#" in /proc/pid/maps.
no_id:
ipc_update_pid(&shp->shm_cprid, NULL);
ipc_update_pid(&shp->shm_lprid, NULL);
- if (is_file_hugepages(file) && shp->mlock_ucounts)
- user_shm_unlock(size, shp->mlock_ucounts);
fput(file);
ipc_rcu_putref(&shp->shm_perm, shm_rcu_free);
return error;
down_write(&shm_ids(ns).rwsem);
shp = shm_lock(ns, shmid);
shp->shm_nattch--;
- if (shm_may_destroy(ns, shp))
+
+ if (shm_may_destroy(shp))
shm_destroy(ns, shp);
else
shm_unlock(shp);
static void ipc_kht_remove(struct ipc_ids *ids, struct kern_ipc_perm *ipcp)
{
if (ipcp->key != IPC_PRIVATE)
- rhashtable_remove_fast(&ids->key_ht, &ipcp->khtnode,
- ipc_kht_params);
+ WARN_ON_ONCE(rhashtable_remove_fast(&ids->key_ht, &ipcp->khtnode,
+ ipc_kht_params));
}
/**
{
int idx = ipcid_to_idx(ipcp->id);
- idr_remove(&ids->ipcs_idr, idx);
+ WARN_ON_ONCE(idr_remove(&ids->ipcs_idr, idx) != ipcp);
ipc_kht_remove(ids, ipcp);
ids->in_use--;
ipcp->deleted = true;
# SPDX-License-Identifier: GPL-2.0-only
+config PREEMPT_NONE_BUILD
+ bool
+
+config PREEMPT_VOLUNTARY_BUILD
+ bool
+
+config PREEMPT_BUILD
+ bool
+ select PREEMPTION
+ select UNINLINE_SPIN_UNLOCK if !ARCH_INLINE_SPIN_UNLOCK
+
choice
prompt "Preemption Model"
- default PREEMPT_NONE_BEHAVIOUR
+ default PREEMPT_NONE
-config PREEMPT_NONE_BEHAVIOUR
+config PREEMPT_NONE
bool "No Forced Preemption (Server)"
- select PREEMPT_NONE if !PREEMPT_DYNAMIC
+ select PREEMPT_NONE_BUILD if !PREEMPT_DYNAMIC
help
This is the traditional Linux preemption model, geared towards
throughput. It will still provide good latencies most of the
raw processing power of the kernel, irrespective of scheduling
latencies.
-config PREEMPT_VOLUNTARY_BEHAVIOUR
+config PREEMPT_VOLUNTARY
bool "Voluntary Kernel Preemption (Desktop)"
depends on !ARCH_NO_PREEMPT
- select PREEMPT_VOLUNTARY if !PREEMPT_DYNAMIC
+ select PREEMPT_VOLUNTARY_BUILD if !PREEMPT_DYNAMIC
help
This option reduces the latency of the kernel by adding more
"explicit preemption points" to the kernel code. These new
Select this if you are building a kernel for a desktop system.
-config PREEMPT_BEHAVIOUR
+config PREEMPT
bool "Preemptible Kernel (Low-Latency Desktop)"
depends on !ARCH_NO_PREEMPT
- select PREEMPT
+ select PREEMPT_BUILD
help
This option reduces the latency of the kernel by making
all kernel code (that is not executing in a critical section)
config PREEMPT_RT
bool "Fully Preemptible Kernel (Real-Time)"
- depends on EXPERT && ARCH_SUPPORTS_RT && !PREEMPT_DYNAMIC
+ depends on EXPERT && ARCH_SUPPORTS_RT
select PREEMPTION
help
This option turns the kernel into a real-time kernel by replacing
endchoice
-config PREEMPT_NONE
- bool
-
-config PREEMPT_VOLUNTARY
- bool
-
-config PREEMPT
- bool
- select PREEMPTION
- select UNINLINE_SPIN_UNLOCK if !ARCH_INLINE_SPIN_UNLOCK
-
config PREEMPT_COUNT
bool
config PREEMPT_DYNAMIC
bool "Preemption behaviour defined on boot"
- depends on HAVE_PREEMPT_DYNAMIC
- select PREEMPT
+ depends on HAVE_PREEMPT_DYNAMIC && !PREEMPT_RT
+ select PREEMPT_BUILD
default y
help
This option allows to define the preemption model on the kernel
audit_mark = container_of(inode_mark, struct audit_fsnotify_mark, mark);
- if (WARN_ON_ONCE(inode_mark->group != audit_fsnotify_group) ||
- WARN_ON_ONCE(!inode))
+ if (WARN_ON_ONCE(inode_mark->group != audit_fsnotify_group))
return 0;
if (mask & (FS_CREATE|FS_MOVED_TO|FS_DELETE|FS_MOVED_FROM)) {
parent = container_of(inode_mark, struct audit_parent, mark);
- if (WARN_ON_ONCE(inode_mark->group != audit_watch_group) ||
- WARN_ON_ONCE(!inode))
+ if (WARN_ON_ONCE(inode_mark->group != audit_watch_group))
return 0;
if (mask & (FS_CREATE|FS_MOVED_TO) && inode)
return &bpf_sysctl_get_new_value_proto;
case BPF_FUNC_sysctl_set_new_value:
return &bpf_sysctl_set_new_value_proto;
+ case BPF_FUNC_ktime_get_coarse_ns:
+ return &bpf_ktime_get_coarse_ns_proto;
default:
return cgroup_base_func_proto(func_id, prog);
}
i = end_new;
insn = prog->insnsi + end_old;
}
+ if (bpf_pseudo_func(insn)) {
+ ret = bpf_adj_delta_to_imm(insn, pos, end_old,
+ end_new, i, probe_pass);
+ if (ret)
+ return ret;
+ continue;
+ }
code = insn->code;
if ((BPF_CLASS(code) != BPF_JMP &&
BPF_CLASS(code) != BPF_JMP32) ||
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
- case BPF_FUNC_ktime_get_coarse_ns:
- return &bpf_ktime_get_coarse_ns_proto;
case BPF_FUNC_ringbuf_output:
return &bpf_ringbuf_output_proto;
case BPF_FUNC_ringbuf_reserve:
return map;
}
+static void bpf_map_write_active_inc(struct bpf_map *map)
+{
+ atomic64_inc(&map->writecnt);
+}
+
+static void bpf_map_write_active_dec(struct bpf_map *map)
+{
+ atomic64_dec(&map->writecnt);
+}
+
+bool bpf_map_write_active(const struct bpf_map *map)
+{
+ return atomic64_read(&map->writecnt) != 0;
+}
+
static u32 bpf_map_value_size(const struct bpf_map *map)
{
if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH ||
{
struct bpf_map *map = vma->vm_file->private_data;
- if (vma->vm_flags & VM_MAYWRITE) {
- mutex_lock(&map->freeze_mutex);
- map->writecnt++;
- mutex_unlock(&map->freeze_mutex);
- }
+ if (vma->vm_flags & VM_MAYWRITE)
+ bpf_map_write_active_inc(map);
}
/* called for all unmapped memory region (including initial) */
{
struct bpf_map *map = vma->vm_file->private_data;
- if (vma->vm_flags & VM_MAYWRITE) {
- mutex_lock(&map->freeze_mutex);
- map->writecnt--;
- mutex_unlock(&map->freeze_mutex);
- }
+ if (vma->vm_flags & VM_MAYWRITE)
+ bpf_map_write_active_dec(map);
}
static const struct vm_operations_struct bpf_map_default_vmops = {
goto out;
if (vma->vm_flags & VM_MAYWRITE)
- map->writecnt++;
+ bpf_map_write_active_inc(map);
out:
mutex_unlock(&map->freeze_mutex);
return err;
map = __bpf_map_get(f);
if (IS_ERR(map))
return PTR_ERR(map);
+ bpf_map_write_active_inc(map);
if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
err = -EPERM;
goto err_put;
free_key:
kvfree(key);
err_put:
+ bpf_map_write_active_dec(map);
fdput(f);
return err;
}
map = __bpf_map_get(f);
if (IS_ERR(map))
return PTR_ERR(map);
+ bpf_map_write_active_inc(map);
if (!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
err = -EPERM;
goto err_put;
out:
kvfree(key);
err_put:
+ bpf_map_write_active_dec(map);
fdput(f);
return err;
}
map = __bpf_map_get(f);
if (IS_ERR(map))
return PTR_ERR(map);
+ bpf_map_write_active_inc(map);
if (!(map_get_sys_perms(map, f) & FMODE_CAN_READ) ||
!(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
err = -EPERM;
free_key:
kvfree(key);
err_put:
+ bpf_map_write_active_dec(map);
fdput(f);
return err;
}
}
mutex_lock(&map->freeze_mutex);
-
- if (map->writecnt) {
+ if (bpf_map_write_active(map)) {
err = -EBUSY;
goto err_put;
}
union bpf_attr __user *uattr,
int cmd)
{
+ bool has_read = cmd == BPF_MAP_LOOKUP_BATCH ||
+ cmd == BPF_MAP_LOOKUP_AND_DELETE_BATCH;
+ bool has_write = cmd != BPF_MAP_LOOKUP_BATCH;
struct bpf_map *map;
int err, ufd;
struct fd f;
map = __bpf_map_get(f);
if (IS_ERR(map))
return PTR_ERR(map);
-
- if ((cmd == BPF_MAP_LOOKUP_BATCH ||
- cmd == BPF_MAP_LOOKUP_AND_DELETE_BATCH) &&
- !(map_get_sys_perms(map, f) & FMODE_CAN_READ)) {
+ if (has_write)
+ bpf_map_write_active_inc(map);
+ if (has_read && !(map_get_sys_perms(map, f) & FMODE_CAN_READ)) {
err = -EPERM;
goto err_put;
}
-
- if (cmd != BPF_MAP_LOOKUP_BATCH &&
- !(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
+ if (has_write && !(map_get_sys_perms(map, f) & FMODE_CAN_WRITE)) {
err = -EPERM;
goto err_put;
}
BPF_DO_BATCH(map->ops->map_update_batch);
else
BPF_DO_BATCH(map->ops->map_delete_batch);
-
err_put:
+ if (has_write)
+ bpf_map_write_active_dec(map);
fdput(f);
return err;
}
insn->src_reg == BPF_PSEUDO_KFUNC_CALL;
}
-static bool bpf_pseudo_func(const struct bpf_insn *insn)
-{
- return insn->code == (BPF_LD | BPF_IMM | BPF_DW) &&
- insn->src_reg == BPF_PSEUDO_FUNC;
-}
-
struct bpf_call_arg_meta {
struct bpf_map *map_ptr;
bool raw_mode;
/* transfer reg's id which is unique for every map_lookup_elem
* as UID of the inner map.
*/
- reg->map_uid = reg->id;
+ if (map_value_has_timer(map->inner_map_meta))
+ reg->map_uid = reg->id;
} else if (map->map_type == BPF_MAP_TYPE_XSKMAP) {
reg->type = PTR_TO_XDP_SOCK;
} else if (map->map_type == BPF_MAP_TYPE_SOCKMAP ||
return -EPERM;
}
- if (bpf_pseudo_func(insn)) {
+ if (bpf_pseudo_func(insn) || bpf_pseudo_call(insn))
ret = add_subprog(env, i + insn->imm + 1);
- if (ret >= 0)
- /* remember subprog */
- insn[1].imm = ret;
- } else if (bpf_pseudo_call(insn)) {
- ret = add_subprog(env, i + insn->imm + 1);
- } else {
+ else
ret = add_kfunc_call(env, insn->imm, insn->off);
- }
if (ret < 0)
return ret;
reg = ®_state->stack[spi].spilled_ptr;
if (is_spilled_reg(®_state->stack[spi])) {
- if (size != BPF_REG_SIZE) {
- u8 scalar_size = 0;
+ u8 spill_size = 1;
+ for (i = BPF_REG_SIZE - 1; i > 0 && stype[i - 1] == STACK_SPILL; i--)
+ spill_size++;
+
+ if (size != BPF_REG_SIZE || spill_size != BPF_REG_SIZE) {
if (reg->type != SCALAR_VALUE) {
verbose_linfo(env, env->insn_idx, "; ");
verbose(env, "invalid size of register fill\n");
if (dst_regno < 0)
return 0;
- for (i = BPF_REG_SIZE; i > 0 && stype[i - 1] == STACK_SPILL; i--)
- scalar_size++;
-
- if (!(off % BPF_REG_SIZE) && size == scalar_size) {
+ if (!(off % BPF_REG_SIZE) && size == spill_size) {
/* The earlier check_reg_arg() has decided the
* subreg_def for this insn. Save it first.
*/
state->regs[dst_regno].live |= REG_LIVE_WRITTEN;
return 0;
}
- for (i = 1; i < BPF_REG_SIZE; i++) {
- if (stype[(slot - i) % BPF_REG_SIZE] != STACK_SPILL) {
- verbose(env, "corrupted spill memory\n");
- return -EACCES;
- }
- }
if (dst_regno >= 0) {
/* restore register state from stack */
static bool bpf_map_is_rdonly(const struct bpf_map *map)
{
- return (map->map_flags & BPF_F_RDONLY_PROG) && map->frozen;
+ /* A map is considered read-only if the following condition are true:
+ *
+ * 1) BPF program side cannot change any of the map content. The
+ * BPF_F_RDONLY_PROG flag is throughout the lifetime of a map
+ * and was set at map creation time.
+ * 2) The map value(s) have been initialized from user space by a
+ * loader and then "frozen", such that no new map update/delete
+ * operations from syscall side are possible for the rest of
+ * the map's lifetime from that point onwards.
+ * 3) Any parallel/pending map update/delete operations from syscall
+ * side have been completed. Only after that point, it's safe to
+ * assume that map value(s) are immutable.
+ */
+ return (map->map_flags & BPF_F_RDONLY_PROG) &&
+ READ_ONCE(map->frozen) &&
+ !bpf_map_write_active(map);
}
static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val)
if (insn->src_reg == BPF_PSEUDO_FUNC) {
struct bpf_prog_aux *aux = env->prog->aux;
- u32 subprogno = insn[1].imm;
+ u32 subprogno = find_subprog(env,
+ env->insn_idx + insn->imm + 1);
if (!aux->func_info) {
verbose(env, "missing btf func_info\n");
}
}
+ if (map_value_has_timer(map)) {
+ if (is_tracing_prog_type(prog_type)) {
+ verbose(env, "tracing progs cannot use bpf_timer yet\n");
+ return -EINVAL;
+ }
+ }
+
if ((bpf_prog_is_dev_bound(prog->aux) || bpf_map_is_dev_bound(map)) &&
!bpf_offload_prog_map_match(prog, map)) {
verbose(env, "offload device mismatch between prog and map\n");
return 0;
for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
- if (bpf_pseudo_func(insn)) {
- env->insn_aux_data[i].call_imm = insn->imm;
- /* subprog is encoded in insn[1].imm */
+ if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn))
continue;
- }
- if (!bpf_pseudo_call(insn))
- continue;
/* Upon error here we cannot fall back to interpreter but
* need a hard reject of the program. Thus -EFAULT is
* propagated in any case.
env->insn_aux_data[i].call_imm = insn->imm;
/* point imm to __bpf_call_base+1 from JITs point of view */
insn->imm = 1;
+ if (bpf_pseudo_func(insn))
+ /* jit (e.g. x86_64) may emit fewer instructions
+ * if it learns a u32 imm is the same as a u64 imm.
+ * Force a non zero here.
+ */
+ insn[1].imm = 1;
}
err = bpf_prog_alloc_jited_linfo(prog);
insn = func[i]->insnsi;
for (j = 0; j < func[i]->len; j++, insn++) {
if (bpf_pseudo_func(insn)) {
- subprog = insn[1].imm;
+ subprog = insn->off;
insn[0].imm = (u32)(long)func[subprog]->bpf_func;
insn[1].imm = ((u64)(long)func[subprog]->bpf_func) >> 32;
continue;
for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) {
if (bpf_pseudo_func(insn)) {
insn[0].imm = env->insn_aux_data[i].call_imm;
- insn[1].imm = find_subprog(env, i + insn[0].imm + 1);
+ insn[1].imm = insn->off;
+ insn->off = 0;
continue;
}
if (!bpf_pseudo_call(insn))
DEFINE_STATIC_KEY_FALSE(cpusets_pre_enable_key);
DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key);
+/*
+ * There could be abnormal cpuset configurations for cpu or memory
+ * node binding, add this key to provide a quick low-cost judgement
+ * of the situation.
+ */
+DEFINE_STATIC_KEY_FALSE(cpusets_insane_config_key);
+
/* See "Frequency meter" comments, below. */
struct fmeter {
static DECLARE_WAIT_QUEUE_HEAD(cpuset_attach_wq);
+static inline void check_insane_mems_config(nodemask_t *nodes)
+{
+ if (!cpusets_insane_config() &&
+ movable_only_nodes(nodes)) {
+ static_branch_enable(&cpusets_insane_config_key);
+ pr_info("Unsupported (movable nodes only) cpuset configuration detected (nmask=%*pbl)!\n"
+ "Cpuset allocations might fail even with a lot of memory available.\n",
+ nodemask_pr_args(nodes));
+ }
+}
+
/*
* Cgroup v2 behavior is used on the "cpus" and "mems" control files when
* on default hierarchy or when the cpuset_v2_mode flag is set by mounting
if (retval < 0)
goto done;
+ check_insane_mems_config(&trialcs->mems_allowed);
+
spin_lock_irq(&callback_lock);
cs->mems_allowed = trialcs->mems_allowed;
spin_unlock_irq(&callback_lock);
cpus_updated = !cpumask_equal(&new_cpus, cs->effective_cpus);
mems_updated = !nodes_equal(new_mems, cs->effective_mems);
+ if (mems_updated)
+ check_insane_mems_config(&new_mems);
+
if (is_in_v2_mode())
hotplug_update_tasks(cs, &new_cpus, &new_mems,
cpus_updated, mems_updated);
* btp <pid> Kernel stack for <pid>
* btt <address-expression> Kernel stack for task structure at
* <address-expression>
- * bta [DRSTCZEUIMA] All useful processes, optionally
+ * bta [state_chars>|A] All useful processes, optionally
* filtered by state
* btc [<cpu>] The current process on one cpu,
* default is all cpus
*/
static int
-kdb_bt1(struct task_struct *p, unsigned long mask, bool btaprompt)
+kdb_bt1(struct task_struct *p, const char *mask, bool btaprompt)
{
char ch;
return;
}
- kdb_bt1(kdb_tsk, ~0UL, false);
+ kdb_bt1(kdb_tsk, "A", false);
}
int
if (strcmp(argv[0], "bta") == 0) {
struct task_struct *g, *p;
unsigned long cpu;
- unsigned long mask = kdb_task_state_string(argc ? argv[1] :
- NULL);
+ const char *mask = argc ? argv[1] : kdbgetenv("PS");
+
if (argc == 0)
kdb_ps_suppressed();
/* Run the active tasks first */
return diag;
p = find_task_by_pid_ns(pid, &init_pid_ns);
if (p)
- return kdb_bt1(p, ~0UL, false);
+ return kdb_bt1(p, "A", false);
kdb_printf("No process with pid == %ld found\n", pid);
return 0;
} else if (strcmp(argv[0], "btt") == 0) {
diag = kdbgetularg((char *)argv[1], &addr);
if (diag)
return diag;
- return kdb_bt1((struct task_struct *)addr, ~0UL, false);
+ return kdb_bt1((struct task_struct *)addr, "A", false);
} else if (strcmp(argv[0], "btc") == 0) {
unsigned long cpu = ~0;
if (argc > 1)
kdb_show_stack(kdb_current_task, (void *)addr);
return 0;
} else {
- return kdb_bt1(kdb_current_task, ~0UL, false);
+ return kdb_bt1(kdb_current_task, "A", false);
}
}
state = 'D'; /* cpu is online but unresponsive */
} else {
state = ' '; /* cpu is responding to kdb */
- if (kdb_task_state_char(KDB_TSK(i)) == 'I')
- state = 'I'; /* idle task */
+ if (kdb_task_state_char(KDB_TSK(i)) == '-')
+ state = '-'; /* idle task */
}
if (state != prev_state) {
if (prev_state != '?') {
void kdb_ps_suppressed(void)
{
int idle = 0, daemon = 0;
- unsigned long mask_I = kdb_task_state_string("I"),
- mask_M = kdb_task_state_string("M");
unsigned long cpu;
const struct task_struct *p, *g;
for_each_online_cpu(cpu) {
p = kdb_curr_task(cpu);
- if (kdb_task_state(p, mask_I))
+ if (kdb_task_state(p, "-"))
++idle;
}
for_each_process_thread(g, p) {
- if (kdb_task_state(p, mask_M))
+ if (kdb_task_state(p, "ims"))
++daemon;
}
if (idle || daemon) {
if (idle)
- kdb_printf("%d idle process%s (state I)%s\n",
+ kdb_printf("%d idle process%s (state -)%s\n",
idle, idle == 1 ? "" : "es",
daemon ? " and " : "");
if (daemon)
- kdb_printf("%d sleeping system daemon (state M) "
+ kdb_printf("%d sleeping system daemon (state [ims]) "
"process%s", daemon,
daemon == 1 ? "" : "es");
kdb_printf(" suppressed,\nuse 'ps A' to see all.\n");
}
}
-/*
- * kdb_ps - This function implements the 'ps' command which shows a
- * list of the active processes.
- * ps [DRSTCZEUIMA] All processes, optionally filtered by state
- */
void kdb_ps1(const struct task_struct *p)
{
int cpu;
}
}
+/*
+ * kdb_ps - This function implements the 'ps' command which shows a
+ * list of the active processes.
+ *
+ * ps [<state_chars>] Show processes, optionally selecting only those whose
+ * state character is found in <state_chars>.
+ */
static int kdb_ps(int argc, const char **argv)
{
struct task_struct *g, *p;
- unsigned long mask, cpu;
+ const char *mask;
+ unsigned long cpu;
if (argc == 0)
kdb_ps_suppressed();
kdb_printf("%-*s Pid Parent [*] cpu State %-*s Command\n",
(int)(2*sizeof(void *))+2, "Task Addr",
(int)(2*sizeof(void *))+2, "Thread");
- mask = kdb_task_state_string(argc ? argv[1] : NULL);
+ mask = argc ? argv[1] : kdbgetenv("PS");
/* Run the active tasks first */
for_each_online_cpu(cpu) {
if (KDB_FLAG(CMD_INTERRUPT))
},
{ .name = "bta",
.func = kdb_bt,
- .usage = "[D|R|S|T|C|Z|E|U|I|M|A]",
- .help = "Backtrace all processes matching state flag",
+ .usage = "[<state_chars>|A]",
+ .help = "Backtrace all processes whose state matches",
.flags = KDB_ENABLE_INSPECT,
},
{ .name = "btc",
},
{ .name = "ps",
.func = kdb_ps,
- .usage = "[<flags>|A]",
+ .usage = "[<state_chars>|A]",
.help = "Display active task list",
.flags = KDB_ENABLE_INSPECT,
},
extern int kdb_grep_leading;
extern int kdb_grep_trailing;
extern char *kdb_cmds[];
-extern unsigned long kdb_task_state_string(const char *);
extern char kdb_task_state_char (const struct task_struct *);
-extern unsigned long kdb_task_state(const struct task_struct *p,
- unsigned long mask);
+extern bool kdb_task_state(const struct task_struct *p, const char *mask);
extern void kdb_ps_suppressed(void);
extern void kdb_ps1(const struct task_struct *p);
extern void kdb_send_sig(struct task_struct *p, int sig);
#include <linux/uaccess.h>
#include <linux/kdb.h>
#include <linux/slab.h>
+#include <linux/ctype.h>
#include "kdb_private.h"
/*
return diag;
}
-/*
- * kdb_task_state_string - Convert a string containing any of the
- * letters DRSTCZEUIMA to a mask for the process state field and
- * return the value. If no argument is supplied, return the mask
- * that corresponds to environment variable PS, DRSTCZEU by
- * default.
- * Inputs:
- * s String to convert
- * Returns:
- * Mask for process state.
- * Notes:
- * The mask folds data from several sources into a single long value, so
- * be careful not to overlap the bits. TASK_* bits are in the LSB,
- * special cases like UNRUNNABLE are in the MSB. As of 2.6.10-rc1 there
- * is no overlap between TASK_* and EXIT_* but that may not always be
- * true, so EXIT_* bits are shifted left 16 bits before being stored in
- * the mask.
- */
-
-/* unrunnable is < 0 */
-#define UNRUNNABLE (1UL << (8*sizeof(unsigned long) - 1))
-#define RUNNING (1UL << (8*sizeof(unsigned long) - 2))
-#define IDLE (1UL << (8*sizeof(unsigned long) - 3))
-#define DAEMON (1UL << (8*sizeof(unsigned long) - 4))
-unsigned long kdb_task_state_string(const char *s)
-{
- long res = 0;
- if (!s) {
- s = kdbgetenv("PS");
- if (!s)
- s = "DRSTCZEU"; /* default value for ps */
- }
- while (*s) {
- switch (*s) {
- case 'D':
- res |= TASK_UNINTERRUPTIBLE;
- break;
- case 'R':
- res |= RUNNING;
- break;
- case 'S':
- res |= TASK_INTERRUPTIBLE;
- break;
- case 'T':
- res |= TASK_STOPPED;
- break;
- case 'C':
- res |= TASK_TRACED;
- break;
- case 'Z':
- res |= EXIT_ZOMBIE << 16;
- break;
- case 'E':
- res |= EXIT_DEAD << 16;
- break;
- case 'U':
- res |= UNRUNNABLE;
- break;
- case 'I':
- res |= IDLE;
- break;
- case 'M':
- res |= DAEMON;
- break;
- case 'A':
- res = ~0UL;
- break;
- default:
- kdb_func_printf("unknown flag '%c' ignored\n", *s);
- break;
- }
- ++s;
- }
- return res;
-}
/*
* kdb_task_state_char - Return the character that represents the task state.
*/
char kdb_task_state_char (const struct task_struct *p)
{
- unsigned int p_state;
unsigned long tmp;
char state;
int cpu;
copy_from_kernel_nofault(&tmp, (char *)p, sizeof(unsigned long)))
return 'E';
- cpu = kdb_process_cpu(p);
- p_state = READ_ONCE(p->__state);
- state = (p_state == 0) ? 'R' :
- (p_state < 0) ? 'U' :
- (p_state & TASK_UNINTERRUPTIBLE) ? 'D' :
- (p_state & TASK_STOPPED) ? 'T' :
- (p_state & TASK_TRACED) ? 'C' :
- (p->exit_state & EXIT_ZOMBIE) ? 'Z' :
- (p->exit_state & EXIT_DEAD) ? 'E' :
- (p_state & TASK_INTERRUPTIBLE) ? 'S' : '?';
+ state = task_state_to_char((struct task_struct *) p);
+
if (is_idle_task(p)) {
/* Idle task. Is it really idle, apart from the kdb
* interrupt? */
+ cpu = kdb_process_cpu(p);
if (!kdb_task_has_cpu(p) || kgdb_info[cpu].irq_depth == 1) {
if (cpu != kdb_initial_cpu)
- state = 'I'; /* idle task */
+ state = '-'; /* idle task */
}
- } else if (!p->mm && state == 'S') {
- state = 'M'; /* sleeping system daemon */
+ } else if (!p->mm && strchr("IMS", state)) {
+ state = tolower(state); /* sleeping system daemon */
}
return state;
}
* given by the mask.
* Inputs:
* p struct task for the process
- * mask mask from kdb_task_state_string to select processes
+ * mask set of characters used to select processes; both NULL
+ * and the empty string mean adopt a default filter, which
+ * is to suppress sleeping system daemons and the idle tasks
* Returns:
* True if the process matches at least one criteria defined by the mask.
*/
-unsigned long kdb_task_state(const struct task_struct *p, unsigned long mask)
+bool kdb_task_state(const struct task_struct *p, const char *mask)
{
- char state[] = { kdb_task_state_char(p), '\0' };
- return (mask & kdb_task_state_string(state)) != 0;
+ char state = kdb_task_state_char(p);
+
+ /* If there is no mask, then we will filter code that runs when the
+ * scheduler is idling and any system daemons that are currently
+ * sleeping.
+ */
+ if (!mask || mask[0] == '\0')
+ return !strchr("-ims", state);
+
+ /* A is a special case that matches all states */
+ if (strchr(mask, 'A'))
+ return true;
+
+ return strchr(mask, state);
}
/* Maintain a small stack of kdb_flags to allow recursion without disturbing
{
struct dma_coherent_mem *dma_mem;
int pages = size >> PAGE_SHIFT;
- int bitmap_size = BITS_TO_LONGS(pages) * sizeof(long);
void *mem_base;
if (!size)
dma_mem = kzalloc(sizeof(struct dma_coherent_mem), GFP_KERNEL);
if (!dma_mem)
goto out_unmap_membase;
- dma_mem->bitmap = kzalloc(bitmap_size, GFP_KERNEL);
+ dma_mem->bitmap = bitmap_zalloc(pages, GFP_KERNEL);
if (!dma_mem->bitmap)
goto out_free_dma_mem;
return;
memunmap(mem->virt_base);
- kfree(mem->bitmap);
+ bitmap_free(mem->bitmap);
kfree(mem);
}
return;
fail_free_mem:
- memblock_free_early(__pa(tlb), bytes);
+ memblock_free(tlb, bytes);
fail:
pr_warn("Cannot allocate buffer");
}
* access_ok() is performed once, at prctl time, when
* the selector is loaded by userspace.
*/
- if (unlikely(__get_user(state, sd->selector)))
- do_exit(SIGSEGV);
+ if (unlikely(__get_user(state, sd->selector))) {
+ force_exit_sig(SIGSEGV);
+ return true;
+ }
if (likely(state == SYSCALL_DISPATCH_FILTER_ALLOW))
return false;
- if (state != SYSCALL_DISPATCH_FILTER_BLOCK)
- do_exit(SIGSYS);
+ if (state != SYSCALL_DISPATCH_FILTER_BLOCK) {
+ force_exit_sig(SIGSYS);
+ return true;
+ }
}
sd->on_dispatch = true;
static u64 perf_virt_to_phys(u64 virt)
{
u64 phys_addr = 0;
- struct page *p = NULL;
if (!virt)
return 0;
* If failed, leave phys_addr as 0.
*/
if (current->mm != NULL) {
+ struct page *p;
+
pagefault_disable();
- if (get_user_page_fast_only(virt, 0, &p))
+ if (get_user_page_fast_only(virt, 0, &p)) {
phys_addr = page_to_phys(p) + virt % PAGE_SIZE;
+ put_page(p);
+ }
pagefault_enable();
}
-
- if (p)
- put_page(p);
}
return phys_addr;
return e;
}
-int init_kernel_text(unsigned long addr)
-{
- if (addr >= (unsigned long)_sinittext &&
- addr < (unsigned long)_einittext)
- return 1;
- return 0;
-}
-
int notrace core_kernel_text(unsigned long addr)
{
- if (addr >= (unsigned long)_stext &&
- addr < (unsigned long)_etext)
+ if (is_kernel_text(addr))
return 1;
- if (system_state < SYSTEM_RUNNING &&
- init_kernel_text(addr))
- return 1;
- return 0;
-}
-
-/**
- * core_kernel_data - tell if addr points to kernel data
- * @addr: address to test
- *
- * Returns true if @addr passed in is from the core kernel data
- * section.
- *
- * Note: On some archs it may return true for core RODATA, and false
- * for others. But will always be true for core RW data.
- */
-int core_kernel_data(unsigned long addr)
-{
- if (addr >= (unsigned long)_sdata &&
- addr < (unsigned long)_edata)
+ if (system_state < SYSTEM_FREEING_INITMEM &&
+ is_kernel_inittext(addr))
return 1;
return 0;
}
* Since we are after the module-symbols check, there's
* no danger of address overlap:
*/
- if (init_kernel_text(addr))
+ if (is_kernel_inittext(addr))
return 1;
return 0;
}
p->pdeath_signal = 0;
INIT_LIST_HEAD(&p->thread_group);
p->task_works = NULL;
+ clear_posix_cputimers_work(p);
#ifdef CONFIG_KRETPROBES
p->kretprobe_instances.first = NULL;
int ksys_unshare(unsigned long unshare_flags)
{
struct fs_struct *fs, *new_fs = NULL;
- struct files_struct *fd, *new_fd = NULL;
+ struct files_struct *new_fd = NULL;
struct cred *new_cred = NULL;
struct nsproxy *new_nsproxy = NULL;
int do_sysvsem = 0;
spin_unlock(&fs->lock);
}
- if (new_fd) {
- fd = current->files;
- current->files = new_fd;
- new_fd = fd;
- }
+ if (new_fd)
+ swap(current->files, new_fd);
task_unlock(current);
return 0;
}
-static void of_phandle_args_to_fwspec(struct device_node *np, const u32 *args,
- unsigned int count,
- struct irq_fwspec *fwspec)
+void of_phandle_args_to_fwspec(struct device_node *np, const u32 *args,
+ unsigned int count, struct irq_fwspec *fwspec)
{
int i;
for (i = 0; i < count; i++)
fwspec->param[i] = args[i];
}
+EXPORT_SYMBOL_GPL(of_phandle_args_to_fwspec);
unsigned int irq_create_fwspec_mapping(struct irq_fwspec *fwspec)
{
irq_free_descs(virq, nr_irqs);
return ret;
}
+EXPORT_SYMBOL_GPL(__irq_domain_alloc_irqs);
/* The irq_data was moved, fix the revmap to refer to the new location */
static void irq_domain_fix_revmap(struct irq_data *d)
/*
* Checking the first MSI descriptor is sufficient. MSIX supports
- * masking and MSI does so when the maskbit is set.
+ * masking and MSI does so when the can_mask attribute is set.
*/
desc = first_msi_entry(dev);
- return desc->msi_attrib.is_msix || desc->msi_attrib.maskbit;
+ return desc->msi_attrib.is_msix || desc->msi_attrib.can_mask;
}
int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
struct kcov_percpu_data {
void *irq_area;
+ local_lock_t lock;
unsigned int saved_mode;
unsigned int saved_size;
int saved_sequence;
};
-static DEFINE_PER_CPU(struct kcov_percpu_data, kcov_percpu_data);
+static DEFINE_PER_CPU(struct kcov_percpu_data, kcov_percpu_data) = {
+ .lock = INIT_LOCAL_LOCK(lock),
+};
/* Must be called with kcov_remote_lock locked. */
static struct kcov_remote *kcov_remote_find(u64 handle)
if (!in_task() && !in_serving_softirq())
return;
- local_irq_save(flags);
+ local_lock_irqsave(&kcov_percpu_data.lock, flags);
/*
* Check that kcov_remote_start() is not called twice in background
*/
mode = READ_ONCE(t->kcov_mode);
if (WARN_ON(in_task() && kcov_mode_enabled(mode))) {
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
/*
* happened while collecting coverage from a background thread.
*/
if (WARN_ON(in_serving_softirq() && t->kcov_softirq)) {
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
spin_lock(&kcov_remote_lock);
remote = kcov_remote_find(handle);
if (!remote) {
- spin_unlock_irqrestore(&kcov_remote_lock, flags);
+ spin_unlock(&kcov_remote_lock);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
kcov_debug("handle = %llx, context: %s\n", handle,
size = CONFIG_KCOV_IRQ_AREA_SIZE;
area = this_cpu_ptr(&kcov_percpu_data)->irq_area;
}
- spin_unlock_irqrestore(&kcov_remote_lock, flags);
+ spin_unlock(&kcov_remote_lock);
/* Can only happen when in_task(). */
if (!area) {
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
area = vmalloc(size * sizeof(unsigned long));
if (!area) {
kcov_put(kcov);
return;
}
+ local_lock_irqsave(&kcov_percpu_data.lock, flags);
}
- local_irq_save(flags);
-
/* Reset coverage size. */
*(u64 *)area = 0;
}
kcov_start(t, kcov, size, area, mode, sequence);
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
}
EXPORT_SYMBOL(kcov_remote_start);
if (!in_task() && !in_serving_softirq())
return;
- local_irq_save(flags);
+ local_lock_irqsave(&kcov_percpu_data.lock, flags);
mode = READ_ONCE(t->kcov_mode);
barrier();
if (!kcov_mode_enabled(mode)) {
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
/*
* actually found the remote handle and started collecting coverage.
*/
if (in_serving_softirq() && !t->kcov_softirq) {
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
/* Make sure that kcov_softirq is only set when in softirq. */
if (WARN_ON(!in_serving_softirq() && t->kcov_softirq)) {
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
return;
}
spin_unlock(&kcov_remote_lock);
}
- local_irq_restore(flags);
+ local_unlock_irqrestore(&kcov_percpu_data.lock, flags);
/* Get in kcov_remote_start(). */
kcov_put(kcov);
int cpu;
for_each_possible_cpu(cpu) {
- void *area = vmalloc(CONFIG_KCOV_IRQ_AREA_SIZE *
- sizeof(unsigned long));
+ void *area = vmalloc_node(CONFIG_KCOV_IRQ_AREA_SIZE *
+ sizeof(unsigned long), cpu_to_node(cpu));
if (!area)
return -ENOMEM;
per_cpu_ptr(&kcov_percpu_data, cpu)->irq_area = area;
return in_task() ? ¤t->kcsan_ctx : raw_cpu_ptr(&kcsan_cpu_ctx);
}
+static __always_inline void
+check_access(const volatile void *ptr, size_t size, int type, unsigned long ip);
+
/* Check scoped accesses; never inline because this is a slow-path! */
static noinline void kcsan_check_scoped_accesses(void)
{
struct kcsan_scoped_access *scoped_access;
ctx->scoped_accesses.prev = NULL; /* Avoid recursion. */
- list_for_each_entry(scoped_access, &ctx->scoped_accesses, list)
- __kcsan_check_access(scoped_access->ptr, scoped_access->size, scoped_access->type);
+ list_for_each_entry(scoped_access, &ctx->scoped_accesses, list) {
+ check_access(scoped_access->ptr, scoped_access->size,
+ scoped_access->type, scoped_access->ip);
+ }
ctx->scoped_accesses.prev = prev_save;
}
/* Rules for generic atomic accesses. Called from fast-path. */
static __always_inline bool
-is_atomic(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *ctx)
+is_atomic(struct kcsan_ctx *ctx, const volatile void *ptr, size_t size, int type)
{
if (type & KCSAN_ACCESS_ATOMIC)
return true;
}
static __always_inline bool
-should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *ctx)
+should_watch(struct kcsan_ctx *ctx, const volatile void *ptr, size_t size, int type)
{
/*
* Never set up watchpoints when memory operations are atomic.
* should not count towards skipped instructions, and (2) to actually
* decrement kcsan_atomic_next for consecutive instruction stream.
*/
- if (is_atomic(ptr, size, type, ctx))
+ if (is_atomic(ctx, ptr, size, type))
return false;
if (this_cpu_dec_return(kcsan_skip) >= 0)
static noinline void kcsan_found_watchpoint(const volatile void *ptr,
size_t size,
int type,
+ unsigned long ip,
atomic_long_t *watchpoint,
long encoded_watchpoint)
{
if (consumed) {
kcsan_save_irqtrace(current);
- kcsan_report_set_info(ptr, size, type, watchpoint - watchpoints);
+ kcsan_report_set_info(ptr, size, type, ip, watchpoint - watchpoints);
kcsan_restore_irqtrace(current);
} else {
/*
}
static noinline void
-kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type)
+kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type, unsigned long ip)
{
const bool is_write = (type & KCSAN_ACCESS_WRITE) != 0;
const bool is_assert = (type & KCSAN_ACCESS_ASSERT) != 0;
if (is_assert && value_change == KCSAN_VALUE_CHANGE_TRUE)
atomic_long_inc(&kcsan_counters[KCSAN_COUNTER_ASSERT_FAILURES]);
- kcsan_report_known_origin(ptr, size, type, value_change,
- watchpoint - watchpoints,
+ kcsan_report_known_origin(ptr, size, type, ip,
+ value_change, watchpoint - watchpoints,
old, new, access_mask);
} else if (value_change == KCSAN_VALUE_CHANGE_TRUE) {
/* Inferring a race, since the value should not have changed. */
if (is_assert)
atomic_long_inc(&kcsan_counters[KCSAN_COUNTER_ASSERT_FAILURES]);
- if (IS_ENABLED(CONFIG_KCSAN_REPORT_RACE_UNKNOWN_ORIGIN) || is_assert)
- kcsan_report_unknown_origin(ptr, size, type, old, new, access_mask);
+ if (IS_ENABLED(CONFIG_KCSAN_REPORT_RACE_UNKNOWN_ORIGIN) || is_assert) {
+ kcsan_report_unknown_origin(ptr, size, type, ip,
+ old, new, access_mask);
+ }
}
/*
user_access_restore(ua_flags);
}
-static __always_inline void check_access(const volatile void *ptr, size_t size,
- int type)
+static __always_inline void
+check_access(const volatile void *ptr, size_t size, int type, unsigned long ip)
{
const bool is_write = (type & KCSAN_ACCESS_WRITE) != 0;
atomic_long_t *watchpoint;
*/
if (unlikely(watchpoint != NULL))
- kcsan_found_watchpoint(ptr, size, type, watchpoint,
- encoded_watchpoint);
+ kcsan_found_watchpoint(ptr, size, type, ip, watchpoint, encoded_watchpoint);
else {
struct kcsan_ctx *ctx = get_ctx(); /* Call only once in fast-path. */
- if (unlikely(should_watch(ptr, size, type, ctx)))
- kcsan_setup_watchpoint(ptr, size, type);
+ if (unlikely(should_watch(ctx, ptr, size, type)))
+ kcsan_setup_watchpoint(ptr, size, type, ip);
else if (unlikely(ctx->scoped_accesses.prev))
kcsan_check_scoped_accesses();
}
{
struct kcsan_ctx *ctx = get_ctx();
- __kcsan_check_access(ptr, size, type);
+ check_access(ptr, size, type, _RET_IP_);
ctx->disable_count++; /* Disable KCSAN, in case list debugging is on. */
sa->ptr = ptr;
sa->size = size;
sa->type = type;
+ sa->ip = _RET_IP_;
if (!ctx->scoped_accesses.prev) /* Lazy initialize list head. */
INIT_LIST_HEAD(&ctx->scoped_accesses);
ctx->disable_count--;
- __kcsan_check_access(sa->ptr, sa->size, sa->type);
+ check_access(sa->ptr, sa->size, sa->type, sa->ip);
}
EXPORT_SYMBOL(kcsan_end_scoped_access);
void __kcsan_check_access(const volatile void *ptr, size_t size, int type)
{
- check_access(ptr, size, type);
+ check_access(ptr, size, type, _RET_IP_);
}
EXPORT_SYMBOL(__kcsan_check_access);
void __tsan_read##size(void *ptr); \
void __tsan_read##size(void *ptr) \
{ \
- check_access(ptr, size, 0); \
+ check_access(ptr, size, 0, _RET_IP_); \
} \
EXPORT_SYMBOL(__tsan_read##size); \
void __tsan_unaligned_read##size(void *ptr) \
void __tsan_write##size(void *ptr); \
void __tsan_write##size(void *ptr) \
{ \
- check_access(ptr, size, KCSAN_ACCESS_WRITE); \
+ check_access(ptr, size, KCSAN_ACCESS_WRITE, _RET_IP_); \
} \
EXPORT_SYMBOL(__tsan_write##size); \
void __tsan_unaligned_write##size(void *ptr) \
void __tsan_read_write##size(void *ptr) \
{ \
check_access(ptr, size, \
- KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE); \
+ KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE, \
+ _RET_IP_); \
} \
EXPORT_SYMBOL(__tsan_read_write##size); \
void __tsan_unaligned_read_write##size(void *ptr) \
void __tsan_read_range(void *ptr, size_t size);
void __tsan_read_range(void *ptr, size_t size)
{
- check_access(ptr, size, 0);
+ check_access(ptr, size, 0, _RET_IP_);
}
EXPORT_SYMBOL(__tsan_read_range);
void __tsan_write_range(void *ptr, size_t size);
void __tsan_write_range(void *ptr, size_t size)
{
- check_access(ptr, size, KCSAN_ACCESS_WRITE);
+ check_access(ptr, size, KCSAN_ACCESS_WRITE, _RET_IP_);
}
EXPORT_SYMBOL(__tsan_write_range);
IS_ALIGNED((unsigned long)ptr, size); \
if (IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS) && is_atomic) \
return; \
- check_access(ptr, size, is_atomic ? KCSAN_ACCESS_ATOMIC : 0); \
+ check_access(ptr, size, is_atomic ? KCSAN_ACCESS_ATOMIC : 0, \
+ _RET_IP_); \
} \
EXPORT_SYMBOL(__tsan_volatile_read##size); \
void __tsan_unaligned_volatile_read##size(void *ptr) \
return; \
check_access(ptr, size, \
KCSAN_ACCESS_WRITE | \
- (is_atomic ? KCSAN_ACCESS_ATOMIC : 0)); \
+ (is_atomic ? KCSAN_ACCESS_ATOMIC : 0), \
+ _RET_IP_); \
} \
EXPORT_SYMBOL(__tsan_volatile_write##size); \
void __tsan_unaligned_volatile_write##size(void *ptr) \
u##bits __tsan_atomic##bits##_load(const u##bits *ptr, int memorder) \
{ \
if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \
- check_access(ptr, bits / BITS_PER_BYTE, KCSAN_ACCESS_ATOMIC); \
+ check_access(ptr, bits / BITS_PER_BYTE, KCSAN_ACCESS_ATOMIC, _RET_IP_); \
} \
return __atomic_load_n(ptr, memorder); \
} \
{ \
if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \
check_access(ptr, bits / BITS_PER_BYTE, \
- KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ATOMIC); \
+ KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ATOMIC, _RET_IP_); \
} \
__atomic_store_n(ptr, v, memorder); \
} \
if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \
check_access(ptr, bits / BITS_PER_BYTE, \
KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
- KCSAN_ACCESS_ATOMIC); \
+ KCSAN_ACCESS_ATOMIC, _RET_IP_); \
} \
return __atomic_##op##suffix(ptr, v, memorder); \
} \
if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \
check_access(ptr, bits / BITS_PER_BYTE, \
KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
- KCSAN_ACCESS_ATOMIC); \
+ KCSAN_ACCESS_ATOMIC, _RET_IP_); \
} \
return __atomic_compare_exchange_n(ptr, exp, val, weak, mo, fail_mo); \
} \
if (!IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS)) { \
check_access(ptr, bits / BITS_PER_BYTE, \
KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | \
- KCSAN_ACCESS_ATOMIC); \
+ KCSAN_ACCESS_ATOMIC, _RET_IP_); \
} \
__atomic_compare_exchange_n(ptr, &exp, val, 0, mo, fail_mo); \
return exp; \
* to be consumed by the reporting thread. No report is printed yet.
*/
void kcsan_report_set_info(const volatile void *ptr, size_t size, int access_type,
- int watchpoint_idx);
+ unsigned long ip, int watchpoint_idx);
/*
* The calling thread observed that the watchpoint it set up was hit and
* thread.
*/
void kcsan_report_known_origin(const volatile void *ptr, size_t size, int access_type,
- enum kcsan_value_change value_change, int watchpoint_idx,
- u64 old, u64 new, u64 mask);
+ unsigned long ip, enum kcsan_value_change value_change,
+ int watchpoint_idx, u64 old, u64 new, u64 mask);
/*
* No other thread was observed to race with the access, but the data value
* before and after the stall differs. Reports a race of "unknown origin".
*/
void kcsan_report_unknown_origin(const volatile void *ptr, size_t size, int access_type,
- u64 old, u64 new, u64 mask);
+ unsigned long ip, u64 old, u64 new, u64 mask);
#endif /* _KERNEL_KCSAN_KCSAN_H */
#include <linux/types.h>
#include <trace/events/printk.h>
+#define KCSAN_TEST_REQUIRES(test, cond) do { \
+ if (!(cond)) \
+ kunit_skip((test), "Test requires: " #cond); \
+} while (0)
+
#ifdef CONFIG_CC_HAS_TSAN_COMPOUND_READ_BEFORE_WRITE
#define __KCSAN_ACCESS_RW(alt) (KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE)
#else
"read-write" :
"write") :
"read");
+ const bool is_atomic = (ty & KCSAN_ACCESS_ATOMIC);
+ const bool is_scoped = (ty & KCSAN_ACCESS_SCOPED);
const char *const access_type_aux =
- (ty & KCSAN_ACCESS_ATOMIC) ?
- " (marked)" :
- ((ty & KCSAN_ACCESS_SCOPED) ? " (scoped)" : "");
+ (is_atomic && is_scoped) ? " (marked, scoped)"
+ : (is_atomic ? " (marked)"
+ : (is_scoped ? " (scoped)" : ""));
if (i == 1) {
/* Access 2 */
ASSERT_EXCLUSIVE_BITS(test_var, ~TEST_CHANGE_BITS);
}
-/* To check that scoped assertions do trigger anywhere in scope. */
+/*
+ * Scoped assertions do trigger anywhere in scope. However, the report should
+ * still only point at the start of the scope.
+ */
static noinline void test_enter_scope(void)
{
int x = 0;
__no_kcsan
static void test_novalue_change(struct kunit *test)
{
- const struct expect_report expect = {
+ const struct expect_report expect_rw = {
.access = {
{ test_kernel_write_nochange, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
{ test_kernel_read, &test_var, sizeof(test_var), 0 },
},
};
+ const struct expect_report expect_ww = {
+ .access = {
+ { test_kernel_write_nochange, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
+ { test_kernel_write_nochange, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
+ },
+ };
bool match_expect = false;
+ test_kernel_write_nochange(); /* Reset value. */
begin_test_checks(test_kernel_write_nochange, test_kernel_read);
do {
- match_expect = report_matches(&expect);
+ match_expect = report_matches(&expect_rw) || report_matches(&expect_ww);
} while (!end_test_checks(match_expect));
if (IS_ENABLED(CONFIG_KCSAN_REPORT_VALUE_CHANGE_ONLY))
KUNIT_EXPECT_FALSE(test, match_expect);
__no_kcsan
static void test_novalue_change_exception(struct kunit *test)
{
- const struct expect_report expect = {
+ const struct expect_report expect_rw = {
.access = {
{ test_kernel_write_nochange_rcu, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
{ test_kernel_read, &test_var, sizeof(test_var), 0 },
},
};
+ const struct expect_report expect_ww = {
+ .access = {
+ { test_kernel_write_nochange_rcu, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
+ { test_kernel_write_nochange_rcu, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
+ },
+ };
bool match_expect = false;
+ test_kernel_write_nochange_rcu(); /* Reset value. */
begin_test_checks(test_kernel_write_nochange_rcu, test_kernel_read);
do {
- match_expect = report_matches(&expect);
+ match_expect = report_matches(&expect_rw) || report_matches(&expect_ww);
} while (!end_test_checks(match_expect));
KUNIT_EXPECT_TRUE(test, match_expect);
}
};
bool match_expect = false;
- if (IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS))
- return;
+ KCSAN_TEST_REQUIRES(test, !IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS));
begin_test_checks(test_kernel_read, test_kernel_write_atomic);
do {
};
bool match_expect = false;
- if (IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS))
- return;
+ KCSAN_TEST_REQUIRES(test, !IS_ENABLED(CONFIG_KCSAN_IGNORE_ATOMICS));
begin_test_checks(test_kernel_read, test_kernel_atomic_rmw);
do {
{ test_kernel_write_nochange, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
},
};
- const struct expect_report expect_anywhere = {
+ const struct expect_report expect_inscope = {
.access = {
{ test_enter_scope, &test_var, sizeof(test_var), KCSAN_ACCESS_ASSERT | KCSAN_ACCESS_SCOPED },
{ test_kernel_write_nochange, &test_var, sizeof(test_var), KCSAN_ACCESS_WRITE },
},
};
bool match_expect_start = false;
- bool match_expect_anywhere = false;
+ bool match_expect_inscope = false;
begin_test_checks(test_kernel_assert_writer_scoped, test_kernel_write_nochange);
do {
match_expect_start |= report_matches(&expect_start);
- match_expect_anywhere |= report_matches(&expect_anywhere);
- } while (!end_test_checks(match_expect_start && match_expect_anywhere));
+ match_expect_inscope |= report_matches(&expect_inscope);
+ } while (!end_test_checks(match_expect_inscope));
KUNIT_EXPECT_TRUE(test, match_expect_start);
- KUNIT_EXPECT_TRUE(test, match_expect_anywhere);
+ KUNIT_EXPECT_FALSE(test, match_expect_inscope);
}
__no_kcsan
do {
match_expect_start |= report_matches(&expect_start1) || report_matches(&expect_start2);
match_expect_inscope |= report_matches(&expect_inscope);
- } while (!end_test_checks(match_expect_start && match_expect_inscope));
+ } while (!end_test_checks(match_expect_inscope));
KUNIT_EXPECT_TRUE(test, match_expect_start);
- KUNIT_EXPECT_TRUE(test, match_expect_inscope);
+ KUNIT_EXPECT_FALSE(test, match_expect_inscope);
}
/*
tracepoint_synchronize_unregister();
}
-late_initcall(kcsan_test_init);
+late_initcall_sync(kcsan_test_init);
module_exit(kcsan_test_exit);
MODULE_LICENSE("GPL v2");
#include <linux/debug_locks.h>
#include <linux/delay.h>
#include <linux/jiffies.h>
+#include <linux/kallsyms.h>
#include <linux/kernel.h>
#include <linux/lockdep.h>
#include <linux/preempt.h>
int access_type;
int task_pid;
int cpu_id;
+ unsigned long ip;
};
/*
return "write (scoped)";
case KCSAN_ACCESS_SCOPED | KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ATOMIC:
return "write (marked, scoped)";
+ case KCSAN_ACCESS_SCOPED | KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE:
+ return "read-write (scoped)";
+ case KCSAN_ACCESS_SCOPED | KCSAN_ACCESS_COMPOUND | KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ATOMIC:
+ return "read-write (marked, scoped)";
default:
BUG();
}
return skip;
}
+/*
+ * Skips to the first entry that matches the function of @ip, and then replaces
+ * that entry with @ip, returning the entries to skip.
+ */
+static int
+replace_stack_entry(unsigned long stack_entries[], int num_entries, unsigned long ip)
+{
+ unsigned long symbolsize, offset;
+ unsigned long target_func;
+ int skip;
+
+ if (kallsyms_lookup_size_offset(ip, &symbolsize, &offset))
+ target_func = ip - offset;
+ else
+ goto fallback;
+
+ for (skip = 0; skip < num_entries; ++skip) {
+ unsigned long func = stack_entries[skip];
+
+ if (!kallsyms_lookup_size_offset(func, &symbolsize, &offset))
+ goto fallback;
+ func -= offset;
+
+ if (func == target_func) {
+ stack_entries[skip] = ip;
+ return skip;
+ }
+ }
+
+fallback:
+ /* Should not happen; the resulting stack trace is likely misleading. */
+ WARN_ONCE(1, "Cannot find frame for %pS in stack trace", (void *)ip);
+ return get_stack_skipnr(stack_entries, num_entries);
+}
+
+static int
+sanitize_stack_entries(unsigned long stack_entries[], int num_entries, unsigned long ip)
+{
+ return ip ? replace_stack_entry(stack_entries, num_entries, ip) :
+ get_stack_skipnr(stack_entries, num_entries);
+}
+
/* Compares symbolized strings of addr1 and addr2. */
static int sym_strcmp(void *addr1, void *addr2)
{
static void print_report(enum kcsan_value_change value_change,
const struct access_info *ai,
- const struct other_info *other_info,
+ struct other_info *other_info,
u64 old, u64 new, u64 mask)
{
unsigned long stack_entries[NUM_STACK_ENTRIES] = { 0 };
int num_stack_entries = stack_trace_save(stack_entries, NUM_STACK_ENTRIES, 1);
- int skipnr = get_stack_skipnr(stack_entries, num_stack_entries);
+ int skipnr = sanitize_stack_entries(stack_entries, num_stack_entries, ai->ip);
unsigned long this_frame = stack_entries[skipnr];
unsigned long other_frame = 0;
int other_skipnr = 0; /* silence uninit warnings */
return;
if (other_info) {
- other_skipnr = get_stack_skipnr(other_info->stack_entries,
- other_info->num_stack_entries);
+ other_skipnr = sanitize_stack_entries(other_info->stack_entries,
+ other_info->num_stack_entries,
+ other_info->ai.ip);
other_frame = other_info->stack_entries[other_skipnr];
/* @value_change is only known for the other thread */
}
static struct access_info prepare_access_info(const volatile void *ptr, size_t size,
- int access_type)
+ int access_type, unsigned long ip)
{
return (struct access_info) {
.ptr = ptr,
.size = size,
.access_type = access_type,
.task_pid = in_task() ? task_pid_nr(current) : -1,
- .cpu_id = raw_smp_processor_id()
+ .cpu_id = raw_smp_processor_id(),
+ /* Only replace stack entry with @ip if scoped access. */
+ .ip = (access_type & KCSAN_ACCESS_SCOPED) ? ip : 0,
};
}
void kcsan_report_set_info(const volatile void *ptr, size_t size, int access_type,
- int watchpoint_idx)
+ unsigned long ip, int watchpoint_idx)
{
- const struct access_info ai = prepare_access_info(ptr, size, access_type);
+ const struct access_info ai = prepare_access_info(ptr, size, access_type, ip);
unsigned long flags;
kcsan_disable_current();
}
void kcsan_report_known_origin(const volatile void *ptr, size_t size, int access_type,
- enum kcsan_value_change value_change, int watchpoint_idx,
- u64 old, u64 new, u64 mask)
+ unsigned long ip, enum kcsan_value_change value_change,
+ int watchpoint_idx, u64 old, u64 new, u64 mask)
{
- const struct access_info ai = prepare_access_info(ptr, size, access_type);
+ const struct access_info ai = prepare_access_info(ptr, size, access_type, ip);
struct other_info *other_info = &other_infos[watchpoint_idx];
unsigned long flags = 0;
}
void kcsan_report_unknown_origin(const volatile void *ptr, size_t size, int access_type,
- u64 old, u64 new, u64 mask)
+ unsigned long ip, u64 old, u64 new, u64 mask)
{
- const struct access_info ai = prepare_access_info(ptr, size, access_type);
+ const struct access_info ai = prepare_access_info(ptr, size, access_type, ip);
unsigned long flags;
kcsan_disable_current();
#define ITERS_PER_TEST 2000
/* Test requirements. */
-static bool test_requires(void)
+static bool __init test_requires(void)
{
/* random should be initialized for the below tests */
return prandom_u32() + prandom_u32() != 0;
* Test watchpoint encode and decode: check that encoding some access's info,
* and then subsequent decode preserves the access's info.
*/
-static bool test_encode_decode(void)
+static bool __init test_encode_decode(void)
{
int i;
for (i = 0; i < ITERS_PER_TEST; ++i) {
size_t size = prandom_u32_max(MAX_ENCODABLE_SIZE) + 1;
bool is_write = !!prandom_u32_max(2);
+ unsigned long verif_masked_addr;
+ long encoded_watchpoint;
+ bool verif_is_write;
unsigned long addr;
+ size_t verif_size;
prandom_bytes(&addr, sizeof(addr));
if (addr < PAGE_SIZE)
if (WARN_ON(!check_encodable(addr, size)))
return false;
- /* Encode and decode */
- {
- const long encoded_watchpoint =
- encode_watchpoint(addr, size, is_write);
- unsigned long verif_masked_addr;
- size_t verif_size;
- bool verif_is_write;
-
- /* Check special watchpoints */
- if (WARN_ON(decode_watchpoint(
- INVALID_WATCHPOINT, &verif_masked_addr,
- &verif_size, &verif_is_write)))
- return false;
- if (WARN_ON(decode_watchpoint(
- CONSUMED_WATCHPOINT, &verif_masked_addr,
- &verif_size, &verif_is_write)))
- return false;
-
- /* Check decoding watchpoint returns same data */
- if (WARN_ON(!decode_watchpoint(
- encoded_watchpoint, &verif_masked_addr,
- &verif_size, &verif_is_write)))
- return false;
- if (WARN_ON(verif_masked_addr !=
- (addr & WATCHPOINT_ADDR_MASK)))
- goto fail;
- if (WARN_ON(verif_size != size))
- goto fail;
- if (WARN_ON(is_write != verif_is_write))
- goto fail;
-
- continue;
-fail:
- pr_err("%s fail: %s %zu bytes @ %lx -> encoded: %lx -> %s %zu bytes @ %lx\n",
- __func__, is_write ? "write" : "read", size,
- addr, encoded_watchpoint,
- verif_is_write ? "write" : "read", verif_size,
- verif_masked_addr);
+ encoded_watchpoint = encode_watchpoint(addr, size, is_write);
+
+ /* Check special watchpoints */
+ if (WARN_ON(decode_watchpoint(INVALID_WATCHPOINT, &verif_masked_addr, &verif_size, &verif_is_write)))
return false;
- }
+ if (WARN_ON(decode_watchpoint(CONSUMED_WATCHPOINT, &verif_masked_addr, &verif_size, &verif_is_write)))
+ return false;
+
+ /* Check decoding watchpoint returns same data */
+ if (WARN_ON(!decode_watchpoint(encoded_watchpoint, &verif_masked_addr, &verif_size, &verif_is_write)))
+ return false;
+ if (WARN_ON(verif_masked_addr != (addr & WATCHPOINT_ADDR_MASK)))
+ goto fail;
+ if (WARN_ON(verif_size != size))
+ goto fail;
+ if (WARN_ON(is_write != verif_is_write))
+ goto fail;
+
+ continue;
+fail:
+ pr_err("%s fail: %s %zu bytes @ %lx -> encoded: %lx -> %s %zu bytes @ %lx\n",
+ __func__, is_write ? "write" : "read", size, addr, encoded_watchpoint,
+ verif_is_write ? "write" : "read", verif_size, verif_masked_addr);
+ return false;
}
return true;
}
/* Test access matching function. */
-static bool test_matching_access(void)
+static bool __init test_matching_access(void)
{
if (WARN_ON(!matching_access(10, 1, 10, 1)))
return false;
if (kbuf->image->type == KEXEC_TYPE_CRASH)
return func(&crashk_res, kbuf);
+ /*
+ * Using MEMBLOCK_NONE will properly skip MEMBLOCK_DRIVER_MANAGED. See
+ * IORESOURCE_SYSRAM_DRIVER_MANAGED handling in
+ * locate_mem_hole_callback().
+ */
if (kbuf->top_down) {
for_each_free_mem_range_reverse(i, NUMA_NO_NODE, MEMBLOCK_NONE,
&mstart, &mend, NULL) {
* If thread is going to be bound on a particular cpu, give its node
* in @node, to get NUMA affinity for kthread stack, or else give NUMA_NO_NODE.
* When woken, the thread will run @threadfn() with @data as its
- * argument. @threadfn() can either call do_exit() directly if it is a
+ * argument. @threadfn() can either return directly if it is a
* standalone thread for which no one will call kthread_stop(), or
* return when 'kthread_should_stop()' is true (which means
* kthread_stop() has been called). The return value should be zero
* Is this the address of a static object:
*/
#ifdef __KERNEL__
+/*
+ * Check if an address is part of freed initmem. After initmem is freed,
+ * memory can be allocated from it, and such allocations would then have
+ * addresses within the range [_stext, _end].
+ */
+#ifndef arch_is_kernel_initmem_freed
+static int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+ if (system_state < SYSTEM_FREEING_INITMEM)
+ return 0;
+
+ return init_section_contains((void *)addr, 1);
+}
+#endif
+
static int static_obj(const void *obj)
{
unsigned long start = (unsigned long) &_stext,
if ((addr >= start) && (addr < end))
return 1;
- if (arch_is_kernel_data(addr))
- return 1;
-
/*
* in-kernel percpu var?
*/
static int validate_section_offset(struct load_info *info, Elf_Shdr *shdr)
{
+#if defined(CONFIG_64BIT)
+ unsigned long long secend;
+#else
unsigned long secend;
+#endif
/*
* Check for both overflow and offset/size being
Elf_Shdr *shdr, *strhdr;
int err;
- if (info->len < sizeof(*(info->hdr)))
- return -ENOEXEC;
+ if (info->len < sizeof(*(info->hdr))) {
+ pr_err("Invalid ELF header len %lu\n", info->len);
+ goto no_exec;
+ }
- if (memcmp(info->hdr->e_ident, ELFMAG, SELFMAG) != 0
- || info->hdr->e_type != ET_REL
- || !elf_check_arch(info->hdr)
- || info->hdr->e_shentsize != sizeof(Elf_Shdr))
- return -ENOEXEC;
+ if (memcmp(info->hdr->e_ident, ELFMAG, SELFMAG) != 0) {
+ pr_err("Invalid ELF header magic: != %s\n", ELFMAG);
+ goto no_exec;
+ }
+ if (info->hdr->e_type != ET_REL) {
+ pr_err("Invalid ELF header type: %u != %u\n",
+ info->hdr->e_type, ET_REL);
+ goto no_exec;
+ }
+ if (!elf_check_arch(info->hdr)) {
+ pr_err("Invalid architecture in ELF header: %u\n",
+ info->hdr->e_machine);
+ goto no_exec;
+ }
+ if (info->hdr->e_shentsize != sizeof(Elf_Shdr)) {
+ pr_err("Invalid ELF section header size\n");
+ goto no_exec;
+ }
/*
* e_shnum is 16 bits, and sizeof(Elf_Shdr) is
*/
if (info->hdr->e_shoff >= info->len
|| (info->hdr->e_shnum * sizeof(Elf_Shdr) >
- info->len - info->hdr->e_shoff))
- return -ENOEXEC;
+ info->len - info->hdr->e_shoff)) {
+ pr_err("Invalid ELF section header overflow\n");
+ goto no_exec;
+ }
info->sechdrs = (void *)info->hdr + info->hdr->e_shoff;
* Verify if the section name table index is valid.
*/
if (info->hdr->e_shstrndx == SHN_UNDEF
- || info->hdr->e_shstrndx >= info->hdr->e_shnum)
- return -ENOEXEC;
+ || info->hdr->e_shstrndx >= info->hdr->e_shnum) {
+ pr_err("Invalid ELF section name index: %d || e_shstrndx (%d) >= e_shnum (%d)\n",
+ info->hdr->e_shstrndx, info->hdr->e_shstrndx,
+ info->hdr->e_shnum);
+ goto no_exec;
+ }
strhdr = &info->sechdrs[info->hdr->e_shstrndx];
err = validate_section_offset(info, strhdr);
- if (err < 0)
+ if (err < 0) {
+ pr_err("Invalid ELF section hdr(type %u)\n", strhdr->sh_type);
return err;
+ }
/*
* The section name table must be NUL-terminated, as required
* strings in the section safe.
*/
info->secstrings = (void *)info->hdr + strhdr->sh_offset;
- if (info->secstrings[strhdr->sh_size - 1] != '\0')
- return -ENOEXEC;
+ if (info->secstrings[strhdr->sh_size - 1] != '\0') {
+ pr_err("ELF Spec violation: section name table isn't null terminated\n");
+ goto no_exec;
+ }
/*
* The code assumes that section 0 has a length of zero and
*/
if (info->sechdrs[0].sh_type != SHT_NULL
|| info->sechdrs[0].sh_size != 0
- || info->sechdrs[0].sh_addr != 0)
- return -ENOEXEC;
+ || info->sechdrs[0].sh_addr != 0) {
+ pr_err("ELF Spec violation: section 0 type(%d)!=SH_NULL or non-zero len or addr\n",
+ info->sechdrs[0].sh_type);
+ goto no_exec;
+ }
for (i = 1; i < info->hdr->e_shnum; i++) {
shdr = &info->sechdrs[i];
continue;
case SHT_SYMTAB:
if (shdr->sh_link == SHN_UNDEF
- || shdr->sh_link >= info->hdr->e_shnum)
- return -ENOEXEC;
+ || shdr->sh_link >= info->hdr->e_shnum) {
+ pr_err("Invalid ELF sh_link!=SHN_UNDEF(%d) or (sh_link(%d) >= hdr->e_shnum(%d)\n",
+ shdr->sh_link, shdr->sh_link,
+ info->hdr->e_shnum);
+ goto no_exec;
+ }
fallthrough;
default:
err = validate_section_offset(info, shdr);
}
return 0;
+
+no_exec:
+ return -ENOEXEC;
}
#define COPY_CHUNK_SIZE (16*PAGE_SIZE)
* sections.
*/
err = elf_validity_check(info);
- if (err) {
- pr_err("Module has invalid ELF structures\n");
+ if (err)
goto free_copy;
- }
/*
* Everything checks out, so set up the section info
}
/**
+ * pidfd_get_task() - Get the task associated with a pidfd
+ *
+ * @pidfd: pidfd for which to get the task
+ * @flags: flags associated with this pidfd
+ *
+ * Return the task associated with @pidfd. The function takes a reference on
+ * the returned task. The caller is responsible for releasing that reference.
+ *
+ * Currently, the process identified by @pidfd is always a thread-group leader.
+ * This restriction currently exists for all aspects of pidfds including pidfd
+ * creation (CLONE_PIDFD cannot be used with CLONE_THREAD) and pidfd polling
+ * (only supports thread group leaders).
+ *
+ * Return: On success, the task_struct associated with the pidfd.
+ * On error, a negative errno number will be returned.
+ */
+struct task_struct *pidfd_get_task(int pidfd, unsigned int *flags)
+{
+ unsigned int f_flags;
+ struct pid *pid;
+ struct task_struct *task;
+
+ pid = pidfd_get_pid(pidfd, &f_flags);
+ if (IS_ERR(pid))
+ return ERR_CAST(pid);
+
+ task = get_pid_task(pid, PIDTYPE_TGID);
+ put_pid(pid);
+ if (!task)
+ return ERR_PTR(-ESRCH);
+
+ *flags = f_flags;
+ return task;
+}
+
+/**
* pidfd_create() - Create a new pid file descriptor.
*
* @pid: struct pid that the pidfd will reference
return;
err_free_descs:
- memblock_free_ptr(new_descs, new_descs_size);
+ memblock_free(new_descs, new_descs_size);
err_free_log_buf:
- memblock_free_ptr(new_log_buf, new_log_buf_len);
+ memblock_free(new_log_buf, new_log_buf_len);
}
static bool __read_mostly ignore_loglevel;
preempt_enable();
}
+void printk_trigger_flush(void)
+{
+ defer_console_output();
+}
+
int vprintk_deferred(const char *fmt, va_list args)
{
int r;
#define DEFAULT_REBOOT_MODE
#endif
enum reboot_mode reboot_mode DEFAULT_REBOOT_MODE;
+EXPORT_SYMBOL_GPL(reboot_mode);
enum reboot_mode panic_reboot_mode = REBOOT_UNDEFINED;
/*
case LINUX_REBOOT_CMD_HALT:
kernel_halt();
do_exit(0);
- panic("cannot halt");
case LINUX_REBOOT_CMD_POWER_OFF:
kernel_power_off();
return p->sibling;
}
+static struct resource *next_resource_skip_children(struct resource *p)
+{
+ while (!p->sibling && p->parent)
+ p = p->parent;
+ return p->sibling;
+}
+
+#define for_each_resource(_root, _p, _skip_children) \
+ for ((_p) = (_root)->child; (_p); \
+ (_p) = (_skip_children) ? next_resource_skip_children(_p) : \
+ next_resource(_p))
+
static void *r_next(struct seq_file *m, void *v, loff_t *pos)
{
struct resource *p = v;
#endif
/*
- * check if an address is reserved in the iomem resource tree
- * returns true if reserved, false if not reserved.
+ * Check if an address is exclusive to the kernel and must not be mapped to
+ * user space, for example, via /dev/mem.
+ *
+ * Returns true if exclusive to the kernel, otherwise returns false.
*/
bool iomem_is_exclusive(u64 addr)
{
- struct resource *p = &iomem_resource;
- bool err = false;
- loff_t l;
+ const unsigned int exclusive_system_ram = IORESOURCE_SYSTEM_RAM |
+ IORESOURCE_EXCLUSIVE;
+ bool skip_children = false, err = false;
int size = PAGE_SIZE;
-
- if (!strict_iomem_checks)
- return false;
+ struct resource *p;
addr = addr & PAGE_MASK;
read_lock(&resource_lock);
- for (p = p->child; p ; p = r_next(NULL, p, &l)) {
- /*
- * We can probably skip the resources without
- * IORESOURCE_IO attribute?
- */
+ for_each_resource(&iomem_resource, p, skip_children) {
if (p->start >= addr + size)
break;
- if (p->end < addr)
+ if (p->end < addr) {
+ skip_children = true;
continue;
+ }
+ skip_children = false;
+
+ /*
+ * IORESOURCE_SYSTEM_RAM resources are exclusive if
+ * IORESOURCE_EXCLUSIVE is set, even if they
+ * are not busy and even if "iomem=relaxed" is set. The
+ * responsible driver dynamically adds/removes system RAM within
+ * such an area and uncontrolled access is dangerous.
+ */
+ if ((p->flags & exclusive_system_ram) == exclusive_system_ram) {
+ err = true;
+ break;
+ }
+
/*
* A resource is exclusive if IORESOURCE_EXCLUSIVE is set
* or CONFIG_IO_STRICT_DEVMEM is enabled and the
* resource is busy.
*/
- if ((p->flags & IORESOURCE_BUSY) == 0)
+ if (!strict_iomem_checks || !(p->flags & IORESOURCE_BUSY))
continue;
if (IS_ENABLED(CONFIG_IO_STRICT_DEVMEM)
|| p->flags & IORESOURCE_EXCLUSIVE) {
ag->tg->rt_se = NULL;
ag->tg->rt_rq = NULL;
#endif
- sched_offline_group(ag->tg);
+ sched_release_group(ag->tg);
sched_destroy_group(ag->tg);
}
bool cpus_share_cache(int this_cpu, int that_cpu)
{
+ if (this_cpu == that_cpu)
+ return true;
+
return per_cpu(sd_llc_id, this_cpu) == per_cpu(sd_llc_id, that_cpu);
}
static void __init preempt_dynamic_init(void)
{
if (preempt_dynamic_mode == preempt_dynamic_undefined) {
- if (IS_ENABLED(CONFIG_PREEMPT_NONE_BEHAVIOUR)) {
+ if (IS_ENABLED(CONFIG_PREEMPT_NONE)) {
sched_dynamic_update(preempt_dynamic_none);
- } else if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY_BEHAVIOUR)) {
+ } else if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) {
sched_dynamic_update(preempt_dynamic_voluntary);
} else {
/* Default static call setting, nothing to do */
- WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPT_BEHAVIOUR));
+ WARN_ON_ONCE(!IS_ENABLED(CONFIG_PREEMPT));
preempt_dynamic_mode = preempt_dynamic_full;
pr_info("Dynamic Preempt: full\n");
}
kmem_cache_free(task_group_cache, tg);
}
+static void sched_free_group_rcu(struct rcu_head *rcu)
+{
+ sched_free_group(container_of(rcu, struct task_group, rcu));
+}
+
+static void sched_unregister_group(struct task_group *tg)
+{
+ unregister_fair_sched_group(tg);
+ unregister_rt_sched_group(tg);
+ /*
+ * We have to wait for yet another RCU grace period to expire, as
+ * print_cfs_stats() might run concurrently.
+ */
+ call_rcu(&tg->rcu, sched_free_group_rcu);
+}
+
/* allocate runqueue etc for a new task group */
struct task_group *sched_create_group(struct task_group *parent)
{
}
/* rcu callback to free various structures associated with a task group */
-static void sched_free_group_rcu(struct rcu_head *rhp)
+static void sched_unregister_group_rcu(struct rcu_head *rhp)
{
/* Now it should be safe to free those cfs_rqs: */
- sched_free_group(container_of(rhp, struct task_group, rcu));
+ sched_unregister_group(container_of(rhp, struct task_group, rcu));
}
void sched_destroy_group(struct task_group *tg)
{
/* Wait for possible concurrent references to cfs_rqs complete: */
- call_rcu(&tg->rcu, sched_free_group_rcu);
+ call_rcu(&tg->rcu, sched_unregister_group_rcu);
}
-void sched_offline_group(struct task_group *tg)
+void sched_release_group(struct task_group *tg)
{
unsigned long flags;
- /* End participation in shares distribution: */
- unregister_fair_sched_group(tg);
-
+ /*
+ * Unlink first, to avoid walk_tg_tree_from() from finding us (via
+ * sched_cfs_period_timer()).
+ *
+ * For this to be effective, we have to wait for all pending users of
+ * this task group to leave their RCU critical section to ensure no new
+ * user will see our dying task group any more. Specifically ensure
+ * that tg_unthrottle_up() won't add decayed cfs_rq's to it.
+ *
+ * We therefore defer calling unregister_fair_sched_group() to
+ * sched_unregister_group() which is guarantied to get called only after the
+ * current RCU grace period has expired.
+ */
spin_lock_irqsave(&task_group_lock, flags);
list_del_rcu(&tg->list);
list_del_rcu(&tg->siblings);
{
struct task_group *tg = css_tg(css);
- sched_offline_group(tg);
+ sched_release_group(tg);
}
static void cpu_cgroup_css_free(struct cgroup_subsys_state *css)
/*
* Relies on the RCU grace period between css_released() and this.
*/
- sched_free_group(tg);
+ sched_unregister_group(tg);
}
/*
if (!static_branch_likely(&sched_smt_present))
return -ENODEV;
+ BUILD_BUG_ON(PR_SCHED_CORE_SCOPE_THREAD != PIDTYPE_PID);
+ BUILD_BUG_ON(PR_SCHED_CORE_SCOPE_THREAD_GROUP != PIDTYPE_TGID);
+ BUILD_BUG_ON(PR_SCHED_CORE_SCOPE_PROCESS_GROUP != PIDTYPE_PGID);
+
if (type > PIDTYPE_PGID || cmd >= PR_SCHED_CORE_MAX || pid < 0 ||
(cmd != PR_SCHED_CORE_GET && uaddr))
return -EINVAL;
{
int i;
- destroy_cfs_bandwidth(tg_cfs_bandwidth(tg));
-
for_each_possible_cpu(i) {
if (tg->cfs_rq)
kfree(tg->cfs_rq[i]);
struct rq *rq;
int cpu;
+ destroy_cfs_bandwidth(tg_cfs_bandwidth(tg));
+
for_each_possible_cpu(cpu) {
if (tg->se[cpu])
remove_entity_load_avg(tg->se[cpu]);
return rt_rq->rq;
}
-void free_rt_sched_group(struct task_group *tg)
+void unregister_rt_sched_group(struct task_group *tg)
{
- int i;
-
if (tg->rt_se)
destroy_rt_bandwidth(&tg->rt_bandwidth);
+}
+
+void free_rt_sched_group(struct task_group *tg)
+{
+ int i;
+
for_each_possible_cpu(i) {
if (tg->rt_rq)
kfree(tg->rt_rq[i]);
return &rq->rt;
}
+void unregister_rt_sched_group(struct task_group *tg) { }
+
void free_rt_sched_group(struct task_group *tg) { }
int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent)
extern void start_cfs_bandwidth(struct cfs_bandwidth *cfs_b);
extern void unthrottle_cfs_rq(struct cfs_rq *cfs_rq);
+extern void unregister_rt_sched_group(struct task_group *tg);
extern void free_rt_sched_group(struct task_group *tg);
extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
extern void init_tg_rt_entry(struct task_group *tg, struct rt_rq *rt_rq,
extern void sched_online_group(struct task_group *tg,
struct task_group *parent);
extern void sched_destroy_group(struct task_group *tg);
-extern void sched_offline_group(struct task_group *tg);
+extern void sched_release_group(struct task_group *tg);
extern void sched_move_task(struct task_struct *tsk);
int sched_max_numa_distance;
static int *sched_domains_numa_distance;
static struct cpumask ***sched_domains_numa_masks;
-int __read_mostly node_reclaim_distance = RECLAIM_DISTANCE;
static unsigned long __read_mostly *sched_numa_onlined_nodes;
#endif
return ret;
}
+enum sig_handler {
+ HANDLER_CURRENT, /* If reachable use the current handler */
+ HANDLER_SIG_DFL, /* Always use SIG_DFL handler semantics */
+ HANDLER_EXIT, /* Only visible as the process exit code */
+};
+
/*
* Force a signal that the process can't ignore: if necessary
* we unblock the signal and change any SIG_IGN to SIG_DFL.
* that is why we also clear SIGNAL_UNKILLABLE.
*/
static int
-force_sig_info_to_task(struct kernel_siginfo *info, struct task_struct *t, bool sigdfl)
+force_sig_info_to_task(struct kernel_siginfo *info, struct task_struct *t,
+ enum sig_handler handler)
{
unsigned long int flags;
int ret, blocked, ignored;
action = &t->sighand->action[sig-1];
ignored = action->sa.sa_handler == SIG_IGN;
blocked = sigismember(&t->blocked, sig);
- if (blocked || ignored || sigdfl) {
+ if (blocked || ignored || (handler != HANDLER_CURRENT)) {
action->sa.sa_handler = SIG_DFL;
+ if (handler == HANDLER_EXIT)
+ action->sa.sa_flags |= SA_IMMUTABLE;
if (blocked) {
sigdelset(&t->blocked, sig);
recalc_sigpending_and_wake(t);
int force_sig_info(struct kernel_siginfo *info)
{
- return force_sig_info_to_task(info, current, false);
+ return force_sig_info_to_task(info, current, HANDLER_CURRENT);
}
/*
}
EXPORT_SYMBOL(force_sig);
+void force_fatal_sig(int sig)
+{
+ struct kernel_siginfo info;
+
+ clear_siginfo(&info);
+ info.si_signo = sig;
+ info.si_errno = 0;
+ info.si_code = SI_KERNEL;
+ info.si_pid = 0;
+ info.si_uid = 0;
+ force_sig_info_to_task(&info, current, HANDLER_SIG_DFL);
+}
+
+void force_exit_sig(int sig)
+{
+ struct kernel_siginfo info;
+
+ clear_siginfo(&info);
+ info.si_signo = sig;
+ info.si_errno = 0;
+ info.si_code = SI_KERNEL;
+ info.si_pid = 0;
+ info.si_uid = 0;
+ force_sig_info_to_task(&info, current, HANDLER_EXIT);
+}
+
/*
* When things go south during signal handling, we
* will force a SIGSEGV. And if the signal that caused
*/
void force_sigsegv(int sig)
{
- struct task_struct *p = current;
-
- if (sig == SIGSEGV) {
- unsigned long flags;
- spin_lock_irqsave(&p->sighand->siglock, flags);
- p->sighand->action[sig - 1].sa.sa_handler = SIG_DFL;
- spin_unlock_irqrestore(&p->sighand->siglock, flags);
- }
- force_sig(SIGSEGV);
+ if (sig == SIGSEGV)
+ force_fatal_sig(SIGSEGV);
+ else
+ force_sig(SIGSEGV);
}
int force_sig_fault_to_task(int sig, int code, void __user *addr
info.si_flags = flags;
info.si_isr = isr;
#endif
- return force_sig_info_to_task(&info, t, false);
+ return force_sig_info_to_task(&info, t, HANDLER_CURRENT);
}
int force_sig_fault(int sig, int code, void __user *addr
info.si_errno = reason;
info.si_arch = syscall_get_arch(current);
info.si_syscall = syscall;
- return force_sig_info_to_task(&info, current, force_coredump);
+ return force_sig_info_to_task(&info, current,
+ force_coredump ? HANDLER_EXIT : HANDLER_CURRENT);
}
/* For the crazy architectures that include trap information in
if (!signr)
break; /* will return 0 */
- if (unlikely(current->ptrace) && signr != SIGKILL) {
+ if (unlikely(current->ptrace) && (signr != SIGKILL) &&
+ !(sighand->action[signr -1].sa.sa_flags & SA_IMMUTABLE)) {
signr = ptrace_signal(signr, &ksig->info);
if (!signr)
continue;
k = &p->sighand->action[sig-1];
spin_lock_irq(&p->sighand->siglock);
+ if (k->sa.sa_flags & SA_IMMUTABLE) {
+ spin_unlock_irq(&p->sighand->siglock);
+ return -EINVAL;
+ }
if (oact)
*oact = *k;
#include <linux/export.h>
#include <linux/kallsyms.h>
#include <linux/stacktrace.h>
+#include <linux/interrupt.h>
/**
* stack_trace_print - Print the entries in the stack trace
#endif /* CONFIG_USER_STACKTRACE_SUPPORT */
#endif /* !CONFIG_ARCH_STACKWALK */
+
+static inline bool in_irqentry_text(unsigned long ptr)
+{
+ return (ptr >= (unsigned long)&__irqentry_text_start &&
+ ptr < (unsigned long)&__irqentry_text_end) ||
+ (ptr >= (unsigned long)&__softirqentry_text_start &&
+ ptr < (unsigned long)&__softirqentry_text_end);
+}
+
+/**
+ * filter_irq_stacks - Find first IRQ stack entry in trace
+ * @entries: Pointer to stack trace array
+ * @nr_entries: Number of entries in the storage array
+ *
+ * Return: Number of trace entries until IRQ stack starts.
+ */
+unsigned int filter_irq_stacks(unsigned long *entries, unsigned int nr_entries)
+{
+ unsigned int i;
+
+ for (i = 0; i < nr_entries; i++) {
+ if (in_irqentry_text(entries[i])) {
+ /* Include the irqentry function into the stack. */
+ return i + 1;
+ }
+ }
+ return nr_entries;
+}
+EXPORT_SYMBOL_GPL(filter_irq_stacks);
}
/*
+ * Clear existing posix CPU timers task work.
+ */
+void clear_posix_cputimers_work(struct task_struct *p)
+{
+ /*
+ * A copied work entry from the old task is not meaningful, clear it.
+ * N.B. init_task_work will not do this.
+ */
+ memset(&p->posix_cputimers_work.work, 0,
+ sizeof(p->posix_cputimers_work.work));
+ init_task_work(&p->posix_cputimers_work.work,
+ posix_cpu_timers_work);
+ p->posix_cputimers_work.scheduled = false;
+}
+
+/*
* Initialize posix CPU timers task work in init task. Out of line to
* keep the callback static and to avoid header recursion hell.
*/
void __init posix_cputimers_init_work(void)
{
- init_task_work(¤t->posix_cputimers_work.work,
- posix_cpu_timers_work);
+ clear_posix_cputimers_work(current);
}
/*
return &bpf_ktime_get_ns_proto;
case BPF_FUNC_ktime_get_boot_ns:
return &bpf_ktime_get_boot_ns_proto;
- case BPF_FUNC_ktime_get_coarse_ns:
- return &bpf_ktime_get_coarse_ns_proto;
case BPF_FUNC_tail_call:
return &bpf_tail_call_proto;
case BPF_FUNC_get_current_pid_tgid:
if (!ftrace_enabled && (ops->flags & FTRACE_OPS_FL_PERMANENT))
return -EBUSY;
- if (!core_kernel_data((unsigned long)ops))
+ if (!is_kernel_core_data((unsigned long)ops))
ops->flags |= FTRACE_OPS_FL_DYNAMIC;
add_ftrace_ops(&ftrace_ops_list, ops);
}
}
+ mutex_unlock(&ftrace_lock);
+
/* Removing the tmp_ops will add the updated direct callers to the functions */
unregister_ftrace_function(&tmp_ops);
- mutex_unlock(&ftrace_lock);
out_direct:
mutex_unlock(&direct_mutex);
return err;
struct ring_buffer_per_cpu *cpu_buffer;
int cpu;
+ /* prevent another thread from changing buffer sizes */
+ mutex_lock(&buffer->mutex);
+
for_each_buffer_cpu(buffer, cpu) {
cpu_buffer = buffer->buffers[cpu];
atomic_dec(&cpu_buffer->record_disabled);
atomic_dec(&cpu_buffer->resize_disabled);
}
+
+ mutex_unlock(&buffer->mutex);
}
EXPORT_SYMBOL_GPL(ring_buffer_reset);
iter->fmt[i] = '\0';
trace_seq_vprintf(&iter->seq, iter->fmt, ap);
+ /*
+ * If iter->seq is full, the above call no longer guarantees
+ * that ap is in sync with fmt processing, and further calls
+ * to va_arg() can return wrong positional arguments.
+ *
+ * Ensure that ap is no longer used in this case.
+ */
+ if (iter->seq.full) {
+ p = "";
+ break;
+ }
+
if (star)
len = va_arg(ap, int);
cnt = PAGE_SIZE - 1;
/* reset all but tr, trace, and overruns */
- memset(&iter->seq, 0,
- sizeof(struct trace_iterator) -
- offsetof(struct trace_iterator, seq));
+ memset_startat(iter, 0, seq);
cpumask_clear(iter->started);
trace_seq_init(&iter->seq);
iter->pos = -1;
if (!hist_field->type)
goto free;
- if (field->filter_type == FILTER_STATIC_STRING)
+ if (field->filter_type == FILTER_STATIC_STRING) {
hist_field->fn = hist_field_string;
- else if (field->filter_type == FILTER_DYN_STRING)
+ hist_field->size = field->size;
+ } else if (field->filter_type == FILTER_DYN_STRING)
hist_field->fn = hist_field_dynstring;
else
hist_field->fn = hist_field_pstring;
/* Split the expression string at the root operator */
if (!sep)
- goto free;
+ return ERR_PTR(-EINVAL);
+
*sep = '\0';
operand1_str = str;
str = sep+1;
- if (!operand1_str || !str)
- goto free;
+ /* Binary operator requires both operands */
+ if (*operand1_str == '\0' || *str == '\0')
+ return ERR_PTR(-EINVAL);
operand_flags = 0;
/* LHS of string is an expression e.g. a+b in a+b+c */
operand1 = parse_expr(hist_data, file, operand1_str, operand_flags, NULL, n_subexprs);
- if (IS_ERR(operand1)) {
- ret = PTR_ERR(operand1);
- operand1 = NULL;
- goto free;
- }
+ if (IS_ERR(operand1))
+ return ERR_CAST(operand1);
+
if (operand1->flags & HIST_FIELD_FL_STRING) {
hist_err(file->tr, HIST_ERR_INVALID_STR_OPERAND, errpos(operand1_str));
ret = -EINVAL;
- goto free;
+ goto free_op1;
}
/* RHS of string is another expression e.g. c in a+b+c */
operand2 = parse_expr(hist_data, file, str, operand_flags, NULL, n_subexprs);
if (IS_ERR(operand2)) {
ret = PTR_ERR(operand2);
- operand2 = NULL;
- goto free;
+ goto free_op1;
}
if (operand2->flags & HIST_FIELD_FL_STRING) {
hist_err(file->tr, HIST_ERR_INVALID_STR_OPERAND, errpos(str));
ret = -EINVAL;
- goto free;
+ goto free_operands;
}
switch (field_op) {
break;
default:
ret = -EINVAL;
- goto free;
+ goto free_operands;
}
ret = check_expr_operands(file->tr, operand1, operand2, &var1, &var2);
if (ret)
- goto free;
+ goto free_operands;
operand_flags = var1 ? var1->flags : operand1->flags;
operand2_flags = var2 ? var2->flags : operand2->flags;
expr = create_hist_field(hist_data, NULL, flags, var_name);
if (!expr) {
ret = -ENOMEM;
- goto free;
+ goto free_operands;
}
operand1->read_once = true;
operand2->read_once = true;
+ /* The operands are now owned and free'd by 'expr' */
expr->operands[0] = operand1;
expr->operands[1] = operand2;
if (!divisor) {
hist_err(file->tr, HIST_ERR_DIVISION_BY_ZERO, errpos(str));
ret = -EDOM;
- goto free;
+ goto free_expr;
}
/*
expr->type = kstrdup_const(operand1->type, GFP_KERNEL);
if (!expr->type) {
ret = -ENOMEM;
- goto free;
+ goto free_expr;
}
expr->name = expr_str(expr, 0);
}
return expr;
-free:
- destroy_hist_field(operand1, 0);
+
+free_operands:
destroy_hist_field(operand2, 0);
- destroy_hist_field(expr, 0);
+free_op1:
+ destroy_hist_field(operand1, 0);
+ return ERR_PTR(ret);
+free_expr:
+ destroy_hist_field(expr, 0);
return ERR_PTR(ret);
}
if (val->flags & HIST_FIELD_FL_STRING) {
char *str = elt_data->field_var_str[j++];
char *val_str = (char *)(uintptr_t)var_val;
+ unsigned int size;
- strscpy(str, val_str, STR_VAR_LEN_MAX);
+ size = min(val->size, STR_VAR_LEN_MAX);
+ strscpy(str, val_str, size);
var_val = (u64)(uintptr_t)str;
}
tracing_map_set_var(elt, var_idx, var_val);
if (hist_field->flags & HIST_FIELD_FL_STRING) {
unsigned int str_start, var_str_idx, idx;
char *str, *val_str;
+ unsigned int size;
str_start = hist_data->n_field_var_str +
hist_data->n_save_var_str;
str = elt_data->field_var_str[idx];
val_str = (char *)(uintptr_t)hist_val;
- strscpy(str, val_str, STR_VAR_LEN_MAX);
+
+ size = min(hist_field->size, STR_VAR_LEN_MAX);
+ strscpy(str, val_str, size);
hist_val = (u64)(uintptr_t)str;
}
struct list_head list;
struct trace_array *tr;
};
-struct list_head osnoise_instances;
+
+static struct list_head osnoise_instances;
static bool osnoise_has_registered_instances(void)
{
* the rest of the math is done in xacct_add_tsk.
*/
tsk->acct_rss_mem1 += delta * get_mm_rss(tsk->mm) >> 10;
- tsk->acct_vm_mem1 += delta * tsk->mm->total_vm >> 10;
+ tsk->acct_vm_mem1 += delta * READ_ONCE(tsk->mm->total_vm) >> 10;
}
/**
struct worker_pool *pool = pwq->pool;
/* record the work call stack in order to print it in KASAN reports */
- kasan_record_aux_stack(work);
+ kasan_record_aux_stack_noalloc(work);
/* we own @work, set data and link */
set_work_pwq(work, pwq, extra_flags);
config MEMORY_NOTIFIER_ERROR_INJECT
tristate "Memory hotplug notifier error injection module"
- depends on MEMORY_HOTPLUG_SPARSE && NOTIFIER_ERROR_INJECTION
+ depends on MEMORY_HOTPLUG && NOTIFIER_ERROR_INJECTION
help
This option provides the ability to inject artificial errors to
memory hotplug notifier chain callbacks. It is controlled through
if KFENCE
-config KFENCE_STATIC_KEYS
- bool "Use static keys to set up allocations"
- default y
- depends on JUMP_LABEL # To ensure performance, require jump labels
- help
- Use static keys (static branches) to set up KFENCE allocations. Using
- static keys is normally recommended, because it avoids a dynamic
- branch in the allocator's fast path. However, with very low sample
- intervals, or on systems that do not support jump labels, a dynamic
- branch may still be an acceptable performance trade-off.
-
config KFENCE_SAMPLE_INTERVAL
int "Default sample interval in milliseconds"
default 100
pages are required; with one containing the object and two adjacent
ones used as guard pages.
+config KFENCE_STATIC_KEYS
+ bool "Use static keys to set up allocations" if EXPERT
+ depends on JUMP_LABEL
+ help
+ Use static keys (static branches) to set up KFENCE allocations. This
+ option is only recommended when using very large sample intervals, or
+ performance has carefully been evaluated with this option.
+
+ Using static keys comes with trade-offs that need to be carefully
+ evaluated given target workloads and system architectures. Notably,
+ enabling and disabling static keys invoke IPI broadcasts, the latency
+ and impact of which is much harder to predict than a dynamic branch.
+
+ Say N if you are unsure.
+
config KFENCE_STRESS_TEST_FAULTS
int "Stress testing of fault handling and error reporting" if EXPERT
default 0
static inline void __init xbc_free_mem(void *addr, size_t size)
{
- memblock_free_ptr(addr, size);
+ memblock_free(addr, size);
}
#else /* !__KERNEL__ */
*/
void __init free_bootmem_cpumask_var(cpumask_var_t mask)
{
- memblock_free_early(__pa(mask), cpumask_size());
+ memblock_free(mask, cpumask_size());
}
#endif
#ifdef STATIC
# define UNZSTD_PREBOOT
# include "xxhash.c"
-# include "zstd/entropy_common.c"
-# include "zstd/fse_decompress.c"
-# include "zstd/huf_decompress.c"
-# include "zstd/zstd_common.c"
-# include "zstd/decompress.c"
+# include "zstd/decompress_sources.h"
#endif
#include <linux/decompress/mm.h>
static int INIT handle_zstd_error(size_t ret, void (*error)(char *x))
{
- const int err = ZSTD_getErrorCode(ret);
+ const zstd_error_code err = zstd_get_error_code(ret);
- if (!ZSTD_isError(ret))
+ if (!zstd_is_error(ret))
return 0;
+ /*
+ * zstd_get_error_name() cannot be used because error takes a char *
+ * not a const char *
+ */
switch (err) {
case ZSTD_error_memory_allocation:
error("ZSTD decompressor ran out of memory");
long out_len, long *in_pos,
void (*error)(char *x))
{
- const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+ const size_t wksp_size = zstd_dctx_workspace_bound();
void *wksp = large_malloc(wksp_size);
- ZSTD_DCtx *dctx = ZSTD_initDCtx(wksp, wksp_size);
+ zstd_dctx *dctx = zstd_init_dctx(wksp, wksp_size);
int err;
size_t ret;
if (dctx == NULL) {
- error("Out of memory while allocating ZSTD_DCtx");
+ error("Out of memory while allocating zstd_dctx");
err = -1;
goto out;
}
/*
* Find out how large the frame actually is, there may be junk at
- * the end of the frame that ZSTD_decompressDCtx() can't handle.
+ * the end of the frame that zstd_decompress_dctx() can't handle.
*/
- ret = ZSTD_findFrameCompressedSize(in_buf, in_len);
+ ret = zstd_find_frame_compressed_size(in_buf, in_len);
err = handle_zstd_error(ret, error);
if (err)
goto out;
in_len = (long)ret;
- ret = ZSTD_decompressDCtx(dctx, out_buf, out_len, in_buf, in_len);
+ ret = zstd_decompress_dctx(dctx, out_buf, out_len, in_buf, in_len);
err = handle_zstd_error(ret, error);
if (err)
goto out;
long *in_pos,
void (*error)(char *x))
{
- ZSTD_inBuffer in;
- ZSTD_outBuffer out;
- ZSTD_frameParams params;
+ zstd_in_buffer in;
+ zstd_out_buffer out;
+ zstd_frame_header header;
void *in_allocated = NULL;
void *out_allocated = NULL;
void *wksp = NULL;
size_t wksp_size;
- ZSTD_DStream *dstream;
+ zstd_dstream *dstream;
int err;
size_t ret;
out.size = out_len;
/*
- * We need to know the window size to allocate the ZSTD_DStream.
+ * We need to know the window size to allocate the zstd_dstream.
* Since we are streaming, we need to allocate a buffer for the sliding
* window. The window size varies from 1 KB to ZSTD_WINDOWSIZE_MAX
* (8 MB), so it is important to use the actual value so as not to
* waste memory when it is smaller.
*/
- ret = ZSTD_getFrameParams(¶ms, in.src, in.size);
+ ret = zstd_get_frame_header(&header, in.src, in.size);
err = handle_zstd_error(ret, error);
if (err)
goto out;
err = -1;
goto out;
}
- if (params.windowSize > ZSTD_WINDOWSIZE_MAX) {
+ if (header.windowSize > ZSTD_WINDOWSIZE_MAX) {
error("ZSTD-compressed data has too large a window size");
err = -1;
goto out;
}
/*
- * Allocate the ZSTD_DStream now that we know how much memory is
+ * Allocate the zstd_dstream now that we know how much memory is
* required.
*/
- wksp_size = ZSTD_DStreamWorkspaceBound(params.windowSize);
+ wksp_size = zstd_dstream_workspace_bound(header.windowSize);
wksp = large_malloc(wksp_size);
- dstream = ZSTD_initDStream(params.windowSize, wksp, wksp_size);
+ dstream = zstd_init_dstream(header.windowSize, wksp, wksp_size);
if (dstream == NULL) {
error("Out of memory while allocating ZSTD_DStream");
err = -1;
in.size = in_len;
}
/* Returns zero when the frame is complete. */
- ret = ZSTD_decompressStream(dstream, &out, &in);
+ ret = zstd_decompress_stream(dstream, &out, &in);
err = handle_zstd_error(ret, error);
if (err)
goto out;
touch_softlockup_watchdog();
}
+ /*
+ * Force flush any remote buffers that might be stuck in IRQ context
+ * and therefore could not run their irq_work.
+ */
+ printk_trigger_flush();
+
clear_bit_unlock(0, &backtrace_flag);
put_cpu();
}
ifeq ($(CONFIG_ALTIVEC),y)
altivec_flags := -maltivec $(call cc-option,-mabi=altivec)
+# Enable <altivec.h>
+altivec_flags += -isystem $(shell $(CC) -print-file-name=include)
ifdef CONFIG_CC_IS_CLANG
# clang ppc port does not yet support -maltivec when -msoft-float is
# ARM/NEON intrinsics in a non C99-compliant environment (such as the kernel)
ifeq ($(CONFIG_KERNEL_MODE_NEON),y)
NEON_FLAGS := -ffreestanding
+# Enable <arm_neon.h>
+NEON_FLAGS += -isystem $(shell $(CC) -print-file-name=include)
ifeq ($(ARCH),arm)
NEON_FLAGS += -march=armv7-a -mfloat-abi=softfp -mfpu=neon
endif
* stops @miter.
*
* Context:
- * Don't care if @miter is stopped, or not proceeded yet.
- * Otherwise, preemption disabled if the SG_MITER_ATOMIC is set.
+ * Don't care.
*
* Returns:
* true if @miter contains the valid mapping. false if end of sg
* @miter->addr and @miter->length point to the current mapping.
*
* Context:
- * Preemption disabled if SG_MITER_ATOMIC. Preemption must stay disabled
- * till @miter is stopped. May sleep if !SG_MITER_ATOMIC.
+ * May sleep if !SG_MITER_ATOMIC.
*
* Returns:
* true if @miter contains the next mapping. false if end of sg
* need to be released during iteration.
*
* Context:
- * Preemption disabled if the SG_MITER_ATOMIC is set. Don't care
- * otherwise.
+ * Don't care otherwise.
*/
void sg_miter_stop(struct sg_mapping_iter *miter)
{
flush_dcache_page(miter->page);
if (miter->__flags & SG_MITER_ATOMIC) {
- WARN_ON_ONCE(preemptible());
+ WARN_ON_ONCE(!pagefault_disabled());
kunmap_atomic(miter->addr);
} else
kunmap(miter->page);
*/
#include <linux/gfp.h>
-#include <linux/interrupt.h>
#include <linux/jhash.h>
#include <linux/kernel.h>
#include <linux/mm.h>
}
/* Allocation of a new stack in raw storage */
-static struct stack_record *depot_alloc_stack(unsigned long *entries, int size,
- u32 hash, void **prealloc, gfp_t alloc_flags)
+static struct stack_record *
+depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc)
{
struct stack_record *stack;
size_t required_size = struct_size(stack, entries, size);
}
/**
+ * stack_depot_snprint - print stack entries from a depot into a buffer
+ *
+ * @handle: Stack depot handle which was returned from
+ * stack_depot_save().
+ * @buf: Pointer to the print buffer
+ *
+ * @size: Size of the print buffer
+ *
+ * @spaces: Number of leading spaces to print
+ *
+ * Return: Number of bytes printed.
+ */
+int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size,
+ int spaces)
+{
+ unsigned long *entries;
+ unsigned int nr_entries;
+
+ nr_entries = stack_depot_fetch(handle, &entries);
+ return nr_entries ? stack_trace_snprint(buf, size, entries, nr_entries,
+ spaces) : 0;
+}
+EXPORT_SYMBOL_GPL(stack_depot_snprint);
+
+/**
+ * stack_depot_print - print stack entries from a depot
+ *
+ * @stack: Stack depot handle which was returned from
+ * stack_depot_save().
+ *
+ */
+void stack_depot_print(depot_stack_handle_t stack)
+{
+ unsigned long *entries;
+ unsigned int nr_entries;
+
+ nr_entries = stack_depot_fetch(stack, &entries);
+ if (nr_entries > 0)
+ stack_trace_print(entries, nr_entries, 0);
+}
+EXPORT_SYMBOL_GPL(stack_depot_print);
+
+/**
* stack_depot_fetch - Fetch stack entries from a depot
*
* @handle: Stack depot handle which was returned from
struct stack_record *stack;
*entries = NULL;
+ if (!handle)
+ return 0;
+
if (parts.slabindex > depot_index) {
WARN(1, "slab index %d out of bounds (%d) for stack id %08x\n",
parts.slabindex, depot_index, handle);
EXPORT_SYMBOL_GPL(stack_depot_fetch);
/**
- * stack_depot_save - Save a stack trace from an array
+ * __stack_depot_save - Save a stack trace from an array
*
* @entries: Pointer to storage array
* @nr_entries: Size of the storage array
* @alloc_flags: Allocation gfp flags
+ * @can_alloc: Allocate stack slabs (increased chance of failure if false)
+ *
+ * Saves a stack trace from @entries array of size @nr_entries. If @can_alloc is
+ * %true, is allowed to replenish the stack slab pool in case no space is left
+ * (allocates using GFP flags of @alloc_flags). If @can_alloc is %false, avoids
+ * any allocations and will fail if no space is left to store the stack trace.
+ *
+ * Context: Any context, but setting @can_alloc to %false is required if
+ * alloc_pages() cannot be used from the current context. Currently
+ * this is the case from contexts where neither %GFP_ATOMIC nor
+ * %GFP_NOWAIT can be used (NMI, raw_spin_lock).
*
- * Return: The handle of the stack struct stored in depot
+ * Return: The handle of the stack struct stored in depot, 0 on failure.
*/
-depot_stack_handle_t stack_depot_save(unsigned long *entries,
- unsigned int nr_entries,
- gfp_t alloc_flags)
+depot_stack_handle_t __stack_depot_save(unsigned long *entries,
+ unsigned int nr_entries,
+ gfp_t alloc_flags, bool can_alloc)
{
struct stack_record *found = NULL, **bucket;
depot_stack_handle_t retval = 0;
* The smp_load_acquire() here pairs with smp_store_release() to
* |next_slab_inited| in depot_alloc_stack() and init_stack_slab().
*/
- if (unlikely(!smp_load_acquire(&next_slab_inited))) {
+ if (unlikely(can_alloc && !smp_load_acquire(&next_slab_inited))) {
/*
* Zero out zone modifiers, as we don't have specific zone
* requirements. Keep the flags related to allocation in atomic
found = find_stack(*bucket, entries, nr_entries, hash);
if (!found) {
- struct stack_record *new =
- depot_alloc_stack(entries, nr_entries,
- hash, &prealloc, alloc_flags);
+ struct stack_record *new = depot_alloc_stack(entries, nr_entries, hash, &prealloc);
+
if (new) {
new->next = *bucket;
/*
fast_exit:
return retval;
}
-EXPORT_SYMBOL_GPL(stack_depot_save);
-
-static inline int in_irqentry_text(unsigned long ptr)
-{
- return (ptr >= (unsigned long)&__irqentry_text_start &&
- ptr < (unsigned long)&__irqentry_text_end) ||
- (ptr >= (unsigned long)&__softirqentry_text_start &&
- ptr < (unsigned long)&__softirqentry_text_end);
-}
+EXPORT_SYMBOL_GPL(__stack_depot_save);
-unsigned int filter_irq_stacks(unsigned long *entries,
- unsigned int nr_entries)
+/**
+ * stack_depot_save - Save a stack trace from an array
+ *
+ * @entries: Pointer to storage array
+ * @nr_entries: Size of the storage array
+ * @alloc_flags: Allocation gfp flags
+ *
+ * Context: Contexts where allocations via alloc_pages() are allowed.
+ * See __stack_depot_save() for more details.
+ *
+ * Return: The handle of the stack struct stored in depot, 0 on failure.
+ */
+depot_stack_handle_t stack_depot_save(unsigned long *entries,
+ unsigned int nr_entries,
+ gfp_t alloc_flags)
{
- unsigned int i;
-
- for (i = 0; i < nr_entries; i++) {
- if (in_irqentry_text(entries[i])) {
- /* Include the irqentry function into the stack. */
- return i + 1;
- }
- }
- return nr_entries;
+ return __stack_depot_save(entries, nr_entries, alloc_flags, true);
}
-EXPORT_SYMBOL_GPL(filter_irq_stacks);
+EXPORT_SYMBOL_GPL(stack_depot_save);
*/
rpage->zone_device_data = dmirror;
- *dst = migrate_pfn(page_to_pfn(dpage)) |
- MIGRATE_PFN_LOCKED;
+ *dst = migrate_pfn(page_to_pfn(dpage));
if ((*src & MIGRATE_PFN_WRITE) ||
(!spage && args->vma->vm_flags & VM_WRITE))
*dst |= MIGRATE_PFN_WRITE;
lock_page(dpage);
xa_erase(&dmirror->pt, addr >> PAGE_SHIFT);
copy_highpage(dpage, spage);
- *dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED;
+ *dst = migrate_pfn(page_to_pfn(dpage));
if (*src & MIGRATE_PFN_WRITE)
*dst |= MIGRATE_PFN_WRITE;
}
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test, memset(ptr + size - 1, 0, 2));
kfree(ptr);
}
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test, memset(ptr + size - 3, 0, 4));
kfree(ptr);
}
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test, memset(ptr + size - 7, 0, 8));
kfree(ptr);
}
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test, memset(ptr + size - 15, 0, 16));
kfree(ptr);
}
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test,
memset(ptr, 0, size + KASAN_GRANULE_SIZE));
kfree(ptr);
}
-static void kmalloc_memmove_invalid_size(struct kunit *test)
+static void kmalloc_memmove_negative_size(struct kunit *test)
{
char *ptr;
size_t size = 64;
- volatile size_t invalid_size = -2;
+ size_t invalid_size = -2;
/*
* Hardware tag-based mode doesn't check memmove for negative size.
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
memset((char *)ptr, 0, 64);
+ OPTIMIZER_HIDE_VAR(invalid_size);
+ KUNIT_EXPECT_KASAN_FAIL(test,
+ memmove((char *)ptr, (char *)ptr + 4, invalid_size));
+ kfree(ptr);
+}
+
+static void kmalloc_memmove_invalid_size(struct kunit *test)
+{
+ char *ptr;
+ size_t size = 64;
+ volatile size_t invalid_size = size;
+
+ ptr = kmalloc(size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+
+ memset((char *)ptr, 0, 64);
KUNIT_EXPECT_KASAN_FAIL(test,
memmove((char *)ptr, (char *)ptr + 4, invalid_size));
kfree(ptr);
ptr = kmalloc(size, GFP_KERNEL | __GFP_ZERO);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test,
kasan_ptr_result = memchr(ptr, '1', size + 1));
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
memset(arr, 0, sizeof(arr));
+ OPTIMIZER_HIDE_VAR(size);
KUNIT_EXPECT_KASAN_FAIL(test,
kasan_int_result = memcmp(ptr, arr, size+1));
kfree(ptr);
KUNIT_CASE(kmalloc_oob_memset_4),
KUNIT_CASE(kmalloc_oob_memset_8),
KUNIT_CASE(kmalloc_oob_memset_16),
+ KUNIT_CASE(kmalloc_memmove_negative_size),
KUNIT_CASE(kmalloc_memmove_invalid_size),
KUNIT_CASE(kmalloc_uaf),
KUNIT_CASE(kmalloc_uaf_memset),
return;
}
+ OPTIMIZER_HIDE_VAR(size);
+
pr_info("out-of-bounds in copy_from_user()\n");
unused = copy_from_user(kmem, usermem, size + 1);
static void shuffle_array(int *arr, int n)
{
unsigned int rnd;
- int i, j, x;
+ int i, j;
for (i = n - 1; i > 0; i--) {
get_random_bytes(&rnd, sizeof(rnd));
j = rnd % i;
/* Swap indexes. */
- x = arr[i];
- arr[i] = arr[j];
- arr[j] = x;
+ swap(arr[i], arr[j]);
}
}
#include <linux/string_helpers.h>
#include "kstrtox.h"
-static unsigned long long simple_strntoull(const char *startp, size_t max_chars,
- char **endp, unsigned int base)
+static noinline unsigned long long simple_strntoull(const char *startp, size_t max_chars, char **endp, unsigned int base)
{
const char *cp;
unsigned long long result = 0ULL;
-# SPDX-License-Identifier: GPL-2.0-only
+# SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause
+# ################################################################
+# Copyright (c) Facebook, Inc.
+# All rights reserved.
+#
+# This source code is licensed under both the BSD-style license (found in the
+# LICENSE file in the root directory of this source tree) and the GPLv2 (found
+# in the COPYING file in the root directory of this source tree).
+# You may select, at your option, one of the above-listed licenses.
+# ################################################################
obj-$(CONFIG_ZSTD_COMPRESS) += zstd_compress.o
obj-$(CONFIG_ZSTD_DECOMPRESS) += zstd_decompress.o
-ccflags-y += -O3
+zstd_compress-y := \
+ zstd_compress_module.o \
+ common/debug.o \
+ common/entropy_common.o \
+ common/error_private.o \
+ common/fse_decompress.o \
+ common/zstd_common.o \
+ compress/fse_compress.o \
+ compress/hist.o \
+ compress/huf_compress.o \
+ compress/zstd_compress.o \
+ compress/zstd_compress_literals.o \
+ compress/zstd_compress_sequences.o \
+ compress/zstd_compress_superblock.o \
+ compress/zstd_double_fast.o \
+ compress/zstd_fast.o \
+ compress/zstd_lazy.o \
+ compress/zstd_ldm.o \
+ compress/zstd_opt.o \
-zstd_compress-y := fse_compress.o huf_compress.o compress.o \
- entropy_common.o fse_decompress.o zstd_common.o
-zstd_decompress-y := huf_decompress.o decompress.o \
- entropy_common.o fse_decompress.o zstd_common.o
+zstd_decompress-y := \
+ zstd_decompress_module.o \
+ common/debug.o \
+ common/entropy_common.o \
+ common/error_private.o \
+ common/fse_decompress.o \
+ common/zstd_common.o \
+ decompress/huf_decompress.o \
+ decompress/zstd_ddict.o \
+ decompress/zstd_decompress.o \
+ decompress/zstd_decompress_block.o \
+++ /dev/null
-/*
- * bitstream
- * Part of FSE library
- * header file (to include)
- * Copyright (C) 2013-2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-#ifndef BITSTREAM_H_MODULE
-#define BITSTREAM_H_MODULE
-
-/*
-* This API consists of small unitary functions, which must be inlined for best performance.
-* Since link-time-optimization is not available for all compilers,
-* these functions are defined into a .h to be included.
-*/
-
-/*-****************************************
-* Dependencies
-******************************************/
-#include "error_private.h" /* error codes and messages */
-#include "mem.h" /* unaligned access routines */
-
-/*=========================================
-* Target specific
-=========================================*/
-#define STREAM_ACCUMULATOR_MIN_32 25
-#define STREAM_ACCUMULATOR_MIN_64 57
-#define STREAM_ACCUMULATOR_MIN ((U32)(ZSTD_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))
-
-/*-******************************************
-* bitStream encoding API (write forward)
-********************************************/
-/* bitStream can mix input from multiple sources.
-* A critical property of these streams is that they encode and decode in **reverse** direction.
-* So the first bit sequence you add will be the last to be read, like a LIFO stack.
-*/
-typedef struct {
- size_t bitContainer;
- int bitPos;
- char *startPtr;
- char *ptr;
- char *endPtr;
-} BIT_CStream_t;
-
-ZSTD_STATIC size_t BIT_initCStream(BIT_CStream_t *bitC, void *dstBuffer, size_t dstCapacity);
-ZSTD_STATIC void BIT_addBits(BIT_CStream_t *bitC, size_t value, unsigned nbBits);
-ZSTD_STATIC void BIT_flushBits(BIT_CStream_t *bitC);
-ZSTD_STATIC size_t BIT_closeCStream(BIT_CStream_t *bitC);
-
-/* Start with initCStream, providing the size of buffer to write into.
-* bitStream will never write outside of this buffer.
-* `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
-*
-* bits are first added to a local register.
-* Local register is size_t, hence 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
-* Writing data into memory is an explicit operation, performed by the flushBits function.
-* Hence keep track how many bits are potentially stored into local register to avoid register overflow.
-* After a flushBits, a maximum of 7 bits might still be stored into local register.
-*
-* Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
-*
-* Last operation is to close the bitStream.
-* The function returns the final size of CStream in bytes.
-* If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
-*/
-
-/*-********************************************
-* bitStream decoding API (read backward)
-**********************************************/
-typedef struct {
- size_t bitContainer;
- unsigned bitsConsumed;
- const char *ptr;
- const char *start;
-} BIT_DStream_t;
-
-typedef enum {
- BIT_DStream_unfinished = 0,
- BIT_DStream_endOfBuffer = 1,
- BIT_DStream_completed = 2,
- BIT_DStream_overflow = 3
-} BIT_DStream_status; /* result of BIT_reloadDStream() */
-/* 1,2,4,8 would be better for bitmap combinations, but slows down performance a bit ... :( */
-
-ZSTD_STATIC size_t BIT_initDStream(BIT_DStream_t *bitD, const void *srcBuffer, size_t srcSize);
-ZSTD_STATIC size_t BIT_readBits(BIT_DStream_t *bitD, unsigned nbBits);
-ZSTD_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t *bitD);
-ZSTD_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t *bitD);
-
-/* Start by invoking BIT_initDStream().
-* A chunk of the bitStream is then stored into a local register.
-* Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
-* You can then retrieve bitFields stored into the local register, **in reverse order**.
-* Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
-* A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
-* Otherwise, it can be less than that, so proceed accordingly.
-* Checking if DStream has reached its end can be performed with BIT_endOfDStream().
-*/
-
-/*-****************************************
-* unsafe API
-******************************************/
-ZSTD_STATIC void BIT_addBitsFast(BIT_CStream_t *bitC, size_t value, unsigned nbBits);
-/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */
-
-ZSTD_STATIC void BIT_flushBitsFast(BIT_CStream_t *bitC);
-/* unsafe version; does not check buffer overflow */
-
-ZSTD_STATIC size_t BIT_readBitsFast(BIT_DStream_t *bitD, unsigned nbBits);
-/* faster, but works only if nbBits >= 1 */
-
-/*-**************************************************************
-* Internal functions
-****************************************************************/
-ZSTD_STATIC unsigned BIT_highbit32(register U32 val) { return 31 - __builtin_clz(val); }
-
-/*===== Local Constants =====*/
-static const unsigned BIT_mask[] = {0, 1, 3, 7, 0xF, 0x1F, 0x3F, 0x7F, 0xFF,
- 0x1FF, 0x3FF, 0x7FF, 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF,
- 0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF, 0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF}; /* up to 26 bits */
-
-/*-**************************************************************
-* bitStream encoding
-****************************************************************/
-/*! BIT_initCStream() :
- * `dstCapacity` must be > sizeof(void*)
- * @return : 0 if success,
- otherwise an error code (can be tested using ERR_isError() ) */
-ZSTD_STATIC size_t BIT_initCStream(BIT_CStream_t *bitC, void *startPtr, size_t dstCapacity)
-{
- bitC->bitContainer = 0;
- bitC->bitPos = 0;
- bitC->startPtr = (char *)startPtr;
- bitC->ptr = bitC->startPtr;
- bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->ptr);
- if (dstCapacity <= sizeof(bitC->ptr))
- return ERROR(dstSize_tooSmall);
- return 0;
-}
-
-/*! BIT_addBits() :
- can add up to 26 bits into `bitC`.
- Does not check for register overflow ! */
-ZSTD_STATIC void BIT_addBits(BIT_CStream_t *bitC, size_t value, unsigned nbBits)
-{
- bitC->bitContainer |= (value & BIT_mask[nbBits]) << bitC->bitPos;
- bitC->bitPos += nbBits;
-}
-
-/*! BIT_addBitsFast() :
- * works only if `value` is _clean_, meaning all high bits above nbBits are 0 */
-ZSTD_STATIC void BIT_addBitsFast(BIT_CStream_t *bitC, size_t value, unsigned nbBits)
-{
- bitC->bitContainer |= value << bitC->bitPos;
- bitC->bitPos += nbBits;
-}
-
-/*! BIT_flushBitsFast() :
- * unsafe version; does not check buffer overflow */
-ZSTD_STATIC void BIT_flushBitsFast(BIT_CStream_t *bitC)
-{
- size_t const nbBytes = bitC->bitPos >> 3;
- ZSTD_writeLEST(bitC->ptr, bitC->bitContainer);
- bitC->ptr += nbBytes;
- bitC->bitPos &= 7;
- bitC->bitContainer >>= nbBytes * 8; /* if bitPos >= sizeof(bitContainer)*8 --> undefined behavior */
-}
-
-/*! BIT_flushBits() :
- * safe version; check for buffer overflow, and prevents it.
- * note : does not signal buffer overflow. This will be revealed later on using BIT_closeCStream() */
-ZSTD_STATIC void BIT_flushBits(BIT_CStream_t *bitC)
-{
- size_t const nbBytes = bitC->bitPos >> 3;
- ZSTD_writeLEST(bitC->ptr, bitC->bitContainer);
- bitC->ptr += nbBytes;
- if (bitC->ptr > bitC->endPtr)
- bitC->ptr = bitC->endPtr;
- bitC->bitPos &= 7;
- bitC->bitContainer >>= nbBytes * 8; /* if bitPos >= sizeof(bitContainer)*8 --> undefined behavior */
-}
-
-/*! BIT_closeCStream() :
- * @return : size of CStream, in bytes,
- or 0 if it could not fit into dstBuffer */
-ZSTD_STATIC size_t BIT_closeCStream(BIT_CStream_t *bitC)
-{
- BIT_addBitsFast(bitC, 1, 1); /* endMark */
- BIT_flushBits(bitC);
-
- if (bitC->ptr >= bitC->endPtr)
- return 0; /* doesn't fit within authorized budget : cancel */
-
- return (bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
-}
-
-/*-********************************************************
-* bitStream decoding
-**********************************************************/
-/*! BIT_initDStream() :
-* Initialize a BIT_DStream_t.
-* `bitD` : a pointer to an already allocated BIT_DStream_t structure.
-* `srcSize` must be the *exact* size of the bitStream, in bytes.
-* @return : size of stream (== srcSize) or an errorCode if a problem is detected
-*/
-ZSTD_STATIC size_t BIT_initDStream(BIT_DStream_t *bitD, const void *srcBuffer, size_t srcSize)
-{
- if (srcSize < 1) {
- memset(bitD, 0, sizeof(*bitD));
- return ERROR(srcSize_wrong);
- }
-
- if (srcSize >= sizeof(bitD->bitContainer)) { /* normal case */
- bitD->start = (const char *)srcBuffer;
- bitD->ptr = (const char *)srcBuffer + srcSize - sizeof(bitD->bitContainer);
- bitD->bitContainer = ZSTD_readLEST(bitD->ptr);
- {
- BYTE const lastByte = ((const BYTE *)srcBuffer)[srcSize - 1];
- bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0; /* ensures bitsConsumed is always set */
- if (lastByte == 0)
- return ERROR(GENERIC); /* endMark not present */
- }
- } else {
- bitD->start = (const char *)srcBuffer;
- bitD->ptr = bitD->start;
- bitD->bitContainer = *(const BYTE *)(bitD->start);
- switch (srcSize) {
- case 7: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[6]) << (sizeof(bitD->bitContainer) * 8 - 16);
- fallthrough;
- case 6: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[5]) << (sizeof(bitD->bitContainer) * 8 - 24);
- fallthrough;
- case 5: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[4]) << (sizeof(bitD->bitContainer) * 8 - 32);
- fallthrough;
- case 4: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[3]) << 24;
- fallthrough;
- case 3: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[2]) << 16;
- fallthrough;
- case 2: bitD->bitContainer += (size_t)(((const BYTE *)(srcBuffer))[1]) << 8;
- fallthrough;
- default:;
- }
- {
- BYTE const lastByte = ((const BYTE *)srcBuffer)[srcSize - 1];
- bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;
- if (lastByte == 0)
- return ERROR(GENERIC); /* endMark not present */
- }
- bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize) * 8;
- }
-
- return srcSize;
-}
-
-ZSTD_STATIC size_t BIT_getUpperBits(size_t bitContainer, U32 const start) { return bitContainer >> start; }
-
-ZSTD_STATIC size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 const nbBits) { return (bitContainer >> start) & BIT_mask[nbBits]; }
-
-ZSTD_STATIC size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits) { return bitContainer & BIT_mask[nbBits]; }
-
-/*! BIT_lookBits() :
- * Provides next n bits from local register.
- * local register is not modified.
- * On 32-bits, maxNbBits==24.
- * On 64-bits, maxNbBits==56.
- * @return : value extracted
- */
-ZSTD_STATIC size_t BIT_lookBits(const BIT_DStream_t *bitD, U32 nbBits)
-{
- U32 const bitMask = sizeof(bitD->bitContainer) * 8 - 1;
- return ((bitD->bitContainer << (bitD->bitsConsumed & bitMask)) >> 1) >> ((bitMask - nbBits) & bitMask);
-}
-
-/*! BIT_lookBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
-ZSTD_STATIC size_t BIT_lookBitsFast(const BIT_DStream_t *bitD, U32 nbBits)
-{
- U32 const bitMask = sizeof(bitD->bitContainer) * 8 - 1;
- return (bitD->bitContainer << (bitD->bitsConsumed & bitMask)) >> (((bitMask + 1) - nbBits) & bitMask);
-}
-
-ZSTD_STATIC void BIT_skipBits(BIT_DStream_t *bitD, U32 nbBits) { bitD->bitsConsumed += nbBits; }
-
-/*! BIT_readBits() :
- * Read (consume) next n bits from local register and update.
- * Pay attention to not read more than nbBits contained into local register.
- * @return : extracted value.
- */
-ZSTD_STATIC size_t BIT_readBits(BIT_DStream_t *bitD, U32 nbBits)
-{
- size_t const value = BIT_lookBits(bitD, nbBits);
- BIT_skipBits(bitD, nbBits);
- return value;
-}
-
-/*! BIT_readBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
-ZSTD_STATIC size_t BIT_readBitsFast(BIT_DStream_t *bitD, U32 nbBits)
-{
- size_t const value = BIT_lookBitsFast(bitD, nbBits);
- BIT_skipBits(bitD, nbBits);
- return value;
-}
-
-/*! BIT_reloadDStream() :
-* Refill `bitD` from buffer previously set in BIT_initDStream() .
-* This function is safe, it guarantees it will not read beyond src buffer.
-* @return : status of `BIT_DStream_t` internal register.
- if status == BIT_DStream_unfinished, internal register is filled with >= (sizeof(bitD->bitContainer)*8 - 7) bits */
-ZSTD_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t *bitD)
-{
- if (bitD->bitsConsumed > (sizeof(bitD->bitContainer) * 8)) /* should not happen => corruption detected */
- return BIT_DStream_overflow;
-
- if (bitD->ptr >= bitD->start + sizeof(bitD->bitContainer)) {
- bitD->ptr -= bitD->bitsConsumed >> 3;
- bitD->bitsConsumed &= 7;
- bitD->bitContainer = ZSTD_readLEST(bitD->ptr);
- return BIT_DStream_unfinished;
- }
- if (bitD->ptr == bitD->start) {
- if (bitD->bitsConsumed < sizeof(bitD->bitContainer) * 8)
- return BIT_DStream_endOfBuffer;
- return BIT_DStream_completed;
- }
- {
- U32 nbBytes = bitD->bitsConsumed >> 3;
- BIT_DStream_status result = BIT_DStream_unfinished;
- if (bitD->ptr - nbBytes < bitD->start) {
- nbBytes = (U32)(bitD->ptr - bitD->start); /* ptr > start */
- result = BIT_DStream_endOfBuffer;
- }
- bitD->ptr -= nbBytes;
- bitD->bitsConsumed -= nbBytes * 8;
- bitD->bitContainer = ZSTD_readLEST(bitD->ptr); /* reminder : srcSize > sizeof(bitD) */
- return result;
- }
-}
-
-/*! BIT_endOfDStream() :
-* @return Tells if DStream has exactly reached its end (all bits consumed).
-*/
-ZSTD_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t *DStream)
-{
- return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer) * 8));
-}
-
-#endif /* BITSTREAM_H_MODULE */
--- /dev/null
+/* ******************************************************************
+ * bitstream
+ * Part of FSE library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+#ifndef BITSTREAM_H_MODULE
+#define BITSTREAM_H_MODULE
+
+/*
+* This API consists of small unitary functions, which must be inlined for best performance.
+* Since link-time-optimization is not available for all compilers,
+* these functions are defined into a .h to be included.
+*/
+
+/*-****************************************
+* Dependencies
+******************************************/
+#include "mem.h" /* unaligned access routines */
+#include "compiler.h" /* UNLIKELY() */
+#include "debug.h" /* assert(), DEBUGLOG(), RAWLOG() */
+#include "error_private.h" /* error codes and messages */
+
+
+/*=========================================
+* Target specific
+=========================================*/
+
+#define STREAM_ACCUMULATOR_MIN_32 25
+#define STREAM_ACCUMULATOR_MIN_64 57
+#define STREAM_ACCUMULATOR_MIN ((U32)(MEM_32bits() ? STREAM_ACCUMULATOR_MIN_32 : STREAM_ACCUMULATOR_MIN_64))
+
+
+/*-******************************************
+* bitStream encoding API (write forward)
+********************************************/
+/* bitStream can mix input from multiple sources.
+ * A critical property of these streams is that they encode and decode in **reverse** direction.
+ * So the first bit sequence you add will be the last to be read, like a LIFO stack.
+ */
+typedef struct {
+ size_t bitContainer;
+ unsigned bitPos;
+ char* startPtr;
+ char* ptr;
+ char* endPtr;
+} BIT_CStream_t;
+
+MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC, void* dstBuffer, size_t dstCapacity);
+MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
+MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC);
+MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC);
+
+/* Start with initCStream, providing the size of buffer to write into.
+* bitStream will never write outside of this buffer.
+* `dstCapacity` must be >= sizeof(bitD->bitContainer), otherwise @return will be an error code.
+*
+* bits are first added to a local register.
+* Local register is size_t, hence 64-bits on 64-bits systems, or 32-bits on 32-bits systems.
+* Writing data into memory is an explicit operation, performed by the flushBits function.
+* Hence keep track how many bits are potentially stored into local register to avoid register overflow.
+* After a flushBits, a maximum of 7 bits might still be stored into local register.
+*
+* Avoid storing elements of more than 24 bits if you want compatibility with 32-bits bitstream readers.
+*
+* Last operation is to close the bitStream.
+* The function returns the final size of CStream in bytes.
+* If data couldn't fit into `dstBuffer`, it will return a 0 ( == not storable)
+*/
+
+
+/*-********************************************
+* bitStream decoding API (read backward)
+**********************************************/
+typedef struct {
+ size_t bitContainer;
+ unsigned bitsConsumed;
+ const char* ptr;
+ const char* start;
+ const char* limitPtr;
+} BIT_DStream_t;
+
+typedef enum { BIT_DStream_unfinished = 0,
+ BIT_DStream_endOfBuffer = 1,
+ BIT_DStream_completed = 2,
+ BIT_DStream_overflow = 3 } BIT_DStream_status; /* result of BIT_reloadDStream() */
+ /* 1,2,4,8 would be better for bitmap combinations, but slows down performance a bit ... :( */
+
+MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize);
+MEM_STATIC size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits);
+MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD);
+MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* bitD);
+
+
+/* Start by invoking BIT_initDStream().
+* A chunk of the bitStream is then stored into a local register.
+* Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
+* You can then retrieve bitFields stored into the local register, **in reverse order**.
+* Local register is explicitly reloaded from memory by the BIT_reloadDStream() method.
+* A reload guarantee a minimum of ((8*sizeof(bitD->bitContainer))-7) bits when its result is BIT_DStream_unfinished.
+* Otherwise, it can be less than that, so proceed accordingly.
+* Checking if DStream has reached its end can be performed with BIT_endOfDStream().
+*/
+
+
+/*-****************************************
+* unsafe API
+******************************************/
+MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC, size_t value, unsigned nbBits);
+/* faster, but works only if value is "clean", meaning all high bits above nbBits are 0 */
+
+MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC);
+/* unsafe version; does not check buffer overflow */
+
+MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
+/* faster, but works only if nbBits >= 1 */
+
+
+
+/*-**************************************************************
+* Internal functions
+****************************************************************/
+MEM_STATIC unsigned BIT_highbit32 (U32 val)
+{
+ assert(val != 0);
+ {
+# if (__GNUC__ >= 3) /* Use GCC Intrinsic */
+ return __builtin_clz (val) ^ 31;
+# else /* Software version */
+ static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
+ 11, 14, 16, 18, 22, 25, 3, 30,
+ 8, 12, 20, 28, 15, 17, 24, 7,
+ 19, 27, 23, 6, 26, 5, 4, 31 };
+ U32 v = val;
+ v |= v >> 1;
+ v |= v >> 2;
+ v |= v >> 4;
+ v |= v >> 8;
+ v |= v >> 16;
+ return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
+# endif
+ }
+}
+
+/*===== Local Constants =====*/
+static const unsigned BIT_mask[] = {
+ 0, 1, 3, 7, 0xF, 0x1F,
+ 0x3F, 0x7F, 0xFF, 0x1FF, 0x3FF, 0x7FF,
+ 0xFFF, 0x1FFF, 0x3FFF, 0x7FFF, 0xFFFF, 0x1FFFF,
+ 0x3FFFF, 0x7FFFF, 0xFFFFF, 0x1FFFFF, 0x3FFFFF, 0x7FFFFF,
+ 0xFFFFFF, 0x1FFFFFF, 0x3FFFFFF, 0x7FFFFFF, 0xFFFFFFF, 0x1FFFFFFF,
+ 0x3FFFFFFF, 0x7FFFFFFF}; /* up to 31 bits */
+#define BIT_MASK_SIZE (sizeof(BIT_mask) / sizeof(BIT_mask[0]))
+
+/*-**************************************************************
+* bitStream encoding
+****************************************************************/
+/*! BIT_initCStream() :
+ * `dstCapacity` must be > sizeof(size_t)
+ * @return : 0 if success,
+ * otherwise an error code (can be tested using ERR_isError()) */
+MEM_STATIC size_t BIT_initCStream(BIT_CStream_t* bitC,
+ void* startPtr, size_t dstCapacity)
+{
+ bitC->bitContainer = 0;
+ bitC->bitPos = 0;
+ bitC->startPtr = (char*)startPtr;
+ bitC->ptr = bitC->startPtr;
+ bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer);
+ if (dstCapacity <= sizeof(bitC->bitContainer)) return ERROR(dstSize_tooSmall);
+ return 0;
+}
+
+/*! BIT_addBits() :
+ * can add up to 31 bits into `bitC`.
+ * Note : does not check for register overflow ! */
+MEM_STATIC void BIT_addBits(BIT_CStream_t* bitC,
+ size_t value, unsigned nbBits)
+{
+ DEBUG_STATIC_ASSERT(BIT_MASK_SIZE == 32);
+ assert(nbBits < BIT_MASK_SIZE);
+ assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
+ bitC->bitContainer |= (value & BIT_mask[nbBits]) << bitC->bitPos;
+ bitC->bitPos += nbBits;
+}
+
+/*! BIT_addBitsFast() :
+ * works only if `value` is _clean_,
+ * meaning all high bits above nbBits are 0 */
+MEM_STATIC void BIT_addBitsFast(BIT_CStream_t* bitC,
+ size_t value, unsigned nbBits)
+{
+ assert((value>>nbBits) == 0);
+ assert(nbBits + bitC->bitPos < sizeof(bitC->bitContainer) * 8);
+ bitC->bitContainer |= value << bitC->bitPos;
+ bitC->bitPos += nbBits;
+}
+
+/*! BIT_flushBitsFast() :
+ * assumption : bitContainer has not overflowed
+ * unsafe version; does not check buffer overflow */
+MEM_STATIC void BIT_flushBitsFast(BIT_CStream_t* bitC)
+{
+ size_t const nbBytes = bitC->bitPos >> 3;
+ assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
+ assert(bitC->ptr <= bitC->endPtr);
+ MEM_writeLEST(bitC->ptr, bitC->bitContainer);
+ bitC->ptr += nbBytes;
+ bitC->bitPos &= 7;
+ bitC->bitContainer >>= nbBytes*8;
+}
+
+/*! BIT_flushBits() :
+ * assumption : bitContainer has not overflowed
+ * safe version; check for buffer overflow, and prevents it.
+ * note : does not signal buffer overflow.
+ * overflow will be revealed later on using BIT_closeCStream() */
+MEM_STATIC void BIT_flushBits(BIT_CStream_t* bitC)
+{
+ size_t const nbBytes = bitC->bitPos >> 3;
+ assert(bitC->bitPos < sizeof(bitC->bitContainer) * 8);
+ assert(bitC->ptr <= bitC->endPtr);
+ MEM_writeLEST(bitC->ptr, bitC->bitContainer);
+ bitC->ptr += nbBytes;
+ if (bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
+ bitC->bitPos &= 7;
+ bitC->bitContainer >>= nbBytes*8;
+}
+
+/*! BIT_closeCStream() :
+ * @return : size of CStream, in bytes,
+ * or 0 if it could not fit into dstBuffer */
+MEM_STATIC size_t BIT_closeCStream(BIT_CStream_t* bitC)
+{
+ BIT_addBitsFast(bitC, 1, 1); /* endMark */
+ BIT_flushBits(bitC);
+ if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
+ return (bitC->ptr - bitC->startPtr) + (bitC->bitPos > 0);
+}
+
+
+/*-********************************************************
+* bitStream decoding
+**********************************************************/
+/*! BIT_initDStream() :
+ * Initialize a BIT_DStream_t.
+ * `bitD` : a pointer to an already allocated BIT_DStream_t structure.
+ * `srcSize` must be the *exact* size of the bitStream, in bytes.
+ * @return : size of stream (== srcSize), or an errorCode if a problem is detected
+ */
+MEM_STATIC size_t BIT_initDStream(BIT_DStream_t* bitD, const void* srcBuffer, size_t srcSize)
+{
+ if (srcSize < 1) { ZSTD_memset(bitD, 0, sizeof(*bitD)); return ERROR(srcSize_wrong); }
+
+ bitD->start = (const char*)srcBuffer;
+ bitD->limitPtr = bitD->start + sizeof(bitD->bitContainer);
+
+ if (srcSize >= sizeof(bitD->bitContainer)) { /* normal case */
+ bitD->ptr = (const char*)srcBuffer + srcSize - sizeof(bitD->bitContainer);
+ bitD->bitContainer = MEM_readLEST(bitD->ptr);
+ { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
+ bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0; /* ensures bitsConsumed is always set */
+ if (lastByte == 0) return ERROR(GENERIC); /* endMark not present */ }
+ } else {
+ bitD->ptr = bitD->start;
+ bitD->bitContainer = *(const BYTE*)(bitD->start);
+ switch(srcSize)
+ {
+ case 7: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[6]) << (sizeof(bitD->bitContainer)*8 - 16);
+ ZSTD_FALLTHROUGH;
+
+ case 6: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[5]) << (sizeof(bitD->bitContainer)*8 - 24);
+ ZSTD_FALLTHROUGH;
+
+ case 5: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[4]) << (sizeof(bitD->bitContainer)*8 - 32);
+ ZSTD_FALLTHROUGH;
+
+ case 4: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[3]) << 24;
+ ZSTD_FALLTHROUGH;
+
+ case 3: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[2]) << 16;
+ ZSTD_FALLTHROUGH;
+
+ case 2: bitD->bitContainer += (size_t)(((const BYTE*)(srcBuffer))[1]) << 8;
+ ZSTD_FALLTHROUGH;
+
+ default: break;
+ }
+ { BYTE const lastByte = ((const BYTE*)srcBuffer)[srcSize-1];
+ bitD->bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;
+ if (lastByte == 0) return ERROR(corruption_detected); /* endMark not present */
+ }
+ bitD->bitsConsumed += (U32)(sizeof(bitD->bitContainer) - srcSize)*8;
+ }
+
+ return srcSize;
+}
+
+MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getUpperBits(size_t bitContainer, U32 const start)
+{
+ return bitContainer >> start;
+}
+
+MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getMiddleBits(size_t bitContainer, U32 const start, U32 const nbBits)
+{
+ U32 const regMask = sizeof(bitContainer)*8 - 1;
+ /* if start > regMask, bitstream is corrupted, and result is undefined */
+ assert(nbBits < BIT_MASK_SIZE);
+ return (bitContainer >> (start & regMask)) & BIT_mask[nbBits];
+}
+
+MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits)
+{
+ assert(nbBits < BIT_MASK_SIZE);
+ return bitContainer & BIT_mask[nbBits];
+}
+
+/*! BIT_lookBits() :
+ * Provides next n bits from local register.
+ * local register is not modified.
+ * On 32-bits, maxNbBits==24.
+ * On 64-bits, maxNbBits==56.
+ * @return : value extracted */
+MEM_STATIC FORCE_INLINE_ATTR size_t BIT_lookBits(const BIT_DStream_t* bitD, U32 nbBits)
+{
+ /* arbitrate between double-shift and shift+mask */
+#if 1
+ /* if bitD->bitsConsumed + nbBits > sizeof(bitD->bitContainer)*8,
+ * bitstream is likely corrupted, and result is undefined */
+ return BIT_getMiddleBits(bitD->bitContainer, (sizeof(bitD->bitContainer)*8) - bitD->bitsConsumed - nbBits, nbBits);
+#else
+ /* this code path is slower on my os-x laptop */
+ U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
+ return ((bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> 1) >> ((regMask-nbBits) & regMask);
+#endif
+}
+
+/*! BIT_lookBitsFast() :
+ * unsafe version; only works if nbBits >= 1 */
+MEM_STATIC size_t BIT_lookBitsFast(const BIT_DStream_t* bitD, U32 nbBits)
+{
+ U32 const regMask = sizeof(bitD->bitContainer)*8 - 1;
+ assert(nbBits >= 1);
+ return (bitD->bitContainer << (bitD->bitsConsumed & regMask)) >> (((regMask+1)-nbBits) & regMask);
+}
+
+MEM_STATIC FORCE_INLINE_ATTR void BIT_skipBits(BIT_DStream_t* bitD, U32 nbBits)
+{
+ bitD->bitsConsumed += nbBits;
+}
+
+/*! BIT_readBits() :
+ * Read (consume) next n bits from local register and update.
+ * Pay attention to not read more than nbBits contained into local register.
+ * @return : extracted value. */
+MEM_STATIC FORCE_INLINE_ATTR size_t BIT_readBits(BIT_DStream_t* bitD, unsigned nbBits)
+{
+ size_t const value = BIT_lookBits(bitD, nbBits);
+ BIT_skipBits(bitD, nbBits);
+ return value;
+}
+
+/*! BIT_readBitsFast() :
+ * unsafe version; only works only if nbBits >= 1 */
+MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
+{
+ size_t const value = BIT_lookBitsFast(bitD, nbBits);
+ assert(nbBits >= 1);
+ BIT_skipBits(bitD, nbBits);
+ return value;
+}
+
+/*! BIT_reloadDStreamFast() :
+ * Similar to BIT_reloadDStream(), but with two differences:
+ * 1. bitsConsumed <= sizeof(bitD->bitContainer)*8 must hold!
+ * 2. Returns BIT_DStream_overflow when bitD->ptr < bitD->limitPtr, at this
+ * point you must use BIT_reloadDStream() to reload.
+ */
+MEM_STATIC BIT_DStream_status BIT_reloadDStreamFast(BIT_DStream_t* bitD)
+{
+ if (UNLIKELY(bitD->ptr < bitD->limitPtr))
+ return BIT_DStream_overflow;
+ assert(bitD->bitsConsumed <= sizeof(bitD->bitContainer)*8);
+ bitD->ptr -= bitD->bitsConsumed >> 3;
+ bitD->bitsConsumed &= 7;
+ bitD->bitContainer = MEM_readLEST(bitD->ptr);
+ return BIT_DStream_unfinished;
+}
+
+/*! BIT_reloadDStream() :
+ * Refill `bitD` from buffer previously set in BIT_initDStream() .
+ * This function is safe, it guarantees it will not read beyond src buffer.
+ * @return : status of `BIT_DStream_t` internal register.
+ * when status == BIT_DStream_unfinished, internal register is filled with at least 25 or 57 bits */
+MEM_STATIC BIT_DStream_status BIT_reloadDStream(BIT_DStream_t* bitD)
+{
+ if (bitD->bitsConsumed > (sizeof(bitD->bitContainer)*8)) /* overflow detected, like end of stream */
+ return BIT_DStream_overflow;
+
+ if (bitD->ptr >= bitD->limitPtr) {
+ return BIT_reloadDStreamFast(bitD);
+ }
+ if (bitD->ptr == bitD->start) {
+ if (bitD->bitsConsumed < sizeof(bitD->bitContainer)*8) return BIT_DStream_endOfBuffer;
+ return BIT_DStream_completed;
+ }
+ /* start < ptr < limitPtr */
+ { U32 nbBytes = bitD->bitsConsumed >> 3;
+ BIT_DStream_status result = BIT_DStream_unfinished;
+ if (bitD->ptr - nbBytes < bitD->start) {
+ nbBytes = (U32)(bitD->ptr - bitD->start); /* ptr > start */
+ result = BIT_DStream_endOfBuffer;
+ }
+ bitD->ptr -= nbBytes;
+ bitD->bitsConsumed -= nbBytes*8;
+ bitD->bitContainer = MEM_readLEST(bitD->ptr); /* reminder : srcSize > sizeof(bitD->bitContainer), otherwise bitD->ptr == bitD->start */
+ return result;
+ }
+}
+
+/*! BIT_endOfDStream() :
+ * @return : 1 if DStream has _exactly_ reached its end (all bits consumed).
+ */
+MEM_STATIC unsigned BIT_endOfDStream(const BIT_DStream_t* DStream)
+{
+ return ((DStream->ptr == DStream->start) && (DStream->bitsConsumed == sizeof(DStream->bitContainer)*8));
+}
+
+
+#endif /* BITSTREAM_H_MODULE */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMPILER_H
+#define ZSTD_COMPILER_H
+
+/*-*******************************************************
+* Compiler specifics
+*********************************************************/
+/* force inlining */
+
+#if !defined(ZSTD_NO_INLINE)
+#if (defined(__GNUC__) && !defined(__STRICT_ANSI__)) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
+# define INLINE_KEYWORD inline
+#else
+# define INLINE_KEYWORD
+#endif
+
+#define FORCE_INLINE_ATTR __attribute__((always_inline))
+
+#else
+
+#define INLINE_KEYWORD
+#define FORCE_INLINE_ATTR
+
+#endif
+
+/*
+ On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC).
+ This explictly marks such functions as __cdecl so that the code will still compile
+ if a CC other than __cdecl has been made the default.
+*/
+#define WIN_CDECL
+
+/*
+ * FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
+ * parameters. They must be inlined for the compiler to eliminate the constant
+ * branches.
+ */
+#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR
+/*
+ * HINT_INLINE is used to help the compiler generate better code. It is *not*
+ * used for "templates", so it can be tweaked based on the compilers
+ * performance.
+ *
+ * gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
+ * always_inline attribute.
+ *
+ * clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
+ * attribute.
+ */
+#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
+# define HINT_INLINE static INLINE_KEYWORD
+#else
+# define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR
+#endif
+
+/* UNUSED_ATTR tells the compiler it is okay if the function is unused. */
+#define UNUSED_ATTR __attribute__((unused))
+
+/* force no inlining */
+#define FORCE_NOINLINE static __attribute__((__noinline__))
+
+
+/* target attribute */
+#ifndef __has_attribute
+ #define __has_attribute(x) 0 /* Compatibility with non-clang compilers. */
+#endif
+#define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
+
+/* Enable runtime BMI2 dispatch based on the CPU.
+ * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
+ */
+#ifndef DYNAMIC_BMI2
+ #if ((defined(__clang__) && __has_attribute(__target__)) \
+ || (defined(__GNUC__) \
+ && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
+ && (defined(__x86_64__) || defined(_M_X86)) \
+ && !defined(__BMI2__)
+ # define DYNAMIC_BMI2 1
+ #else
+ # define DYNAMIC_BMI2 0
+ #endif
+#endif
+
+/* prefetch
+ * can be disabled, by declaring NO_PREFETCH build macro */
+#if ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) )
+# define PREFETCH_L1(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */)
+# define PREFETCH_L2(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 2 /* locality */)
+#elif defined(__aarch64__)
+# define PREFETCH_L1(ptr) __asm__ __volatile__("prfm pldl1keep, %0" ::"Q"(*(ptr)))
+# define PREFETCH_L2(ptr) __asm__ __volatile__("prfm pldl2keep, %0" ::"Q"(*(ptr)))
+#else
+# define PREFETCH_L1(ptr) (void)(ptr) /* disabled */
+# define PREFETCH_L2(ptr) (void)(ptr) /* disabled */
+#endif /* NO_PREFETCH */
+
+#define CACHELINE_SIZE 64
+
+#define PREFETCH_AREA(p, s) { \
+ const char* const _ptr = (const char*)(p); \
+ size_t const _size = (size_t)(s); \
+ size_t _pos; \
+ for (_pos=0; _pos<_size; _pos+=CACHELINE_SIZE) { \
+ PREFETCH_L2(_ptr + _pos); \
+ } \
+}
+
+/* vectorization
+ * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax */
+#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__)
+# if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
+# define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
+# else
+# define DONT_VECTORIZE _Pragma("GCC optimize(\"no-tree-vectorize\")")
+# endif
+#else
+# define DONT_VECTORIZE
+#endif
+
+/* Tell the compiler that a branch is likely or unlikely.
+ * Only use these macros if it causes the compiler to generate better code.
+ * If you can remove a LIKELY/UNLIKELY annotation without speed changes in gcc
+ * and clang, please do.
+ */
+#define LIKELY(x) (__builtin_expect((x), 1))
+#define UNLIKELY(x) (__builtin_expect((x), 0))
+
+/* disable warnings */
+
+/*Like DYNAMIC_BMI2 but for compile time determination of BMI2 support*/
+
+
+/* compat. with non-clang compilers */
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+/* compat. with non-clang compilers */
+#ifndef __has_feature
+# define __has_feature(x) 0
+#endif
+
+/* C-language Attributes are added in C23. */
+#if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute)
+# define ZSTD_HAS_C_ATTRIBUTE(x) __has_c_attribute(x)
+#else
+# define ZSTD_HAS_C_ATTRIBUTE(x) 0
+#endif
+
+/* Only use C++ attributes in C++. Some compilers report support for C++
+ * attributes when compiling with C.
+ */
+#define ZSTD_HAS_CPP_ATTRIBUTE(x) 0
+
+/* Define ZSTD_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute.
+ * - C23: https://en.cppreference.com/w/c/language/attributes/fallthrough
+ * - CPP17: https://en.cppreference.com/w/cpp/language/attributes/fallthrough
+ * - Else: __attribute__((__fallthrough__))
+ */
+#define ZSTD_FALLTHROUGH fallthrough
+
+/* detects whether we are being compiled under msan */
+
+
+/* detects whether we are being compiled under asan */
+
+
+#endif /* ZSTD_COMPILER_H */
--- /dev/null
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMMON_CPU_H
+#define ZSTD_COMMON_CPU_H
+
+/*
+ * Implementation taken from folly/CpuId.h
+ * https://github.com/facebook/folly/blob/master/folly/CpuId.h
+ */
+
+#include "mem.h"
+
+
+typedef struct {
+ U32 f1c;
+ U32 f1d;
+ U32 f7b;
+ U32 f7c;
+} ZSTD_cpuid_t;
+
+MEM_STATIC ZSTD_cpuid_t ZSTD_cpuid(void) {
+ U32 f1c = 0;
+ U32 f1d = 0;
+ U32 f7b = 0;
+ U32 f7c = 0;
+#if defined(__i386__) && defined(__PIC__) && !defined(__clang__) && defined(__GNUC__)
+ /* The following block like the normal cpuid branch below, but gcc
+ * reserves ebx for use of its pic register so we must specially
+ * handle the save and restore to avoid clobbering the register
+ */
+ U32 n;
+ __asm__(
+ "pushl %%ebx\n\t"
+ "cpuid\n\t"
+ "popl %%ebx\n\t"
+ : "=a"(n)
+ : "a"(0)
+ : "ecx", "edx");
+ if (n >= 1) {
+ U32 f1a;
+ __asm__(
+ "pushl %%ebx\n\t"
+ "cpuid\n\t"
+ "popl %%ebx\n\t"
+ : "=a"(f1a), "=c"(f1c), "=d"(f1d)
+ : "a"(1));
+ }
+ if (n >= 7) {
+ __asm__(
+ "pushl %%ebx\n\t"
+ "cpuid\n\t"
+ "movl %%ebx, %%eax\n\t"
+ "popl %%ebx"
+ : "=a"(f7b), "=c"(f7c)
+ : "a"(7), "c"(0)
+ : "edx");
+ }
+#elif defined(__x86_64__) || defined(_M_X64) || defined(__i386__)
+ U32 n;
+ __asm__("cpuid" : "=a"(n) : "a"(0) : "ebx", "ecx", "edx");
+ if (n >= 1) {
+ U32 f1a;
+ __asm__("cpuid" : "=a"(f1a), "=c"(f1c), "=d"(f1d) : "a"(1) : "ebx");
+ }
+ if (n >= 7) {
+ U32 f7a;
+ __asm__("cpuid"
+ : "=a"(f7a), "=b"(f7b), "=c"(f7c)
+ : "a"(7), "c"(0)
+ : "edx");
+ }
+#endif
+ {
+ ZSTD_cpuid_t cpuid;
+ cpuid.f1c = f1c;
+ cpuid.f1d = f1d;
+ cpuid.f7b = f7b;
+ cpuid.f7c = f7c;
+ return cpuid;
+ }
+}
+
+#define X(name, r, bit) \
+ MEM_STATIC int ZSTD_cpuid_##name(ZSTD_cpuid_t const cpuid) { \
+ return ((cpuid.r) & (1U << bit)) != 0; \
+ }
+
+/* cpuid(1): Processor Info and Feature Bits. */
+#define C(name, bit) X(name, f1c, bit)
+ C(sse3, 0)
+ C(pclmuldq, 1)
+ C(dtes64, 2)
+ C(monitor, 3)
+ C(dscpl, 4)
+ C(vmx, 5)
+ C(smx, 6)
+ C(eist, 7)
+ C(tm2, 8)
+ C(ssse3, 9)
+ C(cnxtid, 10)
+ C(fma, 12)
+ C(cx16, 13)
+ C(xtpr, 14)
+ C(pdcm, 15)
+ C(pcid, 17)
+ C(dca, 18)
+ C(sse41, 19)
+ C(sse42, 20)
+ C(x2apic, 21)
+ C(movbe, 22)
+ C(popcnt, 23)
+ C(tscdeadline, 24)
+ C(aes, 25)
+ C(xsave, 26)
+ C(osxsave, 27)
+ C(avx, 28)
+ C(f16c, 29)
+ C(rdrand, 30)
+#undef C
+#define D(name, bit) X(name, f1d, bit)
+ D(fpu, 0)
+ D(vme, 1)
+ D(de, 2)
+ D(pse, 3)
+ D(tsc, 4)
+ D(msr, 5)
+ D(pae, 6)
+ D(mce, 7)
+ D(cx8, 8)
+ D(apic, 9)
+ D(sep, 11)
+ D(mtrr, 12)
+ D(pge, 13)
+ D(mca, 14)
+ D(cmov, 15)
+ D(pat, 16)
+ D(pse36, 17)
+ D(psn, 18)
+ D(clfsh, 19)
+ D(ds, 21)
+ D(acpi, 22)
+ D(mmx, 23)
+ D(fxsr, 24)
+ D(sse, 25)
+ D(sse2, 26)
+ D(ss, 27)
+ D(htt, 28)
+ D(tm, 29)
+ D(pbe, 31)
+#undef D
+
+/* cpuid(7): Extended Features. */
+#define B(name, bit) X(name, f7b, bit)
+ B(bmi1, 3)
+ B(hle, 4)
+ B(avx2, 5)
+ B(smep, 7)
+ B(bmi2, 8)
+ B(erms, 9)
+ B(invpcid, 10)
+ B(rtm, 11)
+ B(mpx, 14)
+ B(avx512f, 16)
+ B(avx512dq, 17)
+ B(rdseed, 18)
+ B(adx, 19)
+ B(smap, 20)
+ B(avx512ifma, 21)
+ B(pcommit, 22)
+ B(clflushopt, 23)
+ B(clwb, 24)
+ B(avx512pf, 26)
+ B(avx512er, 27)
+ B(avx512cd, 28)
+ B(sha, 29)
+ B(avx512bw, 30)
+ B(avx512vl, 31)
+#undef B
+#define C(name, bit) X(name, f7c, bit)
+ C(prefetchwt1, 0)
+ C(avx512vbmi, 1)
+#undef C
+
+#undef X
+
+#endif /* ZSTD_COMMON_CPU_H */
--- /dev/null
+/* ******************************************************************
+ * debug
+ * Part of FSE library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+
+/*
+ * This module only hosts one global variable
+ * which can be used to dynamically influence the verbosity of traces,
+ * such as DEBUGLOG and RAWLOG
+ */
+
+#include "debug.h"
+
+int g_debuglevel = DEBUGLEVEL;
--- /dev/null
+/* ******************************************************************
+ * debug
+ * Part of FSE library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+
+/*
+ * The purpose of this header is to enable debug functions.
+ * They regroup assert(), DEBUGLOG() and RAWLOG() for run-time,
+ * and DEBUG_STATIC_ASSERT() for compile-time.
+ *
+ * By default, DEBUGLEVEL==0, which means run-time debug is disabled.
+ *
+ * Level 1 enables assert() only.
+ * Starting level 2, traces can be generated and pushed to stderr.
+ * The higher the level, the more verbose the traces.
+ *
+ * It's possible to dynamically adjust level using variable g_debug_level,
+ * which is only declared if DEBUGLEVEL>=2,
+ * and is a global variable, not multi-thread protected (use with care)
+ */
+
+#ifndef DEBUG_H_12987983217
+#define DEBUG_H_12987983217
+
+
+
+/* static assert is triggered at compile time, leaving no runtime artefact.
+ * static assert only works with compile-time constants.
+ * Also, this variant can only be used inside a function. */
+#define DEBUG_STATIC_ASSERT(c) (void)sizeof(char[(c) ? 1 : -1])
+
+
+/* DEBUGLEVEL is expected to be defined externally,
+ * typically through compiler command line.
+ * Value must be a number. */
+#ifndef DEBUGLEVEL
+# define DEBUGLEVEL 0
+#endif
+
+
+/* recommended values for DEBUGLEVEL :
+ * 0 : release mode, no debug, all run-time checks disabled
+ * 1 : enables assert() only, no display
+ * 2 : reserved, for currently active debug path
+ * 3 : events once per object lifetime (CCtx, CDict, etc.)
+ * 4 : events once per frame
+ * 5 : events once per block
+ * 6 : events once per sequence (verbose)
+ * 7+: events at every position (*very* verbose)
+ *
+ * It's generally inconvenient to output traces > 5.
+ * In which case, it's possible to selectively trigger high verbosity levels
+ * by modifying g_debug_level.
+ */
+
+#if (DEBUGLEVEL>=1)
+# define ZSTD_DEPS_NEED_ASSERT
+# include "zstd_deps.h"
+#else
+# ifndef assert /* assert may be already defined, due to prior #include <assert.h> */
+# define assert(condition) ((void)0) /* disable assert (default) */
+# endif
+#endif
+
+#if (DEBUGLEVEL>=2)
+# define ZSTD_DEPS_NEED_IO
+# include "zstd_deps.h"
+extern int g_debuglevel; /* the variable is only declared,
+ it actually lives in debug.c,
+ and is shared by the whole process.
+ It's not thread-safe.
+ It's useful when enabling very verbose levels
+ on selective conditions (such as position in src) */
+
+# define RAWLOG(l, ...) { \
+ if (l<=g_debuglevel) { \
+ ZSTD_DEBUG_PRINT(__VA_ARGS__); \
+ } }
+# define DEBUGLOG(l, ...) { \
+ if (l<=g_debuglevel) { \
+ ZSTD_DEBUG_PRINT(__FILE__ ": " __VA_ARGS__); \
+ ZSTD_DEBUG_PRINT(" \n"); \
+ } }
+#else
+# define RAWLOG(l, ...) {} /* disabled */
+# define DEBUGLOG(l, ...) {} /* disabled */
+#endif
+
+
+
+#endif /* DEBUG_H_12987983217 */
--- /dev/null
+/* ******************************************************************
+ * Common functions of New Generation Entropy library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* *************************************
+* Dependencies
+***************************************/
+#include "mem.h"
+#include "error_private.h" /* ERR_*, ERROR */
+#define FSE_STATIC_LINKING_ONLY /* FSE_MIN_TABLELOG */
+#include "fse.h"
+#define HUF_STATIC_LINKING_ONLY /* HUF_TABLELOG_ABSOLUTEMAX */
+#include "huf.h"
+
+
+/*=== Version ===*/
+unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }
+
+
+/*=== Error Management ===*/
+unsigned FSE_isError(size_t code) { return ERR_isError(code); }
+const char* FSE_getErrorName(size_t code) { return ERR_getErrorName(code); }
+
+unsigned HUF_isError(size_t code) { return ERR_isError(code); }
+const char* HUF_getErrorName(size_t code) { return ERR_getErrorName(code); }
+
+
+/*-**************************************************************
+* FSE NCount encoding-decoding
+****************************************************************/
+static U32 FSE_ctz(U32 val)
+{
+ assert(val != 0);
+ {
+# if (__GNUC__ >= 3) /* GCC Intrinsic */
+ return __builtin_ctz(val);
+# else /* Software version */
+ U32 count = 0;
+ while ((val & 1) == 0) {
+ val >>= 1;
+ ++count;
+ }
+ return count;
+# endif
+ }
+}
+
+FORCE_INLINE_TEMPLATE
+size_t FSE_readNCount_body(short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
+ const void* headerBuffer, size_t hbSize)
+{
+ const BYTE* const istart = (const BYTE*) headerBuffer;
+ const BYTE* const iend = istart + hbSize;
+ const BYTE* ip = istart;
+ int nbBits;
+ int remaining;
+ int threshold;
+ U32 bitStream;
+ int bitCount;
+ unsigned charnum = 0;
+ unsigned const maxSV1 = *maxSVPtr + 1;
+ int previous0 = 0;
+
+ if (hbSize < 8) {
+ /* This function only works when hbSize >= 8 */
+ char buffer[8] = {0};
+ ZSTD_memcpy(buffer, headerBuffer, hbSize);
+ { size_t const countSize = FSE_readNCount(normalizedCounter, maxSVPtr, tableLogPtr,
+ buffer, sizeof(buffer));
+ if (FSE_isError(countSize)) return countSize;
+ if (countSize > hbSize) return ERROR(corruption_detected);
+ return countSize;
+ } }
+ assert(hbSize >= 8);
+
+ /* init */
+ ZSTD_memset(normalizedCounter, 0, (*maxSVPtr+1) * sizeof(normalizedCounter[0])); /* all symbols not present in NCount have a frequency of 0 */
+ bitStream = MEM_readLE32(ip);
+ nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG; /* extract tableLog */
+ if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX) return ERROR(tableLog_tooLarge);
+ bitStream >>= 4;
+ bitCount = 4;
+ *tableLogPtr = nbBits;
+ remaining = (1<<nbBits)+1;
+ threshold = 1<<nbBits;
+ nbBits++;
+
+ for (;;) {
+ if (previous0) {
+ /* Count the number of repeats. Each time the
+ * 2-bit repeat code is 0b11 there is another
+ * repeat.
+ * Avoid UB by setting the high bit to 1.
+ */
+ int repeats = FSE_ctz(~bitStream | 0x80000000) >> 1;
+ while (repeats >= 12) {
+ charnum += 3 * 12;
+ if (LIKELY(ip <= iend-7)) {
+ ip += 3;
+ } else {
+ bitCount -= (int)(8 * (iend - 7 - ip));
+ bitCount &= 31;
+ ip = iend - 4;
+ }
+ bitStream = MEM_readLE32(ip) >> bitCount;
+ repeats = FSE_ctz(~bitStream | 0x80000000) >> 1;
+ }
+ charnum += 3 * repeats;
+ bitStream >>= 2 * repeats;
+ bitCount += 2 * repeats;
+
+ /* Add the final repeat which isn't 0b11. */
+ assert((bitStream & 3) < 3);
+ charnum += bitStream & 3;
+ bitCount += 2;
+
+ /* This is an error, but break and return an error
+ * at the end, because returning out of a loop makes
+ * it harder for the compiler to optimize.
+ */
+ if (charnum >= maxSV1) break;
+
+ /* We don't need to set the normalized count to 0
+ * because we already memset the whole buffer to 0.
+ */
+
+ if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
+ assert((bitCount >> 3) <= 3); /* For first condition to work */
+ ip += bitCount>>3;
+ bitCount &= 7;
+ } else {
+ bitCount -= (int)(8 * (iend - 4 - ip));
+ bitCount &= 31;
+ ip = iend - 4;
+ }
+ bitStream = MEM_readLE32(ip) >> bitCount;
+ }
+ {
+ int const max = (2*threshold-1) - remaining;
+ int count;
+
+ if ((bitStream & (threshold-1)) < (U32)max) {
+ count = bitStream & (threshold-1);
+ bitCount += nbBits-1;
+ } else {
+ count = bitStream & (2*threshold-1);
+ if (count >= threshold) count -= max;
+ bitCount += nbBits;
+ }
+
+ count--; /* extra accuracy */
+ /* When it matters (small blocks), this is a
+ * predictable branch, because we don't use -1.
+ */
+ if (count >= 0) {
+ remaining -= count;
+ } else {
+ assert(count == -1);
+ remaining += count;
+ }
+ normalizedCounter[charnum++] = (short)count;
+ previous0 = !count;
+
+ assert(threshold > 1);
+ if (remaining < threshold) {
+ /* This branch can be folded into the
+ * threshold update condition because we
+ * know that threshold > 1.
+ */
+ if (remaining <= 1) break;
+ nbBits = BIT_highbit32(remaining) + 1;
+ threshold = 1 << (nbBits - 1);
+ }
+ if (charnum >= maxSV1) break;
+
+ if (LIKELY(ip <= iend-7) || (ip + (bitCount>>3) <= iend-4)) {
+ ip += bitCount>>3;
+ bitCount &= 7;
+ } else {
+ bitCount -= (int)(8 * (iend - 4 - ip));
+ bitCount &= 31;
+ ip = iend - 4;
+ }
+ bitStream = MEM_readLE32(ip) >> bitCount;
+ } }
+ if (remaining != 1) return ERROR(corruption_detected);
+ /* Only possible when there are too many zeros. */
+ if (charnum > maxSV1) return ERROR(maxSymbolValue_tooSmall);
+ if (bitCount > 32) return ERROR(corruption_detected);
+ *maxSVPtr = charnum-1;
+
+ ip += (bitCount+7)>>3;
+ return ip-istart;
+}
+
+/* Avoids the FORCE_INLINE of the _body() function. */
+static size_t FSE_readNCount_body_default(
+ short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
+ const void* headerBuffer, size_t hbSize)
+{
+ return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
+}
+
+#if DYNAMIC_BMI2
+TARGET_ATTRIBUTE("bmi2") static size_t FSE_readNCount_body_bmi2(
+ short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
+ const void* headerBuffer, size_t hbSize)
+{
+ return FSE_readNCount_body(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
+}
+#endif
+
+size_t FSE_readNCount_bmi2(
+ short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
+ const void* headerBuffer, size_t hbSize, int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+ return FSE_readNCount_body_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
+ }
+#endif
+ (void)bmi2;
+ return FSE_readNCount_body_default(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize);
+}
+
+size_t FSE_readNCount(
+ short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
+ const void* headerBuffer, size_t hbSize)
+{
+ return FSE_readNCount_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize, /* bmi2 */ 0);
+}
+
+
+/*! HUF_readStats() :
+ Read compact Huffman tree, saved by HUF_writeCTable().
+ `huffWeight` is destination buffer.
+ `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
+ @return : size read from `src` , or an error Code .
+ Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
+*/
+size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+ U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize)
+{
+ U32 wksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
+ return HUF_readStats_wksp(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, wksp, sizeof(wksp), /* bmi2 */ 0);
+}
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+ U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize,
+ int bmi2)
+{
+ U32 weightTotal;
+ const BYTE* ip = (const BYTE*) src;
+ size_t iSize;
+ size_t oSize;
+
+ if (!srcSize) return ERROR(srcSize_wrong);
+ iSize = ip[0];
+ /* ZSTD_memset(huffWeight, 0, hwSize); *//* is not necessary, even though some analyzer complain ... */
+
+ if (iSize >= 128) { /* special header */
+ oSize = iSize - 127;
+ iSize = ((oSize+1)/2);
+ if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
+ if (oSize >= hwSize) return ERROR(corruption_detected);
+ ip += 1;
+ { U32 n;
+ for (n=0; n<oSize; n+=2) {
+ huffWeight[n] = ip[n/2] >> 4;
+ huffWeight[n+1] = ip[n/2] & 15;
+ } } }
+ else { /* header compressed with FSE (normal case) */
+ if (iSize+1 > srcSize) return ERROR(srcSize_wrong);
+ /* max (hwSize-1) values decoded, as last one is implied */
+ oSize = FSE_decompress_wksp_bmi2(huffWeight, hwSize-1, ip+1, iSize, 6, workSpace, wkspSize, bmi2);
+ if (FSE_isError(oSize)) return oSize;
+ }
+
+ /* collect weight stats */
+ ZSTD_memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
+ weightTotal = 0;
+ { U32 n; for (n=0; n<oSize; n++) {
+ if (huffWeight[n] >= HUF_TABLELOG_MAX) return ERROR(corruption_detected);
+ rankStats[huffWeight[n]]++;
+ weightTotal += (1 << huffWeight[n]) >> 1;
+ } }
+ if (weightTotal == 0) return ERROR(corruption_detected);
+
+ /* get last non-null symbol weight (implied, total must be 2^n) */
+ { U32 const tableLog = BIT_highbit32(weightTotal) + 1;
+ if (tableLog > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
+ *tableLogPtr = tableLog;
+ /* determine last weight */
+ { U32 const total = 1 << tableLog;
+ U32 const rest = total - weightTotal;
+ U32 const verif = 1 << BIT_highbit32(rest);
+ U32 const lastWeight = BIT_highbit32(rest) + 1;
+ if (verif != rest) return ERROR(corruption_detected); /* last value must be a clean power of 2 */
+ huffWeight[oSize] = (BYTE)lastWeight;
+ rankStats[lastWeight]++;
+ } }
+
+ /* check tree construction validity */
+ if ((rankStats[1] < 2) || (rankStats[1] & 1)) return ERROR(corruption_detected); /* by construction : at least 2 elts of rank 1, must be even */
+
+ /* results */
+ *nbSymbolsPtr = (U32)(oSize+1);
+ return iSize+1;
+}
+
+/* Avoids the FORCE_INLINE of the _body() function. */
+static size_t HUF_readStats_body_default(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+ U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 0);
+}
+
+#if DYNAMIC_BMI2
+static TARGET_ATTRIBUTE("bmi2") size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+ U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_readStats_body(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize, 1);
+}
+#endif
+
+size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+ U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize,
+ int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+ return HUF_readStats_body_bmi2(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
+ }
+#endif
+ (void)bmi2;
+ return HUF_readStats_body_default(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/* The purpose of this file is to have a single list of error strings embedded in binary */
+
+#include "error_private.h"
+
+const char* ERR_getErrorString(ERR_enum code)
+{
+#ifdef ZSTD_STRIP_ERROR_STRINGS
+ (void)code;
+ return "Error strings stripped";
+#else
+ static const char* const notErrorCode = "Unspecified error code";
+ switch( code )
+ {
+ case PREFIX(no_error): return "No error detected";
+ case PREFIX(GENERIC): return "Error (generic)";
+ case PREFIX(prefix_unknown): return "Unknown frame descriptor";
+ case PREFIX(version_unsupported): return "Version not supported";
+ case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter";
+ case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding";
+ case PREFIX(corruption_detected): return "Corrupted block detected";
+ case PREFIX(checksum_wrong): return "Restored data doesn't match checksum";
+ case PREFIX(parameter_unsupported): return "Unsupported parameter";
+ case PREFIX(parameter_outOfBound): return "Parameter is out of bound";
+ case PREFIX(init_missing): return "Context should be init first";
+ case PREFIX(memory_allocation): return "Allocation error : not enough memory";
+ case PREFIX(workSpace_tooSmall): return "workSpace buffer is not large enough";
+ case PREFIX(stage_wrong): return "Operation not authorized at current processing stage";
+ case PREFIX(tableLog_tooLarge): return "tableLog requires too much memory : unsupported";
+ case PREFIX(maxSymbolValue_tooLarge): return "Unsupported max Symbol Value : too large";
+ case PREFIX(maxSymbolValue_tooSmall): return "Specified maxSymbolValue is too small";
+ case PREFIX(dictionary_corrupted): return "Dictionary is corrupted";
+ case PREFIX(dictionary_wrong): return "Dictionary mismatch";
+ case PREFIX(dictionaryCreation_failed): return "Cannot create Dictionary from provided samples";
+ case PREFIX(dstSize_tooSmall): return "Destination buffer is too small";
+ case PREFIX(srcSize_wrong): return "Src size is incorrect";
+ case PREFIX(dstBuffer_null): return "Operation on NULL destination buffer";
+ /* following error codes are not stable and may be removed or changed in a future version */
+ case PREFIX(frameIndex_tooLarge): return "Frame index is too large";
+ case PREFIX(seekableIO): return "An I/O error occurred when reading/seeking";
+ case PREFIX(dstBuffer_wrong): return "Destination buffer is wrong";
+ case PREFIX(srcBuffer_wrong): return "Source buffer is wrong";
+ case PREFIX(maxCode):
+ default: return notErrorCode;
+ }
+#endif
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/* Note : this module is expected to remain private, do not expose it */
+
+#ifndef ERROR_H_MODULE
+#define ERROR_H_MODULE
+
+
+
+/* ****************************************
+* Dependencies
+******************************************/
+#include "zstd_deps.h" /* size_t */
+#include <linux/zstd_errors.h> /* enum list */
+
+
+/* ****************************************
+* Compiler-specific
+******************************************/
+#define ERR_STATIC static __attribute__((unused))
+
+
+/*-****************************************
+* Customization (error_public.h)
+******************************************/
+typedef ZSTD_ErrorCode ERR_enum;
+#define PREFIX(name) ZSTD_error_##name
+
+
+/*-****************************************
+* Error codes handling
+******************************************/
+#undef ERROR /* already defined on Visual Studio */
+#define ERROR(name) ZSTD_ERROR(name)
+#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
+
+ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
+
+ERR_STATIC ERR_enum ERR_getErrorCode(size_t code) { if (!ERR_isError(code)) return (ERR_enum)0; return (ERR_enum) (0-code); }
+
+/* check and forward error code */
+#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
+#define CHECK_F(f) { CHECK_V_F(_var_err__, f); }
+
+
+/*-****************************************
+* Error Strings
+******************************************/
+
+const char* ERR_getErrorString(ERR_enum code); /* error_private.c */
+
+ERR_STATIC const char* ERR_getErrorName(size_t code)
+{
+ return ERR_getErrorString(ERR_getErrorCode(code));
+}
+
+
+#endif /* ERROR_H_MODULE */
--- /dev/null
+/* ******************************************************************
+ * FSE : Finite State Entropy codec
+ * Public Prototypes declaration
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+
+#ifndef FSE_H
+#define FSE_H
+
+
+/*-*****************************************
+* Dependencies
+******************************************/
+#include "zstd_deps.h" /* size_t, ptrdiff_t */
+
+
+/*-*****************************************
+* FSE_PUBLIC_API : control library symbols visibility
+******************************************/
+#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
+# define FSE_PUBLIC_API __attribute__ ((visibility ("default")))
+#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */
+# define FSE_PUBLIC_API __declspec(dllexport)
+#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
+# define FSE_PUBLIC_API __declspec(dllimport) /* It isn't required but allows to generate better code, saving a function pointer load from the IAT and an indirect jump.*/
+#else
+# define FSE_PUBLIC_API
+#endif
+
+/*------ Version ------*/
+#define FSE_VERSION_MAJOR 0
+#define FSE_VERSION_MINOR 9
+#define FSE_VERSION_RELEASE 0
+
+#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
+#define FSE_QUOTE(str) #str
+#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
+#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)
+
+#define FSE_VERSION_NUMBER (FSE_VERSION_MAJOR *100*100 + FSE_VERSION_MINOR *100 + FSE_VERSION_RELEASE)
+FSE_PUBLIC_API unsigned FSE_versionNumber(void); /*< library version number; to be used when checking dll version */
+
+
+/*-****************************************
+* FSE simple functions
+******************************************/
+/*! FSE_compress() :
+ Compress content of buffer 'src', of size 'srcSize', into destination buffer 'dst'.
+ 'dst' buffer must be already allocated. Compression runs faster is dstCapacity >= FSE_compressBound(srcSize).
+ @return : size of compressed data (<= dstCapacity).
+ Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
+ if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression instead.
+ if FSE_isError(return), compression failed (more details using FSE_getErrorName())
+*/
+FSE_PUBLIC_API size_t FSE_compress(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize);
+
+/*! FSE_decompress():
+ Decompress FSE data from buffer 'cSrc', of size 'cSrcSize',
+ into already allocated destination buffer 'dst', of size 'dstCapacity'.
+ @return : size of regenerated data (<= maxDstSize),
+ or an error code, which can be tested using FSE_isError() .
+
+ ** Important ** : FSE_decompress() does not decompress non-compressible nor RLE data !!!
+ Why ? : making this distinction requires a header.
+ Header management is intentionally delegated to the user layer, which can better manage special cases.
+*/
+FSE_PUBLIC_API size_t FSE_decompress(void* dst, size_t dstCapacity,
+ const void* cSrc, size_t cSrcSize);
+
+
+/*-*****************************************
+* Tool functions
+******************************************/
+FSE_PUBLIC_API size_t FSE_compressBound(size_t size); /* maximum compressed size */
+
+/* Error Management */
+FSE_PUBLIC_API unsigned FSE_isError(size_t code); /* tells if a return value is an error code */
+FSE_PUBLIC_API const char* FSE_getErrorName(size_t code); /* provides error code string (useful for debugging) */
+
+
+/*-*****************************************
+* FSE advanced functions
+******************************************/
+/*! FSE_compress2() :
+ Same as FSE_compress(), but allows the selection of 'maxSymbolValue' and 'tableLog'
+ Both parameters can be defined as '0' to mean : use default value
+ @return : size of compressed data
+ Special values : if return == 0, srcData is not compressible => Nothing is stored within cSrc !!!
+ if return == 1, srcData is a single byte symbol * srcSize times. Use RLE compression.
+ if FSE_isError(return), it's an error code.
+*/
+FSE_PUBLIC_API size_t FSE_compress2 (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
+
+
+/*-*****************************************
+* FSE detailed API
+******************************************/
+/*!
+FSE_compress() does the following:
+1. count symbol occurrence from source[] into table count[] (see hist.h)
+2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
+3. save normalized counters to memory buffer using writeNCount()
+4. build encoding table 'CTable' from normalized counters
+5. encode the data stream using encoding table 'CTable'
+
+FSE_decompress() does the following:
+1. read normalized counters with readNCount()
+2. build decoding table 'DTable' from normalized counters
+3. decode the data stream using decoding table 'DTable'
+
+The following API allows targeting specific sub-functions for advanced tasks.
+For example, it's possible to compress several blocks using the same 'CTable',
+or to save and provide normalized distribution using external method.
+*/
+
+/* *** COMPRESSION *** */
+
+/*! FSE_optimalTableLog():
+ dynamically downsize 'tableLog' when conditions are met.
+ It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
+ @return : recommended tableLog (necessarily <= 'maxTableLog') */
+FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
+
+/*! FSE_normalizeCount():
+ normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
+ 'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
+ useLowProbCount is a boolean parameter which trades off compressed size for
+ faster header decoding. When it is set to 1, the compressed data will be slightly
+ smaller. And when it is set to 0, FSE_readNCount() and FSE_buildDTable() will be
+ faster. If you are compressing a small amount of data (< 2 KB) then useLowProbCount=0
+ is a good default, since header deserialization makes a big speed difference.
+ Otherwise, useLowProbCount=1 is a good default, since the speed difference is small.
+ @return : tableLog,
+ or an errorCode, which can be tested using FSE_isError() */
+FSE_PUBLIC_API size_t FSE_normalizeCount(short* normalizedCounter, unsigned tableLog,
+ const unsigned* count, size_t srcSize, unsigned maxSymbolValue, unsigned useLowProbCount);
+
+/*! FSE_NCountWriteBound():
+ Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
+ Typically useful for allocation purpose. */
+FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);
+
+/*! FSE_writeNCount():
+ Compactly save 'normalizedCounter' into 'buffer'.
+ @return : size of the compressed table,
+ or an errorCode, which can be tested using FSE_isError(). */
+FSE_PUBLIC_API size_t FSE_writeNCount (void* buffer, size_t bufferSize,
+ const short* normalizedCounter,
+ unsigned maxSymbolValue, unsigned tableLog);
+
+/*! Constructor and Destructor of FSE_CTable.
+ Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
+typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */
+FSE_PUBLIC_API FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog);
+FSE_PUBLIC_API void FSE_freeCTable (FSE_CTable* ct);
+
+/*! FSE_buildCTable():
+ Builds `ct`, which must be already allocated, using FSE_createCTable().
+ @return : 0, or an errorCode, which can be tested using FSE_isError() */
+FSE_PUBLIC_API size_t FSE_buildCTable(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
+
+/*! FSE_compress_usingCTable():
+ Compress `src` using `ct` into `dst` which must be already allocated.
+ @return : size of compressed data (<= `dstCapacity`),
+ or 0 if compressed data could not fit into `dst`,
+ or an errorCode, which can be tested using FSE_isError() */
+FSE_PUBLIC_API size_t FSE_compress_usingCTable (void* dst, size_t dstCapacity, const void* src, size_t srcSize, const FSE_CTable* ct);
+
+/*!
+Tutorial :
+----------
+The first step is to count all symbols. FSE_count() does this job very fast.
+Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
+'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
+maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
+FSE_count() will return the number of occurrence of the most frequent symbol.
+This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
+If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
+
+The next step is to normalize the frequencies.
+FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
+It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
+You can use 'tableLog'==0 to mean "use default tableLog value".
+If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
+which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").
+
+The result of FSE_normalizeCount() will be saved into a table,
+called 'normalizedCounter', which is a table of signed short.
+'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
+The return value is tableLog if everything proceeded as expected.
+It is 0 if there is a single symbol within distribution.
+If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).
+
+'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
+'buffer' must be already allocated.
+For guaranteed success, buffer size must be at least FSE_headerBound().
+The result of the function is the number of bytes written into 'buffer'.
+If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).
+
+'normalizedCounter' can then be used to create the compression table 'CTable'.
+The space required by 'CTable' must be already allocated, using FSE_createCTable().
+You can then use FSE_buildCTable() to fill 'CTable'.
+If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).
+
+'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
+Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
+The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
+If it returns '0', compressed data could not fit into 'dst'.
+If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
+*/
+
+
+/* *** DECOMPRESSION *** */
+
+/*! FSE_readNCount():
+ Read compactly saved 'normalizedCounter' from 'rBuffer'.
+ @return : size read from 'rBuffer',
+ or an errorCode, which can be tested using FSE_isError().
+ maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
+FSE_PUBLIC_API size_t FSE_readNCount (short* normalizedCounter,
+ unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
+ const void* rBuffer, size_t rBuffSize);
+
+/*! FSE_readNCount_bmi2():
+ * Same as FSE_readNCount() but pass bmi2=1 when your CPU supports BMI2 and 0 otherwise.
+ */
+FSE_PUBLIC_API size_t FSE_readNCount_bmi2(short* normalizedCounter,
+ unsigned* maxSymbolValuePtr, unsigned* tableLogPtr,
+ const void* rBuffer, size_t rBuffSize, int bmi2);
+
+/*! Constructor and Destructor of FSE_DTable.
+ Note that its size depends on 'tableLog' */
+typedef unsigned FSE_DTable; /* don't allocate that. It's just a way to be more restrictive than void* */
+FSE_PUBLIC_API FSE_DTable* FSE_createDTable(unsigned tableLog);
+FSE_PUBLIC_API void FSE_freeDTable(FSE_DTable* dt);
+
+/*! FSE_buildDTable():
+ Builds 'dt', which must be already allocated, using FSE_createDTable().
+ return : 0, or an errorCode, which can be tested using FSE_isError() */
+FSE_PUBLIC_API size_t FSE_buildDTable (FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
+
+/*! FSE_decompress_usingDTable():
+ Decompress compressed source `cSrc` of size `cSrcSize` using `dt`
+ into `dst` which must be already allocated.
+ @return : size of regenerated data (necessarily <= `dstCapacity`),
+ or an errorCode, which can be tested using FSE_isError() */
+FSE_PUBLIC_API size_t FSE_decompress_usingDTable(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, const FSE_DTable* dt);
+
+/*!
+Tutorial :
+----------
+(Note : these functions only decompress FSE-compressed blocks.
+ If block is uncompressed, use memcpy() instead
+ If block is a single repeated byte, use memset() instead )
+
+The first step is to obtain the normalized frequencies of symbols.
+This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
+'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
+In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
+or size the table to handle worst case situations (typically 256).
+FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
+The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
+Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
+If there is an error, the function will return an error code, which can be tested using FSE_isError().
+
+The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
+This is performed by the function FSE_buildDTable().
+The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
+If there is an error, the function will return an error code, which can be tested using FSE_isError().
+
+`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
+`cSrcSize` must be strictly correct, otherwise decompression will fail.
+FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
+If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
+*/
+
+#endif /* FSE_H */
+
+#if !defined(FSE_H_FSE_STATIC_LINKING_ONLY)
+#define FSE_H_FSE_STATIC_LINKING_ONLY
+
+/* *** Dependency *** */
+#include "bitstream.h"
+
+
+/* *****************************************
+* Static allocation
+*******************************************/
+/* FSE buffer bounds */
+#define FSE_NCOUNTBOUND 512
+#define FSE_BLOCKBOUND(size) ((size) + ((size)>>7) + 4 /* fse states */ + sizeof(size_t) /* bitContainer */)
+#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
+
+/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
+#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1<<((maxTableLog)-1)) + (((maxSymbolValue)+1)*2))
+#define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1<<(maxTableLog)))
+
+/* or use the size to malloc() space directly. Pay attention to alignment restrictions though */
+#define FSE_CTABLE_SIZE(maxTableLog, maxSymbolValue) (FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(FSE_CTable))
+#define FSE_DTABLE_SIZE(maxTableLog) (FSE_DTABLE_SIZE_U32(maxTableLog) * sizeof(FSE_DTable))
+
+
+/* *****************************************
+ * FSE advanced API
+ ***************************************** */
+
+unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
+/*< same as FSE_optimalTableLog(), which used `minus==2` */
+
+/* FSE_compress_wksp() :
+ * Same as FSE_compress2(), but using an externally allocated scratch buffer (`workSpace`).
+ * FSE_COMPRESS_WKSP_SIZE_U32() provides the minimum size required for `workSpace` as a table of FSE_CTable.
+ */
+#define FSE_COMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ( FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) + ((maxTableLog > 12) ? (1 << (maxTableLog - 2)) : 1024) )
+size_t FSE_compress_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
+
+size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits);
+/*< build a fake FSE_CTable, designed for a flat distribution, where each symbol uses nbBits */
+
+size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
+/*< build a fake FSE_CTable, designed to compress always the same symbolValue */
+
+/* FSE_buildCTable_wksp() :
+ * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
+ * `wkspSize` must be >= `FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)` of `unsigned`.
+ */
+#define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (maxSymbolValue + 2 + (1ull << (tableLog - 2)))
+#define FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) (sizeof(unsigned) * FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog))
+size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
+
+#define FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) (sizeof(short) * (maxSymbolValue + 1) + (1ULL << maxTableLog) + 8)
+#define FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) ((FSE_BUILD_DTABLE_WKSP_SIZE(maxTableLog, maxSymbolValue) + sizeof(unsigned) - 1) / sizeof(unsigned))
+FSE_PUBLIC_API size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
+/*< Same as FSE_buildDTable(), using an externally allocated `workspace` produced with `FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxSymbolValue)` */
+
+size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits);
+/*< build a fake FSE_DTable, designed to read a flat distribution where each symbol uses nbBits */
+
+size_t FSE_buildDTable_rle (FSE_DTable* dt, unsigned char symbolValue);
+/*< build a fake FSE_DTable, designed to always generate the same symbolValue */
+
+#define FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) (FSE_DTABLE_SIZE_U32(maxTableLog) + FSE_BUILD_DTABLE_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) + (FSE_MAX_SYMBOL_VALUE + 1) / 2 + 1)
+#define FSE_DECOMPRESS_WKSP_SIZE(maxTableLog, maxSymbolValue) (FSE_DECOMPRESS_WKSP_SIZE_U32(maxTableLog, maxSymbolValue) * sizeof(unsigned))
+size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize);
+/*< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DECOMPRESS_WKSP_SIZE_U32(maxLog, maxSymbolValue)` */
+
+size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2);
+/*< Same as FSE_decompress_wksp() but with dynamic BMI2 support. Pass 1 if your CPU supports BMI2 or 0 if it doesn't. */
+
+typedef enum {
+ FSE_repeat_none, /*< Cannot use the previous table */
+ FSE_repeat_check, /*< Can use the previous table but it must be checked */
+ FSE_repeat_valid /*< Can use the previous table and it is assumed to be valid */
+ } FSE_repeat;
+
+/* *****************************************
+* FSE symbol compression API
+*******************************************/
+/*!
+ This API consists of small unitary functions, which highly benefit from being inlined.
+ Hence their body are included in next section.
+*/
+typedef struct {
+ ptrdiff_t value;
+ const void* stateTable;
+ const void* symbolTT;
+ unsigned stateLog;
+} FSE_CState_t;
+
+static void FSE_initCState(FSE_CState_t* CStatePtr, const FSE_CTable* ct);
+
+static void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* CStatePtr, unsigned symbol);
+
+static void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* CStatePtr);
+
+/*<
+These functions are inner components of FSE_compress_usingCTable().
+They allow the creation of custom streams, mixing multiple tables and bit sources.
+
+A key property to keep in mind is that encoding and decoding are done **in reverse direction**.
+So the first symbol you will encode is the last you will decode, like a LIFO stack.
+
+You will need a few variables to track your CStream. They are :
+
+FSE_CTable ct; // Provided by FSE_buildCTable()
+BIT_CStream_t bitStream; // bitStream tracking structure
+FSE_CState_t state; // State tracking structure (can have several)
+
+
+The first thing to do is to init bitStream and state.
+ size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
+ FSE_initCState(&state, ct);
+
+Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
+You can then encode your input data, byte after byte.
+FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
+Remember decoding will be done in reverse direction.
+ FSE_encodeByte(&bitStream, &state, symbol);
+
+At any time, you can also add any bit sequence.
+Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
+ BIT_addBits(&bitStream, bitField, nbBits);
+
+The above methods don't commit data to memory, they just store it into local register, for speed.
+Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
+Writing data to memory is a manual operation, performed by the flushBits function.
+ BIT_flushBits(&bitStream);
+
+Your last FSE encoding operation shall be to flush your last state value(s).
+ FSE_flushState(&bitStream, &state);
+
+Finally, you must close the bitStream.
+The function returns the size of CStream in bytes.
+If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
+If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
+ size_t size = BIT_closeCStream(&bitStream);
+*/
+
+
+/* *****************************************
+* FSE symbol decompression API
+*******************************************/
+typedef struct {
+ size_t state;
+ const void* table; /* precise table may vary, depending on U16 */
+} FSE_DState_t;
+
+
+static void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt);
+
+static unsigned char FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
+
+static unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr);
+
+/*<
+Let's now decompose FSE_decompress_usingDTable() into its unitary components.
+You will decode FSE-encoded symbols from the bitStream,
+and also any other bitFields you put in, **in reverse order**.
+
+You will need a few variables to track your bitStream. They are :
+
+BIT_DStream_t DStream; // Stream context
+FSE_DState_t DState; // State context. Multiple ones are possible
+FSE_DTable* DTablePtr; // Decoding table, provided by FSE_buildDTable()
+
+The first thing to do is to init the bitStream.
+ errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);
+
+You should then retrieve your initial state(s)
+(in reverse flushing order if you have several ones) :
+ errorCode = FSE_initDState(&DState, &DStream, DTablePtr);
+
+You can then decode your data, symbol after symbol.
+For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
+Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
+ unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);
+
+You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
+Note : maximum allowed nbBits is 25, for 32-bits compatibility
+ size_t bitField = BIT_readBits(&DStream, nbBits);
+
+All above operations only read from local register (which size depends on size_t).
+Refueling the register from memory is manually performed by the reload method.
+ endSignal = FSE_reloadDStream(&DStream);
+
+BIT_reloadDStream() result tells if there is still some more data to read from DStream.
+BIT_DStream_unfinished : there is still some data left into the DStream.
+BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
+BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
+BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.
+
+When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
+to properly detect the exact end of stream.
+After each decoded symbol, check if DStream is fully consumed using this simple test :
+ BIT_reloadDStream(&DStream) >= BIT_DStream_completed
+
+When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
+Checking if DStream has reached its end is performed by :
+ BIT_endOfDStream(&DStream);
+Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
+ FSE_endOfDState(&DState);
+*/
+
+
+/* *****************************************
+* FSE unsafe API
+*******************************************/
+static unsigned char FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD);
+/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */
+
+
+/* *****************************************
+* Implementation of inlined functions
+*******************************************/
+typedef struct {
+ int deltaFindState;
+ U32 deltaNbBits;
+} FSE_symbolCompressionTransform; /* total 8 bytes */
+
+MEM_STATIC void FSE_initCState(FSE_CState_t* statePtr, const FSE_CTable* ct)
+{
+ const void* ptr = ct;
+ const U16* u16ptr = (const U16*) ptr;
+ const U32 tableLog = MEM_read16(ptr);
+ statePtr->value = (ptrdiff_t)1<<tableLog;
+ statePtr->stateTable = u16ptr+2;
+ statePtr->symbolTT = ct + 1 + (tableLog ? (1<<(tableLog-1)) : 1);
+ statePtr->stateLog = tableLog;
+}
+
+
+/*! FSE_initCState2() :
+* Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
+* uses the smallest state value possible, saving the cost of this symbol */
+MEM_STATIC void FSE_initCState2(FSE_CState_t* statePtr, const FSE_CTable* ct, U32 symbol)
+{
+ FSE_initCState(statePtr, ct);
+ { const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
+ const U16* stateTable = (const U16*)(statePtr->stateTable);
+ U32 nbBitsOut = (U32)((symbolTT.deltaNbBits + (1<<15)) >> 16);
+ statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
+ statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
+ }
+}
+
+MEM_STATIC void FSE_encodeSymbol(BIT_CStream_t* bitC, FSE_CState_t* statePtr, unsigned symbol)
+{
+ FSE_symbolCompressionTransform const symbolTT = ((const FSE_symbolCompressionTransform*)(statePtr->symbolTT))[symbol];
+ const U16* const stateTable = (const U16*)(statePtr->stateTable);
+ U32 const nbBitsOut = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
+ BIT_addBits(bitC, statePtr->value, nbBitsOut);
+ statePtr->value = stateTable[ (statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
+}
+
+MEM_STATIC void FSE_flushCState(BIT_CStream_t* bitC, const FSE_CState_t* statePtr)
+{
+ BIT_addBits(bitC, statePtr->value, statePtr->stateLog);
+ BIT_flushBits(bitC);
+}
+
+
+/* FSE_getMaxNbBits() :
+ * Approximate maximum cost of a symbol, in bits.
+ * Fractional get rounded up (i.e : a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
+ * note 1 : assume symbolValue is valid (<= maxSymbolValue)
+ * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
+MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue)
+{
+ const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
+ return (symbolTT[symbolValue].deltaNbBits + ((1<<16)-1)) >> 16;
+}
+
+/* FSE_bitCost() :
+ * Approximate symbol cost, as fractional value, using fixed-point format (accuracyLog fractional bits)
+ * note 1 : assume symbolValue is valid (<= maxSymbolValue)
+ * note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
+MEM_STATIC U32 FSE_bitCost(const void* symbolTTPtr, U32 tableLog, U32 symbolValue, U32 accuracyLog)
+{
+ const FSE_symbolCompressionTransform* symbolTT = (const FSE_symbolCompressionTransform*) symbolTTPtr;
+ U32 const minNbBits = symbolTT[symbolValue].deltaNbBits >> 16;
+ U32 const threshold = (minNbBits+1) << 16;
+ assert(tableLog < 16);
+ assert(accuracyLog < 31-tableLog); /* ensure enough room for renormalization double shift */
+ { U32 const tableSize = 1 << tableLog;
+ U32 const deltaFromThreshold = threshold - (symbolTT[symbolValue].deltaNbBits + tableSize);
+ U32 const normalizedDeltaFromThreshold = (deltaFromThreshold << accuracyLog) >> tableLog; /* linear interpolation (very approximate) */
+ U32 const bitMultiplier = 1 << accuracyLog;
+ assert(symbolTT[symbolValue].deltaNbBits + tableSize <= threshold);
+ assert(normalizedDeltaFromThreshold <= bitMultiplier);
+ return (minNbBits+1)*bitMultiplier - normalizedDeltaFromThreshold;
+ }
+}
+
+
+/* ====== Decompression ====== */
+
+typedef struct {
+ U16 tableLog;
+ U16 fastMode;
+} FSE_DTableHeader; /* sizeof U32 */
+
+typedef struct
+{
+ unsigned short newState;
+ unsigned char symbol;
+ unsigned char nbBits;
+} FSE_decode_t; /* size == U32 */
+
+MEM_STATIC void FSE_initDState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD, const FSE_DTable* dt)
+{
+ const void* ptr = dt;
+ const FSE_DTableHeader* const DTableH = (const FSE_DTableHeader*)ptr;
+ DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
+ BIT_reloadDStream(bitD);
+ DStatePtr->table = dt + 1;
+}
+
+MEM_STATIC BYTE FSE_peekSymbol(const FSE_DState_t* DStatePtr)
+{
+ FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
+ return DInfo.symbol;
+}
+
+MEM_STATIC void FSE_updateState(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
+{
+ FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
+ U32 const nbBits = DInfo.nbBits;
+ size_t const lowBits = BIT_readBits(bitD, nbBits);
+ DStatePtr->state = DInfo.newState + lowBits;
+}
+
+MEM_STATIC BYTE FSE_decodeSymbol(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
+{
+ FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
+ U32 const nbBits = DInfo.nbBits;
+ BYTE const symbol = DInfo.symbol;
+ size_t const lowBits = BIT_readBits(bitD, nbBits);
+
+ DStatePtr->state = DInfo.newState + lowBits;
+ return symbol;
+}
+
+/*! FSE_decodeSymbolFast() :
+ unsafe, only works if no symbol has a probability > 50% */
+MEM_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t* DStatePtr, BIT_DStream_t* bitD)
+{
+ FSE_decode_t const DInfo = ((const FSE_decode_t*)(DStatePtr->table))[DStatePtr->state];
+ U32 const nbBits = DInfo.nbBits;
+ BYTE const symbol = DInfo.symbol;
+ size_t const lowBits = BIT_readBitsFast(bitD, nbBits);
+
+ DStatePtr->state = DInfo.newState + lowBits;
+ return symbol;
+}
+
+MEM_STATIC unsigned FSE_endOfDState(const FSE_DState_t* DStatePtr)
+{
+ return DStatePtr->state == 0;
+}
+
+
+
+#ifndef FSE_COMMONDEFS_ONLY
+
+/* **************************************************************
+* Tuning parameters
+****************************************************************/
+/*!MEMORY_USAGE :
+* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
+* Increasing memory usage improves compression ratio
+* Reduced memory usage can improve speed, due to cache effect
+* Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
+#ifndef FSE_MAX_MEMORY_USAGE
+# define FSE_MAX_MEMORY_USAGE 14
+#endif
+#ifndef FSE_DEFAULT_MEMORY_USAGE
+# define FSE_DEFAULT_MEMORY_USAGE 13
+#endif
+#if (FSE_DEFAULT_MEMORY_USAGE > FSE_MAX_MEMORY_USAGE)
+# error "FSE_DEFAULT_MEMORY_USAGE must be <= FSE_MAX_MEMORY_USAGE"
+#endif
+
+/*!FSE_MAX_SYMBOL_VALUE :
+* Maximum symbol value authorized.
+* Required for proper stack allocation */
+#ifndef FSE_MAX_SYMBOL_VALUE
+# define FSE_MAX_SYMBOL_VALUE 255
+#endif
+
+/* **************************************************************
+* template functions type & suffix
+****************************************************************/
+#define FSE_FUNCTION_TYPE BYTE
+#define FSE_FUNCTION_EXTENSION
+#define FSE_DECODE_TYPE FSE_decode_t
+
+
+#endif /* !FSE_COMMONDEFS_ONLY */
+
+
+/* ***************************************************************
+* Constants
+*****************************************************************/
+#define FSE_MAX_TABLELOG (FSE_MAX_MEMORY_USAGE-2)
+#define FSE_MAX_TABLESIZE (1U<<FSE_MAX_TABLELOG)
+#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE-1)
+#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE-2)
+#define FSE_MIN_TABLELOG 5
+
+#define FSE_TABLELOG_ABSOLUTE_MAX 15
+#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
+# error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
+#endif
+
+#define FSE_TABLESTEP(tableSize) (((tableSize)>>1) + ((tableSize)>>3) + 3)
+
+
+#endif /* FSE_STATIC_LINKING_ONLY */
+
+
--- /dev/null
+/* ******************************************************************
+ * FSE : Finite State Entropy decoder
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+
+/* **************************************************************
+* Includes
+****************************************************************/
+#include "debug.h" /* assert */
+#include "bitstream.h"
+#include "compiler.h"
+#define FSE_STATIC_LINKING_ONLY
+#include "fse.h"
+#include "error_private.h"
+#define ZSTD_DEPS_NEED_MALLOC
+#include "zstd_deps.h"
+
+
+/* **************************************************************
+* Error Management
+****************************************************************/
+#define FSE_isError ERR_isError
+#define FSE_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */
+
+
+/* **************************************************************
+* Templates
+****************************************************************/
+/*
+ designed to be included
+ for type-specific functions (template emulation in C)
+ Objective is to write these functions only once, for improved maintenance
+*/
+
+/* safety checks */
+#ifndef FSE_FUNCTION_EXTENSION
+# error "FSE_FUNCTION_EXTENSION must be defined"
+#endif
+#ifndef FSE_FUNCTION_TYPE
+# error "FSE_FUNCTION_TYPE must be defined"
+#endif
+
+/* Function names */
+#define FSE_CAT(X,Y) X##Y
+#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
+#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
+
+
+/* Function templates */
+FSE_DTable* FSE_createDTable (unsigned tableLog)
+{
+ if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
+ return (FSE_DTable*)ZSTD_malloc( FSE_DTABLE_SIZE_U32(tableLog) * sizeof (U32) );
+}
+
+void FSE_freeDTable (FSE_DTable* dt)
+{
+ ZSTD_free(dt);
+}
+
+static size_t FSE_buildDTable_internal(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
+{
+ void* const tdPtr = dt+1; /* because *dt is unsigned, 32-bits aligned on 32-bits */
+ FSE_DECODE_TYPE* const tableDecode = (FSE_DECODE_TYPE*) (tdPtr);
+ U16* symbolNext = (U16*)workSpace;
+ BYTE* spread = (BYTE*)(symbolNext + maxSymbolValue + 1);
+
+ U32 const maxSV1 = maxSymbolValue + 1;
+ U32 const tableSize = 1 << tableLog;
+ U32 highThreshold = tableSize-1;
+
+ /* Sanity Checks */
+ if (FSE_BUILD_DTABLE_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(maxSymbolValue_tooLarge);
+ if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE) return ERROR(maxSymbolValue_tooLarge);
+ if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge);
+
+ /* Init, lay down lowprob symbols */
+ { FSE_DTableHeader DTableH;
+ DTableH.tableLog = (U16)tableLog;
+ DTableH.fastMode = 1;
+ { S16 const largeLimit= (S16)(1 << (tableLog-1));
+ U32 s;
+ for (s=0; s<maxSV1; s++) {
+ if (normalizedCounter[s]==-1) {
+ tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
+ symbolNext[s] = 1;
+ } else {
+ if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
+ symbolNext[s] = normalizedCounter[s];
+ } } }
+ ZSTD_memcpy(dt, &DTableH, sizeof(DTableH));
+ }
+
+ /* Spread symbols */
+ if (highThreshold == tableSize - 1) {
+ size_t const tableMask = tableSize-1;
+ size_t const step = FSE_TABLESTEP(tableSize);
+ /* First lay down the symbols in order.
+ * We use a uint64_t to lay down 8 bytes at a time. This reduces branch
+ * misses since small blocks generally have small table logs, so nearly
+ * all symbols have counts <= 8. We ensure we have 8 bytes at the end of
+ * our buffer to handle the over-write.
+ */
+ {
+ U64 const add = 0x0101010101010101ull;
+ size_t pos = 0;
+ U64 sv = 0;
+ U32 s;
+ for (s=0; s<maxSV1; ++s, sv += add) {
+ int i;
+ int const n = normalizedCounter[s];
+ MEM_write64(spread + pos, sv);
+ for (i = 8; i < n; i += 8) {
+ MEM_write64(spread + pos + i, sv);
+ }
+ pos += n;
+ }
+ }
+ /* Now we spread those positions across the table.
+ * The benefit of doing it in two stages is that we avoid the the
+ * variable size inner loop, which caused lots of branch misses.
+ * Now we can run through all the positions without any branch misses.
+ * We unroll the loop twice, since that is what emperically worked best.
+ */
+ {
+ size_t position = 0;
+ size_t s;
+ size_t const unroll = 2;
+ assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
+ for (s = 0; s < (size_t)tableSize; s += unroll) {
+ size_t u;
+ for (u = 0; u < unroll; ++u) {
+ size_t const uPosition = (position + (u * step)) & tableMask;
+ tableDecode[uPosition].symbol = spread[s + u];
+ }
+ position = (position + (unroll * step)) & tableMask;
+ }
+ assert(position == 0);
+ }
+ } else {
+ U32 const tableMask = tableSize-1;
+ U32 const step = FSE_TABLESTEP(tableSize);
+ U32 s, position = 0;
+ for (s=0; s<maxSV1; s++) {
+ int i;
+ for (i=0; i<normalizedCounter[s]; i++) {
+ tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
+ position = (position + step) & tableMask;
+ while (position > highThreshold) position = (position + step) & tableMask; /* lowprob area */
+ } }
+ if (position!=0) return ERROR(GENERIC); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
+ }
+
+ /* Build Decoding table */
+ { U32 u;
+ for (u=0; u<tableSize; u++) {
+ FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
+ U32 const nextState = symbolNext[symbol]++;
+ tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
+ tableDecode[u].newState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
+ } }
+
+ return 0;
+}
+
+size_t FSE_buildDTable_wksp(FSE_DTable* dt, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize)
+{
+ return FSE_buildDTable_internal(dt, normalizedCounter, maxSymbolValue, tableLog, workSpace, wkspSize);
+}
+
+
+#ifndef FSE_COMMONDEFS_ONLY
+
+/*-*******************************************************
+* Decompression (Byte symbols)
+*********************************************************/
+size_t FSE_buildDTable_rle (FSE_DTable* dt, BYTE symbolValue)
+{
+ void* ptr = dt;
+ FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
+ void* dPtr = dt + 1;
+ FSE_decode_t* const cell = (FSE_decode_t*)dPtr;
+
+ DTableH->tableLog = 0;
+ DTableH->fastMode = 0;
+
+ cell->newState = 0;
+ cell->symbol = symbolValue;
+ cell->nbBits = 0;
+
+ return 0;
+}
+
+
+size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits)
+{
+ void* ptr = dt;
+ FSE_DTableHeader* const DTableH = (FSE_DTableHeader*)ptr;
+ void* dPtr = dt + 1;
+ FSE_decode_t* const dinfo = (FSE_decode_t*)dPtr;
+ const unsigned tableSize = 1 << nbBits;
+ const unsigned tableMask = tableSize - 1;
+ const unsigned maxSV1 = tableMask+1;
+ unsigned s;
+
+ /* Sanity checks */
+ if (nbBits < 1) return ERROR(GENERIC); /* min size */
+
+ /* Build Decoding Table */
+ DTableH->tableLog = (U16)nbBits;
+ DTableH->fastMode = 1;
+ for (s=0; s<maxSV1; s++) {
+ dinfo[s].newState = 0;
+ dinfo[s].symbol = (BYTE)s;
+ dinfo[s].nbBits = (BYTE)nbBits;
+ }
+
+ return 0;
+}
+
+FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
+ void* dst, size_t maxDstSize,
+ const void* cSrc, size_t cSrcSize,
+ const FSE_DTable* dt, const unsigned fast)
+{
+ BYTE* const ostart = (BYTE*) dst;
+ BYTE* op = ostart;
+ BYTE* const omax = op + maxDstSize;
+ BYTE* const olimit = omax-3;
+
+ BIT_DStream_t bitD;
+ FSE_DState_t state1;
+ FSE_DState_t state2;
+
+ /* Init */
+ CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));
+
+ FSE_initDState(&state1, &bitD, dt);
+ FSE_initDState(&state2, &bitD, dt);
+
+#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)
+
+ /* 4 symbols per loop */
+ for ( ; (BIT_reloadDStream(&bitD)==BIT_DStream_unfinished) & (op<olimit) ; op+=4) {
+ op[0] = FSE_GETSYMBOL(&state1);
+
+ if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
+ BIT_reloadDStream(&bitD);
+
+ op[1] = FSE_GETSYMBOL(&state2);
+
+ if (FSE_MAX_TABLELOG*4+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
+ { if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) { op+=2; break; } }
+
+ op[2] = FSE_GETSYMBOL(&state1);
+
+ if (FSE_MAX_TABLELOG*2+7 > sizeof(bitD.bitContainer)*8) /* This test must be static */
+ BIT_reloadDStream(&bitD);
+
+ op[3] = FSE_GETSYMBOL(&state2);
+ }
+
+ /* tail */
+ /* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
+ while (1) {
+ if (op>(omax-2)) return ERROR(dstSize_tooSmall);
+ *op++ = FSE_GETSYMBOL(&state1);
+ if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
+ *op++ = FSE_GETSYMBOL(&state2);
+ break;
+ }
+
+ if (op>(omax-2)) return ERROR(dstSize_tooSmall);
+ *op++ = FSE_GETSYMBOL(&state2);
+ if (BIT_reloadDStream(&bitD)==BIT_DStream_overflow) {
+ *op++ = FSE_GETSYMBOL(&state1);
+ break;
+ } }
+
+ return op-ostart;
+}
+
+
+size_t FSE_decompress_usingDTable(void* dst, size_t originalSize,
+ const void* cSrc, size_t cSrcSize,
+ const FSE_DTable* dt)
+{
+ const void* ptr = dt;
+ const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
+ const U32 fastMode = DTableH->fastMode;
+
+ /* select fast mode (static) */
+ if (fastMode) return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 1);
+ return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 0);
+}
+
+
+size_t FSE_decompress_wksp(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
+{
+ return FSE_decompress_wksp_bmi2(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, /* bmi2 */ 0);
+}
+
+typedef struct {
+ short ncount[FSE_MAX_SYMBOL_VALUE + 1];
+ FSE_DTable dtable[1]; /* Dynamically sized */
+} FSE_DecompressWksp;
+
+
+FORCE_INLINE_TEMPLATE size_t FSE_decompress_wksp_body(
+ void* dst, size_t dstCapacity,
+ const void* cSrc, size_t cSrcSize,
+ unsigned maxLog, void* workSpace, size_t wkspSize,
+ int bmi2)
+{
+ const BYTE* const istart = (const BYTE*)cSrc;
+ const BYTE* ip = istart;
+ unsigned tableLog;
+ unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
+ FSE_DecompressWksp* const wksp = (FSE_DecompressWksp*)workSpace;
+
+ DEBUG_STATIC_ASSERT((FSE_MAX_SYMBOL_VALUE + 1) % 2 == 0);
+ if (wkspSize < sizeof(*wksp)) return ERROR(GENERIC);
+
+ /* normal FSE decoding mode */
+ {
+ size_t const NCountLength = FSE_readNCount_bmi2(wksp->ncount, &maxSymbolValue, &tableLog, istart, cSrcSize, bmi2);
+ if (FSE_isError(NCountLength)) return NCountLength;
+ if (tableLog > maxLog) return ERROR(tableLog_tooLarge);
+ assert(NCountLength <= cSrcSize);
+ ip += NCountLength;
+ cSrcSize -= NCountLength;
+ }
+
+ if (FSE_DECOMPRESS_WKSP_SIZE(tableLog, maxSymbolValue) > wkspSize) return ERROR(tableLog_tooLarge);
+ workSpace = wksp->dtable + FSE_DTABLE_SIZE_U32(tableLog);
+ wkspSize -= sizeof(*wksp) + FSE_DTABLE_SIZE(tableLog);
+
+ CHECK_F( FSE_buildDTable_internal(wksp->dtable, wksp->ncount, maxSymbolValue, tableLog, workSpace, wkspSize) );
+
+ {
+ const void* ptr = wksp->dtable;
+ const FSE_DTableHeader* DTableH = (const FSE_DTableHeader*)ptr;
+ const U32 fastMode = DTableH->fastMode;
+
+ /* select fast mode (static) */
+ if (fastMode) return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, wksp->dtable, 1);
+ return FSE_decompress_usingDTable_generic(dst, dstCapacity, ip, cSrcSize, wksp->dtable, 0);
+ }
+}
+
+/* Avoids the FORCE_INLINE of the _body() function. */
+static size_t FSE_decompress_wksp_body_default(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
+{
+ return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 0);
+}
+
+#if DYNAMIC_BMI2
+TARGET_ATTRIBUTE("bmi2") static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
+{
+ return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 1);
+}
+#endif
+
+size_t FSE_decompress_wksp_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize, int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+ return FSE_decompress_wksp_body_bmi2(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
+ }
+#endif
+ (void)bmi2;
+ return FSE_decompress_wksp_body_default(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize);
+}
+
+
+typedef FSE_DTable DTable_max_t[FSE_DTABLE_SIZE_U32(FSE_MAX_TABLELOG)];
+
+
+
+#endif /* FSE_COMMONDEFS_ONLY */
--- /dev/null
+/* ******************************************************************
+ * huff0 huffman codec,
+ * part of Finite State Entropy library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+
+#ifndef HUF_H_298734234
+#define HUF_H_298734234
+
+/* *** Dependencies *** */
+#include "zstd_deps.h" /* size_t */
+
+
+/* *** library symbols visibility *** */
+/* Note : when linking with -fvisibility=hidden on gcc, or by default on Visual,
+ * HUF symbols remain "private" (internal symbols for library only).
+ * Set macro FSE_DLL_EXPORT to 1 if you want HUF symbols visible on DLL interface */
+#if defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) && defined(__GNUC__) && (__GNUC__ >= 4)
+# define HUF_PUBLIC_API __attribute__ ((visibility ("default")))
+#elif defined(FSE_DLL_EXPORT) && (FSE_DLL_EXPORT==1) /* Visual expected */
+# define HUF_PUBLIC_API __declspec(dllexport)
+#elif defined(FSE_DLL_IMPORT) && (FSE_DLL_IMPORT==1)
+# define HUF_PUBLIC_API __declspec(dllimport) /* not required, just to generate faster code (saves a function pointer load from IAT and an indirect jump) */
+#else
+# define HUF_PUBLIC_API
+#endif
+
+
+/* ========================== */
+/* *** simple functions *** */
+/* ========================== */
+
+/* HUF_compress() :
+ * Compress content from buffer 'src', of size 'srcSize', into buffer 'dst'.
+ * 'dst' buffer must be already allocated.
+ * Compression runs faster if `dstCapacity` >= HUF_compressBound(srcSize).
+ * `srcSize` must be <= `HUF_BLOCKSIZE_MAX` == 128 KB.
+ * @return : size of compressed data (<= `dstCapacity`).
+ * Special values : if return == 0, srcData is not compressible => Nothing is stored within dst !!!
+ * if HUF_isError(return), compression failed (more details using HUF_getErrorName())
+ */
+HUF_PUBLIC_API size_t HUF_compress(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize);
+
+/* HUF_decompress() :
+ * Decompress HUF data from buffer 'cSrc', of size 'cSrcSize',
+ * into already allocated buffer 'dst', of minimum size 'dstSize'.
+ * `originalSize` : **must** be the ***exact*** size of original (uncompressed) data.
+ * Note : in contrast with FSE, HUF_decompress can regenerate
+ * RLE (cSrcSize==1) and uncompressed (cSrcSize==dstSize) data,
+ * because it knows size to regenerate (originalSize).
+ * @return : size of regenerated data (== originalSize),
+ * or an error code, which can be tested using HUF_isError()
+ */
+HUF_PUBLIC_API size_t HUF_decompress(void* dst, size_t originalSize,
+ const void* cSrc, size_t cSrcSize);
+
+
+/* *** Tool functions *** */
+#define HUF_BLOCKSIZE_MAX (128 * 1024) /*< maximum input size for a single block compressed with HUF_compress */
+HUF_PUBLIC_API size_t HUF_compressBound(size_t size); /*< maximum compressed size (worst case) */
+
+/* Error Management */
+HUF_PUBLIC_API unsigned HUF_isError(size_t code); /*< tells if a return value is an error code */
+HUF_PUBLIC_API const char* HUF_getErrorName(size_t code); /*< provides error code string (useful for debugging) */
+
+
+/* *** Advanced function *** */
+
+/* HUF_compress2() :
+ * Same as HUF_compress(), but offers control over `maxSymbolValue` and `tableLog`.
+ * `maxSymbolValue` must be <= HUF_SYMBOLVALUE_MAX .
+ * `tableLog` must be `<= HUF_TABLELOG_MAX` . */
+HUF_PUBLIC_API size_t HUF_compress2 (void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned tableLog);
+
+/* HUF_compress4X_wksp() :
+ * Same as HUF_compress2(), but uses externally allocated `workSpace`.
+ * `workspace` must have minimum alignment of 4, and be at least as large as HUF_WORKSPACE_SIZE */
+#define HUF_WORKSPACE_SIZE ((6 << 10) + 256)
+#define HUF_WORKSPACE_SIZE_U32 (HUF_WORKSPACE_SIZE / sizeof(U32))
+HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned tableLog,
+ void* workSpace, size_t wkspSize);
+
+#endif /* HUF_H_298734234 */
+
+/* ******************************************************************
+ * WARNING !!
+ * The following section contains advanced and experimental definitions
+ * which shall never be used in the context of a dynamic library,
+ * because they are not guaranteed to remain stable in the future.
+ * Only consider them in association with static linking.
+ * *****************************************************************/
+#if !defined(HUF_H_HUF_STATIC_LINKING_ONLY)
+#define HUF_H_HUF_STATIC_LINKING_ONLY
+
+/* *** Dependencies *** */
+#include "mem.h" /* U32 */
+#define FSE_STATIC_LINKING_ONLY
+#include "fse.h"
+
+
+/* *** Constants *** */
+#define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
+#define HUF_TABLELOG_DEFAULT 11 /* default tableLog value when none specified */
+#define HUF_SYMBOLVALUE_MAX 255
+
+#define HUF_TABLELOG_ABSOLUTEMAX 15 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
+#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
+# error "HUF_TABLELOG_MAX is too large !"
+#endif
+
+
+/* ****************************************
+* Static allocation
+******************************************/
+/* HUF buffer bounds */
+#define HUF_CTABLEBOUND 129
+#define HUF_BLOCKBOUND(size) (size + (size>>8) + 8) /* only true when incompressible is pre-filtered with fast heuristic */
+#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
+
+/* static allocation of HUF's Compression Table */
+/* this is a private definition, just exposed for allocation and strict aliasing purpose. never EVER access its members directly */
+struct HUF_CElt_s {
+ U16 val;
+ BYTE nbBits;
+}; /* typedef'd to HUF_CElt */
+typedef struct HUF_CElt_s HUF_CElt; /* consider it an incomplete type */
+#define HUF_CTABLE_SIZE_U32(maxSymbolValue) ((maxSymbolValue)+1) /* Use tables of U32, for proper alignment */
+#define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_U32(maxSymbolValue) * sizeof(U32))
+#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
+ HUF_CElt name[HUF_CTABLE_SIZE_U32(maxSymbolValue)] /* no final ; */
+
+/* static allocation of HUF's DTable */
+typedef U32 HUF_DTable;
+#define HUF_DTABLE_SIZE(maxTableLog) (1 + (1<<(maxTableLog)))
+#define HUF_CREATE_STATIC_DTABLEX1(DTable, maxTableLog) \
+ HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = { ((U32)((maxTableLog)-1) * 0x01000001) }
+#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) \
+ HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = { ((U32)(maxTableLog) * 0x01000001) }
+
+
+/* ****************************************
+* Advanced decompression functions
+******************************************/
+size_t HUF_decompress4X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder */
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress4X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder */
+#endif
+
+size_t HUF_decompress4X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< decodes RLE and uncompressed */
+size_t HUF_decompress4X_hufOnly(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< considers RLE and uncompressed as errors */
+size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< considers RLE and uncompressed as errors */
+size_t HUF_decompress4X1_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder */
+size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< single-symbol decoder */
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress4X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder */
+size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< double-symbols decoder */
+#endif
+
+
+/* ****************************************
+ * HUF detailed API
+ * ****************************************/
+
+/*! HUF_compress() does the following:
+ * 1. count symbol occurrence from source[] into table count[] using FSE_count() (exposed within "fse.h")
+ * 2. (optional) refine tableLog using HUF_optimalTableLog()
+ * 3. build Huffman table from count using HUF_buildCTable()
+ * 4. save Huffman table to memory buffer using HUF_writeCTable()
+ * 5. encode the data stream using HUF_compress4X_usingCTable()
+ *
+ * The following API allows targeting specific sub-functions for advanced tasks.
+ * For example, it's possible to compress several blocks using the same 'CTable',
+ * or to save and regenerate 'CTable' using external methods.
+ */
+unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
+size_t HUF_buildCTable (HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue, unsigned maxNbBits); /* @return : maxNbBits; CTable and count can overlap. In which case, CTable will overwrite count content */
+size_t HUF_writeCTable (void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog);
+size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize);
+size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
+size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
+int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
+
+typedef enum {
+ HUF_repeat_none, /*< Cannot use the previous table */
+ HUF_repeat_check, /*< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1, 4}X_repeat */
+ HUF_repeat_valid /*< Can use the previous table and it is assumed to be valid */
+ } HUF_repeat;
+/* HUF_compress4X_repeat() :
+ * Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
+ * If it uses hufTable it does not modify hufTable or repeat.
+ * If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
+ * If preferRepeat then the old table will always be used if valid. */
+size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned tableLog,
+ void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
+
+/* HUF_buildCTable_wksp() :
+ * Same as HUF_buildCTable(), but using externally allocated scratch buffer.
+ * `workSpace` must be aligned on 4-bytes boundaries, and its size must be >= HUF_CTABLE_WORKSPACE_SIZE.
+ */
+#define HUF_CTABLE_WORKSPACE_SIZE_U32 (2*HUF_SYMBOLVALUE_MAX +1 +1)
+#define HUF_CTABLE_WORKSPACE_SIZE (HUF_CTABLE_WORKSPACE_SIZE_U32 * sizeof(unsigned))
+size_t HUF_buildCTable_wksp (HUF_CElt* tree,
+ const unsigned* count, U32 maxSymbolValue, U32 maxNbBits,
+ void* workSpace, size_t wkspSize);
+
+/*! HUF_readStats() :
+ * Read compact Huffman tree, saved by HUF_writeCTable().
+ * `huffWeight` is destination buffer.
+ * @return : size read from `src` , or an error Code .
+ * Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
+size_t HUF_readStats(BYTE* huffWeight, size_t hwSize,
+ U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize);
+
+/*! HUF_readStats_wksp() :
+ * Same as HUF_readStats() but takes an external workspace which must be
+ * 4-byte aligned and its size must be >= HUF_READ_STATS_WORKSPACE_SIZE.
+ * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
+ */
+#define HUF_READ_STATS_WORKSPACE_SIZE_U32 FSE_DECOMPRESS_WKSP_SIZE_U32(6, HUF_TABLELOG_MAX-1)
+#define HUF_READ_STATS_WORKSPACE_SIZE (HUF_READ_STATS_WORKSPACE_SIZE_U32 * sizeof(unsigned))
+size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize,
+ U32* rankStats, U32* nbSymbolsPtr, U32* tableLogPtr,
+ const void* src, size_t srcSize,
+ void* workspace, size_t wkspSize,
+ int bmi2);
+
+/* HUF_readCTable() :
+ * Loading a CTable saved with HUF_writeCTable() */
+size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);
+
+/* HUF_getNbBits() :
+ * Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
+ * Note 1 : is not inlined, as HUF_CElt definition is private
+ * Note 2 : const void* used, so that it can provide a statically allocated table as argument (which uses type U32) */
+U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue);
+
+/*
+ * HUF_decompress() does the following:
+ * 1. select the decompression algorithm (X1, X2) based on pre-computed heuristics
+ * 2. build Huffman table from save, using HUF_readDTableX?()
+ * 3. decode 1 or 4 segments in parallel using HUF_decompress?X?_usingDTable()
+ */
+
+/* HUF_selectDecoder() :
+ * Tells which decoder is likely to decode faster,
+ * based on a set of pre-computed metrics.
+ * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
+ * Assumption : 0 < dstSize <= 128 KB */
+U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize);
+
+/*
+ * The minimum workspace size for the `workSpace` used in
+ * HUF_readDTableX1_wksp() and HUF_readDTableX2_wksp().
+ *
+ * The space used depends on HUF_TABLELOG_MAX, ranging from ~1500 bytes when
+ * HUF_TABLE_LOG_MAX=12 to ~1850 bytes when HUF_TABLE_LOG_MAX=15.
+ * Buffer overflow errors may potentially occur if code modifications result in
+ * a required workspace size greater than that specified in the following
+ * macro.
+ */
+#define HUF_DECOMPRESS_WORKSPACE_SIZE ((2 << 10) + (1 << 9))
+#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))
+
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_readDTableX1 (HUF_DTable* DTable, const void* src, size_t srcSize);
+size_t HUF_readDTableX1_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
+#endif
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_readDTableX2 (HUF_DTable* DTable, const void* src, size_t srcSize);
+size_t HUF_readDTableX2_wksp (HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize);
+#endif
+
+size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_decompress4X1_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
+#endif
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress4X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
+#endif
+
+
+/* ====================== */
+/* single stream variants */
+/* ====================== */
+
+size_t HUF_compress1X (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
+size_t HUF_compress1X_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); /*< `workSpace` must be a table of at least HUF_WORKSPACE_SIZE_U32 unsigned */
+size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
+/* HUF_compress1X_repeat() :
+ * Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
+ * If it uses hufTable it does not modify hufTable or repeat.
+ * If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
+ * If preferRepeat then the old table will always be used if valid. */
+size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned tableLog,
+ void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
+
+size_t HUF_decompress1X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /* single-symbol decoder */
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress1X2 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /* double-symbol decoder */
+#endif
+
+size_t HUF_decompress1X_DCtx (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize);
+size_t HUF_decompress1X_DCtx_wksp (HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize);
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_decompress1X1_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< single-symbol decoder */
+size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< single-symbol decoder */
+#endif
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress1X2_DCtx(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /*< double-symbols decoder */
+size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize); /*< double-symbols decoder */
+#endif
+
+size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable); /*< automatic selection of sing or double symbol decoder, based on DTable */
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_decompress1X1_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
+#endif
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_decompress1X2_usingDTable(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable);
+#endif
+
+/* BMI2 variants.
+ * If the CPU has BMI2 support, pass bmi2=1, otherwise pass bmi2=0.
+ */
+size_t HUF_decompress1X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_decompress1X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);
+#endif
+size_t HUF_decompress4X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2);
+size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2);
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int bmi2);
+#endif
+
+#endif /* HUF_STATIC_LINKING_ONLY */
+
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause */
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef MEM_H_MODULE
+#define MEM_H_MODULE
+
+/*-****************************************
+* Dependencies
+******************************************/
+#include <asm/unaligned.h> /* get_unaligned, put_unaligned* */
+#include <linux/compiler.h> /* inline */
+#include <linux/swab.h> /* swab32, swab64 */
+#include <linux/types.h> /* size_t, ptrdiff_t */
+#include "debug.h" /* DEBUG_STATIC_ASSERT */
+
+/*-****************************************
+* Compiler specifics
+******************************************/
+#define MEM_STATIC static inline
+
+/*-**************************************************************
+* Basic Types
+*****************************************************************/
+typedef uint8_t BYTE;
+typedef uint16_t U16;
+typedef int16_t S16;
+typedef uint32_t U32;
+typedef int32_t S32;
+typedef uint64_t U64;
+typedef int64_t S64;
+
+/*-**************************************************************
+* Memory I/O API
+*****************************************************************/
+/*=== Static platform detection ===*/
+MEM_STATIC unsigned MEM_32bits(void);
+MEM_STATIC unsigned MEM_64bits(void);
+MEM_STATIC unsigned MEM_isLittleEndian(void);
+
+/*=== Native unaligned read/write ===*/
+MEM_STATIC U16 MEM_read16(const void* memPtr);
+MEM_STATIC U32 MEM_read32(const void* memPtr);
+MEM_STATIC U64 MEM_read64(const void* memPtr);
+MEM_STATIC size_t MEM_readST(const void* memPtr);
+
+MEM_STATIC void MEM_write16(void* memPtr, U16 value);
+MEM_STATIC void MEM_write32(void* memPtr, U32 value);
+MEM_STATIC void MEM_write64(void* memPtr, U64 value);
+
+/*=== Little endian unaligned read/write ===*/
+MEM_STATIC U16 MEM_readLE16(const void* memPtr);
+MEM_STATIC U32 MEM_readLE24(const void* memPtr);
+MEM_STATIC U32 MEM_readLE32(const void* memPtr);
+MEM_STATIC U64 MEM_readLE64(const void* memPtr);
+MEM_STATIC size_t MEM_readLEST(const void* memPtr);
+
+MEM_STATIC void MEM_writeLE16(void* memPtr, U16 val);
+MEM_STATIC void MEM_writeLE24(void* memPtr, U32 val);
+MEM_STATIC void MEM_writeLE32(void* memPtr, U32 val32);
+MEM_STATIC void MEM_writeLE64(void* memPtr, U64 val64);
+MEM_STATIC void MEM_writeLEST(void* memPtr, size_t val);
+
+/*=== Big endian unaligned read/write ===*/
+MEM_STATIC U32 MEM_readBE32(const void* memPtr);
+MEM_STATIC U64 MEM_readBE64(const void* memPtr);
+MEM_STATIC size_t MEM_readBEST(const void* memPtr);
+
+MEM_STATIC void MEM_writeBE32(void* memPtr, U32 val32);
+MEM_STATIC void MEM_writeBE64(void* memPtr, U64 val64);
+MEM_STATIC void MEM_writeBEST(void* memPtr, size_t val);
+
+/*=== Byteswap ===*/
+MEM_STATIC U32 MEM_swap32(U32 in);
+MEM_STATIC U64 MEM_swap64(U64 in);
+MEM_STATIC size_t MEM_swapST(size_t in);
+
+/*-**************************************************************
+* Memory I/O Implementation
+*****************************************************************/
+MEM_STATIC unsigned MEM_32bits(void)
+{
+ return sizeof(size_t) == 4;
+}
+
+MEM_STATIC unsigned MEM_64bits(void)
+{
+ return sizeof(size_t) == 8;
+}
+
+#if defined(__LITTLE_ENDIAN)
+#define MEM_LITTLE_ENDIAN 1
+#else
+#define MEM_LITTLE_ENDIAN 0
+#endif
+
+MEM_STATIC unsigned MEM_isLittleEndian(void)
+{
+ return MEM_LITTLE_ENDIAN;
+}
+
+MEM_STATIC U16 MEM_read16(const void *memPtr)
+{
+ return get_unaligned((const U16 *)memPtr);
+}
+
+MEM_STATIC U32 MEM_read32(const void *memPtr)
+{
+ return get_unaligned((const U32 *)memPtr);
+}
+
+MEM_STATIC U64 MEM_read64(const void *memPtr)
+{
+ return get_unaligned((const U64 *)memPtr);
+}
+
+MEM_STATIC size_t MEM_readST(const void *memPtr)
+{
+ return get_unaligned((const size_t *)memPtr);
+}
+
+MEM_STATIC void MEM_write16(void *memPtr, U16 value)
+{
+ put_unaligned(value, (U16 *)memPtr);
+}
+
+MEM_STATIC void MEM_write32(void *memPtr, U32 value)
+{
+ put_unaligned(value, (U32 *)memPtr);
+}
+
+MEM_STATIC void MEM_write64(void *memPtr, U64 value)
+{
+ put_unaligned(value, (U64 *)memPtr);
+}
+
+/*=== Little endian r/w ===*/
+
+MEM_STATIC U16 MEM_readLE16(const void *memPtr)
+{
+ return get_unaligned_le16(memPtr);
+}
+
+MEM_STATIC void MEM_writeLE16(void *memPtr, U16 val)
+{
+ put_unaligned_le16(val, memPtr);
+}
+
+MEM_STATIC U32 MEM_readLE24(const void *memPtr)
+{
+ return MEM_readLE16(memPtr) + (((const BYTE *)memPtr)[2] << 16);
+}
+
+MEM_STATIC void MEM_writeLE24(void *memPtr, U32 val)
+{
+ MEM_writeLE16(memPtr, (U16)val);
+ ((BYTE *)memPtr)[2] = (BYTE)(val >> 16);
+}
+
+MEM_STATIC U32 MEM_readLE32(const void *memPtr)
+{
+ return get_unaligned_le32(memPtr);
+}
+
+MEM_STATIC void MEM_writeLE32(void *memPtr, U32 val32)
+{
+ put_unaligned_le32(val32, memPtr);
+}
+
+MEM_STATIC U64 MEM_readLE64(const void *memPtr)
+{
+ return get_unaligned_le64(memPtr);
+}
+
+MEM_STATIC void MEM_writeLE64(void *memPtr, U64 val64)
+{
+ put_unaligned_le64(val64, memPtr);
+}
+
+MEM_STATIC size_t MEM_readLEST(const void *memPtr)
+{
+ if (MEM_32bits())
+ return (size_t)MEM_readLE32(memPtr);
+ else
+ return (size_t)MEM_readLE64(memPtr);
+}
+
+MEM_STATIC void MEM_writeLEST(void *memPtr, size_t val)
+{
+ if (MEM_32bits())
+ MEM_writeLE32(memPtr, (U32)val);
+ else
+ MEM_writeLE64(memPtr, (U64)val);
+}
+
+/*=== Big endian r/w ===*/
+
+MEM_STATIC U32 MEM_readBE32(const void *memPtr)
+{
+ return get_unaligned_be32(memPtr);
+}
+
+MEM_STATIC void MEM_writeBE32(void *memPtr, U32 val32)
+{
+ put_unaligned_be32(val32, memPtr);
+}
+
+MEM_STATIC U64 MEM_readBE64(const void *memPtr)
+{
+ return get_unaligned_be64(memPtr);
+}
+
+MEM_STATIC void MEM_writeBE64(void *memPtr, U64 val64)
+{
+ put_unaligned_be64(val64, memPtr);
+}
+
+MEM_STATIC size_t MEM_readBEST(const void *memPtr)
+{
+ if (MEM_32bits())
+ return (size_t)MEM_readBE32(memPtr);
+ else
+ return (size_t)MEM_readBE64(memPtr);
+}
+
+MEM_STATIC void MEM_writeBEST(void *memPtr, size_t val)
+{
+ if (MEM_32bits())
+ MEM_writeBE32(memPtr, (U32)val);
+ else
+ MEM_writeBE64(memPtr, (U64)val);
+}
+
+MEM_STATIC U32 MEM_swap32(U32 in)
+{
+ return swab32(in);
+}
+
+MEM_STATIC U64 MEM_swap64(U64 in)
+{
+ return swab64(in);
+}
+
+MEM_STATIC size_t MEM_swapST(size_t in)
+{
+ if (MEM_32bits())
+ return (size_t)MEM_swap32((U32)in);
+ else
+ return (size_t)MEM_swap64((U64)in);
+}
+
+#endif /* MEM_H_MODULE */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+
+
+/*-*************************************
+* Dependencies
+***************************************/
+#define ZSTD_DEPS_NEED_MALLOC
+#include "zstd_deps.h" /* ZSTD_malloc, ZSTD_calloc, ZSTD_free, ZSTD_memset */
+#include "error_private.h"
+#include "zstd_internal.h"
+
+
+/*-****************************************
+* Version
+******************************************/
+unsigned ZSTD_versionNumber(void) { return ZSTD_VERSION_NUMBER; }
+
+const char* ZSTD_versionString(void) { return ZSTD_VERSION_STRING; }
+
+
+/*-****************************************
+* ZSTD Error Management
+******************************************/
+#undef ZSTD_isError /* defined within zstd_internal.h */
+/*! ZSTD_isError() :
+ * tells if a return value is an error code
+ * symbol is required for external callers */
+unsigned ZSTD_isError(size_t code) { return ERR_isError(code); }
+
+/*! ZSTD_getErrorName() :
+ * provides error code string from function result (useful for debugging) */
+const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); }
+
+/*! ZSTD_getError() :
+ * convert a `size_t` function result into a proper ZSTD_errorCode enum */
+ZSTD_ErrorCode ZSTD_getErrorCode(size_t code) { return ERR_getErrorCode(code); }
+
+/*! ZSTD_getErrorString() :
+ * provides error code string from enum */
+const char* ZSTD_getErrorString(ZSTD_ErrorCode code) { return ERR_getErrorString(code); }
+
+
+
+/*=**************************************************************
+* Custom allocator
+****************************************************************/
+void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem)
+{
+ if (customMem.customAlloc)
+ return customMem.customAlloc(customMem.opaque, size);
+ return ZSTD_malloc(size);
+}
+
+void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem)
+{
+ if (customMem.customAlloc) {
+ /* calloc implemented as malloc+memset;
+ * not as efficient as calloc, but next best guess for custom malloc */
+ void* const ptr = customMem.customAlloc(customMem.opaque, size);
+ ZSTD_memset(ptr, 0, size);
+ return ptr;
+ }
+ return ZSTD_calloc(1, size);
+}
+
+void ZSTD_customFree(void* ptr, ZSTD_customMem customMem)
+{
+ if (ptr!=NULL) {
+ if (customMem.customFree)
+ customMem.customFree(customMem.opaque, ptr);
+ else
+ ZSTD_free(ptr);
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause */
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/*
+ * This file provides common libc dependencies that zstd requires.
+ * The purpose is to allow replacing this file with a custom implementation
+ * to compile zstd without libc support.
+ */
+
+/* Need:
+ * NULL
+ * INT_MAX
+ * UINT_MAX
+ * ZSTD_memcpy()
+ * ZSTD_memset()
+ * ZSTD_memmove()
+ */
+#ifndef ZSTD_DEPS_COMMON
+#define ZSTD_DEPS_COMMON
+
+#include <linux/limits.h>
+#include <linux/stddef.h>
+
+#define ZSTD_memcpy(d,s,n) __builtin_memcpy((d),(s),(n))
+#define ZSTD_memmove(d,s,n) __builtin_memmove((d),(s),(n))
+#define ZSTD_memset(d,s,n) __builtin_memset((d),(s),(n))
+
+#endif /* ZSTD_DEPS_COMMON */
+
+/*
+ * Define malloc as always failing. That means the user must
+ * either use ZSTD_customMem or statically allocate memory.
+ * Need:
+ * ZSTD_malloc()
+ * ZSTD_free()
+ * ZSTD_calloc()
+ */
+#ifdef ZSTD_DEPS_NEED_MALLOC
+#ifndef ZSTD_DEPS_MALLOC
+#define ZSTD_DEPS_MALLOC
+
+#define ZSTD_malloc(s) ({ (void)(s); NULL; })
+#define ZSTD_free(p) ((void)(p))
+#define ZSTD_calloc(n,s) ({ (void)(n); (void)(s); NULL; })
+
+#endif /* ZSTD_DEPS_MALLOC */
+#endif /* ZSTD_DEPS_NEED_MALLOC */
+
+/*
+ * Provides 64-bit math support.
+ * Need:
+ * U64 ZSTD_div64(U64 dividend, U32 divisor)
+ */
+#ifdef ZSTD_DEPS_NEED_MATH64
+#ifndef ZSTD_DEPS_MATH64
+#define ZSTD_DEPS_MATH64
+
+#include <linux/math64.h>
+
+static uint64_t ZSTD_div64(uint64_t dividend, uint32_t divisor) {
+ return div_u64(dividend, divisor);
+}
+
+#endif /* ZSTD_DEPS_MATH64 */
+#endif /* ZSTD_DEPS_NEED_MATH64 */
+
+/*
+ * This is only requested when DEBUGLEVEL >= 1, meaning
+ * it is disabled in production.
+ * Need:
+ * assert()
+ */
+#ifdef ZSTD_DEPS_NEED_ASSERT
+#ifndef ZSTD_DEPS_ASSERT
+#define ZSTD_DEPS_ASSERT
+
+#include <linux/kernel.h>
+
+#define assert(x) WARN_ON((x))
+
+#endif /* ZSTD_DEPS_ASSERT */
+#endif /* ZSTD_DEPS_NEED_ASSERT */
+
+/*
+ * This is only requested when DEBUGLEVEL >= 2, meaning
+ * it is disabled in production.
+ * Need:
+ * ZSTD_DEBUG_PRINT()
+ */
+#ifdef ZSTD_DEPS_NEED_IO
+#ifndef ZSTD_DEPS_IO
+#define ZSTD_DEPS_IO
+
+#include <linux/printk.h>
+
+#define ZSTD_DEBUG_PRINT(...) pr_debug(__VA_ARGS__)
+
+#endif /* ZSTD_DEPS_IO */
+#endif /* ZSTD_DEPS_NEED_IO */
+
+/*
+ * Only requested when MSAN is enabled.
+ * Need:
+ * intptr_t
+ */
+#ifdef ZSTD_DEPS_NEED_STDINT
+#ifndef ZSTD_DEPS_STDINT
+#define ZSTD_DEPS_STDINT
+
+/*
+ * The Linux Kernel doesn't provide intptr_t, only uintptr_t, which
+ * is an unsigned long.
+ */
+typedef long intptr_t;
+
+#endif /* ZSTD_DEPS_STDINT */
+#endif /* ZSTD_DEPS_NEED_STDINT */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_CCOMMON_H_MODULE
+#define ZSTD_CCOMMON_H_MODULE
+
+/* this module contains definitions which must be identical
+ * across compression, decompression and dictBuilder.
+ * It also contains a few functions useful to at least 2 of them
+ * and which benefit from being inlined */
+
+/*-*************************************
+* Dependencies
+***************************************/
+#include "compiler.h"
+#include "mem.h"
+#include "debug.h" /* assert, DEBUGLOG, RAWLOG, g_debuglevel */
+#include "error_private.h"
+#define ZSTD_STATIC_LINKING_ONLY
+#include <linux/zstd.h>
+#define FSE_STATIC_LINKING_ONLY
+#include "fse.h"
+#define HUF_STATIC_LINKING_ONLY
+#include "huf.h"
+#include <linux/xxhash.h> /* XXH_reset, update, digest */
+#define ZSTD_TRACE 0
+
+
+/* ---- static assert (debug) --- */
+#define ZSTD_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c)
+#define ZSTD_isError ERR_isError /* for inlining */
+#define FSE_isError ERR_isError
+#define HUF_isError ERR_isError
+
+
+/*-*************************************
+* shared macros
+***************************************/
+#undef MIN
+#undef MAX
+#define MIN(a,b) ((a)<(b) ? (a) : (b))
+#define MAX(a,b) ((a)>(b) ? (a) : (b))
+
+/*
+ * Ignore: this is an internal helper.
+ *
+ * This is a helper function to help force C99-correctness during compilation.
+ * Under strict compilation modes, variadic macro arguments can't be empty.
+ * However, variadic function arguments can be. Using a function therefore lets
+ * us statically check that at least one (string) argument was passed,
+ * independent of the compilation flags.
+ */
+static INLINE_KEYWORD UNUSED_ATTR
+void _force_has_format_string(const char *format, ...) {
+ (void)format;
+}
+
+/*
+ * Ignore: this is an internal helper.
+ *
+ * We want to force this function invocation to be syntactically correct, but
+ * we don't want to force runtime evaluation of its arguments.
+ */
+#define _FORCE_HAS_FORMAT_STRING(...) \
+ if (0) { \
+ _force_has_format_string(__VA_ARGS__); \
+ }
+
+/*
+ * Return the specified error if the condition evaluates to true.
+ *
+ * In debug modes, prints additional information.
+ * In order to do that (particularly, printing the conditional that failed),
+ * this can't just wrap RETURN_ERROR().
+ */
+#define RETURN_ERROR_IF(cond, err, ...) \
+ if (cond) { \
+ RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
+ __FILE__, __LINE__, ZSTD_QUOTE(cond), ZSTD_QUOTE(ERROR(err))); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return ERROR(err); \
+ }
+
+/*
+ * Unconditionally return the specified error.
+ *
+ * In debug modes, prints additional information.
+ */
+#define RETURN_ERROR(err, ...) \
+ do { \
+ RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
+ __FILE__, __LINE__, ZSTD_QUOTE(ERROR(err))); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return ERROR(err); \
+ } while(0);
+
+/*
+ * If the provided expression evaluates to an error code, returns that error code.
+ *
+ * In debug modes, prints additional information.
+ */
+#define FORWARD_IF_ERROR(err, ...) \
+ do { \
+ size_t const err_code = (err); \
+ if (ERR_isError(err_code)) { \
+ RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
+ __FILE__, __LINE__, ZSTD_QUOTE(err), ERR_getErrorName(err_code)); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return err_code; \
+ } \
+ } while(0);
+
+
+/*-*************************************
+* Common constants
+***************************************/
+#define ZSTD_OPT_NUM (1<<12)
+
+#define ZSTD_REP_NUM 3 /* number of repcodes */
+#define ZSTD_REP_MOVE (ZSTD_REP_NUM-1)
+static UNUSED_ATTR const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };
+
+#define KB *(1 <<10)
+#define MB *(1 <<20)
+#define GB *(1U<<30)
+
+#define BIT7 128
+#define BIT6 64
+#define BIT5 32
+#define BIT4 16
+#define BIT1 2
+#define BIT0 1
+
+#define ZSTD_WINDOWLOG_ABSOLUTEMIN 10
+static UNUSED_ATTR const size_t ZSTD_fcs_fieldSize[4] = { 0, 2, 4, 8 };
+static UNUSED_ATTR const size_t ZSTD_did_fieldSize[4] = { 0, 1, 2, 4 };
+
+#define ZSTD_FRAMEIDSIZE 4 /* magic number size */
+
+#define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */
+static UNUSED_ATTR const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
+typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
+
+#define ZSTD_FRAMECHECKSUMSIZE 4
+
+#define MIN_SEQUENCES_SIZE 1 /* nbSeq==0 */
+#define MIN_CBLOCK_SIZE (1 /*litCSize*/ + 1 /* RLE or RAW */ + MIN_SEQUENCES_SIZE /* nbSeq==0 */) /* for a non-null block */
+
+#define HufLog 12
+typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingType_e;
+
+#define LONGNBSEQ 0x7F00
+
+#define MINMATCH 3
+
+#define Litbits 8
+#define MaxLit ((1<<Litbits) - 1)
+#define MaxML 52
+#define MaxLL 35
+#define DefaultMaxOff 28
+#define MaxOff 31
+#define MaxSeq MAX(MaxLL, MaxML) /* Assumption : MaxOff < MaxLL,MaxML */
+#define MLFSELog 9
+#define LLFSELog 9
+#define OffFSELog 8
+#define MaxFSELog MAX(MAX(MLFSELog, LLFSELog), OffFSELog)
+
+#define ZSTD_MAX_HUF_HEADER_SIZE 128 /* header + <= 127 byte tree description */
+/* Each table cannot take more than #symbols * FSELog bits */
+#define ZSTD_MAX_FSE_HEADERS_SIZE (((MaxML + 1) * MLFSELog + (MaxLL + 1) * LLFSELog + (MaxOff + 1) * OffFSELog + 7) / 8)
+
+static UNUSED_ATTR const U32 LL_bits[MaxLL+1] = {
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 1, 1, 1, 1, 2, 2, 3, 3,
+ 4, 6, 7, 8, 9,10,11,12,
+ 13,14,15,16
+};
+static UNUSED_ATTR const S16 LL_defaultNorm[MaxLL+1] = {
+ 4, 3, 2, 2, 2, 2, 2, 2,
+ 2, 2, 2, 2, 2, 1, 1, 1,
+ 2, 2, 2, 2, 2, 2, 2, 2,
+ 2, 3, 2, 1, 1, 1, 1, 1,
+ -1,-1,-1,-1
+};
+#define LL_DEFAULTNORMLOG 6 /* for static allocation */
+static UNUSED_ATTR const U32 LL_defaultNormLog = LL_DEFAULTNORMLOG;
+
+static UNUSED_ATTR const U32 ML_bits[MaxML+1] = {
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 0, 0, 0, 0, 0, 0, 0, 0,
+ 1, 1, 1, 1, 2, 2, 3, 3,
+ 4, 4, 5, 7, 8, 9,10,11,
+ 12,13,14,15,16
+};
+static UNUSED_ATTR const S16 ML_defaultNorm[MaxML+1] = {
+ 1, 4, 3, 2, 2, 2, 2, 2,
+ 2, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1,-1,-1,
+ -1,-1,-1,-1,-1
+};
+#define ML_DEFAULTNORMLOG 6 /* for static allocation */
+static UNUSED_ATTR const U32 ML_defaultNormLog = ML_DEFAULTNORMLOG;
+
+static UNUSED_ATTR const S16 OF_defaultNorm[DefaultMaxOff+1] = {
+ 1, 1, 1, 1, 1, 1, 2, 2,
+ 2, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ -1,-1,-1,-1,-1
+};
+#define OF_DEFAULTNORMLOG 5 /* for static allocation */
+static UNUSED_ATTR const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
+
+
+/*-*******************************************
+* Shared functions to include for inlining
+*********************************************/
+static void ZSTD_copy8(void* dst, const void* src) {
+ ZSTD_memcpy(dst, src, 8);
+}
+
+#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
+static void ZSTD_copy16(void* dst, const void* src) {
+ ZSTD_memcpy(dst, src, 16);
+}
+#define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; }
+
+#define WILDCOPY_OVERLENGTH 32
+#define WILDCOPY_VECLEN 16
+
+typedef enum {
+ ZSTD_no_overlap,
+ ZSTD_overlap_src_before_dst
+ /* ZSTD_overlap_dst_before_src, */
+} ZSTD_overlap_e;
+
+/*! ZSTD_wildcopy() :
+ * Custom version of ZSTD_memcpy(), can over read/write up to WILDCOPY_OVERLENGTH bytes (if length==0)
+ * @param ovtype controls the overlap detection
+ * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
+ * - ZSTD_overlap_src_before_dst: The src and dst may overlap, but they MUST be at least 8 bytes apart.
+ * The src buffer must be before the dst buffer.
+ */
+MEM_STATIC FORCE_INLINE_ATTR
+void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e const ovtype)
+{
+ ptrdiff_t diff = (BYTE*)dst - (const BYTE*)src;
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* op = (BYTE*)dst;
+ BYTE* const oend = op + length;
+
+ assert(diff >= 8 || (ovtype == ZSTD_no_overlap && diff <= -WILDCOPY_VECLEN));
+
+ if (ovtype == ZSTD_overlap_src_before_dst && diff < WILDCOPY_VECLEN) {
+ /* Handle short offset copies. */
+ do {
+ COPY8(op, ip)
+ } while (op < oend);
+ } else {
+ assert(diff >= WILDCOPY_VECLEN || diff <= -WILDCOPY_VECLEN);
+ /* Separate out the first COPY16() call because the copy length is
+ * almost certain to be short, so the branches have different
+ * probabilities. Since it is almost certain to be short, only do
+ * one COPY16() in the first call. Then, do two calls per loop since
+ * at that point it is more likely to have a high trip count.
+ */
+#ifdef __aarch64__
+ do {
+ COPY16(op, ip);
+ }
+ while (op < oend);
+#else
+ ZSTD_copy16(op, ip);
+ if (16 >= length) return;
+ op += 16;
+ ip += 16;
+ do {
+ COPY16(op, ip);
+ COPY16(op, ip);
+ }
+ while (op < oend);
+#endif
+ }
+}
+
+MEM_STATIC size_t ZSTD_limitCopy(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ size_t const length = MIN(dstCapacity, srcSize);
+ if (length > 0) {
+ ZSTD_memcpy(dst, src, length);
+ }
+ return length;
+}
+
+/* define "workspace is too large" as this number of times larger than needed */
+#define ZSTD_WORKSPACETOOLARGE_FACTOR 3
+
+/* when workspace is continuously too large
+ * during at least this number of times,
+ * context's memory usage is considered wasteful,
+ * because it's sized to handle a worst case scenario which rarely happens.
+ * In which case, resize it down to free some memory */
+#define ZSTD_WORKSPACETOOLARGE_MAXDURATION 128
+
+/* Controls whether the input/output buffer is buffered or stable. */
+typedef enum {
+ ZSTD_bm_buffered = 0, /* Buffer the input/output */
+ ZSTD_bm_stable = 1 /* ZSTD_inBuffer/ZSTD_outBuffer is stable */
+} ZSTD_bufferMode_e;
+
+
+/*-*******************************************
+* Private declarations
+*********************************************/
+typedef struct seqDef_s {
+ U32 offset; /* Offset code of the sequence */
+ U16 litLength;
+ U16 matchLength;
+} seqDef;
+
+typedef struct {
+ seqDef* sequencesStart;
+ seqDef* sequences; /* ptr to end of sequences */
+ BYTE* litStart;
+ BYTE* lit; /* ptr to end of literals */
+ BYTE* llCode;
+ BYTE* mlCode;
+ BYTE* ofCode;
+ size_t maxNbSeq;
+ size_t maxNbLit;
+
+ /* longLengthPos and longLengthID to allow us to represent either a single litLength or matchLength
+ * in the seqStore that has a value larger than U16 (if it exists). To do so, we increment
+ * the existing value of the litLength or matchLength by 0x10000.
+ */
+ U32 longLengthID; /* 0 == no longLength; 1 == Represent the long literal; 2 == Represent the long match; */
+ U32 longLengthPos; /* Index of the sequence to apply long length modification to */
+} seqStore_t;
+
+typedef struct {
+ U32 litLength;
+ U32 matchLength;
+} ZSTD_sequenceLength;
+
+/*
+ * Returns the ZSTD_sequenceLength for the given sequences. It handles the decoding of long sequences
+ * indicated by longLengthPos and longLengthID, and adds MINMATCH back to matchLength.
+ */
+MEM_STATIC ZSTD_sequenceLength ZSTD_getSequenceLength(seqStore_t const* seqStore, seqDef const* seq)
+{
+ ZSTD_sequenceLength seqLen;
+ seqLen.litLength = seq->litLength;
+ seqLen.matchLength = seq->matchLength + MINMATCH;
+ if (seqStore->longLengthPos == (U32)(seq - seqStore->sequencesStart)) {
+ if (seqStore->longLengthID == 1) {
+ seqLen.litLength += 0xFFFF;
+ }
+ if (seqStore->longLengthID == 2) {
+ seqLen.matchLength += 0xFFFF;
+ }
+ }
+ return seqLen;
+}
+
+/*
+ * Contains the compressed frame size and an upper-bound for the decompressed frame size.
+ * Note: before using `compressedSize`, check for errors using ZSTD_isError().
+ * similarly, before using `decompressedBound`, check for errors using:
+ * `decompressedBound != ZSTD_CONTENTSIZE_ERROR`
+ */
+typedef struct {
+ size_t compressedSize;
+ unsigned long long decompressedBound;
+} ZSTD_frameSizeInfo; /* decompress & legacy */
+
+const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx); /* compress & dictBuilder */
+void ZSTD_seqToCodes(const seqStore_t* seqStorePtr); /* compress, dictBuilder, decodeCorpus (shouldn't get its definition from here) */
+
+/* custom memory allocation functions */
+void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem);
+void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem);
+void ZSTD_customFree(void* ptr, ZSTD_customMem customMem);
+
+
+MEM_STATIC U32 ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus */
+{
+ assert(val != 0);
+ {
+# if (__GNUC__ >= 3) /* GCC Intrinsic */
+ return __builtin_clz (val) ^ 31;
+# else /* Software version */
+ static const U32 DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
+ U32 v = val;
+ v |= v >> 1;
+ v |= v >> 2;
+ v |= v >> 4;
+ v |= v >> 8;
+ v |= v >> 16;
+ return DeBruijnClz[(v * 0x07C4ACDDU) >> 27];
+# endif
+ }
+}
+
+
+/* ZSTD_invalidateRepCodes() :
+ * ensures next compression will not use repcodes from previous block.
+ * Note : only works with regular variant;
+ * do not use with extDict variant ! */
+void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx); /* zstdmt, adaptive_compression (shouldn't get this definition from here) */
+
+
+typedef struct {
+ blockType_e blockType;
+ U32 lastBlock;
+ U32 origSize;
+} blockProperties_t; /* declared here for decompress and fullbench */
+
+/*! ZSTD_getcBlockSize() :
+ * Provides the size of compressed block from block header `src` */
+/* Used by: decompress, fullbench (does not get its definition from here) */
+size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
+ blockProperties_t* bpPtr);
+
+/*! ZSTD_decodeSeqHeaders() :
+ * decode sequence header from src */
+/* Used by: decompress, fullbench (does not get its definition from here) */
+size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
+ const void* src, size_t srcSize);
+
+
+
+#endif /* ZSTD_CCOMMON_H_MODULE */
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-/*-*************************************
-* Dependencies
-***************************************/
-#include "fse.h"
-#include "huf.h"
-#include "mem.h"
-#include "zstd_internal.h" /* includes zstd.h */
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h> /* memset */
-
-/*-*************************************
-* Constants
-***************************************/
-static const U32 g_searchStrength = 8; /* control skip over incompressible data */
-#define HASH_READ_SIZE 8
-typedef enum { ZSTDcs_created = 0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e;
-
-/*-*************************************
-* Helper functions
-***************************************/
-size_t ZSTD_compressBound(size_t srcSize) { return FSE_compressBound(srcSize) + 12; }
-
-/*-*************************************
-* Sequence storage
-***************************************/
-static void ZSTD_resetSeqStore(seqStore_t *ssPtr)
-{
- ssPtr->lit = ssPtr->litStart;
- ssPtr->sequences = ssPtr->sequencesStart;
- ssPtr->longLengthID = 0;
-}
-
-/*-*************************************
-* Context memory management
-***************************************/
-struct ZSTD_CCtx_s {
- const BYTE *nextSrc; /* next block here to continue on curr prefix */
- const BYTE *base; /* All regular indexes relative to this position */
- const BYTE *dictBase; /* extDict indexes relative to this position */
- U32 dictLimit; /* below that point, need extDict */
- U32 lowLimit; /* below that point, no more data */
- U32 nextToUpdate; /* index from which to continue dictionary update */
- U32 nextToUpdate3; /* index from which to continue dictionary update */
- U32 hashLog3; /* dispatch table : larger == faster, more memory */
- U32 loadedDictEnd; /* index of end of dictionary */
- U32 forceWindow; /* force back-references to respect limit of 1<<wLog, even for dictionary */
- U32 forceRawDict; /* Force loading dictionary in "content-only" mode (no header analysis) */
- ZSTD_compressionStage_e stage;
- U32 rep[ZSTD_REP_NUM];
- U32 repToConfirm[ZSTD_REP_NUM];
- U32 dictID;
- ZSTD_parameters params;
- void *workSpace;
- size_t workSpaceSize;
- size_t blockSize;
- U64 frameContentSize;
- struct xxh64_state xxhState;
- ZSTD_customMem customMem;
-
- seqStore_t seqStore; /* sequences storage ptrs */
- U32 *hashTable;
- U32 *hashTable3;
- U32 *chainTable;
- HUF_CElt *hufTable;
- U32 flagStaticTables;
- HUF_repeat flagStaticHufTable;
- FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)];
- FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)];
- FSE_CTable litlengthCTable[FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)];
- unsigned tmpCounters[HUF_COMPRESS_WORKSPACE_SIZE_U32];
-};
-
-size_t ZSTD_CCtxWorkspaceBound(ZSTD_compressionParameters cParams)
-{
- size_t const blockSize = MIN(ZSTD_BLOCKSIZE_ABSOLUTEMAX, (size_t)1 << cParams.windowLog);
- U32 const divider = (cParams.searchLength == 3) ? 3 : 4;
- size_t const maxNbSeq = blockSize / divider;
- size_t const tokenSpace = blockSize + 11 * maxNbSeq;
- size_t const chainSize = (cParams.strategy == ZSTD_fast) ? 0 : (1 << cParams.chainLog);
- size_t const hSize = ((size_t)1) << cParams.hashLog;
- U32 const hashLog3 = (cParams.searchLength > 3) ? 0 : MIN(ZSTD_HASHLOG3_MAX, cParams.windowLog);
- size_t const h3Size = ((size_t)1) << hashLog3;
- size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32);
- size_t const optSpace =
- ((MaxML + 1) + (MaxLL + 1) + (MaxOff + 1) + (1 << Litbits)) * sizeof(U32) + (ZSTD_OPT_NUM + 1) * (sizeof(ZSTD_match_t) + sizeof(ZSTD_optimal_t));
- size_t const workspaceSize = tableSpace + (256 * sizeof(U32)) /* huffTable */ + tokenSpace +
- (((cParams.strategy == ZSTD_btopt) || (cParams.strategy == ZSTD_btopt2)) ? optSpace : 0);
-
- return ZSTD_ALIGN(sizeof(ZSTD_stack)) + ZSTD_ALIGN(sizeof(ZSTD_CCtx)) + ZSTD_ALIGN(workspaceSize);
-}
-
-static ZSTD_CCtx *ZSTD_createCCtx_advanced(ZSTD_customMem customMem)
-{
- ZSTD_CCtx *cctx;
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
- cctx = (ZSTD_CCtx *)ZSTD_malloc(sizeof(ZSTD_CCtx), customMem);
- if (!cctx)
- return NULL;
- memset(cctx, 0, sizeof(ZSTD_CCtx));
- cctx->customMem = customMem;
- return cctx;
-}
-
-ZSTD_CCtx *ZSTD_initCCtx(void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- ZSTD_CCtx *cctx = ZSTD_createCCtx_advanced(stackMem);
- if (cctx) {
- cctx->workSpace = ZSTD_stackAllocAll(cctx->customMem.opaque, &cctx->workSpaceSize);
- }
- return cctx;
-}
-
-size_t ZSTD_freeCCtx(ZSTD_CCtx *cctx)
-{
- if (cctx == NULL)
- return 0; /* support free on NULL */
- ZSTD_free(cctx->workSpace, cctx->customMem);
- ZSTD_free(cctx, cctx->customMem);
- return 0; /* reserved as a potential error code in the future */
-}
-
-const seqStore_t *ZSTD_getSeqStore(const ZSTD_CCtx *ctx) /* hidden interface */ { return &(ctx->seqStore); }
-
-static ZSTD_parameters ZSTD_getParamsFromCCtx(const ZSTD_CCtx *cctx) { return cctx->params; }
-
-/** ZSTD_checkParams() :
- ensure param values remain within authorized range.
- @return : 0, or an error code if one value is beyond authorized range */
-size_t ZSTD_checkCParams(ZSTD_compressionParameters cParams)
-{
-#define CLAMPCHECK(val, min, max) \
- { \
- if ((val < min) | (val > max)) \
- return ERROR(compressionParameter_unsupported); \
- }
- CLAMPCHECK(cParams.windowLog, ZSTD_WINDOWLOG_MIN, ZSTD_WINDOWLOG_MAX);
- CLAMPCHECK(cParams.chainLog, ZSTD_CHAINLOG_MIN, ZSTD_CHAINLOG_MAX);
- CLAMPCHECK(cParams.hashLog, ZSTD_HASHLOG_MIN, ZSTD_HASHLOG_MAX);
- CLAMPCHECK(cParams.searchLog, ZSTD_SEARCHLOG_MIN, ZSTD_SEARCHLOG_MAX);
- CLAMPCHECK(cParams.searchLength, ZSTD_SEARCHLENGTH_MIN, ZSTD_SEARCHLENGTH_MAX);
- CLAMPCHECK(cParams.targetLength, ZSTD_TARGETLENGTH_MIN, ZSTD_TARGETLENGTH_MAX);
- if ((U32)(cParams.strategy) > (U32)ZSTD_btopt2)
- return ERROR(compressionParameter_unsupported);
- return 0;
-}
-
-/** ZSTD_cycleLog() :
- * condition for correct operation : hashLog > 1 */
-static U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
-{
- U32 const btScale = ((U32)strat >= (U32)ZSTD_btlazy2);
- return hashLog - btScale;
-}
-
-/** ZSTD_adjustCParams() :
- optimize `cPar` for a given input (`srcSize` and `dictSize`).
- mostly downsizing to reduce memory consumption and initialization.
- Both `srcSize` and `dictSize` are optional (use 0 if unknown),
- but if both are 0, no optimization can be done.
- Note : cPar is considered validated at this stage. Use ZSTD_checkParams() to ensure that. */
-ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize)
-{
- if (srcSize + dictSize == 0)
- return cPar; /* no size information available : no adjustment */
-
- /* resize params, to use less memory when necessary */
- {
- U32 const minSrcSize = (srcSize == 0) ? 500 : 0;
- U64 const rSize = srcSize + dictSize + minSrcSize;
- if (rSize < ((U64)1 << ZSTD_WINDOWLOG_MAX)) {
- U32 const srcLog = MAX(ZSTD_HASHLOG_MIN, ZSTD_highbit32((U32)(rSize)-1) + 1);
- if (cPar.windowLog > srcLog)
- cPar.windowLog = srcLog;
- }
- }
- if (cPar.hashLog > cPar.windowLog)
- cPar.hashLog = cPar.windowLog;
- {
- U32 const cycleLog = ZSTD_cycleLog(cPar.chainLog, cPar.strategy);
- if (cycleLog > cPar.windowLog)
- cPar.chainLog -= (cycleLog - cPar.windowLog);
- }
-
- if (cPar.windowLog < ZSTD_WINDOWLOG_ABSOLUTEMIN)
- cPar.windowLog = ZSTD_WINDOWLOG_ABSOLUTEMIN; /* required for frame header */
-
- return cPar;
-}
-
-static U32 ZSTD_equivalentParams(ZSTD_parameters param1, ZSTD_parameters param2)
-{
- return (param1.cParams.hashLog == param2.cParams.hashLog) & (param1.cParams.chainLog == param2.cParams.chainLog) &
- (param1.cParams.strategy == param2.cParams.strategy) & ((param1.cParams.searchLength == 3) == (param2.cParams.searchLength == 3));
-}
-
-/*! ZSTD_continueCCtx() :
- reuse CCtx without reset (note : requires no dictionary) */
-static size_t ZSTD_continueCCtx(ZSTD_CCtx *cctx, ZSTD_parameters params, U64 frameContentSize)
-{
- U32 const end = (U32)(cctx->nextSrc - cctx->base);
- cctx->params = params;
- cctx->frameContentSize = frameContentSize;
- cctx->lowLimit = end;
- cctx->dictLimit = end;
- cctx->nextToUpdate = end + 1;
- cctx->stage = ZSTDcs_init;
- cctx->dictID = 0;
- cctx->loadedDictEnd = 0;
- {
- int i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- cctx->rep[i] = repStartValue[i];
- }
- cctx->seqStore.litLengthSum = 0; /* force reset of btopt stats */
- xxh64_reset(&cctx->xxhState, 0);
- return 0;
-}
-
-typedef enum { ZSTDcrp_continue, ZSTDcrp_noMemset, ZSTDcrp_fullReset } ZSTD_compResetPolicy_e;
-
-/*! ZSTD_resetCCtx_advanced() :
- note : `params` must be validated */
-static size_t ZSTD_resetCCtx_advanced(ZSTD_CCtx *zc, ZSTD_parameters params, U64 frameContentSize, ZSTD_compResetPolicy_e const crp)
-{
- if (crp == ZSTDcrp_continue)
- if (ZSTD_equivalentParams(params, zc->params)) {
- zc->flagStaticTables = 0;
- zc->flagStaticHufTable = HUF_repeat_none;
- return ZSTD_continueCCtx(zc, params, frameContentSize);
- }
-
- {
- size_t const blockSize = MIN(ZSTD_BLOCKSIZE_ABSOLUTEMAX, (size_t)1 << params.cParams.windowLog);
- U32 const divider = (params.cParams.searchLength == 3) ? 3 : 4;
- size_t const maxNbSeq = blockSize / divider;
- size_t const tokenSpace = blockSize + 11 * maxNbSeq;
- size_t const chainSize = (params.cParams.strategy == ZSTD_fast) ? 0 : (1 << params.cParams.chainLog);
- size_t const hSize = ((size_t)1) << params.cParams.hashLog;
- U32 const hashLog3 = (params.cParams.searchLength > 3) ? 0 : MIN(ZSTD_HASHLOG3_MAX, params.cParams.windowLog);
- size_t const h3Size = ((size_t)1) << hashLog3;
- size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32);
- void *ptr;
-
- /* Check if workSpace is large enough, alloc a new one if needed */
- {
- size_t const optSpace = ((MaxML + 1) + (MaxLL + 1) + (MaxOff + 1) + (1 << Litbits)) * sizeof(U32) +
- (ZSTD_OPT_NUM + 1) * (sizeof(ZSTD_match_t) + sizeof(ZSTD_optimal_t));
- size_t const neededSpace = tableSpace + (256 * sizeof(U32)) /* huffTable */ + tokenSpace +
- (((params.cParams.strategy == ZSTD_btopt) || (params.cParams.strategy == ZSTD_btopt2)) ? optSpace : 0);
- if (zc->workSpaceSize < neededSpace) {
- ZSTD_free(zc->workSpace, zc->customMem);
- zc->workSpace = ZSTD_malloc(neededSpace, zc->customMem);
- if (zc->workSpace == NULL)
- return ERROR(memory_allocation);
- zc->workSpaceSize = neededSpace;
- }
- }
-
- if (crp != ZSTDcrp_noMemset)
- memset(zc->workSpace, 0, tableSpace); /* reset tables only */
- xxh64_reset(&zc->xxhState, 0);
- zc->hashLog3 = hashLog3;
- zc->hashTable = (U32 *)(zc->workSpace);
- zc->chainTable = zc->hashTable + hSize;
- zc->hashTable3 = zc->chainTable + chainSize;
- ptr = zc->hashTable3 + h3Size;
- zc->hufTable = (HUF_CElt *)ptr;
- zc->flagStaticTables = 0;
- zc->flagStaticHufTable = HUF_repeat_none;
- ptr = ((U32 *)ptr) + 256; /* note : HUF_CElt* is incomplete type, size is simulated using U32 */
-
- zc->nextToUpdate = 1;
- zc->nextSrc = NULL;
- zc->base = NULL;
- zc->dictBase = NULL;
- zc->dictLimit = 0;
- zc->lowLimit = 0;
- zc->params = params;
- zc->blockSize = blockSize;
- zc->frameContentSize = frameContentSize;
- {
- int i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- zc->rep[i] = repStartValue[i];
- }
-
- if ((params.cParams.strategy == ZSTD_btopt) || (params.cParams.strategy == ZSTD_btopt2)) {
- zc->seqStore.litFreq = (U32 *)ptr;
- zc->seqStore.litLengthFreq = zc->seqStore.litFreq + (1 << Litbits);
- zc->seqStore.matchLengthFreq = zc->seqStore.litLengthFreq + (MaxLL + 1);
- zc->seqStore.offCodeFreq = zc->seqStore.matchLengthFreq + (MaxML + 1);
- ptr = zc->seqStore.offCodeFreq + (MaxOff + 1);
- zc->seqStore.matchTable = (ZSTD_match_t *)ptr;
- ptr = zc->seqStore.matchTable + ZSTD_OPT_NUM + 1;
- zc->seqStore.priceTable = (ZSTD_optimal_t *)ptr;
- ptr = zc->seqStore.priceTable + ZSTD_OPT_NUM + 1;
- zc->seqStore.litLengthSum = 0;
- }
- zc->seqStore.sequencesStart = (seqDef *)ptr;
- ptr = zc->seqStore.sequencesStart + maxNbSeq;
- zc->seqStore.llCode = (BYTE *)ptr;
- zc->seqStore.mlCode = zc->seqStore.llCode + maxNbSeq;
- zc->seqStore.ofCode = zc->seqStore.mlCode + maxNbSeq;
- zc->seqStore.litStart = zc->seqStore.ofCode + maxNbSeq;
-
- zc->stage = ZSTDcs_init;
- zc->dictID = 0;
- zc->loadedDictEnd = 0;
-
- return 0;
- }
-}
-
-/* ZSTD_invalidateRepCodes() :
- * ensures next compression will not use repcodes from previous block.
- * Note : only works with regular variant;
- * do not use with extDict variant ! */
-void ZSTD_invalidateRepCodes(ZSTD_CCtx *cctx)
-{
- int i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- cctx->rep[i] = 0;
-}
-
-/*! ZSTD_copyCCtx() :
-* Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
-* Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
-* @return : 0, or an error code */
-size_t ZSTD_copyCCtx(ZSTD_CCtx *dstCCtx, const ZSTD_CCtx *srcCCtx, unsigned long long pledgedSrcSize)
-{
- if (srcCCtx->stage != ZSTDcs_init)
- return ERROR(stage_wrong);
-
- memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
- {
- ZSTD_parameters params = srcCCtx->params;
- params.fParams.contentSizeFlag = (pledgedSrcSize > 0);
- ZSTD_resetCCtx_advanced(dstCCtx, params, pledgedSrcSize, ZSTDcrp_noMemset);
- }
-
- /* copy tables */
- {
- size_t const chainSize = (srcCCtx->params.cParams.strategy == ZSTD_fast) ? 0 : (1 << srcCCtx->params.cParams.chainLog);
- size_t const hSize = ((size_t)1) << srcCCtx->params.cParams.hashLog;
- size_t const h3Size = (size_t)1 << srcCCtx->hashLog3;
- size_t const tableSpace = (chainSize + hSize + h3Size) * sizeof(U32);
- memcpy(dstCCtx->workSpace, srcCCtx->workSpace, tableSpace);
- }
-
- /* copy dictionary offsets */
- dstCCtx->nextToUpdate = srcCCtx->nextToUpdate;
- dstCCtx->nextToUpdate3 = srcCCtx->nextToUpdate3;
- dstCCtx->nextSrc = srcCCtx->nextSrc;
- dstCCtx->base = srcCCtx->base;
- dstCCtx->dictBase = srcCCtx->dictBase;
- dstCCtx->dictLimit = srcCCtx->dictLimit;
- dstCCtx->lowLimit = srcCCtx->lowLimit;
- dstCCtx->loadedDictEnd = srcCCtx->loadedDictEnd;
- dstCCtx->dictID = srcCCtx->dictID;
-
- /* copy entropy tables */
- dstCCtx->flagStaticTables = srcCCtx->flagStaticTables;
- dstCCtx->flagStaticHufTable = srcCCtx->flagStaticHufTable;
- if (srcCCtx->flagStaticTables) {
- memcpy(dstCCtx->litlengthCTable, srcCCtx->litlengthCTable, sizeof(dstCCtx->litlengthCTable));
- memcpy(dstCCtx->matchlengthCTable, srcCCtx->matchlengthCTable, sizeof(dstCCtx->matchlengthCTable));
- memcpy(dstCCtx->offcodeCTable, srcCCtx->offcodeCTable, sizeof(dstCCtx->offcodeCTable));
- }
- if (srcCCtx->flagStaticHufTable) {
- memcpy(dstCCtx->hufTable, srcCCtx->hufTable, 256 * 4);
- }
-
- return 0;
-}
-
-/*! ZSTD_reduceTable() :
-* reduce table indexes by `reducerValue` */
-static void ZSTD_reduceTable(U32 *const table, U32 const size, U32 const reducerValue)
-{
- U32 u;
- for (u = 0; u < size; u++) {
- if (table[u] < reducerValue)
- table[u] = 0;
- else
- table[u] -= reducerValue;
- }
-}
-
-/*! ZSTD_reduceIndex() :
-* rescale all indexes to avoid future overflow (indexes are U32) */
-static void ZSTD_reduceIndex(ZSTD_CCtx *zc, const U32 reducerValue)
-{
- {
- U32 const hSize = 1 << zc->params.cParams.hashLog;
- ZSTD_reduceTable(zc->hashTable, hSize, reducerValue);
- }
-
- {
- U32 const chainSize = (zc->params.cParams.strategy == ZSTD_fast) ? 0 : (1 << zc->params.cParams.chainLog);
- ZSTD_reduceTable(zc->chainTable, chainSize, reducerValue);
- }
-
- {
- U32 const h3Size = (zc->hashLog3) ? 1 << zc->hashLog3 : 0;
- ZSTD_reduceTable(zc->hashTable3, h3Size, reducerValue);
- }
-}
-
-/*-*******************************************************
-* Block entropic compression
-*********************************************************/
-
-/* See doc/zstd_compression_format.md for detailed format description */
-
-size_t ZSTD_noCompressBlock(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- if (srcSize + ZSTD_blockHeaderSize > dstCapacity)
- return ERROR(dstSize_tooSmall);
- memcpy((BYTE *)dst + ZSTD_blockHeaderSize, src, srcSize);
- ZSTD_writeLE24(dst, (U32)(srcSize << 2) + (U32)bt_raw);
- return ZSTD_blockHeaderSize + srcSize;
-}
-
-static size_t ZSTD_noCompressLiterals(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- BYTE *const ostart = (BYTE * const)dst;
- U32 const flSize = 1 + (srcSize > 31) + (srcSize > 4095);
-
- if (srcSize + flSize > dstCapacity)
- return ERROR(dstSize_tooSmall);
-
- switch (flSize) {
- case 1: /* 2 - 1 - 5 */ ostart[0] = (BYTE)((U32)set_basic + (srcSize << 3)); break;
- case 2: /* 2 - 2 - 12 */ ZSTD_writeLE16(ostart, (U16)((U32)set_basic + (1 << 2) + (srcSize << 4))); break;
- default: /*note : should not be necessary : flSize is within {1,2,3} */
- case 3: /* 2 - 2 - 20 */ ZSTD_writeLE32(ostart, (U32)((U32)set_basic + (3 << 2) + (srcSize << 4))); break;
- }
-
- memcpy(ostart + flSize, src, srcSize);
- return srcSize + flSize;
-}
-
-static size_t ZSTD_compressRleLiteralsBlock(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- BYTE *const ostart = (BYTE * const)dst;
- U32 const flSize = 1 + (srcSize > 31) + (srcSize > 4095);
-
- (void)dstCapacity; /* dstCapacity already guaranteed to be >=4, hence large enough */
-
- switch (flSize) {
- case 1: /* 2 - 1 - 5 */ ostart[0] = (BYTE)((U32)set_rle + (srcSize << 3)); break;
- case 2: /* 2 - 2 - 12 */ ZSTD_writeLE16(ostart, (U16)((U32)set_rle + (1 << 2) + (srcSize << 4))); break;
- default: /*note : should not be necessary : flSize is necessarily within {1,2,3} */
- case 3: /* 2 - 2 - 20 */ ZSTD_writeLE32(ostart, (U32)((U32)set_rle + (3 << 2) + (srcSize << 4))); break;
- }
-
- ostart[flSize] = *(const BYTE *)src;
- return flSize + 1;
-}
-
-static size_t ZSTD_minGain(size_t srcSize) { return (srcSize >> 6) + 2; }
-
-static size_t ZSTD_compressLiterals(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t const minGain = ZSTD_minGain(srcSize);
- size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB);
- BYTE *const ostart = (BYTE *)dst;
- U32 singleStream = srcSize < 256;
- symbolEncodingType_e hType = set_compressed;
- size_t cLitSize;
-
-/* small ? don't even attempt compression (speed opt) */
-#define LITERAL_NOENTROPY 63
- {
- size_t const minLitSize = zc->flagStaticHufTable == HUF_repeat_valid ? 6 : LITERAL_NOENTROPY;
- if (srcSize <= minLitSize)
- return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
- }
-
- if (dstCapacity < lhSize + 1)
- return ERROR(dstSize_tooSmall); /* not enough space for compression */
- {
- HUF_repeat repeat = zc->flagStaticHufTable;
- int const preferRepeat = zc->params.cParams.strategy < ZSTD_lazy ? srcSize <= 1024 : 0;
- if (repeat == HUF_repeat_valid && lhSize == 3)
- singleStream = 1;
- cLitSize = singleStream ? HUF_compress1X_repeat(ostart + lhSize, dstCapacity - lhSize, src, srcSize, 255, 11, zc->tmpCounters,
- sizeof(zc->tmpCounters), zc->hufTable, &repeat, preferRepeat)
- : HUF_compress4X_repeat(ostart + lhSize, dstCapacity - lhSize, src, srcSize, 255, 11, zc->tmpCounters,
- sizeof(zc->tmpCounters), zc->hufTable, &repeat, preferRepeat);
- if (repeat != HUF_repeat_none) {
- hType = set_repeat;
- } /* reused the existing table */
- else {
- zc->flagStaticHufTable = HUF_repeat_check;
- } /* now have a table to reuse */
- }
-
- if ((cLitSize == 0) | (cLitSize >= srcSize - minGain)) {
- zc->flagStaticHufTable = HUF_repeat_none;
- return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
- }
- if (cLitSize == 1) {
- zc->flagStaticHufTable = HUF_repeat_none;
- return ZSTD_compressRleLiteralsBlock(dst, dstCapacity, src, srcSize);
- }
-
- /* Build header */
- switch (lhSize) {
- case 3: /* 2 - 2 - 10 - 10 */
- {
- U32 const lhc = hType + ((!singleStream) << 2) + ((U32)srcSize << 4) + ((U32)cLitSize << 14);
- ZSTD_writeLE24(ostart, lhc);
- break;
- }
- case 4: /* 2 - 2 - 14 - 14 */
- {
- U32 const lhc = hType + (2 << 2) + ((U32)srcSize << 4) + ((U32)cLitSize << 18);
- ZSTD_writeLE32(ostart, lhc);
- break;
- }
- default: /* should not be necessary, lhSize is only {3,4,5} */
- case 5: /* 2 - 2 - 18 - 18 */
- {
- U32 const lhc = hType + (3 << 2) + ((U32)srcSize << 4) + ((U32)cLitSize << 22);
- ZSTD_writeLE32(ostart, lhc);
- ostart[4] = (BYTE)(cLitSize >> 10);
- break;
- }
- }
- return lhSize + cLitSize;
-}
-
-static const BYTE LL_Code[64] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16, 17, 17, 18, 18,
- 19, 19, 20, 20, 20, 20, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23,
- 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24};
-
-static const BYTE ML_Code[128] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
- 26, 27, 28, 29, 30, 31, 32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38,
- 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
- 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42,
- 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42};
-
-void ZSTD_seqToCodes(const seqStore_t *seqStorePtr)
-{
- BYTE const LL_deltaCode = 19;
- BYTE const ML_deltaCode = 36;
- const seqDef *const sequences = seqStorePtr->sequencesStart;
- BYTE *const llCodeTable = seqStorePtr->llCode;
- BYTE *const ofCodeTable = seqStorePtr->ofCode;
- BYTE *const mlCodeTable = seqStorePtr->mlCode;
- U32 const nbSeq = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
- U32 u;
- for (u = 0; u < nbSeq; u++) {
- U32 const llv = sequences[u].litLength;
- U32 const mlv = sequences[u].matchLength;
- llCodeTable[u] = (llv > 63) ? (BYTE)ZSTD_highbit32(llv) + LL_deltaCode : LL_Code[llv];
- ofCodeTable[u] = (BYTE)ZSTD_highbit32(sequences[u].offset);
- mlCodeTable[u] = (mlv > 127) ? (BYTE)ZSTD_highbit32(mlv) + ML_deltaCode : ML_Code[mlv];
- }
- if (seqStorePtr->longLengthID == 1)
- llCodeTable[seqStorePtr->longLengthPos] = MaxLL;
- if (seqStorePtr->longLengthID == 2)
- mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
-}
-
-ZSTD_STATIC size_t ZSTD_compressSequences_internal(ZSTD_CCtx *zc, void *dst, size_t dstCapacity)
-{
- const int longOffsets = zc->params.cParams.windowLog > STREAM_ACCUMULATOR_MIN;
- const seqStore_t *seqStorePtr = &(zc->seqStore);
- FSE_CTable *CTable_LitLength = zc->litlengthCTable;
- FSE_CTable *CTable_OffsetBits = zc->offcodeCTable;
- FSE_CTable *CTable_MatchLength = zc->matchlengthCTable;
- U32 LLtype, Offtype, MLtype; /* compressed, raw or rle */
- const seqDef *const sequences = seqStorePtr->sequencesStart;
- const BYTE *const ofCodeTable = seqStorePtr->ofCode;
- const BYTE *const llCodeTable = seqStorePtr->llCode;
- const BYTE *const mlCodeTable = seqStorePtr->mlCode;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstCapacity;
- BYTE *op = ostart;
- size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
- BYTE *seqHead;
-
- U32 *count;
- S16 *norm;
- U32 *workspace;
- size_t workspaceSize = sizeof(zc->tmpCounters);
- {
- size_t spaceUsed32 = 0;
- count = (U32 *)zc->tmpCounters + spaceUsed32;
- spaceUsed32 += MaxSeq + 1;
- norm = (S16 *)((U32 *)zc->tmpCounters + spaceUsed32);
- spaceUsed32 += ALIGN(sizeof(S16) * (MaxSeq + 1), sizeof(U32)) >> 2;
-
- workspace = (U32 *)zc->tmpCounters + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
- }
-
- /* Compress literals */
- {
- const BYTE *const literals = seqStorePtr->litStart;
- size_t const litSize = seqStorePtr->lit - literals;
- size_t const cSize = ZSTD_compressLiterals(zc, op, dstCapacity, literals, litSize);
- if (ZSTD_isError(cSize))
- return cSize;
- op += cSize;
- }
-
- /* Sequences Header */
- if ((oend - op) < 3 /*max nbSeq Size*/ + 1 /*seqHead */)
- return ERROR(dstSize_tooSmall);
- if (nbSeq < 0x7F)
- *op++ = (BYTE)nbSeq;
- else if (nbSeq < LONGNBSEQ)
- op[0] = (BYTE)((nbSeq >> 8) + 0x80), op[1] = (BYTE)nbSeq, op += 2;
- else
- op[0] = 0xFF, ZSTD_writeLE16(op + 1, (U16)(nbSeq - LONGNBSEQ)), op += 3;
- if (nbSeq == 0)
- return op - ostart;
-
- /* seqHead : flags for FSE encoding type */
- seqHead = op++;
-
-#define MIN_SEQ_FOR_DYNAMIC_FSE 64
-#define MAX_SEQ_FOR_STATIC_FSE 1000
-
- /* convert length/distances into codes */
- ZSTD_seqToCodes(seqStorePtr);
-
- /* CTable for Literal Lengths */
- {
- U32 max = MaxLL;
- size_t const mostFrequent = FSE_countFast_wksp(count, &max, llCodeTable, nbSeq, workspace);
- if ((mostFrequent == nbSeq) && (nbSeq > 2)) {
- *op++ = llCodeTable[0];
- FSE_buildCTable_rle(CTable_LitLength, (BYTE)max);
- LLtype = set_rle;
- } else if ((zc->flagStaticTables) && (nbSeq < MAX_SEQ_FOR_STATIC_FSE)) {
- LLtype = set_repeat;
- } else if ((nbSeq < MIN_SEQ_FOR_DYNAMIC_FSE) || (mostFrequent < (nbSeq >> (LL_defaultNormLog - 1)))) {
- FSE_buildCTable_wksp(CTable_LitLength, LL_defaultNorm, MaxLL, LL_defaultNormLog, workspace, workspaceSize);
- LLtype = set_basic;
- } else {
- size_t nbSeq_1 = nbSeq;
- const U32 tableLog = FSE_optimalTableLog(LLFSELog, nbSeq, max);
- if (count[llCodeTable[nbSeq - 1]] > 1) {
- count[llCodeTable[nbSeq - 1]]--;
- nbSeq_1--;
- }
- FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
- {
- size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */
- if (FSE_isError(NCountSize))
- return NCountSize;
- op += NCountSize;
- }
- FSE_buildCTable_wksp(CTable_LitLength, norm, max, tableLog, workspace, workspaceSize);
- LLtype = set_compressed;
- }
- }
-
- /* CTable for Offsets */
- {
- U32 max = MaxOff;
- size_t const mostFrequent = FSE_countFast_wksp(count, &max, ofCodeTable, nbSeq, workspace);
- if ((mostFrequent == nbSeq) && (nbSeq > 2)) {
- *op++ = ofCodeTable[0];
- FSE_buildCTable_rle(CTable_OffsetBits, (BYTE)max);
- Offtype = set_rle;
- } else if ((zc->flagStaticTables) && (nbSeq < MAX_SEQ_FOR_STATIC_FSE)) {
- Offtype = set_repeat;
- } else if ((nbSeq < MIN_SEQ_FOR_DYNAMIC_FSE) || (mostFrequent < (nbSeq >> (OF_defaultNormLog - 1)))) {
- FSE_buildCTable_wksp(CTable_OffsetBits, OF_defaultNorm, MaxOff, OF_defaultNormLog, workspace, workspaceSize);
- Offtype = set_basic;
- } else {
- size_t nbSeq_1 = nbSeq;
- const U32 tableLog = FSE_optimalTableLog(OffFSELog, nbSeq, max);
- if (count[ofCodeTable[nbSeq - 1]] > 1) {
- count[ofCodeTable[nbSeq - 1]]--;
- nbSeq_1--;
- }
- FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
- {
- size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */
- if (FSE_isError(NCountSize))
- return NCountSize;
- op += NCountSize;
- }
- FSE_buildCTable_wksp(CTable_OffsetBits, norm, max, tableLog, workspace, workspaceSize);
- Offtype = set_compressed;
- }
- }
-
- /* CTable for MatchLengths */
- {
- U32 max = MaxML;
- size_t const mostFrequent = FSE_countFast_wksp(count, &max, mlCodeTable, nbSeq, workspace);
- if ((mostFrequent == nbSeq) && (nbSeq > 2)) {
- *op++ = *mlCodeTable;
- FSE_buildCTable_rle(CTable_MatchLength, (BYTE)max);
- MLtype = set_rle;
- } else if ((zc->flagStaticTables) && (nbSeq < MAX_SEQ_FOR_STATIC_FSE)) {
- MLtype = set_repeat;
- } else if ((nbSeq < MIN_SEQ_FOR_DYNAMIC_FSE) || (mostFrequent < (nbSeq >> (ML_defaultNormLog - 1)))) {
- FSE_buildCTable_wksp(CTable_MatchLength, ML_defaultNorm, MaxML, ML_defaultNormLog, workspace, workspaceSize);
- MLtype = set_basic;
- } else {
- size_t nbSeq_1 = nbSeq;
- const U32 tableLog = FSE_optimalTableLog(MLFSELog, nbSeq, max);
- if (count[mlCodeTable[nbSeq - 1]] > 1) {
- count[mlCodeTable[nbSeq - 1]]--;
- nbSeq_1--;
- }
- FSE_normalizeCount(norm, tableLog, count, nbSeq_1, max);
- {
- size_t const NCountSize = FSE_writeNCount(op, oend - op, norm, max, tableLog); /* overflow protected */
- if (FSE_isError(NCountSize))
- return NCountSize;
- op += NCountSize;
- }
- FSE_buildCTable_wksp(CTable_MatchLength, norm, max, tableLog, workspace, workspaceSize);
- MLtype = set_compressed;
- }
- }
-
- *seqHead = (BYTE)((LLtype << 6) + (Offtype << 4) + (MLtype << 2));
- zc->flagStaticTables = 0;
-
- /* Encoding Sequences */
- {
- BIT_CStream_t blockStream;
- FSE_CState_t stateMatchLength;
- FSE_CState_t stateOffsetBits;
- FSE_CState_t stateLitLength;
-
- CHECK_E(BIT_initCStream(&blockStream, op, oend - op), dstSize_tooSmall); /* not enough space remaining */
-
- /* first symbols */
- FSE_initCState2(&stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq - 1]);
- FSE_initCState2(&stateOffsetBits, CTable_OffsetBits, ofCodeTable[nbSeq - 1]);
- FSE_initCState2(&stateLitLength, CTable_LitLength, llCodeTable[nbSeq - 1]);
- BIT_addBits(&blockStream, sequences[nbSeq - 1].litLength, LL_bits[llCodeTable[nbSeq - 1]]);
- if (ZSTD_32bits())
- BIT_flushBits(&blockStream);
- BIT_addBits(&blockStream, sequences[nbSeq - 1].matchLength, ML_bits[mlCodeTable[nbSeq - 1]]);
- if (ZSTD_32bits())
- BIT_flushBits(&blockStream);
- if (longOffsets) {
- U32 const ofBits = ofCodeTable[nbSeq - 1];
- int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN - 1);
- if (extraBits) {
- BIT_addBits(&blockStream, sequences[nbSeq - 1].offset, extraBits);
- BIT_flushBits(&blockStream);
- }
- BIT_addBits(&blockStream, sequences[nbSeq - 1].offset >> extraBits, ofBits - extraBits);
- } else {
- BIT_addBits(&blockStream, sequences[nbSeq - 1].offset, ofCodeTable[nbSeq - 1]);
- }
- BIT_flushBits(&blockStream);
-
- {
- size_t n;
- for (n = nbSeq - 2; n < nbSeq; n--) { /* intentional underflow */
- BYTE const llCode = llCodeTable[n];
- BYTE const ofCode = ofCodeTable[n];
- BYTE const mlCode = mlCodeTable[n];
- U32 const llBits = LL_bits[llCode];
- U32 const ofBits = ofCode; /* 32b*/ /* 64b*/
- U32 const mlBits = ML_bits[mlCode];
- /* (7)*/ /* (7)*/
- FSE_encodeSymbol(&blockStream, &stateOffsetBits, ofCode); /* 15 */ /* 15 */
- FSE_encodeSymbol(&blockStream, &stateMatchLength, mlCode); /* 24 */ /* 24 */
- if (ZSTD_32bits())
- BIT_flushBits(&blockStream); /* (7)*/
- FSE_encodeSymbol(&blockStream, &stateLitLength, llCode); /* 16 */ /* 33 */
- if (ZSTD_32bits() || (ofBits + mlBits + llBits >= 64 - 7 - (LLFSELog + MLFSELog + OffFSELog)))
- BIT_flushBits(&blockStream); /* (7)*/
- BIT_addBits(&blockStream, sequences[n].litLength, llBits);
- if (ZSTD_32bits() && ((llBits + mlBits) > 24))
- BIT_flushBits(&blockStream);
- BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
- if (ZSTD_32bits())
- BIT_flushBits(&blockStream); /* (7)*/
- if (longOffsets) {
- int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN - 1);
- if (extraBits) {
- BIT_addBits(&blockStream, sequences[n].offset, extraBits);
- BIT_flushBits(&blockStream); /* (7)*/
- }
- BIT_addBits(&blockStream, sequences[n].offset >> extraBits, ofBits - extraBits); /* 31 */
- } else {
- BIT_addBits(&blockStream, sequences[n].offset, ofBits); /* 31 */
- }
- BIT_flushBits(&blockStream); /* (7)*/
- }
- }
-
- FSE_flushCState(&blockStream, &stateMatchLength);
- FSE_flushCState(&blockStream, &stateOffsetBits);
- FSE_flushCState(&blockStream, &stateLitLength);
-
- {
- size_t const streamSize = BIT_closeCStream(&blockStream);
- if (streamSize == 0)
- return ERROR(dstSize_tooSmall); /* not enough space */
- op += streamSize;
- }
- }
- return op - ostart;
-}
-
-ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, size_t srcSize)
-{
- size_t const cSize = ZSTD_compressSequences_internal(zc, dst, dstCapacity);
- size_t const minGain = ZSTD_minGain(srcSize);
- size_t const maxCSize = srcSize - minGain;
- /* If the srcSize <= dstCapacity, then there is enough space to write a
- * raw uncompressed block. Since we ran out of space, the block must not
- * be compressible, so fall back to a raw uncompressed block.
- */
- int const uncompressibleError = cSize == ERROR(dstSize_tooSmall) && srcSize <= dstCapacity;
- int i;
-
- if (ZSTD_isError(cSize) && !uncompressibleError)
- return cSize;
- if (cSize >= maxCSize || uncompressibleError) {
- zc->flagStaticHufTable = HUF_repeat_none;
- return 0;
- }
- /* confirm repcodes */
- for (i = 0; i < ZSTD_REP_NUM; i++)
- zc->rep[i] = zc->repToConfirm[i];
- return cSize;
-}
-
-/*! ZSTD_storeSeq() :
- Store a sequence (literal length, literals, offset code and match length code) into seqStore_t.
- `offsetCode` : distance to match, or 0 == repCode.
- `matchCode` : matchLength - MINMATCH
-*/
-ZSTD_STATIC void ZSTD_storeSeq(seqStore_t *seqStorePtr, size_t litLength, const void *literals, U32 offsetCode, size_t matchCode)
-{
- /* copy Literals */
- ZSTD_wildcopy(seqStorePtr->lit, literals, litLength);
- seqStorePtr->lit += litLength;
-
- /* literal Length */
- if (litLength > 0xFFFF) {
- seqStorePtr->longLengthID = 1;
- seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
- }
- seqStorePtr->sequences[0].litLength = (U16)litLength;
-
- /* match offset */
- seqStorePtr->sequences[0].offset = offsetCode + 1;
-
- /* match Length */
- if (matchCode > 0xFFFF) {
- seqStorePtr->longLengthID = 2;
- seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
- }
- seqStorePtr->sequences[0].matchLength = (U16)matchCode;
-
- seqStorePtr->sequences++;
-}
-
-/*-*************************************
-* Match length counter
-***************************************/
-static unsigned ZSTD_NbCommonBytes(register size_t val)
-{
- if (ZSTD_isLittleEndian()) {
- if (ZSTD_64bits()) {
- return (__builtin_ctzll((U64)val) >> 3);
- } else { /* 32 bits */
- return (__builtin_ctz((U32)val) >> 3);
- }
- } else { /* Big Endian CPU */
- if (ZSTD_64bits()) {
- return (__builtin_clzll(val) >> 3);
- } else { /* 32 bits */
- return (__builtin_clz((U32)val) >> 3);
- }
- }
-}
-
-static size_t ZSTD_count(const BYTE *pIn, const BYTE *pMatch, const BYTE *const pInLimit)
-{
- const BYTE *const pStart = pIn;
- const BYTE *const pInLoopLimit = pInLimit - (sizeof(size_t) - 1);
-
- while (pIn < pInLoopLimit) {
- size_t const diff = ZSTD_readST(pMatch) ^ ZSTD_readST(pIn);
- if (!diff) {
- pIn += sizeof(size_t);
- pMatch += sizeof(size_t);
- continue;
- }
- pIn += ZSTD_NbCommonBytes(diff);
- return (size_t)(pIn - pStart);
- }
- if (ZSTD_64bits())
- if ((pIn < (pInLimit - 3)) && (ZSTD_read32(pMatch) == ZSTD_read32(pIn))) {
- pIn += 4;
- pMatch += 4;
- }
- if ((pIn < (pInLimit - 1)) && (ZSTD_read16(pMatch) == ZSTD_read16(pIn))) {
- pIn += 2;
- pMatch += 2;
- }
- if ((pIn < pInLimit) && (*pMatch == *pIn))
- pIn++;
- return (size_t)(pIn - pStart);
-}
-
-/** ZSTD_count_2segments() :
-* can count match length with `ip` & `match` in 2 different segments.
-* convention : on reaching mEnd, match count continue starting from iStart
-*/
-static size_t ZSTD_count_2segments(const BYTE *ip, const BYTE *match, const BYTE *iEnd, const BYTE *mEnd, const BYTE *iStart)
-{
- const BYTE *const vEnd = MIN(ip + (mEnd - match), iEnd);
- size_t const matchLength = ZSTD_count(ip, match, vEnd);
- if (match + matchLength != mEnd)
- return matchLength;
- return matchLength + ZSTD_count(ip + matchLength, iStart, iEnd);
-}
-
-/*-*************************************
-* Hashes
-***************************************/
-static const U32 prime3bytes = 506832829U;
-static U32 ZSTD_hash3(U32 u, U32 h) { return ((u << (32 - 24)) * prime3bytes) >> (32 - h); }
-ZSTD_STATIC size_t ZSTD_hash3Ptr(const void *ptr, U32 h) { return ZSTD_hash3(ZSTD_readLE32(ptr), h); } /* only in zstd_opt.h */
-
-static const U32 prime4bytes = 2654435761U;
-static U32 ZSTD_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32 - h); }
-static size_t ZSTD_hash4Ptr(const void *ptr, U32 h) { return ZSTD_hash4(ZSTD_read32(ptr), h); }
-
-static const U64 prime5bytes = 889523592379ULL;
-static size_t ZSTD_hash5(U64 u, U32 h) { return (size_t)(((u << (64 - 40)) * prime5bytes) >> (64 - h)); }
-static size_t ZSTD_hash5Ptr(const void *p, U32 h) { return ZSTD_hash5(ZSTD_readLE64(p), h); }
-
-static const U64 prime6bytes = 227718039650203ULL;
-static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64 - 48)) * prime6bytes) >> (64 - h)); }
-static size_t ZSTD_hash6Ptr(const void *p, U32 h) { return ZSTD_hash6(ZSTD_readLE64(p), h); }
-
-static const U64 prime7bytes = 58295818150454627ULL;
-static size_t ZSTD_hash7(U64 u, U32 h) { return (size_t)(((u << (64 - 56)) * prime7bytes) >> (64 - h)); }
-static size_t ZSTD_hash7Ptr(const void *p, U32 h) { return ZSTD_hash7(ZSTD_readLE64(p), h); }
-
-static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
-static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u)*prime8bytes) >> (64 - h)); }
-static size_t ZSTD_hash8Ptr(const void *p, U32 h) { return ZSTD_hash8(ZSTD_readLE64(p), h); }
-
-static size_t ZSTD_hashPtr(const void *p, U32 hBits, U32 mls)
-{
- switch (mls) {
- // case 3: return ZSTD_hash3Ptr(p, hBits);
- default:
- case 4: return ZSTD_hash4Ptr(p, hBits);
- case 5: return ZSTD_hash5Ptr(p, hBits);
- case 6: return ZSTD_hash6Ptr(p, hBits);
- case 7: return ZSTD_hash7Ptr(p, hBits);
- case 8: return ZSTD_hash8Ptr(p, hBits);
- }
-}
-
-/*-*************************************
-* Fast Scan
-***************************************/
-static void ZSTD_fillHashTable(ZSTD_CCtx *zc, const void *end, const U32 mls)
-{
- U32 *const hashTable = zc->hashTable;
- U32 const hBits = zc->params.cParams.hashLog;
- const BYTE *const base = zc->base;
- const BYTE *ip = base + zc->nextToUpdate;
- const BYTE *const iend = ((const BYTE *)end) - HASH_READ_SIZE;
- const size_t fastHashFillStep = 3;
-
- while (ip <= iend) {
- hashTable[ZSTD_hashPtr(ip, hBits, mls)] = (U32)(ip - base);
- ip += fastHashFillStep;
- }
-}
-
-FORCE_INLINE
-void ZSTD_compressBlock_fast_generic(ZSTD_CCtx *cctx, const void *src, size_t srcSize, const U32 mls)
-{
- U32 *const hashTable = cctx->hashTable;
- U32 const hBits = cctx->params.cParams.hashLog;
- seqStore_t *seqStorePtr = &(cctx->seqStore);
- const BYTE *const base = cctx->base;
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const U32 lowestIndex = cctx->dictLimit;
- const BYTE *const lowest = base + lowestIndex;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - HASH_READ_SIZE;
- U32 offset_1 = cctx->rep[0], offset_2 = cctx->rep[1];
- U32 offsetSaved = 0;
-
- /* init */
- ip += (ip == lowest);
- {
- U32 const maxRep = (U32)(ip - lowest);
- if (offset_2 > maxRep)
- offsetSaved = offset_2, offset_2 = 0;
- if (offset_1 > maxRep)
- offsetSaved = offset_1, offset_1 = 0;
- }
-
- /* Main Search Loop */
- while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
- size_t mLength;
- size_t const h = ZSTD_hashPtr(ip, hBits, mls);
- U32 const curr = (U32)(ip - base);
- U32 const matchIndex = hashTable[h];
- const BYTE *match = base + matchIndex;
- hashTable[h] = curr; /* update hash table */
-
- if ((offset_1 > 0) & (ZSTD_read32(ip + 1 - offset_1) == ZSTD_read32(ip + 1))) {
- mLength = ZSTD_count(ip + 1 + 4, ip + 1 + 4 - offset_1, iend) + 4;
- ip++;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, 0, mLength - MINMATCH);
- } else {
- U32 offset;
- if ((matchIndex <= lowestIndex) || (ZSTD_read32(match) != ZSTD_read32(ip))) {
- ip += ((ip - anchor) >> g_searchStrength) + 1;
- continue;
- }
- mLength = ZSTD_count(ip + 4, match + 4, iend) + 4;
- offset = (U32)(ip - match);
- while (((ip > anchor) & (match > lowest)) && (ip[-1] == match[-1])) {
- ip--;
- match--;
- mLength++;
- } /* catch up */
- offset_2 = offset_1;
- offset_1 = offset;
-
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, offset + ZSTD_REP_MOVE, mLength - MINMATCH);
- }
-
- /* match found */
- ip += mLength;
- anchor = ip;
-
- if (ip <= ilimit) {
- /* Fill Table */
- hashTable[ZSTD_hashPtr(base + curr + 2, hBits, mls)] = curr + 2; /* here because curr+2 could be > iend-8 */
- hashTable[ZSTD_hashPtr(ip - 2, hBits, mls)] = (U32)(ip - 2 - base);
- /* check immediate repcode */
- while ((ip <= ilimit) && ((offset_2 > 0) & (ZSTD_read32(ip) == ZSTD_read32(ip - offset_2)))) {
- /* store sequence */
- size_t const rLength = ZSTD_count(ip + 4, ip + 4 - offset_2, iend) + 4;
- {
- U32 const tmpOff = offset_2;
- offset_2 = offset_1;
- offset_1 = tmpOff;
- } /* swap offset_2 <=> offset_1 */
- hashTable[ZSTD_hashPtr(ip, hBits, mls)] = (U32)(ip - base);
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, rLength - MINMATCH);
- ip += rLength;
- anchor = ip;
- continue; /* faster when present ... (?) */
- }
- }
- }
-
- /* save reps for next block */
- cctx->repToConfirm[0] = offset_1 ? offset_1 : offsetSaved;
- cctx->repToConfirm[1] = offset_2 ? offset_2 : offsetSaved;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-static void ZSTD_compressBlock_fast(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- const U32 mls = ctx->params.cParams.searchLength;
- switch (mls) {
- default: /* includes case 3 */
- case 4: ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 4); return;
- case 5: ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 5); return;
- case 6: ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 6); return;
- case 7: ZSTD_compressBlock_fast_generic(ctx, src, srcSize, 7); return;
- }
-}
-
-static void ZSTD_compressBlock_fast_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const U32 mls)
-{
- U32 *hashTable = ctx->hashTable;
- const U32 hBits = ctx->params.cParams.hashLog;
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const base = ctx->base;
- const BYTE *const dictBase = ctx->dictBase;
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const U32 lowestIndex = ctx->lowLimit;
- const BYTE *const dictStart = dictBase + lowestIndex;
- const U32 dictLimit = ctx->dictLimit;
- const BYTE *const lowPrefixPtr = base + dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- U32 offset_1 = ctx->rep[0], offset_2 = ctx->rep[1];
-
- /* Search Loop */
- while (ip < ilimit) { /* < instead of <=, because (ip+1) */
- const size_t h = ZSTD_hashPtr(ip, hBits, mls);
- const U32 matchIndex = hashTable[h];
- const BYTE *matchBase = matchIndex < dictLimit ? dictBase : base;
- const BYTE *match = matchBase + matchIndex;
- const U32 curr = (U32)(ip - base);
- const U32 repIndex = curr + 1 - offset_1; /* offset_1 expected <= curr +1 */
- const BYTE *repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *repMatch = repBase + repIndex;
- size_t mLength;
- hashTable[h] = curr; /* update hash table */
-
- if ((((U32)((dictLimit - 1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > lowestIndex)) &&
- (ZSTD_read32(repMatch) == ZSTD_read32(ip + 1))) {
- const BYTE *repMatchEnd = repIndex < dictLimit ? dictEnd : iend;
- mLength = ZSTD_count_2segments(ip + 1 + EQUAL_READ32, repMatch + EQUAL_READ32, iend, repMatchEnd, lowPrefixPtr) + EQUAL_READ32;
- ip++;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, 0, mLength - MINMATCH);
- } else {
- if ((matchIndex < lowestIndex) || (ZSTD_read32(match) != ZSTD_read32(ip))) {
- ip += ((ip - anchor) >> g_searchStrength) + 1;
- continue;
- }
- {
- const BYTE *matchEnd = matchIndex < dictLimit ? dictEnd : iend;
- const BYTE *lowMatchPtr = matchIndex < dictLimit ? dictStart : lowPrefixPtr;
- U32 offset;
- mLength = ZSTD_count_2segments(ip + EQUAL_READ32, match + EQUAL_READ32, iend, matchEnd, lowPrefixPtr) + EQUAL_READ32;
- while (((ip > anchor) & (match > lowMatchPtr)) && (ip[-1] == match[-1])) {
- ip--;
- match--;
- mLength++;
- } /* catch up */
- offset = curr - matchIndex;
- offset_2 = offset_1;
- offset_1 = offset;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, offset + ZSTD_REP_MOVE, mLength - MINMATCH);
- }
- }
-
- /* found a match : store it */
- ip += mLength;
- anchor = ip;
-
- if (ip <= ilimit) {
- /* Fill Table */
- hashTable[ZSTD_hashPtr(base + curr + 2, hBits, mls)] = curr + 2;
- hashTable[ZSTD_hashPtr(ip - 2, hBits, mls)] = (U32)(ip - 2 - base);
- /* check immediate repcode */
- while (ip <= ilimit) {
- U32 const curr2 = (U32)(ip - base);
- U32 const repIndex2 = curr2 - offset_2;
- const BYTE *repMatch2 = repIndex2 < dictLimit ? dictBase + repIndex2 : base + repIndex2;
- if ((((U32)((dictLimit - 1) - repIndex2) >= 3) & (repIndex2 > lowestIndex)) /* intentional overflow */
- && (ZSTD_read32(repMatch2) == ZSTD_read32(ip))) {
- const BYTE *const repEnd2 = repIndex2 < dictLimit ? dictEnd : iend;
- size_t repLength2 =
- ZSTD_count_2segments(ip + EQUAL_READ32, repMatch2 + EQUAL_READ32, iend, repEnd2, lowPrefixPtr) + EQUAL_READ32;
- U32 tmpOffset = offset_2;
- offset_2 = offset_1;
- offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, repLength2 - MINMATCH);
- hashTable[ZSTD_hashPtr(ip, hBits, mls)] = curr2;
- ip += repLength2;
- anchor = ip;
- continue;
- }
- break;
- }
- }
- }
-
- /* save reps for next block */
- ctx->repToConfirm[0] = offset_1;
- ctx->repToConfirm[1] = offset_2;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-static void ZSTD_compressBlock_fast_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- U32 const mls = ctx->params.cParams.searchLength;
- switch (mls) {
- default: /* includes case 3 */
- case 4: ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 4); return;
- case 5: ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 5); return;
- case 6: ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 6); return;
- case 7: ZSTD_compressBlock_fast_extDict_generic(ctx, src, srcSize, 7); return;
- }
-}
-
-/*-*************************************
-* Double Fast
-***************************************/
-static void ZSTD_fillDoubleHashTable(ZSTD_CCtx *cctx, const void *end, const U32 mls)
-{
- U32 *const hashLarge = cctx->hashTable;
- U32 const hBitsL = cctx->params.cParams.hashLog;
- U32 *const hashSmall = cctx->chainTable;
- U32 const hBitsS = cctx->params.cParams.chainLog;
- const BYTE *const base = cctx->base;
- const BYTE *ip = base + cctx->nextToUpdate;
- const BYTE *const iend = ((const BYTE *)end) - HASH_READ_SIZE;
- const size_t fastHashFillStep = 3;
-
- while (ip <= iend) {
- hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip - base);
- hashLarge[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip - base);
- ip += fastHashFillStep;
- }
-}
-
-FORCE_INLINE
-void ZSTD_compressBlock_doubleFast_generic(ZSTD_CCtx *cctx, const void *src, size_t srcSize, const U32 mls)
-{
- U32 *const hashLong = cctx->hashTable;
- const U32 hBitsL = cctx->params.cParams.hashLog;
- U32 *const hashSmall = cctx->chainTable;
- const U32 hBitsS = cctx->params.cParams.chainLog;
- seqStore_t *seqStorePtr = &(cctx->seqStore);
- const BYTE *const base = cctx->base;
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const U32 lowestIndex = cctx->dictLimit;
- const BYTE *const lowest = base + lowestIndex;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - HASH_READ_SIZE;
- U32 offset_1 = cctx->rep[0], offset_2 = cctx->rep[1];
- U32 offsetSaved = 0;
-
- /* init */
- ip += (ip == lowest);
- {
- U32 const maxRep = (U32)(ip - lowest);
- if (offset_2 > maxRep)
- offsetSaved = offset_2, offset_2 = 0;
- if (offset_1 > maxRep)
- offsetSaved = offset_1, offset_1 = 0;
- }
-
- /* Main Search Loop */
- while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
- size_t mLength;
- size_t const h2 = ZSTD_hashPtr(ip, hBitsL, 8);
- size_t const h = ZSTD_hashPtr(ip, hBitsS, mls);
- U32 const curr = (U32)(ip - base);
- U32 const matchIndexL = hashLong[h2];
- U32 const matchIndexS = hashSmall[h];
- const BYTE *matchLong = base + matchIndexL;
- const BYTE *match = base + matchIndexS;
- hashLong[h2] = hashSmall[h] = curr; /* update hash tables */
-
- if ((offset_1 > 0) & (ZSTD_read32(ip + 1 - offset_1) == ZSTD_read32(ip + 1))) { /* note : by construction, offset_1 <= curr */
- mLength = ZSTD_count(ip + 1 + 4, ip + 1 + 4 - offset_1, iend) + 4;
- ip++;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, 0, mLength - MINMATCH);
- } else {
- U32 offset;
- if ((matchIndexL > lowestIndex) && (ZSTD_read64(matchLong) == ZSTD_read64(ip))) {
- mLength = ZSTD_count(ip + 8, matchLong + 8, iend) + 8;
- offset = (U32)(ip - matchLong);
- while (((ip > anchor) & (matchLong > lowest)) && (ip[-1] == matchLong[-1])) {
- ip--;
- matchLong--;
- mLength++;
- } /* catch up */
- } else if ((matchIndexS > lowestIndex) && (ZSTD_read32(match) == ZSTD_read32(ip))) {
- size_t const h3 = ZSTD_hashPtr(ip + 1, hBitsL, 8);
- U32 const matchIndex3 = hashLong[h3];
- const BYTE *match3 = base + matchIndex3;
- hashLong[h3] = curr + 1;
- if ((matchIndex3 > lowestIndex) && (ZSTD_read64(match3) == ZSTD_read64(ip + 1))) {
- mLength = ZSTD_count(ip + 9, match3 + 8, iend) + 8;
- ip++;
- offset = (U32)(ip - match3);
- while (((ip > anchor) & (match3 > lowest)) && (ip[-1] == match3[-1])) {
- ip--;
- match3--;
- mLength++;
- } /* catch up */
- } else {
- mLength = ZSTD_count(ip + 4, match + 4, iend) + 4;
- offset = (U32)(ip - match);
- while (((ip > anchor) & (match > lowest)) && (ip[-1] == match[-1])) {
- ip--;
- match--;
- mLength++;
- } /* catch up */
- }
- } else {
- ip += ((ip - anchor) >> g_searchStrength) + 1;
- continue;
- }
-
- offset_2 = offset_1;
- offset_1 = offset;
-
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, offset + ZSTD_REP_MOVE, mLength - MINMATCH);
- }
-
- /* match found */
- ip += mLength;
- anchor = ip;
-
- if (ip <= ilimit) {
- /* Fill Table */
- hashLong[ZSTD_hashPtr(base + curr + 2, hBitsL, 8)] = hashSmall[ZSTD_hashPtr(base + curr + 2, hBitsS, mls)] =
- curr + 2; /* here because curr+2 could be > iend-8 */
- hashLong[ZSTD_hashPtr(ip - 2, hBitsL, 8)] = hashSmall[ZSTD_hashPtr(ip - 2, hBitsS, mls)] = (U32)(ip - 2 - base);
-
- /* check immediate repcode */
- while ((ip <= ilimit) && ((offset_2 > 0) & (ZSTD_read32(ip) == ZSTD_read32(ip - offset_2)))) {
- /* store sequence */
- size_t const rLength = ZSTD_count(ip + 4, ip + 4 - offset_2, iend) + 4;
- {
- U32 const tmpOff = offset_2;
- offset_2 = offset_1;
- offset_1 = tmpOff;
- } /* swap offset_2 <=> offset_1 */
- hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip - base);
- hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip - base);
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, rLength - MINMATCH);
- ip += rLength;
- anchor = ip;
- continue; /* faster when present ... (?) */
- }
- }
- }
-
- /* save reps for next block */
- cctx->repToConfirm[0] = offset_1 ? offset_1 : offsetSaved;
- cctx->repToConfirm[1] = offset_2 ? offset_2 : offsetSaved;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-static void ZSTD_compressBlock_doubleFast(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- const U32 mls = ctx->params.cParams.searchLength;
- switch (mls) {
- default: /* includes case 3 */
- case 4: ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 4); return;
- case 5: ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 5); return;
- case 6: ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 6); return;
- case 7: ZSTD_compressBlock_doubleFast_generic(ctx, src, srcSize, 7); return;
- }
-}
-
-static void ZSTD_compressBlock_doubleFast_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const U32 mls)
-{
- U32 *const hashLong = ctx->hashTable;
- U32 const hBitsL = ctx->params.cParams.hashLog;
- U32 *const hashSmall = ctx->chainTable;
- U32 const hBitsS = ctx->params.cParams.chainLog;
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const base = ctx->base;
- const BYTE *const dictBase = ctx->dictBase;
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const U32 lowestIndex = ctx->lowLimit;
- const BYTE *const dictStart = dictBase + lowestIndex;
- const U32 dictLimit = ctx->dictLimit;
- const BYTE *const lowPrefixPtr = base + dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- U32 offset_1 = ctx->rep[0], offset_2 = ctx->rep[1];
-
- /* Search Loop */
- while (ip < ilimit) { /* < instead of <=, because (ip+1) */
- const size_t hSmall = ZSTD_hashPtr(ip, hBitsS, mls);
- const U32 matchIndex = hashSmall[hSmall];
- const BYTE *matchBase = matchIndex < dictLimit ? dictBase : base;
- const BYTE *match = matchBase + matchIndex;
-
- const size_t hLong = ZSTD_hashPtr(ip, hBitsL, 8);
- const U32 matchLongIndex = hashLong[hLong];
- const BYTE *matchLongBase = matchLongIndex < dictLimit ? dictBase : base;
- const BYTE *matchLong = matchLongBase + matchLongIndex;
-
- const U32 curr = (U32)(ip - base);
- const U32 repIndex = curr + 1 - offset_1; /* offset_1 expected <= curr +1 */
- const BYTE *repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *repMatch = repBase + repIndex;
- size_t mLength;
- hashSmall[hSmall] = hashLong[hLong] = curr; /* update hash table */
-
- if ((((U32)((dictLimit - 1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > lowestIndex)) &&
- (ZSTD_read32(repMatch) == ZSTD_read32(ip + 1))) {
- const BYTE *repMatchEnd = repIndex < dictLimit ? dictEnd : iend;
- mLength = ZSTD_count_2segments(ip + 1 + 4, repMatch + 4, iend, repMatchEnd, lowPrefixPtr) + 4;
- ip++;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, 0, mLength - MINMATCH);
- } else {
- if ((matchLongIndex > lowestIndex) && (ZSTD_read64(matchLong) == ZSTD_read64(ip))) {
- const BYTE *matchEnd = matchLongIndex < dictLimit ? dictEnd : iend;
- const BYTE *lowMatchPtr = matchLongIndex < dictLimit ? dictStart : lowPrefixPtr;
- U32 offset;
- mLength = ZSTD_count_2segments(ip + 8, matchLong + 8, iend, matchEnd, lowPrefixPtr) + 8;
- offset = curr - matchLongIndex;
- while (((ip > anchor) & (matchLong > lowMatchPtr)) && (ip[-1] == matchLong[-1])) {
- ip--;
- matchLong--;
- mLength++;
- } /* catch up */
- offset_2 = offset_1;
- offset_1 = offset;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, offset + ZSTD_REP_MOVE, mLength - MINMATCH);
-
- } else if ((matchIndex > lowestIndex) && (ZSTD_read32(match) == ZSTD_read32(ip))) {
- size_t const h3 = ZSTD_hashPtr(ip + 1, hBitsL, 8);
- U32 const matchIndex3 = hashLong[h3];
- const BYTE *const match3Base = matchIndex3 < dictLimit ? dictBase : base;
- const BYTE *match3 = match3Base + matchIndex3;
- U32 offset;
- hashLong[h3] = curr + 1;
- if ((matchIndex3 > lowestIndex) && (ZSTD_read64(match3) == ZSTD_read64(ip + 1))) {
- const BYTE *matchEnd = matchIndex3 < dictLimit ? dictEnd : iend;
- const BYTE *lowMatchPtr = matchIndex3 < dictLimit ? dictStart : lowPrefixPtr;
- mLength = ZSTD_count_2segments(ip + 9, match3 + 8, iend, matchEnd, lowPrefixPtr) + 8;
- ip++;
- offset = curr + 1 - matchIndex3;
- while (((ip > anchor) & (match3 > lowMatchPtr)) && (ip[-1] == match3[-1])) {
- ip--;
- match3--;
- mLength++;
- } /* catch up */
- } else {
- const BYTE *matchEnd = matchIndex < dictLimit ? dictEnd : iend;
- const BYTE *lowMatchPtr = matchIndex < dictLimit ? dictStart : lowPrefixPtr;
- mLength = ZSTD_count_2segments(ip + 4, match + 4, iend, matchEnd, lowPrefixPtr) + 4;
- offset = curr - matchIndex;
- while (((ip > anchor) & (match > lowMatchPtr)) && (ip[-1] == match[-1])) {
- ip--;
- match--;
- mLength++;
- } /* catch up */
- }
- offset_2 = offset_1;
- offset_1 = offset;
- ZSTD_storeSeq(seqStorePtr, ip - anchor, anchor, offset + ZSTD_REP_MOVE, mLength - MINMATCH);
-
- } else {
- ip += ((ip - anchor) >> g_searchStrength) + 1;
- continue;
- }
- }
-
- /* found a match : store it */
- ip += mLength;
- anchor = ip;
-
- if (ip <= ilimit) {
- /* Fill Table */
- hashSmall[ZSTD_hashPtr(base + curr + 2, hBitsS, mls)] = curr + 2;
- hashLong[ZSTD_hashPtr(base + curr + 2, hBitsL, 8)] = curr + 2;
- hashSmall[ZSTD_hashPtr(ip - 2, hBitsS, mls)] = (U32)(ip - 2 - base);
- hashLong[ZSTD_hashPtr(ip - 2, hBitsL, 8)] = (U32)(ip - 2 - base);
- /* check immediate repcode */
- while (ip <= ilimit) {
- U32 const curr2 = (U32)(ip - base);
- U32 const repIndex2 = curr2 - offset_2;
- const BYTE *repMatch2 = repIndex2 < dictLimit ? dictBase + repIndex2 : base + repIndex2;
- if ((((U32)((dictLimit - 1) - repIndex2) >= 3) & (repIndex2 > lowestIndex)) /* intentional overflow */
- && (ZSTD_read32(repMatch2) == ZSTD_read32(ip))) {
- const BYTE *const repEnd2 = repIndex2 < dictLimit ? dictEnd : iend;
- size_t const repLength2 =
- ZSTD_count_2segments(ip + EQUAL_READ32, repMatch2 + EQUAL_READ32, iend, repEnd2, lowPrefixPtr) + EQUAL_READ32;
- U32 tmpOffset = offset_2;
- offset_2 = offset_1;
- offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, repLength2 - MINMATCH);
- hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = curr2;
- hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = curr2;
- ip += repLength2;
- anchor = ip;
- continue;
- }
- break;
- }
- }
- }
-
- /* save reps for next block */
- ctx->repToConfirm[0] = offset_1;
- ctx->repToConfirm[1] = offset_2;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-static void ZSTD_compressBlock_doubleFast_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- U32 const mls = ctx->params.cParams.searchLength;
- switch (mls) {
- default: /* includes case 3 */
- case 4: ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 4); return;
- case 5: ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 5); return;
- case 6: ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 6); return;
- case 7: ZSTD_compressBlock_doubleFast_extDict_generic(ctx, src, srcSize, 7); return;
- }
-}
-
-/*-*************************************
-* Binary Tree search
-***************************************/
-/** ZSTD_insertBt1() : add one or multiple positions to tree.
-* ip : assumed <= iend-8 .
-* @return : nb of positions added */
-static U32 ZSTD_insertBt1(ZSTD_CCtx *zc, const BYTE *const ip, const U32 mls, const BYTE *const iend, U32 nbCompares, U32 extDict)
-{
- U32 *const hashTable = zc->hashTable;
- U32 const hashLog = zc->params.cParams.hashLog;
- size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
- U32 *const bt = zc->chainTable;
- U32 const btLog = zc->params.cParams.chainLog - 1;
- U32 const btMask = (1 << btLog) - 1;
- U32 matchIndex = hashTable[h];
- size_t commonLengthSmaller = 0, commonLengthLarger = 0;
- const BYTE *const base = zc->base;
- const BYTE *const dictBase = zc->dictBase;
- const U32 dictLimit = zc->dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const BYTE *match;
- const U32 curr = (U32)(ip - base);
- const U32 btLow = btMask >= curr ? 0 : curr - btMask;
- U32 *smallerPtr = bt + 2 * (curr & btMask);
- U32 *largerPtr = smallerPtr + 1;
- U32 dummy32; /* to be nullified at the end */
- U32 const windowLow = zc->lowLimit;
- U32 matchEndIdx = curr + 8;
- size_t bestLength = 8;
-
- hashTable[h] = curr; /* Update Hash Table */
-
- while (nbCompares-- && (matchIndex > windowLow)) {
- U32 *const nextPtr = bt + 2 * (matchIndex & btMask);
- size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
-
- if ((!extDict) || (matchIndex + matchLength >= dictLimit)) {
- match = base + matchIndex;
- if (match[matchLength] == ip[matchLength])
- matchLength += ZSTD_count(ip + matchLength + 1, match + matchLength + 1, iend) + 1;
- } else {
- match = dictBase + matchIndex;
- matchLength += ZSTD_count_2segments(ip + matchLength, match + matchLength, iend, dictEnd, prefixStart);
- if (matchIndex + matchLength >= dictLimit)
- match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
- }
-
- if (matchLength > bestLength) {
- bestLength = matchLength;
- if (matchLength > matchEndIdx - matchIndex)
- matchEndIdx = matchIndex + (U32)matchLength;
- }
-
- if (ip + matchLength == iend) /* equal : no way to know if inf or sup */
- break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt the tree */
-
- if (match[matchLength] < ip[matchLength]) { /* necessarily within correct buffer */
- /* match is smaller than curr */
- *smallerPtr = matchIndex; /* update smaller idx */
- commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
- if (matchIndex <= btLow) {
- smallerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- smallerPtr = nextPtr + 1; /* new "smaller" => larger of match */
- matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to curr) */
- } else {
- /* match is larger than curr */
- *largerPtr = matchIndex;
- commonLengthLarger = matchLength;
- if (matchIndex <= btLow) {
- largerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- largerPtr = nextPtr;
- matchIndex = nextPtr[0];
- }
- }
-
- *smallerPtr = *largerPtr = 0;
- if (bestLength > 384)
- return MIN(192, (U32)(bestLength - 384)); /* speed optimization */
- if (matchEndIdx > curr + 8)
- return matchEndIdx - curr - 8;
- return 1;
-}
-
-static size_t ZSTD_insertBtAndFindBestMatch(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iend, size_t *offsetPtr, U32 nbCompares, const U32 mls,
- U32 extDict)
-{
- U32 *const hashTable = zc->hashTable;
- U32 const hashLog = zc->params.cParams.hashLog;
- size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
- U32 *const bt = zc->chainTable;
- U32 const btLog = zc->params.cParams.chainLog - 1;
- U32 const btMask = (1 << btLog) - 1;
- U32 matchIndex = hashTable[h];
- size_t commonLengthSmaller = 0, commonLengthLarger = 0;
- const BYTE *const base = zc->base;
- const BYTE *const dictBase = zc->dictBase;
- const U32 dictLimit = zc->dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const U32 curr = (U32)(ip - base);
- const U32 btLow = btMask >= curr ? 0 : curr - btMask;
- const U32 windowLow = zc->lowLimit;
- U32 *smallerPtr = bt + 2 * (curr & btMask);
- U32 *largerPtr = bt + 2 * (curr & btMask) + 1;
- U32 matchEndIdx = curr + 8;
- U32 dummy32; /* to be nullified at the end */
- size_t bestLength = 0;
-
- hashTable[h] = curr; /* Update Hash Table */
-
- while (nbCompares-- && (matchIndex > windowLow)) {
- U32 *const nextPtr = bt + 2 * (matchIndex & btMask);
- size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
- const BYTE *match;
-
- if ((!extDict) || (matchIndex + matchLength >= dictLimit)) {
- match = base + matchIndex;
- if (match[matchLength] == ip[matchLength])
- matchLength += ZSTD_count(ip + matchLength + 1, match + matchLength + 1, iend) + 1;
- } else {
- match = dictBase + matchIndex;
- matchLength += ZSTD_count_2segments(ip + matchLength, match + matchLength, iend, dictEnd, prefixStart);
- if (matchIndex + matchLength >= dictLimit)
- match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
- }
-
- if (matchLength > bestLength) {
- if (matchLength > matchEndIdx - matchIndex)
- matchEndIdx = matchIndex + (U32)matchLength;
- if ((4 * (int)(matchLength - bestLength)) > (int)(ZSTD_highbit32(curr - matchIndex + 1) - ZSTD_highbit32((U32)offsetPtr[0] + 1)))
- bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + curr - matchIndex;
- if (ip + matchLength == iend) /* equal : no way to know if inf or sup */
- break; /* drop, to guarantee consistency (miss a little bit of compression) */
- }
-
- if (match[matchLength] < ip[matchLength]) {
- /* match is smaller than curr */
- *smallerPtr = matchIndex; /* update smaller idx */
- commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
- if (matchIndex <= btLow) {
- smallerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- smallerPtr = nextPtr + 1; /* new "smaller" => larger of match */
- matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to curr) */
- } else {
- /* match is larger than curr */
- *largerPtr = matchIndex;
- commonLengthLarger = matchLength;
- if (matchIndex <= btLow) {
- largerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- largerPtr = nextPtr;
- matchIndex = nextPtr[0];
- }
- }
-
- *smallerPtr = *largerPtr = 0;
-
- zc->nextToUpdate = (matchEndIdx > curr + 8) ? matchEndIdx - 8 : curr + 1;
- return bestLength;
-}
-
-static void ZSTD_updateTree(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iend, const U32 nbCompares, const U32 mls)
-{
- const BYTE *const base = zc->base;
- const U32 target = (U32)(ip - base);
- U32 idx = zc->nextToUpdate;
-
- while (idx < target)
- idx += ZSTD_insertBt1(zc, base + idx, mls, iend, nbCompares, 0);
-}
-
-/** ZSTD_BtFindBestMatch() : Tree updater, providing best match */
-static size_t ZSTD_BtFindBestMatch(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts, const U32 mls)
-{
- if (ip < zc->base + zc->nextToUpdate)
- return 0; /* skipped area */
- ZSTD_updateTree(zc, ip, iLimit, maxNbAttempts, mls);
- return ZSTD_insertBtAndFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, mls, 0);
-}
-
-static size_t ZSTD_BtFindBestMatch_selectMLS(ZSTD_CCtx *zc, /* Index table will be updated */
- const BYTE *ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts, const U32 matchLengthSearch)
-{
- switch (matchLengthSearch) {
- default: /* includes case 3 */
- case 4: return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4);
- case 5: return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5);
- case 7:
- case 6: return ZSTD_BtFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6);
- }
-}
-
-static void ZSTD_updateTree_extDict(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iend, const U32 nbCompares, const U32 mls)
-{
- const BYTE *const base = zc->base;
- const U32 target = (U32)(ip - base);
- U32 idx = zc->nextToUpdate;
-
- while (idx < target)
- idx += ZSTD_insertBt1(zc, base + idx, mls, iend, nbCompares, 1);
-}
-
-/** Tree updater, providing best match */
-static size_t ZSTD_BtFindBestMatch_extDict(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts,
- const U32 mls)
-{
- if (ip < zc->base + zc->nextToUpdate)
- return 0; /* skipped area */
- ZSTD_updateTree_extDict(zc, ip, iLimit, maxNbAttempts, mls);
- return ZSTD_insertBtAndFindBestMatch(zc, ip, iLimit, offsetPtr, maxNbAttempts, mls, 1);
-}
-
-static size_t ZSTD_BtFindBestMatch_selectMLS_extDict(ZSTD_CCtx *zc, /* Index table will be updated */
- const BYTE *ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts,
- const U32 matchLengthSearch)
-{
- switch (matchLengthSearch) {
- default: /* includes case 3 */
- case 4: return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4);
- case 5: return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5);
- case 7:
- case 6: return ZSTD_BtFindBestMatch_extDict(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6);
- }
-}
-
-/* *********************************
-* Hash Chain
-***********************************/
-#define NEXT_IN_CHAIN(d, mask) chainTable[(d)&mask]
-
-/* Update chains up to ip (excluded)
- Assumption : always within prefix (i.e. not within extDict) */
-FORCE_INLINE
-U32 ZSTD_insertAndFindFirstIndex(ZSTD_CCtx *zc, const BYTE *ip, U32 mls)
-{
- U32 *const hashTable = zc->hashTable;
- const U32 hashLog = zc->params.cParams.hashLog;
- U32 *const chainTable = zc->chainTable;
- const U32 chainMask = (1 << zc->params.cParams.chainLog) - 1;
- const BYTE *const base = zc->base;
- const U32 target = (U32)(ip - base);
- U32 idx = zc->nextToUpdate;
-
- while (idx < target) { /* catch up */
- size_t const h = ZSTD_hashPtr(base + idx, hashLog, mls);
- NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
- hashTable[h] = idx;
- idx++;
- }
-
- zc->nextToUpdate = target;
- return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
-}
-
-/* inlining is important to hardwire a hot branch (template emulation) */
-FORCE_INLINE
-size_t ZSTD_HcFindBestMatch_generic(ZSTD_CCtx *zc, /* Index table will be updated */
- const BYTE *const ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts, const U32 mls,
- const U32 extDict)
-{
- U32 *const chainTable = zc->chainTable;
- const U32 chainSize = (1 << zc->params.cParams.chainLog);
- const U32 chainMask = chainSize - 1;
- const BYTE *const base = zc->base;
- const BYTE *const dictBase = zc->dictBase;
- const U32 dictLimit = zc->dictLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const U32 lowLimit = zc->lowLimit;
- const U32 curr = (U32)(ip - base);
- const U32 minChain = curr > chainSize ? curr - chainSize : 0;
- int nbAttempts = maxNbAttempts;
- size_t ml = EQUAL_READ32 - 1;
-
- /* HC4 match finder */
- U32 matchIndex = ZSTD_insertAndFindFirstIndex(zc, ip, mls);
-
- for (; (matchIndex > lowLimit) & (nbAttempts > 0); nbAttempts--) {
- const BYTE *match;
- size_t currMl = 0;
- if ((!extDict) || matchIndex >= dictLimit) {
- match = base + matchIndex;
- if (match[ml] == ip[ml]) /* potentially better */
- currMl = ZSTD_count(ip, match, iLimit);
- } else {
- match = dictBase + matchIndex;
- if (ZSTD_read32(match) == ZSTD_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
- currMl = ZSTD_count_2segments(ip + EQUAL_READ32, match + EQUAL_READ32, iLimit, dictEnd, prefixStart) + EQUAL_READ32;
- }
-
- /* save best solution */
- if (currMl > ml) {
- ml = currMl;
- *offsetPtr = curr - matchIndex + ZSTD_REP_MOVE;
- if (ip + currMl == iLimit)
- break; /* best possible, and avoid read overflow*/
- }
-
- if (matchIndex <= minChain)
- break;
- matchIndex = NEXT_IN_CHAIN(matchIndex, chainMask);
- }
-
- return ml;
-}
-
-FORCE_INLINE size_t ZSTD_HcFindBestMatch_selectMLS(ZSTD_CCtx *zc, const BYTE *ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts,
- const U32 matchLengthSearch)
-{
- switch (matchLengthSearch) {
- default: /* includes case 3 */
- case 4: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4, 0);
- case 5: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5, 0);
- case 7:
- case 6: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6, 0);
- }
-}
-
-FORCE_INLINE size_t ZSTD_HcFindBestMatch_extDict_selectMLS(ZSTD_CCtx *zc, const BYTE *ip, const BYTE *const iLimit, size_t *offsetPtr, const U32 maxNbAttempts,
- const U32 matchLengthSearch)
-{
- switch (matchLengthSearch) {
- default: /* includes case 3 */
- case 4: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 4, 1);
- case 5: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 5, 1);
- case 7:
- case 6: return ZSTD_HcFindBestMatch_generic(zc, ip, iLimit, offsetPtr, maxNbAttempts, 6, 1);
- }
-}
-
-/* *******************************
-* Common parser - lazy strategy
-*********************************/
-FORCE_INLINE
-void ZSTD_compressBlock_lazy_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const U32 searchMethod, const U32 depth)
-{
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- const BYTE *const base = ctx->base + ctx->dictLimit;
-
- U32 const maxSearches = 1 << ctx->params.cParams.searchLog;
- U32 const mls = ctx->params.cParams.searchLength;
-
- typedef size_t (*searchMax_f)(ZSTD_CCtx * zc, const BYTE *ip, const BYTE *iLimit, size_t *offsetPtr, U32 maxNbAttempts, U32 matchLengthSearch);
- searchMax_f const searchMax = searchMethod ? ZSTD_BtFindBestMatch_selectMLS : ZSTD_HcFindBestMatch_selectMLS;
- U32 offset_1 = ctx->rep[0], offset_2 = ctx->rep[1], savedOffset = 0;
-
- /* init */
- ip += (ip == base);
- ctx->nextToUpdate3 = ctx->nextToUpdate;
- {
- U32 const maxRep = (U32)(ip - base);
- if (offset_2 > maxRep)
- savedOffset = offset_2, offset_2 = 0;
- if (offset_1 > maxRep)
- savedOffset = offset_1, offset_1 = 0;
- }
-
- /* Match Loop */
- while (ip < ilimit) {
- size_t matchLength = 0;
- size_t offset = 0;
- const BYTE *start = ip + 1;
-
- /* check repCode */
- if ((offset_1 > 0) & (ZSTD_read32(ip + 1) == ZSTD_read32(ip + 1 - offset_1))) {
- /* repcode : we take it */
- matchLength = ZSTD_count(ip + 1 + EQUAL_READ32, ip + 1 + EQUAL_READ32 - offset_1, iend) + EQUAL_READ32;
- if (depth == 0)
- goto _storeSequence;
- }
-
- /* first search (depth 0) */
- {
- size_t offsetFound = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offsetFound, maxSearches, mls);
- if (ml2 > matchLength)
- matchLength = ml2, start = ip, offset = offsetFound;
- }
-
- if (matchLength < EQUAL_READ32) {
- ip += ((ip - anchor) >> g_searchStrength) + 1; /* jump faster over incompressible sections */
- continue;
- }
-
- /* let's try to find a better solution */
- if (depth >= 1)
- while (ip < ilimit) {
- ip++;
- if ((offset) && ((offset_1 > 0) & (ZSTD_read32(ip) == ZSTD_read32(ip - offset_1)))) {
- size_t const mlRep = ZSTD_count(ip + EQUAL_READ32, ip + EQUAL_READ32 - offset_1, iend) + EQUAL_READ32;
- int const gain2 = (int)(mlRep * 3);
- int const gain1 = (int)(matchLength * 3 - ZSTD_highbit32((U32)offset + 1) + 1);
- if ((mlRep >= EQUAL_READ32) && (gain2 > gain1))
- matchLength = mlRep, offset = 0, start = ip;
- }
- {
- size_t offset2 = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
- int const gain2 = (int)(ml2 * 4 - ZSTD_highbit32((U32)offset2 + 1)); /* raw approx */
- int const gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 4);
- if ((ml2 >= EQUAL_READ32) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
- continue; /* search a better one */
- }
- }
-
- /* let's find an even better one */
- if ((depth == 2) && (ip < ilimit)) {
- ip++;
- if ((offset) && ((offset_1 > 0) & (ZSTD_read32(ip) == ZSTD_read32(ip - offset_1)))) {
- size_t const ml2 = ZSTD_count(ip + EQUAL_READ32, ip + EQUAL_READ32 - offset_1, iend) + EQUAL_READ32;
- int const gain2 = (int)(ml2 * 4);
- int const gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 1);
- if ((ml2 >= EQUAL_READ32) && (gain2 > gain1))
- matchLength = ml2, offset = 0, start = ip;
- }
- {
- size_t offset2 = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
- int const gain2 = (int)(ml2 * 4 - ZSTD_highbit32((U32)offset2 + 1)); /* raw approx */
- int const gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 7);
- if ((ml2 >= EQUAL_READ32) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
- continue;
- }
- }
- }
- break; /* nothing found : store previous solution */
- }
-
- /* NOTE:
- * start[-offset+ZSTD_REP_MOVE-1] is undefined behavior.
- * (-offset+ZSTD_REP_MOVE-1) is unsigned, and is added to start, which
- * overflows the pointer, which is undefined behavior.
- */
- /* catch up */
- if (offset) {
- while ((start > anchor) && (start > base + offset - ZSTD_REP_MOVE) &&
- (start[-1] == (start-offset+ZSTD_REP_MOVE)[-1])) /* only search for offset within prefix */
- {
- start--;
- matchLength++;
- }
- offset_2 = offset_1;
- offset_1 = (U32)(offset - ZSTD_REP_MOVE);
- }
-
- /* store sequence */
-_storeSequence:
- {
- size_t const litLength = start - anchor;
- ZSTD_storeSeq(seqStorePtr, litLength, anchor, (U32)offset, matchLength - MINMATCH);
- anchor = ip = start + matchLength;
- }
-
- /* check immediate repcode */
- while ((ip <= ilimit) && ((offset_2 > 0) & (ZSTD_read32(ip) == ZSTD_read32(ip - offset_2)))) {
- /* store sequence */
- matchLength = ZSTD_count(ip + EQUAL_READ32, ip + EQUAL_READ32 - offset_2, iend) + EQUAL_READ32;
- offset = offset_2;
- offset_2 = offset_1;
- offset_1 = (U32)offset; /* swap repcodes */
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, matchLength - MINMATCH);
- ip += matchLength;
- anchor = ip;
- continue; /* faster when present ... (?) */
- }
- }
-
- /* Save reps for next block */
- ctx->repToConfirm[0] = offset_1 ? offset_1 : savedOffset;
- ctx->repToConfirm[1] = offset_2 ? offset_2 : savedOffset;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-static void ZSTD_compressBlock_btlazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 1, 2); }
-
-static void ZSTD_compressBlock_lazy2(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 2); }
-
-static void ZSTD_compressBlock_lazy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 1); }
-
-static void ZSTD_compressBlock_greedy(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { ZSTD_compressBlock_lazy_generic(ctx, src, srcSize, 0, 0); }
-
-FORCE_INLINE
-void ZSTD_compressBlock_lazy_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const U32 searchMethod, const U32 depth)
-{
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- const BYTE *const base = ctx->base;
- const U32 dictLimit = ctx->dictLimit;
- const U32 lowestIndex = ctx->lowLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const BYTE *const dictBase = ctx->dictBase;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const dictStart = dictBase + ctx->lowLimit;
-
- const U32 maxSearches = 1 << ctx->params.cParams.searchLog;
- const U32 mls = ctx->params.cParams.searchLength;
-
- typedef size_t (*searchMax_f)(ZSTD_CCtx * zc, const BYTE *ip, const BYTE *iLimit, size_t *offsetPtr, U32 maxNbAttempts, U32 matchLengthSearch);
- searchMax_f searchMax = searchMethod ? ZSTD_BtFindBestMatch_selectMLS_extDict : ZSTD_HcFindBestMatch_extDict_selectMLS;
-
- U32 offset_1 = ctx->rep[0], offset_2 = ctx->rep[1];
-
- /* init */
- ctx->nextToUpdate3 = ctx->nextToUpdate;
- ip += (ip == prefixStart);
-
- /* Match Loop */
- while (ip < ilimit) {
- size_t matchLength = 0;
- size_t offset = 0;
- const BYTE *start = ip + 1;
- U32 curr = (U32)(ip - base);
-
- /* check repCode */
- {
- const U32 repIndex = (U32)(curr + 1 - offset_1);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- if (ZSTD_read32(ip + 1) == ZSTD_read32(repMatch)) {
- /* repcode detected we should take it */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- matchLength =
- ZSTD_count_2segments(ip + 1 + EQUAL_READ32, repMatch + EQUAL_READ32, iend, repEnd, prefixStart) + EQUAL_READ32;
- if (depth == 0)
- goto _storeSequence;
- }
- }
-
- /* first search (depth 0) */
- {
- size_t offsetFound = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offsetFound, maxSearches, mls);
- if (ml2 > matchLength)
- matchLength = ml2, start = ip, offset = offsetFound;
- }
-
- if (matchLength < EQUAL_READ32) {
- ip += ((ip - anchor) >> g_searchStrength) + 1; /* jump faster over incompressible sections */
- continue;
- }
-
- /* let's try to find a better solution */
- if (depth >= 1)
- while (ip < ilimit) {
- ip++;
- curr++;
- /* check repCode */
- if (offset) {
- const U32 repIndex = (U32)(curr - offset_1);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- if (ZSTD_read32(ip) == ZSTD_read32(repMatch)) {
- /* repcode detected */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- size_t const repLength =
- ZSTD_count_2segments(ip + EQUAL_READ32, repMatch + EQUAL_READ32, iend, repEnd, prefixStart) +
- EQUAL_READ32;
- int const gain2 = (int)(repLength * 3);
- int const gain1 = (int)(matchLength * 3 - ZSTD_highbit32((U32)offset + 1) + 1);
- if ((repLength >= EQUAL_READ32) && (gain2 > gain1))
- matchLength = repLength, offset = 0, start = ip;
- }
- }
-
- /* search match, depth 1 */
- {
- size_t offset2 = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
- int const gain2 = (int)(ml2 * 4 - ZSTD_highbit32((U32)offset2 + 1)); /* raw approx */
- int const gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 4);
- if ((ml2 >= EQUAL_READ32) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
- continue; /* search a better one */
- }
- }
-
- /* let's find an even better one */
- if ((depth == 2) && (ip < ilimit)) {
- ip++;
- curr++;
- /* check repCode */
- if (offset) {
- const U32 repIndex = (U32)(curr - offset_1);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- if (ZSTD_read32(ip) == ZSTD_read32(repMatch)) {
- /* repcode detected */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- size_t repLength = ZSTD_count_2segments(ip + EQUAL_READ32, repMatch + EQUAL_READ32, iend,
- repEnd, prefixStart) +
- EQUAL_READ32;
- int gain2 = (int)(repLength * 4);
- int gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 1);
- if ((repLength >= EQUAL_READ32) && (gain2 > gain1))
- matchLength = repLength, offset = 0, start = ip;
- }
- }
-
- /* search match, depth 2 */
- {
- size_t offset2 = 99999999;
- size_t const ml2 = searchMax(ctx, ip, iend, &offset2, maxSearches, mls);
- int const gain2 = (int)(ml2 * 4 - ZSTD_highbit32((U32)offset2 + 1)); /* raw approx */
- int const gain1 = (int)(matchLength * 4 - ZSTD_highbit32((U32)offset + 1) + 7);
- if ((ml2 >= EQUAL_READ32) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
- continue;
- }
- }
- }
- break; /* nothing found : store previous solution */
- }
-
- /* catch up */
- if (offset) {
- U32 const matchIndex = (U32)((start - base) - (offset - ZSTD_REP_MOVE));
- const BYTE *match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex;
- const BYTE *const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
- while ((start > anchor) && (match > mStart) && (start[-1] == match[-1])) {
- start--;
- match--;
- matchLength++;
- } /* catch up */
- offset_2 = offset_1;
- offset_1 = (U32)(offset - ZSTD_REP_MOVE);
- }
-
- /* store sequence */
- _storeSequence : {
- size_t const litLength = start - anchor;
- ZSTD_storeSeq(seqStorePtr, litLength, anchor, (U32)offset, matchLength - MINMATCH);
- anchor = ip = start + matchLength;
- }
-
- /* check immediate repcode */
- while (ip <= ilimit) {
- const U32 repIndex = (U32)((ip - base) - offset_2);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- if (ZSTD_read32(ip) == ZSTD_read32(repMatch)) {
- /* repcode detected we should take it */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- matchLength =
- ZSTD_count_2segments(ip + EQUAL_READ32, repMatch + EQUAL_READ32, iend, repEnd, prefixStart) + EQUAL_READ32;
- offset = offset_2;
- offset_2 = offset_1;
- offset_1 = (U32)offset; /* swap offset history */
- ZSTD_storeSeq(seqStorePtr, 0, anchor, 0, matchLength - MINMATCH);
- ip += matchLength;
- anchor = ip;
- continue; /* faster when present ... (?) */
- }
- break;
- }
- }
-
- /* Save reps for next block */
- ctx->repToConfirm[0] = offset_1;
- ctx->repToConfirm[1] = offset_2;
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-void ZSTD_compressBlock_greedy_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize) { ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 0); }
-
-static void ZSTD_compressBlock_lazy_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 1);
-}
-
-static void ZSTD_compressBlock_lazy2_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 0, 2);
-}
-
-static void ZSTD_compressBlock_btlazy2_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
- ZSTD_compressBlock_lazy_extDict_generic(ctx, src, srcSize, 1, 2);
-}
-
-/* The optimal parser */
-#include "zstd_opt.h"
-
-static void ZSTD_compressBlock_btopt(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
-#ifdef ZSTD_OPT_H_91842398743
- ZSTD_compressBlock_opt_generic(ctx, src, srcSize, 0);
-#else
- (void)ctx;
- (void)src;
- (void)srcSize;
- return;
-#endif
-}
-
-static void ZSTD_compressBlock_btopt2(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
-#ifdef ZSTD_OPT_H_91842398743
- ZSTD_compressBlock_opt_generic(ctx, src, srcSize, 1);
-#else
- (void)ctx;
- (void)src;
- (void)srcSize;
- return;
-#endif
-}
-
-static void ZSTD_compressBlock_btopt_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
-#ifdef ZSTD_OPT_H_91842398743
- ZSTD_compressBlock_opt_extDict_generic(ctx, src, srcSize, 0);
-#else
- (void)ctx;
- (void)src;
- (void)srcSize;
- return;
-#endif
-}
-
-static void ZSTD_compressBlock_btopt2_extDict(ZSTD_CCtx *ctx, const void *src, size_t srcSize)
-{
-#ifdef ZSTD_OPT_H_91842398743
- ZSTD_compressBlock_opt_extDict_generic(ctx, src, srcSize, 1);
-#else
- (void)ctx;
- (void)src;
- (void)srcSize;
- return;
-#endif
-}
-
-typedef void (*ZSTD_blockCompressor)(ZSTD_CCtx *ctx, const void *src, size_t srcSize);
-
-static ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, int extDict)
-{
- static const ZSTD_blockCompressor blockCompressor[2][8] = {
- {ZSTD_compressBlock_fast, ZSTD_compressBlock_doubleFast, ZSTD_compressBlock_greedy, ZSTD_compressBlock_lazy, ZSTD_compressBlock_lazy2,
- ZSTD_compressBlock_btlazy2, ZSTD_compressBlock_btopt, ZSTD_compressBlock_btopt2},
- {ZSTD_compressBlock_fast_extDict, ZSTD_compressBlock_doubleFast_extDict, ZSTD_compressBlock_greedy_extDict, ZSTD_compressBlock_lazy_extDict,
- ZSTD_compressBlock_lazy2_extDict, ZSTD_compressBlock_btlazy2_extDict, ZSTD_compressBlock_btopt_extDict, ZSTD_compressBlock_btopt2_extDict}};
-
- return blockCompressor[extDict][(U32)strat];
-}
-
-static size_t ZSTD_compressBlock_internal(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->params.cParams.strategy, zc->lowLimit < zc->dictLimit);
- const BYTE *const base = zc->base;
- const BYTE *const istart = (const BYTE *)src;
- const U32 curr = (U32)(istart - base);
- if (srcSize < MIN_CBLOCK_SIZE + ZSTD_blockHeaderSize + 1)
- return 0; /* don't even attempt compression below a certain srcSize */
- ZSTD_resetSeqStore(&(zc->seqStore));
- if (curr > zc->nextToUpdate + 384)
- zc->nextToUpdate = curr - MIN(192, (U32)(curr - zc->nextToUpdate - 384)); /* update tree not updated after finding very long rep matches */
- blockCompressor(zc, src, srcSize);
- return ZSTD_compressSequences(zc, dst, dstCapacity, srcSize);
-}
-
-/*! ZSTD_compress_generic() :
-* Compress a chunk of data into one or multiple blocks.
-* All blocks will be terminated, all input will be consumed.
-* Function will issue an error if there is not enough `dstCapacity` to hold the compressed content.
-* Frame is supposed already started (header already produced)
-* @return : compressed size, or an error code
-*/
-static size_t ZSTD_compress_generic(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, U32 lastFrameChunk)
-{
- size_t blockSize = cctx->blockSize;
- size_t remaining = srcSize;
- const BYTE *ip = (const BYTE *)src;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *op = ostart;
- U32 const maxDist = 1 << cctx->params.cParams.windowLog;
-
- if (cctx->params.fParams.checksumFlag && srcSize)
- xxh64_update(&cctx->xxhState, src, srcSize);
-
- while (remaining) {
- U32 const lastBlock = lastFrameChunk & (blockSize >= remaining);
- size_t cSize;
-
- if (dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE)
- return ERROR(dstSize_tooSmall); /* not enough space to store compressed block */
- if (remaining < blockSize)
- blockSize = remaining;
-
- /* preemptive overflow correction */
- if (cctx->lowLimit > (3U << 29)) {
- U32 const cycleMask = (1 << ZSTD_cycleLog(cctx->params.cParams.hashLog, cctx->params.cParams.strategy)) - 1;
- U32 const curr = (U32)(ip - cctx->base);
- U32 const newCurr = (curr & cycleMask) + (1 << cctx->params.cParams.windowLog);
- U32 const correction = curr - newCurr;
- ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_64 <= 30);
- ZSTD_reduceIndex(cctx, correction);
- cctx->base += correction;
- cctx->dictBase += correction;
- cctx->lowLimit -= correction;
- cctx->dictLimit -= correction;
- if (cctx->nextToUpdate < correction)
- cctx->nextToUpdate = 0;
- else
- cctx->nextToUpdate -= correction;
- }
-
- if ((U32)(ip + blockSize - cctx->base) > cctx->loadedDictEnd + maxDist) {
- /* enforce maxDist */
- U32 const newLowLimit = (U32)(ip + blockSize - cctx->base) - maxDist;
- if (cctx->lowLimit < newLowLimit)
- cctx->lowLimit = newLowLimit;
- if (cctx->dictLimit < cctx->lowLimit)
- cctx->dictLimit = cctx->lowLimit;
- }
-
- cSize = ZSTD_compressBlock_internal(cctx, op + ZSTD_blockHeaderSize, dstCapacity - ZSTD_blockHeaderSize, ip, blockSize);
- if (ZSTD_isError(cSize))
- return cSize;
-
- if (cSize == 0) { /* block is not compressible */
- U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw) << 1) + (U32)(blockSize << 3);
- if (blockSize + ZSTD_blockHeaderSize > dstCapacity)
- return ERROR(dstSize_tooSmall);
- ZSTD_writeLE32(op, cBlockHeader24); /* no pb, 4th byte will be overwritten */
- memcpy(op + ZSTD_blockHeaderSize, ip, blockSize);
- cSize = ZSTD_blockHeaderSize + blockSize;
- } else {
- U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed) << 1) + (U32)(cSize << 3);
- ZSTD_writeLE24(op, cBlockHeader24);
- cSize += ZSTD_blockHeaderSize;
- }
-
- remaining -= blockSize;
- dstCapacity -= cSize;
- ip += blockSize;
- op += cSize;
- }
-
- if (lastFrameChunk && (op > ostart))
- cctx->stage = ZSTDcs_ending;
- return op - ostart;
-}
-
-static size_t ZSTD_writeFrameHeader(void *dst, size_t dstCapacity, ZSTD_parameters params, U64 pledgedSrcSize, U32 dictID)
-{
- BYTE *const op = (BYTE *)dst;
- U32 const dictIDSizeCode = (dictID > 0) + (dictID >= 256) + (dictID >= 65536); /* 0-3 */
- U32 const checksumFlag = params.fParams.checksumFlag > 0;
- U32 const windowSize = 1U << params.cParams.windowLog;
- U32 const singleSegment = params.fParams.contentSizeFlag && (windowSize >= pledgedSrcSize);
- BYTE const windowLogByte = (BYTE)((params.cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3);
- U32 const fcsCode =
- params.fParams.contentSizeFlag ? (pledgedSrcSize >= 256) + (pledgedSrcSize >= 65536 + 256) + (pledgedSrcSize >= 0xFFFFFFFFU) : 0; /* 0-3 */
- BYTE const frameHeaderDecriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag << 2) + (singleSegment << 5) + (fcsCode << 6));
- size_t pos;
-
- if (dstCapacity < ZSTD_frameHeaderSize_max)
- return ERROR(dstSize_tooSmall);
-
- ZSTD_writeLE32(dst, ZSTD_MAGICNUMBER);
- op[4] = frameHeaderDecriptionByte;
- pos = 5;
- if (!singleSegment)
- op[pos++] = windowLogByte;
- switch (dictIDSizeCode) {
- default: /* impossible */
- case 0: break;
- case 1:
- op[pos] = (BYTE)(dictID);
- pos++;
- break;
- case 2:
- ZSTD_writeLE16(op + pos, (U16)dictID);
- pos += 2;
- break;
- case 3:
- ZSTD_writeLE32(op + pos, dictID);
- pos += 4;
- break;
- }
- switch (fcsCode) {
- default: /* impossible */
- case 0:
- if (singleSegment)
- op[pos++] = (BYTE)(pledgedSrcSize);
- break;
- case 1:
- ZSTD_writeLE16(op + pos, (U16)(pledgedSrcSize - 256));
- pos += 2;
- break;
- case 2:
- ZSTD_writeLE32(op + pos, (U32)(pledgedSrcSize));
- pos += 4;
- break;
- case 3:
- ZSTD_writeLE64(op + pos, (U64)(pledgedSrcSize));
- pos += 8;
- break;
- }
- return pos;
-}
-
-static size_t ZSTD_compressContinue_internal(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, U32 frame, U32 lastFrameChunk)
-{
- const BYTE *const ip = (const BYTE *)src;
- size_t fhSize = 0;
-
- if (cctx->stage == ZSTDcs_created)
- return ERROR(stage_wrong); /* missing init (ZSTD_compressBegin) */
-
- if (frame && (cctx->stage == ZSTDcs_init)) {
- fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, cctx->params, cctx->frameContentSize, cctx->dictID);
- if (ZSTD_isError(fhSize))
- return fhSize;
- dstCapacity -= fhSize;
- dst = (char *)dst + fhSize;
- cctx->stage = ZSTDcs_ongoing;
- }
-
- /* Check if blocks follow each other */
- if (src != cctx->nextSrc) {
- /* not contiguous */
- ptrdiff_t const delta = cctx->nextSrc - ip;
- cctx->lowLimit = cctx->dictLimit;
- cctx->dictLimit = (U32)(cctx->nextSrc - cctx->base);
- cctx->dictBase = cctx->base;
- cctx->base -= delta;
- cctx->nextToUpdate = cctx->dictLimit;
- if (cctx->dictLimit - cctx->lowLimit < HASH_READ_SIZE)
- cctx->lowLimit = cctx->dictLimit; /* too small extDict */
- }
-
- /* if input and dictionary overlap : reduce dictionary (area presumed modified by input) */
- if ((ip + srcSize > cctx->dictBase + cctx->lowLimit) & (ip < cctx->dictBase + cctx->dictLimit)) {
- ptrdiff_t const highInputIdx = (ip + srcSize) - cctx->dictBase;
- U32 const lowLimitMax = (highInputIdx > (ptrdiff_t)cctx->dictLimit) ? cctx->dictLimit : (U32)highInputIdx;
- cctx->lowLimit = lowLimitMax;
- }
-
- cctx->nextSrc = ip + srcSize;
-
- if (srcSize) {
- size_t const cSize = frame ? ZSTD_compress_generic(cctx, dst, dstCapacity, src, srcSize, lastFrameChunk)
- : ZSTD_compressBlock_internal(cctx, dst, dstCapacity, src, srcSize);
- if (ZSTD_isError(cSize))
- return cSize;
- return cSize + fhSize;
- } else
- return fhSize;
-}
-
-size_t ZSTD_compressContinue(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1, 0);
-}
-
-size_t ZSTD_getBlockSizeMax(ZSTD_CCtx *cctx) { return MIN(ZSTD_BLOCKSIZE_ABSOLUTEMAX, 1 << cctx->params.cParams.windowLog); }
-
-size_t ZSTD_compressBlock(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t const blockSizeMax = ZSTD_getBlockSizeMax(cctx);
- if (srcSize > blockSizeMax)
- return ERROR(srcSize_wrong);
- return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0, 0);
-}
-
-/*! ZSTD_loadDictionaryContent() :
- * @return : 0, or an error code
- */
-static size_t ZSTD_loadDictionaryContent(ZSTD_CCtx *zc, const void *src, size_t srcSize)
-{
- const BYTE *const ip = (const BYTE *)src;
- const BYTE *const iend = ip + srcSize;
-
- /* input becomes curr prefix */
- zc->lowLimit = zc->dictLimit;
- zc->dictLimit = (U32)(zc->nextSrc - zc->base);
- zc->dictBase = zc->base;
- zc->base += ip - zc->nextSrc;
- zc->nextToUpdate = zc->dictLimit;
- zc->loadedDictEnd = zc->forceWindow ? 0 : (U32)(iend - zc->base);
-
- zc->nextSrc = iend;
- if (srcSize <= HASH_READ_SIZE)
- return 0;
-
- switch (zc->params.cParams.strategy) {
- case ZSTD_fast: ZSTD_fillHashTable(zc, iend, zc->params.cParams.searchLength); break;
-
- case ZSTD_dfast: ZSTD_fillDoubleHashTable(zc, iend, zc->params.cParams.searchLength); break;
-
- case ZSTD_greedy:
- case ZSTD_lazy:
- case ZSTD_lazy2:
- if (srcSize >= HASH_READ_SIZE)
- ZSTD_insertAndFindFirstIndex(zc, iend - HASH_READ_SIZE, zc->params.cParams.searchLength);
- break;
-
- case ZSTD_btlazy2:
- case ZSTD_btopt:
- case ZSTD_btopt2:
- if (srcSize >= HASH_READ_SIZE)
- ZSTD_updateTree(zc, iend - HASH_READ_SIZE, iend, 1 << zc->params.cParams.searchLog, zc->params.cParams.searchLength);
- break;
-
- default:
- return ERROR(GENERIC); /* strategy doesn't exist; impossible */
- }
-
- zc->nextToUpdate = (U32)(iend - zc->base);
- return 0;
-}
-
-/* Dictionaries that assign zero probability to symbols that show up causes problems
- when FSE encoding. Refuse dictionaries that assign zero probability to symbols
- that we may encounter during compression.
- NOTE: This behavior is not standard and could be improved in the future. */
-static size_t ZSTD_checkDictNCount(short *normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue)
-{
- U32 s;
- if (dictMaxSymbolValue < maxSymbolValue)
- return ERROR(dictionary_corrupted);
- for (s = 0; s <= maxSymbolValue; ++s) {
- if (normalizedCounter[s] == 0)
- return ERROR(dictionary_corrupted);
- }
- return 0;
-}
-
-/* Dictionary format :
- * See :
- * https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#dictionary-format
- */
-/*! ZSTD_loadZstdDictionary() :
- * @return : 0, or an error code
- * assumptions : magic number supposed already checked
- * dictSize supposed > 8
- */
-static size_t ZSTD_loadZstdDictionary(ZSTD_CCtx *cctx, const void *dict, size_t dictSize)
-{
- const BYTE *dictPtr = (const BYTE *)dict;
- const BYTE *const dictEnd = dictPtr + dictSize;
- short offcodeNCount[MaxOff + 1];
- unsigned offcodeMaxValue = MaxOff;
-
- dictPtr += 4; /* skip magic number */
- cctx->dictID = cctx->params.fParams.noDictIDFlag ? 0 : ZSTD_readLE32(dictPtr);
- dictPtr += 4;
-
- {
- size_t const hufHeaderSize = HUF_readCTable_wksp(cctx->hufTable, 255, dictPtr, dictEnd - dictPtr, cctx->tmpCounters, sizeof(cctx->tmpCounters));
- if (HUF_isError(hufHeaderSize))
- return ERROR(dictionary_corrupted);
- dictPtr += hufHeaderSize;
- }
-
- {
- unsigned offcodeLog;
- size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(offcodeHeaderSize))
- return ERROR(dictionary_corrupted);
- if (offcodeLog > OffFSELog)
- return ERROR(dictionary_corrupted);
- /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
- CHECK_E(FSE_buildCTable_wksp(cctx->offcodeCTable, offcodeNCount, offcodeMaxValue, offcodeLog, cctx->tmpCounters, sizeof(cctx->tmpCounters)),
- dictionary_corrupted);
- dictPtr += offcodeHeaderSize;
- }
-
- {
- short matchlengthNCount[MaxML + 1];
- unsigned matchlengthMaxValue = MaxML, matchlengthLog;
- size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(matchlengthHeaderSize))
- return ERROR(dictionary_corrupted);
- if (matchlengthLog > MLFSELog)
- return ERROR(dictionary_corrupted);
- /* Every match length code must have non-zero probability */
- CHECK_F(ZSTD_checkDictNCount(matchlengthNCount, matchlengthMaxValue, MaxML));
- CHECK_E(
- FSE_buildCTable_wksp(cctx->matchlengthCTable, matchlengthNCount, matchlengthMaxValue, matchlengthLog, cctx->tmpCounters, sizeof(cctx->tmpCounters)),
- dictionary_corrupted);
- dictPtr += matchlengthHeaderSize;
- }
-
- {
- short litlengthNCount[MaxLL + 1];
- unsigned litlengthMaxValue = MaxLL, litlengthLog;
- size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(litlengthHeaderSize))
- return ERROR(dictionary_corrupted);
- if (litlengthLog > LLFSELog)
- return ERROR(dictionary_corrupted);
- /* Every literal length code must have non-zero probability */
- CHECK_F(ZSTD_checkDictNCount(litlengthNCount, litlengthMaxValue, MaxLL));
- CHECK_E(FSE_buildCTable_wksp(cctx->litlengthCTable, litlengthNCount, litlengthMaxValue, litlengthLog, cctx->tmpCounters, sizeof(cctx->tmpCounters)),
- dictionary_corrupted);
- dictPtr += litlengthHeaderSize;
- }
-
- if (dictPtr + 12 > dictEnd)
- return ERROR(dictionary_corrupted);
- cctx->rep[0] = ZSTD_readLE32(dictPtr + 0);
- cctx->rep[1] = ZSTD_readLE32(dictPtr + 4);
- cctx->rep[2] = ZSTD_readLE32(dictPtr + 8);
- dictPtr += 12;
-
- {
- size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
- U32 offcodeMax = MaxOff;
- if (dictContentSize <= ((U32)-1) - 128 KB) {
- U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */
- offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
- }
- /* All offset values <= dictContentSize + 128 KB must be representable */
- CHECK_F(ZSTD_checkDictNCount(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff)));
- /* All repCodes must be <= dictContentSize and != 0*/
- {
- U32 u;
- for (u = 0; u < 3; u++) {
- if (cctx->rep[u] == 0)
- return ERROR(dictionary_corrupted);
- if (cctx->rep[u] > dictContentSize)
- return ERROR(dictionary_corrupted);
- }
- }
-
- cctx->flagStaticTables = 1;
- cctx->flagStaticHufTable = HUF_repeat_valid;
- return ZSTD_loadDictionaryContent(cctx, dictPtr, dictContentSize);
- }
-}
-
-/** ZSTD_compress_insertDictionary() :
-* @return : 0, or an error code */
-static size_t ZSTD_compress_insertDictionary(ZSTD_CCtx *cctx, const void *dict, size_t dictSize)
-{
- if ((dict == NULL) || (dictSize <= 8))
- return 0;
-
- /* dict as pure content */
- if ((ZSTD_readLE32(dict) != ZSTD_DICT_MAGIC) || (cctx->forceRawDict))
- return ZSTD_loadDictionaryContent(cctx, dict, dictSize);
-
- /* dict as zstd dictionary */
- return ZSTD_loadZstdDictionary(cctx, dict, dictSize);
-}
-
-/*! ZSTD_compressBegin_internal() :
-* @return : 0, or an error code */
-static size_t ZSTD_compressBegin_internal(ZSTD_CCtx *cctx, const void *dict, size_t dictSize, ZSTD_parameters params, U64 pledgedSrcSize)
-{
- ZSTD_compResetPolicy_e const crp = dictSize ? ZSTDcrp_fullReset : ZSTDcrp_continue;
- CHECK_F(ZSTD_resetCCtx_advanced(cctx, params, pledgedSrcSize, crp));
- return ZSTD_compress_insertDictionary(cctx, dict, dictSize);
-}
-
-/*! ZSTD_compressBegin_advanced() :
-* @return : 0, or an error code */
-size_t ZSTD_compressBegin_advanced(ZSTD_CCtx *cctx, const void *dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize)
-{
- /* compression parameters verification and optimization */
- CHECK_F(ZSTD_checkCParams(params.cParams));
- return ZSTD_compressBegin_internal(cctx, dict, dictSize, params, pledgedSrcSize);
-}
-
-size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx *cctx, const void *dict, size_t dictSize, int compressionLevel)
-{
- ZSTD_parameters const params = ZSTD_getParams(compressionLevel, 0, dictSize);
- return ZSTD_compressBegin_internal(cctx, dict, dictSize, params, 0);
-}
-
-size_t ZSTD_compressBegin(ZSTD_CCtx *cctx, int compressionLevel) { return ZSTD_compressBegin_usingDict(cctx, NULL, 0, compressionLevel); }
-
-/*! ZSTD_writeEpilogue() :
-* Ends a frame.
-* @return : nb of bytes written into dst (or an error code) */
-static size_t ZSTD_writeEpilogue(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity)
-{
- BYTE *const ostart = (BYTE *)dst;
- BYTE *op = ostart;
- size_t fhSize = 0;
-
- if (cctx->stage == ZSTDcs_created)
- return ERROR(stage_wrong); /* init missing */
-
- /* special case : empty frame */
- if (cctx->stage == ZSTDcs_init) {
- fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, cctx->params, 0, 0);
- if (ZSTD_isError(fhSize))
- return fhSize;
- dstCapacity -= fhSize;
- op += fhSize;
- cctx->stage = ZSTDcs_ongoing;
- }
-
- if (cctx->stage != ZSTDcs_ending) {
- /* write one last empty block, make it the "last" block */
- U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw) << 1) + 0;
- if (dstCapacity < 4)
- return ERROR(dstSize_tooSmall);
- ZSTD_writeLE32(op, cBlockHeader24);
- op += ZSTD_blockHeaderSize;
- dstCapacity -= ZSTD_blockHeaderSize;
- }
-
- if (cctx->params.fParams.checksumFlag) {
- U32 const checksum = (U32)xxh64_digest(&cctx->xxhState);
- if (dstCapacity < 4)
- return ERROR(dstSize_tooSmall);
- ZSTD_writeLE32(op, checksum);
- op += 4;
- }
-
- cctx->stage = ZSTDcs_created; /* return to "created but no init" status */
- return op - ostart;
-}
-
-size_t ZSTD_compressEnd(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t endResult;
- size_t const cSize = ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1, 1);
- if (ZSTD_isError(cSize))
- return cSize;
- endResult = ZSTD_writeEpilogue(cctx, (char *)dst + cSize, dstCapacity - cSize);
- if (ZSTD_isError(endResult))
- return endResult;
- return cSize + endResult;
-}
-
-static size_t ZSTD_compress_internal(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const void *dict, size_t dictSize,
- ZSTD_parameters params)
-{
- CHECK_F(ZSTD_compressBegin_internal(cctx, dict, dictSize, params, srcSize));
- return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
-}
-
-size_t ZSTD_compress_usingDict(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const void *dict, size_t dictSize,
- ZSTD_parameters params)
-{
- return ZSTD_compress_internal(ctx, dst, dstCapacity, src, srcSize, dict, dictSize, params);
-}
-
-size_t ZSTD_compressCCtx(ZSTD_CCtx *ctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, ZSTD_parameters params)
-{
- return ZSTD_compress_internal(ctx, dst, dstCapacity, src, srcSize, NULL, 0, params);
-}
-
-/* ===== Dictionary API ===== */
-
-struct ZSTD_CDict_s {
- void *dictBuffer;
- const void *dictContent;
- size_t dictContentSize;
- ZSTD_CCtx *refContext;
-}; /* typedef'd tp ZSTD_CDict within "zstd.h" */
-
-size_t ZSTD_CDictWorkspaceBound(ZSTD_compressionParameters cParams) { return ZSTD_CCtxWorkspaceBound(cParams) + ZSTD_ALIGN(sizeof(ZSTD_CDict)); }
-
-static ZSTD_CDict *ZSTD_createCDict_advanced(const void *dictBuffer, size_t dictSize, unsigned byReference, ZSTD_parameters params, ZSTD_customMem customMem)
-{
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
-
- {
- ZSTD_CDict *const cdict = (ZSTD_CDict *)ZSTD_malloc(sizeof(ZSTD_CDict), customMem);
- ZSTD_CCtx *const cctx = ZSTD_createCCtx_advanced(customMem);
-
- if (!cdict || !cctx) {
- ZSTD_free(cdict, customMem);
- ZSTD_freeCCtx(cctx);
- return NULL;
- }
-
- if ((byReference) || (!dictBuffer) || (!dictSize)) {
- cdict->dictBuffer = NULL;
- cdict->dictContent = dictBuffer;
- } else {
- void *const internalBuffer = ZSTD_malloc(dictSize, customMem);
- if (!internalBuffer) {
- ZSTD_free(cctx, customMem);
- ZSTD_free(cdict, customMem);
- return NULL;
- }
- memcpy(internalBuffer, dictBuffer, dictSize);
- cdict->dictBuffer = internalBuffer;
- cdict->dictContent = internalBuffer;
- }
-
- {
- size_t const errorCode = ZSTD_compressBegin_advanced(cctx, cdict->dictContent, dictSize, params, 0);
- if (ZSTD_isError(errorCode)) {
- ZSTD_free(cdict->dictBuffer, customMem);
- ZSTD_free(cdict, customMem);
- ZSTD_freeCCtx(cctx);
- return NULL;
- }
- }
-
- cdict->refContext = cctx;
- cdict->dictContentSize = dictSize;
- return cdict;
- }
-}
-
-ZSTD_CDict *ZSTD_initCDict(const void *dict, size_t dictSize, ZSTD_parameters params, void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- return ZSTD_createCDict_advanced(dict, dictSize, 1, params, stackMem);
-}
-
-size_t ZSTD_freeCDict(ZSTD_CDict *cdict)
-{
- if (cdict == NULL)
- return 0; /* support free on NULL */
- {
- ZSTD_customMem const cMem = cdict->refContext->customMem;
- ZSTD_freeCCtx(cdict->refContext);
- ZSTD_free(cdict->dictBuffer, cMem);
- ZSTD_free(cdict, cMem);
- return 0;
- }
-}
-
-static ZSTD_parameters ZSTD_getParamsFromCDict(const ZSTD_CDict *cdict) { return ZSTD_getParamsFromCCtx(cdict->refContext); }
-
-size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx *cctx, const ZSTD_CDict *cdict, unsigned long long pledgedSrcSize)
-{
- if (cdict->dictContentSize)
- CHECK_F(ZSTD_copyCCtx(cctx, cdict->refContext, pledgedSrcSize))
- else {
- ZSTD_parameters params = cdict->refContext->params;
- params.fParams.contentSizeFlag = (pledgedSrcSize > 0);
- CHECK_F(ZSTD_compressBegin_advanced(cctx, NULL, 0, params, pledgedSrcSize));
- }
- return 0;
-}
-
-/*! ZSTD_compress_usingCDict() :
-* Compression using a digested Dictionary.
-* Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times.
-* Note that compression level is decided during dictionary creation */
-size_t ZSTD_compress_usingCDict(ZSTD_CCtx *cctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const ZSTD_CDict *cdict)
-{
- CHECK_F(ZSTD_compressBegin_usingCDict(cctx, cdict, srcSize));
-
- if (cdict->refContext->params.fParams.contentSizeFlag == 1) {
- cctx->params.fParams.contentSizeFlag = 1;
- cctx->frameContentSize = srcSize;
- } else {
- cctx->params.fParams.contentSizeFlag = 0;
- }
-
- return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
-}
-
-/* ******************************************************************
-* Streaming
-********************************************************************/
-
-typedef enum { zcss_init, zcss_load, zcss_flush, zcss_final } ZSTD_cStreamStage;
-
-struct ZSTD_CStream_s {
- ZSTD_CCtx *cctx;
- ZSTD_CDict *cdictLocal;
- const ZSTD_CDict *cdict;
- char *inBuff;
- size_t inBuffSize;
- size_t inToCompress;
- size_t inBuffPos;
- size_t inBuffTarget;
- size_t blockSize;
- char *outBuff;
- size_t outBuffSize;
- size_t outBuffContentSize;
- size_t outBuffFlushedSize;
- ZSTD_cStreamStage stage;
- U32 checksum;
- U32 frameEnded;
- U64 pledgedSrcSize;
- U64 inputProcessed;
- ZSTD_parameters params;
- ZSTD_customMem customMem;
-}; /* typedef'd to ZSTD_CStream within "zstd.h" */
-
-size_t ZSTD_CStreamWorkspaceBound(ZSTD_compressionParameters cParams)
-{
- size_t const inBuffSize = (size_t)1 << cParams.windowLog;
- size_t const blockSize = MIN(ZSTD_BLOCKSIZE_ABSOLUTEMAX, inBuffSize);
- size_t const outBuffSize = ZSTD_compressBound(blockSize) + 1;
-
- return ZSTD_CCtxWorkspaceBound(cParams) + ZSTD_ALIGN(sizeof(ZSTD_CStream)) + ZSTD_ALIGN(inBuffSize) + ZSTD_ALIGN(outBuffSize);
-}
-
-ZSTD_CStream *ZSTD_createCStream_advanced(ZSTD_customMem customMem)
-{
- ZSTD_CStream *zcs;
-
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
-
- zcs = (ZSTD_CStream *)ZSTD_malloc(sizeof(ZSTD_CStream), customMem);
- if (zcs == NULL)
- return NULL;
- memset(zcs, 0, sizeof(ZSTD_CStream));
- memcpy(&zcs->customMem, &customMem, sizeof(ZSTD_customMem));
- zcs->cctx = ZSTD_createCCtx_advanced(customMem);
- if (zcs->cctx == NULL) {
- ZSTD_freeCStream(zcs);
- return NULL;
- }
- return zcs;
-}
-
-size_t ZSTD_freeCStream(ZSTD_CStream *zcs)
-{
- if (zcs == NULL)
- return 0; /* support free on NULL */
- {
- ZSTD_customMem const cMem = zcs->customMem;
- ZSTD_freeCCtx(zcs->cctx);
- zcs->cctx = NULL;
- ZSTD_freeCDict(zcs->cdictLocal);
- zcs->cdictLocal = NULL;
- ZSTD_free(zcs->inBuff, cMem);
- zcs->inBuff = NULL;
- ZSTD_free(zcs->outBuff, cMem);
- zcs->outBuff = NULL;
- ZSTD_free(zcs, cMem);
- return 0;
- }
-}
-
-/*====== Initialization ======*/
-
-size_t ZSTD_CStreamInSize(void) { return ZSTD_BLOCKSIZE_ABSOLUTEMAX; }
-size_t ZSTD_CStreamOutSize(void) { return ZSTD_compressBound(ZSTD_BLOCKSIZE_ABSOLUTEMAX) + ZSTD_blockHeaderSize + 4 /* 32-bits hash */; }
-
-static size_t ZSTD_resetCStream_internal(ZSTD_CStream *zcs, unsigned long long pledgedSrcSize)
-{
- if (zcs->inBuffSize == 0)
- return ERROR(stage_wrong); /* zcs has not been init at least once => can't reset */
-
- if (zcs->cdict)
- CHECK_F(ZSTD_compressBegin_usingCDict(zcs->cctx, zcs->cdict, pledgedSrcSize))
- else
- CHECK_F(ZSTD_compressBegin_advanced(zcs->cctx, NULL, 0, zcs->params, pledgedSrcSize));
-
- zcs->inToCompress = 0;
- zcs->inBuffPos = 0;
- zcs->inBuffTarget = zcs->blockSize;
- zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0;
- zcs->stage = zcss_load;
- zcs->frameEnded = 0;
- zcs->pledgedSrcSize = pledgedSrcSize;
- zcs->inputProcessed = 0;
- return 0; /* ready to go */
-}
-
-size_t ZSTD_resetCStream(ZSTD_CStream *zcs, unsigned long long pledgedSrcSize)
-{
-
- zcs->params.fParams.contentSizeFlag = (pledgedSrcSize > 0);
-
- return ZSTD_resetCStream_internal(zcs, pledgedSrcSize);
-}
-
-static size_t ZSTD_initCStream_advanced(ZSTD_CStream *zcs, const void *dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize)
-{
- /* allocate buffers */
- {
- size_t const neededInBuffSize = (size_t)1 << params.cParams.windowLog;
- if (zcs->inBuffSize < neededInBuffSize) {
- zcs->inBuffSize = neededInBuffSize;
- ZSTD_free(zcs->inBuff, zcs->customMem);
- zcs->inBuff = (char *)ZSTD_malloc(neededInBuffSize, zcs->customMem);
- if (zcs->inBuff == NULL)
- return ERROR(memory_allocation);
- }
- zcs->blockSize = MIN(ZSTD_BLOCKSIZE_ABSOLUTEMAX, neededInBuffSize);
- }
- if (zcs->outBuffSize < ZSTD_compressBound(zcs->blockSize) + 1) {
- zcs->outBuffSize = ZSTD_compressBound(zcs->blockSize) + 1;
- ZSTD_free(zcs->outBuff, zcs->customMem);
- zcs->outBuff = (char *)ZSTD_malloc(zcs->outBuffSize, zcs->customMem);
- if (zcs->outBuff == NULL)
- return ERROR(memory_allocation);
- }
-
- if (dict && dictSize >= 8) {
- ZSTD_freeCDict(zcs->cdictLocal);
- zcs->cdictLocal = ZSTD_createCDict_advanced(dict, dictSize, 0, params, zcs->customMem);
- if (zcs->cdictLocal == NULL)
- return ERROR(memory_allocation);
- zcs->cdict = zcs->cdictLocal;
- } else
- zcs->cdict = NULL;
-
- zcs->checksum = params.fParams.checksumFlag > 0;
- zcs->params = params;
-
- return ZSTD_resetCStream_internal(zcs, pledgedSrcSize);
-}
-
-ZSTD_CStream *ZSTD_initCStream(ZSTD_parameters params, unsigned long long pledgedSrcSize, void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- ZSTD_CStream *const zcs = ZSTD_createCStream_advanced(stackMem);
- if (zcs) {
- size_t const code = ZSTD_initCStream_advanced(zcs, NULL, 0, params, pledgedSrcSize);
- if (ZSTD_isError(code)) {
- return NULL;
- }
- }
- return zcs;
-}
-
-ZSTD_CStream *ZSTD_initCStream_usingCDict(const ZSTD_CDict *cdict, unsigned long long pledgedSrcSize, void *workspace, size_t workspaceSize)
-{
- ZSTD_parameters const params = ZSTD_getParamsFromCDict(cdict);
- ZSTD_CStream *const zcs = ZSTD_initCStream(params, pledgedSrcSize, workspace, workspaceSize);
- if (zcs) {
- zcs->cdict = cdict;
- if (ZSTD_isError(ZSTD_resetCStream_internal(zcs, pledgedSrcSize))) {
- return NULL;
- }
- }
- return zcs;
-}
-
-/*====== Compression ======*/
-
-typedef enum { zsf_gather, zsf_flush, zsf_end } ZSTD_flush_e;
-
-ZSTD_STATIC size_t ZSTD_limitCopy(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t const length = MIN(dstCapacity, srcSize);
- memcpy(dst, src, length);
- return length;
-}
-
-static size_t ZSTD_compressStream_generic(ZSTD_CStream *zcs, void *dst, size_t *dstCapacityPtr, const void *src, size_t *srcSizePtr, ZSTD_flush_e const flush)
-{
- U32 someMoreWork = 1;
- const char *const istart = (const char *)src;
- const char *const iend = istart + *srcSizePtr;
- const char *ip = istart;
- char *const ostart = (char *)dst;
- char *const oend = ostart + *dstCapacityPtr;
- char *op = ostart;
-
- while (someMoreWork) {
- switch (zcs->stage) {
- case zcss_init:
- return ERROR(init_missing); /* call ZBUFF_compressInit() first ! */
-
- case zcss_load:
- /* complete inBuffer */
- {
- size_t const toLoad = zcs->inBuffTarget - zcs->inBuffPos;
- size_t const loaded = ZSTD_limitCopy(zcs->inBuff + zcs->inBuffPos, toLoad, ip, iend - ip);
- zcs->inBuffPos += loaded;
- ip += loaded;
- if ((zcs->inBuffPos == zcs->inToCompress) || (!flush && (toLoad != loaded))) {
- someMoreWork = 0;
- break; /* not enough input to get a full block : stop there, wait for more */
- }
- }
- /* compress curr block (note : this stage cannot be stopped in the middle) */
- {
- void *cDst;
- size_t cSize;
- size_t const iSize = zcs->inBuffPos - zcs->inToCompress;
- size_t oSize = oend - op;
- if (oSize >= ZSTD_compressBound(iSize))
- cDst = op; /* compress directly into output buffer (avoid flush stage) */
- else
- cDst = zcs->outBuff, oSize = zcs->outBuffSize;
- cSize = (flush == zsf_end) ? ZSTD_compressEnd(zcs->cctx, cDst, oSize, zcs->inBuff + zcs->inToCompress, iSize)
- : ZSTD_compressContinue(zcs->cctx, cDst, oSize, zcs->inBuff + zcs->inToCompress, iSize);
- if (ZSTD_isError(cSize))
- return cSize;
- if (flush == zsf_end)
- zcs->frameEnded = 1;
- /* prepare next block */
- zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSize;
- if (zcs->inBuffTarget > zcs->inBuffSize)
- zcs->inBuffPos = 0, zcs->inBuffTarget = zcs->blockSize; /* note : inBuffSize >= blockSize */
- zcs->inToCompress = zcs->inBuffPos;
- if (cDst == op) {
- op += cSize;
- break;
- } /* no need to flush */
- zcs->outBuffContentSize = cSize;
- zcs->outBuffFlushedSize = 0;
- zcs->stage = zcss_flush; /* pass-through to flush stage */
- }
- fallthrough;
-
- case zcss_flush: {
- size_t const toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize;
- size_t const flushed = ZSTD_limitCopy(op, oend - op, zcs->outBuff + zcs->outBuffFlushedSize, toFlush);
- op += flushed;
- zcs->outBuffFlushedSize += flushed;
- if (toFlush != flushed) {
- someMoreWork = 0;
- break;
- } /* dst too small to store flushed data : stop there */
- zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0;
- zcs->stage = zcss_load;
- break;
- }
-
- case zcss_final:
- someMoreWork = 0; /* do nothing */
- break;
-
- default:
- return ERROR(GENERIC); /* impossible */
- }
- }
-
- *srcSizePtr = ip - istart;
- *dstCapacityPtr = op - ostart;
- zcs->inputProcessed += *srcSizePtr;
- if (zcs->frameEnded)
- return 0;
- {
- size_t hintInSize = zcs->inBuffTarget - zcs->inBuffPos;
- if (hintInSize == 0)
- hintInSize = zcs->blockSize;
- return hintInSize;
- }
-}
-
-size_t ZSTD_compressStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output, ZSTD_inBuffer *input)
-{
- size_t sizeRead = input->size - input->pos;
- size_t sizeWritten = output->size - output->pos;
- size_t const result =
- ZSTD_compressStream_generic(zcs, (char *)(output->dst) + output->pos, &sizeWritten, (const char *)(input->src) + input->pos, &sizeRead, zsf_gather);
- input->pos += sizeRead;
- output->pos += sizeWritten;
- return result;
-}
-
-/*====== Finalize ======*/
-
-/*! ZSTD_flushStream() :
-* @return : amount of data remaining to flush */
-size_t ZSTD_flushStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output)
-{
- size_t srcSize = 0;
- size_t sizeWritten = output->size - output->pos;
- size_t const result = ZSTD_compressStream_generic(zcs, (char *)(output->dst) + output->pos, &sizeWritten, &srcSize,
- &srcSize, /* use a valid src address instead of NULL */
- zsf_flush);
- output->pos += sizeWritten;
- if (ZSTD_isError(result))
- return result;
- return zcs->outBuffContentSize - zcs->outBuffFlushedSize; /* remaining to flush */
-}
-
-size_t ZSTD_endStream(ZSTD_CStream *zcs, ZSTD_outBuffer *output)
-{
- BYTE *const ostart = (BYTE *)(output->dst) + output->pos;
- BYTE *const oend = (BYTE *)(output->dst) + output->size;
- BYTE *op = ostart;
-
- if ((zcs->pledgedSrcSize) && (zcs->inputProcessed != zcs->pledgedSrcSize))
- return ERROR(srcSize_wrong); /* pledgedSrcSize not respected */
-
- if (zcs->stage != zcss_final) {
- /* flush whatever remains */
- size_t srcSize = 0;
- size_t sizeWritten = output->size - output->pos;
- size_t const notEnded =
- ZSTD_compressStream_generic(zcs, ostart, &sizeWritten, &srcSize, &srcSize, zsf_end); /* use a valid src address instead of NULL */
- size_t const remainingToFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize;
- op += sizeWritten;
- if (remainingToFlush) {
- output->pos += sizeWritten;
- return remainingToFlush + ZSTD_BLOCKHEADERSIZE /* final empty block */ + (zcs->checksum * 4);
- }
- /* create epilogue */
- zcs->stage = zcss_final;
- zcs->outBuffContentSize = !notEnded ? 0 : ZSTD_compressEnd(zcs->cctx, zcs->outBuff, zcs->outBuffSize, NULL,
- 0); /* write epilogue, including final empty block, into outBuff */
- }
-
- /* flush epilogue */
- {
- size_t const toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize;
- size_t const flushed = ZSTD_limitCopy(op, oend - op, zcs->outBuff + zcs->outBuffFlushedSize, toFlush);
- op += flushed;
- zcs->outBuffFlushedSize += flushed;
- output->pos += op - ostart;
- if (toFlush == flushed)
- zcs->stage = zcss_init; /* end reached */
- return toFlush - flushed;
- }
-}
-
-/*-===== Pre-defined compression levels =====-*/
-
-#define ZSTD_DEFAULT_CLEVEL 1
-#define ZSTD_MAX_CLEVEL 22
-int ZSTD_maxCLevel(void) { return ZSTD_MAX_CLEVEL; }
-
-static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL + 1] = {
- {
- /* "default" */
- /* W, C, H, S, L, TL, strat */
- {18, 12, 12, 1, 7, 16, ZSTD_fast}, /* level 0 - never used */
- {19, 13, 14, 1, 7, 16, ZSTD_fast}, /* level 1 */
- {19, 15, 16, 1, 6, 16, ZSTD_fast}, /* level 2 */
- {20, 16, 17, 1, 5, 16, ZSTD_dfast}, /* level 3.*/
- {20, 18, 18, 1, 5, 16, ZSTD_dfast}, /* level 4.*/
- {20, 15, 18, 3, 5, 16, ZSTD_greedy}, /* level 5 */
- {21, 16, 19, 2, 5, 16, ZSTD_lazy}, /* level 6 */
- {21, 17, 20, 3, 5, 16, ZSTD_lazy}, /* level 7 */
- {21, 18, 20, 3, 5, 16, ZSTD_lazy2}, /* level 8 */
- {21, 20, 20, 3, 5, 16, ZSTD_lazy2}, /* level 9 */
- {21, 19, 21, 4, 5, 16, ZSTD_lazy2}, /* level 10 */
- {22, 20, 22, 4, 5, 16, ZSTD_lazy2}, /* level 11 */
- {22, 20, 22, 5, 5, 16, ZSTD_lazy2}, /* level 12 */
- {22, 21, 22, 5, 5, 16, ZSTD_lazy2}, /* level 13 */
- {22, 21, 22, 6, 5, 16, ZSTD_lazy2}, /* level 14 */
- {22, 21, 21, 5, 5, 16, ZSTD_btlazy2}, /* level 15 */
- {23, 22, 22, 5, 5, 16, ZSTD_btlazy2}, /* level 16 */
- {23, 21, 22, 4, 5, 24, ZSTD_btopt}, /* level 17 */
- {23, 23, 22, 6, 5, 32, ZSTD_btopt}, /* level 18 */
- {23, 23, 22, 6, 3, 48, ZSTD_btopt}, /* level 19 */
- {25, 25, 23, 7, 3, 64, ZSTD_btopt2}, /* level 20 */
- {26, 26, 23, 7, 3, 256, ZSTD_btopt2}, /* level 21 */
- {27, 27, 25, 9, 3, 512, ZSTD_btopt2}, /* level 22 */
- },
- {
- /* for srcSize <= 256 KB */
- /* W, C, H, S, L, T, strat */
- {0, 0, 0, 0, 0, 0, ZSTD_fast}, /* level 0 - not used */
- {18, 13, 14, 1, 6, 8, ZSTD_fast}, /* level 1 */
- {18, 14, 13, 1, 5, 8, ZSTD_dfast}, /* level 2 */
- {18, 16, 15, 1, 5, 8, ZSTD_dfast}, /* level 3 */
- {18, 15, 17, 1, 5, 8, ZSTD_greedy}, /* level 4.*/
- {18, 16, 17, 4, 5, 8, ZSTD_greedy}, /* level 5.*/
- {18, 16, 17, 3, 5, 8, ZSTD_lazy}, /* level 6.*/
- {18, 17, 17, 4, 4, 8, ZSTD_lazy}, /* level 7 */
- {18, 17, 17, 4, 4, 8, ZSTD_lazy2}, /* level 8 */
- {18, 17, 17, 5, 4, 8, ZSTD_lazy2}, /* level 9 */
- {18, 17, 17, 6, 4, 8, ZSTD_lazy2}, /* level 10 */
- {18, 18, 17, 6, 4, 8, ZSTD_lazy2}, /* level 11.*/
- {18, 18, 17, 7, 4, 8, ZSTD_lazy2}, /* level 12.*/
- {18, 19, 17, 6, 4, 8, ZSTD_btlazy2}, /* level 13 */
- {18, 18, 18, 4, 4, 16, ZSTD_btopt}, /* level 14.*/
- {18, 18, 18, 4, 3, 16, ZSTD_btopt}, /* level 15.*/
- {18, 19, 18, 6, 3, 32, ZSTD_btopt}, /* level 16.*/
- {18, 19, 18, 8, 3, 64, ZSTD_btopt}, /* level 17.*/
- {18, 19, 18, 9, 3, 128, ZSTD_btopt}, /* level 18.*/
- {18, 19, 18, 10, 3, 256, ZSTD_btopt}, /* level 19.*/
- {18, 19, 18, 11, 3, 512, ZSTD_btopt2}, /* level 20.*/
- {18, 19, 18, 12, 3, 512, ZSTD_btopt2}, /* level 21.*/
- {18, 19, 18, 13, 3, 512, ZSTD_btopt2}, /* level 22.*/
- },
- {
- /* for srcSize <= 128 KB */
- /* W, C, H, S, L, T, strat */
- {17, 12, 12, 1, 7, 8, ZSTD_fast}, /* level 0 - not used */
- {17, 12, 13, 1, 6, 8, ZSTD_fast}, /* level 1 */
- {17, 13, 16, 1, 5, 8, ZSTD_fast}, /* level 2 */
- {17, 16, 16, 2, 5, 8, ZSTD_dfast}, /* level 3 */
- {17, 13, 15, 3, 4, 8, ZSTD_greedy}, /* level 4 */
- {17, 15, 17, 4, 4, 8, ZSTD_greedy}, /* level 5 */
- {17, 16, 17, 3, 4, 8, ZSTD_lazy}, /* level 6 */
- {17, 15, 17, 4, 4, 8, ZSTD_lazy2}, /* level 7 */
- {17, 17, 17, 4, 4, 8, ZSTD_lazy2}, /* level 8 */
- {17, 17, 17, 5, 4, 8, ZSTD_lazy2}, /* level 9 */
- {17, 17, 17, 6, 4, 8, ZSTD_lazy2}, /* level 10 */
- {17, 17, 17, 7, 4, 8, ZSTD_lazy2}, /* level 11 */
- {17, 17, 17, 8, 4, 8, ZSTD_lazy2}, /* level 12 */
- {17, 18, 17, 6, 4, 8, ZSTD_btlazy2}, /* level 13.*/
- {17, 17, 17, 7, 3, 8, ZSTD_btopt}, /* level 14.*/
- {17, 17, 17, 7, 3, 16, ZSTD_btopt}, /* level 15.*/
- {17, 18, 17, 7, 3, 32, ZSTD_btopt}, /* level 16.*/
- {17, 18, 17, 7, 3, 64, ZSTD_btopt}, /* level 17.*/
- {17, 18, 17, 7, 3, 256, ZSTD_btopt}, /* level 18.*/
- {17, 18, 17, 8, 3, 256, ZSTD_btopt}, /* level 19.*/
- {17, 18, 17, 9, 3, 256, ZSTD_btopt2}, /* level 20.*/
- {17, 18, 17, 10, 3, 256, ZSTD_btopt2}, /* level 21.*/
- {17, 18, 17, 11, 3, 512, ZSTD_btopt2}, /* level 22.*/
- },
- {
- /* for srcSize <= 16 KB */
- /* W, C, H, S, L, T, strat */
- {14, 12, 12, 1, 7, 6, ZSTD_fast}, /* level 0 - not used */
- {14, 14, 14, 1, 6, 6, ZSTD_fast}, /* level 1 */
- {14, 14, 14, 1, 4, 6, ZSTD_fast}, /* level 2 */
- {14, 14, 14, 1, 4, 6, ZSTD_dfast}, /* level 3.*/
- {14, 14, 14, 4, 4, 6, ZSTD_greedy}, /* level 4.*/
- {14, 14, 14, 3, 4, 6, ZSTD_lazy}, /* level 5.*/
- {14, 14, 14, 4, 4, 6, ZSTD_lazy2}, /* level 6 */
- {14, 14, 14, 5, 4, 6, ZSTD_lazy2}, /* level 7 */
- {14, 14, 14, 6, 4, 6, ZSTD_lazy2}, /* level 8.*/
- {14, 15, 14, 6, 4, 6, ZSTD_btlazy2}, /* level 9.*/
- {14, 15, 14, 3, 3, 6, ZSTD_btopt}, /* level 10.*/
- {14, 15, 14, 6, 3, 8, ZSTD_btopt}, /* level 11.*/
- {14, 15, 14, 6, 3, 16, ZSTD_btopt}, /* level 12.*/
- {14, 15, 14, 6, 3, 24, ZSTD_btopt}, /* level 13.*/
- {14, 15, 15, 6, 3, 48, ZSTD_btopt}, /* level 14.*/
- {14, 15, 15, 6, 3, 64, ZSTD_btopt}, /* level 15.*/
- {14, 15, 15, 6, 3, 96, ZSTD_btopt}, /* level 16.*/
- {14, 15, 15, 6, 3, 128, ZSTD_btopt}, /* level 17.*/
- {14, 15, 15, 6, 3, 256, ZSTD_btopt}, /* level 18.*/
- {14, 15, 15, 7, 3, 256, ZSTD_btopt}, /* level 19.*/
- {14, 15, 15, 8, 3, 256, ZSTD_btopt2}, /* level 20.*/
- {14, 15, 15, 9, 3, 256, ZSTD_btopt2}, /* level 21.*/
- {14, 15, 15, 10, 3, 256, ZSTD_btopt2}, /* level 22.*/
- },
-};
-
-/*! ZSTD_getCParams() :
-* @return ZSTD_compressionParameters structure for a selected compression level, `srcSize` and `dictSize`.
-* Size values are optional, provide 0 if not known or unused */
-ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSize, size_t dictSize)
-{
- ZSTD_compressionParameters cp;
- size_t const addedSize = srcSize ? 0 : 500;
- U64 const rSize = srcSize + dictSize ? srcSize + dictSize + addedSize : (U64)-1;
- U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB); /* intentional underflow for srcSizeHint == 0 */
- if (compressionLevel <= 0)
- compressionLevel = ZSTD_DEFAULT_CLEVEL; /* 0 == default; no negative compressionLevel yet */
- if (compressionLevel > ZSTD_MAX_CLEVEL)
- compressionLevel = ZSTD_MAX_CLEVEL;
- cp = ZSTD_defaultCParameters[tableID][compressionLevel];
- if (ZSTD_32bits()) { /* auto-correction, for 32-bits mode */
- if (cp.windowLog > ZSTD_WINDOWLOG_MAX)
- cp.windowLog = ZSTD_WINDOWLOG_MAX;
- if (cp.chainLog > ZSTD_CHAINLOG_MAX)
- cp.chainLog = ZSTD_CHAINLOG_MAX;
- if (cp.hashLog > ZSTD_HASHLOG_MAX)
- cp.hashLog = ZSTD_HASHLOG_MAX;
- }
- cp = ZSTD_adjustCParams(cp, srcSize, dictSize);
- return cp;
-}
-
-/*! ZSTD_getParams() :
-* same as ZSTD_getCParams(), but @return a `ZSTD_parameters` object (instead of `ZSTD_compressionParameters`).
-* All fields of `ZSTD_frameParameters` are set to default (0) */
-ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSize, size_t dictSize)
-{
- ZSTD_parameters params;
- ZSTD_compressionParameters const cParams = ZSTD_getCParams(compressionLevel, srcSize, dictSize);
- memset(¶ms, 0, sizeof(params));
- params.cParams = cParams;
- return params;
-}
-
-EXPORT_SYMBOL(ZSTD_maxCLevel);
-EXPORT_SYMBOL(ZSTD_compressBound);
-
-EXPORT_SYMBOL(ZSTD_CCtxWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initCCtx);
-EXPORT_SYMBOL(ZSTD_compressCCtx);
-EXPORT_SYMBOL(ZSTD_compress_usingDict);
-
-EXPORT_SYMBOL(ZSTD_CDictWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initCDict);
-EXPORT_SYMBOL(ZSTD_compress_usingCDict);
-
-EXPORT_SYMBOL(ZSTD_CStreamWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initCStream);
-EXPORT_SYMBOL(ZSTD_initCStream_usingCDict);
-EXPORT_SYMBOL(ZSTD_resetCStream);
-EXPORT_SYMBOL(ZSTD_compressStream);
-EXPORT_SYMBOL(ZSTD_flushStream);
-EXPORT_SYMBOL(ZSTD_endStream);
-EXPORT_SYMBOL(ZSTD_CStreamInSize);
-EXPORT_SYMBOL(ZSTD_CStreamOutSize);
-
-EXPORT_SYMBOL(ZSTD_getCParams);
-EXPORT_SYMBOL(ZSTD_getParams);
-EXPORT_SYMBOL(ZSTD_checkCParams);
-EXPORT_SYMBOL(ZSTD_adjustCParams);
-
-EXPORT_SYMBOL(ZSTD_compressBegin);
-EXPORT_SYMBOL(ZSTD_compressBegin_usingDict);
-EXPORT_SYMBOL(ZSTD_compressBegin_advanced);
-EXPORT_SYMBOL(ZSTD_copyCCtx);
-EXPORT_SYMBOL(ZSTD_compressBegin_usingCDict);
-EXPORT_SYMBOL(ZSTD_compressContinue);
-EXPORT_SYMBOL(ZSTD_compressEnd);
-
-EXPORT_SYMBOL(ZSTD_getBlockSizeMax);
-EXPORT_SYMBOL(ZSTD_compressBlock);
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("Zstd Compressor");
--- /dev/null
+/* ******************************************************************
+ * FSE : Finite State Entropy encoder
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* **************************************************************
+* Includes
+****************************************************************/
+#include "../common/compiler.h"
+#include "../common/mem.h" /* U32, U16, etc. */
+#include "../common/debug.h" /* assert, DEBUGLOG */
+#include "hist.h" /* HIST_count_wksp */
+#include "../common/bitstream.h"
+#define FSE_STATIC_LINKING_ONLY
+#include "../common/fse.h"
+#include "../common/error_private.h"
+#define ZSTD_DEPS_NEED_MALLOC
+#define ZSTD_DEPS_NEED_MATH64
+#include "../common/zstd_deps.h" /* ZSTD_malloc, ZSTD_free, ZSTD_memcpy, ZSTD_memset */
+
+
+/* **************************************************************
+* Error Management
+****************************************************************/
+#define FSE_isError ERR_isError
+
+
+/* **************************************************************
+* Templates
+****************************************************************/
+/*
+ designed to be included
+ for type-specific functions (template emulation in C)
+ Objective is to write these functions only once, for improved maintenance
+*/
+
+/* safety checks */
+#ifndef FSE_FUNCTION_EXTENSION
+# error "FSE_FUNCTION_EXTENSION must be defined"
+#endif
+#ifndef FSE_FUNCTION_TYPE
+# error "FSE_FUNCTION_TYPE must be defined"
+#endif
+
+/* Function names */
+#define FSE_CAT(X,Y) X##Y
+#define FSE_FUNCTION_NAME(X,Y) FSE_CAT(X,Y)
+#define FSE_TYPE_NAME(X,Y) FSE_CAT(X,Y)
+
+
+/* Function templates */
+
+/* FSE_buildCTable_wksp() :
+ * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
+ * wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
+ * workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
+ */
+size_t FSE_buildCTable_wksp(FSE_CTable* ct,
+ const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
+ void* workSpace, size_t wkspSize)
+{
+ U32 const tableSize = 1 << tableLog;
+ U32 const tableMask = tableSize - 1;
+ void* const ptr = ct;
+ U16* const tableU16 = ( (U16*) ptr) + 2;
+ void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableLog ? tableSize>>1 : 1) ;
+ FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
+ U32 const step = FSE_TABLESTEP(tableSize);
+
+ U32* cumul = (U32*)workSpace;
+ FSE_FUNCTION_TYPE* tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSymbolValue + 2));
+
+ U32 highThreshold = tableSize-1;
+
+ if ((size_t)workSpace & 3) return ERROR(GENERIC); /* Must be 4 byte aligned */
+ if (FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) > wkspSize) return ERROR(tableLog_tooLarge);
+ /* CTable header */
+ tableU16[-2] = (U16) tableLog;
+ tableU16[-1] = (U16) maxSymbolValue;
+ assert(tableLog < 16); /* required for threshold strategy to work */
+
+ /* For explanations on how to distribute symbol values over the table :
+ * http://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */
+
+ #ifdef __clang_analyzer__
+ ZSTD_memset(tableSymbol, 0, sizeof(*tableSymbol) * tableSize); /* useless initialization, just to keep scan-build happy */
+ #endif
+
+ /* symbol start positions */
+ { U32 u;
+ cumul[0] = 0;
+ for (u=1; u <= maxSymbolValue+1; u++) {
+ if (normalizedCounter[u-1]==-1) { /* Low proba symbol */
+ cumul[u] = cumul[u-1] + 1;
+ tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
+ } else {
+ cumul[u] = cumul[u-1] + normalizedCounter[u-1];
+ } }
+ cumul[maxSymbolValue+1] = tableSize+1;
+ }
+
+ /* Spread symbols */
+ { U32 position = 0;
+ U32 symbol;
+ for (symbol=0; symbol<=maxSymbolValue; symbol++) {
+ int nbOccurrences;
+ int const freq = normalizedCounter[symbol];
+ for (nbOccurrences=0; nbOccurrences<freq; nbOccurrences++) {
+ tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
+ position = (position + step) & tableMask;
+ while (position > highThreshold)
+ position = (position + step) & tableMask; /* Low proba area */
+ } }
+
+ assert(position==0); /* Must have initialized all positions */
+ }
+
+ /* Build table */
+ { U32 u; for (u=0; u<tableSize; u++) {
+ FSE_FUNCTION_TYPE s = tableSymbol[u]; /* note : static analyzer may not understand tableSymbol is properly initialized */
+ tableU16[cumul[s]++] = (U16) (tableSize+u); /* TableU16 : sorted by symbol order; gives next state value */
+ } }
+
+ /* Build Symbol Transformation Table */
+ { unsigned total = 0;
+ unsigned s;
+ for (s=0; s<=maxSymbolValue; s++) {
+ switch (normalizedCounter[s])
+ {
+ case 0:
+ /* filling nonetheless, for compatibility with FSE_getMaxNbBits() */
+ symbolTT[s].deltaNbBits = ((tableLog+1) << 16) - (1<<tableLog);
+ break;
+
+ case -1:
+ case 1:
+ symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
+ symbolTT[s].deltaFindState = total - 1;
+ total ++;
+ break;
+ default :
+ {
+ U32 const maxBitsOut = tableLog - BIT_highbit32 (normalizedCounter[s]-1);
+ U32 const minStatePlus = normalizedCounter[s] << maxBitsOut;
+ symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
+ symbolTT[s].deltaFindState = total - normalizedCounter[s];
+ total += normalizedCounter[s];
+ } } } }
+
+#if 0 /* debug : symbol costs */
+ DEBUGLOG(5, "\n --- table statistics : ");
+ { U32 symbol;
+ for (symbol=0; symbol<=maxSymbolValue; symbol++) {
+ DEBUGLOG(5, "%3u: w=%3i, maxBits=%u, fracBits=%.2f",
+ symbol, normalizedCounter[symbol],
+ FSE_getMaxNbBits(symbolTT, symbol),
+ (double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256);
+ }
+ }
+#endif
+
+ return 0;
+}
+
+
+
+
+#ifndef FSE_COMMONDEFS_ONLY
+
+
+/*-**************************************************************
+* FSE NCount encoding
+****************************************************************/
+size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
+{
+ size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog) >> 3) + 3;
+ return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */
+}
+
+static size_t
+FSE_writeNCount_generic (void* header, size_t headerBufferSize,
+ const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
+ unsigned writeIsSafe)
+{
+ BYTE* const ostart = (BYTE*) header;
+ BYTE* out = ostart;
+ BYTE* const oend = ostart + headerBufferSize;
+ int nbBits;
+ const int tableSize = 1 << tableLog;
+ int remaining;
+ int threshold;
+ U32 bitStream = 0;
+ int bitCount = 0;
+ unsigned symbol = 0;
+ unsigned const alphabetSize = maxSymbolValue + 1;
+ int previousIs0 = 0;
+
+ /* Table Size */
+ bitStream += (tableLog-FSE_MIN_TABLELOG) << bitCount;
+ bitCount += 4;
+
+ /* Init */
+ remaining = tableSize+1; /* +1 for extra accuracy */
+ threshold = tableSize;
+ nbBits = tableLog+1;
+
+ while ((symbol < alphabetSize) && (remaining>1)) { /* stops at 1 */
+ if (previousIs0) {
+ unsigned start = symbol;
+ while ((symbol < alphabetSize) && !normalizedCounter[symbol]) symbol++;
+ if (symbol == alphabetSize) break; /* incorrect distribution */
+ while (symbol >= start+24) {
+ start+=24;
+ bitStream += 0xFFFFU << bitCount;
+ if ((!writeIsSafe) && (out > oend-2))
+ return ERROR(dstSize_tooSmall); /* Buffer overflow */
+ out[0] = (BYTE) bitStream;
+ out[1] = (BYTE)(bitStream>>8);
+ out+=2;
+ bitStream>>=16;
+ }
+ while (symbol >= start+3) {
+ start+=3;
+ bitStream += 3 << bitCount;
+ bitCount += 2;
+ }
+ bitStream += (symbol-start) << bitCount;
+ bitCount += 2;
+ if (bitCount>16) {
+ if ((!writeIsSafe) && (out > oend - 2))
+ return ERROR(dstSize_tooSmall); /* Buffer overflow */
+ out[0] = (BYTE)bitStream;
+ out[1] = (BYTE)(bitStream>>8);
+ out += 2;
+ bitStream >>= 16;
+ bitCount -= 16;
+ } }
+ { int count = normalizedCounter[symbol++];
+ int const max = (2*threshold-1) - remaining;
+ remaining -= count < 0 ? -count : count;
+ count++; /* +1 for extra accuracy */
+ if (count>=threshold)
+ count += max; /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */
+ bitStream += count << bitCount;
+ bitCount += nbBits;
+ bitCount -= (count<max);
+ previousIs0 = (count==1);
+ if (remaining<1) return ERROR(GENERIC);
+ while (remaining<threshold) { nbBits--; threshold>>=1; }
+ }
+ if (bitCount>16) {
+ if ((!writeIsSafe) && (out > oend - 2))
+ return ERROR(dstSize_tooSmall); /* Buffer overflow */
+ out[0] = (BYTE)bitStream;
+ out[1] = (BYTE)(bitStream>>8);
+ out += 2;
+ bitStream >>= 16;
+ bitCount -= 16;
+ } }
+
+ if (remaining != 1)
+ return ERROR(GENERIC); /* incorrect normalized distribution */
+ assert(symbol <= alphabetSize);
+
+ /* flush remaining bitStream */
+ if ((!writeIsSafe) && (out > oend - 2))
+ return ERROR(dstSize_tooSmall); /* Buffer overflow */
+ out[0] = (BYTE)bitStream;
+ out[1] = (BYTE)(bitStream>>8);
+ out+= (bitCount+7) /8;
+
+ return (out-ostart);
+}
+
+
+size_t FSE_writeNCount (void* buffer, size_t bufferSize,
+ const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
+{
+ if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported */
+ if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported */
+
+ if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
+ return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);
+
+ return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1 /* write in buffer is safe */);
+}
+
+
+/*-**************************************************************
+* FSE Compression Code
+****************************************************************/
+
+FSE_CTable* FSE_createCTable (unsigned maxSymbolValue, unsigned tableLog)
+{
+ size_t size;
+ if (tableLog > FSE_TABLELOG_ABSOLUTE_MAX) tableLog = FSE_TABLELOG_ABSOLUTE_MAX;
+ size = FSE_CTABLE_SIZE_U32 (tableLog, maxSymbolValue) * sizeof(U32);
+ return (FSE_CTable*)ZSTD_malloc(size);
+}
+
+void FSE_freeCTable (FSE_CTable* ct) { ZSTD_free(ct); }
+
+/* provides the minimum logSize to safely represent a distribution */
+static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
+{
+ U32 minBitsSrc = BIT_highbit32((U32)(srcSize)) + 1;
+ U32 minBitsSymbols = BIT_highbit32(maxSymbolValue) + 2;
+ U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
+ assert(srcSize > 1); /* Not supported, RLE should be used instead */
+ return minBits;
+}
+
+unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
+{
+ U32 maxBitsSrc = BIT_highbit32((U32)(srcSize - 1)) - minus;
+ U32 tableLog = maxTableLog;
+ U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
+ assert(srcSize > 1); /* Not supported, RLE should be used instead */
+ if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
+ if (maxBitsSrc < tableLog) tableLog = maxBitsSrc; /* Accuracy can be reduced */
+ if (minBits > tableLog) tableLog = minBits; /* Need a minimum to safely represent all symbol values */
+ if (tableLog < FSE_MIN_TABLELOG) tableLog = FSE_MIN_TABLELOG;
+ if (tableLog > FSE_MAX_TABLELOG) tableLog = FSE_MAX_TABLELOG;
+ return tableLog;
+}
+
+unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
+{
+ return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
+}
+
+/* Secondary normalization method.
+ To be used when primary method fails. */
+
+static size_t FSE_normalizeM2(short* norm, U32 tableLog, const unsigned* count, size_t total, U32 maxSymbolValue, short lowProbCount)
+{
+ short const NOT_YET_ASSIGNED = -2;
+ U32 s;
+ U32 distributed = 0;
+ U32 ToDistribute;
+
+ /* Init */
+ U32 const lowThreshold = (U32)(total >> tableLog);
+ U32 lowOne = (U32)((total * 3) >> (tableLog + 1));
+
+ for (s=0; s<=maxSymbolValue; s++) {
+ if (count[s] == 0) {
+ norm[s]=0;
+ continue;
+ }
+ if (count[s] <= lowThreshold) {
+ norm[s] = lowProbCount;
+ distributed++;
+ total -= count[s];
+ continue;
+ }
+ if (count[s] <= lowOne) {
+ norm[s] = 1;
+ distributed++;
+ total -= count[s];
+ continue;
+ }
+
+ norm[s]=NOT_YET_ASSIGNED;
+ }
+ ToDistribute = (1 << tableLog) - distributed;
+
+ if (ToDistribute == 0)
+ return 0;
+
+ if ((total / ToDistribute) > lowOne) {
+ /* risk of rounding to zero */
+ lowOne = (U32)((total * 3) / (ToDistribute * 2));
+ for (s=0; s<=maxSymbolValue; s++) {
+ if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
+ norm[s] = 1;
+ distributed++;
+ total -= count[s];
+ continue;
+ } }
+ ToDistribute = (1 << tableLog) - distributed;
+ }
+
+ if (distributed == maxSymbolValue+1) {
+ /* all values are pretty poor;
+ probably incompressible data (should have already been detected);
+ find max, then give all remaining points to max */
+ U32 maxV = 0, maxC = 0;
+ for (s=0; s<=maxSymbolValue; s++)
+ if (count[s] > maxC) { maxV=s; maxC=count[s]; }
+ norm[maxV] += (short)ToDistribute;
+ return 0;
+ }
+
+ if (total == 0) {
+ /* all of the symbols were low enough for the lowOne or lowThreshold */
+ for (s=0; ToDistribute > 0; s = (s+1)%(maxSymbolValue+1))
+ if (norm[s] > 0) { ToDistribute--; norm[s]++; }
+ return 0;
+ }
+
+ { U64 const vStepLog = 62 - tableLog;
+ U64 const mid = (1ULL << (vStepLog-1)) - 1;
+ U64 const rStep = ZSTD_div64((((U64)1<<vStepLog) * ToDistribute) + mid, (U32)total); /* scale on remaining */
+ U64 tmpTotal = mid;
+ for (s=0; s<=maxSymbolValue; s++) {
+ if (norm[s]==NOT_YET_ASSIGNED) {
+ U64 const end = tmpTotal + (count[s] * rStep);
+ U32 const sStart = (U32)(tmpTotal >> vStepLog);
+ U32 const sEnd = (U32)(end >> vStepLog);
+ U32 const weight = sEnd - sStart;
+ if (weight < 1)
+ return ERROR(GENERIC);
+ norm[s] = (short)weight;
+ tmpTotal = end;
+ } } }
+
+ return 0;
+}
+
+size_t FSE_normalizeCount (short* normalizedCounter, unsigned tableLog,
+ const unsigned* count, size_t total,
+ unsigned maxSymbolValue, unsigned useLowProbCount)
+{
+ /* Sanity checks */
+ if (tableLog==0) tableLog = FSE_DEFAULT_TABLELOG;
+ if (tableLog < FSE_MIN_TABLELOG) return ERROR(GENERIC); /* Unsupported size */
+ if (tableLog > FSE_MAX_TABLELOG) return ERROR(tableLog_tooLarge); /* Unsupported size */
+ if (tableLog < FSE_minTableLog(total, maxSymbolValue)) return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */
+
+ { static U32 const rtbTable[] = { 0, 473195, 504333, 520860, 550000, 700000, 750000, 830000 };
+ short const lowProbCount = useLowProbCount ? -1 : 1;
+ U64 const scale = 62 - tableLog;
+ U64 const step = ZSTD_div64((U64)1<<62, (U32)total); /* <== here, one division ! */
+ U64 const vStep = 1ULL<<(scale-20);
+ int stillToDistribute = 1<<tableLog;
+ unsigned s;
+ unsigned largest=0;
+ short largestP=0;
+ U32 lowThreshold = (U32)(total >> tableLog);
+
+ for (s=0; s<=maxSymbolValue; s++) {
+ if (count[s] == total) return 0; /* rle special case */
+ if (count[s] == 0) { normalizedCounter[s]=0; continue; }
+ if (count[s] <= lowThreshold) {
+ normalizedCounter[s] = lowProbCount;
+ stillToDistribute--;
+ } else {
+ short proba = (short)((count[s]*step) >> scale);
+ if (proba<8) {
+ U64 restToBeat = vStep * rtbTable[proba];
+ proba += (count[s]*step) - ((U64)proba<<scale) > restToBeat;
+ }
+ if (proba > largestP) { largestP=proba; largest=s; }
+ normalizedCounter[s] = proba;
+ stillToDistribute -= proba;
+ } }
+ if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
+ /* corner case, need another normalization method */
+ size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue, lowProbCount);
+ if (FSE_isError(errorCode)) return errorCode;
+ }
+ else normalizedCounter[largest] += (short)stillToDistribute;
+ }
+
+#if 0
+ { /* Print Table (debug) */
+ U32 s;
+ U32 nTotal = 0;
+ for (s=0; s<=maxSymbolValue; s++)
+ RAWLOG(2, "%3i: %4i \n", s, normalizedCounter[s]);
+ for (s=0; s<=maxSymbolValue; s++)
+ nTotal += abs(normalizedCounter[s]);
+ if (nTotal != (1U<<tableLog))
+ RAWLOG(2, "Warning !!! Total == %u != %u !!!", nTotal, 1U<<tableLog);
+ getchar();
+ }
+#endif
+
+ return tableLog;
+}
+
+
+/* fake FSE_CTable, for raw (uncompressed) input */
+size_t FSE_buildCTable_raw (FSE_CTable* ct, unsigned nbBits)
+{
+ const unsigned tableSize = 1 << nbBits;
+ const unsigned tableMask = tableSize - 1;
+ const unsigned maxSymbolValue = tableMask;
+ void* const ptr = ct;
+ U16* const tableU16 = ( (U16*) ptr) + 2;
+ void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableSize>>1); /* assumption : tableLog >= 1 */
+ FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
+ unsigned s;
+
+ /* Sanity checks */
+ if (nbBits < 1) return ERROR(GENERIC); /* min size */
+
+ /* header */
+ tableU16[-2] = (U16) nbBits;
+ tableU16[-1] = (U16) maxSymbolValue;
+
+ /* Build table */
+ for (s=0; s<tableSize; s++)
+ tableU16[s] = (U16)(tableSize + s);
+
+ /* Build Symbol Transformation Table */
+ { const U32 deltaNbBits = (nbBits << 16) - (1 << nbBits);
+ for (s=0; s<=maxSymbolValue; s++) {
+ symbolTT[s].deltaNbBits = deltaNbBits;
+ symbolTT[s].deltaFindState = s-1;
+ } }
+
+ return 0;
+}
+
+/* fake FSE_CTable, for rle input (always same symbol) */
+size_t FSE_buildCTable_rle (FSE_CTable* ct, BYTE symbolValue)
+{
+ void* ptr = ct;
+ U16* tableU16 = ( (U16*) ptr) + 2;
+ void* FSCTptr = (U32*)ptr + 2;
+ FSE_symbolCompressionTransform* symbolTT = (FSE_symbolCompressionTransform*) FSCTptr;
+
+ /* header */
+ tableU16[-2] = (U16) 0;
+ tableU16[-1] = (U16) symbolValue;
+
+ /* Build table */
+ tableU16[0] = 0;
+ tableU16[1] = 0; /* just in case */
+
+ /* Build Symbol Transformation Table */
+ symbolTT[symbolValue].deltaNbBits = 0;
+ symbolTT[symbolValue].deltaFindState = 0;
+
+ return 0;
+}
+
+
+static size_t FSE_compress_usingCTable_generic (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const FSE_CTable* ct, const unsigned fast)
+{
+ const BYTE* const istart = (const BYTE*) src;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* ip=iend;
+
+ BIT_CStream_t bitC;
+ FSE_CState_t CState1, CState2;
+
+ /* init */
+ if (srcSize <= 2) return 0;
+ { size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
+ if (FSE_isError(initError)) return 0; /* not enough space available to write a bitstream */ }
+
+#define FSE_FLUSHBITS(s) (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))
+
+ if (srcSize & 1) {
+ FSE_initCState2(&CState1, ct, *--ip);
+ FSE_initCState2(&CState2, ct, *--ip);
+ FSE_encodeSymbol(&bitC, &CState1, *--ip);
+ FSE_FLUSHBITS(&bitC);
+ } else {
+ FSE_initCState2(&CState2, ct, *--ip);
+ FSE_initCState2(&CState1, ct, *--ip);
+ }
+
+ /* join to mod 4 */
+ srcSize -= 2;
+ if ((sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) && (srcSize & 2)) { /* test bit 2 */
+ FSE_encodeSymbol(&bitC, &CState2, *--ip);
+ FSE_encodeSymbol(&bitC, &CState1, *--ip);
+ FSE_FLUSHBITS(&bitC);
+ }
+
+ /* 2 or 4 encoding per loop */
+ while ( ip>istart ) {
+
+ FSE_encodeSymbol(&bitC, &CState2, *--ip);
+
+ if (sizeof(bitC.bitContainer)*8 < FSE_MAX_TABLELOG*2+7 ) /* this test must be static */
+ FSE_FLUSHBITS(&bitC);
+
+ FSE_encodeSymbol(&bitC, &CState1, *--ip);
+
+ if (sizeof(bitC.bitContainer)*8 > FSE_MAX_TABLELOG*4+7 ) { /* this test must be static */
+ FSE_encodeSymbol(&bitC, &CState2, *--ip);
+ FSE_encodeSymbol(&bitC, &CState1, *--ip);
+ }
+
+ FSE_FLUSHBITS(&bitC);
+ }
+
+ FSE_flushCState(&bitC, &CState2);
+ FSE_flushCState(&bitC, &CState1);
+ return BIT_closeCStream(&bitC);
+}
+
+size_t FSE_compress_usingCTable (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const FSE_CTable* ct)
+{
+ unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));
+
+ if (fast)
+ return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
+ else
+ return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
+}
+
+
+size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
+
+
+#endif /* FSE_COMMONDEFS_ONLY */
--- /dev/null
+/* ******************************************************************
+ * hist : Histogram functions
+ * part of Finite State Entropy project
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* --- dependencies --- */
+#include "../common/mem.h" /* U32, BYTE, etc. */
+#include "../common/debug.h" /* assert, DEBUGLOG */
+#include "../common/error_private.h" /* ERROR */
+#include "hist.h"
+
+
+/* --- Error management --- */
+unsigned HIST_isError(size_t code) { return ERR_isError(code); }
+
+/*-**************************************************************
+ * Histogram functions
+ ****************************************************************/
+unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize)
+{
+ const BYTE* ip = (const BYTE*)src;
+ const BYTE* const end = ip + srcSize;
+ unsigned maxSymbolValue = *maxSymbolValuePtr;
+ unsigned largestCount=0;
+
+ ZSTD_memset(count, 0, (maxSymbolValue+1) * sizeof(*count));
+ if (srcSize==0) { *maxSymbolValuePtr = 0; return 0; }
+
+ while (ip<end) {
+ assert(*ip <= maxSymbolValue);
+ count[*ip++]++;
+ }
+
+ while (!count[maxSymbolValue]) maxSymbolValue--;
+ *maxSymbolValuePtr = maxSymbolValue;
+
+ { U32 s;
+ for (s=0; s<=maxSymbolValue; s++)
+ if (count[s] > largestCount) largestCount = count[s];
+ }
+
+ return largestCount;
+}
+
+typedef enum { trustInput, checkMaxSymbolValue } HIST_checkInput_e;
+
+/* HIST_count_parallel_wksp() :
+ * store histogram into 4 intermediate tables, recombined at the end.
+ * this design makes better use of OoO cpus,
+ * and is noticeably faster when some values are heavily repeated.
+ * But it needs some additional workspace for intermediate tables.
+ * `workSpace` must be a U32 table of size >= HIST_WKSP_SIZE_U32.
+ * @return : largest histogram frequency,
+ * or an error code (notably when histogram's alphabet is larger than *maxSymbolValuePtr) */
+static size_t HIST_count_parallel_wksp(
+ unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* source, size_t sourceSize,
+ HIST_checkInput_e check,
+ U32* const workSpace)
+{
+ const BYTE* ip = (const BYTE*)source;
+ const BYTE* const iend = ip+sourceSize;
+ size_t const countSize = (*maxSymbolValuePtr + 1) * sizeof(*count);
+ unsigned max=0;
+ U32* const Counting1 = workSpace;
+ U32* const Counting2 = Counting1 + 256;
+ U32* const Counting3 = Counting2 + 256;
+ U32* const Counting4 = Counting3 + 256;
+
+ /* safety checks */
+ assert(*maxSymbolValuePtr <= 255);
+ if (!sourceSize) {
+ ZSTD_memset(count, 0, countSize);
+ *maxSymbolValuePtr = 0;
+ return 0;
+ }
+ ZSTD_memset(workSpace, 0, 4*256*sizeof(unsigned));
+
+ /* by stripes of 16 bytes */
+ { U32 cached = MEM_read32(ip); ip += 4;
+ while (ip < iend-15) {
+ U32 c = cached; cached = MEM_read32(ip); ip += 4;
+ Counting1[(BYTE) c ]++;
+ Counting2[(BYTE)(c>>8) ]++;
+ Counting3[(BYTE)(c>>16)]++;
+ Counting4[ c>>24 ]++;
+ c = cached; cached = MEM_read32(ip); ip += 4;
+ Counting1[(BYTE) c ]++;
+ Counting2[(BYTE)(c>>8) ]++;
+ Counting3[(BYTE)(c>>16)]++;
+ Counting4[ c>>24 ]++;
+ c = cached; cached = MEM_read32(ip); ip += 4;
+ Counting1[(BYTE) c ]++;
+ Counting2[(BYTE)(c>>8) ]++;
+ Counting3[(BYTE)(c>>16)]++;
+ Counting4[ c>>24 ]++;
+ c = cached; cached = MEM_read32(ip); ip += 4;
+ Counting1[(BYTE) c ]++;
+ Counting2[(BYTE)(c>>8) ]++;
+ Counting3[(BYTE)(c>>16)]++;
+ Counting4[ c>>24 ]++;
+ }
+ ip-=4;
+ }
+
+ /* finish last symbols */
+ while (ip<iend) Counting1[*ip++]++;
+
+ { U32 s;
+ for (s=0; s<256; s++) {
+ Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
+ if (Counting1[s] > max) max = Counting1[s];
+ } }
+
+ { unsigned maxSymbolValue = 255;
+ while (!Counting1[maxSymbolValue]) maxSymbolValue--;
+ if (check && maxSymbolValue > *maxSymbolValuePtr) return ERROR(maxSymbolValue_tooSmall);
+ *maxSymbolValuePtr = maxSymbolValue;
+ ZSTD_memmove(count, Counting1, countSize); /* in case count & Counting1 are overlapping */
+ }
+ return (size_t)max;
+}
+
+/* HIST_countFast_wksp() :
+ * Same as HIST_countFast(), but using an externally provided scratch buffer.
+ * `workSpace` is a writable buffer which must be 4-bytes aligned,
+ * `workSpaceSize` must be >= HIST_WKSP_SIZE
+ */
+size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* source, size_t sourceSize,
+ void* workSpace, size_t workSpaceSize)
+{
+ if (sourceSize < 1500) /* heuristic threshold */
+ return HIST_count_simple(count, maxSymbolValuePtr, source, sourceSize);
+ if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
+ if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
+ return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, trustInput, (U32*)workSpace);
+}
+
+/* HIST_count_wksp() :
+ * Same as HIST_count(), but using an externally provided scratch buffer.
+ * `workSpace` size must be table of >= HIST_WKSP_SIZE_U32 unsigned */
+size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* source, size_t sourceSize,
+ void* workSpace, size_t workSpaceSize)
+{
+ if ((size_t)workSpace & 3) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
+ if (workSpaceSize < HIST_WKSP_SIZE) return ERROR(workSpace_tooSmall);
+ if (*maxSymbolValuePtr < 255)
+ return HIST_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, checkMaxSymbolValue, (U32*)workSpace);
+ *maxSymbolValuePtr = 255;
+ return HIST_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace, workSpaceSize);
+}
+
--- /dev/null
+/* ******************************************************************
+ * hist : Histogram functions
+ * part of Finite State Entropy project
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* --- dependencies --- */
+#include "../common/zstd_deps.h" /* size_t */
+
+
+/* --- simple histogram functions --- */
+
+/*! HIST_count():
+ * Provides the precise count of each byte within a table 'count'.
+ * 'count' is a table of unsigned int, of minimum size (*maxSymbolValuePtr+1).
+ * Updates *maxSymbolValuePtr with actual largest symbol value detected.
+ * @return : count of the most frequent symbol (which isn't identified).
+ * or an error code, which can be tested using HIST_isError().
+ * note : if return == srcSize, there is only one symbol.
+ */
+size_t HIST_count(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize);
+
+unsigned HIST_isError(size_t code); /*< tells if a return value is an error code */
+
+
+/* --- advanced histogram functions --- */
+
+#define HIST_WKSP_SIZE_U32 1024
+#define HIST_WKSP_SIZE (HIST_WKSP_SIZE_U32 * sizeof(unsigned))
+/* HIST_count_wksp() :
+ * Same as HIST_count(), but using an externally provided scratch buffer.
+ * Benefit is this function will use very little stack space.
+ * `workSpace` is a writable buffer which must be 4-bytes aligned,
+ * `workSpaceSize` must be >= HIST_WKSP_SIZE
+ */
+size_t HIST_count_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t workSpaceSize);
+
+/* HIST_countFast() :
+ * same as HIST_count(), but blindly trusts that all byte values within src are <= *maxSymbolValuePtr.
+ * This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr`
+ */
+size_t HIST_countFast(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize);
+
+/* HIST_countFast_wksp() :
+ * Same as HIST_countFast(), but using an externally provided scratch buffer.
+ * `workSpace` is a writable buffer which must be 4-bytes aligned,
+ * `workSpaceSize` must be >= HIST_WKSP_SIZE
+ */
+size_t HIST_countFast_wksp(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t workSpaceSize);
+
+/*! HIST_count_simple() :
+ * Same as HIST_countFast(), this function is unsafe,
+ * and will segfault if any value within `src` is `> *maxSymbolValuePtr`.
+ * It is also a bit slower for large inputs.
+ * However, it does not need any additional memory (not even on stack).
+ * @return : count of the most frequent symbol.
+ * Note this function doesn't produce any error (i.e. it must succeed).
+ */
+unsigned HIST_count_simple(unsigned* count, unsigned* maxSymbolValuePtr,
+ const void* src, size_t srcSize);
--- /dev/null
+/* ******************************************************************
+ * Huffman encoder, part of New Generation Entropy library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ * - Public forum : https://groups.google.com/forum/#!forum/lz4c
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* **************************************************************
+* Compiler specifics
+****************************************************************/
+
+
+/* **************************************************************
+* Includes
+****************************************************************/
+#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memset */
+#include "../common/compiler.h"
+#include "../common/bitstream.h"
+#include "hist.h"
+#define FSE_STATIC_LINKING_ONLY /* FSE_optimalTableLog_internal */
+#include "../common/fse.h" /* header compression */
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include "../common/error_private.h"
+
+
+/* **************************************************************
+* Error Management
+****************************************************************/
+#define HUF_isError ERR_isError
+#define HUF_STATIC_ASSERT(c) DEBUG_STATIC_ASSERT(c) /* use only *after* variable declarations */
+
+
+/* **************************************************************
+* Utils
+****************************************************************/
+unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
+{
+ return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1);
+}
+
+
+/* *******************************************************
+* HUF : Huffman block compression
+*********************************************************/
+/* HUF_compressWeights() :
+ * Same as FSE_compress(), but dedicated to huff0's weights compression.
+ * The use case needs much less stack memory.
+ * Note : all elements within weightTable are supposed to be <= HUF_TABLELOG_MAX.
+ */
+#define MAX_FSE_TABLELOG_FOR_HUFF_HEADER 6
+
+typedef struct {
+ FSE_CTable CTable[FSE_CTABLE_SIZE_U32(MAX_FSE_TABLELOG_FOR_HUFF_HEADER, HUF_TABLELOG_MAX)];
+ U32 scratchBuffer[FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(HUF_TABLELOG_MAX, MAX_FSE_TABLELOG_FOR_HUFF_HEADER)];
+ unsigned count[HUF_TABLELOG_MAX+1];
+ S16 norm[HUF_TABLELOG_MAX+1];
+} HUF_CompressWeightsWksp;
+
+static size_t HUF_compressWeights(void* dst, size_t dstSize, const void* weightTable, size_t wtSize, void* workspace, size_t workspaceSize)
+{
+ BYTE* const ostart = (BYTE*) dst;
+ BYTE* op = ostart;
+ BYTE* const oend = ostart + dstSize;
+
+ unsigned maxSymbolValue = HUF_TABLELOG_MAX;
+ U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER;
+ HUF_CompressWeightsWksp* wksp = (HUF_CompressWeightsWksp*)workspace;
+
+ if (workspaceSize < sizeof(HUF_CompressWeightsWksp)) return ERROR(GENERIC);
+
+ /* init conditions */
+ if (wtSize <= 1) return 0; /* Not compressible */
+
+ /* Scan input and build symbol stats */
+ { unsigned const maxCount = HIST_count_simple(wksp->count, &maxSymbolValue, weightTable, wtSize); /* never fails */
+ if (maxCount == wtSize) return 1; /* only a single symbol in src : rle */
+ if (maxCount == 1) return 0; /* each symbol present maximum once => not compressible */
+ }
+
+ tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue);
+ CHECK_F( FSE_normalizeCount(wksp->norm, tableLog, wksp->count, wtSize, maxSymbolValue, /* useLowProbCount */ 0) );
+
+ /* Write table description header */
+ { CHECK_V_F(hSize, FSE_writeNCount(op, (size_t)(oend-op), wksp->norm, maxSymbolValue, tableLog) );
+ op += hSize;
+ }
+
+ /* Compress */
+ CHECK_F( FSE_buildCTable_wksp(wksp->CTable, wksp->norm, maxSymbolValue, tableLog, wksp->scratchBuffer, sizeof(wksp->scratchBuffer)) );
+ { CHECK_V_F(cSize, FSE_compress_usingCTable(op, (size_t)(oend - op), weightTable, wtSize, wksp->CTable) );
+ if (cSize == 0) return 0; /* not enough space for compressed data */
+ op += cSize;
+ }
+
+ return (size_t)(op-ostart);
+}
+
+
+typedef struct {
+ HUF_CompressWeightsWksp wksp;
+ BYTE bitsToWeight[HUF_TABLELOG_MAX + 1]; /* precomputed conversion table */
+ BYTE huffWeight[HUF_SYMBOLVALUE_MAX];
+} HUF_WriteCTableWksp;
+
+size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize,
+ const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog,
+ void* workspace, size_t workspaceSize)
+{
+ BYTE* op = (BYTE*)dst;
+ U32 n;
+ HUF_WriteCTableWksp* wksp = (HUF_WriteCTableWksp*)workspace;
+
+ /* check conditions */
+ if (workspaceSize < sizeof(HUF_WriteCTableWksp)) return ERROR(GENERIC);
+ if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
+
+ /* convert to weight */
+ wksp->bitsToWeight[0] = 0;
+ for (n=1; n<huffLog+1; n++)
+ wksp->bitsToWeight[n] = (BYTE)(huffLog + 1 - n);
+ for (n=0; n<maxSymbolValue; n++)
+ wksp->huffWeight[n] = wksp->bitsToWeight[CTable[n].nbBits];
+
+ /* attempt weights compression by FSE */
+ { CHECK_V_F(hSize, HUF_compressWeights(op+1, maxDstSize-1, wksp->huffWeight, maxSymbolValue, &wksp->wksp, sizeof(wksp->wksp)) );
+ if ((hSize>1) & (hSize < maxSymbolValue/2)) { /* FSE compressed */
+ op[0] = (BYTE)hSize;
+ return hSize+1;
+ } }
+
+ /* write raw values as 4-bits (max : 15) */
+ if (maxSymbolValue > (256-128)) return ERROR(GENERIC); /* should not happen : likely means source cannot be compressed */
+ if (((maxSymbolValue+1)/2) + 1 > maxDstSize) return ERROR(dstSize_tooSmall); /* not enough space within dst buffer */
+ op[0] = (BYTE)(128 /*special case*/ + (maxSymbolValue-1));
+ wksp->huffWeight[maxSymbolValue] = 0; /* to be sure it doesn't cause msan issue in final combination */
+ for (n=0; n<maxSymbolValue; n+=2)
+ op[(n/2)+1] = (BYTE)((wksp->huffWeight[n] << 4) + wksp->huffWeight[n+1]);
+ return ((maxSymbolValue+1)/2) + 1;
+}
+
+/*! HUF_writeCTable() :
+ `CTable` : Huffman tree to save, using huf representation.
+ @return : size of saved CTable */
+size_t HUF_writeCTable (void* dst, size_t maxDstSize,
+ const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog)
+{
+ HUF_WriteCTableWksp wksp;
+ return HUF_writeCTable_wksp(dst, maxDstSize, CTable, maxSymbolValue, huffLog, &wksp, sizeof(wksp));
+}
+
+
+size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned* hasZeroWeights)
+{
+ BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1]; /* init not required, even though some static analyzer may complain */
+ U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; /* large enough for values from 0 to 16 */
+ U32 tableLog = 0;
+ U32 nbSymbols = 0;
+
+ /* get symbol weights */
+ CHECK_V_F(readSize, HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX+1, rankVal, &nbSymbols, &tableLog, src, srcSize));
+ *hasZeroWeights = (rankVal[0] > 0);
+
+ /* check result */
+ if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
+ if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall);
+
+ /* Prepare base value per rank */
+ { U32 n, nextRankStart = 0;
+ for (n=1; n<=tableLog; n++) {
+ U32 curr = nextRankStart;
+ nextRankStart += (rankVal[n] << (n-1));
+ rankVal[n] = curr;
+ } }
+
+ /* fill nbBits */
+ { U32 n; for (n=0; n<nbSymbols; n++) {
+ const U32 w = huffWeight[n];
+ CTable[n].nbBits = (BYTE)(tableLog + 1 - w) & -(w != 0);
+ } }
+
+ /* fill val */
+ { U16 nbPerRank[HUF_TABLELOG_MAX+2] = {0}; /* support w=0=>n=tableLog+1 */
+ U16 valPerRank[HUF_TABLELOG_MAX+2] = {0};
+ { U32 n; for (n=0; n<nbSymbols; n++) nbPerRank[CTable[n].nbBits]++; }
+ /* determine stating value per rank */
+ valPerRank[tableLog+1] = 0; /* for w==0 */
+ { U16 min = 0;
+ U32 n; for (n=tableLog; n>0; n--) { /* start at n=tablelog <-> w=1 */
+ valPerRank[n] = min; /* get starting value within each rank */
+ min += nbPerRank[n];
+ min >>= 1;
+ } }
+ /* assign value within rank, symbol order */
+ { U32 n; for (n=0; n<nbSymbols; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
+ }
+
+ *maxSymbolValuePtr = nbSymbols - 1;
+ return readSize;
+}
+
+U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue)
+{
+ const HUF_CElt* table = (const HUF_CElt*)symbolTable;
+ assert(symbolValue <= HUF_SYMBOLVALUE_MAX);
+ return table[symbolValue].nbBits;
+}
+
+
+typedef struct nodeElt_s {
+ U32 count;
+ U16 parent;
+ BYTE byte;
+ BYTE nbBits;
+} nodeElt;
+
+/*
+ * HUF_setMaxHeight():
+ * Enforces maxNbBits on the Huffman tree described in huffNode.
+ *
+ * It sets all nodes with nbBits > maxNbBits to be maxNbBits. Then it adjusts
+ * the tree to so that it is a valid canonical Huffman tree.
+ *
+ * @pre The sum of the ranks of each symbol == 2^largestBits,
+ * where largestBits == huffNode[lastNonNull].nbBits.
+ * @post The sum of the ranks of each symbol == 2^largestBits,
+ * where largestBits is the return value <= maxNbBits.
+ *
+ * @param huffNode The Huffman tree modified in place to enforce maxNbBits.
+ * @param lastNonNull The symbol with the lowest count in the Huffman tree.
+ * @param maxNbBits The maximum allowed number of bits, which the Huffman tree
+ * may not respect. After this function the Huffman tree will
+ * respect maxNbBits.
+ * @return The maximum number of bits of the Huffman tree after adjustment,
+ * necessarily no more than maxNbBits.
+ */
+static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
+{
+ const U32 largestBits = huffNode[lastNonNull].nbBits;
+ /* early exit : no elt > maxNbBits, so the tree is already valid. */
+ if (largestBits <= maxNbBits) return largestBits;
+
+ /* there are several too large elements (at least >= 2) */
+ { int totalCost = 0;
+ const U32 baseCost = 1 << (largestBits - maxNbBits);
+ int n = (int)lastNonNull;
+
+ /* Adjust any ranks > maxNbBits to maxNbBits.
+ * Compute totalCost, which is how far the sum of the ranks is
+ * we are over 2^largestBits after adjust the offending ranks.
+ */
+ while (huffNode[n].nbBits > maxNbBits) {
+ totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits));
+ huffNode[n].nbBits = (BYTE)maxNbBits;
+ n--;
+ }
+ /* n stops at huffNode[n].nbBits <= maxNbBits */
+ assert(huffNode[n].nbBits <= maxNbBits);
+ /* n end at index of smallest symbol using < maxNbBits */
+ while (huffNode[n].nbBits == maxNbBits) --n;
+
+ /* renorm totalCost from 2^largestBits to 2^maxNbBits
+ * note : totalCost is necessarily a multiple of baseCost */
+ assert((totalCost & (baseCost - 1)) == 0);
+ totalCost >>= (largestBits - maxNbBits);
+ assert(totalCost > 0);
+
+ /* repay normalized cost */
+ { U32 const noSymbol = 0xF0F0F0F0;
+ U32 rankLast[HUF_TABLELOG_MAX+2];
+
+ /* Get pos of last (smallest = lowest cum. count) symbol per rank */
+ ZSTD_memset(rankLast, 0xF0, sizeof(rankLast));
+ { U32 currentNbBits = maxNbBits;
+ int pos;
+ for (pos=n ; pos >= 0; pos--) {
+ if (huffNode[pos].nbBits >= currentNbBits) continue;
+ currentNbBits = huffNode[pos].nbBits; /* < maxNbBits */
+ rankLast[maxNbBits-currentNbBits] = (U32)pos;
+ } }
+
+ while (totalCost > 0) {
+ /* Try to reduce the next power of 2 above totalCost because we
+ * gain back half the rank.
+ */
+ U32 nBitsToDecrease = BIT_highbit32((U32)totalCost) + 1;
+ for ( ; nBitsToDecrease > 1; nBitsToDecrease--) {
+ U32 const highPos = rankLast[nBitsToDecrease];
+ U32 const lowPos = rankLast[nBitsToDecrease-1];
+ if (highPos == noSymbol) continue;
+ /* Decrease highPos if no symbols of lowPos or if it is
+ * not cheaper to remove 2 lowPos than highPos.
+ */
+ if (lowPos == noSymbol) break;
+ { U32 const highTotal = huffNode[highPos].count;
+ U32 const lowTotal = 2 * huffNode[lowPos].count;
+ if (highTotal <= lowTotal) break;
+ } }
+ /* only triggered when no more rank 1 symbol left => find closest one (note : there is necessarily at least one !) */
+ assert(rankLast[nBitsToDecrease] != noSymbol || nBitsToDecrease == 1);
+ /* HUF_MAX_TABLELOG test just to please gcc 5+; but it should not be necessary */
+ while ((nBitsToDecrease<=HUF_TABLELOG_MAX) && (rankLast[nBitsToDecrease] == noSymbol))
+ nBitsToDecrease++;
+ assert(rankLast[nBitsToDecrease] != noSymbol);
+ /* Increase the number of bits to gain back half the rank cost. */
+ totalCost -= 1 << (nBitsToDecrease-1);
+ huffNode[rankLast[nBitsToDecrease]].nbBits++;
+
+ /* Fix up the new rank.
+ * If the new rank was empty, this symbol is now its smallest.
+ * Otherwise, this symbol will be the largest in the new rank so no adjustment.
+ */
+ if (rankLast[nBitsToDecrease-1] == noSymbol)
+ rankLast[nBitsToDecrease-1] = rankLast[nBitsToDecrease];
+ /* Fix up the old rank.
+ * If the symbol was at position 0, meaning it was the highest weight symbol in the tree,
+ * it must be the only symbol in its rank, so the old rank now has no symbols.
+ * Otherwise, since the Huffman nodes are sorted by count, the previous position is now
+ * the smallest node in the rank. If the previous position belongs to a different rank,
+ * then the rank is now empty.
+ */
+ if (rankLast[nBitsToDecrease] == 0) /* special case, reached largest symbol */
+ rankLast[nBitsToDecrease] = noSymbol;
+ else {
+ rankLast[nBitsToDecrease]--;
+ if (huffNode[rankLast[nBitsToDecrease]].nbBits != maxNbBits-nBitsToDecrease)
+ rankLast[nBitsToDecrease] = noSymbol; /* this rank is now empty */
+ }
+ } /* while (totalCost > 0) */
+
+ /* If we've removed too much weight, then we have to add it back.
+ * To avoid overshooting again, we only adjust the smallest rank.
+ * We take the largest nodes from the lowest rank 0 and move them
+ * to rank 1. There's guaranteed to be enough rank 0 symbols because
+ * TODO.
+ */
+ while (totalCost < 0) { /* Sometimes, cost correction overshoot */
+ /* special case : no rank 1 symbol (using maxNbBits-1);
+ * let's create one from largest rank 0 (using maxNbBits).
+ */
+ if (rankLast[1] == noSymbol) {
+ while (huffNode[n].nbBits == maxNbBits) n--;
+ huffNode[n+1].nbBits--;
+ assert(n >= 0);
+ rankLast[1] = (U32)(n+1);
+ totalCost++;
+ continue;
+ }
+ huffNode[ rankLast[1] + 1 ].nbBits--;
+ rankLast[1]++;
+ totalCost ++;
+ }
+ } /* repay normalized cost */
+ } /* there are several too large elements (at least >= 2) */
+
+ return maxNbBits;
+}
+
+typedef struct {
+ U32 base;
+ U32 curr;
+} rankPos;
+
+typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];
+
+#define RANK_POSITION_TABLE_SIZE 32
+
+typedef struct {
+ huffNodeTable huffNodeTbl;
+ rankPos rankPosition[RANK_POSITION_TABLE_SIZE];
+} HUF_buildCTable_wksp_tables;
+
+/*
+ * HUF_sort():
+ * Sorts the symbols [0, maxSymbolValue] by count[symbol] in decreasing order.
+ *
+ * @param[out] huffNode Sorted symbols by decreasing count. Only members `.count` and `.byte` are filled.
+ * Must have (maxSymbolValue + 1) entries.
+ * @param[in] count Histogram of the symbols.
+ * @param[in] maxSymbolValue Maximum symbol value.
+ * @param rankPosition This is a scratch workspace. Must have RANK_POSITION_TABLE_SIZE entries.
+ */
+static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValue, rankPos* rankPosition)
+{
+ int n;
+ int const maxSymbolValue1 = (int)maxSymbolValue + 1;
+
+ /* Compute base and set curr to base.
+ * For symbol s let lowerRank = BIT_highbit32(count[n]+1) and rank = lowerRank + 1.
+ * Then 2^lowerRank <= count[n]+1 <= 2^rank.
+ * We attribute each symbol to lowerRank's base value, because we want to know where
+ * each rank begins in the output, so for rank R we want to count ranks R+1 and above.
+ */
+ ZSTD_memset(rankPosition, 0, sizeof(*rankPosition) * RANK_POSITION_TABLE_SIZE);
+ for (n = 0; n < maxSymbolValue1; ++n) {
+ U32 lowerRank = BIT_highbit32(count[n] + 1);
+ rankPosition[lowerRank].base++;
+ }
+ assert(rankPosition[RANK_POSITION_TABLE_SIZE - 1].base == 0);
+ for (n = RANK_POSITION_TABLE_SIZE - 1; n > 0; --n) {
+ rankPosition[n-1].base += rankPosition[n].base;
+ rankPosition[n-1].curr = rankPosition[n-1].base;
+ }
+ /* Sort */
+ for (n = 0; n < maxSymbolValue1; ++n) {
+ U32 const c = count[n];
+ U32 const r = BIT_highbit32(c+1) + 1;
+ U32 pos = rankPosition[r].curr++;
+ /* Insert into the correct position in the rank.
+ * We have at most 256 symbols, so this insertion should be fine.
+ */
+ while ((pos > rankPosition[r].base) && (c > huffNode[pos-1].count)) {
+ huffNode[pos] = huffNode[pos-1];
+ pos--;
+ }
+ huffNode[pos].count = c;
+ huffNode[pos].byte = (BYTE)n;
+ }
+}
+
+
+/* HUF_buildCTable_wksp() :
+ * Same as HUF_buildCTable(), but using externally allocated scratch buffer.
+ * `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as sizeof(HUF_buildCTable_wksp_tables).
+ */
+#define STARTNODE (HUF_SYMBOLVALUE_MAX+1)
+
+/* HUF_buildTree():
+ * Takes the huffNode array sorted by HUF_sort() and builds an unlimited-depth Huffman tree.
+ *
+ * @param huffNode The array sorted by HUF_sort(). Builds the Huffman tree in this array.
+ * @param maxSymbolValue The maximum symbol value.
+ * @return The smallest node in the Huffman tree (by count).
+ */
+static int HUF_buildTree(nodeElt* huffNode, U32 maxSymbolValue)
+{
+ nodeElt* const huffNode0 = huffNode - 1;
+ int nonNullRank;
+ int lowS, lowN;
+ int nodeNb = STARTNODE;
+ int n, nodeRoot;
+ /* init for parents */
+ nonNullRank = (int)maxSymbolValue;
+ while(huffNode[nonNullRank].count == 0) nonNullRank--;
+ lowS = nonNullRank; nodeRoot = nodeNb + lowS - 1; lowN = nodeNb;
+ huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS-1].count;
+ huffNode[lowS].parent = huffNode[lowS-1].parent = (U16)nodeNb;
+ nodeNb++; lowS-=2;
+ for (n=nodeNb; n<=nodeRoot; n++) huffNode[n].count = (U32)(1U<<30);
+ huffNode0[0].count = (U32)(1U<<31); /* fake entry, strong barrier */
+
+ /* create parents */
+ while (nodeNb <= nodeRoot) {
+ int const n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
+ int const n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
+ huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count;
+ huffNode[n1].parent = huffNode[n2].parent = (U16)nodeNb;
+ nodeNb++;
+ }
+
+ /* distribute weights (unlimited tree height) */
+ huffNode[nodeRoot].nbBits = 0;
+ for (n=nodeRoot-1; n>=STARTNODE; n--)
+ huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
+ for (n=0; n<=nonNullRank; n++)
+ huffNode[n].nbBits = huffNode[ huffNode[n].parent ].nbBits + 1;
+
+ return nonNullRank;
+}
+
+/*
+ * HUF_buildCTableFromTree():
+ * Build the CTable given the Huffman tree in huffNode.
+ *
+ * @param[out] CTable The output Huffman CTable.
+ * @param huffNode The Huffman tree.
+ * @param nonNullRank The last and smallest node in the Huffman tree.
+ * @param maxSymbolValue The maximum symbol value.
+ * @param maxNbBits The exact maximum number of bits used in the Huffman tree.
+ */
+static void HUF_buildCTableFromTree(HUF_CElt* CTable, nodeElt const* huffNode, int nonNullRank, U32 maxSymbolValue, U32 maxNbBits)
+{
+ /* fill result into ctable (val, nbBits) */
+ int n;
+ U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0};
+ U16 valPerRank[HUF_TABLELOG_MAX+1] = {0};
+ int const alphabetSize = (int)(maxSymbolValue + 1);
+ for (n=0; n<=nonNullRank; n++)
+ nbPerRank[huffNode[n].nbBits]++;
+ /* determine starting value per rank */
+ { U16 min = 0;
+ for (n=(int)maxNbBits; n>0; n--) {
+ valPerRank[n] = min; /* get starting value within each rank */
+ min += nbPerRank[n];
+ min >>= 1;
+ } }
+ for (n=0; n<alphabetSize; n++)
+ CTable[huffNode[n].byte].nbBits = huffNode[n].nbBits; /* push nbBits per symbol, symbol order */
+ for (n=0; n<alphabetSize; n++)
+ CTable[n].val = valPerRank[CTable[n].nbBits]++; /* assign value within rank, symbol order */
+}
+
+size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
+{
+ HUF_buildCTable_wksp_tables* const wksp_tables = (HUF_buildCTable_wksp_tables*)workSpace;
+ nodeElt* const huffNode0 = wksp_tables->huffNodeTbl;
+ nodeElt* const huffNode = huffNode0+1;
+ int nonNullRank;
+
+ /* safety checks */
+ if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
+ if (wkspSize < sizeof(HUF_buildCTable_wksp_tables))
+ return ERROR(workSpace_tooSmall);
+ if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT;
+ if (maxSymbolValue > HUF_SYMBOLVALUE_MAX)
+ return ERROR(maxSymbolValue_tooLarge);
+ ZSTD_memset(huffNode0, 0, sizeof(huffNodeTable));
+
+ /* sort, decreasing order */
+ HUF_sort(huffNode, count, maxSymbolValue, wksp_tables->rankPosition);
+
+ /* build tree */
+ nonNullRank = HUF_buildTree(huffNode, maxSymbolValue);
+
+ /* enforce maxTableLog */
+ maxNbBits = HUF_setMaxHeight(huffNode, (U32)nonNullRank, maxNbBits);
+ if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC); /* check fit into table */
+
+ HUF_buildCTableFromTree(tree, huffNode, nonNullRank, maxSymbolValue, maxNbBits);
+
+ return maxNbBits;
+}
+
+size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
+{
+ size_t nbBits = 0;
+ int s;
+ for (s = 0; s <= (int)maxSymbolValue; ++s) {
+ nbBits += CTable[s].nbBits * count[s];
+ }
+ return nbBits >> 3;
+}
+
+int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
+ int bad = 0;
+ int s;
+ for (s = 0; s <= (int)maxSymbolValue; ++s) {
+ bad |= (count[s] != 0) & (CTable[s].nbBits == 0);
+ }
+ return !bad;
+}
+
+size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); }
+
+FORCE_INLINE_TEMPLATE void
+HUF_encodeSymbol(BIT_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable)
+{
+ BIT_addBitsFast(bitCPtr, CTable[symbol].val, CTable[symbol].nbBits);
+}
+
+#define HUF_FLUSHBITS(s) BIT_flushBits(s)
+
+#define HUF_FLUSHBITS_1(stream) \
+ if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*2+7) HUF_FLUSHBITS(stream)
+
+#define HUF_FLUSHBITS_2(stream) \
+ if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream)
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable)
+{
+ const BYTE* ip = (const BYTE*) src;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* op = ostart;
+ size_t n;
+ BIT_CStream_t bitC;
+
+ /* init */
+ if (dstSize < 8) return 0; /* not enough space to compress */
+ { size_t const initErr = BIT_initCStream(&bitC, op, (size_t)(oend-op));
+ if (HUF_isError(initErr)) return 0; }
+
+ n = srcSize & ~3; /* join to mod 4 */
+ switch (srcSize & 3)
+ {
+ case 3:
+ HUF_encodeSymbol(&bitC, ip[n+ 2], CTable);
+ HUF_FLUSHBITS_2(&bitC);
+ ZSTD_FALLTHROUGH;
+ case 2:
+ HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);
+ HUF_FLUSHBITS_1(&bitC);
+ ZSTD_FALLTHROUGH;
+ case 1:
+ HUF_encodeSymbol(&bitC, ip[n+ 0], CTable);
+ HUF_FLUSHBITS(&bitC);
+ ZSTD_FALLTHROUGH;
+ case 0: ZSTD_FALLTHROUGH;
+ default: break;
+ }
+
+ for (; n>0; n-=4) { /* note : n&3==0 at this stage */
+ HUF_encodeSymbol(&bitC, ip[n- 1], CTable);
+ HUF_FLUSHBITS_1(&bitC);
+ HUF_encodeSymbol(&bitC, ip[n- 2], CTable);
+ HUF_FLUSHBITS_2(&bitC);
+ HUF_encodeSymbol(&bitC, ip[n- 3], CTable);
+ HUF_FLUSHBITS_1(&bitC);
+ HUF_encodeSymbol(&bitC, ip[n- 4], CTable);
+ HUF_FLUSHBITS(&bitC);
+ }
+
+ return BIT_closeCStream(&bitC);
+}
+
+#if DYNAMIC_BMI2
+
+static TARGET_ATTRIBUTE("bmi2") size_t
+HUF_compress1X_usingCTable_internal_bmi2(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable)
+{
+ return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
+}
+
+static size_t
+HUF_compress1X_usingCTable_internal_default(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable)
+{
+ return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
+}
+
+static size_t
+HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable, const int bmi2)
+{
+ if (bmi2) {
+ return HUF_compress1X_usingCTable_internal_bmi2(dst, dstSize, src, srcSize, CTable);
+ }
+ return HUF_compress1X_usingCTable_internal_default(dst, dstSize, src, srcSize, CTable);
+}
+
+#else
+
+static size_t
+HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable, const int bmi2)
+{
+ (void)bmi2;
+ return HUF_compress1X_usingCTable_internal_body(dst, dstSize, src, srcSize, CTable);
+}
+
+#endif
+
+size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
+{
+ return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
+}
+
+
+static size_t
+HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ const HUF_CElt* CTable, int bmi2)
+{
+ size_t const segmentSize = (srcSize+3)/4; /* first 3 segments */
+ const BYTE* ip = (const BYTE*) src;
+ const BYTE* const iend = ip + srcSize;
+ BYTE* const ostart = (BYTE*) dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* op = ostart;
+
+ if (dstSize < 6 + 1 + 1 + 1 + 8) return 0; /* minimum space to compress successfully */
+ if (srcSize < 12) return 0; /* no saving possible : too small input */
+ op += 6; /* jumpTable */
+
+ assert(op <= oend);
+ { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
+ if (cSize==0) return 0;
+ assert(cSize <= 65535);
+ MEM_writeLE16(ostart, (U16)cSize);
+ op += cSize;
+ }
+
+ ip += segmentSize;
+ assert(op <= oend);
+ { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
+ if (cSize==0) return 0;
+ assert(cSize <= 65535);
+ MEM_writeLE16(ostart+2, (U16)cSize);
+ op += cSize;
+ }
+
+ ip += segmentSize;
+ assert(op <= oend);
+ { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
+ if (cSize==0) return 0;
+ assert(cSize <= 65535);
+ MEM_writeLE16(ostart+4, (U16)cSize);
+ op += cSize;
+ }
+
+ ip += segmentSize;
+ assert(op <= oend);
+ assert(ip <= iend);
+ { CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, (size_t)(iend-ip), CTable, bmi2) );
+ if (cSize==0) return 0;
+ op += cSize;
+ }
+
+ return (size_t)(op-ostart);
+}
+
+size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
+{
+ return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
+}
+
+typedef enum { HUF_singleStream, HUF_fourStreams } HUF_nbStreams_e;
+
+static size_t HUF_compressCTable_internal(
+ BYTE* const ostart, BYTE* op, BYTE* const oend,
+ const void* src, size_t srcSize,
+ HUF_nbStreams_e nbStreams, const HUF_CElt* CTable, const int bmi2)
+{
+ size_t const cSize = (nbStreams==HUF_singleStream) ?
+ HUF_compress1X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2) :
+ HUF_compress4X_usingCTable_internal(op, (size_t)(oend - op), src, srcSize, CTable, bmi2);
+ if (HUF_isError(cSize)) { return cSize; }
+ if (cSize==0) { return 0; } /* uncompressible */
+ op += cSize;
+ /* check compressibility */
+ assert(op >= ostart);
+ if ((size_t)(op-ostart) >= srcSize-1) { return 0; }
+ return (size_t)(op-ostart);
+}
+
+typedef struct {
+ unsigned count[HUF_SYMBOLVALUE_MAX + 1];
+ HUF_CElt CTable[HUF_SYMBOLVALUE_MAX + 1];
+ union {
+ HUF_buildCTable_wksp_tables buildCTable_wksp;
+ HUF_WriteCTableWksp writeCTable_wksp;
+ } wksps;
+} HUF_compress_tables_t;
+
+/* HUF_compress_internal() :
+ * `workSpace_align4` must be aligned on 4-bytes boundaries,
+ * and occupies the same space as a table of HUF_WORKSPACE_SIZE_U32 unsigned */
+static size_t
+HUF_compress_internal (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned huffLog,
+ HUF_nbStreams_e nbStreams,
+ void* workSpace_align4, size_t wkspSize,
+ HUF_CElt* oldHufTable, HUF_repeat* repeat, int preferRepeat,
+ const int bmi2)
+{
+ HUF_compress_tables_t* const table = (HUF_compress_tables_t*)workSpace_align4;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* op = ostart;
+
+ HUF_STATIC_ASSERT(sizeof(*table) <= HUF_WORKSPACE_SIZE);
+ assert(((size_t)workSpace_align4 & 3) == 0); /* must be aligned on 4-bytes boundaries */
+
+ /* checks & inits */
+ if (wkspSize < HUF_WORKSPACE_SIZE) return ERROR(workSpace_tooSmall);
+ if (!srcSize) return 0; /* Uncompressed */
+ if (!dstSize) return 0; /* cannot fit anything within dst budget */
+ if (srcSize > HUF_BLOCKSIZE_MAX) return ERROR(srcSize_wrong); /* current block size limit */
+ if (huffLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
+ if (maxSymbolValue > HUF_SYMBOLVALUE_MAX) return ERROR(maxSymbolValue_tooLarge);
+ if (!maxSymbolValue) maxSymbolValue = HUF_SYMBOLVALUE_MAX;
+ if (!huffLog) huffLog = HUF_TABLELOG_DEFAULT;
+
+ /* Heuristic : If old table is valid, use it for small inputs */
+ if (preferRepeat && repeat && *repeat == HUF_repeat_valid) {
+ return HUF_compressCTable_internal(ostart, op, oend,
+ src, srcSize,
+ nbStreams, oldHufTable, bmi2);
+ }
+
+ /* Scan input and build symbol stats */
+ { CHECK_V_F(largest, HIST_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, workSpace_align4, wkspSize) );
+ if (largest == srcSize) { *ostart = ((const BYTE*)src)[0]; return 1; } /* single symbol, rle */
+ if (largest <= (srcSize >> 7)+4) return 0; /* heuristic : probably not compressible enough */
+ }
+
+ /* Check validity of previous table */
+ if ( repeat
+ && *repeat == HUF_repeat_check
+ && !HUF_validateCTable(oldHufTable, table->count, maxSymbolValue)) {
+ *repeat = HUF_repeat_none;
+ }
+ /* Heuristic : use existing table for small inputs */
+ if (preferRepeat && repeat && *repeat != HUF_repeat_none) {
+ return HUF_compressCTable_internal(ostart, op, oend,
+ src, srcSize,
+ nbStreams, oldHufTable, bmi2);
+ }
+
+ /* Build Huffman Tree */
+ huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
+ { size_t const maxBits = HUF_buildCTable_wksp(table->CTable, table->count,
+ maxSymbolValue, huffLog,
+ &table->wksps.buildCTable_wksp, sizeof(table->wksps.buildCTable_wksp));
+ CHECK_F(maxBits);
+ huffLog = (U32)maxBits;
+ /* Zero unused symbols in CTable, so we can check it for validity */
+ ZSTD_memset(table->CTable + (maxSymbolValue + 1), 0,
+ sizeof(table->CTable) - ((maxSymbolValue + 1) * sizeof(HUF_CElt)));
+ }
+
+ /* Write table description header */
+ { CHECK_V_F(hSize, HUF_writeCTable_wksp(op, dstSize, table->CTable, maxSymbolValue, huffLog,
+ &table->wksps.writeCTable_wksp, sizeof(table->wksps.writeCTable_wksp)) );
+ /* Check if using previous huffman table is beneficial */
+ if (repeat && *repeat != HUF_repeat_none) {
+ size_t const oldSize = HUF_estimateCompressedSize(oldHufTable, table->count, maxSymbolValue);
+ size_t const newSize = HUF_estimateCompressedSize(table->CTable, table->count, maxSymbolValue);
+ if (oldSize <= hSize + newSize || hSize + 12 >= srcSize) {
+ return HUF_compressCTable_internal(ostart, op, oend,
+ src, srcSize,
+ nbStreams, oldHufTable, bmi2);
+ } }
+
+ /* Use the new huffman table */
+ if (hSize + 12ul >= srcSize) { return 0; }
+ op += hSize;
+ if (repeat) { *repeat = HUF_repeat_none; }
+ if (oldHufTable)
+ ZSTD_memcpy(oldHufTable, table->CTable, sizeof(table->CTable)); /* Save new table */
+ }
+ return HUF_compressCTable_internal(ostart, op, oend,
+ src, srcSize,
+ nbStreams, table->CTable, bmi2);
+}
+
+
+size_t HUF_compress1X_wksp (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned huffLog,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_compress_internal(dst, dstSize, src, srcSize,
+ maxSymbolValue, huffLog, HUF_singleStream,
+ workSpace, wkspSize,
+ NULL, NULL, 0, 0 /*bmi2*/);
+}
+
+size_t HUF_compress1X_repeat (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned huffLog,
+ void* workSpace, size_t wkspSize,
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
+{
+ return HUF_compress_internal(dst, dstSize, src, srcSize,
+ maxSymbolValue, huffLog, HUF_singleStream,
+ workSpace, wkspSize, hufTable,
+ repeat, preferRepeat, bmi2);
+}
+
+/* HUF_compress4X_repeat():
+ * compress input using 4 streams.
+ * provide workspace to generate compression tables */
+size_t HUF_compress4X_wksp (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned huffLog,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_compress_internal(dst, dstSize, src, srcSize,
+ maxSymbolValue, huffLog, HUF_fourStreams,
+ workSpace, wkspSize,
+ NULL, NULL, 0, 0 /*bmi2*/);
+}
+
+/* HUF_compress4X_repeat():
+ * compress input using 4 streams.
+ * re-use an existing huffman compression table */
+size_t HUF_compress4X_repeat (void* dst, size_t dstSize,
+ const void* src, size_t srcSize,
+ unsigned maxSymbolValue, unsigned huffLog,
+ void* workSpace, size_t wkspSize,
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
+{
+ return HUF_compress_internal(dst, dstSize, src, srcSize,
+ maxSymbolValue, huffLog, HUF_fourStreams,
+ workSpace, wkspSize,
+ hufTable, repeat, preferRepeat, bmi2);
+}
+
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/*-*************************************
+* Dependencies
+***************************************/
+#include "../common/zstd_deps.h" /* INT_MAX, ZSTD_memset, ZSTD_memcpy */
+#include "../common/cpu.h"
+#include "../common/mem.h"
+#include "hist.h" /* HIST_countFast_wksp */
+#define FSE_STATIC_LINKING_ONLY /* FSE_encodeSymbol */
+#include "../common/fse.h"
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include "zstd_compress_internal.h"
+#include "zstd_compress_sequences.h"
+#include "zstd_compress_literals.h"
+#include "zstd_fast.h"
+#include "zstd_double_fast.h"
+#include "zstd_lazy.h"
+#include "zstd_opt.h"
+#include "zstd_ldm.h"
+#include "zstd_compress_superblock.h"
+
+/* ***************************************************************
+* Tuning parameters
+*****************************************************************/
+/*!
+ * COMPRESS_HEAPMODE :
+ * Select how default decompression function ZSTD_compress() allocates its context,
+ * on stack (0, default), or into heap (1).
+ * Note that functions with explicit context such as ZSTD_compressCCtx() are unaffected.
+ */
+
+
+/*-*************************************
+* Helper functions
+***************************************/
+/* ZSTD_compressBound()
+ * Note that the result from this function is only compatible with the "normal"
+ * full-block strategy.
+ * When there are a lot of small blocks due to frequent flush in streaming mode
+ * the overhead of headers can make the compressed data to be larger than the
+ * return value of ZSTD_compressBound().
+ */
+size_t ZSTD_compressBound(size_t srcSize) {
+ return ZSTD_COMPRESSBOUND(srcSize);
+}
+
+
+/*-*************************************
+* Context memory management
+***************************************/
+struct ZSTD_CDict_s {
+ const void* dictContent;
+ size_t dictContentSize;
+ ZSTD_dictContentType_e dictContentType; /* The dictContentType the CDict was created with */
+ U32* entropyWorkspace; /* entropy workspace of HUF_WORKSPACE_SIZE bytes */
+ ZSTD_cwksp workspace;
+ ZSTD_matchState_t matchState;
+ ZSTD_compressedBlockState_t cBlockState;
+ ZSTD_customMem customMem;
+ U32 dictID;
+ int compressionLevel; /* 0 indicates that advanced API was used to select CDict params */
+}; /* typedef'd to ZSTD_CDict within "zstd.h" */
+
+ZSTD_CCtx* ZSTD_createCCtx(void)
+{
+ return ZSTD_createCCtx_advanced(ZSTD_defaultCMem);
+}
+
+static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager)
+{
+ assert(cctx != NULL);
+ ZSTD_memset(cctx, 0, sizeof(*cctx));
+ cctx->customMem = memManager;
+ cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+ { size_t const err = ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters);
+ assert(!ZSTD_isError(err));
+ (void)err;
+ }
+}
+
+ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem)
+{
+ ZSTD_STATIC_ASSERT(zcss_init==0);
+ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN==(0ULL - 1));
+ if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
+ { ZSTD_CCtx* const cctx = (ZSTD_CCtx*)ZSTD_customMalloc(sizeof(ZSTD_CCtx), customMem);
+ if (!cctx) return NULL;
+ ZSTD_initCCtx(cctx, customMem);
+ return cctx;
+ }
+}
+
+ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize)
+{
+ ZSTD_cwksp ws;
+ ZSTD_CCtx* cctx;
+ if (workspaceSize <= sizeof(ZSTD_CCtx)) return NULL; /* minimum size */
+ if ((size_t)workspace & 7) return NULL; /* must be 8-aligned */
+ ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_static_alloc);
+
+ cctx = (ZSTD_CCtx*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CCtx));
+ if (cctx == NULL) return NULL;
+
+ ZSTD_memset(cctx, 0, sizeof(ZSTD_CCtx));
+ ZSTD_cwksp_move(&cctx->workspace, &ws);
+ cctx->staticSize = workspaceSize;
+
+ /* statically sized space. entropyWorkspace never moves (but prev/next block swap places) */
+ if (!ZSTD_cwksp_check_available(&cctx->workspace, ENTROPY_WORKSPACE_SIZE + 2 * sizeof(ZSTD_compressedBlockState_t))) return NULL;
+ cctx->blockState.prevCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
+ cctx->blockState.nextCBlock = (ZSTD_compressedBlockState_t*)ZSTD_cwksp_reserve_object(&cctx->workspace, sizeof(ZSTD_compressedBlockState_t));
+ cctx->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cctx->workspace, ENTROPY_WORKSPACE_SIZE);
+ cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+ return cctx;
+}
+
+/*
+ * Clears and frees all of the dictionaries in the CCtx.
+ */
+static void ZSTD_clearAllDicts(ZSTD_CCtx* cctx)
+{
+ ZSTD_customFree(cctx->localDict.dictBuffer, cctx->customMem);
+ ZSTD_freeCDict(cctx->localDict.cdict);
+ ZSTD_memset(&cctx->localDict, 0, sizeof(cctx->localDict));
+ ZSTD_memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict));
+ cctx->cdict = NULL;
+}
+
+static size_t ZSTD_sizeof_localDict(ZSTD_localDict dict)
+{
+ size_t const bufferSize = dict.dictBuffer != NULL ? dict.dictSize : 0;
+ size_t const cdictSize = ZSTD_sizeof_CDict(dict.cdict);
+ return bufferSize + cdictSize;
+}
+
+static void ZSTD_freeCCtxContent(ZSTD_CCtx* cctx)
+{
+ assert(cctx != NULL);
+ assert(cctx->staticSize == 0);
+ ZSTD_clearAllDicts(cctx);
+ ZSTD_cwksp_free(&cctx->workspace, cctx->customMem);
+}
+
+size_t ZSTD_freeCCtx(ZSTD_CCtx* cctx)
+{
+ if (cctx==NULL) return 0; /* support free on NULL */
+ RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
+ "not compatible with static CCtx");
+ {
+ int cctxInWorkspace = ZSTD_cwksp_owns_buffer(&cctx->workspace, cctx);
+ ZSTD_freeCCtxContent(cctx);
+ if (!cctxInWorkspace) {
+ ZSTD_customFree(cctx, cctx->customMem);
+ }
+ }
+ return 0;
+}
+
+
+static size_t ZSTD_sizeof_mtctx(const ZSTD_CCtx* cctx)
+{
+ (void)cctx;
+ return 0;
+}
+
+
+size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx)
+{
+ if (cctx==NULL) return 0; /* support sizeof on NULL */
+ /* cctx may be in the workspace */
+ return (cctx->workspace.workspace == cctx ? 0 : sizeof(*cctx))
+ + ZSTD_cwksp_sizeof(&cctx->workspace)
+ + ZSTD_sizeof_localDict(cctx->localDict)
+ + ZSTD_sizeof_mtctx(cctx);
+}
+
+size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs)
+{
+ return ZSTD_sizeof_CCtx(zcs); /* same object */
+}
+
+/* private API call, for dictBuilder only */
+const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx) { return &(ctx->seqStore); }
+
+/* Returns 1 if compression parameters are such that we should
+ * enable long distance matching (wlog >= 27, strategy >= btopt).
+ * Returns 0 otherwise.
+ */
+static U32 ZSTD_CParams_shouldEnableLdm(const ZSTD_compressionParameters* const cParams) {
+ return cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 27;
+}
+
+static ZSTD_CCtx_params ZSTD_makeCCtxParamsFromCParams(
+ ZSTD_compressionParameters cParams)
+{
+ ZSTD_CCtx_params cctxParams;
+ /* should not matter, as all cParams are presumed properly defined */
+ ZSTD_CCtxParams_init(&cctxParams, ZSTD_CLEVEL_DEFAULT);
+ cctxParams.cParams = cParams;
+
+ if (ZSTD_CParams_shouldEnableLdm(&cParams)) {
+ DEBUGLOG(4, "ZSTD_makeCCtxParamsFromCParams(): Including LDM into cctx params");
+ cctxParams.ldmParams.enableLdm = 1;
+ /* LDM is enabled by default for optimal parser and window size >= 128MB */
+ ZSTD_ldm_adjustParameters(&cctxParams.ldmParams, &cParams);
+ assert(cctxParams.ldmParams.hashLog >= cctxParams.ldmParams.bucketSizeLog);
+ assert(cctxParams.ldmParams.hashRateLog < 32);
+ }
+
+ assert(!ZSTD_checkCParams(cParams));
+ return cctxParams;
+}
+
+static ZSTD_CCtx_params* ZSTD_createCCtxParams_advanced(
+ ZSTD_customMem customMem)
+{
+ ZSTD_CCtx_params* params;
+ if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
+ params = (ZSTD_CCtx_params*)ZSTD_customCalloc(
+ sizeof(ZSTD_CCtx_params), customMem);
+ if (!params) { return NULL; }
+ ZSTD_CCtxParams_init(params, ZSTD_CLEVEL_DEFAULT);
+ params->customMem = customMem;
+ return params;
+}
+
+ZSTD_CCtx_params* ZSTD_createCCtxParams(void)
+{
+ return ZSTD_createCCtxParams_advanced(ZSTD_defaultCMem);
+}
+
+size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params)
+{
+ if (params == NULL) { return 0; }
+ ZSTD_customFree(params, params->customMem);
+ return 0;
+}
+
+size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params)
+{
+ return ZSTD_CCtxParams_init(params, ZSTD_CLEVEL_DEFAULT);
+}
+
+size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel) {
+ RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
+ ZSTD_memset(cctxParams, 0, sizeof(*cctxParams));
+ cctxParams->compressionLevel = compressionLevel;
+ cctxParams->fParams.contentSizeFlag = 1;
+ return 0;
+}
+
+#define ZSTD_NO_CLEVEL 0
+
+/*
+ * Initializes the cctxParams from params and compressionLevel.
+ * @param compressionLevel If params are derived from a compression level then that compression level, otherwise ZSTD_NO_CLEVEL.
+ */
+static void ZSTD_CCtxParams_init_internal(ZSTD_CCtx_params* cctxParams, ZSTD_parameters const* params, int compressionLevel)
+{
+ assert(!ZSTD_checkCParams(params->cParams));
+ ZSTD_memset(cctxParams, 0, sizeof(*cctxParams));
+ cctxParams->cParams = params->cParams;
+ cctxParams->fParams = params->fParams;
+ /* Should not matter, as all cParams are presumed properly defined.
+ * But, set it for tracing anyway.
+ */
+ cctxParams->compressionLevel = compressionLevel;
+}
+
+size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
+{
+ RETURN_ERROR_IF(!cctxParams, GENERIC, "NULL pointer!");
+ FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
+ ZSTD_CCtxParams_init_internal(cctxParams, ¶ms, ZSTD_NO_CLEVEL);
+ return 0;
+}
+
+/*
+ * Sets cctxParams' cParams and fParams from params, but otherwise leaves them alone.
+ * @param param Validated zstd parameters.
+ */
+static void ZSTD_CCtxParams_setZstdParams(
+ ZSTD_CCtx_params* cctxParams, const ZSTD_parameters* params)
+{
+ assert(!ZSTD_checkCParams(params->cParams));
+ cctxParams->cParams = params->cParams;
+ cctxParams->fParams = params->fParams;
+ /* Should not matter, as all cParams are presumed properly defined.
+ * But, set it for tracing anyway.
+ */
+ cctxParams->compressionLevel = ZSTD_NO_CLEVEL;
+}
+
+ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
+{
+ ZSTD_bounds bounds = { 0, 0, 0 };
+
+ switch(param)
+ {
+ case ZSTD_c_compressionLevel:
+ bounds.lowerBound = ZSTD_minCLevel();
+ bounds.upperBound = ZSTD_maxCLevel();
+ return bounds;
+
+ case ZSTD_c_windowLog:
+ bounds.lowerBound = ZSTD_WINDOWLOG_MIN;
+ bounds.upperBound = ZSTD_WINDOWLOG_MAX;
+ return bounds;
+
+ case ZSTD_c_hashLog:
+ bounds.lowerBound = ZSTD_HASHLOG_MIN;
+ bounds.upperBound = ZSTD_HASHLOG_MAX;
+ return bounds;
+
+ case ZSTD_c_chainLog:
+ bounds.lowerBound = ZSTD_CHAINLOG_MIN;
+ bounds.upperBound = ZSTD_CHAINLOG_MAX;
+ return bounds;
+
+ case ZSTD_c_searchLog:
+ bounds.lowerBound = ZSTD_SEARCHLOG_MIN;
+ bounds.upperBound = ZSTD_SEARCHLOG_MAX;
+ return bounds;
+
+ case ZSTD_c_minMatch:
+ bounds.lowerBound = ZSTD_MINMATCH_MIN;
+ bounds.upperBound = ZSTD_MINMATCH_MAX;
+ return bounds;
+
+ case ZSTD_c_targetLength:
+ bounds.lowerBound = ZSTD_TARGETLENGTH_MIN;
+ bounds.upperBound = ZSTD_TARGETLENGTH_MAX;
+ return bounds;
+
+ case ZSTD_c_strategy:
+ bounds.lowerBound = ZSTD_STRATEGY_MIN;
+ bounds.upperBound = ZSTD_STRATEGY_MAX;
+ return bounds;
+
+ case ZSTD_c_contentSizeFlag:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_checksumFlag:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_dictIDFlag:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_nbWorkers:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 0;
+ return bounds;
+
+ case ZSTD_c_jobSize:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 0;
+ return bounds;
+
+ case ZSTD_c_overlapLog:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 0;
+ return bounds;
+
+ case ZSTD_c_enableDedicatedDictSearch:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_enableLongDistanceMatching:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_ldmHashLog:
+ bounds.lowerBound = ZSTD_LDM_HASHLOG_MIN;
+ bounds.upperBound = ZSTD_LDM_HASHLOG_MAX;
+ return bounds;
+
+ case ZSTD_c_ldmMinMatch:
+ bounds.lowerBound = ZSTD_LDM_MINMATCH_MIN;
+ bounds.upperBound = ZSTD_LDM_MINMATCH_MAX;
+ return bounds;
+
+ case ZSTD_c_ldmBucketSizeLog:
+ bounds.lowerBound = ZSTD_LDM_BUCKETSIZELOG_MIN;
+ bounds.upperBound = ZSTD_LDM_BUCKETSIZELOG_MAX;
+ return bounds;
+
+ case ZSTD_c_ldmHashRateLog:
+ bounds.lowerBound = ZSTD_LDM_HASHRATELOG_MIN;
+ bounds.upperBound = ZSTD_LDM_HASHRATELOG_MAX;
+ return bounds;
+
+ /* experimental parameters */
+ case ZSTD_c_rsyncable:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_forceMaxWindow :
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ case ZSTD_c_format:
+ ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
+ bounds.lowerBound = ZSTD_f_zstd1;
+ bounds.upperBound = ZSTD_f_zstd1_magicless; /* note : how to ensure at compile time that this is the highest value enum ? */
+ return bounds;
+
+ case ZSTD_c_forceAttachDict:
+ ZSTD_STATIC_ASSERT(ZSTD_dictDefaultAttach < ZSTD_dictForceLoad);
+ bounds.lowerBound = ZSTD_dictDefaultAttach;
+ bounds.upperBound = ZSTD_dictForceLoad; /* note : how to ensure at compile time that this is the highest value enum ? */
+ return bounds;
+
+ case ZSTD_c_literalCompressionMode:
+ ZSTD_STATIC_ASSERT(ZSTD_lcm_auto < ZSTD_lcm_huffman && ZSTD_lcm_huffman < ZSTD_lcm_uncompressed);
+ bounds.lowerBound = ZSTD_lcm_auto;
+ bounds.upperBound = ZSTD_lcm_uncompressed;
+ return bounds;
+
+ case ZSTD_c_targetCBlockSize:
+ bounds.lowerBound = ZSTD_TARGETCBLOCKSIZE_MIN;
+ bounds.upperBound = ZSTD_TARGETCBLOCKSIZE_MAX;
+ return bounds;
+
+ case ZSTD_c_srcSizeHint:
+ bounds.lowerBound = ZSTD_SRCSIZEHINT_MIN;
+ bounds.upperBound = ZSTD_SRCSIZEHINT_MAX;
+ return bounds;
+
+ case ZSTD_c_stableInBuffer:
+ case ZSTD_c_stableOutBuffer:
+ bounds.lowerBound = (int)ZSTD_bm_buffered;
+ bounds.upperBound = (int)ZSTD_bm_stable;
+ return bounds;
+
+ case ZSTD_c_blockDelimiters:
+ bounds.lowerBound = (int)ZSTD_sf_noBlockDelimiters;
+ bounds.upperBound = (int)ZSTD_sf_explicitBlockDelimiters;
+ return bounds;
+
+ case ZSTD_c_validateSequences:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
+ default:
+ bounds.error = ERROR(parameter_unsupported);
+ return bounds;
+ }
+}
+
+/* ZSTD_cParam_clampBounds:
+ * Clamps the value into the bounded range.
+ */
+static size_t ZSTD_cParam_clampBounds(ZSTD_cParameter cParam, int* value)
+{
+ ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam);
+ if (ZSTD_isError(bounds.error)) return bounds.error;
+ if (*value < bounds.lowerBound) *value = bounds.lowerBound;
+ if (*value > bounds.upperBound) *value = bounds.upperBound;
+ return 0;
+}
+
+#define BOUNDCHECK(cParam, val) { \
+ RETURN_ERROR_IF(!ZSTD_cParam_withinBounds(cParam,val), \
+ parameter_outOfBound, "Param out of bounds"); \
+}
+
+
+static int ZSTD_isUpdateAuthorized(ZSTD_cParameter param)
+{
+ switch(param)
+ {
+ case ZSTD_c_compressionLevel:
+ case ZSTD_c_hashLog:
+ case ZSTD_c_chainLog:
+ case ZSTD_c_searchLog:
+ case ZSTD_c_minMatch:
+ case ZSTD_c_targetLength:
+ case ZSTD_c_strategy:
+ return 1;
+
+ case ZSTD_c_format:
+ case ZSTD_c_windowLog:
+ case ZSTD_c_contentSizeFlag:
+ case ZSTD_c_checksumFlag:
+ case ZSTD_c_dictIDFlag:
+ case ZSTD_c_forceMaxWindow :
+ case ZSTD_c_nbWorkers:
+ case ZSTD_c_jobSize:
+ case ZSTD_c_overlapLog:
+ case ZSTD_c_rsyncable:
+ case ZSTD_c_enableDedicatedDictSearch:
+ case ZSTD_c_enableLongDistanceMatching:
+ case ZSTD_c_ldmHashLog:
+ case ZSTD_c_ldmMinMatch:
+ case ZSTD_c_ldmBucketSizeLog:
+ case ZSTD_c_ldmHashRateLog:
+ case ZSTD_c_forceAttachDict:
+ case ZSTD_c_literalCompressionMode:
+ case ZSTD_c_targetCBlockSize:
+ case ZSTD_c_srcSizeHint:
+ case ZSTD_c_stableInBuffer:
+ case ZSTD_c_stableOutBuffer:
+ case ZSTD_c_blockDelimiters:
+ case ZSTD_c_validateSequences:
+ default:
+ return 0;
+ }
+}
+
+size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
+{
+ DEBUGLOG(4, "ZSTD_CCtx_setParameter (%i, %i)", (int)param, value);
+ if (cctx->streamStage != zcss_init) {
+ if (ZSTD_isUpdateAuthorized(param)) {
+ cctx->cParamsChanged = 1;
+ } else {
+ RETURN_ERROR(stage_wrong, "can only set params in ctx init stage");
+ } }
+
+ switch(param)
+ {
+ case ZSTD_c_nbWorkers:
+ RETURN_ERROR_IF((value!=0) && cctx->staticSize, parameter_unsupported,
+ "MT not compatible with static alloc");
+ break;
+
+ case ZSTD_c_compressionLevel:
+ case ZSTD_c_windowLog:
+ case ZSTD_c_hashLog:
+ case ZSTD_c_chainLog:
+ case ZSTD_c_searchLog:
+ case ZSTD_c_minMatch:
+ case ZSTD_c_targetLength:
+ case ZSTD_c_strategy:
+ case ZSTD_c_ldmHashRateLog:
+ case ZSTD_c_format:
+ case ZSTD_c_contentSizeFlag:
+ case ZSTD_c_checksumFlag:
+ case ZSTD_c_dictIDFlag:
+ case ZSTD_c_forceMaxWindow:
+ case ZSTD_c_forceAttachDict:
+ case ZSTD_c_literalCompressionMode:
+ case ZSTD_c_jobSize:
+ case ZSTD_c_overlapLog:
+ case ZSTD_c_rsyncable:
+ case ZSTD_c_enableDedicatedDictSearch:
+ case ZSTD_c_enableLongDistanceMatching:
+ case ZSTD_c_ldmHashLog:
+ case ZSTD_c_ldmMinMatch:
+ case ZSTD_c_ldmBucketSizeLog:
+ case ZSTD_c_targetCBlockSize:
+ case ZSTD_c_srcSizeHint:
+ case ZSTD_c_stableInBuffer:
+ case ZSTD_c_stableOutBuffer:
+ case ZSTD_c_blockDelimiters:
+ case ZSTD_c_validateSequences:
+ break;
+
+ default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
+ }
+ return ZSTD_CCtxParams_setParameter(&cctx->requestedParams, param, value);
+}
+
+size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
+ ZSTD_cParameter param, int value)
+{
+ DEBUGLOG(4, "ZSTD_CCtxParams_setParameter (%i, %i)", (int)param, value);
+ switch(param)
+ {
+ case ZSTD_c_format :
+ BOUNDCHECK(ZSTD_c_format, value);
+ CCtxParams->format = (ZSTD_format_e)value;
+ return (size_t)CCtxParams->format;
+
+ case ZSTD_c_compressionLevel : {
+ FORWARD_IF_ERROR(ZSTD_cParam_clampBounds(param, &value), "");
+ if (value == 0)
+ CCtxParams->compressionLevel = ZSTD_CLEVEL_DEFAULT; /* 0 == default */
+ else
+ CCtxParams->compressionLevel = value;
+ if (CCtxParams->compressionLevel >= 0) return (size_t)CCtxParams->compressionLevel;
+ return 0; /* return type (size_t) cannot represent negative values */
+ }
+
+ case ZSTD_c_windowLog :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_windowLog, value);
+ CCtxParams->cParams.windowLog = (U32)value;
+ return CCtxParams->cParams.windowLog;
+
+ case ZSTD_c_hashLog :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_hashLog, value);
+ CCtxParams->cParams.hashLog = (U32)value;
+ return CCtxParams->cParams.hashLog;
+
+ case ZSTD_c_chainLog :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_chainLog, value);
+ CCtxParams->cParams.chainLog = (U32)value;
+ return CCtxParams->cParams.chainLog;
+
+ case ZSTD_c_searchLog :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_searchLog, value);
+ CCtxParams->cParams.searchLog = (U32)value;
+ return (size_t)value;
+
+ case ZSTD_c_minMatch :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_minMatch, value);
+ CCtxParams->cParams.minMatch = value;
+ return CCtxParams->cParams.minMatch;
+
+ case ZSTD_c_targetLength :
+ BOUNDCHECK(ZSTD_c_targetLength, value);
+ CCtxParams->cParams.targetLength = value;
+ return CCtxParams->cParams.targetLength;
+
+ case ZSTD_c_strategy :
+ if (value!=0) /* 0 => use default */
+ BOUNDCHECK(ZSTD_c_strategy, value);
+ CCtxParams->cParams.strategy = (ZSTD_strategy)value;
+ return (size_t)CCtxParams->cParams.strategy;
+
+ case ZSTD_c_contentSizeFlag :
+ /* Content size written in frame header _when known_ (default:1) */
+ DEBUGLOG(4, "set content size flag = %u", (value!=0));
+ CCtxParams->fParams.contentSizeFlag = value != 0;
+ return CCtxParams->fParams.contentSizeFlag;
+
+ case ZSTD_c_checksumFlag :
+ /* A 32-bits content checksum will be calculated and written at end of frame (default:0) */
+ CCtxParams->fParams.checksumFlag = value != 0;
+ return CCtxParams->fParams.checksumFlag;
+
+ case ZSTD_c_dictIDFlag : /* When applicable, dictionary's dictID is provided in frame header (default:1) */
+ DEBUGLOG(4, "set dictIDFlag = %u", (value!=0));
+ CCtxParams->fParams.noDictIDFlag = !value;
+ return !CCtxParams->fParams.noDictIDFlag;
+
+ case ZSTD_c_forceMaxWindow :
+ CCtxParams->forceWindow = (value != 0);
+ return CCtxParams->forceWindow;
+
+ case ZSTD_c_forceAttachDict : {
+ const ZSTD_dictAttachPref_e pref = (ZSTD_dictAttachPref_e)value;
+ BOUNDCHECK(ZSTD_c_forceAttachDict, pref);
+ CCtxParams->attachDictPref = pref;
+ return CCtxParams->attachDictPref;
+ }
+
+ case ZSTD_c_literalCompressionMode : {
+ const ZSTD_literalCompressionMode_e lcm = (ZSTD_literalCompressionMode_e)value;
+ BOUNDCHECK(ZSTD_c_literalCompressionMode, lcm);
+ CCtxParams->literalCompressionMode = lcm;
+ return CCtxParams->literalCompressionMode;
+ }
+
+ case ZSTD_c_nbWorkers :
+ RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
+ return 0;
+
+ case ZSTD_c_jobSize :
+ RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
+ return 0;
+
+ case ZSTD_c_overlapLog :
+ RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
+ return 0;
+
+ case ZSTD_c_rsyncable :
+ RETURN_ERROR_IF(value!=0, parameter_unsupported, "not compiled with multithreading");
+ return 0;
+
+ case ZSTD_c_enableDedicatedDictSearch :
+ CCtxParams->enableDedicatedDictSearch = (value!=0);
+ return CCtxParams->enableDedicatedDictSearch;
+
+ case ZSTD_c_enableLongDistanceMatching :
+ CCtxParams->ldmParams.enableLdm = (value!=0);
+ return CCtxParams->ldmParams.enableLdm;
+
+ case ZSTD_c_ldmHashLog :
+ if (value!=0) /* 0 ==> auto */
+ BOUNDCHECK(ZSTD_c_ldmHashLog, value);
+ CCtxParams->ldmParams.hashLog = value;
+ return CCtxParams->ldmParams.hashLog;
+
+ case ZSTD_c_ldmMinMatch :
+ if (value!=0) /* 0 ==> default */
+ BOUNDCHECK(ZSTD_c_ldmMinMatch, value);
+ CCtxParams->ldmParams.minMatchLength = value;
+ return CCtxParams->ldmParams.minMatchLength;
+
+ case ZSTD_c_ldmBucketSizeLog :
+ if (value!=0) /* 0 ==> default */
+ BOUNDCHECK(ZSTD_c_ldmBucketSizeLog, value);
+ CCtxParams->ldmParams.bucketSizeLog = value;
+ return CCtxParams->ldmParams.bucketSizeLog;
+
+ case ZSTD_c_ldmHashRateLog :
+ if (value!=0) /* 0 ==> default */
+ BOUNDCHECK(ZSTD_c_ldmHashRateLog, value);
+ CCtxParams->ldmParams.hashRateLog = value;
+ return CCtxParams->ldmParams.hashRateLog;
+
+ case ZSTD_c_targetCBlockSize :
+ if (value!=0) /* 0 ==> default */
+ BOUNDCHECK(ZSTD_c_targetCBlockSize, value);
+ CCtxParams->targetCBlockSize = value;
+ return CCtxParams->targetCBlockSize;
+
+ case ZSTD_c_srcSizeHint :
+ if (value!=0) /* 0 ==> default */
+ BOUNDCHECK(ZSTD_c_srcSizeHint, value);
+ CCtxParams->srcSizeHint = value;
+ return CCtxParams->srcSizeHint;
+
+ case ZSTD_c_stableInBuffer:
+ BOUNDCHECK(ZSTD_c_stableInBuffer, value);
+ CCtxParams->inBufferMode = (ZSTD_bufferMode_e)value;
+ return CCtxParams->inBufferMode;
+
+ case ZSTD_c_stableOutBuffer:
+ BOUNDCHECK(ZSTD_c_stableOutBuffer, value);
+ CCtxParams->outBufferMode = (ZSTD_bufferMode_e)value;
+ return CCtxParams->outBufferMode;
+
+ case ZSTD_c_blockDelimiters:
+ BOUNDCHECK(ZSTD_c_blockDelimiters, value);
+ CCtxParams->blockDelimiters = (ZSTD_sequenceFormat_e)value;
+ return CCtxParams->blockDelimiters;
+
+ case ZSTD_c_validateSequences:
+ BOUNDCHECK(ZSTD_c_validateSequences, value);
+ CCtxParams->validateSequences = value;
+ return CCtxParams->validateSequences;
+
+ default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
+ }
+}
+
+size_t ZSTD_CCtx_getParameter(ZSTD_CCtx const* cctx, ZSTD_cParameter param, int* value)
+{
+ return ZSTD_CCtxParams_getParameter(&cctx->requestedParams, param, value);
+}
+
+size_t ZSTD_CCtxParams_getParameter(
+ ZSTD_CCtx_params const* CCtxParams, ZSTD_cParameter param, int* value)
+{
+ switch(param)
+ {
+ case ZSTD_c_format :
+ *value = CCtxParams->format;
+ break;
+ case ZSTD_c_compressionLevel :
+ *value = CCtxParams->compressionLevel;
+ break;
+ case ZSTD_c_windowLog :
+ *value = (int)CCtxParams->cParams.windowLog;
+ break;
+ case ZSTD_c_hashLog :
+ *value = (int)CCtxParams->cParams.hashLog;
+ break;
+ case ZSTD_c_chainLog :
+ *value = (int)CCtxParams->cParams.chainLog;
+ break;
+ case ZSTD_c_searchLog :
+ *value = CCtxParams->cParams.searchLog;
+ break;
+ case ZSTD_c_minMatch :
+ *value = CCtxParams->cParams.minMatch;
+ break;
+ case ZSTD_c_targetLength :
+ *value = CCtxParams->cParams.targetLength;
+ break;
+ case ZSTD_c_strategy :
+ *value = (unsigned)CCtxParams->cParams.strategy;
+ break;
+ case ZSTD_c_contentSizeFlag :
+ *value = CCtxParams->fParams.contentSizeFlag;
+ break;
+ case ZSTD_c_checksumFlag :
+ *value = CCtxParams->fParams.checksumFlag;
+ break;
+ case ZSTD_c_dictIDFlag :
+ *value = !CCtxParams->fParams.noDictIDFlag;
+ break;
+ case ZSTD_c_forceMaxWindow :
+ *value = CCtxParams->forceWindow;
+ break;
+ case ZSTD_c_forceAttachDict :
+ *value = CCtxParams->attachDictPref;
+ break;
+ case ZSTD_c_literalCompressionMode :
+ *value = CCtxParams->literalCompressionMode;
+ break;
+ case ZSTD_c_nbWorkers :
+ assert(CCtxParams->nbWorkers == 0);
+ *value = CCtxParams->nbWorkers;
+ break;
+ case ZSTD_c_jobSize :
+ RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
+ case ZSTD_c_overlapLog :
+ RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
+ case ZSTD_c_rsyncable :
+ RETURN_ERROR(parameter_unsupported, "not compiled with multithreading");
+ case ZSTD_c_enableDedicatedDictSearch :
+ *value = CCtxParams->enableDedicatedDictSearch;
+ break;
+ case ZSTD_c_enableLongDistanceMatching :
+ *value = CCtxParams->ldmParams.enableLdm;
+ break;
+ case ZSTD_c_ldmHashLog :
+ *value = CCtxParams->ldmParams.hashLog;
+ break;
+ case ZSTD_c_ldmMinMatch :
+ *value = CCtxParams->ldmParams.minMatchLength;
+ break;
+ case ZSTD_c_ldmBucketSizeLog :
+ *value = CCtxParams->ldmParams.bucketSizeLog;
+ break;
+ case ZSTD_c_ldmHashRateLog :
+ *value = CCtxParams->ldmParams.hashRateLog;
+ break;
+ case ZSTD_c_targetCBlockSize :
+ *value = (int)CCtxParams->targetCBlockSize;
+ break;
+ case ZSTD_c_srcSizeHint :
+ *value = (int)CCtxParams->srcSizeHint;
+ break;
+ case ZSTD_c_stableInBuffer :
+ *value = (int)CCtxParams->inBufferMode;
+ break;
+ case ZSTD_c_stableOutBuffer :
+ *value = (int)CCtxParams->outBufferMode;
+ break;
+ case ZSTD_c_blockDelimiters :
+ *value = (int)CCtxParams->blockDelimiters;
+ break;
+ case ZSTD_c_validateSequences :
+ *value = (int)CCtxParams->validateSequences;
+ break;
+ default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
+ }
+ return 0;
+}
+
+/* ZSTD_CCtx_setParametersUsingCCtxParams() :
+ * just applies `params` into `cctx`
+ * no action is performed, parameters are merely stored.
+ * If ZSTDMT is enabled, parameters are pushed to cctx->mtctx.
+ * This is possible even if a compression is ongoing.
+ * In which case, new parameters will be applied on the fly, starting with next compression job.
+ */
+size_t ZSTD_CCtx_setParametersUsingCCtxParams(
+ ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params)
+{
+ DEBUGLOG(4, "ZSTD_CCtx_setParametersUsingCCtxParams");
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "The context is in the wrong stage!");
+ RETURN_ERROR_IF(cctx->cdict, stage_wrong,
+ "Can't override parameters with cdict attached (some must "
+ "be inherited from the cdict).");
+
+ cctx->requestedParams = *params;
+ return 0;
+}
+
+ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize)
+{
+ DEBUGLOG(4, "ZSTD_CCtx_setPledgedSrcSize to %u bytes", (U32)pledgedSrcSize);
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't set pledgedSrcSize when not in init stage.");
+ cctx->pledgedSrcSizePlusOne = pledgedSrcSize+1;
+ return 0;
+}
+
+static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams(
+ int const compressionLevel,
+ size_t const dictSize);
+static int ZSTD_dedicatedDictSearch_isSupported(
+ const ZSTD_compressionParameters* cParams);
+static void ZSTD_dedicatedDictSearch_revertCParams(
+ ZSTD_compressionParameters* cParams);
+
+/*
+ * Initializes the local dict using the requested parameters.
+ * NOTE: This does not use the pledged src size, because it may be used for more
+ * than one compression.
+ */
+static size_t ZSTD_initLocalDict(ZSTD_CCtx* cctx)
+{
+ ZSTD_localDict* const dl = &cctx->localDict;
+ if (dl->dict == NULL) {
+ /* No local dictionary. */
+ assert(dl->dictBuffer == NULL);
+ assert(dl->cdict == NULL);
+ assert(dl->dictSize == 0);
+ return 0;
+ }
+ if (dl->cdict != NULL) {
+ assert(cctx->cdict == dl->cdict);
+ /* Local dictionary already initialized. */
+ return 0;
+ }
+ assert(dl->dictSize > 0);
+ assert(cctx->cdict == NULL);
+ assert(cctx->prefixDict.dict == NULL);
+
+ dl->cdict = ZSTD_createCDict_advanced2(
+ dl->dict,
+ dl->dictSize,
+ ZSTD_dlm_byRef,
+ dl->dictContentType,
+ &cctx->requestedParams,
+ cctx->customMem);
+ RETURN_ERROR_IF(!dl->cdict, memory_allocation, "ZSTD_createCDict_advanced failed");
+ cctx->cdict = dl->cdict;
+ return 0;
+}
+
+size_t ZSTD_CCtx_loadDictionary_advanced(
+ ZSTD_CCtx* cctx, const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType)
+{
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't load a dictionary when ctx is not in init stage.");
+ DEBUGLOG(4, "ZSTD_CCtx_loadDictionary_advanced (size: %u)", (U32)dictSize);
+ ZSTD_clearAllDicts(cctx); /* in case one already exists */
+ if (dict == NULL || dictSize == 0) /* no dictionary mode */
+ return 0;
+ if (dictLoadMethod == ZSTD_dlm_byRef) {
+ cctx->localDict.dict = dict;
+ } else {
+ void* dictBuffer;
+ RETURN_ERROR_IF(cctx->staticSize, memory_allocation,
+ "no malloc for static CCtx");
+ dictBuffer = ZSTD_customMalloc(dictSize, cctx->customMem);
+ RETURN_ERROR_IF(!dictBuffer, memory_allocation, "NULL pointer!");
+ ZSTD_memcpy(dictBuffer, dict, dictSize);
+ cctx->localDict.dictBuffer = dictBuffer;
+ cctx->localDict.dict = dictBuffer;
+ }
+ cctx->localDict.dictSize = dictSize;
+ cctx->localDict.dictContentType = dictContentType;
+ return 0;
+}
+
+ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(
+ ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
+{
+ return ZSTD_CCtx_loadDictionary_advanced(
+ cctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto);
+}
+
+ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
+{
+ return ZSTD_CCtx_loadDictionary_advanced(
+ cctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
+}
+
+
+size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
+{
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't ref a dict when ctx not in init stage.");
+ /* Free the existing local cdict (if any) to save memory. */
+ ZSTD_clearAllDicts(cctx);
+ cctx->cdict = cdict;
+ return 0;
+}
+
+size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool)
+{
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't ref a pool when ctx not in init stage.");
+ cctx->pool = pool;
+ return 0;
+}
+
+size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize)
+{
+ return ZSTD_CCtx_refPrefix_advanced(cctx, prefix, prefixSize, ZSTD_dct_rawContent);
+}
+
+size_t ZSTD_CCtx_refPrefix_advanced(
+ ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
+{
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't ref a prefix when ctx not in init stage.");
+ ZSTD_clearAllDicts(cctx);
+ if (prefix != NULL && prefixSize > 0) {
+ cctx->prefixDict.dict = prefix;
+ cctx->prefixDict.dictSize = prefixSize;
+ cctx->prefixDict.dictContentType = dictContentType;
+ }
+ return 0;
+}
+
+/*! ZSTD_CCtx_reset() :
+ * Also dumps dictionary */
+size_t ZSTD_CCtx_reset(ZSTD_CCtx* cctx, ZSTD_ResetDirective reset)
+{
+ if ( (reset == ZSTD_reset_session_only)
+ || (reset == ZSTD_reset_session_and_parameters) ) {
+ cctx->streamStage = zcss_init;
+ cctx->pledgedSrcSizePlusOne = 0;
+ }
+ if ( (reset == ZSTD_reset_parameters)
+ || (reset == ZSTD_reset_session_and_parameters) ) {
+ RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
+ "Can't reset parameters only when not in init stage.");
+ ZSTD_clearAllDicts(cctx);
+ return ZSTD_CCtxParams_reset(&cctx->requestedParams);
+ }
+ return 0;
+}
+
+
+/* ZSTD_checkCParams() :
+ control CParam values remain within authorized range.
+ @return : 0, or an error code if one value is beyond authorized range */
+size_t ZSTD_checkCParams(ZSTD_compressionParameters cParams)
+{
+ BOUNDCHECK(ZSTD_c_windowLog, (int)cParams.windowLog);
+ BOUNDCHECK(ZSTD_c_chainLog, (int)cParams.chainLog);
+ BOUNDCHECK(ZSTD_c_hashLog, (int)cParams.hashLog);
+ BOUNDCHECK(ZSTD_c_searchLog, (int)cParams.searchLog);
+ BOUNDCHECK(ZSTD_c_minMatch, (int)cParams.minMatch);
+ BOUNDCHECK(ZSTD_c_targetLength,(int)cParams.targetLength);
+ BOUNDCHECK(ZSTD_c_strategy, cParams.strategy);
+ return 0;
+}
+
+/* ZSTD_clampCParams() :
+ * make CParam values within valid range.
+ * @return : valid CParams */
+static ZSTD_compressionParameters
+ZSTD_clampCParams(ZSTD_compressionParameters cParams)
+{
+# define CLAMP_TYPE(cParam, val, type) { \
+ ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam); \
+ if ((int)val<bounds.lowerBound) val=(type)bounds.lowerBound; \
+ else if ((int)val>bounds.upperBound) val=(type)bounds.upperBound; \
+ }
+# define CLAMP(cParam, val) CLAMP_TYPE(cParam, val, unsigned)
+ CLAMP(ZSTD_c_windowLog, cParams.windowLog);
+ CLAMP(ZSTD_c_chainLog, cParams.chainLog);
+ CLAMP(ZSTD_c_hashLog, cParams.hashLog);
+ CLAMP(ZSTD_c_searchLog, cParams.searchLog);
+ CLAMP(ZSTD_c_minMatch, cParams.minMatch);
+ CLAMP(ZSTD_c_targetLength,cParams.targetLength);
+ CLAMP_TYPE(ZSTD_c_strategy,cParams.strategy, ZSTD_strategy);
+ return cParams;
+}
+
+/* ZSTD_cycleLog() :
+ * condition for correct operation : hashLog > 1 */
+U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat)
+{
+ U32 const btScale = ((U32)strat >= (U32)ZSTD_btlazy2);
+ return hashLog - btScale;
+}
+
+/* ZSTD_dictAndWindowLog() :
+ * Returns an adjusted window log that is large enough to fit the source and the dictionary.
+ * The zstd format says that the entire dictionary is valid if one byte of the dictionary
+ * is within the window. So the hashLog and chainLog should be large enough to reference both
+ * the dictionary and the window. So we must use this adjusted dictAndWindowLog when downsizing
+ * the hashLog and windowLog.
+ * NOTE: srcSize must not be ZSTD_CONTENTSIZE_UNKNOWN.
+ */
+static U32 ZSTD_dictAndWindowLog(U32 windowLog, U64 srcSize, U64 dictSize)
+{
+ const U64 maxWindowSize = 1ULL << ZSTD_WINDOWLOG_MAX;
+ /* No dictionary ==> No change */
+ if (dictSize == 0) {
+ return windowLog;
+ }
+ assert(windowLog <= ZSTD_WINDOWLOG_MAX);
+ assert(srcSize != ZSTD_CONTENTSIZE_UNKNOWN); /* Handled in ZSTD_adjustCParams_internal() */
+ {
+ U64 const windowSize = 1ULL << windowLog;
+ U64 const dictAndWindowSize = dictSize + windowSize;
+ /* If the window size is already large enough to fit both the source and the dictionary
+ * then just use the window size. Otherwise adjust so that it fits the dictionary and
+ * the window.
+ */
+ if (windowSize >= dictSize + srcSize) {
+ return windowLog; /* Window size large enough already */
+ } else if (dictAndWindowSize >= maxWindowSize) {
+ return ZSTD_WINDOWLOG_MAX; /* Larger than max window log */
+ } else {
+ return ZSTD_highbit32((U32)dictAndWindowSize - 1) + 1;
+ }
+ }
+}
+
+/* ZSTD_adjustCParams_internal() :
+ * optimize `cPar` for a specified input (`srcSize` and `dictSize`).
+ * mostly downsize to reduce memory consumption and initialization latency.
+ * `srcSize` can be ZSTD_CONTENTSIZE_UNKNOWN when not known.
+ * `mode` is the mode for parameter adjustment. See docs for `ZSTD_cParamMode_e`.
+ * note : `srcSize==0` means 0!
+ * condition : cPar is presumed validated (can be checked using ZSTD_checkCParams()). */
+static ZSTD_compressionParameters
+ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar,
+ unsigned long long srcSize,
+ size_t dictSize,
+ ZSTD_cParamMode_e mode)
+{
+ const U64 minSrcSize = 513; /* (1<<9) + 1 */
+ const U64 maxWindowResize = 1ULL << (ZSTD_WINDOWLOG_MAX-1);
+ assert(ZSTD_checkCParams(cPar)==0);
+
+ switch (mode) {
+ case ZSTD_cpm_unknown:
+ case ZSTD_cpm_noAttachDict:
+ /* If we don't know the source size, don't make any
+ * assumptions about it. We will already have selected
+ * smaller parameters if a dictionary is in use.
+ */
+ break;
+ case ZSTD_cpm_createCDict:
+ /* Assume a small source size when creating a dictionary
+ * with an unkown source size.
+ */
+ if (dictSize && srcSize == ZSTD_CONTENTSIZE_UNKNOWN)
+ srcSize = minSrcSize;
+ break;
+ case ZSTD_cpm_attachDict:
+ /* Dictionary has its own dedicated parameters which have
+ * already been selected. We are selecting parameters
+ * for only the source.
+ */
+ dictSize = 0;
+ break;
+ default:
+ assert(0);
+ break;
+ }
+
+ /* resize windowLog if input is small enough, to use less memory */
+ if ( (srcSize < maxWindowResize)
+ && (dictSize < maxWindowResize) ) {
+ U32 const tSize = (U32)(srcSize + dictSize);
+ static U32 const hashSizeMin = 1 << ZSTD_HASHLOG_MIN;
+ U32 const srcLog = (tSize < hashSizeMin) ? ZSTD_HASHLOG_MIN :
+ ZSTD_highbit32(tSize-1) + 1;
+ if (cPar.windowLog > srcLog) cPar.windowLog = srcLog;
+ }
+ if (srcSize != ZSTD_CONTENTSIZE_UNKNOWN) {
+ U32 const dictAndWindowLog = ZSTD_dictAndWindowLog(cPar.windowLog, (U64)srcSize, (U64)dictSize);
+ U32 const cycleLog = ZSTD_cycleLog(cPar.chainLog, cPar.strategy);
+ if (cPar.hashLog > dictAndWindowLog+1) cPar.hashLog = dictAndWindowLog+1;
+ if (cycleLog > dictAndWindowLog)
+ cPar.chainLog -= (cycleLog - dictAndWindowLog);
+ }
+
+ if (cPar.windowLog < ZSTD_WINDOWLOG_ABSOLUTEMIN)
+ cPar.windowLog = ZSTD_WINDOWLOG_ABSOLUTEMIN; /* minimum wlog required for valid frame header */
+
+ return cPar;
+}
+
+ZSTD_compressionParameters
+ZSTD_adjustCParams(ZSTD_compressionParameters cPar,
+ unsigned long long srcSize,
+ size_t dictSize)
+{
+ cPar = ZSTD_clampCParams(cPar); /* resulting cPar is necessarily valid (all parameters within range) */
+ if (srcSize == 0) srcSize = ZSTD_CONTENTSIZE_UNKNOWN;
+ return ZSTD_adjustCParams_internal(cPar, srcSize, dictSize, ZSTD_cpm_unknown);
+}
+
+static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode);
+static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode);
+
+static void ZSTD_overrideCParams(
+ ZSTD_compressionParameters* cParams,
+ const ZSTD_compressionParameters* overrides)
+{
+ if (overrides->windowLog) cParams->windowLog = overrides->windowLog;
+ if (overrides->hashLog) cParams->hashLog = overrides->hashLog;
+ if (overrides->chainLog) cParams->chainLog = overrides->chainLog;
+ if (overrides->searchLog) cParams->searchLog = overrides->searchLog;
+ if (overrides->minMatch) cParams->minMatch = overrides->minMatch;
+ if (overrides->targetLength) cParams->targetLength = overrides->targetLength;
+ if (overrides->strategy) cParams->strategy = overrides->strategy;
+}
+
+ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
+ const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode)
+{
+ ZSTD_compressionParameters cParams;
+ if (srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN && CCtxParams->srcSizeHint > 0) {
+ srcSizeHint = CCtxParams->srcSizeHint;
+ }
+ cParams = ZSTD_getCParams_internal(CCtxParams->compressionLevel, srcSizeHint, dictSize, mode);
+ if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
+ ZSTD_overrideCParams(&cParams, &CCtxParams->cParams);
+ assert(!ZSTD_checkCParams(cParams));
+ /* srcSizeHint == 0 means 0 */
+ return ZSTD_adjustCParams_internal(cParams, srcSizeHint, dictSize, mode);
+}
+
+static size_t
+ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
+ const U32 forCCtx)
+{
+ size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
+ size_t const hSize = ((size_t)1) << cParams->hashLog;
+ U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
+ size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
+ /* We don't use ZSTD_cwksp_alloc_size() here because the tables aren't
+ * surrounded by redzones in ASAN. */
+ size_t const tableSpace = chainSize * sizeof(U32)
+ + hSize * sizeof(U32)
+ + h3Size * sizeof(U32);
+ size_t const optPotentialSpace =
+ ZSTD_cwksp_alloc_size((MaxML+1) * sizeof(U32))
+ + ZSTD_cwksp_alloc_size((MaxLL+1) * sizeof(U32))
+ + ZSTD_cwksp_alloc_size((MaxOff+1) * sizeof(U32))
+ + ZSTD_cwksp_alloc_size((1<<Litbits) * sizeof(U32))
+ + ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t))
+ + ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
+ size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt))
+ ? optPotentialSpace
+ : 0;
+ DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u",
+ (U32)chainSize, (U32)hSize, (U32)h3Size);
+ return tableSpace + optSpace;
+}
+
+static size_t ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+ const ZSTD_compressionParameters* cParams,
+ const ldmParams_t* ldmParams,
+ const int isStatic,
+ const size_t buffInSize,
+ const size_t buffOutSize,
+ const U64 pledgedSrcSize)
+{
+ size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << cParams->windowLog), pledgedSrcSize));
+ size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
+ U32 const divider = (cParams->minMatch==3) ? 3 : 4;
+ size_t const maxNbSeq = blockSize / divider;
+ size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
+ + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
+ + 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
+ size_t const entropySpace = ZSTD_cwksp_alloc_size(ENTROPY_WORKSPACE_SIZE);
+ size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
+ size_t const matchStateSize = ZSTD_sizeof_matchState(cParams, /* forCCtx */ 1);
+
+ size_t const ldmSpace = ZSTD_ldm_getTableSize(*ldmParams);
+ size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(*ldmParams, blockSize);
+ size_t const ldmSeqSpace = ldmParams->enableLdm ?
+ ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq)) : 0;
+
+
+ size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize)
+ + ZSTD_cwksp_alloc_size(buffOutSize);
+
+ size_t const cctxSpace = isStatic ? ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)) : 0;
+
+ size_t const neededSpace =
+ cctxSpace +
+ entropySpace +
+ blockStateSpace +
+ ldmSpace +
+ ldmSeqSpace +
+ matchStateSize +
+ tokenSpace +
+ bufferSpace;
+
+ DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace);
+ return neededSpace;
+}
+
+size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
+{
+ ZSTD_compressionParameters const cParams =
+ ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict);
+
+ RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
+ /* estimateCCtxSize is for one-shot compression. So no buffers should
+ * be needed. However, we still allocate two 0-sized buffers, which can
+ * take space under ASAN. */
+ return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+ &cParams, ¶ms->ldmParams, 1, 0, 0, ZSTD_CONTENTSIZE_UNKNOWN);
+}
+
+size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
+{
+ ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
+ return ZSTD_estimateCCtxSize_usingCCtxParams(¶ms);
+}
+
+static size_t ZSTD_estimateCCtxSize_internal(int compressionLevel)
+{
+ int tier = 0;
+ size_t largestSize = 0;
+ static const unsigned long long srcSizeTiers[4] = {16 KB, 128 KB, 256 KB, ZSTD_CONTENTSIZE_UNKNOWN};
+ for (; tier < 4; ++tier) {
+ /* Choose the set of cParams for a given level across all srcSizes that give the largest cctxSize */
+ ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeTiers[tier], 0, ZSTD_cpm_noAttachDict);
+ largestSize = MAX(ZSTD_estimateCCtxSize_usingCParams(cParams), largestSize);
+ }
+ return largestSize;
+}
+
+size_t ZSTD_estimateCCtxSize(int compressionLevel)
+{
+ int level;
+ size_t memBudget = 0;
+ for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) {
+ /* Ensure monotonically increasing memory usage as compression level increases */
+ size_t const newMB = ZSTD_estimateCCtxSize_internal(level);
+ if (newMB > memBudget) memBudget = newMB;
+ }
+ return memBudget;
+}
+
+size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
+{
+ RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
+ { ZSTD_compressionParameters const cParams =
+ ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict);
+ size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, (size_t)1 << cParams.windowLog);
+ size_t const inBuffSize = (params->inBufferMode == ZSTD_bm_buffered)
+ ? ((size_t)1 << cParams.windowLog) + blockSize
+ : 0;
+ size_t const outBuffSize = (params->outBufferMode == ZSTD_bm_buffered)
+ ? ZSTD_compressBound(blockSize) + 1
+ : 0;
+
+ return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+ &cParams, ¶ms->ldmParams, 1, inBuffSize, outBuffSize,
+ ZSTD_CONTENTSIZE_UNKNOWN);
+ }
+}
+
+size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
+{
+ ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
+ return ZSTD_estimateCStreamSize_usingCCtxParams(¶ms);
+}
+
+static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel)
+{
+ ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict);
+ return ZSTD_estimateCStreamSize_usingCParams(cParams);
+}
+
+size_t ZSTD_estimateCStreamSize(int compressionLevel)
+{
+ int level;
+ size_t memBudget = 0;
+ for (level=MIN(compressionLevel, 1); level<=compressionLevel; level++) {
+ size_t const newMB = ZSTD_estimateCStreamSize_internal(level);
+ if (newMB > memBudget) memBudget = newMB;
+ }
+ return memBudget;
+}
+
+/* ZSTD_getFrameProgression():
+ * tells how much data has been consumed (input) and produced (output) for current frame.
+ * able to count progression inside worker threads (non-blocking mode).
+ */
+ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx)
+{
+ { ZSTD_frameProgression fp;
+ size_t const buffered = (cctx->inBuff == NULL) ? 0 :
+ cctx->inBuffPos - cctx->inToCompress;
+ if (buffered) assert(cctx->inBuffPos >= cctx->inToCompress);
+ assert(buffered <= ZSTD_BLOCKSIZE_MAX);
+ fp.ingested = cctx->consumedSrcSize + buffered;
+ fp.consumed = cctx->consumedSrcSize;
+ fp.produced = cctx->producedCSize;
+ fp.flushed = cctx->producedCSize; /* simplified; some data might still be left within streaming output buffer */
+ fp.currentJobID = 0;
+ fp.nbActiveWorkers = 0;
+ return fp;
+} }
+
+/*! ZSTD_toFlushNow()
+ * Only useful for multithreading scenarios currently (nbWorkers >= 1).
+ */
+size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx)
+{
+ (void)cctx;
+ return 0; /* over-simplification; could also check if context is currently running in streaming mode, and in which case, report how many bytes are left to be flushed within output buffer */
+}
+
+static void ZSTD_assertEqualCParams(ZSTD_compressionParameters cParams1,
+ ZSTD_compressionParameters cParams2)
+{
+ (void)cParams1;
+ (void)cParams2;
+ assert(cParams1.windowLog == cParams2.windowLog);
+ assert(cParams1.chainLog == cParams2.chainLog);
+ assert(cParams1.hashLog == cParams2.hashLog);
+ assert(cParams1.searchLog == cParams2.searchLog);
+ assert(cParams1.minMatch == cParams2.minMatch);
+ assert(cParams1.targetLength == cParams2.targetLength);
+ assert(cParams1.strategy == cParams2.strategy);
+}
+
+void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs)
+{
+ int i;
+ for (i = 0; i < ZSTD_REP_NUM; ++i)
+ bs->rep[i] = repStartValue[i];
+ bs->entropy.huf.repeatMode = HUF_repeat_none;
+ bs->entropy.fse.offcode_repeatMode = FSE_repeat_none;
+ bs->entropy.fse.matchlength_repeatMode = FSE_repeat_none;
+ bs->entropy.fse.litlength_repeatMode = FSE_repeat_none;
+}
+
+/*! ZSTD_invalidateMatchState()
+ * Invalidate all the matches in the match finder tables.
+ * Requires nextSrc and base to be set (can be NULL).
+ */
+static void ZSTD_invalidateMatchState(ZSTD_matchState_t* ms)
+{
+ ZSTD_window_clear(&ms->window);
+
+ ms->nextToUpdate = ms->window.dictLimit;
+ ms->loadedDictEnd = 0;
+ ms->opt.litLengthSum = 0; /* force reset of btopt stats */
+ ms->dictMatchState = NULL;
+}
+
+/*
+ * Controls, for this matchState reset, whether the tables need to be cleared /
+ * prepared for the coming compression (ZSTDcrp_makeClean), or whether the
+ * tables can be left unclean (ZSTDcrp_leaveDirty), because we know that a
+ * subsequent operation will overwrite the table space anyways (e.g., copying
+ * the matchState contents in from a CDict).
+ */
+typedef enum {
+ ZSTDcrp_makeClean,
+ ZSTDcrp_leaveDirty
+} ZSTD_compResetPolicy_e;
+
+/*
+ * Controls, for this matchState reset, whether indexing can continue where it
+ * left off (ZSTDirp_continue), or whether it needs to be restarted from zero
+ * (ZSTDirp_reset).
+ */
+typedef enum {
+ ZSTDirp_continue,
+ ZSTDirp_reset
+} ZSTD_indexResetPolicy_e;
+
+typedef enum {
+ ZSTD_resetTarget_CDict,
+ ZSTD_resetTarget_CCtx
+} ZSTD_resetTarget_e;
+
+static size_t
+ZSTD_reset_matchState(ZSTD_matchState_t* ms,
+ ZSTD_cwksp* ws,
+ const ZSTD_compressionParameters* cParams,
+ const ZSTD_compResetPolicy_e crp,
+ const ZSTD_indexResetPolicy_e forceResetIndex,
+ const ZSTD_resetTarget_e forWho)
+{
+ size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
+ size_t const hSize = ((size_t)1) << cParams->hashLog;
+ U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
+ size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
+
+ DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset);
+ if (forceResetIndex == ZSTDirp_reset) {
+ ZSTD_window_init(&ms->window);
+ ZSTD_cwksp_mark_tables_dirty(ws);
+ }
+
+ ms->hashLog3 = hashLog3;
+
+ ZSTD_invalidateMatchState(ms);
+
+ assert(!ZSTD_cwksp_reserve_failed(ws)); /* check that allocation hasn't already failed */
+
+ ZSTD_cwksp_clear_tables(ws);
+
+ DEBUGLOG(5, "reserving table space");
+ /* table Space */
+ ms->hashTable = (U32*)ZSTD_cwksp_reserve_table(ws, hSize * sizeof(U32));
+ ms->chainTable = (U32*)ZSTD_cwksp_reserve_table(ws, chainSize * sizeof(U32));
+ ms->hashTable3 = (U32*)ZSTD_cwksp_reserve_table(ws, h3Size * sizeof(U32));
+ RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
+ "failed a workspace allocation in ZSTD_reset_matchState");
+
+ DEBUGLOG(4, "reset table : %u", crp!=ZSTDcrp_leaveDirty);
+ if (crp!=ZSTDcrp_leaveDirty) {
+ /* reset tables only */
+ ZSTD_cwksp_clean_tables(ws);
+ }
+
+ /* opt parser space */
+ if ((forWho == ZSTD_resetTarget_CCtx) && (cParams->strategy >= ZSTD_btopt)) {
+ DEBUGLOG(4, "reserving optimal parser space");
+ ms->opt.litFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (1<<Litbits) * sizeof(unsigned));
+ ms->opt.litLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxLL+1) * sizeof(unsigned));
+ ms->opt.matchLengthFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxML+1) * sizeof(unsigned));
+ ms->opt.offCodeFreq = (unsigned*)ZSTD_cwksp_reserve_aligned(ws, (MaxOff+1) * sizeof(unsigned));
+ ms->opt.matchTable = (ZSTD_match_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t));
+ ms->opt.priceTable = (ZSTD_optimal_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
+ }
+
+ ms->cParams = *cParams;
+
+ RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
+ "failed a workspace allocation in ZSTD_reset_matchState");
+
+ return 0;
+}
+
+/* ZSTD_indexTooCloseToMax() :
+ * minor optimization : prefer memset() rather than reduceIndex()
+ * which is measurably slow in some circumstances (reported for Visual Studio).
+ * Works when re-using a context for a lot of smallish inputs :
+ * if all inputs are smaller than ZSTD_INDEXOVERFLOW_MARGIN,
+ * memset() will be triggered before reduceIndex().
+ */
+#define ZSTD_INDEXOVERFLOW_MARGIN (16 MB)
+static int ZSTD_indexTooCloseToMax(ZSTD_window_t w)
+{
+ return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN);
+}
+
+/*! ZSTD_resetCCtx_internal() :
+ note : `params` are assumed fully validated at this stage */
+static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
+ ZSTD_CCtx_params params,
+ U64 const pledgedSrcSize,
+ ZSTD_compResetPolicy_e const crp,
+ ZSTD_buffered_policy_e const zbuff)
+{
+ ZSTD_cwksp* const ws = &zc->workspace;
+ DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u",
+ (U32)pledgedSrcSize, params.cParams.windowLog);
+ assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
+
+ zc->isFirstBlock = 1;
+
+ if (params.ldmParams.enableLdm) {
+ /* Adjust long distance matching parameters */
+ ZSTD_ldm_adjustParameters(¶ms.ldmParams, ¶ms.cParams);
+ assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog);
+ assert(params.ldmParams.hashRateLog < 32);
+ }
+
+ { size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize));
+ size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
+ U32 const divider = (params.cParams.minMatch==3) ? 3 : 4;
+ size_t const maxNbSeq = blockSize / divider;
+ size_t const buffOutSize = (zbuff == ZSTDb_buffered && params.outBufferMode == ZSTD_bm_buffered)
+ ? ZSTD_compressBound(blockSize) + 1
+ : 0;
+ size_t const buffInSize = (zbuff == ZSTDb_buffered && params.inBufferMode == ZSTD_bm_buffered)
+ ? windowSize + blockSize
+ : 0;
+ size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);
+
+ int const indexTooClose = ZSTD_indexTooCloseToMax(zc->blockState.matchState.window);
+ ZSTD_indexResetPolicy_e needsIndexReset =
+ (!indexTooClose && zc->initialized) ? ZSTDirp_continue : ZSTDirp_reset;
+
+ size_t const neededSpace =
+ ZSTD_estimateCCtxSize_usingCCtxParams_internal(
+ ¶ms.cParams, ¶ms.ldmParams, zc->staticSize != 0,
+ buffInSize, buffOutSize, pledgedSrcSize);
+ FORWARD_IF_ERROR(neededSpace, "cctx size estimate failed!");
+
+ if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0);
+
+ /* Check if workspace is large enough, alloc a new one if needed */
+ {
+ int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace;
+ int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace);
+
+ DEBUGLOG(4, "Need %zu B workspace", neededSpace);
+ DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize);
+
+ if (workspaceTooSmall || workspaceWasteful) {
+ DEBUGLOG(4, "Resize workspaceSize from %zuKB to %zuKB",
+ ZSTD_cwksp_sizeof(ws) >> 10,
+ neededSpace >> 10);
+
+ RETURN_ERROR_IF(zc->staticSize, memory_allocation, "static cctx : no resize");
+
+ needsIndexReset = ZSTDirp_reset;
+
+ ZSTD_cwksp_free(ws, zc->customMem);
+ FORWARD_IF_ERROR(ZSTD_cwksp_create(ws, neededSpace, zc->customMem), "");
+
+ DEBUGLOG(5, "reserving object space");
+ /* Statically sized space.
+ * entropyWorkspace never moves,
+ * though prev/next block swap places */
+ assert(ZSTD_cwksp_check_available(ws, 2 * sizeof(ZSTD_compressedBlockState_t)));
+ zc->blockState.prevCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
+ RETURN_ERROR_IF(zc->blockState.prevCBlock == NULL, memory_allocation, "couldn't allocate prevCBlock");
+ zc->blockState.nextCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
+ RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate nextCBlock");
+ zc->entropyWorkspace = (U32*) ZSTD_cwksp_reserve_object(ws, ENTROPY_WORKSPACE_SIZE);
+ RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate entropyWorkspace");
+ } }
+
+ ZSTD_cwksp_clear(ws);
+
+ /* init params */
+ zc->appliedParams = params;
+ zc->blockState.matchState.cParams = params.cParams;
+ zc->pledgedSrcSizePlusOne = pledgedSrcSize+1;
+ zc->consumedSrcSize = 0;
+ zc->producedCSize = 0;
+ if (pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN)
+ zc->appliedParams.fParams.contentSizeFlag = 0;
+ DEBUGLOG(4, "pledged content size : %u ; flag : %u",
+ (unsigned)pledgedSrcSize, zc->appliedParams.fParams.contentSizeFlag);
+ zc->blockSize = blockSize;
+
+ xxh64_reset(&zc->xxhState, 0);
+ zc->stage = ZSTDcs_init;
+ zc->dictID = 0;
+ zc->dictContentSize = 0;
+
+ ZSTD_reset_compressedBlockState(zc->blockState.prevCBlock);
+
+ /* ZSTD_wildcopy() is used to copy into the literals buffer,
+ * so we have to oversize the buffer by WILDCOPY_OVERLENGTH bytes.
+ */
+ zc->seqStore.litStart = ZSTD_cwksp_reserve_buffer(ws, blockSize + WILDCOPY_OVERLENGTH);
+ zc->seqStore.maxNbLit = blockSize;
+
+ /* buffers */
+ zc->bufferedPolicy = zbuff;
+ zc->inBuffSize = buffInSize;
+ zc->inBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffInSize);
+ zc->outBuffSize = buffOutSize;
+ zc->outBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize);
+
+ /* ldm bucketOffsets table */
+ if (params.ldmParams.enableLdm) {
+ /* TODO: avoid memset? */
+ size_t const numBuckets =
+ ((size_t)1) << (params.ldmParams.hashLog -
+ params.ldmParams.bucketSizeLog);
+ zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, numBuckets);
+ ZSTD_memset(zc->ldmState.bucketOffsets, 0, numBuckets);
+ }
+
+ /* sequences storage */
+ ZSTD_referenceExternalSequences(zc, NULL, 0);
+ zc->seqStore.maxNbSeq = maxNbSeq;
+ zc->seqStore.llCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
+ zc->seqStore.mlCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
+ zc->seqStore.ofCode = ZSTD_cwksp_reserve_buffer(ws, maxNbSeq * sizeof(BYTE));
+ zc->seqStore.sequencesStart = (seqDef*)ZSTD_cwksp_reserve_aligned(ws, maxNbSeq * sizeof(seqDef));
+
+ FORWARD_IF_ERROR(ZSTD_reset_matchState(
+ &zc->blockState.matchState,
+ ws,
+ ¶ms.cParams,
+ crp,
+ needsIndexReset,
+ ZSTD_resetTarget_CCtx), "");
+
+ /* ldm hash table */
+ if (params.ldmParams.enableLdm) {
+ /* TODO: avoid memset? */
+ size_t const ldmHSize = ((size_t)1) << params.ldmParams.hashLog;
+ zc->ldmState.hashTable = (ldmEntry_t*)ZSTD_cwksp_reserve_aligned(ws, ldmHSize * sizeof(ldmEntry_t));
+ ZSTD_memset(zc->ldmState.hashTable, 0, ldmHSize * sizeof(ldmEntry_t));
+ zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq * sizeof(rawSeq));
+ zc->maxNbLdmSequences = maxNbLdmSeq;
+
+ ZSTD_window_init(&zc->ldmState.window);
+ ZSTD_window_clear(&zc->ldmState.window);
+ zc->ldmState.loadedDictEnd = 0;
+ }
+
+ /* Due to alignment, when reusing a workspace, we can actually consume
+ * up to 3 extra bytes for alignment. See the comments in zstd_cwksp.h
+ */
+ assert(ZSTD_cwksp_used(ws) >= neededSpace &&
+ ZSTD_cwksp_used(ws) <= neededSpace + 3);
+
+ DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
+ zc->initialized = 1;
+
+ return 0;
+ }
+}
+
+/* ZSTD_invalidateRepCodes() :
+ * ensures next compression will not use repcodes from previous block.
+ * Note : only works with regular variant;
+ * do not use with extDict variant ! */
+void ZSTD_invalidateRepCodes(ZSTD_CCtx* cctx) {
+ int i;
+ for (i=0; i<ZSTD_REP_NUM; i++) cctx->blockState.prevCBlock->rep[i] = 0;
+ assert(!ZSTD_window_hasExtDict(cctx->blockState.matchState.window));
+}
+
+/* These are the approximate sizes for each strategy past which copying the
+ * dictionary tables into the working context is faster than using them
+ * in-place.
+ */
+static const size_t attachDictSizeCutoffs[ZSTD_STRATEGY_MAX+1] = {
+ 8 KB, /* unused */
+ 8 KB, /* ZSTD_fast */
+ 16 KB, /* ZSTD_dfast */
+ 32 KB, /* ZSTD_greedy */
+ 32 KB, /* ZSTD_lazy */
+ 32 KB, /* ZSTD_lazy2 */
+ 32 KB, /* ZSTD_btlazy2 */
+ 32 KB, /* ZSTD_btopt */
+ 8 KB, /* ZSTD_btultra */
+ 8 KB /* ZSTD_btultra2 */
+};
+
+static int ZSTD_shouldAttachDict(const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params,
+ U64 pledgedSrcSize)
+{
+ size_t cutoff = attachDictSizeCutoffs[cdict->matchState.cParams.strategy];
+ int const dedicatedDictSearch = cdict->matchState.dedicatedDictSearch;
+ return dedicatedDictSearch
+ || ( ( pledgedSrcSize <= cutoff
+ || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
+ || params->attachDictPref == ZSTD_dictForceAttach )
+ && params->attachDictPref != ZSTD_dictForceCopy
+ && !params->forceWindow ); /* dictMatchState isn't correctly
+ * handled in _enforceMaxDist */
+}
+
+static size_t
+ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
+ const ZSTD_CDict* cdict,
+ ZSTD_CCtx_params params,
+ U64 pledgedSrcSize,
+ ZSTD_buffered_policy_e zbuff)
+{
+ {
+ ZSTD_compressionParameters adjusted_cdict_cParams = cdict->matchState.cParams;
+ unsigned const windowLog = params.cParams.windowLog;
+ assert(windowLog != 0);
+ /* Resize working context table params for input only, since the dict
+ * has its own tables. */
+ /* pledgedSrcSize == 0 means 0! */
+
+ if (cdict->matchState.dedicatedDictSearch) {
+ ZSTD_dedicatedDictSearch_revertCParams(&adjusted_cdict_cParams);
+ }
+
+ params.cParams = ZSTD_adjustCParams_internal(adjusted_cdict_cParams, pledgedSrcSize,
+ cdict->dictContentSize, ZSTD_cpm_attachDict);
+ params.cParams.windowLog = windowLog;
+ FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
+ ZSTDcrp_makeClean, zbuff), "");
+ assert(cctx->appliedParams.cParams.strategy == adjusted_cdict_cParams.strategy);
+ }
+
+ { const U32 cdictEnd = (U32)( cdict->matchState.window.nextSrc
+ - cdict->matchState.window.base);
+ const U32 cdictLen = cdictEnd - cdict->matchState.window.dictLimit;
+ if (cdictLen == 0) {
+ /* don't even attach dictionaries with no contents */
+ DEBUGLOG(4, "skipping attaching empty dictionary");
+ } else {
+ DEBUGLOG(4, "attaching dictionary into context");
+ cctx->blockState.matchState.dictMatchState = &cdict->matchState;
+
+ /* prep working match state so dict matches never have negative indices
+ * when they are translated to the working context's index space. */
+ if (cctx->blockState.matchState.window.dictLimit < cdictEnd) {
+ cctx->blockState.matchState.window.nextSrc =
+ cctx->blockState.matchState.window.base + cdictEnd;
+ ZSTD_window_clear(&cctx->blockState.matchState.window);
+ }
+ /* loadedDictEnd is expressed within the referential of the active context */
+ cctx->blockState.matchState.loadedDictEnd = cctx->blockState.matchState.window.dictLimit;
+ } }
+
+ cctx->dictID = cdict->dictID;
+ cctx->dictContentSize = cdict->dictContentSize;
+
+ /* copy block state */
+ ZSTD_memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState));
+
+ return 0;
+}
+
+static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx,
+ const ZSTD_CDict* cdict,
+ ZSTD_CCtx_params params,
+ U64 pledgedSrcSize,
+ ZSTD_buffered_policy_e zbuff)
+{
+ const ZSTD_compressionParameters *cdict_cParams = &cdict->matchState.cParams;
+
+ assert(!cdict->matchState.dedicatedDictSearch);
+
+ DEBUGLOG(4, "copying dictionary into context");
+
+ { unsigned const windowLog = params.cParams.windowLog;
+ assert(windowLog != 0);
+ /* Copy only compression parameters related to tables. */
+ params.cParams = *cdict_cParams;
+ params.cParams.windowLog = windowLog;
+ FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
+ ZSTDcrp_leaveDirty, zbuff), "");
+ assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
+ assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog);
+ assert(cctx->appliedParams.cParams.chainLog == cdict_cParams->chainLog);
+ }
+
+ ZSTD_cwksp_mark_tables_dirty(&cctx->workspace);
+
+ /* copy tables */
+ { size_t const chainSize = (cdict_cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cdict_cParams->chainLog);
+ size_t const hSize = (size_t)1 << cdict_cParams->hashLog;
+
+ ZSTD_memcpy(cctx->blockState.matchState.hashTable,
+ cdict->matchState.hashTable,
+ hSize * sizeof(U32));
+ ZSTD_memcpy(cctx->blockState.matchState.chainTable,
+ cdict->matchState.chainTable,
+ chainSize * sizeof(U32));
+ }
+
+ /* Zero the hashTable3, since the cdict never fills it */
+ { int const h3log = cctx->blockState.matchState.hashLog3;
+ size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
+ assert(cdict->matchState.hashLog3 == 0);
+ ZSTD_memset(cctx->blockState.matchState.hashTable3, 0, h3Size * sizeof(U32));
+ }
+
+ ZSTD_cwksp_mark_tables_clean(&cctx->workspace);
+
+ /* copy dictionary offsets */
+ { ZSTD_matchState_t const* srcMatchState = &cdict->matchState;
+ ZSTD_matchState_t* dstMatchState = &cctx->blockState.matchState;
+ dstMatchState->window = srcMatchState->window;
+ dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
+ dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
+ }
+
+ cctx->dictID = cdict->dictID;
+ cctx->dictContentSize = cdict->dictContentSize;
+
+ /* copy block state */
+ ZSTD_memcpy(cctx->blockState.prevCBlock, &cdict->cBlockState, sizeof(cdict->cBlockState));
+
+ return 0;
+}
+
+/* We have a choice between copying the dictionary context into the working
+ * context, or referencing the dictionary context from the working context
+ * in-place. We decide here which strategy to use. */
+static size_t ZSTD_resetCCtx_usingCDict(ZSTD_CCtx* cctx,
+ const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params,
+ U64 pledgedSrcSize,
+ ZSTD_buffered_policy_e zbuff)
+{
+
+ DEBUGLOG(4, "ZSTD_resetCCtx_usingCDict (pledgedSrcSize=%u)",
+ (unsigned)pledgedSrcSize);
+
+ if (ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize)) {
+ return ZSTD_resetCCtx_byAttachingCDict(
+ cctx, cdict, *params, pledgedSrcSize, zbuff);
+ } else {
+ return ZSTD_resetCCtx_byCopyingCDict(
+ cctx, cdict, *params, pledgedSrcSize, zbuff);
+ }
+}
+
+/*! ZSTD_copyCCtx_internal() :
+ * Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
+ * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
+ * The "context", in this case, refers to the hash and chain tables,
+ * entropy tables, and dictionary references.
+ * `windowLog` value is enforced if != 0, otherwise value is copied from srcCCtx.
+ * @return : 0, or an error code */
+static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx,
+ const ZSTD_CCtx* srcCCtx,
+ ZSTD_frameParameters fParams,
+ U64 pledgedSrcSize,
+ ZSTD_buffered_policy_e zbuff)
+{
+ DEBUGLOG(5, "ZSTD_copyCCtx_internal");
+ RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong,
+ "Can't copy a ctx that's not in init stage.");
+
+ ZSTD_memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
+ { ZSTD_CCtx_params params = dstCCtx->requestedParams;
+ /* Copy only compression parameters related to tables. */
+ params.cParams = srcCCtx->appliedParams.cParams;
+ params.fParams = fParams;
+ ZSTD_resetCCtx_internal(dstCCtx, params, pledgedSrcSize,
+ ZSTDcrp_leaveDirty, zbuff);
+ assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog);
+ assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy);
+ assert(dstCCtx->appliedParams.cParams.hashLog == srcCCtx->appliedParams.cParams.hashLog);
+ assert(dstCCtx->appliedParams.cParams.chainLog == srcCCtx->appliedParams.cParams.chainLog);
+ assert(dstCCtx->blockState.matchState.hashLog3 == srcCCtx->blockState.matchState.hashLog3);
+ }
+
+ ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace);
+
+ /* copy tables */
+ { size_t const chainSize = (srcCCtx->appliedParams.cParams.strategy == ZSTD_fast) ? 0 : ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog);
+ size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog;
+ int const h3log = srcCCtx->blockState.matchState.hashLog3;
+ size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
+
+ ZSTD_memcpy(dstCCtx->blockState.matchState.hashTable,
+ srcCCtx->blockState.matchState.hashTable,
+ hSize * sizeof(U32));
+ ZSTD_memcpy(dstCCtx->blockState.matchState.chainTable,
+ srcCCtx->blockState.matchState.chainTable,
+ chainSize * sizeof(U32));
+ ZSTD_memcpy(dstCCtx->blockState.matchState.hashTable3,
+ srcCCtx->blockState.matchState.hashTable3,
+ h3Size * sizeof(U32));
+ }
+
+ ZSTD_cwksp_mark_tables_clean(&dstCCtx->workspace);
+
+ /* copy dictionary offsets */
+ {
+ const ZSTD_matchState_t* srcMatchState = &srcCCtx->blockState.matchState;
+ ZSTD_matchState_t* dstMatchState = &dstCCtx->blockState.matchState;
+ dstMatchState->window = srcMatchState->window;
+ dstMatchState->nextToUpdate = srcMatchState->nextToUpdate;
+ dstMatchState->loadedDictEnd= srcMatchState->loadedDictEnd;
+ }
+ dstCCtx->dictID = srcCCtx->dictID;
+ dstCCtx->dictContentSize = srcCCtx->dictContentSize;
+
+ /* copy block state */
+ ZSTD_memcpy(dstCCtx->blockState.prevCBlock, srcCCtx->blockState.prevCBlock, sizeof(*srcCCtx->blockState.prevCBlock));
+
+ return 0;
+}
+
+/*! ZSTD_copyCCtx() :
+ * Duplicate an existing context `srcCCtx` into another one `dstCCtx`.
+ * Only works during stage ZSTDcs_init (i.e. after creation, but before first call to ZSTD_compressContinue()).
+ * pledgedSrcSize==0 means "unknown".
+* @return : 0, or an error code */
+size_t ZSTD_copyCCtx(ZSTD_CCtx* dstCCtx, const ZSTD_CCtx* srcCCtx, unsigned long long pledgedSrcSize)
+{
+ ZSTD_frameParameters fParams = { 1 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ };
+ ZSTD_buffered_policy_e const zbuff = srcCCtx->bufferedPolicy;
+ ZSTD_STATIC_ASSERT((U32)ZSTDb_buffered==1);
+ if (pledgedSrcSize==0) pledgedSrcSize = ZSTD_CONTENTSIZE_UNKNOWN;
+ fParams.contentSizeFlag = (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN);
+
+ return ZSTD_copyCCtx_internal(dstCCtx, srcCCtx,
+ fParams, pledgedSrcSize,
+ zbuff);
+}
+
+
+#define ZSTD_ROWSIZE 16
+/*! ZSTD_reduceTable() :
+ * reduce table indexes by `reducerValue`, or squash to zero.
+ * PreserveMark preserves "unsorted mark" for btlazy2 strategy.
+ * It must be set to a clear 0/1 value, to remove branch during inlining.
+ * Presume table size is a multiple of ZSTD_ROWSIZE
+ * to help auto-vectorization */
+FORCE_INLINE_TEMPLATE void
+ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerValue, int const preserveMark)
+{
+ int const nbRows = (int)size / ZSTD_ROWSIZE;
+ int cellNb = 0;
+ int rowNb;
+ assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */
+ assert(size < (1U<<31)); /* can be casted to int */
+
+
+ for (rowNb=0 ; rowNb < nbRows ; rowNb++) {
+ int column;
+ for (column=0; column<ZSTD_ROWSIZE; column++) {
+ if (preserveMark) {
+ U32 const adder = (table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) ? reducerValue : 0;
+ table[cellNb] += adder;
+ }
+ if (table[cellNb] < reducerValue) table[cellNb] = 0;
+ else table[cellNb] -= reducerValue;
+ cellNb++;
+ } }
+}
+
+static void ZSTD_reduceTable(U32* const table, U32 const size, U32 const reducerValue)
+{
+ ZSTD_reduceTable_internal(table, size, reducerValue, 0);
+}
+
+static void ZSTD_reduceTable_btlazy2(U32* const table, U32 const size, U32 const reducerValue)
+{
+ ZSTD_reduceTable_internal(table, size, reducerValue, 1);
+}
+
+/*! ZSTD_reduceIndex() :
+* rescale all indexes to avoid future overflow (indexes are U32) */
+static void ZSTD_reduceIndex (ZSTD_matchState_t* ms, ZSTD_CCtx_params const* params, const U32 reducerValue)
+{
+ { U32 const hSize = (U32)1 << params->cParams.hashLog;
+ ZSTD_reduceTable(ms->hashTable, hSize, reducerValue);
+ }
+
+ if (params->cParams.strategy != ZSTD_fast) {
+ U32 const chainSize = (U32)1 << params->cParams.chainLog;
+ if (params->cParams.strategy == ZSTD_btlazy2)
+ ZSTD_reduceTable_btlazy2(ms->chainTable, chainSize, reducerValue);
+ else
+ ZSTD_reduceTable(ms->chainTable, chainSize, reducerValue);
+ }
+
+ if (ms->hashLog3) {
+ U32 const h3Size = (U32)1 << ms->hashLog3;
+ ZSTD_reduceTable(ms->hashTable3, h3Size, reducerValue);
+ }
+}
+
+
+/*-*******************************************************
+* Block entropic compression
+*********************************************************/
+
+/* See doc/zstd_compression_format.md for detailed format description */
+
+void ZSTD_seqToCodes(const seqStore_t* seqStorePtr)
+{
+ const seqDef* const sequences = seqStorePtr->sequencesStart;
+ BYTE* const llCodeTable = seqStorePtr->llCode;
+ BYTE* const ofCodeTable = seqStorePtr->ofCode;
+ BYTE* const mlCodeTable = seqStorePtr->mlCode;
+ U32 const nbSeq = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ U32 u;
+ assert(nbSeq <= seqStorePtr->maxNbSeq);
+ for (u=0; u<nbSeq; u++) {
+ U32 const llv = sequences[u].litLength;
+ U32 const mlv = sequences[u].matchLength;
+ llCodeTable[u] = (BYTE)ZSTD_LLcode(llv);
+ ofCodeTable[u] = (BYTE)ZSTD_highbit32(sequences[u].offset);
+ mlCodeTable[u] = (BYTE)ZSTD_MLcode(mlv);
+ }
+ if (seqStorePtr->longLengthID==1)
+ llCodeTable[seqStorePtr->longLengthPos] = MaxLL;
+ if (seqStorePtr->longLengthID==2)
+ mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
+}
+
+/* ZSTD_useTargetCBlockSize():
+ * Returns if target compressed block size param is being used.
+ * If used, compression will do best effort to make a compressed block size to be around targetCBlockSize.
+ * Returns 1 if true, 0 otherwise. */
+static int ZSTD_useTargetCBlockSize(const ZSTD_CCtx_params* cctxParams)
+{
+ DEBUGLOG(5, "ZSTD_useTargetCBlockSize (targetCBlockSize=%zu)", cctxParams->targetCBlockSize);
+ return (cctxParams->targetCBlockSize != 0);
+}
+
+/* ZSTD_entropyCompressSequences_internal():
+ * actually compresses both literals and sequences */
+MEM_STATIC size_t
+ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
+ const ZSTD_entropyCTables_t* prevEntropy,
+ ZSTD_entropyCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ void* dst, size_t dstCapacity,
+ void* entropyWorkspace, size_t entropyWkspSize,
+ const int bmi2)
+{
+ const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
+ ZSTD_strategy const strategy = cctxParams->cParams.strategy;
+ unsigned* count = (unsigned*)entropyWorkspace;
+ FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable;
+ FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable;
+ FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable;
+ U32 LLtype, Offtype, MLtype; /* compressed, raw or rle */
+ const seqDef* const sequences = seqStorePtr->sequencesStart;
+ const BYTE* const ofCodeTable = seqStorePtr->ofCode;
+ const BYTE* const llCodeTable = seqStorePtr->llCode;
+ const BYTE* const mlCodeTable = seqStorePtr->mlCode;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstCapacity;
+ BYTE* op = ostart;
+ size_t const nbSeq = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ BYTE* seqHead;
+ BYTE* lastNCount = NULL;
+
+ entropyWorkspace = count + (MaxSeq + 1);
+ entropyWkspSize -= (MaxSeq + 1) * sizeof(*count);
+
+ DEBUGLOG(4, "ZSTD_entropyCompressSequences_internal (nbSeq=%zu)", nbSeq);
+ ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
+ assert(entropyWkspSize >= HUF_WORKSPACE_SIZE);
+
+ /* Compress literals */
+ { const BYTE* const literals = seqStorePtr->litStart;
+ size_t const litSize = (size_t)(seqStorePtr->lit - literals);
+ size_t const cSize = ZSTD_compressLiterals(
+ &prevEntropy->huf, &nextEntropy->huf,
+ cctxParams->cParams.strategy,
+ ZSTD_disableLiteralsCompression(cctxParams),
+ op, dstCapacity,
+ literals, litSize,
+ entropyWorkspace, entropyWkspSize,
+ bmi2);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressLiterals failed");
+ assert(cSize <= dstCapacity);
+ op += cSize;
+ }
+
+ /* Sequences Header */
+ RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/,
+ dstSize_tooSmall, "Can't fit seq hdr in output buf!");
+ if (nbSeq < 128) {
+ *op++ = (BYTE)nbSeq;
+ } else if (nbSeq < LONGNBSEQ) {
+ op[0] = (BYTE)((nbSeq>>8) + 0x80);
+ op[1] = (BYTE)nbSeq;
+ op+=2;
+ } else {
+ op[0]=0xFF;
+ MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ));
+ op+=3;
+ }
+ assert(op <= oend);
+ if (nbSeq==0) {
+ /* Copy the old tables over as if we repeated them */
+ ZSTD_memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse));
+ return (size_t)(op - ostart);
+ }
+
+ /* seqHead : flags for FSE encoding type */
+ seqHead = op++;
+ assert(op <= oend);
+
+ /* convert length/distances into codes */
+ ZSTD_seqToCodes(seqStorePtr);
+ /* build CTable for Literal Lengths */
+ { unsigned max = MaxLL;
+ size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ DEBUGLOG(5, "Building LL table");
+ nextEntropy->fse.litlength_repeatMode = prevEntropy->fse.litlength_repeatMode;
+ LLtype = ZSTD_selectEncodingType(&nextEntropy->fse.litlength_repeatMode,
+ count, max, mostFrequent, nbSeq,
+ LLFSELog, prevEntropy->fse.litlengthCTable,
+ LL_defaultNorm, LL_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(set_basic < set_compressed && set_rle < set_compressed);
+ assert(!(LLtype < set_compressed && nextEntropy->fse.litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
+ count, max, llCodeTable, nbSeq,
+ LL_defaultNorm, LL_defaultNormLog, MaxLL,
+ prevEntropy->fse.litlengthCTable,
+ sizeof(prevEntropy->fse.litlengthCTable),
+ entropyWorkspace, entropyWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
+ if (LLtype == set_compressed)
+ lastNCount = op;
+ op += countSize;
+ assert(op <= oend);
+ } }
+ /* build CTable for Offsets */
+ { unsigned max = MaxOff;
+ size_t const mostFrequent = HIST_countFast_wksp(
+ count, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
+ ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
+ DEBUGLOG(5, "Building OF table");
+ nextEntropy->fse.offcode_repeatMode = prevEntropy->fse.offcode_repeatMode;
+ Offtype = ZSTD_selectEncodingType(&nextEntropy->fse.offcode_repeatMode,
+ count, max, mostFrequent, nbSeq,
+ OffFSELog, prevEntropy->fse.offcodeCTable,
+ OF_defaultNorm, OF_defaultNormLog,
+ defaultPolicy, strategy);
+ assert(!(Offtype < set_compressed && nextEntropy->fse.offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
+ count, max, ofCodeTable, nbSeq,
+ OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+ prevEntropy->fse.offcodeCTable,
+ sizeof(prevEntropy->fse.offcodeCTable),
+ entropyWorkspace, entropyWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
+ if (Offtype == set_compressed)
+ lastNCount = op;
+ op += countSize;
+ assert(op <= oend);
+ } }
+ /* build CTable for MatchLengths */
+ { unsigned max = MaxML;
+ size_t const mostFrequent = HIST_countFast_wksp(
+ count, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
+ nextEntropy->fse.matchlength_repeatMode = prevEntropy->fse.matchlength_repeatMode;
+ MLtype = ZSTD_selectEncodingType(&nextEntropy->fse.matchlength_repeatMode,
+ count, max, mostFrequent, nbSeq,
+ MLFSELog, prevEntropy->fse.matchlengthCTable,
+ ML_defaultNorm, ML_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(!(MLtype < set_compressed && nextEntropy->fse.matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
+ count, max, mlCodeTable, nbSeq,
+ ML_defaultNorm, ML_defaultNormLog, MaxML,
+ prevEntropy->fse.matchlengthCTable,
+ sizeof(prevEntropy->fse.matchlengthCTable),
+ entropyWorkspace, entropyWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
+ if (MLtype == set_compressed)
+ lastNCount = op;
+ op += countSize;
+ assert(op <= oend);
+ } }
+
+ *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
+
+ { size_t const bitstreamSize = ZSTD_encodeSequences(
+ op, (size_t)(oend - op),
+ CTable_MatchLength, mlCodeTable,
+ CTable_OffsetBits, ofCodeTable,
+ CTable_LitLength, llCodeTable,
+ sequences, nbSeq,
+ longOffsets, bmi2);
+ FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
+ op += bitstreamSize;
+ assert(op <= oend);
+ /* zstd versions <= 1.3.4 mistakenly report corruption when
+ * FSE_readNCount() receives a buffer < 4 bytes.
+ * Fixed by https://github.com/facebook/zstd/pull/1146.
+ * This can happen when the last set_compressed table present is 2
+ * bytes and the bitstream is only one byte.
+ * In this exceedingly rare case, we will simply emit an uncompressed
+ * block, since it isn't worth optimizing.
+ */
+ if (lastNCount && (op - lastNCount) < 4) {
+ /* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
+ assert(op - lastNCount == 3);
+ DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
+ "emitting an uncompressed block.");
+ return 0;
+ }
+ }
+
+ DEBUGLOG(5, "compressed block size : %u", (unsigned)(op - ostart));
+ return (size_t)(op - ostart);
+}
+
+MEM_STATIC size_t
+ZSTD_entropyCompressSequences(seqStore_t* seqStorePtr,
+ const ZSTD_entropyCTables_t* prevEntropy,
+ ZSTD_entropyCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ void* dst, size_t dstCapacity,
+ size_t srcSize,
+ void* entropyWorkspace, size_t entropyWkspSize,
+ int bmi2)
+{
+ size_t const cSize = ZSTD_entropyCompressSequences_internal(
+ seqStorePtr, prevEntropy, nextEntropy, cctxParams,
+ dst, dstCapacity,
+ entropyWorkspace, entropyWkspSize, bmi2);
+ if (cSize == 0) return 0;
+ /* When srcSize <= dstCapacity, there is enough space to write a raw uncompressed block.
+ * Since we ran out of space, block must be not compressible, so fall back to raw uncompressed block.
+ */
+ if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity))
+ return 0; /* block not compressed */
+ FORWARD_IF_ERROR(cSize, "ZSTD_entropyCompressSequences_internal failed");
+
+ /* Check compressibility */
+ { size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, cctxParams->cParams.strategy);
+ if (cSize >= maxCSize) return 0; /* block not compressed */
+ }
+ DEBUGLOG(4, "ZSTD_entropyCompressSequences() cSize: %zu\n", cSize);
+ return cSize;
+}
+
+/* ZSTD_selectBlockCompressor() :
+ * Not static, but internal use only (used by long distance matcher)
+ * assumption : strat is a valid strategy */
+ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode)
+{
+ static const ZSTD_blockCompressor blockCompressor[4][ZSTD_STRATEGY_MAX+1] = {
+ { ZSTD_compressBlock_fast /* default for 0 */,
+ ZSTD_compressBlock_fast,
+ ZSTD_compressBlock_doubleFast,
+ ZSTD_compressBlock_greedy,
+ ZSTD_compressBlock_lazy,
+ ZSTD_compressBlock_lazy2,
+ ZSTD_compressBlock_btlazy2,
+ ZSTD_compressBlock_btopt,
+ ZSTD_compressBlock_btultra,
+ ZSTD_compressBlock_btultra2 },
+ { ZSTD_compressBlock_fast_extDict /* default for 0 */,
+ ZSTD_compressBlock_fast_extDict,
+ ZSTD_compressBlock_doubleFast_extDict,
+ ZSTD_compressBlock_greedy_extDict,
+ ZSTD_compressBlock_lazy_extDict,
+ ZSTD_compressBlock_lazy2_extDict,
+ ZSTD_compressBlock_btlazy2_extDict,
+ ZSTD_compressBlock_btopt_extDict,
+ ZSTD_compressBlock_btultra_extDict,
+ ZSTD_compressBlock_btultra_extDict },
+ { ZSTD_compressBlock_fast_dictMatchState /* default for 0 */,
+ ZSTD_compressBlock_fast_dictMatchState,
+ ZSTD_compressBlock_doubleFast_dictMatchState,
+ ZSTD_compressBlock_greedy_dictMatchState,
+ ZSTD_compressBlock_lazy_dictMatchState,
+ ZSTD_compressBlock_lazy2_dictMatchState,
+ ZSTD_compressBlock_btlazy2_dictMatchState,
+ ZSTD_compressBlock_btopt_dictMatchState,
+ ZSTD_compressBlock_btultra_dictMatchState,
+ ZSTD_compressBlock_btultra_dictMatchState },
+ { NULL /* default for 0 */,
+ NULL,
+ NULL,
+ ZSTD_compressBlock_greedy_dedicatedDictSearch,
+ ZSTD_compressBlock_lazy_dedicatedDictSearch,
+ ZSTD_compressBlock_lazy2_dedicatedDictSearch,
+ NULL,
+ NULL,
+ NULL,
+ NULL }
+ };
+ ZSTD_blockCompressor selectedCompressor;
+ ZSTD_STATIC_ASSERT((unsigned)ZSTD_fast == 1);
+
+ assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
+ selectedCompressor = blockCompressor[(int)dictMode][(int)strat];
+ assert(selectedCompressor != NULL);
+ return selectedCompressor;
+}
+
+static void ZSTD_storeLastLiterals(seqStore_t* seqStorePtr,
+ const BYTE* anchor, size_t lastLLSize)
+{
+ ZSTD_memcpy(seqStorePtr->lit, anchor, lastLLSize);
+ seqStorePtr->lit += lastLLSize;
+}
+
+void ZSTD_resetSeqStore(seqStore_t* ssPtr)
+{
+ ssPtr->lit = ssPtr->litStart;
+ ssPtr->sequences = ssPtr->sequencesStart;
+ ssPtr->longLengthID = 0;
+}
+
+typedef enum { ZSTDbss_compress, ZSTDbss_noCompress } ZSTD_buildSeqStore_e;
+
+static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize)
+{
+ ZSTD_matchState_t* const ms = &zc->blockState.matchState;
+ DEBUGLOG(5, "ZSTD_buildSeqStore (srcSize=%zu)", srcSize);
+ assert(srcSize <= ZSTD_BLOCKSIZE_MAX);
+ /* Assert that we have correctly flushed the ctx params into the ms's copy */
+ ZSTD_assertEqualCParams(zc->appliedParams.cParams, ms->cParams);
+ if (srcSize < MIN_CBLOCK_SIZE+ZSTD_blockHeaderSize+1) {
+ if (zc->appliedParams.cParams.strategy >= ZSTD_btopt) {
+ ZSTD_ldm_skipRawSeqStoreBytes(&zc->externSeqStore, srcSize);
+ } else {
+ ZSTD_ldm_skipSequences(&zc->externSeqStore, srcSize, zc->appliedParams.cParams.minMatch);
+ }
+ return ZSTDbss_noCompress; /* don't even attempt compression below a certain srcSize */
+ }
+ ZSTD_resetSeqStore(&(zc->seqStore));
+ /* required for optimal parser to read stats from dictionary */
+ ms->opt.symbolCosts = &zc->blockState.prevCBlock->entropy;
+ /* tell the optimal parser how we expect to compress literals */
+ ms->opt.literalCompressionMode = zc->appliedParams.literalCompressionMode;
+ /* a gap between an attached dict and the current window is not safe,
+ * they must remain adjacent,
+ * and when that stops being the case, the dict must be unset */
+ assert(ms->dictMatchState == NULL || ms->loadedDictEnd == ms->window.dictLimit);
+
+ /* limited update after a very long match */
+ { const BYTE* const base = ms->window.base;
+ const BYTE* const istart = (const BYTE*)src;
+ const U32 curr = (U32)(istart-base);
+ if (sizeof(ptrdiff_t)==8) assert(istart - base < (ptrdiff_t)(U32)(-1)); /* ensure no overflow */
+ if (curr > ms->nextToUpdate + 384)
+ ms->nextToUpdate = curr - MIN(192, (U32)(curr - ms->nextToUpdate - 384));
+ }
+
+ /* select and store sequences */
+ { ZSTD_dictMode_e const dictMode = ZSTD_matchState_dictMode(ms);
+ size_t lastLLSize;
+ { int i;
+ for (i = 0; i < ZSTD_REP_NUM; ++i)
+ zc->blockState.nextCBlock->rep[i] = zc->blockState.prevCBlock->rep[i];
+ }
+ if (zc->externSeqStore.pos < zc->externSeqStore.size) {
+ assert(!zc->appliedParams.ldmParams.enableLdm);
+ /* Updates ldmSeqStore.pos */
+ lastLLSize =
+ ZSTD_ldm_blockCompress(&zc->externSeqStore,
+ ms, &zc->seqStore,
+ zc->blockState.nextCBlock->rep,
+ src, srcSize);
+ assert(zc->externSeqStore.pos <= zc->externSeqStore.size);
+ } else if (zc->appliedParams.ldmParams.enableLdm) {
+ rawSeqStore_t ldmSeqStore = kNullRawSeqStore;
+
+ ldmSeqStore.seq = zc->ldmSequences;
+ ldmSeqStore.capacity = zc->maxNbLdmSequences;
+ /* Updates ldmSeqStore.size */
+ FORWARD_IF_ERROR(ZSTD_ldm_generateSequences(&zc->ldmState, &ldmSeqStore,
+ &zc->appliedParams.ldmParams,
+ src, srcSize), "");
+ /* Updates ldmSeqStore.pos */
+ lastLLSize =
+ ZSTD_ldm_blockCompress(&ldmSeqStore,
+ ms, &zc->seqStore,
+ zc->blockState.nextCBlock->rep,
+ src, srcSize);
+ assert(ldmSeqStore.pos == ldmSeqStore.size);
+ } else { /* not long range mode */
+ ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy, dictMode);
+ ms->ldmSeqStore = NULL;
+ lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize);
+ }
+ { const BYTE* const lastLiterals = (const BYTE*)src + srcSize - lastLLSize;
+ ZSTD_storeLastLiterals(&zc->seqStore, lastLiterals, lastLLSize);
+ } }
+ return ZSTDbss_compress;
+}
+
+static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc)
+{
+ const seqStore_t* seqStore = ZSTD_getSeqStore(zc);
+ const seqDef* seqStoreSeqs = seqStore->sequencesStart;
+ size_t seqStoreSeqSize = seqStore->sequences - seqStoreSeqs;
+ size_t seqStoreLiteralsSize = (size_t)(seqStore->lit - seqStore->litStart);
+ size_t literalsRead = 0;
+ size_t lastLLSize;
+
+ ZSTD_Sequence* outSeqs = &zc->seqCollector.seqStart[zc->seqCollector.seqIndex];
+ size_t i;
+ repcodes_t updatedRepcodes;
+
+ assert(zc->seqCollector.seqIndex + 1 < zc->seqCollector.maxSequences);
+ /* Ensure we have enough space for last literals "sequence" */
+ assert(zc->seqCollector.maxSequences >= seqStoreSeqSize + 1);
+ ZSTD_memcpy(updatedRepcodes.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t));
+ for (i = 0; i < seqStoreSeqSize; ++i) {
+ U32 rawOffset = seqStoreSeqs[i].offset - ZSTD_REP_NUM;
+ outSeqs[i].litLength = seqStoreSeqs[i].litLength;
+ outSeqs[i].matchLength = seqStoreSeqs[i].matchLength + MINMATCH;
+ outSeqs[i].rep = 0;
+
+ if (i == seqStore->longLengthPos) {
+ if (seqStore->longLengthID == 1) {
+ outSeqs[i].litLength += 0x10000;
+ } else if (seqStore->longLengthID == 2) {
+ outSeqs[i].matchLength += 0x10000;
+ }
+ }
+
+ if (seqStoreSeqs[i].offset <= ZSTD_REP_NUM) {
+ /* Derive the correct offset corresponding to a repcode */
+ outSeqs[i].rep = seqStoreSeqs[i].offset;
+ if (outSeqs[i].litLength != 0) {
+ rawOffset = updatedRepcodes.rep[outSeqs[i].rep - 1];
+ } else {
+ if (outSeqs[i].rep == 3) {
+ rawOffset = updatedRepcodes.rep[0] - 1;
+ } else {
+ rawOffset = updatedRepcodes.rep[outSeqs[i].rep];
+ }
+ }
+ }
+ outSeqs[i].offset = rawOffset;
+ /* seqStoreSeqs[i].offset == offCode+1, and ZSTD_updateRep() expects offCode
+ so we provide seqStoreSeqs[i].offset - 1 */
+ updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep,
+ seqStoreSeqs[i].offset - 1,
+ seqStoreSeqs[i].litLength == 0);
+ literalsRead += outSeqs[i].litLength;
+ }
+ /* Insert last literals (if any exist) in the block as a sequence with ml == off == 0.
+ * If there are no last literals, then we'll emit (of: 0, ml: 0, ll: 0), which is a marker
+ * for the block boundary, according to the API.
+ */
+ assert(seqStoreLiteralsSize >= literalsRead);
+ lastLLSize = seqStoreLiteralsSize - literalsRead;
+ outSeqs[i].litLength = (U32)lastLLSize;
+ outSeqs[i].matchLength = outSeqs[i].offset = outSeqs[i].rep = 0;
+ seqStoreSeqSize++;
+ zc->seqCollector.seqIndex += seqStoreSeqSize;
+}
+
+size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
+ size_t outSeqsSize, const void* src, size_t srcSize)
+{
+ const size_t dstCapacity = ZSTD_compressBound(srcSize);
+ void* dst = ZSTD_customMalloc(dstCapacity, ZSTD_defaultCMem);
+ SeqCollector seqCollector;
+
+ RETURN_ERROR_IF(dst == NULL, memory_allocation, "NULL pointer!");
+
+ seqCollector.collectSequences = 1;
+ seqCollector.seqStart = outSeqs;
+ seqCollector.seqIndex = 0;
+ seqCollector.maxSequences = outSeqsSize;
+ zc->seqCollector = seqCollector;
+
+ ZSTD_compress2(zc, dst, dstCapacity, src, srcSize);
+ ZSTD_customFree(dst, ZSTD_defaultCMem);
+ return zc->seqCollector.seqIndex;
+}
+
+size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize) {
+ size_t in = 0;
+ size_t out = 0;
+ for (; in < seqsSize; ++in) {
+ if (sequences[in].offset == 0 && sequences[in].matchLength == 0) {
+ if (in != seqsSize - 1) {
+ sequences[in+1].litLength += sequences[in].litLength;
+ }
+ } else {
+ sequences[out] = sequences[in];
+ ++out;
+ }
+ }
+ return out;
+}
+
+/* Unrolled loop to read four size_ts of input at a time. Returns 1 if is RLE, 0 if not. */
+static int ZSTD_isRLE(const BYTE* src, size_t length) {
+ const BYTE* ip = src;
+ const BYTE value = ip[0];
+ const size_t valueST = (size_t)((U64)value * 0x0101010101010101ULL);
+ const size_t unrollSize = sizeof(size_t) * 4;
+ const size_t unrollMask = unrollSize - 1;
+ const size_t prefixLength = length & unrollMask;
+ size_t i;
+ size_t u;
+ if (length == 1) return 1;
+ /* Check if prefix is RLE first before using unrolled loop */
+ if (prefixLength && ZSTD_count(ip+1, ip, ip+prefixLength) != prefixLength-1) {
+ return 0;
+ }
+ for (i = prefixLength; i != length; i += unrollSize) {
+ for (u = 0; u < unrollSize; u += sizeof(size_t)) {
+ if (MEM_readST(ip + i + u) != valueST) {
+ return 0;
+ }
+ }
+ }
+ return 1;
+}
+
+/* Returns true if the given block may be RLE.
+ * This is just a heuristic based on the compressibility.
+ * It may return both false positives and false negatives.
+ */
+static int ZSTD_maybeRLE(seqStore_t const* seqStore)
+{
+ size_t const nbSeqs = (size_t)(seqStore->sequences - seqStore->sequencesStart);
+ size_t const nbLits = (size_t)(seqStore->lit - seqStore->litStart);
+
+ return nbSeqs < 4 && nbLits < 10;
+}
+
+static void ZSTD_confirmRepcodesAndEntropyTables(ZSTD_CCtx* zc)
+{
+ ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
+ zc->blockState.prevCBlock = zc->blockState.nextCBlock;
+ zc->blockState.nextCBlock = tmp;
+}
+
+static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, U32 frame)
+{
+ /* This the upper bound for the length of an rle block.
+ * This isn't the actual upper bound. Finding the real threshold
+ * needs further investigation.
+ */
+ const U32 rleMaxLength = 25;
+ size_t cSize;
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* op = (BYTE*)dst;
+ DEBUGLOG(5, "ZSTD_compressBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)",
+ (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit,
+ (unsigned)zc->blockState.matchState.nextToUpdate);
+
+ { const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
+ FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
+ if (bss == ZSTDbss_noCompress) { cSize = 0; goto out; }
+ }
+
+ if (zc->seqCollector.collectSequences) {
+ ZSTD_copyBlockSequences(zc);
+ ZSTD_confirmRepcodesAndEntropyTables(zc);
+ return 0;
+ }
+
+ /* encode sequences and literals */
+ cSize = ZSTD_entropyCompressSequences(&zc->seqStore,
+ &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy,
+ &zc->appliedParams,
+ dst, dstCapacity,
+ srcSize,
+ zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
+ zc->bmi2);
+
+ if (zc->seqCollector.collectSequences) {
+ ZSTD_copyBlockSequences(zc);
+ return 0;
+ }
+
+
+ if (frame &&
+ /* We don't want to emit our first block as a RLE even if it qualifies because
+ * doing so will cause the decoder (cli only) to throw a "should consume all input error."
+ * This is only an issue for zstd <= v1.4.3
+ */
+ !zc->isFirstBlock &&
+ cSize < rleMaxLength &&
+ ZSTD_isRLE(ip, srcSize))
+ {
+ cSize = 1;
+ op[0] = ip[0];
+ }
+
+out:
+ if (!ZSTD_isError(cSize) && cSize > 1) {
+ ZSTD_confirmRepcodesAndEntropyTables(zc);
+ }
+ /* We check that dictionaries have offset codes available for the first
+ * block. After the first block, the offcode table might not have large
+ * enough codes to represent the offsets in the data.
+ */
+ if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+ zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+
+ return cSize;
+}
+
+static size_t ZSTD_compressBlock_targetCBlockSize_body(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const size_t bss, U32 lastBlock)
+{
+ DEBUGLOG(6, "Attempting ZSTD_compressSuperBlock()");
+ if (bss == ZSTDbss_compress) {
+ if (/* We don't want to emit our first block as a RLE even if it qualifies because
+ * doing so will cause the decoder (cli only) to throw a "should consume all input error."
+ * This is only an issue for zstd <= v1.4.3
+ */
+ !zc->isFirstBlock &&
+ ZSTD_maybeRLE(&zc->seqStore) &&
+ ZSTD_isRLE((BYTE const*)src, srcSize))
+ {
+ return ZSTD_rleCompressBlock(dst, dstCapacity, *(BYTE const*)src, srcSize, lastBlock);
+ }
+ /* Attempt superblock compression.
+ *
+ * Note that compressed size of ZSTD_compressSuperBlock() is not bound by the
+ * standard ZSTD_compressBound(). This is a problem, because even if we have
+ * space now, taking an extra byte now could cause us to run out of space later
+ * and violate ZSTD_compressBound().
+ *
+ * Define blockBound(blockSize) = blockSize + ZSTD_blockHeaderSize.
+ *
+ * In order to respect ZSTD_compressBound() we must attempt to emit a raw
+ * uncompressed block in these cases:
+ * * cSize == 0: Return code for an uncompressed block.
+ * * cSize == dstSize_tooSmall: We may have expanded beyond blockBound(srcSize).
+ * ZSTD_noCompressBlock() will return dstSize_tooSmall if we are really out of
+ * output space.
+ * * cSize >= blockBound(srcSize): We have expanded the block too much so
+ * emit an uncompressed block.
+ */
+ {
+ size_t const cSize = ZSTD_compressSuperBlock(zc, dst, dstCapacity, src, srcSize, lastBlock);
+ if (cSize != ERROR(dstSize_tooSmall)) {
+ size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressSuperBlock failed");
+ if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) {
+ ZSTD_confirmRepcodesAndEntropyTables(zc);
+ return cSize;
+ }
+ }
+ }
+ }
+
+ DEBUGLOG(6, "Resorting to ZSTD_noCompressBlock()");
+ /* Superblock compression failed, attempt to emit a single no compress block.
+ * The decoder will be able to stream this block since it is uncompressed.
+ */
+ return ZSTD_noCompressBlock(dst, dstCapacity, src, srcSize, lastBlock);
+}
+
+static size_t ZSTD_compressBlock_targetCBlockSize(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ U32 lastBlock)
+{
+ size_t cSize = 0;
+ const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
+ DEBUGLOG(5, "ZSTD_compressBlock_targetCBlockSize (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u, srcSize=%zu)",
+ (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit, (unsigned)zc->blockState.matchState.nextToUpdate, srcSize);
+ FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
+
+ cSize = ZSTD_compressBlock_targetCBlockSize_body(zc, dst, dstCapacity, src, srcSize, bss, lastBlock);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize_body failed");
+
+ if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+ zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+
+ return cSize;
+}
+
+static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
+ ZSTD_cwksp* ws,
+ ZSTD_CCtx_params const* params,
+ void const* ip,
+ void const* iend)
+{
+ if (ZSTD_window_needOverflowCorrection(ms->window, iend)) {
+ U32 const maxDist = (U32)1 << params->cParams.windowLog;
+ U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy);
+ U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip);
+ ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30);
+ ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30);
+ ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX <= 31);
+ ZSTD_cwksp_mark_tables_dirty(ws);
+ ZSTD_reduceIndex(ms, params, correction);
+ ZSTD_cwksp_mark_tables_clean(ws);
+ if (ms->nextToUpdate < correction) ms->nextToUpdate = 0;
+ else ms->nextToUpdate -= correction;
+ /* invalidate dictionaries on overflow correction */
+ ms->loadedDictEnd = 0;
+ ms->dictMatchState = NULL;
+ }
+}
+
+/*! ZSTD_compress_frameChunk() :
+* Compress a chunk of data into one or multiple blocks.
+* All blocks will be terminated, all input will be consumed.
+* Function will issue an error if there is not enough `dstCapacity` to hold the compressed content.
+* Frame is supposed already started (header already produced)
+* @return : compressed size, or an error code
+*/
+static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ U32 lastFrameChunk)
+{
+ size_t blockSize = cctx->blockSize;
+ size_t remaining = srcSize;
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* op = ostart;
+ U32 const maxDist = (U32)1 << cctx->appliedParams.cParams.windowLog;
+
+ assert(cctx->appliedParams.cParams.windowLog <= ZSTD_WINDOWLOG_MAX);
+
+ DEBUGLOG(4, "ZSTD_compress_frameChunk (blockSize=%u)", (unsigned)blockSize);
+ if (cctx->appliedParams.fParams.checksumFlag && srcSize)
+ xxh64_update(&cctx->xxhState, src, srcSize);
+
+ while (remaining) {
+ ZSTD_matchState_t* const ms = &cctx->blockState.matchState;
+ U32 const lastBlock = lastFrameChunk & (blockSize >= remaining);
+
+ RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize + MIN_CBLOCK_SIZE,
+ dstSize_tooSmall,
+ "not enough space to store compressed block");
+ if (remaining < blockSize) blockSize = remaining;
+
+ ZSTD_overflowCorrectIfNeeded(
+ ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize);
+ ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
+
+ /* Ensure hash/chain table insertion resumes no sooner than lowlimit */
+ if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;
+
+ { size_t cSize;
+ if (ZSTD_useTargetCBlockSize(&cctx->appliedParams)) {
+ cSize = ZSTD_compressBlock_targetCBlockSize(cctx, op, dstCapacity, ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize failed");
+ assert(cSize > 0);
+ assert(cSize <= blockSize + ZSTD_blockHeaderSize);
+ } else {
+ cSize = ZSTD_compressBlock_internal(cctx,
+ op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
+ ip, blockSize, 1 /* frame */);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_internal failed");
+
+ if (cSize == 0) { /* block is not compressible */
+ cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
+ } else {
+ U32 const cBlockHeader = cSize == 1 ?
+ lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
+ lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
+ MEM_writeLE24(op, cBlockHeader);
+ cSize += ZSTD_blockHeaderSize;
+ }
+ }
+
+
+ ip += blockSize;
+ assert(remaining >= blockSize);
+ remaining -= blockSize;
+ op += cSize;
+ assert(dstCapacity >= cSize);
+ dstCapacity -= cSize;
+ cctx->isFirstBlock = 0;
+ DEBUGLOG(5, "ZSTD_compress_frameChunk: adding a block of size %u",
+ (unsigned)cSize);
+ } }
+
+ if (lastFrameChunk && (op>ostart)) cctx->stage = ZSTDcs_ending;
+ return (size_t)(op-ostart);
+}
+
+
+static size_t ZSTD_writeFrameHeader(void* dst, size_t dstCapacity,
+ const ZSTD_CCtx_params* params, U64 pledgedSrcSize, U32 dictID)
+{ BYTE* const op = (BYTE*)dst;
+ U32 const dictIDSizeCodeLength = (dictID>0) + (dictID>=256) + (dictID>=65536); /* 0-3 */
+ U32 const dictIDSizeCode = params->fParams.noDictIDFlag ? 0 : dictIDSizeCodeLength; /* 0-3 */
+ U32 const checksumFlag = params->fParams.checksumFlag>0;
+ U32 const windowSize = (U32)1 << params->cParams.windowLog;
+ U32 const singleSegment = params->fParams.contentSizeFlag && (windowSize >= pledgedSrcSize);
+ BYTE const windowLogByte = (BYTE)((params->cParams.windowLog - ZSTD_WINDOWLOG_ABSOLUTEMIN) << 3);
+ U32 const fcsCode = params->fParams.contentSizeFlag ?
+ (pledgedSrcSize>=256) + (pledgedSrcSize>=65536+256) + (pledgedSrcSize>=0xFFFFFFFFU) : 0; /* 0-3 */
+ BYTE const frameHeaderDescriptionByte = (BYTE)(dictIDSizeCode + (checksumFlag<<2) + (singleSegment<<5) + (fcsCode<<6) );
+ size_t pos=0;
+
+ assert(!(params->fParams.contentSizeFlag && pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN));
+ RETURN_ERROR_IF(dstCapacity < ZSTD_FRAMEHEADERSIZE_MAX, dstSize_tooSmall,
+ "dst buf is too small to fit worst-case frame header size.");
+ DEBUGLOG(4, "ZSTD_writeFrameHeader : dictIDFlag : %u ; dictID : %u ; dictIDSizeCode : %u",
+ !params->fParams.noDictIDFlag, (unsigned)dictID, (unsigned)dictIDSizeCode);
+ if (params->format == ZSTD_f_zstd1) {
+ MEM_writeLE32(dst, ZSTD_MAGICNUMBER);
+ pos = 4;
+ }
+ op[pos++] = frameHeaderDescriptionByte;
+ if (!singleSegment) op[pos++] = windowLogByte;
+ switch(dictIDSizeCode)
+ {
+ default:
+ assert(0); /* impossible */
+ ZSTD_FALLTHROUGH;
+ case 0 : break;
+ case 1 : op[pos] = (BYTE)(dictID); pos++; break;
+ case 2 : MEM_writeLE16(op+pos, (U16)dictID); pos+=2; break;
+ case 3 : MEM_writeLE32(op+pos, dictID); pos+=4; break;
+ }
+ switch(fcsCode)
+ {
+ default:
+ assert(0); /* impossible */
+ ZSTD_FALLTHROUGH;
+ case 0 : if (singleSegment) op[pos++] = (BYTE)(pledgedSrcSize); break;
+ case 1 : MEM_writeLE16(op+pos, (U16)(pledgedSrcSize-256)); pos+=2; break;
+ case 2 : MEM_writeLE32(op+pos, (U32)(pledgedSrcSize)); pos+=4; break;
+ case 3 : MEM_writeLE64(op+pos, (U64)(pledgedSrcSize)); pos+=8; break;
+ }
+ return pos;
+}
+
+/* ZSTD_writeSkippableFrame_advanced() :
+ * Writes out a skippable frame with the specified magic number variant (16 are supported),
+ * from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15, and the desired source data.
+ *
+ * Returns the total number of bytes written, or a ZSTD error code.
+ */
+size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, unsigned magicVariant) {
+ BYTE* op = (BYTE*)dst;
+ RETURN_ERROR_IF(dstCapacity < srcSize + ZSTD_SKIPPABLEHEADERSIZE /* Skippable frame overhead */,
+ dstSize_tooSmall, "Not enough room for skippable frame");
+ RETURN_ERROR_IF(srcSize > (unsigned)0xFFFFFFFF, srcSize_wrong, "Src size too large for skippable frame");
+ RETURN_ERROR_IF(magicVariant > 15, parameter_outOfBound, "Skippable frame magic number variant not supported");
+
+ MEM_writeLE32(op, (U32)(ZSTD_MAGIC_SKIPPABLE_START + magicVariant));
+ MEM_writeLE32(op+4, (U32)srcSize);
+ ZSTD_memcpy(op+8, src, srcSize);
+ return srcSize + ZSTD_SKIPPABLEHEADERSIZE;
+}
+
+/* ZSTD_writeLastEmptyBlock() :
+ * output an empty Block with end-of-frame mark to complete a frame
+ * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h))
+ * or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize)
+ */
+size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity)
+{
+ RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall,
+ "dst buf is too small to write frame trailer empty block.");
+ { U32 const cBlockHeader24 = 1 /*lastBlock*/ + (((U32)bt_raw)<<1); /* 0 size */
+ MEM_writeLE24(dst, cBlockHeader24);
+ return ZSTD_blockHeaderSize;
+ }
+}
+
+size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq)
+{
+ RETURN_ERROR_IF(cctx->stage != ZSTDcs_init, stage_wrong,
+ "wrong cctx stage");
+ RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm,
+ parameter_unsupported,
+ "incompatible with ldm");
+ cctx->externSeqStore.seq = seq;
+ cctx->externSeqStore.size = nbSeq;
+ cctx->externSeqStore.capacity = nbSeq;
+ cctx->externSeqStore.pos = 0;
+ cctx->externSeqStore.posInSequence = 0;
+ return 0;
+}
+
+
+static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ U32 frame, U32 lastFrameChunk)
+{
+ ZSTD_matchState_t* const ms = &cctx->blockState.matchState;
+ size_t fhSize = 0;
+
+ DEBUGLOG(5, "ZSTD_compressContinue_internal, stage: %u, srcSize: %u",
+ cctx->stage, (unsigned)srcSize);
+ RETURN_ERROR_IF(cctx->stage==ZSTDcs_created, stage_wrong,
+ "missing init (ZSTD_compressBegin)");
+
+ if (frame && (cctx->stage==ZSTDcs_init)) {
+ fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams,
+ cctx->pledgedSrcSizePlusOne-1, cctx->dictID);
+ FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
+ assert(fhSize <= dstCapacity);
+ dstCapacity -= fhSize;
+ dst = (char*)dst + fhSize;
+ cctx->stage = ZSTDcs_ongoing;
+ }
+
+ if (!srcSize) return fhSize; /* do not generate an empty block if no input */
+
+ if (!ZSTD_window_update(&ms->window, src, srcSize)) {
+ ms->nextToUpdate = ms->window.dictLimit;
+ }
+ if (cctx->appliedParams.ldmParams.enableLdm) {
+ ZSTD_window_update(&cctx->ldmState.window, src, srcSize);
+ }
+
+ if (!frame) {
+ /* overflow check and correction for block mode */
+ ZSTD_overflowCorrectIfNeeded(
+ ms, &cctx->workspace, &cctx->appliedParams,
+ src, (BYTE const*)src + srcSize);
+ }
+
+ DEBUGLOG(5, "ZSTD_compressContinue_internal (blockSize=%u)", (unsigned)cctx->blockSize);
+ { size_t const cSize = frame ?
+ ZSTD_compress_frameChunk (cctx, dst, dstCapacity, src, srcSize, lastFrameChunk) :
+ ZSTD_compressBlock_internal (cctx, dst, dstCapacity, src, srcSize, 0 /* frame */);
+ FORWARD_IF_ERROR(cSize, "%s", frame ? "ZSTD_compress_frameChunk failed" : "ZSTD_compressBlock_internal failed");
+ cctx->consumedSrcSize += srcSize;
+ cctx->producedCSize += (cSize + fhSize);
+ assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
+ if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */
+ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
+ RETURN_ERROR_IF(
+ cctx->consumedSrcSize+1 > cctx->pledgedSrcSizePlusOne,
+ srcSize_wrong,
+ "error : pledgedSrcSize = %u, while realSrcSize >= %u",
+ (unsigned)cctx->pledgedSrcSizePlusOne-1,
+ (unsigned)cctx->consumedSrcSize);
+ }
+ return cSize + fhSize;
+ }
+}
+
+size_t ZSTD_compressContinue (ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_compressContinue (srcSize=%u)", (unsigned)srcSize);
+ return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 1 /* frame mode */, 0 /* last chunk */);
+}
+
+
+size_t ZSTD_getBlockSize(const ZSTD_CCtx* cctx)
+{
+ ZSTD_compressionParameters const cParams = cctx->appliedParams.cParams;
+ assert(!ZSTD_checkCParams(cParams));
+ return MIN (ZSTD_BLOCKSIZE_MAX, (U32)1 << cParams.windowLog);
+}
+
+size_t ZSTD_compressBlock(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_compressBlock: srcSize = %u", (unsigned)srcSize);
+ { size_t const blockSizeMax = ZSTD_getBlockSize(cctx);
+ RETURN_ERROR_IF(srcSize > blockSizeMax, srcSize_wrong, "input is larger than a block"); }
+
+ return ZSTD_compressContinue_internal(cctx, dst, dstCapacity, src, srcSize, 0 /* frame mode */, 0 /* last chunk */);
+}
+
+/*! ZSTD_loadDictionaryContent() :
+ * @return : 0, or an error code
+ */
+static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
+ ldmState_t* ls,
+ ZSTD_cwksp* ws,
+ ZSTD_CCtx_params const* params,
+ const void* src, size_t srcSize,
+ ZSTD_dictTableLoadMethod_e dtlm)
+{
+ const BYTE* ip = (const BYTE*) src;
+ const BYTE* const iend = ip + srcSize;
+
+ ZSTD_window_update(&ms->window, src, srcSize);
+ ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);
+
+ if (params->ldmParams.enableLdm && ls != NULL) {
+ ZSTD_window_update(&ls->window, src, srcSize);
+ ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base);
+ }
+
+ /* Assert that we the ms params match the params we're being given */
+ ZSTD_assertEqualCParams(params->cParams, ms->cParams);
+
+ if (srcSize <= HASH_READ_SIZE) return 0;
+
+ while (iend - ip > HASH_READ_SIZE) {
+ size_t const remaining = (size_t)(iend - ip);
+ size_t const chunk = MIN(remaining, ZSTD_CHUNKSIZE_MAX);
+ const BYTE* const ichunk = ip + chunk;
+
+ ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, ichunk);
+
+ if (params->ldmParams.enableLdm && ls != NULL)
+ ZSTD_ldm_fillHashTable(ls, (const BYTE*)src, (const BYTE*)src + srcSize, ¶ms->ldmParams);
+
+ switch(params->cParams.strategy)
+ {
+ case ZSTD_fast:
+ ZSTD_fillHashTable(ms, ichunk, dtlm);
+ break;
+ case ZSTD_dfast:
+ ZSTD_fillDoubleHashTable(ms, ichunk, dtlm);
+ break;
+
+ case ZSTD_greedy:
+ case ZSTD_lazy:
+ case ZSTD_lazy2:
+ if (chunk >= HASH_READ_SIZE && ms->dedicatedDictSearch) {
+ assert(chunk == remaining); /* must load everything in one go */
+ ZSTD_dedicatedDictSearch_lazy_loadDictionary(ms, ichunk-HASH_READ_SIZE);
+ } else if (chunk >= HASH_READ_SIZE) {
+ ZSTD_insertAndFindFirstIndex(ms, ichunk-HASH_READ_SIZE);
+ }
+ break;
+
+ case ZSTD_btlazy2: /* we want the dictionary table fully sorted */
+ case ZSTD_btopt:
+ case ZSTD_btultra:
+ case ZSTD_btultra2:
+ if (chunk >= HASH_READ_SIZE)
+ ZSTD_updateTree(ms, ichunk-HASH_READ_SIZE, ichunk);
+ break;
+
+ default:
+ assert(0); /* not possible : not a valid strategy id */
+ }
+
+ ip = ichunk;
+ }
+
+ ms->nextToUpdate = (U32)(iend - ms->window.base);
+ return 0;
+}
+
+
+/* Dictionaries that assign zero probability to symbols that show up causes problems
+ * when FSE encoding. Mark dictionaries with zero probability symbols as FSE_repeat_check
+ * and only dictionaries with 100% valid symbols can be assumed valid.
+ */
+static FSE_repeat ZSTD_dictNCountRepeat(short* normalizedCounter, unsigned dictMaxSymbolValue, unsigned maxSymbolValue)
+{
+ U32 s;
+ if (dictMaxSymbolValue < maxSymbolValue) {
+ return FSE_repeat_check;
+ }
+ for (s = 0; s <= maxSymbolValue; ++s) {
+ if (normalizedCounter[s] == 0) {
+ return FSE_repeat_check;
+ }
+ }
+ return FSE_repeat_valid;
+}
+
+size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
+ const void* const dict, size_t dictSize)
+{
+ short offcodeNCount[MaxOff+1];
+ unsigned offcodeMaxValue = MaxOff;
+ const BYTE* dictPtr = (const BYTE*)dict; /* skip magic num and dict ID */
+ const BYTE* const dictEnd = dictPtr + dictSize;
+ dictPtr += 8;
+ bs->entropy.huf.repeatMode = HUF_repeat_check;
+
+ { unsigned maxSymbolValue = 255;
+ unsigned hasZeroWeights = 1;
+ size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr,
+ dictEnd-dictPtr, &hasZeroWeights);
+
+ /* We only set the loaded table as valid if it contains all non-zero
+ * weights. Otherwise, we set it to check */
+ if (!hasZeroWeights)
+ bs->entropy.huf.repeatMode = HUF_repeat_valid;
+
+ RETURN_ERROR_IF(HUF_isError(hufHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(maxSymbolValue < 255, dictionary_corrupted, "");
+ dictPtr += hufHeaderSize;
+ }
+
+ { unsigned offcodeLog;
+ size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd-dictPtr);
+ RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
+ /* fill all offset symbols to avoid garbage at end of table */
+ RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
+ bs->entropy.fse.offcodeCTable,
+ offcodeNCount, MaxOff, offcodeLog,
+ workspace, HUF_WORKSPACE_SIZE)),
+ dictionary_corrupted, "");
+ /* Defer checking offcodeMaxValue because we need to know the size of the dictionary content */
+ dictPtr += offcodeHeaderSize;
+ }
+
+ { short matchlengthNCount[MaxML+1];
+ unsigned matchlengthMaxValue = MaxML, matchlengthLog;
+ size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd-dictPtr);
+ RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
+ RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
+ bs->entropy.fse.matchlengthCTable,
+ matchlengthNCount, matchlengthMaxValue, matchlengthLog,
+ workspace, HUF_WORKSPACE_SIZE)),
+ dictionary_corrupted, "");
+ bs->entropy.fse.matchlength_repeatMode = ZSTD_dictNCountRepeat(matchlengthNCount, matchlengthMaxValue, MaxML);
+ dictPtr += matchlengthHeaderSize;
+ }
+
+ { short litlengthNCount[MaxLL+1];
+ unsigned litlengthMaxValue = MaxLL, litlengthLog;
+ size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd-dictPtr);
+ RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
+ RETURN_ERROR_IF(FSE_isError(FSE_buildCTable_wksp(
+ bs->entropy.fse.litlengthCTable,
+ litlengthNCount, litlengthMaxValue, litlengthLog,
+ workspace, HUF_WORKSPACE_SIZE)),
+ dictionary_corrupted, "");
+ bs->entropy.fse.litlength_repeatMode = ZSTD_dictNCountRepeat(litlengthNCount, litlengthMaxValue, MaxLL);
+ dictPtr += litlengthHeaderSize;
+ }
+
+ RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
+ bs->rep[0] = MEM_readLE32(dictPtr+0);
+ bs->rep[1] = MEM_readLE32(dictPtr+4);
+ bs->rep[2] = MEM_readLE32(dictPtr+8);
+ dictPtr += 12;
+
+ { size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
+ U32 offcodeMax = MaxOff;
+ if (dictContentSize <= ((U32)-1) - 128 KB) {
+ U32 const maxOffset = (U32)dictContentSize + 128 KB; /* The maximum offset that must be supported */
+ offcodeMax = ZSTD_highbit32(maxOffset); /* Calculate minimum offset code required to represent maxOffset */
+ }
+ /* All offset values <= dictContentSize + 128 KB must be representable for a valid table */
+ bs->entropy.fse.offcode_repeatMode = ZSTD_dictNCountRepeat(offcodeNCount, offcodeMaxValue, MIN(offcodeMax, MaxOff));
+
+ /* All repCodes must be <= dictContentSize and != 0 */
+ { U32 u;
+ for (u=0; u<3; u++) {
+ RETURN_ERROR_IF(bs->rep[u] == 0, dictionary_corrupted, "");
+ RETURN_ERROR_IF(bs->rep[u] > dictContentSize, dictionary_corrupted, "");
+ } } }
+
+ return dictPtr - (const BYTE*)dict;
+}
+
+/* Dictionary format :
+ * See :
+ * https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#dictionary-format
+ */
+/*! ZSTD_loadZstdDictionary() :
+ * @return : dictID, or an error code
+ * assumptions : magic number supposed already checked
+ * dictSize supposed >= 8
+ */
+static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
+ ZSTD_matchState_t* ms,
+ ZSTD_cwksp* ws,
+ ZSTD_CCtx_params const* params,
+ const void* dict, size_t dictSize,
+ ZSTD_dictTableLoadMethod_e dtlm,
+ void* workspace)
+{
+ const BYTE* dictPtr = (const BYTE*)dict;
+ const BYTE* const dictEnd = dictPtr + dictSize;
+ size_t dictID;
+ size_t eSize;
+
+ ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
+ assert(dictSize >= 8);
+ assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);
+
+ dictID = params->fParams.noDictIDFlag ? 0 : MEM_readLE32(dictPtr + 4 /* skip magic number */ );
+ eSize = ZSTD_loadCEntropy(bs, workspace, dict, dictSize);
+ FORWARD_IF_ERROR(eSize, "ZSTD_loadCEntropy failed");
+ dictPtr += eSize;
+
+ {
+ size_t const dictContentSize = (size_t)(dictEnd - dictPtr);
+ FORWARD_IF_ERROR(ZSTD_loadDictionaryContent(
+ ms, NULL, ws, params, dictPtr, dictContentSize, dtlm), "");
+ }
+ return dictID;
+}
+
+/* ZSTD_compress_insertDictionary() :
+* @return : dictID, or an error code */
+static size_t
+ZSTD_compress_insertDictionary(ZSTD_compressedBlockState_t* bs,
+ ZSTD_matchState_t* ms,
+ ldmState_t* ls,
+ ZSTD_cwksp* ws,
+ const ZSTD_CCtx_params* params,
+ const void* dict, size_t dictSize,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_dictTableLoadMethod_e dtlm,
+ void* workspace)
+{
+ DEBUGLOG(4, "ZSTD_compress_insertDictionary (dictSize=%u)", (U32)dictSize);
+ if ((dict==NULL) || (dictSize<8)) {
+ RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
+ return 0;
+ }
+
+ ZSTD_reset_compressedBlockState(bs);
+
+ /* dict restricted modes */
+ if (dictContentType == ZSTD_dct_rawContent)
+ return ZSTD_loadDictionaryContent(ms, ls, ws, params, dict, dictSize, dtlm);
+
+ if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) {
+ if (dictContentType == ZSTD_dct_auto) {
+ DEBUGLOG(4, "raw content dictionary detected");
+ return ZSTD_loadDictionaryContent(
+ ms, ls, ws, params, dict, dictSize, dtlm);
+ }
+ RETURN_ERROR_IF(dictContentType == ZSTD_dct_fullDict, dictionary_wrong, "");
+ assert(0); /* impossible */
+ }
+
+ /* dict as full zstd dictionary */
+ return ZSTD_loadZstdDictionary(
+ bs, ms, ws, params, dict, dictSize, dtlm, workspace);
+}
+
+#define ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF (128 KB)
+#define ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER (6ULL)
+
+/*! ZSTD_compressBegin_internal() :
+ * @return : 0, or an error code */
+static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
+ const void* dict, size_t dictSize,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_dictTableLoadMethod_e dtlm,
+ const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params, U64 pledgedSrcSize,
+ ZSTD_buffered_policy_e zbuff)
+{
+ DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params->cParams.windowLog);
+ /* params are supposed to be fully validated at this point */
+ assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
+ assert(!((dict) && (cdict))); /* either dict or cdict, not both */
+ if ( (cdict)
+ && (cdict->dictContentSize > 0)
+ && ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
+ || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
+ || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
+ || cdict->compressionLevel == 0)
+ && (params->attachDictPref != ZSTD_dictForceLoad) ) {
+ return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff);
+ }
+
+ FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, *params, pledgedSrcSize,
+ ZSTDcrp_makeClean, zbuff) , "");
+ { size_t const dictID = cdict ?
+ ZSTD_compress_insertDictionary(
+ cctx->blockState.prevCBlock, &cctx->blockState.matchState,
+ &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, cdict->dictContent,
+ cdict->dictContentSize, cdict->dictContentType, dtlm,
+ cctx->entropyWorkspace)
+ : ZSTD_compress_insertDictionary(
+ cctx->blockState.prevCBlock, &cctx->blockState.matchState,
+ &cctx->ldmState, &cctx->workspace, &cctx->appliedParams, dict, dictSize,
+ dictContentType, dtlm, cctx->entropyWorkspace);
+ FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
+ assert(dictID <= UINT_MAX);
+ cctx->dictID = (U32)dictID;
+ cctx->dictContentSize = cdict ? cdict->dictContentSize : dictSize;
+ }
+ return 0;
+}
+
+size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
+ const void* dict, size_t dictSize,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_dictTableLoadMethod_e dtlm,
+ const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params,
+ unsigned long long pledgedSrcSize)
+{
+ DEBUGLOG(4, "ZSTD_compressBegin_advanced_internal: wlog=%u", params->cParams.windowLog);
+ /* compression parameters verification and optimization */
+ FORWARD_IF_ERROR( ZSTD_checkCParams(params->cParams) , "");
+ return ZSTD_compressBegin_internal(cctx,
+ dict, dictSize, dictContentType, dtlm,
+ cdict,
+ params, pledgedSrcSize,
+ ZSTDb_not_buffered);
+}
+
+/*! ZSTD_compressBegin_advanced() :
+* @return : 0, or an error code */
+size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx,
+ const void* dict, size_t dictSize,
+ ZSTD_parameters params, unsigned long long pledgedSrcSize)
+{
+ ZSTD_CCtx_params cctxParams;
+ ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, ZSTD_NO_CLEVEL);
+ return ZSTD_compressBegin_advanced_internal(cctx,
+ dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast,
+ NULL /*cdict*/,
+ &cctxParams, pledgedSrcSize);
+}
+
+size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel)
+{
+ ZSTD_CCtx_params cctxParams;
+ {
+ ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_noAttachDict);
+ ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel);
+ }
+ DEBUGLOG(4, "ZSTD_compressBegin_usingDict (dictSize=%u)", (unsigned)dictSize);
+ return ZSTD_compressBegin_internal(cctx, dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
+ &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, ZSTDb_not_buffered);
+}
+
+size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel)
+{
+ return ZSTD_compressBegin_usingDict(cctx, NULL, 0, compressionLevel);
+}
+
+
+/*! ZSTD_writeEpilogue() :
+* Ends a frame.
+* @return : nb of bytes written into dst (or an error code) */
+static size_t ZSTD_writeEpilogue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity)
+{
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* op = ostart;
+ size_t fhSize = 0;
+
+ DEBUGLOG(4, "ZSTD_writeEpilogue");
+ RETURN_ERROR_IF(cctx->stage == ZSTDcs_created, stage_wrong, "init missing");
+
+ /* special case : empty frame */
+ if (cctx->stage == ZSTDcs_init) {
+ fhSize = ZSTD_writeFrameHeader(dst, dstCapacity, &cctx->appliedParams, 0, 0);
+ FORWARD_IF_ERROR(fhSize, "ZSTD_writeFrameHeader failed");
+ dstCapacity -= fhSize;
+ op += fhSize;
+ cctx->stage = ZSTDcs_ongoing;
+ }
+
+ if (cctx->stage != ZSTDcs_ending) {
+ /* write one last empty block, make it the "last" block */
+ U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1) + 0;
+ RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for epilogue");
+ MEM_writeLE32(op, cBlockHeader24);
+ op += ZSTD_blockHeaderSize;
+ dstCapacity -= ZSTD_blockHeaderSize;
+ }
+
+ if (cctx->appliedParams.fParams.checksumFlag) {
+ U32 const checksum = (U32) xxh64_digest(&cctx->xxhState);
+ RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum");
+ DEBUGLOG(4, "ZSTD_writeEpilogue: write checksum : %08X", (unsigned)checksum);
+ MEM_writeLE32(op, checksum);
+ op += 4;
+ }
+
+ cctx->stage = ZSTDcs_created; /* return to "created but no init" status */
+ return op-ostart;
+}
+
+void ZSTD_CCtx_trace(ZSTD_CCtx* cctx, size_t extraCSize)
+{
+ (void)cctx;
+ (void)extraCSize;
+}
+
+size_t ZSTD_compressEnd (ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize)
+{
+ size_t endResult;
+ size_t const cSize = ZSTD_compressContinue_internal(cctx,
+ dst, dstCapacity, src, srcSize,
+ 1 /* frame mode */, 1 /* last chunk */);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressContinue_internal failed");
+ endResult = ZSTD_writeEpilogue(cctx, (char*)dst + cSize, dstCapacity-cSize);
+ FORWARD_IF_ERROR(endResult, "ZSTD_writeEpilogue failed");
+ assert(!(cctx->appliedParams.fParams.contentSizeFlag && cctx->pledgedSrcSizePlusOne == 0));
+ if (cctx->pledgedSrcSizePlusOne != 0) { /* control src size */
+ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_UNKNOWN == (unsigned long long)-1);
+ DEBUGLOG(4, "end of frame : controlling src size");
+ RETURN_ERROR_IF(
+ cctx->pledgedSrcSizePlusOne != cctx->consumedSrcSize+1,
+ srcSize_wrong,
+ "error : pledgedSrcSize = %u, while realSrcSize = %u",
+ (unsigned)cctx->pledgedSrcSizePlusOne-1,
+ (unsigned)cctx->consumedSrcSize);
+ }
+ ZSTD_CCtx_trace(cctx, endResult);
+ return cSize + endResult;
+}
+
+size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize,
+ ZSTD_parameters params)
+{
+ ZSTD_CCtx_params cctxParams;
+ DEBUGLOG(4, "ZSTD_compress_advanced");
+ FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), "");
+ ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, ZSTD_NO_CLEVEL);
+ return ZSTD_compress_advanced_internal(cctx,
+ dst, dstCapacity,
+ src, srcSize,
+ dict, dictSize,
+ &cctxParams);
+}
+
+/* Internal */
+size_t ZSTD_compress_advanced_internal(
+ ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize,
+ const ZSTD_CCtx_params* params)
+{
+ DEBUGLOG(4, "ZSTD_compress_advanced_internal (srcSize:%u)", (unsigned)srcSize);
+ FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
+ dict, dictSize, ZSTD_dct_auto, ZSTD_dtlm_fast, NULL,
+ params, srcSize, ZSTDb_not_buffered) , "");
+ return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
+}
+
+size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict, size_t dictSize,
+ int compressionLevel)
+{
+ ZSTD_CCtx_params cctxParams;
+ {
+ ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, srcSize, dict ? dictSize : 0, ZSTD_cpm_noAttachDict);
+ assert(params.fParams.contentSizeFlag == 1);
+ ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT: compressionLevel);
+ }
+ DEBUGLOG(4, "ZSTD_compress_usingDict (srcSize=%u)", (unsigned)srcSize);
+ return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctxParams);
+}
+
+size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ int compressionLevel)
+{
+ DEBUGLOG(4, "ZSTD_compressCCtx (srcSize=%u)", (unsigned)srcSize);
+ assert(cctx != NULL);
+ return ZSTD_compress_usingDict(cctx, dst, dstCapacity, src, srcSize, NULL, 0, compressionLevel);
+}
+
+size_t ZSTD_compress(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ int compressionLevel)
+{
+ size_t result;
+ ZSTD_CCtx* cctx = ZSTD_createCCtx();
+ RETURN_ERROR_IF(!cctx, memory_allocation, "ZSTD_createCCtx failed");
+ result = ZSTD_compressCCtx(cctx, dst, dstCapacity, src, srcSize, compressionLevel);
+ ZSTD_freeCCtx(cctx);
+ return result;
+}
+
+
+/* ===== Dictionary API ===== */
+
+/*! ZSTD_estimateCDictSize_advanced() :
+ * Estimate amount of memory that will be needed to create a dictionary with following arguments */
+size_t ZSTD_estimateCDictSize_advanced(
+ size_t dictSize, ZSTD_compressionParameters cParams,
+ ZSTD_dictLoadMethod_e dictLoadMethod)
+{
+ DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict));
+ return ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
+ + ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0)
+ + (dictLoadMethod == ZSTD_dlm_byRef ? 0
+ : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void *))));
+}
+
+size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel)
+{
+ ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
+ return ZSTD_estimateCDictSize_advanced(dictSize, cParams, ZSTD_dlm_byCopy);
+}
+
+size_t ZSTD_sizeof_CDict(const ZSTD_CDict* cdict)
+{
+ if (cdict==NULL) return 0; /* support sizeof on NULL */
+ DEBUGLOG(5, "sizeof(*cdict) : %u", (unsigned)sizeof(*cdict));
+ /* cdict may be in the workspace */
+ return (cdict->workspace.workspace == cdict ? 0 : sizeof(*cdict))
+ + ZSTD_cwksp_sizeof(&cdict->workspace);
+}
+
+static size_t ZSTD_initCDict_internal(
+ ZSTD_CDict* cdict,
+ const void* dictBuffer, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_CCtx_params params)
+{
+ DEBUGLOG(3, "ZSTD_initCDict_internal (dictContentType:%u)", (unsigned)dictContentType);
+ assert(!ZSTD_checkCParams(params.cParams));
+ cdict->matchState.cParams = params.cParams;
+ cdict->matchState.dedicatedDictSearch = params.enableDedicatedDictSearch;
+ if (cdict->matchState.dedicatedDictSearch && dictSize > ZSTD_CHUNKSIZE_MAX) {
+ cdict->matchState.dedicatedDictSearch = 0;
+ }
+ if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dictBuffer) || (!dictSize)) {
+ cdict->dictContent = dictBuffer;
+ } else {
+ void *internalBuffer = ZSTD_cwksp_reserve_object(&cdict->workspace, ZSTD_cwksp_align(dictSize, sizeof(void*)));
+ RETURN_ERROR_IF(!internalBuffer, memory_allocation, "NULL pointer!");
+ cdict->dictContent = internalBuffer;
+ ZSTD_memcpy(internalBuffer, dictBuffer, dictSize);
+ }
+ cdict->dictContentSize = dictSize;
+ cdict->dictContentType = dictContentType;
+
+ cdict->entropyWorkspace = (U32*)ZSTD_cwksp_reserve_object(&cdict->workspace, HUF_WORKSPACE_SIZE);
+
+
+ /* Reset the state to no dictionary */
+ ZSTD_reset_compressedBlockState(&cdict->cBlockState);
+ FORWARD_IF_ERROR(ZSTD_reset_matchState(
+ &cdict->matchState,
+ &cdict->workspace,
+ ¶ms.cParams,
+ ZSTDcrp_makeClean,
+ ZSTDirp_reset,
+ ZSTD_resetTarget_CDict), "");
+ /* (Maybe) load the dictionary
+ * Skips loading the dictionary if it is < 8 bytes.
+ */
+ { params.compressionLevel = ZSTD_CLEVEL_DEFAULT;
+ params.fParams.contentSizeFlag = 1;
+ { size_t const dictID = ZSTD_compress_insertDictionary(
+ &cdict->cBlockState, &cdict->matchState, NULL, &cdict->workspace,
+ ¶ms, cdict->dictContent, cdict->dictContentSize,
+ dictContentType, ZSTD_dtlm_full, cdict->entropyWorkspace);
+ FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
+ assert(dictID <= (size_t)(U32)-1);
+ cdict->dictID = (U32)dictID;
+ }
+ }
+
+ return 0;
+}
+
+static ZSTD_CDict* ZSTD_createCDict_advanced_internal(size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_compressionParameters cParams, ZSTD_customMem customMem)
+{
+ if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
+
+ { size_t const workspaceSize =
+ ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) +
+ ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) +
+ ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0) +
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
+ : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))));
+ void* const workspace = ZSTD_customMalloc(workspaceSize, customMem);
+ ZSTD_cwksp ws;
+ ZSTD_CDict* cdict;
+
+ if (!workspace) {
+ ZSTD_customFree(workspace, customMem);
+ return NULL;
+ }
+
+ ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_dynamic_alloc);
+
+ cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
+ assert(cdict != NULL);
+ ZSTD_cwksp_move(&cdict->workspace, &ws);
+ cdict->customMem = customMem;
+ cdict->compressionLevel = ZSTD_NO_CLEVEL; /* signals advanced API usage */
+
+ return cdict;
+ }
+}
+
+ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_compressionParameters cParams,
+ ZSTD_customMem customMem)
+{
+ ZSTD_CCtx_params cctxParams;
+ ZSTD_memset(&cctxParams, 0, sizeof(cctxParams));
+ ZSTD_CCtxParams_init(&cctxParams, 0);
+ cctxParams.cParams = cParams;
+ cctxParams.customMem = customMem;
+ return ZSTD_createCDict_advanced2(
+ dictBuffer, dictSize,
+ dictLoadMethod, dictContentType,
+ &cctxParams, customMem);
+}
+
+ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced2(
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ const ZSTD_CCtx_params* originalCctxParams,
+ ZSTD_customMem customMem)
+{
+ ZSTD_CCtx_params cctxParams = *originalCctxParams;
+ ZSTD_compressionParameters cParams;
+ ZSTD_CDict* cdict;
+
+ DEBUGLOG(3, "ZSTD_createCDict_advanced2, mode %u", (unsigned)dictContentType);
+ if (!customMem.customAlloc ^ !customMem.customFree) return NULL;
+
+ if (cctxParams.enableDedicatedDictSearch) {
+ cParams = ZSTD_dedicatedDictSearch_getCParams(
+ cctxParams.compressionLevel, dictSize);
+ ZSTD_overrideCParams(&cParams, &cctxParams.cParams);
+ } else {
+ cParams = ZSTD_getCParamsFromCCtxParams(
+ &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
+ }
+
+ if (!ZSTD_dedicatedDictSearch_isSupported(&cParams)) {
+ /* Fall back to non-DDSS params */
+ cctxParams.enableDedicatedDictSearch = 0;
+ cParams = ZSTD_getCParamsFromCCtxParams(
+ &cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
+ }
+
+ cctxParams.cParams = cParams;
+
+ cdict = ZSTD_createCDict_advanced_internal(dictSize,
+ dictLoadMethod, cctxParams.cParams,
+ customMem);
+
+ if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
+ dict, dictSize,
+ dictLoadMethod, dictContentType,
+ cctxParams) )) {
+ ZSTD_freeCDict(cdict);
+ return NULL;
+ }
+
+ return cdict;
+}
+
+ZSTD_CDict* ZSTD_createCDict(const void* dict, size_t dictSize, int compressionLevel)
+{
+ ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
+ ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dict, dictSize,
+ ZSTD_dlm_byCopy, ZSTD_dct_auto,
+ cParams, ZSTD_defaultCMem);
+ if (cdict)
+ cdict->compressionLevel = (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel;
+ return cdict;
+}
+
+ZSTD_CDict* ZSTD_createCDict_byReference(const void* dict, size_t dictSize, int compressionLevel)
+{
+ ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
+ ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dict, dictSize,
+ ZSTD_dlm_byRef, ZSTD_dct_auto,
+ cParams, ZSTD_defaultCMem);
+ if (cdict)
+ cdict->compressionLevel = (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT : compressionLevel;
+ return cdict;
+}
+
+size_t ZSTD_freeCDict(ZSTD_CDict* cdict)
+{
+ if (cdict==NULL) return 0; /* support free on NULL */
+ { ZSTD_customMem const cMem = cdict->customMem;
+ int cdictInWorkspace = ZSTD_cwksp_owns_buffer(&cdict->workspace, cdict);
+ ZSTD_cwksp_free(&cdict->workspace, cMem);
+ if (!cdictInWorkspace) {
+ ZSTD_customFree(cdict, cMem);
+ }
+ return 0;
+ }
+}
+
+/*! ZSTD_initStaticCDict_advanced() :
+ * Generate a digested dictionary in provided memory area.
+ * workspace: The memory area to emplace the dictionary into.
+ * Provided pointer must 8-bytes aligned.
+ * It must outlive dictionary usage.
+ * workspaceSize: Use ZSTD_estimateCDictSize()
+ * to determine how large workspace must be.
+ * cParams : use ZSTD_getCParams() to transform a compression level
+ * into its relevants cParams.
+ * @return : pointer to ZSTD_CDict*, or NULL if error (size too small)
+ * Note : there is no corresponding "free" function.
+ * Since workspace was allocated externally, it must be freed externally.
+ */
+const ZSTD_CDict* ZSTD_initStaticCDict(
+ void* workspace, size_t workspaceSize,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_compressionParameters cParams)
+{
+ size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0);
+ size_t const neededSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ + (dictLoadMethod == ZSTD_dlm_byRef ? 0
+ : ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))))
+ + ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
+ + matchStateSize;
+ ZSTD_CDict* cdict;
+ ZSTD_CCtx_params params;
+
+ if ((size_t)workspace & 7) return NULL; /* 8-aligned */
+
+ {
+ ZSTD_cwksp ws;
+ ZSTD_cwksp_init(&ws, workspace, workspaceSize, ZSTD_cwksp_static_alloc);
+ cdict = (ZSTD_CDict*)ZSTD_cwksp_reserve_object(&ws, sizeof(ZSTD_CDict));
+ if (cdict == NULL) return NULL;
+ ZSTD_cwksp_move(&cdict->workspace, &ws);
+ }
+
+ DEBUGLOG(4, "(workspaceSize < neededSize) : (%u < %u) => %u",
+ (unsigned)workspaceSize, (unsigned)neededSize, (unsigned)(workspaceSize < neededSize));
+ if (workspaceSize < neededSize) return NULL;
+
+ ZSTD_CCtxParams_init(¶ms, 0);
+ params.cParams = cParams;
+
+ if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
+ dict, dictSize,
+ dictLoadMethod, dictContentType,
+ params) ))
+ return NULL;
+
+ return cdict;
+}
+
+ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict)
+{
+ assert(cdict != NULL);
+ return cdict->matchState.cParams;
+}
+
+/*! ZSTD_getDictID_fromCDict() :
+ * Provides the dictID of the dictionary loaded into `cdict`.
+ * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
+ * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
+unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict)
+{
+ if (cdict==NULL) return 0;
+ return cdict->dictID;
+}
+
+
+/* ZSTD_compressBegin_usingCDict_advanced() :
+ * cdict must be != NULL */
+size_t ZSTD_compressBegin_usingCDict_advanced(
+ ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict,
+ ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
+{
+ ZSTD_CCtx_params cctxParams;
+ DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced");
+ RETURN_ERROR_IF(cdict==NULL, dictionary_wrong, "NULL pointer!");
+ /* Initialize the cctxParams from the cdict */
+ {
+ ZSTD_parameters params;
+ params.fParams = fParams;
+ params.cParams = ( pledgedSrcSize < ZSTD_USE_CDICT_PARAMS_SRCSIZE_CUTOFF
+ || pledgedSrcSize < cdict->dictContentSize * ZSTD_USE_CDICT_PARAMS_DICTSIZE_MULTIPLIER
+ || pledgedSrcSize == ZSTD_CONTENTSIZE_UNKNOWN
+ || cdict->compressionLevel == 0 ) ?
+ ZSTD_getCParamsFromCDict(cdict)
+ : ZSTD_getCParams(cdict->compressionLevel,
+ pledgedSrcSize,
+ cdict->dictContentSize);
+ ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, cdict->compressionLevel);
+ }
+ /* Increase window log to fit the entire dictionary and source if the
+ * source size is known. Limit the increase to 19, which is the
+ * window log for compression level 1 with the largest source size.
+ */
+ if (pledgedSrcSize != ZSTD_CONTENTSIZE_UNKNOWN) {
+ U32 const limitedSrcSize = (U32)MIN(pledgedSrcSize, 1U << 19);
+ U32 const limitedSrcLog = limitedSrcSize > 1 ? ZSTD_highbit32(limitedSrcSize - 1) + 1 : 1;
+ cctxParams.cParams.windowLog = MAX(cctxParams.cParams.windowLog, limitedSrcLog);
+ }
+ return ZSTD_compressBegin_internal(cctx,
+ NULL, 0, ZSTD_dct_auto, ZSTD_dtlm_fast,
+ cdict,
+ &cctxParams, pledgedSrcSize,
+ ZSTDb_not_buffered);
+}
+
+/* ZSTD_compressBegin_usingCDict() :
+ * pledgedSrcSize=0 means "unknown"
+ * if pledgedSrcSize>0, it will enable contentSizeFlag */
+size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
+{
+ ZSTD_frameParameters const fParams = { 0 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ };
+ DEBUGLOG(4, "ZSTD_compressBegin_usingCDict : dictIDFlag == %u", !fParams.noDictIDFlag);
+ return ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN);
+}
+
+size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_CDict* cdict, ZSTD_frameParameters fParams)
+{
+ FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, srcSize), ""); /* will check if cdict != NULL */
+ return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
+}
+
+/*! ZSTD_compress_usingCDict() :
+ * Compression using a digested Dictionary.
+ * Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times.
+ * Note that compression parameters are decided at CDict creation time
+ * while frame parameters are hardcoded */
+size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_CDict* cdict)
+{
+ ZSTD_frameParameters const fParams = { 1 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ };
+ return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, fParams);
+}
+
+
+
+/* ******************************************************************
+* Streaming
+********************************************************************/
+
+ZSTD_CStream* ZSTD_createCStream(void)
+{
+ DEBUGLOG(3, "ZSTD_createCStream");
+ return ZSTD_createCStream_advanced(ZSTD_defaultCMem);
+}
+
+ZSTD_CStream* ZSTD_initStaticCStream(void *workspace, size_t workspaceSize)
+{
+ return ZSTD_initStaticCCtx(workspace, workspaceSize);
+}
+
+ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem)
+{ /* CStream and CCtx are now same object */
+ return ZSTD_createCCtx_advanced(customMem);
+}
+
+size_t ZSTD_freeCStream(ZSTD_CStream* zcs)
+{
+ return ZSTD_freeCCtx(zcs); /* same object */
+}
+
+
+
+/*====== Initialization ======*/
+
+size_t ZSTD_CStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX; }
+
+size_t ZSTD_CStreamOutSize(void)
+{
+ return ZSTD_compressBound(ZSTD_BLOCKSIZE_MAX) + ZSTD_blockHeaderSize + 4 /* 32-bits hash */ ;
+}
+
+static ZSTD_cParamMode_e ZSTD_getCParamMode(ZSTD_CDict const* cdict, ZSTD_CCtx_params const* params, U64 pledgedSrcSize)
+{
+ if (cdict != NULL && ZSTD_shouldAttachDict(cdict, params, pledgedSrcSize))
+ return ZSTD_cpm_attachDict;
+ else
+ return ZSTD_cpm_noAttachDict;
+}
+
+/* ZSTD_resetCStream():
+ * pledgedSrcSize == 0 means "unknown" */
+size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pss)
+{
+ /* temporary : 0 interpreted as "unknown" during transition period.
+ * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN.
+ * 0 will be interpreted as "empty" in the future.
+ */
+ U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
+ DEBUGLOG(4, "ZSTD_resetCStream: pledgedSrcSize = %u", (unsigned)pledgedSrcSize);
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+ return 0;
+}
+
+/*! ZSTD_initCStream_internal() :
+ * Note : for lib/compress only. Used by zstdmt_compress.c.
+ * Assumption 1 : params are valid
+ * Assumption 2 : either dict, or cdict, is defined, not both */
+size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
+ const void* dict, size_t dictSize, const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params,
+ unsigned long long pledgedSrcSize)
+{
+ DEBUGLOG(4, "ZSTD_initCStream_internal");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+ assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
+ zcs->requestedParams = *params;
+ assert(!((dict) && (cdict))); /* either dict or cdict, not both */
+ if (dict) {
+ FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
+ } else {
+ /* Dictionary is cleared if !cdict */
+ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
+ }
+ return 0;
+}
+
+/* ZSTD_initCStream_usingCDict_advanced() :
+ * same as ZSTD_initCStream_usingCDict(), with control over frame parameters */
+size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
+ const ZSTD_CDict* cdict,
+ ZSTD_frameParameters fParams,
+ unsigned long long pledgedSrcSize)
+{
+ DEBUGLOG(4, "ZSTD_initCStream_usingCDict_advanced");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+ zcs->requestedParams.fParams = fParams;
+ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
+ return 0;
+}
+
+/* note : cdict must outlive compression session */
+size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict)
+{
+ DEBUGLOG(4, "ZSTD_initCStream_usingCDict");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, cdict) , "");
+ return 0;
+}
+
+
+/* ZSTD_initCStream_advanced() :
+ * pledgedSrcSize must be exact.
+ * if srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
+ * dict is loaded with default parameters ZSTD_dct_auto and ZSTD_dlm_byCopy. */
+size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
+ const void* dict, size_t dictSize,
+ ZSTD_parameters params, unsigned long long pss)
+{
+ /* for compatibility with older programs relying on this behavior.
+ * Users should now specify ZSTD_CONTENTSIZE_UNKNOWN.
+ * This line will be removed in the future.
+ */
+ U64 const pledgedSrcSize = (pss==0 && params.fParams.contentSizeFlag==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
+ DEBUGLOG(4, "ZSTD_initCStream_advanced");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+ FORWARD_IF_ERROR( ZSTD_checkCParams(params.cParams) , "");
+ ZSTD_CCtxParams_setZstdParams(&zcs->requestedParams, ¶ms);
+ FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
+ return 0;
+}
+
+size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs, const void* dict, size_t dictSize, int compressionLevel)
+{
+ DEBUGLOG(4, "ZSTD_initCStream_usingDict");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_loadDictionary(zcs, dict, dictSize) , "");
+ return 0;
+}
+
+size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs, int compressionLevel, unsigned long long pss)
+{
+ /* temporary : 0 interpreted as "unknown" during transition period.
+ * Users willing to specify "unknown" **must** use ZSTD_CONTENTSIZE_UNKNOWN.
+ * 0 will be interpreted as "empty" in the future.
+ */
+ U64 const pledgedSrcSize = (pss==0) ? ZSTD_CONTENTSIZE_UNKNOWN : pss;
+ DEBUGLOG(4, "ZSTD_initCStream_srcSize");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize) , "");
+ return 0;
+}
+
+size_t ZSTD_initCStream(ZSTD_CStream* zcs, int compressionLevel)
+{
+ DEBUGLOG(4, "ZSTD_initCStream");
+ FORWARD_IF_ERROR( ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_refCDict(zcs, NULL) , "");
+ FORWARD_IF_ERROR( ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel) , "");
+ return 0;
+}
+
+/*====== Compression ======*/
+
+static size_t ZSTD_nextInputSizeHint(const ZSTD_CCtx* cctx)
+{
+ size_t hintInSize = cctx->inBuffTarget - cctx->inBuffPos;
+ if (hintInSize==0) hintInSize = cctx->blockSize;
+ return hintInSize;
+}
+
+/* ZSTD_compressStream_generic():
+ * internal function for all *compressStream*() variants
+ * non-static, because can be called from zstdmt_compress.c
+ * @return : hint size for next input */
+static size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs,
+ ZSTD_outBuffer* output,
+ ZSTD_inBuffer* input,
+ ZSTD_EndDirective const flushMode)
+{
+ const char* const istart = (const char*)input->src;
+ const char* const iend = input->size != 0 ? istart + input->size : istart;
+ const char* ip = input->pos != 0 ? istart + input->pos : istart;
+ char* const ostart = (char*)output->dst;
+ char* const oend = output->size != 0 ? ostart + output->size : ostart;
+ char* op = output->pos != 0 ? ostart + output->pos : ostart;
+ U32 someMoreWork = 1;
+
+ /* check expectations */
+ DEBUGLOG(5, "ZSTD_compressStream_generic, flush=%u", (unsigned)flushMode);
+ if (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered) {
+ assert(zcs->inBuff != NULL);
+ assert(zcs->inBuffSize > 0);
+ }
+ if (zcs->appliedParams.outBufferMode == ZSTD_bm_buffered) {
+ assert(zcs->outBuff != NULL);
+ assert(zcs->outBuffSize > 0);
+ }
+ assert(output->pos <= output->size);
+ assert(input->pos <= input->size);
+ assert((U32)flushMode <= (U32)ZSTD_e_end);
+
+ while (someMoreWork) {
+ switch(zcs->streamStage)
+ {
+ case zcss_init:
+ RETURN_ERROR(init_missing, "call ZSTD_initCStream() first!");
+
+ case zcss_load:
+ if ( (flushMode == ZSTD_e_end)
+ && ( (size_t)(oend-op) >= ZSTD_compressBound(iend-ip) /* Enough output space */
+ || zcs->appliedParams.outBufferMode == ZSTD_bm_stable) /* OR we are allowed to return dstSizeTooSmall */
+ && (zcs->inBuffPos == 0) ) {
+ /* shortcut to compression pass directly into output buffer */
+ size_t const cSize = ZSTD_compressEnd(zcs,
+ op, oend-op, ip, iend-ip);
+ DEBUGLOG(4, "ZSTD_compressEnd : cSize=%u", (unsigned)cSize);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressEnd failed");
+ ip = iend;
+ op += cSize;
+ zcs->frameEnded = 1;
+ ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ someMoreWork = 0; break;
+ }
+ /* complete loading into inBuffer in buffered mode */
+ if (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered) {
+ size_t const toLoad = zcs->inBuffTarget - zcs->inBuffPos;
+ size_t const loaded = ZSTD_limitCopy(
+ zcs->inBuff + zcs->inBuffPos, toLoad,
+ ip, iend-ip);
+ zcs->inBuffPos += loaded;
+ if (loaded != 0)
+ ip += loaded;
+ if ( (flushMode == ZSTD_e_continue)
+ && (zcs->inBuffPos < zcs->inBuffTarget) ) {
+ /* not enough input to fill full block : stop here */
+ someMoreWork = 0; break;
+ }
+ if ( (flushMode == ZSTD_e_flush)
+ && (zcs->inBuffPos == zcs->inToCompress) ) {
+ /* empty */
+ someMoreWork = 0; break;
+ }
+ }
+ /* compress current block (note : this stage cannot be stopped in the middle) */
+ DEBUGLOG(5, "stream compression stage (flushMode==%u)", flushMode);
+ { int const inputBuffered = (zcs->appliedParams.inBufferMode == ZSTD_bm_buffered);
+ void* cDst;
+ size_t cSize;
+ size_t oSize = oend-op;
+ size_t const iSize = inputBuffered
+ ? zcs->inBuffPos - zcs->inToCompress
+ : MIN((size_t)(iend - ip), zcs->blockSize);
+ if (oSize >= ZSTD_compressBound(iSize) || zcs->appliedParams.outBufferMode == ZSTD_bm_stable)
+ cDst = op; /* compress into output buffer, to skip flush stage */
+ else
+ cDst = zcs->outBuff, oSize = zcs->outBuffSize;
+ if (inputBuffered) {
+ unsigned const lastBlock = (flushMode == ZSTD_e_end) && (ip==iend);
+ cSize = lastBlock ?
+ ZSTD_compressEnd(zcs, cDst, oSize,
+ zcs->inBuff + zcs->inToCompress, iSize) :
+ ZSTD_compressContinue(zcs, cDst, oSize,
+ zcs->inBuff + zcs->inToCompress, iSize);
+ FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed");
+ zcs->frameEnded = lastBlock;
+ /* prepare next block */
+ zcs->inBuffTarget = zcs->inBuffPos + zcs->blockSize;
+ if (zcs->inBuffTarget > zcs->inBuffSize)
+ zcs->inBuffPos = 0, zcs->inBuffTarget = zcs->blockSize;
+ DEBUGLOG(5, "inBuffTarget:%u / inBuffSize:%u",
+ (unsigned)zcs->inBuffTarget, (unsigned)zcs->inBuffSize);
+ if (!lastBlock)
+ assert(zcs->inBuffTarget <= zcs->inBuffSize);
+ zcs->inToCompress = zcs->inBuffPos;
+ } else {
+ unsigned const lastBlock = (ip + iSize == iend);
+ assert(flushMode == ZSTD_e_end /* Already validated */);
+ cSize = lastBlock ?
+ ZSTD_compressEnd(zcs, cDst, oSize, ip, iSize) :
+ ZSTD_compressContinue(zcs, cDst, oSize, ip, iSize);
+ /* Consume the input prior to error checking to mirror buffered mode. */
+ if (iSize > 0)
+ ip += iSize;
+ FORWARD_IF_ERROR(cSize, "%s", lastBlock ? "ZSTD_compressEnd failed" : "ZSTD_compressContinue failed");
+ zcs->frameEnded = lastBlock;
+ if (lastBlock)
+ assert(ip == iend);
+ }
+ if (cDst == op) { /* no need to flush */
+ op += cSize;
+ if (zcs->frameEnded) {
+ DEBUGLOG(5, "Frame completed directly in outBuffer");
+ someMoreWork = 0;
+ ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ }
+ break;
+ }
+ zcs->outBuffContentSize = cSize;
+ zcs->outBuffFlushedSize = 0;
+ zcs->streamStage = zcss_flush; /* pass-through to flush stage */
+ }
+ ZSTD_FALLTHROUGH;
+ case zcss_flush:
+ DEBUGLOG(5, "flush stage");
+ assert(zcs->appliedParams.outBufferMode == ZSTD_bm_buffered);
+ { size_t const toFlush = zcs->outBuffContentSize - zcs->outBuffFlushedSize;
+ size_t const flushed = ZSTD_limitCopy(op, (size_t)(oend-op),
+ zcs->outBuff + zcs->outBuffFlushedSize, toFlush);
+ DEBUGLOG(5, "toFlush: %u into %u ==> flushed: %u",
+ (unsigned)toFlush, (unsigned)(oend-op), (unsigned)flushed);
+ if (flushed)
+ op += flushed;
+ zcs->outBuffFlushedSize += flushed;
+ if (toFlush!=flushed) {
+ /* flush not fully completed, presumably because dst is too small */
+ assert(op==oend);
+ someMoreWork = 0;
+ break;
+ }
+ zcs->outBuffContentSize = zcs->outBuffFlushedSize = 0;
+ if (zcs->frameEnded) {
+ DEBUGLOG(5, "Frame completed on flush");
+ someMoreWork = 0;
+ ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
+ break;
+ }
+ zcs->streamStage = zcss_load;
+ break;
+ }
+
+ default: /* impossible */
+ assert(0);
+ }
+ }
+
+ input->pos = ip - istart;
+ output->pos = op - ostart;
+ if (zcs->frameEnded) return 0;
+ return ZSTD_nextInputSizeHint(zcs);
+}
+
+static size_t ZSTD_nextInputSizeHint_MTorST(const ZSTD_CCtx* cctx)
+{
+ return ZSTD_nextInputSizeHint(cctx);
+
+}
+
+size_t ZSTD_compressStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
+{
+ FORWARD_IF_ERROR( ZSTD_compressStream2(zcs, output, input, ZSTD_e_continue) , "");
+ return ZSTD_nextInputSizeHint_MTorST(zcs);
+}
+
+/* After a compression call set the expected input/output buffer.
+ * This is validated at the start of the next compression call.
+ */
+static void ZSTD_setBufferExpectations(ZSTD_CCtx* cctx, ZSTD_outBuffer const* output, ZSTD_inBuffer const* input)
+{
+ if (cctx->appliedParams.inBufferMode == ZSTD_bm_stable) {
+ cctx->expectedInBuffer = *input;
+ }
+ if (cctx->appliedParams.outBufferMode == ZSTD_bm_stable) {
+ cctx->expectedOutBufferSize = output->size - output->pos;
+ }
+}
+
+/* Validate that the input/output buffers match the expectations set by
+ * ZSTD_setBufferExpectations.
+ */
+static size_t ZSTD_checkBufferStability(ZSTD_CCtx const* cctx,
+ ZSTD_outBuffer const* output,
+ ZSTD_inBuffer const* input,
+ ZSTD_EndDirective endOp)
+{
+ if (cctx->appliedParams.inBufferMode == ZSTD_bm_stable) {
+ ZSTD_inBuffer const expect = cctx->expectedInBuffer;
+ if (expect.src != input->src || expect.pos != input->pos || expect.size != input->size)
+ RETURN_ERROR(srcBuffer_wrong, "ZSTD_c_stableInBuffer enabled but input differs!");
+ if (endOp != ZSTD_e_end)
+ RETURN_ERROR(srcBuffer_wrong, "ZSTD_c_stableInBuffer can only be used with ZSTD_e_end!");
+ }
+ if (cctx->appliedParams.outBufferMode == ZSTD_bm_stable) {
+ size_t const outBufferSize = output->size - output->pos;
+ if (cctx->expectedOutBufferSize != outBufferSize)
+ RETURN_ERROR(dstBuffer_wrong, "ZSTD_c_stableOutBuffer enabled but output size differs!");
+ }
+ return 0;
+}
+
+static size_t ZSTD_CCtx_init_compressStream2(ZSTD_CCtx* cctx,
+ ZSTD_EndDirective endOp,
+ size_t inSize) {
+ ZSTD_CCtx_params params = cctx->requestedParams;
+ ZSTD_prefixDict const prefixDict = cctx->prefixDict;
+ FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) , ""); /* Init the local dict if present. */
+ ZSTD_memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */
+ assert(prefixDict.dict==NULL || cctx->cdict==NULL); /* only one can be set */
+ if (cctx->cdict)
+ params.compressionLevel = cctx->cdict->compressionLevel; /* let cdict take priority in terms of compression level */
+ DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage");
+ if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = inSize + 1; /* auto-fix pledgedSrcSize */
+ {
+ size_t const dictSize = prefixDict.dict
+ ? prefixDict.dictSize
+ : (cctx->cdict ? cctx->cdict->dictContentSize : 0);
+ ZSTD_cParamMode_e const mode = ZSTD_getCParamMode(cctx->cdict, ¶ms, cctx->pledgedSrcSizePlusOne - 1);
+ params.cParams = ZSTD_getCParamsFromCCtxParams(
+ ¶ms, cctx->pledgedSrcSizePlusOne-1,
+ dictSize, mode);
+ }
+
+ if (ZSTD_CParams_shouldEnableLdm(¶ms.cParams)) {
+ /* Enable LDM by default for optimal parser and window size >= 128MB */
+ DEBUGLOG(4, "LDM enabled by default (window size >= 128MB, strategy >= btopt)");
+ params.ldmParams.enableLdm = 1;
+ }
+
+ { U64 const pledgedSrcSize = cctx->pledgedSrcSizePlusOne - 1;
+ assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
+ FORWARD_IF_ERROR( ZSTD_compressBegin_internal(cctx,
+ prefixDict.dict, prefixDict.dictSize, prefixDict.dictContentType, ZSTD_dtlm_fast,
+ cctx->cdict,
+ ¶ms, pledgedSrcSize,
+ ZSTDb_buffered) , "");
+ assert(cctx->appliedParams.nbWorkers == 0);
+ cctx->inToCompress = 0;
+ cctx->inBuffPos = 0;
+ if (cctx->appliedParams.inBufferMode == ZSTD_bm_buffered) {
+ /* for small input: avoid automatic flush on reaching end of block, since
+ * it would require to add a 3-bytes null block to end frame
+ */
+ cctx->inBuffTarget = cctx->blockSize + (cctx->blockSize == pledgedSrcSize);
+ } else {
+ cctx->inBuffTarget = 0;
+ }
+ cctx->outBuffContentSize = cctx->outBuffFlushedSize = 0;
+ cctx->streamStage = zcss_load;
+ cctx->frameEnded = 0;
+ }
+ return 0;
+}
+
+size_t ZSTD_compressStream2( ZSTD_CCtx* cctx,
+ ZSTD_outBuffer* output,
+ ZSTD_inBuffer* input,
+ ZSTD_EndDirective endOp)
+{
+ DEBUGLOG(5, "ZSTD_compressStream2, endOp=%u ", (unsigned)endOp);
+ /* check conditions */
+ RETURN_ERROR_IF(output->pos > output->size, dstSize_tooSmall, "invalid output buffer");
+ RETURN_ERROR_IF(input->pos > input->size, srcSize_wrong, "invalid input buffer");
+ RETURN_ERROR_IF((U32)endOp > (U32)ZSTD_e_end, parameter_outOfBound, "invalid endDirective");
+ assert(cctx != NULL);
+
+ /* transparent initialization stage */
+ if (cctx->streamStage == zcss_init) {
+ FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, endOp, input->size), "CompressStream2 initialization failed");
+ ZSTD_setBufferExpectations(cctx, output, input); /* Set initial buffer expectations now that we've initialized */
+ }
+ /* end of transparent initialization stage */
+
+ FORWARD_IF_ERROR(ZSTD_checkBufferStability(cctx, output, input, endOp), "invalid buffers");
+ /* compression stage */
+ FORWARD_IF_ERROR( ZSTD_compressStream_generic(cctx, output, input, endOp) , "");
+ DEBUGLOG(5, "completed ZSTD_compressStream2");
+ ZSTD_setBufferExpectations(cctx, output, input);
+ return cctx->outBuffContentSize - cctx->outBuffFlushedSize; /* remaining to flush */
+}
+
+size_t ZSTD_compressStream2_simpleArgs (
+ ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity, size_t* dstPos,
+ const void* src, size_t srcSize, size_t* srcPos,
+ ZSTD_EndDirective endOp)
+{
+ ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
+ ZSTD_inBuffer input = { src, srcSize, *srcPos };
+ /* ZSTD_compressStream2() will check validity of dstPos and srcPos */
+ size_t const cErr = ZSTD_compressStream2(cctx, &output, &input, endOp);
+ *dstPos = output.pos;
+ *srcPos = input.pos;
+ return cErr;
+}
+
+size_t ZSTD_compress2(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize)
+{
+ ZSTD_bufferMode_e const originalInBufferMode = cctx->requestedParams.inBufferMode;
+ ZSTD_bufferMode_e const originalOutBufferMode = cctx->requestedParams.outBufferMode;
+ DEBUGLOG(4, "ZSTD_compress2 (srcSize=%u)", (unsigned)srcSize);
+ ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
+ /* Enable stable input/output buffers. */
+ cctx->requestedParams.inBufferMode = ZSTD_bm_stable;
+ cctx->requestedParams.outBufferMode = ZSTD_bm_stable;
+ { size_t oPos = 0;
+ size_t iPos = 0;
+ size_t const result = ZSTD_compressStream2_simpleArgs(cctx,
+ dst, dstCapacity, &oPos,
+ src, srcSize, &iPos,
+ ZSTD_e_end);
+ /* Reset to the original values. */
+ cctx->requestedParams.inBufferMode = originalInBufferMode;
+ cctx->requestedParams.outBufferMode = originalOutBufferMode;
+ FORWARD_IF_ERROR(result, "ZSTD_compressStream2_simpleArgs failed");
+ if (result != 0) { /* compression not completed, due to lack of output space */
+ assert(oPos == dstCapacity);
+ RETURN_ERROR(dstSize_tooSmall, "");
+ }
+ assert(iPos == srcSize); /* all input is expected consumed */
+ return oPos;
+ }
+}
+
+typedef struct {
+ U32 idx; /* Index in array of ZSTD_Sequence */
+ U32 posInSequence; /* Position within sequence at idx */
+ size_t posInSrc; /* Number of bytes given by sequences provided so far */
+} ZSTD_sequencePosition;
+
+/* Returns a ZSTD error code if sequence is not valid */
+static size_t ZSTD_validateSequence(U32 offCode, U32 matchLength,
+ size_t posInSrc, U32 windowLog, size_t dictSize, U32 minMatch) {
+ size_t offsetBound;
+ U32 windowSize = 1 << windowLog;
+ /* posInSrc represents the amount of data the the decoder would decode up to this point.
+ * As long as the amount of data decoded is less than or equal to window size, offsets may be
+ * larger than the total length of output decoded in order to reference the dict, even larger than
+ * window size. After output surpasses windowSize, we're limited to windowSize offsets again.
+ */
+ offsetBound = posInSrc > windowSize ? (size_t)windowSize : posInSrc + (size_t)dictSize;
+ RETURN_ERROR_IF(offCode > offsetBound + ZSTD_REP_MOVE, corruption_detected, "Offset too large!");
+ RETURN_ERROR_IF(matchLength < minMatch, corruption_detected, "Matchlength too small");
+ return 0;
+}
+
+/* Returns an offset code, given a sequence's raw offset, the ongoing repcode array, and whether litLength == 0 */
+static U32 ZSTD_finalizeOffCode(U32 rawOffset, const U32 rep[ZSTD_REP_NUM], U32 ll0) {
+ U32 offCode = rawOffset + ZSTD_REP_MOVE;
+ U32 repCode = 0;
+
+ if (!ll0 && rawOffset == rep[0]) {
+ repCode = 1;
+ } else if (rawOffset == rep[1]) {
+ repCode = 2 - ll0;
+ } else if (rawOffset == rep[2]) {
+ repCode = 3 - ll0;
+ } else if (ll0 && rawOffset == rep[0] - 1) {
+ repCode = 3;
+ }
+ if (repCode) {
+ /* ZSTD_storeSeq expects a number in the range [0, 2] to represent a repcode */
+ offCode = repCode - 1;
+ }
+ return offCode;
+}
+
+/* Returns 0 on success, and a ZSTD_error otherwise. This function scans through an array of
+ * ZSTD_Sequence, storing the sequences it finds, until it reaches a block delimiter.
+ */
+static size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
+ const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
+ const void* src, size_t blockSize) {
+ U32 idx = seqPos->idx;
+ BYTE const* ip = (BYTE const*)(src);
+ const BYTE* const iend = ip + blockSize;
+ repcodes_t updatedRepcodes;
+ U32 dictSize;
+ U32 litLength;
+ U32 matchLength;
+ U32 ll0;
+ U32 offCode;
+
+ if (cctx->cdict) {
+ dictSize = (U32)cctx->cdict->dictContentSize;
+ } else if (cctx->prefixDict.dict) {
+ dictSize = (U32)cctx->prefixDict.dictSize;
+ } else {
+ dictSize = 0;
+ }
+ ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t));
+ for (; (inSeqs[idx].matchLength != 0 || inSeqs[idx].offset != 0) && idx < inSeqsSize; ++idx) {
+ litLength = inSeqs[idx].litLength;
+ matchLength = inSeqs[idx].matchLength;
+ ll0 = litLength == 0;
+ offCode = ZSTD_finalizeOffCode(inSeqs[idx].offset, updatedRepcodes.rep, ll0);
+ updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
+
+ DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offCode, matchLength, litLength);
+ if (cctx->appliedParams.validateSequences) {
+ seqPos->posInSrc += litLength + matchLength;
+ FORWARD_IF_ERROR(ZSTD_validateSequence(offCode, matchLength, seqPos->posInSrc,
+ cctx->appliedParams.cParams.windowLog, dictSize,
+ cctx->appliedParams.cParams.minMatch),
+ "Sequence validation failed");
+ }
+ RETURN_ERROR_IF(idx - seqPos->idx > cctx->seqStore.maxNbSeq, memory_allocation,
+ "Not enough memory allocated. Try adjusting ZSTD_c_minMatch.");
+ ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength - MINMATCH);
+ ip += matchLength + litLength;
+ }
+ ZSTD_memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, sizeof(repcodes_t));
+
+ if (inSeqs[idx].litLength) {
+ DEBUGLOG(6, "Storing last literals of size: %u", inSeqs[idx].litLength);
+ ZSTD_storeLastLiterals(&cctx->seqStore, ip, inSeqs[idx].litLength);
+ ip += inSeqs[idx].litLength;
+ seqPos->posInSrc += inSeqs[idx].litLength;
+ }
+ RETURN_ERROR_IF(ip != iend, corruption_detected, "Blocksize doesn't agree with block delimiter!");
+ seqPos->idx = idx+1;
+ return 0;
+}
+
+/* Returns the number of bytes to move the current read position back by. Only non-zero
+ * if we ended up splitting a sequence. Otherwise, it may return a ZSTD error if something
+ * went wrong.
+ *
+ * This function will attempt to scan through blockSize bytes represented by the sequences
+ * in inSeqs, storing any (partial) sequences.
+ *
+ * Occasionally, we may want to change the actual number of bytes we consumed from inSeqs to
+ * avoid splitting a match, or to avoid splitting a match such that it would produce a match
+ * smaller than MINMATCH. In this case, we return the number of bytes that we didn't read from this block.
+ */
+static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
+ const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
+ const void* src, size_t blockSize) {
+ U32 idx = seqPos->idx;
+ U32 startPosInSequence = seqPos->posInSequence;
+ U32 endPosInSequence = seqPos->posInSequence + (U32)blockSize;
+ size_t dictSize;
+ BYTE const* ip = (BYTE const*)(src);
+ BYTE const* iend = ip + blockSize; /* May be adjusted if we decide to process fewer than blockSize bytes */
+ repcodes_t updatedRepcodes;
+ U32 bytesAdjustment = 0;
+ U32 finalMatchSplit = 0;
+ U32 litLength;
+ U32 matchLength;
+ U32 rawOffset;
+ U32 offCode;
+
+ if (cctx->cdict) {
+ dictSize = cctx->cdict->dictContentSize;
+ } else if (cctx->prefixDict.dict) {
+ dictSize = cctx->prefixDict.dictSize;
+ } else {
+ dictSize = 0;
+ }
+ DEBUGLOG(5, "ZSTD_copySequencesToSeqStore: idx: %u PIS: %u blockSize: %zu", idx, startPosInSequence, blockSize);
+ DEBUGLOG(5, "Start seq: idx: %u (of: %u ml: %u ll: %u)", idx, inSeqs[idx].offset, inSeqs[idx].matchLength, inSeqs[idx].litLength);
+ ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t));
+ while (endPosInSequence && idx < inSeqsSize && !finalMatchSplit) {
+ const ZSTD_Sequence currSeq = inSeqs[idx];
+ litLength = currSeq.litLength;
+ matchLength = currSeq.matchLength;
+ rawOffset = currSeq.offset;
+
+ /* Modify the sequence depending on where endPosInSequence lies */
+ if (endPosInSequence >= currSeq.litLength + currSeq.matchLength) {
+ if (startPosInSequence >= litLength) {
+ startPosInSequence -= litLength;
+ litLength = 0;
+ matchLength -= startPosInSequence;
+ } else {
+ litLength -= startPosInSequence;
+ }
+ /* Move to the next sequence */
+ endPosInSequence -= currSeq.litLength + currSeq.matchLength;
+ startPosInSequence = 0;
+ idx++;
+ } else {
+ /* This is the final (partial) sequence we're adding from inSeqs, and endPosInSequence
+ does not reach the end of the match. So, we have to split the sequence */
+ DEBUGLOG(6, "Require a split: diff: %u, idx: %u PIS: %u",
+ currSeq.litLength + currSeq.matchLength - endPosInSequence, idx, endPosInSequence);
+ if (endPosInSequence > litLength) {
+ U32 firstHalfMatchLength;
+ litLength = startPosInSequence >= litLength ? 0 : litLength - startPosInSequence;
+ firstHalfMatchLength = endPosInSequence - startPosInSequence - litLength;
+ if (matchLength > blockSize && firstHalfMatchLength >= cctx->appliedParams.cParams.minMatch) {
+ /* Only ever split the match if it is larger than the block size */
+ U32 secondHalfMatchLength = currSeq.matchLength + currSeq.litLength - endPosInSequence;
+ if (secondHalfMatchLength < cctx->appliedParams.cParams.minMatch) {
+ /* Move the endPosInSequence backward so that it creates match of minMatch length */
+ endPosInSequence -= cctx->appliedParams.cParams.minMatch - secondHalfMatchLength;
+ bytesAdjustment = cctx->appliedParams.cParams.minMatch - secondHalfMatchLength;
+ firstHalfMatchLength -= bytesAdjustment;
+ }
+ matchLength = firstHalfMatchLength;
+ /* Flag that we split the last match - after storing the sequence, exit the loop,
+ but keep the value of endPosInSequence */
+ finalMatchSplit = 1;
+ } else {
+ /* Move the position in sequence backwards so that we don't split match, and break to store
+ * the last literals. We use the original currSeq.litLength as a marker for where endPosInSequence
+ * should go. We prefer to do this whenever it is not necessary to split the match, or if doing so
+ * would cause the first half of the match to be too small
+ */
+ bytesAdjustment = endPosInSequence - currSeq.litLength;
+ endPosInSequence = currSeq.litLength;
+ break;
+ }
+ } else {
+ /* This sequence ends inside the literals, break to store the last literals */
+ break;
+ }
+ }
+ /* Check if this offset can be represented with a repcode */
+ { U32 ll0 = (litLength == 0);
+ offCode = ZSTD_finalizeOffCode(rawOffset, updatedRepcodes.rep, ll0);
+ updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
+ }
+
+ if (cctx->appliedParams.validateSequences) {
+ seqPos->posInSrc += litLength + matchLength;
+ FORWARD_IF_ERROR(ZSTD_validateSequence(offCode, matchLength, seqPos->posInSrc,
+ cctx->appliedParams.cParams.windowLog, dictSize,
+ cctx->appliedParams.cParams.minMatch),
+ "Sequence validation failed");
+ }
+ DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offCode, matchLength, litLength);
+ RETURN_ERROR_IF(idx - seqPos->idx > cctx->seqStore.maxNbSeq, memory_allocation,
+ "Not enough memory allocated. Try adjusting ZSTD_c_minMatch.");
+ ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength - MINMATCH);
+ ip += matchLength + litLength;
+ }
+ DEBUGLOG(5, "Ending seq: idx: %u (of: %u ml: %u ll: %u)", idx, inSeqs[idx].offset, inSeqs[idx].matchLength, inSeqs[idx].litLength);
+ assert(idx == inSeqsSize || endPosInSequence <= inSeqs[idx].litLength + inSeqs[idx].matchLength);
+ seqPos->idx = idx;
+ seqPos->posInSequence = endPosInSequence;
+ ZSTD_memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, sizeof(repcodes_t));
+
+ iend -= bytesAdjustment;
+ if (ip != iend) {
+ /* Store any last literals */
+ U32 lastLLSize = (U32)(iend - ip);
+ assert(ip <= iend);
+ DEBUGLOG(6, "Storing last literals of size: %u", lastLLSize);
+ ZSTD_storeLastLiterals(&cctx->seqStore, ip, lastLLSize);
+ seqPos->posInSrc += lastLLSize;
+ }
+
+ return bytesAdjustment;
+}
+
+typedef size_t (*ZSTD_sequenceCopier) (ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
+ const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
+ const void* src, size_t blockSize);
+static ZSTD_sequenceCopier ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode) {
+ ZSTD_sequenceCopier sequenceCopier = NULL;
+ assert(ZSTD_cParam_withinBounds(ZSTD_c_blockDelimiters, mode));
+ if (mode == ZSTD_sf_explicitBlockDelimiters) {
+ return ZSTD_copySequencesToSeqStoreExplicitBlockDelim;
+ } else if (mode == ZSTD_sf_noBlockDelimiters) {
+ return ZSTD_copySequencesToSeqStoreNoBlockDelim;
+ }
+ assert(sequenceCopier != NULL);
+ return sequenceCopier;
+}
+
+/* Compress, block-by-block, all of the sequences given.
+ *
+ * Returns the cumulative size of all compressed blocks (including their headers), otherwise a ZSTD error.
+ */
+static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
+ const void* src, size_t srcSize) {
+ size_t cSize = 0;
+ U32 lastBlock;
+ size_t blockSize;
+ size_t compressedSeqsSize;
+ size_t remaining = srcSize;
+ ZSTD_sequencePosition seqPos = {0, 0, 0};
+
+ BYTE const* ip = (BYTE const*)src;
+ BYTE* op = (BYTE*)dst;
+ ZSTD_sequenceCopier sequenceCopier = ZSTD_selectSequenceCopier(cctx->appliedParams.blockDelimiters);
+
+ DEBUGLOG(4, "ZSTD_compressSequences_internal srcSize: %zu, inSeqsSize: %zu", srcSize, inSeqsSize);
+ /* Special case: empty frame */
+ if (remaining == 0) {
+ U32 const cBlockHeader24 = 1 /* last block */ + (((U32)bt_raw)<<1);
+ RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "No room for empty frame block header");
+ MEM_writeLE32(op, cBlockHeader24);
+ op += ZSTD_blockHeaderSize;
+ dstCapacity -= ZSTD_blockHeaderSize;
+ cSize += ZSTD_blockHeaderSize;
+ }
+
+ while (remaining) {
+ size_t cBlockSize;
+ size_t additionalByteAdjustment;
+ lastBlock = remaining <= cctx->blockSize;
+ blockSize = lastBlock ? (U32)remaining : (U32)cctx->blockSize;
+ ZSTD_resetSeqStore(&cctx->seqStore);
+ DEBUGLOG(4, "Working on new block. Blocksize: %zu", blockSize);
+
+ additionalByteAdjustment = sequenceCopier(cctx, &seqPos, inSeqs, inSeqsSize, ip, blockSize);
+ FORWARD_IF_ERROR(additionalByteAdjustment, "Bad sequence copy");
+ blockSize -= additionalByteAdjustment;
+
+ /* If blocks are too small, emit as a nocompress block */
+ if (blockSize < MIN_CBLOCK_SIZE+ZSTD_blockHeaderSize+1) {
+ cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cBlockSize, "Nocompress block failed");
+ DEBUGLOG(4, "Block too small, writing out nocompress block: cSize: %zu", cBlockSize);
+ cSize += cBlockSize;
+ ip += blockSize;
+ op += cBlockSize;
+ remaining -= blockSize;
+ dstCapacity -= cBlockSize;
+ continue;
+ }
+
+ compressedSeqsSize = ZSTD_entropyCompressSequences(&cctx->seqStore,
+ &cctx->blockState.prevCBlock->entropy, &cctx->blockState.nextCBlock->entropy,
+ &cctx->appliedParams,
+ op + ZSTD_blockHeaderSize /* Leave space for block header */, dstCapacity - ZSTD_blockHeaderSize,
+ blockSize,
+ cctx->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
+ cctx->bmi2);
+ FORWARD_IF_ERROR(compressedSeqsSize, "Compressing sequences of block failed");
+ DEBUGLOG(4, "Compressed sequences size: %zu", compressedSeqsSize);
+
+ if (!cctx->isFirstBlock &&
+ ZSTD_maybeRLE(&cctx->seqStore) &&
+ ZSTD_isRLE((BYTE const*)src, srcSize)) {
+ /* We don't want to emit our first block as a RLE even if it qualifies because
+ * doing so will cause the decoder (cli only) to throw a "should consume all input error."
+ * This is only an issue for zstd <= v1.4.3
+ */
+ compressedSeqsSize = 1;
+ }
+
+ if (compressedSeqsSize == 0) {
+ /* ZSTD_noCompressBlock writes the block header as well */
+ cBlockSize = ZSTD_noCompressBlock(op, dstCapacity, ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cBlockSize, "Nocompress block failed");
+ DEBUGLOG(4, "Writing out nocompress block, size: %zu", cBlockSize);
+ } else if (compressedSeqsSize == 1) {
+ cBlockSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cBlockSize, "RLE compress block failed");
+ DEBUGLOG(4, "Writing out RLE block, size: %zu", cBlockSize);
+ } else {
+ U32 cBlockHeader;
+ /* Error checking and repcodes update */
+ ZSTD_confirmRepcodesAndEntropyTables(cctx);
+ if (cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+ cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+
+ /* Write block header into beginning of block*/
+ cBlockHeader = lastBlock + (((U32)bt_compressed)<<1) + (U32)(compressedSeqsSize << 3);
+ MEM_writeLE24(op, cBlockHeader);
+ cBlockSize = ZSTD_blockHeaderSize + compressedSeqsSize;
+ DEBUGLOG(4, "Writing out compressed block, size: %zu", cBlockSize);
+ }
+
+ cSize += cBlockSize;
+ DEBUGLOG(4, "cSize running total: %zu", cSize);
+
+ if (lastBlock) {
+ break;
+ } else {
+ ip += blockSize;
+ op += cBlockSize;
+ remaining -= blockSize;
+ dstCapacity -= cBlockSize;
+ cctx->isFirstBlock = 0;
+ }
+ }
+
+ return cSize;
+}
+
+size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstCapacity,
+ const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
+ const void* src, size_t srcSize) {
+ BYTE* op = (BYTE*)dst;
+ size_t cSize = 0;
+ size_t compressedBlocksSize = 0;
+ size_t frameHeaderSize = 0;
+
+ /* Transparent initialization stage, same as compressStream2() */
+ DEBUGLOG(3, "ZSTD_compressSequences()");
+ assert(cctx != NULL);
+ FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, ZSTD_e_end, srcSize), "CCtx initialization failed");
+ /* Begin writing output, starting with frame header */
+ frameHeaderSize = ZSTD_writeFrameHeader(op, dstCapacity, &cctx->appliedParams, srcSize, cctx->dictID);
+ op += frameHeaderSize;
+ dstCapacity -= frameHeaderSize;
+ cSize += frameHeaderSize;
+ if (cctx->appliedParams.fParams.checksumFlag && srcSize) {
+ xxh64_update(&cctx->xxhState, src, srcSize);
+ }
+ /* cSize includes block header size and compressed sequences size */
+ compressedBlocksSize = ZSTD_compressSequences_internal(cctx,
+ op, dstCapacity,
+ inSeqs, inSeqsSize,
+ src, srcSize);
+ FORWARD_IF_ERROR(compressedBlocksSize, "Compressing blocks failed!");
+ cSize += compressedBlocksSize;
+ dstCapacity -= compressedBlocksSize;
+
+ if (cctx->appliedParams.fParams.checksumFlag) {
+ U32 const checksum = (U32) xxh64_digest(&cctx->xxhState);
+ RETURN_ERROR_IF(dstCapacity<4, dstSize_tooSmall, "no room for checksum");
+ DEBUGLOG(4, "Write checksum : %08X", (unsigned)checksum);
+ MEM_writeLE32((char*)dst + cSize, checksum);
+ cSize += 4;
+ }
+
+ DEBUGLOG(3, "Final compressed size: %zu", cSize);
+ return cSize;
+}
+
+/*====== Finalize ======*/
+
+/*! ZSTD_flushStream() :
+ * @return : amount of data remaining to flush */
+size_t ZSTD_flushStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
+{
+ ZSTD_inBuffer input = { NULL, 0, 0 };
+ return ZSTD_compressStream2(zcs, output, &input, ZSTD_e_flush);
+}
+
+
+size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
+{
+ ZSTD_inBuffer input = { NULL, 0, 0 };
+ size_t const remainingToFlush = ZSTD_compressStream2(zcs, output, &input, ZSTD_e_end);
+ FORWARD_IF_ERROR( remainingToFlush , "ZSTD_compressStream2 failed");
+ if (zcs->appliedParams.nbWorkers > 0) return remainingToFlush; /* minimal estimation */
+ /* single thread mode : attempt to calculate remaining to flush more precisely */
+ { size_t const lastBlockSize = zcs->frameEnded ? 0 : ZSTD_BLOCKHEADERSIZE;
+ size_t const checksumSize = (size_t)(zcs->frameEnded ? 0 : zcs->appliedParams.fParams.checksumFlag * 4);
+ size_t const toFlush = remainingToFlush + lastBlockSize + checksumSize;
+ DEBUGLOG(4, "ZSTD_endStream : remaining to flush : %u", (unsigned)toFlush);
+ return toFlush;
+ }
+}
+
+
+/*-===== Pre-defined compression levels =====-*/
+
+#define ZSTD_MAX_CLEVEL 22
+int ZSTD_maxCLevel(void) { return ZSTD_MAX_CLEVEL; }
+int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; }
+
+static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
+{ /* "default" - for any srcSize > 256 KB */
+ /* W, C, H, S, L, TL, strat */
+ { 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
+ { 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
+ { 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
+ { 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
+ { 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
+ { 21, 18, 19, 2, 5, 2, ZSTD_greedy }, /* level 5 */
+ { 21, 19, 19, 3, 5, 4, ZSTD_greedy }, /* level 6 */
+ { 21, 19, 19, 3, 5, 8, ZSTD_lazy }, /* level 7 */
+ { 21, 19, 19, 3, 5, 16, ZSTD_lazy2 }, /* level 8 */
+ { 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
+ { 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 10 */
+ { 22, 21, 22, 4, 5, 16, ZSTD_lazy2 }, /* level 11 */
+ { 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 12 */
+ { 22, 21, 22, 5, 5, 32, ZSTD_btlazy2 }, /* level 13 */
+ { 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
+ { 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
+ { 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
+ { 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
+ { 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
+ { 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
+ { 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
+ { 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
+ { 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
+},
+{ /* for srcSize <= 256 KB */
+ /* W, C, H, S, L, T, strat */
+ { 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
+ { 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
+ { 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
+ { 18, 16, 17, 2, 5, 2, ZSTD_greedy }, /* level 4.*/
+ { 18, 18, 18, 3, 5, 2, ZSTD_greedy }, /* level 5.*/
+ { 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
+ { 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
+ { 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
+ { 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
+ { 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
+ { 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
+ { 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/
+ { 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */
+ { 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
+ { 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/
+ { 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/
+ { 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/
+ { 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/
+ { 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+{ /* for srcSize <= 128 KB */
+ /* W, C, H, S, L, T, strat */
+ { 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
+ { 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
+ { 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
+ { 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
+ { 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
+ { 17, 17, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
+ { 17, 17, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
+ { 17, 17, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
+ { 17, 17, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
+ { 17, 17, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
+ { 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
+ { 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
+ { 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
+ { 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
+ { 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
+ { 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
+ { 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
+ { 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
+ { 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+{ /* for srcSize <= 16 KB */
+ /* W, C, H, S, L, T, strat */
+ { 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
+ { 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
+ { 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
+ { 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
+ { 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
+ { 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
+ { 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
+ { 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
+ { 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
+ { 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
+ { 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/
+ { 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/
+ { 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/
+ { 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/
+ { 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/
+ { 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/
+ { 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/
+ { 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/
+ { 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+};
+
+static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams(int const compressionLevel, size_t const dictSize)
+{
+ ZSTD_compressionParameters cParams = ZSTD_getCParams_internal(compressionLevel, 0, dictSize, ZSTD_cpm_createCDict);
+ switch (cParams.strategy) {
+ case ZSTD_fast:
+ case ZSTD_dfast:
+ break;
+ case ZSTD_greedy:
+ case ZSTD_lazy:
+ case ZSTD_lazy2:
+ cParams.hashLog += ZSTD_LAZY_DDSS_BUCKET_LOG;
+ break;
+ case ZSTD_btlazy2:
+ case ZSTD_btopt:
+ case ZSTD_btultra:
+ case ZSTD_btultra2:
+ break;
+ }
+ return cParams;
+}
+
+static int ZSTD_dedicatedDictSearch_isSupported(
+ ZSTD_compressionParameters const* cParams)
+{
+ return (cParams->strategy >= ZSTD_greedy)
+ && (cParams->strategy <= ZSTD_lazy2)
+ && (cParams->hashLog >= cParams->chainLog)
+ && (cParams->chainLog <= 24);
+}
+
+/*
+ * Reverses the adjustment applied to cparams when enabling dedicated dict
+ * search. This is used to recover the params set to be used in the working
+ * context. (Otherwise, those tables would also grow.)
+ */
+static void ZSTD_dedicatedDictSearch_revertCParams(
+ ZSTD_compressionParameters* cParams) {
+ switch (cParams->strategy) {
+ case ZSTD_fast:
+ case ZSTD_dfast:
+ break;
+ case ZSTD_greedy:
+ case ZSTD_lazy:
+ case ZSTD_lazy2:
+ cParams->hashLog -= ZSTD_LAZY_DDSS_BUCKET_LOG;
+ break;
+ case ZSTD_btlazy2:
+ case ZSTD_btopt:
+ case ZSTD_btultra:
+ case ZSTD_btultra2:
+ break;
+ }
+}
+
+static U64 ZSTD_getCParamRowSize(U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode)
+{
+ switch (mode) {
+ case ZSTD_cpm_unknown:
+ case ZSTD_cpm_noAttachDict:
+ case ZSTD_cpm_createCDict:
+ break;
+ case ZSTD_cpm_attachDict:
+ dictSize = 0;
+ break;
+ default:
+ assert(0);
+ break;
+ }
+ { int const unknown = srcSizeHint == ZSTD_CONTENTSIZE_UNKNOWN;
+ size_t const addedSize = unknown && dictSize > 0 ? 500 : 0;
+ return unknown && dictSize == 0 ? ZSTD_CONTENTSIZE_UNKNOWN : srcSizeHint+dictSize+addedSize;
+ }
+}
+
+/*! ZSTD_getCParams_internal() :
+ * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
+ * Note: srcSizeHint 0 means 0, use ZSTD_CONTENTSIZE_UNKNOWN for unknown.
+ * Use dictSize == 0 for unknown or unused.
+ * Note: `mode` controls how we treat the `dictSize`. See docs for `ZSTD_cParamMode_e`. */
+static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode)
+{
+ U64 const rSize = ZSTD_getCParamRowSize(srcSizeHint, dictSize, mode);
+ U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB);
+ int row;
+ DEBUGLOG(5, "ZSTD_getCParams_internal (cLevel=%i)", compressionLevel);
+
+ /* row */
+ if (compressionLevel == 0) row = ZSTD_CLEVEL_DEFAULT; /* 0 == default */
+ else if (compressionLevel < 0) row = 0; /* entry 0 is baseline for fast mode */
+ else if (compressionLevel > ZSTD_MAX_CLEVEL) row = ZSTD_MAX_CLEVEL;
+ else row = compressionLevel;
+
+ { ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row];
+ /* acceleration factor */
+ if (compressionLevel < 0) {
+ int const clampedCompressionLevel = MAX(ZSTD_minCLevel(), compressionLevel);
+ cp.targetLength = (unsigned)(-clampedCompressionLevel);
+ }
+ /* refine parameters based on srcSize & dictSize */
+ return ZSTD_adjustCParams_internal(cp, srcSizeHint, dictSize, mode);
+ }
+}
+
+/*! ZSTD_getCParams() :
+ * @return ZSTD_compressionParameters structure for a selected compression level, srcSize and dictSize.
+ * Size values are optional, provide 0 if not known or unused */
+ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize)
+{
+ if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
+ return ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize, ZSTD_cpm_unknown);
+}
+
+/*! ZSTD_getParams() :
+ * same idea as ZSTD_getCParams()
+ * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
+ * Fields of `ZSTD_frameParameters` are set to default values */
+static ZSTD_parameters ZSTD_getParams_internal(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode) {
+ ZSTD_parameters params;
+ ZSTD_compressionParameters const cParams = ZSTD_getCParams_internal(compressionLevel, srcSizeHint, dictSize, mode);
+ DEBUGLOG(5, "ZSTD_getParams (cLevel=%i)", compressionLevel);
+ ZSTD_memset(¶ms, 0, sizeof(params));
+ params.cParams = cParams;
+ params.fParams.contentSizeFlag = 1;
+ return params;
+}
+
+/*! ZSTD_getParams() :
+ * same idea as ZSTD_getCParams()
+ * @return a `ZSTD_parameters` structure (instead of `ZSTD_compressionParameters`).
+ * Fields of `ZSTD_frameParameters` are set to default values */
+ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long srcSizeHint, size_t dictSize) {
+ if (srcSizeHint == 0) srcSizeHint = ZSTD_CONTENTSIZE_UNKNOWN;
+ return ZSTD_getParams_internal(compressionLevel, srcSizeHint, dictSize, ZSTD_cpm_unknown);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/* This header contains definitions
+ * that shall **only** be used by modules within lib/compress.
+ */
+
+#ifndef ZSTD_COMPRESS_H
+#define ZSTD_COMPRESS_H
+
+/*-*************************************
+* Dependencies
+***************************************/
+#include "../common/zstd_internal.h"
+#include "zstd_cwksp.h"
+
+
+/*-*************************************
+* Constants
+***************************************/
+#define kSearchStrength 8
+#define HASH_READ_SIZE 8
+#define ZSTD_DUBT_UNSORTED_MARK 1 /* For btlazy2 strategy, index ZSTD_DUBT_UNSORTED_MARK==1 means "unsorted".
+ It could be confused for a real successor at index "1", if sorted as larger than its predecessor.
+ It's not a big deal though : candidate will just be sorted again.
+ Additionally, candidate position 1 will be lost.
+ But candidate 1 cannot hide a large tree of candidates, so it's a minimal loss.
+ The benefit is that ZSTD_DUBT_UNSORTED_MARK cannot be mishandled after table re-use with a different strategy.
+ This constant is required by ZSTD_compressBlock_btlazy2() and ZSTD_reduceTable_internal() */
+
+
+/*-*************************************
+* Context memory management
+***************************************/
+typedef enum { ZSTDcs_created=0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e;
+typedef enum { zcss_init=0, zcss_load, zcss_flush } ZSTD_cStreamStage;
+
+typedef struct ZSTD_prefixDict_s {
+ const void* dict;
+ size_t dictSize;
+ ZSTD_dictContentType_e dictContentType;
+} ZSTD_prefixDict;
+
+typedef struct {
+ void* dictBuffer;
+ void const* dict;
+ size_t dictSize;
+ ZSTD_dictContentType_e dictContentType;
+ ZSTD_CDict* cdict;
+} ZSTD_localDict;
+
+typedef struct {
+ HUF_CElt CTable[HUF_CTABLE_SIZE_U32(255)];
+ HUF_repeat repeatMode;
+} ZSTD_hufCTables_t;
+
+typedef struct {
+ FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)];
+ FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)];
+ FSE_CTable litlengthCTable[FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)];
+ FSE_repeat offcode_repeatMode;
+ FSE_repeat matchlength_repeatMode;
+ FSE_repeat litlength_repeatMode;
+} ZSTD_fseCTables_t;
+
+typedef struct {
+ ZSTD_hufCTables_t huf;
+ ZSTD_fseCTables_t fse;
+} ZSTD_entropyCTables_t;
+
+typedef struct {
+ U32 off; /* Offset code (offset + ZSTD_REP_MOVE) for the match */
+ U32 len; /* Raw length of match */
+} ZSTD_match_t;
+
+typedef struct {
+ U32 offset; /* Offset of sequence */
+ U32 litLength; /* Length of literals prior to match */
+ U32 matchLength; /* Raw length of match */
+} rawSeq;
+
+typedef struct {
+ rawSeq* seq; /* The start of the sequences */
+ size_t pos; /* The index in seq where reading stopped. pos <= size. */
+ size_t posInSequence; /* The position within the sequence at seq[pos] where reading
+ stopped. posInSequence <= seq[pos].litLength + seq[pos].matchLength */
+ size_t size; /* The number of sequences. <= capacity. */
+ size_t capacity; /* The capacity starting from `seq` pointer */
+} rawSeqStore_t;
+
+UNUSED_ATTR static const rawSeqStore_t kNullRawSeqStore = {NULL, 0, 0, 0, 0};
+
+typedef struct {
+ int price;
+ U32 off;
+ U32 mlen;
+ U32 litlen;
+ U32 rep[ZSTD_REP_NUM];
+} ZSTD_optimal_t;
+
+typedef enum { zop_dynamic=0, zop_predef } ZSTD_OptPrice_e;
+
+typedef struct {
+ /* All tables are allocated inside cctx->workspace by ZSTD_resetCCtx_internal() */
+ unsigned* litFreq; /* table of literals statistics, of size 256 */
+ unsigned* litLengthFreq; /* table of litLength statistics, of size (MaxLL+1) */
+ unsigned* matchLengthFreq; /* table of matchLength statistics, of size (MaxML+1) */
+ unsigned* offCodeFreq; /* table of offCode statistics, of size (MaxOff+1) */
+ ZSTD_match_t* matchTable; /* list of found matches, of size ZSTD_OPT_NUM+1 */
+ ZSTD_optimal_t* priceTable; /* All positions tracked by optimal parser, of size ZSTD_OPT_NUM+1 */
+
+ U32 litSum; /* nb of literals */
+ U32 litLengthSum; /* nb of litLength codes */
+ U32 matchLengthSum; /* nb of matchLength codes */
+ U32 offCodeSum; /* nb of offset codes */
+ U32 litSumBasePrice; /* to compare to log2(litfreq) */
+ U32 litLengthSumBasePrice; /* to compare to log2(llfreq) */
+ U32 matchLengthSumBasePrice;/* to compare to log2(mlfreq) */
+ U32 offCodeSumBasePrice; /* to compare to log2(offreq) */
+ ZSTD_OptPrice_e priceType; /* prices can be determined dynamically, or follow a pre-defined cost structure */
+ const ZSTD_entropyCTables_t* symbolCosts; /* pre-calculated dictionary statistics */
+ ZSTD_literalCompressionMode_e literalCompressionMode;
+} optState_t;
+
+typedef struct {
+ ZSTD_entropyCTables_t entropy;
+ U32 rep[ZSTD_REP_NUM];
+} ZSTD_compressedBlockState_t;
+
+typedef struct {
+ BYTE const* nextSrc; /* next block here to continue on current prefix */
+ BYTE const* base; /* All regular indexes relative to this position */
+ BYTE const* dictBase; /* extDict indexes relative to this position */
+ U32 dictLimit; /* below that point, need extDict */
+ U32 lowLimit; /* below that point, no more valid data */
+} ZSTD_window_t;
+
+typedef struct ZSTD_matchState_t ZSTD_matchState_t;
+struct ZSTD_matchState_t {
+ ZSTD_window_t window; /* State for window round buffer management */
+ U32 loadedDictEnd; /* index of end of dictionary, within context's referential.
+ * When loadedDictEnd != 0, a dictionary is in use, and still valid.
+ * This relies on a mechanism to set loadedDictEnd=0 when dictionary is no longer within distance.
+ * Such mechanism is provided within ZSTD_window_enforceMaxDist() and ZSTD_checkDictValidity().
+ * When dict referential is copied into active context (i.e. not attached),
+ * loadedDictEnd == dictSize, since referential starts from zero.
+ */
+ U32 nextToUpdate; /* index from which to continue table update */
+ U32 hashLog3; /* dispatch table for matches of len==3 : larger == faster, more memory */
+ U32* hashTable;
+ U32* hashTable3;
+ U32* chainTable;
+ int dedicatedDictSearch; /* Indicates whether this matchState is using the
+ * dedicated dictionary search structure.
+ */
+ optState_t opt; /* optimal parser state */
+ const ZSTD_matchState_t* dictMatchState;
+ ZSTD_compressionParameters cParams;
+ const rawSeqStore_t* ldmSeqStore;
+};
+
+typedef struct {
+ ZSTD_compressedBlockState_t* prevCBlock;
+ ZSTD_compressedBlockState_t* nextCBlock;
+ ZSTD_matchState_t matchState;
+} ZSTD_blockState_t;
+
+typedef struct {
+ U32 offset;
+ U32 checksum;
+} ldmEntry_t;
+
+typedef struct {
+ BYTE const* split;
+ U32 hash;
+ U32 checksum;
+ ldmEntry_t* bucket;
+} ldmMatchCandidate_t;
+
+#define LDM_BATCH_SIZE 64
+
+typedef struct {
+ ZSTD_window_t window; /* State for the window round buffer management */
+ ldmEntry_t* hashTable;
+ U32 loadedDictEnd;
+ BYTE* bucketOffsets; /* Next position in bucket to insert entry */
+ size_t splitIndices[LDM_BATCH_SIZE];
+ ldmMatchCandidate_t matchCandidates[LDM_BATCH_SIZE];
+} ldmState_t;
+
+typedef struct {
+ U32 enableLdm; /* 1 if enable long distance matching */
+ U32 hashLog; /* Log size of hashTable */
+ U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */
+ U32 minMatchLength; /* Minimum match length */
+ U32 hashRateLog; /* Log number of entries to skip */
+ U32 windowLog; /* Window log for the LDM */
+} ldmParams_t;
+
+typedef struct {
+ int collectSequences;
+ ZSTD_Sequence* seqStart;
+ size_t seqIndex;
+ size_t maxSequences;
+} SeqCollector;
+
+struct ZSTD_CCtx_params_s {
+ ZSTD_format_e format;
+ ZSTD_compressionParameters cParams;
+ ZSTD_frameParameters fParams;
+
+ int compressionLevel;
+ int forceWindow; /* force back-references to respect limit of
+ * 1<<wLog, even for dictionary */
+ size_t targetCBlockSize; /* Tries to fit compressed block size to be around targetCBlockSize.
+ * No target when targetCBlockSize == 0.
+ * There is no guarantee on compressed block size */
+ int srcSizeHint; /* User's best guess of source size.
+ * Hint is not valid when srcSizeHint == 0.
+ * There is no guarantee that hint is close to actual source size */
+
+ ZSTD_dictAttachPref_e attachDictPref;
+ ZSTD_literalCompressionMode_e literalCompressionMode;
+
+ /* Multithreading: used to pass parameters to mtctx */
+ int nbWorkers;
+ size_t jobSize;
+ int overlapLog;
+ int rsyncable;
+
+ /* Long distance matching parameters */
+ ldmParams_t ldmParams;
+
+ /* Dedicated dict search algorithm trigger */
+ int enableDedicatedDictSearch;
+
+ /* Input/output buffer modes */
+ ZSTD_bufferMode_e inBufferMode;
+ ZSTD_bufferMode_e outBufferMode;
+
+ /* Sequence compression API */
+ ZSTD_sequenceFormat_e blockDelimiters;
+ int validateSequences;
+
+ /* Internal use, for createCCtxParams() and freeCCtxParams() only */
+ ZSTD_customMem customMem;
+}; /* typedef'd to ZSTD_CCtx_params within "zstd.h" */
+
+#define COMPRESS_SEQUENCES_WORKSPACE_SIZE (sizeof(unsigned) * (MaxSeq + 2))
+#define ENTROPY_WORKSPACE_SIZE (HUF_WORKSPACE_SIZE + COMPRESS_SEQUENCES_WORKSPACE_SIZE)
+
+/*
+ * Indicates whether this compression proceeds directly from user-provided
+ * source buffer to user-provided destination buffer (ZSTDb_not_buffered), or
+ * whether the context needs to buffer the input/output (ZSTDb_buffered).
+ */
+typedef enum {
+ ZSTDb_not_buffered,
+ ZSTDb_buffered
+} ZSTD_buffered_policy_e;
+
+struct ZSTD_CCtx_s {
+ ZSTD_compressionStage_e stage;
+ int cParamsChanged; /* == 1 if cParams(except wlog) or compression level are changed in requestedParams. Triggers transmission of new params to ZSTDMT (if available) then reset to 0. */
+ int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
+ ZSTD_CCtx_params requestedParams;
+ ZSTD_CCtx_params appliedParams;
+ U32 dictID;
+ size_t dictContentSize;
+
+ ZSTD_cwksp workspace; /* manages buffer for dynamic allocations */
+ size_t blockSize;
+ unsigned long long pledgedSrcSizePlusOne; /* this way, 0 (default) == unknown */
+ unsigned long long consumedSrcSize;
+ unsigned long long producedCSize;
+ struct xxh64_state xxhState;
+ ZSTD_customMem customMem;
+ ZSTD_threadPool* pool;
+ size_t staticSize;
+ SeqCollector seqCollector;
+ int isFirstBlock;
+ int initialized;
+
+ seqStore_t seqStore; /* sequences storage ptrs */
+ ldmState_t ldmState; /* long distance matching state */
+ rawSeq* ldmSequences; /* Storage for the ldm output sequences */
+ size_t maxNbLdmSequences;
+ rawSeqStore_t externSeqStore; /* Mutable reference to external sequences */
+ ZSTD_blockState_t blockState;
+ U32* entropyWorkspace; /* entropy workspace of ENTROPY_WORKSPACE_SIZE bytes */
+
+ /* Wether we are streaming or not */
+ ZSTD_buffered_policy_e bufferedPolicy;
+
+ /* streaming */
+ char* inBuff;
+ size_t inBuffSize;
+ size_t inToCompress;
+ size_t inBuffPos;
+ size_t inBuffTarget;
+ char* outBuff;
+ size_t outBuffSize;
+ size_t outBuffContentSize;
+ size_t outBuffFlushedSize;
+ ZSTD_cStreamStage streamStage;
+ U32 frameEnded;
+
+ /* Stable in/out buffer verification */
+ ZSTD_inBuffer expectedInBuffer;
+ size_t expectedOutBufferSize;
+
+ /* Dictionary */
+ ZSTD_localDict localDict;
+ const ZSTD_CDict* cdict;
+ ZSTD_prefixDict prefixDict; /* single-usage dictionary */
+
+ /* Multi-threading */
+
+ /* Tracing */
+};
+
+typedef enum { ZSTD_dtlm_fast, ZSTD_dtlm_full } ZSTD_dictTableLoadMethod_e;
+
+typedef enum {
+ ZSTD_noDict = 0,
+ ZSTD_extDict = 1,
+ ZSTD_dictMatchState = 2,
+ ZSTD_dedicatedDictSearch = 3
+} ZSTD_dictMode_e;
+
+typedef enum {
+ ZSTD_cpm_noAttachDict = 0, /* Compression with ZSTD_noDict or ZSTD_extDict.
+ * In this mode we use both the srcSize and the dictSize
+ * when selecting and adjusting parameters.
+ */
+ ZSTD_cpm_attachDict = 1, /* Compression with ZSTD_dictMatchState or ZSTD_dedicatedDictSearch.
+ * In this mode we only take the srcSize into account when selecting
+ * and adjusting parameters.
+ */
+ ZSTD_cpm_createCDict = 2, /* Creating a CDict.
+ * In this mode we take both the source size and the dictionary size
+ * into account when selecting and adjusting the parameters.
+ */
+ ZSTD_cpm_unknown = 3, /* ZSTD_getCParams, ZSTD_getParams, ZSTD_adjustParams.
+ * We don't know what these parameters are for. We default to the legacy
+ * behavior of taking both the source size and the dict size into account
+ * when selecting and adjusting parameters.
+ */
+} ZSTD_cParamMode_e;
+
+typedef size_t (*ZSTD_blockCompressor) (
+ ZSTD_matchState_t* bs, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode);
+
+
+MEM_STATIC U32 ZSTD_LLcode(U32 litLength)
+{
+ static const BYTE LL_Code[64] = { 0, 1, 2, 3, 4, 5, 6, 7,
+ 8, 9, 10, 11, 12, 13, 14, 15,
+ 16, 16, 17, 17, 18, 18, 19, 19,
+ 20, 20, 20, 20, 21, 21, 21, 21,
+ 22, 22, 22, 22, 22, 22, 22, 22,
+ 23, 23, 23, 23, 23, 23, 23, 23,
+ 24, 24, 24, 24, 24, 24, 24, 24,
+ 24, 24, 24, 24, 24, 24, 24, 24 };
+ static const U32 LL_deltaCode = 19;
+ return (litLength > 63) ? ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
+}
+
+/* ZSTD_MLcode() :
+ * note : mlBase = matchLength - MINMATCH;
+ * because it's the format it's stored in seqStore->sequences */
+MEM_STATIC U32 ZSTD_MLcode(U32 mlBase)
+{
+ static const BYTE ML_Code[128] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
+ 32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37,
+ 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39,
+ 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40,
+ 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41,
+ 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42,
+ 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42 };
+ static const U32 ML_deltaCode = 36;
+ return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase];
+}
+
+typedef struct repcodes_s {
+ U32 rep[3];
+} repcodes_t;
+
+MEM_STATIC repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0)
+{
+ repcodes_t newReps;
+ if (offset >= ZSTD_REP_NUM) { /* full offset */
+ newReps.rep[2] = rep[1];
+ newReps.rep[1] = rep[0];
+ newReps.rep[0] = offset - ZSTD_REP_MOVE;
+ } else { /* repcode */
+ U32 const repCode = offset + ll0;
+ if (repCode > 0) { /* note : if repCode==0, no change */
+ U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
+ newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2];
+ newReps.rep[1] = rep[0];
+ newReps.rep[0] = currentOffset;
+ } else { /* repCode == 0 */
+ ZSTD_memcpy(&newReps, rep, sizeof(newReps));
+ }
+ }
+ return newReps;
+}
+
+/* ZSTD_cParam_withinBounds:
+ * @return 1 if value is within cParam bounds,
+ * 0 otherwise */
+MEM_STATIC int ZSTD_cParam_withinBounds(ZSTD_cParameter cParam, int value)
+{
+ ZSTD_bounds const bounds = ZSTD_cParam_getBounds(cParam);
+ if (ZSTD_isError(bounds.error)) return 0;
+ if (value < bounds.lowerBound) return 0;
+ if (value > bounds.upperBound) return 0;
+ return 1;
+}
+
+/* ZSTD_noCompressBlock() :
+ * Writes uncompressed block to dst buffer from given src.
+ * Returns the size of the block */
+MEM_STATIC size_t ZSTD_noCompressBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize, U32 lastBlock)
+{
+ U32 const cBlockHeader24 = lastBlock + (((U32)bt_raw)<<1) + (U32)(srcSize << 3);
+ RETURN_ERROR_IF(srcSize + ZSTD_blockHeaderSize > dstCapacity,
+ dstSize_tooSmall, "dst buf too small for uncompressed block");
+ MEM_writeLE24(dst, cBlockHeader24);
+ ZSTD_memcpy((BYTE*)dst + ZSTD_blockHeaderSize, src, srcSize);
+ return ZSTD_blockHeaderSize + srcSize;
+}
+
+MEM_STATIC size_t ZSTD_rleCompressBlock (void* dst, size_t dstCapacity, BYTE src, size_t srcSize, U32 lastBlock)
+{
+ BYTE* const op = (BYTE*)dst;
+ U32 const cBlockHeader = lastBlock + (((U32)bt_rle)<<1) + (U32)(srcSize << 3);
+ RETURN_ERROR_IF(dstCapacity < 4, dstSize_tooSmall, "");
+ MEM_writeLE24(op, cBlockHeader);
+ op[3] = src;
+ return 4;
+}
+
+
+/* ZSTD_minGain() :
+ * minimum compression required
+ * to generate a compress block or a compressed literals section.
+ * note : use same formula for both situations */
+MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat)
+{
+ U32 const minlog = (strat>=ZSTD_btultra) ? (U32)(strat) - 1 : 6;
+ ZSTD_STATIC_ASSERT(ZSTD_btultra == 8);
+ assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
+ return (srcSize >> minlog) + 2;
+}
+
+MEM_STATIC int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParams)
+{
+ switch (cctxParams->literalCompressionMode) {
+ case ZSTD_lcm_huffman:
+ return 0;
+ case ZSTD_lcm_uncompressed:
+ return 1;
+ default:
+ assert(0 /* impossible: pre-validated */);
+ ZSTD_FALLTHROUGH;
+ case ZSTD_lcm_auto:
+ return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0);
+ }
+}
+
+/*! ZSTD_safecopyLiterals() :
+ * memcpy() function that won't read beyond more than WILDCOPY_OVERLENGTH bytes past ilimit_w.
+ * Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single
+ * large copies.
+ */
+static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w) {
+ assert(iend > ilimit_w);
+ if (ip <= ilimit_w) {
+ ZSTD_wildcopy(op, ip, ilimit_w - ip, ZSTD_no_overlap);
+ op += ilimit_w - ip;
+ ip = ilimit_w;
+ }
+ while (ip < iend) *op++ = *ip++;
+}
+
+/*! ZSTD_storeSeq() :
+ * Store a sequence (litlen, litPtr, offCode and mlBase) into seqStore_t.
+ * `offCode` : distance to match + ZSTD_REP_MOVE (values <= ZSTD_REP_MOVE are repCodes).
+ * `mlBase` : matchLength - MINMATCH
+ * Allowed to overread literals up to litLimit.
+*/
+HINT_INLINE UNUSED_ATTR
+void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, const BYTE* litLimit, U32 offCode, size_t mlBase)
+{
+ BYTE const* const litLimit_w = litLimit - WILDCOPY_OVERLENGTH;
+ BYTE const* const litEnd = literals + litLength;
+#if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6)
+ static const BYTE* g_start = NULL;
+ if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */
+ { U32 const pos = (U32)((const BYTE*)literals - g_start);
+ DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u",
+ pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offCode);
+ }
+#endif
+ assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq);
+ /* copy Literals */
+ assert(seqStorePtr->maxNbLit <= 128 KB);
+ assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit);
+ assert(literals + litLength <= litLimit);
+ if (litEnd <= litLimit_w) {
+ /* Common case we can use wildcopy.
+ * First copy 16 bytes, because literals are likely short.
+ */
+ assert(WILDCOPY_OVERLENGTH >= 16);
+ ZSTD_copy16(seqStorePtr->lit, literals);
+ if (litLength > 16) {
+ ZSTD_wildcopy(seqStorePtr->lit+16, literals+16, (ptrdiff_t)litLength-16, ZSTD_no_overlap);
+ }
+ } else {
+ ZSTD_safecopyLiterals(seqStorePtr->lit, literals, litEnd, litLimit_w);
+ }
+ seqStorePtr->lit += litLength;
+
+ /* literal Length */
+ if (litLength>0xFFFF) {
+ assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
+ seqStorePtr->longLengthID = 1;
+ seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ }
+ seqStorePtr->sequences[0].litLength = (U16)litLength;
+
+ /* match offset */
+ seqStorePtr->sequences[0].offset = offCode + 1;
+
+ /* match Length */
+ if (mlBase>0xFFFF) {
+ assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
+ seqStorePtr->longLengthID = 2;
+ seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ }
+ seqStorePtr->sequences[0].matchLength = (U16)mlBase;
+
+ seqStorePtr->sequences++;
+}
+
+
+/*-*************************************
+* Match length counter
+***************************************/
+static unsigned ZSTD_NbCommonBytes (size_t val)
+{
+ if (MEM_isLittleEndian()) {
+ if (MEM_64bits()) {
+# if (__GNUC__ >= 4)
+ return (__builtin_ctzll((U64)val) >> 3);
+# else
+ static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2,
+ 0, 3, 1, 3, 1, 4, 2, 7,
+ 0, 2, 3, 6, 1, 5, 3, 5,
+ 1, 3, 4, 4, 2, 5, 6, 7,
+ 7, 0, 1, 2, 3, 3, 4, 6,
+ 2, 6, 5, 5, 3, 4, 5, 6,
+ 7, 1, 2, 4, 6, 4, 4, 5,
+ 7, 2, 6, 5, 7, 6, 7, 7 };
+ return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
+# endif
+ } else { /* 32 bits */
+# if (__GNUC__ >= 3)
+ return (__builtin_ctz((U32)val) >> 3);
+# else
+ static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0,
+ 3, 2, 2, 1, 3, 2, 0, 1,
+ 3, 3, 1, 2, 2, 2, 2, 0,
+ 3, 1, 2, 0, 1, 0, 1, 1 };
+ return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
+# endif
+ }
+ } else { /* Big Endian CPU */
+ if (MEM_64bits()) {
+# if (__GNUC__ >= 4)
+ return (__builtin_clzll(val) >> 3);
+# else
+ unsigned r;
+ const unsigned n32 = sizeof(size_t)*4; /* calculate this way due to compiler complaining in 32-bits mode */
+ if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
+ if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
+ r += (!val);
+ return r;
+# endif
+ } else { /* 32 bits */
+# if (__GNUC__ >= 3)
+ return (__builtin_clz((U32)val) >> 3);
+# else
+ unsigned r;
+ if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
+ r += (!val);
+ return r;
+# endif
+ } }
+}
+
+
+MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* const pInLimit)
+{
+ const BYTE* const pStart = pIn;
+ const BYTE* const pInLoopLimit = pInLimit - (sizeof(size_t)-1);
+
+ if (pIn < pInLoopLimit) {
+ { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
+ if (diff) return ZSTD_NbCommonBytes(diff); }
+ pIn+=sizeof(size_t); pMatch+=sizeof(size_t);
+ while (pIn < pInLoopLimit) {
+ size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
+ if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; }
+ pIn += ZSTD_NbCommonBytes(diff);
+ return (size_t)(pIn - pStart);
+ } }
+ if (MEM_64bits() && (pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; }
+ if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; }
+ if ((pIn<pInLimit) && (*pMatch == *pIn)) pIn++;
+ return (size_t)(pIn - pStart);
+}
+
+/* ZSTD_count_2segments() :
+ * can count match length with `ip` & `match` in 2 different segments.
+ * convention : on reaching mEnd, match count continue starting from iStart
+ */
+MEM_STATIC size_t
+ZSTD_count_2segments(const BYTE* ip, const BYTE* match,
+ const BYTE* iEnd, const BYTE* mEnd, const BYTE* iStart)
+{
+ const BYTE* const vEnd = MIN( ip + (mEnd - match), iEnd);
+ size_t const matchLength = ZSTD_count(ip, match, vEnd);
+ if (match + matchLength != mEnd) return matchLength;
+ DEBUGLOG(7, "ZSTD_count_2segments: found a 2-parts match (current length==%zu)", matchLength);
+ DEBUGLOG(7, "distance from match beginning to end dictionary = %zi", mEnd - match);
+ DEBUGLOG(7, "distance from current pos to end buffer = %zi", iEnd - ip);
+ DEBUGLOG(7, "next byte : ip==%02X, istart==%02X", ip[matchLength], *iStart);
+ DEBUGLOG(7, "final match length = %zu", matchLength + ZSTD_count(ip+matchLength, iStart, iEnd));
+ return matchLength + ZSTD_count(ip+matchLength, iStart, iEnd);
+}
+
+
+/*-*************************************
+ * Hashes
+ ***************************************/
+static const U32 prime3bytes = 506832829U;
+static U32 ZSTD_hash3(U32 u, U32 h) { return ((u << (32-24)) * prime3bytes) >> (32-h) ; }
+MEM_STATIC size_t ZSTD_hash3Ptr(const void* ptr, U32 h) { return ZSTD_hash3(MEM_readLE32(ptr), h); } /* only in zstd_opt.h */
+
+static const U32 prime4bytes = 2654435761U;
+static U32 ZSTD_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32-h) ; }
+static size_t ZSTD_hash4Ptr(const void* ptr, U32 h) { return ZSTD_hash4(MEM_read32(ptr), h); }
+
+static const U64 prime5bytes = 889523592379ULL;
+static size_t ZSTD_hash5(U64 u, U32 h) { return (size_t)(((u << (64-40)) * prime5bytes) >> (64-h)) ; }
+static size_t ZSTD_hash5Ptr(const void* p, U32 h) { return ZSTD_hash5(MEM_readLE64(p), h); }
+
+static const U64 prime6bytes = 227718039650203ULL;
+static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64-48)) * prime6bytes) >> (64-h)) ; }
+static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); }
+
+static const U64 prime7bytes = 58295818150454627ULL;
+static size_t ZSTD_hash7(U64 u, U32 h) { return (size_t)(((u << (64-56)) * prime7bytes) >> (64-h)) ; }
+static size_t ZSTD_hash7Ptr(const void* p, U32 h) { return ZSTD_hash7(MEM_readLE64(p), h); }
+
+static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL;
+static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; }
+static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); }
+
+MEM_STATIC FORCE_INLINE_ATTR
+size_t ZSTD_hashPtr(const void* p, U32 hBits, U32 mls)
+{
+ switch(mls)
+ {
+ default:
+ case 4: return ZSTD_hash4Ptr(p, hBits);
+ case 5: return ZSTD_hash5Ptr(p, hBits);
+ case 6: return ZSTD_hash6Ptr(p, hBits);
+ case 7: return ZSTD_hash7Ptr(p, hBits);
+ case 8: return ZSTD_hash8Ptr(p, hBits);
+ }
+}
+
+/* ZSTD_ipow() :
+ * Return base^exponent.
+ */
+static U64 ZSTD_ipow(U64 base, U64 exponent)
+{
+ U64 power = 1;
+ while (exponent) {
+ if (exponent & 1) power *= base;
+ exponent >>= 1;
+ base *= base;
+ }
+ return power;
+}
+
+#define ZSTD_ROLL_HASH_CHAR_OFFSET 10
+
+/* ZSTD_rollingHash_append() :
+ * Add the buffer to the hash value.
+ */
+static U64 ZSTD_rollingHash_append(U64 hash, void const* buf, size_t size)
+{
+ BYTE const* istart = (BYTE const*)buf;
+ size_t pos;
+ for (pos = 0; pos < size; ++pos) {
+ hash *= prime8bytes;
+ hash += istart[pos] + ZSTD_ROLL_HASH_CHAR_OFFSET;
+ }
+ return hash;
+}
+
+/* ZSTD_rollingHash_compute() :
+ * Compute the rolling hash value of the buffer.
+ */
+MEM_STATIC U64 ZSTD_rollingHash_compute(void const* buf, size_t size)
+{
+ return ZSTD_rollingHash_append(0, buf, size);
+}
+
+/* ZSTD_rollingHash_primePower() :
+ * Compute the primePower to be passed to ZSTD_rollingHash_rotate() for a hash
+ * over a window of length bytes.
+ */
+MEM_STATIC U64 ZSTD_rollingHash_primePower(U32 length)
+{
+ return ZSTD_ipow(prime8bytes, length - 1);
+}
+
+/* ZSTD_rollingHash_rotate() :
+ * Rotate the rolling hash by one byte.
+ */
+MEM_STATIC U64 ZSTD_rollingHash_rotate(U64 hash, BYTE toRemove, BYTE toAdd, U64 primePower)
+{
+ hash -= (toRemove + ZSTD_ROLL_HASH_CHAR_OFFSET) * primePower;
+ hash *= prime8bytes;
+ hash += toAdd + ZSTD_ROLL_HASH_CHAR_OFFSET;
+ return hash;
+}
+
+/*-*************************************
+* Round buffer management
+***************************************/
+#if (ZSTD_WINDOWLOG_MAX_64 > 31)
+# error "ZSTD_WINDOWLOG_MAX is too large : would overflow ZSTD_CURRENT_MAX"
+#endif
+/* Max current allowed */
+#define ZSTD_CURRENT_MAX ((3U << 29) + (1U << ZSTD_WINDOWLOG_MAX))
+/* Maximum chunk size before overflow correction needs to be called again */
+#define ZSTD_CHUNKSIZE_MAX \
+ ( ((U32)-1) /* Maximum ending current index */ \
+ - ZSTD_CURRENT_MAX) /* Maximum beginning lowLimit */
+
+/*
+ * ZSTD_window_clear():
+ * Clears the window containing the history by simply setting it to empty.
+ */
+MEM_STATIC void ZSTD_window_clear(ZSTD_window_t* window)
+{
+ size_t const endT = (size_t)(window->nextSrc - window->base);
+ U32 const end = (U32)endT;
+
+ window->lowLimit = end;
+ window->dictLimit = end;
+}
+
+/*
+ * ZSTD_window_hasExtDict():
+ * Returns non-zero if the window has a non-empty extDict.
+ */
+MEM_STATIC U32 ZSTD_window_hasExtDict(ZSTD_window_t const window)
+{
+ return window.lowLimit < window.dictLimit;
+}
+
+/*
+ * ZSTD_matchState_dictMode():
+ * Inspects the provided matchState and figures out what dictMode should be
+ * passed to the compressor.
+ */
+MEM_STATIC ZSTD_dictMode_e ZSTD_matchState_dictMode(const ZSTD_matchState_t *ms)
+{
+ return ZSTD_window_hasExtDict(ms->window) ?
+ ZSTD_extDict :
+ ms->dictMatchState != NULL ?
+ (ms->dictMatchState->dedicatedDictSearch ? ZSTD_dedicatedDictSearch : ZSTD_dictMatchState) :
+ ZSTD_noDict;
+}
+
+/*
+ * ZSTD_window_needOverflowCorrection():
+ * Returns non-zero if the indices are getting too large and need overflow
+ * protection.
+ */
+MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window,
+ void const* srcEnd)
+{
+ U32 const curr = (U32)((BYTE const*)srcEnd - window.base);
+ return curr > ZSTD_CURRENT_MAX;
+}
+
+/*
+ * ZSTD_window_correctOverflow():
+ * Reduces the indices to protect from index overflow.
+ * Returns the correction made to the indices, which must be applied to every
+ * stored index.
+ *
+ * The least significant cycleLog bits of the indices must remain the same,
+ * which may be 0. Every index up to maxDist in the past must be valid.
+ * NOTE: (maxDist & cycleMask) must be zero.
+ */
+MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
+ U32 maxDist, void const* src)
+{
+ /* preemptive overflow correction:
+ * 1. correction is large enough:
+ * lowLimit > (3<<29) ==> current > 3<<29 + 1<<windowLog
+ * 1<<windowLog <= newCurrent < 1<<chainLog + 1<<windowLog
+ *
+ * current - newCurrent
+ * > (3<<29 + 1<<windowLog) - (1<<windowLog + 1<<chainLog)
+ * > (3<<29) - (1<<chainLog)
+ * > (3<<29) - (1<<30) (NOTE: chainLog <= 30)
+ * > 1<<29
+ *
+ * 2. (ip+ZSTD_CHUNKSIZE_MAX - cctx->base) doesn't overflow:
+ * After correction, current is less than (1<<chainLog + 1<<windowLog).
+ * In 64-bit mode we are safe, because we have 64-bit ptrdiff_t.
+ * In 32-bit mode we are safe, because (chainLog <= 29), so
+ * ip+ZSTD_CHUNKSIZE_MAX - cctx->base < 1<<32.
+ * 3. (cctx->lowLimit + 1<<windowLog) < 1<<32:
+ * windowLog <= 31 ==> 3<<29 + 1<<windowLog < 7<<29 < 1<<32.
+ */
+ U32 const cycleMask = (1U << cycleLog) - 1;
+ U32 const curr = (U32)((BYTE const*)src - window->base);
+ U32 const currentCycle0 = curr & cycleMask;
+ /* Exclude zero so that newCurrent - maxDist >= 1. */
+ U32 const currentCycle1 = currentCycle0 == 0 ? (1U << cycleLog) : currentCycle0;
+ U32 const newCurrent = currentCycle1 + maxDist;
+ U32 const correction = curr - newCurrent;
+ assert((maxDist & cycleMask) == 0);
+ assert(curr > newCurrent);
+ /* Loose bound, should be around 1<<29 (see above) */
+ assert(correction > 1<<28);
+
+ window->base += correction;
+ window->dictBase += correction;
+ if (window->lowLimit <= correction) window->lowLimit = 1;
+ else window->lowLimit -= correction;
+ if (window->dictLimit <= correction) window->dictLimit = 1;
+ else window->dictLimit -= correction;
+
+ /* Ensure we can still reference the full window. */
+ assert(newCurrent >= maxDist);
+ assert(newCurrent - maxDist >= 1);
+ /* Ensure that lowLimit and dictLimit didn't underflow. */
+ assert(window->lowLimit <= newCurrent);
+ assert(window->dictLimit <= newCurrent);
+
+ DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction,
+ window->lowLimit);
+ return correction;
+}
+
+/*
+ * ZSTD_window_enforceMaxDist():
+ * Updates lowLimit so that:
+ * (srcEnd - base) - lowLimit == maxDist + loadedDictEnd
+ *
+ * It ensures index is valid as long as index >= lowLimit.
+ * This must be called before a block compression call.
+ *
+ * loadedDictEnd is only defined if a dictionary is in use for current compression.
+ * As the name implies, loadedDictEnd represents the index at end of dictionary.
+ * The value lies within context's referential, it can be directly compared to blockEndIdx.
+ *
+ * If loadedDictEndPtr is NULL, no dictionary is in use, and we use loadedDictEnd == 0.
+ * If loadedDictEndPtr is not NULL, we set it to zero after updating lowLimit.
+ * This is because dictionaries are allowed to be referenced fully
+ * as long as the last byte of the dictionary is in the window.
+ * Once input has progressed beyond window size, dictionary cannot be referenced anymore.
+ *
+ * In normal dict mode, the dictionary lies between lowLimit and dictLimit.
+ * In dictMatchState mode, lowLimit and dictLimit are the same,
+ * and the dictionary is below them.
+ * forceWindow and dictMatchState are therefore incompatible.
+ */
+MEM_STATIC void
+ZSTD_window_enforceMaxDist(ZSTD_window_t* window,
+ const void* blockEnd,
+ U32 maxDist,
+ U32* loadedDictEndPtr,
+ const ZSTD_matchState_t** dictMatchStatePtr)
+{
+ U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base);
+ U32 const loadedDictEnd = (loadedDictEndPtr != NULL) ? *loadedDictEndPtr : 0;
+ DEBUGLOG(5, "ZSTD_window_enforceMaxDist: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u",
+ (unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd);
+
+ /* - When there is no dictionary : loadedDictEnd == 0.
+ In which case, the test (blockEndIdx > maxDist) is merely to avoid
+ overflowing next operation `newLowLimit = blockEndIdx - maxDist`.
+ - When there is a standard dictionary :
+ Index referential is copied from the dictionary,
+ which means it starts from 0.
+ In which case, loadedDictEnd == dictSize,
+ and it makes sense to compare `blockEndIdx > maxDist + dictSize`
+ since `blockEndIdx` also starts from zero.
+ - When there is an attached dictionary :
+ loadedDictEnd is expressed within the referential of the context,
+ so it can be directly compared against blockEndIdx.
+ */
+ if (blockEndIdx > maxDist + loadedDictEnd) {
+ U32 const newLowLimit = blockEndIdx - maxDist;
+ if (window->lowLimit < newLowLimit) window->lowLimit = newLowLimit;
+ if (window->dictLimit < window->lowLimit) {
+ DEBUGLOG(5, "Update dictLimit to match lowLimit, from %u to %u",
+ (unsigned)window->dictLimit, (unsigned)window->lowLimit);
+ window->dictLimit = window->lowLimit;
+ }
+ /* On reaching window size, dictionaries are invalidated */
+ if (loadedDictEndPtr) *loadedDictEndPtr = 0;
+ if (dictMatchStatePtr) *dictMatchStatePtr = NULL;
+ }
+}
+
+/* Similar to ZSTD_window_enforceMaxDist(),
+ * but only invalidates dictionary
+ * when input progresses beyond window size.
+ * assumption : loadedDictEndPtr and dictMatchStatePtr are valid (non NULL)
+ * loadedDictEnd uses same referential as window->base
+ * maxDist is the window size */
+MEM_STATIC void
+ZSTD_checkDictValidity(const ZSTD_window_t* window,
+ const void* blockEnd,
+ U32 maxDist,
+ U32* loadedDictEndPtr,
+ const ZSTD_matchState_t** dictMatchStatePtr)
+{
+ assert(loadedDictEndPtr != NULL);
+ assert(dictMatchStatePtr != NULL);
+ { U32 const blockEndIdx = (U32)((BYTE const*)blockEnd - window->base);
+ U32 const loadedDictEnd = *loadedDictEndPtr;
+ DEBUGLOG(5, "ZSTD_checkDictValidity: blockEndIdx=%u, maxDist=%u, loadedDictEnd=%u",
+ (unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd);
+ assert(blockEndIdx >= loadedDictEnd);
+
+ if (blockEndIdx > loadedDictEnd + maxDist) {
+ /* On reaching window size, dictionaries are invalidated.
+ * For simplification, if window size is reached anywhere within next block,
+ * the dictionary is invalidated for the full block.
+ */
+ DEBUGLOG(6, "invalidating dictionary for current block (distance > windowSize)");
+ *loadedDictEndPtr = 0;
+ *dictMatchStatePtr = NULL;
+ } else {
+ if (*loadedDictEndPtr != 0) {
+ DEBUGLOG(6, "dictionary considered valid for current block");
+ } } }
+}
+
+MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {
+ ZSTD_memset(window, 0, sizeof(*window));
+ window->base = (BYTE const*)"";
+ window->dictBase = (BYTE const*)"";
+ window->dictLimit = 1; /* start from 1, so that 1st position is valid */
+ window->lowLimit = 1; /* it ensures first and later CCtx usages compress the same */
+ window->nextSrc = window->base + 1; /* see issue #1241 */
+}
+
+/*
+ * ZSTD_window_update():
+ * Updates the window by appending [src, src + srcSize) to the window.
+ * If it is not contiguous, the current prefix becomes the extDict, and we
+ * forget about the extDict. Handles overlap of the prefix and extDict.
+ * Returns non-zero if the segment is contiguous.
+ */
+MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
+ void const* src, size_t srcSize)
+{
+ BYTE const* const ip = (BYTE const*)src;
+ U32 contiguous = 1;
+ DEBUGLOG(5, "ZSTD_window_update");
+ if (srcSize == 0)
+ return contiguous;
+ assert(window->base != NULL);
+ assert(window->dictBase != NULL);
+ /* Check if blocks follow each other */
+ if (src != window->nextSrc) {
+ /* not contiguous */
+ size_t const distanceFromBase = (size_t)(window->nextSrc - window->base);
+ DEBUGLOG(5, "Non contiguous blocks, new segment starts at %u", window->dictLimit);
+ window->lowLimit = window->dictLimit;
+ assert(distanceFromBase == (size_t)(U32)distanceFromBase); /* should never overflow */
+ window->dictLimit = (U32)distanceFromBase;
+ window->dictBase = window->base;
+ window->base = ip - distanceFromBase;
+ /* ms->nextToUpdate = window->dictLimit; */
+ if (window->dictLimit - window->lowLimit < HASH_READ_SIZE) window->lowLimit = window->dictLimit; /* too small extDict */
+ contiguous = 0;
+ }
+ window->nextSrc = ip + srcSize;
+ /* if input and dictionary overlap : reduce dictionary (area presumed modified by input) */
+ if ( (ip+srcSize > window->dictBase + window->lowLimit)
+ & (ip < window->dictBase + window->dictLimit)) {
+ ptrdiff_t const highInputIdx = (ip + srcSize) - window->dictBase;
+ U32 const lowLimitMax = (highInputIdx > (ptrdiff_t)window->dictLimit) ? window->dictLimit : (U32)highInputIdx;
+ window->lowLimit = lowLimitMax;
+ DEBUGLOG(5, "Overlapping extDict and input : new lowLimit = %u", window->lowLimit);
+ }
+ return contiguous;
+}
+
+/*
+ * Returns the lowest allowed match index. It may either be in the ext-dict or the prefix.
+ */
+MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 curr, unsigned windowLog)
+{
+ U32 const maxDistance = 1U << windowLog;
+ U32 const lowestValid = ms->window.lowLimit;
+ U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
+ U32 const isDictionary = (ms->loadedDictEnd != 0);
+ /* When using a dictionary the entire dictionary is valid if a single byte of the dictionary
+ * is within the window. We invalidate the dictionary (and set loadedDictEnd to 0) when it isn't
+ * valid for the entire block. So this check is sufficient to find the lowest valid match index.
+ */
+ U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
+ return matchLowest;
+}
+
+/*
+ * Returns the lowest allowed match index in the prefix.
+ */
+MEM_STATIC U32 ZSTD_getLowestPrefixIndex(const ZSTD_matchState_t* ms, U32 curr, unsigned windowLog)
+{
+ U32 const maxDistance = 1U << windowLog;
+ U32 const lowestValid = ms->window.dictLimit;
+ U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
+ U32 const isDictionary = (ms->loadedDictEnd != 0);
+ /* When computing the lowest prefix index we need to take the dictionary into account to handle
+ * the edge case where the dictionary and the source are contiguous in memory.
+ */
+ U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
+ return matchLowest;
+}
+
+
+
+/* debug functions */
+#if (DEBUGLEVEL>=2)
+
+MEM_STATIC double ZSTD_fWeight(U32 rawStat)
+{
+ U32 const fp_accuracy = 8;
+ U32 const fp_multiplier = (1 << fp_accuracy);
+ U32 const newStat = rawStat + 1;
+ U32 const hb = ZSTD_highbit32(newStat);
+ U32 const BWeight = hb * fp_multiplier;
+ U32 const FWeight = (newStat << fp_accuracy) >> hb;
+ U32 const weight = BWeight + FWeight;
+ assert(hb + fp_accuracy < 31);
+ return (double)weight / fp_multiplier;
+}
+
+/* display a table content,
+ * listing each element, its frequency, and its predicted bit cost */
+MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max)
+{
+ unsigned u, sum;
+ for (u=0, sum=0; u<=max; u++) sum += table[u];
+ DEBUGLOG(2, "total nb elts: %u", sum);
+ for (u=0; u<=max; u++) {
+ DEBUGLOG(2, "%2u: %5u (%.2f)",
+ u, table[u], ZSTD_fWeight(sum) - ZSTD_fWeight(table[u]) );
+ }
+}
+
+#endif
+
+
+
+/* ===============================================================
+ * Shared internal declarations
+ * These prototypes may be called from sources not in lib/compress
+ * =============================================================== */
+
+/* ZSTD_loadCEntropy() :
+ * dict : must point at beginning of a valid zstd dictionary.
+ * return : size of dictionary header (size of magic number + dict ID + entropy tables)
+ * assumptions : magic number supposed already checked
+ * and dictSize >= 8 */
+size_t ZSTD_loadCEntropy(ZSTD_compressedBlockState_t* bs, void* workspace,
+ const void* const dict, size_t dictSize);
+
+void ZSTD_reset_compressedBlockState(ZSTD_compressedBlockState_t* bs);
+
+/* ==============================================================
+ * Private declarations
+ * These prototypes shall only be called from within lib/compress
+ * ============================================================== */
+
+/* ZSTD_getCParamsFromCCtxParams() :
+ * cParams are built depending on compressionLevel, src size hints,
+ * LDM and manually set compression parameters.
+ * Note: srcSizeHint == 0 means 0!
+ */
+ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
+ const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize, ZSTD_cParamMode_e mode);
+
+/*! ZSTD_initCStream_internal() :
+ * Private use only. Init streaming operation.
+ * expects params to be valid.
+ * must receive dict, or cdict, or none, but not both.
+ * @return : 0, or an error code */
+size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs,
+ const void* dict, size_t dictSize,
+ const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params, unsigned long long pledgedSrcSize);
+
+void ZSTD_resetSeqStore(seqStore_t* ssPtr);
+
+/*! ZSTD_getCParamsFromCDict() :
+ * as the name implies */
+ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict);
+
+/* ZSTD_compressBegin_advanced_internal() :
+ * Private use only. To be called from zstdmt_compress.c. */
+size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx,
+ const void* dict, size_t dictSize,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_dictTableLoadMethod_e dtlm,
+ const ZSTD_CDict* cdict,
+ const ZSTD_CCtx_params* params,
+ unsigned long long pledgedSrcSize);
+
+/* ZSTD_compress_advanced_internal() :
+ * Private use only. To be called from zstdmt_compress.c. */
+size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict,size_t dictSize,
+ const ZSTD_CCtx_params* params);
+
+
+/* ZSTD_writeLastEmptyBlock() :
+ * output an empty Block with end-of-frame mark to complete a frame
+ * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h))
+ * or an error code if `dstCapacity` is too small (<ZSTD_blockHeaderSize)
+ */
+size_t ZSTD_writeLastEmptyBlock(void* dst, size_t dstCapacity);
+
+
+/* ZSTD_referenceExternalSequences() :
+ * Must be called before starting a compression operation.
+ * seqs must parse a prefix of the source.
+ * This cannot be used when long range matching is enabled.
+ * Zstd will use these sequences, and pass the literals to a secondary block
+ * compressor.
+ * @return : An error code on failure.
+ * NOTE: seqs are not verified! Invalid sequences can cause out-of-bounds memory
+ * access and data corruption.
+ */
+size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSeq);
+
+/* ZSTD_cycleLog() :
+ * condition for correct operation : hashLog > 1 */
+U32 ZSTD_cycleLog(U32 hashLog, ZSTD_strategy strat);
+
+/* ZSTD_CCtx_trace() :
+ * Trace the end of a compression call.
+ */
+void ZSTD_CCtx_trace(ZSTD_CCtx* cctx, size_t extraCSize);
+
+#endif /* ZSTD_COMPRESS_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+ /*-*************************************
+ * Dependencies
+ ***************************************/
+#include "zstd_compress_literals.h"
+
+size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ BYTE* const ostart = (BYTE*)dst;
+ U32 const flSize = 1 + (srcSize>31) + (srcSize>4095);
+
+ RETURN_ERROR_IF(srcSize + flSize > dstCapacity, dstSize_tooSmall, "");
+
+ switch(flSize)
+ {
+ case 1: /* 2 - 1 - 5 */
+ ostart[0] = (BYTE)((U32)set_basic + (srcSize<<3));
+ break;
+ case 2: /* 2 - 2 - 12 */
+ MEM_writeLE16(ostart, (U16)((U32)set_basic + (1<<2) + (srcSize<<4)));
+ break;
+ case 3: /* 2 - 2 - 20 */
+ MEM_writeLE32(ostart, (U32)((U32)set_basic + (3<<2) + (srcSize<<4)));
+ break;
+ default: /* not necessary : flSize is {1,2,3} */
+ assert(0);
+ }
+
+ ZSTD_memcpy(ostart + flSize, src, srcSize);
+ DEBUGLOG(5, "Raw literals: %u -> %u", (U32)srcSize, (U32)(srcSize + flSize));
+ return srcSize + flSize;
+}
+
+size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ BYTE* const ostart = (BYTE*)dst;
+ U32 const flSize = 1 + (srcSize>31) + (srcSize>4095);
+
+ (void)dstCapacity; /* dstCapacity already guaranteed to be >=4, hence large enough */
+
+ switch(flSize)
+ {
+ case 1: /* 2 - 1 - 5 */
+ ostart[0] = (BYTE)((U32)set_rle + (srcSize<<3));
+ break;
+ case 2: /* 2 - 2 - 12 */
+ MEM_writeLE16(ostart, (U16)((U32)set_rle + (1<<2) + (srcSize<<4)));
+ break;
+ case 3: /* 2 - 2 - 20 */
+ MEM_writeLE32(ostart, (U32)((U32)set_rle + (3<<2) + (srcSize<<4)));
+ break;
+ default: /* not necessary : flSize is {1,2,3} */
+ assert(0);
+ }
+
+ ostart[flSize] = *(const BYTE*)src;
+ DEBUGLOG(5, "RLE literals: %u -> %u", (U32)srcSize, (U32)flSize + 1);
+ return flSize+1;
+}
+
+size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
+ ZSTD_hufCTables_t* nextHuf,
+ ZSTD_strategy strategy, int disableLiteralCompression,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ void* entropyWorkspace, size_t entropyWorkspaceSize,
+ const int bmi2)
+{
+ size_t const minGain = ZSTD_minGain(srcSize, strategy);
+ size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB);
+ BYTE* const ostart = (BYTE*)dst;
+ U32 singleStream = srcSize < 256;
+ symbolEncodingType_e hType = set_compressed;
+ size_t cLitSize;
+
+ DEBUGLOG(5,"ZSTD_compressLiterals (disableLiteralCompression=%i srcSize=%u)",
+ disableLiteralCompression, (U32)srcSize);
+
+ /* Prepare nextEntropy assuming reusing the existing table */
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+
+ if (disableLiteralCompression)
+ return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
+
+ /* small ? don't even attempt compression (speed opt) */
+# define COMPRESS_LITERALS_SIZE_MIN 63
+ { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
+ if (srcSize <= minLitSize) return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
+ }
+
+ RETURN_ERROR_IF(dstCapacity < lhSize+1, dstSize_tooSmall, "not enough space for compression");
+ { HUF_repeat repeat = prevHuf->repeatMode;
+ int const preferRepeat = strategy < ZSTD_lazy ? srcSize <= 1024 : 0;
+ if (repeat == HUF_repeat_valid && lhSize == 3) singleStream = 1;
+ cLitSize = singleStream ?
+ HUF_compress1X_repeat(
+ ostart+lhSize, dstCapacity-lhSize, src, srcSize,
+ HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
+ (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
+ HUF_compress4X_repeat(
+ ostart+lhSize, dstCapacity-lhSize, src, srcSize,
+ HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
+ (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
+ if (repeat != HUF_repeat_none) {
+ /* reused the existing table */
+ DEBUGLOG(5, "Reusing previous huffman table");
+ hType = set_repeat;
+ }
+ }
+
+ if ((cLitSize==0) | (cLitSize >= srcSize - minGain) | ERR_isError(cLitSize)) {
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
+ }
+ if (cLitSize==1) {
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ return ZSTD_compressRleLiteralsBlock(dst, dstCapacity, src, srcSize);
+ }
+
+ if (hType == set_compressed) {
+ /* using a newly constructed table */
+ nextHuf->repeatMode = HUF_repeat_check;
+ }
+
+ /* Build header */
+ switch(lhSize)
+ {
+ case 3: /* 2 - 2 - 10 - 10 */
+ { U32 const lhc = hType + ((!singleStream) << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<14);
+ MEM_writeLE24(ostart, lhc);
+ break;
+ }
+ case 4: /* 2 - 2 - 14 - 14 */
+ { U32 const lhc = hType + (2 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<18);
+ MEM_writeLE32(ostart, lhc);
+ break;
+ }
+ case 5: /* 2 - 2 - 18 - 18 */
+ { U32 const lhc = hType + (3 << 2) + ((U32)srcSize<<4) + ((U32)cLitSize<<22);
+ MEM_writeLE32(ostart, lhc);
+ ostart[4] = (BYTE)(cLitSize >> 10);
+ break;
+ }
+ default: /* not possible : lhSize is {3,4,5} */
+ assert(0);
+ }
+ DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)srcSize, (U32)(lhSize+cLitSize));
+ return lhSize+cLitSize;
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMPRESS_LITERALS_H
+#define ZSTD_COMPRESS_LITERALS_H
+
+#include "zstd_compress_internal.h" /* ZSTD_hufCTables_t, ZSTD_minGain() */
+
+
+size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+
+size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+
+size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
+ ZSTD_hufCTables_t* nextHuf,
+ ZSTD_strategy strategy, int disableLiteralCompression,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ void* entropyWorkspace, size_t entropyWorkspaceSize,
+ const int bmi2);
+
+#endif /* ZSTD_COMPRESS_LITERALS_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+ /*-*************************************
+ * Dependencies
+ ***************************************/
+#include "zstd_compress_sequences.h"
+
+/*
+ * -log2(x / 256) lookup table for x in [0, 256).
+ * If x == 0: Return 0
+ * Else: Return floor(-log2(x / 256) * 256)
+ */
+static unsigned const kInverseProbabilityLog256[256] = {
+ 0, 2048, 1792, 1642, 1536, 1453, 1386, 1329, 1280, 1236, 1197, 1162,
+ 1130, 1100, 1073, 1047, 1024, 1001, 980, 960, 941, 923, 906, 889,
+ 874, 859, 844, 830, 817, 804, 791, 779, 768, 756, 745, 734,
+ 724, 714, 704, 694, 685, 676, 667, 658, 650, 642, 633, 626,
+ 618, 610, 603, 595, 588, 581, 574, 567, 561, 554, 548, 542,
+ 535, 529, 523, 517, 512, 506, 500, 495, 489, 484, 478, 473,
+ 468, 463, 458, 453, 448, 443, 438, 434, 429, 424, 420, 415,
+ 411, 407, 402, 398, 394, 390, 386, 382, 377, 373, 370, 366,
+ 362, 358, 354, 350, 347, 343, 339, 336, 332, 329, 325, 322,
+ 318, 315, 311, 308, 305, 302, 298, 295, 292, 289, 286, 282,
+ 279, 276, 273, 270, 267, 264, 261, 258, 256, 253, 250, 247,
+ 244, 241, 239, 236, 233, 230, 228, 225, 222, 220, 217, 215,
+ 212, 209, 207, 204, 202, 199, 197, 194, 192, 190, 187, 185,
+ 182, 180, 178, 175, 173, 171, 168, 166, 164, 162, 159, 157,
+ 155, 153, 151, 149, 146, 144, 142, 140, 138, 136, 134, 132,
+ 130, 128, 126, 123, 121, 119, 117, 115, 114, 112, 110, 108,
+ 106, 104, 102, 100, 98, 96, 94, 93, 91, 89, 87, 85,
+ 83, 82, 80, 78, 76, 74, 73, 71, 69, 67, 66, 64,
+ 62, 61, 59, 57, 55, 54, 52, 50, 49, 47, 46, 44,
+ 42, 41, 39, 37, 36, 34, 33, 31, 30, 28, 26, 25,
+ 23, 22, 20, 19, 17, 16, 14, 13, 11, 10, 8, 7,
+ 5, 4, 2, 1,
+};
+
+static unsigned ZSTD_getFSEMaxSymbolValue(FSE_CTable const* ctable) {
+ void const* ptr = ctable;
+ U16 const* u16ptr = (U16 const*)ptr;
+ U32 const maxSymbolValue = MEM_read16(u16ptr + 1);
+ return maxSymbolValue;
+}
+
+/*
+ * Returns true if we should use ncount=-1 else we should
+ * use ncount=1 for low probability symbols instead.
+ */
+static unsigned ZSTD_useLowProbCount(size_t const nbSeq)
+{
+ /* Heuristic: This should cover most blocks <= 16K and
+ * start to fade out after 16K to about 32K depending on
+ * comprssibility.
+ */
+ return nbSeq >= 2048;
+}
+
+/*
+ * Returns the cost in bytes of encoding the normalized count header.
+ * Returns an error if any of the helper functions return an error.
+ */
+static size_t ZSTD_NCountCost(unsigned const* count, unsigned const max,
+ size_t const nbSeq, unsigned const FSELog)
+{
+ BYTE wksp[FSE_NCOUNTBOUND];
+ S16 norm[MaxSeq + 1];
+ const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
+ FORWARD_IF_ERROR(FSE_normalizeCount(norm, tableLog, count, nbSeq, max, ZSTD_useLowProbCount(nbSeq)), "");
+ return FSE_writeNCount(wksp, sizeof(wksp), norm, max, tableLog);
+}
+
+/*
+ * Returns the cost in bits of encoding the distribution described by count
+ * using the entropy bound.
+ */
+static size_t ZSTD_entropyCost(unsigned const* count, unsigned const max, size_t const total)
+{
+ unsigned cost = 0;
+ unsigned s;
+ for (s = 0; s <= max; ++s) {
+ unsigned norm = (unsigned)((256 * count[s]) / total);
+ if (count[s] != 0 && norm == 0)
+ norm = 1;
+ assert(count[s] < total);
+ cost += count[s] * kInverseProbabilityLog256[norm];
+ }
+ return cost >> 8;
+}
+
+/*
+ * Returns the cost in bits of encoding the distribution in count using ctable.
+ * Returns an error if ctable cannot represent all the symbols in count.
+ */
+size_t ZSTD_fseBitCost(
+ FSE_CTable const* ctable,
+ unsigned const* count,
+ unsigned const max)
+{
+ unsigned const kAccuracyLog = 8;
+ size_t cost = 0;
+ unsigned s;
+ FSE_CState_t cstate;
+ FSE_initCState(&cstate, ctable);
+ if (ZSTD_getFSEMaxSymbolValue(ctable) < max) {
+ DEBUGLOG(5, "Repeat FSE_CTable has maxSymbolValue %u < %u",
+ ZSTD_getFSEMaxSymbolValue(ctable), max);
+ return ERROR(GENERIC);
+ }
+ for (s = 0; s <= max; ++s) {
+ unsigned const tableLog = cstate.stateLog;
+ unsigned const badCost = (tableLog + 1) << kAccuracyLog;
+ unsigned const bitCost = FSE_bitCost(cstate.symbolTT, tableLog, s, kAccuracyLog);
+ if (count[s] == 0)
+ continue;
+ if (bitCost >= badCost) {
+ DEBUGLOG(5, "Repeat FSE_CTable has Prob[%u] == 0", s);
+ return ERROR(GENERIC);
+ }
+ cost += (size_t)count[s] * bitCost;
+ }
+ return cost >> kAccuracyLog;
+}
+
+/*
+ * Returns the cost in bits of encoding the distribution in count using the
+ * table described by norm. The max symbol support by norm is assumed >= max.
+ * norm must be valid for every symbol with non-zero probability in count.
+ */
+size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
+ unsigned const* count, unsigned const max)
+{
+ unsigned const shift = 8 - accuracyLog;
+ size_t cost = 0;
+ unsigned s;
+ assert(accuracyLog <= 8);
+ for (s = 0; s <= max; ++s) {
+ unsigned const normAcc = (norm[s] != -1) ? (unsigned)norm[s] : 1;
+ unsigned const norm256 = normAcc << shift;
+ assert(norm256 > 0);
+ assert(norm256 < 256);
+ cost += count[s] * kInverseProbabilityLog256[norm256];
+ }
+ return cost >> 8;
+}
+
+symbolEncodingType_e
+ZSTD_selectEncodingType(
+ FSE_repeat* repeatMode, unsigned const* count, unsigned const max,
+ size_t const mostFrequent, size_t nbSeq, unsigned const FSELog,
+ FSE_CTable const* prevCTable,
+ short const* defaultNorm, U32 defaultNormLog,
+ ZSTD_defaultPolicy_e const isDefaultAllowed,
+ ZSTD_strategy const strategy)
+{
+ ZSTD_STATIC_ASSERT(ZSTD_defaultDisallowed == 0 && ZSTD_defaultAllowed != 0);
+ if (mostFrequent == nbSeq) {
+ *repeatMode = FSE_repeat_none;
+ if (isDefaultAllowed && nbSeq <= 2) {
+ /* Prefer set_basic over set_rle when there are 2 or less symbols,
+ * since RLE uses 1 byte, but set_basic uses 5-6 bits per symbol.
+ * If basic encoding isn't possible, always choose RLE.
+ */
+ DEBUGLOG(5, "Selected set_basic");
+ return set_basic;
+ }
+ DEBUGLOG(5, "Selected set_rle");
+ return set_rle;
+ }
+ if (strategy < ZSTD_lazy) {
+ if (isDefaultAllowed) {
+ size_t const staticFse_nbSeq_max = 1000;
+ size_t const mult = 10 - strategy;
+ size_t const baseLog = 3;
+ size_t const dynamicFse_nbSeq_min = (((size_t)1 << defaultNormLog) * mult) >> baseLog; /* 28-36 for offset, 56-72 for lengths */
+ assert(defaultNormLog >= 5 && defaultNormLog <= 6); /* xx_DEFAULTNORMLOG */
+ assert(mult <= 9 && mult >= 7);
+ if ( (*repeatMode == FSE_repeat_valid)
+ && (nbSeq < staticFse_nbSeq_max) ) {
+ DEBUGLOG(5, "Selected set_repeat");
+ return set_repeat;
+ }
+ if ( (nbSeq < dynamicFse_nbSeq_min)
+ || (mostFrequent < (nbSeq >> (defaultNormLog-1))) ) {
+ DEBUGLOG(5, "Selected set_basic");
+ /* The format allows default tables to be repeated, but it isn't useful.
+ * When using simple heuristics to select encoding type, we don't want
+ * to confuse these tables with dictionaries. When running more careful
+ * analysis, we don't need to waste time checking both repeating tables
+ * and default tables.
+ */
+ *repeatMode = FSE_repeat_none;
+ return set_basic;
+ }
+ }
+ } else {
+ size_t const basicCost = isDefaultAllowed ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, count, max) : ERROR(GENERIC);
+ size_t const repeatCost = *repeatMode != FSE_repeat_none ? ZSTD_fseBitCost(prevCTable, count, max) : ERROR(GENERIC);
+ size_t const NCountCost = ZSTD_NCountCost(count, max, nbSeq, FSELog);
+ size_t const compressedCost = (NCountCost << 3) + ZSTD_entropyCost(count, max, nbSeq);
+
+ if (isDefaultAllowed) {
+ assert(!ZSTD_isError(basicCost));
+ assert(!(*repeatMode == FSE_repeat_valid && ZSTD_isError(repeatCost)));
+ }
+ assert(!ZSTD_isError(NCountCost));
+ assert(compressedCost < ERROR(maxCode));
+ DEBUGLOG(5, "Estimated bit costs: basic=%u\trepeat=%u\tcompressed=%u",
+ (unsigned)basicCost, (unsigned)repeatCost, (unsigned)compressedCost);
+ if (basicCost <= repeatCost && basicCost <= compressedCost) {
+ DEBUGLOG(5, "Selected set_basic");
+ assert(isDefaultAllowed);
+ *repeatMode = FSE_repeat_none;
+ return set_basic;
+ }
+ if (repeatCost <= compressedCost) {
+ DEBUGLOG(5, "Selected set_repeat");
+ assert(!ZSTD_isError(repeatCost));
+ return set_repeat;
+ }
+ assert(compressedCost < basicCost && compressedCost < repeatCost);
+ }
+ DEBUGLOG(5, "Selected set_compressed");
+ *repeatMode = FSE_repeat_check;
+ return set_compressed;
+}
+
+typedef struct {
+ S16 norm[MaxSeq + 1];
+ U32 wksp[FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(MaxSeq, MaxFSELog)];
+} ZSTD_BuildCTableWksp;
+
+size_t
+ZSTD_buildCTable(void* dst, size_t dstCapacity,
+ FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
+ unsigned* count, U32 max,
+ const BYTE* codeTable, size_t nbSeq,
+ const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
+ const FSE_CTable* prevCTable, size_t prevCTableSize,
+ void* entropyWorkspace, size_t entropyWorkspaceSize)
+{
+ BYTE* op = (BYTE*)dst;
+ const BYTE* const oend = op + dstCapacity;
+ DEBUGLOG(6, "ZSTD_buildCTable (dstCapacity=%u)", (unsigned)dstCapacity);
+
+ switch (type) {
+ case set_rle:
+ FORWARD_IF_ERROR(FSE_buildCTable_rle(nextCTable, (BYTE)max), "");
+ RETURN_ERROR_IF(dstCapacity==0, dstSize_tooSmall, "not enough space");
+ *op = codeTable[0];
+ return 1;
+ case set_repeat:
+ ZSTD_memcpy(nextCTable, prevCTable, prevCTableSize);
+ return 0;
+ case set_basic:
+ FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, defaultNorm, defaultMax, defaultNormLog, entropyWorkspace, entropyWorkspaceSize), ""); /* note : could be pre-calculated */
+ return 0;
+ case set_compressed: {
+ ZSTD_BuildCTableWksp* wksp = (ZSTD_BuildCTableWksp*)entropyWorkspace;
+ size_t nbSeq_1 = nbSeq;
+ const U32 tableLog = FSE_optimalTableLog(FSELog, nbSeq, max);
+ if (count[codeTable[nbSeq-1]] > 1) {
+ count[codeTable[nbSeq-1]]--;
+ nbSeq_1--;
+ }
+ assert(nbSeq_1 > 1);
+ assert(entropyWorkspaceSize >= sizeof(ZSTD_BuildCTableWksp));
+ (void)entropyWorkspaceSize;
+ FORWARD_IF_ERROR(FSE_normalizeCount(wksp->norm, tableLog, count, nbSeq_1, max, ZSTD_useLowProbCount(nbSeq_1)), "");
+ { size_t const NCountSize = FSE_writeNCount(op, oend - op, wksp->norm, max, tableLog); /* overflow protected */
+ FORWARD_IF_ERROR(NCountSize, "FSE_writeNCount failed");
+ FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, wksp->norm, max, tableLog, wksp->wksp, sizeof(wksp->wksp)), "");
+ return NCountSize;
+ }
+ }
+ default: assert(0); RETURN_ERROR(GENERIC, "impossible to reach");
+ }
+}
+
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_encodeSequences_body(
+ void* dst, size_t dstCapacity,
+ FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
+ FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
+ FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
+ seqDef const* sequences, size_t nbSeq, int longOffsets)
+{
+ BIT_CStream_t blockStream;
+ FSE_CState_t stateMatchLength;
+ FSE_CState_t stateOffsetBits;
+ FSE_CState_t stateLitLength;
+
+ RETURN_ERROR_IF(
+ ERR_isError(BIT_initCStream(&blockStream, dst, dstCapacity)),
+ dstSize_tooSmall, "not enough space remaining");
+ DEBUGLOG(6, "available space for bitstream : %i (dstCapacity=%u)",
+ (int)(blockStream.endPtr - blockStream.startPtr),
+ (unsigned)dstCapacity);
+
+ /* first symbols */
+ FSE_initCState2(&stateMatchLength, CTable_MatchLength, mlCodeTable[nbSeq-1]);
+ FSE_initCState2(&stateOffsetBits, CTable_OffsetBits, ofCodeTable[nbSeq-1]);
+ FSE_initCState2(&stateLitLength, CTable_LitLength, llCodeTable[nbSeq-1]);
+ BIT_addBits(&blockStream, sequences[nbSeq-1].litLength, LL_bits[llCodeTable[nbSeq-1]]);
+ if (MEM_32bits()) BIT_flushBits(&blockStream);
+ BIT_addBits(&blockStream, sequences[nbSeq-1].matchLength, ML_bits[mlCodeTable[nbSeq-1]]);
+ if (MEM_32bits()) BIT_flushBits(&blockStream);
+ if (longOffsets) {
+ U32 const ofBits = ofCodeTable[nbSeq-1];
+ unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
+ if (extraBits) {
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offset, extraBits);
+ BIT_flushBits(&blockStream);
+ }
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offset >> extraBits,
+ ofBits - extraBits);
+ } else {
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offset, ofCodeTable[nbSeq-1]);
+ }
+ BIT_flushBits(&blockStream);
+
+ { size_t n;
+ for (n=nbSeq-2 ; n<nbSeq ; n--) { /* intentional underflow */
+ BYTE const llCode = llCodeTable[n];
+ BYTE const ofCode = ofCodeTable[n];
+ BYTE const mlCode = mlCodeTable[n];
+ U32 const llBits = LL_bits[llCode];
+ U32 const ofBits = ofCode;
+ U32 const mlBits = ML_bits[mlCode];
+ DEBUGLOG(6, "encoding: litlen:%2u - matchlen:%2u - offCode:%7u",
+ (unsigned)sequences[n].litLength,
+ (unsigned)sequences[n].matchLength + MINMATCH,
+ (unsigned)sequences[n].offset);
+ /* 32b*/ /* 64b*/
+ /* (7)*/ /* (7)*/
+ FSE_encodeSymbol(&blockStream, &stateOffsetBits, ofCode); /* 15 */ /* 15 */
+ FSE_encodeSymbol(&blockStream, &stateMatchLength, mlCode); /* 24 */ /* 24 */
+ if (MEM_32bits()) BIT_flushBits(&blockStream); /* (7)*/
+ FSE_encodeSymbol(&blockStream, &stateLitLength, llCode); /* 16 */ /* 33 */
+ if (MEM_32bits() || (ofBits+mlBits+llBits >= 64-7-(LLFSELog+MLFSELog+OffFSELog)))
+ BIT_flushBits(&blockStream); /* (7)*/
+ BIT_addBits(&blockStream, sequences[n].litLength, llBits);
+ if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream);
+ BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
+ if (MEM_32bits() || (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream);
+ if (longOffsets) {
+ unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
+ if (extraBits) {
+ BIT_addBits(&blockStream, sequences[n].offset, extraBits);
+ BIT_flushBits(&blockStream); /* (7)*/
+ }
+ BIT_addBits(&blockStream, sequences[n].offset >> extraBits,
+ ofBits - extraBits); /* 31 */
+ } else {
+ BIT_addBits(&blockStream, sequences[n].offset, ofBits); /* 31 */
+ }
+ BIT_flushBits(&blockStream); /* (7)*/
+ DEBUGLOG(7, "remaining space : %i", (int)(blockStream.endPtr - blockStream.ptr));
+ } }
+
+ DEBUGLOG(6, "ZSTD_encodeSequences: flushing ML state with %u bits", stateMatchLength.stateLog);
+ FSE_flushCState(&blockStream, &stateMatchLength);
+ DEBUGLOG(6, "ZSTD_encodeSequences: flushing Off state with %u bits", stateOffsetBits.stateLog);
+ FSE_flushCState(&blockStream, &stateOffsetBits);
+ DEBUGLOG(6, "ZSTD_encodeSequences: flushing LL state with %u bits", stateLitLength.stateLog);
+ FSE_flushCState(&blockStream, &stateLitLength);
+
+ { size_t const streamSize = BIT_closeCStream(&blockStream);
+ RETURN_ERROR_IF(streamSize==0, dstSize_tooSmall, "not enough space");
+ return streamSize;
+ }
+}
+
+static size_t
+ZSTD_encodeSequences_default(
+ void* dst, size_t dstCapacity,
+ FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
+ FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
+ FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
+ seqDef const* sequences, size_t nbSeq, int longOffsets)
+{
+ return ZSTD_encodeSequences_body(dst, dstCapacity,
+ CTable_MatchLength, mlCodeTable,
+ CTable_OffsetBits, ofCodeTable,
+ CTable_LitLength, llCodeTable,
+ sequences, nbSeq, longOffsets);
+}
+
+
+#if DYNAMIC_BMI2
+
+static TARGET_ATTRIBUTE("bmi2") size_t
+ZSTD_encodeSequences_bmi2(
+ void* dst, size_t dstCapacity,
+ FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
+ FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
+ FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
+ seqDef const* sequences, size_t nbSeq, int longOffsets)
+{
+ return ZSTD_encodeSequences_body(dst, dstCapacity,
+ CTable_MatchLength, mlCodeTable,
+ CTable_OffsetBits, ofCodeTable,
+ CTable_LitLength, llCodeTable,
+ sequences, nbSeq, longOffsets);
+}
+
+#endif
+
+size_t ZSTD_encodeSequences(
+ void* dst, size_t dstCapacity,
+ FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
+ FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
+ FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
+ seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2)
+{
+ DEBUGLOG(5, "ZSTD_encodeSequences: dstCapacity = %u", (unsigned)dstCapacity);
+#if DYNAMIC_BMI2
+ if (bmi2) {
+ return ZSTD_encodeSequences_bmi2(dst, dstCapacity,
+ CTable_MatchLength, mlCodeTable,
+ CTable_OffsetBits, ofCodeTable,
+ CTable_LitLength, llCodeTable,
+ sequences, nbSeq, longOffsets);
+ }
+#endif
+ (void)bmi2;
+ return ZSTD_encodeSequences_default(dst, dstCapacity,
+ CTable_MatchLength, mlCodeTable,
+ CTable_OffsetBits, ofCodeTable,
+ CTable_LitLength, llCodeTable,
+ sequences, nbSeq, longOffsets);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMPRESS_SEQUENCES_H
+#define ZSTD_COMPRESS_SEQUENCES_H
+
+#include "../common/fse.h" /* FSE_repeat, FSE_CTable */
+#include "../common/zstd_internal.h" /* symbolEncodingType_e, ZSTD_strategy */
+
+typedef enum {
+ ZSTD_defaultDisallowed = 0,
+ ZSTD_defaultAllowed = 1
+} ZSTD_defaultPolicy_e;
+
+symbolEncodingType_e
+ZSTD_selectEncodingType(
+ FSE_repeat* repeatMode, unsigned const* count, unsigned const max,
+ size_t const mostFrequent, size_t nbSeq, unsigned const FSELog,
+ FSE_CTable const* prevCTable,
+ short const* defaultNorm, U32 defaultNormLog,
+ ZSTD_defaultPolicy_e const isDefaultAllowed,
+ ZSTD_strategy const strategy);
+
+size_t
+ZSTD_buildCTable(void* dst, size_t dstCapacity,
+ FSE_CTable* nextCTable, U32 FSELog, symbolEncodingType_e type,
+ unsigned* count, U32 max,
+ const BYTE* codeTable, size_t nbSeq,
+ const S16* defaultNorm, U32 defaultNormLog, U32 defaultMax,
+ const FSE_CTable* prevCTable, size_t prevCTableSize,
+ void* entropyWorkspace, size_t entropyWorkspaceSize);
+
+size_t ZSTD_encodeSequences(
+ void* dst, size_t dstCapacity,
+ FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
+ FSE_CTable const* CTable_OffsetBits, BYTE const* ofCodeTable,
+ FSE_CTable const* CTable_LitLength, BYTE const* llCodeTable,
+ seqDef const* sequences, size_t nbSeq, int longOffsets, int bmi2);
+
+size_t ZSTD_fseBitCost(
+ FSE_CTable const* ctable,
+ unsigned const* count,
+ unsigned const max);
+
+size_t ZSTD_crossEntropyCost(short const* norm, unsigned accuracyLog,
+ unsigned const* count, unsigned const max);
+#endif /* ZSTD_COMPRESS_SEQUENCES_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+ /*-*************************************
+ * Dependencies
+ ***************************************/
+#include "zstd_compress_superblock.h"
+
+#include "../common/zstd_internal.h" /* ZSTD_getSequenceLength */
+#include "hist.h" /* HIST_countFast_wksp */
+#include "zstd_compress_internal.h"
+#include "zstd_compress_sequences.h"
+#include "zstd_compress_literals.h"
+
+/*-*************************************
+* Superblock entropy buffer structs
+***************************************/
+/* ZSTD_hufCTablesMetadata_t :
+ * Stores Literals Block Type for a super-block in hType, and
+ * huffman tree description in hufDesBuffer.
+ * hufDesSize refers to the size of huffman tree description in bytes.
+ * This metadata is populated in ZSTD_buildSuperBlockEntropy_literal() */
+typedef struct {
+ symbolEncodingType_e hType;
+ BYTE hufDesBuffer[ZSTD_MAX_HUF_HEADER_SIZE];
+ size_t hufDesSize;
+} ZSTD_hufCTablesMetadata_t;
+
+/* ZSTD_fseCTablesMetadata_t :
+ * Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and
+ * fse tables in fseTablesBuffer.
+ * fseTablesSize refers to the size of fse tables in bytes.
+ * This metadata is populated in ZSTD_buildSuperBlockEntropy_sequences() */
+typedef struct {
+ symbolEncodingType_e llType;
+ symbolEncodingType_e ofType;
+ symbolEncodingType_e mlType;
+ BYTE fseTablesBuffer[ZSTD_MAX_FSE_HEADERS_SIZE];
+ size_t fseTablesSize;
+ size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_compressSubBlock_sequences() */
+} ZSTD_fseCTablesMetadata_t;
+
+typedef struct {
+ ZSTD_hufCTablesMetadata_t hufMetadata;
+ ZSTD_fseCTablesMetadata_t fseMetadata;
+} ZSTD_entropyCTablesMetadata_t;
+
+
+/* ZSTD_buildSuperBlockEntropy_literal() :
+ * Builds entropy for the super-block literals.
+ * Stores literals block type (raw, rle, compressed, repeat) and
+ * huffman description table to hufMetadata.
+ * @return : size of huffman description table or error code */
+static size_t ZSTD_buildSuperBlockEntropy_literal(void* const src, size_t srcSize,
+ const ZSTD_hufCTables_t* prevHuf,
+ ZSTD_hufCTables_t* nextHuf,
+ ZSTD_hufCTablesMetadata_t* hufMetadata,
+ const int disableLiteralsCompression,
+ void* workspace, size_t wkspSize)
+{
+ BYTE* const wkspStart = (BYTE*)workspace;
+ BYTE* const wkspEnd = wkspStart + wkspSize;
+ BYTE* const countWkspStart = wkspStart;
+ unsigned* const countWksp = (unsigned*)workspace;
+ const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned);
+ BYTE* const nodeWksp = countWkspStart + countWkspSize;
+ const size_t nodeWkspSize = wkspEnd-nodeWksp;
+ unsigned maxSymbolValue = 255;
+ unsigned huffLog = HUF_TABLELOG_DEFAULT;
+ HUF_repeat repeat = prevHuf->repeatMode;
+
+ DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_literal (srcSize=%zu)", srcSize);
+
+ /* Prepare nextEntropy assuming reusing the existing table */
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+
+ if (disableLiteralsCompression) {
+ DEBUGLOG(5, "set_basic - disabled");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+
+ /* small ? don't even attempt compression (speed opt) */
+# define COMPRESS_LITERALS_SIZE_MIN 63
+ { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
+ if (srcSize <= minLitSize) {
+ DEBUGLOG(5, "set_basic - too small");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ }
+
+ /* Scan input and build symbol stats */
+ { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize);
+ FORWARD_IF_ERROR(largest, "HIST_count_wksp failed");
+ if (largest == srcSize) {
+ DEBUGLOG(5, "set_rle");
+ hufMetadata->hType = set_rle;
+ return 0;
+ }
+ if (largest <= (srcSize >> 7)+4) {
+ DEBUGLOG(5, "set_basic - no gain");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ }
+
+ /* Validate the previous Huffman table */
+ if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) {
+ repeat = HUF_repeat_none;
+ }
+
+ /* Build Huffman Tree */
+ ZSTD_memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable));
+ huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
+ { size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp,
+ maxSymbolValue, huffLog,
+ nodeWksp, nodeWkspSize);
+ FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp");
+ huffLog = (U32)maxBits;
+ { /* Build and write the CTable */
+ size_t const newCSize = HUF_estimateCompressedSize(
+ (HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue);
+ size_t const hSize = HUF_writeCTable_wksp(
+ hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer),
+ (HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog,
+ nodeWksp, nodeWkspSize);
+ /* Check against repeating the previous CTable */
+ if (repeat != HUF_repeat_none) {
+ size_t const oldCSize = HUF_estimateCompressedSize(
+ (HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue);
+ if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) {
+ DEBUGLOG(5, "set_repeat - smaller");
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ hufMetadata->hType = set_repeat;
+ return 0;
+ }
+ }
+ if (newCSize + hSize >= srcSize) {
+ DEBUGLOG(5, "set_basic - no gains");
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize);
+ hufMetadata->hType = set_compressed;
+ nextHuf->repeatMode = HUF_repeat_check;
+ return hSize;
+ }
+ }
+}
+
+/* ZSTD_buildSuperBlockEntropy_sequences() :
+ * Builds entropy for the super-block sequences.
+ * Stores symbol compression modes and fse table to fseMetadata.
+ * @return : size of fse tables or error code */
+static size_t ZSTD_buildSuperBlockEntropy_sequences(seqStore_t* seqStorePtr,
+ const ZSTD_fseCTables_t* prevEntropy,
+ ZSTD_fseCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_fseCTablesMetadata_t* fseMetadata,
+ void* workspace, size_t wkspSize)
+{
+ BYTE* const wkspStart = (BYTE*)workspace;
+ BYTE* const wkspEnd = wkspStart + wkspSize;
+ BYTE* const countWkspStart = wkspStart;
+ unsigned* const countWksp = (unsigned*)workspace;
+ const size_t countWkspSize = (MaxSeq + 1) * sizeof(unsigned);
+ BYTE* const cTableWksp = countWkspStart + countWkspSize;
+ const size_t cTableWkspSize = wkspEnd-cTableWksp;
+ ZSTD_strategy const strategy = cctxParams->cParams.strategy;
+ FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable;
+ FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable;
+ FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable;
+ const BYTE* const ofCodeTable = seqStorePtr->ofCode;
+ const BYTE* const llCodeTable = seqStorePtr->llCode;
+ const BYTE* const mlCodeTable = seqStorePtr->mlCode;
+ size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
+ BYTE* const ostart = fseMetadata->fseTablesBuffer;
+ BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer);
+ BYTE* op = ostart;
+
+ assert(cTableWkspSize >= (1 << MaxFSELog) * sizeof(FSE_FUNCTION_TYPE));
+ DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_sequences (nbSeq=%zu)", nbSeq);
+ ZSTD_memset(workspace, 0, wkspSize);
+
+ fseMetadata->lastCountSize = 0;
+ /* convert length/distances into codes */
+ ZSTD_seqToCodes(seqStorePtr);
+ /* build CTable for Literal Lengths */
+ { U32 LLtype;
+ unsigned max = MaxLL;
+ size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, llCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
+ DEBUGLOG(5, "Building LL table");
+ nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode;
+ LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode,
+ countWksp, max, mostFrequent, nbSeq,
+ LLFSELog, prevEntropy->litlengthCTable,
+ LL_defaultNorm, LL_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(set_basic < set_compressed && set_rle < set_compressed);
+ assert(!(LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
+ countWksp, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL,
+ prevEntropy->litlengthCTable, sizeof(prevEntropy->litlengthCTable),
+ cTableWksp, cTableWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
+ if (LLtype == set_compressed)
+ fseMetadata->lastCountSize = countSize;
+ op += countSize;
+ fseMetadata->llType = (symbolEncodingType_e) LLtype;
+ } }
+ /* build CTable for Offsets */
+ { U32 Offtype;
+ unsigned max = MaxOff;
+ size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, ofCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
+ /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
+ ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
+ DEBUGLOG(5, "Building OF table");
+ nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode;
+ Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode,
+ countWksp, max, mostFrequent, nbSeq,
+ OffFSELog, prevEntropy->offcodeCTable,
+ OF_defaultNorm, OF_defaultNormLog,
+ defaultPolicy, strategy);
+ assert(!(Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
+ countWksp, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+ prevEntropy->offcodeCTable, sizeof(prevEntropy->offcodeCTable),
+ cTableWksp, cTableWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
+ if (Offtype == set_compressed)
+ fseMetadata->lastCountSize = countSize;
+ op += countSize;
+ fseMetadata->ofType = (symbolEncodingType_e) Offtype;
+ } }
+ /* build CTable for MatchLengths */
+ { U32 MLtype;
+ unsigned max = MaxML;
+ size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, mlCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
+ DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
+ nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode;
+ MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode,
+ countWksp, max, mostFrequent, nbSeq,
+ MLFSELog, prevEntropy->matchlengthCTable,
+ ML_defaultNorm, ML_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(!(MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
+ countWksp, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML,
+ prevEntropy->matchlengthCTable, sizeof(prevEntropy->matchlengthCTable),
+ cTableWksp, cTableWkspSize);
+ FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
+ if (MLtype == set_compressed)
+ fseMetadata->lastCountSize = countSize;
+ op += countSize;
+ fseMetadata->mlType = (symbolEncodingType_e) MLtype;
+ } }
+ assert((size_t) (op-ostart) <= sizeof(fseMetadata->fseTablesBuffer));
+ return op-ostart;
+}
+
+
+/* ZSTD_buildSuperBlockEntropy() :
+ * Builds entropy for the super-block.
+ * @return : 0 on success or error code */
+static size_t
+ZSTD_buildSuperBlockEntropy(seqStore_t* seqStorePtr,
+ const ZSTD_entropyCTables_t* prevEntropy,
+ ZSTD_entropyCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ void* workspace, size_t wkspSize)
+{
+ size_t const litSize = seqStorePtr->lit - seqStorePtr->litStart;
+ DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy");
+ entropyMetadata->hufMetadata.hufDesSize =
+ ZSTD_buildSuperBlockEntropy_literal(seqStorePtr->litStart, litSize,
+ &prevEntropy->huf, &nextEntropy->huf,
+ &entropyMetadata->hufMetadata,
+ ZSTD_disableLiteralsCompression(cctxParams),
+ workspace, wkspSize);
+ FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildSuperBlockEntropy_literal failed");
+ entropyMetadata->fseMetadata.fseTablesSize =
+ ZSTD_buildSuperBlockEntropy_sequences(seqStorePtr,
+ &prevEntropy->fse, &nextEntropy->fse,
+ cctxParams,
+ &entropyMetadata->fseMetadata,
+ workspace, wkspSize);
+ FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildSuperBlockEntropy_sequences failed");
+ return 0;
+}
+
+/* ZSTD_compressSubBlock_literal() :
+ * Compresses literals section for a sub-block.
+ * When we have to write the Huffman table we will sometimes choose a header
+ * size larger than necessary. This is because we have to pick the header size
+ * before we know the table size + compressed size, so we have a bound on the
+ * table size. If we guessed incorrectly, we fall back to uncompressed literals.
+ *
+ * We write the header when writeEntropy=1 and set entropyWritten=1 when we succeeded
+ * in writing the header, otherwise it is set to 0.
+ *
+ * hufMetadata->hType has literals block type info.
+ * If it is set_basic, all sub-blocks literals section will be Raw_Literals_Block.
+ * If it is set_rle, all sub-blocks literals section will be RLE_Literals_Block.
+ * If it is set_compressed, first sub-block's literals section will be Compressed_Literals_Block
+ * If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block
+ * and the following sub-blocks' literals sections will be Treeless_Literals_Block.
+ * @return : compressed size of literals section of a sub-block
+ * Or 0 if it unable to compress.
+ * Or error code */
+static size_t ZSTD_compressSubBlock_literal(const HUF_CElt* hufTable,
+ const ZSTD_hufCTablesMetadata_t* hufMetadata,
+ const BYTE* literals, size_t litSize,
+ void* dst, size_t dstSize,
+ const int bmi2, int writeEntropy, int* entropyWritten)
+{
+ size_t const header = writeEntropy ? 200 : 0;
+ size_t const lhSize = 3 + (litSize >= (1 KB - header)) + (litSize >= (16 KB - header));
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* op = ostart + lhSize;
+ U32 const singleStream = lhSize == 3;
+ symbolEncodingType_e hType = writeEntropy ? hufMetadata->hType : set_repeat;
+ size_t cLitSize = 0;
+
+ (void)bmi2; /* TODO bmi2... */
+
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal (litSize=%zu, lhSize=%zu, writeEntropy=%d)", litSize, lhSize, writeEntropy);
+
+ *entropyWritten = 0;
+ if (litSize == 0 || hufMetadata->hType == set_basic) {
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal");
+ return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+ } else if (hufMetadata->hType == set_rle) {
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal using rle literal");
+ return ZSTD_compressRleLiteralsBlock(dst, dstSize, literals, litSize);
+ }
+
+ assert(litSize > 0);
+ assert(hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat);
+
+ if (writeEntropy && hufMetadata->hType == set_compressed) {
+ ZSTD_memcpy(op, hufMetadata->hufDesBuffer, hufMetadata->hufDesSize);
+ op += hufMetadata->hufDesSize;
+ cLitSize += hufMetadata->hufDesSize;
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal (hSize=%zu)", hufMetadata->hufDesSize);
+ }
+
+ /* TODO bmi2 */
+ { const size_t cSize = singleStream ? HUF_compress1X_usingCTable(op, oend-op, literals, litSize, hufTable)
+ : HUF_compress4X_usingCTable(op, oend-op, literals, litSize, hufTable);
+ op += cSize;
+ cLitSize += cSize;
+ if (cSize == 0 || ERR_isError(cSize)) {
+ DEBUGLOG(5, "Failed to write entropy tables %s", ZSTD_getErrorName(cSize));
+ return 0;
+ }
+ /* If we expand and we aren't writing a header then emit uncompressed */
+ if (!writeEntropy && cLitSize >= litSize) {
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal using raw literal because uncompressible");
+ return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+ }
+ /* If we are writing headers then allow expansion that doesn't change our header size. */
+ if (lhSize < (size_t)(3 + (cLitSize >= 1 KB) + (cLitSize >= 16 KB))) {
+ assert(cLitSize > litSize);
+ DEBUGLOG(5, "Literals expanded beyond allowed header size");
+ return ZSTD_noCompressLiterals(dst, dstSize, literals, litSize);
+ }
+ DEBUGLOG(5, "ZSTD_compressSubBlock_literal (cSize=%zu)", cSize);
+ }
+
+ /* Build header */
+ switch(lhSize)
+ {
+ case 3: /* 2 - 2 - 10 - 10 */
+ { U32 const lhc = hType + ((!singleStream) << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<14);
+ MEM_writeLE24(ostart, lhc);
+ break;
+ }
+ case 4: /* 2 - 2 - 14 - 14 */
+ { U32 const lhc = hType + (2 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<18);
+ MEM_writeLE32(ostart, lhc);
+ break;
+ }
+ case 5: /* 2 - 2 - 18 - 18 */
+ { U32 const lhc = hType + (3 << 2) + ((U32)litSize<<4) + ((U32)cLitSize<<22);
+ MEM_writeLE32(ostart, lhc);
+ ostart[4] = (BYTE)(cLitSize >> 10);
+ break;
+ }
+ default: /* not possible : lhSize is {3,4,5} */
+ assert(0);
+ }
+ *entropyWritten = 1;
+ DEBUGLOG(5, "Compressed literals: %u -> %u", (U32)litSize, (U32)(op-ostart));
+ return op-ostart;
+}
+
+static size_t ZSTD_seqDecompressedSize(seqStore_t const* seqStore, const seqDef* sequences, size_t nbSeq, size_t litSize, int lastSequence) {
+ const seqDef* const sstart = sequences;
+ const seqDef* const send = sequences + nbSeq;
+ const seqDef* sp = sstart;
+ size_t matchLengthSum = 0;
+ size_t litLengthSum = 0;
+ /* Only used by assert(), suppress unused variable warnings in production. */
+ (void)litLengthSum;
+ while (send-sp > 0) {
+ ZSTD_sequenceLength const seqLen = ZSTD_getSequenceLength(seqStore, sp);
+ litLengthSum += seqLen.litLength;
+ matchLengthSum += seqLen.matchLength;
+ sp++;
+ }
+ assert(litLengthSum <= litSize);
+ if (!lastSequence) {
+ assert(litLengthSum == litSize);
+ }
+ return matchLengthSum + litSize;
+}
+
+/* ZSTD_compressSubBlock_sequences() :
+ * Compresses sequences section for a sub-block.
+ * fseMetadata->llType, fseMetadata->ofType, and fseMetadata->mlType have
+ * symbol compression modes for the super-block.
+ * The first successfully compressed block will have these in its header.
+ * We set entropyWritten=1 when we succeed in compressing the sequences.
+ * The following sub-blocks will always have repeat mode.
+ * @return : compressed size of sequences section of a sub-block
+ * Or 0 if it is unable to compress
+ * Or error code. */
+static size_t ZSTD_compressSubBlock_sequences(const ZSTD_fseCTables_t* fseTables,
+ const ZSTD_fseCTablesMetadata_t* fseMetadata,
+ const seqDef* sequences, size_t nbSeq,
+ const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
+ const ZSTD_CCtx_params* cctxParams,
+ void* dst, size_t dstCapacity,
+ const int bmi2, int writeEntropy, int* entropyWritten)
+{
+ const int longOffsets = cctxParams->cParams.windowLog > STREAM_ACCUMULATOR_MIN;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstCapacity;
+ BYTE* op = ostart;
+ BYTE* seqHead;
+
+ DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (nbSeq=%zu, writeEntropy=%d, longOffsets=%d)", nbSeq, writeEntropy, longOffsets);
+
+ *entropyWritten = 0;
+ /* Sequences Header */
+ RETURN_ERROR_IF((oend-op) < 3 /*max nbSeq Size*/ + 1 /*seqHead*/,
+ dstSize_tooSmall, "");
+ if (nbSeq < 0x7F)
+ *op++ = (BYTE)nbSeq;
+ else if (nbSeq < LONGNBSEQ)
+ op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2;
+ else
+ op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3;
+ if (nbSeq==0) {
+ return op - ostart;
+ }
+
+ /* seqHead : flags for FSE encoding type */
+ seqHead = op++;
+
+ DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (seqHeadSize=%u)", (unsigned)(op-ostart));
+
+ if (writeEntropy) {
+ const U32 LLtype = fseMetadata->llType;
+ const U32 Offtype = fseMetadata->ofType;
+ const U32 MLtype = fseMetadata->mlType;
+ DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (fseTablesSize=%zu)", fseMetadata->fseTablesSize);
+ *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
+ ZSTD_memcpy(op, fseMetadata->fseTablesBuffer, fseMetadata->fseTablesSize);
+ op += fseMetadata->fseTablesSize;
+ } else {
+ const U32 repeat = set_repeat;
+ *seqHead = (BYTE)((repeat<<6) + (repeat<<4) + (repeat<<2));
+ }
+
+ { size_t const bitstreamSize = ZSTD_encodeSequences(
+ op, oend - op,
+ fseTables->matchlengthCTable, mlCode,
+ fseTables->offcodeCTable, ofCode,
+ fseTables->litlengthCTable, llCode,
+ sequences, nbSeq,
+ longOffsets, bmi2);
+ FORWARD_IF_ERROR(bitstreamSize, "ZSTD_encodeSequences failed");
+ op += bitstreamSize;
+ /* zstd versions <= 1.3.4 mistakenly report corruption when
+ * FSE_readNCount() receives a buffer < 4 bytes.
+ * Fixed by https://github.com/facebook/zstd/pull/1146.
+ * This can happen when the last set_compressed table present is 2
+ * bytes and the bitstream is only one byte.
+ * In this exceedingly rare case, we will simply emit an uncompressed
+ * block, since it isn't worth optimizing.
+ */
+#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ if (writeEntropy && fseMetadata->lastCountSize && fseMetadata->lastCountSize + bitstreamSize < 4) {
+ /* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
+ assert(fseMetadata->lastCountSize + bitstreamSize == 3);
+ DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
+ "emitting an uncompressed block.");
+ return 0;
+ }
+#endif
+ DEBUGLOG(5, "ZSTD_compressSubBlock_sequences (bitstreamSize=%zu)", bitstreamSize);
+ }
+
+ /* zstd versions <= 1.4.0 mistakenly report error when
+ * sequences section body size is less than 3 bytes.
+ * Fixed by https://github.com/facebook/zstd/pull/1664.
+ * This can happen when the previous sequences section block is compressed
+ * with rle mode and the current block's sequences section is compressed
+ * with repeat mode where sequences section body size can be 1 byte.
+ */
+#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ if (op-seqHead < 4) {
+ DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.4.0 by emitting "
+ "an uncompressed block when sequences are < 4 bytes");
+ return 0;
+ }
+#endif
+
+ *entropyWritten = 1;
+ return op - ostart;
+}
+
+/* ZSTD_compressSubBlock() :
+ * Compresses a single sub-block.
+ * @return : compressed size of the sub-block
+ * Or 0 if it failed to compress. */
+static size_t ZSTD_compressSubBlock(const ZSTD_entropyCTables_t* entropy,
+ const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ const seqDef* sequences, size_t nbSeq,
+ const BYTE* literals, size_t litSize,
+ const BYTE* llCode, const BYTE* mlCode, const BYTE* ofCode,
+ const ZSTD_CCtx_params* cctxParams,
+ void* dst, size_t dstCapacity,
+ const int bmi2,
+ int writeLitEntropy, int writeSeqEntropy,
+ int* litEntropyWritten, int* seqEntropyWritten,
+ U32 lastBlock)
+{
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstCapacity;
+ BYTE* op = ostart + ZSTD_blockHeaderSize;
+ DEBUGLOG(5, "ZSTD_compressSubBlock (litSize=%zu, nbSeq=%zu, writeLitEntropy=%d, writeSeqEntropy=%d, lastBlock=%d)",
+ litSize, nbSeq, writeLitEntropy, writeSeqEntropy, lastBlock);
+ { size_t cLitSize = ZSTD_compressSubBlock_literal((const HUF_CElt*)entropy->huf.CTable,
+ &entropyMetadata->hufMetadata, literals, litSize,
+ op, oend-op, bmi2, writeLitEntropy, litEntropyWritten);
+ FORWARD_IF_ERROR(cLitSize, "ZSTD_compressSubBlock_literal failed");
+ if (cLitSize == 0) return 0;
+ op += cLitSize;
+ }
+ { size_t cSeqSize = ZSTD_compressSubBlock_sequences(&entropy->fse,
+ &entropyMetadata->fseMetadata,
+ sequences, nbSeq,
+ llCode, mlCode, ofCode,
+ cctxParams,
+ op, oend-op,
+ bmi2, writeSeqEntropy, seqEntropyWritten);
+ FORWARD_IF_ERROR(cSeqSize, "ZSTD_compressSubBlock_sequences failed");
+ if (cSeqSize == 0) return 0;
+ op += cSeqSize;
+ }
+ /* Write block header */
+ { size_t cSize = (op-ostart)-ZSTD_blockHeaderSize;
+ U32 const cBlockHeader24 = lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
+ MEM_writeLE24(ostart, cBlockHeader24);
+ }
+ return op-ostart;
+}
+
+static size_t ZSTD_estimateSubBlockSize_literal(const BYTE* literals, size_t litSize,
+ const ZSTD_hufCTables_t* huf,
+ const ZSTD_hufCTablesMetadata_t* hufMetadata,
+ void* workspace, size_t wkspSize,
+ int writeEntropy)
+{
+ unsigned* const countWksp = (unsigned*)workspace;
+ unsigned maxSymbolValue = 255;
+ size_t literalSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
+
+ if (hufMetadata->hType == set_basic) return litSize;
+ else if (hufMetadata->hType == set_rle) return 1;
+ else if (hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat) {
+ size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize);
+ if (ZSTD_isError(largest)) return litSize;
+ { size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue);
+ if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize;
+ return cLitSizeEstimate + literalSectionHeaderSize;
+ } }
+ assert(0); /* impossible */
+ return 0;
+}
+
+static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,
+ const BYTE* codeTable, unsigned maxCode,
+ size_t nbSeq, const FSE_CTable* fseCTable,
+ const U32* additionalBits,
+ short const* defaultNorm, U32 defaultNormLog, U32 defaultMax,
+ void* workspace, size_t wkspSize)
+{
+ unsigned* const countWksp = (unsigned*)workspace;
+ const BYTE* ctp = codeTable;
+ const BYTE* const ctStart = ctp;
+ const BYTE* const ctEnd = ctStart + nbSeq;
+ size_t cSymbolTypeSizeEstimateInBits = 0;
+ unsigned max = maxCode;
+
+ HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); /* can't fail */
+ if (type == set_basic) {
+ /* We selected this encoding type, so it must be valid. */
+ assert(max <= defaultMax);
+ cSymbolTypeSizeEstimateInBits = max <= defaultMax
+ ? ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max)
+ : ERROR(GENERIC);
+ } else if (type == set_rle) {
+ cSymbolTypeSizeEstimateInBits = 0;
+ } else if (type == set_compressed || type == set_repeat) {
+ cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max);
+ }
+ if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) return nbSeq * 10;
+ while (ctp < ctEnd) {
+ if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp];
+ else cSymbolTypeSizeEstimateInBits += *ctp; /* for offset, offset code is also the number of additional bits */
+ ctp++;
+ }
+ return cSymbolTypeSizeEstimateInBits / 8;
+}
+
+static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable,
+ const BYTE* llCodeTable,
+ const BYTE* mlCodeTable,
+ size_t nbSeq,
+ const ZSTD_fseCTables_t* fseTables,
+ const ZSTD_fseCTablesMetadata_t* fseMetadata,
+ void* workspace, size_t wkspSize,
+ int writeEntropy)
+{
+ size_t sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
+ size_t cSeqSizeEstimate = 0;
+ cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff,
+ nbSeq, fseTables->offcodeCTable, NULL,
+ OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+ workspace, wkspSize);
+ cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->llType, llCodeTable, MaxLL,
+ nbSeq, fseTables->litlengthCTable, LL_bits,
+ LL_defaultNorm, LL_defaultNormLog, MaxLL,
+ workspace, wkspSize);
+ cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, MaxML,
+ nbSeq, fseTables->matchlengthCTable, ML_bits,
+ ML_defaultNorm, ML_defaultNormLog, MaxML,
+ workspace, wkspSize);
+ if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize;
+ return cSeqSizeEstimate + sequencesSectionHeaderSize;
+}
+
+static size_t ZSTD_estimateSubBlockSize(const BYTE* literals, size_t litSize,
+ const BYTE* ofCodeTable,
+ const BYTE* llCodeTable,
+ const BYTE* mlCodeTable,
+ size_t nbSeq,
+ const ZSTD_entropyCTables_t* entropy,
+ const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ void* workspace, size_t wkspSize,
+ int writeLitEntropy, int writeSeqEntropy) {
+ size_t cSizeEstimate = 0;
+ cSizeEstimate += ZSTD_estimateSubBlockSize_literal(literals, litSize,
+ &entropy->huf, &entropyMetadata->hufMetadata,
+ workspace, wkspSize, writeLitEntropy);
+ cSizeEstimate += ZSTD_estimateSubBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable,
+ nbSeq, &entropy->fse, &entropyMetadata->fseMetadata,
+ workspace, wkspSize, writeSeqEntropy);
+ return cSizeEstimate + ZSTD_blockHeaderSize;
+}
+
+static int ZSTD_needSequenceEntropyTables(ZSTD_fseCTablesMetadata_t const* fseMetadata)
+{
+ if (fseMetadata->llType == set_compressed || fseMetadata->llType == set_rle)
+ return 1;
+ if (fseMetadata->mlType == set_compressed || fseMetadata->mlType == set_rle)
+ return 1;
+ if (fseMetadata->ofType == set_compressed || fseMetadata->ofType == set_rle)
+ return 1;
+ return 0;
+}
+
+/* ZSTD_compressSubBlock_multi() :
+ * Breaks super-block into multiple sub-blocks and compresses them.
+ * Entropy will be written to the first block.
+ * The following blocks will use repeat mode to compress.
+ * All sub-blocks are compressed blocks (no raw or rle blocks).
+ * @return : compressed size of the super block (which is multiple ZSTD blocks)
+ * Or 0 if it failed to compress. */
+static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
+ const ZSTD_compressedBlockState_t* prevCBlock,
+ ZSTD_compressedBlockState_t* nextCBlock,
+ const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ const ZSTD_CCtx_params* cctxParams,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const int bmi2, U32 lastBlock,
+ void* workspace, size_t wkspSize)
+{
+ const seqDef* const sstart = seqStorePtr->sequencesStart;
+ const seqDef* const send = seqStorePtr->sequences;
+ const seqDef* sp = sstart;
+ const BYTE* const lstart = seqStorePtr->litStart;
+ const BYTE* const lend = seqStorePtr->lit;
+ const BYTE* lp = lstart;
+ BYTE const* ip = (BYTE const*)src;
+ BYTE const* const iend = ip + srcSize;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + dstCapacity;
+ BYTE* op = ostart;
+ const BYTE* llCodePtr = seqStorePtr->llCode;
+ const BYTE* mlCodePtr = seqStorePtr->mlCode;
+ const BYTE* ofCodePtr = seqStorePtr->ofCode;
+ size_t targetCBlockSize = cctxParams->targetCBlockSize;
+ size_t litSize, seqCount;
+ int writeLitEntropy = entropyMetadata->hufMetadata.hType == set_compressed;
+ int writeSeqEntropy = 1;
+ int lastSequence = 0;
+
+ DEBUGLOG(5, "ZSTD_compressSubBlock_multi (litSize=%u, nbSeq=%u)",
+ (unsigned)(lend-lp), (unsigned)(send-sstart));
+
+ litSize = 0;
+ seqCount = 0;
+ do {
+ size_t cBlockSizeEstimate = 0;
+ if (sstart == send) {
+ lastSequence = 1;
+ } else {
+ const seqDef* const sequence = sp + seqCount;
+ lastSequence = sequence == send - 1;
+ litSize += ZSTD_getSequenceLength(seqStorePtr, sequence).litLength;
+ seqCount++;
+ }
+ if (lastSequence) {
+ assert(lp <= lend);
+ assert(litSize <= (size_t)(lend - lp));
+ litSize = (size_t)(lend - lp);
+ }
+ /* I think there is an optimization opportunity here.
+ * Calling ZSTD_estimateSubBlockSize for every sequence can be wasteful
+ * since it recalculates estimate from scratch.
+ * For example, it would recount literal distribution and symbol codes everytime.
+ */
+ cBlockSizeEstimate = ZSTD_estimateSubBlockSize(lp, litSize, ofCodePtr, llCodePtr, mlCodePtr, seqCount,
+ &nextCBlock->entropy, entropyMetadata,
+ workspace, wkspSize, writeLitEntropy, writeSeqEntropy);
+ if (cBlockSizeEstimate > targetCBlockSize || lastSequence) {
+ int litEntropyWritten = 0;
+ int seqEntropyWritten = 0;
+ const size_t decompressedSize = ZSTD_seqDecompressedSize(seqStorePtr, sp, seqCount, litSize, lastSequence);
+ const size_t cSize = ZSTD_compressSubBlock(&nextCBlock->entropy, entropyMetadata,
+ sp, seqCount,
+ lp, litSize,
+ llCodePtr, mlCodePtr, ofCodePtr,
+ cctxParams,
+ op, oend-op,
+ bmi2, writeLitEntropy, writeSeqEntropy,
+ &litEntropyWritten, &seqEntropyWritten,
+ lastBlock && lastSequence);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressSubBlock failed");
+ if (cSize > 0 && cSize < decompressedSize) {
+ DEBUGLOG(5, "Committed the sub-block");
+ assert(ip + decompressedSize <= iend);
+ ip += decompressedSize;
+ sp += seqCount;
+ lp += litSize;
+ op += cSize;
+ llCodePtr += seqCount;
+ mlCodePtr += seqCount;
+ ofCodePtr += seqCount;
+ litSize = 0;
+ seqCount = 0;
+ /* Entropy only needs to be written once */
+ if (litEntropyWritten) {
+ writeLitEntropy = 0;
+ }
+ if (seqEntropyWritten) {
+ writeSeqEntropy = 0;
+ }
+ }
+ }
+ } while (!lastSequence);
+ if (writeLitEntropy) {
+ DEBUGLOG(5, "ZSTD_compressSubBlock_multi has literal entropy tables unwritten");
+ ZSTD_memcpy(&nextCBlock->entropy.huf, &prevCBlock->entropy.huf, sizeof(prevCBlock->entropy.huf));
+ }
+ if (writeSeqEntropy && ZSTD_needSequenceEntropyTables(&entropyMetadata->fseMetadata)) {
+ /* If we haven't written our entropy tables, then we've violated our contract and
+ * must emit an uncompressed block.
+ */
+ DEBUGLOG(5, "ZSTD_compressSubBlock_multi has sequence entropy tables unwritten");
+ return 0;
+ }
+ if (ip < iend) {
+ size_t const cSize = ZSTD_noCompressBlock(op, oend - op, ip, iend - ip, lastBlock);
+ DEBUGLOG(5, "ZSTD_compressSubBlock_multi last sub-block uncompressed, %zu bytes", (size_t)(iend - ip));
+ FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
+ assert(cSize != 0);
+ op += cSize;
+ /* We have to regenerate the repcodes because we've skipped some sequences */
+ if (sp < send) {
+ seqDef const* seq;
+ repcodes_t rep;
+ ZSTD_memcpy(&rep, prevCBlock->rep, sizeof(rep));
+ for (seq = sstart; seq < sp; ++seq) {
+ rep = ZSTD_updateRep(rep.rep, seq->offset - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
+ }
+ ZSTD_memcpy(nextCBlock->rep, &rep, sizeof(rep));
+ }
+ }
+ DEBUGLOG(5, "ZSTD_compressSubBlock_multi compressed");
+ return op-ostart;
+}
+
+size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ void const* src, size_t srcSize,
+ unsigned lastBlock) {
+ ZSTD_entropyCTablesMetadata_t entropyMetadata;
+
+ FORWARD_IF_ERROR(ZSTD_buildSuperBlockEntropy(&zc->seqStore,
+ &zc->blockState.prevCBlock->entropy,
+ &zc->blockState.nextCBlock->entropy,
+ &zc->appliedParams,
+ &entropyMetadata,
+ zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */), "");
+
+ return ZSTD_compressSubBlock_multi(&zc->seqStore,
+ zc->blockState.prevCBlock,
+ zc->blockState.nextCBlock,
+ &entropyMetadata,
+ &zc->appliedParams,
+ dst, dstCapacity,
+ src, srcSize,
+ zc->bmi2, lastBlock,
+ zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_COMPRESS_ADVANCED_H
+#define ZSTD_COMPRESS_ADVANCED_H
+
+/*-*************************************
+* Dependencies
+***************************************/
+
+#include <linux/zstd.h> /* ZSTD_CCtx */
+
+/*-*************************************
+* Target Compressed Block Size
+***************************************/
+
+/* ZSTD_compressSuperBlock() :
+ * Used to compress a super block when targetCBlockSize is being used.
+ * The given block will be compressed into multiple sub blocks that are around targetCBlockSize. */
+size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ void const* src, size_t srcSize,
+ unsigned lastBlock);
+
+#endif /* ZSTD_COMPRESS_ADVANCED_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_CWKSP_H
+#define ZSTD_CWKSP_H
+
+/*-*************************************
+* Dependencies
+***************************************/
+#include "../common/zstd_internal.h"
+
+
+/*-*************************************
+* Constants
+***************************************/
+
+/* Since the workspace is effectively its own little malloc implementation /
+ * arena, when we run under ASAN, we should similarly insert redzones between
+ * each internal element of the workspace, so ASAN will catch overruns that
+ * reach outside an object but that stay inside the workspace.
+ *
+ * This defines the size of that redzone.
+ */
+#ifndef ZSTD_CWKSP_ASAN_REDZONE_SIZE
+#define ZSTD_CWKSP_ASAN_REDZONE_SIZE 128
+#endif
+
+/*-*************************************
+* Structures
+***************************************/
+typedef enum {
+ ZSTD_cwksp_alloc_objects,
+ ZSTD_cwksp_alloc_buffers,
+ ZSTD_cwksp_alloc_aligned
+} ZSTD_cwksp_alloc_phase_e;
+
+/*
+ * Used to describe whether the workspace is statically allocated (and will not
+ * necessarily ever be freed), or if it's dynamically allocated and we can
+ * expect a well-formed caller to free this.
+ */
+typedef enum {
+ ZSTD_cwksp_dynamic_alloc,
+ ZSTD_cwksp_static_alloc
+} ZSTD_cwksp_static_alloc_e;
+
+/*
+ * Zstd fits all its internal datastructures into a single continuous buffer,
+ * so that it only needs to perform a single OS allocation (or so that a buffer
+ * can be provided to it and it can perform no allocations at all). This buffer
+ * is called the workspace.
+ *
+ * Several optimizations complicate that process of allocating memory ranges
+ * from this workspace for each internal datastructure:
+ *
+ * - These different internal datastructures have different setup requirements:
+ *
+ * - The static objects need to be cleared once and can then be trivially
+ * reused for each compression.
+ *
+ * - Various buffers don't need to be initialized at all--they are always
+ * written into before they're read.
+ *
+ * - The matchstate tables have a unique requirement that they don't need
+ * their memory to be totally cleared, but they do need the memory to have
+ * some bound, i.e., a guarantee that all values in the memory they've been
+ * allocated is less than some maximum value (which is the starting value
+ * for the indices that they will then use for compression). When this
+ * guarantee is provided to them, they can use the memory without any setup
+ * work. When it can't, they have to clear the area.
+ *
+ * - These buffers also have different alignment requirements.
+ *
+ * - We would like to reuse the objects in the workspace for multiple
+ * compressions without having to perform any expensive reallocation or
+ * reinitialization work.
+ *
+ * - We would like to be able to efficiently reuse the workspace across
+ * multiple compressions **even when the compression parameters change** and
+ * we need to resize some of the objects (where possible).
+ *
+ * To attempt to manage this buffer, given these constraints, the ZSTD_cwksp
+ * abstraction was created. It works as follows:
+ *
+ * Workspace Layout:
+ *
+ * [ ... workspace ... ]
+ * [objects][tables ... ->] free space [<- ... aligned][<- ... buffers]
+ *
+ * The various objects that live in the workspace are divided into the
+ * following categories, and are allocated separately:
+ *
+ * - Static objects: this is optionally the enclosing ZSTD_CCtx or ZSTD_CDict,
+ * so that literally everything fits in a single buffer. Note: if present,
+ * this must be the first object in the workspace, since ZSTD_customFree{CCtx,
+ * CDict}() rely on a pointer comparison to see whether one or two frees are
+ * required.
+ *
+ * - Fixed size objects: these are fixed-size, fixed-count objects that are
+ * nonetheless "dynamically" allocated in the workspace so that we can
+ * control how they're initialized separately from the broader ZSTD_CCtx.
+ * Examples:
+ * - Entropy Workspace
+ * - 2 x ZSTD_compressedBlockState_t
+ * - CDict dictionary contents
+ *
+ * - Tables: these are any of several different datastructures (hash tables,
+ * chain tables, binary trees) that all respect a common format: they are
+ * uint32_t arrays, all of whose values are between 0 and (nextSrc - base).
+ * Their sizes depend on the cparams.
+ *
+ * - Aligned: these buffers are used for various purposes that require 4 byte
+ * alignment, but don't require any initialization before they're used.
+ *
+ * - Buffers: these buffers are used for various purposes that don't require
+ * any alignment or initialization before they're used. This means they can
+ * be moved around at no cost for a new compression.
+ *
+ * Allocating Memory:
+ *
+ * The various types of objects must be allocated in order, so they can be
+ * correctly packed into the workspace buffer. That order is:
+ *
+ * 1. Objects
+ * 2. Buffers
+ * 3. Aligned
+ * 4. Tables
+ *
+ * Attempts to reserve objects of different types out of order will fail.
+ */
+typedef struct {
+ void* workspace;
+ void* workspaceEnd;
+
+ void* objectEnd;
+ void* tableEnd;
+ void* tableValidEnd;
+ void* allocStart;
+
+ BYTE allocFailed;
+ int workspaceOversizedDuration;
+ ZSTD_cwksp_alloc_phase_e phase;
+ ZSTD_cwksp_static_alloc_e isStatic;
+} ZSTD_cwksp;
+
+/*-*************************************
+* Functions
+***************************************/
+
+MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws);
+
+MEM_STATIC void ZSTD_cwksp_assert_internal_consistency(ZSTD_cwksp* ws) {
+ (void)ws;
+ assert(ws->workspace <= ws->objectEnd);
+ assert(ws->objectEnd <= ws->tableEnd);
+ assert(ws->objectEnd <= ws->tableValidEnd);
+ assert(ws->tableEnd <= ws->allocStart);
+ assert(ws->tableValidEnd <= ws->allocStart);
+ assert(ws->allocStart <= ws->workspaceEnd);
+}
+
+/*
+ * Align must be a power of 2.
+ */
+MEM_STATIC size_t ZSTD_cwksp_align(size_t size, size_t const align) {
+ size_t const mask = align - 1;
+ assert((align & mask) == 0);
+ return (size + mask) & ~mask;
+}
+
+/*
+ * Use this to determine how much space in the workspace we will consume to
+ * allocate this object. (Normally it should be exactly the size of the object,
+ * but under special conditions, like ASAN, where we pad each object, it might
+ * be larger.)
+ *
+ * Since tables aren't currently redzoned, you don't need to call through this
+ * to figure out how much space you need for the matchState tables. Everything
+ * else is though.
+ */
+MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) {
+ if (size == 0)
+ return 0;
+ return size;
+}
+
+MEM_STATIC void ZSTD_cwksp_internal_advance_phase(
+ ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase) {
+ assert(phase >= ws->phase);
+ if (phase > ws->phase) {
+ if (ws->phase < ZSTD_cwksp_alloc_buffers &&
+ phase >= ZSTD_cwksp_alloc_buffers) {
+ ws->tableValidEnd = ws->objectEnd;
+ }
+ if (ws->phase < ZSTD_cwksp_alloc_aligned &&
+ phase >= ZSTD_cwksp_alloc_aligned) {
+ /* If unaligned allocations down from a too-large top have left us
+ * unaligned, we need to realign our alloc ptr. Technically, this
+ * can consume space that is unaccounted for in the neededSpace
+ * calculation. However, I believe this can only happen when the
+ * workspace is too large, and specifically when it is too large
+ * by a larger margin than the space that will be consumed. */
+ /* TODO: cleaner, compiler warning friendly way to do this??? */
+ ws->allocStart = (BYTE*)ws->allocStart - ((size_t)ws->allocStart & (sizeof(U32)-1));
+ if (ws->allocStart < ws->tableValidEnd) {
+ ws->tableValidEnd = ws->allocStart;
+ }
+ }
+ ws->phase = phase;
+ }
+}
+
+/*
+ * Returns whether this object/buffer/etc was allocated in this workspace.
+ */
+MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr) {
+ return (ptr != NULL) && (ws->workspace <= ptr) && (ptr <= ws->workspaceEnd);
+}
+
+/*
+ * Internal function. Do not use directly.
+ */
+MEM_STATIC void* ZSTD_cwksp_reserve_internal(
+ ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase) {
+ void* alloc;
+ void* bottom = ws->tableEnd;
+ ZSTD_cwksp_internal_advance_phase(ws, phase);
+ alloc = (BYTE *)ws->allocStart - bytes;
+
+ if (bytes == 0)
+ return NULL;
+
+
+ DEBUGLOG(5, "cwksp: reserving %p %zd bytes, %zd bytes remaining",
+ alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
+ ZSTD_cwksp_assert_internal_consistency(ws);
+ assert(alloc >= bottom);
+ if (alloc < bottom) {
+ DEBUGLOG(4, "cwksp: alloc failed!");
+ ws->allocFailed = 1;
+ return NULL;
+ }
+ if (alloc < ws->tableValidEnd) {
+ ws->tableValidEnd = alloc;
+ }
+ ws->allocStart = alloc;
+
+
+ return alloc;
+}
+
+/*
+ * Reserves and returns unaligned memory.
+ */
+MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes) {
+ return (BYTE*)ZSTD_cwksp_reserve_internal(ws, bytes, ZSTD_cwksp_alloc_buffers);
+}
+
+/*
+ * Reserves and returns memory sized on and aligned on sizeof(unsigned).
+ */
+MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes) {
+ assert((bytes & (sizeof(U32)-1)) == 0);
+ return ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, sizeof(U32)), ZSTD_cwksp_alloc_aligned);
+}
+
+/*
+ * Aligned on sizeof(unsigned). These buffers have the special property that
+ * their values remain constrained, allowing us to re-use them without
+ * memset()-ing them.
+ */
+MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) {
+ const ZSTD_cwksp_alloc_phase_e phase = ZSTD_cwksp_alloc_aligned;
+ void* alloc = ws->tableEnd;
+ void* end = (BYTE *)alloc + bytes;
+ void* top = ws->allocStart;
+
+ DEBUGLOG(5, "cwksp: reserving %p table %zd bytes, %zd bytes remaining",
+ alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
+ assert((bytes & (sizeof(U32)-1)) == 0);
+ ZSTD_cwksp_internal_advance_phase(ws, phase);
+ ZSTD_cwksp_assert_internal_consistency(ws);
+ assert(end <= top);
+ if (end > top) {
+ DEBUGLOG(4, "cwksp: table alloc failed!");
+ ws->allocFailed = 1;
+ return NULL;
+ }
+ ws->tableEnd = end;
+
+
+ return alloc;
+}
+
+/*
+ * Aligned on sizeof(void*).
+ */
+MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) {
+ size_t roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*));
+ void* alloc = ws->objectEnd;
+ void* end = (BYTE*)alloc + roundedBytes;
+
+
+ DEBUGLOG(5,
+ "cwksp: reserving %p object %zd bytes (rounded to %zd), %zd bytes remaining",
+ alloc, bytes, roundedBytes, ZSTD_cwksp_available_space(ws) - roundedBytes);
+ assert(((size_t)alloc & (sizeof(void*)-1)) == 0);
+ assert((bytes & (sizeof(void*)-1)) == 0);
+ ZSTD_cwksp_assert_internal_consistency(ws);
+ /* we must be in the first phase, no advance is possible */
+ if (ws->phase != ZSTD_cwksp_alloc_objects || end > ws->workspaceEnd) {
+ DEBUGLOG(4, "cwksp: object alloc failed!");
+ ws->allocFailed = 1;
+ return NULL;
+ }
+ ws->objectEnd = end;
+ ws->tableEnd = end;
+ ws->tableValidEnd = end;
+
+
+ return alloc;
+}
+
+MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) {
+ DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_dirty");
+
+
+ assert(ws->tableValidEnd >= ws->objectEnd);
+ assert(ws->tableValidEnd <= ws->allocStart);
+ ws->tableValidEnd = ws->objectEnd;
+ ZSTD_cwksp_assert_internal_consistency(ws);
+}
+
+MEM_STATIC void ZSTD_cwksp_mark_tables_clean(ZSTD_cwksp* ws) {
+ DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_clean");
+ assert(ws->tableValidEnd >= ws->objectEnd);
+ assert(ws->tableValidEnd <= ws->allocStart);
+ if (ws->tableValidEnd < ws->tableEnd) {
+ ws->tableValidEnd = ws->tableEnd;
+ }
+ ZSTD_cwksp_assert_internal_consistency(ws);
+}
+
+/*
+ * Zero the part of the allocated tables not already marked clean.
+ */
+MEM_STATIC void ZSTD_cwksp_clean_tables(ZSTD_cwksp* ws) {
+ DEBUGLOG(4, "cwksp: ZSTD_cwksp_clean_tables");
+ assert(ws->tableValidEnd >= ws->objectEnd);
+ assert(ws->tableValidEnd <= ws->allocStart);
+ if (ws->tableValidEnd < ws->tableEnd) {
+ ZSTD_memset(ws->tableValidEnd, 0, (BYTE*)ws->tableEnd - (BYTE*)ws->tableValidEnd);
+ }
+ ZSTD_cwksp_mark_tables_clean(ws);
+}
+
+/*
+ * Invalidates table allocations.
+ * All other allocations remain valid.
+ */
+MEM_STATIC void ZSTD_cwksp_clear_tables(ZSTD_cwksp* ws) {
+ DEBUGLOG(4, "cwksp: clearing tables!");
+
+
+ ws->tableEnd = ws->objectEnd;
+ ZSTD_cwksp_assert_internal_consistency(ws);
+}
+
+/*
+ * Invalidates all buffer, aligned, and table allocations.
+ * Object allocations remain valid.
+ */
+MEM_STATIC void ZSTD_cwksp_clear(ZSTD_cwksp* ws) {
+ DEBUGLOG(4, "cwksp: clearing!");
+
+
+
+ ws->tableEnd = ws->objectEnd;
+ ws->allocStart = ws->workspaceEnd;
+ ws->allocFailed = 0;
+ if (ws->phase > ZSTD_cwksp_alloc_buffers) {
+ ws->phase = ZSTD_cwksp_alloc_buffers;
+ }
+ ZSTD_cwksp_assert_internal_consistency(ws);
+}
+
+/*
+ * The provided workspace takes ownership of the buffer [start, start+size).
+ * Any existing values in the workspace are ignored (the previously managed
+ * buffer, if present, must be separately freed).
+ */
+MEM_STATIC void ZSTD_cwksp_init(ZSTD_cwksp* ws, void* start, size_t size, ZSTD_cwksp_static_alloc_e isStatic) {
+ DEBUGLOG(4, "cwksp: init'ing workspace with %zd bytes", size);
+ assert(((size_t)start & (sizeof(void*)-1)) == 0); /* ensure correct alignment */
+ ws->workspace = start;
+ ws->workspaceEnd = (BYTE*)start + size;
+ ws->objectEnd = ws->workspace;
+ ws->tableValidEnd = ws->objectEnd;
+ ws->phase = ZSTD_cwksp_alloc_objects;
+ ws->isStatic = isStatic;
+ ZSTD_cwksp_clear(ws);
+ ws->workspaceOversizedDuration = 0;
+ ZSTD_cwksp_assert_internal_consistency(ws);
+}
+
+MEM_STATIC size_t ZSTD_cwksp_create(ZSTD_cwksp* ws, size_t size, ZSTD_customMem customMem) {
+ void* workspace = ZSTD_customMalloc(size, customMem);
+ DEBUGLOG(4, "cwksp: creating new workspace with %zd bytes", size);
+ RETURN_ERROR_IF(workspace == NULL, memory_allocation, "NULL pointer!");
+ ZSTD_cwksp_init(ws, workspace, size, ZSTD_cwksp_dynamic_alloc);
+ return 0;
+}
+
+MEM_STATIC void ZSTD_cwksp_free(ZSTD_cwksp* ws, ZSTD_customMem customMem) {
+ void *ptr = ws->workspace;
+ DEBUGLOG(4, "cwksp: freeing workspace");
+ ZSTD_memset(ws, 0, sizeof(ZSTD_cwksp));
+ ZSTD_customFree(ptr, customMem);
+}
+
+/*
+ * Moves the management of a workspace from one cwksp to another. The src cwksp
+ * is left in an invalid state (src must be re-init()'ed before it's used again).
+ */
+MEM_STATIC void ZSTD_cwksp_move(ZSTD_cwksp* dst, ZSTD_cwksp* src) {
+ *dst = *src;
+ ZSTD_memset(src, 0, sizeof(ZSTD_cwksp));
+}
+
+MEM_STATIC size_t ZSTD_cwksp_sizeof(const ZSTD_cwksp* ws) {
+ return (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->workspace);
+}
+
+MEM_STATIC size_t ZSTD_cwksp_used(const ZSTD_cwksp* ws) {
+ return (size_t)((BYTE*)ws->tableEnd - (BYTE*)ws->workspace)
+ + (size_t)((BYTE*)ws->workspaceEnd - (BYTE*)ws->allocStart);
+}
+
+MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) {
+ return ws->allocFailed;
+}
+
+/*-*************************************
+* Functions Checking Free Space
+***************************************/
+
+MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws) {
+ return (size_t)((BYTE*)ws->allocStart - (BYTE*)ws->tableEnd);
+}
+
+MEM_STATIC int ZSTD_cwksp_check_available(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
+ return ZSTD_cwksp_available_space(ws) >= additionalNeededSpace;
+}
+
+MEM_STATIC int ZSTD_cwksp_check_too_large(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
+ return ZSTD_cwksp_check_available(
+ ws, additionalNeededSpace * ZSTD_WORKSPACETOOLARGE_FACTOR);
+}
+
+MEM_STATIC int ZSTD_cwksp_check_wasteful(ZSTD_cwksp* ws, size_t additionalNeededSpace) {
+ return ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)
+ && ws->workspaceOversizedDuration > ZSTD_WORKSPACETOOLARGE_MAXDURATION;
+}
+
+MEM_STATIC void ZSTD_cwksp_bump_oversized_duration(
+ ZSTD_cwksp* ws, size_t additionalNeededSpace) {
+ if (ZSTD_cwksp_check_too_large(ws, additionalNeededSpace)) {
+ ws->workspaceOversizedDuration++;
+ } else {
+ ws->workspaceOversizedDuration = 0;
+ }
+}
+
+
+#endif /* ZSTD_CWKSP_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include "zstd_compress_internal.h"
+#include "zstd_double_fast.h"
+
+
+void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
+ void const* end, ZSTD_dictTableLoadMethod_e dtlm)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashLarge = ms->hashTable;
+ U32 const hBitsL = cParams->hashLog;
+ U32 const mls = cParams->minMatch;
+ U32* const hashSmall = ms->chainTable;
+ U32 const hBitsS = cParams->chainLog;
+ const BYTE* const base = ms->window.base;
+ const BYTE* ip = base + ms->nextToUpdate;
+ const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
+ const U32 fastHashFillStep = 3;
+
+ /* Always insert every fastHashFillStep position into the hash tables.
+ * Insert the other positions into the large hash table if their entry
+ * is empty.
+ */
+ for (; ip + fastHashFillStep - 1 <= iend; ip += fastHashFillStep) {
+ U32 const curr = (U32)(ip - base);
+ U32 i;
+ for (i = 0; i < fastHashFillStep; ++i) {
+ size_t const smHash = ZSTD_hashPtr(ip + i, hBitsS, mls);
+ size_t const lgHash = ZSTD_hashPtr(ip + i, hBitsL, 8);
+ if (i == 0)
+ hashSmall[smHash] = curr + i;
+ if (i == 0 || hashLarge[lgHash] == 0)
+ hashLarge[lgHash] = curr + i;
+ /* Only load extra positions for ZSTD_dtlm_full */
+ if (dtlm == ZSTD_dtlm_fast)
+ break;
+ } }
+}
+
+
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_compressBlock_doubleFast_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize,
+ U32 const mls /* template */, ZSTD_dictMode_e const dictMode)
+{
+ ZSTD_compressionParameters const* cParams = &ms->cParams;
+ U32* const hashLong = ms->hashTable;
+ const U32 hBitsL = cParams->hashLog;
+ U32* const hashSmall = ms->chainTable;
+ const U32 hBitsS = cParams->chainLog;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
+ /* presumes that, if there is a dictionary, it must be using Attach mode */
+ const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
+ const BYTE* const prefixLowest = base + prefixLowestIndex;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - HASH_READ_SIZE;
+ U32 offset_1=rep[0], offset_2=rep[1];
+ U32 offsetSaved = 0;
+
+ const ZSTD_matchState_t* const dms = ms->dictMatchState;
+ const ZSTD_compressionParameters* const dictCParams =
+ dictMode == ZSTD_dictMatchState ?
+ &dms->cParams : NULL;
+ const U32* const dictHashLong = dictMode == ZSTD_dictMatchState ?
+ dms->hashTable : NULL;
+ const U32* const dictHashSmall = dictMode == ZSTD_dictMatchState ?
+ dms->chainTable : NULL;
+ const U32 dictStartIndex = dictMode == ZSTD_dictMatchState ?
+ dms->window.dictLimit : 0;
+ const BYTE* const dictBase = dictMode == ZSTD_dictMatchState ?
+ dms->window.base : NULL;
+ const BYTE* const dictStart = dictMode == ZSTD_dictMatchState ?
+ dictBase + dictStartIndex : NULL;
+ const BYTE* const dictEnd = dictMode == ZSTD_dictMatchState ?
+ dms->window.nextSrc : NULL;
+ const U32 dictIndexDelta = dictMode == ZSTD_dictMatchState ?
+ prefixLowestIndex - (U32)(dictEnd - dictBase) :
+ 0;
+ const U32 dictHBitsL = dictMode == ZSTD_dictMatchState ?
+ dictCParams->hashLog : hBitsL;
+ const U32 dictHBitsS = dictMode == ZSTD_dictMatchState ?
+ dictCParams->chainLog : hBitsS;
+ const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictStart));
+
+ DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_generic");
+
+ assert(dictMode == ZSTD_noDict || dictMode == ZSTD_dictMatchState);
+
+ /* if a dictionary is attached, it must be within window range */
+ if (dictMode == ZSTD_dictMatchState) {
+ assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex);
+ }
+
+ /* init */
+ ip += (dictAndPrefixLength == 0);
+ if (dictMode == ZSTD_noDict) {
+ U32 const curr = (U32)(ip - base);
+ U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog);
+ U32 const maxRep = curr - windowLow;
+ if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
+ if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
+ }
+ if (dictMode == ZSTD_dictMatchState) {
+ /* dictMatchState repCode checks don't currently handle repCode == 0
+ * disabling. */
+ assert(offset_1 <= dictAndPrefixLength);
+ assert(offset_2 <= dictAndPrefixLength);
+ }
+
+ /* Main Search Loop */
+ while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
+ size_t mLength;
+ U32 offset;
+ size_t const h2 = ZSTD_hashPtr(ip, hBitsL, 8);
+ size_t const h = ZSTD_hashPtr(ip, hBitsS, mls);
+ size_t const dictHL = ZSTD_hashPtr(ip, dictHBitsL, 8);
+ size_t const dictHS = ZSTD_hashPtr(ip, dictHBitsS, mls);
+ U32 const curr = (U32)(ip-base);
+ U32 const matchIndexL = hashLong[h2];
+ U32 matchIndexS = hashSmall[h];
+ const BYTE* matchLong = base + matchIndexL;
+ const BYTE* match = base + matchIndexS;
+ const U32 repIndex = curr + 1 - offset_1;
+ const BYTE* repMatch = (dictMode == ZSTD_dictMatchState
+ && repIndex < prefixLowestIndex) ?
+ dictBase + (repIndex - dictIndexDelta) :
+ base + repIndex;
+ hashLong[h2] = hashSmall[h] = curr; /* update hash tables */
+
+ /* check dictMatchState repcode */
+ if (dictMode == ZSTD_dictMatchState
+ && ((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
+ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
+ const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
+ mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ goto _match_stored;
+ }
+
+ /* check noDict repcode */
+ if ( dictMode == ZSTD_noDict
+ && ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
+ mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ goto _match_stored;
+ }
+
+ if (matchIndexL > prefixLowestIndex) {
+ /* check prefix long match */
+ if (MEM_read64(matchLong) == MEM_read64(ip)) {
+ mLength = ZSTD_count(ip+8, matchLong+8, iend) + 8;
+ offset = (U32)(ip-matchLong);
+ while (((ip>anchor) & (matchLong>prefixLowest)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
+ goto _match_found;
+ }
+ } else if (dictMode == ZSTD_dictMatchState) {
+ /* check dictMatchState long match */
+ U32 const dictMatchIndexL = dictHashLong[dictHL];
+ const BYTE* dictMatchL = dictBase + dictMatchIndexL;
+ assert(dictMatchL < dictEnd);
+
+ if (dictMatchL > dictStart && MEM_read64(dictMatchL) == MEM_read64(ip)) {
+ mLength = ZSTD_count_2segments(ip+8, dictMatchL+8, iend, dictEnd, prefixLowest) + 8;
+ offset = (U32)(curr - dictMatchIndexL - dictIndexDelta);
+ while (((ip>anchor) & (dictMatchL>dictStart)) && (ip[-1] == dictMatchL[-1])) { ip--; dictMatchL--; mLength++; } /* catch up */
+ goto _match_found;
+ } }
+
+ if (matchIndexS > prefixLowestIndex) {
+ /* check prefix short match */
+ if (MEM_read32(match) == MEM_read32(ip)) {
+ goto _search_next_long;
+ }
+ } else if (dictMode == ZSTD_dictMatchState) {
+ /* check dictMatchState short match */
+ U32 const dictMatchIndexS = dictHashSmall[dictHS];
+ match = dictBase + dictMatchIndexS;
+ matchIndexS = dictMatchIndexS + dictIndexDelta;
+
+ if (match > dictStart && MEM_read32(match) == MEM_read32(ip)) {
+ goto _search_next_long;
+ } }
+
+ ip += ((ip-anchor) >> kSearchStrength) + 1;
+#if defined(__aarch64__)
+ PREFETCH_L1(ip+256);
+#endif
+ continue;
+
+_search_next_long:
+
+ { size_t const hl3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
+ size_t const dictHLNext = ZSTD_hashPtr(ip+1, dictHBitsL, 8);
+ U32 const matchIndexL3 = hashLong[hl3];
+ const BYTE* matchL3 = base + matchIndexL3;
+ hashLong[hl3] = curr + 1;
+
+ /* check prefix long +1 match */
+ if (matchIndexL3 > prefixLowestIndex) {
+ if (MEM_read64(matchL3) == MEM_read64(ip+1)) {
+ mLength = ZSTD_count(ip+9, matchL3+8, iend) + 8;
+ ip++;
+ offset = (U32)(ip-matchL3);
+ while (((ip>anchor) & (matchL3>prefixLowest)) && (ip[-1] == matchL3[-1])) { ip--; matchL3--; mLength++; } /* catch up */
+ goto _match_found;
+ }
+ } else if (dictMode == ZSTD_dictMatchState) {
+ /* check dict long +1 match */
+ U32 const dictMatchIndexL3 = dictHashLong[dictHLNext];
+ const BYTE* dictMatchL3 = dictBase + dictMatchIndexL3;
+ assert(dictMatchL3 < dictEnd);
+ if (dictMatchL3 > dictStart && MEM_read64(dictMatchL3) == MEM_read64(ip+1)) {
+ mLength = ZSTD_count_2segments(ip+1+8, dictMatchL3+8, iend, dictEnd, prefixLowest) + 8;
+ ip++;
+ offset = (U32)(curr + 1 - dictMatchIndexL3 - dictIndexDelta);
+ while (((ip>anchor) & (dictMatchL3>dictStart)) && (ip[-1] == dictMatchL3[-1])) { ip--; dictMatchL3--; mLength++; } /* catch up */
+ goto _match_found;
+ } } }
+
+ /* if no long +1 match, explore the short match we found */
+ if (dictMode == ZSTD_dictMatchState && matchIndexS < prefixLowestIndex) {
+ mLength = ZSTD_count_2segments(ip+4, match+4, iend, dictEnd, prefixLowest) + 4;
+ offset = (U32)(curr - matchIndexS);
+ while (((ip>anchor) & (match>dictStart)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
+ } else {
+ mLength = ZSTD_count(ip+4, match+4, iend) + 4;
+ offset = (U32)(ip - match);
+ while (((ip>anchor) & (match>prefixLowest)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
+ }
+
+_match_found:
+ offset_2 = offset_1;
+ offset_1 = offset;
+
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+
+_match_stored:
+ /* match found */
+ ip += mLength;
+ anchor = ip;
+
+ if (ip <= ilimit) {
+ /* Complementary insertion */
+ /* done after iLimit test, as candidates could be > iend-8 */
+ { U32 const indexToInsert = curr+2;
+ hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert;
+ hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
+ hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert;
+ hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base);
+ }
+
+ /* check immediate repcode */
+ if (dictMode == ZSTD_dictMatchState) {
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex2 = current2 - offset_2;
+ const BYTE* repMatch2 = dictMode == ZSTD_dictMatchState
+ && repIndex2 < prefixLowestIndex ?
+ dictBase + repIndex2 - dictIndexDelta :
+ base + repIndex2;
+ if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
+ && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend;
+ size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4;
+ U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
+ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
+ hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
+ ip += repLength2;
+ anchor = ip;
+ continue;
+ }
+ break;
+ } }
+
+ if (dictMode == ZSTD_noDict) {
+ while ( (ip <= ilimit)
+ && ( (offset_2>0)
+ & (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
+ /* store sequence */
+ size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
+ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */
+ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
+ hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, rLength-MINMATCH);
+ ip += rLength;
+ anchor = ip;
+ continue; /* faster when present ... (?) */
+ } } }
+ } /* while (ip < ilimit) */
+
+ /* save reps for next block */
+ rep[0] = offset_1 ? offset_1 : offsetSaved;
+ rep[1] = offset_2 ? offset_2 : offsetSaved;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_doubleFast(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ const U32 mls = ms->cParams.minMatch;
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_noDict);
+ case 5 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_noDict);
+ case 6 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_noDict);
+ case 7 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_noDict);
+ }
+}
+
+
+size_t ZSTD_compressBlock_doubleFast_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ const U32 mls = ms->cParams.minMatch;
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_dictMatchState);
+ case 5 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_dictMatchState);
+ case 6 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_dictMatchState);
+ case 7 :
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_dictMatchState);
+ }
+}
+
+
+static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize,
+ U32 const mls /* template */)
+{
+ ZSTD_compressionParameters const* cParams = &ms->cParams;
+ U32* const hashLong = ms->hashTable;
+ U32 const hBitsL = cParams->hashLog;
+ U32* const hashSmall = ms->chainTable;
+ U32 const hBitsS = cParams->chainLog;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - 8;
+ const BYTE* const base = ms->window.base;
+ const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
+ const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog);
+ const U32 dictStartIndex = lowLimit;
+ const U32 dictLimit = ms->window.dictLimit;
+ const U32 prefixStartIndex = (dictLimit > lowLimit) ? dictLimit : lowLimit;
+ const BYTE* const prefixStart = base + prefixStartIndex;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const BYTE* const dictStart = dictBase + dictStartIndex;
+ const BYTE* const dictEnd = dictBase + prefixStartIndex;
+ U32 offset_1=rep[0], offset_2=rep[1];
+
+ DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_extDict_generic (srcSize=%zu)", srcSize);
+
+ /* if extDict is invalidated due to maxDistance, switch to "regular" variant */
+ if (prefixStartIndex == dictStartIndex)
+ return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, mls, ZSTD_noDict);
+
+ /* Search Loop */
+ while (ip < ilimit) { /* < instead of <=, because (ip+1) */
+ const size_t hSmall = ZSTD_hashPtr(ip, hBitsS, mls);
+ const U32 matchIndex = hashSmall[hSmall];
+ const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base;
+ const BYTE* match = matchBase + matchIndex;
+
+ const size_t hLong = ZSTD_hashPtr(ip, hBitsL, 8);
+ const U32 matchLongIndex = hashLong[hLong];
+ const BYTE* const matchLongBase = matchLongIndex < prefixStartIndex ? dictBase : base;
+ const BYTE* matchLong = matchLongBase + matchLongIndex;
+
+ const U32 curr = (U32)(ip-base);
+ const U32 repIndex = curr + 1 - offset_1; /* offset_1 expected <= curr +1 */
+ const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ size_t mLength;
+ hashSmall[hSmall] = hashLong[hLong] = curr; /* update hash table */
+
+ if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */
+ & (repIndex > dictStartIndex))
+ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
+ const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
+ mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ } else {
+ if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) {
+ const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend;
+ const BYTE* const lowMatchPtr = matchLongIndex < prefixStartIndex ? dictStart : prefixStart;
+ U32 offset;
+ mLength = ZSTD_count_2segments(ip+8, matchLong+8, iend, matchEnd, prefixStart) + 8;
+ offset = curr - matchLongIndex;
+ while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
+ offset_2 = offset_1;
+ offset_1 = offset;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+
+ } else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) {
+ size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
+ U32 const matchIndex3 = hashLong[h3];
+ const BYTE* const match3Base = matchIndex3 < prefixStartIndex ? dictBase : base;
+ const BYTE* match3 = match3Base + matchIndex3;
+ U32 offset;
+ hashLong[h3] = curr + 1;
+ if ( (matchIndex3 > dictStartIndex) && (MEM_read64(match3) == MEM_read64(ip+1)) ) {
+ const BYTE* const matchEnd = matchIndex3 < prefixStartIndex ? dictEnd : iend;
+ const BYTE* const lowMatchPtr = matchIndex3 < prefixStartIndex ? dictStart : prefixStart;
+ mLength = ZSTD_count_2segments(ip+9, match3+8, iend, matchEnd, prefixStart) + 8;
+ ip++;
+ offset = curr+1 - matchIndex3;
+ while (((ip>anchor) & (match3>lowMatchPtr)) && (ip[-1] == match3[-1])) { ip--; match3--; mLength++; } /* catch up */
+ } else {
+ const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
+ const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
+ mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
+ offset = curr - matchIndex;
+ while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
+ }
+ offset_2 = offset_1;
+ offset_1 = offset;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+
+ } else {
+ ip += ((ip-anchor) >> kSearchStrength) + 1;
+ continue;
+ } }
+
+ /* move to next sequence start */
+ ip += mLength;
+ anchor = ip;
+
+ if (ip <= ilimit) {
+ /* Complementary insertion */
+ /* done after iLimit test, as candidates could be > iend-8 */
+ { U32 const indexToInsert = curr+2;
+ hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert;
+ hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
+ hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert;
+ hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base);
+ }
+
+ /* check immediate repcode */
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex2 = current2 - offset_2;
+ const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
+ if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */
+ & (repIndex2 > dictStartIndex))
+ && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
+ size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
+ U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
+ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
+ hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
+ ip += repLength2;
+ anchor = ip;
+ continue;
+ }
+ break;
+ } } }
+
+ /* save reps for next block */
+ rep[0] = offset_1;
+ rep[1] = offset_2;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_doubleFast_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ U32 const mls = ms->cParams.minMatch;
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
+ case 5 :
+ return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
+ case 6 :
+ return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
+ case 7 :
+ return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
+ }
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_DOUBLE_FAST_H
+#define ZSTD_DOUBLE_FAST_H
+
+
+#include "../common/mem.h" /* U32 */
+#include "zstd_compress_internal.h" /* ZSTD_CCtx, size_t */
+
+void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
+ void const* end, ZSTD_dictTableLoadMethod_e dtlm);
+size_t ZSTD_compressBlock_doubleFast(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_doubleFast_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_doubleFast_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+
+
+#endif /* ZSTD_DOUBLE_FAST_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include "zstd_compress_internal.h" /* ZSTD_hashPtr, ZSTD_count, ZSTD_storeSeq */
+#include "zstd_fast.h"
+
+
+void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
+ const void* const end,
+ ZSTD_dictTableLoadMethod_e dtlm)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hBits = cParams->hashLog;
+ U32 const mls = cParams->minMatch;
+ const BYTE* const base = ms->window.base;
+ const BYTE* ip = base + ms->nextToUpdate;
+ const BYTE* const iend = ((const BYTE*)end) - HASH_READ_SIZE;
+ const U32 fastHashFillStep = 3;
+
+ /* Always insert every fastHashFillStep position into the hash table.
+ * Insert the other positions if their hash entry is empty.
+ */
+ for ( ; ip + fastHashFillStep < iend + 2; ip += fastHashFillStep) {
+ U32 const curr = (U32)(ip - base);
+ size_t const hash0 = ZSTD_hashPtr(ip, hBits, mls);
+ hashTable[hash0] = curr;
+ if (dtlm == ZSTD_dtlm_fast) continue;
+ /* Only load extra positions for ZSTD_dtlm_full */
+ { U32 p;
+ for (p = 1; p < fastHashFillStep; ++p) {
+ size_t const hash = ZSTD_hashPtr(ip + p, hBits, mls);
+ if (hashTable[hash] == 0) { /* not yet filled */
+ hashTable[hash] = curr + p;
+ } } } }
+}
+
+
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_compressBlock_fast_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize,
+ U32 const mls)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hlog = cParams->hashLog;
+ /* support stepSize of 0 */
+ size_t const stepSize = cParams->targetLength + !(cParams->targetLength) + 1;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const istart = (const BYTE*)src;
+ /* We check ip0 (ip + 0) and ip1 (ip + 1) each loop */
+ const BYTE* ip0 = istart;
+ const BYTE* ip1;
+ const BYTE* anchor = istart;
+ const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
+ const U32 prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
+ const BYTE* const prefixStart = base + prefixStartIndex;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - HASH_READ_SIZE;
+ U32 offset_1=rep[0], offset_2=rep[1];
+ U32 offsetSaved = 0;
+
+ /* init */
+ DEBUGLOG(5, "ZSTD_compressBlock_fast_generic");
+ ip0 += (ip0 == prefixStart);
+ ip1 = ip0 + 1;
+ { U32 const curr = (U32)(ip0 - base);
+ U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog);
+ U32 const maxRep = curr - windowLow;
+ if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
+ if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
+ }
+
+ /* Main Search Loop */
+#ifdef __INTEL_COMPILER
+ /* From intel 'The vector pragma indicates that the loop should be
+ * vectorized if it is legal to do so'. Can be used together with
+ * #pragma ivdep (but have opted to exclude that because intel
+ * warns against using it).*/
+ #pragma vector always
+#endif
+ while (ip1 < ilimit) { /* < instead of <=, because check at ip0+2 */
+ size_t mLength;
+ BYTE const* ip2 = ip0 + 2;
+ size_t const h0 = ZSTD_hashPtr(ip0, hlog, mls);
+ U32 const val0 = MEM_read32(ip0);
+ size_t const h1 = ZSTD_hashPtr(ip1, hlog, mls);
+ U32 const val1 = MEM_read32(ip1);
+ U32 const current0 = (U32)(ip0-base);
+ U32 const current1 = (U32)(ip1-base);
+ U32 const matchIndex0 = hashTable[h0];
+ U32 const matchIndex1 = hashTable[h1];
+ BYTE const* repMatch = ip2 - offset_1;
+ const BYTE* match0 = base + matchIndex0;
+ const BYTE* match1 = base + matchIndex1;
+ U32 offcode;
+
+#if defined(__aarch64__)
+ PREFETCH_L1(ip0+256);
+#endif
+
+ hashTable[h0] = current0; /* update hash table */
+ hashTable[h1] = current1; /* update hash table */
+
+ assert(ip0 + 1 == ip1);
+
+ if ((offset_1 > 0) & (MEM_read32(repMatch) == MEM_read32(ip2))) {
+ mLength = (ip2[-1] == repMatch[-1]) ? 1 : 0;
+ ip0 = ip2 - mLength;
+ match0 = repMatch - mLength;
+ mLength += 4;
+ offcode = 0;
+ goto _match;
+ }
+ if ((matchIndex0 > prefixStartIndex) && MEM_read32(match0) == val0) {
+ /* found a regular match */
+ goto _offset;
+ }
+ if ((matchIndex1 > prefixStartIndex) && MEM_read32(match1) == val1) {
+ /* found a regular match after one literal */
+ ip0 = ip1;
+ match0 = match1;
+ goto _offset;
+ }
+ { size_t const step = ((size_t)(ip0-anchor) >> (kSearchStrength - 1)) + stepSize;
+ assert(step >= 2);
+ ip0 += step;
+ ip1 += step;
+ continue;
+ }
+_offset: /* Requires: ip0, match0 */
+ /* Compute the offset code */
+ offset_2 = offset_1;
+ offset_1 = (U32)(ip0-match0);
+ offcode = offset_1 + ZSTD_REP_MOVE;
+ mLength = 4;
+ /* Count the backwards match length */
+ while (((ip0>anchor) & (match0>prefixStart))
+ && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */
+
+_match: /* Requires: ip0, match0, offcode */
+ /* Count the forward length */
+ mLength += ZSTD_count(ip0+mLength, match0+mLength, iend);
+ ZSTD_storeSeq(seqStore, (size_t)(ip0-anchor), anchor, iend, offcode, mLength-MINMATCH);
+ /* match found */
+ ip0 += mLength;
+ anchor = ip0;
+
+ if (ip0 <= ilimit) {
+ /* Fill Table */
+ assert(base+current0+2 > istart); /* check base overflow */
+ hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */
+ hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);
+
+ if (offset_2 > 0) { /* offset_2==0 means offset_2 is invalidated */
+ while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
+ /* store sequence */
+ size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
+ { U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
+ hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
+ ip0 += rLength;
+ ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, 0 /*offCode*/, rLength-MINMATCH);
+ anchor = ip0;
+ continue; /* faster when present (confirmed on gcc-8) ... (?) */
+ } } }
+ ip1 = ip0 + 1;
+ }
+
+ /* save reps for next block */
+ rep[0] = offset_1 ? offset_1 : offsetSaved;
+ rep[1] = offset_2 ? offset_2 : offsetSaved;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_fast(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ U32 const mls = ms->cParams.minMatch;
+ assert(ms->dictMatchState == NULL);
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 4);
+ case 5 :
+ return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 5);
+ case 6 :
+ return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 6);
+ case 7 :
+ return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 7);
+ }
+}
+
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_compressBlock_fast_dictMatchState_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize, U32 const mls)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hlog = cParams->hashLog;
+ /* support stepSize of 0 */
+ U32 const stepSize = cParams->targetLength + !(cParams->targetLength);
+ const BYTE* const base = ms->window.base;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const U32 prefixStartIndex = ms->window.dictLimit;
+ const BYTE* const prefixStart = base + prefixStartIndex;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - HASH_READ_SIZE;
+ U32 offset_1=rep[0], offset_2=rep[1];
+ U32 offsetSaved = 0;
+
+ const ZSTD_matchState_t* const dms = ms->dictMatchState;
+ const ZSTD_compressionParameters* const dictCParams = &dms->cParams ;
+ const U32* const dictHashTable = dms->hashTable;
+ const U32 dictStartIndex = dms->window.dictLimit;
+ const BYTE* const dictBase = dms->window.base;
+ const BYTE* const dictStart = dictBase + dictStartIndex;
+ const BYTE* const dictEnd = dms->window.nextSrc;
+ const U32 dictIndexDelta = prefixStartIndex - (U32)(dictEnd - dictBase);
+ const U32 dictAndPrefixLength = (U32)(ip - prefixStart + dictEnd - dictStart);
+ const U32 dictHLog = dictCParams->hashLog;
+
+ /* if a dictionary is still attached, it necessarily means that
+ * it is within window size. So we just check it. */
+ const U32 maxDistance = 1U << cParams->windowLog;
+ const U32 endIndex = (U32)((size_t)(ip - base) + srcSize);
+ assert(endIndex - prefixStartIndex <= maxDistance);
+ (void)maxDistance; (void)endIndex; /* these variables are not used when assert() is disabled */
+
+ /* ensure there will be no underflow
+ * when translating a dict index into a local index */
+ assert(prefixStartIndex >= (U32)(dictEnd - dictBase));
+
+ /* init */
+ DEBUGLOG(5, "ZSTD_compressBlock_fast_dictMatchState_generic");
+ ip += (dictAndPrefixLength == 0);
+ /* dictMatchState repCode checks don't currently handle repCode == 0
+ * disabling. */
+ assert(offset_1 <= dictAndPrefixLength);
+ assert(offset_2 <= dictAndPrefixLength);
+
+ /* Main Search Loop */
+ while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
+ size_t mLength;
+ size_t const h = ZSTD_hashPtr(ip, hlog, mls);
+ U32 const curr = (U32)(ip-base);
+ U32 const matchIndex = hashTable[h];
+ const BYTE* match = base + matchIndex;
+ const U32 repIndex = curr + 1 - offset_1;
+ const BYTE* repMatch = (repIndex < prefixStartIndex) ?
+ dictBase + (repIndex - dictIndexDelta) :
+ base + repIndex;
+ hashTable[h] = curr; /* update hash table */
+
+ if ( ((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex isn't overlapping dict + prefix */
+ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
+ const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
+ mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ } else if ( (matchIndex <= prefixStartIndex) ) {
+ size_t const dictHash = ZSTD_hashPtr(ip, dictHLog, mls);
+ U32 const dictMatchIndex = dictHashTable[dictHash];
+ const BYTE* dictMatch = dictBase + dictMatchIndex;
+ if (dictMatchIndex <= dictStartIndex ||
+ MEM_read32(dictMatch) != MEM_read32(ip)) {
+ assert(stepSize >= 1);
+ ip += ((ip-anchor) >> kSearchStrength) + stepSize;
+ continue;
+ } else {
+ /* found a dict match */
+ U32 const offset = (U32)(curr-dictMatchIndex-dictIndexDelta);
+ mLength = ZSTD_count_2segments(ip+4, dictMatch+4, iend, dictEnd, prefixStart) + 4;
+ while (((ip>anchor) & (dictMatch>dictStart))
+ && (ip[-1] == dictMatch[-1])) {
+ ip--; dictMatch--; mLength++;
+ } /* catch up */
+ offset_2 = offset_1;
+ offset_1 = offset;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ }
+ } else if (MEM_read32(match) != MEM_read32(ip)) {
+ /* it's not a match, and we're not going to check the dictionary */
+ assert(stepSize >= 1);
+ ip += ((ip-anchor) >> kSearchStrength) + stepSize;
+ continue;
+ } else {
+ /* found a regular match */
+ U32 const offset = (U32)(ip-match);
+ mLength = ZSTD_count(ip+4, match+4, iend) + 4;
+ while (((ip>anchor) & (match>prefixStart))
+ && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
+ offset_2 = offset_1;
+ offset_1 = offset;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ }
+
+ /* match found */
+ ip += mLength;
+ anchor = ip;
+
+ if (ip <= ilimit) {
+ /* Fill Table */
+ assert(base+curr+2 > istart); /* check base overflow */
+ hashTable[ZSTD_hashPtr(base+curr+2, hlog, mls)] = curr+2; /* here because curr+2 could be > iend-8 */
+ hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);
+
+ /* check immediate repcode */
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex2 = current2 - offset_2;
+ const BYTE* repMatch2 = repIndex2 < prefixStartIndex ?
+ dictBase - dictIndexDelta + repIndex2 :
+ base + repIndex2;
+ if ( ((U32)((prefixStartIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
+ && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
+ size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
+ U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
+ hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
+ ip += repLength2;
+ anchor = ip;
+ continue;
+ }
+ break;
+ }
+ }
+ }
+
+ /* save reps for next block */
+ rep[0] = offset_1 ? offset_1 : offsetSaved;
+ rep[1] = offset_2 ? offset_2 : offsetSaved;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+size_t ZSTD_compressBlock_fast_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ U32 const mls = ms->cParams.minMatch;
+ assert(ms->dictMatchState != NULL);
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 4);
+ case 5 :
+ return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 5);
+ case 6 :
+ return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 6);
+ case 7 :
+ return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 7);
+ }
+}
+
+
+static size_t ZSTD_compressBlock_fast_extDict_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize, U32 const mls)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hlog = cParams->hashLog;
+ /* support stepSize of 0 */
+ U32 const stepSize = cParams->targetLength + !(cParams->targetLength);
+ const BYTE* const base = ms->window.base;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
+ const U32 lowLimit = ZSTD_getLowestMatchIndex(ms, endIndex, cParams->windowLog);
+ const U32 dictStartIndex = lowLimit;
+ const BYTE* const dictStart = dictBase + dictStartIndex;
+ const U32 dictLimit = ms->window.dictLimit;
+ const U32 prefixStartIndex = dictLimit < lowLimit ? lowLimit : dictLimit;
+ const BYTE* const prefixStart = base + prefixStartIndex;
+ const BYTE* const dictEnd = dictBase + prefixStartIndex;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - 8;
+ U32 offset_1=rep[0], offset_2=rep[1];
+
+ DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic (offset_1=%u)", offset_1);
+
+ /* switch to "regular" variant if extDict is invalidated due to maxDistance */
+ if (prefixStartIndex == dictStartIndex)
+ return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, mls);
+
+ /* Search Loop */
+ while (ip < ilimit) { /* < instead of <=, because (ip+1) */
+ const size_t h = ZSTD_hashPtr(ip, hlog, mls);
+ const U32 matchIndex = hashTable[h];
+ const BYTE* const matchBase = matchIndex < prefixStartIndex ? dictBase : base;
+ const BYTE* match = matchBase + matchIndex;
+ const U32 curr = (U32)(ip-base);
+ const U32 repIndex = curr + 1 - offset_1;
+ const BYTE* const repBase = repIndex < prefixStartIndex ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ hashTable[h] = curr; /* update hash table */
+ DEBUGLOG(7, "offset_1 = %u , curr = %u", offset_1, curr);
+ assert(offset_1 <= curr +1); /* check repIndex */
+
+ if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex))
+ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
+ const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
+ size_t const rLength = ZSTD_count_2segments(ip+1 +4, repMatch +4, iend, repMatchEnd, prefixStart) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, rLength-MINMATCH);
+ ip += rLength;
+ anchor = ip;
+ } else {
+ if ( (matchIndex < dictStartIndex) ||
+ (MEM_read32(match) != MEM_read32(ip)) ) {
+ assert(stepSize >= 1);
+ ip += ((ip-anchor) >> kSearchStrength) + stepSize;
+ continue;
+ }
+ { const BYTE* const matchEnd = matchIndex < prefixStartIndex ? dictEnd : iend;
+ const BYTE* const lowMatchPtr = matchIndex < prefixStartIndex ? dictStart : prefixStart;
+ U32 const offset = curr - matchIndex;
+ size_t mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
+ while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
+ offset_2 = offset_1; offset_1 = offset; /* update offset history */
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ip += mLength;
+ anchor = ip;
+ } }
+
+ if (ip <= ilimit) {
+ /* Fill Table */
+ hashTable[ZSTD_hashPtr(base+curr+2, hlog, mls)] = curr+2;
+ hashTable[ZSTD_hashPtr(ip-2, hlog, mls)] = (U32)(ip-2-base);
+ /* check immediate repcode */
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex2 = current2 - offset_2;
+ const BYTE* const repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
+ if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (repIndex2 > dictStartIndex)) /* intentional overflow */
+ && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
+ size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
+ { U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; } /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0 /*litlen*/, anchor, iend, 0 /*offcode*/, repLength2-MINMATCH);
+ hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
+ ip += repLength2;
+ anchor = ip;
+ continue;
+ }
+ break;
+ } } }
+
+ /* save reps for next block */
+ rep[0] = offset_1;
+ rep[1] = offset_2;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_fast_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ U32 const mls = ms->cParams.minMatch;
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
+ case 5 :
+ return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
+ case 6 :
+ return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
+ case 7 :
+ return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
+ }
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_FAST_H
+#define ZSTD_FAST_H
+
+
+#include "../common/mem.h" /* U32 */
+#include "zstd_compress_internal.h"
+
+void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
+ void const* end, ZSTD_dictTableLoadMethod_e dtlm);
+size_t ZSTD_compressBlock_fast(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_fast_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_fast_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+
+#endif /* ZSTD_FAST_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include "zstd_compress_internal.h"
+#include "zstd_lazy.h"
+
+
+/*-*************************************
+* Binary Tree search
+***************************************/
+
+static void
+ZSTD_updateDUBT(ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* iend,
+ U32 mls)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hashLog = cParams->hashLog;
+
+ U32* const bt = ms->chainTable;
+ U32 const btLog = cParams->chainLog - 1;
+ U32 const btMask = (1 << btLog) - 1;
+
+ const BYTE* const base = ms->window.base;
+ U32 const target = (U32)(ip - base);
+ U32 idx = ms->nextToUpdate;
+
+ if (idx != target)
+ DEBUGLOG(7, "ZSTD_updateDUBT, from %u to %u (dictLimit:%u)",
+ idx, target, ms->window.dictLimit);
+ assert(ip + 8 <= iend); /* condition for ZSTD_hashPtr */
+ (void)iend;
+
+ assert(idx >= ms->window.dictLimit); /* condition for valid base+idx */
+ for ( ; idx < target ; idx++) {
+ size_t const h = ZSTD_hashPtr(base + idx, hashLog, mls); /* assumption : ip + 8 <= iend */
+ U32 const matchIndex = hashTable[h];
+
+ U32* const nextCandidatePtr = bt + 2*(idx&btMask);
+ U32* const sortMarkPtr = nextCandidatePtr + 1;
+
+ DEBUGLOG(8, "ZSTD_updateDUBT: insert %u", idx);
+ hashTable[h] = idx; /* Update Hash Table */
+ *nextCandidatePtr = matchIndex; /* update BT like a chain */
+ *sortMarkPtr = ZSTD_DUBT_UNSORTED_MARK;
+ }
+ ms->nextToUpdate = target;
+}
+
+
+/* ZSTD_insertDUBT1() :
+ * sort one already inserted but unsorted position
+ * assumption : curr >= btlow == (curr - btmask)
+ * doesn't fail */
+static void
+ZSTD_insertDUBT1(ZSTD_matchState_t* ms,
+ U32 curr, const BYTE* inputEnd,
+ U32 nbCompares, U32 btLow,
+ const ZSTD_dictMode_e dictMode)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const bt = ms->chainTable;
+ U32 const btLog = cParams->chainLog - 1;
+ U32 const btMask = (1 << btLog) - 1;
+ size_t commonLengthSmaller=0, commonLengthLarger=0;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const ip = (curr>=dictLimit) ? base + curr : dictBase + curr;
+ const BYTE* const iend = (curr>=dictLimit) ? inputEnd : dictBase + dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ const BYTE* match;
+ U32* smallerPtr = bt + 2*(curr&btMask);
+ U32* largerPtr = smallerPtr + 1;
+ U32 matchIndex = *smallerPtr; /* this candidate is unsorted : next sorted candidate is reached through *smallerPtr, while *largerPtr contains previous unsorted candidate (which is already saved and can be overwritten) */
+ U32 dummy32; /* to be nullified at the end */
+ U32 const windowValid = ms->window.lowLimit;
+ U32 const maxDistance = 1U << cParams->windowLog;
+ U32 const windowLow = (curr - windowValid > maxDistance) ? curr - maxDistance : windowValid;
+
+
+ DEBUGLOG(8, "ZSTD_insertDUBT1(%u) (dictLimit=%u, lowLimit=%u)",
+ curr, dictLimit, windowLow);
+ assert(curr >= btLow);
+ assert(ip < iend); /* condition for ZSTD_count */
+
+ for (; nbCompares && (matchIndex > windowLow); --nbCompares) {
+ U32* const nextPtr = bt + 2*(matchIndex & btMask);
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ assert(matchIndex < curr);
+ /* note : all candidates are now supposed sorted,
+ * but it's still possible to have nextPtr[1] == ZSTD_DUBT_UNSORTED_MARK
+ * when a real index has the same value as ZSTD_DUBT_UNSORTED_MARK */
+
+ if ( (dictMode != ZSTD_extDict)
+ || (matchIndex+matchLength >= dictLimit) /* both in current segment*/
+ || (curr < dictLimit) /* both in extDict */) {
+ const BYTE* const mBase = ( (dictMode != ZSTD_extDict)
+ || (matchIndex+matchLength >= dictLimit)) ?
+ base : dictBase;
+ assert( (matchIndex+matchLength >= dictLimit) /* might be wrong if extDict is incorrectly set to 0 */
+ || (curr < dictLimit) );
+ match = mBase + matchIndex;
+ matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
+ } else {
+ match = dictBase + matchIndex;
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
+ if (matchIndex+matchLength >= dictLimit)
+ match = base + matchIndex; /* preparation for next read of match[matchLength] */
+ }
+
+ DEBUGLOG(8, "ZSTD_insertDUBT1: comparing %u with %u : found %u common bytes ",
+ curr, matchIndex, (U32)matchLength);
+
+ if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
+ break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */
+ }
+
+ if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */
+ /* match is smaller than current */
+ *smallerPtr = matchIndex; /* update smaller idx */
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */
+ DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is smaller : next => %u",
+ matchIndex, btLow, nextPtr[1]);
+ smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */
+ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */
+ } else {
+ /* match is larger than current */
+ *largerPtr = matchIndex;
+ commonLengthLarger = matchLength;
+ if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */
+ DEBUGLOG(8, "ZSTD_insertDUBT1: %u (>btLow=%u) is larger => %u",
+ matchIndex, btLow, nextPtr[0]);
+ largerPtr = nextPtr;
+ matchIndex = nextPtr[0];
+ } }
+
+ *smallerPtr = *largerPtr = 0;
+}
+
+
+static size_t
+ZSTD_DUBT_findBetterDictMatch (
+ ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iend,
+ size_t* offsetPtr,
+ size_t bestLength,
+ U32 nbCompares,
+ U32 const mls,
+ const ZSTD_dictMode_e dictMode)
+{
+ const ZSTD_matchState_t * const dms = ms->dictMatchState;
+ const ZSTD_compressionParameters* const dmsCParams = &dms->cParams;
+ const U32 * const dictHashTable = dms->hashTable;
+ U32 const hashLog = dmsCParams->hashLog;
+ size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
+ U32 dictMatchIndex = dictHashTable[h];
+
+ const BYTE* const base = ms->window.base;
+ const BYTE* const prefixStart = base + ms->window.dictLimit;
+ U32 const curr = (U32)(ip-base);
+ const BYTE* const dictBase = dms->window.base;
+ const BYTE* const dictEnd = dms->window.nextSrc;
+ U32 const dictHighLimit = (U32)(dms->window.nextSrc - dms->window.base);
+ U32 const dictLowLimit = dms->window.lowLimit;
+ U32 const dictIndexDelta = ms->window.lowLimit - dictHighLimit;
+
+ U32* const dictBt = dms->chainTable;
+ U32 const btLog = dmsCParams->chainLog - 1;
+ U32 const btMask = (1 << btLog) - 1;
+ U32 const btLow = (btMask >= dictHighLimit - dictLowLimit) ? dictLowLimit : dictHighLimit - btMask;
+
+ size_t commonLengthSmaller=0, commonLengthLarger=0;
+
+ (void)dictMode;
+ assert(dictMode == ZSTD_dictMatchState);
+
+ for (; nbCompares && (dictMatchIndex > dictLowLimit); --nbCompares) {
+ U32* const nextPtr = dictBt + 2*(dictMatchIndex & btMask);
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ const BYTE* match = dictBase + dictMatchIndex;
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
+ if (dictMatchIndex+matchLength >= dictHighLimit)
+ match = base + dictMatchIndex + dictIndexDelta; /* to prepare for next usage of match[matchLength] */
+
+ if (matchLength > bestLength) {
+ U32 matchIndex = dictMatchIndex + dictIndexDelta;
+ if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) ) {
+ DEBUGLOG(9, "ZSTD_DUBT_findBetterDictMatch(%u) : found better match length %u -> %u and offsetCode %u -> %u (dictMatchIndex %u, matchIndex %u)",
+ curr, (U32)bestLength, (U32)matchLength, (U32)*offsetPtr, ZSTD_REP_MOVE + curr - matchIndex, dictMatchIndex, matchIndex);
+ bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + curr - matchIndex;
+ }
+ if (ip+matchLength == iend) { /* reached end of input : ip[matchLength] is not valid, no way to know if it's larger or smaller than match */
+ break; /* drop, to guarantee consistency (miss a little bit of compression) */
+ }
+ }
+
+ if (match[matchLength] < ip[matchLength]) {
+ if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
+ } else {
+ /* match is larger than current */
+ if (dictMatchIndex <= btLow) { break; } /* beyond tree size, stop the search */
+ commonLengthLarger = matchLength;
+ dictMatchIndex = nextPtr[0];
+ }
+ }
+
+ if (bestLength >= MINMATCH) {
+ U32 const mIndex = curr - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
+ DEBUGLOG(8, "ZSTD_DUBT_findBetterDictMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
+ curr, (U32)bestLength, (U32)*offsetPtr, mIndex);
+ }
+ return bestLength;
+
+}
+
+
+static size_t
+ZSTD_DUBT_findBestMatch(ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iend,
+ size_t* offsetPtr,
+ U32 const mls,
+ const ZSTD_dictMode_e dictMode)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hashLog = cParams->hashLog;
+ size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
+ U32 matchIndex = hashTable[h];
+
+ const BYTE* const base = ms->window.base;
+ U32 const curr = (U32)(ip-base);
+ U32 const windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog);
+
+ U32* const bt = ms->chainTable;
+ U32 const btLog = cParams->chainLog - 1;
+ U32 const btMask = (1 << btLog) - 1;
+ U32 const btLow = (btMask >= curr) ? 0 : curr - btMask;
+ U32 const unsortLimit = MAX(btLow, windowLow);
+
+ U32* nextCandidate = bt + 2*(matchIndex&btMask);
+ U32* unsortedMark = bt + 2*(matchIndex&btMask) + 1;
+ U32 nbCompares = 1U << cParams->searchLog;
+ U32 nbCandidates = nbCompares;
+ U32 previousCandidate = 0;
+
+ DEBUGLOG(7, "ZSTD_DUBT_findBestMatch (%u) ", curr);
+ assert(ip <= iend-8); /* required for h calculation */
+ assert(dictMode != ZSTD_dedicatedDictSearch);
+
+ /* reach end of unsorted candidates list */
+ while ( (matchIndex > unsortLimit)
+ && (*unsortedMark == ZSTD_DUBT_UNSORTED_MARK)
+ && (nbCandidates > 1) ) {
+ DEBUGLOG(8, "ZSTD_DUBT_findBestMatch: candidate %u is unsorted",
+ matchIndex);
+ *unsortedMark = previousCandidate; /* the unsortedMark becomes a reversed chain, to move up back to original position */
+ previousCandidate = matchIndex;
+ matchIndex = *nextCandidate;
+ nextCandidate = bt + 2*(matchIndex&btMask);
+ unsortedMark = bt + 2*(matchIndex&btMask) + 1;
+ nbCandidates --;
+ }
+
+ /* nullify last candidate if it's still unsorted
+ * simplification, detrimental to compression ratio, beneficial for speed */
+ if ( (matchIndex > unsortLimit)
+ && (*unsortedMark==ZSTD_DUBT_UNSORTED_MARK) ) {
+ DEBUGLOG(7, "ZSTD_DUBT_findBestMatch: nullify last unsorted candidate %u",
+ matchIndex);
+ *nextCandidate = *unsortedMark = 0;
+ }
+
+ /* batch sort stacked candidates */
+ matchIndex = previousCandidate;
+ while (matchIndex) { /* will end on matchIndex == 0 */
+ U32* const nextCandidateIdxPtr = bt + 2*(matchIndex&btMask) + 1;
+ U32 const nextCandidateIdx = *nextCandidateIdxPtr;
+ ZSTD_insertDUBT1(ms, matchIndex, iend,
+ nbCandidates, unsortLimit, dictMode);
+ matchIndex = nextCandidateIdx;
+ nbCandidates++;
+ }
+
+ /* find longest match */
+ { size_t commonLengthSmaller = 0, commonLengthLarger = 0;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ U32* smallerPtr = bt + 2*(curr&btMask);
+ U32* largerPtr = bt + 2*(curr&btMask) + 1;
+ U32 matchEndIdx = curr + 8 + 1;
+ U32 dummy32; /* to be nullified at the end */
+ size_t bestLength = 0;
+
+ matchIndex = hashTable[h];
+ hashTable[h] = curr; /* Update Hash Table */
+
+ for (; nbCompares && (matchIndex > windowLow); --nbCompares) {
+ U32* const nextPtr = bt + 2*(matchIndex & btMask);
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ const BYTE* match;
+
+ if ((dictMode != ZSTD_extDict) || (matchIndex+matchLength >= dictLimit)) {
+ match = base + matchIndex;
+ matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
+ } else {
+ match = dictBase + matchIndex;
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
+ if (matchIndex+matchLength >= dictLimit)
+ match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
+ }
+
+ if (matchLength > bestLength) {
+ if (matchLength > matchEndIdx - matchIndex)
+ matchEndIdx = matchIndex + (U32)matchLength;
+ if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) )
+ bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + curr - matchIndex;
+ if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
+ if (dictMode == ZSTD_dictMatchState) {
+ nbCompares = 0; /* in addition to avoiding checking any
+ * further in this loop, make sure we
+ * skip checking in the dictionary. */
+ }
+ break; /* drop, to guarantee consistency (miss a little bit of compression) */
+ }
+ }
+
+ if (match[matchLength] < ip[matchLength]) {
+ /* match is smaller than current */
+ *smallerPtr = matchIndex; /* update smaller idx */
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
+ matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
+ } else {
+ /* match is larger than current */
+ *largerPtr = matchIndex;
+ commonLengthLarger = matchLength;
+ if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ largerPtr = nextPtr;
+ matchIndex = nextPtr[0];
+ } }
+
+ *smallerPtr = *largerPtr = 0;
+
+ assert(nbCompares <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */
+ if (dictMode == ZSTD_dictMatchState && nbCompares) {
+ bestLength = ZSTD_DUBT_findBetterDictMatch(
+ ms, ip, iend,
+ offsetPtr, bestLength, nbCompares,
+ mls, dictMode);
+ }
+
+ assert(matchEndIdx > curr+8); /* ensure nextToUpdate is increased */
+ ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
+ if (bestLength >= MINMATCH) {
+ U32 const mIndex = curr - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
+ DEBUGLOG(8, "ZSTD_DUBT_findBestMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
+ curr, (U32)bestLength, (U32)*offsetPtr, mIndex);
+ }
+ return bestLength;
+ }
+}
+
+
+/* ZSTD_BtFindBestMatch() : Tree updater, providing best match */
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_BtFindBestMatch( ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iLimit,
+ size_t* offsetPtr,
+ const U32 mls /* template */,
+ const ZSTD_dictMode_e dictMode)
+{
+ DEBUGLOG(7, "ZSTD_BtFindBestMatch");
+ if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */
+ ZSTD_updateDUBT(ms, ip, iLimit, mls);
+ return ZSTD_DUBT_findBestMatch(ms, ip, iLimit, offsetPtr, mls, dictMode);
+}
+
+
+static size_t
+ZSTD_BtFindBestMatch_selectMLS ( ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
+ case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
+ case 7 :
+ case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
+ }
+}
+
+
+static size_t ZSTD_BtFindBestMatch_dictMatchState_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
+ case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
+ case 7 :
+ case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
+ }
+}
+
+
+static size_t ZSTD_BtFindBestMatch_extDict_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
+ case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
+ case 7 :
+ case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
+ }
+}
+
+
+
+/* *********************************
+* Hash Chain
+***********************************/
+#define NEXT_IN_CHAIN(d, mask) chainTable[(d) & (mask)]
+
+/* Update chains up to ip (excluded)
+ Assumption : always within prefix (i.e. not within extDict) */
+FORCE_INLINE_TEMPLATE U32 ZSTD_insertAndFindFirstIndex_internal(
+ ZSTD_matchState_t* ms,
+ const ZSTD_compressionParameters* const cParams,
+ const BYTE* ip, U32 const mls)
+{
+ U32* const hashTable = ms->hashTable;
+ const U32 hashLog = cParams->hashLog;
+ U32* const chainTable = ms->chainTable;
+ const U32 chainMask = (1 << cParams->chainLog) - 1;
+ const BYTE* const base = ms->window.base;
+ const U32 target = (U32)(ip - base);
+ U32 idx = ms->nextToUpdate;
+
+ while(idx < target) { /* catch up */
+ size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls);
+ NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
+ hashTable[h] = idx;
+ idx++;
+ }
+
+ ms->nextToUpdate = target;
+ return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
+}
+
+U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip) {
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch);
+}
+
+void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip)
+{
+ const BYTE* const base = ms->window.base;
+ U32 const target = (U32)(ip - base);
+ U32* const hashTable = ms->hashTable;
+ U32* const chainTable = ms->chainTable;
+ U32 const chainSize = 1 << ms->cParams.chainLog;
+ U32 idx = ms->nextToUpdate;
+ U32 const minChain = chainSize < target ? target - chainSize : idx;
+ U32 const bucketSize = 1 << ZSTD_LAZY_DDSS_BUCKET_LOG;
+ U32 const cacheSize = bucketSize - 1;
+ U32 const chainAttempts = (1 << ms->cParams.searchLog) - cacheSize;
+ U32 const chainLimit = chainAttempts > 255 ? 255 : chainAttempts;
+
+ /* We know the hashtable is oversized by a factor of `bucketSize`.
+ * We are going to temporarily pretend `bucketSize == 1`, keeping only a
+ * single entry. We will use the rest of the space to construct a temporary
+ * chaintable.
+ */
+ U32 const hashLog = ms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG;
+ U32* const tmpHashTable = hashTable;
+ U32* const tmpChainTable = hashTable + ((size_t)1 << hashLog);
+ U32 const tmpChainSize = ((1 << ZSTD_LAZY_DDSS_BUCKET_LOG) - 1) << hashLog;
+ U32 const tmpMinChain = tmpChainSize < target ? target - tmpChainSize : idx;
+
+ U32 hashIdx;
+
+ assert(ms->cParams.chainLog <= 24);
+ assert(ms->cParams.hashLog >= ms->cParams.chainLog);
+ assert(idx != 0);
+ assert(tmpMinChain <= minChain);
+
+ /* fill conventional hash table and conventional chain table */
+ for ( ; idx < target; idx++) {
+ U32 const h = (U32)ZSTD_hashPtr(base + idx, hashLog, ms->cParams.minMatch);
+ if (idx >= tmpMinChain) {
+ tmpChainTable[idx - tmpMinChain] = hashTable[h];
+ }
+ tmpHashTable[h] = idx;
+ }
+
+ /* sort chains into ddss chain table */
+ {
+ U32 chainPos = 0;
+ for (hashIdx = 0; hashIdx < (1U << hashLog); hashIdx++) {
+ U32 count;
+ U32 countBeyondMinChain = 0;
+ U32 i = tmpHashTable[hashIdx];
+ for (count = 0; i >= tmpMinChain && count < cacheSize; count++) {
+ /* skip through the chain to the first position that won't be
+ * in the hash cache bucket */
+ if (i < minChain) {
+ countBeyondMinChain++;
+ }
+ i = tmpChainTable[i - tmpMinChain];
+ }
+ if (count == cacheSize) {
+ for (count = 0; count < chainLimit;) {
+ if (i < minChain) {
+ if (!i || countBeyondMinChain++ > cacheSize) {
+ /* only allow pulling `cacheSize` number of entries
+ * into the cache or chainTable beyond `minChain`,
+ * to replace the entries pulled out of the
+ * chainTable into the cache. This lets us reach
+ * back further without increasing the total number
+ * of entries in the chainTable, guaranteeing the
+ * DDSS chain table will fit into the space
+ * allocated for the regular one. */
+ break;
+ }
+ }
+ chainTable[chainPos++] = i;
+ count++;
+ if (i < tmpMinChain) {
+ break;
+ }
+ i = tmpChainTable[i - tmpMinChain];
+ }
+ } else {
+ count = 0;
+ }
+ if (count) {
+ tmpHashTable[hashIdx] = ((chainPos - count) << 8) + count;
+ } else {
+ tmpHashTable[hashIdx] = 0;
+ }
+ }
+ assert(chainPos <= chainSize); /* I believe this is guaranteed... */
+ }
+
+ /* move chain pointers into the last entry of each hash bucket */
+ for (hashIdx = (1 << hashLog); hashIdx; ) {
+ U32 const bucketIdx = --hashIdx << ZSTD_LAZY_DDSS_BUCKET_LOG;
+ U32 const chainPackedPointer = tmpHashTable[hashIdx];
+ U32 i;
+ for (i = 0; i < cacheSize; i++) {
+ hashTable[bucketIdx + i] = 0;
+ }
+ hashTable[bucketIdx + bucketSize - 1] = chainPackedPointer;
+ }
+
+ /* fill the buckets of the hash table */
+ for (idx = ms->nextToUpdate; idx < target; idx++) {
+ U32 const h = (U32)ZSTD_hashPtr(base + idx, hashLog, ms->cParams.minMatch)
+ << ZSTD_LAZY_DDSS_BUCKET_LOG;
+ U32 i;
+ /* Shift hash cache down 1. */
+ for (i = cacheSize - 1; i; i--)
+ hashTable[h + i] = hashTable[h + i - 1];
+ hashTable[h] = idx;
+ }
+
+ ms->nextToUpdate = target;
+}
+
+
+/* inlining is important to hardwire a hot branch (template emulation) */
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_HcFindBestMatch_generic (
+ ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iLimit,
+ size_t* offsetPtr,
+ const U32 mls, const ZSTD_dictMode_e dictMode)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const chainTable = ms->chainTable;
+ const U32 chainSize = (1 << cParams->chainLog);
+ const U32 chainMask = chainSize-1;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const U32 curr = (U32)(ip-base);
+ const U32 maxDistance = 1U << cParams->windowLog;
+ const U32 lowestValid = ms->window.lowLimit;
+ const U32 withinMaxDistance = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
+ const U32 isDictionary = (ms->loadedDictEnd != 0);
+ const U32 lowLimit = isDictionary ? lowestValid : withinMaxDistance;
+ const U32 minChain = curr > chainSize ? curr - chainSize : 0;
+ U32 nbAttempts = 1U << cParams->searchLog;
+ size_t ml=4-1;
+
+ const ZSTD_matchState_t* const dms = ms->dictMatchState;
+ const U32 ddsHashLog = dictMode == ZSTD_dedicatedDictSearch
+ ? dms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG : 0;
+ const size_t ddsIdx = dictMode == ZSTD_dedicatedDictSearch
+ ? ZSTD_hashPtr(ip, ddsHashLog, mls) << ZSTD_LAZY_DDSS_BUCKET_LOG : 0;
+
+ U32 matchIndex;
+
+ if (dictMode == ZSTD_dedicatedDictSearch) {
+ const U32* entry = &dms->hashTable[ddsIdx];
+ PREFETCH_L1(entry);
+ }
+
+ /* HC4 match finder */
+ matchIndex = ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, mls);
+
+ for ( ; (matchIndex>=lowLimit) & (nbAttempts>0) ; nbAttempts--) {
+ size_t currentMl=0;
+ if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) {
+ const BYTE* const match = base + matchIndex;
+ assert(matchIndex >= dictLimit); /* ensures this is true if dictMode != ZSTD_extDict */
+ if (match[ml] == ip[ml]) /* potentially better */
+ currentMl = ZSTD_count(ip, match, iLimit);
+ } else {
+ const BYTE* const match = dictBase + matchIndex;
+ assert(match+4 <= dictEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4;
+ }
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = curr - matchIndex + ZSTD_REP_MOVE;
+ if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
+ }
+
+ if (matchIndex <= minChain) break;
+ matchIndex = NEXT_IN_CHAIN(matchIndex, chainMask);
+ }
+
+ assert(nbAttempts <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */
+ if (dictMode == ZSTD_dedicatedDictSearch) {
+ const U32 ddsLowestIndex = dms->window.dictLimit;
+ const BYTE* const ddsBase = dms->window.base;
+ const BYTE* const ddsEnd = dms->window.nextSrc;
+ const U32 ddsSize = (U32)(ddsEnd - ddsBase);
+ const U32 ddsIndexDelta = dictLimit - ddsSize;
+ const U32 bucketSize = (1 << ZSTD_LAZY_DDSS_BUCKET_LOG);
+ const U32 bucketLimit = nbAttempts < bucketSize - 1 ? nbAttempts : bucketSize - 1;
+ U32 ddsAttempt;
+
+ for (ddsAttempt = 0; ddsAttempt < bucketSize - 1; ddsAttempt++) {
+ PREFETCH_L1(ddsBase + dms->hashTable[ddsIdx + ddsAttempt]);
+ }
+
+ {
+ U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
+ U32 const chainIndex = chainPackedPointer >> 8;
+
+ PREFETCH_L1(&dms->chainTable[chainIndex]);
+ }
+
+ for (ddsAttempt = 0; ddsAttempt < bucketLimit; ddsAttempt++) {
+ size_t currentMl=0;
+ const BYTE* match;
+ matchIndex = dms->hashTable[ddsIdx + ddsAttempt];
+ match = ddsBase + matchIndex;
+
+ if (!matchIndex) {
+ return ml;
+ }
+
+ /* guaranteed by table construction */
+ (void)ddsLowestIndex;
+ assert(matchIndex >= ddsLowestIndex);
+ assert(match+4 <= ddsEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) {
+ /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
+ }
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = curr - (matchIndex + ddsIndexDelta) + ZSTD_REP_MOVE;
+ if (ip+currentMl == iLimit) {
+ /* best possible, avoids read overflow on next attempt */
+ return ml;
+ }
+ }
+ }
+
+ {
+ U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
+ U32 chainIndex = chainPackedPointer >> 8;
+ U32 const chainLength = chainPackedPointer & 0xFF;
+ U32 const chainAttempts = nbAttempts - ddsAttempt;
+ U32 const chainLimit = chainAttempts > chainLength ? chainLength : chainAttempts;
+ U32 chainAttempt;
+
+ for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++) {
+ PREFETCH_L1(ddsBase + dms->chainTable[chainIndex + chainAttempt]);
+ }
+
+ for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++, chainIndex++) {
+ size_t currentMl=0;
+ const BYTE* match;
+ matchIndex = dms->chainTable[chainIndex];
+ match = ddsBase + matchIndex;
+
+ /* guaranteed by table construction */
+ assert(matchIndex >= ddsLowestIndex);
+ assert(match+4 <= ddsEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) {
+ /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
+ }
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = curr - (matchIndex + ddsIndexDelta) + ZSTD_REP_MOVE;
+ if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
+ }
+ }
+ }
+ } else if (dictMode == ZSTD_dictMatchState) {
+ const U32* const dmsChainTable = dms->chainTable;
+ const U32 dmsChainSize = (1 << dms->cParams.chainLog);
+ const U32 dmsChainMask = dmsChainSize - 1;
+ const U32 dmsLowestIndex = dms->window.dictLimit;
+ const BYTE* const dmsBase = dms->window.base;
+ const BYTE* const dmsEnd = dms->window.nextSrc;
+ const U32 dmsSize = (U32)(dmsEnd - dmsBase);
+ const U32 dmsIndexDelta = dictLimit - dmsSize;
+ const U32 dmsMinChain = dmsSize > dmsChainSize ? dmsSize - dmsChainSize : 0;
+
+ matchIndex = dms->hashTable[ZSTD_hashPtr(ip, dms->cParams.hashLog, mls)];
+
+ for ( ; (matchIndex>=dmsLowestIndex) & (nbAttempts>0) ; nbAttempts--) {
+ size_t currentMl=0;
+ const BYTE* const match = dmsBase + matchIndex;
+ assert(match+4 <= dmsEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dmsEnd, prefixStart) + 4;
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = curr - (matchIndex + dmsIndexDelta) + ZSTD_REP_MOVE;
+ if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
+ }
+
+ if (matchIndex <= dmsMinChain) break;
+
+ matchIndex = dmsChainTable[matchIndex & dmsChainMask];
+ }
+ }
+
+ return ml;
+}
+
+
+FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
+ case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
+ case 7 :
+ case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
+ }
+}
+
+
+static size_t ZSTD_HcFindBestMatch_dictMatchState_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
+ case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
+ case 7 :
+ case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
+ }
+}
+
+
+static size_t ZSTD_HcFindBestMatch_dedicatedDictSearch_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_dedicatedDictSearch);
+ case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_dedicatedDictSearch);
+ case 7 :
+ case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_dedicatedDictSearch);
+ }
+}
+
+
+FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* const iLimit,
+ size_t* offsetPtr)
+{
+ switch(ms->cParams.minMatch)
+ {
+ default : /* includes case 3 */
+ case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
+ case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
+ case 7 :
+ case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
+ }
+}
+
+
+/* *******************************
+* Common parser - lazy strategy
+*********************************/
+typedef enum { search_hashChain, search_binaryTree } searchMethod_e;
+
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_compressBlock_lazy_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore,
+ U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize,
+ const searchMethod_e searchMethod, const U32 depth,
+ ZSTD_dictMode_e const dictMode)
+{
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - 8;
+ const BYTE* const base = ms->window.base;
+ const U32 prefixLowestIndex = ms->window.dictLimit;
+ const BYTE* const prefixLowest = base + prefixLowestIndex;
+
+ typedef size_t (*searchMax_f)(
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
+
+ /*
+ * This table is indexed first by the four ZSTD_dictMode_e values, and then
+ * by the two searchMethod_e values. NULLs are placed for configurations
+ * that should never occur (extDict modes go to the other implementation
+ * below and there is no DDSS for binary tree search yet).
+ */
+ const searchMax_f searchFuncs[4][2] = {
+ {
+ ZSTD_HcFindBestMatch_selectMLS,
+ ZSTD_BtFindBestMatch_selectMLS
+ },
+ {
+ NULL,
+ NULL
+ },
+ {
+ ZSTD_HcFindBestMatch_dictMatchState_selectMLS,
+ ZSTD_BtFindBestMatch_dictMatchState_selectMLS
+ },
+ {
+ ZSTD_HcFindBestMatch_dedicatedDictSearch_selectMLS,
+ NULL
+ }
+ };
+
+ searchMax_f const searchMax = searchFuncs[dictMode][searchMethod == search_binaryTree];
+ U32 offset_1 = rep[0], offset_2 = rep[1], savedOffset=0;
+
+ const int isDMS = dictMode == ZSTD_dictMatchState;
+ const int isDDS = dictMode == ZSTD_dedicatedDictSearch;
+ const int isDxS = isDMS || isDDS;
+ const ZSTD_matchState_t* const dms = ms->dictMatchState;
+ const U32 dictLowestIndex = isDxS ? dms->window.dictLimit : 0;
+ const BYTE* const dictBase = isDxS ? dms->window.base : NULL;
+ const BYTE* const dictLowest = isDxS ? dictBase + dictLowestIndex : NULL;
+ const BYTE* const dictEnd = isDxS ? dms->window.nextSrc : NULL;
+ const U32 dictIndexDelta = isDxS ?
+ prefixLowestIndex - (U32)(dictEnd - dictBase) :
+ 0;
+ const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictLowest));
+
+ assert(searchMax != NULL);
+
+ DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u)", (U32)dictMode);
+
+ /* init */
+ ip += (dictAndPrefixLength == 0);
+ if (dictMode == ZSTD_noDict) {
+ U32 const curr = (U32)(ip - base);
+ U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, ms->cParams.windowLog);
+ U32 const maxRep = curr - windowLow;
+ if (offset_2 > maxRep) savedOffset = offset_2, offset_2 = 0;
+ if (offset_1 > maxRep) savedOffset = offset_1, offset_1 = 0;
+ }
+ if (isDxS) {
+ /* dictMatchState repCode checks don't currently handle repCode == 0
+ * disabling. */
+ assert(offset_1 <= dictAndPrefixLength);
+ assert(offset_2 <= dictAndPrefixLength);
+ }
+
+ /* Match Loop */
+#if defined(__x86_64__)
+ /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
+ * code alignment is perturbed. To fix the instability align the loop on 32-bytes.
+ */
+ __asm__(".p2align 5");
+#endif
+ while (ip < ilimit) {
+ size_t matchLength=0;
+ size_t offset=0;
+ const BYTE* start=ip+1;
+
+ /* check repCode */
+ if (isDxS) {
+ const U32 repIndex = (U32)(ip - base) + 1 - offset_1;
+ const BYTE* repMatch = ((dictMode == ZSTD_dictMatchState || dictMode == ZSTD_dedicatedDictSearch)
+ && repIndex < prefixLowestIndex) ?
+ dictBase + (repIndex - dictIndexDelta) :
+ base + repIndex;
+ if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
+ && (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
+ const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
+ matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
+ if (depth==0) goto _storeSequence;
+ }
+ }
+ if ( dictMode == ZSTD_noDict
+ && ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
+ matchLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
+ if (depth==0) goto _storeSequence;
+ }
+
+ /* first search (depth 0) */
+ { size_t offsetFound = 999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
+ if (ml2 > matchLength)
+ matchLength = ml2, start = ip, offset=offsetFound;
+ }
+
+ if (matchLength < 4) {
+ ip += ((ip-anchor) >> kSearchStrength) + 1; /* jump faster over incompressible sections */
+ continue;
+ }
+
+ /* let's try to find a better solution */
+ if (depth>=1)
+ while (ip<ilimit) {
+ ip ++;
+ if ( (dictMode == ZSTD_noDict)
+ && (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
+ size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
+ int const gain2 = (int)(mlRep * 3);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((mlRep >= 4) && (gain2 > gain1))
+ matchLength = mlRep, offset = 0, start = ip;
+ }
+ if (isDxS) {
+ const U32 repIndex = (U32)(ip - base) - offset_1;
+ const BYTE* repMatch = repIndex < prefixLowestIndex ?
+ dictBase + (repIndex - dictIndexDelta) :
+ base + repIndex;
+ if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
+ && (MEM_read32(repMatch) == MEM_read32(ip)) ) {
+ const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
+ size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
+ int const gain2 = (int)(mlRep * 3);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((mlRep >= 4) && (gain2 > gain1))
+ matchLength = mlRep, offset = 0, start = ip;
+ }
+ }
+ { size_t offset2=999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offset2);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
+ if ((ml2 >= 4) && (gain2 > gain1)) {
+ matchLength = ml2, offset = offset2, start = ip;
+ continue; /* search a better one */
+ } }
+
+ /* let's find an even better one */
+ if ((depth==2) && (ip<ilimit)) {
+ ip ++;
+ if ( (dictMode == ZSTD_noDict)
+ && (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
+ size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
+ int const gain2 = (int)(mlRep * 4);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((mlRep >= 4) && (gain2 > gain1))
+ matchLength = mlRep, offset = 0, start = ip;
+ }
+ if (isDxS) {
+ const U32 repIndex = (U32)(ip - base) - offset_1;
+ const BYTE* repMatch = repIndex < prefixLowestIndex ?
+ dictBase + (repIndex - dictIndexDelta) :
+ base + repIndex;
+ if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
+ && (MEM_read32(repMatch) == MEM_read32(ip)) ) {
+ const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
+ size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
+ int const gain2 = (int)(mlRep * 4);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((mlRep >= 4) && (gain2 > gain1))
+ matchLength = mlRep, offset = 0, start = ip;
+ }
+ }
+ { size_t offset2=999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offset2);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
+ if ((ml2 >= 4) && (gain2 > gain1)) {
+ matchLength = ml2, offset = offset2, start = ip;
+ continue;
+ } } }
+ break; /* nothing found : store previous solution */
+ }
+
+ /* NOTE:
+ * start[-offset+ZSTD_REP_MOVE-1] is undefined behavior.
+ * (-offset+ZSTD_REP_MOVE-1) is unsigned, and is added to start, which
+ * overflows the pointer, which is undefined behavior.
+ */
+ /* catch up */
+ if (offset) {
+ if (dictMode == ZSTD_noDict) {
+ while ( ((start > anchor) & (start - (offset-ZSTD_REP_MOVE) > prefixLowest))
+ && (start[-1] == (start-(offset-ZSTD_REP_MOVE))[-1]) ) /* only search for offset within prefix */
+ { start--; matchLength++; }
+ }
+ if (isDxS) {
+ U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
+ const BYTE* match = (matchIndex < prefixLowestIndex) ? dictBase + matchIndex - dictIndexDelta : base + matchIndex;
+ const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest;
+ while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
+ }
+ offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
+ }
+ /* store sequence */
+_storeSequence:
+ { size_t const litLength = start - anchor;
+ ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
+ anchor = ip = start + matchLength;
+ }
+
+ /* check immediate repcode */
+ if (isDxS) {
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex = current2 - offset_2;
+ const BYTE* repMatch = repIndex < prefixLowestIndex ?
+ dictBase - dictIndexDelta + repIndex :
+ base + repIndex;
+ if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex) >= 3 /* intentional overflow */)
+ && (MEM_read32(repMatch) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend;
+ matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4;
+ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ ip += matchLength;
+ anchor = ip;
+ continue;
+ }
+ break;
+ }
+ }
+
+ if (dictMode == ZSTD_noDict) {
+ while ( ((ip <= ilimit) & (offset_2>0))
+ && (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) {
+ /* store sequence */
+ matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
+ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ ip += matchLength;
+ anchor = ip;
+ continue; /* faster when present ... (?) */
+ } } }
+
+ /* Save reps for next block */
+ rep[0] = offset_1 ? offset_1 : savedOffset;
+ rep[1] = offset_2 ? offset_2 : savedOffset;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_btlazy2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_lazy2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_lazy(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_greedy(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_btlazy2_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_lazy2_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_lazy_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_greedy_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dictMatchState);
+}
+
+
+size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2, ZSTD_dedicatedDictSearch);
+}
+
+size_t ZSTD_compressBlock_lazy_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1, ZSTD_dedicatedDictSearch);
+}
+
+size_t ZSTD_compressBlock_greedy_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dedicatedDictSearch);
+}
+
+
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_compressBlock_lazy_extDict_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore,
+ U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize,
+ const searchMethod_e searchMethod, const U32 depth)
+{
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - 8;
+ const BYTE* const base = ms->window.base;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const BYTE* const dictStart = dictBase + ms->window.lowLimit;
+ const U32 windowLog = ms->cParams.windowLog;
+
+ typedef size_t (*searchMax_f)(
+ ZSTD_matchState_t* ms,
+ const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
+ searchMax_f searchMax = searchMethod==search_binaryTree ? ZSTD_BtFindBestMatch_extDict_selectMLS : ZSTD_HcFindBestMatch_extDict_selectMLS;
+
+ U32 offset_1 = rep[0], offset_2 = rep[1];
+
+ DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic");
+
+ /* init */
+ ip += (ip == prefixStart);
+
+ /* Match Loop */
+#if defined(__x86_64__)
+ /* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
+ * code alignment is perturbed. To fix the instability align the loop on 32-bytes.
+ */
+ __asm__(".p2align 5");
+#endif
+ while (ip < ilimit) {
+ size_t matchLength=0;
+ size_t offset=0;
+ const BYTE* start=ip+1;
+ U32 curr = (U32)(ip-base);
+
+ /* check repCode */
+ { const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr+1, windowLog);
+ const U32 repIndex = (U32)(curr+1 - offset_1);
+ const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if (MEM_read32(ip+1) == MEM_read32(repMatch)) {
+ /* repcode detected we should take it */
+ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
+ matchLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repEnd, prefixStart) + 4;
+ if (depth==0) goto _storeSequence;
+ } }
+
+ /* first search (depth 0) */
+ { size_t offsetFound = 999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
+ if (ml2 > matchLength)
+ matchLength = ml2, start = ip, offset=offsetFound;
+ }
+
+ if (matchLength < 4) {
+ ip += ((ip-anchor) >> kSearchStrength) + 1; /* jump faster over incompressible sections */
+ continue;
+ }
+
+ /* let's try to find a better solution */
+ if (depth>=1)
+ while (ip<ilimit) {
+ ip ++;
+ curr++;
+ /* check repCode */
+ if (offset) {
+ const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog);
+ const U32 repIndex = (U32)(curr - offset_1);
+ const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if (MEM_read32(ip) == MEM_read32(repMatch)) {
+ /* repcode detected */
+ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
+ size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
+ int const gain2 = (int)(repLength * 3);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((repLength >= 4) && (gain2 > gain1))
+ matchLength = repLength, offset = 0, start = ip;
+ } }
+
+ /* search match, depth 1 */
+ { size_t offset2=999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offset2);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
+ if ((ml2 >= 4) && (gain2 > gain1)) {
+ matchLength = ml2, offset = offset2, start = ip;
+ continue; /* search a better one */
+ } }
+
+ /* let's find an even better one */
+ if ((depth==2) && (ip<ilimit)) {
+ ip ++;
+ curr++;
+ /* check repCode */
+ if (offset) {
+ const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog);
+ const U32 repIndex = (U32)(curr - offset_1);
+ const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if (MEM_read32(ip) == MEM_read32(repMatch)) {
+ /* repcode detected */
+ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
+ size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
+ int const gain2 = (int)(repLength * 4);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ if ((repLength >= 4) && (gain2 > gain1))
+ matchLength = repLength, offset = 0, start = ip;
+ } }
+
+ /* search match, depth 2 */
+ { size_t offset2=999999999;
+ size_t const ml2 = searchMax(ms, ip, iend, &offset2);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
+ if ((ml2 >= 4) && (gain2 > gain1)) {
+ matchLength = ml2, offset = offset2, start = ip;
+ continue;
+ } } }
+ break; /* nothing found : store previous solution */
+ }
+
+ /* catch up */
+ if (offset) {
+ U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
+ const BYTE* match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex;
+ const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
+ while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
+ offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
+ }
+
+ /* store sequence */
+_storeSequence:
+ { size_t const litLength = start - anchor;
+ ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
+ anchor = ip = start + matchLength;
+ }
+
+ /* check immediate repcode */
+ while (ip <= ilimit) {
+ const U32 repCurrent = (U32)(ip-base);
+ const U32 windowLow = ZSTD_getLowestMatchIndex(ms, repCurrent, windowLog);
+ const U32 repIndex = repCurrent - offset_2;
+ const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
+ const BYTE* const repMatch = repBase + repIndex;
+ if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if (MEM_read32(ip) == MEM_read32(repMatch)) {
+ /* repcode detected we should take it */
+ const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
+ matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
+ offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ ip += matchLength;
+ anchor = ip;
+ continue; /* faster when present ... (?) */
+ }
+ break;
+ } }
+
+ /* Save reps for next block */
+ rep[0] = offset_1;
+ rep[1] = offset_2;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_greedy_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0);
+}
+
+size_t ZSTD_compressBlock_lazy_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 1);
+}
+
+size_t ZSTD_compressBlock_lazy2_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 2);
+}
+
+size_t ZSTD_compressBlock_btlazy2_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_LAZY_H
+#define ZSTD_LAZY_H
+
+
+#include "zstd_compress_internal.h"
+
+/*
+ * Dedicated Dictionary Search Structure bucket log. In the
+ * ZSTD_dedicatedDictSearch mode, the hashTable has
+ * 2 ** ZSTD_LAZY_DDSS_BUCKET_LOG entries in each bucket, rather than just
+ * one.
+ */
+#define ZSTD_LAZY_DDSS_BUCKET_LOG 2
+
+U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip);
+
+void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip);
+
+void ZSTD_preserveUnsortedMark (U32* const table, U32 const size, U32 const reducerValue); /*! used in ZSTD_reduceIndex(). preemptively increase value of ZSTD_DUBT_UNSORTED_MARK */
+
+size_t ZSTD_compressBlock_btlazy2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+size_t ZSTD_compressBlock_btlazy2_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_dedicatedDictSearch(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+size_t ZSTD_compressBlock_greedy_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_btlazy2_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+
+#endif /* ZSTD_LAZY_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include "zstd_ldm.h"
+
+#include "../common/debug.h"
+#include <linux/xxhash.h>
+#include "zstd_fast.h" /* ZSTD_fillHashTable() */
+#include "zstd_double_fast.h" /* ZSTD_fillDoubleHashTable() */
+#include "zstd_ldm_geartab.h"
+
+#define LDM_BUCKET_SIZE_LOG 3
+#define LDM_MIN_MATCH_LENGTH 64
+#define LDM_HASH_RLOG 7
+
+typedef struct {
+ U64 rolling;
+ U64 stopMask;
+} ldmRollingHashState_t;
+
+/* ZSTD_ldm_gear_init():
+ *
+ * Initializes the rolling hash state such that it will honor the
+ * settings in params. */
+static void ZSTD_ldm_gear_init(ldmRollingHashState_t* state, ldmParams_t const* params)
+{
+ unsigned maxBitsInMask = MIN(params->minMatchLength, 64);
+ unsigned hashRateLog = params->hashRateLog;
+
+ state->rolling = ~(U32)0;
+
+ /* The choice of the splitting criterion is subject to two conditions:
+ * 1. it has to trigger on average every 2^(hashRateLog) bytes;
+ * 2. ideally, it has to depend on a window of minMatchLength bytes.
+ *
+ * In the gear hash algorithm, bit n depends on the last n bytes;
+ * so in order to obtain a good quality splitting criterion it is
+ * preferable to use bits with high weight.
+ *
+ * To match condition 1 we use a mask with hashRateLog bits set
+ * and, because of the previous remark, we make sure these bits
+ * have the highest possible weight while still respecting
+ * condition 2.
+ */
+ if (hashRateLog > 0 && hashRateLog <= maxBitsInMask) {
+ state->stopMask = (((U64)1 << hashRateLog) - 1) << (maxBitsInMask - hashRateLog);
+ } else {
+ /* In this degenerate case we simply honor the hash rate. */
+ state->stopMask = ((U64)1 << hashRateLog) - 1;
+ }
+}
+
+/* ZSTD_ldm_gear_feed():
+ *
+ * Registers in the splits array all the split points found in the first
+ * size bytes following the data pointer. This function terminates when
+ * either all the data has been processed or LDM_BATCH_SIZE splits are
+ * present in the splits array.
+ *
+ * Precondition: The splits array must not be full.
+ * Returns: The number of bytes processed. */
+static size_t ZSTD_ldm_gear_feed(ldmRollingHashState_t* state,
+ BYTE const* data, size_t size,
+ size_t* splits, unsigned* numSplits)
+{
+ size_t n;
+ U64 hash, mask;
+
+ hash = state->rolling;
+ mask = state->stopMask;
+ n = 0;
+
+#define GEAR_ITER_ONCE() do { \
+ hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; \
+ n += 1; \
+ if (UNLIKELY((hash & mask) == 0)) { \
+ splits[*numSplits] = n; \
+ *numSplits += 1; \
+ if (*numSplits == LDM_BATCH_SIZE) \
+ goto done; \
+ } \
+ } while (0)
+
+ while (n + 3 < size) {
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ }
+ while (n < size) {
+ GEAR_ITER_ONCE();
+ }
+
+#undef GEAR_ITER_ONCE
+
+done:
+ state->rolling = hash;
+ return n;
+}
+
+void ZSTD_ldm_adjustParameters(ldmParams_t* params,
+ ZSTD_compressionParameters const* cParams)
+{
+ params->windowLog = cParams->windowLog;
+ ZSTD_STATIC_ASSERT(LDM_BUCKET_SIZE_LOG <= ZSTD_LDM_BUCKETSIZELOG_MAX);
+ DEBUGLOG(4, "ZSTD_ldm_adjustParameters");
+ if (!params->bucketSizeLog) params->bucketSizeLog = LDM_BUCKET_SIZE_LOG;
+ if (!params->minMatchLength) params->minMatchLength = LDM_MIN_MATCH_LENGTH;
+ if (params->hashLog == 0) {
+ params->hashLog = MAX(ZSTD_HASHLOG_MIN, params->windowLog - LDM_HASH_RLOG);
+ assert(params->hashLog <= ZSTD_HASHLOG_MAX);
+ }
+ if (params->hashRateLog == 0) {
+ params->hashRateLog = params->windowLog < params->hashLog
+ ? 0
+ : params->windowLog - params->hashLog;
+ }
+ params->bucketSizeLog = MIN(params->bucketSizeLog, params->hashLog);
+}
+
+size_t ZSTD_ldm_getTableSize(ldmParams_t params)
+{
+ size_t const ldmHSize = ((size_t)1) << params.hashLog;
+ size_t const ldmBucketSizeLog = MIN(params.bucketSizeLog, params.hashLog);
+ size_t const ldmBucketSize = ((size_t)1) << (params.hashLog - ldmBucketSizeLog);
+ size_t const totalSize = ZSTD_cwksp_alloc_size(ldmBucketSize)
+ + ZSTD_cwksp_alloc_size(ldmHSize * sizeof(ldmEntry_t));
+ return params.enableLdm ? totalSize : 0;
+}
+
+size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize)
+{
+ return params.enableLdm ? (maxChunkSize / params.minMatchLength) : 0;
+}
+
+/* ZSTD_ldm_getBucket() :
+ * Returns a pointer to the start of the bucket associated with hash. */
+static ldmEntry_t* ZSTD_ldm_getBucket(
+ ldmState_t* ldmState, size_t hash, ldmParams_t const ldmParams)
+{
+ return ldmState->hashTable + (hash << ldmParams.bucketSizeLog);
+}
+
+/* ZSTD_ldm_insertEntry() :
+ * Insert the entry with corresponding hash into the hash table */
+static void ZSTD_ldm_insertEntry(ldmState_t* ldmState,
+ size_t const hash, const ldmEntry_t entry,
+ ldmParams_t const ldmParams)
+{
+ BYTE* const pOffset = ldmState->bucketOffsets + hash;
+ unsigned const offset = *pOffset;
+
+ *(ZSTD_ldm_getBucket(ldmState, hash, ldmParams) + offset) = entry;
+ *pOffset = (BYTE)((offset + 1) & ((1u << ldmParams.bucketSizeLog) - 1));
+
+}
+
+/* ZSTD_ldm_countBackwardsMatch() :
+ * Returns the number of bytes that match backwards before pIn and pMatch.
+ *
+ * We count only bytes where pMatch >= pBase and pIn >= pAnchor. */
+static size_t ZSTD_ldm_countBackwardsMatch(
+ const BYTE* pIn, const BYTE* pAnchor,
+ const BYTE* pMatch, const BYTE* pMatchBase)
+{
+ size_t matchLength = 0;
+ while (pIn > pAnchor && pMatch > pMatchBase && pIn[-1] == pMatch[-1]) {
+ pIn--;
+ pMatch--;
+ matchLength++;
+ }
+ return matchLength;
+}
+
+/* ZSTD_ldm_countBackwardsMatch_2segments() :
+ * Returns the number of bytes that match backwards from pMatch,
+ * even with the backwards match spanning 2 different segments.
+ *
+ * On reaching `pMatchBase`, start counting from mEnd */
+static size_t ZSTD_ldm_countBackwardsMatch_2segments(
+ const BYTE* pIn, const BYTE* pAnchor,
+ const BYTE* pMatch, const BYTE* pMatchBase,
+ const BYTE* pExtDictStart, const BYTE* pExtDictEnd)
+{
+ size_t matchLength = ZSTD_ldm_countBackwardsMatch(pIn, pAnchor, pMatch, pMatchBase);
+ if (pMatch - matchLength != pMatchBase || pMatchBase == pExtDictStart) {
+ /* If backwards match is entirely in the extDict or prefix, immediately return */
+ return matchLength;
+ }
+ DEBUGLOG(7, "ZSTD_ldm_countBackwardsMatch_2segments: found 2-parts backwards match (length in prefix==%zu)", matchLength);
+ matchLength += ZSTD_ldm_countBackwardsMatch(pIn - matchLength, pAnchor, pExtDictEnd, pExtDictStart);
+ DEBUGLOG(7, "final backwards match length = %zu", matchLength);
+ return matchLength;
+}
+
+/* ZSTD_ldm_fillFastTables() :
+ *
+ * Fills the relevant tables for the ZSTD_fast and ZSTD_dfast strategies.
+ * This is similar to ZSTD_loadDictionaryContent.
+ *
+ * The tables for the other strategies are filled within their
+ * block compressors. */
+static size_t ZSTD_ldm_fillFastTables(ZSTD_matchState_t* ms,
+ void const* end)
+{
+ const BYTE* const iend = (const BYTE*)end;
+
+ switch(ms->cParams.strategy)
+ {
+ case ZSTD_fast:
+ ZSTD_fillHashTable(ms, iend, ZSTD_dtlm_fast);
+ break;
+
+ case ZSTD_dfast:
+ ZSTD_fillDoubleHashTable(ms, iend, ZSTD_dtlm_fast);
+ break;
+
+ case ZSTD_greedy:
+ case ZSTD_lazy:
+ case ZSTD_lazy2:
+ case ZSTD_btlazy2:
+ case ZSTD_btopt:
+ case ZSTD_btultra:
+ case ZSTD_btultra2:
+ break;
+ default:
+ assert(0); /* not possible : not a valid strategy id */
+ }
+
+ return 0;
+}
+
+void ZSTD_ldm_fillHashTable(
+ ldmState_t* ldmState, const BYTE* ip,
+ const BYTE* iend, ldmParams_t const* params)
+{
+ U32 const minMatchLength = params->minMatchLength;
+ U32 const hBits = params->hashLog - params->bucketSizeLog;
+ BYTE const* const base = ldmState->window.base;
+ BYTE const* const istart = ip;
+ ldmRollingHashState_t hashState;
+ size_t* const splits = ldmState->splitIndices;
+ unsigned numSplits;
+
+ DEBUGLOG(5, "ZSTD_ldm_fillHashTable");
+
+ ZSTD_ldm_gear_init(&hashState, params);
+ while (ip < iend) {
+ size_t hashed;
+ unsigned n;
+
+ numSplits = 0;
+ hashed = ZSTD_ldm_gear_feed(&hashState, ip, iend - ip, splits, &numSplits);
+
+ for (n = 0; n < numSplits; n++) {
+ if (ip + splits[n] >= istart + minMatchLength) {
+ BYTE const* const split = ip + splits[n] - minMatchLength;
+ U64 const xxhash = xxh64(split, minMatchLength, 0);
+ U32 const hash = (U32)(xxhash & (((U32)1 << hBits) - 1));
+ ldmEntry_t entry;
+
+ entry.offset = (U32)(split - base);
+ entry.checksum = (U32)(xxhash >> 32);
+ ZSTD_ldm_insertEntry(ldmState, hash, entry, *params);
+ }
+ }
+
+ ip += hashed;
+ }
+}
+
+
+/* ZSTD_ldm_limitTableUpdate() :
+ *
+ * Sets cctx->nextToUpdate to a position corresponding closer to anchor
+ * if it is far way
+ * (after a long match, only update tables a limited amount). */
+static void ZSTD_ldm_limitTableUpdate(ZSTD_matchState_t* ms, const BYTE* anchor)
+{
+ U32 const curr = (U32)(anchor - ms->window.base);
+ if (curr > ms->nextToUpdate + 1024) {
+ ms->nextToUpdate =
+ curr - MIN(512, curr - ms->nextToUpdate - 1024);
+ }
+}
+
+static size_t ZSTD_ldm_generateSequences_internal(
+ ldmState_t* ldmState, rawSeqStore_t* rawSeqStore,
+ ldmParams_t const* params, void const* src, size_t srcSize)
+{
+ /* LDM parameters */
+ int const extDict = ZSTD_window_hasExtDict(ldmState->window);
+ U32 const minMatchLength = params->minMatchLength;
+ U32 const entsPerBucket = 1U << params->bucketSizeLog;
+ U32 const hBits = params->hashLog - params->bucketSizeLog;
+ /* Prefix and extDict parameters */
+ U32 const dictLimit = ldmState->window.dictLimit;
+ U32 const lowestIndex = extDict ? ldmState->window.lowLimit : dictLimit;
+ BYTE const* const base = ldmState->window.base;
+ BYTE const* const dictBase = extDict ? ldmState->window.dictBase : NULL;
+ BYTE const* const dictStart = extDict ? dictBase + lowestIndex : NULL;
+ BYTE const* const dictEnd = extDict ? dictBase + dictLimit : NULL;
+ BYTE const* const lowPrefixPtr = base + dictLimit;
+ /* Input bounds */
+ BYTE const* const istart = (BYTE const*)src;
+ BYTE const* const iend = istart + srcSize;
+ BYTE const* const ilimit = iend - HASH_READ_SIZE;
+ /* Input positions */
+ BYTE const* anchor = istart;
+ BYTE const* ip = istart;
+ /* Rolling hash state */
+ ldmRollingHashState_t hashState;
+ /* Arrays for staged-processing */
+ size_t* const splits = ldmState->splitIndices;
+ ldmMatchCandidate_t* const candidates = ldmState->matchCandidates;
+ unsigned numSplits;
+
+ if (srcSize < minMatchLength)
+ return iend - anchor;
+
+ /* Initialize the rolling hash state with the first minMatchLength bytes */
+ ZSTD_ldm_gear_init(&hashState, params);
+ {
+ size_t n = 0;
+
+ while (n < minMatchLength) {
+ numSplits = 0;
+ n += ZSTD_ldm_gear_feed(&hashState, ip + n, minMatchLength - n,
+ splits, &numSplits);
+ }
+ ip += minMatchLength;
+ }
+
+ while (ip < ilimit) {
+ size_t hashed;
+ unsigned n;
+
+ numSplits = 0;
+ hashed = ZSTD_ldm_gear_feed(&hashState, ip, ilimit - ip,
+ splits, &numSplits);
+
+ for (n = 0; n < numSplits; n++) {
+ BYTE const* const split = ip + splits[n] - minMatchLength;
+ U64 const xxhash = xxh64(split, minMatchLength, 0);
+ U32 const hash = (U32)(xxhash & (((U32)1 << hBits) - 1));
+
+ candidates[n].split = split;
+ candidates[n].hash = hash;
+ candidates[n].checksum = (U32)(xxhash >> 32);
+ candidates[n].bucket = ZSTD_ldm_getBucket(ldmState, hash, *params);
+ PREFETCH_L1(candidates[n].bucket);
+ }
+
+ for (n = 0; n < numSplits; n++) {
+ size_t forwardMatchLength = 0, backwardMatchLength = 0,
+ bestMatchLength = 0, mLength;
+ BYTE const* const split = candidates[n].split;
+ U32 const checksum = candidates[n].checksum;
+ U32 const hash = candidates[n].hash;
+ ldmEntry_t* const bucket = candidates[n].bucket;
+ ldmEntry_t const* cur;
+ ldmEntry_t const* bestEntry = NULL;
+ ldmEntry_t newEntry;
+
+ newEntry.offset = (U32)(split - base);
+ newEntry.checksum = checksum;
+
+ /* If a split point would generate a sequence overlapping with
+ * the previous one, we merely register it in the hash table and
+ * move on */
+ if (split < anchor) {
+ ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params);
+ continue;
+ }
+
+ for (cur = bucket; cur < bucket + entsPerBucket; cur++) {
+ size_t curForwardMatchLength, curBackwardMatchLength,
+ curTotalMatchLength;
+ if (cur->checksum != checksum || cur->offset <= lowestIndex) {
+ continue;
+ }
+ if (extDict) {
+ BYTE const* const curMatchBase =
+ cur->offset < dictLimit ? dictBase : base;
+ BYTE const* const pMatch = curMatchBase + cur->offset;
+ BYTE const* const matchEnd =
+ cur->offset < dictLimit ? dictEnd : iend;
+ BYTE const* const lowMatchPtr =
+ cur->offset < dictLimit ? dictStart : lowPrefixPtr;
+ curForwardMatchLength =
+ ZSTD_count_2segments(split, pMatch, iend, matchEnd, lowPrefixPtr);
+ if (curForwardMatchLength < minMatchLength) {
+ continue;
+ }
+ curBackwardMatchLength = ZSTD_ldm_countBackwardsMatch_2segments(
+ split, anchor, pMatch, lowMatchPtr, dictStart, dictEnd);
+ } else { /* !extDict */
+ BYTE const* const pMatch = base + cur->offset;
+ curForwardMatchLength = ZSTD_count(split, pMatch, iend);
+ if (curForwardMatchLength < minMatchLength) {
+ continue;
+ }
+ curBackwardMatchLength =
+ ZSTD_ldm_countBackwardsMatch(split, anchor, pMatch, lowPrefixPtr);
+ }
+ curTotalMatchLength = curForwardMatchLength + curBackwardMatchLength;
+
+ if (curTotalMatchLength > bestMatchLength) {
+ bestMatchLength = curTotalMatchLength;
+ forwardMatchLength = curForwardMatchLength;
+ backwardMatchLength = curBackwardMatchLength;
+ bestEntry = cur;
+ }
+ }
+
+ /* No match found -- insert an entry into the hash table
+ * and process the next candidate match */
+ if (bestEntry == NULL) {
+ ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params);
+ continue;
+ }
+
+ /* Match found */
+ mLength = forwardMatchLength + backwardMatchLength;
+ {
+ U32 const offset = (U32)(split - base) - bestEntry->offset;
+ rawSeq* const seq = rawSeqStore->seq + rawSeqStore->size;
+
+ /* Out of sequence storage */
+ if (rawSeqStore->size == rawSeqStore->capacity)
+ return ERROR(dstSize_tooSmall);
+ seq->litLength = (U32)(split - backwardMatchLength - anchor);
+ seq->matchLength = (U32)mLength;
+ seq->offset = offset;
+ rawSeqStore->size++;
+ }
+
+ /* Insert the current entry into the hash table --- it must be
+ * done after the previous block to avoid clobbering bestEntry */
+ ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params);
+
+ anchor = split + forwardMatchLength;
+ }
+
+ ip += hashed;
+ }
+
+ return iend - anchor;
+}
+
+/*! ZSTD_ldm_reduceTable() :
+ * reduce table indexes by `reducerValue` */
+static void ZSTD_ldm_reduceTable(ldmEntry_t* const table, U32 const size,
+ U32 const reducerValue)
+{
+ U32 u;
+ for (u = 0; u < size; u++) {
+ if (table[u].offset < reducerValue) table[u].offset = 0;
+ else table[u].offset -= reducerValue;
+ }
+}
+
+size_t ZSTD_ldm_generateSequences(
+ ldmState_t* ldmState, rawSeqStore_t* sequences,
+ ldmParams_t const* params, void const* src, size_t srcSize)
+{
+ U32 const maxDist = 1U << params->windowLog;
+ BYTE const* const istart = (BYTE const*)src;
+ BYTE const* const iend = istart + srcSize;
+ size_t const kMaxChunkSize = 1 << 20;
+ size_t const nbChunks = (srcSize / kMaxChunkSize) + ((srcSize % kMaxChunkSize) != 0);
+ size_t chunk;
+ size_t leftoverSize = 0;
+
+ assert(ZSTD_CHUNKSIZE_MAX >= kMaxChunkSize);
+ /* Check that ZSTD_window_update() has been called for this chunk prior
+ * to passing it to this function.
+ */
+ assert(ldmState->window.nextSrc >= (BYTE const*)src + srcSize);
+ /* The input could be very large (in zstdmt), so it must be broken up into
+ * chunks to enforce the maximum distance and handle overflow correction.
+ */
+ assert(sequences->pos <= sequences->size);
+ assert(sequences->size <= sequences->capacity);
+ for (chunk = 0; chunk < nbChunks && sequences->size < sequences->capacity; ++chunk) {
+ BYTE const* const chunkStart = istart + chunk * kMaxChunkSize;
+ size_t const remaining = (size_t)(iend - chunkStart);
+ BYTE const *const chunkEnd =
+ (remaining < kMaxChunkSize) ? iend : chunkStart + kMaxChunkSize;
+ size_t const chunkSize = chunkEnd - chunkStart;
+ size_t newLeftoverSize;
+ size_t const prevSize = sequences->size;
+
+ assert(chunkStart < iend);
+ /* 1. Perform overflow correction if necessary. */
+ if (ZSTD_window_needOverflowCorrection(ldmState->window, chunkEnd)) {
+ U32 const ldmHSize = 1U << params->hashLog;
+ U32 const correction = ZSTD_window_correctOverflow(
+ &ldmState->window, /* cycleLog */ 0, maxDist, chunkStart);
+ ZSTD_ldm_reduceTable(ldmState->hashTable, ldmHSize, correction);
+ /* invalidate dictionaries on overflow correction */
+ ldmState->loadedDictEnd = 0;
+ }
+ /* 2. We enforce the maximum offset allowed.
+ *
+ * kMaxChunkSize should be small enough that we don't lose too much of
+ * the window through early invalidation.
+ * TODO: * Test the chunk size.
+ * * Try invalidation after the sequence generation and test the
+ * the offset against maxDist directly.
+ *
+ * NOTE: Because of dictionaries + sequence splitting we MUST make sure
+ * that any offset used is valid at the END of the sequence, since it may
+ * be split into two sequences. This condition holds when using
+ * ZSTD_window_enforceMaxDist(), but if we move to checking offsets
+ * against maxDist directly, we'll have to carefully handle that case.
+ */
+ ZSTD_window_enforceMaxDist(&ldmState->window, chunkEnd, maxDist, &ldmState->loadedDictEnd, NULL);
+ /* 3. Generate the sequences for the chunk, and get newLeftoverSize. */
+ newLeftoverSize = ZSTD_ldm_generateSequences_internal(
+ ldmState, sequences, params, chunkStart, chunkSize);
+ if (ZSTD_isError(newLeftoverSize))
+ return newLeftoverSize;
+ /* 4. We add the leftover literals from previous iterations to the first
+ * newly generated sequence, or add the `newLeftoverSize` if none are
+ * generated.
+ */
+ /* Prepend the leftover literals from the last call */
+ if (prevSize < sequences->size) {
+ sequences->seq[prevSize].litLength += (U32)leftoverSize;
+ leftoverSize = newLeftoverSize;
+ } else {
+ assert(newLeftoverSize == chunkSize);
+ leftoverSize += chunkSize;
+ }
+ }
+ return 0;
+}
+
+void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch) {
+ while (srcSize > 0 && rawSeqStore->pos < rawSeqStore->size) {
+ rawSeq* seq = rawSeqStore->seq + rawSeqStore->pos;
+ if (srcSize <= seq->litLength) {
+ /* Skip past srcSize literals */
+ seq->litLength -= (U32)srcSize;
+ return;
+ }
+ srcSize -= seq->litLength;
+ seq->litLength = 0;
+ if (srcSize < seq->matchLength) {
+ /* Skip past the first srcSize of the match */
+ seq->matchLength -= (U32)srcSize;
+ if (seq->matchLength < minMatch) {
+ /* The match is too short, omit it */
+ if (rawSeqStore->pos + 1 < rawSeqStore->size) {
+ seq[1].litLength += seq[0].matchLength;
+ }
+ rawSeqStore->pos++;
+ }
+ return;
+ }
+ srcSize -= seq->matchLength;
+ seq->matchLength = 0;
+ rawSeqStore->pos++;
+ }
+}
+
+/*
+ * If the sequence length is longer than remaining then the sequence is split
+ * between this block and the next.
+ *
+ * Returns the current sequence to handle, or if the rest of the block should
+ * be literals, it returns a sequence with offset == 0.
+ */
+static rawSeq maybeSplitSequence(rawSeqStore_t* rawSeqStore,
+ U32 const remaining, U32 const minMatch)
+{
+ rawSeq sequence = rawSeqStore->seq[rawSeqStore->pos];
+ assert(sequence.offset > 0);
+ /* Likely: No partial sequence */
+ if (remaining >= sequence.litLength + sequence.matchLength) {
+ rawSeqStore->pos++;
+ return sequence;
+ }
+ /* Cut the sequence short (offset == 0 ==> rest is literals). */
+ if (remaining <= sequence.litLength) {
+ sequence.offset = 0;
+ } else if (remaining < sequence.litLength + sequence.matchLength) {
+ sequence.matchLength = remaining - sequence.litLength;
+ if (sequence.matchLength < minMatch) {
+ sequence.offset = 0;
+ }
+ }
+ /* Skip past `remaining` bytes for the future sequences. */
+ ZSTD_ldm_skipSequences(rawSeqStore, remaining, minMatch);
+ return sequence;
+}
+
+void ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) {
+ U32 currPos = (U32)(rawSeqStore->posInSequence + nbBytes);
+ while (currPos && rawSeqStore->pos < rawSeqStore->size) {
+ rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos];
+ if (currPos >= currSeq.litLength + currSeq.matchLength) {
+ currPos -= currSeq.litLength + currSeq.matchLength;
+ rawSeqStore->pos++;
+ } else {
+ rawSeqStore->posInSequence = currPos;
+ break;
+ }
+ }
+ if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) {
+ rawSeqStore->posInSequence = 0;
+ }
+}
+
+size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ unsigned const minMatch = cParams->minMatch;
+ ZSTD_blockCompressor const blockCompressor =
+ ZSTD_selectBlockCompressor(cParams->strategy, ZSTD_matchState_dictMode(ms));
+ /* Input bounds */
+ BYTE const* const istart = (BYTE const*)src;
+ BYTE const* const iend = istart + srcSize;
+ /* Input positions */
+ BYTE const* ip = istart;
+
+ DEBUGLOG(5, "ZSTD_ldm_blockCompress: srcSize=%zu", srcSize);
+ /* If using opt parser, use LDMs only as candidates rather than always accepting them */
+ if (cParams->strategy >= ZSTD_btopt) {
+ size_t lastLLSize;
+ ms->ldmSeqStore = rawSeqStore;
+ lastLLSize = blockCompressor(ms, seqStore, rep, src, srcSize);
+ ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore, srcSize);
+ return lastLLSize;
+ }
+
+ assert(rawSeqStore->pos <= rawSeqStore->size);
+ assert(rawSeqStore->size <= rawSeqStore->capacity);
+ /* Loop through each sequence and apply the block compressor to the literals */
+ while (rawSeqStore->pos < rawSeqStore->size && ip < iend) {
+ /* maybeSplitSequence updates rawSeqStore->pos */
+ rawSeq const sequence = maybeSplitSequence(rawSeqStore,
+ (U32)(iend - ip), minMatch);
+ int i;
+ /* End signal */
+ if (sequence.offset == 0)
+ break;
+
+ assert(ip + sequence.litLength + sequence.matchLength <= iend);
+
+ /* Fill tables for block compressor */
+ ZSTD_ldm_limitTableUpdate(ms, ip);
+ ZSTD_ldm_fillFastTables(ms, ip);
+ /* Run the block compressor */
+ DEBUGLOG(5, "pos %u : calling block compressor on segment of size %u", (unsigned)(ip-istart), sequence.litLength);
+ {
+ size_t const newLitLength =
+ blockCompressor(ms, seqStore, rep, ip, sequence.litLength);
+ ip += sequence.litLength;
+ /* Update the repcodes */
+ for (i = ZSTD_REP_NUM - 1; i > 0; i--)
+ rep[i] = rep[i-1];
+ rep[0] = sequence.offset;
+ /* Store the sequence */
+ ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, iend,
+ sequence.offset + ZSTD_REP_MOVE,
+ sequence.matchLength - MINMATCH);
+ ip += sequence.matchLength;
+ }
+ }
+ /* Fill the tables for the block compressor */
+ ZSTD_ldm_limitTableUpdate(ms, ip);
+ ZSTD_ldm_fillFastTables(ms, ip);
+ /* Compress the last literals */
+ return blockCompressor(ms, seqStore, rep, ip, iend - ip);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_LDM_H
+#define ZSTD_LDM_H
+
+
+#include "zstd_compress_internal.h" /* ldmParams_t, U32 */
+#include <linux/zstd.h> /* ZSTD_CCtx, size_t */
+
+/*-*************************************
+* Long distance matching
+***************************************/
+
+#define ZSTD_LDM_DEFAULT_WINDOW_LOG ZSTD_WINDOWLOG_LIMIT_DEFAULT
+
+void ZSTD_ldm_fillHashTable(
+ ldmState_t* state, const BYTE* ip,
+ const BYTE* iend, ldmParams_t const* params);
+
+/*
+ * ZSTD_ldm_generateSequences():
+ *
+ * Generates the sequences using the long distance match finder.
+ * Generates long range matching sequences in `sequences`, which parse a prefix
+ * of the source. `sequences` must be large enough to store every sequence,
+ * which can be checked with `ZSTD_ldm_getMaxNbSeq()`.
+ * @returns 0 or an error code.
+ *
+ * NOTE: The user must have called ZSTD_window_update() for all of the input
+ * they have, even if they pass it to ZSTD_ldm_generateSequences() in chunks.
+ * NOTE: This function returns an error if it runs out of space to store
+ * sequences.
+ */
+size_t ZSTD_ldm_generateSequences(
+ ldmState_t* ldms, rawSeqStore_t* sequences,
+ ldmParams_t const* params, void const* src, size_t srcSize);
+
+/*
+ * ZSTD_ldm_blockCompress():
+ *
+ * Compresses a block using the predefined sequences, along with a secondary
+ * block compressor. The literals section of every sequence is passed to the
+ * secondary block compressor, and those sequences are interspersed with the
+ * predefined sequences. Returns the length of the last literals.
+ * Updates `rawSeqStore.pos` to indicate how many sequences have been consumed.
+ * `rawSeqStore.seq` may also be updated to split the last sequence between two
+ * blocks.
+ * @return The length of the last literals.
+ *
+ * NOTE: The source must be at most the maximum block size, but the predefined
+ * sequences can be any size, and may be longer than the block. In the case that
+ * they are longer than the block, the last sequences may need to be split into
+ * two. We handle that case correctly, and update `rawSeqStore` appropriately.
+ * NOTE: This function does not return any errors.
+ */
+size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+/*
+ * ZSTD_ldm_skipSequences():
+ *
+ * Skip past `srcSize` bytes worth of sequences in `rawSeqStore`.
+ * Avoids emitting matches less than `minMatch` bytes.
+ * Must be called for data that is not passed to ZSTD_ldm_blockCompress().
+ */
+void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize,
+ U32 const minMatch);
+
+/* ZSTD_ldm_skipRawSeqStoreBytes():
+ * Moves forward in rawSeqStore by nbBytes, updating fields 'pos' and 'posInSequence'.
+ * Not to be used in conjunction with ZSTD_ldm_skipSequences().
+ * Must be called for data with is not passed to ZSTD_ldm_blockCompress().
+ */
+void ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes);
+
+/* ZSTD_ldm_getTableSize() :
+ * Estimate the space needed for long distance matching tables or 0 if LDM is
+ * disabled.
+ */
+size_t ZSTD_ldm_getTableSize(ldmParams_t params);
+
+/* ZSTD_ldm_getSeqSpace() :
+ * Return an upper bound on the number of sequences that can be produced by
+ * the long distance matcher, or 0 if LDM is disabled.
+ */
+size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize);
+
+/* ZSTD_ldm_adjustParameters() :
+ * If the params->hashRateLog is not set, set it to its default value based on
+ * windowLog and params->hashLog.
+ *
+ * Ensures that params->bucketSizeLog is <= params->hashLog (setting it to
+ * params->hashLog if it is not).
+ *
+ * Ensures that the minMatchLength >= targetLength during optimal parsing.
+ */
+void ZSTD_ldm_adjustParameters(ldmParams_t* params,
+ ZSTD_compressionParameters const* cParams);
+
+
+#endif /* ZSTD_FAST_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_LDM_GEARTAB_H
+#define ZSTD_LDM_GEARTAB_H
+
+static U64 ZSTD_ldm_gearTab[256] = {
+ 0xf5b8f72c5f77775c, 0x84935f266b7ac412, 0xb647ada9ca730ccc,
+ 0xb065bb4b114fb1de, 0x34584e7e8c3a9fd0, 0x4e97e17c6ae26b05,
+ 0x3a03d743bc99a604, 0xcecd042422c4044f, 0x76de76c58524259e,
+ 0x9c8528f65badeaca, 0x86563706e2097529, 0x2902475fa375d889,
+ 0xafb32a9739a5ebe6, 0xce2714da3883e639, 0x21eaf821722e69e,
+ 0x37b628620b628, 0x49a8d455d88caf5, 0x8556d711e6958140,
+ 0x4f7ae74fc605c1f, 0x829f0c3468bd3a20, 0x4ffdc885c625179e,
+ 0x8473de048a3daf1b, 0x51008822b05646b2, 0x69d75d12b2d1cc5f,
+ 0x8c9d4a19159154bc, 0xc3cc10f4abbd4003, 0xd06ddc1cecb97391,
+ 0xbe48e6e7ed80302e, 0x3481db31cee03547, 0xacc3f67cdaa1d210,
+ 0x65cb771d8c7f96cc, 0x8eb27177055723dd, 0xc789950d44cd94be,
+ 0x934feadc3700b12b, 0x5e485f11edbdf182, 0x1e2e2a46fd64767a,
+ 0x2969ca71d82efa7c, 0x9d46e9935ebbba2e, 0xe056b67e05e6822b,
+ 0x94d73f55739d03a0, 0xcd7010bdb69b5a03, 0x455ef9fcd79b82f4,
+ 0x869cb54a8749c161, 0x38d1a4fa6185d225, 0xb475166f94bbe9bb,
+ 0xa4143548720959f1, 0x7aed4780ba6b26ba, 0xd0ce264439e02312,
+ 0x84366d746078d508, 0xa8ce973c72ed17be, 0x21c323a29a430b01,
+ 0x9962d617e3af80ee, 0xab0ce91d9c8cf75b, 0x530e8ee6d19a4dbc,
+ 0x2ef68c0cf53f5d72, 0xc03a681640a85506, 0x496e4e9f9c310967,
+ 0x78580472b59b14a0, 0x273824c23b388577, 0x66bf923ad45cb553,
+ 0x47ae1a5a2492ba86, 0x35e304569e229659, 0x4765182a46870b6f,
+ 0x6cbab625e9099412, 0xddac9a2e598522c1, 0x7172086e666624f2,
+ 0xdf5003ca503b7837, 0x88c0c1db78563d09, 0x58d51865acfc289d,
+ 0x177671aec65224f1, 0xfb79d8a241e967d7, 0x2be1e101cad9a49a,
+ 0x6625682f6e29186b, 0x399553457ac06e50, 0x35dffb4c23abb74,
+ 0x429db2591f54aade, 0xc52802a8037d1009, 0x6acb27381f0b25f3,
+ 0xf45e2551ee4f823b, 0x8b0ea2d99580c2f7, 0x3bed519cbcb4e1e1,
+ 0xff452823dbb010a, 0x9d42ed614f3dd267, 0x5b9313c06257c57b,
+ 0xa114b8008b5e1442, 0xc1fe311c11c13d4b, 0x66e8763ea34c5568,
+ 0x8b982af1c262f05d, 0xee8876faaa75fbb7, 0x8a62a4d0d172bb2a,
+ 0xc13d94a3b7449a97, 0x6dbbba9dc15d037c, 0xc786101f1d92e0f1,
+ 0xd78681a907a0b79b, 0xf61aaf2962c9abb9, 0x2cfd16fcd3cb7ad9,
+ 0x868c5b6744624d21, 0x25e650899c74ddd7, 0xba042af4a7c37463,
+ 0x4eb1a539465a3eca, 0xbe09dbf03b05d5ca, 0x774e5a362b5472ba,
+ 0x47a1221229d183cd, 0x504b0ca18ef5a2df, 0xdffbdfbde2456eb9,
+ 0x46cd2b2fbee34634, 0xf2aef8fe819d98c3, 0x357f5276d4599d61,
+ 0x24a5483879c453e3, 0x88026889192b4b9, 0x28da96671782dbec,
+ 0x4ef37c40588e9aaa, 0x8837b90651bc9fb3, 0xc164f741d3f0e5d6,
+ 0xbc135a0a704b70ba, 0x69cd868f7622ada, 0xbc37ba89e0b9c0ab,
+ 0x47c14a01323552f6, 0x4f00794bacee98bb, 0x7107de7d637a69d5,
+ 0x88af793bb6f2255e, 0xf3c6466b8799b598, 0xc288c616aa7f3b59,
+ 0x81ca63cf42fca3fd, 0x88d85ace36a2674b, 0xd056bd3792389e7,
+ 0xe55c396c4e9dd32d, 0xbefb504571e6c0a6, 0x96ab32115e91e8cc,
+ 0xbf8acb18de8f38d1, 0x66dae58801672606, 0x833b6017872317fb,
+ 0xb87c16f2d1c92864, 0xdb766a74e58b669c, 0x89659f85c61417be,
+ 0xc8daad856011ea0c, 0x76a4b565b6fe7eae, 0xa469d085f6237312,
+ 0xaaf0365683a3e96c, 0x4dbb746f8424f7b8, 0x638755af4e4acc1,
+ 0x3d7807f5bde64486, 0x17be6d8f5bbb7639, 0x903f0cd44dc35dc,
+ 0x67b672eafdf1196c, 0xa676ff93ed4c82f1, 0x521d1004c5053d9d,
+ 0x37ba9ad09ccc9202, 0x84e54d297aacfb51, 0xa0b4b776a143445,
+ 0x820d471e20b348e, 0x1874383cb83d46dc, 0x97edeec7a1efe11c,
+ 0xb330e50b1bdc42aa, 0x1dd91955ce70e032, 0xa514cdb88f2939d5,
+ 0x2791233fd90db9d3, 0x7b670a4cc50f7a9b, 0x77c07d2a05c6dfa5,
+ 0xe3778b6646d0a6fa, 0xb39c8eda47b56749, 0x933ed448addbef28,
+ 0xaf846af6ab7d0bf4, 0xe5af208eb666e49, 0x5e6622f73534cd6a,
+ 0x297daeca42ef5b6e, 0x862daef3d35539a6, 0xe68722498f8e1ea9,
+ 0x981c53093dc0d572, 0xfa09b0bfbf86fbf5, 0x30b1e96166219f15,
+ 0x70e7d466bdc4fb83, 0x5a66736e35f2a8e9, 0xcddb59d2b7c1baef,
+ 0xd6c7d247d26d8996, 0xea4e39eac8de1ba3, 0x539c8bb19fa3aff2,
+ 0x9f90e4c5fd508d8, 0xa34e5956fbaf3385, 0x2e2f8e151d3ef375,
+ 0x173691e9b83faec1, 0xb85a8d56bf016379, 0x8382381267408ae3,
+ 0xb90f901bbdc0096d, 0x7c6ad32933bcec65, 0x76bb5e2f2c8ad595,
+ 0x390f851a6cf46d28, 0xc3e6064da1c2da72, 0xc52a0c101cfa5389,
+ 0xd78eaf84a3fbc530, 0x3781b9e2288b997e, 0x73c2f6dea83d05c4,
+ 0x4228e364c5b5ed7, 0x9d7a3edf0da43911, 0x8edcfeda24686756,
+ 0x5e7667a7b7a9b3a1, 0x4c4f389fa143791d, 0xb08bc1023da7cddc,
+ 0x7ab4be3ae529b1cc, 0x754e6132dbe74ff9, 0x71635442a839df45,
+ 0x2f6fb1643fbe52de, 0x961e0a42cf7a8177, 0xf3b45d83d89ef2ea,
+ 0xee3de4cf4a6e3e9b, 0xcd6848542c3295e7, 0xe4cee1664c78662f,
+ 0x9947548b474c68c4, 0x25d73777a5ed8b0b, 0xc915b1d636b7fc,
+ 0x21c2ba75d9b0d2da, 0x5f6b5dcf608a64a1, 0xdcf333255ff9570c,
+ 0x633b922418ced4ee, 0xc136dde0b004b34a, 0x58cc83b05d4b2f5a,
+ 0x5eb424dda28e42d2, 0x62df47369739cd98, 0xb4e0b42485e4ce17,
+ 0x16e1f0c1f9a8d1e7, 0x8ec3916707560ebf, 0x62ba6e2df2cc9db3,
+ 0xcbf9f4ff77d83a16, 0x78d9d7d07d2bbcc4, 0xef554ce1e02c41f4,
+ 0x8d7581127eccf94d, 0xa9b53336cb3c8a05, 0x38c42c0bf45c4f91,
+ 0x640893cdf4488863, 0x80ec34bc575ea568, 0x39f324f5b48eaa40,
+ 0xe9d9ed1f8eff527f, 0x9224fc058cc5a214, 0xbaba00b04cfe7741,
+ 0x309a9f120fcf52af, 0xa558f3ec65626212, 0x424bec8b7adabe2f,
+ 0x41622513a6aea433, 0xb88da2d5324ca798, 0xd287733b245528a4,
+ 0x9a44697e6d68aec3, 0x7b1093be2f49bb28, 0x50bbec632e3d8aad,
+ 0x6cd90723e1ea8283, 0x897b9e7431b02bf3, 0x219efdcb338a7047,
+ 0x3b0311f0a27c0656, 0xdb17bf91c0db96e7, 0x8cd4fd6b4e85a5b2,
+ 0xfab071054ba6409d, 0x40d6fe831fa9dfd9, 0xaf358debad7d791e,
+ 0xeb8d0e25a65e3e58, 0xbbcbd3df14e08580, 0xcf751f27ecdab2b,
+ 0x2b4da14f2613d8f4
+};
+
+#endif /* ZSTD_LDM_GEARTAB_H */
--- /dev/null
+/*
+ * Copyright (c) Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/*
+ * Disable inlining for the optimal parser for the kernel build.
+ * It is unlikely to be used in the kernel, and where it is used
+ * latency shouldn't matter because it is very slow to begin with.
+ * We prefer a ~180KB binary size win over faster optimal parsing.
+ *
+ * TODO(https://github.com/facebook/zstd/issues/2862):
+ * Improve the code size of the optimal parser in general, so we
+ * don't need this hack for the kernel build.
+ */
+#define ZSTD_NO_INLINE 1
+
+#include "zstd_compress_internal.h"
+#include "hist.h"
+#include "zstd_opt.h"
+
+
+#define ZSTD_LITFREQ_ADD 2 /* scaling factor for litFreq, so that frequencies adapt faster to new stats */
+#define ZSTD_FREQ_DIV 4 /* log factor when using previous stats to init next stats */
+#define ZSTD_MAX_PRICE (1<<30)
+
+#define ZSTD_PREDEF_THRESHOLD 1024 /* if srcSize < ZSTD_PREDEF_THRESHOLD, symbols' cost is assumed static, directly determined by pre-defined distributions */
+
+
+/*-*************************************
+* Price functions for optimal parser
+***************************************/
+
+#if 0 /* approximation at bit level */
+# define BITCOST_ACCURACY 0
+# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
+# define WEIGHT(stat) ((void)opt, ZSTD_bitWeight(stat))
+#elif 0 /* fractional bit accuracy */
+# define BITCOST_ACCURACY 8
+# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
+# define WEIGHT(stat,opt) ((void)opt, ZSTD_fracWeight(stat))
+#else /* opt==approx, ultra==accurate */
+# define BITCOST_ACCURACY 8
+# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
+# define WEIGHT(stat,opt) (opt ? ZSTD_fracWeight(stat) : ZSTD_bitWeight(stat))
+#endif
+
+MEM_STATIC U32 ZSTD_bitWeight(U32 stat)
+{
+ return (ZSTD_highbit32(stat+1) * BITCOST_MULTIPLIER);
+}
+
+MEM_STATIC U32 ZSTD_fracWeight(U32 rawStat)
+{
+ U32 const stat = rawStat + 1;
+ U32 const hb = ZSTD_highbit32(stat);
+ U32 const BWeight = hb * BITCOST_MULTIPLIER;
+ U32 const FWeight = (stat << BITCOST_ACCURACY) >> hb;
+ U32 const weight = BWeight + FWeight;
+ assert(hb + BITCOST_ACCURACY < 31);
+ return weight;
+}
+
+#if (DEBUGLEVEL>=2)
+/* debugging function,
+ * @return price in bytes as fractional value
+ * for debug messages only */
+MEM_STATIC double ZSTD_fCost(U32 price)
+{
+ return (double)price / (BITCOST_MULTIPLIER*8);
+}
+#endif
+
+static int ZSTD_compressedLiterals(optState_t const* const optPtr)
+{
+ return optPtr->literalCompressionMode != ZSTD_lcm_uncompressed;
+}
+
+static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel)
+{
+ if (ZSTD_compressedLiterals(optPtr))
+ optPtr->litSumBasePrice = WEIGHT(optPtr->litSum, optLevel);
+ optPtr->litLengthSumBasePrice = WEIGHT(optPtr->litLengthSum, optLevel);
+ optPtr->matchLengthSumBasePrice = WEIGHT(optPtr->matchLengthSum, optLevel);
+ optPtr->offCodeSumBasePrice = WEIGHT(optPtr->offCodeSum, optLevel);
+}
+
+
+/* ZSTD_downscaleStat() :
+ * reduce all elements in table by a factor 2^(ZSTD_FREQ_DIV+malus)
+ * return the resulting sum of elements */
+static U32 ZSTD_downscaleStat(unsigned* table, U32 lastEltIndex, int malus)
+{
+ U32 s, sum=0;
+ DEBUGLOG(5, "ZSTD_downscaleStat (nbElts=%u)", (unsigned)lastEltIndex+1);
+ assert(ZSTD_FREQ_DIV+malus > 0 && ZSTD_FREQ_DIV+malus < 31);
+ for (s=0; s<lastEltIndex+1; s++) {
+ table[s] = 1 + (table[s] >> (ZSTD_FREQ_DIV+malus));
+ sum += table[s];
+ }
+ return sum;
+}
+
+/* ZSTD_rescaleFreqs() :
+ * if first block (detected by optPtr->litLengthSum == 0) : init statistics
+ * take hints from dictionary if there is one
+ * or init from zero, using src for literals stats, or flat 1 for match symbols
+ * otherwise downscale existing stats, to be used as seed for next block.
+ */
+static void
+ZSTD_rescaleFreqs(optState_t* const optPtr,
+ const BYTE* const src, size_t const srcSize,
+ int const optLevel)
+{
+ int const compressedLiterals = ZSTD_compressedLiterals(optPtr);
+ DEBUGLOG(5, "ZSTD_rescaleFreqs (srcSize=%u)", (unsigned)srcSize);
+ optPtr->priceType = zop_dynamic;
+
+ if (optPtr->litLengthSum == 0) { /* first block : init */
+ if (srcSize <= ZSTD_PREDEF_THRESHOLD) { /* heuristic */
+ DEBUGLOG(5, "(srcSize <= ZSTD_PREDEF_THRESHOLD) => zop_predef");
+ optPtr->priceType = zop_predef;
+ }
+
+ assert(optPtr->symbolCosts != NULL);
+ if (optPtr->symbolCosts->huf.repeatMode == HUF_repeat_valid) {
+ /* huffman table presumed generated by dictionary */
+ optPtr->priceType = zop_dynamic;
+
+ if (compressedLiterals) {
+ unsigned lit;
+ assert(optPtr->litFreq != NULL);
+ optPtr->litSum = 0;
+ for (lit=0; lit<=MaxLit; lit++) {
+ U32 const scaleLog = 11; /* scale to 2K */
+ U32 const bitCost = HUF_getNbBits(optPtr->symbolCosts->huf.CTable, lit);
+ assert(bitCost <= scaleLog);
+ optPtr->litFreq[lit] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/;
+ optPtr->litSum += optPtr->litFreq[lit];
+ } }
+
+ { unsigned ll;
+ FSE_CState_t llstate;
+ FSE_initCState(&llstate, optPtr->symbolCosts->fse.litlengthCTable);
+ optPtr->litLengthSum = 0;
+ for (ll=0; ll<=MaxLL; ll++) {
+ U32 const scaleLog = 10; /* scale to 1K */
+ U32 const bitCost = FSE_getMaxNbBits(llstate.symbolTT, ll);
+ assert(bitCost < scaleLog);
+ optPtr->litLengthFreq[ll] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/;
+ optPtr->litLengthSum += optPtr->litLengthFreq[ll];
+ } }
+
+ { unsigned ml;
+ FSE_CState_t mlstate;
+ FSE_initCState(&mlstate, optPtr->symbolCosts->fse.matchlengthCTable);
+ optPtr->matchLengthSum = 0;
+ for (ml=0; ml<=MaxML; ml++) {
+ U32 const scaleLog = 10;
+ U32 const bitCost = FSE_getMaxNbBits(mlstate.symbolTT, ml);
+ assert(bitCost < scaleLog);
+ optPtr->matchLengthFreq[ml] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/;
+ optPtr->matchLengthSum += optPtr->matchLengthFreq[ml];
+ } }
+
+ { unsigned of;
+ FSE_CState_t ofstate;
+ FSE_initCState(&ofstate, optPtr->symbolCosts->fse.offcodeCTable);
+ optPtr->offCodeSum = 0;
+ for (of=0; of<=MaxOff; of++) {
+ U32 const scaleLog = 10;
+ U32 const bitCost = FSE_getMaxNbBits(ofstate.symbolTT, of);
+ assert(bitCost < scaleLog);
+ optPtr->offCodeFreq[of] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/;
+ optPtr->offCodeSum += optPtr->offCodeFreq[of];
+ } }
+
+ } else { /* not a dictionary */
+
+ assert(optPtr->litFreq != NULL);
+ if (compressedLiterals) {
+ unsigned lit = MaxLit;
+ HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); /* use raw first block to init statistics */
+ optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
+ }
+
+ { unsigned ll;
+ for (ll=0; ll<=MaxLL; ll++)
+ optPtr->litLengthFreq[ll] = 1;
+ }
+ optPtr->litLengthSum = MaxLL+1;
+
+ { unsigned ml;
+ for (ml=0; ml<=MaxML; ml++)
+ optPtr->matchLengthFreq[ml] = 1;
+ }
+ optPtr->matchLengthSum = MaxML+1;
+
+ { unsigned of;
+ for (of=0; of<=MaxOff; of++)
+ optPtr->offCodeFreq[of] = 1;
+ }
+ optPtr->offCodeSum = MaxOff+1;
+
+ }
+
+ } else { /* new block : re-use previous statistics, scaled down */
+
+ if (compressedLiterals)
+ optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
+ optPtr->litLengthSum = ZSTD_downscaleStat(optPtr->litLengthFreq, MaxLL, 0);
+ optPtr->matchLengthSum = ZSTD_downscaleStat(optPtr->matchLengthFreq, MaxML, 0);
+ optPtr->offCodeSum = ZSTD_downscaleStat(optPtr->offCodeFreq, MaxOff, 0);
+ }
+
+ ZSTD_setBasePrices(optPtr, optLevel);
+}
+
+/* ZSTD_rawLiteralsCost() :
+ * price of literals (only) in specified segment (which length can be 0).
+ * does not include price of literalLength symbol */
+static U32 ZSTD_rawLiteralsCost(const BYTE* const literals, U32 const litLength,
+ const optState_t* const optPtr,
+ int optLevel)
+{
+ if (litLength == 0) return 0;
+
+ if (!ZSTD_compressedLiterals(optPtr))
+ return (litLength << 3) * BITCOST_MULTIPLIER; /* Uncompressed - 8 bytes per literal. */
+
+ if (optPtr->priceType == zop_predef)
+ return (litLength*6) * BITCOST_MULTIPLIER; /* 6 bit per literal - no statistic used */
+
+ /* dynamic statistics */
+ { U32 price = litLength * optPtr->litSumBasePrice;
+ U32 u;
+ for (u=0; u < litLength; u++) {
+ assert(WEIGHT(optPtr->litFreq[literals[u]], optLevel) <= optPtr->litSumBasePrice); /* literal cost should never be negative */
+ price -= WEIGHT(optPtr->litFreq[literals[u]], optLevel);
+ }
+ return price;
+ }
+}
+
+/* ZSTD_litLengthPrice() :
+ * cost of literalLength symbol */
+static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optPtr, int optLevel)
+{
+ if (optPtr->priceType == zop_predef) return WEIGHT(litLength, optLevel);
+
+ /* dynamic statistics */
+ { U32 const llCode = ZSTD_LLcode(litLength);
+ return (LL_bits[llCode] * BITCOST_MULTIPLIER)
+ + optPtr->litLengthSumBasePrice
+ - WEIGHT(optPtr->litLengthFreq[llCode], optLevel);
+ }
+}
+
+/* ZSTD_getMatchPrice() :
+ * Provides the cost of the match part (offset + matchLength) of a sequence
+ * Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence.
+ * optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) */
+FORCE_INLINE_TEMPLATE U32
+ZSTD_getMatchPrice(U32 const offset,
+ U32 const matchLength,
+ const optState_t* const optPtr,
+ int const optLevel)
+{
+ U32 price;
+ U32 const offCode = ZSTD_highbit32(offset+1);
+ U32 const mlBase = matchLength - MINMATCH;
+ assert(matchLength >= MINMATCH);
+
+ if (optPtr->priceType == zop_predef) /* fixed scheme, do not use statistics */
+ return WEIGHT(mlBase, optLevel) + ((16 + offCode) * BITCOST_MULTIPLIER);
+
+ /* dynamic statistics */
+ price = (offCode * BITCOST_MULTIPLIER) + (optPtr->offCodeSumBasePrice - WEIGHT(optPtr->offCodeFreq[offCode], optLevel));
+ if ((optLevel<2) /*static*/ && offCode >= 20)
+ price += (offCode-19)*2 * BITCOST_MULTIPLIER; /* handicap for long distance offsets, favor decompression speed */
+
+ /* match Length */
+ { U32 const mlCode = ZSTD_MLcode(mlBase);
+ price += (ML_bits[mlCode] * BITCOST_MULTIPLIER) + (optPtr->matchLengthSumBasePrice - WEIGHT(optPtr->matchLengthFreq[mlCode], optLevel));
+ }
+
+ price += BITCOST_MULTIPLIER / 5; /* heuristic : make matches a bit more costly to favor less sequences -> faster decompression speed */
+
+ DEBUGLOG(8, "ZSTD_getMatchPrice(ml:%u) = %u", matchLength, price);
+ return price;
+}
+
+/* ZSTD_updateStats() :
+ * assumption : literals + litLengtn <= iend */
+static void ZSTD_updateStats(optState_t* const optPtr,
+ U32 litLength, const BYTE* literals,
+ U32 offsetCode, U32 matchLength)
+{
+ /* literals */
+ if (ZSTD_compressedLiterals(optPtr)) {
+ U32 u;
+ for (u=0; u < litLength; u++)
+ optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD;
+ optPtr->litSum += litLength*ZSTD_LITFREQ_ADD;
+ }
+
+ /* literal Length */
+ { U32 const llCode = ZSTD_LLcode(litLength);
+ optPtr->litLengthFreq[llCode]++;
+ optPtr->litLengthSum++;
+ }
+
+ /* match offset code (0-2=>repCode; 3+=>offset+2) */
+ { U32 const offCode = ZSTD_highbit32(offsetCode+1);
+ assert(offCode <= MaxOff);
+ optPtr->offCodeFreq[offCode]++;
+ optPtr->offCodeSum++;
+ }
+
+ /* match Length */
+ { U32 const mlBase = matchLength - MINMATCH;
+ U32 const mlCode = ZSTD_MLcode(mlBase);
+ optPtr->matchLengthFreq[mlCode]++;
+ optPtr->matchLengthSum++;
+ }
+}
+
+
+/* ZSTD_readMINMATCH() :
+ * function safe only for comparisons
+ * assumption : memPtr must be at least 4 bytes before end of buffer */
+MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
+{
+ switch (length)
+ {
+ default :
+ case 4 : return MEM_read32(memPtr);
+ case 3 : if (MEM_isLittleEndian())
+ return MEM_read32(memPtr)<<8;
+ else
+ return MEM_read32(memPtr)>>8;
+ }
+}
+
+
+/* Update hashTable3 up to ip (excluded)
+ Assumption : always within prefix (i.e. not within extDict) */
+static U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_matchState_t* ms,
+ U32* nextToUpdate3,
+ const BYTE* const ip)
+{
+ U32* const hashTable3 = ms->hashTable3;
+ U32 const hashLog3 = ms->hashLog3;
+ const BYTE* const base = ms->window.base;
+ U32 idx = *nextToUpdate3;
+ U32 const target = (U32)(ip - base);
+ size_t const hash3 = ZSTD_hash3Ptr(ip, hashLog3);
+ assert(hashLog3 > 0);
+
+ while(idx < target) {
+ hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx;
+ idx++;
+ }
+
+ *nextToUpdate3 = target;
+ return hashTable3[hash3];
+}
+
+
+/*-*************************************
+* Binary Tree search
+***************************************/
+/* ZSTD_insertBt1() : add one or multiple positions to tree.
+ * ip : assumed <= iend-8 .
+ * @return : nb of positions added */
+static U32 ZSTD_insertBt1(
+ ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iend,
+ U32 const mls, const int extDict)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32* const hashTable = ms->hashTable;
+ U32 const hashLog = cParams->hashLog;
+ size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
+ U32* const bt = ms->chainTable;
+ U32 const btLog = cParams->chainLog - 1;
+ U32 const btMask = (1 << btLog) - 1;
+ U32 matchIndex = hashTable[h];
+ size_t commonLengthSmaller=0, commonLengthLarger=0;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ const BYTE* match;
+ const U32 curr = (U32)(ip-base);
+ const U32 btLow = btMask >= curr ? 0 : curr - btMask;
+ U32* smallerPtr = bt + 2*(curr&btMask);
+ U32* largerPtr = smallerPtr + 1;
+ U32 dummy32; /* to be nullified at the end */
+ U32 const windowLow = ms->window.lowLimit;
+ U32 matchEndIdx = curr+8+1;
+ size_t bestLength = 8;
+ U32 nbCompares = 1U << cParams->searchLog;
+#ifdef ZSTD_C_PREDICT
+ U32 predictedSmall = *(bt + 2*((curr-1)&btMask) + 0);
+ U32 predictedLarge = *(bt + 2*((curr-1)&btMask) + 1);
+ predictedSmall += (predictedSmall>0);
+ predictedLarge += (predictedLarge>0);
+#endif /* ZSTD_C_PREDICT */
+
+ DEBUGLOG(8, "ZSTD_insertBt1 (%u)", curr);
+
+ assert(ip <= iend-8); /* required for h calculation */
+ hashTable[h] = curr; /* Update Hash Table */
+
+ assert(windowLow > 0);
+ for (; nbCompares && (matchIndex >= windowLow); --nbCompares) {
+ U32* const nextPtr = bt + 2*(matchIndex & btMask);
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ assert(matchIndex < curr);
+
+#ifdef ZSTD_C_PREDICT /* note : can create issues when hlog small <= 11 */
+ const U32* predictPtr = bt + 2*((matchIndex-1) & btMask); /* written this way, as bt is a roll buffer */
+ if (matchIndex == predictedSmall) {
+ /* no need to check length, result known */
+ *smallerPtr = matchIndex;
+ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ smallerPtr = nextPtr+1; /* new "smaller" => larger of match */
+ matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
+ predictedSmall = predictPtr[1] + (predictPtr[1]>0);
+ continue;
+ }
+ if (matchIndex == predictedLarge) {
+ *largerPtr = matchIndex;
+ if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ largerPtr = nextPtr;
+ matchIndex = nextPtr[0];
+ predictedLarge = predictPtr[0] + (predictPtr[0]>0);
+ continue;
+ }
+#endif
+
+ if (!extDict || (matchIndex+matchLength >= dictLimit)) {
+ assert(matchIndex+matchLength >= dictLimit); /* might be wrong if actually extDict */
+ match = base + matchIndex;
+ matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend);
+ } else {
+ match = dictBase + matchIndex;
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart);
+ if (matchIndex+matchLength >= dictLimit)
+ match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
+ }
+
+ if (matchLength > bestLength) {
+ bestLength = matchLength;
+ if (matchLength > matchEndIdx - matchIndex)
+ matchEndIdx = matchIndex + (U32)matchLength;
+ }
+
+ if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
+ break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */
+ }
+
+ if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */
+ /* match is smaller than current */
+ *smallerPtr = matchIndex; /* update smaller idx */
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */
+ smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */
+ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */
+ } else {
+ /* match is larger than current */
+ *largerPtr = matchIndex;
+ commonLengthLarger = matchLength;
+ if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */
+ largerPtr = nextPtr;
+ matchIndex = nextPtr[0];
+ } }
+
+ *smallerPtr = *largerPtr = 0;
+ { U32 positions = 0;
+ if (bestLength > 384) positions = MIN(192, (U32)(bestLength - 384)); /* speed optimization */
+ assert(matchEndIdx > curr + 8);
+ return MAX(positions, matchEndIdx - (curr + 8));
+ }
+}
+
+FORCE_INLINE_TEMPLATE
+void ZSTD_updateTree_internal(
+ ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iend,
+ const U32 mls, const ZSTD_dictMode_e dictMode)
+{
+ const BYTE* const base = ms->window.base;
+ U32 const target = (U32)(ip - base);
+ U32 idx = ms->nextToUpdate;
+ DEBUGLOG(6, "ZSTD_updateTree_internal, from %u to %u (dictMode:%u)",
+ idx, target, dictMode);
+
+ while(idx < target) {
+ U32 const forward = ZSTD_insertBt1(ms, base+idx, iend, mls, dictMode == ZSTD_extDict);
+ assert(idx < (U32)(idx + forward));
+ idx += forward;
+ }
+ assert((size_t)(ip - base) <= (size_t)(U32)(-1));
+ assert((size_t)(iend - base) <= (size_t)(U32)(-1));
+ ms->nextToUpdate = target;
+}
+
+void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend) {
+ ZSTD_updateTree_internal(ms, ip, iend, ms->cParams.minMatch, ZSTD_noDict);
+}
+
+FORCE_INLINE_TEMPLATE
+U32 ZSTD_insertBtAndGetAllMatches (
+ ZSTD_match_t* matches, /* store result (found matches) in this table (presumed large enough) */
+ ZSTD_matchState_t* ms,
+ U32* nextToUpdate3,
+ const BYTE* const ip, const BYTE* const iLimit, const ZSTD_dictMode_e dictMode,
+ const U32 rep[ZSTD_REP_NUM],
+ U32 const ll0, /* tells if associated literal length is 0 or not. This value must be 0 or 1 */
+ const U32 lengthToBeat,
+ U32 const mls /* template */)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1);
+ const BYTE* const base = ms->window.base;
+ U32 const curr = (U32)(ip-base);
+ U32 const hashLog = cParams->hashLog;
+ U32 const minMatch = (mls==3) ? 3 : 4;
+ U32* const hashTable = ms->hashTable;
+ size_t const h = ZSTD_hashPtr(ip, hashLog, mls);
+ U32 matchIndex = hashTable[h];
+ U32* const bt = ms->chainTable;
+ U32 const btLog = cParams->chainLog - 1;
+ U32 const btMask= (1U << btLog) - 1;
+ size_t commonLengthSmaller=0, commonLengthLarger=0;
+ const BYTE* const dictBase = ms->window.dictBase;
+ U32 const dictLimit = ms->window.dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ U32 const btLow = (btMask >= curr) ? 0 : curr - btMask;
+ U32 const windowLow = ZSTD_getLowestMatchIndex(ms, curr, cParams->windowLog);
+ U32 const matchLow = windowLow ? windowLow : 1;
+ U32* smallerPtr = bt + 2*(curr&btMask);
+ U32* largerPtr = bt + 2*(curr&btMask) + 1;
+ U32 matchEndIdx = curr+8+1; /* farthest referenced position of any match => detects repetitive patterns */
+ U32 dummy32; /* to be nullified at the end */
+ U32 mnum = 0;
+ U32 nbCompares = 1U << cParams->searchLog;
+
+ const ZSTD_matchState_t* dms = dictMode == ZSTD_dictMatchState ? ms->dictMatchState : NULL;
+ const ZSTD_compressionParameters* const dmsCParams =
+ dictMode == ZSTD_dictMatchState ? &dms->cParams : NULL;
+ const BYTE* const dmsBase = dictMode == ZSTD_dictMatchState ? dms->window.base : NULL;
+ const BYTE* const dmsEnd = dictMode == ZSTD_dictMatchState ? dms->window.nextSrc : NULL;
+ U32 const dmsHighLimit = dictMode == ZSTD_dictMatchState ? (U32)(dmsEnd - dmsBase) : 0;
+ U32 const dmsLowLimit = dictMode == ZSTD_dictMatchState ? dms->window.lowLimit : 0;
+ U32 const dmsIndexDelta = dictMode == ZSTD_dictMatchState ? windowLow - dmsHighLimit : 0;
+ U32 const dmsHashLog = dictMode == ZSTD_dictMatchState ? dmsCParams->hashLog : hashLog;
+ U32 const dmsBtLog = dictMode == ZSTD_dictMatchState ? dmsCParams->chainLog - 1 : btLog;
+ U32 const dmsBtMask = dictMode == ZSTD_dictMatchState ? (1U << dmsBtLog) - 1 : 0;
+ U32 const dmsBtLow = dictMode == ZSTD_dictMatchState && dmsBtMask < dmsHighLimit - dmsLowLimit ? dmsHighLimit - dmsBtMask : dmsLowLimit;
+
+ size_t bestLength = lengthToBeat-1;
+ DEBUGLOG(8, "ZSTD_insertBtAndGetAllMatches: current=%u", curr);
+
+ /* check repCode */
+ assert(ll0 <= 1); /* necessarily 1 or 0 */
+ { U32 const lastR = ZSTD_REP_NUM + ll0;
+ U32 repCode;
+ for (repCode = ll0; repCode < lastR; repCode++) {
+ U32 const repOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
+ U32 const repIndex = curr - repOffset;
+ U32 repLen = 0;
+ assert(curr >= dictLimit);
+ if (repOffset-1 /* intentional overflow, discards 0 and -1 */ < curr-dictLimit) { /* equivalent to `curr > repIndex >= dictLimit` */
+ /* We must validate the repcode offset because when we're using a dictionary the
+ * valid offset range shrinks when the dictionary goes out of bounds.
+ */
+ if ((repIndex >= windowLow) & (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch))) {
+ repLen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repOffset, iLimit) + minMatch;
+ }
+ } else { /* repIndex < dictLimit || repIndex >= curr */
+ const BYTE* const repMatch = dictMode == ZSTD_dictMatchState ?
+ dmsBase + repIndex - dmsIndexDelta :
+ dictBase + repIndex;
+ assert(curr >= windowLow);
+ if ( dictMode == ZSTD_extDict
+ && ( ((repOffset-1) /*intentional overflow*/ < curr - windowLow) /* equivalent to `curr > repIndex >= windowLow` */
+ & (((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */)
+ && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
+ repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dictEnd, prefixStart) + minMatch;
+ }
+ if (dictMode == ZSTD_dictMatchState
+ && ( ((repOffset-1) /*intentional overflow*/ < curr - (dmsLowLimit + dmsIndexDelta)) /* equivalent to `curr > repIndex >= dmsLowLimit` */
+ & ((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */
+ && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) {
+ repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dmsEnd, prefixStart) + minMatch;
+ } }
+ /* save longer solution */
+ if (repLen > bestLength) {
+ DEBUGLOG(8, "found repCode %u (ll0:%u, offset:%u) of length %u",
+ repCode, ll0, repOffset, repLen);
+ bestLength = repLen;
+ matches[mnum].off = repCode - ll0;
+ matches[mnum].len = (U32)repLen;
+ mnum++;
+ if ( (repLen > sufficient_len)
+ | (ip+repLen == iLimit) ) { /* best possible */
+ return mnum;
+ } } } }
+
+ /* HC3 match finder */
+ if ((mls == 3) /*static*/ && (bestLength < mls)) {
+ U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(ms, nextToUpdate3, ip);
+ if ((matchIndex3 >= matchLow)
+ & (curr - matchIndex3 < (1<<18)) /*heuristic : longer distance likely too expensive*/ ) {
+ size_t mlen;
+ if ((dictMode == ZSTD_noDict) /*static*/ || (dictMode == ZSTD_dictMatchState) /*static*/ || (matchIndex3 >= dictLimit)) {
+ const BYTE* const match = base + matchIndex3;
+ mlen = ZSTD_count(ip, match, iLimit);
+ } else {
+ const BYTE* const match = dictBase + matchIndex3;
+ mlen = ZSTD_count_2segments(ip, match, iLimit, dictEnd, prefixStart);
+ }
+
+ /* save best solution */
+ if (mlen >= mls /* == 3 > bestLength */) {
+ DEBUGLOG(8, "found small match with hlog3, of length %u",
+ (U32)mlen);
+ bestLength = mlen;
+ assert(curr > matchIndex3);
+ assert(mnum==0); /* no prior solution */
+ matches[0].off = (curr - matchIndex3) + ZSTD_REP_MOVE;
+ matches[0].len = (U32)mlen;
+ mnum = 1;
+ if ( (mlen > sufficient_len) |
+ (ip+mlen == iLimit) ) { /* best possible length */
+ ms->nextToUpdate = curr+1; /* skip insertion */
+ return 1;
+ } } }
+ /* no dictMatchState lookup: dicts don't have a populated HC3 table */
+ }
+
+ hashTable[h] = curr; /* Update Hash Table */
+
+ for (; nbCompares && (matchIndex >= matchLow); --nbCompares) {
+ U32* const nextPtr = bt + 2*(matchIndex & btMask);
+ const BYTE* match;
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ assert(curr > matchIndex);
+
+ if ((dictMode == ZSTD_noDict) || (dictMode == ZSTD_dictMatchState) || (matchIndex+matchLength >= dictLimit)) {
+ assert(matchIndex+matchLength >= dictLimit); /* ensure the condition is correct when !extDict */
+ match = base + matchIndex;
+ if (matchIndex >= dictLimit) assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */
+ matchLength += ZSTD_count(ip+matchLength, match+matchLength, iLimit);
+ } else {
+ match = dictBase + matchIndex;
+ assert(memcmp(match, ip, matchLength) == 0); /* ensure early section of match is equal as expected */
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart);
+ if (matchIndex+matchLength >= dictLimit)
+ match = base + matchIndex; /* prepare for match[matchLength] read */
+ }
+
+ if (matchLength > bestLength) {
+ DEBUGLOG(8, "found match of length %u at distance %u (offCode=%u)",
+ (U32)matchLength, curr - matchIndex, curr - matchIndex + ZSTD_REP_MOVE);
+ assert(matchEndIdx > matchIndex);
+ if (matchLength > matchEndIdx - matchIndex)
+ matchEndIdx = matchIndex + (U32)matchLength;
+ bestLength = matchLength;
+ matches[mnum].off = (curr - matchIndex) + ZSTD_REP_MOVE;
+ matches[mnum].len = (U32)matchLength;
+ mnum++;
+ if ( (matchLength > ZSTD_OPT_NUM)
+ | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
+ if (dictMode == ZSTD_dictMatchState) nbCompares = 0; /* break should also skip searching dms */
+ break; /* drop, to preserve bt consistency (miss a little bit of compression) */
+ }
+ }
+
+ if (match[matchLength] < ip[matchLength]) {
+ /* match smaller than current */
+ *smallerPtr = matchIndex; /* update smaller idx */
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ smallerPtr = nextPtr+1; /* new candidate => larger than match, which was smaller than current */
+ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous, closer to current */
+ } else {
+ *largerPtr = matchIndex;
+ commonLengthLarger = matchLength;
+ if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */
+ largerPtr = nextPtr;
+ matchIndex = nextPtr[0];
+ } }
+
+ *smallerPtr = *largerPtr = 0;
+
+ assert(nbCompares <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */
+ if (dictMode == ZSTD_dictMatchState && nbCompares) {
+ size_t const dmsH = ZSTD_hashPtr(ip, dmsHashLog, mls);
+ U32 dictMatchIndex = dms->hashTable[dmsH];
+ const U32* const dmsBt = dms->chainTable;
+ commonLengthSmaller = commonLengthLarger = 0;
+ for (; nbCompares && (dictMatchIndex > dmsLowLimit); --nbCompares) {
+ const U32* const nextPtr = dmsBt + 2*(dictMatchIndex & dmsBtMask);
+ size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
+ const BYTE* match = dmsBase + dictMatchIndex;
+ matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dmsEnd, prefixStart);
+ if (dictMatchIndex+matchLength >= dmsHighLimit)
+ match = base + dictMatchIndex + dmsIndexDelta; /* to prepare for next usage of match[matchLength] */
+
+ if (matchLength > bestLength) {
+ matchIndex = dictMatchIndex + dmsIndexDelta;
+ DEBUGLOG(8, "found dms match of length %u at distance %u (offCode=%u)",
+ (U32)matchLength, curr - matchIndex, curr - matchIndex + ZSTD_REP_MOVE);
+ if (matchLength > matchEndIdx - matchIndex)
+ matchEndIdx = matchIndex + (U32)matchLength;
+ bestLength = matchLength;
+ matches[mnum].off = (curr - matchIndex) + ZSTD_REP_MOVE;
+ matches[mnum].len = (U32)matchLength;
+ mnum++;
+ if ( (matchLength > ZSTD_OPT_NUM)
+ | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
+ break; /* drop, to guarantee consistency (miss a little bit of compression) */
+ }
+ }
+
+ if (dictMatchIndex <= dmsBtLow) { break; } /* beyond tree size, stop the search */
+ if (match[matchLength] < ip[matchLength]) {
+ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
+ dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */
+ } else {
+ /* match is larger than current */
+ commonLengthLarger = matchLength;
+ dictMatchIndex = nextPtr[0];
+ }
+ }
+ }
+
+ assert(matchEndIdx > curr+8);
+ ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
+ return mnum;
+}
+
+
+FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches (
+ ZSTD_match_t* matches, /* store result (match found, increasing size) in this table */
+ ZSTD_matchState_t* ms,
+ U32* nextToUpdate3,
+ const BYTE* ip, const BYTE* const iHighLimit, const ZSTD_dictMode_e dictMode,
+ const U32 rep[ZSTD_REP_NUM],
+ U32 const ll0,
+ U32 const lengthToBeat)
+{
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ U32 const matchLengthSearch = cParams->minMatch;
+ DEBUGLOG(8, "ZSTD_BtGetAllMatches");
+ if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */
+ ZSTD_updateTree_internal(ms, ip, iHighLimit, matchLengthSearch, dictMode);
+ switch(matchLengthSearch)
+ {
+ case 3 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 3);
+ default :
+ case 4 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 4);
+ case 5 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 5);
+ case 7 :
+ case 6 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 6);
+ }
+}
+
+/* ***********************
+* LDM helper functions *
+*************************/
+
+/* Struct containing info needed to make decision about ldm inclusion */
+typedef struct {
+ rawSeqStore_t seqStore; /* External match candidates store for this block */
+ U32 startPosInBlock; /* Start position of the current match candidate */
+ U32 endPosInBlock; /* End position of the current match candidate */
+ U32 offset; /* Offset of the match candidate */
+} ZSTD_optLdm_t;
+
+/* ZSTD_optLdm_skipRawSeqStoreBytes():
+ * Moves forward in rawSeqStore by nbBytes, which will update the fields 'pos' and 'posInSequence'.
+ */
+static void ZSTD_optLdm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) {
+ U32 currPos = (U32)(rawSeqStore->posInSequence + nbBytes);
+ while (currPos && rawSeqStore->pos < rawSeqStore->size) {
+ rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos];
+ if (currPos >= currSeq.litLength + currSeq.matchLength) {
+ currPos -= currSeq.litLength + currSeq.matchLength;
+ rawSeqStore->pos++;
+ } else {
+ rawSeqStore->posInSequence = currPos;
+ break;
+ }
+ }
+ if (currPos == 0 || rawSeqStore->pos == rawSeqStore->size) {
+ rawSeqStore->posInSequence = 0;
+ }
+}
+
+/* ZSTD_opt_getNextMatchAndUpdateSeqStore():
+ * Calculates the beginning and end of the next match in the current block.
+ * Updates 'pos' and 'posInSequence' of the ldmSeqStore.
+ */
+static void ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 currPosInBlock,
+ U32 blockBytesRemaining) {
+ rawSeq currSeq;
+ U32 currBlockEndPos;
+ U32 literalsBytesRemaining;
+ U32 matchBytesRemaining;
+
+ /* Setting match end position to MAX to ensure we never use an LDM during this block */
+ if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) {
+ optLdm->startPosInBlock = UINT_MAX;
+ optLdm->endPosInBlock = UINT_MAX;
+ return;
+ }
+ /* Calculate appropriate bytes left in matchLength and litLength after adjusting
+ based on ldmSeqStore->posInSequence */
+ currSeq = optLdm->seqStore.seq[optLdm->seqStore.pos];
+ assert(optLdm->seqStore.posInSequence <= currSeq.litLength + currSeq.matchLength);
+ currBlockEndPos = currPosInBlock + blockBytesRemaining;
+ literalsBytesRemaining = (optLdm->seqStore.posInSequence < currSeq.litLength) ?
+ currSeq.litLength - (U32)optLdm->seqStore.posInSequence :
+ 0;
+ matchBytesRemaining = (literalsBytesRemaining == 0) ?
+ currSeq.matchLength - ((U32)optLdm->seqStore.posInSequence - currSeq.litLength) :
+ currSeq.matchLength;
+
+ /* If there are more literal bytes than bytes remaining in block, no ldm is possible */
+ if (literalsBytesRemaining >= blockBytesRemaining) {
+ optLdm->startPosInBlock = UINT_MAX;
+ optLdm->endPosInBlock = UINT_MAX;
+ ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, blockBytesRemaining);
+ return;
+ }
+
+ /* Matches may be < MINMATCH by this process. In that case, we will reject them
+ when we are deciding whether or not to add the ldm */
+ optLdm->startPosInBlock = currPosInBlock + literalsBytesRemaining;
+ optLdm->endPosInBlock = optLdm->startPosInBlock + matchBytesRemaining;
+ optLdm->offset = currSeq.offset;
+
+ if (optLdm->endPosInBlock > currBlockEndPos) {
+ /* Match ends after the block ends, we can't use the whole match */
+ optLdm->endPosInBlock = currBlockEndPos;
+ ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, currBlockEndPos - currPosInBlock);
+ } else {
+ /* Consume nb of bytes equal to size of sequence left */
+ ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, literalsBytesRemaining + matchBytesRemaining);
+ }
+}
+
+/* ZSTD_optLdm_maybeAddMatch():
+ * Adds a match if it's long enough, based on it's 'matchStartPosInBlock'
+ * and 'matchEndPosInBlock', into 'matches'. Maintains the correct ordering of 'matches'
+ */
+static void ZSTD_optLdm_maybeAddMatch(ZSTD_match_t* matches, U32* nbMatches,
+ ZSTD_optLdm_t* optLdm, U32 currPosInBlock) {
+ U32 posDiff = currPosInBlock - optLdm->startPosInBlock;
+ /* Note: ZSTD_match_t actually contains offCode and matchLength (before subtracting MINMATCH) */
+ U32 candidateMatchLength = optLdm->endPosInBlock - optLdm->startPosInBlock - posDiff;
+ U32 candidateOffCode = optLdm->offset + ZSTD_REP_MOVE;
+
+ /* Ensure that current block position is not outside of the match */
+ if (currPosInBlock < optLdm->startPosInBlock
+ || currPosInBlock >= optLdm->endPosInBlock
+ || candidateMatchLength < MINMATCH) {
+ return;
+ }
+
+ if (*nbMatches == 0 || ((candidateMatchLength > matches[*nbMatches-1].len) && *nbMatches < ZSTD_OPT_NUM)) {
+ DEBUGLOG(6, "ZSTD_optLdm_maybeAddMatch(): Adding ldm candidate match (offCode: %u matchLength %u) at block position=%u",
+ candidateOffCode, candidateMatchLength, currPosInBlock);
+ matches[*nbMatches].len = candidateMatchLength;
+ matches[*nbMatches].off = candidateOffCode;
+ (*nbMatches)++;
+ }
+}
+
+/* ZSTD_optLdm_processMatchCandidate():
+ * Wrapper function to update ldm seq store and call ldm functions as necessary.
+ */
+static void ZSTD_optLdm_processMatchCandidate(ZSTD_optLdm_t* optLdm, ZSTD_match_t* matches, U32* nbMatches,
+ U32 currPosInBlock, U32 remainingBytes) {
+ if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) {
+ return;
+ }
+
+ if (currPosInBlock >= optLdm->endPosInBlock) {
+ if (currPosInBlock > optLdm->endPosInBlock) {
+ /* The position at which ZSTD_optLdm_processMatchCandidate() is called is not necessarily
+ * at the end of a match from the ldm seq store, and will often be some bytes
+ * over beyond matchEndPosInBlock. As such, we need to correct for these "overshoots"
+ */
+ U32 posOvershoot = currPosInBlock - optLdm->endPosInBlock;
+ ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, posOvershoot);
+ }
+ ZSTD_opt_getNextMatchAndUpdateSeqStore(optLdm, currPosInBlock, remainingBytes);
+ }
+ ZSTD_optLdm_maybeAddMatch(matches, nbMatches, optLdm, currPosInBlock);
+}
+
+/*-*******************************
+* Optimal parser
+*********************************/
+
+
+static U32 ZSTD_totalLen(ZSTD_optimal_t sol)
+{
+ return sol.litlen + sol.mlen;
+}
+
+#if 0 /* debug */
+
+static void
+listStats(const U32* table, int lastEltID)
+{
+ int const nbElts = lastEltID + 1;
+ int enb;
+ for (enb=0; enb < nbElts; enb++) {
+ (void)table;
+ /* RAWLOG(2, "%3i:%3i, ", enb, table[enb]); */
+ RAWLOG(2, "%4i,", table[enb]);
+ }
+ RAWLOG(2, " \n");
+}
+
+#endif
+
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
+ seqStore_t* seqStore,
+ U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize,
+ const int optLevel,
+ const ZSTD_dictMode_e dictMode)
+{
+ optState_t* const optStatePtr = &ms->opt;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* ip = istart;
+ const BYTE* anchor = istart;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - 8;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const prefixStart = base + ms->window.dictLimit;
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+
+ U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1);
+ U32 const minMatch = (cParams->minMatch == 3) ? 3 : 4;
+ U32 nextToUpdate3 = ms->nextToUpdate;
+
+ ZSTD_optimal_t* const opt = optStatePtr->priceTable;
+ ZSTD_match_t* const matches = optStatePtr->matchTable;
+ ZSTD_optimal_t lastSequence;
+ ZSTD_optLdm_t optLdm;
+
+ optLdm.seqStore = ms->ldmSeqStore ? *ms->ldmSeqStore : kNullRawSeqStore;
+ optLdm.endPosInBlock = optLdm.startPosInBlock = optLdm.offset = 0;
+ ZSTD_opt_getNextMatchAndUpdateSeqStore(&optLdm, (U32)(ip-istart), (U32)(iend-ip));
+
+ /* init */
+ DEBUGLOG(5, "ZSTD_compressBlock_opt_generic: current=%u, prefix=%u, nextToUpdate=%u",
+ (U32)(ip - base), ms->window.dictLimit, ms->nextToUpdate);
+ assert(optLevel <= 2);
+ ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize, optLevel);
+ ip += (ip==prefixStart);
+
+ /* Match Loop */
+ while (ip < ilimit) {
+ U32 cur, last_pos = 0;
+
+ /* find first match */
+ { U32 const litlen = (U32)(ip - anchor);
+ U32 const ll0 = !litlen;
+ U32 nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, ip, iend, dictMode, rep, ll0, minMatch);
+ ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches,
+ (U32)(ip-istart), (U32)(iend - ip));
+ if (!nbMatches) { ip++; continue; }
+
+ /* initialize opt[0] */
+ { U32 i ; for (i=0; i<ZSTD_REP_NUM; i++) opt[0].rep[i] = rep[i]; }
+ opt[0].mlen = 0; /* means is_a_literal */
+ opt[0].litlen = litlen;
+ /* We don't need to include the actual price of the literals because
+ * it is static for the duration of the forward pass, and is included
+ * in every price. We include the literal length to avoid negative
+ * prices when we subtract the previous literal length.
+ */
+ opt[0].price = ZSTD_litLengthPrice(litlen, optStatePtr, optLevel);
+
+ /* large match -> immediate encoding */
+ { U32 const maxML = matches[nbMatches-1].len;
+ U32 const maxOffset = matches[nbMatches-1].off;
+ DEBUGLOG(6, "found %u matches of maxLength=%u and maxOffCode=%u at cPos=%u => start new series",
+ nbMatches, maxML, maxOffset, (U32)(ip-prefixStart));
+
+ if (maxML > sufficient_len) {
+ lastSequence.litlen = litlen;
+ lastSequence.mlen = maxML;
+ lastSequence.off = maxOffset;
+ DEBUGLOG(6, "large match (%u>%u), immediate encoding",
+ maxML, sufficient_len);
+ cur = 0;
+ last_pos = ZSTD_totalLen(lastSequence);
+ goto _shortestPath;
+ } }
+
+ /* set prices for first matches starting position == 0 */
+ { U32 const literalsPrice = opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
+ U32 pos;
+ U32 matchNb;
+ for (pos = 1; pos < minMatch; pos++) {
+ opt[pos].price = ZSTD_MAX_PRICE; /* mlen, litlen and price will be fixed during forward scanning */
+ }
+ for (matchNb = 0; matchNb < nbMatches; matchNb++) {
+ U32 const offset = matches[matchNb].off;
+ U32 const end = matches[matchNb].len;
+ for ( ; pos <= end ; pos++ ) {
+ U32 const matchPrice = ZSTD_getMatchPrice(offset, pos, optStatePtr, optLevel);
+ U32 const sequencePrice = literalsPrice + matchPrice;
+ DEBUGLOG(7, "rPos:%u => set initial price : %.2f",
+ pos, ZSTD_fCost(sequencePrice));
+ opt[pos].mlen = pos;
+ opt[pos].off = offset;
+ opt[pos].litlen = litlen;
+ opt[pos].price = sequencePrice;
+ } }
+ last_pos = pos-1;
+ }
+ }
+
+ /* check further positions */
+ for (cur = 1; cur <= last_pos; cur++) {
+ const BYTE* const inr = ip + cur;
+ assert(cur < ZSTD_OPT_NUM);
+ DEBUGLOG(7, "cPos:%zi==rPos:%u", inr-istart, cur)
+
+ /* Fix current position with one literal if cheaper */
+ { U32 const litlen = (opt[cur-1].mlen == 0) ? opt[cur-1].litlen + 1 : 1;
+ int const price = opt[cur-1].price
+ + ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel)
+ + ZSTD_litLengthPrice(litlen, optStatePtr, optLevel)
+ - ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel);
+ assert(price < 1000000000); /* overflow check */
+ if (price <= opt[cur].price) {
+ DEBUGLOG(7, "cPos:%zi==rPos:%u : better price (%.2f<=%.2f) using literal (ll==%u) (hist:%u,%u,%u)",
+ inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), litlen,
+ opt[cur-1].rep[0], opt[cur-1].rep[1], opt[cur-1].rep[2]);
+ opt[cur].mlen = 0;
+ opt[cur].off = 0;
+ opt[cur].litlen = litlen;
+ opt[cur].price = price;
+ } else {
+ DEBUGLOG(7, "cPos:%zi==rPos:%u : literal would cost more (%.2f>%.2f) (hist:%u,%u,%u)",
+ inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price),
+ opt[cur].rep[0], opt[cur].rep[1], opt[cur].rep[2]);
+ }
+ }
+
+ /* Set the repcodes of the current position. We must do it here
+ * because we rely on the repcodes of the 2nd to last sequence being
+ * correct to set the next chunks repcodes during the backward
+ * traversal.
+ */
+ ZSTD_STATIC_ASSERT(sizeof(opt[cur].rep) == sizeof(repcodes_t));
+ assert(cur >= opt[cur].mlen);
+ if (opt[cur].mlen != 0) {
+ U32 const prev = cur - opt[cur].mlen;
+ repcodes_t newReps = ZSTD_updateRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0);
+ ZSTD_memcpy(opt[cur].rep, &newReps, sizeof(repcodes_t));
+ } else {
+ ZSTD_memcpy(opt[cur].rep, opt[cur - 1].rep, sizeof(repcodes_t));
+ }
+
+ /* last match must start at a minimum distance of 8 from oend */
+ if (inr > ilimit) continue;
+
+ if (cur == last_pos) break;
+
+ if ( (optLevel==0) /*static_test*/
+ && (opt[cur+1].price <= opt[cur].price + (BITCOST_MULTIPLIER/2)) ) {
+ DEBUGLOG(7, "move to next rPos:%u : price is <=", cur+1);
+ continue; /* skip unpromising positions; about ~+6% speed, -0.01 ratio */
+ }
+
+ { U32 const ll0 = (opt[cur].mlen != 0);
+ U32 const litlen = (opt[cur].mlen == 0) ? opt[cur].litlen : 0;
+ U32 const previousPrice = opt[cur].price;
+ U32 const basePrice = previousPrice + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
+ U32 nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, inr, iend, dictMode, opt[cur].rep, ll0, minMatch);
+ U32 matchNb;
+
+ ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches,
+ (U32)(inr-istart), (U32)(iend-inr));
+
+ if (!nbMatches) {
+ DEBUGLOG(7, "rPos:%u : no match found", cur);
+ continue;
+ }
+
+ { U32 const maxML = matches[nbMatches-1].len;
+ DEBUGLOG(7, "cPos:%zi==rPos:%u, found %u matches, of maxLength=%u",
+ inr-istart, cur, nbMatches, maxML);
+
+ if ( (maxML > sufficient_len)
+ || (cur + maxML >= ZSTD_OPT_NUM) ) {
+ lastSequence.mlen = maxML;
+ lastSequence.off = matches[nbMatches-1].off;
+ lastSequence.litlen = litlen;
+ cur -= (opt[cur].mlen==0) ? opt[cur].litlen : 0; /* last sequence is actually only literals, fix cur to last match - note : may underflow, in which case, it's first sequence, and it's okay */
+ last_pos = cur + ZSTD_totalLen(lastSequence);
+ if (cur > ZSTD_OPT_NUM) cur = 0; /* underflow => first match */
+ goto _shortestPath;
+ } }
+
+ /* set prices using matches found at position == cur */
+ for (matchNb = 0; matchNb < nbMatches; matchNb++) {
+ U32 const offset = matches[matchNb].off;
+ U32 const lastML = matches[matchNb].len;
+ U32 const startML = (matchNb>0) ? matches[matchNb-1].len+1 : minMatch;
+ U32 mlen;
+
+ DEBUGLOG(7, "testing match %u => offCode=%4u, mlen=%2u, llen=%2u",
+ matchNb, matches[matchNb].off, lastML, litlen);
+
+ for (mlen = lastML; mlen >= startML; mlen--) { /* scan downward */
+ U32 const pos = cur + mlen;
+ int const price = basePrice + ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel);
+
+ if ((pos > last_pos) || (price < opt[pos].price)) {
+ DEBUGLOG(7, "rPos:%u (ml=%2u) => new better price (%.2f<%.2f)",
+ pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price));
+ while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } /* fill empty positions */
+ opt[pos].mlen = mlen;
+ opt[pos].off = offset;
+ opt[pos].litlen = litlen;
+ opt[pos].price = price;
+ } else {
+ DEBUGLOG(7, "rPos:%u (ml=%2u) => new price is worse (%.2f>=%.2f)",
+ pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price));
+ if (optLevel==0) break; /* early update abort; gets ~+10% speed for about -0.01 ratio loss */
+ }
+ } } }
+ } /* for (cur = 1; cur <= last_pos; cur++) */
+
+ lastSequence = opt[last_pos];
+ cur = last_pos > ZSTD_totalLen(lastSequence) ? last_pos - ZSTD_totalLen(lastSequence) : 0; /* single sequence, and it starts before `ip` */
+ assert(cur < ZSTD_OPT_NUM); /* control overflow*/
+
+_shortestPath: /* cur, last_pos, best_mlen, best_off have to be set */
+ assert(opt[0].mlen == 0);
+
+ /* Set the next chunk's repcodes based on the repcodes of the beginning
+ * of the last match, and the last sequence. This avoids us having to
+ * update them while traversing the sequences.
+ */
+ if (lastSequence.mlen != 0) {
+ repcodes_t reps = ZSTD_updateRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0);
+ ZSTD_memcpy(rep, &reps, sizeof(reps));
+ } else {
+ ZSTD_memcpy(rep, opt[cur].rep, sizeof(repcodes_t));
+ }
+
+ { U32 const storeEnd = cur + 1;
+ U32 storeStart = storeEnd;
+ U32 seqPos = cur;
+
+ DEBUGLOG(6, "start reverse traversal (last_pos:%u, cur:%u)",
+ last_pos, cur); (void)last_pos;
+ assert(storeEnd < ZSTD_OPT_NUM);
+ DEBUGLOG(6, "last sequence copied into pos=%u (llen=%u,mlen=%u,ofc=%u)",
+ storeEnd, lastSequence.litlen, lastSequence.mlen, lastSequence.off);
+ opt[storeEnd] = lastSequence;
+ while (seqPos > 0) {
+ U32 const backDist = ZSTD_totalLen(opt[seqPos]);
+ storeStart--;
+ DEBUGLOG(6, "sequence from rPos=%u copied into pos=%u (llen=%u,mlen=%u,ofc=%u)",
+ seqPos, storeStart, opt[seqPos].litlen, opt[seqPos].mlen, opt[seqPos].off);
+ opt[storeStart] = opt[seqPos];
+ seqPos = (seqPos > backDist) ? seqPos - backDist : 0;
+ }
+
+ /* save sequences */
+ DEBUGLOG(6, "sending selected sequences into seqStore")
+ { U32 storePos;
+ for (storePos=storeStart; storePos <= storeEnd; storePos++) {
+ U32 const llen = opt[storePos].litlen;
+ U32 const mlen = opt[storePos].mlen;
+ U32 const offCode = opt[storePos].off;
+ U32 const advance = llen + mlen;
+ DEBUGLOG(6, "considering seq starting at %zi, llen=%u, mlen=%u",
+ anchor - istart, (unsigned)llen, (unsigned)mlen);
+
+ if (mlen==0) { /* only literals => must be last "sequence", actually starting a new stream of sequences */
+ assert(storePos == storeEnd); /* must be last sequence */
+ ip = anchor + llen; /* last "sequence" is a bunch of literals => don't progress anchor */
+ continue; /* will finish */
+ }
+
+ assert(anchor + llen <= iend);
+ ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen);
+ ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen-MINMATCH);
+ anchor += advance;
+ ip = anchor;
+ } }
+ ZSTD_setBasePrices(optStatePtr, optLevel);
+ }
+ } /* while (ip < ilimit) */
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+}
+
+
+size_t ZSTD_compressBlock_btopt(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_compressBlock_btopt");
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_noDict);
+}
+
+
+/* used in 2-pass strategy */
+static U32 ZSTD_upscaleStat(unsigned* table, U32 lastEltIndex, int bonus)
+{
+ U32 s, sum=0;
+ assert(ZSTD_FREQ_DIV+bonus >= 0);
+ for (s=0; s<lastEltIndex+1; s++) {
+ table[s] <<= ZSTD_FREQ_DIV+bonus;
+ table[s]--;
+ sum += table[s];
+ }
+ return sum;
+}
+
+/* used in 2-pass strategy */
+MEM_STATIC void ZSTD_upscaleStats(optState_t* optPtr)
+{
+ if (ZSTD_compressedLiterals(optPtr))
+ optPtr->litSum = ZSTD_upscaleStat(optPtr->litFreq, MaxLit, 0);
+ optPtr->litLengthSum = ZSTD_upscaleStat(optPtr->litLengthFreq, MaxLL, 0);
+ optPtr->matchLengthSum = ZSTD_upscaleStat(optPtr->matchLengthFreq, MaxML, 0);
+ optPtr->offCodeSum = ZSTD_upscaleStat(optPtr->offCodeFreq, MaxOff, 0);
+}
+
+/* ZSTD_initStats_ultra():
+ * make a first compression pass, just to seed stats with more accurate starting values.
+ * only works on first block, with no dictionary and no ldm.
+ * this function cannot error, hence its contract must be respected.
+ */
+static void
+ZSTD_initStats_ultra(ZSTD_matchState_t* ms,
+ seqStore_t* seqStore,
+ U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ U32 tmpRep[ZSTD_REP_NUM]; /* updated rep codes will sink here */
+ ZSTD_memcpy(tmpRep, rep, sizeof(tmpRep));
+
+ DEBUGLOG(4, "ZSTD_initStats_ultra (srcSize=%zu)", srcSize);
+ assert(ms->opt.litLengthSum == 0); /* first block */
+ assert(seqStore->sequences == seqStore->sequencesStart); /* no ldm */
+ assert(ms->window.dictLimit == ms->window.lowLimit); /* no dictionary */
+ assert(ms->window.dictLimit - ms->nextToUpdate <= 1); /* no prefix (note: intentional overflow, defined as 2-complement) */
+
+ ZSTD_compressBlock_opt_generic(ms, seqStore, tmpRep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict); /* generate stats into ms->opt*/
+
+ /* invalidate first scan from history */
+ ZSTD_resetSeqStore(seqStore);
+ ms->window.base -= srcSize;
+ ms->window.dictLimit += (U32)srcSize;
+ ms->window.lowLimit = ms->window.dictLimit;
+ ms->nextToUpdate = ms->window.dictLimit;
+
+ /* re-inforce weight of collected statistics */
+ ZSTD_upscaleStats(&ms->opt);
+}
+
+size_t ZSTD_compressBlock_btultra(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_compressBlock_btultra (srcSize=%zu)", srcSize);
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_btultra2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ U32 const curr = (U32)((const BYTE*)src - ms->window.base);
+ DEBUGLOG(5, "ZSTD_compressBlock_btultra2 (srcSize=%zu)", srcSize);
+
+ /* 2-pass strategy:
+ * this strategy makes a first pass over first block to collect statistics
+ * and seed next round's statistics with it.
+ * After 1st pass, function forgets everything, and starts a new block.
+ * Consequently, this can only work if no data has been previously loaded in tables,
+ * aka, no dictionary, no prefix, no ldm preprocessing.
+ * The compression ratio gain is generally small (~0.5% on first block),
+ * the cost is 2x cpu time on first block. */
+ assert(srcSize <= ZSTD_BLOCKSIZE_MAX);
+ if ( (ms->opt.litLengthSum==0) /* first block */
+ && (seqStore->sequences == seqStore->sequencesStart) /* no ldm */
+ && (ms->window.dictLimit == ms->window.lowLimit) /* no dictionary */
+ && (curr == ms->window.dictLimit) /* start of frame, nothing already loaded nor skipped */
+ && (srcSize > ZSTD_PREDEF_THRESHOLD)
+ ) {
+ ZSTD_initStats_ultra(ms, seqStore, rep, src, srcSize);
+ }
+
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_btopt_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_btultra_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_btopt_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_extDict);
+}
+
+size_t ZSTD_compressBlock_btultra_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_extDict);
+}
+
+/* note : no btultra2 variant for extDict nor dictMatchState,
+ * because btultra2 is not meant to work with dictionaries
+ * and is only specific for the first block (no prefix) */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_OPT_H
+#define ZSTD_OPT_H
+
+
+#include "zstd_compress_internal.h"
+
+/* used in ZSTD_loadDictionaryContent() */
+void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend);
+
+size_t ZSTD_compressBlock_btopt(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_btultra(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_btultra2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+
+size_t ZSTD_compressBlock_btopt_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_btultra_dictMatchState(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+size_t ZSTD_compressBlock_btopt_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_btultra_extDict(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+
+ /* note : no btultra2 variant for extDict nor dictMatchState,
+ * because btultra2 is not meant to work with dictionaries
+ * and is only specific for the first block (no prefix) */
+
+
+#endif /* ZSTD_OPT_H */
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-/* ***************************************************************
-* Tuning parameters
-*****************************************************************/
-/*!
-* MAXWINDOWSIZE_DEFAULT :
-* maximum window size accepted by DStream, by default.
-* Frames requiring more memory will be rejected.
-*/
-#ifndef ZSTD_MAXWINDOWSIZE_DEFAULT
-#define ZSTD_MAXWINDOWSIZE_DEFAULT ((1 << ZSTD_WINDOWLOG_MAX) + 1) /* defined within zstd.h */
-#endif
-
-/*-*******************************************************
-* Dependencies
-*********************************************************/
-#include "fse.h"
-#include "huf.h"
-#include "mem.h" /* low level memory routines */
-#include "zstd_internal.h"
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h> /* memcpy, memmove, memset */
-
-#define ZSTD_PREFETCH(ptr) __builtin_prefetch(ptr, 0, 0)
-
-/*-*************************************
-* Macros
-***************************************/
-#define ZSTD_isError ERR_isError /* for inlining */
-#define FSE_isError ERR_isError
-#define HUF_isError ERR_isError
-
-/*_*******************************************************
-* Memory operations
-**********************************************************/
-static void ZSTD_copy4(void *dst, const void *src) { memcpy(dst, src, 4); }
-
-/*-*************************************************************
-* Context management
-***************************************************************/
-typedef enum {
- ZSTDds_getFrameHeaderSize,
- ZSTDds_decodeFrameHeader,
- ZSTDds_decodeBlockHeader,
- ZSTDds_decompressBlock,
- ZSTDds_decompressLastBlock,
- ZSTDds_checkChecksum,
- ZSTDds_decodeSkippableHeader,
- ZSTDds_skipFrame
-} ZSTD_dStage;
-
-typedef struct {
- FSE_DTable LLTable[FSE_DTABLE_SIZE_U32(LLFSELog)];
- FSE_DTable OFTable[FSE_DTABLE_SIZE_U32(OffFSELog)];
- FSE_DTable MLTable[FSE_DTABLE_SIZE_U32(MLFSELog)];
- HUF_DTable hufTable[HUF_DTABLE_SIZE(HufLog)]; /* can accommodate HUF_decompress4X */
- U64 workspace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32 / 2];
- U32 rep[ZSTD_REP_NUM];
-} ZSTD_entropyTables_t;
-
-struct ZSTD_DCtx_s {
- const FSE_DTable *LLTptr;
- const FSE_DTable *MLTptr;
- const FSE_DTable *OFTptr;
- const HUF_DTable *HUFptr;
- ZSTD_entropyTables_t entropy;
- const void *previousDstEnd; /* detect continuity */
- const void *base; /* start of curr segment */
- const void *vBase; /* virtual start of previous segment if it was just before curr one */
- const void *dictEnd; /* end of previous segment */
- size_t expected;
- ZSTD_frameParams fParams;
- blockType_e bType; /* used in ZSTD_decompressContinue(), to transfer blockType between header decoding and block decoding stages */
- ZSTD_dStage stage;
- U32 litEntropy;
- U32 fseEntropy;
- struct xxh64_state xxhState;
- size_t headerSize;
- U32 dictID;
- const BYTE *litPtr;
- ZSTD_customMem customMem;
- size_t litSize;
- size_t rleSize;
- BYTE litBuffer[ZSTD_BLOCKSIZE_ABSOLUTEMAX + WILDCOPY_OVERLENGTH];
- BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
-}; /* typedef'd to ZSTD_DCtx within "zstd.h" */
-
-size_t ZSTD_DCtxWorkspaceBound(void) { return ZSTD_ALIGN(sizeof(ZSTD_stack)) + ZSTD_ALIGN(sizeof(ZSTD_DCtx)); }
-
-size_t ZSTD_decompressBegin(ZSTD_DCtx *dctx)
-{
- dctx->expected = ZSTD_frameHeaderSize_prefix;
- dctx->stage = ZSTDds_getFrameHeaderSize;
- dctx->previousDstEnd = NULL;
- dctx->base = NULL;
- dctx->vBase = NULL;
- dctx->dictEnd = NULL;
- dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
- dctx->litEntropy = dctx->fseEntropy = 0;
- dctx->dictID = 0;
- ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
- memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */
- dctx->LLTptr = dctx->entropy.LLTable;
- dctx->MLTptr = dctx->entropy.MLTable;
- dctx->OFTptr = dctx->entropy.OFTable;
- dctx->HUFptr = dctx->entropy.hufTable;
- return 0;
-}
-
-ZSTD_DCtx *ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
-{
- ZSTD_DCtx *dctx;
-
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
-
- dctx = (ZSTD_DCtx *)ZSTD_malloc(sizeof(ZSTD_DCtx), customMem);
- if (!dctx)
- return NULL;
- memcpy(&dctx->customMem, &customMem, sizeof(customMem));
- ZSTD_decompressBegin(dctx);
- return dctx;
-}
-
-ZSTD_DCtx *ZSTD_initDCtx(void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- return ZSTD_createDCtx_advanced(stackMem);
-}
-
-size_t ZSTD_freeDCtx(ZSTD_DCtx *dctx)
-{
- if (dctx == NULL)
- return 0; /* support free on NULL */
- ZSTD_free(dctx, dctx->customMem);
- return 0; /* reserved as a potential error code in the future */
-}
-
-void ZSTD_copyDCtx(ZSTD_DCtx *dstDCtx, const ZSTD_DCtx *srcDCtx)
-{
- size_t const workSpaceSize = (ZSTD_BLOCKSIZE_ABSOLUTEMAX + WILDCOPY_OVERLENGTH) + ZSTD_frameHeaderSize_max;
- memcpy(dstDCtx, srcDCtx, sizeof(ZSTD_DCtx) - workSpaceSize); /* no need to copy workspace */
-}
-
-static void ZSTD_refDDict(ZSTD_DCtx *dstDCtx, const ZSTD_DDict *ddict);
-
-/*-*************************************************************
-* Decompression section
-***************************************************************/
-
-/*! ZSTD_isFrame() :
- * Tells if the content of `buffer` starts with a valid Frame Identifier.
- * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
- * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
- * Note 3 : Skippable Frame Identifiers are considered valid. */
-unsigned ZSTD_isFrame(const void *buffer, size_t size)
-{
- if (size < 4)
- return 0;
- {
- U32 const magic = ZSTD_readLE32(buffer);
- if (magic == ZSTD_MAGICNUMBER)
- return 1;
- if ((magic & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START)
- return 1;
- }
- return 0;
-}
-
-/** ZSTD_frameHeaderSize() :
-* srcSize must be >= ZSTD_frameHeaderSize_prefix.
-* @return : size of the Frame Header */
-static size_t ZSTD_frameHeaderSize(const void *src, size_t srcSize)
-{
- if (srcSize < ZSTD_frameHeaderSize_prefix)
- return ERROR(srcSize_wrong);
- {
- BYTE const fhd = ((const BYTE *)src)[4];
- U32 const dictID = fhd & 3;
- U32 const singleSegment = (fhd >> 5) & 1;
- U32 const fcsId = fhd >> 6;
- return ZSTD_frameHeaderSize_prefix + !singleSegment + ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId] + (singleSegment && !fcsId);
- }
-}
-
-/** ZSTD_getFrameParams() :
-* decode Frame Header, or require larger `srcSize`.
-* @return : 0, `fparamsPtr` is correctly filled,
-* >0, `srcSize` is too small, result is expected `srcSize`,
-* or an error code, which can be tested using ZSTD_isError() */
-size_t ZSTD_getFrameParams(ZSTD_frameParams *fparamsPtr, const void *src, size_t srcSize)
-{
- const BYTE *ip = (const BYTE *)src;
-
- if (srcSize < ZSTD_frameHeaderSize_prefix)
- return ZSTD_frameHeaderSize_prefix;
- if (ZSTD_readLE32(src) != ZSTD_MAGICNUMBER) {
- if ((ZSTD_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
- if (srcSize < ZSTD_skippableHeaderSize)
- return ZSTD_skippableHeaderSize; /* magic number + skippable frame length */
- memset(fparamsPtr, 0, sizeof(*fparamsPtr));
- fparamsPtr->frameContentSize = ZSTD_readLE32((const char *)src + 4);
- fparamsPtr->windowSize = 0; /* windowSize==0 means a frame is skippable */
- return 0;
- }
- return ERROR(prefix_unknown);
- }
-
- /* ensure there is enough `srcSize` to fully read/decode frame header */
- {
- size_t const fhsize = ZSTD_frameHeaderSize(src, srcSize);
- if (srcSize < fhsize)
- return fhsize;
- }
-
- {
- BYTE const fhdByte = ip[4];
- size_t pos = 5;
- U32 const dictIDSizeCode = fhdByte & 3;
- U32 const checksumFlag = (fhdByte >> 2) & 1;
- U32 const singleSegment = (fhdByte >> 5) & 1;
- U32 const fcsID = fhdByte >> 6;
- U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX;
- U32 windowSize = 0;
- U32 dictID = 0;
- U64 frameContentSize = 0;
- if ((fhdByte & 0x08) != 0)
- return ERROR(frameParameter_unsupported); /* reserved bits, which must be zero */
- if (!singleSegment) {
- BYTE const wlByte = ip[pos++];
- U32 const windowLog = (wlByte >> 3) + ZSTD_WINDOWLOG_ABSOLUTEMIN;
- if (windowLog > ZSTD_WINDOWLOG_MAX)
- return ERROR(frameParameter_windowTooLarge); /* avoids issue with 1 << windowLog */
- windowSize = (1U << windowLog);
- windowSize += (windowSize >> 3) * (wlByte & 7);
- }
-
- switch (dictIDSizeCode) {
- default: /* impossible */
- case 0: break;
- case 1:
- dictID = ip[pos];
- pos++;
- break;
- case 2:
- dictID = ZSTD_readLE16(ip + pos);
- pos += 2;
- break;
- case 3:
- dictID = ZSTD_readLE32(ip + pos);
- pos += 4;
- break;
- }
- switch (fcsID) {
- default: /* impossible */
- case 0:
- if (singleSegment)
- frameContentSize = ip[pos];
- break;
- case 1: frameContentSize = ZSTD_readLE16(ip + pos) + 256; break;
- case 2: frameContentSize = ZSTD_readLE32(ip + pos); break;
- case 3: frameContentSize = ZSTD_readLE64(ip + pos); break;
- }
- if (!windowSize)
- windowSize = (U32)frameContentSize;
- if (windowSize > windowSizeMax)
- return ERROR(frameParameter_windowTooLarge);
- fparamsPtr->frameContentSize = frameContentSize;
- fparamsPtr->windowSize = windowSize;
- fparamsPtr->dictID = dictID;
- fparamsPtr->checksumFlag = checksumFlag;
- }
- return 0;
-}
-
-/** ZSTD_getFrameContentSize() :
-* compatible with legacy mode
-* @return : decompressed size of the single frame pointed to be `src` if known, otherwise
-* - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
-* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */
-unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize)
-{
- {
- ZSTD_frameParams fParams;
- if (ZSTD_getFrameParams(&fParams, src, srcSize) != 0)
- return ZSTD_CONTENTSIZE_ERROR;
- if (fParams.windowSize == 0) {
- /* Either skippable or empty frame, size == 0 either way */
- return 0;
- } else if (fParams.frameContentSize != 0) {
- return fParams.frameContentSize;
- } else {
- return ZSTD_CONTENTSIZE_UNKNOWN;
- }
- }
-}
-
-/** ZSTD_findDecompressedSize() :
- * compatible with legacy mode
- * `srcSize` must be the exact length of some number of ZSTD compressed and/or
- * skippable frames
- * @return : decompressed size of the frames contained */
-unsigned long long ZSTD_findDecompressedSize(const void *src, size_t srcSize)
-{
- {
- unsigned long long totalDstSize = 0;
- while (srcSize >= ZSTD_frameHeaderSize_prefix) {
- const U32 magicNumber = ZSTD_readLE32(src);
-
- if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
- size_t skippableSize;
- if (srcSize < ZSTD_skippableHeaderSize)
- return ERROR(srcSize_wrong);
- skippableSize = ZSTD_readLE32((const BYTE *)src + 4) + ZSTD_skippableHeaderSize;
- if (srcSize < skippableSize) {
- return ZSTD_CONTENTSIZE_ERROR;
- }
-
- src = (const BYTE *)src + skippableSize;
- srcSize -= skippableSize;
- continue;
- }
-
- {
- unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
- if (ret >= ZSTD_CONTENTSIZE_ERROR)
- return ret;
-
- /* check for overflow */
- if (totalDstSize + ret < totalDstSize)
- return ZSTD_CONTENTSIZE_ERROR;
- totalDstSize += ret;
- }
- {
- size_t const frameSrcSize = ZSTD_findFrameCompressedSize(src, srcSize);
- if (ZSTD_isError(frameSrcSize)) {
- return ZSTD_CONTENTSIZE_ERROR;
- }
-
- src = (const BYTE *)src + frameSrcSize;
- srcSize -= frameSrcSize;
- }
- }
-
- if (srcSize) {
- return ZSTD_CONTENTSIZE_ERROR;
- }
-
- return totalDstSize;
- }
-}
-
-/** ZSTD_decodeFrameHeader() :
-* `headerSize` must be the size provided by ZSTD_frameHeaderSize().
-* @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */
-static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx *dctx, const void *src, size_t headerSize)
-{
- size_t const result = ZSTD_getFrameParams(&(dctx->fParams), src, headerSize);
- if (ZSTD_isError(result))
- return result; /* invalid header */
- if (result > 0)
- return ERROR(srcSize_wrong); /* headerSize too small */
- if (dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID))
- return ERROR(dictionary_wrong);
- if (dctx->fParams.checksumFlag)
- xxh64_reset(&dctx->xxhState, 0);
- return 0;
-}
-
-typedef struct {
- blockType_e blockType;
- U32 lastBlock;
- U32 origSize;
-} blockProperties_t;
-
-/*! ZSTD_getcBlockSize() :
-* Provides the size of compressed block from block header `src` */
-size_t ZSTD_getcBlockSize(const void *src, size_t srcSize, blockProperties_t *bpPtr)
-{
- if (srcSize < ZSTD_blockHeaderSize)
- return ERROR(srcSize_wrong);
- {
- U32 const cBlockHeader = ZSTD_readLE24(src);
- U32 const cSize = cBlockHeader >> 3;
- bpPtr->lastBlock = cBlockHeader & 1;
- bpPtr->blockType = (blockType_e)((cBlockHeader >> 1) & 3);
- bpPtr->origSize = cSize; /* only useful for RLE */
- if (bpPtr->blockType == bt_rle)
- return 1;
- if (bpPtr->blockType == bt_reserved)
- return ERROR(corruption_detected);
- return cSize;
- }
-}
-
-static size_t ZSTD_copyRawBlock(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- if (srcSize > dstCapacity)
- return ERROR(dstSize_tooSmall);
- memcpy(dst, src, srcSize);
- return srcSize;
-}
-
-static size_t ZSTD_setRleBlock(void *dst, size_t dstCapacity, const void *src, size_t srcSize, size_t regenSize)
-{
- if (srcSize != 1)
- return ERROR(srcSize_wrong);
- if (regenSize > dstCapacity)
- return ERROR(dstSize_tooSmall);
- memset(dst, *(const BYTE *)src, regenSize);
- return regenSize;
-}
-
-/*! ZSTD_decodeLiteralsBlock() :
- @return : nb of bytes read from src (< srcSize ) */
-size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx *dctx, const void *src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
-{
- if (srcSize < MIN_CBLOCK_SIZE)
- return ERROR(corruption_detected);
-
- {
- const BYTE *const istart = (const BYTE *)src;
- symbolEncodingType_e const litEncType = (symbolEncodingType_e)(istart[0] & 3);
-
- switch (litEncType) {
- case set_repeat:
- if (dctx->litEntropy == 0)
- return ERROR(dictionary_corrupted);
- fallthrough;
- case set_compressed:
- if (srcSize < 5)
- return ERROR(corruption_detected); /* srcSize >= MIN_CBLOCK_SIZE == 3; here we need up to 5 for case 3 */
- {
- size_t lhSize, litSize, litCSize;
- U32 singleStream = 0;
- U32 const lhlCode = (istart[0] >> 2) & 3;
- U32 const lhc = ZSTD_readLE32(istart);
- switch (lhlCode) {
- case 0:
- case 1:
- default: /* note : default is impossible, since lhlCode into [0..3] */
- /* 2 - 2 - 10 - 10 */
- singleStream = !lhlCode;
- lhSize = 3;
- litSize = (lhc >> 4) & 0x3FF;
- litCSize = (lhc >> 14) & 0x3FF;
- break;
- case 2:
- /* 2 - 2 - 14 - 14 */
- lhSize = 4;
- litSize = (lhc >> 4) & 0x3FFF;
- litCSize = lhc >> 18;
- break;
- case 3:
- /* 2 - 2 - 18 - 18 */
- lhSize = 5;
- litSize = (lhc >> 4) & 0x3FFFF;
- litCSize = (lhc >> 22) + (istart[4] << 10);
- break;
- }
- if (litSize > ZSTD_BLOCKSIZE_ABSOLUTEMAX)
- return ERROR(corruption_detected);
- if (litCSize + lhSize > srcSize)
- return ERROR(corruption_detected);
-
- if (HUF_isError(
- (litEncType == set_repeat)
- ? (singleStream ? HUF_decompress1X_usingDTable(dctx->litBuffer, litSize, istart + lhSize, litCSize, dctx->HUFptr)
- : HUF_decompress4X_usingDTable(dctx->litBuffer, litSize, istart + lhSize, litCSize, dctx->HUFptr))
- : (singleStream
- ? HUF_decompress1X2_DCtx_wksp(dctx->entropy.hufTable, dctx->litBuffer, litSize, istart + lhSize, litCSize,
- dctx->entropy.workspace, sizeof(dctx->entropy.workspace))
- : HUF_decompress4X_hufOnly_wksp(dctx->entropy.hufTable, dctx->litBuffer, litSize, istart + lhSize, litCSize,
- dctx->entropy.workspace, sizeof(dctx->entropy.workspace)))))
- return ERROR(corruption_detected);
-
- dctx->litPtr = dctx->litBuffer;
- dctx->litSize = litSize;
- dctx->litEntropy = 1;
- if (litEncType == set_compressed)
- dctx->HUFptr = dctx->entropy.hufTable;
- memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
- return litCSize + lhSize;
- }
-
- case set_basic: {
- size_t litSize, lhSize;
- U32 const lhlCode = ((istart[0]) >> 2) & 3;
- switch (lhlCode) {
- case 0:
- case 2:
- default: /* note : default is impossible, since lhlCode into [0..3] */
- lhSize = 1;
- litSize = istart[0] >> 3;
- break;
- case 1:
- lhSize = 2;
- litSize = ZSTD_readLE16(istart) >> 4;
- break;
- case 3:
- lhSize = 3;
- litSize = ZSTD_readLE24(istart) >> 4;
- break;
- }
-
- if (lhSize + litSize + WILDCOPY_OVERLENGTH > srcSize) { /* risk reading beyond src buffer with wildcopy */
- if (litSize + lhSize > srcSize)
- return ERROR(corruption_detected);
- memcpy(dctx->litBuffer, istart + lhSize, litSize);
- dctx->litPtr = dctx->litBuffer;
- dctx->litSize = litSize;
- memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
- return lhSize + litSize;
- }
- /* direct reference into compressed stream */
- dctx->litPtr = istart + lhSize;
- dctx->litSize = litSize;
- return lhSize + litSize;
- }
-
- case set_rle: {
- U32 const lhlCode = ((istart[0]) >> 2) & 3;
- size_t litSize, lhSize;
- switch (lhlCode) {
- case 0:
- case 2:
- default: /* note : default is impossible, since lhlCode into [0..3] */
- lhSize = 1;
- litSize = istart[0] >> 3;
- break;
- case 1:
- lhSize = 2;
- litSize = ZSTD_readLE16(istart) >> 4;
- break;
- case 3:
- lhSize = 3;
- litSize = ZSTD_readLE24(istart) >> 4;
- if (srcSize < 4)
- return ERROR(corruption_detected); /* srcSize >= MIN_CBLOCK_SIZE == 3; here we need lhSize+1 = 4 */
- break;
- }
- if (litSize > ZSTD_BLOCKSIZE_ABSOLUTEMAX)
- return ERROR(corruption_detected);
- memset(dctx->litBuffer, istart[lhSize], litSize + WILDCOPY_OVERLENGTH);
- dctx->litPtr = dctx->litBuffer;
- dctx->litSize = litSize;
- return lhSize + 1;
- }
- default:
- return ERROR(corruption_detected); /* impossible */
- }
- }
-}
-
-typedef union {
- FSE_decode_t realData;
- U32 alignedBy4;
-} FSE_decode_t4;
-
-static const FSE_decode_t4 LL_defaultDTable[(1 << LL_DEFAULTNORMLOG) + 1] = {
- {{LL_DEFAULTNORMLOG, 1, 1}}, /* header : tableLog, fastMode, fastMode */
- {{0, 0, 4}}, /* 0 : base, symbol, bits */
- {{16, 0, 4}},
- {{32, 1, 5}},
- {{0, 3, 5}},
- {{0, 4, 5}},
- {{0, 6, 5}},
- {{0, 7, 5}},
- {{0, 9, 5}},
- {{0, 10, 5}},
- {{0, 12, 5}},
- {{0, 14, 6}},
- {{0, 16, 5}},
- {{0, 18, 5}},
- {{0, 19, 5}},
- {{0, 21, 5}},
- {{0, 22, 5}},
- {{0, 24, 5}},
- {{32, 25, 5}},
- {{0, 26, 5}},
- {{0, 27, 6}},
- {{0, 29, 6}},
- {{0, 31, 6}},
- {{32, 0, 4}},
- {{0, 1, 4}},
- {{0, 2, 5}},
- {{32, 4, 5}},
- {{0, 5, 5}},
- {{32, 7, 5}},
- {{0, 8, 5}},
- {{32, 10, 5}},
- {{0, 11, 5}},
- {{0, 13, 6}},
- {{32, 16, 5}},
- {{0, 17, 5}},
- {{32, 19, 5}},
- {{0, 20, 5}},
- {{32, 22, 5}},
- {{0, 23, 5}},
- {{0, 25, 4}},
- {{16, 25, 4}},
- {{32, 26, 5}},
- {{0, 28, 6}},
- {{0, 30, 6}},
- {{48, 0, 4}},
- {{16, 1, 4}},
- {{32, 2, 5}},
- {{32, 3, 5}},
- {{32, 5, 5}},
- {{32, 6, 5}},
- {{32, 8, 5}},
- {{32, 9, 5}},
- {{32, 11, 5}},
- {{32, 12, 5}},
- {{0, 15, 6}},
- {{32, 17, 5}},
- {{32, 18, 5}},
- {{32, 20, 5}},
- {{32, 21, 5}},
- {{32, 23, 5}},
- {{32, 24, 5}},
- {{0, 35, 6}},
- {{0, 34, 6}},
- {{0, 33, 6}},
- {{0, 32, 6}},
-}; /* LL_defaultDTable */
-
-static const FSE_decode_t4 ML_defaultDTable[(1 << ML_DEFAULTNORMLOG) + 1] = {
- {{ML_DEFAULTNORMLOG, 1, 1}}, /* header : tableLog, fastMode, fastMode */
- {{0, 0, 6}}, /* 0 : base, symbol, bits */
- {{0, 1, 4}},
- {{32, 2, 5}},
- {{0, 3, 5}},
- {{0, 5, 5}},
- {{0, 6, 5}},
- {{0, 8, 5}},
- {{0, 10, 6}},
- {{0, 13, 6}},
- {{0, 16, 6}},
- {{0, 19, 6}},
- {{0, 22, 6}},
- {{0, 25, 6}},
- {{0, 28, 6}},
- {{0, 31, 6}},
- {{0, 33, 6}},
- {{0, 35, 6}},
- {{0, 37, 6}},
- {{0, 39, 6}},
- {{0, 41, 6}},
- {{0, 43, 6}},
- {{0, 45, 6}},
- {{16, 1, 4}},
- {{0, 2, 4}},
- {{32, 3, 5}},
- {{0, 4, 5}},
- {{32, 6, 5}},
- {{0, 7, 5}},
- {{0, 9, 6}},
- {{0, 12, 6}},
- {{0, 15, 6}},
- {{0, 18, 6}},
- {{0, 21, 6}},
- {{0, 24, 6}},
- {{0, 27, 6}},
- {{0, 30, 6}},
- {{0, 32, 6}},
- {{0, 34, 6}},
- {{0, 36, 6}},
- {{0, 38, 6}},
- {{0, 40, 6}},
- {{0, 42, 6}},
- {{0, 44, 6}},
- {{32, 1, 4}},
- {{48, 1, 4}},
- {{16, 2, 4}},
- {{32, 4, 5}},
- {{32, 5, 5}},
- {{32, 7, 5}},
- {{32, 8, 5}},
- {{0, 11, 6}},
- {{0, 14, 6}},
- {{0, 17, 6}},
- {{0, 20, 6}},
- {{0, 23, 6}},
- {{0, 26, 6}},
- {{0, 29, 6}},
- {{0, 52, 6}},
- {{0, 51, 6}},
- {{0, 50, 6}},
- {{0, 49, 6}},
- {{0, 48, 6}},
- {{0, 47, 6}},
- {{0, 46, 6}},
-}; /* ML_defaultDTable */
-
-static const FSE_decode_t4 OF_defaultDTable[(1 << OF_DEFAULTNORMLOG) + 1] = {
- {{OF_DEFAULTNORMLOG, 1, 1}}, /* header : tableLog, fastMode, fastMode */
- {{0, 0, 5}}, /* 0 : base, symbol, bits */
- {{0, 6, 4}},
- {{0, 9, 5}},
- {{0, 15, 5}},
- {{0, 21, 5}},
- {{0, 3, 5}},
- {{0, 7, 4}},
- {{0, 12, 5}},
- {{0, 18, 5}},
- {{0, 23, 5}},
- {{0, 5, 5}},
- {{0, 8, 4}},
- {{0, 14, 5}},
- {{0, 20, 5}},
- {{0, 2, 5}},
- {{16, 7, 4}},
- {{0, 11, 5}},
- {{0, 17, 5}},
- {{0, 22, 5}},
- {{0, 4, 5}},
- {{16, 8, 4}},
- {{0, 13, 5}},
- {{0, 19, 5}},
- {{0, 1, 5}},
- {{16, 6, 4}},
- {{0, 10, 5}},
- {{0, 16, 5}},
- {{0, 28, 5}},
- {{0, 27, 5}},
- {{0, 26, 5}},
- {{0, 25, 5}},
- {{0, 24, 5}},
-}; /* OF_defaultDTable */
-
-/*! ZSTD_buildSeqTable() :
- @return : nb bytes read from src,
- or an error code if it fails, testable with ZSTD_isError()
-*/
-static size_t ZSTD_buildSeqTable(FSE_DTable *DTableSpace, const FSE_DTable **DTablePtr, symbolEncodingType_e type, U32 max, U32 maxLog, const void *src,
- size_t srcSize, const FSE_decode_t4 *defaultTable, U32 flagRepeatTable, void *workspace, size_t workspaceSize)
-{
- const void *const tmpPtr = defaultTable; /* bypass strict aliasing */
- switch (type) {
- case set_rle:
- if (!srcSize)
- return ERROR(srcSize_wrong);
- if ((*(const BYTE *)src) > max)
- return ERROR(corruption_detected);
- FSE_buildDTable_rle(DTableSpace, *(const BYTE *)src);
- *DTablePtr = DTableSpace;
- return 1;
- case set_basic: *DTablePtr = (const FSE_DTable *)tmpPtr; return 0;
- case set_repeat:
- if (!flagRepeatTable)
- return ERROR(corruption_detected);
- return 0;
- default: /* impossible */
- case set_compressed: {
- U32 tableLog;
- S16 *norm = (S16 *)workspace;
- size_t const spaceUsed32 = ALIGN(sizeof(S16) * (MaxSeq + 1), sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(GENERIC);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
- {
- size_t const headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize);
- if (FSE_isError(headerSize))
- return ERROR(corruption_detected);
- if (tableLog > maxLog)
- return ERROR(corruption_detected);
- FSE_buildDTable_wksp(DTableSpace, norm, max, tableLog, workspace, workspaceSize);
- *DTablePtr = DTableSpace;
- return headerSize;
- }
- }
- }
-}
-
-size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx *dctx, int *nbSeqPtr, const void *src, size_t srcSize)
-{
- const BYTE *const istart = (const BYTE *const)src;
- const BYTE *const iend = istart + srcSize;
- const BYTE *ip = istart;
-
- /* check */
- if (srcSize < MIN_SEQUENCES_SIZE)
- return ERROR(srcSize_wrong);
-
- /* SeqHead */
- {
- int nbSeq = *ip++;
- if (!nbSeq) {
- *nbSeqPtr = 0;
- return 1;
- }
- if (nbSeq > 0x7F) {
- if (nbSeq == 0xFF) {
- if (ip + 2 > iend)
- return ERROR(srcSize_wrong);
- nbSeq = ZSTD_readLE16(ip) + LONGNBSEQ, ip += 2;
- } else {
- if (ip >= iend)
- return ERROR(srcSize_wrong);
- nbSeq = ((nbSeq - 0x80) << 8) + *ip++;
- }
- }
- *nbSeqPtr = nbSeq;
- }
-
- /* FSE table descriptors */
- if (ip + 4 > iend)
- return ERROR(srcSize_wrong); /* minimum possible size */
- {
- symbolEncodingType_e const LLtype = (symbolEncodingType_e)(*ip >> 6);
- symbolEncodingType_e const OFtype = (symbolEncodingType_e)((*ip >> 4) & 3);
- symbolEncodingType_e const MLtype = (symbolEncodingType_e)((*ip >> 2) & 3);
- ip++;
-
- /* Build DTables */
- {
- size_t const llhSize = ZSTD_buildSeqTable(dctx->entropy.LLTable, &dctx->LLTptr, LLtype, MaxLL, LLFSELog, ip, iend - ip,
- LL_defaultDTable, dctx->fseEntropy, dctx->entropy.workspace, sizeof(dctx->entropy.workspace));
- if (ZSTD_isError(llhSize))
- return ERROR(corruption_detected);
- ip += llhSize;
- }
- {
- size_t const ofhSize = ZSTD_buildSeqTable(dctx->entropy.OFTable, &dctx->OFTptr, OFtype, MaxOff, OffFSELog, ip, iend - ip,
- OF_defaultDTable, dctx->fseEntropy, dctx->entropy.workspace, sizeof(dctx->entropy.workspace));
- if (ZSTD_isError(ofhSize))
- return ERROR(corruption_detected);
- ip += ofhSize;
- }
- {
- size_t const mlhSize = ZSTD_buildSeqTable(dctx->entropy.MLTable, &dctx->MLTptr, MLtype, MaxML, MLFSELog, ip, iend - ip,
- ML_defaultDTable, dctx->fseEntropy, dctx->entropy.workspace, sizeof(dctx->entropy.workspace));
- if (ZSTD_isError(mlhSize))
- return ERROR(corruption_detected);
- ip += mlhSize;
- }
- }
-
- return ip - istart;
-}
-
-typedef struct {
- size_t litLength;
- size_t matchLength;
- size_t offset;
- const BYTE *match;
-} seq_t;
-
-typedef struct {
- BIT_DStream_t DStream;
- FSE_DState_t stateLL;
- FSE_DState_t stateOffb;
- FSE_DState_t stateML;
- size_t prevOffset[ZSTD_REP_NUM];
- const BYTE *base;
- size_t pos;
- uPtrDiff gotoDict;
-} seqState_t;
-
-FORCE_NOINLINE
-size_t ZSTD_execSequenceLast7(BYTE *op, BYTE *const oend, seq_t sequence, const BYTE **litPtr, const BYTE *const litLimit, const BYTE *const base,
- const BYTE *const vBase, const BYTE *const dictEnd)
-{
- BYTE *const oLitEnd = op + sequence.litLength;
- size_t const sequenceLength = sequence.litLength + sequence.matchLength;
- BYTE *const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
- BYTE *const oend_w = oend - WILDCOPY_OVERLENGTH;
- const BYTE *const iLitEnd = *litPtr + sequence.litLength;
- const BYTE *match = oLitEnd - sequence.offset;
-
- /* check */
- if (oMatchEnd > oend)
- return ERROR(dstSize_tooSmall); /* last match must start at a minimum distance of WILDCOPY_OVERLENGTH from oend */
- if (iLitEnd > litLimit)
- return ERROR(corruption_detected); /* over-read beyond lit buffer */
- if (oLitEnd <= oend_w)
- return ERROR(GENERIC); /* Precondition */
-
- /* copy literals */
- if (op < oend_w) {
- ZSTD_wildcopy(op, *litPtr, oend_w - op);
- *litPtr += oend_w - op;
- op = oend_w;
- }
- while (op < oLitEnd)
- *op++ = *(*litPtr)++;
-
- /* copy Match */
- if (sequence.offset > (size_t)(oLitEnd - base)) {
- /* offset beyond prefix */
- if (sequence.offset > (size_t)(oLitEnd - vBase))
- return ERROR(corruption_detected);
- match = dictEnd - (base - match);
- if (match + sequence.matchLength <= dictEnd) {
- memmove(oLitEnd, match, sequence.matchLength);
- return sequenceLength;
- }
- /* span extDict & currPrefixSegment */
- {
- size_t const length1 = dictEnd - match;
- memmove(oLitEnd, match, length1);
- op = oLitEnd + length1;
- sequence.matchLength -= length1;
- match = base;
- }
- }
- while (op < oMatchEnd)
- *op++ = *match++;
- return sequenceLength;
-}
-
-static seq_t ZSTD_decodeSequence(seqState_t *seqState)
-{
- seq_t seq;
-
- U32 const llCode = FSE_peekSymbol(&seqState->stateLL);
- U32 const mlCode = FSE_peekSymbol(&seqState->stateML);
- U32 const ofCode = FSE_peekSymbol(&seqState->stateOffb); /* <= maxOff, by table construction */
-
- U32 const llBits = LL_bits[llCode];
- U32 const mlBits = ML_bits[mlCode];
- U32 const ofBits = ofCode;
- U32 const totalBits = llBits + mlBits + ofBits;
-
- static const U32 LL_base[MaxLL + 1] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18,
- 20, 22, 24, 28, 32, 40, 48, 64, 0x80, 0x100, 0x200, 0x400, 0x800, 0x1000, 0x2000, 0x4000, 0x8000, 0x10000};
-
- static const U32 ML_base[MaxML + 1] = {3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
- 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 39, 41,
- 43, 47, 51, 59, 67, 83, 99, 0x83, 0x103, 0x203, 0x403, 0x803, 0x1003, 0x2003, 0x4003, 0x8003, 0x10003};
-
- static const U32 OF_base[MaxOff + 1] = {0, 1, 1, 5, 0xD, 0x1D, 0x3D, 0x7D, 0xFD, 0x1FD,
- 0x3FD, 0x7FD, 0xFFD, 0x1FFD, 0x3FFD, 0x7FFD, 0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD,
- 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD, 0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD};
-
- /* sequence */
- {
- size_t offset;
- if (!ofCode)
- offset = 0;
- else {
- offset = OF_base[ofCode] + BIT_readBitsFast(&seqState->DStream, ofBits); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
- if (ZSTD_32bits())
- BIT_reloadDStream(&seqState->DStream);
- }
-
- if (ofCode <= 1) {
- offset += (llCode == 0);
- if (offset) {
- size_t temp = (offset == 3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
- temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
- if (offset != 1)
- seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset = temp;
- } else {
- offset = seqState->prevOffset[0];
- }
- } else {
- seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset;
- }
- seq.offset = offset;
- }
-
- seq.matchLength = ML_base[mlCode] + ((mlCode > 31) ? BIT_readBitsFast(&seqState->DStream, mlBits) : 0); /* <= 16 bits */
- if (ZSTD_32bits() && (mlBits + llBits > 24))
- BIT_reloadDStream(&seqState->DStream);
-
- seq.litLength = LL_base[llCode] + ((llCode > 15) ? BIT_readBitsFast(&seqState->DStream, llBits) : 0); /* <= 16 bits */
- if (ZSTD_32bits() || (totalBits > 64 - 7 - (LLFSELog + MLFSELog + OffFSELog)))
- BIT_reloadDStream(&seqState->DStream);
-
- /* ANS state update */
- FSE_updateState(&seqState->stateLL, &seqState->DStream); /* <= 9 bits */
- FSE_updateState(&seqState->stateML, &seqState->DStream); /* <= 9 bits */
- if (ZSTD_32bits())
- BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
- FSE_updateState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
-
- seq.match = NULL;
-
- return seq;
-}
-
-FORCE_INLINE
-size_t ZSTD_execSequence(BYTE *op, BYTE *const oend, seq_t sequence, const BYTE **litPtr, const BYTE *const litLimit, const BYTE *const base,
- const BYTE *const vBase, const BYTE *const dictEnd)
-{
- BYTE *const oLitEnd = op + sequence.litLength;
- size_t const sequenceLength = sequence.litLength + sequence.matchLength;
- BYTE *const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
- BYTE *const oend_w = oend - WILDCOPY_OVERLENGTH;
- const BYTE *const iLitEnd = *litPtr + sequence.litLength;
- const BYTE *match = oLitEnd - sequence.offset;
-
- /* check */
- if (oMatchEnd > oend)
- return ERROR(dstSize_tooSmall); /* last match must start at a minimum distance of WILDCOPY_OVERLENGTH from oend */
- if (iLitEnd > litLimit)
- return ERROR(corruption_detected); /* over-read beyond lit buffer */
- if (oLitEnd > oend_w)
- return ZSTD_execSequenceLast7(op, oend, sequence, litPtr, litLimit, base, vBase, dictEnd);
-
- /* copy Literals */
- ZSTD_copy8(op, *litPtr);
- if (sequence.litLength > 8)
- ZSTD_wildcopy(op + 8, (*litPtr) + 8,
- sequence.litLength - 8); /* note : since oLitEnd <= oend-WILDCOPY_OVERLENGTH, no risk of overwrite beyond oend */
- op = oLitEnd;
- *litPtr = iLitEnd; /* update for next sequence */
-
- /* copy Match */
- if (sequence.offset > (size_t)(oLitEnd - base)) {
- /* offset beyond prefix */
- if (sequence.offset > (size_t)(oLitEnd - vBase))
- return ERROR(corruption_detected);
- match = dictEnd + (match - base);
- if (match + sequence.matchLength <= dictEnd) {
- memmove(oLitEnd, match, sequence.matchLength);
- return sequenceLength;
- }
- /* span extDict & currPrefixSegment */
- {
- size_t const length1 = dictEnd - match;
- memmove(oLitEnd, match, length1);
- op = oLitEnd + length1;
- sequence.matchLength -= length1;
- match = base;
- if (op > oend_w || sequence.matchLength < MINMATCH) {
- U32 i;
- for (i = 0; i < sequence.matchLength; ++i)
- op[i] = match[i];
- return sequenceLength;
- }
- }
- }
- /* Requirement: op <= oend_w && sequence.matchLength >= MINMATCH */
-
- /* match within prefix */
- if (sequence.offset < 8) {
- /* close range match, overlap */
- static const U32 dec32table[] = {0, 1, 2, 1, 4, 4, 4, 4}; /* added */
- static const int dec64table[] = {8, 8, 8, 7, 8, 9, 10, 11}; /* subtracted */
- int const sub2 = dec64table[sequence.offset];
- op[0] = match[0];
- op[1] = match[1];
- op[2] = match[2];
- op[3] = match[3];
- match += dec32table[sequence.offset];
- ZSTD_copy4(op + 4, match);
- match -= sub2;
- } else {
- ZSTD_copy8(op, match);
- }
- op += 8;
- match += 8;
-
- if (oMatchEnd > oend - (16 - MINMATCH)) {
- if (op < oend_w) {
- ZSTD_wildcopy(op, match, oend_w - op);
- match += oend_w - op;
- op = oend_w;
- }
- while (op < oMatchEnd)
- *op++ = *match++;
- } else {
- ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength - 8); /* works even if matchLength < 8 */
- }
- return sequenceLength;
-}
-
-static size_t ZSTD_decompressSequences(ZSTD_DCtx *dctx, void *dst, size_t maxDstSize, const void *seqStart, size_t seqSize)
-{
- const BYTE *ip = (const BYTE *)seqStart;
- const BYTE *const iend = ip + seqSize;
- BYTE *const ostart = (BYTE * const)dst;
- BYTE *const oend = ostart + maxDstSize;
- BYTE *op = ostart;
- const BYTE *litPtr = dctx->litPtr;
- const BYTE *const litEnd = litPtr + dctx->litSize;
- const BYTE *const base = (const BYTE *)(dctx->base);
- const BYTE *const vBase = (const BYTE *)(dctx->vBase);
- const BYTE *const dictEnd = (const BYTE *)(dctx->dictEnd);
- int nbSeq;
-
- /* Build Decoding Tables */
- {
- size_t const seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, seqSize);
- if (ZSTD_isError(seqHSize))
- return seqHSize;
- ip += seqHSize;
- }
-
- /* Regen sequences */
- if (nbSeq) {
- seqState_t seqState;
- dctx->fseEntropy = 1;
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- seqState.prevOffset[i] = dctx->entropy.rep[i];
- }
- CHECK_E(BIT_initDStream(&seqState.DStream, ip, iend - ip), corruption_detected);
- FSE_initDState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
- FSE_initDState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
- FSE_initDState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
-
- for (; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && nbSeq;) {
- nbSeq--;
- {
- seq_t const sequence = ZSTD_decodeSequence(&seqState);
- size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, base, vBase, dictEnd);
- if (ZSTD_isError(oneSeqSize))
- return oneSeqSize;
- op += oneSeqSize;
- }
- }
-
- /* check if reached exact end */
- if (nbSeq)
- return ERROR(corruption_detected);
- /* save reps for next block */
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]);
- }
- }
-
- /* last literal segment */
- {
- size_t const lastLLSize = litEnd - litPtr;
- if (lastLLSize > (size_t)(oend - op))
- return ERROR(dstSize_tooSmall);
- memcpy(op, litPtr, lastLLSize);
- op += lastLLSize;
- }
-
- return op - ostart;
-}
-
-FORCE_INLINE seq_t ZSTD_decodeSequenceLong_generic(seqState_t *seqState, int const longOffsets)
-{
- seq_t seq;
-
- U32 const llCode = FSE_peekSymbol(&seqState->stateLL);
- U32 const mlCode = FSE_peekSymbol(&seqState->stateML);
- U32 const ofCode = FSE_peekSymbol(&seqState->stateOffb); /* <= maxOff, by table construction */
-
- U32 const llBits = LL_bits[llCode];
- U32 const mlBits = ML_bits[mlCode];
- U32 const ofBits = ofCode;
- U32 const totalBits = llBits + mlBits + ofBits;
-
- static const U32 LL_base[MaxLL + 1] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18,
- 20, 22, 24, 28, 32, 40, 48, 64, 0x80, 0x100, 0x200, 0x400, 0x800, 0x1000, 0x2000, 0x4000, 0x8000, 0x10000};
-
- static const U32 ML_base[MaxML + 1] = {3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
- 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37, 39, 41,
- 43, 47, 51, 59, 67, 83, 99, 0x83, 0x103, 0x203, 0x403, 0x803, 0x1003, 0x2003, 0x4003, 0x8003, 0x10003};
-
- static const U32 OF_base[MaxOff + 1] = {0, 1, 1, 5, 0xD, 0x1D, 0x3D, 0x7D, 0xFD, 0x1FD,
- 0x3FD, 0x7FD, 0xFFD, 0x1FFD, 0x3FFD, 0x7FFD, 0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD,
- 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD, 0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD};
-
- /* sequence */
- {
- size_t offset;
- if (!ofCode)
- offset = 0;
- else {
- if (longOffsets) {
- int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN);
- offset = OF_base[ofCode] + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
- if (ZSTD_32bits() || extraBits)
- BIT_reloadDStream(&seqState->DStream);
- if (extraBits)
- offset += BIT_readBitsFast(&seqState->DStream, extraBits);
- } else {
- offset = OF_base[ofCode] + BIT_readBitsFast(&seqState->DStream, ofBits); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
- if (ZSTD_32bits())
- BIT_reloadDStream(&seqState->DStream);
- }
- }
-
- if (ofCode <= 1) {
- offset += (llCode == 0);
- if (offset) {
- size_t temp = (offset == 3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
- temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
- if (offset != 1)
- seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset = temp;
- } else {
- offset = seqState->prevOffset[0];
- }
- } else {
- seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset;
- }
- seq.offset = offset;
- }
-
- seq.matchLength = ML_base[mlCode] + ((mlCode > 31) ? BIT_readBitsFast(&seqState->DStream, mlBits) : 0); /* <= 16 bits */
- if (ZSTD_32bits() && (mlBits + llBits > 24))
- BIT_reloadDStream(&seqState->DStream);
-
- seq.litLength = LL_base[llCode] + ((llCode > 15) ? BIT_readBitsFast(&seqState->DStream, llBits) : 0); /* <= 16 bits */
- if (ZSTD_32bits() || (totalBits > 64 - 7 - (LLFSELog + MLFSELog + OffFSELog)))
- BIT_reloadDStream(&seqState->DStream);
-
- {
- size_t const pos = seqState->pos + seq.litLength;
- seq.match = seqState->base + pos - seq.offset; /* single memory segment */
- if (seq.offset > pos)
- seq.match += seqState->gotoDict; /* separate memory segment */
- seqState->pos = pos + seq.matchLength;
- }
-
- /* ANS state update */
- FSE_updateState(&seqState->stateLL, &seqState->DStream); /* <= 9 bits */
- FSE_updateState(&seqState->stateML, &seqState->DStream); /* <= 9 bits */
- if (ZSTD_32bits())
- BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
- FSE_updateState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
-
- return seq;
-}
-
-static seq_t ZSTD_decodeSequenceLong(seqState_t *seqState, unsigned const windowSize)
-{
- if (ZSTD_highbit32(windowSize) > STREAM_ACCUMULATOR_MIN) {
- return ZSTD_decodeSequenceLong_generic(seqState, 1);
- } else {
- return ZSTD_decodeSequenceLong_generic(seqState, 0);
- }
-}
-
-FORCE_INLINE
-size_t ZSTD_execSequenceLong(BYTE *op, BYTE *const oend, seq_t sequence, const BYTE **litPtr, const BYTE *const litLimit, const BYTE *const base,
- const BYTE *const vBase, const BYTE *const dictEnd)
-{
- BYTE *const oLitEnd = op + sequence.litLength;
- size_t const sequenceLength = sequence.litLength + sequence.matchLength;
- BYTE *const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
- BYTE *const oend_w = oend - WILDCOPY_OVERLENGTH;
- const BYTE *const iLitEnd = *litPtr + sequence.litLength;
- const BYTE *match = sequence.match;
-
- /* check */
- if (oMatchEnd > oend)
- return ERROR(dstSize_tooSmall); /* last match must start at a minimum distance of WILDCOPY_OVERLENGTH from oend */
- if (iLitEnd > litLimit)
- return ERROR(corruption_detected); /* over-read beyond lit buffer */
- if (oLitEnd > oend_w)
- return ZSTD_execSequenceLast7(op, oend, sequence, litPtr, litLimit, base, vBase, dictEnd);
-
- /* copy Literals */
- ZSTD_copy8(op, *litPtr);
- if (sequence.litLength > 8)
- ZSTD_wildcopy(op + 8, (*litPtr) + 8,
- sequence.litLength - 8); /* note : since oLitEnd <= oend-WILDCOPY_OVERLENGTH, no risk of overwrite beyond oend */
- op = oLitEnd;
- *litPtr = iLitEnd; /* update for next sequence */
-
- /* copy Match */
- if (sequence.offset > (size_t)(oLitEnd - base)) {
- /* offset beyond prefix */
- if (sequence.offset > (size_t)(oLitEnd - vBase))
- return ERROR(corruption_detected);
- if (match + sequence.matchLength <= dictEnd) {
- memmove(oLitEnd, match, sequence.matchLength);
- return sequenceLength;
- }
- /* span extDict & currPrefixSegment */
- {
- size_t const length1 = dictEnd - match;
- memmove(oLitEnd, match, length1);
- op = oLitEnd + length1;
- sequence.matchLength -= length1;
- match = base;
- if (op > oend_w || sequence.matchLength < MINMATCH) {
- U32 i;
- for (i = 0; i < sequence.matchLength; ++i)
- op[i] = match[i];
- return sequenceLength;
- }
- }
- }
- /* Requirement: op <= oend_w && sequence.matchLength >= MINMATCH */
-
- /* match within prefix */
- if (sequence.offset < 8) {
- /* close range match, overlap */
- static const U32 dec32table[] = {0, 1, 2, 1, 4, 4, 4, 4}; /* added */
- static const int dec64table[] = {8, 8, 8, 7, 8, 9, 10, 11}; /* subtracted */
- int const sub2 = dec64table[sequence.offset];
- op[0] = match[0];
- op[1] = match[1];
- op[2] = match[2];
- op[3] = match[3];
- match += dec32table[sequence.offset];
- ZSTD_copy4(op + 4, match);
- match -= sub2;
- } else {
- ZSTD_copy8(op, match);
- }
- op += 8;
- match += 8;
-
- if (oMatchEnd > oend - (16 - MINMATCH)) {
- if (op < oend_w) {
- ZSTD_wildcopy(op, match, oend_w - op);
- match += oend_w - op;
- op = oend_w;
- }
- while (op < oMatchEnd)
- *op++ = *match++;
- } else {
- ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength - 8); /* works even if matchLength < 8 */
- }
- return sequenceLength;
-}
-
-static size_t ZSTD_decompressSequencesLong(ZSTD_DCtx *dctx, void *dst, size_t maxDstSize, const void *seqStart, size_t seqSize)
-{
- const BYTE *ip = (const BYTE *)seqStart;
- const BYTE *const iend = ip + seqSize;
- BYTE *const ostart = (BYTE * const)dst;
- BYTE *const oend = ostart + maxDstSize;
- BYTE *op = ostart;
- const BYTE *litPtr = dctx->litPtr;
- const BYTE *const litEnd = litPtr + dctx->litSize;
- const BYTE *const base = (const BYTE *)(dctx->base);
- const BYTE *const vBase = (const BYTE *)(dctx->vBase);
- const BYTE *const dictEnd = (const BYTE *)(dctx->dictEnd);
- unsigned const windowSize = dctx->fParams.windowSize;
- int nbSeq;
-
- /* Build Decoding Tables */
- {
- size_t const seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, seqSize);
- if (ZSTD_isError(seqHSize))
- return seqHSize;
- ip += seqHSize;
- }
-
- /* Regen sequences */
- if (nbSeq) {
-#define STORED_SEQS 4
-#define STOSEQ_MASK (STORED_SEQS - 1)
-#define ADVANCED_SEQS 4
- seq_t *sequences = (seq_t *)dctx->entropy.workspace;
- int const seqAdvance = MIN(nbSeq, ADVANCED_SEQS);
- seqState_t seqState;
- int seqNb;
- ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.workspace) >= sizeof(seq_t) * STORED_SEQS);
- dctx->fseEntropy = 1;
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- seqState.prevOffset[i] = dctx->entropy.rep[i];
- }
- seqState.base = base;
- seqState.pos = (size_t)(op - base);
- seqState.gotoDict = (uPtrDiff)dictEnd - (uPtrDiff)base; /* cast to avoid undefined behaviour */
- CHECK_E(BIT_initDStream(&seqState.DStream, ip, iend - ip), corruption_detected);
- FSE_initDState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
- FSE_initDState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
- FSE_initDState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
-
- /* prepare in advance */
- for (seqNb = 0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && seqNb < seqAdvance; seqNb++) {
- sequences[seqNb] = ZSTD_decodeSequenceLong(&seqState, windowSize);
- }
- if (seqNb < seqAdvance)
- return ERROR(corruption_detected);
-
- /* decode and decompress */
- for (; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && seqNb < nbSeq; seqNb++) {
- seq_t const sequence = ZSTD_decodeSequenceLong(&seqState, windowSize);
- size_t const oneSeqSize =
- ZSTD_execSequenceLong(op, oend, sequences[(seqNb - ADVANCED_SEQS) & STOSEQ_MASK], &litPtr, litEnd, base, vBase, dictEnd);
- if (ZSTD_isError(oneSeqSize))
- return oneSeqSize;
- ZSTD_PREFETCH(sequence.match);
- sequences[seqNb & STOSEQ_MASK] = sequence;
- op += oneSeqSize;
- }
- if (seqNb < nbSeq)
- return ERROR(corruption_detected);
-
- /* finish queue */
- seqNb -= seqAdvance;
- for (; seqNb < nbSeq; seqNb++) {
- size_t const oneSeqSize = ZSTD_execSequenceLong(op, oend, sequences[seqNb & STOSEQ_MASK], &litPtr, litEnd, base, vBase, dictEnd);
- if (ZSTD_isError(oneSeqSize))
- return oneSeqSize;
- op += oneSeqSize;
- }
-
- /* save reps for next block */
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]);
- }
- }
-
- /* last literal segment */
- {
- size_t const lastLLSize = litEnd - litPtr;
- if (lastLLSize > (size_t)(oend - op))
- return ERROR(dstSize_tooSmall);
- memcpy(op, litPtr, lastLLSize);
- op += lastLLSize;
- }
-
- return op - ostart;
-}
-
-static size_t ZSTD_decompressBlock_internal(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{ /* blockType == blockCompressed */
- const BYTE *ip = (const BYTE *)src;
-
- if (srcSize >= ZSTD_BLOCKSIZE_ABSOLUTEMAX)
- return ERROR(srcSize_wrong);
-
- /* Decode literals section */
- {
- size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize);
- if (ZSTD_isError(litCSize))
- return litCSize;
- ip += litCSize;
- srcSize -= litCSize;
- }
- if (sizeof(size_t) > 4) /* do not enable prefetching on 32-bits x86, as it's performance detrimental */
- /* likely because of register pressure */
- /* if that's the correct cause, then 32-bits ARM should be affected differently */
- /* it would be good to test this on ARM real hardware, to see if prefetch version improves speed */
- if (dctx->fParams.windowSize > (1 << 23))
- return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize);
- return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize);
-}
-
-static void ZSTD_checkContinuity(ZSTD_DCtx *dctx, const void *dst)
-{
- if (dst != dctx->previousDstEnd) { /* not contiguous */
- dctx->dictEnd = dctx->previousDstEnd;
- dctx->vBase = (const char *)dst - ((const char *)(dctx->previousDstEnd) - (const char *)(dctx->base));
- dctx->base = dst;
- dctx->previousDstEnd = dst;
- }
-}
-
-size_t ZSTD_decompressBlock(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t dSize;
- ZSTD_checkContinuity(dctx, dst);
- dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize);
- dctx->previousDstEnd = (char *)dst + dSize;
- return dSize;
-}
-
-/** ZSTD_insertBlock() :
- insert `src` block into `dctx` history. Useful to track uncompressed blocks. */
-size_t ZSTD_insertBlock(ZSTD_DCtx *dctx, const void *blockStart, size_t blockSize)
-{
- ZSTD_checkContinuity(dctx, blockStart);
- dctx->previousDstEnd = (const char *)blockStart + blockSize;
- return blockSize;
-}
-
-size_t ZSTD_generateNxBytes(void *dst, size_t dstCapacity, BYTE byte, size_t length)
-{
- if (length > dstCapacity)
- return ERROR(dstSize_tooSmall);
- memset(dst, byte, length);
- return length;
-}
-
-/** ZSTD_findFrameCompressedSize() :
- * compatible with legacy mode
- * `src` must point to the start of a ZSTD frame, ZSTD legacy frame, or skippable frame
- * `srcSize` must be at least as large as the frame contained
- * @return : the compressed size of the frame starting at `src` */
-size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize)
-{
- if (srcSize >= ZSTD_skippableHeaderSize && (ZSTD_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
- return ZSTD_skippableHeaderSize + ZSTD_readLE32((const BYTE *)src + 4);
- } else {
- const BYTE *ip = (const BYTE *)src;
- const BYTE *const ipstart = ip;
- size_t remainingSize = srcSize;
- ZSTD_frameParams fParams;
-
- size_t const headerSize = ZSTD_frameHeaderSize(ip, remainingSize);
- if (ZSTD_isError(headerSize))
- return headerSize;
-
- /* Frame Header */
- {
- size_t const ret = ZSTD_getFrameParams(&fParams, ip, remainingSize);
- if (ZSTD_isError(ret))
- return ret;
- if (ret > 0)
- return ERROR(srcSize_wrong);
- }
-
- ip += headerSize;
- remainingSize -= headerSize;
-
- /* Loop on each block */
- while (1) {
- blockProperties_t blockProperties;
- size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties);
- if (ZSTD_isError(cBlockSize))
- return cBlockSize;
-
- if (ZSTD_blockHeaderSize + cBlockSize > remainingSize)
- return ERROR(srcSize_wrong);
-
- ip += ZSTD_blockHeaderSize + cBlockSize;
- remainingSize -= ZSTD_blockHeaderSize + cBlockSize;
-
- if (blockProperties.lastBlock)
- break;
- }
-
- if (fParams.checksumFlag) { /* Frame content checksum */
- if (remainingSize < 4)
- return ERROR(srcSize_wrong);
- ip += 4;
- remainingSize -= 4;
- }
-
- return ip - ipstart;
- }
-}
-
-/*! ZSTD_decompressFrame() :
-* @dctx must be properly initialized */
-static size_t ZSTD_decompressFrame(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void **srcPtr, size_t *srcSizePtr)
-{
- const BYTE *ip = (const BYTE *)(*srcPtr);
- BYTE *const ostart = (BYTE * const)dst;
- BYTE *const oend = ostart + dstCapacity;
- BYTE *op = ostart;
- size_t remainingSize = *srcSizePtr;
-
- /* check */
- if (remainingSize < ZSTD_frameHeaderSize_min + ZSTD_blockHeaderSize)
- return ERROR(srcSize_wrong);
-
- /* Frame Header */
- {
- size_t const frameHeaderSize = ZSTD_frameHeaderSize(ip, ZSTD_frameHeaderSize_prefix);
- if (ZSTD_isError(frameHeaderSize))
- return frameHeaderSize;
- if (remainingSize < frameHeaderSize + ZSTD_blockHeaderSize)
- return ERROR(srcSize_wrong);
- CHECK_F(ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize));
- ip += frameHeaderSize;
- remainingSize -= frameHeaderSize;
- }
-
- /* Loop on each block */
- while (1) {
- size_t decodedSize;
- blockProperties_t blockProperties;
- size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties);
- if (ZSTD_isError(cBlockSize))
- return cBlockSize;
-
- ip += ZSTD_blockHeaderSize;
- remainingSize -= ZSTD_blockHeaderSize;
- if (cBlockSize > remainingSize)
- return ERROR(srcSize_wrong);
-
- switch (blockProperties.blockType) {
- case bt_compressed: decodedSize = ZSTD_decompressBlock_internal(dctx, op, oend - op, ip, cBlockSize); break;
- case bt_raw: decodedSize = ZSTD_copyRawBlock(op, oend - op, ip, cBlockSize); break;
- case bt_rle: decodedSize = ZSTD_generateNxBytes(op, oend - op, *ip, blockProperties.origSize); break;
- case bt_reserved:
- default: return ERROR(corruption_detected);
- }
-
- if (ZSTD_isError(decodedSize))
- return decodedSize;
- if (dctx->fParams.checksumFlag)
- xxh64_update(&dctx->xxhState, op, decodedSize);
- op += decodedSize;
- ip += cBlockSize;
- remainingSize -= cBlockSize;
- if (blockProperties.lastBlock)
- break;
- }
-
- if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */
- U32 const checkCalc = (U32)xxh64_digest(&dctx->xxhState);
- U32 checkRead;
- if (remainingSize < 4)
- return ERROR(checksum_wrong);
- checkRead = ZSTD_readLE32(ip);
- if (checkRead != checkCalc)
- return ERROR(checksum_wrong);
- ip += 4;
- remainingSize -= 4;
- }
-
- /* Allow caller to get size read */
- *srcPtr = ip;
- *srcSizePtr = remainingSize;
- return op - ostart;
-}
-
-static const void *ZSTD_DDictDictContent(const ZSTD_DDict *ddict);
-static size_t ZSTD_DDictDictSize(const ZSTD_DDict *ddict);
-
-static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const void *dict, size_t dictSize,
- const ZSTD_DDict *ddict)
-{
- void *const dststart = dst;
-
- if (ddict) {
- if (dict) {
- /* programmer error, these two cases should be mutually exclusive */
- return ERROR(GENERIC);
- }
-
- dict = ZSTD_DDictDictContent(ddict);
- dictSize = ZSTD_DDictDictSize(ddict);
- }
-
- while (srcSize >= ZSTD_frameHeaderSize_prefix) {
- U32 magicNumber;
-
- magicNumber = ZSTD_readLE32(src);
- if (magicNumber != ZSTD_MAGICNUMBER) {
- if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
- size_t skippableSize;
- if (srcSize < ZSTD_skippableHeaderSize)
- return ERROR(srcSize_wrong);
- skippableSize = ZSTD_readLE32((const BYTE *)src + 4) + ZSTD_skippableHeaderSize;
- if (srcSize < skippableSize) {
- return ERROR(srcSize_wrong);
- }
-
- src = (const BYTE *)src + skippableSize;
- srcSize -= skippableSize;
- continue;
- } else {
- return ERROR(prefix_unknown);
- }
- }
-
- if (ddict) {
- /* we were called from ZSTD_decompress_usingDDict */
- ZSTD_refDDict(dctx, ddict);
- } else {
- /* this will initialize correctly with no dict if dict == NULL, so
- * use this in all cases but ddict */
- CHECK_F(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize));
- }
- ZSTD_checkContinuity(dctx, dst);
-
- {
- const size_t res = ZSTD_decompressFrame(dctx, dst, dstCapacity, &src, &srcSize);
- if (ZSTD_isError(res))
- return res;
- /* don't need to bounds check this, ZSTD_decompressFrame will have
- * already */
- dst = (BYTE *)dst + res;
- dstCapacity -= res;
- }
- }
-
- if (srcSize)
- return ERROR(srcSize_wrong); /* input not entirely consumed */
-
- return (BYTE *)dst - (BYTE *)dststart;
-}
-
-size_t ZSTD_decompress_usingDict(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const void *dict, size_t dictSize)
-{
- return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, dict, dictSize, NULL);
-}
-
-size_t ZSTD_decompressDCtx(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- return ZSTD_decompress_usingDict(dctx, dst, dstCapacity, src, srcSize, NULL, 0);
-}
-
-/*-**************************************
-* Advanced Streaming Decompression API
-* Bufferless and synchronous
-****************************************/
-size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx *dctx) { return dctx->expected; }
-
-ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx *dctx)
-{
- switch (dctx->stage) {
- default: /* should not happen */
- case ZSTDds_getFrameHeaderSize:
- case ZSTDds_decodeFrameHeader: return ZSTDnit_frameHeader;
- case ZSTDds_decodeBlockHeader: return ZSTDnit_blockHeader;
- case ZSTDds_decompressBlock: return ZSTDnit_block;
- case ZSTDds_decompressLastBlock: return ZSTDnit_lastBlock;
- case ZSTDds_checkChecksum: return ZSTDnit_checksum;
- case ZSTDds_decodeSkippableHeader:
- case ZSTDds_skipFrame: return ZSTDnit_skippableFrame;
- }
-}
-
-int ZSTD_isSkipFrame(ZSTD_DCtx *dctx) { return dctx->stage == ZSTDds_skipFrame; } /* for zbuff */
-
-/** ZSTD_decompressContinue() :
-* @return : nb of bytes generated into `dst` (necessarily <= `dstCapacity)
-* or an error code, which can be tested using ZSTD_isError() */
-size_t ZSTD_decompressContinue(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- /* Sanity check */
- if (srcSize != dctx->expected)
- return ERROR(srcSize_wrong);
- if (dstCapacity)
- ZSTD_checkContinuity(dctx, dst);
-
- switch (dctx->stage) {
- case ZSTDds_getFrameHeaderSize:
- if (srcSize != ZSTD_frameHeaderSize_prefix)
- return ERROR(srcSize_wrong); /* impossible */
- if ((ZSTD_readLE32(src) & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
- memcpy(dctx->headerBuffer, src, ZSTD_frameHeaderSize_prefix);
- dctx->expected = ZSTD_skippableHeaderSize - ZSTD_frameHeaderSize_prefix; /* magic number + skippable frame length */
- dctx->stage = ZSTDds_decodeSkippableHeader;
- return 0;
- }
- dctx->headerSize = ZSTD_frameHeaderSize(src, ZSTD_frameHeaderSize_prefix);
- if (ZSTD_isError(dctx->headerSize))
- return dctx->headerSize;
- memcpy(dctx->headerBuffer, src, ZSTD_frameHeaderSize_prefix);
- if (dctx->headerSize > ZSTD_frameHeaderSize_prefix) {
- dctx->expected = dctx->headerSize - ZSTD_frameHeaderSize_prefix;
- dctx->stage = ZSTDds_decodeFrameHeader;
- return 0;
- }
- dctx->expected = 0; /* not necessary to copy more */
- fallthrough;
-
- case ZSTDds_decodeFrameHeader:
- memcpy(dctx->headerBuffer + ZSTD_frameHeaderSize_prefix, src, dctx->expected);
- CHECK_F(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize));
- dctx->expected = ZSTD_blockHeaderSize;
- dctx->stage = ZSTDds_decodeBlockHeader;
- return 0;
-
- case ZSTDds_decodeBlockHeader: {
- blockProperties_t bp;
- size_t const cBlockSize = ZSTD_getcBlockSize(src, ZSTD_blockHeaderSize, &bp);
- if (ZSTD_isError(cBlockSize))
- return cBlockSize;
- dctx->expected = cBlockSize;
- dctx->bType = bp.blockType;
- dctx->rleSize = bp.origSize;
- if (cBlockSize) {
- dctx->stage = bp.lastBlock ? ZSTDds_decompressLastBlock : ZSTDds_decompressBlock;
- return 0;
- }
- /* empty block */
- if (bp.lastBlock) {
- if (dctx->fParams.checksumFlag) {
- dctx->expected = 4;
- dctx->stage = ZSTDds_checkChecksum;
- } else {
- dctx->expected = 0; /* end of frame */
- dctx->stage = ZSTDds_getFrameHeaderSize;
- }
- } else {
- dctx->expected = 3; /* go directly to next header */
- dctx->stage = ZSTDds_decodeBlockHeader;
- }
- return 0;
- }
- case ZSTDds_decompressLastBlock:
- case ZSTDds_decompressBlock: {
- size_t rSize;
- switch (dctx->bType) {
- case bt_compressed: rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize); break;
- case bt_raw: rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize); break;
- case bt_rle: rSize = ZSTD_setRleBlock(dst, dstCapacity, src, srcSize, dctx->rleSize); break;
- case bt_reserved: /* should never happen */
- default: return ERROR(corruption_detected);
- }
- if (ZSTD_isError(rSize))
- return rSize;
- if (dctx->fParams.checksumFlag)
- xxh64_update(&dctx->xxhState, dst, rSize);
-
- if (dctx->stage == ZSTDds_decompressLastBlock) { /* end of frame */
- if (dctx->fParams.checksumFlag) { /* another round for frame checksum */
- dctx->expected = 4;
- dctx->stage = ZSTDds_checkChecksum;
- } else {
- dctx->expected = 0; /* ends here */
- dctx->stage = ZSTDds_getFrameHeaderSize;
- }
- } else {
- dctx->stage = ZSTDds_decodeBlockHeader;
- dctx->expected = ZSTD_blockHeaderSize;
- dctx->previousDstEnd = (char *)dst + rSize;
- }
- return rSize;
- }
- case ZSTDds_checkChecksum: {
- U32 const h32 = (U32)xxh64_digest(&dctx->xxhState);
- U32 const check32 = ZSTD_readLE32(src); /* srcSize == 4, guaranteed by dctx->expected */
- if (check32 != h32)
- return ERROR(checksum_wrong);
- dctx->expected = 0;
- dctx->stage = ZSTDds_getFrameHeaderSize;
- return 0;
- }
- case ZSTDds_decodeSkippableHeader: {
- memcpy(dctx->headerBuffer + ZSTD_frameHeaderSize_prefix, src, dctx->expected);
- dctx->expected = ZSTD_readLE32(dctx->headerBuffer + 4);
- dctx->stage = ZSTDds_skipFrame;
- return 0;
- }
- case ZSTDds_skipFrame: {
- dctx->expected = 0;
- dctx->stage = ZSTDds_getFrameHeaderSize;
- return 0;
- }
- default:
- return ERROR(GENERIC); /* impossible */
- }
-}
-
-static size_t ZSTD_refDictContent(ZSTD_DCtx *dctx, const void *dict, size_t dictSize)
-{
- dctx->dictEnd = dctx->previousDstEnd;
- dctx->vBase = (const char *)dict - ((const char *)(dctx->previousDstEnd) - (const char *)(dctx->base));
- dctx->base = dict;
- dctx->previousDstEnd = (const char *)dict + dictSize;
- return 0;
-}
-
-/* ZSTD_loadEntropy() :
- * dict : must point at beginning of a valid zstd dictionary
- * @return : size of entropy tables read */
-static size_t ZSTD_loadEntropy(ZSTD_entropyTables_t *entropy, const void *const dict, size_t const dictSize)
-{
- const BYTE *dictPtr = (const BYTE *)dict;
- const BYTE *const dictEnd = dictPtr + dictSize;
-
- if (dictSize <= 8)
- return ERROR(dictionary_corrupted);
- dictPtr += 8; /* skip header = magic + dictID */
-
- {
- size_t const hSize = HUF_readDTableX4_wksp(entropy->hufTable, dictPtr, dictEnd - dictPtr, entropy->workspace, sizeof(entropy->workspace));
- if (HUF_isError(hSize))
- return ERROR(dictionary_corrupted);
- dictPtr += hSize;
- }
-
- {
- short offcodeNCount[MaxOff + 1];
- U32 offcodeMaxValue = MaxOff, offcodeLog;
- size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(offcodeHeaderSize))
- return ERROR(dictionary_corrupted);
- if (offcodeLog > OffFSELog)
- return ERROR(dictionary_corrupted);
- CHECK_E(FSE_buildDTable_wksp(entropy->OFTable, offcodeNCount, offcodeMaxValue, offcodeLog, entropy->workspace, sizeof(entropy->workspace)), dictionary_corrupted);
- dictPtr += offcodeHeaderSize;
- }
-
- {
- short matchlengthNCount[MaxML + 1];
- unsigned matchlengthMaxValue = MaxML, matchlengthLog;
- size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(matchlengthHeaderSize))
- return ERROR(dictionary_corrupted);
- if (matchlengthLog > MLFSELog)
- return ERROR(dictionary_corrupted);
- CHECK_E(FSE_buildDTable_wksp(entropy->MLTable, matchlengthNCount, matchlengthMaxValue, matchlengthLog, entropy->workspace, sizeof(entropy->workspace)), dictionary_corrupted);
- dictPtr += matchlengthHeaderSize;
- }
-
- {
- short litlengthNCount[MaxLL + 1];
- unsigned litlengthMaxValue = MaxLL, litlengthLog;
- size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, dictEnd - dictPtr);
- if (FSE_isError(litlengthHeaderSize))
- return ERROR(dictionary_corrupted);
- if (litlengthLog > LLFSELog)
- return ERROR(dictionary_corrupted);
- CHECK_E(FSE_buildDTable_wksp(entropy->LLTable, litlengthNCount, litlengthMaxValue, litlengthLog, entropy->workspace, sizeof(entropy->workspace)), dictionary_corrupted);
- dictPtr += litlengthHeaderSize;
- }
-
- if (dictPtr + 12 > dictEnd)
- return ERROR(dictionary_corrupted);
- {
- int i;
- size_t const dictContentSize = (size_t)(dictEnd - (dictPtr + 12));
- for (i = 0; i < 3; i++) {
- U32 const rep = ZSTD_readLE32(dictPtr);
- dictPtr += 4;
- if (rep == 0 || rep >= dictContentSize)
- return ERROR(dictionary_corrupted);
- entropy->rep[i] = rep;
- }
- }
-
- return dictPtr - (const BYTE *)dict;
-}
-
-static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx *dctx, const void *dict, size_t dictSize)
-{
- if (dictSize < 8)
- return ZSTD_refDictContent(dctx, dict, dictSize);
- {
- U32 const magic = ZSTD_readLE32(dict);
- if (magic != ZSTD_DICT_MAGIC) {
- return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */
- }
- }
- dctx->dictID = ZSTD_readLE32((const char *)dict + 4);
-
- /* load entropy tables */
- {
- size_t const eSize = ZSTD_loadEntropy(&dctx->entropy, dict, dictSize);
- if (ZSTD_isError(eSize))
- return ERROR(dictionary_corrupted);
- dict = (const char *)dict + eSize;
- dictSize -= eSize;
- }
- dctx->litEntropy = dctx->fseEntropy = 1;
-
- /* reference dictionary content */
- return ZSTD_refDictContent(dctx, dict, dictSize);
-}
-
-size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx *dctx, const void *dict, size_t dictSize)
-{
- CHECK_F(ZSTD_decompressBegin(dctx));
- if (dict && dictSize)
- CHECK_E(ZSTD_decompress_insertDictionary(dctx, dict, dictSize), dictionary_corrupted);
- return 0;
-}
-
-/* ====== ZSTD_DDict ====== */
-
-struct ZSTD_DDict_s {
- void *dictBuffer;
- const void *dictContent;
- size_t dictSize;
- ZSTD_entropyTables_t entropy;
- U32 dictID;
- U32 entropyPresent;
- ZSTD_customMem cMem;
-}; /* typedef'd to ZSTD_DDict within "zstd.h" */
-
-size_t ZSTD_DDictWorkspaceBound(void) { return ZSTD_ALIGN(sizeof(ZSTD_stack)) + ZSTD_ALIGN(sizeof(ZSTD_DDict)); }
-
-static const void *ZSTD_DDictDictContent(const ZSTD_DDict *ddict) { return ddict->dictContent; }
-
-static size_t ZSTD_DDictDictSize(const ZSTD_DDict *ddict) { return ddict->dictSize; }
-
-static void ZSTD_refDDict(ZSTD_DCtx *dstDCtx, const ZSTD_DDict *ddict)
-{
- ZSTD_decompressBegin(dstDCtx); /* init */
- if (ddict) { /* support refDDict on NULL */
- dstDCtx->dictID = ddict->dictID;
- dstDCtx->base = ddict->dictContent;
- dstDCtx->vBase = ddict->dictContent;
- dstDCtx->dictEnd = (const BYTE *)ddict->dictContent + ddict->dictSize;
- dstDCtx->previousDstEnd = dstDCtx->dictEnd;
- if (ddict->entropyPresent) {
- dstDCtx->litEntropy = 1;
- dstDCtx->fseEntropy = 1;
- dstDCtx->LLTptr = ddict->entropy.LLTable;
- dstDCtx->MLTptr = ddict->entropy.MLTable;
- dstDCtx->OFTptr = ddict->entropy.OFTable;
- dstDCtx->HUFptr = ddict->entropy.hufTable;
- dstDCtx->entropy.rep[0] = ddict->entropy.rep[0];
- dstDCtx->entropy.rep[1] = ddict->entropy.rep[1];
- dstDCtx->entropy.rep[2] = ddict->entropy.rep[2];
- } else {
- dstDCtx->litEntropy = 0;
- dstDCtx->fseEntropy = 0;
- }
- }
-}
-
-static size_t ZSTD_loadEntropy_inDDict(ZSTD_DDict *ddict)
-{
- ddict->dictID = 0;
- ddict->entropyPresent = 0;
- if (ddict->dictSize < 8)
- return 0;
- {
- U32 const magic = ZSTD_readLE32(ddict->dictContent);
- if (magic != ZSTD_DICT_MAGIC)
- return 0; /* pure content mode */
- }
- ddict->dictID = ZSTD_readLE32((const char *)ddict->dictContent + 4);
-
- /* load entropy tables */
- CHECK_E(ZSTD_loadEntropy(&ddict->entropy, ddict->dictContent, ddict->dictSize), dictionary_corrupted);
- ddict->entropyPresent = 1;
- return 0;
-}
-
-static ZSTD_DDict *ZSTD_createDDict_advanced(const void *dict, size_t dictSize, unsigned byReference, ZSTD_customMem customMem)
-{
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
-
- {
- ZSTD_DDict *const ddict = (ZSTD_DDict *)ZSTD_malloc(sizeof(ZSTD_DDict), customMem);
- if (!ddict)
- return NULL;
- ddict->cMem = customMem;
-
- if ((byReference) || (!dict) || (!dictSize)) {
- ddict->dictBuffer = NULL;
- ddict->dictContent = dict;
- } else {
- void *const internalBuffer = ZSTD_malloc(dictSize, customMem);
- if (!internalBuffer) {
- ZSTD_freeDDict(ddict);
- return NULL;
- }
- memcpy(internalBuffer, dict, dictSize);
- ddict->dictBuffer = internalBuffer;
- ddict->dictContent = internalBuffer;
- }
- ddict->dictSize = dictSize;
- ddict->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
- /* parse dictionary content */
- {
- size_t const errorCode = ZSTD_loadEntropy_inDDict(ddict);
- if (ZSTD_isError(errorCode)) {
- ZSTD_freeDDict(ddict);
- return NULL;
- }
- }
-
- return ddict;
- }
-}
-
-/*! ZSTD_initDDict() :
-* Create a digested dictionary, to start decompression without startup delay.
-* `dict` content is copied inside DDict.
-* Consequently, `dict` can be released after `ZSTD_DDict` creation */
-ZSTD_DDict *ZSTD_initDDict(const void *dict, size_t dictSize, void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- return ZSTD_createDDict_advanced(dict, dictSize, 1, stackMem);
-}
-
-size_t ZSTD_freeDDict(ZSTD_DDict *ddict)
-{
- if (ddict == NULL)
- return 0; /* support free on NULL */
- {
- ZSTD_customMem const cMem = ddict->cMem;
- ZSTD_free(ddict->dictBuffer, cMem);
- ZSTD_free(ddict, cMem);
- return 0;
- }
-}
-
-/*! ZSTD_getDictID_fromDict() :
- * Provides the dictID stored within dictionary.
- * if @return == 0, the dictionary is not conformant with Zstandard specification.
- * It can still be loaded, but as a content-only dictionary. */
-unsigned ZSTD_getDictID_fromDict(const void *dict, size_t dictSize)
-{
- if (dictSize < 8)
- return 0;
- if (ZSTD_readLE32(dict) != ZSTD_DICT_MAGIC)
- return 0;
- return ZSTD_readLE32((const char *)dict + 4);
-}
-
-/*! ZSTD_getDictID_fromDDict() :
- * Provides the dictID of the dictionary loaded into `ddict`.
- * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
- * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
-unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict *ddict)
-{
- if (ddict == NULL)
- return 0;
- return ZSTD_getDictID_fromDict(ddict->dictContent, ddict->dictSize);
-}
-
-/*! ZSTD_getDictID_fromFrame() :
- * Provides the dictID required to decompressed the frame stored within `src`.
- * If @return == 0, the dictID could not be decoded.
- * This could for one of the following reasons :
- * - The frame does not require a dictionary to be decoded (most common case).
- * - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
- * Note : this use case also happens when using a non-conformant dictionary.
- * - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
- * - This is not a Zstandard frame.
- * When identifying the exact failure cause, it's possible to used ZSTD_getFrameParams(), which will provide a more precise error code. */
-unsigned ZSTD_getDictID_fromFrame(const void *src, size_t srcSize)
-{
- ZSTD_frameParams zfp = {0, 0, 0, 0};
- size_t const hError = ZSTD_getFrameParams(&zfp, src, srcSize);
- if (ZSTD_isError(hError))
- return 0;
- return zfp.dictID;
-}
-
-/*! ZSTD_decompress_usingDDict() :
-* Decompression using a pre-digested Dictionary
-* Use dictionary without significant overhead. */
-size_t ZSTD_decompress_usingDDict(ZSTD_DCtx *dctx, void *dst, size_t dstCapacity, const void *src, size_t srcSize, const ZSTD_DDict *ddict)
-{
- /* pass content and size in case legacy frames are encountered */
- return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, NULL, 0, ddict);
-}
-
-/*=====================================
-* Streaming decompression
-*====================================*/
-
-typedef enum { zdss_init, zdss_loadHeader, zdss_read, zdss_load, zdss_flush } ZSTD_dStreamStage;
-
-/* *** Resource management *** */
-struct ZSTD_DStream_s {
- ZSTD_DCtx *dctx;
- ZSTD_DDict *ddictLocal;
- const ZSTD_DDict *ddict;
- ZSTD_frameParams fParams;
- ZSTD_dStreamStage stage;
- char *inBuff;
- size_t inBuffSize;
- size_t inPos;
- size_t maxWindowSize;
- char *outBuff;
- size_t outBuffSize;
- size_t outStart;
- size_t outEnd;
- size_t blockSize;
- BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX]; /* tmp buffer to store frame header */
- size_t lhSize;
- ZSTD_customMem customMem;
- void *legacyContext;
- U32 previousLegacyVersion;
- U32 legacyVersion;
- U32 hostageByte;
-}; /* typedef'd to ZSTD_DStream within "zstd.h" */
-
-size_t ZSTD_DStreamWorkspaceBound(size_t maxWindowSize)
-{
- size_t const blockSize = MIN(maxWindowSize, ZSTD_BLOCKSIZE_ABSOLUTEMAX);
- size_t const inBuffSize = blockSize;
- size_t const outBuffSize = maxWindowSize + blockSize + WILDCOPY_OVERLENGTH * 2;
- return ZSTD_DCtxWorkspaceBound() + ZSTD_ALIGN(sizeof(ZSTD_DStream)) + ZSTD_ALIGN(inBuffSize) + ZSTD_ALIGN(outBuffSize);
-}
-
-static ZSTD_DStream *ZSTD_createDStream_advanced(ZSTD_customMem customMem)
-{
- ZSTD_DStream *zds;
-
- if (!customMem.customAlloc || !customMem.customFree)
- return NULL;
-
- zds = (ZSTD_DStream *)ZSTD_malloc(sizeof(ZSTD_DStream), customMem);
- if (zds == NULL)
- return NULL;
- memset(zds, 0, sizeof(ZSTD_DStream));
- memcpy(&zds->customMem, &customMem, sizeof(ZSTD_customMem));
- zds->dctx = ZSTD_createDCtx_advanced(customMem);
- if (zds->dctx == NULL) {
- ZSTD_freeDStream(zds);
- return NULL;
- }
- zds->stage = zdss_init;
- zds->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
- return zds;
-}
-
-ZSTD_DStream *ZSTD_initDStream(size_t maxWindowSize, void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem const stackMem = ZSTD_initStack(workspace, workspaceSize);
- ZSTD_DStream *zds = ZSTD_createDStream_advanced(stackMem);
- if (!zds) {
- return NULL;
- }
-
- zds->maxWindowSize = maxWindowSize;
- zds->stage = zdss_loadHeader;
- zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
- ZSTD_freeDDict(zds->ddictLocal);
- zds->ddictLocal = NULL;
- zds->ddict = zds->ddictLocal;
- zds->legacyVersion = 0;
- zds->hostageByte = 0;
-
- {
- size_t const blockSize = MIN(zds->maxWindowSize, ZSTD_BLOCKSIZE_ABSOLUTEMAX);
- size_t const neededOutSize = zds->maxWindowSize + blockSize + WILDCOPY_OVERLENGTH * 2;
-
- zds->inBuff = (char *)ZSTD_malloc(blockSize, zds->customMem);
- zds->inBuffSize = blockSize;
- zds->outBuff = (char *)ZSTD_malloc(neededOutSize, zds->customMem);
- zds->outBuffSize = neededOutSize;
- if (zds->inBuff == NULL || zds->outBuff == NULL) {
- ZSTD_freeDStream(zds);
- return NULL;
- }
- }
- return zds;
-}
-
-ZSTD_DStream *ZSTD_initDStream_usingDDict(size_t maxWindowSize, const ZSTD_DDict *ddict, void *workspace, size_t workspaceSize)
-{
- ZSTD_DStream *zds = ZSTD_initDStream(maxWindowSize, workspace, workspaceSize);
- if (zds) {
- zds->ddict = ddict;
- }
- return zds;
-}
-
-size_t ZSTD_freeDStream(ZSTD_DStream *zds)
-{
- if (zds == NULL)
- return 0; /* support free on null */
- {
- ZSTD_customMem const cMem = zds->customMem;
- ZSTD_freeDCtx(zds->dctx);
- zds->dctx = NULL;
- ZSTD_freeDDict(zds->ddictLocal);
- zds->ddictLocal = NULL;
- ZSTD_free(zds->inBuff, cMem);
- zds->inBuff = NULL;
- ZSTD_free(zds->outBuff, cMem);
- zds->outBuff = NULL;
- ZSTD_free(zds, cMem);
- return 0;
- }
-}
-
-/* *** Initialization *** */
-
-size_t ZSTD_DStreamInSize(void) { return ZSTD_BLOCKSIZE_ABSOLUTEMAX + ZSTD_blockHeaderSize; }
-size_t ZSTD_DStreamOutSize(void) { return ZSTD_BLOCKSIZE_ABSOLUTEMAX; }
-
-size_t ZSTD_resetDStream(ZSTD_DStream *zds)
-{
- zds->stage = zdss_loadHeader;
- zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
- zds->legacyVersion = 0;
- zds->hostageByte = 0;
- return ZSTD_frameHeaderSize_prefix;
-}
-
-/* ***** Decompression ***** */
-
-ZSTD_STATIC size_t ZSTD_limitCopy(void *dst, size_t dstCapacity, const void *src, size_t srcSize)
-{
- size_t const length = MIN(dstCapacity, srcSize);
- memcpy(dst, src, length);
- return length;
-}
-
-size_t ZSTD_decompressStream(ZSTD_DStream *zds, ZSTD_outBuffer *output, ZSTD_inBuffer *input)
-{
- const char *const istart = (const char *)(input->src) + input->pos;
- const char *const iend = (const char *)(input->src) + input->size;
- const char *ip = istart;
- char *const ostart = (char *)(output->dst) + output->pos;
- char *const oend = (char *)(output->dst) + output->size;
- char *op = ostart;
- U32 someMoreWork = 1;
-
- while (someMoreWork) {
- switch (zds->stage) {
- case zdss_init:
- ZSTD_resetDStream(zds); /* transparent reset on starting decoding a new frame */
- fallthrough;
-
- case zdss_loadHeader: {
- size_t const hSize = ZSTD_getFrameParams(&zds->fParams, zds->headerBuffer, zds->lhSize);
- if (ZSTD_isError(hSize))
- return hSize;
- if (hSize != 0) { /* need more input */
- size_t const toLoad = hSize - zds->lhSize; /* if hSize!=0, hSize > zds->lhSize */
- if (toLoad > (size_t)(iend - ip)) { /* not enough input to load full header */
- memcpy(zds->headerBuffer + zds->lhSize, ip, iend - ip);
- zds->lhSize += iend - ip;
- input->pos = input->size;
- return (MAX(ZSTD_frameHeaderSize_min, hSize) - zds->lhSize) +
- ZSTD_blockHeaderSize; /* remaining header bytes + next block header */
- }
- memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad);
- zds->lhSize = hSize;
- ip += toLoad;
- break;
- }
-
- /* check for single-pass mode opportunity */
- if (zds->fParams.frameContentSize && zds->fParams.windowSize /* skippable frame if == 0 */
- && (U64)(size_t)(oend - op) >= zds->fParams.frameContentSize) {
- size_t const cSize = ZSTD_findFrameCompressedSize(istart, iend - istart);
- if (cSize <= (size_t)(iend - istart)) {
- size_t const decompressedSize = ZSTD_decompress_usingDDict(zds->dctx, op, oend - op, istart, cSize, zds->ddict);
- if (ZSTD_isError(decompressedSize))
- return decompressedSize;
- ip = istart + cSize;
- op += decompressedSize;
- zds->dctx->expected = 0;
- zds->stage = zdss_init;
- someMoreWork = 0;
- break;
- }
- }
-
- /* Consume header */
- ZSTD_refDDict(zds->dctx, zds->ddict);
- {
- size_t const h1Size = ZSTD_nextSrcSizeToDecompress(zds->dctx); /* == ZSTD_frameHeaderSize_prefix */
- CHECK_F(ZSTD_decompressContinue(zds->dctx, NULL, 0, zds->headerBuffer, h1Size));
- {
- size_t const h2Size = ZSTD_nextSrcSizeToDecompress(zds->dctx);
- CHECK_F(ZSTD_decompressContinue(zds->dctx, NULL, 0, zds->headerBuffer + h1Size, h2Size));
- }
- }
-
- zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN);
- if (zds->fParams.windowSize > zds->maxWindowSize)
- return ERROR(frameParameter_windowTooLarge);
-
- /* Buffers are preallocated, but double check */
- {
- size_t const blockSize = MIN(zds->maxWindowSize, ZSTD_BLOCKSIZE_ABSOLUTEMAX);
- size_t const neededOutSize = zds->maxWindowSize + blockSize + WILDCOPY_OVERLENGTH * 2;
- if (zds->inBuffSize < blockSize) {
- return ERROR(GENERIC);
- }
- if (zds->outBuffSize < neededOutSize) {
- return ERROR(GENERIC);
- }
- zds->blockSize = blockSize;
- }
- zds->stage = zdss_read;
- }
- fallthrough;
-
- case zdss_read: {
- size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds->dctx);
- if (neededInSize == 0) { /* end of frame */
- zds->stage = zdss_init;
- someMoreWork = 0;
- break;
- }
- if ((size_t)(iend - ip) >= neededInSize) { /* decode directly from src */
- const int isSkipFrame = ZSTD_isSkipFrame(zds->dctx);
- size_t const decodedSize = ZSTD_decompressContinue(zds->dctx, zds->outBuff + zds->outStart,
- (isSkipFrame ? 0 : zds->outBuffSize - zds->outStart), ip, neededInSize);
- if (ZSTD_isError(decodedSize))
- return decodedSize;
- ip += neededInSize;
- if (!decodedSize && !isSkipFrame)
- break; /* this was just a header */
- zds->outEnd = zds->outStart + decodedSize;
- zds->stage = zdss_flush;
- break;
- }
- if (ip == iend) {
- someMoreWork = 0;
- break;
- } /* no more input */
- zds->stage = zdss_load;
- /* pass-through */
- }
- fallthrough;
-
- case zdss_load: {
- size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds->dctx);
- size_t const toLoad = neededInSize - zds->inPos; /* should always be <= remaining space within inBuff */
- size_t loadedSize;
- if (toLoad > zds->inBuffSize - zds->inPos)
- return ERROR(corruption_detected); /* should never happen */
- loadedSize = ZSTD_limitCopy(zds->inBuff + zds->inPos, toLoad, ip, iend - ip);
- ip += loadedSize;
- zds->inPos += loadedSize;
- if (loadedSize < toLoad) {
- someMoreWork = 0;
- break;
- } /* not enough input, wait for more */
-
- /* decode loaded input */
- {
- const int isSkipFrame = ZSTD_isSkipFrame(zds->dctx);
- size_t const decodedSize = ZSTD_decompressContinue(zds->dctx, zds->outBuff + zds->outStart, zds->outBuffSize - zds->outStart,
- zds->inBuff, neededInSize);
- if (ZSTD_isError(decodedSize))
- return decodedSize;
- zds->inPos = 0; /* input is consumed */
- if (!decodedSize && !isSkipFrame) {
- zds->stage = zdss_read;
- break;
- } /* this was just a header */
- zds->outEnd = zds->outStart + decodedSize;
- zds->stage = zdss_flush;
- /* pass-through */
- }
- }
- fallthrough;
-
- case zdss_flush: {
- size_t const toFlushSize = zds->outEnd - zds->outStart;
- size_t const flushedSize = ZSTD_limitCopy(op, oend - op, zds->outBuff + zds->outStart, toFlushSize);
- op += flushedSize;
- zds->outStart += flushedSize;
- if (flushedSize == toFlushSize) { /* flush completed */
- zds->stage = zdss_read;
- if (zds->outStart + zds->blockSize > zds->outBuffSize)
- zds->outStart = zds->outEnd = 0;
- break;
- }
- /* cannot complete flush */
- someMoreWork = 0;
- break;
- }
- default:
- return ERROR(GENERIC); /* impossible */
- }
- }
-
- /* result */
- input->pos += (size_t)(ip - istart);
- output->pos += (size_t)(op - ostart);
- {
- size_t nextSrcSizeHint = ZSTD_nextSrcSizeToDecompress(zds->dctx);
- if (!nextSrcSizeHint) { /* frame fully decoded */
- if (zds->outEnd == zds->outStart) { /* output fully flushed */
- if (zds->hostageByte) {
- if (input->pos >= input->size) {
- zds->stage = zdss_read;
- return 1;
- } /* can't release hostage (not present) */
- input->pos++; /* release hostage */
- }
- return 0;
- }
- if (!zds->hostageByte) { /* output not fully flushed; keep last byte as hostage; will be released when all output is flushed */
- input->pos--; /* note : pos > 0, otherwise, impossible to finish reading last block */
- zds->hostageByte = 1;
- }
- return 1;
- }
- nextSrcSizeHint += ZSTD_blockHeaderSize * (ZSTD_nextInputType(zds->dctx) == ZSTDnit_block); /* preload header of next block */
- if (zds->inPos > nextSrcSizeHint)
- return ERROR(GENERIC); /* should never happen */
- nextSrcSizeHint -= zds->inPos; /* already loaded*/
- return nextSrcSizeHint;
- }
-}
-
-EXPORT_SYMBOL(ZSTD_DCtxWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initDCtx);
-EXPORT_SYMBOL(ZSTD_decompressDCtx);
-EXPORT_SYMBOL(ZSTD_decompress_usingDict);
-
-EXPORT_SYMBOL(ZSTD_DDictWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initDDict);
-EXPORT_SYMBOL(ZSTD_decompress_usingDDict);
-
-EXPORT_SYMBOL(ZSTD_DStreamWorkspaceBound);
-EXPORT_SYMBOL(ZSTD_initDStream);
-EXPORT_SYMBOL(ZSTD_initDStream_usingDDict);
-EXPORT_SYMBOL(ZSTD_resetDStream);
-EXPORT_SYMBOL(ZSTD_decompressStream);
-EXPORT_SYMBOL(ZSTD_DStreamInSize);
-EXPORT_SYMBOL(ZSTD_DStreamOutSize);
-
-EXPORT_SYMBOL(ZSTD_findFrameCompressedSize);
-EXPORT_SYMBOL(ZSTD_getFrameContentSize);
-EXPORT_SYMBOL(ZSTD_findDecompressedSize);
-
-EXPORT_SYMBOL(ZSTD_isFrame);
-EXPORT_SYMBOL(ZSTD_getDictID_fromDict);
-EXPORT_SYMBOL(ZSTD_getDictID_fromDDict);
-EXPORT_SYMBOL(ZSTD_getDictID_fromFrame);
-
-EXPORT_SYMBOL(ZSTD_getFrameParams);
-EXPORT_SYMBOL(ZSTD_decompressBegin);
-EXPORT_SYMBOL(ZSTD_decompressBegin_usingDict);
-EXPORT_SYMBOL(ZSTD_copyDCtx);
-EXPORT_SYMBOL(ZSTD_nextSrcSizeToDecompress);
-EXPORT_SYMBOL(ZSTD_decompressContinue);
-EXPORT_SYMBOL(ZSTD_nextInputType);
-
-EXPORT_SYMBOL(ZSTD_decompressBlock);
-EXPORT_SYMBOL(ZSTD_insertBlock);
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("Zstd Decompressor");
--- /dev/null
+/* ******************************************************************
+ * huff0 huffman decoder,
+ * part of Finite State Entropy library
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ *
+ * You can contact the author at :
+ * - FSE+HUF source repository : https://github.com/Cyan4973/FiniteStateEntropy
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+****************************************************************** */
+
+/* **************************************************************
+* Dependencies
+****************************************************************/
+#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memset */
+#include "../common/compiler.h"
+#include "../common/bitstream.h" /* BIT_* */
+#include "../common/fse.h" /* to compress headers */
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include "../common/error_private.h"
+
+/* **************************************************************
+* Macros
+****************************************************************/
+
+/* These two optional macros force the use one way or another of the two
+ * Huffman decompression implementations. You can't force in both directions
+ * at the same time.
+ */
+#if defined(HUF_FORCE_DECOMPRESS_X1) && \
+ defined(HUF_FORCE_DECOMPRESS_X2)
+#error "Cannot force the use of the X1 and X2 decoders at the same time!"
+#endif
+
+
+/* **************************************************************
+* Error Management
+****************************************************************/
+#define HUF_isError ERR_isError
+
+
+/* **************************************************************
+* Byte alignment for workSpace management
+****************************************************************/
+#define HUF_ALIGN(x, a) HUF_ALIGN_MASK((x), (a) - 1)
+#define HUF_ALIGN_MASK(x, mask) (((x) + (mask)) & ~(mask))
+
+
+/* **************************************************************
+* BMI2 Variant Wrappers
+****************************************************************/
+#if DYNAMIC_BMI2
+
+#define HUF_DGEN(fn) \
+ \
+ static size_t fn##_default( \
+ void* dst, size_t dstSize, \
+ const void* cSrc, size_t cSrcSize, \
+ const HUF_DTable* DTable) \
+ { \
+ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
+ } \
+ \
+ static TARGET_ATTRIBUTE("bmi2") size_t fn##_bmi2( \
+ void* dst, size_t dstSize, \
+ const void* cSrc, size_t cSrcSize, \
+ const HUF_DTable* DTable) \
+ { \
+ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
+ } \
+ \
+ static size_t fn(void* dst, size_t dstSize, void const* cSrc, \
+ size_t cSrcSize, HUF_DTable const* DTable, int bmi2) \
+ { \
+ if (bmi2) { \
+ return fn##_bmi2(dst, dstSize, cSrc, cSrcSize, DTable); \
+ } \
+ return fn##_default(dst, dstSize, cSrc, cSrcSize, DTable); \
+ }
+
+#else
+
+#define HUF_DGEN(fn) \
+ static size_t fn(void* dst, size_t dstSize, void const* cSrc, \
+ size_t cSrcSize, HUF_DTable const* DTable, int bmi2) \
+ { \
+ (void)bmi2; \
+ return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
+ }
+
+#endif
+
+
+/*-***************************/
+/* generic DTableDesc */
+/*-***************************/
+typedef struct { BYTE maxTableLog; BYTE tableType; BYTE tableLog; BYTE reserved; } DTableDesc;
+
+static DTableDesc HUF_getDTableDesc(const HUF_DTable* table)
+{
+ DTableDesc dtd;
+ ZSTD_memcpy(&dtd, table, sizeof(dtd));
+ return dtd;
+}
+
+
+#ifndef HUF_FORCE_DECOMPRESS_X2
+
+/*-***************************/
+/* single-symbol decoding */
+/*-***************************/
+typedef struct { BYTE byte; BYTE nbBits; } HUF_DEltX1; /* single-symbol decoding */
+
+/*
+ * Packs 4 HUF_DEltX1 structs into a U64. This is used to lay down 4 entries at
+ * a time.
+ */
+static U64 HUF_DEltX1_set4(BYTE symbol, BYTE nbBits) {
+ U64 D4;
+ if (MEM_isLittleEndian()) {
+ D4 = symbol + (nbBits << 8);
+ } else {
+ D4 = (symbol << 8) + nbBits;
+ }
+ D4 *= 0x0001000100010001ULL;
+ return D4;
+}
+
+typedef struct {
+ U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1];
+ U32 rankStart[HUF_TABLELOG_ABSOLUTEMAX + 1];
+ U32 statsWksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
+ BYTE symbols[HUF_SYMBOLVALUE_MAX + 1];
+ BYTE huffWeight[HUF_SYMBOLVALUE_MAX + 1];
+} HUF_ReadDTableX1_Workspace;
+
+
+size_t HUF_readDTableX1_wksp(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize)
+{
+ return HUF_readDTableX1_wksp_bmi2(DTable, src, srcSize, workSpace, wkspSize, /* bmi2 */ 0);
+}
+
+size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int bmi2)
+{
+ U32 tableLog = 0;
+ U32 nbSymbols = 0;
+ size_t iSize;
+ void* const dtPtr = DTable + 1;
+ HUF_DEltX1* const dt = (HUF_DEltX1*)dtPtr;
+ HUF_ReadDTableX1_Workspace* wksp = (HUF_ReadDTableX1_Workspace*)workSpace;
+
+ DEBUG_STATIC_ASSERT(HUF_DECOMPRESS_WORKSPACE_SIZE >= sizeof(*wksp));
+ if (sizeof(*wksp) > wkspSize) return ERROR(tableLog_tooLarge);
+
+ DEBUG_STATIC_ASSERT(sizeof(DTableDesc) == sizeof(HUF_DTable));
+ /* ZSTD_memset(huffWeight, 0, sizeof(huffWeight)); */ /* is not necessary, even though some analyzer complain ... */
+
+ iSize = HUF_readStats_wksp(wksp->huffWeight, HUF_SYMBOLVALUE_MAX + 1, wksp->rankVal, &nbSymbols, &tableLog, src, srcSize, wksp->statsWksp, sizeof(wksp->statsWksp), bmi2);
+ if (HUF_isError(iSize)) return iSize;
+
+ /* Table header */
+ { DTableDesc dtd = HUF_getDTableDesc(DTable);
+ if (tableLog > (U32)(dtd.maxTableLog+1)) return ERROR(tableLog_tooLarge); /* DTable too small, Huffman tree cannot fit in */
+ dtd.tableType = 0;
+ dtd.tableLog = (BYTE)tableLog;
+ ZSTD_memcpy(DTable, &dtd, sizeof(dtd));
+ }
+
+ /* Compute symbols and rankStart given rankVal:
+ *
+ * rankVal already contains the number of values of each weight.
+ *
+ * symbols contains the symbols ordered by weight. First are the rankVal[0]
+ * weight 0 symbols, followed by the rankVal[1] weight 1 symbols, and so on.
+ * symbols[0] is filled (but unused) to avoid a branch.
+ *
+ * rankStart contains the offset where each rank belongs in the DTable.
+ * rankStart[0] is not filled because there are no entries in the table for
+ * weight 0.
+ */
+ {
+ int n;
+ int nextRankStart = 0;
+ int const unroll = 4;
+ int const nLimit = (int)nbSymbols - unroll + 1;
+ for (n=0; n<(int)tableLog+1; n++) {
+ U32 const curr = nextRankStart;
+ nextRankStart += wksp->rankVal[n];
+ wksp->rankStart[n] = curr;
+ }
+ for (n=0; n < nLimit; n += unroll) {
+ int u;
+ for (u=0; u < unroll; ++u) {
+ size_t const w = wksp->huffWeight[n+u];
+ wksp->symbols[wksp->rankStart[w]++] = (BYTE)(n+u);
+ }
+ }
+ for (; n < (int)nbSymbols; ++n) {
+ size_t const w = wksp->huffWeight[n];
+ wksp->symbols[wksp->rankStart[w]++] = (BYTE)n;
+ }
+ }
+
+ /* fill DTable
+ * We fill all entries of each weight in order.
+ * That way length is a constant for each iteration of the outter loop.
+ * We can switch based on the length to a different inner loop which is
+ * optimized for that particular case.
+ */
+ {
+ U32 w;
+ int symbol=wksp->rankVal[0];
+ int rankStart=0;
+ for (w=1; w<tableLog+1; ++w) {
+ int const symbolCount = wksp->rankVal[w];
+ int const length = (1 << w) >> 1;
+ int uStart = rankStart;
+ BYTE const nbBits = (BYTE)(tableLog + 1 - w);
+ int s;
+ int u;
+ switch (length) {
+ case 1:
+ for (s=0; s<symbolCount; ++s) {
+ HUF_DEltX1 D;
+ D.byte = wksp->symbols[symbol + s];
+ D.nbBits = nbBits;
+ dt[uStart] = D;
+ uStart += 1;
+ }
+ break;
+ case 2:
+ for (s=0; s<symbolCount; ++s) {
+ HUF_DEltX1 D;
+ D.byte = wksp->symbols[symbol + s];
+ D.nbBits = nbBits;
+ dt[uStart+0] = D;
+ dt[uStart+1] = D;
+ uStart += 2;
+ }
+ break;
+ case 4:
+ for (s=0; s<symbolCount; ++s) {
+ U64 const D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits);
+ MEM_write64(dt + uStart, D4);
+ uStart += 4;
+ }
+ break;
+ case 8:
+ for (s=0; s<symbolCount; ++s) {
+ U64 const D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits);
+ MEM_write64(dt + uStart, D4);
+ MEM_write64(dt + uStart + 4, D4);
+ uStart += 8;
+ }
+ break;
+ default:
+ for (s=0; s<symbolCount; ++s) {
+ U64 const D4 = HUF_DEltX1_set4(wksp->symbols[symbol + s], nbBits);
+ for (u=0; u < length; u += 16) {
+ MEM_write64(dt + uStart + u + 0, D4);
+ MEM_write64(dt + uStart + u + 4, D4);
+ MEM_write64(dt + uStart + u + 8, D4);
+ MEM_write64(dt + uStart + u + 12, D4);
+ }
+ assert(u == length);
+ uStart += length;
+ }
+ break;
+ }
+ symbol += symbolCount;
+ rankStart += symbolCount * length;
+ }
+ }
+ return iSize;
+}
+
+FORCE_INLINE_TEMPLATE BYTE
+HUF_decodeSymbolX1(BIT_DStream_t* Dstream, const HUF_DEltX1* dt, const U32 dtLog)
+{
+ size_t const val = BIT_lookBitsFast(Dstream, dtLog); /* note : dtLog >= 1 */
+ BYTE const c = dt[val].byte;
+ BIT_skipBits(Dstream, dt[val].nbBits);
+ return c;
+}
+
+#define HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr) \
+ *ptr++ = HUF_decodeSymbolX1(DStreamPtr, dt, dtLog)
+
+#define HUF_DECODE_SYMBOLX1_1(ptr, DStreamPtr) \
+ if (MEM_64bits() || (HUF_TABLELOG_MAX<=12)) \
+ HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr)
+
+#define HUF_DECODE_SYMBOLX1_2(ptr, DStreamPtr) \
+ if (MEM_64bits()) \
+ HUF_DECODE_SYMBOLX1_0(ptr, DStreamPtr)
+
+HINT_INLINE size_t
+HUF_decodeStreamX1(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, const HUF_DEltX1* const dt, const U32 dtLog)
+{
+ BYTE* const pStart = p;
+
+ /* up to 4 symbols at a time */
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-3)) {
+ HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_1(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
+ }
+
+ /* [0-3] symbols remaining */
+ if (MEM_32bits())
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd))
+ HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
+
+ /* no more data to retrieve from bitstream, no need to reload */
+ while (p < pEnd)
+ HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
+
+ return pEnd-pStart;
+}
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_decompress1X1_usingDTable_internal_body(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ BYTE* op = (BYTE*)dst;
+ BYTE* const oend = op + dstSize;
+ const void* dtPtr = DTable + 1;
+ const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr;
+ BIT_DStream_t bitD;
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+ U32 const dtLog = dtd.tableLog;
+
+ CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) );
+
+ HUF_decodeStreamX1(op, &bitD, oend, dt, dtLog);
+
+ if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected);
+
+ return dstSize;
+}
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_decompress4X1_usingDTable_internal_body(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ /* Check */
+ if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */
+
+ { const BYTE* const istart = (const BYTE*) cSrc;
+ BYTE* const ostart = (BYTE*) dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* const olimit = oend - 3;
+ const void* const dtPtr = DTable + 1;
+ const HUF_DEltX1* const dt = (const HUF_DEltX1*)dtPtr;
+
+ /* Init */
+ BIT_DStream_t bitD1;
+ BIT_DStream_t bitD2;
+ BIT_DStream_t bitD3;
+ BIT_DStream_t bitD4;
+ size_t const length1 = MEM_readLE16(istart);
+ size_t const length2 = MEM_readLE16(istart+2);
+ size_t const length3 = MEM_readLE16(istart+4);
+ size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
+ const BYTE* const istart1 = istart + 6; /* jumpTable */
+ const BYTE* const istart2 = istart1 + length1;
+ const BYTE* const istart3 = istart2 + length2;
+ const BYTE* const istart4 = istart3 + length3;
+ const size_t segmentSize = (dstSize+3) / 4;
+ BYTE* const opStart2 = ostart + segmentSize;
+ BYTE* const opStart3 = opStart2 + segmentSize;
+ BYTE* const opStart4 = opStart3 + segmentSize;
+ BYTE* op1 = ostart;
+ BYTE* op2 = opStart2;
+ BYTE* op3 = opStart3;
+ BYTE* op4 = opStart4;
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+ U32 const dtLog = dtd.tableLog;
+ U32 endSignal = 1;
+
+ if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
+ CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
+ CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
+ CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
+ CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
+
+ /* up to 16 symbols per loop (4 symbols per stream) in 64-bit mode */
+ for ( ; (endSignal) & (op4 < olimit) ; ) {
+ HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_0(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_0(op4, &bitD4);
+ endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+ }
+
+ /* check corruption */
+ /* note : should not be necessary : op# advance in lock step, and we control op4.
+ * but curiously, binary generated by gcc 7.2 & 7.3 with -mbmi2 runs faster when >=1 test is present */
+ if (op1 > opStart2) return ERROR(corruption_detected);
+ if (op2 > opStart3) return ERROR(corruption_detected);
+ if (op3 > opStart4) return ERROR(corruption_detected);
+ /* note : op4 supposed already verified within main loop */
+
+ /* finish bitStreams one by one */
+ HUF_decodeStreamX1(op1, &bitD1, opStart2, dt, dtLog);
+ HUF_decodeStreamX1(op2, &bitD2, opStart3, dt, dtLog);
+ HUF_decodeStreamX1(op3, &bitD3, opStart4, dt, dtLog);
+ HUF_decodeStreamX1(op4, &bitD4, oend, dt, dtLog);
+
+ /* check */
+ { U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
+ if (!endCheck) return ERROR(corruption_detected); }
+
+ /* decoded size */
+ return dstSize;
+ }
+}
+
+
+typedef size_t (*HUF_decompress_usingDTable_t)(void *dst, size_t dstSize,
+ const void *cSrc,
+ size_t cSrcSize,
+ const HUF_DTable *DTable);
+
+HUF_DGEN(HUF_decompress1X1_usingDTable_internal)
+HUF_DGEN(HUF_decompress4X1_usingDTable_internal)
+
+
+
+size_t HUF_decompress1X1_usingDTable(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ if (dtd.tableType != 0) return ERROR(GENERIC);
+ return HUF_decompress1X1_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+}
+
+size_t HUF_decompress1X1_DCtx_wksp(HUF_DTable* DCtx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize)
+{
+ const BYTE* ip = (const BYTE*) cSrc;
+
+ size_t const hSize = HUF_readDTableX1_wksp(DCtx, cSrc, cSrcSize, workSpace, wkspSize);
+ if (HUF_isError(hSize)) return hSize;
+ if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
+ ip += hSize; cSrcSize -= hSize;
+
+ return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, /* bmi2 */ 0);
+}
+
+
+size_t HUF_decompress4X1_usingDTable(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ if (dtd.tableType != 0) return ERROR(GENERIC);
+ return HUF_decompress4X1_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+}
+
+static size_t HUF_decompress4X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize, int bmi2)
+{
+ const BYTE* ip = (const BYTE*) cSrc;
+
+ size_t const hSize = HUF_readDTableX1_wksp_bmi2(dctx, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
+ if (HUF_isError(hSize)) return hSize;
+ if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
+ ip += hSize; cSrcSize -= hSize;
+
+ return HUF_decompress4X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
+}
+
+size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, 0);
+}
+
+
+#endif /* HUF_FORCE_DECOMPRESS_X2 */
+
+
+#ifndef HUF_FORCE_DECOMPRESS_X1
+
+/* *************************/
+/* double-symbols decoding */
+/* *************************/
+
+typedef struct { U16 sequence; BYTE nbBits; BYTE length; } HUF_DEltX2; /* double-symbols decoding */
+typedef struct { BYTE symbol; BYTE weight; } sortedSymbol_t;
+typedef U32 rankValCol_t[HUF_TABLELOG_MAX + 1];
+typedef rankValCol_t rankVal_t[HUF_TABLELOG_MAX];
+
+
+/* HUF_fillDTableX2Level2() :
+ * `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */
+static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 sizeLog, const U32 consumed,
+ const U32* rankValOrigin, const int minWeight,
+ const sortedSymbol_t* sortedSymbols, const U32 sortedListSize,
+ U32 nbBitsBaseline, U16 baseSeq, U32* wksp, size_t wkspSize)
+{
+ HUF_DEltX2 DElt;
+ U32* rankVal = wksp;
+
+ assert(wkspSize >= HUF_TABLELOG_MAX + 1);
+ (void)wkspSize;
+ /* get pre-calculated rankVal */
+ ZSTD_memcpy(rankVal, rankValOrigin, sizeof(U32) * (HUF_TABLELOG_MAX + 1));
+
+ /* fill skipped values */
+ if (minWeight>1) {
+ U32 i, skipSize = rankVal[minWeight];
+ MEM_writeLE16(&(DElt.sequence), baseSeq);
+ DElt.nbBits = (BYTE)(consumed);
+ DElt.length = 1;
+ for (i = 0; i < skipSize; i++)
+ DTable[i] = DElt;
+ }
+
+ /* fill DTable */
+ { U32 s; for (s=0; s<sortedListSize; s++) { /* note : sortedSymbols already skipped */
+ const U32 symbol = sortedSymbols[s].symbol;
+ const U32 weight = sortedSymbols[s].weight;
+ const U32 nbBits = nbBitsBaseline - weight;
+ const U32 length = 1 << (sizeLog-nbBits);
+ const U32 start = rankVal[weight];
+ U32 i = start;
+ const U32 end = start + length;
+
+ MEM_writeLE16(&(DElt.sequence), (U16)(baseSeq + (symbol << 8)));
+ DElt.nbBits = (BYTE)(nbBits + consumed);
+ DElt.length = 2;
+ do { DTable[i++] = DElt; } while (i<end); /* since length >= 1 */
+
+ rankVal[weight] += length;
+ } }
+}
+
+
+static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,
+ const sortedSymbol_t* sortedList, const U32 sortedListSize,
+ const U32* rankStart, rankVal_t rankValOrigin, const U32 maxWeight,
+ const U32 nbBitsBaseline, U32* wksp, size_t wkspSize)
+{
+ U32* rankVal = wksp;
+ const int scaleLog = nbBitsBaseline - targetLog; /* note : targetLog >= srcLog, hence scaleLog <= 1 */
+ const U32 minBits = nbBitsBaseline - maxWeight;
+ U32 s;
+
+ assert(wkspSize >= HUF_TABLELOG_MAX + 1);
+ wksp += HUF_TABLELOG_MAX + 1;
+ wkspSize -= HUF_TABLELOG_MAX + 1;
+
+ ZSTD_memcpy(rankVal, rankValOrigin, sizeof(U32) * (HUF_TABLELOG_MAX + 1));
+
+ /* fill DTable */
+ for (s=0; s<sortedListSize; s++) {
+ const U16 symbol = sortedList[s].symbol;
+ const U32 weight = sortedList[s].weight;
+ const U32 nbBits = nbBitsBaseline - weight;
+ const U32 start = rankVal[weight];
+ const U32 length = 1 << (targetLog-nbBits);
+
+ if (targetLog-nbBits >= minBits) { /* enough room for a second symbol */
+ U32 sortedRank;
+ int minWeight = nbBits + scaleLog;
+ if (minWeight < 1) minWeight = 1;
+ sortedRank = rankStart[minWeight];
+ HUF_fillDTableX2Level2(DTable+start, targetLog-nbBits, nbBits,
+ rankValOrigin[nbBits], minWeight,
+ sortedList+sortedRank, sortedListSize-sortedRank,
+ nbBitsBaseline, symbol, wksp, wkspSize);
+ } else {
+ HUF_DEltX2 DElt;
+ MEM_writeLE16(&(DElt.sequence), symbol);
+ DElt.nbBits = (BYTE)(nbBits);
+ DElt.length = 1;
+ { U32 const end = start + length;
+ U32 u;
+ for (u = start; u < end; u++) DTable[u] = DElt;
+ } }
+ rankVal[weight] += length;
+ }
+}
+
+typedef struct {
+ rankValCol_t rankVal[HUF_TABLELOG_MAX];
+ U32 rankStats[HUF_TABLELOG_MAX + 1];
+ U32 rankStart0[HUF_TABLELOG_MAX + 2];
+ sortedSymbol_t sortedSymbol[HUF_SYMBOLVALUE_MAX + 1];
+ BYTE weightList[HUF_SYMBOLVALUE_MAX + 1];
+ U32 calleeWksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
+} HUF_ReadDTableX2_Workspace;
+
+size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize)
+{
+ U32 tableLog, maxW, sizeOfSort, nbSymbols;
+ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ U32 const maxTableLog = dtd.maxTableLog;
+ size_t iSize;
+ void* dtPtr = DTable+1; /* force compiler to avoid strict-aliasing */
+ HUF_DEltX2* const dt = (HUF_DEltX2*)dtPtr;
+ U32 *rankStart;
+
+ HUF_ReadDTableX2_Workspace* const wksp = (HUF_ReadDTableX2_Workspace*)workSpace;
+
+ if (sizeof(*wksp) > wkspSize) return ERROR(GENERIC);
+
+ rankStart = wksp->rankStart0 + 1;
+ ZSTD_memset(wksp->rankStats, 0, sizeof(wksp->rankStats));
+ ZSTD_memset(wksp->rankStart0, 0, sizeof(wksp->rankStart0));
+
+ DEBUG_STATIC_ASSERT(sizeof(HUF_DEltX2) == sizeof(HUF_DTable)); /* if compiler fails here, assertion is wrong */
+ if (maxTableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
+ /* ZSTD_memset(weightList, 0, sizeof(weightList)); */ /* is not necessary, even though some analyzer complain ... */
+
+ iSize = HUF_readStats_wksp(wksp->weightList, HUF_SYMBOLVALUE_MAX + 1, wksp->rankStats, &nbSymbols, &tableLog, src, srcSize, wksp->calleeWksp, sizeof(wksp->calleeWksp), /* bmi2 */ 0);
+ if (HUF_isError(iSize)) return iSize;
+
+ /* check result */
+ if (tableLog > maxTableLog) return ERROR(tableLog_tooLarge); /* DTable can't fit code depth */
+
+ /* find maxWeight */
+ for (maxW = tableLog; wksp->rankStats[maxW]==0; maxW--) {} /* necessarily finds a solution before 0 */
+
+ /* Get start index of each weight */
+ { U32 w, nextRankStart = 0;
+ for (w=1; w<maxW+1; w++) {
+ U32 curr = nextRankStart;
+ nextRankStart += wksp->rankStats[w];
+ rankStart[w] = curr;
+ }
+ rankStart[0] = nextRankStart; /* put all 0w symbols at the end of sorted list*/
+ sizeOfSort = nextRankStart;
+ }
+
+ /* sort symbols by weight */
+ { U32 s;
+ for (s=0; s<nbSymbols; s++) {
+ U32 const w = wksp->weightList[s];
+ U32 const r = rankStart[w]++;
+ wksp->sortedSymbol[r].symbol = (BYTE)s;
+ wksp->sortedSymbol[r].weight = (BYTE)w;
+ }
+ rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
+ }
+
+ /* Build rankVal */
+ { U32* const rankVal0 = wksp->rankVal[0];
+ { int const rescale = (maxTableLog-tableLog) - 1; /* tableLog <= maxTableLog */
+ U32 nextRankVal = 0;
+ U32 w;
+ for (w=1; w<maxW+1; w++) {
+ U32 curr = nextRankVal;
+ nextRankVal += wksp->rankStats[w] << (w+rescale);
+ rankVal0[w] = curr;
+ } }
+ { U32 const minBits = tableLog+1 - maxW;
+ U32 consumed;
+ for (consumed = minBits; consumed < maxTableLog - minBits + 1; consumed++) {
+ U32* const rankValPtr = wksp->rankVal[consumed];
+ U32 w;
+ for (w = 1; w < maxW+1; w++) {
+ rankValPtr[w] = rankVal0[w] >> consumed;
+ } } } }
+
+ HUF_fillDTableX2(dt, maxTableLog,
+ wksp->sortedSymbol, sizeOfSort,
+ wksp->rankStart0, wksp->rankVal, maxW,
+ tableLog+1,
+ wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
+
+ dtd.tableLog = (BYTE)maxTableLog;
+ dtd.tableType = 1;
+ ZSTD_memcpy(DTable, &dtd, sizeof(dtd));
+ return iSize;
+}
+
+
+FORCE_INLINE_TEMPLATE U32
+HUF_decodeSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
+{
+ size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
+ ZSTD_memcpy(op, dt+val, 2);
+ BIT_skipBits(DStream, dt[val].nbBits);
+ return dt[val].length;
+}
+
+FORCE_INLINE_TEMPLATE U32
+HUF_decodeLastSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
+{
+ size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
+ ZSTD_memcpy(op, dt+val, 1);
+ if (dt[val].length==1) BIT_skipBits(DStream, dt[val].nbBits);
+ else {
+ if (DStream->bitsConsumed < (sizeof(DStream->bitContainer)*8)) {
+ BIT_skipBits(DStream, dt[val].nbBits);
+ if (DStream->bitsConsumed > (sizeof(DStream->bitContainer)*8))
+ /* ugly hack; works only because it's the last symbol. Note : can't easily extract nbBits from just this symbol */
+ DStream->bitsConsumed = (sizeof(DStream->bitContainer)*8);
+ } }
+ return 1;
+}
+
+#define HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr) \
+ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)
+
+#define HUF_DECODE_SYMBOLX2_1(ptr, DStreamPtr) \
+ if (MEM_64bits() || (HUF_TABLELOG_MAX<=12)) \
+ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)
+
+#define HUF_DECODE_SYMBOLX2_2(ptr, DStreamPtr) \
+ if (MEM_64bits()) \
+ ptr += HUF_decodeSymbolX2(ptr, DStreamPtr, dt, dtLog)
+
+HINT_INLINE size_t
+HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd,
+ const HUF_DEltX2* const dt, const U32 dtLog)
+{
+ BYTE* const pStart = p;
+
+ /* up to 8 symbols at a time */
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-(sizeof(bitDPtr->bitContainer)-1))) {
+ HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_1(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ }
+
+ /* closer to end : up to 2 symbols at a time */
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd-2))
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+
+ while (p <= pEnd-2)
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr); /* no need to reload : reached the end of DStream */
+
+ if (p < pEnd)
+ p += HUF_decodeLastSymbolX2(p, bitDPtr, dt, dtLog);
+
+ return p-pStart;
+}
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_decompress1X2_usingDTable_internal_body(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ BIT_DStream_t bitD;
+
+ /* Init */
+ CHECK_F( BIT_initDStream(&bitD, cSrc, cSrcSize) );
+
+ /* decode */
+ { BYTE* const ostart = (BYTE*) dst;
+ BYTE* const oend = ostart + dstSize;
+ const void* const dtPtr = DTable+1; /* force compiler to not use strict-aliasing */
+ const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr;
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+ HUF_decodeStreamX2(ostart, &bitD, oend, dt, dtd.tableLog);
+ }
+
+ /* check */
+ if (!BIT_endOfDStream(&bitD)) return ERROR(corruption_detected);
+
+ /* decoded size */
+ return dstSize;
+}
+
+FORCE_INLINE_TEMPLATE size_t
+HUF_decompress4X2_usingDTable_internal_body(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ if (cSrcSize < 10) return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */
+
+ { const BYTE* const istart = (const BYTE*) cSrc;
+ BYTE* const ostart = (BYTE*) dst;
+ BYTE* const oend = ostart + dstSize;
+ BYTE* const olimit = oend - (sizeof(size_t)-1);
+ const void* const dtPtr = DTable+1;
+ const HUF_DEltX2* const dt = (const HUF_DEltX2*)dtPtr;
+
+ /* Init */
+ BIT_DStream_t bitD1;
+ BIT_DStream_t bitD2;
+ BIT_DStream_t bitD3;
+ BIT_DStream_t bitD4;
+ size_t const length1 = MEM_readLE16(istart);
+ size_t const length2 = MEM_readLE16(istart+2);
+ size_t const length3 = MEM_readLE16(istart+4);
+ size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
+ const BYTE* const istart1 = istart + 6; /* jumpTable */
+ const BYTE* const istart2 = istart1 + length1;
+ const BYTE* const istart3 = istart2 + length2;
+ const BYTE* const istart4 = istart3 + length3;
+ size_t const segmentSize = (dstSize+3) / 4;
+ BYTE* const opStart2 = ostart + segmentSize;
+ BYTE* const opStart3 = opStart2 + segmentSize;
+ BYTE* const opStart4 = opStart3 + segmentSize;
+ BYTE* op1 = ostart;
+ BYTE* op2 = opStart2;
+ BYTE* op3 = opStart3;
+ BYTE* op4 = opStart4;
+ U32 endSignal = 1;
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+ U32 const dtLog = dtd.tableLog;
+
+ if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
+ CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
+ CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
+ CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
+ CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
+
+ /* 16-32 symbols per loop (4-8 symbols per stream) */
+ for ( ; (endSignal) & (op4 < olimit); ) {
+#if defined(__clang__) && (defined(__x86_64__) || defined(__i386__))
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
+ endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
+ endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+#else
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
+ endSignal = (U32)LIKELY((U32)
+ (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished));
+#endif
+ }
+
+ /* check corruption */
+ if (op1 > opStart2) return ERROR(corruption_detected);
+ if (op2 > opStart3) return ERROR(corruption_detected);
+ if (op3 > opStart4) return ERROR(corruption_detected);
+ /* note : op4 already verified within main loop */
+
+ /* finish bitStreams one by one */
+ HUF_decodeStreamX2(op1, &bitD1, opStart2, dt, dtLog);
+ HUF_decodeStreamX2(op2, &bitD2, opStart3, dt, dtLog);
+ HUF_decodeStreamX2(op3, &bitD3, opStart4, dt, dtLog);
+ HUF_decodeStreamX2(op4, &bitD4, oend, dt, dtLog);
+
+ /* check */
+ { U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
+ if (!endCheck) return ERROR(corruption_detected); }
+
+ /* decoded size */
+ return dstSize;
+ }
+}
+
+HUF_DGEN(HUF_decompress1X2_usingDTable_internal)
+HUF_DGEN(HUF_decompress4X2_usingDTable_internal)
+
+size_t HUF_decompress1X2_usingDTable(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ if (dtd.tableType != 1) return ERROR(GENERIC);
+ return HUF_decompress1X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+}
+
+size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable* DCtx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize)
+{
+ const BYTE* ip = (const BYTE*) cSrc;
+
+ size_t const hSize = HUF_readDTableX2_wksp(DCtx, cSrc, cSrcSize,
+ workSpace, wkspSize);
+ if (HUF_isError(hSize)) return hSize;
+ if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
+ ip += hSize; cSrcSize -= hSize;
+
+ return HUF_decompress1X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx, /* bmi2 */ 0);
+}
+
+
+size_t HUF_decompress4X2_usingDTable(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ if (dtd.tableType != 1) return ERROR(GENERIC);
+ return HUF_decompress4X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+}
+
+static size_t HUF_decompress4X2_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize, int bmi2)
+{
+ const BYTE* ip = (const BYTE*) cSrc;
+
+ size_t hSize = HUF_readDTableX2_wksp(dctx, cSrc, cSrcSize,
+ workSpace, wkspSize);
+ if (HUF_isError(hSize)) return hSize;
+ if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
+ ip += hSize; cSrcSize -= hSize;
+
+ return HUF_decompress4X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
+}
+
+size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize)
+{
+ return HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, /* bmi2 */ 0);
+}
+
+
+#endif /* HUF_FORCE_DECOMPRESS_X1 */
+
+
+/* ***********************************/
+/* Universal decompression selectors */
+/* ***********************************/
+
+size_t HUF_decompress1X_usingDTable(void* dst, size_t maxDstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)dtd;
+ assert(dtd.tableType == 0);
+ return HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)dtd;
+ assert(dtd.tableType == 1);
+ return HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#else
+ return dtd.tableType ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0) :
+ HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#endif
+}
+
+size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)dtd;
+ assert(dtd.tableType == 0);
+ return HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)dtd;
+ assert(dtd.tableType == 1);
+ return HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#else
+ return dtd.tableType ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0) :
+ HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, /* bmi2 */ 0);
+#endif
+}
+
+
+#if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2)
+typedef struct { U32 tableTime; U32 decode256Time; } algo_time_t;
+static const algo_time_t algoTime[16 /* Quantization */][3 /* single, double, quad */] =
+{
+ /* single, double, quad */
+ {{0,0}, {1,1}, {2,2}}, /* Q==0 : impossible */
+ {{0,0}, {1,1}, {2,2}}, /* Q==1 : impossible */
+ {{ 38,130}, {1313, 74}, {2151, 38}}, /* Q == 2 : 12-18% */
+ {{ 448,128}, {1353, 74}, {2238, 41}}, /* Q == 3 : 18-25% */
+ {{ 556,128}, {1353, 74}, {2238, 47}}, /* Q == 4 : 25-32% */
+ {{ 714,128}, {1418, 74}, {2436, 53}}, /* Q == 5 : 32-38% */
+ {{ 883,128}, {1437, 74}, {2464, 61}}, /* Q == 6 : 38-44% */
+ {{ 897,128}, {1515, 75}, {2622, 68}}, /* Q == 7 : 44-50% */
+ {{ 926,128}, {1613, 75}, {2730, 75}}, /* Q == 8 : 50-56% */
+ {{ 947,128}, {1729, 77}, {3359, 77}}, /* Q == 9 : 56-62% */
+ {{1107,128}, {2083, 81}, {4006, 84}}, /* Q ==10 : 62-69% */
+ {{1177,128}, {2379, 87}, {4785, 88}}, /* Q ==11 : 69-75% */
+ {{1242,128}, {2415, 93}, {5155, 84}}, /* Q ==12 : 75-81% */
+ {{1349,128}, {2644,106}, {5260,106}}, /* Q ==13 : 81-87% */
+ {{1455,128}, {2422,124}, {4174,124}}, /* Q ==14 : 87-93% */
+ {{ 722,128}, {1891,145}, {1936,146}}, /* Q ==15 : 93-99% */
+};
+#endif
+
+/* HUF_selectDecoder() :
+ * Tells which decoder is likely to decode faster,
+ * based on a set of pre-computed metrics.
+ * @return : 0==HUF_decompress4X1, 1==HUF_decompress4X2 .
+ * Assumption : 0 < dstSize <= 128 KB */
+U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize)
+{
+ assert(dstSize > 0);
+ assert(dstSize <= 128*1024);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)dstSize;
+ (void)cSrcSize;
+ return 0;
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)dstSize;
+ (void)cSrcSize;
+ return 1;
+#else
+ /* decoder timing evaluation */
+ { U32 const Q = (cSrcSize >= dstSize) ? 15 : (U32)(cSrcSize * 16 / dstSize); /* Q < 16 */
+ U32 const D256 = (U32)(dstSize >> 8);
+ U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256);
+ U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256);
+ DTime1 += DTime1 >> 3; /* advantage to algorithm using less memory, to reduce cache eviction */
+ return DTime1 < DTime0;
+ }
+#endif
+}
+
+
+size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst,
+ size_t dstSize, const void* cSrc,
+ size_t cSrcSize, void* workSpace,
+ size_t wkspSize)
+{
+ /* validation checks */
+ if (dstSize == 0) return ERROR(dstSize_tooSmall);
+ if (cSrcSize == 0) return ERROR(corruption_detected);
+
+ { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)algoNb;
+ assert(algoNb == 0);
+ return HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)algoNb;
+ assert(algoNb == 1);
+ return HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
+#else
+ return algoNb ? HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
+ cSrcSize, workSpace, wkspSize):
+ HUF_decompress4X1_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize);
+#endif
+ }
+}
+
+size_t HUF_decompress1X_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ void* workSpace, size_t wkspSize)
+{
+ /* validation checks */
+ if (dstSize == 0) return ERROR(dstSize_tooSmall);
+ if (cSrcSize > dstSize) return ERROR(corruption_detected); /* invalid */
+ if (cSrcSize == dstSize) { ZSTD_memcpy(dst, cSrc, dstSize); return dstSize; } /* not compressed */
+ if (cSrcSize == 1) { ZSTD_memset(dst, *(const BYTE*)cSrc, dstSize); return dstSize; } /* RLE */
+
+ { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)algoNb;
+ assert(algoNb == 0);
+ return HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc,
+ cSrcSize, workSpace, wkspSize);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)algoNb;
+ assert(algoNb == 1);
+ return HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
+ cSrcSize, workSpace, wkspSize);
+#else
+ return algoNb ? HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc,
+ cSrcSize, workSpace, wkspSize):
+ HUF_decompress1X1_DCtx_wksp(dctx, dst, dstSize, cSrc,
+ cSrcSize, workSpace, wkspSize);
+#endif
+ }
+}
+
+
+size_t HUF_decompress1X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2)
+{
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)dtd;
+ assert(dtd.tableType == 0);
+ return HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)dtd;
+ assert(dtd.tableType == 1);
+ return HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#else
+ return dtd.tableType ? HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2) :
+ HUF_decompress1X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#endif
+}
+
+#ifndef HUF_FORCE_DECOMPRESS_X2
+size_t HUF_decompress1X1_DCtx_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2)
+{
+ const BYTE* ip = (const BYTE*) cSrc;
+
+ size_t const hSize = HUF_readDTableX1_wksp_bmi2(dctx, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
+ if (HUF_isError(hSize)) return hSize;
+ if (hSize >= cSrcSize) return ERROR(srcSize_wrong);
+ ip += hSize; cSrcSize -= hSize;
+
+ return HUF_decompress1X1_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx, bmi2);
+}
+#endif
+
+size_t HUF_decompress4X_usingDTable_bmi2(void* dst, size_t maxDstSize, const void* cSrc, size_t cSrcSize, const HUF_DTable* DTable, int bmi2)
+{
+ DTableDesc const dtd = HUF_getDTableDesc(DTable);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)dtd;
+ assert(dtd.tableType == 0);
+ return HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)dtd;
+ assert(dtd.tableType == 1);
+ return HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#else
+ return dtd.tableType ? HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2) :
+ HUF_decompress4X1_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable, bmi2);
+#endif
+}
+
+size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize, void* workSpace, size_t wkspSize, int bmi2)
+{
+ /* validation checks */
+ if (dstSize == 0) return ERROR(dstSize_tooSmall);
+ if (cSrcSize == 0) return ERROR(corruption_detected);
+
+ { U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
+#if defined(HUF_FORCE_DECOMPRESS_X1)
+ (void)algoNb;
+ assert(algoNb == 0);
+ return HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
+#elif defined(HUF_FORCE_DECOMPRESS_X2)
+ (void)algoNb;
+ assert(algoNb == 1);
+ return HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
+#else
+ return algoNb ? HUF_decompress4X2_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2) :
+ HUF_decompress4X1_DCtx_wksp_bmi2(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize, bmi2);
+#endif
+ }
+}
+
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/* zstd_ddict.c :
+ * concentrates all logic that needs to know the internals of ZSTD_DDict object */
+
+/*-*******************************************************
+* Dependencies
+*********************************************************/
+#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */
+#include "../common/cpu.h" /* bmi2 */
+#include "../common/mem.h" /* low level memory routines */
+#define FSE_STATIC_LINKING_ONLY
+#include "../common/fse.h"
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include "zstd_decompress_internal.h"
+#include "zstd_ddict.h"
+
+
+
+
+/*-*******************************************************
+* Types
+*********************************************************/
+struct ZSTD_DDict_s {
+ void* dictBuffer;
+ const void* dictContent;
+ size_t dictSize;
+ ZSTD_entropyDTables_t entropy;
+ U32 dictID;
+ U32 entropyPresent;
+ ZSTD_customMem cMem;
+}; /* typedef'd to ZSTD_DDict within "zstd.h" */
+
+const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict)
+{
+ assert(ddict != NULL);
+ return ddict->dictContent;
+}
+
+size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict)
+{
+ assert(ddict != NULL);
+ return ddict->dictSize;
+}
+
+void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
+{
+ DEBUGLOG(4, "ZSTD_copyDDictParameters");
+ assert(dctx != NULL);
+ assert(ddict != NULL);
+ dctx->dictID = ddict->dictID;
+ dctx->prefixStart = ddict->dictContent;
+ dctx->virtualStart = ddict->dictContent;
+ dctx->dictEnd = (const BYTE*)ddict->dictContent + ddict->dictSize;
+ dctx->previousDstEnd = dctx->dictEnd;
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ dctx->dictContentBeginForFuzzing = dctx->prefixStart;
+ dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
+#endif
+ if (ddict->entropyPresent) {
+ dctx->litEntropy = 1;
+ dctx->fseEntropy = 1;
+ dctx->LLTptr = ddict->entropy.LLTable;
+ dctx->MLTptr = ddict->entropy.MLTable;
+ dctx->OFTptr = ddict->entropy.OFTable;
+ dctx->HUFptr = ddict->entropy.hufTable;
+ dctx->entropy.rep[0] = ddict->entropy.rep[0];
+ dctx->entropy.rep[1] = ddict->entropy.rep[1];
+ dctx->entropy.rep[2] = ddict->entropy.rep[2];
+ } else {
+ dctx->litEntropy = 0;
+ dctx->fseEntropy = 0;
+ }
+}
+
+
+static size_t
+ZSTD_loadEntropy_intoDDict(ZSTD_DDict* ddict,
+ ZSTD_dictContentType_e dictContentType)
+{
+ ddict->dictID = 0;
+ ddict->entropyPresent = 0;
+ if (dictContentType == ZSTD_dct_rawContent) return 0;
+
+ if (ddict->dictSize < 8) {
+ if (dictContentType == ZSTD_dct_fullDict)
+ return ERROR(dictionary_corrupted); /* only accept specified dictionaries */
+ return 0; /* pure content mode */
+ }
+ { U32 const magic = MEM_readLE32(ddict->dictContent);
+ if (magic != ZSTD_MAGIC_DICTIONARY) {
+ if (dictContentType == ZSTD_dct_fullDict)
+ return ERROR(dictionary_corrupted); /* only accept specified dictionaries */
+ return 0; /* pure content mode */
+ }
+ }
+ ddict->dictID = MEM_readLE32((const char*)ddict->dictContent + ZSTD_FRAMEIDSIZE);
+
+ /* load entropy tables */
+ RETURN_ERROR_IF(ZSTD_isError(ZSTD_loadDEntropy(
+ &ddict->entropy, ddict->dictContent, ddict->dictSize)),
+ dictionary_corrupted, "");
+ ddict->entropyPresent = 1;
+ return 0;
+}
+
+
+static size_t ZSTD_initDDict_internal(ZSTD_DDict* ddict,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType)
+{
+ if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dict) || (!dictSize)) {
+ ddict->dictBuffer = NULL;
+ ddict->dictContent = dict;
+ if (!dict) dictSize = 0;
+ } else {
+ void* const internalBuffer = ZSTD_customMalloc(dictSize, ddict->cMem);
+ ddict->dictBuffer = internalBuffer;
+ ddict->dictContent = internalBuffer;
+ if (!internalBuffer) return ERROR(memory_allocation);
+ ZSTD_memcpy(internalBuffer, dict, dictSize);
+ }
+ ddict->dictSize = dictSize;
+ ddict->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
+
+ /* parse dictionary content */
+ FORWARD_IF_ERROR( ZSTD_loadEntropy_intoDDict(ddict, dictContentType) , "");
+
+ return 0;
+}
+
+ZSTD_DDict* ZSTD_createDDict_advanced(const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType,
+ ZSTD_customMem customMem)
+{
+ if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
+
+ { ZSTD_DDict* const ddict = (ZSTD_DDict*) ZSTD_customMalloc(sizeof(ZSTD_DDict), customMem);
+ if (ddict == NULL) return NULL;
+ ddict->cMem = customMem;
+ { size_t const initResult = ZSTD_initDDict_internal(ddict,
+ dict, dictSize,
+ dictLoadMethod, dictContentType);
+ if (ZSTD_isError(initResult)) {
+ ZSTD_freeDDict(ddict);
+ return NULL;
+ } }
+ return ddict;
+ }
+}
+
+/*! ZSTD_createDDict() :
+* Create a digested dictionary, to start decompression without startup delay.
+* `dict` content is copied inside DDict.
+* Consequently, `dict` can be released after `ZSTD_DDict` creation */
+ZSTD_DDict* ZSTD_createDDict(const void* dict, size_t dictSize)
+{
+ ZSTD_customMem const allocator = { NULL, NULL, NULL };
+ return ZSTD_createDDict_advanced(dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto, allocator);
+}
+
+/*! ZSTD_createDDict_byReference() :
+ * Create a digested dictionary, to start decompression without startup delay.
+ * Dictionary content is simply referenced, it will be accessed during decompression.
+ * Warning : dictBuffer must outlive DDict (DDict must be freed before dictBuffer) */
+ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize)
+{
+ ZSTD_customMem const allocator = { NULL, NULL, NULL };
+ return ZSTD_createDDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto, allocator);
+}
+
+
+const ZSTD_DDict* ZSTD_initStaticDDict(
+ void* sBuffer, size_t sBufferSize,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType)
+{
+ size_t const neededSpace = sizeof(ZSTD_DDict)
+ + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize);
+ ZSTD_DDict* const ddict = (ZSTD_DDict*)sBuffer;
+ assert(sBuffer != NULL);
+ assert(dict != NULL);
+ if ((size_t)sBuffer & 7) return NULL; /* 8-aligned */
+ if (sBufferSize < neededSpace) return NULL;
+ if (dictLoadMethod == ZSTD_dlm_byCopy) {
+ ZSTD_memcpy(ddict+1, dict, dictSize); /* local copy */
+ dict = ddict+1;
+ }
+ if (ZSTD_isError( ZSTD_initDDict_internal(ddict,
+ dict, dictSize,
+ ZSTD_dlm_byRef, dictContentType) ))
+ return NULL;
+ return ddict;
+}
+
+
+size_t ZSTD_freeDDict(ZSTD_DDict* ddict)
+{
+ if (ddict==NULL) return 0; /* support free on NULL */
+ { ZSTD_customMem const cMem = ddict->cMem;
+ ZSTD_customFree(ddict->dictBuffer, cMem);
+ ZSTD_customFree(ddict, cMem);
+ return 0;
+ }
+}
+
+/*! ZSTD_estimateDDictSize() :
+ * Estimate amount of memory that will be needed to create a dictionary for decompression.
+ * Note : dictionary created by reference using ZSTD_dlm_byRef are smaller */
+size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod)
+{
+ return sizeof(ZSTD_DDict) + (dictLoadMethod == ZSTD_dlm_byRef ? 0 : dictSize);
+}
+
+size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict)
+{
+ if (ddict==NULL) return 0; /* support sizeof on NULL */
+ return sizeof(*ddict) + (ddict->dictBuffer ? ddict->dictSize : 0) ;
+}
+
+/*! ZSTD_getDictID_fromDDict() :
+ * Provides the dictID of the dictionary loaded into `ddict`.
+ * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
+ * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
+unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict)
+{
+ if (ddict==NULL) return 0;
+ return ZSTD_getDictID_fromDict(ddict->dictContent, ddict->dictSize);
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+
+#ifndef ZSTD_DDICT_H
+#define ZSTD_DDICT_H
+
+/*-*******************************************************
+ * Dependencies
+ *********************************************************/
+#include "../common/zstd_deps.h" /* size_t */
+#include <linux/zstd.h> /* ZSTD_DDict, and several public functions */
+
+
+/*-*******************************************************
+ * Interface
+ *********************************************************/
+
+/* note: several prototypes are already published in `zstd.h` :
+ * ZSTD_createDDict()
+ * ZSTD_createDDict_byReference()
+ * ZSTD_createDDict_advanced()
+ * ZSTD_freeDDict()
+ * ZSTD_initStaticDDict()
+ * ZSTD_sizeof_DDict()
+ * ZSTD_estimateDDictSize()
+ * ZSTD_getDictID_fromDict()
+ */
+
+const void* ZSTD_DDict_dictContent(const ZSTD_DDict* ddict);
+size_t ZSTD_DDict_dictSize(const ZSTD_DDict* ddict);
+
+void ZSTD_copyDDictParameters(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
+
+
+
+#endif /* ZSTD_DDICT_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+
+/* ***************************************************************
+* Tuning parameters
+*****************************************************************/
+/*!
+ * HEAPMODE :
+ * Select how default decompression function ZSTD_decompress() allocates its context,
+ * on stack (0), or into heap (1, default; requires malloc()).
+ * Note that functions with explicit context such as ZSTD_decompressDCtx() are unaffected.
+ */
+#ifndef ZSTD_HEAPMODE
+# define ZSTD_HEAPMODE 1
+#endif
+
+/*!
+* LEGACY_SUPPORT :
+* if set to 1+, ZSTD_decompress() can decode older formats (v0.1+)
+*/
+
+/*!
+ * MAXWINDOWSIZE_DEFAULT :
+ * maximum window size accepted by DStream __by default__.
+ * Frames requiring more memory will be rejected.
+ * It's possible to set a different limit using ZSTD_DCtx_setMaxWindowSize().
+ */
+#ifndef ZSTD_MAXWINDOWSIZE_DEFAULT
+# define ZSTD_MAXWINDOWSIZE_DEFAULT (((U32)1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) + 1)
+#endif
+
+/*!
+ * NO_FORWARD_PROGRESS_MAX :
+ * maximum allowed nb of calls to ZSTD_decompressStream()
+ * without any forward progress
+ * (defined as: no byte read from input, and no byte flushed to output)
+ * before triggering an error.
+ */
+#ifndef ZSTD_NO_FORWARD_PROGRESS_MAX
+# define ZSTD_NO_FORWARD_PROGRESS_MAX 16
+#endif
+
+
+/*-*******************************************************
+* Dependencies
+*********************************************************/
+#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */
+#include "../common/cpu.h" /* bmi2 */
+#include "../common/mem.h" /* low level memory routines */
+#define FSE_STATIC_LINKING_ONLY
+#include "../common/fse.h"
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include <linux/xxhash.h> /* xxh64_reset, xxh64_update, xxh64_digest, XXH64 */
+#include "../common/zstd_internal.h" /* blockProperties_t */
+#include "zstd_decompress_internal.h" /* ZSTD_DCtx */
+#include "zstd_ddict.h" /* ZSTD_DDictDictContent */
+#include "zstd_decompress_block.h" /* ZSTD_decompressBlock_internal */
+
+
+
+
+/* ***********************************
+ * Multiple DDicts Hashset internals *
+ *************************************/
+
+#define DDICT_HASHSET_MAX_LOAD_FACTOR_COUNT_MULT 4
+#define DDICT_HASHSET_MAX_LOAD_FACTOR_SIZE_MULT 3 /* These two constants represent SIZE_MULT/COUNT_MULT load factor without using a float.
+ * Currently, that means a 0.75 load factor.
+ * So, if count * COUNT_MULT / size * SIZE_MULT != 0, then we've exceeded
+ * the load factor of the ddict hash set.
+ */
+
+#define DDICT_HASHSET_TABLE_BASE_SIZE 64
+#define DDICT_HASHSET_RESIZE_FACTOR 2
+
+/* Hash function to determine starting position of dict insertion within the table
+ * Returns an index between [0, hashSet->ddictPtrTableSize]
+ */
+static size_t ZSTD_DDictHashSet_getIndex(const ZSTD_DDictHashSet* hashSet, U32 dictID) {
+ const U64 hash = xxh64(&dictID, sizeof(U32), 0);
+ /* DDict ptr table size is a multiple of 2, use size - 1 as mask to get index within [0, hashSet->ddictPtrTableSize) */
+ return hash & (hashSet->ddictPtrTableSize - 1);
+}
+
+/* Adds DDict to a hashset without resizing it.
+ * If inserting a DDict with a dictID that already exists in the set, replaces the one in the set.
+ * Returns 0 if successful, or a zstd error code if something went wrong.
+ */
+static size_t ZSTD_DDictHashSet_emplaceDDict(ZSTD_DDictHashSet* hashSet, const ZSTD_DDict* ddict) {
+ const U32 dictID = ZSTD_getDictID_fromDDict(ddict);
+ size_t idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID);
+ const size_t idxRangeMask = hashSet->ddictPtrTableSize - 1;
+ RETURN_ERROR_IF(hashSet->ddictPtrCount == hashSet->ddictPtrTableSize, GENERIC, "Hash set is full!");
+ DEBUGLOG(4, "Hashed index: for dictID: %u is %zu", dictID, idx);
+ while (hashSet->ddictPtrTable[idx] != NULL) {
+ /* Replace existing ddict if inserting ddict with same dictID */
+ if (ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]) == dictID) {
+ DEBUGLOG(4, "DictID already exists, replacing rather than adding");
+ hashSet->ddictPtrTable[idx] = ddict;
+ return 0;
+ }
+ idx &= idxRangeMask;
+ idx++;
+ }
+ DEBUGLOG(4, "Final idx after probing for dictID %u is: %zu", dictID, idx);
+ hashSet->ddictPtrTable[idx] = ddict;
+ hashSet->ddictPtrCount++;
+ return 0;
+}
+
+/* Expands hash table by factor of DDICT_HASHSET_RESIZE_FACTOR and
+ * rehashes all values, allocates new table, frees old table.
+ * Returns 0 on success, otherwise a zstd error code.
+ */
+static size_t ZSTD_DDictHashSet_expand(ZSTD_DDictHashSet* hashSet, ZSTD_customMem customMem) {
+ size_t newTableSize = hashSet->ddictPtrTableSize * DDICT_HASHSET_RESIZE_FACTOR;
+ const ZSTD_DDict** newTable = (const ZSTD_DDict**)ZSTD_customCalloc(sizeof(ZSTD_DDict*) * newTableSize, customMem);
+ const ZSTD_DDict** oldTable = hashSet->ddictPtrTable;
+ size_t oldTableSize = hashSet->ddictPtrTableSize;
+ size_t i;
+
+ DEBUGLOG(4, "Expanding DDict hash table! Old size: %zu new size: %zu", oldTableSize, newTableSize);
+ RETURN_ERROR_IF(!newTable, memory_allocation, "Expanded hashset allocation failed!");
+ hashSet->ddictPtrTable = newTable;
+ hashSet->ddictPtrTableSize = newTableSize;
+ hashSet->ddictPtrCount = 0;
+ for (i = 0; i < oldTableSize; ++i) {
+ if (oldTable[i] != NULL) {
+ FORWARD_IF_ERROR(ZSTD_DDictHashSet_emplaceDDict(hashSet, oldTable[i]), "");
+ }
+ }
+ ZSTD_customFree((void*)oldTable, customMem);
+ DEBUGLOG(4, "Finished re-hash");
+ return 0;
+}
+
+/* Fetches a DDict with the given dictID
+ * Returns the ZSTD_DDict* with the requested dictID. If it doesn't exist, then returns NULL.
+ */
+static const ZSTD_DDict* ZSTD_DDictHashSet_getDDict(ZSTD_DDictHashSet* hashSet, U32 dictID) {
+ size_t idx = ZSTD_DDictHashSet_getIndex(hashSet, dictID);
+ const size_t idxRangeMask = hashSet->ddictPtrTableSize - 1;
+ DEBUGLOG(4, "Hashed index: for dictID: %u is %zu", dictID, idx);
+ for (;;) {
+ size_t currDictID = ZSTD_getDictID_fromDDict(hashSet->ddictPtrTable[idx]);
+ if (currDictID == dictID || currDictID == 0) {
+ /* currDictID == 0 implies a NULL ddict entry */
+ break;
+ } else {
+ idx &= idxRangeMask; /* Goes to start of table when we reach the end */
+ idx++;
+ }
+ }
+ DEBUGLOG(4, "Final idx after probing for dictID %u is: %zu", dictID, idx);
+ return hashSet->ddictPtrTable[idx];
+}
+
+/* Allocates space for and returns a ddict hash set
+ * The hash set's ZSTD_DDict* table has all values automatically set to NULL to begin with.
+ * Returns NULL if allocation failed.
+ */
+static ZSTD_DDictHashSet* ZSTD_createDDictHashSet(ZSTD_customMem customMem) {
+ ZSTD_DDictHashSet* ret = (ZSTD_DDictHashSet*)ZSTD_customMalloc(sizeof(ZSTD_DDictHashSet), customMem);
+ DEBUGLOG(4, "Allocating new hash set");
+ if (!ret)
+ return NULL;
+ ret->ddictPtrTable = (const ZSTD_DDict**)ZSTD_customCalloc(DDICT_HASHSET_TABLE_BASE_SIZE * sizeof(ZSTD_DDict*), customMem);
+ if (!ret->ddictPtrTable) {
+ ZSTD_customFree(ret, customMem);
+ return NULL;
+ }
+ ret->ddictPtrTableSize = DDICT_HASHSET_TABLE_BASE_SIZE;
+ ret->ddictPtrCount = 0;
+ return ret;
+}
+
+/* Frees the table of ZSTD_DDict* within a hashset, then frees the hashset itself.
+ * Note: The ZSTD_DDict* within the table are NOT freed.
+ */
+static void ZSTD_freeDDictHashSet(ZSTD_DDictHashSet* hashSet, ZSTD_customMem customMem) {
+ DEBUGLOG(4, "Freeing ddict hash set");
+ if (hashSet && hashSet->ddictPtrTable) {
+ ZSTD_customFree((void*)hashSet->ddictPtrTable, customMem);
+ }
+ if (hashSet) {
+ ZSTD_customFree(hashSet, customMem);
+ }
+}
+
+/* Public function: Adds a DDict into the ZSTD_DDictHashSet, possibly triggering a resize of the hash set.
+ * Returns 0 on success, or a ZSTD error.
+ */
+static size_t ZSTD_DDictHashSet_addDDict(ZSTD_DDictHashSet* hashSet, const ZSTD_DDict* ddict, ZSTD_customMem customMem) {
+ DEBUGLOG(4, "Adding dict ID: %u to hashset with - Count: %zu Tablesize: %zu", ZSTD_getDictID_fromDDict(ddict), hashSet->ddictPtrCount, hashSet->ddictPtrTableSize);
+ if (hashSet->ddictPtrCount * DDICT_HASHSET_MAX_LOAD_FACTOR_COUNT_MULT / hashSet->ddictPtrTableSize * DDICT_HASHSET_MAX_LOAD_FACTOR_SIZE_MULT != 0) {
+ FORWARD_IF_ERROR(ZSTD_DDictHashSet_expand(hashSet, customMem), "");
+ }
+ FORWARD_IF_ERROR(ZSTD_DDictHashSet_emplaceDDict(hashSet, ddict), "");
+ return 0;
+}
+
+/*-*************************************************************
+* Context management
+***************************************************************/
+size_t ZSTD_sizeof_DCtx (const ZSTD_DCtx* dctx)
+{
+ if (dctx==NULL) return 0; /* support sizeof NULL */
+ return sizeof(*dctx)
+ + ZSTD_sizeof_DDict(dctx->ddictLocal)
+ + dctx->inBuffSize + dctx->outBuffSize;
+}
+
+size_t ZSTD_estimateDCtxSize(void) { return sizeof(ZSTD_DCtx); }
+
+
+static size_t ZSTD_startingInputLength(ZSTD_format_e format)
+{
+ size_t const startingInputLength = ZSTD_FRAMEHEADERSIZE_PREFIX(format);
+ /* only supports formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless */
+ assert( (format == ZSTD_f_zstd1) || (format == ZSTD_f_zstd1_magicless) );
+ return startingInputLength;
+}
+
+static void ZSTD_DCtx_resetParameters(ZSTD_DCtx* dctx)
+{
+ assert(dctx->streamStage == zdss_init);
+ dctx->format = ZSTD_f_zstd1;
+ dctx->maxWindowSize = ZSTD_MAXWINDOWSIZE_DEFAULT;
+ dctx->outBufferMode = ZSTD_bm_buffered;
+ dctx->forceIgnoreChecksum = ZSTD_d_validateChecksum;
+ dctx->refMultipleDDicts = ZSTD_rmd_refSingleDDict;
+}
+
+static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
+{
+ dctx->staticSize = 0;
+ dctx->ddict = NULL;
+ dctx->ddictLocal = NULL;
+ dctx->dictEnd = NULL;
+ dctx->ddictIsCold = 0;
+ dctx->dictUses = ZSTD_dont_use;
+ dctx->inBuff = NULL;
+ dctx->inBuffSize = 0;
+ dctx->outBuffSize = 0;
+ dctx->streamStage = zdss_init;
+ dctx->legacyContext = NULL;
+ dctx->previousLegacyVersion = 0;
+ dctx->noForwardProgress = 0;
+ dctx->oversizedDuration = 0;
+ dctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+ dctx->ddictSet = NULL;
+ ZSTD_DCtx_resetParameters(dctx);
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ dctx->dictContentEndForFuzzing = NULL;
+#endif
+}
+
+ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
+{
+ ZSTD_DCtx* const dctx = (ZSTD_DCtx*) workspace;
+
+ if ((size_t)workspace & 7) return NULL; /* 8-aligned */
+ if (workspaceSize < sizeof(ZSTD_DCtx)) return NULL; /* minimum size */
+
+ ZSTD_initDCtx_internal(dctx);
+ dctx->staticSize = workspaceSize;
+ dctx->inBuff = (char*)(dctx+1);
+ return dctx;
+}
+
+ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
+{
+ if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
+
+ { ZSTD_DCtx* const dctx = (ZSTD_DCtx*)ZSTD_customMalloc(sizeof(*dctx), customMem);
+ if (!dctx) return NULL;
+ dctx->customMem = customMem;
+ ZSTD_initDCtx_internal(dctx);
+ return dctx;
+ }
+}
+
+ZSTD_DCtx* ZSTD_createDCtx(void)
+{
+ DEBUGLOG(3, "ZSTD_createDCtx");
+ return ZSTD_createDCtx_advanced(ZSTD_defaultCMem);
+}
+
+static void ZSTD_clearDict(ZSTD_DCtx* dctx)
+{
+ ZSTD_freeDDict(dctx->ddictLocal);
+ dctx->ddictLocal = NULL;
+ dctx->ddict = NULL;
+ dctx->dictUses = ZSTD_dont_use;
+}
+
+size_t ZSTD_freeDCtx(ZSTD_DCtx* dctx)
+{
+ if (dctx==NULL) return 0; /* support free on NULL */
+ RETURN_ERROR_IF(dctx->staticSize, memory_allocation, "not compatible with static DCtx");
+ { ZSTD_customMem const cMem = dctx->customMem;
+ ZSTD_clearDict(dctx);
+ ZSTD_customFree(dctx->inBuff, cMem);
+ dctx->inBuff = NULL;
+ if (dctx->ddictSet) {
+ ZSTD_freeDDictHashSet(dctx->ddictSet, cMem);
+ dctx->ddictSet = NULL;
+ }
+ ZSTD_customFree(dctx, cMem);
+ return 0;
+ }
+}
+
+/* no longer useful */
+void ZSTD_copyDCtx(ZSTD_DCtx* dstDCtx, const ZSTD_DCtx* srcDCtx)
+{
+ size_t const toCopy = (size_t)((char*)(&dstDCtx->inBuff) - (char*)dstDCtx);
+ ZSTD_memcpy(dstDCtx, srcDCtx, toCopy); /* no need to copy workspace */
+}
+
+/* Given a dctx with a digested frame params, re-selects the correct ZSTD_DDict based on
+ * the requested dict ID from the frame. If there exists a reference to the correct ZSTD_DDict, then
+ * accordingly sets the ddict to be used to decompress the frame.
+ *
+ * If no DDict is found, then no action is taken, and the ZSTD_DCtx::ddict remains as-is.
+ *
+ * ZSTD_d_refMultipleDDicts must be enabled for this function to be called.
+ */
+static void ZSTD_DCtx_selectFrameDDict(ZSTD_DCtx* dctx) {
+ assert(dctx->refMultipleDDicts && dctx->ddictSet);
+ DEBUGLOG(4, "Adjusting DDict based on requested dict ID from frame");
+ if (dctx->ddict) {
+ const ZSTD_DDict* frameDDict = ZSTD_DDictHashSet_getDDict(dctx->ddictSet, dctx->fParams.dictID);
+ if (frameDDict) {
+ DEBUGLOG(4, "DDict found!");
+ ZSTD_clearDict(dctx);
+ dctx->dictID = dctx->fParams.dictID;
+ dctx->ddict = frameDDict;
+ dctx->dictUses = ZSTD_use_indefinitely;
+ }
+ }
+}
+
+
+/*-*************************************************************
+ * Frame header decoding
+ ***************************************************************/
+
+/*! ZSTD_isFrame() :
+ * Tells if the content of `buffer` starts with a valid Frame Identifier.
+ * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
+ * Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
+ * Note 3 : Skippable Frame Identifiers are considered valid. */
+unsigned ZSTD_isFrame(const void* buffer, size_t size)
+{
+ if (size < ZSTD_FRAMEIDSIZE) return 0;
+ { U32 const magic = MEM_readLE32(buffer);
+ if (magic == ZSTD_MAGICNUMBER) return 1;
+ if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1;
+ }
+ return 0;
+}
+
+/* ZSTD_frameHeaderSize_internal() :
+ * srcSize must be large enough to reach header size fields.
+ * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless.
+ * @return : size of the Frame Header
+ * or an error code, which can be tested with ZSTD_isError() */
+static size_t ZSTD_frameHeaderSize_internal(const void* src, size_t srcSize, ZSTD_format_e format)
+{
+ size_t const minInputSize = ZSTD_startingInputLength(format);
+ RETURN_ERROR_IF(srcSize < minInputSize, srcSize_wrong, "");
+
+ { BYTE const fhd = ((const BYTE*)src)[minInputSize-1];
+ U32 const dictID= fhd & 3;
+ U32 const singleSegment = (fhd >> 5) & 1;
+ U32 const fcsId = fhd >> 6;
+ return minInputSize + !singleSegment
+ + ZSTD_did_fieldSize[dictID] + ZSTD_fcs_fieldSize[fcsId]
+ + (singleSegment && !fcsId);
+ }
+}
+
+/* ZSTD_frameHeaderSize() :
+ * srcSize must be >= ZSTD_frameHeaderSize_prefix.
+ * @return : size of the Frame Header,
+ * or an error code (if srcSize is too small) */
+size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize)
+{
+ return ZSTD_frameHeaderSize_internal(src, srcSize, ZSTD_f_zstd1);
+}
+
+
+/* ZSTD_getFrameHeader_advanced() :
+ * decode Frame Header, or require larger `srcSize`.
+ * note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless
+ * @return : 0, `zfhPtr` is correctly filled,
+ * >0, `srcSize` is too small, value is wanted `srcSize` amount,
+ * or an error code, which can be tested using ZSTD_isError() */
+size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format)
+{
+ const BYTE* ip = (const BYTE*)src;
+ size_t const minInputSize = ZSTD_startingInputLength(format);
+
+ ZSTD_memset(zfhPtr, 0, sizeof(*zfhPtr)); /* not strictly necessary, but static analyzer do not understand that zfhPtr is only going to be read only if return value is zero, since they are 2 different signals */
+ if (srcSize < minInputSize) return minInputSize;
+ RETURN_ERROR_IF(src==NULL, GENERIC, "invalid parameter");
+
+ if ( (format != ZSTD_f_zstd1_magicless)
+ && (MEM_readLE32(src) != ZSTD_MAGICNUMBER) ) {
+ if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
+ /* skippable frame */
+ if (srcSize < ZSTD_SKIPPABLEHEADERSIZE)
+ return ZSTD_SKIPPABLEHEADERSIZE; /* magic number + frame length */
+ ZSTD_memset(zfhPtr, 0, sizeof(*zfhPtr));
+ zfhPtr->frameContentSize = MEM_readLE32((const char *)src + ZSTD_FRAMEIDSIZE);
+ zfhPtr->frameType = ZSTD_skippableFrame;
+ return 0;
+ }
+ RETURN_ERROR(prefix_unknown, "");
+ }
+
+ /* ensure there is enough `srcSize` to fully read/decode frame header */
+ { size_t const fhsize = ZSTD_frameHeaderSize_internal(src, srcSize, format);
+ if (srcSize < fhsize) return fhsize;
+ zfhPtr->headerSize = (U32)fhsize;
+ }
+
+ { BYTE const fhdByte = ip[minInputSize-1];
+ size_t pos = minInputSize;
+ U32 const dictIDSizeCode = fhdByte&3;
+ U32 const checksumFlag = (fhdByte>>2)&1;
+ U32 const singleSegment = (fhdByte>>5)&1;
+ U32 const fcsID = fhdByte>>6;
+ U64 windowSize = 0;
+ U32 dictID = 0;
+ U64 frameContentSize = ZSTD_CONTENTSIZE_UNKNOWN;
+ RETURN_ERROR_IF((fhdByte & 0x08) != 0, frameParameter_unsupported,
+ "reserved bits, must be zero");
+
+ if (!singleSegment) {
+ BYTE const wlByte = ip[pos++];
+ U32 const windowLog = (wlByte >> 3) + ZSTD_WINDOWLOG_ABSOLUTEMIN;
+ RETURN_ERROR_IF(windowLog > ZSTD_WINDOWLOG_MAX, frameParameter_windowTooLarge, "");
+ windowSize = (1ULL << windowLog);
+ windowSize += (windowSize >> 3) * (wlByte&7);
+ }
+ switch(dictIDSizeCode)
+ {
+ default:
+ assert(0); /* impossible */
+ ZSTD_FALLTHROUGH;
+ case 0 : break;
+ case 1 : dictID = ip[pos]; pos++; break;
+ case 2 : dictID = MEM_readLE16(ip+pos); pos+=2; break;
+ case 3 : dictID = MEM_readLE32(ip+pos); pos+=4; break;
+ }
+ switch(fcsID)
+ {
+ default:
+ assert(0); /* impossible */
+ ZSTD_FALLTHROUGH;
+ case 0 : if (singleSegment) frameContentSize = ip[pos]; break;
+ case 1 : frameContentSize = MEM_readLE16(ip+pos)+256; break;
+ case 2 : frameContentSize = MEM_readLE32(ip+pos); break;
+ case 3 : frameContentSize = MEM_readLE64(ip+pos); break;
+ }
+ if (singleSegment) windowSize = frameContentSize;
+
+ zfhPtr->frameType = ZSTD_frame;
+ zfhPtr->frameContentSize = frameContentSize;
+ zfhPtr->windowSize = windowSize;
+ zfhPtr->blockSizeMax = (unsigned) MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
+ zfhPtr->dictID = dictID;
+ zfhPtr->checksumFlag = checksumFlag;
+ }
+ return 0;
+}
+
+/* ZSTD_getFrameHeader() :
+ * decode Frame Header, or require larger `srcSize`.
+ * note : this function does not consume input, it only reads it.
+ * @return : 0, `zfhPtr` is correctly filled,
+ * >0, `srcSize` is too small, value is wanted `srcSize` amount,
+ * or an error code, which can be tested using ZSTD_isError() */
+size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize)
+{
+ return ZSTD_getFrameHeader_advanced(zfhPtr, src, srcSize, ZSTD_f_zstd1);
+}
+
+
+/* ZSTD_getFrameContentSize() :
+ * compatible with legacy mode
+ * @return : decompressed size of the single frame pointed to be `src` if known, otherwise
+ * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
+ * - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too small) */
+unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize)
+{
+ { ZSTD_frameHeader zfh;
+ if (ZSTD_getFrameHeader(&zfh, src, srcSize) != 0)
+ return ZSTD_CONTENTSIZE_ERROR;
+ if (zfh.frameType == ZSTD_skippableFrame) {
+ return 0;
+ } else {
+ return zfh.frameContentSize;
+ } }
+}
+
+static size_t readSkippableFrameSize(void const* src, size_t srcSize)
+{
+ size_t const skippableHeaderSize = ZSTD_SKIPPABLEHEADERSIZE;
+ U32 sizeU32;
+
+ RETURN_ERROR_IF(srcSize < ZSTD_SKIPPABLEHEADERSIZE, srcSize_wrong, "");
+
+ sizeU32 = MEM_readLE32((BYTE const*)src + ZSTD_FRAMEIDSIZE);
+ RETURN_ERROR_IF((U32)(sizeU32 + ZSTD_SKIPPABLEHEADERSIZE) < sizeU32,
+ frameParameter_unsupported, "");
+ {
+ size_t const skippableSize = skippableHeaderSize + sizeU32;
+ RETURN_ERROR_IF(skippableSize > srcSize, srcSize_wrong, "");
+ return skippableSize;
+ }
+}
+
+/* ZSTD_findDecompressedSize() :
+ * compatible with legacy mode
+ * `srcSize` must be the exact length of some number of ZSTD compressed and/or
+ * skippable frames
+ * @return : decompressed size of the frames contained */
+unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize)
+{
+ unsigned long long totalDstSize = 0;
+
+ while (srcSize >= ZSTD_startingInputLength(ZSTD_f_zstd1)) {
+ U32 const magicNumber = MEM_readLE32(src);
+
+ if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
+ size_t const skippableSize = readSkippableFrameSize(src, srcSize);
+ if (ZSTD_isError(skippableSize)) {
+ return ZSTD_CONTENTSIZE_ERROR;
+ }
+ assert(skippableSize <= srcSize);
+
+ src = (const BYTE *)src + skippableSize;
+ srcSize -= skippableSize;
+ continue;
+ }
+
+ { unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
+ if (ret >= ZSTD_CONTENTSIZE_ERROR) return ret;
+
+ /* check for overflow */
+ if (totalDstSize + ret < totalDstSize) return ZSTD_CONTENTSIZE_ERROR;
+ totalDstSize += ret;
+ }
+ { size_t const frameSrcSize = ZSTD_findFrameCompressedSize(src, srcSize);
+ if (ZSTD_isError(frameSrcSize)) {
+ return ZSTD_CONTENTSIZE_ERROR;
+ }
+
+ src = (const BYTE *)src + frameSrcSize;
+ srcSize -= frameSrcSize;
+ }
+ } /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */
+
+ if (srcSize) return ZSTD_CONTENTSIZE_ERROR;
+
+ return totalDstSize;
+}
+
+/* ZSTD_getDecompressedSize() :
+ * compatible with legacy mode
+ * @return : decompressed size if known, 0 otherwise
+ note : 0 can mean any of the following :
+ - frame content is empty
+ - decompressed size field is not present in frame header
+ - frame header unknown / not supported
+ - frame header not complete (`srcSize` too small) */
+unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize)
+{
+ unsigned long long const ret = ZSTD_getFrameContentSize(src, srcSize);
+ ZSTD_STATIC_ASSERT(ZSTD_CONTENTSIZE_ERROR < ZSTD_CONTENTSIZE_UNKNOWN);
+ return (ret >= ZSTD_CONTENTSIZE_ERROR) ? 0 : ret;
+}
+
+
+/* ZSTD_decodeFrameHeader() :
+ * `headerSize` must be the size provided by ZSTD_frameHeaderSize().
+ * If multiple DDict references are enabled, also will choose the correct DDict to use.
+ * @return : 0 if success, or an error code, which can be tested using ZSTD_isError() */
+static size_t ZSTD_decodeFrameHeader(ZSTD_DCtx* dctx, const void* src, size_t headerSize)
+{
+ size_t const result = ZSTD_getFrameHeader_advanced(&(dctx->fParams), src, headerSize, dctx->format);
+ if (ZSTD_isError(result)) return result; /* invalid header */
+ RETURN_ERROR_IF(result>0, srcSize_wrong, "headerSize too small");
+
+ /* Reference DDict requested by frame if dctx references multiple ddicts */
+ if (dctx->refMultipleDDicts == ZSTD_rmd_refMultipleDDicts && dctx->ddictSet) {
+ ZSTD_DCtx_selectFrameDDict(dctx);
+ }
+
+#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ /* Skip the dictID check in fuzzing mode, because it makes the search
+ * harder.
+ */
+ RETURN_ERROR_IF(dctx->fParams.dictID && (dctx->dictID != dctx->fParams.dictID),
+ dictionary_wrong, "");
+#endif
+ dctx->validateChecksum = (dctx->fParams.checksumFlag && !dctx->forceIgnoreChecksum) ? 1 : 0;
+ if (dctx->validateChecksum) xxh64_reset(&dctx->xxhState, 0);
+ dctx->processedCSize += headerSize;
+ return 0;
+}
+
+static ZSTD_frameSizeInfo ZSTD_errorFrameSizeInfo(size_t ret)
+{
+ ZSTD_frameSizeInfo frameSizeInfo;
+ frameSizeInfo.compressedSize = ret;
+ frameSizeInfo.decompressedBound = ZSTD_CONTENTSIZE_ERROR;
+ return frameSizeInfo;
+}
+
+static ZSTD_frameSizeInfo ZSTD_findFrameSizeInfo(const void* src, size_t srcSize)
+{
+ ZSTD_frameSizeInfo frameSizeInfo;
+ ZSTD_memset(&frameSizeInfo, 0, sizeof(ZSTD_frameSizeInfo));
+
+
+ if ((srcSize >= ZSTD_SKIPPABLEHEADERSIZE)
+ && (MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
+ frameSizeInfo.compressedSize = readSkippableFrameSize(src, srcSize);
+ assert(ZSTD_isError(frameSizeInfo.compressedSize) ||
+ frameSizeInfo.compressedSize <= srcSize);
+ return frameSizeInfo;
+ } else {
+ const BYTE* ip = (const BYTE*)src;
+ const BYTE* const ipstart = ip;
+ size_t remainingSize = srcSize;
+ size_t nbBlocks = 0;
+ ZSTD_frameHeader zfh;
+
+ /* Extract Frame Header */
+ { size_t const ret = ZSTD_getFrameHeader(&zfh, src, srcSize);
+ if (ZSTD_isError(ret))
+ return ZSTD_errorFrameSizeInfo(ret);
+ if (ret > 0)
+ return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));
+ }
+
+ ip += zfh.headerSize;
+ remainingSize -= zfh.headerSize;
+
+ /* Iterate over each block */
+ while (1) {
+ blockProperties_t blockProperties;
+ size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSize, &blockProperties);
+ if (ZSTD_isError(cBlockSize))
+ return ZSTD_errorFrameSizeInfo(cBlockSize);
+
+ if (ZSTD_blockHeaderSize + cBlockSize > remainingSize)
+ return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));
+
+ ip += ZSTD_blockHeaderSize + cBlockSize;
+ remainingSize -= ZSTD_blockHeaderSize + cBlockSize;
+ nbBlocks++;
+
+ if (blockProperties.lastBlock) break;
+ }
+
+ /* Final frame content checksum */
+ if (zfh.checksumFlag) {
+ if (remainingSize < 4)
+ return ZSTD_errorFrameSizeInfo(ERROR(srcSize_wrong));
+ ip += 4;
+ }
+
+ frameSizeInfo.compressedSize = (size_t)(ip - ipstart);
+ frameSizeInfo.decompressedBound = (zfh.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN)
+ ? zfh.frameContentSize
+ : nbBlocks * zfh.blockSizeMax;
+ return frameSizeInfo;
+ }
+}
+
+/* ZSTD_findFrameCompressedSize() :
+ * compatible with legacy mode
+ * `src` must point to the start of a ZSTD frame, ZSTD legacy frame, or skippable frame
+ * `srcSize` must be at least as large as the frame contained
+ * @return : the compressed size of the frame starting at `src` */
+size_t ZSTD_findFrameCompressedSize(const void *src, size_t srcSize)
+{
+ ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize);
+ return frameSizeInfo.compressedSize;
+}
+
+/* ZSTD_decompressBound() :
+ * compatible with legacy mode
+ * `src` must point to the start of a ZSTD frame or a skippeable frame
+ * `srcSize` must be at least as large as the frame contained
+ * @return : the maximum decompressed size of the compressed source
+ */
+unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize)
+{
+ unsigned long long bound = 0;
+ /* Iterate over each frame */
+ while (srcSize > 0) {
+ ZSTD_frameSizeInfo const frameSizeInfo = ZSTD_findFrameSizeInfo(src, srcSize);
+ size_t const compressedSize = frameSizeInfo.compressedSize;
+ unsigned long long const decompressedBound = frameSizeInfo.decompressedBound;
+ if (ZSTD_isError(compressedSize) || decompressedBound == ZSTD_CONTENTSIZE_ERROR)
+ return ZSTD_CONTENTSIZE_ERROR;
+ assert(srcSize >= compressedSize);
+ src = (const BYTE*)src + compressedSize;
+ srcSize -= compressedSize;
+ bound += decompressedBound;
+ }
+ return bound;
+}
+
+
+/*-*************************************************************
+ * Frame decoding
+ ***************************************************************/
+
+/* ZSTD_insertBlock() :
+ * insert `src` block into `dctx` history. Useful to track uncompressed blocks. */
+size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize)
+{
+ DEBUGLOG(5, "ZSTD_insertBlock: %u bytes", (unsigned)blockSize);
+ ZSTD_checkContinuity(dctx, blockStart, blockSize);
+ dctx->previousDstEnd = (const char*)blockStart + blockSize;
+ return blockSize;
+}
+
+
+static size_t ZSTD_copyRawBlock(void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_copyRawBlock");
+ RETURN_ERROR_IF(srcSize > dstCapacity, dstSize_tooSmall, "");
+ if (dst == NULL) {
+ if (srcSize == 0) return 0;
+ RETURN_ERROR(dstBuffer_null, "");
+ }
+ ZSTD_memcpy(dst, src, srcSize);
+ return srcSize;
+}
+
+static size_t ZSTD_setRleBlock(void* dst, size_t dstCapacity,
+ BYTE b,
+ size_t regenSize)
+{
+ RETURN_ERROR_IF(regenSize > dstCapacity, dstSize_tooSmall, "");
+ if (dst == NULL) {
+ if (regenSize == 0) return 0;
+ RETURN_ERROR(dstBuffer_null, "");
+ }
+ ZSTD_memset(dst, b, regenSize);
+ return regenSize;
+}
+
+static void ZSTD_DCtx_trace_end(ZSTD_DCtx const* dctx, U64 uncompressedSize, U64 compressedSize, unsigned streaming)
+{
+ (void)dctx;
+ (void)uncompressedSize;
+ (void)compressedSize;
+ (void)streaming;
+}
+
+
+/*! ZSTD_decompressFrame() :
+ * @dctx must be properly initialized
+ * will update *srcPtr and *srcSizePtr,
+ * to make *srcPtr progress by one frame. */
+static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void** srcPtr, size_t *srcSizePtr)
+{
+ const BYTE* const istart = (const BYTE*)(*srcPtr);
+ const BYTE* ip = istart;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = dstCapacity != 0 ? ostart + dstCapacity : ostart;
+ BYTE* op = ostart;
+ size_t remainingSrcSize = *srcSizePtr;
+
+ DEBUGLOG(4, "ZSTD_decompressFrame (srcSize:%i)", (int)*srcSizePtr);
+
+ /* check */
+ RETURN_ERROR_IF(
+ remainingSrcSize < ZSTD_FRAMEHEADERSIZE_MIN(dctx->format)+ZSTD_blockHeaderSize,
+ srcSize_wrong, "");
+
+ /* Frame Header */
+ { size_t const frameHeaderSize = ZSTD_frameHeaderSize_internal(
+ ip, ZSTD_FRAMEHEADERSIZE_PREFIX(dctx->format), dctx->format);
+ if (ZSTD_isError(frameHeaderSize)) return frameHeaderSize;
+ RETURN_ERROR_IF(remainingSrcSize < frameHeaderSize+ZSTD_blockHeaderSize,
+ srcSize_wrong, "");
+ FORWARD_IF_ERROR( ZSTD_decodeFrameHeader(dctx, ip, frameHeaderSize) , "");
+ ip += frameHeaderSize; remainingSrcSize -= frameHeaderSize;
+ }
+
+ /* Loop on each block */
+ while (1) {
+ size_t decodedSize;
+ blockProperties_t blockProperties;
+ size_t const cBlockSize = ZSTD_getcBlockSize(ip, remainingSrcSize, &blockProperties);
+ if (ZSTD_isError(cBlockSize)) return cBlockSize;
+
+ ip += ZSTD_blockHeaderSize;
+ remainingSrcSize -= ZSTD_blockHeaderSize;
+ RETURN_ERROR_IF(cBlockSize > remainingSrcSize, srcSize_wrong, "");
+
+ switch(blockProperties.blockType)
+ {
+ case bt_compressed:
+ decodedSize = ZSTD_decompressBlock_internal(dctx, op, (size_t)(oend-op), ip, cBlockSize, /* frame */ 1);
+ break;
+ case bt_raw :
+ decodedSize = ZSTD_copyRawBlock(op, (size_t)(oend-op), ip, cBlockSize);
+ break;
+ case bt_rle :
+ decodedSize = ZSTD_setRleBlock(op, (size_t)(oend-op), *ip, blockProperties.origSize);
+ break;
+ case bt_reserved :
+ default:
+ RETURN_ERROR(corruption_detected, "invalid block type");
+ }
+
+ if (ZSTD_isError(decodedSize)) return decodedSize;
+ if (dctx->validateChecksum)
+ xxh64_update(&dctx->xxhState, op, decodedSize);
+ if (decodedSize != 0)
+ op += decodedSize;
+ assert(ip != NULL);
+ ip += cBlockSize;
+ remainingSrcSize -= cBlockSize;
+ if (blockProperties.lastBlock) break;
+ }
+
+ if (dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN) {
+ RETURN_ERROR_IF((U64)(op-ostart) != dctx->fParams.frameContentSize,
+ corruption_detected, "");
+ }
+ if (dctx->fParams.checksumFlag) { /* Frame content checksum verification */
+ RETURN_ERROR_IF(remainingSrcSize<4, checksum_wrong, "");
+ if (!dctx->forceIgnoreChecksum) {
+ U32 const checkCalc = (U32)xxh64_digest(&dctx->xxhState);
+ U32 checkRead;
+ checkRead = MEM_readLE32(ip);
+ RETURN_ERROR_IF(checkRead != checkCalc, checksum_wrong, "");
+ }
+ ip += 4;
+ remainingSrcSize -= 4;
+ }
+ ZSTD_DCtx_trace_end(dctx, (U64)(op-ostart), (U64)(ip-istart), /* streaming */ 0);
+ /* Allow caller to get size read */
+ *srcPtr = ip;
+ *srcSizePtr = remainingSrcSize;
+ return (size_t)(op-ostart);
+}
+
+static size_t ZSTD_decompressMultiFrame(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict, size_t dictSize,
+ const ZSTD_DDict* ddict)
+{
+ void* const dststart = dst;
+ int moreThan1Frame = 0;
+
+ DEBUGLOG(5, "ZSTD_decompressMultiFrame");
+ assert(dict==NULL || ddict==NULL); /* either dict or ddict set, not both */
+
+ if (ddict) {
+ dict = ZSTD_DDict_dictContent(ddict);
+ dictSize = ZSTD_DDict_dictSize(ddict);
+ }
+
+ while (srcSize >= ZSTD_startingInputLength(dctx->format)) {
+
+
+ { U32 const magicNumber = MEM_readLE32(src);
+ DEBUGLOG(4, "reading magic number %08X (expecting %08X)",
+ (unsigned)magicNumber, ZSTD_MAGICNUMBER);
+ if ((magicNumber & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) {
+ size_t const skippableSize = readSkippableFrameSize(src, srcSize);
+ FORWARD_IF_ERROR(skippableSize, "readSkippableFrameSize failed");
+ assert(skippableSize <= srcSize);
+
+ src = (const BYTE *)src + skippableSize;
+ srcSize -= skippableSize;
+ continue;
+ } }
+
+ if (ddict) {
+ /* we were called from ZSTD_decompress_usingDDict */
+ FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(dctx, ddict), "");
+ } else {
+ /* this will initialize correctly with no dict if dict == NULL, so
+ * use this in all cases but ddict */
+ FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDict(dctx, dict, dictSize), "");
+ }
+ ZSTD_checkContinuity(dctx, dst, dstCapacity);
+
+ { const size_t res = ZSTD_decompressFrame(dctx, dst, dstCapacity,
+ &src, &srcSize);
+ RETURN_ERROR_IF(
+ (ZSTD_getErrorCode(res) == ZSTD_error_prefix_unknown)
+ && (moreThan1Frame==1),
+ srcSize_wrong,
+ "At least one frame successfully completed, "
+ "but following bytes are garbage: "
+ "it's more likely to be a srcSize error, "
+ "specifying more input bytes than size of frame(s). "
+ "Note: one could be unlucky, it might be a corruption error instead, "
+ "happening right at the place where we expect zstd magic bytes. "
+ "But this is _much_ less likely than a srcSize field error.");
+ if (ZSTD_isError(res)) return res;
+ assert(res <= dstCapacity);
+ if (res != 0)
+ dst = (BYTE*)dst + res;
+ dstCapacity -= res;
+ }
+ moreThan1Frame = 1;
+ } /* while (srcSize >= ZSTD_frameHeaderSize_prefix) */
+
+ RETURN_ERROR_IF(srcSize, srcSize_wrong, "input not entirely consumed");
+
+ return (size_t)((BYTE*)dst - (BYTE*)dststart);
+}
+
+size_t ZSTD_decompress_usingDict(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const void* dict, size_t dictSize)
+{
+ return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize, dict, dictSize, NULL);
+}
+
+
+static ZSTD_DDict const* ZSTD_getDDict(ZSTD_DCtx* dctx)
+{
+ switch (dctx->dictUses) {
+ default:
+ assert(0 /* Impossible */);
+ ZSTD_FALLTHROUGH;
+ case ZSTD_dont_use:
+ ZSTD_clearDict(dctx);
+ return NULL;
+ case ZSTD_use_indefinitely:
+ return dctx->ddict;
+ case ZSTD_use_once:
+ dctx->dictUses = ZSTD_dont_use;
+ return dctx->ddict;
+ }
+}
+
+size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ return ZSTD_decompress_usingDDict(dctx, dst, dstCapacity, src, srcSize, ZSTD_getDDict(dctx));
+}
+
+
+size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+#if defined(ZSTD_HEAPMODE) && (ZSTD_HEAPMODE>=1)
+ size_t regenSize;
+ ZSTD_DCtx* const dctx = ZSTD_createDCtx();
+ RETURN_ERROR_IF(dctx==NULL, memory_allocation, "NULL pointer!");
+ regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize);
+ ZSTD_freeDCtx(dctx);
+ return regenSize;
+#else /* stack mode */
+ ZSTD_DCtx dctx;
+ ZSTD_initDCtx_internal(&dctx);
+ return ZSTD_decompressDCtx(&dctx, dst, dstCapacity, src, srcSize);
+#endif
+}
+
+
+/*-**************************************
+* Advanced Streaming Decompression API
+* Bufferless and synchronous
+****************************************/
+size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; }
+
+/*
+ * Similar to ZSTD_nextSrcSizeToDecompress(), but when when a block input can be streamed,
+ * we allow taking a partial block as the input. Currently only raw uncompressed blocks can
+ * be streamed.
+ *
+ * For blocks that can be streamed, this allows us to reduce the latency until we produce
+ * output, and avoid copying the input.
+ *
+ * @param inputSize - The total amount of input that the caller currently has.
+ */
+static size_t ZSTD_nextSrcSizeToDecompressWithInputSize(ZSTD_DCtx* dctx, size_t inputSize) {
+ if (!(dctx->stage == ZSTDds_decompressBlock || dctx->stage == ZSTDds_decompressLastBlock))
+ return dctx->expected;
+ if (dctx->bType != bt_raw)
+ return dctx->expected;
+ return MIN(MAX(inputSize, 1), dctx->expected);
+}
+
+ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx) {
+ switch(dctx->stage)
+ {
+ default: /* should not happen */
+ assert(0);
+ ZSTD_FALLTHROUGH;
+ case ZSTDds_getFrameHeaderSize:
+ ZSTD_FALLTHROUGH;
+ case ZSTDds_decodeFrameHeader:
+ return ZSTDnit_frameHeader;
+ case ZSTDds_decodeBlockHeader:
+ return ZSTDnit_blockHeader;
+ case ZSTDds_decompressBlock:
+ return ZSTDnit_block;
+ case ZSTDds_decompressLastBlock:
+ return ZSTDnit_lastBlock;
+ case ZSTDds_checkChecksum:
+ return ZSTDnit_checksum;
+ case ZSTDds_decodeSkippableHeader:
+ ZSTD_FALLTHROUGH;
+ case ZSTDds_skipFrame:
+ return ZSTDnit_skippableFrame;
+ }
+}
+
+static int ZSTD_isSkipFrame(ZSTD_DCtx* dctx) { return dctx->stage == ZSTDds_skipFrame; }
+
+/* ZSTD_decompressContinue() :
+ * srcSize : must be the exact nb of bytes expected (see ZSTD_nextSrcSizeToDecompress())
+ * @return : nb of bytes generated into `dst` (necessarily <= `dstCapacity)
+ * or an error code, which can be tested using ZSTD_isError() */
+size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)
+{
+ DEBUGLOG(5, "ZSTD_decompressContinue (srcSize:%u)", (unsigned)srcSize);
+ /* Sanity check */
+ RETURN_ERROR_IF(srcSize != ZSTD_nextSrcSizeToDecompressWithInputSize(dctx, srcSize), srcSize_wrong, "not allowed");
+ ZSTD_checkContinuity(dctx, dst, dstCapacity);
+
+ dctx->processedCSize += srcSize;
+
+ switch (dctx->stage)
+ {
+ case ZSTDds_getFrameHeaderSize :
+ assert(src != NULL);
+ if (dctx->format == ZSTD_f_zstd1) { /* allows header */
+ assert(srcSize >= ZSTD_FRAMEIDSIZE); /* to read skippable magic number */
+ if ((MEM_readLE32(src) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
+ ZSTD_memcpy(dctx->headerBuffer, src, srcSize);
+ dctx->expected = ZSTD_SKIPPABLEHEADERSIZE - srcSize; /* remaining to load to get full skippable frame header */
+ dctx->stage = ZSTDds_decodeSkippableHeader;
+ return 0;
+ } }
+ dctx->headerSize = ZSTD_frameHeaderSize_internal(src, srcSize, dctx->format);
+ if (ZSTD_isError(dctx->headerSize)) return dctx->headerSize;
+ ZSTD_memcpy(dctx->headerBuffer, src, srcSize);
+ dctx->expected = dctx->headerSize - srcSize;
+ dctx->stage = ZSTDds_decodeFrameHeader;
+ return 0;
+
+ case ZSTDds_decodeFrameHeader:
+ assert(src != NULL);
+ ZSTD_memcpy(dctx->headerBuffer + (dctx->headerSize - srcSize), src, srcSize);
+ FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(dctx, dctx->headerBuffer, dctx->headerSize), "");
+ dctx->expected = ZSTD_blockHeaderSize;
+ dctx->stage = ZSTDds_decodeBlockHeader;
+ return 0;
+
+ case ZSTDds_decodeBlockHeader:
+ { blockProperties_t bp;
+ size_t const cBlockSize = ZSTD_getcBlockSize(src, ZSTD_blockHeaderSize, &bp);
+ if (ZSTD_isError(cBlockSize)) return cBlockSize;
+ RETURN_ERROR_IF(cBlockSize > dctx->fParams.blockSizeMax, corruption_detected, "Block Size Exceeds Maximum");
+ dctx->expected = cBlockSize;
+ dctx->bType = bp.blockType;
+ dctx->rleSize = bp.origSize;
+ if (cBlockSize) {
+ dctx->stage = bp.lastBlock ? ZSTDds_decompressLastBlock : ZSTDds_decompressBlock;
+ return 0;
+ }
+ /* empty block */
+ if (bp.lastBlock) {
+ if (dctx->fParams.checksumFlag) {
+ dctx->expected = 4;
+ dctx->stage = ZSTDds_checkChecksum;
+ } else {
+ dctx->expected = 0; /* end of frame */
+ dctx->stage = ZSTDds_getFrameHeaderSize;
+ }
+ } else {
+ dctx->expected = ZSTD_blockHeaderSize; /* jump to next header */
+ dctx->stage = ZSTDds_decodeBlockHeader;
+ }
+ return 0;
+ }
+
+ case ZSTDds_decompressLastBlock:
+ case ZSTDds_decompressBlock:
+ DEBUGLOG(5, "ZSTD_decompressContinue: case ZSTDds_decompressBlock");
+ { size_t rSize;
+ switch(dctx->bType)
+ {
+ case bt_compressed:
+ DEBUGLOG(5, "ZSTD_decompressContinue: case bt_compressed");
+ rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1);
+ dctx->expected = 0; /* Streaming not supported */
+ break;
+ case bt_raw :
+ assert(srcSize <= dctx->expected);
+ rSize = ZSTD_copyRawBlock(dst, dstCapacity, src, srcSize);
+ FORWARD_IF_ERROR(rSize, "ZSTD_copyRawBlock failed");
+ assert(rSize == srcSize);
+ dctx->expected -= rSize;
+ break;
+ case bt_rle :
+ rSize = ZSTD_setRleBlock(dst, dstCapacity, *(const BYTE*)src, dctx->rleSize);
+ dctx->expected = 0; /* Streaming not supported */
+ break;
+ case bt_reserved : /* should never happen */
+ default:
+ RETURN_ERROR(corruption_detected, "invalid block type");
+ }
+ FORWARD_IF_ERROR(rSize, "");
+ RETURN_ERROR_IF(rSize > dctx->fParams.blockSizeMax, corruption_detected, "Decompressed Block Size Exceeds Maximum");
+ DEBUGLOG(5, "ZSTD_decompressContinue: decoded size from block : %u", (unsigned)rSize);
+ dctx->decodedSize += rSize;
+ if (dctx->validateChecksum) xxh64_update(&dctx->xxhState, dst, rSize);
+ dctx->previousDstEnd = (char*)dst + rSize;
+
+ /* Stay on the same stage until we are finished streaming the block. */
+ if (dctx->expected > 0) {
+ return rSize;
+ }
+
+ if (dctx->stage == ZSTDds_decompressLastBlock) { /* end of frame */
+ DEBUGLOG(4, "ZSTD_decompressContinue: decoded size from frame : %u", (unsigned)dctx->decodedSize);
+ RETURN_ERROR_IF(
+ dctx->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
+ && dctx->decodedSize != dctx->fParams.frameContentSize,
+ corruption_detected, "");
+ if (dctx->fParams.checksumFlag) { /* another round for frame checksum */
+ dctx->expected = 4;
+ dctx->stage = ZSTDds_checkChecksum;
+ } else {
+ ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, /* streaming */ 1);
+ dctx->expected = 0; /* ends here */
+ dctx->stage = ZSTDds_getFrameHeaderSize;
+ }
+ } else {
+ dctx->stage = ZSTDds_decodeBlockHeader;
+ dctx->expected = ZSTD_blockHeaderSize;
+ }
+ return rSize;
+ }
+
+ case ZSTDds_checkChecksum:
+ assert(srcSize == 4); /* guaranteed by dctx->expected */
+ {
+ if (dctx->validateChecksum) {
+ U32 const h32 = (U32)xxh64_digest(&dctx->xxhState);
+ U32 const check32 = MEM_readLE32(src);
+ DEBUGLOG(4, "ZSTD_decompressContinue: checksum : calculated %08X :: %08X read", (unsigned)h32, (unsigned)check32);
+ RETURN_ERROR_IF(check32 != h32, checksum_wrong, "");
+ }
+ ZSTD_DCtx_trace_end(dctx, dctx->decodedSize, dctx->processedCSize, /* streaming */ 1);
+ dctx->expected = 0;
+ dctx->stage = ZSTDds_getFrameHeaderSize;
+ return 0;
+ }
+
+ case ZSTDds_decodeSkippableHeader:
+ assert(src != NULL);
+ assert(srcSize <= ZSTD_SKIPPABLEHEADERSIZE);
+ ZSTD_memcpy(dctx->headerBuffer + (ZSTD_SKIPPABLEHEADERSIZE - srcSize), src, srcSize); /* complete skippable header */
+ dctx->expected = MEM_readLE32(dctx->headerBuffer + ZSTD_FRAMEIDSIZE); /* note : dctx->expected can grow seriously large, beyond local buffer size */
+ dctx->stage = ZSTDds_skipFrame;
+ return 0;
+
+ case ZSTDds_skipFrame:
+ dctx->expected = 0;
+ dctx->stage = ZSTDds_getFrameHeaderSize;
+ return 0;
+
+ default:
+ assert(0); /* impossible */
+ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
+ }
+}
+
+
+static size_t ZSTD_refDictContent(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
+{
+ dctx->dictEnd = dctx->previousDstEnd;
+ dctx->virtualStart = (const char*)dict - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
+ dctx->prefixStart = dict;
+ dctx->previousDstEnd = (const char*)dict + dictSize;
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ dctx->dictContentBeginForFuzzing = dctx->prefixStart;
+ dctx->dictContentEndForFuzzing = dctx->previousDstEnd;
+#endif
+ return 0;
+}
+
+/*! ZSTD_loadDEntropy() :
+ * dict : must point at beginning of a valid zstd dictionary.
+ * @return : size of entropy tables read */
+size_t
+ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
+ const void* const dict, size_t const dictSize)
+{
+ const BYTE* dictPtr = (const BYTE*)dict;
+ const BYTE* const dictEnd = dictPtr + dictSize;
+
+ RETURN_ERROR_IF(dictSize <= 8, dictionary_corrupted, "dict is too small");
+ assert(MEM_readLE32(dict) == ZSTD_MAGIC_DICTIONARY); /* dict must be valid */
+ dictPtr += 8; /* skip header = magic + dictID */
+
+ ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, OFTable) == offsetof(ZSTD_entropyDTables_t, LLTable) + sizeof(entropy->LLTable));
+ ZSTD_STATIC_ASSERT(offsetof(ZSTD_entropyDTables_t, MLTable) == offsetof(ZSTD_entropyDTables_t, OFTable) + sizeof(entropy->OFTable));
+ ZSTD_STATIC_ASSERT(sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable) >= HUF_DECOMPRESS_WORKSPACE_SIZE);
+ { void* const workspace = &entropy->LLTable; /* use fse tables as temporary workspace; implies fse tables are grouped together */
+ size_t const workspaceSize = sizeof(entropy->LLTable) + sizeof(entropy->OFTable) + sizeof(entropy->MLTable);
+#ifdef HUF_FORCE_DECOMPRESS_X1
+ /* in minimal huffman, we always use X1 variants */
+ size_t const hSize = HUF_readDTableX1_wksp(entropy->hufTable,
+ dictPtr, dictEnd - dictPtr,
+ workspace, workspaceSize);
+#else
+ size_t const hSize = HUF_readDTableX2_wksp(entropy->hufTable,
+ dictPtr, (size_t)(dictEnd - dictPtr),
+ workspace, workspaceSize);
+#endif
+ RETURN_ERROR_IF(HUF_isError(hSize), dictionary_corrupted, "");
+ dictPtr += hSize;
+ }
+
+ { short offcodeNCount[MaxOff+1];
+ unsigned offcodeMaxValue = MaxOff, offcodeLog;
+ size_t const offcodeHeaderSize = FSE_readNCount(offcodeNCount, &offcodeMaxValue, &offcodeLog, dictPtr, (size_t)(dictEnd-dictPtr));
+ RETURN_ERROR_IF(FSE_isError(offcodeHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(offcodeMaxValue > MaxOff, dictionary_corrupted, "");
+ RETURN_ERROR_IF(offcodeLog > OffFSELog, dictionary_corrupted, "");
+ ZSTD_buildFSETable( entropy->OFTable,
+ offcodeNCount, offcodeMaxValue,
+ OF_base, OF_bits,
+ offcodeLog,
+ entropy->workspace, sizeof(entropy->workspace),
+ /* bmi2 */0);
+ dictPtr += offcodeHeaderSize;
+ }
+
+ { short matchlengthNCount[MaxML+1];
+ unsigned matchlengthMaxValue = MaxML, matchlengthLog;
+ size_t const matchlengthHeaderSize = FSE_readNCount(matchlengthNCount, &matchlengthMaxValue, &matchlengthLog, dictPtr, (size_t)(dictEnd-dictPtr));
+ RETURN_ERROR_IF(FSE_isError(matchlengthHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(matchlengthMaxValue > MaxML, dictionary_corrupted, "");
+ RETURN_ERROR_IF(matchlengthLog > MLFSELog, dictionary_corrupted, "");
+ ZSTD_buildFSETable( entropy->MLTable,
+ matchlengthNCount, matchlengthMaxValue,
+ ML_base, ML_bits,
+ matchlengthLog,
+ entropy->workspace, sizeof(entropy->workspace),
+ /* bmi2 */ 0);
+ dictPtr += matchlengthHeaderSize;
+ }
+
+ { short litlengthNCount[MaxLL+1];
+ unsigned litlengthMaxValue = MaxLL, litlengthLog;
+ size_t const litlengthHeaderSize = FSE_readNCount(litlengthNCount, &litlengthMaxValue, &litlengthLog, dictPtr, (size_t)(dictEnd-dictPtr));
+ RETURN_ERROR_IF(FSE_isError(litlengthHeaderSize), dictionary_corrupted, "");
+ RETURN_ERROR_IF(litlengthMaxValue > MaxLL, dictionary_corrupted, "");
+ RETURN_ERROR_IF(litlengthLog > LLFSELog, dictionary_corrupted, "");
+ ZSTD_buildFSETable( entropy->LLTable,
+ litlengthNCount, litlengthMaxValue,
+ LL_base, LL_bits,
+ litlengthLog,
+ entropy->workspace, sizeof(entropy->workspace),
+ /* bmi2 */ 0);
+ dictPtr += litlengthHeaderSize;
+ }
+
+ RETURN_ERROR_IF(dictPtr+12 > dictEnd, dictionary_corrupted, "");
+ { int i;
+ size_t const dictContentSize = (size_t)(dictEnd - (dictPtr+12));
+ for (i=0; i<3; i++) {
+ U32 const rep = MEM_readLE32(dictPtr); dictPtr += 4;
+ RETURN_ERROR_IF(rep==0 || rep > dictContentSize,
+ dictionary_corrupted, "");
+ entropy->rep[i] = rep;
+ } }
+
+ return (size_t)(dictPtr - (const BYTE*)dict);
+}
+
+static size_t ZSTD_decompress_insertDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
+{
+ if (dictSize < 8) return ZSTD_refDictContent(dctx, dict, dictSize);
+ { U32 const magic = MEM_readLE32(dict);
+ if (magic != ZSTD_MAGIC_DICTIONARY) {
+ return ZSTD_refDictContent(dctx, dict, dictSize); /* pure content mode */
+ } }
+ dctx->dictID = MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE);
+
+ /* load entropy tables */
+ { size_t const eSize = ZSTD_loadDEntropy(&dctx->entropy, dict, dictSize);
+ RETURN_ERROR_IF(ZSTD_isError(eSize), dictionary_corrupted, "");
+ dict = (const char*)dict + eSize;
+ dictSize -= eSize;
+ }
+ dctx->litEntropy = dctx->fseEntropy = 1;
+
+ /* reference dictionary content */
+ return ZSTD_refDictContent(dctx, dict, dictSize);
+}
+
+size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx)
+{
+ assert(dctx != NULL);
+ dctx->expected = ZSTD_startingInputLength(dctx->format); /* dctx->format must be properly set */
+ dctx->stage = ZSTDds_getFrameHeaderSize;
+ dctx->processedCSize = 0;
+ dctx->decodedSize = 0;
+ dctx->previousDstEnd = NULL;
+ dctx->prefixStart = NULL;
+ dctx->virtualStart = NULL;
+ dctx->dictEnd = NULL;
+ dctx->entropy.hufTable[0] = (HUF_DTable)((HufLog)*0x1000001); /* cover both little and big endian */
+ dctx->litEntropy = dctx->fseEntropy = 0;
+ dctx->dictID = 0;
+ dctx->bType = bt_reserved;
+ ZSTD_STATIC_ASSERT(sizeof(dctx->entropy.rep) == sizeof(repStartValue));
+ ZSTD_memcpy(dctx->entropy.rep, repStartValue, sizeof(repStartValue)); /* initial repcodes */
+ dctx->LLTptr = dctx->entropy.LLTable;
+ dctx->MLTptr = dctx->entropy.MLTable;
+ dctx->OFTptr = dctx->entropy.OFTable;
+ dctx->HUFptr = dctx->entropy.hufTable;
+ return 0;
+}
+
+size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
+{
+ FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
+ if (dict && dictSize)
+ RETURN_ERROR_IF(
+ ZSTD_isError(ZSTD_decompress_insertDictionary(dctx, dict, dictSize)),
+ dictionary_corrupted, "");
+ return 0;
+}
+
+
+/* ====== ZSTD_DDict ====== */
+
+size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
+{
+ DEBUGLOG(4, "ZSTD_decompressBegin_usingDDict");
+ assert(dctx != NULL);
+ if (ddict) {
+ const char* const dictStart = (const char*)ZSTD_DDict_dictContent(ddict);
+ size_t const dictSize = ZSTD_DDict_dictSize(ddict);
+ const void* const dictEnd = dictStart + dictSize;
+ dctx->ddictIsCold = (dctx->dictEnd != dictEnd);
+ DEBUGLOG(4, "DDict is %s",
+ dctx->ddictIsCold ? "~cold~" : "hot!");
+ }
+ FORWARD_IF_ERROR( ZSTD_decompressBegin(dctx) , "");
+ if (ddict) { /* NULL ddict is equivalent to no dictionary */
+ ZSTD_copyDDictParameters(dctx, ddict);
+ }
+ return 0;
+}
+
+/*! ZSTD_getDictID_fromDict() :
+ * Provides the dictID stored within dictionary.
+ * if @return == 0, the dictionary is not conformant with Zstandard specification.
+ * It can still be loaded, but as a content-only dictionary. */
+unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize)
+{
+ if (dictSize < 8) return 0;
+ if (MEM_readLE32(dict) != ZSTD_MAGIC_DICTIONARY) return 0;
+ return MEM_readLE32((const char*)dict + ZSTD_FRAMEIDSIZE);
+}
+
+/*! ZSTD_getDictID_fromFrame() :
+ * Provides the dictID required to decompress frame stored within `src`.
+ * If @return == 0, the dictID could not be decoded.
+ * This could for one of the following reasons :
+ * - The frame does not require a dictionary (most common case).
+ * - The frame was built with dictID intentionally removed.
+ * Needed dictionary is a hidden information.
+ * Note : this use case also happens when using a non-conformant dictionary.
+ * - `srcSize` is too small, and as a result, frame header could not be decoded.
+ * Note : possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`.
+ * - This is not a Zstandard frame.
+ * When identifying the exact failure cause, it's possible to use
+ * ZSTD_getFrameHeader(), which will provide a more precise error code. */
+unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize)
+{
+ ZSTD_frameHeader zfp = { 0, 0, 0, ZSTD_frame, 0, 0, 0 };
+ size_t const hError = ZSTD_getFrameHeader(&zfp, src, srcSize);
+ if (ZSTD_isError(hError)) return 0;
+ return zfp.dictID;
+}
+
+
+/*! ZSTD_decompress_usingDDict() :
+* Decompression using a pre-digested Dictionary
+* Use dictionary without significant overhead. */
+size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_DDict* ddict)
+{
+ /* pass content and size in case legacy frames are encountered */
+ return ZSTD_decompressMultiFrame(dctx, dst, dstCapacity, src, srcSize,
+ NULL, 0,
+ ddict);
+}
+
+
+/*=====================================
+* Streaming decompression
+*====================================*/
+
+ZSTD_DStream* ZSTD_createDStream(void)
+{
+ DEBUGLOG(3, "ZSTD_createDStream");
+ return ZSTD_createDStream_advanced(ZSTD_defaultCMem);
+}
+
+ZSTD_DStream* ZSTD_initStaticDStream(void *workspace, size_t workspaceSize)
+{
+ return ZSTD_initStaticDCtx(workspace, workspaceSize);
+}
+
+ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem)
+{
+ return ZSTD_createDCtx_advanced(customMem);
+}
+
+size_t ZSTD_freeDStream(ZSTD_DStream* zds)
+{
+ return ZSTD_freeDCtx(zds);
+}
+
+
+/* *** Initialization *** */
+
+size_t ZSTD_DStreamInSize(void) { return ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize; }
+size_t ZSTD_DStreamOutSize(void) { return ZSTD_BLOCKSIZE_MAX; }
+
+size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx,
+ const void* dict, size_t dictSize,
+ ZSTD_dictLoadMethod_e dictLoadMethod,
+ ZSTD_dictContentType_e dictContentType)
+{
+ RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+ ZSTD_clearDict(dctx);
+ if (dict && dictSize != 0) {
+ dctx->ddictLocal = ZSTD_createDDict_advanced(dict, dictSize, dictLoadMethod, dictContentType, dctx->customMem);
+ RETURN_ERROR_IF(dctx->ddictLocal == NULL, memory_allocation, "NULL pointer!");
+ dctx->ddict = dctx->ddictLocal;
+ dctx->dictUses = ZSTD_use_indefinitely;
+ }
+ return 0;
+}
+
+size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
+{
+ return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto);
+}
+
+size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize)
+{
+ return ZSTD_DCtx_loadDictionary_advanced(dctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
+}
+
+size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType)
+{
+ FORWARD_IF_ERROR(ZSTD_DCtx_loadDictionary_advanced(dctx, prefix, prefixSize, ZSTD_dlm_byRef, dictContentType), "");
+ dctx->dictUses = ZSTD_use_once;
+ return 0;
+}
+
+size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize)
+{
+ return ZSTD_DCtx_refPrefix_advanced(dctx, prefix, prefixSize, ZSTD_dct_rawContent);
+}
+
+
+/* ZSTD_initDStream_usingDict() :
+ * return : expected size, aka ZSTD_startingInputLength().
+ * this function cannot fail */
+size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize)
+{
+ DEBUGLOG(4, "ZSTD_initDStream_usingDict");
+ FORWARD_IF_ERROR( ZSTD_DCtx_reset(zds, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_DCtx_loadDictionary(zds, dict, dictSize) , "");
+ return ZSTD_startingInputLength(zds->format);
+}
+
+/* note : this variant can't fail */
+size_t ZSTD_initDStream(ZSTD_DStream* zds)
+{
+ DEBUGLOG(4, "ZSTD_initDStream");
+ return ZSTD_initDStream_usingDDict(zds, NULL);
+}
+
+/* ZSTD_initDStream_usingDDict() :
+ * ddict will just be referenced, and must outlive decompression session
+ * this function cannot fail */
+size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* dctx, const ZSTD_DDict* ddict)
+{
+ FORWARD_IF_ERROR( ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only) , "");
+ FORWARD_IF_ERROR( ZSTD_DCtx_refDDict(dctx, ddict) , "");
+ return ZSTD_startingInputLength(dctx->format);
+}
+
+/* ZSTD_resetDStream() :
+ * return : expected size, aka ZSTD_startingInputLength().
+ * this function cannot fail */
+size_t ZSTD_resetDStream(ZSTD_DStream* dctx)
+{
+ FORWARD_IF_ERROR(ZSTD_DCtx_reset(dctx, ZSTD_reset_session_only), "");
+ return ZSTD_startingInputLength(dctx->format);
+}
+
+
+size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict)
+{
+ RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+ ZSTD_clearDict(dctx);
+ if (ddict) {
+ dctx->ddict = ddict;
+ dctx->dictUses = ZSTD_use_indefinitely;
+ if (dctx->refMultipleDDicts == ZSTD_rmd_refMultipleDDicts) {
+ if (dctx->ddictSet == NULL) {
+ dctx->ddictSet = ZSTD_createDDictHashSet(dctx->customMem);
+ if (!dctx->ddictSet) {
+ RETURN_ERROR(memory_allocation, "Failed to allocate memory for hash set!");
+ }
+ }
+ assert(!dctx->staticSize); /* Impossible: ddictSet cannot have been allocated if static dctx */
+ FORWARD_IF_ERROR(ZSTD_DDictHashSet_addDDict(dctx->ddictSet, ddict, dctx->customMem), "");
+ }
+ }
+ return 0;
+}
+
+/* ZSTD_DCtx_setMaxWindowSize() :
+ * note : no direct equivalence in ZSTD_DCtx_setParameter,
+ * since this version sets windowSize, and the other sets windowLog */
+size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize)
+{
+ ZSTD_bounds const bounds = ZSTD_dParam_getBounds(ZSTD_d_windowLogMax);
+ size_t const min = (size_t)1 << bounds.lowerBound;
+ size_t const max = (size_t)1 << bounds.upperBound;
+ RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+ RETURN_ERROR_IF(maxWindowSize < min, parameter_outOfBound, "");
+ RETURN_ERROR_IF(maxWindowSize > max, parameter_outOfBound, "");
+ dctx->maxWindowSize = maxWindowSize;
+ return 0;
+}
+
+size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format)
+{
+ return ZSTD_DCtx_setParameter(dctx, ZSTD_d_format, (int)format);
+}
+
+ZSTD_bounds ZSTD_dParam_getBounds(ZSTD_dParameter dParam)
+{
+ ZSTD_bounds bounds = { 0, 0, 0 };
+ switch(dParam) {
+ case ZSTD_d_windowLogMax:
+ bounds.lowerBound = ZSTD_WINDOWLOG_ABSOLUTEMIN;
+ bounds.upperBound = ZSTD_WINDOWLOG_MAX;
+ return bounds;
+ case ZSTD_d_format:
+ bounds.lowerBound = (int)ZSTD_f_zstd1;
+ bounds.upperBound = (int)ZSTD_f_zstd1_magicless;
+ ZSTD_STATIC_ASSERT(ZSTD_f_zstd1 < ZSTD_f_zstd1_magicless);
+ return bounds;
+ case ZSTD_d_stableOutBuffer:
+ bounds.lowerBound = (int)ZSTD_bm_buffered;
+ bounds.upperBound = (int)ZSTD_bm_stable;
+ return bounds;
+ case ZSTD_d_forceIgnoreChecksum:
+ bounds.lowerBound = (int)ZSTD_d_validateChecksum;
+ bounds.upperBound = (int)ZSTD_d_ignoreChecksum;
+ return bounds;
+ case ZSTD_d_refMultipleDDicts:
+ bounds.lowerBound = (int)ZSTD_rmd_refSingleDDict;
+ bounds.upperBound = (int)ZSTD_rmd_refMultipleDDicts;
+ return bounds;
+ default:;
+ }
+ bounds.error = ERROR(parameter_unsupported);
+ return bounds;
+}
+
+/* ZSTD_dParam_withinBounds:
+ * @return 1 if value is within dParam bounds,
+ * 0 otherwise */
+static int ZSTD_dParam_withinBounds(ZSTD_dParameter dParam, int value)
+{
+ ZSTD_bounds const bounds = ZSTD_dParam_getBounds(dParam);
+ if (ZSTD_isError(bounds.error)) return 0;
+ if (value < bounds.lowerBound) return 0;
+ if (value > bounds.upperBound) return 0;
+ return 1;
+}
+
+#define CHECK_DBOUNDS(p,v) { \
+ RETURN_ERROR_IF(!ZSTD_dParam_withinBounds(p, v), parameter_outOfBound, ""); \
+}
+
+size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value)
+{
+ switch (param) {
+ case ZSTD_d_windowLogMax:
+ *value = (int)ZSTD_highbit32((U32)dctx->maxWindowSize);
+ return 0;
+ case ZSTD_d_format:
+ *value = (int)dctx->format;
+ return 0;
+ case ZSTD_d_stableOutBuffer:
+ *value = (int)dctx->outBufferMode;
+ return 0;
+ case ZSTD_d_forceIgnoreChecksum:
+ *value = (int)dctx->forceIgnoreChecksum;
+ return 0;
+ case ZSTD_d_refMultipleDDicts:
+ *value = (int)dctx->refMultipleDDicts;
+ return 0;
+ default:;
+ }
+ RETURN_ERROR(parameter_unsupported, "");
+}
+
+size_t ZSTD_DCtx_setParameter(ZSTD_DCtx* dctx, ZSTD_dParameter dParam, int value)
+{
+ RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+ switch(dParam) {
+ case ZSTD_d_windowLogMax:
+ if (value == 0) value = ZSTD_WINDOWLOG_LIMIT_DEFAULT;
+ CHECK_DBOUNDS(ZSTD_d_windowLogMax, value);
+ dctx->maxWindowSize = ((size_t)1) << value;
+ return 0;
+ case ZSTD_d_format:
+ CHECK_DBOUNDS(ZSTD_d_format, value);
+ dctx->format = (ZSTD_format_e)value;
+ return 0;
+ case ZSTD_d_stableOutBuffer:
+ CHECK_DBOUNDS(ZSTD_d_stableOutBuffer, value);
+ dctx->outBufferMode = (ZSTD_bufferMode_e)value;
+ return 0;
+ case ZSTD_d_forceIgnoreChecksum:
+ CHECK_DBOUNDS(ZSTD_d_forceIgnoreChecksum, value);
+ dctx->forceIgnoreChecksum = (ZSTD_forceIgnoreChecksum_e)value;
+ return 0;
+ case ZSTD_d_refMultipleDDicts:
+ CHECK_DBOUNDS(ZSTD_d_refMultipleDDicts, value);
+ if (dctx->staticSize != 0) {
+ RETURN_ERROR(parameter_unsupported, "Static dctx does not support multiple DDicts!");
+ }
+ dctx->refMultipleDDicts = (ZSTD_refMultipleDDicts_e)value;
+ return 0;
+ default:;
+ }
+ RETURN_ERROR(parameter_unsupported, "");
+}
+
+size_t ZSTD_DCtx_reset(ZSTD_DCtx* dctx, ZSTD_ResetDirective reset)
+{
+ if ( (reset == ZSTD_reset_session_only)
+ || (reset == ZSTD_reset_session_and_parameters) ) {
+ dctx->streamStage = zdss_init;
+ dctx->noForwardProgress = 0;
+ }
+ if ( (reset == ZSTD_reset_parameters)
+ || (reset == ZSTD_reset_session_and_parameters) ) {
+ RETURN_ERROR_IF(dctx->streamStage != zdss_init, stage_wrong, "");
+ ZSTD_clearDict(dctx);
+ ZSTD_DCtx_resetParameters(dctx);
+ }
+ return 0;
+}
+
+
+size_t ZSTD_sizeof_DStream(const ZSTD_DStream* dctx)
+{
+ return ZSTD_sizeof_DCtx(dctx);
+}
+
+size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize)
+{
+ size_t const blockSize = (size_t) MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
+ unsigned long long const neededRBSize = windowSize + blockSize + (WILDCOPY_OVERLENGTH * 2);
+ unsigned long long const neededSize = MIN(frameContentSize, neededRBSize);
+ size_t const minRBSize = (size_t) neededSize;
+ RETURN_ERROR_IF((unsigned long long)minRBSize != neededSize,
+ frameParameter_windowTooLarge, "");
+ return minRBSize;
+}
+
+size_t ZSTD_estimateDStreamSize(size_t windowSize)
+{
+ size_t const blockSize = MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
+ size_t const inBuffSize = blockSize; /* no block can be larger */
+ size_t const outBuffSize = ZSTD_decodingBufferSize_min(windowSize, ZSTD_CONTENTSIZE_UNKNOWN);
+ return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize;
+}
+
+size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize)
+{
+ U32 const windowSizeMax = 1U << ZSTD_WINDOWLOG_MAX; /* note : should be user-selectable, but requires an additional parameter (or a dctx) */
+ ZSTD_frameHeader zfh;
+ size_t const err = ZSTD_getFrameHeader(&zfh, src, srcSize);
+ if (ZSTD_isError(err)) return err;
+ RETURN_ERROR_IF(err>0, srcSize_wrong, "");
+ RETURN_ERROR_IF(zfh.windowSize > windowSizeMax,
+ frameParameter_windowTooLarge, "");
+ return ZSTD_estimateDStreamSize((size_t)zfh.windowSize);
+}
+
+
+/* ***** Decompression ***** */
+
+static int ZSTD_DCtx_isOverflow(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
+{
+ return (zds->inBuffSize + zds->outBuffSize) >= (neededInBuffSize + neededOutBuffSize) * ZSTD_WORKSPACETOOLARGE_FACTOR;
+}
+
+static void ZSTD_DCtx_updateOversizedDuration(ZSTD_DStream* zds, size_t const neededInBuffSize, size_t const neededOutBuffSize)
+{
+ if (ZSTD_DCtx_isOverflow(zds, neededInBuffSize, neededOutBuffSize))
+ zds->oversizedDuration++;
+ else
+ zds->oversizedDuration = 0;
+}
+
+static int ZSTD_DCtx_isOversizedTooLong(ZSTD_DStream* zds)
+{
+ return zds->oversizedDuration >= ZSTD_WORKSPACETOOLARGE_MAXDURATION;
+}
+
+/* Checks that the output buffer hasn't changed if ZSTD_obm_stable is used. */
+static size_t ZSTD_checkOutBuffer(ZSTD_DStream const* zds, ZSTD_outBuffer const* output)
+{
+ ZSTD_outBuffer const expect = zds->expectedOutBuffer;
+ /* No requirement when ZSTD_obm_stable is not enabled. */
+ if (zds->outBufferMode != ZSTD_bm_stable)
+ return 0;
+ /* Any buffer is allowed in zdss_init, this must be the same for every other call until
+ * the context is reset.
+ */
+ if (zds->streamStage == zdss_init)
+ return 0;
+ /* The buffer must match our expectation exactly. */
+ if (expect.dst == output->dst && expect.pos == output->pos && expect.size == output->size)
+ return 0;
+ RETURN_ERROR(dstBuffer_wrong, "ZSTD_d_stableOutBuffer enabled but output differs!");
+}
+
+/* Calls ZSTD_decompressContinue() with the right parameters for ZSTD_decompressStream()
+ * and updates the stage and the output buffer state. This call is extracted so it can be
+ * used both when reading directly from the ZSTD_inBuffer, and in buffered input mode.
+ * NOTE: You must break after calling this function since the streamStage is modified.
+ */
+static size_t ZSTD_decompressContinueStream(
+ ZSTD_DStream* zds, char** op, char* oend,
+ void const* src, size_t srcSize) {
+ int const isSkipFrame = ZSTD_isSkipFrame(zds);
+ if (zds->outBufferMode == ZSTD_bm_buffered) {
+ size_t const dstSize = isSkipFrame ? 0 : zds->outBuffSize - zds->outStart;
+ size_t const decodedSize = ZSTD_decompressContinue(zds,
+ zds->outBuff + zds->outStart, dstSize, src, srcSize);
+ FORWARD_IF_ERROR(decodedSize, "");
+ if (!decodedSize && !isSkipFrame) {
+ zds->streamStage = zdss_read;
+ } else {
+ zds->outEnd = zds->outStart + decodedSize;
+ zds->streamStage = zdss_flush;
+ }
+ } else {
+ /* Write directly into the output buffer */
+ size_t const dstSize = isSkipFrame ? 0 : (size_t)(oend - *op);
+ size_t const decodedSize = ZSTD_decompressContinue(zds, *op, dstSize, src, srcSize);
+ FORWARD_IF_ERROR(decodedSize, "");
+ *op += decodedSize;
+ /* Flushing is not needed. */
+ zds->streamStage = zdss_read;
+ assert(*op <= oend);
+ assert(zds->outBufferMode == ZSTD_bm_stable);
+ }
+ return 0;
+}
+
+size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input)
+{
+ const char* const src = (const char*)input->src;
+ const char* const istart = input->pos != 0 ? src + input->pos : src;
+ const char* const iend = input->size != 0 ? src + input->size : src;
+ const char* ip = istart;
+ char* const dst = (char*)output->dst;
+ char* const ostart = output->pos != 0 ? dst + output->pos : dst;
+ char* const oend = output->size != 0 ? dst + output->size : dst;
+ char* op = ostart;
+ U32 someMoreWork = 1;
+
+ DEBUGLOG(5, "ZSTD_decompressStream");
+ RETURN_ERROR_IF(
+ input->pos > input->size,
+ srcSize_wrong,
+ "forbidden. in: pos: %u vs size: %u",
+ (U32)input->pos, (U32)input->size);
+ RETURN_ERROR_IF(
+ output->pos > output->size,
+ dstSize_tooSmall,
+ "forbidden. out: pos: %u vs size: %u",
+ (U32)output->pos, (U32)output->size);
+ DEBUGLOG(5, "input size : %u", (U32)(input->size - input->pos));
+ FORWARD_IF_ERROR(ZSTD_checkOutBuffer(zds, output), "");
+
+ while (someMoreWork) {
+ switch(zds->streamStage)
+ {
+ case zdss_init :
+ DEBUGLOG(5, "stage zdss_init => transparent reset ");
+ zds->streamStage = zdss_loadHeader;
+ zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
+ zds->legacyVersion = 0;
+ zds->hostageByte = 0;
+ zds->expectedOutBuffer = *output;
+ ZSTD_FALLTHROUGH;
+
+ case zdss_loadHeader :
+ DEBUGLOG(5, "stage zdss_loadHeader (srcSize : %u)", (U32)(iend - ip));
+ { size_t const hSize = ZSTD_getFrameHeader_advanced(&zds->fParams, zds->headerBuffer, zds->lhSize, zds->format);
+ if (zds->refMultipleDDicts && zds->ddictSet) {
+ ZSTD_DCtx_selectFrameDDict(zds);
+ }
+ DEBUGLOG(5, "header size : %u", (U32)hSize);
+ if (ZSTD_isError(hSize)) {
+ return hSize; /* error */
+ }
+ if (hSize != 0) { /* need more input */
+ size_t const toLoad = hSize - zds->lhSize; /* if hSize!=0, hSize > zds->lhSize */
+ size_t const remainingInput = (size_t)(iend-ip);
+ assert(iend >= ip);
+ if (toLoad > remainingInput) { /* not enough input to load full header */
+ if (remainingInput > 0) {
+ ZSTD_memcpy(zds->headerBuffer + zds->lhSize, ip, remainingInput);
+ zds->lhSize += remainingInput;
+ }
+ input->pos = input->size;
+ return (MAX((size_t)ZSTD_FRAMEHEADERSIZE_MIN(zds->format), hSize) - zds->lhSize) + ZSTD_blockHeaderSize; /* remaining header bytes + next block header */
+ }
+ assert(ip != NULL);
+ ZSTD_memcpy(zds->headerBuffer + zds->lhSize, ip, toLoad); zds->lhSize = hSize; ip += toLoad;
+ break;
+ } }
+
+ /* check for single-pass mode opportunity */
+ if (zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
+ && zds->fParams.frameType != ZSTD_skippableFrame
+ && (U64)(size_t)(oend-op) >= zds->fParams.frameContentSize) {
+ size_t const cSize = ZSTD_findFrameCompressedSize(istart, (size_t)(iend-istart));
+ if (cSize <= (size_t)(iend-istart)) {
+ /* shortcut : using single-pass mode */
+ size_t const decompressedSize = ZSTD_decompress_usingDDict(zds, op, (size_t)(oend-op), istart, cSize, ZSTD_getDDict(zds));
+ if (ZSTD_isError(decompressedSize)) return decompressedSize;
+ DEBUGLOG(4, "shortcut to single-pass ZSTD_decompress_usingDDict()")
+ ip = istart + cSize;
+ op += decompressedSize;
+ zds->expected = 0;
+ zds->streamStage = zdss_init;
+ someMoreWork = 0;
+ break;
+ } }
+
+ /* Check output buffer is large enough for ZSTD_odm_stable. */
+ if (zds->outBufferMode == ZSTD_bm_stable
+ && zds->fParams.frameType != ZSTD_skippableFrame
+ && zds->fParams.frameContentSize != ZSTD_CONTENTSIZE_UNKNOWN
+ && (U64)(size_t)(oend-op) < zds->fParams.frameContentSize) {
+ RETURN_ERROR(dstSize_tooSmall, "ZSTD_obm_stable passed but ZSTD_outBuffer is too small");
+ }
+
+ /* Consume header (see ZSTDds_decodeFrameHeader) */
+ DEBUGLOG(4, "Consume header");
+ FORWARD_IF_ERROR(ZSTD_decompressBegin_usingDDict(zds, ZSTD_getDDict(zds)), "");
+
+ if ((MEM_readLE32(zds->headerBuffer) & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) { /* skippable frame */
+ zds->expected = MEM_readLE32(zds->headerBuffer + ZSTD_FRAMEIDSIZE);
+ zds->stage = ZSTDds_skipFrame;
+ } else {
+ FORWARD_IF_ERROR(ZSTD_decodeFrameHeader(zds, zds->headerBuffer, zds->lhSize), "");
+ zds->expected = ZSTD_blockHeaderSize;
+ zds->stage = ZSTDds_decodeBlockHeader;
+ }
+
+ /* control buffer memory usage */
+ DEBUGLOG(4, "Control max memory usage (%u KB <= max %u KB)",
+ (U32)(zds->fParams.windowSize >>10),
+ (U32)(zds->maxWindowSize >> 10) );
+ zds->fParams.windowSize = MAX(zds->fParams.windowSize, 1U << ZSTD_WINDOWLOG_ABSOLUTEMIN);
+ RETURN_ERROR_IF(zds->fParams.windowSize > zds->maxWindowSize,
+ frameParameter_windowTooLarge, "");
+
+ /* Adapt buffer sizes to frame header instructions */
+ { size_t const neededInBuffSize = MAX(zds->fParams.blockSizeMax, 4 /* frame checksum */);
+ size_t const neededOutBuffSize = zds->outBufferMode == ZSTD_bm_buffered
+ ? ZSTD_decodingBufferSize_min(zds->fParams.windowSize, zds->fParams.frameContentSize)
+ : 0;
+
+ ZSTD_DCtx_updateOversizedDuration(zds, neededInBuffSize, neededOutBuffSize);
+
+ { int const tooSmall = (zds->inBuffSize < neededInBuffSize) || (zds->outBuffSize < neededOutBuffSize);
+ int const tooLarge = ZSTD_DCtx_isOversizedTooLong(zds);
+
+ if (tooSmall || tooLarge) {
+ size_t const bufferSize = neededInBuffSize + neededOutBuffSize;
+ DEBUGLOG(4, "inBuff : from %u to %u",
+ (U32)zds->inBuffSize, (U32)neededInBuffSize);
+ DEBUGLOG(4, "outBuff : from %u to %u",
+ (U32)zds->outBuffSize, (U32)neededOutBuffSize);
+ if (zds->staticSize) { /* static DCtx */
+ DEBUGLOG(4, "staticSize : %u", (U32)zds->staticSize);
+ assert(zds->staticSize >= sizeof(ZSTD_DCtx)); /* controlled at init */
+ RETURN_ERROR_IF(
+ bufferSize > zds->staticSize - sizeof(ZSTD_DCtx),
+ memory_allocation, "");
+ } else {
+ ZSTD_customFree(zds->inBuff, zds->customMem);
+ zds->inBuffSize = 0;
+ zds->outBuffSize = 0;
+ zds->inBuff = (char*)ZSTD_customMalloc(bufferSize, zds->customMem);
+ RETURN_ERROR_IF(zds->inBuff == NULL, memory_allocation, "");
+ }
+ zds->inBuffSize = neededInBuffSize;
+ zds->outBuff = zds->inBuff + zds->inBuffSize;
+ zds->outBuffSize = neededOutBuffSize;
+ } } }
+ zds->streamStage = zdss_read;
+ ZSTD_FALLTHROUGH;
+
+ case zdss_read:
+ DEBUGLOG(5, "stage zdss_read");
+ { size_t const neededInSize = ZSTD_nextSrcSizeToDecompressWithInputSize(zds, (size_t)(iend - ip));
+ DEBUGLOG(5, "neededInSize = %u", (U32)neededInSize);
+ if (neededInSize==0) { /* end of frame */
+ zds->streamStage = zdss_init;
+ someMoreWork = 0;
+ break;
+ }
+ if ((size_t)(iend-ip) >= neededInSize) { /* decode directly from src */
+ FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, ip, neededInSize), "");
+ ip += neededInSize;
+ /* Function modifies the stage so we must break */
+ break;
+ } }
+ if (ip==iend) { someMoreWork = 0; break; } /* no more input */
+ zds->streamStage = zdss_load;
+ ZSTD_FALLTHROUGH;
+
+ case zdss_load:
+ { size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds);
+ size_t const toLoad = neededInSize - zds->inPos;
+ int const isSkipFrame = ZSTD_isSkipFrame(zds);
+ size_t loadedSize;
+ /* At this point we shouldn't be decompressing a block that we can stream. */
+ assert(neededInSize == ZSTD_nextSrcSizeToDecompressWithInputSize(zds, iend - ip));
+ if (isSkipFrame) {
+ loadedSize = MIN(toLoad, (size_t)(iend-ip));
+ } else {
+ RETURN_ERROR_IF(toLoad > zds->inBuffSize - zds->inPos,
+ corruption_detected,
+ "should never happen");
+ loadedSize = ZSTD_limitCopy(zds->inBuff + zds->inPos, toLoad, ip, (size_t)(iend-ip));
+ }
+ ip += loadedSize;
+ zds->inPos += loadedSize;
+ if (loadedSize < toLoad) { someMoreWork = 0; break; } /* not enough input, wait for more */
+
+ /* decode loaded input */
+ zds->inPos = 0; /* input is consumed */
+ FORWARD_IF_ERROR(ZSTD_decompressContinueStream(zds, &op, oend, zds->inBuff, neededInSize), "");
+ /* Function modifies the stage so we must break */
+ break;
+ }
+ case zdss_flush:
+ { size_t const toFlushSize = zds->outEnd - zds->outStart;
+ size_t const flushedSize = ZSTD_limitCopy(op, (size_t)(oend-op), zds->outBuff + zds->outStart, toFlushSize);
+ op += flushedSize;
+ zds->outStart += flushedSize;
+ if (flushedSize == toFlushSize) { /* flush completed */
+ zds->streamStage = zdss_read;
+ if ( (zds->outBuffSize < zds->fParams.frameContentSize)
+ && (zds->outStart + zds->fParams.blockSizeMax > zds->outBuffSize) ) {
+ DEBUGLOG(5, "restart filling outBuff from beginning (left:%i, needed:%u)",
+ (int)(zds->outBuffSize - zds->outStart),
+ (U32)zds->fParams.blockSizeMax);
+ zds->outStart = zds->outEnd = 0;
+ }
+ break;
+ } }
+ /* cannot complete flush */
+ someMoreWork = 0;
+ break;
+
+ default:
+ assert(0); /* impossible */
+ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
+ } }
+
+ /* result */
+ input->pos = (size_t)(ip - (const char*)(input->src));
+ output->pos = (size_t)(op - (char*)(output->dst));
+
+ /* Update the expected output buffer for ZSTD_obm_stable. */
+ zds->expectedOutBuffer = *output;
+
+ if ((ip==istart) && (op==ostart)) { /* no forward progress */
+ zds->noForwardProgress ++;
+ if (zds->noForwardProgress >= ZSTD_NO_FORWARD_PROGRESS_MAX) {
+ RETURN_ERROR_IF(op==oend, dstSize_tooSmall, "");
+ RETURN_ERROR_IF(ip==iend, srcSize_wrong, "");
+ assert(0);
+ }
+ } else {
+ zds->noForwardProgress = 0;
+ }
+ { size_t nextSrcSizeHint = ZSTD_nextSrcSizeToDecompress(zds);
+ if (!nextSrcSizeHint) { /* frame fully decoded */
+ if (zds->outEnd == zds->outStart) { /* output fully flushed */
+ if (zds->hostageByte) {
+ if (input->pos >= input->size) {
+ /* can't release hostage (not present) */
+ zds->streamStage = zdss_read;
+ return 1;
+ }
+ input->pos++; /* release hostage */
+ } /* zds->hostageByte */
+ return 0;
+ } /* zds->outEnd == zds->outStart */
+ if (!zds->hostageByte) { /* output not fully flushed; keep last byte as hostage; will be released when all output is flushed */
+ input->pos--; /* note : pos > 0, otherwise, impossible to finish reading last block */
+ zds->hostageByte=1;
+ }
+ return 1;
+ } /* nextSrcSizeHint==0 */
+ nextSrcSizeHint += ZSTD_blockHeaderSize * (ZSTD_nextInputType(zds) == ZSTDnit_block); /* preload header of next block */
+ assert(zds->inPos <= nextSrcSizeHint);
+ nextSrcSizeHint -= zds->inPos; /* part already loaded*/
+ return nextSrcSizeHint;
+ }
+}
+
+size_t ZSTD_decompressStream_simpleArgs (
+ ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity, size_t* dstPos,
+ const void* src, size_t srcSize, size_t* srcPos)
+{
+ ZSTD_outBuffer output = { dst, dstCapacity, *dstPos };
+ ZSTD_inBuffer input = { src, srcSize, *srcPos };
+ /* ZSTD_compress_generic() will check validity of dstPos and srcPos */
+ size_t const cErr = ZSTD_decompressStream(dctx, &output, &input);
+ *dstPos = output.pos;
+ *srcPos = input.pos;
+ return cErr;
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/* zstd_decompress_block :
+ * this module takes care of decompressing _compressed_ block */
+
+/*-*******************************************************
+* Dependencies
+*********************************************************/
+#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */
+#include "../common/compiler.h" /* prefetch */
+#include "../common/cpu.h" /* bmi2 */
+#include "../common/mem.h" /* low level memory routines */
+#define FSE_STATIC_LINKING_ONLY
+#include "../common/fse.h"
+#define HUF_STATIC_LINKING_ONLY
+#include "../common/huf.h"
+#include "../common/zstd_internal.h"
+#include "zstd_decompress_internal.h" /* ZSTD_DCtx */
+#include "zstd_ddict.h" /* ZSTD_DDictDictContent */
+#include "zstd_decompress_block.h"
+
+/*_*******************************************************
+* Macros
+**********************************************************/
+
+/* These two optional macros force the use one way or another of the two
+ * ZSTD_decompressSequences implementations. You can't force in both directions
+ * at the same time.
+ */
+#if defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
+ defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
+#error "Cannot force the use of the short and the long ZSTD_decompressSequences variants!"
+#endif
+
+
+/*_*******************************************************
+* Memory operations
+**********************************************************/
+static void ZSTD_copy4(void* dst, const void* src) { ZSTD_memcpy(dst, src, 4); }
+
+
+/*-*************************************************************
+ * Block decoding
+ ***************************************************************/
+
+/*! ZSTD_getcBlockSize() :
+ * Provides the size of compressed block from block header `src` */
+size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
+ blockProperties_t* bpPtr)
+{
+ RETURN_ERROR_IF(srcSize < ZSTD_blockHeaderSize, srcSize_wrong, "");
+
+ { U32 const cBlockHeader = MEM_readLE24(src);
+ U32 const cSize = cBlockHeader >> 3;
+ bpPtr->lastBlock = cBlockHeader & 1;
+ bpPtr->blockType = (blockType_e)((cBlockHeader >> 1) & 3);
+ bpPtr->origSize = cSize; /* only useful for RLE */
+ if (bpPtr->blockType == bt_rle) return 1;
+ RETURN_ERROR_IF(bpPtr->blockType == bt_reserved, corruption_detected, "");
+ return cSize;
+ }
+}
+
+
+/* Hidden declaration for fullbench */
+size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
+ const void* src, size_t srcSize);
+/*! ZSTD_decodeLiteralsBlock() :
+ * @return : nb of bytes read from src (< srcSize )
+ * note : symbol not declared but exposed for fullbench */
+size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
+ const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
+{
+ DEBUGLOG(5, "ZSTD_decodeLiteralsBlock");
+ RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected, "");
+
+ { const BYTE* const istart = (const BYTE*) src;
+ symbolEncodingType_e const litEncType = (symbolEncodingType_e)(istart[0] & 3);
+
+ switch(litEncType)
+ {
+ case set_repeat:
+ DEBUGLOG(5, "set_repeat flag : re-using stats from previous compressed literals block");
+ RETURN_ERROR_IF(dctx->litEntropy==0, dictionary_corrupted, "");
+ ZSTD_FALLTHROUGH;
+
+ case set_compressed:
+ RETURN_ERROR_IF(srcSize < 5, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need up to 5 for case 3");
+ { size_t lhSize, litSize, litCSize;
+ U32 singleStream=0;
+ U32 const lhlCode = (istart[0] >> 2) & 3;
+ U32 const lhc = MEM_readLE32(istart);
+ size_t hufSuccess;
+ switch(lhlCode)
+ {
+ case 0: case 1: default: /* note : default is impossible, since lhlCode into [0..3] */
+ /* 2 - 2 - 10 - 10 */
+ singleStream = !lhlCode;
+ lhSize = 3;
+ litSize = (lhc >> 4) & 0x3FF;
+ litCSize = (lhc >> 14) & 0x3FF;
+ break;
+ case 2:
+ /* 2 - 2 - 14 - 14 */
+ lhSize = 4;
+ litSize = (lhc >> 4) & 0x3FFF;
+ litCSize = lhc >> 18;
+ break;
+ case 3:
+ /* 2 - 2 - 18 - 18 */
+ lhSize = 5;
+ litSize = (lhc >> 4) & 0x3FFFF;
+ litCSize = (lhc >> 22) + ((size_t)istart[4] << 10);
+ break;
+ }
+ RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
+ RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected, "");
+
+ /* prefetch huffman table if cold */
+ if (dctx->ddictIsCold && (litSize > 768 /* heuristic */)) {
+ PREFETCH_AREA(dctx->HUFptr, sizeof(dctx->entropy.hufTable));
+ }
+
+ if (litEncType==set_repeat) {
+ if (singleStream) {
+ hufSuccess = HUF_decompress1X_usingDTable_bmi2(
+ dctx->litBuffer, litSize, istart+lhSize, litCSize,
+ dctx->HUFptr, dctx->bmi2);
+ } else {
+ hufSuccess = HUF_decompress4X_usingDTable_bmi2(
+ dctx->litBuffer, litSize, istart+lhSize, litCSize,
+ dctx->HUFptr, dctx->bmi2);
+ }
+ } else {
+ if (singleStream) {
+#if defined(HUF_FORCE_DECOMPRESS_X2)
+ hufSuccess = HUF_decompress1X_DCtx_wksp(
+ dctx->entropy.hufTable, dctx->litBuffer, litSize,
+ istart+lhSize, litCSize, dctx->workspace,
+ sizeof(dctx->workspace));
+#else
+ hufSuccess = HUF_decompress1X1_DCtx_wksp_bmi2(
+ dctx->entropy.hufTable, dctx->litBuffer, litSize,
+ istart+lhSize, litCSize, dctx->workspace,
+ sizeof(dctx->workspace), dctx->bmi2);
+#endif
+ } else {
+ hufSuccess = HUF_decompress4X_hufOnly_wksp_bmi2(
+ dctx->entropy.hufTable, dctx->litBuffer, litSize,
+ istart+lhSize, litCSize, dctx->workspace,
+ sizeof(dctx->workspace), dctx->bmi2);
+ }
+ }
+
+ RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected, "");
+
+ dctx->litPtr = dctx->litBuffer;
+ dctx->litSize = litSize;
+ dctx->litEntropy = 1;
+ if (litEncType==set_compressed) dctx->HUFptr = dctx->entropy.hufTable;
+ ZSTD_memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
+ return litCSize + lhSize;
+ }
+
+ case set_basic:
+ { size_t litSize, lhSize;
+ U32 const lhlCode = ((istart[0]) >> 2) & 3;
+ switch(lhlCode)
+ {
+ case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
+ lhSize = 1;
+ litSize = istart[0] >> 3;
+ break;
+ case 1:
+ lhSize = 2;
+ litSize = MEM_readLE16(istart) >> 4;
+ break;
+ case 3:
+ lhSize = 3;
+ litSize = MEM_readLE24(istart) >> 4;
+ break;
+ }
+
+ if (lhSize+litSize+WILDCOPY_OVERLENGTH > srcSize) { /* risk reading beyond src buffer with wildcopy */
+ RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected, "");
+ ZSTD_memcpy(dctx->litBuffer, istart+lhSize, litSize);
+ dctx->litPtr = dctx->litBuffer;
+ dctx->litSize = litSize;
+ ZSTD_memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
+ return lhSize+litSize;
+ }
+ /* direct reference into compressed stream */
+ dctx->litPtr = istart+lhSize;
+ dctx->litSize = litSize;
+ return lhSize+litSize;
+ }
+
+ case set_rle:
+ { U32 const lhlCode = ((istart[0]) >> 2) & 3;
+ size_t litSize, lhSize;
+ switch(lhlCode)
+ {
+ case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
+ lhSize = 1;
+ litSize = istart[0] >> 3;
+ break;
+ case 1:
+ lhSize = 2;
+ litSize = MEM_readLE16(istart) >> 4;
+ break;
+ case 3:
+ lhSize = 3;
+ litSize = MEM_readLE24(istart) >> 4;
+ RETURN_ERROR_IF(srcSize<4, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need lhSize+1 = 4");
+ break;
+ }
+ RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
+ ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize + WILDCOPY_OVERLENGTH);
+ dctx->litPtr = dctx->litBuffer;
+ dctx->litSize = litSize;
+ return lhSize+1;
+ }
+ default:
+ RETURN_ERROR(corruption_detected, "impossible");
+ }
+ }
+}
+
+/* Default FSE distribution tables.
+ * These are pre-calculated FSE decoding tables using default distributions as defined in specification :
+ * https://github.com/facebook/zstd/blob/release/doc/zstd_compression_format.md#default-distributions
+ * They were generated programmatically with following method :
+ * - start from default distributions, present in /lib/common/zstd_internal.h
+ * - generate tables normally, using ZSTD_buildFSETable()
+ * - printout the content of tables
+ * - pretify output, report below, test with fuzzer to ensure it's correct */
+
+/* Default FSE distribution table for Literal Lengths */
+static const ZSTD_seqSymbol LL_defaultDTable[(1<<LL_DEFAULTNORMLOG)+1] = {
+ { 1, 1, 1, LL_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
+ /* nextState, nbAddBits, nbBits, baseVal */
+ { 0, 0, 4, 0}, { 16, 0, 4, 0},
+ { 32, 0, 5, 1}, { 0, 0, 5, 3},
+ { 0, 0, 5, 4}, { 0, 0, 5, 6},
+ { 0, 0, 5, 7}, { 0, 0, 5, 9},
+ { 0, 0, 5, 10}, { 0, 0, 5, 12},
+ { 0, 0, 6, 14}, { 0, 1, 5, 16},
+ { 0, 1, 5, 20}, { 0, 1, 5, 22},
+ { 0, 2, 5, 28}, { 0, 3, 5, 32},
+ { 0, 4, 5, 48}, { 32, 6, 5, 64},
+ { 0, 7, 5, 128}, { 0, 8, 6, 256},
+ { 0, 10, 6, 1024}, { 0, 12, 6, 4096},
+ { 32, 0, 4, 0}, { 0, 0, 4, 1},
+ { 0, 0, 5, 2}, { 32, 0, 5, 4},
+ { 0, 0, 5, 5}, { 32, 0, 5, 7},
+ { 0, 0, 5, 8}, { 32, 0, 5, 10},
+ { 0, 0, 5, 11}, { 0, 0, 6, 13},
+ { 32, 1, 5, 16}, { 0, 1, 5, 18},
+ { 32, 1, 5, 22}, { 0, 2, 5, 24},
+ { 32, 3, 5, 32}, { 0, 3, 5, 40},
+ { 0, 6, 4, 64}, { 16, 6, 4, 64},
+ { 32, 7, 5, 128}, { 0, 9, 6, 512},
+ { 0, 11, 6, 2048}, { 48, 0, 4, 0},
+ { 16, 0, 4, 1}, { 32, 0, 5, 2},
+ { 32, 0, 5, 3}, { 32, 0, 5, 5},
+ { 32, 0, 5, 6}, { 32, 0, 5, 8},
+ { 32, 0, 5, 9}, { 32, 0, 5, 11},
+ { 32, 0, 5, 12}, { 0, 0, 6, 15},
+ { 32, 1, 5, 18}, { 32, 1, 5, 20},
+ { 32, 2, 5, 24}, { 32, 2, 5, 28},
+ { 32, 3, 5, 40}, { 32, 4, 5, 48},
+ { 0, 16, 6,65536}, { 0, 15, 6,32768},
+ { 0, 14, 6,16384}, { 0, 13, 6, 8192},
+}; /* LL_defaultDTable */
+
+/* Default FSE distribution table for Offset Codes */
+static const ZSTD_seqSymbol OF_defaultDTable[(1<<OF_DEFAULTNORMLOG)+1] = {
+ { 1, 1, 1, OF_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
+ /* nextState, nbAddBits, nbBits, baseVal */
+ { 0, 0, 5, 0}, { 0, 6, 4, 61},
+ { 0, 9, 5, 509}, { 0, 15, 5,32765},
+ { 0, 21, 5,2097149}, { 0, 3, 5, 5},
+ { 0, 7, 4, 125}, { 0, 12, 5, 4093},
+ { 0, 18, 5,262141}, { 0, 23, 5,8388605},
+ { 0, 5, 5, 29}, { 0, 8, 4, 253},
+ { 0, 14, 5,16381}, { 0, 20, 5,1048573},
+ { 0, 2, 5, 1}, { 16, 7, 4, 125},
+ { 0, 11, 5, 2045}, { 0, 17, 5,131069},
+ { 0, 22, 5,4194301}, { 0, 4, 5, 13},
+ { 16, 8, 4, 253}, { 0, 13, 5, 8189},
+ { 0, 19, 5,524285}, { 0, 1, 5, 1},
+ { 16, 6, 4, 61}, { 0, 10, 5, 1021},
+ { 0, 16, 5,65533}, { 0, 28, 5,268435453},
+ { 0, 27, 5,134217725}, { 0, 26, 5,67108861},
+ { 0, 25, 5,33554429}, { 0, 24, 5,16777213},
+}; /* OF_defaultDTable */
+
+
+/* Default FSE distribution table for Match Lengths */
+static const ZSTD_seqSymbol ML_defaultDTable[(1<<ML_DEFAULTNORMLOG)+1] = {
+ { 1, 1, 1, ML_DEFAULTNORMLOG}, /* header : fastMode, tableLog */
+ /* nextState, nbAddBits, nbBits, baseVal */
+ { 0, 0, 6, 3}, { 0, 0, 4, 4},
+ { 32, 0, 5, 5}, { 0, 0, 5, 6},
+ { 0, 0, 5, 8}, { 0, 0, 5, 9},
+ { 0, 0, 5, 11}, { 0, 0, 6, 13},
+ { 0, 0, 6, 16}, { 0, 0, 6, 19},
+ { 0, 0, 6, 22}, { 0, 0, 6, 25},
+ { 0, 0, 6, 28}, { 0, 0, 6, 31},
+ { 0, 0, 6, 34}, { 0, 1, 6, 37},
+ { 0, 1, 6, 41}, { 0, 2, 6, 47},
+ { 0, 3, 6, 59}, { 0, 4, 6, 83},
+ { 0, 7, 6, 131}, { 0, 9, 6, 515},
+ { 16, 0, 4, 4}, { 0, 0, 4, 5},
+ { 32, 0, 5, 6}, { 0, 0, 5, 7},
+ { 32, 0, 5, 9}, { 0, 0, 5, 10},
+ { 0, 0, 6, 12}, { 0, 0, 6, 15},
+ { 0, 0, 6, 18}, { 0, 0, 6, 21},
+ { 0, 0, 6, 24}, { 0, 0, 6, 27},
+ { 0, 0, 6, 30}, { 0, 0, 6, 33},
+ { 0, 1, 6, 35}, { 0, 1, 6, 39},
+ { 0, 2, 6, 43}, { 0, 3, 6, 51},
+ { 0, 4, 6, 67}, { 0, 5, 6, 99},
+ { 0, 8, 6, 259}, { 32, 0, 4, 4},
+ { 48, 0, 4, 4}, { 16, 0, 4, 5},
+ { 32, 0, 5, 7}, { 32, 0, 5, 8},
+ { 32, 0, 5, 10}, { 32, 0, 5, 11},
+ { 0, 0, 6, 14}, { 0, 0, 6, 17},
+ { 0, 0, 6, 20}, { 0, 0, 6, 23},
+ { 0, 0, 6, 26}, { 0, 0, 6, 29},
+ { 0, 0, 6, 32}, { 0, 16, 6,65539},
+ { 0, 15, 6,32771}, { 0, 14, 6,16387},
+ { 0, 13, 6, 8195}, { 0, 12, 6, 4099},
+ { 0, 11, 6, 2051}, { 0, 10, 6, 1027},
+}; /* ML_defaultDTable */
+
+
+static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U32 nbAddBits)
+{
+ void* ptr = dt;
+ ZSTD_seqSymbol_header* const DTableH = (ZSTD_seqSymbol_header*)ptr;
+ ZSTD_seqSymbol* const cell = dt + 1;
+
+ DTableH->tableLog = 0;
+ DTableH->fastMode = 0;
+
+ cell->nbBits = 0;
+ cell->nextState = 0;
+ assert(nbAddBits < 255);
+ cell->nbAdditionalBits = (BYTE)nbAddBits;
+ cell->baseValue = baseValue;
+}
+
+
+/* ZSTD_buildFSETable() :
+ * generate FSE decoding table for one symbol (ll, ml or off)
+ * cannot fail if input is valid =>
+ * all inputs are presumed validated at this stage */
+FORCE_INLINE_TEMPLATE
+void ZSTD_buildFSETable_body(ZSTD_seqSymbol* dt,
+ const short* normalizedCounter, unsigned maxSymbolValue,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ unsigned tableLog, void* wksp, size_t wkspSize)
+{
+ ZSTD_seqSymbol* const tableDecode = dt+1;
+ U32 const maxSV1 = maxSymbolValue + 1;
+ U32 const tableSize = 1 << tableLog;
+
+ U16* symbolNext = (U16*)wksp;
+ BYTE* spread = (BYTE*)(symbolNext + MaxSeq + 1);
+ U32 highThreshold = tableSize - 1;
+
+
+ /* Sanity Checks */
+ assert(maxSymbolValue <= MaxSeq);
+ assert(tableLog <= MaxFSELog);
+ assert(wkspSize >= ZSTD_BUILD_FSE_TABLE_WKSP_SIZE);
+ (void)wkspSize;
+ /* Init, lay down lowprob symbols */
+ { ZSTD_seqSymbol_header DTableH;
+ DTableH.tableLog = tableLog;
+ DTableH.fastMode = 1;
+ { S16 const largeLimit= (S16)(1 << (tableLog-1));
+ U32 s;
+ for (s=0; s<maxSV1; s++) {
+ if (normalizedCounter[s]==-1) {
+ tableDecode[highThreshold--].baseValue = s;
+ symbolNext[s] = 1;
+ } else {
+ if (normalizedCounter[s] >= largeLimit) DTableH.fastMode=0;
+ assert(normalizedCounter[s]>=0);
+ symbolNext[s] = (U16)normalizedCounter[s];
+ } } }
+ ZSTD_memcpy(dt, &DTableH, sizeof(DTableH));
+ }
+
+ /* Spread symbols */
+ assert(tableSize <= 512);
+ /* Specialized symbol spreading for the case when there are
+ * no low probability (-1 count) symbols. When compressing
+ * small blocks we avoid low probability symbols to hit this
+ * case, since header decoding speed matters more.
+ */
+ if (highThreshold == tableSize - 1) {
+ size_t const tableMask = tableSize-1;
+ size_t const step = FSE_TABLESTEP(tableSize);
+ /* First lay down the symbols in order.
+ * We use a uint64_t to lay down 8 bytes at a time. This reduces branch
+ * misses since small blocks generally have small table logs, so nearly
+ * all symbols have counts <= 8. We ensure we have 8 bytes at the end of
+ * our buffer to handle the over-write.
+ */
+ {
+ U64 const add = 0x0101010101010101ull;
+ size_t pos = 0;
+ U64 sv = 0;
+ U32 s;
+ for (s=0; s<maxSV1; ++s, sv += add) {
+ int i;
+ int const n = normalizedCounter[s];
+ MEM_write64(spread + pos, sv);
+ for (i = 8; i < n; i += 8) {
+ MEM_write64(spread + pos + i, sv);
+ }
+ pos += n;
+ }
+ }
+ /* Now we spread those positions across the table.
+ * The benefit of doing it in two stages is that we avoid the the
+ * variable size inner loop, which caused lots of branch misses.
+ * Now we can run through all the positions without any branch misses.
+ * We unroll the loop twice, since that is what emperically worked best.
+ */
+ {
+ size_t position = 0;
+ size_t s;
+ size_t const unroll = 2;
+ assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
+ for (s = 0; s < (size_t)tableSize; s += unroll) {
+ size_t u;
+ for (u = 0; u < unroll; ++u) {
+ size_t const uPosition = (position + (u * step)) & tableMask;
+ tableDecode[uPosition].baseValue = spread[s + u];
+ }
+ position = (position + (unroll * step)) & tableMask;
+ }
+ assert(position == 0);
+ }
+ } else {
+ U32 const tableMask = tableSize-1;
+ U32 const step = FSE_TABLESTEP(tableSize);
+ U32 s, position = 0;
+ for (s=0; s<maxSV1; s++) {
+ int i;
+ int const n = normalizedCounter[s];
+ for (i=0; i<n; i++) {
+ tableDecode[position].baseValue = s;
+ position = (position + step) & tableMask;
+ while (position > highThreshold) position = (position + step) & tableMask; /* lowprob area */
+ } }
+ assert(position == 0); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
+ }
+
+ /* Build Decoding table */
+ {
+ U32 u;
+ for (u=0; u<tableSize; u++) {
+ U32 const symbol = tableDecode[u].baseValue;
+ U32 const nextState = symbolNext[symbol]++;
+ tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
+ tableDecode[u].nextState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
+ assert(nbAdditionalBits[symbol] < 255);
+ tableDecode[u].nbAdditionalBits = (BYTE)nbAdditionalBits[symbol];
+ tableDecode[u].baseValue = baseValue[symbol];
+ }
+ }
+}
+
+/* Avoids the FORCE_INLINE of the _body() function. */
+static void ZSTD_buildFSETable_body_default(ZSTD_seqSymbol* dt,
+ const short* normalizedCounter, unsigned maxSymbolValue,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ unsigned tableLog, void* wksp, size_t wkspSize)
+{
+ ZSTD_buildFSETable_body(dt, normalizedCounter, maxSymbolValue,
+ baseValue, nbAdditionalBits, tableLog, wksp, wkspSize);
+}
+
+#if DYNAMIC_BMI2
+TARGET_ATTRIBUTE("bmi2") static void ZSTD_buildFSETable_body_bmi2(ZSTD_seqSymbol* dt,
+ const short* normalizedCounter, unsigned maxSymbolValue,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ unsigned tableLog, void* wksp, size_t wkspSize)
+{
+ ZSTD_buildFSETable_body(dt, normalizedCounter, maxSymbolValue,
+ baseValue, nbAdditionalBits, tableLog, wksp, wkspSize);
+}
+#endif
+
+void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
+ const short* normalizedCounter, unsigned maxSymbolValue,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ unsigned tableLog, void* wksp, size_t wkspSize, int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+ ZSTD_buildFSETable_body_bmi2(dt, normalizedCounter, maxSymbolValue,
+ baseValue, nbAdditionalBits, tableLog, wksp, wkspSize);
+ return;
+ }
+#endif
+ (void)bmi2;
+ ZSTD_buildFSETable_body_default(dt, normalizedCounter, maxSymbolValue,
+ baseValue, nbAdditionalBits, tableLog, wksp, wkspSize);
+}
+
+
+/*! ZSTD_buildSeqTable() :
+ * @return : nb bytes read from src,
+ * or an error code if it fails */
+static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymbol** DTablePtr,
+ symbolEncodingType_e type, unsigned max, U32 maxLog,
+ const void* src, size_t srcSize,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ const ZSTD_seqSymbol* defaultTable, U32 flagRepeatTable,
+ int ddictIsCold, int nbSeq, U32* wksp, size_t wkspSize,
+ int bmi2)
+{
+ switch(type)
+ {
+ case set_rle :
+ RETURN_ERROR_IF(!srcSize, srcSize_wrong, "");
+ RETURN_ERROR_IF((*(const BYTE*)src) > max, corruption_detected, "");
+ { U32 const symbol = *(const BYTE*)src;
+ U32 const baseline = baseValue[symbol];
+ U32 const nbBits = nbAdditionalBits[symbol];
+ ZSTD_buildSeqTable_rle(DTableSpace, baseline, nbBits);
+ }
+ *DTablePtr = DTableSpace;
+ return 1;
+ case set_basic :
+ *DTablePtr = defaultTable;
+ return 0;
+ case set_repeat:
+ RETURN_ERROR_IF(!flagRepeatTable, corruption_detected, "");
+ /* prefetch FSE table if used */
+ if (ddictIsCold && (nbSeq > 24 /* heuristic */)) {
+ const void* const pStart = *DTablePtr;
+ size_t const pSize = sizeof(ZSTD_seqSymbol) * (SEQSYMBOL_TABLE_SIZE(maxLog));
+ PREFETCH_AREA(pStart, pSize);
+ }
+ return 0;
+ case set_compressed :
+ { unsigned tableLog;
+ S16 norm[MaxSeq+1];
+ size_t const headerSize = FSE_readNCount(norm, &max, &tableLog, src, srcSize);
+ RETURN_ERROR_IF(FSE_isError(headerSize), corruption_detected, "");
+ RETURN_ERROR_IF(tableLog > maxLog, corruption_detected, "");
+ ZSTD_buildFSETable(DTableSpace, norm, max, baseValue, nbAdditionalBits, tableLog, wksp, wkspSize, bmi2);
+ *DTablePtr = DTableSpace;
+ return headerSize;
+ }
+ default :
+ assert(0);
+ RETURN_ERROR(GENERIC, "impossible");
+ }
+}
+
+size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
+ const void* src, size_t srcSize)
+{
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* ip = istart;
+ int nbSeq;
+ DEBUGLOG(5, "ZSTD_decodeSeqHeaders");
+
+ /* check */
+ RETURN_ERROR_IF(srcSize < MIN_SEQUENCES_SIZE, srcSize_wrong, "");
+
+ /* SeqHead */
+ nbSeq = *ip++;
+ if (!nbSeq) {
+ *nbSeqPtr=0;
+ RETURN_ERROR_IF(srcSize != 1, srcSize_wrong, "");
+ return 1;
+ }
+ if (nbSeq > 0x7F) {
+ if (nbSeq == 0xFF) {
+ RETURN_ERROR_IF(ip+2 > iend, srcSize_wrong, "");
+ nbSeq = MEM_readLE16(ip) + LONGNBSEQ;
+ ip+=2;
+ } else {
+ RETURN_ERROR_IF(ip >= iend, srcSize_wrong, "");
+ nbSeq = ((nbSeq-0x80)<<8) + *ip++;
+ }
+ }
+ *nbSeqPtr = nbSeq;
+
+ /* FSE table descriptors */
+ RETURN_ERROR_IF(ip+1 > iend, srcSize_wrong, ""); /* minimum possible size: 1 byte for symbol encoding types */
+ { symbolEncodingType_e const LLtype = (symbolEncodingType_e)(*ip >> 6);
+ symbolEncodingType_e const OFtype = (symbolEncodingType_e)((*ip >> 4) & 3);
+ symbolEncodingType_e const MLtype = (symbolEncodingType_e)((*ip >> 2) & 3);
+ ip++;
+
+ /* Build DTables */
+ { size_t const llhSize = ZSTD_buildSeqTable(dctx->entropy.LLTable, &dctx->LLTptr,
+ LLtype, MaxLL, LLFSELog,
+ ip, iend-ip,
+ LL_base, LL_bits,
+ LL_defaultDTable, dctx->fseEntropy,
+ dctx->ddictIsCold, nbSeq,
+ dctx->workspace, sizeof(dctx->workspace),
+ dctx->bmi2);
+ RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected, "ZSTD_buildSeqTable failed");
+ ip += llhSize;
+ }
+
+ { size_t const ofhSize = ZSTD_buildSeqTable(dctx->entropy.OFTable, &dctx->OFTptr,
+ OFtype, MaxOff, OffFSELog,
+ ip, iend-ip,
+ OF_base, OF_bits,
+ OF_defaultDTable, dctx->fseEntropy,
+ dctx->ddictIsCold, nbSeq,
+ dctx->workspace, sizeof(dctx->workspace),
+ dctx->bmi2);
+ RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected, "ZSTD_buildSeqTable failed");
+ ip += ofhSize;
+ }
+
+ { size_t const mlhSize = ZSTD_buildSeqTable(dctx->entropy.MLTable, &dctx->MLTptr,
+ MLtype, MaxML, MLFSELog,
+ ip, iend-ip,
+ ML_base, ML_bits,
+ ML_defaultDTable, dctx->fseEntropy,
+ dctx->ddictIsCold, nbSeq,
+ dctx->workspace, sizeof(dctx->workspace),
+ dctx->bmi2);
+ RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected, "ZSTD_buildSeqTable failed");
+ ip += mlhSize;
+ }
+ }
+
+ return ip-istart;
+}
+
+
+typedef struct {
+ size_t litLength;
+ size_t matchLength;
+ size_t offset;
+ const BYTE* match;
+} seq_t;
+
+typedef struct {
+ size_t state;
+ const ZSTD_seqSymbol* table;
+} ZSTD_fseState;
+
+typedef struct {
+ BIT_DStream_t DStream;
+ ZSTD_fseState stateLL;
+ ZSTD_fseState stateOffb;
+ ZSTD_fseState stateML;
+ size_t prevOffset[ZSTD_REP_NUM];
+ const BYTE* prefixStart;
+ const BYTE* dictEnd;
+ size_t pos;
+} seqState_t;
+
+/*! ZSTD_overlapCopy8() :
+ * Copies 8 bytes from ip to op and updates op and ip where ip <= op.
+ * If the offset is < 8 then the offset is spread to at least 8 bytes.
+ *
+ * Precondition: *ip <= *op
+ * Postcondition: *op - *op >= 8
+ */
+HINT_INLINE void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) {
+ assert(*ip <= *op);
+ if (offset < 8) {
+ /* close range match, overlap */
+ static const U32 dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 }; /* added */
+ static const int dec64table[] = { 8, 8, 8, 7, 8, 9,10,11 }; /* subtracted */
+ int const sub2 = dec64table[offset];
+ (*op)[0] = (*ip)[0];
+ (*op)[1] = (*ip)[1];
+ (*op)[2] = (*ip)[2];
+ (*op)[3] = (*ip)[3];
+ *ip += dec32table[offset];
+ ZSTD_copy4(*op+4, *ip);
+ *ip -= sub2;
+ } else {
+ ZSTD_copy8(*op, *ip);
+ }
+ *ip += 8;
+ *op += 8;
+ assert(*op - *ip >= 8);
+}
+
+/*! ZSTD_safecopy() :
+ * Specialized version of memcpy() that is allowed to READ up to WILDCOPY_OVERLENGTH past the input buffer
+ * and write up to 16 bytes past oend_w (op >= oend_w is allowed).
+ * This function is only called in the uncommon case where the sequence is near the end of the block. It
+ * should be fast for a single long sequence, but can be slow for several short sequences.
+ *
+ * @param ovtype controls the overlap detection
+ * - ZSTD_no_overlap: The source and destination are guaranteed to be at least WILDCOPY_VECLEN bytes apart.
+ * - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart.
+ * The src buffer must be before the dst buffer.
+ */
+static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) {
+ ptrdiff_t const diff = op - ip;
+ BYTE* const oend = op + length;
+
+ assert((ovtype == ZSTD_no_overlap && (diff <= -8 || diff >= 8 || op >= oend_w)) ||
+ (ovtype == ZSTD_overlap_src_before_dst && diff >= 0));
+
+ if (length < 8) {
+ /* Handle short lengths. */
+ while (op < oend) *op++ = *ip++;
+ return;
+ }
+ if (ovtype == ZSTD_overlap_src_before_dst) {
+ /* Copy 8 bytes and ensure the offset >= 8 when there can be overlap. */
+ assert(length >= 8);
+ ZSTD_overlapCopy8(&op, &ip, diff);
+ assert(op - ip >= 8);
+ assert(op <= oend);
+ }
+
+ if (oend <= oend_w) {
+ /* No risk of overwrite. */
+ ZSTD_wildcopy(op, ip, length, ovtype);
+ return;
+ }
+ if (op <= oend_w) {
+ /* Wildcopy until we get close to the end. */
+ assert(oend > oend_w);
+ ZSTD_wildcopy(op, ip, oend_w - op, ovtype);
+ ip += oend_w - op;
+ op = oend_w;
+ }
+ /* Handle the leftovers. */
+ while (op < oend) *op++ = *ip++;
+}
+
+/* ZSTD_execSequenceEnd():
+ * This version handles cases that are near the end of the output buffer. It requires
+ * more careful checks to make sure there is no overflow. By separating out these hard
+ * and unlikely cases, we can speed up the common cases.
+ *
+ * NOTE: This function needs to be fast for a single long sequence, but doesn't need
+ * to be optimized for many small sequences, since those fall into ZSTD_execSequence().
+ */
+FORCE_NOINLINE
+size_t ZSTD_execSequenceEnd(BYTE* op,
+ BYTE* const oend, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+{
+ BYTE* const oLitEnd = op + sequence.litLength;
+ size_t const sequenceLength = sequence.litLength + sequence.matchLength;
+ const BYTE* const iLitEnd = *litPtr + sequence.litLength;
+ const BYTE* match = oLitEnd - sequence.offset;
+ BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH;
+
+ /* bounds checks : careful of address space overflow in 32-bit mode */
+ RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer");
+ RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer");
+ assert(op < op + sequenceLength);
+ assert(oLitEnd < op + sequenceLength);
+
+ /* copy literals */
+ ZSTD_safecopy(op, oend_w, *litPtr, sequence.litLength, ZSTD_no_overlap);
+ op = oLitEnd;
+ *litPtr = iLitEnd;
+
+ /* copy Match */
+ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
+ /* offset beyond prefix */
+ RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, "");
+ match = dictEnd - (prefixStart-match);
+ if (match + sequence.matchLength <= dictEnd) {
+ ZSTD_memmove(oLitEnd, match, sequence.matchLength);
+ return sequenceLength;
+ }
+ /* span extDict & currentPrefixSegment */
+ { size_t const length1 = dictEnd - match;
+ ZSTD_memmove(oLitEnd, match, length1);
+ op = oLitEnd + length1;
+ sequence.matchLength -= length1;
+ match = prefixStart;
+ } }
+ ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst);
+ return sequenceLength;
+}
+
+HINT_INLINE
+size_t ZSTD_execSequence(BYTE* op,
+ BYTE* const oend, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+{
+ BYTE* const oLitEnd = op + sequence.litLength;
+ size_t const sequenceLength = sequence.litLength + sequence.matchLength;
+ BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
+ BYTE* const oend_w = oend - WILDCOPY_OVERLENGTH; /* risk : address space underflow on oend=NULL */
+ const BYTE* const iLitEnd = *litPtr + sequence.litLength;
+ const BYTE* match = oLitEnd - sequence.offset;
+
+ assert(op != NULL /* Precondition */);
+ assert(oend_w < oend /* No underflow */);
+ /* Handle edge cases in a slow path:
+ * - Read beyond end of literals
+ * - Match end is within WILDCOPY_OVERLIMIT of oend
+ * - 32-bit mode and the match length overflows
+ */
+ if (UNLIKELY(
+ iLitEnd > litLimit ||
+ oMatchEnd > oend_w ||
+ (MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH)))
+ return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
+
+ /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
+ assert(op <= oLitEnd /* No overflow */);
+ assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */);
+ assert(oMatchEnd <= oend /* No underflow */);
+ assert(iLitEnd <= litLimit /* Literal length is in bounds */);
+ assert(oLitEnd <= oend_w /* Can wildcopy literals */);
+ assert(oMatchEnd <= oend_w /* Can wildcopy matches */);
+
+ /* Copy Literals:
+ * Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9.
+ * We likely don't need the full 32-byte wildcopy.
+ */
+ assert(WILDCOPY_OVERLENGTH >= 16);
+ ZSTD_copy16(op, (*litPtr));
+ if (UNLIKELY(sequence.litLength > 16)) {
+ ZSTD_wildcopy(op+16, (*litPtr)+16, sequence.litLength-16, ZSTD_no_overlap);
+ }
+ op = oLitEnd;
+ *litPtr = iLitEnd; /* update for next sequence */
+
+ /* Copy Match */
+ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
+ /* offset beyond prefix -> go into extDict */
+ RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, "");
+ match = dictEnd + (match - prefixStart);
+ if (match + sequence.matchLength <= dictEnd) {
+ ZSTD_memmove(oLitEnd, match, sequence.matchLength);
+ return sequenceLength;
+ }
+ /* span extDict & currentPrefixSegment */
+ { size_t const length1 = dictEnd - match;
+ ZSTD_memmove(oLitEnd, match, length1);
+ op = oLitEnd + length1;
+ sequence.matchLength -= length1;
+ match = prefixStart;
+ } }
+ /* Match within prefix of 1 or more bytes */
+ assert(op <= oMatchEnd);
+ assert(oMatchEnd <= oend_w);
+ assert(match >= prefixStart);
+ assert(sequence.matchLength >= 1);
+
+ /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy
+ * without overlap checking.
+ */
+ if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) {
+ /* We bet on a full wildcopy for matches, since we expect matches to be
+ * longer than literals (in general). In silesia, ~10% of matches are longer
+ * than 16 bytes.
+ */
+ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap);
+ return sequenceLength;
+ }
+ assert(sequence.offset < WILDCOPY_VECLEN);
+
+ /* Copy 8 bytes and spread the offset to be >= 8. */
+ ZSTD_overlapCopy8(&op, &match, sequence.offset);
+
+ /* If the match length is > 8 bytes, then continue with the wildcopy. */
+ if (sequence.matchLength > 8) {
+ assert(op < oMatchEnd);
+ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength-8, ZSTD_overlap_src_before_dst);
+ }
+ return sequenceLength;
+}
+
+static void
+ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt)
+{
+ const void* ptr = dt;
+ const ZSTD_seqSymbol_header* const DTableH = (const ZSTD_seqSymbol_header*)ptr;
+ DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
+ DEBUGLOG(6, "ZSTD_initFseState : val=%u using %u bits",
+ (U32)DStatePtr->state, DTableH->tableLog);
+ BIT_reloadDStream(bitD);
+ DStatePtr->table = dt + 1;
+}
+
+FORCE_INLINE_TEMPLATE void
+ZSTD_updateFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD)
+{
+ ZSTD_seqSymbol const DInfo = DStatePtr->table[DStatePtr->state];
+ U32 const nbBits = DInfo.nbBits;
+ size_t const lowBits = BIT_readBits(bitD, nbBits);
+ DStatePtr->state = DInfo.nextState + lowBits;
+}
+
+FORCE_INLINE_TEMPLATE void
+ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, ZSTD_seqSymbol const DInfo)
+{
+ U32 const nbBits = DInfo.nbBits;
+ size_t const lowBits = BIT_readBits(bitD, nbBits);
+ DStatePtr->state = DInfo.nextState + lowBits;
+}
+
+/* We need to add at most (ZSTD_WINDOWLOG_MAX_32 - 1) bits to read the maximum
+ * offset bits. But we can only read at most (STREAM_ACCUMULATOR_MIN_32 - 1)
+ * bits before reloading. This value is the maximum number of bytes we read
+ * after reloading when we are decoding long offsets.
+ */
+#define LONG_OFFSETS_MAX_EXTRA_BITS_32 \
+ (ZSTD_WINDOWLOG_MAX_32 > STREAM_ACCUMULATOR_MIN_32 \
+ ? ZSTD_WINDOWLOG_MAX_32 - STREAM_ACCUMULATOR_MIN_32 \
+ : 0)
+
+typedef enum { ZSTD_lo_isRegularOffset, ZSTD_lo_isLongOffset=1 } ZSTD_longOffset_e;
+typedef enum { ZSTD_p_noPrefetch=0, ZSTD_p_prefetch=1 } ZSTD_prefetch_e;
+
+FORCE_INLINE_TEMPLATE seq_t
+ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, const ZSTD_prefetch_e prefetch)
+{
+ seq_t seq;
+ ZSTD_seqSymbol const llDInfo = seqState->stateLL.table[seqState->stateLL.state];
+ ZSTD_seqSymbol const mlDInfo = seqState->stateML.table[seqState->stateML.state];
+ ZSTD_seqSymbol const ofDInfo = seqState->stateOffb.table[seqState->stateOffb.state];
+ U32 const llBase = llDInfo.baseValue;
+ U32 const mlBase = mlDInfo.baseValue;
+ U32 const ofBase = ofDInfo.baseValue;
+ BYTE const llBits = llDInfo.nbAdditionalBits;
+ BYTE const mlBits = mlDInfo.nbAdditionalBits;
+ BYTE const ofBits = ofDInfo.nbAdditionalBits;
+ BYTE const totalBits = llBits+mlBits+ofBits;
+
+ /* sequence */
+ { size_t offset;
+ if (ofBits > 1) {
+ ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
+ ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
+ assert(ofBits <= MaxOff);
+ if (MEM_32bits() && longOffsets && (ofBits >= STREAM_ACCUMULATOR_MIN_32)) {
+ U32 const extraBits = ofBits - MIN(ofBits, 32 - seqState->DStream.bitsConsumed);
+ offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
+ BIT_reloadDStream(&seqState->DStream);
+ if (extraBits) offset += BIT_readBitsFast(&seqState->DStream, extraBits);
+ assert(extraBits <= LONG_OFFSETS_MAX_EXTRA_BITS_32); /* to avoid another reload */
+ } else {
+ offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/*>0*/); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
+ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
+ }
+ seqState->prevOffset[2] = seqState->prevOffset[1];
+ seqState->prevOffset[1] = seqState->prevOffset[0];
+ seqState->prevOffset[0] = offset;
+ } else {
+ U32 const ll0 = (llBase == 0);
+ if (LIKELY((ofBits == 0))) {
+ if (LIKELY(!ll0))
+ offset = seqState->prevOffset[0];
+ else {
+ offset = seqState->prevOffset[1];
+ seqState->prevOffset[1] = seqState->prevOffset[0];
+ seqState->prevOffset[0] = offset;
+ }
+ } else {
+ offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1);
+ { size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
+ temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
+ if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
+ seqState->prevOffset[1] = seqState->prevOffset[0];
+ seqState->prevOffset[0] = offset = temp;
+ } } }
+ seq.offset = offset;
+ }
+
+ seq.matchLength = mlBase;
+ if (mlBits > 0)
+ seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
+
+ if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
+ BIT_reloadDStream(&seqState->DStream);
+ if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
+ BIT_reloadDStream(&seqState->DStream);
+ /* Ensure there are enough bits to read the rest of data in 64-bit mode. */
+ ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);
+
+ seq.litLength = llBase;
+ if (llBits > 0)
+ seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/*>0*/);
+
+ if (MEM_32bits())
+ BIT_reloadDStream(&seqState->DStream);
+
+ DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u",
+ (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
+
+ if (prefetch == ZSTD_p_prefetch) {
+ size_t const pos = seqState->pos + seq.litLength;
+ const BYTE* const matchBase = (seq.offset > pos) ? seqState->dictEnd : seqState->prefixStart;
+ seq.match = matchBase + pos - seq.offset; /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
+ * No consequence though : no memory access will occur, offset is only used for prefetching */
+ seqState->pos = pos + seq.matchLength;
+ }
+
+ /* ANS state update
+ * gcc-9.0.0 does 2.5% worse with ZSTD_updateFseStateWithDInfo().
+ * clang-9.2.0 does 7% worse with ZSTD_updateFseState().
+ * Naturally it seems like ZSTD_updateFseStateWithDInfo() should be the
+ * better option, so it is the default for other compilers. But, if you
+ * measure that it is worse, please put up a pull request.
+ */
+ {
+#if !defined(__clang__)
+ const int kUseUpdateFseState = 1;
+#else
+ const int kUseUpdateFseState = 0;
+#endif
+ if (kUseUpdateFseState) {
+ ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream); /* <= 9 bits */
+ ZSTD_updateFseState(&seqState->stateML, &seqState->DStream); /* <= 9 bits */
+ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
+ ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
+ } else {
+ ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llDInfo); /* <= 9 bits */
+ ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlDInfo); /* <= 9 bits */
+ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
+ ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofDInfo); /* <= 8 bits */
+ }
+ }
+
+ return seq;
+}
+
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+MEM_STATIC int ZSTD_dictionaryIsActive(ZSTD_DCtx const* dctx, BYTE const* prefixStart, BYTE const* oLitEnd)
+{
+ size_t const windowSize = dctx->fParams.windowSize;
+ /* No dictionary used. */
+ if (dctx->dictContentEndForFuzzing == NULL) return 0;
+ /* Dictionary is our prefix. */
+ if (prefixStart == dctx->dictContentBeginForFuzzing) return 1;
+ /* Dictionary is not our ext-dict. */
+ if (dctx->dictEnd != dctx->dictContentEndForFuzzing) return 0;
+ /* Dictionary is not within our window size. */
+ if ((size_t)(oLitEnd - prefixStart) >= windowSize) return 0;
+ /* Dictionary is active. */
+ return 1;
+}
+
+MEM_STATIC void ZSTD_assertValidSequence(
+ ZSTD_DCtx const* dctx,
+ BYTE const* op, BYTE const* oend,
+ seq_t const seq,
+ BYTE const* prefixStart, BYTE const* virtualStart)
+{
+#if DEBUGLEVEL >= 1
+ size_t const windowSize = dctx->fParams.windowSize;
+ size_t const sequenceSize = seq.litLength + seq.matchLength;
+ BYTE const* const oLitEnd = op + seq.litLength;
+ DEBUGLOG(6, "Checking sequence: litL=%u matchL=%u offset=%u",
+ (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
+ assert(op <= oend);
+ assert((size_t)(oend - op) >= sequenceSize);
+ assert(sequenceSize <= ZSTD_BLOCKSIZE_MAX);
+ if (ZSTD_dictionaryIsActive(dctx, prefixStart, oLitEnd)) {
+ size_t const dictSize = (size_t)((char const*)dctx->dictContentEndForFuzzing - (char const*)dctx->dictContentBeginForFuzzing);
+ /* Offset must be within the dictionary. */
+ assert(seq.offset <= (size_t)(oLitEnd - virtualStart));
+ assert(seq.offset <= windowSize + dictSize);
+ } else {
+ /* Offset must be within our window. */
+ assert(seq.offset <= windowSize);
+ }
+#else
+ (void)dctx, (void)op, (void)oend, (void)seq, (void)prefixStart, (void)virtualStart;
+#endif
+}
+#endif
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
+FORCE_INLINE_TEMPLATE size_t
+DONT_VECTORIZE
+ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ const BYTE* ip = (const BYTE*)seqStart;
+ const BYTE* const iend = ip + seqSize;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + maxDstSize;
+ BYTE* op = ostart;
+ const BYTE* litPtr = dctx->litPtr;
+ const BYTE* const litEnd = litPtr + dctx->litSize;
+ const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
+ const BYTE* const vBase = (const BYTE*) (dctx->virtualStart);
+ const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
+ DEBUGLOG(5, "ZSTD_decompressSequences_body");
+ (void)frame;
+
+ /* Regen sequences */
+ if (nbSeq) {
+ seqState_t seqState;
+ size_t error = 0;
+ dctx->fseEntropy = 1;
+ { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
+ RETURN_ERROR_IF(
+ ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
+ corruption_detected, "");
+ ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
+ ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
+ ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
+ assert(dst != NULL);
+
+ ZSTD_STATIC_ASSERT(
+ BIT_DStream_unfinished < BIT_DStream_completed &&
+ BIT_DStream_endOfBuffer < BIT_DStream_completed &&
+ BIT_DStream_completed < BIT_DStream_overflow);
+
+#if defined(__x86_64__)
+ /* Align the decompression loop to 32 + 16 bytes.
+ *
+ * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression
+ * speed swings based on the alignment of the decompression loop. This
+ * performance swing is caused by parts of the decompression loop falling
+ * out of the DSB. The entire decompression loop should fit in the DSB,
+ * when it can't we get much worse performance. You can measure if you've
+ * hit the good case or the bad case with this perf command for some
+ * compressed file test.zst:
+ *
+ * perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \
+ * -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst
+ *
+ * If you see most cycles served out of the MITE you've hit the bad case.
+ * If you see most cycles served out of the DSB you've hit the good case.
+ * If it is pretty even then you may be in an okay case.
+ *
+ * I've been able to reproduce this issue on the following CPUs:
+ * - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9
+ * Use Instruments->Counters to get DSB/MITE cycles.
+ * I never got performance swings, but I was able to
+ * go from the good case of mostly DSB to half of the
+ * cycles served from MITE.
+ * - Coffeelake: Intel i9-9900k
+ *
+ * I haven't been able to reproduce the instability or DSB misses on any
+ * of the following CPUS:
+ * - Haswell
+ * - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH
+ * - Skylake
+ *
+ * If you are seeing performance stability this script can help test.
+ * It tests on 4 commits in zstd where I saw performance change.
+ *
+ * https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4
+ */
+ __asm__(".p2align 5");
+ __asm__("nop");
+ __asm__(".p2align 4");
+#endif
+ for ( ; ; ) {
+ seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_noPrefetch);
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
+#endif
+ DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
+ BIT_reloadDStream(&(seqState.DStream));
+ op += oneSeqSize;
+ /* gcc and clang both don't like early returns in this loop.
+ * Instead break and check for an error at the end of the loop.
+ */
+ if (UNLIKELY(ZSTD_isError(oneSeqSize))) {
+ error = oneSeqSize;
+ break;
+ }
+ if (UNLIKELY(!--nbSeq)) break;
+ }
+
+ /* check if reached exact end */
+ DEBUGLOG(5, "ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : %i", nbSeq);
+ if (ZSTD_isError(error)) return error;
+ RETURN_ERROR_IF(nbSeq, corruption_detected, "");
+ RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, "");
+ /* save reps for next block */
+ { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
+ }
+
+ /* last literal segment */
+ { size_t const lastLLSize = litEnd - litPtr;
+ RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
+ if (op != NULL) {
+ ZSTD_memcpy(op, litPtr, lastLLSize);
+ op += lastLLSize;
+ }
+ }
+
+ return op-ostart;
+}
+
+static size_t
+ZSTD_decompressSequences_default(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_decompressSequencesLong_body(
+ ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ const BYTE* ip = (const BYTE*)seqStart;
+ const BYTE* const iend = ip + seqSize;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = ostart + maxDstSize;
+ BYTE* op = ostart;
+ const BYTE* litPtr = dctx->litPtr;
+ const BYTE* const litEnd = litPtr + dctx->litSize;
+ const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
+ const BYTE* const dictStart = (const BYTE*) (dctx->virtualStart);
+ const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
+ (void)frame;
+
+ /* Regen sequences */
+ if (nbSeq) {
+#define STORED_SEQS 4
+#define STORED_SEQS_MASK (STORED_SEQS-1)
+#define ADVANCED_SEQS 4
+ seq_t sequences[STORED_SEQS];
+ int const seqAdvance = MIN(nbSeq, ADVANCED_SEQS);
+ seqState_t seqState;
+ int seqNb;
+ dctx->fseEntropy = 1;
+ { int i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
+ seqState.prefixStart = prefixStart;
+ seqState.pos = (size_t)(op-prefixStart);
+ seqState.dictEnd = dictEnd;
+ assert(dst != NULL);
+ assert(iend >= ip);
+ RETURN_ERROR_IF(
+ ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend-ip)),
+ corruption_detected, "");
+ ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
+ ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
+ ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
+
+ /* prepare in advance */
+ for (seqNb=0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && (seqNb<seqAdvance); seqNb++) {
+ sequences[seqNb] = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
+ PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
+ }
+ RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected, "");
+
+ /* decode and decompress */
+ for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) {
+ seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+ PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
+ sequences[seqNb & STORED_SEQS_MASK] = sequence;
+ op += oneSeqSize;
+ }
+ RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected, "");
+
+ /* finish queue */
+ seqNb -= seqAdvance;
+ for ( ; seqNb<nbSeq ; seqNb++) {
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+ op += oneSeqSize;
+ }
+
+ /* save reps for next block */
+ { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
+ }
+
+ /* last literal segment */
+ { size_t const lastLLSize = litEnd - litPtr;
+ RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
+ if (op != NULL) {
+ ZSTD_memcpy(op, litPtr, lastLLSize);
+ op += lastLLSize;
+ }
+ }
+
+ return op-ostart;
+}
+
+static size_t
+ZSTD_decompressSequencesLong_default(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
+
+
+
+#if DYNAMIC_BMI2
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
+static TARGET_ATTRIBUTE("bmi2") size_t
+DONT_VECTORIZE
+ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
+static TARGET_ATTRIBUTE("bmi2") size_t
+ZSTD_decompressSequencesLong_bmi2(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequencesLong_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
+
+#endif /* DYNAMIC_BMI2 */
+
+typedef size_t (*ZSTD_decompressSequences_t)(
+ ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame);
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
+static size_t
+ZSTD_decompressSequences(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ DEBUGLOG(5, "ZSTD_decompressSequences");
+#if DYNAMIC_BMI2
+ if (dctx->bmi2) {
+ return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+ }
+#endif
+ return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
+
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
+/* ZSTD_decompressSequencesLong() :
+ * decompression function triggered when a minimum share of offsets is considered "long",
+ * aka out of cache.
+ * note : "long" definition seems overloaded here, sometimes meaning "wider than bitstream register", and sometimes meaning "farther than memory cache distance".
+ * This function will try to mitigate main memory latency through the use of prefetching */
+static size_t
+ZSTD_decompressSequencesLong(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ DEBUGLOG(5, "ZSTD_decompressSequencesLong");
+#if DYNAMIC_BMI2
+ if (dctx->bmi2) {
+ return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+ }
+#endif
+ return ZSTD_decompressSequencesLong_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT */
+
+
+
+#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
+ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
+/* ZSTD_getLongOffsetsShare() :
+ * condition : offTable must be valid
+ * @return : "share" of long offsets (arbitrarily defined as > (1<<23))
+ * compared to maximum possible of (1<<OffFSELog) */
+static unsigned
+ZSTD_getLongOffsetsShare(const ZSTD_seqSymbol* offTable)
+{
+ const void* ptr = offTable;
+ U32 const tableLog = ((const ZSTD_seqSymbol_header*)ptr)[0].tableLog;
+ const ZSTD_seqSymbol* table = offTable + 1;
+ U32 const max = 1 << tableLog;
+ U32 u, total = 0;
+ DEBUGLOG(5, "ZSTD_getLongOffsetsShare: (tableLog=%u)", tableLog);
+
+ assert(max <= (1 << OffFSELog)); /* max not too large */
+ for (u=0; u<max; u++) {
+ if (table[u].nbAdditionalBits > 22) total += 1;
+ }
+
+ assert(tableLog <= OffFSELog);
+ total <<= (OffFSELog - tableLog); /* scale to OffFSELog */
+
+ return total;
+}
+#endif
+
+size_t
+ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, const int frame)
+{ /* blockType == blockCompressed */
+ const BYTE* ip = (const BYTE*)src;
+ /* isLongOffset must be true if there are long offsets.
+ * Offsets are long if they are larger than 2^STREAM_ACCUMULATOR_MIN.
+ * We don't expect that to be the case in 64-bit mode.
+ * In block mode, window size is not known, so we have to be conservative.
+ * (note: but it could be evaluated from current-lowLimit)
+ */
+ ZSTD_longOffset_e const isLongOffset = (ZSTD_longOffset_e)(MEM_32bits() && (!frame || (dctx->fParams.windowSize > (1ULL << STREAM_ACCUMULATOR_MIN))));
+ DEBUGLOG(5, "ZSTD_decompressBlock_internal (size : %u)", (U32)srcSize);
+
+ RETURN_ERROR_IF(srcSize >= ZSTD_BLOCKSIZE_MAX, srcSize_wrong, "");
+
+ /* Decode literals section */
+ { size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize);
+ DEBUGLOG(5, "ZSTD_decodeLiteralsBlock : %u", (U32)litCSize);
+ if (ZSTD_isError(litCSize)) return litCSize;
+ ip += litCSize;
+ srcSize -= litCSize;
+ }
+
+ /* Build Decoding Tables */
+ {
+ /* These macros control at build-time which decompressor implementation
+ * we use. If neither is defined, we do some inspection and dispatch at
+ * runtime.
+ */
+#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
+ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
+ int usePrefetchDecoder = dctx->ddictIsCold;
+#endif
+ int nbSeq;
+ size_t const seqHSize = ZSTD_decodeSeqHeaders(dctx, &nbSeq, ip, srcSize);
+ if (ZSTD_isError(seqHSize)) return seqHSize;
+ ip += seqHSize;
+ srcSize -= seqHSize;
+
+ RETURN_ERROR_IF(dst == NULL && nbSeq > 0, dstSize_tooSmall, "NULL not handled");
+
+#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
+ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
+ if ( !usePrefetchDecoder
+ && (!frame || (dctx->fParams.windowSize > (1<<24)))
+ && (nbSeq>ADVANCED_SEQS) ) { /* could probably use a larger nbSeq limit */
+ U32 const shareLongOffsets = ZSTD_getLongOffsetsShare(dctx->OFTptr);
+ U32 const minShare = MEM_64bits() ? 7 : 20; /* heuristic values, correspond to 2.73% and 7.81% */
+ usePrefetchDecoder = (shareLongOffsets >= minShare);
+ }
+#endif
+
+ dctx->ddictIsCold = 0;
+
+#if !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT) && \
+ !defined(ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG)
+ if (usePrefetchDecoder)
+#endif
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
+ return ZSTD_decompressSequencesLong(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
+#endif
+
+#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
+ /* else */
+ return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
+#endif
+ }
+}
+
+
+void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst, size_t dstSize)
+{
+ if (dst != dctx->previousDstEnd && dstSize > 0) { /* not contiguous */
+ dctx->dictEnd = dctx->previousDstEnd;
+ dctx->virtualStart = (const char*)dst - ((const char*)(dctx->previousDstEnd) - (const char*)(dctx->prefixStart));
+ dctx->prefixStart = dst;
+ dctx->previousDstEnd = dst;
+ }
+}
+
+
+size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize)
+{
+ size_t dSize;
+ ZSTD_checkContinuity(dctx, dst, dstCapacity);
+ dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 0);
+ dctx->previousDstEnd = (char*)dst + dSize;
+ return dSize;
+}
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+
+#ifndef ZSTD_DEC_BLOCK_H
+#define ZSTD_DEC_BLOCK_H
+
+/*-*******************************************************
+ * Dependencies
+ *********************************************************/
+#include "../common/zstd_deps.h" /* size_t */
+#include <linux/zstd.h> /* DCtx, and some public functions */
+#include "../common/zstd_internal.h" /* blockProperties_t, and some public functions */
+#include "zstd_decompress_internal.h" /* ZSTD_seqSymbol */
+
+
+/* === Prototypes === */
+
+/* note: prototypes already published within `zstd.h` :
+ * ZSTD_decompressBlock()
+ */
+
+/* note: prototypes already published within `zstd_internal.h` :
+ * ZSTD_getcBlockSize()
+ * ZSTD_decodeSeqHeaders()
+ */
+
+
+/* ZSTD_decompressBlock_internal() :
+ * decompress block, starting at `src`,
+ * into destination buffer `dst`.
+ * @return : decompressed block size,
+ * or an error code (which can be tested using ZSTD_isError())
+ */
+size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, const int frame);
+
+/* ZSTD_buildFSETable() :
+ * generate FSE decoding table for one symbol (ll, ml or off)
+ * this function must be called with valid parameters only
+ * (dt is large enough, normalizedCounter distribution total is a power of 2, max is within range, etc.)
+ * in which case it cannot fail.
+ * The workspace must be 4-byte aligned and at least ZSTD_BUILD_FSE_TABLE_WKSP_SIZE bytes, which is
+ * defined in zstd_decompress_internal.h.
+ * Internal use only.
+ */
+void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
+ const short* normalizedCounter, unsigned maxSymbolValue,
+ const U32* baseValue, const U32* nbAdditionalBits,
+ unsigned tableLog, void* wksp, size_t wkspSize,
+ int bmi2);
+
+
+#endif /* ZSTD_DEC_BLOCK_H */
--- /dev/null
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+
+/* zstd_decompress_internal:
+ * objects and definitions shared within lib/decompress modules */
+
+ #ifndef ZSTD_DECOMPRESS_INTERNAL_H
+ #define ZSTD_DECOMPRESS_INTERNAL_H
+
+
+/*-*******************************************************
+ * Dependencies
+ *********************************************************/
+#include "../common/mem.h" /* BYTE, U16, U32 */
+#include "../common/zstd_internal.h" /* ZSTD_seqSymbol */
+
+
+
+/*-*******************************************************
+ * Constants
+ *********************************************************/
+static UNUSED_ATTR const U32 LL_base[MaxLL+1] = {
+ 0, 1, 2, 3, 4, 5, 6, 7,
+ 8, 9, 10, 11, 12, 13, 14, 15,
+ 16, 18, 20, 22, 24, 28, 32, 40,
+ 48, 64, 0x80, 0x100, 0x200, 0x400, 0x800, 0x1000,
+ 0x2000, 0x4000, 0x8000, 0x10000 };
+
+static UNUSED_ATTR const U32 OF_base[MaxOff+1] = {
+ 0, 1, 1, 5, 0xD, 0x1D, 0x3D, 0x7D,
+ 0xFD, 0x1FD, 0x3FD, 0x7FD, 0xFFD, 0x1FFD, 0x3FFD, 0x7FFD,
+ 0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD, 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD,
+ 0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD, 0x1FFFFFFD, 0x3FFFFFFD, 0x7FFFFFFD };
+
+static UNUSED_ATTR const U32 OF_bits[MaxOff+1] = {
+ 0, 1, 2, 3, 4, 5, 6, 7,
+ 8, 9, 10, 11, 12, 13, 14, 15,
+ 16, 17, 18, 19, 20, 21, 22, 23,
+ 24, 25, 26, 27, 28, 29, 30, 31 };
+
+static UNUSED_ATTR const U32 ML_base[MaxML+1] = {
+ 3, 4, 5, 6, 7, 8, 9, 10,
+ 11, 12, 13, 14, 15, 16, 17, 18,
+ 19, 20, 21, 22, 23, 24, 25, 26,
+ 27, 28, 29, 30, 31, 32, 33, 34,
+ 35, 37, 39, 41, 43, 47, 51, 59,
+ 67, 83, 99, 0x83, 0x103, 0x203, 0x403, 0x803,
+ 0x1003, 0x2003, 0x4003, 0x8003, 0x10003 };
+
+
+/*-*******************************************************
+ * Decompression types
+ *********************************************************/
+ typedef struct {
+ U32 fastMode;
+ U32 tableLog;
+ } ZSTD_seqSymbol_header;
+
+ typedef struct {
+ U16 nextState;
+ BYTE nbAdditionalBits;
+ BYTE nbBits;
+ U32 baseValue;
+ } ZSTD_seqSymbol;
+
+ #define SEQSYMBOL_TABLE_SIZE(log) (1 + (1 << (log)))
+
+#define ZSTD_BUILD_FSE_TABLE_WKSP_SIZE (sizeof(S16) * (MaxSeq + 1) + (1u << MaxFSELog) + sizeof(U64))
+#define ZSTD_BUILD_FSE_TABLE_WKSP_SIZE_U32 ((ZSTD_BUILD_FSE_TABLE_WKSP_SIZE + sizeof(U32) - 1) / sizeof(U32))
+
+typedef struct {
+ ZSTD_seqSymbol LLTable[SEQSYMBOL_TABLE_SIZE(LLFSELog)]; /* Note : Space reserved for FSE Tables */
+ ZSTD_seqSymbol OFTable[SEQSYMBOL_TABLE_SIZE(OffFSELog)]; /* is also used as temporary workspace while building hufTable during DDict creation */
+ ZSTD_seqSymbol MLTable[SEQSYMBOL_TABLE_SIZE(MLFSELog)]; /* and therefore must be at least HUF_DECOMPRESS_WORKSPACE_SIZE large */
+ HUF_DTable hufTable[HUF_DTABLE_SIZE(HufLog)]; /* can accommodate HUF_decompress4X */
+ U32 rep[ZSTD_REP_NUM];
+ U32 workspace[ZSTD_BUILD_FSE_TABLE_WKSP_SIZE_U32];
+} ZSTD_entropyDTables_t;
+
+typedef enum { ZSTDds_getFrameHeaderSize, ZSTDds_decodeFrameHeader,
+ ZSTDds_decodeBlockHeader, ZSTDds_decompressBlock,
+ ZSTDds_decompressLastBlock, ZSTDds_checkChecksum,
+ ZSTDds_decodeSkippableHeader, ZSTDds_skipFrame } ZSTD_dStage;
+
+typedef enum { zdss_init=0, zdss_loadHeader,
+ zdss_read, zdss_load, zdss_flush } ZSTD_dStreamStage;
+
+typedef enum {
+ ZSTD_use_indefinitely = -1, /* Use the dictionary indefinitely */
+ ZSTD_dont_use = 0, /* Do not use the dictionary (if one exists free it) */
+ ZSTD_use_once = 1 /* Use the dictionary once and set to ZSTD_dont_use */
+} ZSTD_dictUses_e;
+
+/* Hashset for storing references to multiple ZSTD_DDict within ZSTD_DCtx */
+typedef struct {
+ const ZSTD_DDict** ddictPtrTable;
+ size_t ddictPtrTableSize;
+ size_t ddictPtrCount;
+} ZSTD_DDictHashSet;
+
+struct ZSTD_DCtx_s
+{
+ const ZSTD_seqSymbol* LLTptr;
+ const ZSTD_seqSymbol* MLTptr;
+ const ZSTD_seqSymbol* OFTptr;
+ const HUF_DTable* HUFptr;
+ ZSTD_entropyDTables_t entropy;
+ U32 workspace[HUF_DECOMPRESS_WORKSPACE_SIZE_U32]; /* space needed when building huffman tables */
+ const void* previousDstEnd; /* detect continuity */
+ const void* prefixStart; /* start of current segment */
+ const void* virtualStart; /* virtual start of previous segment if it was just before current one */
+ const void* dictEnd; /* end of previous segment */
+ size_t expected;
+ ZSTD_frameHeader fParams;
+ U64 processedCSize;
+ U64 decodedSize;
+ blockType_e bType; /* used in ZSTD_decompressContinue(), store blockType between block header decoding and block decompression stages */
+ ZSTD_dStage stage;
+ U32 litEntropy;
+ U32 fseEntropy;
+ struct xxh64_state xxhState;
+ size_t headerSize;
+ ZSTD_format_e format;
+ ZSTD_forceIgnoreChecksum_e forceIgnoreChecksum; /* User specified: if == 1, will ignore checksums in compressed frame. Default == 0 */
+ U32 validateChecksum; /* if == 1, will validate checksum. Is == 1 if (fParams.checksumFlag == 1) and (forceIgnoreChecksum == 0). */
+ const BYTE* litPtr;
+ ZSTD_customMem customMem;
+ size_t litSize;
+ size_t rleSize;
+ size_t staticSize;
+ int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
+
+ /* dictionary */
+ ZSTD_DDict* ddictLocal;
+ const ZSTD_DDict* ddict; /* set by ZSTD_initDStream_usingDDict(), or ZSTD_DCtx_refDDict() */
+ U32 dictID;
+ int ddictIsCold; /* if == 1 : dictionary is "new" for working context, and presumed "cold" (not in cpu cache) */
+ ZSTD_dictUses_e dictUses;
+ ZSTD_DDictHashSet* ddictSet; /* Hash set for multiple ddicts */
+ ZSTD_refMultipleDDicts_e refMultipleDDicts; /* User specified: if == 1, will allow references to multiple DDicts. Default == 0 (disabled) */
+
+ /* streaming */
+ ZSTD_dStreamStage streamStage;
+ char* inBuff;
+ size_t inBuffSize;
+ size_t inPos;
+ size_t maxWindowSize;
+ char* outBuff;
+ size_t outBuffSize;
+ size_t outStart;
+ size_t outEnd;
+ size_t lhSize;
+ void* legacyContext;
+ U32 previousLegacyVersion;
+ U32 legacyVersion;
+ U32 hostageByte;
+ int noForwardProgress;
+ ZSTD_bufferMode_e outBufferMode;
+ ZSTD_outBuffer expectedOutBuffer;
+
+ /* workspace */
+ BYTE litBuffer[ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH];
+ BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
+
+ size_t oversizedDuration;
+
+#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+ void const* dictContentBeginForFuzzing;
+ void const* dictContentEndForFuzzing;
+#endif
+
+ /* Tracing */
+}; /* typedef'd to ZSTD_DCtx within "zstd.h" */
+
+
+/*-*******************************************************
+ * Shared internal functions
+ *********************************************************/
+
+/*! ZSTD_loadDEntropy() :
+ * dict : must point at beginning of a valid zstd dictionary.
+ * @return : size of dictionary header (size of magic number + dict ID + entropy tables) */
+size_t ZSTD_loadDEntropy(ZSTD_entropyDTables_t* entropy,
+ const void* const dict, size_t const dictSize);
+
+/*! ZSTD_checkContinuity() :
+ * check if next `dst` follows previous position, where decompression ended.
+ * If yes, do nothing (continue on current segment).
+ * If not, classify previous segment as "external dictionary", and start a new segment.
+ * This function cannot fail. */
+void ZSTD_checkContinuity(ZSTD_DCtx* dctx, const void* dst, size_t dstSize);
+
+
+#endif /* ZSTD_DECOMPRESS_INTERNAL_H */
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause */
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+/*
+ * This file includes every .c file needed for decompression.
+ * It is used by lib/decompress_unzstd.c to include the decompression
+ * source into the translation-unit, so it can be used for kernel
+ * decompression.
+ */
+
+#include "common/debug.c"
+#include "common/entropy_common.c"
+#include "common/error_private.c"
+#include "common/fse_decompress.c"
+#include "common/zstd_common.c"
+#include "decompress/huf_decompress.c"
+#include "decompress/zstd_ddict.c"
+#include "decompress/zstd_decompress.c"
+#include "decompress/zstd_decompress_block.c"
+#include "zstd_decompress_module.c"
+++ /dev/null
-/*
- * Common functions of New Generation Entropy library
- * Copyright (C) 2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-
-/* *************************************
-* Dependencies
-***************************************/
-#include "error_private.h" /* ERR_*, ERROR */
-#include "fse.h"
-#include "huf.h"
-#include "mem.h"
-
-/*=== Version ===*/
-unsigned FSE_versionNumber(void) { return FSE_VERSION_NUMBER; }
-
-/*=== Error Management ===*/
-unsigned FSE_isError(size_t code) { return ERR_isError(code); }
-
-unsigned HUF_isError(size_t code) { return ERR_isError(code); }
-
-/*-**************************************************************
-* FSE NCount encoding-decoding
-****************************************************************/
-size_t FSE_readNCount(short *normalizedCounter, unsigned *maxSVPtr, unsigned *tableLogPtr, const void *headerBuffer, size_t hbSize)
-{
- const BYTE *const istart = (const BYTE *)headerBuffer;
- const BYTE *const iend = istart + hbSize;
- const BYTE *ip = istart;
- int nbBits;
- int remaining;
- int threshold;
- U32 bitStream;
- int bitCount;
- unsigned charnum = 0;
- int previous0 = 0;
-
- if (hbSize < 4)
- return ERROR(srcSize_wrong);
- bitStream = ZSTD_readLE32(ip);
- nbBits = (bitStream & 0xF) + FSE_MIN_TABLELOG; /* extract tableLog */
- if (nbBits > FSE_TABLELOG_ABSOLUTE_MAX)
- return ERROR(tableLog_tooLarge);
- bitStream >>= 4;
- bitCount = 4;
- *tableLogPtr = nbBits;
- remaining = (1 << nbBits) + 1;
- threshold = 1 << nbBits;
- nbBits++;
-
- while ((remaining > 1) & (charnum <= *maxSVPtr)) {
- if (previous0) {
- unsigned n0 = charnum;
- while ((bitStream & 0xFFFF) == 0xFFFF) {
- n0 += 24;
- if (ip < iend - 5) {
- ip += 2;
- bitStream = ZSTD_readLE32(ip) >> bitCount;
- } else {
- bitStream >>= 16;
- bitCount += 16;
- }
- }
- while ((bitStream & 3) == 3) {
- n0 += 3;
- bitStream >>= 2;
- bitCount += 2;
- }
- n0 += bitStream & 3;
- bitCount += 2;
- if (n0 > *maxSVPtr)
- return ERROR(maxSymbolValue_tooSmall);
- while (charnum < n0)
- normalizedCounter[charnum++] = 0;
- if ((ip <= iend - 7) || (ip + (bitCount >> 3) <= iend - 4)) {
- ip += bitCount >> 3;
- bitCount &= 7;
- bitStream = ZSTD_readLE32(ip) >> bitCount;
- } else {
- bitStream >>= 2;
- }
- }
- {
- int const max = (2 * threshold - 1) - remaining;
- int count;
-
- if ((bitStream & (threshold - 1)) < (U32)max) {
- count = bitStream & (threshold - 1);
- bitCount += nbBits - 1;
- } else {
- count = bitStream & (2 * threshold - 1);
- if (count >= threshold)
- count -= max;
- bitCount += nbBits;
- }
-
- count--; /* extra accuracy */
- remaining -= count < 0 ? -count : count; /* -1 means +1 */
- normalizedCounter[charnum++] = (short)count;
- previous0 = !count;
- while (remaining < threshold) {
- nbBits--;
- threshold >>= 1;
- }
-
- if ((ip <= iend - 7) || (ip + (bitCount >> 3) <= iend - 4)) {
- ip += bitCount >> 3;
- bitCount &= 7;
- } else {
- bitCount -= (int)(8 * (iend - 4 - ip));
- ip = iend - 4;
- }
- bitStream = ZSTD_readLE32(ip) >> (bitCount & 31);
- }
- } /* while ((remaining>1) & (charnum<=*maxSVPtr)) */
- if (remaining != 1)
- return ERROR(corruption_detected);
- if (bitCount > 32)
- return ERROR(corruption_detected);
- *maxSVPtr = charnum - 1;
-
- ip += (bitCount + 7) >> 3;
- return ip - istart;
-}
-
-/*! HUF_readStats() :
- Read compact Huffman tree, saved by HUF_writeCTable().
- `huffWeight` is destination buffer.
- `rankStats` is assumed to be a table of at least HUF_TABLELOG_MAX U32.
- @return : size read from `src` , or an error Code .
- Note : Needed by HUF_readCTable() and HUF_readDTableX?() .
-*/
-size_t HUF_readStats_wksp(BYTE *huffWeight, size_t hwSize, U32 *rankStats, U32 *nbSymbolsPtr, U32 *tableLogPtr, const void *src, size_t srcSize, void *workspace, size_t workspaceSize)
-{
- U32 weightTotal;
- const BYTE *ip = (const BYTE *)src;
- size_t iSize;
- size_t oSize;
-
- if (!srcSize)
- return ERROR(srcSize_wrong);
- iSize = ip[0];
- /* memset(huffWeight, 0, hwSize); */ /* is not necessary, even though some analyzer complain ... */
-
- if (iSize >= 128) { /* special header */
- oSize = iSize - 127;
- iSize = ((oSize + 1) / 2);
- if (iSize + 1 > srcSize)
- return ERROR(srcSize_wrong);
- if (oSize >= hwSize)
- return ERROR(corruption_detected);
- ip += 1;
- {
- U32 n;
- for (n = 0; n < oSize; n += 2) {
- huffWeight[n] = ip[n / 2] >> 4;
- huffWeight[n + 1] = ip[n / 2] & 15;
- }
- }
- } else { /* header compressed with FSE (normal case) */
- if (iSize + 1 > srcSize)
- return ERROR(srcSize_wrong);
- oSize = FSE_decompress_wksp(huffWeight, hwSize - 1, ip + 1, iSize, 6, workspace, workspaceSize); /* max (hwSize-1) values decoded, as last one is implied */
- if (FSE_isError(oSize))
- return oSize;
- }
-
- /* collect weight stats */
- memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
- weightTotal = 0;
- {
- U32 n;
- for (n = 0; n < oSize; n++) {
- if (huffWeight[n] >= HUF_TABLELOG_MAX)
- return ERROR(corruption_detected);
- rankStats[huffWeight[n]]++;
- weightTotal += (1 << huffWeight[n]) >> 1;
- }
- }
- if (weightTotal == 0)
- return ERROR(corruption_detected);
-
- /* get last non-null symbol weight (implied, total must be 2^n) */
- {
- U32 const tableLog = BIT_highbit32(weightTotal) + 1;
- if (tableLog > HUF_TABLELOG_MAX)
- return ERROR(corruption_detected);
- *tableLogPtr = tableLog;
- /* determine last weight */
- {
- U32 const total = 1 << tableLog;
- U32 const rest = total - weightTotal;
- U32 const verif = 1 << BIT_highbit32(rest);
- U32 const lastWeight = BIT_highbit32(rest) + 1;
- if (verif != rest)
- return ERROR(corruption_detected); /* last value must be a clean power of 2 */
- huffWeight[oSize] = (BYTE)lastWeight;
- rankStats[lastWeight]++;
- }
- }
-
- /* check tree construction validity */
- if ((rankStats[1] < 2) || (rankStats[1] & 1))
- return ERROR(corruption_detected); /* by construction : at least 2 elts of rank 1, must be even */
-
- /* results */
- *nbSymbolsPtr = (U32)(oSize + 1);
- return iSize + 1;
-}
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-/* Note : this module is expected to remain private, do not expose it */
-
-#ifndef ERROR_H_MODULE
-#define ERROR_H_MODULE
-
-/* ****************************************
-* Dependencies
-******************************************/
-#include <linux/types.h> /* size_t */
-#include <linux/zstd.h> /* enum list */
-
-/* ****************************************
-* Compiler-specific
-******************************************/
-#define ERR_STATIC static __attribute__((unused))
-
-/*-****************************************
-* Customization (error_public.h)
-******************************************/
-typedef ZSTD_ErrorCode ERR_enum;
-#define PREFIX(name) ZSTD_error_##name
-
-/*-****************************************
-* Error codes handling
-******************************************/
-#define ERROR(name) ((size_t)-PREFIX(name))
-
-ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
-
-ERR_STATIC ERR_enum ERR_getErrorCode(size_t code)
-{
- if (!ERR_isError(code))
- return (ERR_enum)0;
- return (ERR_enum)(0 - code);
-}
-
-#endif /* ERROR_H_MODULE */
+++ /dev/null
-/*
- * FSE : Finite State Entropy codec
- * Public Prototypes declaration
- * Copyright (C) 2013-2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-#ifndef FSE_H
-#define FSE_H
-
-/*-*****************************************
-* Dependencies
-******************************************/
-#include <linux/types.h> /* size_t, ptrdiff_t */
-
-/*-*****************************************
-* FSE_PUBLIC_API : control library symbols visibility
-******************************************/
-#define FSE_PUBLIC_API
-
-/*------ Version ------*/
-#define FSE_VERSION_MAJOR 0
-#define FSE_VERSION_MINOR 9
-#define FSE_VERSION_RELEASE 0
-
-#define FSE_LIB_VERSION FSE_VERSION_MAJOR.FSE_VERSION_MINOR.FSE_VERSION_RELEASE
-#define FSE_QUOTE(str) #str
-#define FSE_EXPAND_AND_QUOTE(str) FSE_QUOTE(str)
-#define FSE_VERSION_STRING FSE_EXPAND_AND_QUOTE(FSE_LIB_VERSION)
-
-#define FSE_VERSION_NUMBER (FSE_VERSION_MAJOR * 100 * 100 + FSE_VERSION_MINOR * 100 + FSE_VERSION_RELEASE)
-FSE_PUBLIC_API unsigned FSE_versionNumber(void); /**< library version number; to be used when checking dll version */
-
-/*-*****************************************
-* Tool functions
-******************************************/
-FSE_PUBLIC_API size_t FSE_compressBound(size_t size); /* maximum compressed size */
-
-/* Error Management */
-FSE_PUBLIC_API unsigned FSE_isError(size_t code); /* tells if a return value is an error code */
-
-/*-*****************************************
-* FSE detailed API
-******************************************/
-/*!
-FSE_compress() does the following:
-1. count symbol occurrence from source[] into table count[]
-2. normalize counters so that sum(count[]) == Power_of_2 (2^tableLog)
-3. save normalized counters to memory buffer using writeNCount()
-4. build encoding table 'CTable' from normalized counters
-5. encode the data stream using encoding table 'CTable'
-
-FSE_decompress() does the following:
-1. read normalized counters with readNCount()
-2. build decoding table 'DTable' from normalized counters
-3. decode the data stream using decoding table 'DTable'
-
-The following API allows targeting specific sub-functions for advanced tasks.
-For example, it's possible to compress several blocks using the same 'CTable',
-or to save and provide normalized distribution using external method.
-*/
-
-/* *** COMPRESSION *** */
-/*! FSE_optimalTableLog():
- dynamically downsize 'tableLog' when conditions are met.
- It saves CPU time, by using smaller tables, while preserving or even improving compression ratio.
- @return : recommended tableLog (necessarily <= 'maxTableLog') */
-FSE_PUBLIC_API unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
-
-/*! FSE_normalizeCount():
- normalize counts so that sum(count[]) == Power_of_2 (2^tableLog)
- 'normalizedCounter' is a table of short, of minimum size (maxSymbolValue+1).
- @return : tableLog,
- or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_normalizeCount(short *normalizedCounter, unsigned tableLog, const unsigned *count, size_t srcSize, unsigned maxSymbolValue);
-
-/*! FSE_NCountWriteBound():
- Provides the maximum possible size of an FSE normalized table, given 'maxSymbolValue' and 'tableLog'.
- Typically useful for allocation purpose. */
-FSE_PUBLIC_API size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog);
-
-/*! FSE_writeNCount():
- Compactly save 'normalizedCounter' into 'buffer'.
- @return : size of the compressed table,
- or an errorCode, which can be tested using FSE_isError(). */
-FSE_PUBLIC_API size_t FSE_writeNCount(void *buffer, size_t bufferSize, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog);
-
-/*! Constructor and Destructor of FSE_CTable.
- Note that FSE_CTable size depends on 'tableLog' and 'maxSymbolValue' */
-typedef unsigned FSE_CTable; /* don't allocate that. It's only meant to be more restrictive than void* */
-
-/*! FSE_compress_usingCTable():
- Compress `src` using `ct` into `dst` which must be already allocated.
- @return : size of compressed data (<= `dstCapacity`),
- or 0 if compressed data could not fit into `dst`,
- or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_compress_usingCTable(void *dst, size_t dstCapacity, const void *src, size_t srcSize, const FSE_CTable *ct);
-
-/*!
-Tutorial :
-----------
-The first step is to count all symbols. FSE_count() does this job very fast.
-Result will be saved into 'count', a table of unsigned int, which must be already allocated, and have 'maxSymbolValuePtr[0]+1' cells.
-'src' is a table of bytes of size 'srcSize'. All values within 'src' MUST be <= maxSymbolValuePtr[0]
-maxSymbolValuePtr[0] will be updated, with its real value (necessarily <= original value)
-FSE_count() will return the number of occurrence of the most frequent symbol.
-This can be used to know if there is a single symbol within 'src', and to quickly evaluate its compressibility.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-The next step is to normalize the frequencies.
-FSE_normalizeCount() will ensure that sum of frequencies is == 2 ^'tableLog'.
-It also guarantees a minimum of 1 to any Symbol with frequency >= 1.
-You can use 'tableLog'==0 to mean "use default tableLog value".
-If you are unsure of which tableLog value to use, you can ask FSE_optimalTableLog(),
-which will provide the optimal valid tableLog given sourceSize, maxSymbolValue, and a user-defined maximum (0 means "default").
-
-The result of FSE_normalizeCount() will be saved into a table,
-called 'normalizedCounter', which is a table of signed short.
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValue+1' cells.
-The return value is tableLog if everything proceeded as expected.
-It is 0 if there is a single symbol within distribution.
-If there is an error (ex: invalid tableLog value), the function will return an ErrorCode (which can be tested using FSE_isError()).
-
-'normalizedCounter' can be saved in a compact manner to a memory area using FSE_writeNCount().
-'buffer' must be already allocated.
-For guaranteed success, buffer size must be at least FSE_headerBound().
-The result of the function is the number of bytes written into 'buffer'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError(); ex : buffer size too small).
-
-'normalizedCounter' can then be used to create the compression table 'CTable'.
-The space required by 'CTable' must be already allocated, using FSE_createCTable().
-You can then use FSE_buildCTable() to fill 'CTable'.
-If there is an error, both functions will return an ErrorCode (which can be tested using FSE_isError()).
-
-'CTable' can then be used to compress 'src', with FSE_compress_usingCTable().
-Similar to FSE_count(), the convention is that 'src' is assumed to be a table of char of size 'srcSize'
-The function returns the size of compressed data (without header), necessarily <= `dstCapacity`.
-If it returns '0', compressed data could not fit into 'dst'.
-If there is an error, the function will return an ErrorCode (which can be tested using FSE_isError()).
-*/
-
-/* *** DECOMPRESSION *** */
-
-/*! FSE_readNCount():
- Read compactly saved 'normalizedCounter' from 'rBuffer'.
- @return : size read from 'rBuffer',
- or an errorCode, which can be tested using FSE_isError().
- maxSymbolValuePtr[0] and tableLogPtr[0] will also be updated with their respective values */
-FSE_PUBLIC_API size_t FSE_readNCount(short *normalizedCounter, unsigned *maxSymbolValuePtr, unsigned *tableLogPtr, const void *rBuffer, size_t rBuffSize);
-
-/*! Constructor and Destructor of FSE_DTable.
- Note that its size depends on 'tableLog' */
-typedef unsigned FSE_DTable; /* don't allocate that. It's just a way to be more restrictive than void* */
-
-/*! FSE_buildDTable():
- Builds 'dt', which must be already allocated, using FSE_createDTable().
- return : 0, or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_buildDTable_wksp(FSE_DTable *dt, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void *workspace, size_t workspaceSize);
-
-/*! FSE_decompress_usingDTable():
- Decompress compressed source `cSrc` of size `cSrcSize` using `dt`
- into `dst` which must be already allocated.
- @return : size of regenerated data (necessarily <= `dstCapacity`),
- or an errorCode, which can be tested using FSE_isError() */
-FSE_PUBLIC_API size_t FSE_decompress_usingDTable(void *dst, size_t dstCapacity, const void *cSrc, size_t cSrcSize, const FSE_DTable *dt);
-
-/*!
-Tutorial :
-----------
-(Note : these functions only decompress FSE-compressed blocks.
- If block is uncompressed, use memcpy() instead
- If block is a single repeated byte, use memset() instead )
-
-The first step is to obtain the normalized frequencies of symbols.
-This can be performed by FSE_readNCount() if it was saved using FSE_writeNCount().
-'normalizedCounter' must be already allocated, and have at least 'maxSymbolValuePtr[0]+1' cells of signed short.
-In practice, that means it's necessary to know 'maxSymbolValue' beforehand,
-or size the table to handle worst case situations (typically 256).
-FSE_readNCount() will provide 'tableLog' and 'maxSymbolValue'.
-The result of FSE_readNCount() is the number of bytes read from 'rBuffer'.
-Note that 'rBufferSize' must be at least 4 bytes, even if useful information is less than that.
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-The next step is to build the decompression tables 'FSE_DTable' from 'normalizedCounter'.
-This is performed by the function FSE_buildDTable().
-The space required by 'FSE_DTable' must be already allocated using FSE_createDTable().
-If there is an error, the function will return an error code, which can be tested using FSE_isError().
-
-`FSE_DTable` can then be used to decompress `cSrc`, with FSE_decompress_usingDTable().
-`cSrcSize` must be strictly correct, otherwise decompression will fail.
-FSE_decompress_usingDTable() result will tell how many bytes were regenerated (<=`dstCapacity`).
-If there is an error, the function will return an error code, which can be tested using FSE_isError(). (ex: dst buffer too small)
-*/
-
-/* *** Dependency *** */
-#include "bitstream.h"
-
-/* *****************************************
-* Static allocation
-*******************************************/
-/* FSE buffer bounds */
-#define FSE_NCOUNTBOUND 512
-#define FSE_BLOCKBOUND(size) (size + (size >> 7))
-#define FSE_COMPRESSBOUND(size) (FSE_NCOUNTBOUND + FSE_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
-
-/* It is possible to statically allocate FSE CTable/DTable as a table of FSE_CTable/FSE_DTable using below macros */
-#define FSE_CTABLE_SIZE_U32(maxTableLog, maxSymbolValue) (1 + (1 << (maxTableLog - 1)) + ((maxSymbolValue + 1) * 2))
-#define FSE_DTABLE_SIZE_U32(maxTableLog) (1 + (1 << maxTableLog))
-
-/* *****************************************
-* FSE advanced API
-*******************************************/
-/* FSE_count_wksp() :
- * Same as FSE_count(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned
- */
-size_t FSE_count_wksp(unsigned *count, unsigned *maxSymbolValuePtr, const void *source, size_t sourceSize, unsigned *workSpace);
-
-/* FSE_countFast_wksp() :
- * Same as FSE_countFast(), but using an externally provided scratch buffer.
- * `workSpace` must be a table of minimum `1024` unsigned
- */
-size_t FSE_countFast_wksp(unsigned *count, unsigned *maxSymbolValuePtr, const void *src, size_t srcSize, unsigned *workSpace);
-
-/*! FSE_count_simple
- * Same as FSE_countFast(), but does not use any additional memory (not even on stack).
- * This function is unsafe, and will segfault if any value within `src` is `> *maxSymbolValuePtr` (presuming it's also the size of `count`).
-*/
-size_t FSE_count_simple(unsigned *count, unsigned *maxSymbolValuePtr, const void *src, size_t srcSize);
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus);
-/**< same as FSE_optimalTableLog(), which used `minus==2` */
-
-size_t FSE_buildCTable_raw(FSE_CTable *ct, unsigned nbBits);
-/**< build a fake FSE_CTable, designed for a flat distribution, where each symbol uses nbBits */
-
-size_t FSE_buildCTable_rle(FSE_CTable *ct, unsigned char symbolValue);
-/**< build a fake FSE_CTable, designed to compress always the same symbolValue */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * `wkspSize` must be >= `(1<<tableLog)`.
- */
-size_t FSE_buildCTable_wksp(FSE_CTable *ct, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void *workSpace, size_t wkspSize);
-
-size_t FSE_buildDTable_raw(FSE_DTable *dt, unsigned nbBits);
-/**< build a fake FSE_DTable, designed to read a flat distribution where each symbol uses nbBits */
-
-size_t FSE_buildDTable_rle(FSE_DTable *dt, unsigned char symbolValue);
-/**< build a fake FSE_DTable, designed to always generate the same symbolValue */
-
-size_t FSE_decompress_wksp(void *dst, size_t dstCapacity, const void *cSrc, size_t cSrcSize, unsigned maxLog, void *workspace, size_t workspaceSize);
-/**< same as FSE_decompress(), using an externally allocated `workSpace` produced with `FSE_DTABLE_SIZE_U32(maxLog)` */
-
-/* *****************************************
-* FSE symbol compression API
-*******************************************/
-/*!
- This API consists of small unitary functions, which highly benefit from being inlined.
- Hence their body are included in next section.
-*/
-typedef struct {
- ptrdiff_t value;
- const void *stateTable;
- const void *symbolTT;
- unsigned stateLog;
-} FSE_CState_t;
-
-static void FSE_initCState(FSE_CState_t *CStatePtr, const FSE_CTable *ct);
-
-static void FSE_encodeSymbol(BIT_CStream_t *bitC, FSE_CState_t *CStatePtr, unsigned symbol);
-
-static void FSE_flushCState(BIT_CStream_t *bitC, const FSE_CState_t *CStatePtr);
-
-/**<
-These functions are inner components of FSE_compress_usingCTable().
-They allow the creation of custom streams, mixing multiple tables and bit sources.
-
-A key property to keep in mind is that encoding and decoding are done **in reverse direction**.
-So the first symbol you will encode is the last you will decode, like a LIFO stack.
-
-You will need a few variables to track your CStream. They are :
-
-FSE_CTable ct; // Provided by FSE_buildCTable()
-BIT_CStream_t bitStream; // bitStream tracking structure
-FSE_CState_t state; // State tracking structure (can have several)
-
-
-The first thing to do is to init bitStream and state.
- size_t errorCode = BIT_initCStream(&bitStream, dstBuffer, maxDstSize);
- FSE_initCState(&state, ct);
-
-Note that BIT_initCStream() can produce an error code, so its result should be tested, using FSE_isError();
-You can then encode your input data, byte after byte.
-FSE_encodeSymbol() outputs a maximum of 'tableLog' bits at a time.
-Remember decoding will be done in reverse direction.
- FSE_encodeByte(&bitStream, &state, symbol);
-
-At any time, you can also add any bit sequence.
-Note : maximum allowed nbBits is 25, for compatibility with 32-bits decoders
- BIT_addBits(&bitStream, bitField, nbBits);
-
-The above methods don't commit data to memory, they just store it into local register, for speed.
-Local register size is 64-bits on 64-bits systems, 32-bits on 32-bits systems (size_t).
-Writing data to memory is a manual operation, performed by the flushBits function.
- BIT_flushBits(&bitStream);
-
-Your last FSE encoding operation shall be to flush your last state value(s).
- FSE_flushState(&bitStream, &state);
-
-Finally, you must close the bitStream.
-The function returns the size of CStream in bytes.
-If data couldn't fit into dstBuffer, it will return a 0 ( == not compressible)
-If there is an error, it returns an errorCode (which can be tested using FSE_isError()).
- size_t size = BIT_closeCStream(&bitStream);
-*/
-
-/* *****************************************
-* FSE symbol decompression API
-*******************************************/
-typedef struct {
- size_t state;
- const void *table; /* precise table may vary, depending on U16 */
-} FSE_DState_t;
-
-static void FSE_initDState(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD, const FSE_DTable *dt);
-
-static unsigned char FSE_decodeSymbol(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD);
-
-static unsigned FSE_endOfDState(const FSE_DState_t *DStatePtr);
-
-/**<
-Let's now decompose FSE_decompress_usingDTable() into its unitary components.
-You will decode FSE-encoded symbols from the bitStream,
-and also any other bitFields you put in, **in reverse order**.
-
-You will need a few variables to track your bitStream. They are :
-
-BIT_DStream_t DStream; // Stream context
-FSE_DState_t DState; // State context. Multiple ones are possible
-FSE_DTable* DTablePtr; // Decoding table, provided by FSE_buildDTable()
-
-The first thing to do is to init the bitStream.
- errorCode = BIT_initDStream(&DStream, srcBuffer, srcSize);
-
-You should then retrieve your initial state(s)
-(in reverse flushing order if you have several ones) :
- errorCode = FSE_initDState(&DState, &DStream, DTablePtr);
-
-You can then decode your data, symbol after symbol.
-For information the maximum number of bits read by FSE_decodeSymbol() is 'tableLog'.
-Keep in mind that symbols are decoded in reverse order, like a LIFO stack (last in, first out).
- unsigned char symbol = FSE_decodeSymbol(&DState, &DStream);
-
-You can retrieve any bitfield you eventually stored into the bitStream (in reverse order)
-Note : maximum allowed nbBits is 25, for 32-bits compatibility
- size_t bitField = BIT_readBits(&DStream, nbBits);
-
-All above operations only read from local register (which size depends on size_t).
-Refueling the register from memory is manually performed by the reload method.
- endSignal = FSE_reloadDStream(&DStream);
-
-BIT_reloadDStream() result tells if there is still some more data to read from DStream.
-BIT_DStream_unfinished : there is still some data left into the DStream.
-BIT_DStream_endOfBuffer : Dstream reached end of buffer. Its container may no longer be completely filled.
-BIT_DStream_completed : Dstream reached its exact end, corresponding in general to decompression completed.
-BIT_DStream_tooFar : Dstream went too far. Decompression result is corrupted.
-
-When reaching end of buffer (BIT_DStream_endOfBuffer), progress slowly, notably if you decode multiple symbols per loop,
-to properly detect the exact end of stream.
-After each decoded symbol, check if DStream is fully consumed using this simple test :
- BIT_reloadDStream(&DStream) >= BIT_DStream_completed
-
-When it's done, verify decompression is fully completed, by checking both DStream and the relevant states.
-Checking if DStream has reached its end is performed by :
- BIT_endOfDStream(&DStream);
-Check also the states. There might be some symbols left there, if some high probability ones (>50%) are possible.
- FSE_endOfDState(&DState);
-*/
-
-/* *****************************************
-* FSE unsafe API
-*******************************************/
-static unsigned char FSE_decodeSymbolFast(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD);
-/* faster, but works only if nbBits is always >= 1 (otherwise, result will be corrupted) */
-
-/* *****************************************
-* Implementation of inlined functions
-*******************************************/
-typedef struct {
- int deltaFindState;
- U32 deltaNbBits;
-} FSE_symbolCompressionTransform; /* total 8 bytes */
-
-ZSTD_STATIC void FSE_initCState(FSE_CState_t *statePtr, const FSE_CTable *ct)
-{
- const void *ptr = ct;
- const U16 *u16ptr = (const U16 *)ptr;
- const U32 tableLog = ZSTD_read16(ptr);
- statePtr->value = (ptrdiff_t)1 << tableLog;
- statePtr->stateTable = u16ptr + 2;
- statePtr->symbolTT = ((const U32 *)ct + 1 + (tableLog ? (1 << (tableLog - 1)) : 1));
- statePtr->stateLog = tableLog;
-}
-
-/*! FSE_initCState2() :
-* Same as FSE_initCState(), but the first symbol to include (which will be the last to be read)
-* uses the smallest state value possible, saving the cost of this symbol */
-ZSTD_STATIC void FSE_initCState2(FSE_CState_t *statePtr, const FSE_CTable *ct, U32 symbol)
-{
- FSE_initCState(statePtr, ct);
- {
- const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform *)(statePtr->symbolTT))[symbol];
- const U16 *stateTable = (const U16 *)(statePtr->stateTable);
- U32 nbBitsOut = (U32)((symbolTT.deltaNbBits + (1 << 15)) >> 16);
- statePtr->value = (nbBitsOut << 16) - symbolTT.deltaNbBits;
- statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
- }
-}
-
-ZSTD_STATIC void FSE_encodeSymbol(BIT_CStream_t *bitC, FSE_CState_t *statePtr, U32 symbol)
-{
- const FSE_symbolCompressionTransform symbolTT = ((const FSE_symbolCompressionTransform *)(statePtr->symbolTT))[symbol];
- const U16 *const stateTable = (const U16 *)(statePtr->stateTable);
- U32 nbBitsOut = (U32)((statePtr->value + symbolTT.deltaNbBits) >> 16);
- BIT_addBits(bitC, statePtr->value, nbBitsOut);
- statePtr->value = stateTable[(statePtr->value >> nbBitsOut) + symbolTT.deltaFindState];
-}
-
-ZSTD_STATIC void FSE_flushCState(BIT_CStream_t *bitC, const FSE_CState_t *statePtr)
-{
- BIT_addBits(bitC, statePtr->value, statePtr->stateLog);
- BIT_flushBits(bitC);
-}
-
-/* ====== Decompression ====== */
-
-typedef struct {
- U16 tableLog;
- U16 fastMode;
-} FSE_DTableHeader; /* sizeof U32 */
-
-typedef struct {
- unsigned short newState;
- unsigned char symbol;
- unsigned char nbBits;
-} FSE_decode_t; /* size == U32 */
-
-ZSTD_STATIC void FSE_initDState(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD, const FSE_DTable *dt)
-{
- const void *ptr = dt;
- const FSE_DTableHeader *const DTableH = (const FSE_DTableHeader *)ptr;
- DStatePtr->state = BIT_readBits(bitD, DTableH->tableLog);
- BIT_reloadDStream(bitD);
- DStatePtr->table = dt + 1;
-}
-
-ZSTD_STATIC BYTE FSE_peekSymbol(const FSE_DState_t *DStatePtr)
-{
- FSE_decode_t const DInfo = ((const FSE_decode_t *)(DStatePtr->table))[DStatePtr->state];
- return DInfo.symbol;
-}
-
-ZSTD_STATIC void FSE_updateState(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD)
-{
- FSE_decode_t const DInfo = ((const FSE_decode_t *)(DStatePtr->table))[DStatePtr->state];
- U32 const nbBits = DInfo.nbBits;
- size_t const lowBits = BIT_readBits(bitD, nbBits);
- DStatePtr->state = DInfo.newState + lowBits;
-}
-
-ZSTD_STATIC BYTE FSE_decodeSymbol(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD)
-{
- FSE_decode_t const DInfo = ((const FSE_decode_t *)(DStatePtr->table))[DStatePtr->state];
- U32 const nbBits = DInfo.nbBits;
- BYTE const symbol = DInfo.symbol;
- size_t const lowBits = BIT_readBits(bitD, nbBits);
-
- DStatePtr->state = DInfo.newState + lowBits;
- return symbol;
-}
-
-/*! FSE_decodeSymbolFast() :
- unsafe, only works if no symbol has a probability > 50% */
-ZSTD_STATIC BYTE FSE_decodeSymbolFast(FSE_DState_t *DStatePtr, BIT_DStream_t *bitD)
-{
- FSE_decode_t const DInfo = ((const FSE_decode_t *)(DStatePtr->table))[DStatePtr->state];
- U32 const nbBits = DInfo.nbBits;
- BYTE const symbol = DInfo.symbol;
- size_t const lowBits = BIT_readBitsFast(bitD, nbBits);
-
- DStatePtr->state = DInfo.newState + lowBits;
- return symbol;
-}
-
-ZSTD_STATIC unsigned FSE_endOfDState(const FSE_DState_t *DStatePtr) { return DStatePtr->state == 0; }
-
-/* **************************************************************
-* Tuning parameters
-****************************************************************/
-/*!MEMORY_USAGE :
-* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
-* Increasing memory usage improves compression ratio
-* Reduced memory usage can improve speed, due to cache effect
-* Recommended max value is 14, for 16KB, which nicely fits into Intel x86 L1 cache */
-#ifndef FSE_MAX_MEMORY_USAGE
-#define FSE_MAX_MEMORY_USAGE 14
-#endif
-#ifndef FSE_DEFAULT_MEMORY_USAGE
-#define FSE_DEFAULT_MEMORY_USAGE 13
-#endif
-
-/*!FSE_MAX_SYMBOL_VALUE :
-* Maximum symbol value authorized.
-* Required for proper stack allocation */
-#ifndef FSE_MAX_SYMBOL_VALUE
-#define FSE_MAX_SYMBOL_VALUE 255
-#endif
-
-/* **************************************************************
-* template functions type & suffix
-****************************************************************/
-#define FSE_FUNCTION_TYPE BYTE
-#define FSE_FUNCTION_EXTENSION
-#define FSE_DECODE_TYPE FSE_decode_t
-
-/* ***************************************************************
-* Constants
-*****************************************************************/
-#define FSE_MAX_TABLELOG (FSE_MAX_MEMORY_USAGE - 2)
-#define FSE_MAX_TABLESIZE (1U << FSE_MAX_TABLELOG)
-#define FSE_MAXTABLESIZE_MASK (FSE_MAX_TABLESIZE - 1)
-#define FSE_DEFAULT_TABLELOG (FSE_DEFAULT_MEMORY_USAGE - 2)
-#define FSE_MIN_TABLELOG 5
-
-#define FSE_TABLELOG_ABSOLUTE_MAX 15
-#if FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX
-#error "FSE_MAX_TABLELOG > FSE_TABLELOG_ABSOLUTE_MAX is not supported"
-#endif
-
-#define FSE_TABLESTEP(tableSize) ((tableSize >> 1) + (tableSize >> 3) + 3)
-
-#endif /* FSE_H */
+++ /dev/null
-/*
- * FSE : Finite State Entropy encoder
- * Copyright (C) 2013-2015, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-
-/* **************************************************************
-* Compiler specifics
-****************************************************************/
-#define FORCE_INLINE static __always_inline
-
-/* **************************************************************
-* Includes
-****************************************************************/
-#include "bitstream.h"
-#include "fse.h"
-#include <linux/compiler.h>
-#include <linux/kernel.h>
-#include <linux/math64.h>
-#include <linux/string.h> /* memcpy, memset */
-
-/* **************************************************************
-* Error Management
-****************************************************************/
-#define FSE_STATIC_ASSERT(c) \
- { \
- enum { FSE_static_assert = 1 / (int)(!!(c)) }; \
- } /* use only *after* variable declarations */
-
-/* **************************************************************
-* Templates
-****************************************************************/
-/*
- designed to be included
- for type-specific functions (template emulation in C)
- Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X, Y) X##Y
-#define FSE_FUNCTION_NAME(X, Y) FSE_CAT(X, Y)
-#define FSE_TYPE_NAME(X, Y) FSE_CAT(X, Y)
-
-/* Function templates */
-
-/* FSE_buildCTable_wksp() :
- * Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
- * wkspSize should be sized to handle worst case situation, which is `1<<max_tableLog * sizeof(FSE_FUNCTION_TYPE)`
- * workSpace must also be properly aligned with FSE_FUNCTION_TYPE requirements
- */
-size_t FSE_buildCTable_wksp(FSE_CTable *ct, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void *workspace, size_t workspaceSize)
-{
- U32 const tableSize = 1 << tableLog;
- U32 const tableMask = tableSize - 1;
- void *const ptr = ct;
- U16 *const tableU16 = ((U16 *)ptr) + 2;
- void *const FSCT = ((U32 *)ptr) + 1 /* header */ + (tableLog ? tableSize >> 1 : 1);
- FSE_symbolCompressionTransform *const symbolTT = (FSE_symbolCompressionTransform *)(FSCT);
- U32 const step = FSE_TABLESTEP(tableSize);
- U32 highThreshold = tableSize - 1;
-
- U32 *cumul;
- FSE_FUNCTION_TYPE *tableSymbol;
- size_t spaceUsed32 = 0;
-
- cumul = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += FSE_MAX_SYMBOL_VALUE + 2;
- tableSymbol = (FSE_FUNCTION_TYPE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(sizeof(FSE_FUNCTION_TYPE) * ((size_t)1 << tableLog), sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- /* CTable header */
- tableU16[-2] = (U16)tableLog;
- tableU16[-1] = (U16)maxSymbolValue;
-
- /* For explanations on how to distribute symbol values over the table :
- * http://fastcompression.blogspot.fr/2014/02/fse-distributing-symbol-values.html */
-
- /* symbol start positions */
- {
- U32 u;
- cumul[0] = 0;
- for (u = 1; u <= maxSymbolValue + 1; u++) {
- if (normalizedCounter[u - 1] == -1) { /* Low proba symbol */
- cumul[u] = cumul[u - 1] + 1;
- tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u - 1);
- } else {
- cumul[u] = cumul[u - 1] + normalizedCounter[u - 1];
- }
- }
- cumul[maxSymbolValue + 1] = tableSize + 1;
- }
-
- /* Spread symbols */
- {
- U32 position = 0;
- U32 symbol;
- for (symbol = 0; symbol <= maxSymbolValue; symbol++) {
- int nbOccurences;
- for (nbOccurences = 0; nbOccurences < normalizedCounter[symbol]; nbOccurences++) {
- tableSymbol[position] = (FSE_FUNCTION_TYPE)symbol;
- position = (position + step) & tableMask;
- while (position > highThreshold)
- position = (position + step) & tableMask; /* Low proba area */
- }
- }
-
- if (position != 0)
- return ERROR(GENERIC); /* Must have gone through all positions */
- }
-
- /* Build table */
- {
- U32 u;
- for (u = 0; u < tableSize; u++) {
- FSE_FUNCTION_TYPE s = tableSymbol[u]; /* note : static analyzer may not understand tableSymbol is properly initialized */
- tableU16[cumul[s]++] = (U16)(tableSize + u); /* TableU16 : sorted by symbol order; gives next state value */
- }
- }
-
- /* Build Symbol Transformation Table */
- {
- unsigned total = 0;
- unsigned s;
- for (s = 0; s <= maxSymbolValue; s++) {
- switch (normalizedCounter[s]) {
- case 0: break;
-
- case -1:
- case 1:
- symbolTT[s].deltaNbBits = (tableLog << 16) - (1 << tableLog);
- symbolTT[s].deltaFindState = total - 1;
- total++;
- break;
- default: {
- U32 const maxBitsOut = tableLog - BIT_highbit32(normalizedCounter[s] - 1);
- U32 const minStatePlus = normalizedCounter[s] << maxBitsOut;
- symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
- symbolTT[s].deltaFindState = total - normalizedCounter[s];
- total += normalizedCounter[s];
- }
- }
- }
- }
-
- return 0;
-}
-
-/*-**************************************************************
-* FSE NCount encoding-decoding
-****************************************************************/
-size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
-{
- size_t const maxHeaderSize = (((maxSymbolValue + 1) * tableLog) >> 3) + 3;
- return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */
-}
-
-static size_t FSE_writeNCount_generic(void *header, size_t headerBufferSize, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog,
- unsigned writeIsSafe)
-{
- BYTE *const ostart = (BYTE *)header;
- BYTE *out = ostart;
- BYTE *const oend = ostart + headerBufferSize;
- int nbBits;
- const int tableSize = 1 << tableLog;
- int remaining;
- int threshold;
- U32 bitStream;
- int bitCount;
- unsigned charnum = 0;
- int previous0 = 0;
-
- bitStream = 0;
- bitCount = 0;
- /* Table Size */
- bitStream += (tableLog - FSE_MIN_TABLELOG) << bitCount;
- bitCount += 4;
-
- /* Init */
- remaining = tableSize + 1; /* +1 for extra accuracy */
- threshold = tableSize;
- nbBits = tableLog + 1;
-
- while (remaining > 1) { /* stops at 1 */
- if (previous0) {
- unsigned start = charnum;
- while (!normalizedCounter[charnum])
- charnum++;
- while (charnum >= start + 24) {
- start += 24;
- bitStream += 0xFFFFU << bitCount;
- if ((!writeIsSafe) && (out > oend - 2))
- return ERROR(dstSize_tooSmall); /* Buffer overflow */
- out[0] = (BYTE)bitStream;
- out[1] = (BYTE)(bitStream >> 8);
- out += 2;
- bitStream >>= 16;
- }
- while (charnum >= start + 3) {
- start += 3;
- bitStream += 3 << bitCount;
- bitCount += 2;
- }
- bitStream += (charnum - start) << bitCount;
- bitCount += 2;
- if (bitCount > 16) {
- if ((!writeIsSafe) && (out > oend - 2))
- return ERROR(dstSize_tooSmall); /* Buffer overflow */
- out[0] = (BYTE)bitStream;
- out[1] = (BYTE)(bitStream >> 8);
- out += 2;
- bitStream >>= 16;
- bitCount -= 16;
- }
- }
- {
- int count = normalizedCounter[charnum++];
- int const max = (2 * threshold - 1) - remaining;
- remaining -= count < 0 ? -count : count;
- count++; /* +1 for extra accuracy */
- if (count >= threshold)
- count += max; /* [0..max[ [max..threshold[ (...) [threshold+max 2*threshold[ */
- bitStream += count << bitCount;
- bitCount += nbBits;
- bitCount -= (count < max);
- previous0 = (count == 1);
- if (remaining < 1)
- return ERROR(GENERIC);
- while (remaining < threshold)
- nbBits--, threshold >>= 1;
- }
- if (bitCount > 16) {
- if ((!writeIsSafe) && (out > oend - 2))
- return ERROR(dstSize_tooSmall); /* Buffer overflow */
- out[0] = (BYTE)bitStream;
- out[1] = (BYTE)(bitStream >> 8);
- out += 2;
- bitStream >>= 16;
- bitCount -= 16;
- }
- }
-
- /* flush remaining bitStream */
- if ((!writeIsSafe) && (out > oend - 2))
- return ERROR(dstSize_tooSmall); /* Buffer overflow */
- out[0] = (BYTE)bitStream;
- out[1] = (BYTE)(bitStream >> 8);
- out += (bitCount + 7) / 8;
-
- if (charnum > maxSymbolValue + 1)
- return ERROR(GENERIC);
-
- return (out - ostart);
-}
-
-size_t FSE_writeNCount(void *buffer, size_t bufferSize, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog)
-{
- if (tableLog > FSE_MAX_TABLELOG)
- return ERROR(tableLog_tooLarge); /* Unsupported */
- if (tableLog < FSE_MIN_TABLELOG)
- return ERROR(GENERIC); /* Unsupported */
-
- if (bufferSize < FSE_NCountWriteBound(maxSymbolValue, tableLog))
- return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 0);
-
- return FSE_writeNCount_generic(buffer, bufferSize, normalizedCounter, maxSymbolValue, tableLog, 1);
-}
-
-/*-**************************************************************
-* Counting histogram
-****************************************************************/
-/*! FSE_count_simple
- This function counts byte values within `src`, and store the histogram into table `count`.
- It doesn't use any additional memory.
- But this function is unsafe : it doesn't check that all values within `src` can fit into `count`.
- For this reason, prefer using a table `count` with 256 elements.
- @return : count of most numerous element
-*/
-size_t FSE_count_simple(unsigned *count, unsigned *maxSymbolValuePtr, const void *src, size_t srcSize)
-{
- const BYTE *ip = (const BYTE *)src;
- const BYTE *const end = ip + srcSize;
- unsigned maxSymbolValue = *maxSymbolValuePtr;
- unsigned max = 0;
-
- memset(count, 0, (maxSymbolValue + 1) * sizeof(*count));
- if (srcSize == 0) {
- *maxSymbolValuePtr = 0;
- return 0;
- }
-
- while (ip < end)
- count[*ip++]++;
-
- while (!count[maxSymbolValue])
- maxSymbolValue--;
- *maxSymbolValuePtr = maxSymbolValue;
-
- {
- U32 s;
- for (s = 0; s <= maxSymbolValue; s++)
- if (count[s] > max)
- max = count[s];
- }
-
- return (size_t)max;
-}
-
-/* FSE_count_parallel_wksp() :
- * Same as FSE_count_parallel(), but using an externally provided scratch buffer.
- * `workSpace` size must be a minimum of `1024 * sizeof(unsigned)`` */
-static size_t FSE_count_parallel_wksp(unsigned *count, unsigned *maxSymbolValuePtr, const void *source, size_t sourceSize, unsigned checkMax,
- unsigned *const workSpace)
-{
- const BYTE *ip = (const BYTE *)source;
- const BYTE *const iend = ip + sourceSize;
- unsigned maxSymbolValue = *maxSymbolValuePtr;
- unsigned max = 0;
- U32 *const Counting1 = workSpace;
- U32 *const Counting2 = Counting1 + 256;
- U32 *const Counting3 = Counting2 + 256;
- U32 *const Counting4 = Counting3 + 256;
-
- memset(Counting1, 0, 4 * 256 * sizeof(unsigned));
-
- /* safety checks */
- if (!sourceSize) {
- memset(count, 0, maxSymbolValue + 1);
- *maxSymbolValuePtr = 0;
- return 0;
- }
- if (!maxSymbolValue)
- maxSymbolValue = 255; /* 0 == default */
-
- /* by stripes of 16 bytes */
- {
- U32 cached = ZSTD_read32(ip);
- ip += 4;
- while (ip < iend - 15) {
- U32 c = cached;
- cached = ZSTD_read32(ip);
- ip += 4;
- Counting1[(BYTE)c]++;
- Counting2[(BYTE)(c >> 8)]++;
- Counting3[(BYTE)(c >> 16)]++;
- Counting4[c >> 24]++;
- c = cached;
- cached = ZSTD_read32(ip);
- ip += 4;
- Counting1[(BYTE)c]++;
- Counting2[(BYTE)(c >> 8)]++;
- Counting3[(BYTE)(c >> 16)]++;
- Counting4[c >> 24]++;
- c = cached;
- cached = ZSTD_read32(ip);
- ip += 4;
- Counting1[(BYTE)c]++;
- Counting2[(BYTE)(c >> 8)]++;
- Counting3[(BYTE)(c >> 16)]++;
- Counting4[c >> 24]++;
- c = cached;
- cached = ZSTD_read32(ip);
- ip += 4;
- Counting1[(BYTE)c]++;
- Counting2[(BYTE)(c >> 8)]++;
- Counting3[(BYTE)(c >> 16)]++;
- Counting4[c >> 24]++;
- }
- ip -= 4;
- }
-
- /* finish last symbols */
- while (ip < iend)
- Counting1[*ip++]++;
-
- if (checkMax) { /* verify stats will fit into destination table */
- U32 s;
- for (s = 255; s > maxSymbolValue; s--) {
- Counting1[s] += Counting2[s] + Counting3[s] + Counting4[s];
- if (Counting1[s])
- return ERROR(maxSymbolValue_tooSmall);
- }
- }
-
- {
- U32 s;
- for (s = 0; s <= maxSymbolValue; s++) {
- count[s] = Counting1[s] + Counting2[s] + Counting3[s] + Counting4[s];
- if (count[s] > max)
- max = count[s];
- }
- }
-
- while (!count[maxSymbolValue])
- maxSymbolValue--;
- *maxSymbolValuePtr = maxSymbolValue;
- return (size_t)max;
-}
-
-/* FSE_countFast_wksp() :
- * Same as FSE_countFast(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned */
-size_t FSE_countFast_wksp(unsigned *count, unsigned *maxSymbolValuePtr, const void *source, size_t sourceSize, unsigned *workSpace)
-{
- if (sourceSize < 1500)
- return FSE_count_simple(count, maxSymbolValuePtr, source, sourceSize);
- return FSE_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, 0, workSpace);
-}
-
-/* FSE_count_wksp() :
- * Same as FSE_count(), but using an externally provided scratch buffer.
- * `workSpace` size must be table of >= `1024` unsigned */
-size_t FSE_count_wksp(unsigned *count, unsigned *maxSymbolValuePtr, const void *source, size_t sourceSize, unsigned *workSpace)
-{
- if (*maxSymbolValuePtr < 255)
- return FSE_count_parallel_wksp(count, maxSymbolValuePtr, source, sourceSize, 1, workSpace);
- *maxSymbolValuePtr = 255;
- return FSE_countFast_wksp(count, maxSymbolValuePtr, source, sourceSize, workSpace);
-}
-
-/*-**************************************************************
-* FSE Compression Code
-****************************************************************/
-/*! FSE_sizeof_CTable() :
- FSE_CTable is a variable size structure which contains :
- `U16 tableLog;`
- `U16 maxSymbolValue;`
- `U16 nextStateNumber[1 << tableLog];` // This size is variable
- `FSE_symbolCompressionTransform symbolTT[maxSymbolValue+1];` // This size is variable
-Allocation is manual (C standard does not support variable-size structures).
-*/
-size_t FSE_sizeof_CTable(unsigned maxSymbolValue, unsigned tableLog)
-{
- if (tableLog > FSE_MAX_TABLELOG)
- return ERROR(tableLog_tooLarge);
- return FSE_CTABLE_SIZE_U32(tableLog, maxSymbolValue) * sizeof(U32);
-}
-
-/* provides the minimum logSize to safely represent a distribution */
-static unsigned FSE_minTableLog(size_t srcSize, unsigned maxSymbolValue)
-{
- U32 minBitsSrc = BIT_highbit32((U32)(srcSize - 1)) + 1;
- U32 minBitsSymbols = BIT_highbit32(maxSymbolValue) + 2;
- U32 minBits = minBitsSrc < minBitsSymbols ? minBitsSrc : minBitsSymbols;
- return minBits;
-}
-
-unsigned FSE_optimalTableLog_internal(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue, unsigned minus)
-{
- U32 maxBitsSrc = BIT_highbit32((U32)(srcSize - 1)) - minus;
- U32 tableLog = maxTableLog;
- U32 minBits = FSE_minTableLog(srcSize, maxSymbolValue);
- if (tableLog == 0)
- tableLog = FSE_DEFAULT_TABLELOG;
- if (maxBitsSrc < tableLog)
- tableLog = maxBitsSrc; /* Accuracy can be reduced */
- if (minBits > tableLog)
- tableLog = minBits; /* Need a minimum to safely represent all symbol values */
- if (tableLog < FSE_MIN_TABLELOG)
- tableLog = FSE_MIN_TABLELOG;
- if (tableLog > FSE_MAX_TABLELOG)
- tableLog = FSE_MAX_TABLELOG;
- return tableLog;
-}
-
-unsigned FSE_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
-{
- return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 2);
-}
-
-/* Secondary normalization method.
- To be used when primary method fails. */
-
-static size_t FSE_normalizeM2(short *norm, U32 tableLog, const unsigned *count, size_t total, U32 maxSymbolValue)
-{
- short const NOT_YET_ASSIGNED = -2;
- U32 s;
- U32 distributed = 0;
- U32 ToDistribute;
-
- /* Init */
- U32 const lowThreshold = (U32)(total >> tableLog);
- U32 lowOne = (U32)((total * 3) >> (tableLog + 1));
-
- for (s = 0; s <= maxSymbolValue; s++) {
- if (count[s] == 0) {
- norm[s] = 0;
- continue;
- }
- if (count[s] <= lowThreshold) {
- norm[s] = -1;
- distributed++;
- total -= count[s];
- continue;
- }
- if (count[s] <= lowOne) {
- norm[s] = 1;
- distributed++;
- total -= count[s];
- continue;
- }
-
- norm[s] = NOT_YET_ASSIGNED;
- }
- ToDistribute = (1 << tableLog) - distributed;
-
- if ((total / ToDistribute) > lowOne) {
- /* risk of rounding to zero */
- lowOne = (U32)((total * 3) / (ToDistribute * 2));
- for (s = 0; s <= maxSymbolValue; s++) {
- if ((norm[s] == NOT_YET_ASSIGNED) && (count[s] <= lowOne)) {
- norm[s] = 1;
- distributed++;
- total -= count[s];
- continue;
- }
- }
- ToDistribute = (1 << tableLog) - distributed;
- }
-
- if (distributed == maxSymbolValue + 1) {
- /* all values are pretty poor;
- probably incompressible data (should have already been detected);
- find max, then give all remaining points to max */
- U32 maxV = 0, maxC = 0;
- for (s = 0; s <= maxSymbolValue; s++)
- if (count[s] > maxC)
- maxV = s, maxC = count[s];
- norm[maxV] += (short)ToDistribute;
- return 0;
- }
-
- if (total == 0) {
- /* all of the symbols were low enough for the lowOne or lowThreshold */
- for (s = 0; ToDistribute > 0; s = (s + 1) % (maxSymbolValue + 1))
- if (norm[s] > 0)
- ToDistribute--, norm[s]++;
- return 0;
- }
-
- {
- U64 const vStepLog = 62 - tableLog;
- U64 const mid = (1ULL << (vStepLog - 1)) - 1;
- U64 const rStep = div_u64((((U64)1 << vStepLog) * ToDistribute) + mid, (U32)total); /* scale on remaining */
- U64 tmpTotal = mid;
- for (s = 0; s <= maxSymbolValue; s++) {
- if (norm[s] == NOT_YET_ASSIGNED) {
- U64 const end = tmpTotal + (count[s] * rStep);
- U32 const sStart = (U32)(tmpTotal >> vStepLog);
- U32 const sEnd = (U32)(end >> vStepLog);
- U32 const weight = sEnd - sStart;
- if (weight < 1)
- return ERROR(GENERIC);
- norm[s] = (short)weight;
- tmpTotal = end;
- }
- }
- }
-
- return 0;
-}
-
-size_t FSE_normalizeCount(short *normalizedCounter, unsigned tableLog, const unsigned *count, size_t total, unsigned maxSymbolValue)
-{
- /* Sanity checks */
- if (tableLog == 0)
- tableLog = FSE_DEFAULT_TABLELOG;
- if (tableLog < FSE_MIN_TABLELOG)
- return ERROR(GENERIC); /* Unsupported size */
- if (tableLog > FSE_MAX_TABLELOG)
- return ERROR(tableLog_tooLarge); /* Unsupported size */
- if (tableLog < FSE_minTableLog(total, maxSymbolValue))
- return ERROR(GENERIC); /* Too small tableLog, compression potentially impossible */
-
- {
- U32 const rtbTable[] = {0, 473195, 504333, 520860, 550000, 700000, 750000, 830000};
- U64 const scale = 62 - tableLog;
- U64 const step = div_u64((U64)1 << 62, (U32)total); /* <== here, one division ! */
- U64 const vStep = 1ULL << (scale - 20);
- int stillToDistribute = 1 << tableLog;
- unsigned s;
- unsigned largest = 0;
- short largestP = 0;
- U32 lowThreshold = (U32)(total >> tableLog);
-
- for (s = 0; s <= maxSymbolValue; s++) {
- if (count[s] == total)
- return 0; /* rle special case */
- if (count[s] == 0) {
- normalizedCounter[s] = 0;
- continue;
- }
- if (count[s] <= lowThreshold) {
- normalizedCounter[s] = -1;
- stillToDistribute--;
- } else {
- short proba = (short)((count[s] * step) >> scale);
- if (proba < 8) {
- U64 restToBeat = vStep * rtbTable[proba];
- proba += (count[s] * step) - ((U64)proba << scale) > restToBeat;
- }
- if (proba > largestP)
- largestP = proba, largest = s;
- normalizedCounter[s] = proba;
- stillToDistribute -= proba;
- }
- }
- if (-stillToDistribute >= (normalizedCounter[largest] >> 1)) {
- /* corner case, need another normalization method */
- size_t const errorCode = FSE_normalizeM2(normalizedCounter, tableLog, count, total, maxSymbolValue);
- if (FSE_isError(errorCode))
- return errorCode;
- } else
- normalizedCounter[largest] += (short)stillToDistribute;
- }
-
- return tableLog;
-}
-
-/* fake FSE_CTable, for raw (uncompressed) input */
-size_t FSE_buildCTable_raw(FSE_CTable *ct, unsigned nbBits)
-{
- const unsigned tableSize = 1 << nbBits;
- const unsigned tableMask = tableSize - 1;
- const unsigned maxSymbolValue = tableMask;
- void *const ptr = ct;
- U16 *const tableU16 = ((U16 *)ptr) + 2;
- void *const FSCT = ((U32 *)ptr) + 1 /* header */ + (tableSize >> 1); /* assumption : tableLog >= 1 */
- FSE_symbolCompressionTransform *const symbolTT = (FSE_symbolCompressionTransform *)(FSCT);
- unsigned s;
-
- /* Sanity checks */
- if (nbBits < 1)
- return ERROR(GENERIC); /* min size */
-
- /* header */
- tableU16[-2] = (U16)nbBits;
- tableU16[-1] = (U16)maxSymbolValue;
-
- /* Build table */
- for (s = 0; s < tableSize; s++)
- tableU16[s] = (U16)(tableSize + s);
-
- /* Build Symbol Transformation Table */
- {
- const U32 deltaNbBits = (nbBits << 16) - (1 << nbBits);
- for (s = 0; s <= maxSymbolValue; s++) {
- symbolTT[s].deltaNbBits = deltaNbBits;
- symbolTT[s].deltaFindState = s - 1;
- }
- }
-
- return 0;
-}
-
-/* fake FSE_CTable, for rle input (always same symbol) */
-size_t FSE_buildCTable_rle(FSE_CTable *ct, BYTE symbolValue)
-{
- void *ptr = ct;
- U16 *tableU16 = ((U16 *)ptr) + 2;
- void *FSCTptr = (U32 *)ptr + 2;
- FSE_symbolCompressionTransform *symbolTT = (FSE_symbolCompressionTransform *)FSCTptr;
-
- /* header */
- tableU16[-2] = (U16)0;
- tableU16[-1] = (U16)symbolValue;
-
- /* Build table */
- tableU16[0] = 0;
- tableU16[1] = 0; /* just in case */
-
- /* Build Symbol Transformation Table */
- symbolTT[symbolValue].deltaNbBits = 0;
- symbolTT[symbolValue].deltaFindState = 0;
-
- return 0;
-}
-
-static size_t FSE_compress_usingCTable_generic(void *dst, size_t dstSize, const void *src, size_t srcSize, const FSE_CTable *ct, const unsigned fast)
-{
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *const iend = istart + srcSize;
- const BYTE *ip = iend;
-
- BIT_CStream_t bitC;
- FSE_CState_t CState1, CState2;
-
- /* init */
- if (srcSize <= 2)
- return 0;
- {
- size_t const initError = BIT_initCStream(&bitC, dst, dstSize);
- if (FSE_isError(initError))
- return 0; /* not enough space available to write a bitstream */
- }
-
-#define FSE_FLUSHBITS(s) (fast ? BIT_flushBitsFast(s) : BIT_flushBits(s))
-
- if (srcSize & 1) {
- FSE_initCState2(&CState1, ct, *--ip);
- FSE_initCState2(&CState2, ct, *--ip);
- FSE_encodeSymbol(&bitC, &CState1, *--ip);
- FSE_FLUSHBITS(&bitC);
- } else {
- FSE_initCState2(&CState2, ct, *--ip);
- FSE_initCState2(&CState1, ct, *--ip);
- }
-
- /* join to mod 4 */
- srcSize -= 2;
- if ((sizeof(bitC.bitContainer) * 8 > FSE_MAX_TABLELOG * 4 + 7) && (srcSize & 2)) { /* test bit 2 */
- FSE_encodeSymbol(&bitC, &CState2, *--ip);
- FSE_encodeSymbol(&bitC, &CState1, *--ip);
- FSE_FLUSHBITS(&bitC);
- }
-
- /* 2 or 4 encoding per loop */
- while (ip > istart) {
-
- FSE_encodeSymbol(&bitC, &CState2, *--ip);
-
- if (sizeof(bitC.bitContainer) * 8 < FSE_MAX_TABLELOG * 2 + 7) /* this test must be static */
- FSE_FLUSHBITS(&bitC);
-
- FSE_encodeSymbol(&bitC, &CState1, *--ip);
-
- if (sizeof(bitC.bitContainer) * 8 > FSE_MAX_TABLELOG * 4 + 7) { /* this test must be static */
- FSE_encodeSymbol(&bitC, &CState2, *--ip);
- FSE_encodeSymbol(&bitC, &CState1, *--ip);
- }
-
- FSE_FLUSHBITS(&bitC);
- }
-
- FSE_flushCState(&bitC, &CState2);
- FSE_flushCState(&bitC, &CState1);
- return BIT_closeCStream(&bitC);
-}
-
-size_t FSE_compress_usingCTable(void *dst, size_t dstSize, const void *src, size_t srcSize, const FSE_CTable *ct)
-{
- unsigned const fast = (dstSize >= FSE_BLOCKBOUND(srcSize));
-
- if (fast)
- return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 1);
- else
- return FSE_compress_usingCTable_generic(dst, dstSize, src, srcSize, ct, 0);
-}
-
-size_t FSE_compressBound(size_t size) { return FSE_COMPRESSBOUND(size); }
+++ /dev/null
-/*
- * FSE : Finite State Entropy decoder
- * Copyright (C) 2013-2015, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-
-/* **************************************************************
-* Compiler specifics
-****************************************************************/
-#define FORCE_INLINE static __always_inline
-
-/* **************************************************************
-* Includes
-****************************************************************/
-#include "bitstream.h"
-#include "fse.h"
-#include "zstd_internal.h"
-#include <linux/compiler.h>
-#include <linux/kernel.h>
-#include <linux/string.h> /* memcpy, memset */
-
-/* **************************************************************
-* Error Management
-****************************************************************/
-#define FSE_isError ERR_isError
-#define FSE_STATIC_ASSERT(c) \
- { \
- enum { FSE_static_assert = 1 / (int)(!!(c)) }; \
- } /* use only *after* variable declarations */
-
-/* **************************************************************
-* Templates
-****************************************************************/
-/*
- designed to be included
- for type-specific functions (template emulation in C)
- Objective is to write these functions only once, for improved maintenance
-*/
-
-/* safety checks */
-#ifndef FSE_FUNCTION_EXTENSION
-#error "FSE_FUNCTION_EXTENSION must be defined"
-#endif
-#ifndef FSE_FUNCTION_TYPE
-#error "FSE_FUNCTION_TYPE must be defined"
-#endif
-
-/* Function names */
-#define FSE_CAT(X, Y) X##Y
-#define FSE_FUNCTION_NAME(X, Y) FSE_CAT(X, Y)
-#define FSE_TYPE_NAME(X, Y) FSE_CAT(X, Y)
-
-/* Function templates */
-
-size_t FSE_buildDTable_wksp(FSE_DTable *dt, const short *normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void *workspace, size_t workspaceSize)
-{
- void *const tdPtr = dt + 1; /* because *dt is unsigned, 32-bits aligned on 32-bits */
- FSE_DECODE_TYPE *const tableDecode = (FSE_DECODE_TYPE *)(tdPtr);
- U16 *symbolNext = (U16 *)workspace;
-
- U32 const maxSV1 = maxSymbolValue + 1;
- U32 const tableSize = 1 << tableLog;
- U32 highThreshold = tableSize - 1;
-
- /* Sanity Checks */
- if (workspaceSize < sizeof(U16) * (FSE_MAX_SYMBOL_VALUE + 1))
- return ERROR(tableLog_tooLarge);
- if (maxSymbolValue > FSE_MAX_SYMBOL_VALUE)
- return ERROR(maxSymbolValue_tooLarge);
- if (tableLog > FSE_MAX_TABLELOG)
- return ERROR(tableLog_tooLarge);
-
- /* Init, lay down lowprob symbols */
- {
- FSE_DTableHeader DTableH;
- DTableH.tableLog = (U16)tableLog;
- DTableH.fastMode = 1;
- {
- S16 const largeLimit = (S16)(1 << (tableLog - 1));
- U32 s;
- for (s = 0; s < maxSV1; s++) {
- if (normalizedCounter[s] == -1) {
- tableDecode[highThreshold--].symbol = (FSE_FUNCTION_TYPE)s;
- symbolNext[s] = 1;
- } else {
- if (normalizedCounter[s] >= largeLimit)
- DTableH.fastMode = 0;
- symbolNext[s] = normalizedCounter[s];
- }
- }
- }
- memcpy(dt, &DTableH, sizeof(DTableH));
- }
-
- /* Spread symbols */
- {
- U32 const tableMask = tableSize - 1;
- U32 const step = FSE_TABLESTEP(tableSize);
- U32 s, position = 0;
- for (s = 0; s < maxSV1; s++) {
- int i;
- for (i = 0; i < normalizedCounter[s]; i++) {
- tableDecode[position].symbol = (FSE_FUNCTION_TYPE)s;
- position = (position + step) & tableMask;
- while (position > highThreshold)
- position = (position + step) & tableMask; /* lowprob area */
- }
- }
- if (position != 0)
- return ERROR(GENERIC); /* position must reach all cells once, otherwise normalizedCounter is incorrect */
- }
-
- /* Build Decoding table */
- {
- U32 u;
- for (u = 0; u < tableSize; u++) {
- FSE_FUNCTION_TYPE const symbol = (FSE_FUNCTION_TYPE)(tableDecode[u].symbol);
- U16 nextState = symbolNext[symbol]++;
- tableDecode[u].nbBits = (BYTE)(tableLog - BIT_highbit32((U32)nextState));
- tableDecode[u].newState = (U16)((nextState << tableDecode[u].nbBits) - tableSize);
- }
- }
-
- return 0;
-}
-
-/*-*******************************************************
-* Decompression (Byte symbols)
-*********************************************************/
-size_t FSE_buildDTable_rle(FSE_DTable *dt, BYTE symbolValue)
-{
- void *ptr = dt;
- FSE_DTableHeader *const DTableH = (FSE_DTableHeader *)ptr;
- void *dPtr = dt + 1;
- FSE_decode_t *const cell = (FSE_decode_t *)dPtr;
-
- DTableH->tableLog = 0;
- DTableH->fastMode = 0;
-
- cell->newState = 0;
- cell->symbol = symbolValue;
- cell->nbBits = 0;
-
- return 0;
-}
-
-size_t FSE_buildDTable_raw(FSE_DTable *dt, unsigned nbBits)
-{
- void *ptr = dt;
- FSE_DTableHeader *const DTableH = (FSE_DTableHeader *)ptr;
- void *dPtr = dt + 1;
- FSE_decode_t *const dinfo = (FSE_decode_t *)dPtr;
- const unsigned tableSize = 1 << nbBits;
- const unsigned tableMask = tableSize - 1;
- const unsigned maxSV1 = tableMask + 1;
- unsigned s;
-
- /* Sanity checks */
- if (nbBits < 1)
- return ERROR(GENERIC); /* min size */
-
- /* Build Decoding Table */
- DTableH->tableLog = (U16)nbBits;
- DTableH->fastMode = 1;
- for (s = 0; s < maxSV1; s++) {
- dinfo[s].newState = 0;
- dinfo[s].symbol = (BYTE)s;
- dinfo[s].nbBits = (BYTE)nbBits;
- }
-
- return 0;
-}
-
-FORCE_INLINE size_t FSE_decompress_usingDTable_generic(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const FSE_DTable *dt,
- const unsigned fast)
-{
- BYTE *const ostart = (BYTE *)dst;
- BYTE *op = ostart;
- BYTE *const omax = op + maxDstSize;
- BYTE *const olimit = omax - 3;
-
- BIT_DStream_t bitD;
- FSE_DState_t state1;
- FSE_DState_t state2;
-
- /* Init */
- CHECK_F(BIT_initDStream(&bitD, cSrc, cSrcSize));
-
- FSE_initDState(&state1, &bitD, dt);
- FSE_initDState(&state2, &bitD, dt);
-
-#define FSE_GETSYMBOL(statePtr) fast ? FSE_decodeSymbolFast(statePtr, &bitD) : FSE_decodeSymbol(statePtr, &bitD)
-
- /* 4 symbols per loop */
- for (; (BIT_reloadDStream(&bitD) == BIT_DStream_unfinished) & (op < olimit); op += 4) {
- op[0] = FSE_GETSYMBOL(&state1);
-
- if (FSE_MAX_TABLELOG * 2 + 7 > sizeof(bitD.bitContainer) * 8) /* This test must be static */
- BIT_reloadDStream(&bitD);
-
- op[1] = FSE_GETSYMBOL(&state2);
-
- if (FSE_MAX_TABLELOG * 4 + 7 > sizeof(bitD.bitContainer) * 8) /* This test must be static */
- {
- if (BIT_reloadDStream(&bitD) > BIT_DStream_unfinished) {
- op += 2;
- break;
- }
- }
-
- op[2] = FSE_GETSYMBOL(&state1);
-
- if (FSE_MAX_TABLELOG * 2 + 7 > sizeof(bitD.bitContainer) * 8) /* This test must be static */
- BIT_reloadDStream(&bitD);
-
- op[3] = FSE_GETSYMBOL(&state2);
- }
-
- /* tail */
- /* note : BIT_reloadDStream(&bitD) >= FSE_DStream_partiallyFilled; Ends at exactly BIT_DStream_completed */
- while (1) {
- if (op > (omax - 2))
- return ERROR(dstSize_tooSmall);
- *op++ = FSE_GETSYMBOL(&state1);
- if (BIT_reloadDStream(&bitD) == BIT_DStream_overflow) {
- *op++ = FSE_GETSYMBOL(&state2);
- break;
- }
-
- if (op > (omax - 2))
- return ERROR(dstSize_tooSmall);
- *op++ = FSE_GETSYMBOL(&state2);
- if (BIT_reloadDStream(&bitD) == BIT_DStream_overflow) {
- *op++ = FSE_GETSYMBOL(&state1);
- break;
- }
- }
-
- return op - ostart;
-}
-
-size_t FSE_decompress_usingDTable(void *dst, size_t originalSize, const void *cSrc, size_t cSrcSize, const FSE_DTable *dt)
-{
- const void *ptr = dt;
- const FSE_DTableHeader *DTableH = (const FSE_DTableHeader *)ptr;
- const U32 fastMode = DTableH->fastMode;
-
- /* select fast mode (static) */
- if (fastMode)
- return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 1);
- return FSE_decompress_usingDTable_generic(dst, originalSize, cSrc, cSrcSize, dt, 0);
-}
-
-size_t FSE_decompress_wksp(void *dst, size_t dstCapacity, const void *cSrc, size_t cSrcSize, unsigned maxLog, void *workspace, size_t workspaceSize)
-{
- const BYTE *const istart = (const BYTE *)cSrc;
- const BYTE *ip = istart;
- unsigned tableLog;
- unsigned maxSymbolValue = FSE_MAX_SYMBOL_VALUE;
- size_t NCountLength;
-
- FSE_DTable *dt;
- short *counting;
- size_t spaceUsed32 = 0;
-
- FSE_STATIC_ASSERT(sizeof(FSE_DTable) == sizeof(U32));
-
- dt = (FSE_DTable *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += FSE_DTABLE_SIZE_U32(maxLog);
- counting = (short *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(sizeof(short) * (FSE_MAX_SYMBOL_VALUE + 1), sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- /* normal FSE decoding mode */
- NCountLength = FSE_readNCount(counting, &maxSymbolValue, &tableLog, istart, cSrcSize);
- if (FSE_isError(NCountLength))
- return NCountLength;
- // if (NCountLength >= cSrcSize) return ERROR(srcSize_wrong); /* too small input size; supposed to be already checked in NCountLength, only remaining
- // case : NCountLength==cSrcSize */
- if (tableLog > maxLog)
- return ERROR(tableLog_tooLarge);
- ip += NCountLength;
- cSrcSize -= NCountLength;
-
- CHECK_F(FSE_buildDTable_wksp(dt, counting, maxSymbolValue, tableLog, workspace, workspaceSize));
-
- return FSE_decompress_usingDTable(dst, dstCapacity, ip, cSrcSize, dt); /* always return, even if it is an error code */
-}
+++ /dev/null
-/*
- * Huffman coder, part of New Generation Entropy library
- * header file
- * Copyright (C) 2013-2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-#ifndef HUF_H_298734234
-#define HUF_H_298734234
-
-/* *** Dependencies *** */
-#include <linux/types.h> /* size_t */
-
-/* *** Tool functions *** */
-#define HUF_BLOCKSIZE_MAX (128 * 1024) /**< maximum input size for a single block compressed with HUF_compress */
-size_t HUF_compressBound(size_t size); /**< maximum compressed size (worst case) */
-
-/* Error Management */
-unsigned HUF_isError(size_t code); /**< tells if a return value is an error code */
-
-/* *** Advanced function *** */
-
-/** HUF_compress4X_wksp() :
-* Same as HUF_compress2(), but uses externally allocated `workSpace`, which must be a table of >= 1024 unsigned */
-size_t HUF_compress4X_wksp(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void *workSpace,
- size_t wkspSize); /**< `workSpace` must be a table of at least HUF_COMPRESS_WORKSPACE_SIZE_U32 unsigned */
-
-/* *** Dependencies *** */
-#include "mem.h" /* U32 */
-
-/* *** Constants *** */
-#define HUF_TABLELOG_MAX 12 /* max configured tableLog (for static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
-#define HUF_TABLELOG_DEFAULT 11 /* tableLog by default, when not specified */
-#define HUF_SYMBOLVALUE_MAX 255
-
-#define HUF_TABLELOG_ABSOLUTEMAX 15 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
-#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
-#error "HUF_TABLELOG_MAX is too large !"
-#endif
-
-/* ****************************************
-* Static allocation
-******************************************/
-/* HUF buffer bounds */
-#define HUF_CTABLEBOUND 129
-#define HUF_BLOCKBOUND(size) (size + (size >> 8) + 8) /* only true if incompressible pre-filtered with fast heuristic */
-#define HUF_COMPRESSBOUND(size) (HUF_CTABLEBOUND + HUF_BLOCKBOUND(size)) /* Macro version, useful for static allocation */
-
-/* static allocation of HUF's Compression Table */
-#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
- U32 name##hb[maxSymbolValue + 1]; \
- void *name##hv = &(name##hb); \
- HUF_CElt *name = (HUF_CElt *)(name##hv) /* no final ; */
-
-/* static allocation of HUF's DTable */
-typedef U32 HUF_DTable;
-#define HUF_DTABLE_SIZE(maxTableLog) (1 + (1 << (maxTableLog)))
-#define HUF_CREATE_STATIC_DTABLEX2(DTable, maxTableLog) HUF_DTable DTable[HUF_DTABLE_SIZE((maxTableLog)-1)] = {((U32)((maxTableLog)-1) * 0x01000001)}
-#define HUF_CREATE_STATIC_DTABLEX4(DTable, maxTableLog) HUF_DTable DTable[HUF_DTABLE_SIZE(maxTableLog)] = {((U32)(maxTableLog)*0x01000001)}
-
-/* The workspace must have alignment at least 4 and be at least this large */
-#define HUF_COMPRESS_WORKSPACE_SIZE (6 << 10)
-#define HUF_COMPRESS_WORKSPACE_SIZE_U32 (HUF_COMPRESS_WORKSPACE_SIZE / sizeof(U32))
-
-/* The workspace must have alignment at least 4 and be at least this large */
-#define HUF_DECOMPRESS_WORKSPACE_SIZE (3 << 10)
-#define HUF_DECOMPRESS_WORKSPACE_SIZE_U32 (HUF_DECOMPRESS_WORKSPACE_SIZE / sizeof(U32))
-
-/* ****************************************
-* Advanced decompression functions
-******************************************/
-size_t HUF_decompress4X_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize); /**< decodes RLE and uncompressed */
-size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace,
- size_t workspaceSize); /**< considers RLE and uncompressed as errors */
-size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace,
- size_t workspaceSize); /**< single-symbol decoder */
-size_t HUF_decompress4X4_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace,
- size_t workspaceSize); /**< double-symbols decoder */
-
-/* ****************************************
-* HUF detailed API
-******************************************/
-/*!
-HUF_compress() does the following:
-1. count symbol occurrence from source[] into table count[] using FSE_count()
-2. (optional) refine tableLog using HUF_optimalTableLog()
-3. build Huffman table from count using HUF_buildCTable()
-4. save Huffman table to memory buffer using HUF_writeCTable_wksp()
-5. encode the data stream using HUF_compress4X_usingCTable()
-
-The following API allows targeting specific sub-functions for advanced tasks.
-For example, it's possible to compress several blocks using the same 'CTable',
-or to save and regenerate 'CTable' using external methods.
-*/
-/* FSE_count() : find it within "fse.h" */
-unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue);
-typedef struct HUF_CElt_s HUF_CElt; /* incomplete type */
-size_t HUF_writeCTable_wksp(void *dst, size_t maxDstSize, const HUF_CElt *CTable, unsigned maxSymbolValue, unsigned huffLog, void *workspace, size_t workspaceSize);
-size_t HUF_compress4X_usingCTable(void *dst, size_t dstSize, const void *src, size_t srcSize, const HUF_CElt *CTable);
-
-typedef enum {
- HUF_repeat_none, /**< Cannot use the previous table */
- HUF_repeat_check, /**< Can use the previous table but it must be checked. Note : The previous table must have been constructed by HUF_compress{1,
- 4}X_repeat */
- HUF_repeat_valid /**< Can use the previous table and it is assumed to be valid */
-} HUF_repeat;
-/** HUF_compress4X_repeat() :
-* Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
-* If it uses hufTable it does not modify hufTable or repeat.
-* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
-* If preferRepeat then the old table will always be used if valid. */
-size_t HUF_compress4X_repeat(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void *workSpace,
- size_t wkspSize, HUF_CElt *hufTable, HUF_repeat *repeat,
- int preferRepeat); /**< `workSpace` must be a table of at least HUF_COMPRESS_WORKSPACE_SIZE_U32 unsigned */
-
-/** HUF_buildCTable_wksp() :
- * Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- * `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as a table of 1024 unsigned.
- */
-size_t HUF_buildCTable_wksp(HUF_CElt *tree, const U32 *count, U32 maxSymbolValue, U32 maxNbBits, void *workSpace, size_t wkspSize);
-
-/*! HUF_readStats() :
- Read compact Huffman tree, saved by HUF_writeCTable().
- `huffWeight` is destination buffer.
- @return : size read from `src` , or an error Code .
- Note : Needed by HUF_readCTable() and HUF_readDTableXn() . */
-size_t HUF_readStats_wksp(BYTE *huffWeight, size_t hwSize, U32 *rankStats, U32 *nbSymbolsPtr, U32 *tableLogPtr, const void *src, size_t srcSize,
- void *workspace, size_t workspaceSize);
-
-/** HUF_readCTable() :
-* Loading a CTable saved with HUF_writeCTable() */
-size_t HUF_readCTable_wksp(HUF_CElt *CTable, unsigned maxSymbolValue, const void *src, size_t srcSize, void *workspace, size_t workspaceSize);
-
-/*
-HUF_decompress() does the following:
-1. select the decompression algorithm (X2, X4) based on pre-computed heuristics
-2. build Huffman table from save, using HUF_readDTableXn()
-3. decode 1 or 4 segments in parallel using HUF_decompressSXn_usingDTable
-*/
-
-/** HUF_selectDecoder() :
-* Tells which decoder is likely to decode faster,
-* based on a set of pre-determined metrics.
-* @return : 0==HUF_decompress4X2, 1==HUF_decompress4X4 .
-* Assumption : 0 < cSrcSize < dstSize <= 128 KB */
-U32 HUF_selectDecoder(size_t dstSize, size_t cSrcSize);
-
-size_t HUF_readDTableX2_wksp(HUF_DTable *DTable, const void *src, size_t srcSize, void *workspace, size_t workspaceSize);
-size_t HUF_readDTableX4_wksp(HUF_DTable *DTable, const void *src, size_t srcSize, void *workspace, size_t workspaceSize);
-
-size_t HUF_decompress4X_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable);
-size_t HUF_decompress4X2_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable);
-size_t HUF_decompress4X4_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable);
-
-/* single stream variants */
-
-size_t HUF_compress1X_wksp(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void *workSpace,
- size_t wkspSize); /**< `workSpace` must be a table of at least HUF_COMPRESS_WORKSPACE_SIZE_U32 unsigned */
-size_t HUF_compress1X_usingCTable(void *dst, size_t dstSize, const void *src, size_t srcSize, const HUF_CElt *CTable);
-/** HUF_compress1X_repeat() :
-* Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
-* If it uses hufTable it does not modify hufTable or repeat.
-* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
-* If preferRepeat then the old table will always be used if valid. */
-size_t HUF_compress1X_repeat(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void *workSpace,
- size_t wkspSize, HUF_CElt *hufTable, HUF_repeat *repeat,
- int preferRepeat); /**< `workSpace` must be a table of at least HUF_COMPRESS_WORKSPACE_SIZE_U32 unsigned */
-
-size_t HUF_decompress1X_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize);
-size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace,
- size_t workspaceSize); /**< single-symbol decoder */
-size_t HUF_decompress1X4_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace,
- size_t workspaceSize); /**< double-symbols decoder */
-
-size_t HUF_decompress1X_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize,
- const HUF_DTable *DTable); /**< automatic selection of sing or double symbol decoder, based on DTable */
-size_t HUF_decompress1X2_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable);
-size_t HUF_decompress1X4_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable);
-
-#endif /* HUF_H_298734234 */
+++ /dev/null
-/*
- * Huffman encoder, part of New Generation Entropy library
- * Copyright (C) 2013-2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-
-/* **************************************************************
-* Includes
-****************************************************************/
-#include "bitstream.h"
-#include "fse.h" /* header compression */
-#include "huf.h"
-#include <linux/kernel.h>
-#include <linux/string.h> /* memcpy, memset */
-
-/* **************************************************************
-* Error Management
-****************************************************************/
-#define HUF_STATIC_ASSERT(c) \
- { \
- enum { HUF_static_assert = 1 / (int)(!!(c)) }; \
- } /* use only *after* variable declarations */
-#define CHECK_V_F(e, f) \
- size_t const e = f; \
- if (ERR_isError(e)) \
- return f
-#define CHECK_F(f) \
- { \
- CHECK_V_F(_var_err__, f); \
- }
-
-/* **************************************************************
-* Utils
-****************************************************************/
-unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxSymbolValue)
-{
- return FSE_optimalTableLog_internal(maxTableLog, srcSize, maxSymbolValue, 1);
-}
-
-/* *******************************************************
-* HUF : Huffman block compression
-*********************************************************/
-/* HUF_compressWeights() :
- * Same as FSE_compress(), but dedicated to huff0's weights compression.
- * The use case needs much less stack memory.
- * Note : all elements within weightTable are supposed to be <= HUF_TABLELOG_MAX.
- */
-#define MAX_FSE_TABLELOG_FOR_HUFF_HEADER 6
-size_t HUF_compressWeights_wksp(void *dst, size_t dstSize, const void *weightTable, size_t wtSize, void *workspace, size_t workspaceSize)
-{
- BYTE *const ostart = (BYTE *)dst;
- BYTE *op = ostart;
- BYTE *const oend = ostart + dstSize;
-
- U32 maxSymbolValue = HUF_TABLELOG_MAX;
- U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER;
-
- FSE_CTable *CTable;
- U32 *count;
- S16 *norm;
- size_t spaceUsed32 = 0;
-
- HUF_STATIC_ASSERT(sizeof(FSE_CTable) == sizeof(U32));
-
- CTable = (FSE_CTable *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += FSE_CTABLE_SIZE_U32(MAX_FSE_TABLELOG_FOR_HUFF_HEADER, HUF_TABLELOG_MAX);
- count = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += HUF_TABLELOG_MAX + 1;
- norm = (S16 *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(sizeof(S16) * (HUF_TABLELOG_MAX + 1), sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- /* init conditions */
- if (wtSize <= 1)
- return 0; /* Not compressible */
-
- /* Scan input and build symbol stats */
- {
- CHECK_V_F(maxCount, FSE_count_simple(count, &maxSymbolValue, weightTable, wtSize));
- if (maxCount == wtSize)
- return 1; /* only a single symbol in src : rle */
- if (maxCount == 1)
- return 0; /* each symbol present maximum once => not compressible */
- }
-
- tableLog = FSE_optimalTableLog(tableLog, wtSize, maxSymbolValue);
- CHECK_F(FSE_normalizeCount(norm, tableLog, count, wtSize, maxSymbolValue));
-
- /* Write table description header */
- {
- CHECK_V_F(hSize, FSE_writeNCount(op, oend - op, norm, maxSymbolValue, tableLog));
- op += hSize;
- }
-
- /* Compress */
- CHECK_F(FSE_buildCTable_wksp(CTable, norm, maxSymbolValue, tableLog, workspace, workspaceSize));
- {
- CHECK_V_F(cSize, FSE_compress_usingCTable(op, oend - op, weightTable, wtSize, CTable));
- if (cSize == 0)
- return 0; /* not enough space for compressed data */
- op += cSize;
- }
-
- return op - ostart;
-}
-
-struct HUF_CElt_s {
- U16 val;
- BYTE nbBits;
-}; /* typedef'd to HUF_CElt within "huf.h" */
-
-/*! HUF_writeCTable_wksp() :
- `CTable` : Huffman tree to save, using huf representation.
- @return : size of saved CTable */
-size_t HUF_writeCTable_wksp(void *dst, size_t maxDstSize, const HUF_CElt *CTable, U32 maxSymbolValue, U32 huffLog, void *workspace, size_t workspaceSize)
-{
- BYTE *op = (BYTE *)dst;
- U32 n;
-
- BYTE *bitsToWeight;
- BYTE *huffWeight;
- size_t spaceUsed32 = 0;
-
- bitsToWeight = (BYTE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(HUF_TABLELOG_MAX + 1, sizeof(U32)) >> 2;
- huffWeight = (BYTE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(HUF_SYMBOLVALUE_MAX, sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- /* check conditions */
- if (maxSymbolValue > HUF_SYMBOLVALUE_MAX)
- return ERROR(maxSymbolValue_tooLarge);
-
- /* convert to weight */
- bitsToWeight[0] = 0;
- for (n = 1; n < huffLog + 1; n++)
- bitsToWeight[n] = (BYTE)(huffLog + 1 - n);
- for (n = 0; n < maxSymbolValue; n++)
- huffWeight[n] = bitsToWeight[CTable[n].nbBits];
-
- /* attempt weights compression by FSE */
- {
- CHECK_V_F(hSize, HUF_compressWeights_wksp(op + 1, maxDstSize - 1, huffWeight, maxSymbolValue, workspace, workspaceSize));
- if ((hSize > 1) & (hSize < maxSymbolValue / 2)) { /* FSE compressed */
- op[0] = (BYTE)hSize;
- return hSize + 1;
- }
- }
-
- /* write raw values as 4-bits (max : 15) */
- if (maxSymbolValue > (256 - 128))
- return ERROR(GENERIC); /* should not happen : likely means source cannot be compressed */
- if (((maxSymbolValue + 1) / 2) + 1 > maxDstSize)
- return ERROR(dstSize_tooSmall); /* not enough space within dst buffer */
- op[0] = (BYTE)(128 /*special case*/ + (maxSymbolValue - 1));
- huffWeight[maxSymbolValue] = 0; /* to be sure it doesn't cause msan issue in final combination */
- for (n = 0; n < maxSymbolValue; n += 2)
- op[(n / 2) + 1] = (BYTE)((huffWeight[n] << 4) + huffWeight[n + 1]);
- return ((maxSymbolValue + 1) / 2) + 1;
-}
-
-size_t HUF_readCTable_wksp(HUF_CElt *CTable, U32 maxSymbolValue, const void *src, size_t srcSize, void *workspace, size_t workspaceSize)
-{
- U32 *rankVal;
- BYTE *huffWeight;
- U32 tableLog = 0;
- U32 nbSymbols = 0;
- size_t readSize;
- size_t spaceUsed32 = 0;
-
- rankVal = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += HUF_TABLELOG_ABSOLUTEMAX + 1;
- huffWeight = (BYTE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(HUF_SYMBOLVALUE_MAX + 1, sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- /* get symbol weights */
- readSize = HUF_readStats_wksp(huffWeight, HUF_SYMBOLVALUE_MAX + 1, rankVal, &nbSymbols, &tableLog, src, srcSize, workspace, workspaceSize);
- if (ERR_isError(readSize))
- return readSize;
-
- /* check result */
- if (tableLog > HUF_TABLELOG_MAX)
- return ERROR(tableLog_tooLarge);
- if (nbSymbols > maxSymbolValue + 1)
- return ERROR(maxSymbolValue_tooSmall);
-
- /* Prepare base value per rank */
- {
- U32 n, nextRankStart = 0;
- for (n = 1; n <= tableLog; n++) {
- U32 curr = nextRankStart;
- nextRankStart += (rankVal[n] << (n - 1));
- rankVal[n] = curr;
- }
- }
-
- /* fill nbBits */
- {
- U32 n;
- for (n = 0; n < nbSymbols; n++) {
- const U32 w = huffWeight[n];
- CTable[n].nbBits = (BYTE)(tableLog + 1 - w);
- }
- }
-
- /* fill val */
- {
- U16 nbPerRank[HUF_TABLELOG_MAX + 2] = {0}; /* support w=0=>n=tableLog+1 */
- U16 valPerRank[HUF_TABLELOG_MAX + 2] = {0};
- {
- U32 n;
- for (n = 0; n < nbSymbols; n++)
- nbPerRank[CTable[n].nbBits]++;
- }
- /* determine stating value per rank */
- valPerRank[tableLog + 1] = 0; /* for w==0 */
- {
- U16 min = 0;
- U32 n;
- for (n = tableLog; n > 0; n--) { /* start at n=tablelog <-> w=1 */
- valPerRank[n] = min; /* get starting value within each rank */
- min += nbPerRank[n];
- min >>= 1;
- }
- }
- /* assign value within rank, symbol order */
- {
- U32 n;
- for (n = 0; n <= maxSymbolValue; n++)
- CTable[n].val = valPerRank[CTable[n].nbBits]++;
- }
- }
-
- return readSize;
-}
-
-typedef struct nodeElt_s {
- U32 count;
- U16 parent;
- BYTE byte;
- BYTE nbBits;
-} nodeElt;
-
-static U32 HUF_setMaxHeight(nodeElt *huffNode, U32 lastNonNull, U32 maxNbBits)
-{
- const U32 largestBits = huffNode[lastNonNull].nbBits;
- if (largestBits <= maxNbBits)
- return largestBits; /* early exit : no elt > maxNbBits */
-
- /* there are several too large elements (at least >= 2) */
- {
- int totalCost = 0;
- const U32 baseCost = 1 << (largestBits - maxNbBits);
- U32 n = lastNonNull;
-
- while (huffNode[n].nbBits > maxNbBits) {
- totalCost += baseCost - (1 << (largestBits - huffNode[n].nbBits));
- huffNode[n].nbBits = (BYTE)maxNbBits;
- n--;
- } /* n stops at huffNode[n].nbBits <= maxNbBits */
- while (huffNode[n].nbBits == maxNbBits)
- n--; /* n end at index of smallest symbol using < maxNbBits */
-
- /* renorm totalCost */
- totalCost >>= (largestBits - maxNbBits); /* note : totalCost is necessarily a multiple of baseCost */
-
- /* repay normalized cost */
- {
- U32 const noSymbol = 0xF0F0F0F0;
- U32 rankLast[HUF_TABLELOG_MAX + 2];
- int pos;
-
- /* Get pos of last (smallest) symbol per rank */
- memset(rankLast, 0xF0, sizeof(rankLast));
- {
- U32 currNbBits = maxNbBits;
- for (pos = n; pos >= 0; pos--) {
- if (huffNode[pos].nbBits >= currNbBits)
- continue;
- currNbBits = huffNode[pos].nbBits; /* < maxNbBits */
- rankLast[maxNbBits - currNbBits] = pos;
- }
- }
-
- while (totalCost > 0) {
- U32 nBitsToDecrease = BIT_highbit32(totalCost) + 1;
- for (; nBitsToDecrease > 1; nBitsToDecrease--) {
- U32 highPos = rankLast[nBitsToDecrease];
- U32 lowPos = rankLast[nBitsToDecrease - 1];
- if (highPos == noSymbol)
- continue;
- if (lowPos == noSymbol)
- break;
- {
- U32 const highTotal = huffNode[highPos].count;
- U32 const lowTotal = 2 * huffNode[lowPos].count;
- if (highTotal <= lowTotal)
- break;
- }
- }
- /* only triggered when no more rank 1 symbol left => find closest one (note : there is necessarily at least one !) */
- /* HUF_MAX_TABLELOG test just to please gcc 5+; but it should not be necessary */
- while ((nBitsToDecrease <= HUF_TABLELOG_MAX) && (rankLast[nBitsToDecrease] == noSymbol))
- nBitsToDecrease++;
- totalCost -= 1 << (nBitsToDecrease - 1);
- if (rankLast[nBitsToDecrease - 1] == noSymbol)
- rankLast[nBitsToDecrease - 1] = rankLast[nBitsToDecrease]; /* this rank is no longer empty */
- huffNode[rankLast[nBitsToDecrease]].nbBits++;
- if (rankLast[nBitsToDecrease] == 0) /* special case, reached largest symbol */
- rankLast[nBitsToDecrease] = noSymbol;
- else {
- rankLast[nBitsToDecrease]--;
- if (huffNode[rankLast[nBitsToDecrease]].nbBits != maxNbBits - nBitsToDecrease)
- rankLast[nBitsToDecrease] = noSymbol; /* this rank is now empty */
- }
- } /* while (totalCost > 0) */
-
- while (totalCost < 0) { /* Sometimes, cost correction overshoot */
- if (rankLast[1] == noSymbol) { /* special case : no rank 1 symbol (using maxNbBits-1); let's create one from largest rank 0
- (using maxNbBits) */
- while (huffNode[n].nbBits == maxNbBits)
- n--;
- huffNode[n + 1].nbBits--;
- rankLast[1] = n + 1;
- totalCost++;
- continue;
- }
- huffNode[rankLast[1] + 1].nbBits--;
- rankLast[1]++;
- totalCost++;
- }
- }
- } /* there are several too large elements (at least >= 2) */
-
- return maxNbBits;
-}
-
-typedef struct {
- U32 base;
- U32 curr;
-} rankPos;
-
-static void HUF_sort(nodeElt *huffNode, const U32 *count, U32 maxSymbolValue)
-{
- rankPos rank[32];
- U32 n;
-
- memset(rank, 0, sizeof(rank));
- for (n = 0; n <= maxSymbolValue; n++) {
- U32 r = BIT_highbit32(count[n] + 1);
- rank[r].base++;
- }
- for (n = 30; n > 0; n--)
- rank[n - 1].base += rank[n].base;
- for (n = 0; n < 32; n++)
- rank[n].curr = rank[n].base;
- for (n = 0; n <= maxSymbolValue; n++) {
- U32 const c = count[n];
- U32 const r = BIT_highbit32(c + 1) + 1;
- U32 pos = rank[r].curr++;
- while ((pos > rank[r].base) && (c > huffNode[pos - 1].count))
- huffNode[pos] = huffNode[pos - 1], pos--;
- huffNode[pos].count = c;
- huffNode[pos].byte = (BYTE)n;
- }
-}
-
-/** HUF_buildCTable_wksp() :
- * Same as HUF_buildCTable(), but using externally allocated scratch buffer.
- * `workSpace` must be aligned on 4-bytes boundaries, and be at least as large as a table of 1024 unsigned.
- */
-#define STARTNODE (HUF_SYMBOLVALUE_MAX + 1)
-typedef nodeElt huffNodeTable[2 * HUF_SYMBOLVALUE_MAX + 1 + 1];
-size_t HUF_buildCTable_wksp(HUF_CElt *tree, const U32 *count, U32 maxSymbolValue, U32 maxNbBits, void *workSpace, size_t wkspSize)
-{
- nodeElt *const huffNode0 = (nodeElt *)workSpace;
- nodeElt *const huffNode = huffNode0 + 1;
- U32 n, nonNullRank;
- int lowS, lowN;
- U16 nodeNb = STARTNODE;
- U32 nodeRoot;
-
- /* safety checks */
- if (wkspSize < sizeof(huffNodeTable))
- return ERROR(GENERIC); /* workSpace is not large enough */
- if (maxNbBits == 0)
- maxNbBits = HUF_TABLELOG_DEFAULT;
- if (maxSymbolValue > HUF_SYMBOLVALUE_MAX)
- return ERROR(GENERIC);
- memset(huffNode0, 0, sizeof(huffNodeTable));
-
- /* sort, decreasing order */
- HUF_sort(huffNode, count, maxSymbolValue);
-
- /* init for parents */
- nonNullRank = maxSymbolValue;
- while (huffNode[nonNullRank].count == 0)
- nonNullRank--;
- lowS = nonNullRank;
- nodeRoot = nodeNb + lowS - 1;
- lowN = nodeNb;
- huffNode[nodeNb].count = huffNode[lowS].count + huffNode[lowS - 1].count;
- huffNode[lowS].parent = huffNode[lowS - 1].parent = nodeNb;
- nodeNb++;
- lowS -= 2;
- for (n = nodeNb; n <= nodeRoot; n++)
- huffNode[n].count = (U32)(1U << 30);
- huffNode0[0].count = (U32)(1U << 31); /* fake entry, strong barrier */
-
- /* create parents */
- while (nodeNb <= nodeRoot) {
- U32 n1 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
- U32 n2 = (huffNode[lowS].count < huffNode[lowN].count) ? lowS-- : lowN++;
- huffNode[nodeNb].count = huffNode[n1].count + huffNode[n2].count;
- huffNode[n1].parent = huffNode[n2].parent = nodeNb;
- nodeNb++;
- }
-
- /* distribute weights (unlimited tree height) */
- huffNode[nodeRoot].nbBits = 0;
- for (n = nodeRoot - 1; n >= STARTNODE; n--)
- huffNode[n].nbBits = huffNode[huffNode[n].parent].nbBits + 1;
- for (n = 0; n <= nonNullRank; n++)
- huffNode[n].nbBits = huffNode[huffNode[n].parent].nbBits + 1;
-
- /* enforce maxTableLog */
- maxNbBits = HUF_setMaxHeight(huffNode, nonNullRank, maxNbBits);
-
- /* fill result into tree (val, nbBits) */
- {
- U16 nbPerRank[HUF_TABLELOG_MAX + 1] = {0};
- U16 valPerRank[HUF_TABLELOG_MAX + 1] = {0};
- if (maxNbBits > HUF_TABLELOG_MAX)
- return ERROR(GENERIC); /* check fit into table */
- for (n = 0; n <= nonNullRank; n++)
- nbPerRank[huffNode[n].nbBits]++;
- /* determine stating value per rank */
- {
- U16 min = 0;
- for (n = maxNbBits; n > 0; n--) {
- valPerRank[n] = min; /* get starting value within each rank */
- min += nbPerRank[n];
- min >>= 1;
- }
- }
- for (n = 0; n <= maxSymbolValue; n++)
- tree[huffNode[n].byte].nbBits = huffNode[n].nbBits; /* push nbBits per symbol, symbol order */
- for (n = 0; n <= maxSymbolValue; n++)
- tree[n].val = valPerRank[tree[n].nbBits]++; /* assign value within rank, symbol order */
- }
-
- return maxNbBits;
-}
-
-static size_t HUF_estimateCompressedSize(HUF_CElt *CTable, const unsigned *count, unsigned maxSymbolValue)
-{
- size_t nbBits = 0;
- int s;
- for (s = 0; s <= (int)maxSymbolValue; ++s) {
- nbBits += CTable[s].nbBits * count[s];
- }
- return nbBits >> 3;
-}
-
-static int HUF_validateCTable(const HUF_CElt *CTable, const unsigned *count, unsigned maxSymbolValue)
-{
- int bad = 0;
- int s;
- for (s = 0; s <= (int)maxSymbolValue; ++s) {
- bad |= (count[s] != 0) & (CTable[s].nbBits == 0);
- }
- return !bad;
-}
-
-static void HUF_encodeSymbol(BIT_CStream_t *bitCPtr, U32 symbol, const HUF_CElt *CTable)
-{
- BIT_addBitsFast(bitCPtr, CTable[symbol].val, CTable[symbol].nbBits);
-}
-
-size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); }
-
-#define HUF_FLUSHBITS(s) BIT_flushBits(s)
-
-#define HUF_FLUSHBITS_1(stream) \
- if (sizeof((stream)->bitContainer) * 8 < HUF_TABLELOG_MAX * 2 + 7) \
- HUF_FLUSHBITS(stream)
-
-#define HUF_FLUSHBITS_2(stream) \
- if (sizeof((stream)->bitContainer) * 8 < HUF_TABLELOG_MAX * 4 + 7) \
- HUF_FLUSHBITS(stream)
-
-size_t HUF_compress1X_usingCTable(void *dst, size_t dstSize, const void *src, size_t srcSize, const HUF_CElt *CTable)
-{
- const BYTE *ip = (const BYTE *)src;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- BYTE *op = ostart;
- size_t n;
- BIT_CStream_t bitC;
-
- /* init */
- if (dstSize < 8)
- return 0; /* not enough space to compress */
- {
- size_t const initErr = BIT_initCStream(&bitC, op, oend - op);
- if (HUF_isError(initErr))
- return 0;
- }
-
- n = srcSize & ~3; /* join to mod 4 */
- switch (srcSize & 3) {
- case 3: HUF_encodeSymbol(&bitC, ip[n + 2], CTable); HUF_FLUSHBITS_2(&bitC);
- fallthrough;
- case 2: HUF_encodeSymbol(&bitC, ip[n + 1], CTable); HUF_FLUSHBITS_1(&bitC);
- fallthrough;
- case 1: HUF_encodeSymbol(&bitC, ip[n + 0], CTable); HUF_FLUSHBITS(&bitC);
- fallthrough;
- case 0:
- default:;
- }
-
- for (; n > 0; n -= 4) { /* note : n&3==0 at this stage */
- HUF_encodeSymbol(&bitC, ip[n - 1], CTable);
- HUF_FLUSHBITS_1(&bitC);
- HUF_encodeSymbol(&bitC, ip[n - 2], CTable);
- HUF_FLUSHBITS_2(&bitC);
- HUF_encodeSymbol(&bitC, ip[n - 3], CTable);
- HUF_FLUSHBITS_1(&bitC);
- HUF_encodeSymbol(&bitC, ip[n - 4], CTable);
- HUF_FLUSHBITS(&bitC);
- }
-
- return BIT_closeCStream(&bitC);
-}
-
-size_t HUF_compress4X_usingCTable(void *dst, size_t dstSize, const void *src, size_t srcSize, const HUF_CElt *CTable)
-{
- size_t const segmentSize = (srcSize + 3) / 4; /* first 3 segments */
- const BYTE *ip = (const BYTE *)src;
- const BYTE *const iend = ip + srcSize;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- BYTE *op = ostart;
-
- if (dstSize < 6 + 1 + 1 + 1 + 8)
- return 0; /* minimum space to compress successfully */
- if (srcSize < 12)
- return 0; /* no saving possible : too small input */
- op += 6; /* jumpTable */
-
- {
- CHECK_V_F(cSize, HUF_compress1X_usingCTable(op, oend - op, ip, segmentSize, CTable));
- if (cSize == 0)
- return 0;
- ZSTD_writeLE16(ostart, (U16)cSize);
- op += cSize;
- }
-
- ip += segmentSize;
- {
- CHECK_V_F(cSize, HUF_compress1X_usingCTable(op, oend - op, ip, segmentSize, CTable));
- if (cSize == 0)
- return 0;
- ZSTD_writeLE16(ostart + 2, (U16)cSize);
- op += cSize;
- }
-
- ip += segmentSize;
- {
- CHECK_V_F(cSize, HUF_compress1X_usingCTable(op, oend - op, ip, segmentSize, CTable));
- if (cSize == 0)
- return 0;
- ZSTD_writeLE16(ostart + 4, (U16)cSize);
- op += cSize;
- }
-
- ip += segmentSize;
- {
- CHECK_V_F(cSize, HUF_compress1X_usingCTable(op, oend - op, ip, iend - ip, CTable));
- if (cSize == 0)
- return 0;
- op += cSize;
- }
-
- return op - ostart;
-}
-
-static size_t HUF_compressCTable_internal(BYTE *const ostart, BYTE *op, BYTE *const oend, const void *src, size_t srcSize, unsigned singleStream,
- const HUF_CElt *CTable)
-{
- size_t const cSize =
- singleStream ? HUF_compress1X_usingCTable(op, oend - op, src, srcSize, CTable) : HUF_compress4X_usingCTable(op, oend - op, src, srcSize, CTable);
- if (HUF_isError(cSize)) {
- return cSize;
- }
- if (cSize == 0) {
- return 0;
- } /* uncompressible */
- op += cSize;
- /* check compressibility */
- if ((size_t)(op - ostart) >= srcSize - 1) {
- return 0;
- }
- return op - ostart;
-}
-
-/* `workSpace` must a table of at least 1024 unsigned */
-static size_t HUF_compress_internal(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog,
- unsigned singleStream, void *workSpace, size_t wkspSize, HUF_CElt *oldHufTable, HUF_repeat *repeat, int preferRepeat)
-{
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- BYTE *op = ostart;
-
- U32 *count;
- size_t const countSize = sizeof(U32) * (HUF_SYMBOLVALUE_MAX + 1);
- HUF_CElt *CTable;
- size_t const CTableSize = sizeof(HUF_CElt) * (HUF_SYMBOLVALUE_MAX + 1);
-
- /* checks & inits */
- if (wkspSize < sizeof(huffNodeTable) + countSize + CTableSize)
- return ERROR(GENERIC);
- if (!srcSize)
- return 0; /* Uncompressed (note : 1 means rle, so first byte must be correct) */
- if (!dstSize)
- return 0; /* cannot fit within dst budget */
- if (srcSize > HUF_BLOCKSIZE_MAX)
- return ERROR(srcSize_wrong); /* curr block size limit */
- if (huffLog > HUF_TABLELOG_MAX)
- return ERROR(tableLog_tooLarge);
- if (!maxSymbolValue)
- maxSymbolValue = HUF_SYMBOLVALUE_MAX;
- if (!huffLog)
- huffLog = HUF_TABLELOG_DEFAULT;
-
- count = (U32 *)workSpace;
- workSpace = (BYTE *)workSpace + countSize;
- wkspSize -= countSize;
- CTable = (HUF_CElt *)workSpace;
- workSpace = (BYTE *)workSpace + CTableSize;
- wkspSize -= CTableSize;
-
- /* Heuristic : If we don't need to check the validity of the old table use the old table for small inputs */
- if (preferRepeat && repeat && *repeat == HUF_repeat_valid) {
- return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, singleStream, oldHufTable);
- }
-
- /* Scan input and build symbol stats */
- {
- CHECK_V_F(largest, FSE_count_wksp(count, &maxSymbolValue, (const BYTE *)src, srcSize, (U32 *)workSpace));
- if (largest == srcSize) {
- *ostart = ((const BYTE *)src)[0];
- return 1;
- } /* single symbol, rle */
- if (largest <= (srcSize >> 7) + 1)
- return 0; /* Fast heuristic : not compressible enough */
- }
-
- /* Check validity of previous table */
- if (repeat && *repeat == HUF_repeat_check && !HUF_validateCTable(oldHufTable, count, maxSymbolValue)) {
- *repeat = HUF_repeat_none;
- }
- /* Heuristic : use existing table for small inputs */
- if (preferRepeat && repeat && *repeat != HUF_repeat_none) {
- return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, singleStream, oldHufTable);
- }
-
- /* Build Huffman Tree */
- huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
- {
- CHECK_V_F(maxBits, HUF_buildCTable_wksp(CTable, count, maxSymbolValue, huffLog, workSpace, wkspSize));
- huffLog = (U32)maxBits;
- /* Zero the unused symbols so we can check it for validity */
- memset(CTable + maxSymbolValue + 1, 0, CTableSize - (maxSymbolValue + 1) * sizeof(HUF_CElt));
- }
-
- /* Write table description header */
- {
- CHECK_V_F(hSize, HUF_writeCTable_wksp(op, dstSize, CTable, maxSymbolValue, huffLog, workSpace, wkspSize));
- /* Check if using the previous table will be beneficial */
- if (repeat && *repeat != HUF_repeat_none) {
- size_t const oldSize = HUF_estimateCompressedSize(oldHufTable, count, maxSymbolValue);
- size_t const newSize = HUF_estimateCompressedSize(CTable, count, maxSymbolValue);
- if (oldSize <= hSize + newSize || hSize + 12 >= srcSize) {
- return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, singleStream, oldHufTable);
- }
- }
- /* Use the new table */
- if (hSize + 12ul >= srcSize) {
- return 0;
- }
- op += hSize;
- if (repeat) {
- *repeat = HUF_repeat_none;
- }
- if (oldHufTable) {
- memcpy(oldHufTable, CTable, CTableSize);
- } /* Save the new table */
- }
- return HUF_compressCTable_internal(ostart, op, oend, src, srcSize, singleStream, CTable);
-}
-
-size_t HUF_compress1X_wksp(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void *workSpace,
- size_t wkspSize)
-{
- return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, 1 /* single stream */, workSpace, wkspSize, NULL, NULL, 0);
-}
-
-size_t HUF_compress1X_repeat(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void *workSpace,
- size_t wkspSize, HUF_CElt *hufTable, HUF_repeat *repeat, int preferRepeat)
-{
- return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, 1 /* single stream */, workSpace, wkspSize, hufTable, repeat,
- preferRepeat);
-}
-
-size_t HUF_compress4X_wksp(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void *workSpace,
- size_t wkspSize)
-{
- return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, 0 /* 4 streams */, workSpace, wkspSize, NULL, NULL, 0);
-}
-
-size_t HUF_compress4X_repeat(void *dst, size_t dstSize, const void *src, size_t srcSize, unsigned maxSymbolValue, unsigned huffLog, void *workSpace,
- size_t wkspSize, HUF_CElt *hufTable, HUF_repeat *repeat, int preferRepeat)
-{
- return HUF_compress_internal(dst, dstSize, src, srcSize, maxSymbolValue, huffLog, 0 /* 4 streams */, workSpace, wkspSize, hufTable, repeat,
- preferRepeat);
-}
+++ /dev/null
-/*
- * Huffman decoder, part of New Generation Entropy library
- * Copyright (C) 2013-2016, Yann Collet.
- *
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions are
- * met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following disclaimer
- * in the documentation and/or other materials provided with the
- * distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
- * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
- * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
- * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
- * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
- * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
- * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
- * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
- * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- *
- * You can contact the author at :
- * - Source repository : https://github.com/Cyan4973/FiniteStateEntropy
- */
-
-/* **************************************************************
-* Compiler specifics
-****************************************************************/
-#define FORCE_INLINE static __always_inline
-
-/* **************************************************************
-* Dependencies
-****************************************************************/
-#include "bitstream.h" /* BIT_* */
-#include "fse.h" /* header compression */
-#include "huf.h"
-#include <linux/compiler.h>
-#include <linux/kernel.h>
-#include <linux/string.h> /* memcpy, memset */
-
-/* **************************************************************
-* Error Management
-****************************************************************/
-#define HUF_STATIC_ASSERT(c) \
- { \
- enum { HUF_static_assert = 1 / (int)(!!(c)) }; \
- } /* use only *after* variable declarations */
-
-/*-***************************/
-/* generic DTableDesc */
-/*-***************************/
-
-typedef struct {
- BYTE maxTableLog;
- BYTE tableType;
- BYTE tableLog;
- BYTE reserved;
-} DTableDesc;
-
-static DTableDesc HUF_getDTableDesc(const HUF_DTable *table)
-{
- DTableDesc dtd;
- memcpy(&dtd, table, sizeof(dtd));
- return dtd;
-}
-
-/*-***************************/
-/* single-symbol decoding */
-/*-***************************/
-
-typedef struct {
- BYTE byte;
- BYTE nbBits;
-} HUF_DEltX2; /* single-symbol decoding */
-
-size_t HUF_readDTableX2_wksp(HUF_DTable *DTable, const void *src, size_t srcSize, void *workspace, size_t workspaceSize)
-{
- U32 tableLog = 0;
- U32 nbSymbols = 0;
- size_t iSize;
- void *const dtPtr = DTable + 1;
- HUF_DEltX2 *const dt = (HUF_DEltX2 *)dtPtr;
-
- U32 *rankVal;
- BYTE *huffWeight;
- size_t spaceUsed32 = 0;
-
- rankVal = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += HUF_TABLELOG_ABSOLUTEMAX + 1;
- huffWeight = (BYTE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(HUF_SYMBOLVALUE_MAX + 1, sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- HUF_STATIC_ASSERT(sizeof(DTableDesc) == sizeof(HUF_DTable));
- /* memset(huffWeight, 0, sizeof(huffWeight)); */ /* is not necessary, even though some analyzer complain ... */
-
- iSize = HUF_readStats_wksp(huffWeight, HUF_SYMBOLVALUE_MAX + 1, rankVal, &nbSymbols, &tableLog, src, srcSize, workspace, workspaceSize);
- if (HUF_isError(iSize))
- return iSize;
-
- /* Table header */
- {
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- if (tableLog > (U32)(dtd.maxTableLog + 1))
- return ERROR(tableLog_tooLarge); /* DTable too small, Huffman tree cannot fit in */
- dtd.tableType = 0;
- dtd.tableLog = (BYTE)tableLog;
- memcpy(DTable, &dtd, sizeof(dtd));
- }
-
- /* Calculate starting value for each rank */
- {
- U32 n, nextRankStart = 0;
- for (n = 1; n < tableLog + 1; n++) {
- U32 const curr = nextRankStart;
- nextRankStart += (rankVal[n] << (n - 1));
- rankVal[n] = curr;
- }
- }
-
- /* fill DTable */
- {
- U32 n;
- for (n = 0; n < nbSymbols; n++) {
- U32 const w = huffWeight[n];
- U32 const length = (1 << w) >> 1;
- U32 u;
- HUF_DEltX2 D;
- D.byte = (BYTE)n;
- D.nbBits = (BYTE)(tableLog + 1 - w);
- for (u = rankVal[w]; u < rankVal[w] + length; u++)
- dt[u] = D;
- rankVal[w] += length;
- }
- }
-
- return iSize;
-}
-
-static BYTE HUF_decodeSymbolX2(BIT_DStream_t *Dstream, const HUF_DEltX2 *dt, const U32 dtLog)
-{
- size_t const val = BIT_lookBitsFast(Dstream, dtLog); /* note : dtLog >= 1 */
- BYTE const c = dt[val].byte;
- BIT_skipBits(Dstream, dt[val].nbBits);
- return c;
-}
-
-#define HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr) *ptr++ = HUF_decodeSymbolX2(DStreamPtr, dt, dtLog)
-
-#define HUF_DECODE_SYMBOLX2_1(ptr, DStreamPtr) \
- if (ZSTD_64bits() || (HUF_TABLELOG_MAX <= 12)) \
- HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr)
-
-#define HUF_DECODE_SYMBOLX2_2(ptr, DStreamPtr) \
- if (ZSTD_64bits()) \
- HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr)
-
-FORCE_INLINE size_t HUF_decodeStreamX2(BYTE *p, BIT_DStream_t *const bitDPtr, BYTE *const pEnd, const HUF_DEltX2 *const dt, const U32 dtLog)
-{
- BYTE *const pStart = p;
-
- /* up to 4 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) && (p <= pEnd - 4)) {
- HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_1(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
- }
-
- /* closer to the end */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) && (p < pEnd))
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
-
- /* no more data to retrieve from bitstream, hence no need to reload */
- while (p < pEnd)
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
-
- return pEnd - pStart;
-}
-
-static size_t HUF_decompress1X2_usingDTable_internal(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- BYTE *op = (BYTE *)dst;
- BYTE *const oend = op + dstSize;
- const void *dtPtr = DTable + 1;
- const HUF_DEltX2 *const dt = (const HUF_DEltX2 *)dtPtr;
- BIT_DStream_t bitD;
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- U32 const dtLog = dtd.tableLog;
-
- {
- size_t const errorCode = BIT_initDStream(&bitD, cSrc, cSrcSize);
- if (HUF_isError(errorCode))
- return errorCode;
- }
-
- HUF_decodeStreamX2(op, &bitD, oend, dt, dtLog);
-
- /* check */
- if (!BIT_endOfDStream(&bitD))
- return ERROR(corruption_detected);
-
- return dstSize;
-}
-
-size_t HUF_decompress1X2_usingDTable(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- if (dtd.tableType != 0)
- return ERROR(GENERIC);
- return HUF_decompress1X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable);
-}
-
-size_t HUF_decompress1X2_DCtx_wksp(HUF_DTable *DCtx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- const BYTE *ip = (const BYTE *)cSrc;
-
- size_t const hSize = HUF_readDTableX2_wksp(DCtx, cSrc, cSrcSize, workspace, workspaceSize);
- if (HUF_isError(hSize))
- return hSize;
- if (hSize >= cSrcSize)
- return ERROR(srcSize_wrong);
- ip += hSize;
- cSrcSize -= hSize;
-
- return HUF_decompress1X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx);
-}
-
-static size_t HUF_decompress4X2_usingDTable_internal(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- /* Check */
- if (cSrcSize < 10)
- return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */
-
- {
- const BYTE *const istart = (const BYTE *)cSrc;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- const void *const dtPtr = DTable + 1;
- const HUF_DEltX2 *const dt = (const HUF_DEltX2 *)dtPtr;
-
- /* Init */
- BIT_DStream_t bitD1;
- BIT_DStream_t bitD2;
- BIT_DStream_t bitD3;
- BIT_DStream_t bitD4;
- size_t const length1 = ZSTD_readLE16(istart);
- size_t const length2 = ZSTD_readLE16(istart + 2);
- size_t const length3 = ZSTD_readLE16(istart + 4);
- size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
- const BYTE *const istart1 = istart + 6; /* jumpTable */
- const BYTE *const istart2 = istart1 + length1;
- const BYTE *const istart3 = istart2 + length2;
- const BYTE *const istart4 = istart3 + length3;
- const size_t segmentSize = (dstSize + 3) / 4;
- BYTE *const opStart2 = ostart + segmentSize;
- BYTE *const opStart3 = opStart2 + segmentSize;
- BYTE *const opStart4 = opStart3 + segmentSize;
- BYTE *op1 = ostart;
- BYTE *op2 = opStart2;
- BYTE *op3 = opStart3;
- BYTE *op4 = opStart4;
- U32 endSignal;
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- U32 const dtLog = dtd.tableLog;
-
- if (length4 > cSrcSize)
- return ERROR(corruption_detected); /* overflow */
- {
- size_t const errorCode = BIT_initDStream(&bitD1, istart1, length1);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD2, istart2, length2);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD3, istart3, length3);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD4, istart4, length4);
- if (HUF_isError(errorCode))
- return errorCode;
- }
-
- /* 16-32 symbols per loop (4-8 symbols per stream) */
- endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
- for (; (endSignal == BIT_DStream_unfinished) && (op4 < (oend - 7));) {
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
- endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
- }
-
- /* check corruption */
- if (op1 > opStart2)
- return ERROR(corruption_detected);
- if (op2 > opStart3)
- return ERROR(corruption_detected);
- if (op3 > opStart4)
- return ERROR(corruption_detected);
- /* note : op4 supposed already verified within main loop */
-
- /* finish bitStreams one by one */
- HUF_decodeStreamX2(op1, &bitD1, opStart2, dt, dtLog);
- HUF_decodeStreamX2(op2, &bitD2, opStart3, dt, dtLog);
- HUF_decodeStreamX2(op3, &bitD3, opStart4, dt, dtLog);
- HUF_decodeStreamX2(op4, &bitD4, oend, dt, dtLog);
-
- /* check */
- endSignal = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
- if (!endSignal)
- return ERROR(corruption_detected);
-
- /* decoded size */
- return dstSize;
- }
-}
-
-size_t HUF_decompress4X2_usingDTable(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- if (dtd.tableType != 0)
- return ERROR(GENERIC);
- return HUF_decompress4X2_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable);
-}
-
-size_t HUF_decompress4X2_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- const BYTE *ip = (const BYTE *)cSrc;
-
- size_t const hSize = HUF_readDTableX2_wksp(dctx, cSrc, cSrcSize, workspace, workspaceSize);
- if (HUF_isError(hSize))
- return hSize;
- if (hSize >= cSrcSize)
- return ERROR(srcSize_wrong);
- ip += hSize;
- cSrcSize -= hSize;
-
- return HUF_decompress4X2_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx);
-}
-
-/* *************************/
-/* double-symbols decoding */
-/* *************************/
-typedef struct {
- U16 sequence;
- BYTE nbBits;
- BYTE length;
-} HUF_DEltX4; /* double-symbols decoding */
-
-typedef struct {
- BYTE symbol;
- BYTE weight;
-} sortedSymbol_t;
-
-/* HUF_fillDTableX4Level2() :
- * `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */
-static void HUF_fillDTableX4Level2(HUF_DEltX4 *DTable, U32 sizeLog, const U32 consumed, const U32 *rankValOrigin, const int minWeight,
- const sortedSymbol_t *sortedSymbols, const U32 sortedListSize, U32 nbBitsBaseline, U16 baseSeq)
-{
- HUF_DEltX4 DElt;
- U32 rankVal[HUF_TABLELOG_MAX + 1];
-
- /* get pre-calculated rankVal */
- memcpy(rankVal, rankValOrigin, sizeof(rankVal));
-
- /* fill skipped values */
- if (minWeight > 1) {
- U32 i, skipSize = rankVal[minWeight];
- ZSTD_writeLE16(&(DElt.sequence), baseSeq);
- DElt.nbBits = (BYTE)(consumed);
- DElt.length = 1;
- for (i = 0; i < skipSize; i++)
- DTable[i] = DElt;
- }
-
- /* fill DTable */
- {
- U32 s;
- for (s = 0; s < sortedListSize; s++) { /* note : sortedSymbols already skipped */
- const U32 symbol = sortedSymbols[s].symbol;
- const U32 weight = sortedSymbols[s].weight;
- const U32 nbBits = nbBitsBaseline - weight;
- const U32 length = 1 << (sizeLog - nbBits);
- const U32 start = rankVal[weight];
- U32 i = start;
- const U32 end = start + length;
-
- ZSTD_writeLE16(&(DElt.sequence), (U16)(baseSeq + (symbol << 8)));
- DElt.nbBits = (BYTE)(nbBits + consumed);
- DElt.length = 2;
- do {
- DTable[i++] = DElt;
- } while (i < end); /* since length >= 1 */
-
- rankVal[weight] += length;
- }
- }
-}
-
-typedef U32 rankVal_t[HUF_TABLELOG_MAX][HUF_TABLELOG_MAX + 1];
-typedef U32 rankValCol_t[HUF_TABLELOG_MAX + 1];
-
-static void HUF_fillDTableX4(HUF_DEltX4 *DTable, const U32 targetLog, const sortedSymbol_t *sortedList, const U32 sortedListSize, const U32 *rankStart,
- rankVal_t rankValOrigin, const U32 maxWeight, const U32 nbBitsBaseline)
-{
- U32 rankVal[HUF_TABLELOG_MAX + 1];
- const int scaleLog = nbBitsBaseline - targetLog; /* note : targetLog >= srcLog, hence scaleLog <= 1 */
- const U32 minBits = nbBitsBaseline - maxWeight;
- U32 s;
-
- memcpy(rankVal, rankValOrigin, sizeof(rankVal));
-
- /* fill DTable */
- for (s = 0; s < sortedListSize; s++) {
- const U16 symbol = sortedList[s].symbol;
- const U32 weight = sortedList[s].weight;
- const U32 nbBits = nbBitsBaseline - weight;
- const U32 start = rankVal[weight];
- const U32 length = 1 << (targetLog - nbBits);
-
- if (targetLog - nbBits >= minBits) { /* enough room for a second symbol */
- U32 sortedRank;
- int minWeight = nbBits + scaleLog;
- if (minWeight < 1)
- minWeight = 1;
- sortedRank = rankStart[minWeight];
- HUF_fillDTableX4Level2(DTable + start, targetLog - nbBits, nbBits, rankValOrigin[nbBits], minWeight, sortedList + sortedRank,
- sortedListSize - sortedRank, nbBitsBaseline, symbol);
- } else {
- HUF_DEltX4 DElt;
- ZSTD_writeLE16(&(DElt.sequence), symbol);
- DElt.nbBits = (BYTE)(nbBits);
- DElt.length = 1;
- {
- U32 const end = start + length;
- U32 u;
- for (u = start; u < end; u++)
- DTable[u] = DElt;
- }
- }
- rankVal[weight] += length;
- }
-}
-
-size_t HUF_readDTableX4_wksp(HUF_DTable *DTable, const void *src, size_t srcSize, void *workspace, size_t workspaceSize)
-{
- U32 tableLog, maxW, sizeOfSort, nbSymbols;
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- U32 const maxTableLog = dtd.maxTableLog;
- size_t iSize;
- void *dtPtr = DTable + 1; /* force compiler to avoid strict-aliasing */
- HUF_DEltX4 *const dt = (HUF_DEltX4 *)dtPtr;
- U32 *rankStart;
-
- rankValCol_t *rankVal;
- U32 *rankStats;
- U32 *rankStart0;
- sortedSymbol_t *sortedSymbol;
- BYTE *weightList;
- size_t spaceUsed32 = 0;
-
- HUF_STATIC_ASSERT((sizeof(rankValCol_t) & 3) == 0);
-
- rankVal = (rankValCol_t *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += (sizeof(rankValCol_t) * HUF_TABLELOG_MAX) >> 2;
- rankStats = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += HUF_TABLELOG_MAX + 1;
- rankStart0 = (U32 *)workspace + spaceUsed32;
- spaceUsed32 += HUF_TABLELOG_MAX + 2;
- sortedSymbol = (sortedSymbol_t *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(sizeof(sortedSymbol_t) * (HUF_SYMBOLVALUE_MAX + 1), sizeof(U32)) >> 2;
- weightList = (BYTE *)((U32 *)workspace + spaceUsed32);
- spaceUsed32 += ALIGN(HUF_SYMBOLVALUE_MAX + 1, sizeof(U32)) >> 2;
-
- if ((spaceUsed32 << 2) > workspaceSize)
- return ERROR(tableLog_tooLarge);
- workspace = (U32 *)workspace + spaceUsed32;
- workspaceSize -= (spaceUsed32 << 2);
-
- rankStart = rankStart0 + 1;
- memset(rankStats, 0, sizeof(U32) * (2 * HUF_TABLELOG_MAX + 2 + 1));
-
- HUF_STATIC_ASSERT(sizeof(HUF_DEltX4) == sizeof(HUF_DTable)); /* if compiler fails here, assertion is wrong */
- if (maxTableLog > HUF_TABLELOG_MAX)
- return ERROR(tableLog_tooLarge);
- /* memset(weightList, 0, sizeof(weightList)); */ /* is not necessary, even though some analyzer complain ... */
-
- iSize = HUF_readStats_wksp(weightList, HUF_SYMBOLVALUE_MAX + 1, rankStats, &nbSymbols, &tableLog, src, srcSize, workspace, workspaceSize);
- if (HUF_isError(iSize))
- return iSize;
-
- /* check result */
- if (tableLog > maxTableLog)
- return ERROR(tableLog_tooLarge); /* DTable can't fit code depth */
-
- /* find maxWeight */
- for (maxW = tableLog; rankStats[maxW] == 0; maxW--) {
- } /* necessarily finds a solution before 0 */
-
- /* Get start index of each weight */
- {
- U32 w, nextRankStart = 0;
- for (w = 1; w < maxW + 1; w++) {
- U32 curr = nextRankStart;
- nextRankStart += rankStats[w];
- rankStart[w] = curr;
- }
- rankStart[0] = nextRankStart; /* put all 0w symbols at the end of sorted list*/
- sizeOfSort = nextRankStart;
- }
-
- /* sort symbols by weight */
- {
- U32 s;
- for (s = 0; s < nbSymbols; s++) {
- U32 const w = weightList[s];
- U32 const r = rankStart[w]++;
- sortedSymbol[r].symbol = (BYTE)s;
- sortedSymbol[r].weight = (BYTE)w;
- }
- rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
- }
-
- /* Build rankVal */
- {
- U32 *const rankVal0 = rankVal[0];
- {
- int const rescale = (maxTableLog - tableLog) - 1; /* tableLog <= maxTableLog */
- U32 nextRankVal = 0;
- U32 w;
- for (w = 1; w < maxW + 1; w++) {
- U32 curr = nextRankVal;
- nextRankVal += rankStats[w] << (w + rescale);
- rankVal0[w] = curr;
- }
- }
- {
- U32 const minBits = tableLog + 1 - maxW;
- U32 consumed;
- for (consumed = minBits; consumed < maxTableLog - minBits + 1; consumed++) {
- U32 *const rankValPtr = rankVal[consumed];
- U32 w;
- for (w = 1; w < maxW + 1; w++) {
- rankValPtr[w] = rankVal0[w] >> consumed;
- }
- }
- }
- }
-
- HUF_fillDTableX4(dt, maxTableLog, sortedSymbol, sizeOfSort, rankStart0, rankVal, maxW, tableLog + 1);
-
- dtd.tableLog = (BYTE)maxTableLog;
- dtd.tableType = 1;
- memcpy(DTable, &dtd, sizeof(dtd));
- return iSize;
-}
-
-static U32 HUF_decodeSymbolX4(void *op, BIT_DStream_t *DStream, const HUF_DEltX4 *dt, const U32 dtLog)
-{
- size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
- memcpy(op, dt + val, 2);
- BIT_skipBits(DStream, dt[val].nbBits);
- return dt[val].length;
-}
-
-static U32 HUF_decodeLastSymbolX4(void *op, BIT_DStream_t *DStream, const HUF_DEltX4 *dt, const U32 dtLog)
-{
- size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
- memcpy(op, dt + val, 1);
- if (dt[val].length == 1)
- BIT_skipBits(DStream, dt[val].nbBits);
- else {
- if (DStream->bitsConsumed < (sizeof(DStream->bitContainer) * 8)) {
- BIT_skipBits(DStream, dt[val].nbBits);
- if (DStream->bitsConsumed > (sizeof(DStream->bitContainer) * 8))
- /* ugly hack; works only because it's the last symbol. Note : can't easily extract nbBits from just this symbol */
- DStream->bitsConsumed = (sizeof(DStream->bitContainer) * 8);
- }
- }
- return 1;
-}
-
-#define HUF_DECODE_SYMBOLX4_0(ptr, DStreamPtr) ptr += HUF_decodeSymbolX4(ptr, DStreamPtr, dt, dtLog)
-
-#define HUF_DECODE_SYMBOLX4_1(ptr, DStreamPtr) \
- if (ZSTD_64bits() || (HUF_TABLELOG_MAX <= 12)) \
- ptr += HUF_decodeSymbolX4(ptr, DStreamPtr, dt, dtLog)
-
-#define HUF_DECODE_SYMBOLX4_2(ptr, DStreamPtr) \
- if (ZSTD_64bits()) \
- ptr += HUF_decodeSymbolX4(ptr, DStreamPtr, dt, dtLog)
-
-FORCE_INLINE size_t HUF_decodeStreamX4(BYTE *p, BIT_DStream_t *bitDPtr, BYTE *const pEnd, const HUF_DEltX4 *const dt, const U32 dtLog)
-{
- BYTE *const pStart = p;
-
- /* up to 8 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd - (sizeof(bitDPtr->bitContainer) - 1))) {
- HUF_DECODE_SYMBOLX4_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX4_1(p, bitDPtr);
- HUF_DECODE_SYMBOLX4_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX4_0(p, bitDPtr);
- }
-
- /* closer to end : up to 2 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd - 2))
- HUF_DECODE_SYMBOLX4_0(p, bitDPtr);
-
- while (p <= pEnd - 2)
- HUF_DECODE_SYMBOLX4_0(p, bitDPtr); /* no need to reload : reached the end of DStream */
-
- if (p < pEnd)
- p += HUF_decodeLastSymbolX4(p, bitDPtr, dt, dtLog);
-
- return p - pStart;
-}
-
-static size_t HUF_decompress1X4_usingDTable_internal(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- BIT_DStream_t bitD;
-
- /* Init */
- {
- size_t const errorCode = BIT_initDStream(&bitD, cSrc, cSrcSize);
- if (HUF_isError(errorCode))
- return errorCode;
- }
-
- /* decode */
- {
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- const void *const dtPtr = DTable + 1; /* force compiler to not use strict-aliasing */
- const HUF_DEltX4 *const dt = (const HUF_DEltX4 *)dtPtr;
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- HUF_decodeStreamX4(ostart, &bitD, oend, dt, dtd.tableLog);
- }
-
- /* check */
- if (!BIT_endOfDStream(&bitD))
- return ERROR(corruption_detected);
-
- /* decoded size */
- return dstSize;
-}
-
-size_t HUF_decompress1X4_usingDTable(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- if (dtd.tableType != 1)
- return ERROR(GENERIC);
- return HUF_decompress1X4_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable);
-}
-
-size_t HUF_decompress1X4_DCtx_wksp(HUF_DTable *DCtx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- const BYTE *ip = (const BYTE *)cSrc;
-
- size_t const hSize = HUF_readDTableX4_wksp(DCtx, cSrc, cSrcSize, workspace, workspaceSize);
- if (HUF_isError(hSize))
- return hSize;
- if (hSize >= cSrcSize)
- return ERROR(srcSize_wrong);
- ip += hSize;
- cSrcSize -= hSize;
-
- return HUF_decompress1X4_usingDTable_internal(dst, dstSize, ip, cSrcSize, DCtx);
-}
-
-static size_t HUF_decompress4X4_usingDTable_internal(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- if (cSrcSize < 10)
- return ERROR(corruption_detected); /* strict minimum : jump table + 1 byte per stream */
-
- {
- const BYTE *const istart = (const BYTE *)cSrc;
- BYTE *const ostart = (BYTE *)dst;
- BYTE *const oend = ostart + dstSize;
- const void *const dtPtr = DTable + 1;
- const HUF_DEltX4 *const dt = (const HUF_DEltX4 *)dtPtr;
-
- /* Init */
- BIT_DStream_t bitD1;
- BIT_DStream_t bitD2;
- BIT_DStream_t bitD3;
- BIT_DStream_t bitD4;
- size_t const length1 = ZSTD_readLE16(istart);
- size_t const length2 = ZSTD_readLE16(istart + 2);
- size_t const length3 = ZSTD_readLE16(istart + 4);
- size_t const length4 = cSrcSize - (length1 + length2 + length3 + 6);
- const BYTE *const istart1 = istart + 6; /* jumpTable */
- const BYTE *const istart2 = istart1 + length1;
- const BYTE *const istart3 = istart2 + length2;
- const BYTE *const istart4 = istart3 + length3;
- size_t const segmentSize = (dstSize + 3) / 4;
- BYTE *const opStart2 = ostart + segmentSize;
- BYTE *const opStart3 = opStart2 + segmentSize;
- BYTE *const opStart4 = opStart3 + segmentSize;
- BYTE *op1 = ostart;
- BYTE *op2 = opStart2;
- BYTE *op3 = opStart3;
- BYTE *op4 = opStart4;
- U32 endSignal;
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- U32 const dtLog = dtd.tableLog;
-
- if (length4 > cSrcSize)
- return ERROR(corruption_detected); /* overflow */
- {
- size_t const errorCode = BIT_initDStream(&bitD1, istart1, length1);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD2, istart2, length2);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD3, istart3, length3);
- if (HUF_isError(errorCode))
- return errorCode;
- }
- {
- size_t const errorCode = BIT_initDStream(&bitD4, istart4, length4);
- if (HUF_isError(errorCode))
- return errorCode;
- }
-
- /* 16-32 symbols per loop (4-8 symbols per stream) */
- endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
- for (; (endSignal == BIT_DStream_unfinished) & (op4 < (oend - (sizeof(bitD4.bitContainer) - 1)));) {
- HUF_DECODE_SYMBOLX4_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX4_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX4_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX4_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX4_1(op1, &bitD1);
- HUF_DECODE_SYMBOLX4_1(op2, &bitD2);
- HUF_DECODE_SYMBOLX4_1(op3, &bitD3);
- HUF_DECODE_SYMBOLX4_1(op4, &bitD4);
- HUF_DECODE_SYMBOLX4_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX4_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX4_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX4_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX4_0(op1, &bitD1);
- HUF_DECODE_SYMBOLX4_0(op2, &bitD2);
- HUF_DECODE_SYMBOLX4_0(op3, &bitD3);
- HUF_DECODE_SYMBOLX4_0(op4, &bitD4);
-
- endSignal = BIT_reloadDStream(&bitD1) | BIT_reloadDStream(&bitD2) | BIT_reloadDStream(&bitD3) | BIT_reloadDStream(&bitD4);
- }
-
- /* check corruption */
- if (op1 > opStart2)
- return ERROR(corruption_detected);
- if (op2 > opStart3)
- return ERROR(corruption_detected);
- if (op3 > opStart4)
- return ERROR(corruption_detected);
- /* note : op4 already verified within main loop */
-
- /* finish bitStreams one by one */
- HUF_decodeStreamX4(op1, &bitD1, opStart2, dt, dtLog);
- HUF_decodeStreamX4(op2, &bitD2, opStart3, dt, dtLog);
- HUF_decodeStreamX4(op3, &bitD3, opStart4, dt, dtLog);
- HUF_decodeStreamX4(op4, &bitD4, oend, dt, dtLog);
-
- /* check */
- {
- U32 const endCheck = BIT_endOfDStream(&bitD1) & BIT_endOfDStream(&bitD2) & BIT_endOfDStream(&bitD3) & BIT_endOfDStream(&bitD4);
- if (!endCheck)
- return ERROR(corruption_detected);
- }
-
- /* decoded size */
- return dstSize;
- }
-}
-
-size_t HUF_decompress4X4_usingDTable(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc dtd = HUF_getDTableDesc(DTable);
- if (dtd.tableType != 1)
- return ERROR(GENERIC);
- return HUF_decompress4X4_usingDTable_internal(dst, dstSize, cSrc, cSrcSize, DTable);
-}
-
-size_t HUF_decompress4X4_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- const BYTE *ip = (const BYTE *)cSrc;
-
- size_t hSize = HUF_readDTableX4_wksp(dctx, cSrc, cSrcSize, workspace, workspaceSize);
- if (HUF_isError(hSize))
- return hSize;
- if (hSize >= cSrcSize)
- return ERROR(srcSize_wrong);
- ip += hSize;
- cSrcSize -= hSize;
-
- return HUF_decompress4X4_usingDTable_internal(dst, dstSize, ip, cSrcSize, dctx);
-}
-
-/* ********************************/
-/* Generic decompression selector */
-/* ********************************/
-
-size_t HUF_decompress1X_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- return dtd.tableType ? HUF_decompress1X4_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable)
- : HUF_decompress1X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable);
-}
-
-size_t HUF_decompress4X_usingDTable(void *dst, size_t maxDstSize, const void *cSrc, size_t cSrcSize, const HUF_DTable *DTable)
-{
- DTableDesc const dtd = HUF_getDTableDesc(DTable);
- return dtd.tableType ? HUF_decompress4X4_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable)
- : HUF_decompress4X2_usingDTable_internal(dst, maxDstSize, cSrc, cSrcSize, DTable);
-}
-
-typedef struct {
- U32 tableTime;
- U32 decode256Time;
-} algo_time_t;
-static const algo_time_t algoTime[16 /* Quantization */][3 /* single, double, quad */] = {
- /* single, double, quad */
- {{0, 0}, {1, 1}, {2, 2}}, /* Q==0 : impossible */
- {{0, 0}, {1, 1}, {2, 2}}, /* Q==1 : impossible */
- {{38, 130}, {1313, 74}, {2151, 38}}, /* Q == 2 : 12-18% */
- {{448, 128}, {1353, 74}, {2238, 41}}, /* Q == 3 : 18-25% */
- {{556, 128}, {1353, 74}, {2238, 47}}, /* Q == 4 : 25-32% */
- {{714, 128}, {1418, 74}, {2436, 53}}, /* Q == 5 : 32-38% */
- {{883, 128}, {1437, 74}, {2464, 61}}, /* Q == 6 : 38-44% */
- {{897, 128}, {1515, 75}, {2622, 68}}, /* Q == 7 : 44-50% */
- {{926, 128}, {1613, 75}, {2730, 75}}, /* Q == 8 : 50-56% */
- {{947, 128}, {1729, 77}, {3359, 77}}, /* Q == 9 : 56-62% */
- {{1107, 128}, {2083, 81}, {4006, 84}}, /* Q ==10 : 62-69% */
- {{1177, 128}, {2379, 87}, {4785, 88}}, /* Q ==11 : 69-75% */
- {{1242, 128}, {2415, 93}, {5155, 84}}, /* Q ==12 : 75-81% */
- {{1349, 128}, {2644, 106}, {5260, 106}}, /* Q ==13 : 81-87% */
- {{1455, 128}, {2422, 124}, {4174, 124}}, /* Q ==14 : 87-93% */
- {{722, 128}, {1891, 145}, {1936, 146}}, /* Q ==15 : 93-99% */
-};
-
-/** HUF_selectDecoder() :
-* Tells which decoder is likely to decode faster,
-* based on a set of pre-determined metrics.
-* @return : 0==HUF_decompress4X2, 1==HUF_decompress4X4 .
-* Assumption : 0 < cSrcSize < dstSize <= 128 KB */
-U32 HUF_selectDecoder(size_t dstSize, size_t cSrcSize)
-{
- /* decoder timing evaluation */
- U32 const Q = (U32)(cSrcSize * 16 / dstSize); /* Q < 16 since dstSize > cSrcSize */
- U32 const D256 = (U32)(dstSize >> 8);
- U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256);
- U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256);
- DTime1 += DTime1 >> 3; /* advantage to algorithm using less memory, for cache eviction */
-
- return DTime1 < DTime0;
-}
-
-typedef size_t (*decompressionAlgo)(void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize);
-
-size_t HUF_decompress4X_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- /* validation checks */
- if (dstSize == 0)
- return ERROR(dstSize_tooSmall);
- if (cSrcSize > dstSize)
- return ERROR(corruption_detected); /* invalid */
- if (cSrcSize == dstSize) {
- memcpy(dst, cSrc, dstSize);
- return dstSize;
- } /* not compressed */
- if (cSrcSize == 1) {
- memset(dst, *(const BYTE *)cSrc, dstSize);
- return dstSize;
- } /* RLE */
-
- {
- U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
- return algoNb ? HUF_decompress4X4_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize)
- : HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize);
- }
-}
-
-size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- /* validation checks */
- if (dstSize == 0)
- return ERROR(dstSize_tooSmall);
- if ((cSrcSize >= dstSize) || (cSrcSize <= 1))
- return ERROR(corruption_detected); /* invalid */
-
- {
- U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
- return algoNb ? HUF_decompress4X4_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize)
- : HUF_decompress4X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize);
- }
-}
-
-size_t HUF_decompress1X_DCtx_wksp(HUF_DTable *dctx, void *dst, size_t dstSize, const void *cSrc, size_t cSrcSize, void *workspace, size_t workspaceSize)
-{
- /* validation checks */
- if (dstSize == 0)
- return ERROR(dstSize_tooSmall);
- if (cSrcSize > dstSize)
- return ERROR(corruption_detected); /* invalid */
- if (cSrcSize == dstSize) {
- memcpy(dst, cSrc, dstSize);
- return dstSize;
- } /* not compressed */
- if (cSrcSize == 1) {
- memset(dst, *(const BYTE *)cSrc, dstSize);
- return dstSize;
- } /* RLE */
-
- {
- U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
- return algoNb ? HUF_decompress1X4_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize)
- : HUF_decompress1X2_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workspace, workspaceSize);
- }
-}
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-#ifndef MEM_H_MODULE
-#define MEM_H_MODULE
-
-/*-****************************************
-* Dependencies
-******************************************/
-#include <asm/unaligned.h>
-#include <linux/string.h> /* memcpy */
-#include <linux/types.h> /* size_t, ptrdiff_t */
-
-/*-****************************************
-* Compiler specifics
-******************************************/
-#define ZSTD_STATIC static inline
-
-/*-**************************************************************
-* Basic Types
-*****************************************************************/
-typedef uint8_t BYTE;
-typedef uint16_t U16;
-typedef int16_t S16;
-typedef uint32_t U32;
-typedef int32_t S32;
-typedef uint64_t U64;
-typedef int64_t S64;
-typedef ptrdiff_t iPtrDiff;
-typedef uintptr_t uPtrDiff;
-
-/*-**************************************************************
-* Memory I/O
-*****************************************************************/
-ZSTD_STATIC unsigned ZSTD_32bits(void) { return sizeof(size_t) == 4; }
-ZSTD_STATIC unsigned ZSTD_64bits(void) { return sizeof(size_t) == 8; }
-
-#if defined(__LITTLE_ENDIAN)
-#define ZSTD_LITTLE_ENDIAN 1
-#else
-#define ZSTD_LITTLE_ENDIAN 0
-#endif
-
-ZSTD_STATIC unsigned ZSTD_isLittleEndian(void) { return ZSTD_LITTLE_ENDIAN; }
-
-ZSTD_STATIC U16 ZSTD_read16(const void *memPtr) { return get_unaligned((const U16 *)memPtr); }
-
-ZSTD_STATIC U32 ZSTD_read32(const void *memPtr) { return get_unaligned((const U32 *)memPtr); }
-
-ZSTD_STATIC U64 ZSTD_read64(const void *memPtr) { return get_unaligned((const U64 *)memPtr); }
-
-ZSTD_STATIC size_t ZSTD_readST(const void *memPtr) { return get_unaligned((const size_t *)memPtr); }
-
-ZSTD_STATIC void ZSTD_write16(void *memPtr, U16 value) { put_unaligned(value, (U16 *)memPtr); }
-
-ZSTD_STATIC void ZSTD_write32(void *memPtr, U32 value) { put_unaligned(value, (U32 *)memPtr); }
-
-ZSTD_STATIC void ZSTD_write64(void *memPtr, U64 value) { put_unaligned(value, (U64 *)memPtr); }
-
-/*=== Little endian r/w ===*/
-
-ZSTD_STATIC U16 ZSTD_readLE16(const void *memPtr) { return get_unaligned_le16(memPtr); }
-
-ZSTD_STATIC void ZSTD_writeLE16(void *memPtr, U16 val) { put_unaligned_le16(val, memPtr); }
-
-ZSTD_STATIC U32 ZSTD_readLE24(const void *memPtr) { return ZSTD_readLE16(memPtr) + (((const BYTE *)memPtr)[2] << 16); }
-
-ZSTD_STATIC void ZSTD_writeLE24(void *memPtr, U32 val)
-{
- ZSTD_writeLE16(memPtr, (U16)val);
- ((BYTE *)memPtr)[2] = (BYTE)(val >> 16);
-}
-
-ZSTD_STATIC U32 ZSTD_readLE32(const void *memPtr) { return get_unaligned_le32(memPtr); }
-
-ZSTD_STATIC void ZSTD_writeLE32(void *memPtr, U32 val32) { put_unaligned_le32(val32, memPtr); }
-
-ZSTD_STATIC U64 ZSTD_readLE64(const void *memPtr) { return get_unaligned_le64(memPtr); }
-
-ZSTD_STATIC void ZSTD_writeLE64(void *memPtr, U64 val64) { put_unaligned_le64(val64, memPtr); }
-
-ZSTD_STATIC size_t ZSTD_readLEST(const void *memPtr)
-{
- if (ZSTD_32bits())
- return (size_t)ZSTD_readLE32(memPtr);
- else
- return (size_t)ZSTD_readLE64(memPtr);
-}
-
-ZSTD_STATIC void ZSTD_writeLEST(void *memPtr, size_t val)
-{
- if (ZSTD_32bits())
- ZSTD_writeLE32(memPtr, (U32)val);
- else
- ZSTD_writeLE64(memPtr, (U64)val);
-}
-
-/*=== Big endian r/w ===*/
-
-ZSTD_STATIC U32 ZSTD_readBE32(const void *memPtr) { return get_unaligned_be32(memPtr); }
-
-ZSTD_STATIC void ZSTD_writeBE32(void *memPtr, U32 val32) { put_unaligned_be32(val32, memPtr); }
-
-ZSTD_STATIC U64 ZSTD_readBE64(const void *memPtr) { return get_unaligned_be64(memPtr); }
-
-ZSTD_STATIC void ZSTD_writeBE64(void *memPtr, U64 val64) { put_unaligned_be64(val64, memPtr); }
-
-ZSTD_STATIC size_t ZSTD_readBEST(const void *memPtr)
-{
- if (ZSTD_32bits())
- return (size_t)ZSTD_readBE32(memPtr);
- else
- return (size_t)ZSTD_readBE64(memPtr);
-}
-
-ZSTD_STATIC void ZSTD_writeBEST(void *memPtr, size_t val)
-{
- if (ZSTD_32bits())
- ZSTD_writeBE32(memPtr, (U32)val);
- else
- ZSTD_writeBE64(memPtr, (U64)val);
-}
-
-/* function safe only for comparisons */
-ZSTD_STATIC U32 ZSTD_readMINMATCH(const void *memPtr, U32 length)
-{
- switch (length) {
- default:
- case 4: return ZSTD_read32(memPtr);
- case 3:
- if (ZSTD_isLittleEndian())
- return ZSTD_read32(memPtr) << 8;
- else
- return ZSTD_read32(memPtr) >> 8;
- }
-}
-
-#endif /* MEM_H_MODULE */
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-/*-*************************************
-* Dependencies
-***************************************/
-#include "error_private.h"
-#include "zstd_internal.h" /* declaration of ZSTD_isError, ZSTD_getErrorName, ZSTD_getErrorCode, ZSTD_getErrorString, ZSTD_versionNumber */
-#include <linux/kernel.h>
-
-/*=**************************************************************
-* Custom allocator
-****************************************************************/
-
-#define stack_push(stack, size) \
- ({ \
- void *const ptr = ZSTD_PTR_ALIGN((stack)->ptr); \
- (stack)->ptr = (char *)ptr + (size); \
- (stack)->ptr <= (stack)->end ? ptr : NULL; \
- })
-
-ZSTD_customMem ZSTD_initStack(void *workspace, size_t workspaceSize)
-{
- ZSTD_customMem stackMem = {ZSTD_stackAlloc, ZSTD_stackFree, workspace};
- ZSTD_stack *stack = (ZSTD_stack *)workspace;
- /* Verify preconditions */
- if (!workspace || workspaceSize < sizeof(ZSTD_stack) || workspace != ZSTD_PTR_ALIGN(workspace)) {
- ZSTD_customMem error = {NULL, NULL, NULL};
- return error;
- }
- /* Initialize the stack */
- stack->ptr = workspace;
- stack->end = (char *)workspace + workspaceSize;
- stack_push(stack, sizeof(ZSTD_stack));
- return stackMem;
-}
-
-void *ZSTD_stackAllocAll(void *opaque, size_t *size)
-{
- ZSTD_stack *stack = (ZSTD_stack *)opaque;
- *size = (BYTE const *)stack->end - (BYTE *)ZSTD_PTR_ALIGN(stack->ptr);
- return stack_push(stack, *size);
-}
-
-void *ZSTD_stackAlloc(void *opaque, size_t size)
-{
- ZSTD_stack *stack = (ZSTD_stack *)opaque;
- return stack_push(stack, size);
-}
-void ZSTD_stackFree(void *opaque, void *address)
-{
- (void)opaque;
- (void)address;
-}
-
-void *ZSTD_malloc(size_t size, ZSTD_customMem customMem) { return customMem.customAlloc(customMem.opaque, size); }
-
-void ZSTD_free(void *ptr, ZSTD_customMem customMem)
-{
- if (ptr != NULL)
- customMem.customFree(customMem.opaque, ptr);
-}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/zstd.h>
+
+#include "common/zstd_deps.h"
+#include "common/zstd_internal.h"
+
+#define ZSTD_FORWARD_IF_ERR(ret) \
+ do { \
+ size_t const __ret = (ret); \
+ if (ZSTD_isError(__ret)) \
+ return __ret; \
+ } while (0)
+
+static size_t zstd_cctx_init(zstd_cctx *cctx, const zstd_parameters *parameters,
+ unsigned long long pledged_src_size)
+{
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_reset(
+ cctx, ZSTD_reset_session_and_parameters));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setPledgedSrcSize(
+ cctx, pledged_src_size));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_windowLog, parameters->cParams.windowLog));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_hashLog, parameters->cParams.hashLog));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_chainLog, parameters->cParams.chainLog));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_searchLog, parameters->cParams.searchLog));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_minMatch, parameters->cParams.minMatch));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_targetLength, parameters->cParams.targetLength));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_strategy, parameters->cParams.strategy));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_contentSizeFlag, parameters->fParams.contentSizeFlag));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_checksumFlag, parameters->fParams.checksumFlag));
+ ZSTD_FORWARD_IF_ERR(ZSTD_CCtx_setParameter(
+ cctx, ZSTD_c_dictIDFlag, !parameters->fParams.noDictIDFlag));
+ return 0;
+}
+
+int zstd_min_clevel(void)
+{
+ return ZSTD_minCLevel();
+}
+EXPORT_SYMBOL(zstd_min_clevel);
+
+int zstd_max_clevel(void)
+{
+ return ZSTD_maxCLevel();
+}
+EXPORT_SYMBOL(zstd_max_clevel);
+
+size_t zstd_compress_bound(size_t src_size)
+{
+ return ZSTD_compressBound(src_size);
+}
+EXPORT_SYMBOL(zstd_compress_bound);
+
+zstd_parameters zstd_get_params(int level,
+ unsigned long long estimated_src_size)
+{
+ return ZSTD_getParams(level, estimated_src_size, 0);
+}
+EXPORT_SYMBOL(zstd_get_params);
+
+size_t zstd_cctx_workspace_bound(const zstd_compression_parameters *cparams)
+{
+ return ZSTD_estimateCCtxSize_usingCParams(*cparams);
+}
+EXPORT_SYMBOL(zstd_cctx_workspace_bound);
+
+zstd_cctx *zstd_init_cctx(void *workspace, size_t workspace_size)
+{
+ if (workspace == NULL)
+ return NULL;
+ return ZSTD_initStaticCCtx(workspace, workspace_size);
+}
+EXPORT_SYMBOL(zstd_init_cctx);
+
+size_t zstd_compress_cctx(zstd_cctx *cctx, void *dst, size_t dst_capacity,
+ const void *src, size_t src_size, const zstd_parameters *parameters)
+{
+ ZSTD_FORWARD_IF_ERR(zstd_cctx_init(cctx, parameters, src_size));
+ return ZSTD_compress2(cctx, dst, dst_capacity, src, src_size);
+}
+EXPORT_SYMBOL(zstd_compress_cctx);
+
+size_t zstd_cstream_workspace_bound(const zstd_compression_parameters *cparams)
+{
+ return ZSTD_estimateCStreamSize_usingCParams(*cparams);
+}
+EXPORT_SYMBOL(zstd_cstream_workspace_bound);
+
+zstd_cstream *zstd_init_cstream(const zstd_parameters *parameters,
+ unsigned long long pledged_src_size, void *workspace, size_t workspace_size)
+{
+ zstd_cstream *cstream;
+
+ if (workspace == NULL)
+ return NULL;
+
+ cstream = ZSTD_initStaticCStream(workspace, workspace_size);
+ if (cstream == NULL)
+ return NULL;
+
+ /* 0 means unknown in linux zstd API but means 0 in new zstd API */
+ if (pledged_src_size == 0)
+ pledged_src_size = ZSTD_CONTENTSIZE_UNKNOWN;
+
+ if (ZSTD_isError(zstd_cctx_init(cstream, parameters, pledged_src_size)))
+ return NULL;
+
+ return cstream;
+}
+EXPORT_SYMBOL(zstd_init_cstream);
+
+size_t zstd_reset_cstream(zstd_cstream *cstream,
+ unsigned long long pledged_src_size)
+{
+ return ZSTD_resetCStream(cstream, pledged_src_size);
+}
+EXPORT_SYMBOL(zstd_reset_cstream);
+
+size_t zstd_compress_stream(zstd_cstream *cstream, zstd_out_buffer *output,
+ zstd_in_buffer *input)
+{
+ return ZSTD_compressStream(cstream, output, input);
+}
+EXPORT_SYMBOL(zstd_compress_stream);
+
+size_t zstd_flush_stream(zstd_cstream *cstream, zstd_out_buffer *output)
+{
+ return ZSTD_flushStream(cstream, output);
+}
+EXPORT_SYMBOL(zstd_flush_stream);
+
+size_t zstd_end_stream(zstd_cstream *cstream, zstd_out_buffer *output)
+{
+ return ZSTD_endStream(cstream, output);
+}
+EXPORT_SYMBOL(zstd_end_stream);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Zstd Compressor");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/zstd.h>
+
+#include "common/zstd_deps.h"
+
+/* Common symbols. zstd_compress must depend on zstd_decompress. */
+
+unsigned int zstd_is_error(size_t code)
+{
+ return ZSTD_isError(code);
+}
+EXPORT_SYMBOL(zstd_is_error);
+
+zstd_error_code zstd_get_error_code(size_t code)
+{
+ return ZSTD_getErrorCode(code);
+}
+EXPORT_SYMBOL(zstd_get_error_code);
+
+const char *zstd_get_error_name(size_t code)
+{
+ return ZSTD_getErrorName(code);
+}
+EXPORT_SYMBOL(zstd_get_error_name);
+
+/* Decompression symbols. */
+
+size_t zstd_dctx_workspace_bound(void)
+{
+ return ZSTD_estimateDCtxSize();
+}
+EXPORT_SYMBOL(zstd_dctx_workspace_bound);
+
+zstd_dctx *zstd_init_dctx(void *workspace, size_t workspace_size)
+{
+ if (workspace == NULL)
+ return NULL;
+ return ZSTD_initStaticDCtx(workspace, workspace_size);
+}
+EXPORT_SYMBOL(zstd_init_dctx);
+
+size_t zstd_decompress_dctx(zstd_dctx *dctx, void *dst, size_t dst_capacity,
+ const void *src, size_t src_size)
+{
+ return ZSTD_decompressDCtx(dctx, dst, dst_capacity, src, src_size);
+}
+EXPORT_SYMBOL(zstd_decompress_dctx);
+
+size_t zstd_dstream_workspace_bound(size_t max_window_size)
+{
+ return ZSTD_estimateDStreamSize(max_window_size);
+}
+EXPORT_SYMBOL(zstd_dstream_workspace_bound);
+
+zstd_dstream *zstd_init_dstream(size_t max_window_size, void *workspace,
+ size_t workspace_size)
+{
+ if (workspace == NULL)
+ return NULL;
+ (void)max_window_size;
+ return ZSTD_initStaticDStream(workspace, workspace_size);
+}
+EXPORT_SYMBOL(zstd_init_dstream);
+
+size_t zstd_reset_dstream(zstd_dstream *dstream)
+{
+ return ZSTD_resetDStream(dstream);
+}
+EXPORT_SYMBOL(zstd_reset_dstream);
+
+size_t zstd_decompress_stream(zstd_dstream *dstream, zstd_out_buffer *output,
+ zstd_in_buffer *input)
+{
+ return ZSTD_decompressStream(dstream, output, input);
+}
+EXPORT_SYMBOL(zstd_decompress_stream);
+
+size_t zstd_find_frame_compressed_size(const void *src, size_t src_size)
+{
+ return ZSTD_findFrameCompressedSize(src, src_size);
+}
+EXPORT_SYMBOL(zstd_find_frame_compressed_size);
+
+size_t zstd_get_frame_header(zstd_frame_header *header, const void *src,
+ size_t src_size)
+{
+ return ZSTD_getFrameHeader(header, src, src_size);
+}
+EXPORT_SYMBOL(zstd_get_frame_header);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Zstd Decompressor");
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-#ifndef ZSTD_CCOMMON_H_MODULE
-#define ZSTD_CCOMMON_H_MODULE
-
-/*-*******************************************************
-* Compiler specifics
-*********************************************************/
-#define FORCE_INLINE static __always_inline
-#define FORCE_NOINLINE static noinline
-
-/*-*************************************
-* Dependencies
-***************************************/
-#include "error_private.h"
-#include "mem.h"
-#include <linux/compiler.h>
-#include <linux/kernel.h>
-#include <linux/xxhash.h>
-#include <linux/zstd.h>
-
-/*-*************************************
-* shared macros
-***************************************/
-#define MIN(a, b) ((a) < (b) ? (a) : (b))
-#define MAX(a, b) ((a) > (b) ? (a) : (b))
-#define CHECK_F(f) \
- { \
- size_t const errcod = f; \
- if (ERR_isError(errcod)) \
- return errcod; \
- } /* check and Forward error code */
-#define CHECK_E(f, e) \
- { \
- size_t const errcod = f; \
- if (ERR_isError(errcod)) \
- return ERROR(e); \
- } /* check and send Error code */
-#define ZSTD_STATIC_ASSERT(c) \
- { \
- enum { ZSTD_static_assert = 1 / (int)(!!(c)) }; \
- }
-
-/*-*************************************
-* Common constants
-***************************************/
-#define ZSTD_OPT_NUM (1 << 12)
-#define ZSTD_DICT_MAGIC 0xEC30A437 /* v0.7+ */
-
-#define ZSTD_REP_NUM 3 /* number of repcodes */
-#define ZSTD_REP_CHECK (ZSTD_REP_NUM) /* number of repcodes to check by the optimal parser */
-#define ZSTD_REP_MOVE (ZSTD_REP_NUM - 1)
-#define ZSTD_REP_MOVE_OPT (ZSTD_REP_NUM)
-static const U32 repStartValue[ZSTD_REP_NUM] = {1, 4, 8};
-
-#define KB *(1 << 10)
-#define MB *(1 << 20)
-#define GB *(1U << 30)
-
-#define BIT7 128
-#define BIT6 64
-#define BIT5 32
-#define BIT4 16
-#define BIT1 2
-#define BIT0 1
-
-#define ZSTD_WINDOWLOG_ABSOLUTEMIN 10
-static const size_t ZSTD_fcs_fieldSize[4] = {0, 2, 4, 8};
-static const size_t ZSTD_did_fieldSize[4] = {0, 1, 2, 4};
-
-#define ZSTD_BLOCKHEADERSIZE 3 /* C standard doesn't allow `static const` variable to be init using another `static const` variable */
-static const size_t ZSTD_blockHeaderSize = ZSTD_BLOCKHEADERSIZE;
-typedef enum { bt_raw, bt_rle, bt_compressed, bt_reserved } blockType_e;
-
-#define MIN_SEQUENCES_SIZE 1 /* nbSeq==0 */
-#define MIN_CBLOCK_SIZE (1 /*litCSize*/ + 1 /* RLE or RAW */ + MIN_SEQUENCES_SIZE /* nbSeq==0 */) /* for a non-null block */
-
-#define HufLog 12
-typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingType_e;
-
-#define LONGNBSEQ 0x7F00
-
-#define MINMATCH 3
-#define EQUAL_READ32 4
-
-#define Litbits 8
-#define MaxLit ((1 << Litbits) - 1)
-#define MaxML 52
-#define MaxLL 35
-#define MaxOff 28
-#define MaxSeq MAX(MaxLL, MaxML) /* Assumption : MaxOff < MaxLL,MaxML */
-#define MLFSELog 9
-#define LLFSELog 9
-#define OffFSELog 8
-
-static const U32 LL_bits[MaxLL + 1] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16};
-static const S16 LL_defaultNorm[MaxLL + 1] = {4, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 2, 1, 1, 1, 1, 1, -1, -1, -1, -1};
-#define LL_DEFAULTNORMLOG 6 /* for static allocation */
-static const U32 LL_defaultNormLog = LL_DEFAULTNORMLOG;
-
-static const U32 ML_bits[MaxML + 1] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
- 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16};
-static const S16 ML_defaultNorm[MaxML + 1] = {1, 4, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
- 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1, -1, -1};
-#define ML_DEFAULTNORMLOG 6 /* for static allocation */
-static const U32 ML_defaultNormLog = ML_DEFAULTNORMLOG;
-
-static const S16 OF_defaultNorm[MaxOff + 1] = {1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, -1, -1, -1, -1, -1};
-#define OF_DEFAULTNORMLOG 5 /* for static allocation */
-static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
-
-/*-*******************************************
-* Shared functions to include for inlining
-*********************************************/
-ZSTD_STATIC void ZSTD_copy8(void *dst, const void *src) {
- /*
- * zstd relies heavily on gcc being able to analyze and inline this
- * memcpy() call, since it is called in a tight loop. Preboot mode
- * is compiled in freestanding mode, which stops gcc from analyzing
- * memcpy(). Use __builtin_memcpy() to tell gcc to analyze this as a
- * regular memcpy().
- */
- __builtin_memcpy(dst, src, 8);
-}
-/*! ZSTD_wildcopy() :
-* custom version of memcpy(), can copy up to 7 bytes too many (8 bytes if length==0) */
-#define WILDCOPY_OVERLENGTH 8
-ZSTD_STATIC void ZSTD_wildcopy(void *dst, const void *src, ptrdiff_t length)
-{
- const BYTE* ip = (const BYTE*)src;
- BYTE* op = (BYTE*)dst;
- BYTE* const oend = op + length;
-#if defined(GCC_VERSION) && GCC_VERSION >= 70000 && GCC_VERSION < 70200
- /*
- * Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388.
- * Avoid the bad case where the loop only runs once by handling the
- * special case separately. This doesn't trigger the bug because it
- * doesn't involve pointer/integer overflow.
- */
- if (length <= 8)
- return ZSTD_copy8(dst, src);
-#endif
- do {
- ZSTD_copy8(op, ip);
- op += 8;
- ip += 8;
- } while (op < oend);
-}
-
-/*-*******************************************
-* Private interfaces
-*********************************************/
-typedef struct ZSTD_stats_s ZSTD_stats_t;
-
-typedef struct {
- U32 off;
- U32 len;
-} ZSTD_match_t;
-
-typedef struct {
- U32 price;
- U32 off;
- U32 mlen;
- U32 litlen;
- U32 rep[ZSTD_REP_NUM];
-} ZSTD_optimal_t;
-
-typedef struct seqDef_s {
- U32 offset;
- U16 litLength;
- U16 matchLength;
-} seqDef;
-
-typedef struct {
- seqDef *sequencesStart;
- seqDef *sequences;
- BYTE *litStart;
- BYTE *lit;
- BYTE *llCode;
- BYTE *mlCode;
- BYTE *ofCode;
- U32 longLengthID; /* 0 == no longLength; 1 == Lit.longLength; 2 == Match.longLength; */
- U32 longLengthPos;
- /* opt */
- ZSTD_optimal_t *priceTable;
- ZSTD_match_t *matchTable;
- U32 *matchLengthFreq;
- U32 *litLengthFreq;
- U32 *litFreq;
- U32 *offCodeFreq;
- U32 matchLengthSum;
- U32 matchSum;
- U32 litLengthSum;
- U32 litSum;
- U32 offCodeSum;
- U32 log2matchLengthSum;
- U32 log2matchSum;
- U32 log2litLengthSum;
- U32 log2litSum;
- U32 log2offCodeSum;
- U32 factor;
- U32 staticPrices;
- U32 cachedPrice;
- U32 cachedLitLength;
- const BYTE *cachedLiterals;
-} seqStore_t;
-
-const seqStore_t *ZSTD_getSeqStore(const ZSTD_CCtx *ctx);
-void ZSTD_seqToCodes(const seqStore_t *seqStorePtr);
-int ZSTD_isSkipFrame(ZSTD_DCtx *dctx);
-
-/*= Custom memory allocation functions */
-typedef void *(*ZSTD_allocFunction)(void *opaque, size_t size);
-typedef void (*ZSTD_freeFunction)(void *opaque, void *address);
-typedef struct {
- ZSTD_allocFunction customAlloc;
- ZSTD_freeFunction customFree;
- void *opaque;
-} ZSTD_customMem;
-
-void *ZSTD_malloc(size_t size, ZSTD_customMem customMem);
-void ZSTD_free(void *ptr, ZSTD_customMem customMem);
-
-/*====== stack allocation ======*/
-
-typedef struct {
- void *ptr;
- const void *end;
-} ZSTD_stack;
-
-#define ZSTD_ALIGN(x) ALIGN(x, sizeof(size_t))
-#define ZSTD_PTR_ALIGN(p) PTR_ALIGN(p, sizeof(size_t))
-
-ZSTD_customMem ZSTD_initStack(void *workspace, size_t workspaceSize);
-
-void *ZSTD_stackAllocAll(void *opaque, size_t *size);
-void *ZSTD_stackAlloc(void *opaque, size_t size);
-void ZSTD_stackFree(void *opaque, void *address);
-
-/*====== common function ======*/
-
-ZSTD_STATIC U32 ZSTD_highbit32(U32 val) { return 31 - __builtin_clz(val); }
-
-/* hidden functions */
-
-/* ZSTD_invalidateRepCodes() :
- * ensures next compression will not use repcodes from previous block.
- * Note : only works with regular variant;
- * do not use with extDict variant ! */
-void ZSTD_invalidateRepCodes(ZSTD_CCtx *cctx);
-
-size_t ZSTD_freeCCtx(ZSTD_CCtx *cctx);
-size_t ZSTD_freeDCtx(ZSTD_DCtx *dctx);
-size_t ZSTD_freeCDict(ZSTD_CDict *cdict);
-size_t ZSTD_freeDDict(ZSTD_DDict *cdict);
-size_t ZSTD_freeCStream(ZSTD_CStream *zcs);
-size_t ZSTD_freeDStream(ZSTD_DStream *zds);
-
-#endif /* ZSTD_CCOMMON_H_MODULE */
+++ /dev/null
-/**
- * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
- * All rights reserved.
- *
- * This source code is licensed under the BSD-style license found in the
- * LICENSE file in the root directory of https://github.com/facebook/zstd.
- * An additional grant of patent rights can be found in the PATENTS file in the
- * same directory.
- *
- * This program is free software; you can redistribute it and/or modify it under
- * the terms of the GNU General Public License version 2 as published by the
- * Free Software Foundation. This program is dual-licensed; you may select
- * either version 2 of the GNU General Public License ("GPL") or BSD license
- * ("BSD").
- */
-
-/* Note : this file is intended to be included within zstd_compress.c */
-
-#ifndef ZSTD_OPT_H_91842398743
-#define ZSTD_OPT_H_91842398743
-
-#define ZSTD_LITFREQ_ADD 2
-#define ZSTD_FREQ_DIV 4
-#define ZSTD_MAX_PRICE (1 << 30)
-
-/*-*************************************
-* Price functions for optimal parser
-***************************************/
-FORCE_INLINE void ZSTD_setLog2Prices(seqStore_t *ssPtr)
-{
- ssPtr->log2matchLengthSum = ZSTD_highbit32(ssPtr->matchLengthSum + 1);
- ssPtr->log2litLengthSum = ZSTD_highbit32(ssPtr->litLengthSum + 1);
- ssPtr->log2litSum = ZSTD_highbit32(ssPtr->litSum + 1);
- ssPtr->log2offCodeSum = ZSTD_highbit32(ssPtr->offCodeSum + 1);
- ssPtr->factor = 1 + ((ssPtr->litSum >> 5) / ssPtr->litLengthSum) + ((ssPtr->litSum << 1) / (ssPtr->litSum + ssPtr->matchSum));
-}
-
-ZSTD_STATIC void ZSTD_rescaleFreqs(seqStore_t *ssPtr, const BYTE *src, size_t srcSize)
-{
- unsigned u;
-
- ssPtr->cachedLiterals = NULL;
- ssPtr->cachedPrice = ssPtr->cachedLitLength = 0;
- ssPtr->staticPrices = 0;
-
- if (ssPtr->litLengthSum == 0) {
- if (srcSize <= 1024)
- ssPtr->staticPrices = 1;
-
- for (u = 0; u <= MaxLit; u++)
- ssPtr->litFreq[u] = 0;
- for (u = 0; u < srcSize; u++)
- ssPtr->litFreq[src[u]]++;
-
- ssPtr->litSum = 0;
- ssPtr->litLengthSum = MaxLL + 1;
- ssPtr->matchLengthSum = MaxML + 1;
- ssPtr->offCodeSum = (MaxOff + 1);
- ssPtr->matchSum = (ZSTD_LITFREQ_ADD << Litbits);
-
- for (u = 0; u <= MaxLit; u++) {
- ssPtr->litFreq[u] = 1 + (ssPtr->litFreq[u] >> ZSTD_FREQ_DIV);
- ssPtr->litSum += ssPtr->litFreq[u];
- }
- for (u = 0; u <= MaxLL; u++)
- ssPtr->litLengthFreq[u] = 1;
- for (u = 0; u <= MaxML; u++)
- ssPtr->matchLengthFreq[u] = 1;
- for (u = 0; u <= MaxOff; u++)
- ssPtr->offCodeFreq[u] = 1;
- } else {
- ssPtr->matchLengthSum = 0;
- ssPtr->litLengthSum = 0;
- ssPtr->offCodeSum = 0;
- ssPtr->matchSum = 0;
- ssPtr->litSum = 0;
-
- for (u = 0; u <= MaxLit; u++) {
- ssPtr->litFreq[u] = 1 + (ssPtr->litFreq[u] >> (ZSTD_FREQ_DIV + 1));
- ssPtr->litSum += ssPtr->litFreq[u];
- }
- for (u = 0; u <= MaxLL; u++) {
- ssPtr->litLengthFreq[u] = 1 + (ssPtr->litLengthFreq[u] >> (ZSTD_FREQ_DIV + 1));
- ssPtr->litLengthSum += ssPtr->litLengthFreq[u];
- }
- for (u = 0; u <= MaxML; u++) {
- ssPtr->matchLengthFreq[u] = 1 + (ssPtr->matchLengthFreq[u] >> ZSTD_FREQ_DIV);
- ssPtr->matchLengthSum += ssPtr->matchLengthFreq[u];
- ssPtr->matchSum += ssPtr->matchLengthFreq[u] * (u + 3);
- }
- ssPtr->matchSum *= ZSTD_LITFREQ_ADD;
- for (u = 0; u <= MaxOff; u++) {
- ssPtr->offCodeFreq[u] = 1 + (ssPtr->offCodeFreq[u] >> ZSTD_FREQ_DIV);
- ssPtr->offCodeSum += ssPtr->offCodeFreq[u];
- }
- }
-
- ZSTD_setLog2Prices(ssPtr);
-}
-
-FORCE_INLINE U32 ZSTD_getLiteralPrice(seqStore_t *ssPtr, U32 litLength, const BYTE *literals)
-{
- U32 price, u;
-
- if (ssPtr->staticPrices)
- return ZSTD_highbit32((U32)litLength + 1) + (litLength * 6);
-
- if (litLength == 0)
- return ssPtr->log2litLengthSum - ZSTD_highbit32(ssPtr->litLengthFreq[0] + 1);
-
- /* literals */
- if (ssPtr->cachedLiterals == literals) {
- U32 const additional = litLength - ssPtr->cachedLitLength;
- const BYTE *literals2 = ssPtr->cachedLiterals + ssPtr->cachedLitLength;
- price = ssPtr->cachedPrice + additional * ssPtr->log2litSum;
- for (u = 0; u < additional; u++)
- price -= ZSTD_highbit32(ssPtr->litFreq[literals2[u]] + 1);
- ssPtr->cachedPrice = price;
- ssPtr->cachedLitLength = litLength;
- } else {
- price = litLength * ssPtr->log2litSum;
- for (u = 0; u < litLength; u++)
- price -= ZSTD_highbit32(ssPtr->litFreq[literals[u]] + 1);
-
- if (litLength >= 12) {
- ssPtr->cachedLiterals = literals;
- ssPtr->cachedPrice = price;
- ssPtr->cachedLitLength = litLength;
- }
- }
-
- /* literal Length */
- {
- const BYTE LL_deltaCode = 19;
- const BYTE llCode = (litLength > 63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
- price += LL_bits[llCode] + ssPtr->log2litLengthSum - ZSTD_highbit32(ssPtr->litLengthFreq[llCode] + 1);
- }
-
- return price;
-}
-
-FORCE_INLINE U32 ZSTD_getPrice(seqStore_t *seqStorePtr, U32 litLength, const BYTE *literals, U32 offset, U32 matchLength, const int ultra)
-{
- /* offset */
- U32 price;
- BYTE const offCode = (BYTE)ZSTD_highbit32(offset + 1);
-
- if (seqStorePtr->staticPrices)
- return ZSTD_getLiteralPrice(seqStorePtr, litLength, literals) + ZSTD_highbit32((U32)matchLength + 1) + 16 + offCode;
-
- price = offCode + seqStorePtr->log2offCodeSum - ZSTD_highbit32(seqStorePtr->offCodeFreq[offCode] + 1);
- if (!ultra && offCode >= 20)
- price += (offCode - 19) * 2;
-
- /* match Length */
- {
- const BYTE ML_deltaCode = 36;
- const BYTE mlCode = (matchLength > 127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
- price += ML_bits[mlCode] + seqStorePtr->log2matchLengthSum - ZSTD_highbit32(seqStorePtr->matchLengthFreq[mlCode] + 1);
- }
-
- return price + ZSTD_getLiteralPrice(seqStorePtr, litLength, literals) + seqStorePtr->factor;
-}
-
-ZSTD_STATIC void ZSTD_updatePrice(seqStore_t *seqStorePtr, U32 litLength, const BYTE *literals, U32 offset, U32 matchLength)
-{
- U32 u;
-
- /* literals */
- seqStorePtr->litSum += litLength * ZSTD_LITFREQ_ADD;
- for (u = 0; u < litLength; u++)
- seqStorePtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD;
-
- /* literal Length */
- {
- const BYTE LL_deltaCode = 19;
- const BYTE llCode = (litLength > 63) ? (BYTE)ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength];
- seqStorePtr->litLengthFreq[llCode]++;
- seqStorePtr->litLengthSum++;
- }
-
- /* match offset */
- {
- BYTE const offCode = (BYTE)ZSTD_highbit32(offset + 1);
- seqStorePtr->offCodeSum++;
- seqStorePtr->offCodeFreq[offCode]++;
- }
-
- /* match Length */
- {
- const BYTE ML_deltaCode = 36;
- const BYTE mlCode = (matchLength > 127) ? (BYTE)ZSTD_highbit32(matchLength) + ML_deltaCode : ML_Code[matchLength];
- seqStorePtr->matchLengthFreq[mlCode]++;
- seqStorePtr->matchLengthSum++;
- }
-
- ZSTD_setLog2Prices(seqStorePtr);
-}
-
-#define SET_PRICE(pos, mlen_, offset_, litlen_, price_) \
- { \
- while (last_pos < pos) { \
- opt[last_pos + 1].price = ZSTD_MAX_PRICE; \
- last_pos++; \
- } \
- opt[pos].mlen = mlen_; \
- opt[pos].off = offset_; \
- opt[pos].litlen = litlen_; \
- opt[pos].price = price_; \
- }
-
-/* Update hashTable3 up to ip (excluded)
- Assumption : always within prefix (i.e. not within extDict) */
-FORCE_INLINE
-U32 ZSTD_insertAndFindFirstIndexHash3(ZSTD_CCtx *zc, const BYTE *ip)
-{
- U32 *const hashTable3 = zc->hashTable3;
- U32 const hashLog3 = zc->hashLog3;
- const BYTE *const base = zc->base;
- U32 idx = zc->nextToUpdate3;
- const U32 target = zc->nextToUpdate3 = (U32)(ip - base);
- const size_t hash3 = ZSTD_hash3Ptr(ip, hashLog3);
-
- while (idx < target) {
- hashTable3[ZSTD_hash3Ptr(base + idx, hashLog3)] = idx;
- idx++;
- }
-
- return hashTable3[hash3];
-}
-
-/*-*************************************
-* Binary Tree search
-***************************************/
-static U32 ZSTD_insertBtAndGetAllMatches(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iLimit, U32 nbCompares, const U32 mls, U32 extDict,
- ZSTD_match_t *matches, const U32 minMatchLen)
-{
- const BYTE *const base = zc->base;
- const U32 curr = (U32)(ip - base);
- const U32 hashLog = zc->params.cParams.hashLog;
- const size_t h = ZSTD_hashPtr(ip, hashLog, mls);
- U32 *const hashTable = zc->hashTable;
- U32 matchIndex = hashTable[h];
- U32 *const bt = zc->chainTable;
- const U32 btLog = zc->params.cParams.chainLog - 1;
- const U32 btMask = (1U << btLog) - 1;
- size_t commonLengthSmaller = 0, commonLengthLarger = 0;
- const BYTE *const dictBase = zc->dictBase;
- const U32 dictLimit = zc->dictLimit;
- const BYTE *const dictEnd = dictBase + dictLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const U32 btLow = btMask >= curr ? 0 : curr - btMask;
- const U32 windowLow = zc->lowLimit;
- U32 *smallerPtr = bt + 2 * (curr & btMask);
- U32 *largerPtr = bt + 2 * (curr & btMask) + 1;
- U32 matchEndIdx = curr + 8;
- U32 dummy32; /* to be nullified at the end */
- U32 mnum = 0;
-
- const U32 minMatch = (mls == 3) ? 3 : 4;
- size_t bestLength = minMatchLen - 1;
-
- if (minMatch == 3) { /* HC3 match finder */
- U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(zc, ip);
- if (matchIndex3 > windowLow && (curr - matchIndex3 < (1 << 18))) {
- const BYTE *match;
- size_t currMl = 0;
- if ((!extDict) || matchIndex3 >= dictLimit) {
- match = base + matchIndex3;
- if (match[bestLength] == ip[bestLength])
- currMl = ZSTD_count(ip, match, iLimit);
- } else {
- match = dictBase + matchIndex3;
- if (ZSTD_readMINMATCH(match, MINMATCH) ==
- ZSTD_readMINMATCH(ip, MINMATCH)) /* assumption : matchIndex3 <= dictLimit-4 (by table construction) */
- currMl = ZSTD_count_2segments(ip + MINMATCH, match + MINMATCH, iLimit, dictEnd, prefixStart) + MINMATCH;
- }
-
- /* save best solution */
- if (currMl > bestLength) {
- bestLength = currMl;
- matches[mnum].off = ZSTD_REP_MOVE_OPT + curr - matchIndex3;
- matches[mnum].len = (U32)currMl;
- mnum++;
- if (currMl > ZSTD_OPT_NUM)
- goto update;
- if (ip + currMl == iLimit)
- goto update; /* best possible, and avoid read overflow*/
- }
- }
- }
-
- hashTable[h] = curr; /* Update Hash Table */
-
- while (nbCompares-- && (matchIndex > windowLow)) {
- U32 *nextPtr = bt + 2 * (matchIndex & btMask);
- size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */
- const BYTE *match;
-
- if ((!extDict) || (matchIndex + matchLength >= dictLimit)) {
- match = base + matchIndex;
- if (match[matchLength] == ip[matchLength]) {
- matchLength += ZSTD_count(ip + matchLength + 1, match + matchLength + 1, iLimit) + 1;
- }
- } else {
- match = dictBase + matchIndex;
- matchLength += ZSTD_count_2segments(ip + matchLength, match + matchLength, iLimit, dictEnd, prefixStart);
- if (matchIndex + matchLength >= dictLimit)
- match = base + matchIndex; /* to prepare for next usage of match[matchLength] */
- }
-
- if (matchLength > bestLength) {
- if (matchLength > matchEndIdx - matchIndex)
- matchEndIdx = matchIndex + (U32)matchLength;
- bestLength = matchLength;
- matches[mnum].off = ZSTD_REP_MOVE_OPT + curr - matchIndex;
- matches[mnum].len = (U32)matchLength;
- mnum++;
- if (matchLength > ZSTD_OPT_NUM)
- break;
- if (ip + matchLength == iLimit) /* equal : no way to know if inf or sup */
- break; /* drop, to guarantee consistency (miss a little bit of compression) */
- }
-
- if (match[matchLength] < ip[matchLength]) {
- /* match is smaller than curr */
- *smallerPtr = matchIndex; /* update smaller idx */
- commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */
- if (matchIndex <= btLow) {
- smallerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- smallerPtr = nextPtr + 1; /* new "smaller" => larger of match */
- matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to curr) */
- } else {
- /* match is larger than curr */
- *largerPtr = matchIndex;
- commonLengthLarger = matchLength;
- if (matchIndex <= btLow) {
- largerPtr = &dummy32;
- break;
- } /* beyond tree size, stop the search */
- largerPtr = nextPtr;
- matchIndex = nextPtr[0];
- }
- }
-
- *smallerPtr = *largerPtr = 0;
-
-update:
- zc->nextToUpdate = (matchEndIdx > curr + 8) ? matchEndIdx - 8 : curr + 1;
- return mnum;
-}
-
-/** Tree updater, providing best match */
-static U32 ZSTD_BtGetAllMatches(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iLimit, const U32 maxNbAttempts, const U32 mls, ZSTD_match_t *matches,
- const U32 minMatchLen)
-{
- if (ip < zc->base + zc->nextToUpdate)
- return 0; /* skipped area */
- ZSTD_updateTree(zc, ip, iLimit, maxNbAttempts, mls);
- return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 0, matches, minMatchLen);
-}
-
-static U32 ZSTD_BtGetAllMatches_selectMLS(ZSTD_CCtx *zc, /* Index table will be updated */
- const BYTE *ip, const BYTE *const iHighLimit, const U32 maxNbAttempts, const U32 matchLengthSearch,
- ZSTD_match_t *matches, const U32 minMatchLen)
-{
- switch (matchLengthSearch) {
- case 3: return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
- default:
- case 4: return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
- case 5: return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
- case 7:
- case 6: return ZSTD_BtGetAllMatches(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
- }
-}
-
-/** Tree updater, providing best match */
-static U32 ZSTD_BtGetAllMatches_extDict(ZSTD_CCtx *zc, const BYTE *const ip, const BYTE *const iLimit, const U32 maxNbAttempts, const U32 mls,
- ZSTD_match_t *matches, const U32 minMatchLen)
-{
- if (ip < zc->base + zc->nextToUpdate)
- return 0; /* skipped area */
- ZSTD_updateTree_extDict(zc, ip, iLimit, maxNbAttempts, mls);
- return ZSTD_insertBtAndGetAllMatches(zc, ip, iLimit, maxNbAttempts, mls, 1, matches, minMatchLen);
-}
-
-static U32 ZSTD_BtGetAllMatches_selectMLS_extDict(ZSTD_CCtx *zc, /* Index table will be updated */
- const BYTE *ip, const BYTE *const iHighLimit, const U32 maxNbAttempts, const U32 matchLengthSearch,
- ZSTD_match_t *matches, const U32 minMatchLen)
-{
- switch (matchLengthSearch) {
- case 3: return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 3, matches, minMatchLen);
- default:
- case 4: return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 4, matches, minMatchLen);
- case 5: return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 5, matches, minMatchLen);
- case 7:
- case 6: return ZSTD_BtGetAllMatches_extDict(zc, ip, iHighLimit, maxNbAttempts, 6, matches, minMatchLen);
- }
-}
-
-/*-*******************************
-* Optimal parser
-*********************************/
-FORCE_INLINE
-void ZSTD_compressBlock_opt_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const int ultra)
-{
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- const BYTE *const base = ctx->base;
- const BYTE *const prefixStart = base + ctx->dictLimit;
-
- const U32 maxSearches = 1U << ctx->params.cParams.searchLog;
- const U32 sufficient_len = ctx->params.cParams.targetLength;
- const U32 mls = ctx->params.cParams.searchLength;
- const U32 minMatch = (ctx->params.cParams.searchLength == 3) ? 3 : 4;
-
- ZSTD_optimal_t *opt = seqStorePtr->priceTable;
- ZSTD_match_t *matches = seqStorePtr->matchTable;
- const BYTE *inr;
- U32 offset, rep[ZSTD_REP_NUM];
-
- /* init */
- ctx->nextToUpdate3 = ctx->nextToUpdate;
- ZSTD_rescaleFreqs(seqStorePtr, (const BYTE *)src, srcSize);
- ip += (ip == prefixStart);
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- rep[i] = ctx->rep[i];
- }
-
- /* Match Loop */
- while (ip < ilimit) {
- U32 cur, match_num, last_pos, litlen, price;
- U32 u, mlen, best_mlen, best_off, litLength;
- memset(opt, 0, sizeof(ZSTD_optimal_t));
- last_pos = 0;
- litlen = (U32)(ip - anchor);
-
- /* check repCode */
- {
- U32 i, last_i = ZSTD_REP_CHECK + (ip == anchor);
- for (i = (ip == anchor); i < last_i; i++) {
- const S32 repCur = (i == ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
- if ((repCur > 0) && (repCur < (S32)(ip - prefixStart)) &&
- (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repCur, minMatch))) {
- mlen = (U32)ZSTD_count(ip + minMatch, ip + minMatch - repCur, iend) + minMatch;
- if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
- best_mlen = mlen;
- best_off = i;
- cur = 0;
- last_pos = 1;
- goto _storeSequence;
- }
- best_off = i - (ip == anchor);
- do {
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
- if (mlen > last_pos || price < opt[mlen].price)
- SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
- mlen--;
- } while (mlen >= minMatch);
- }
- }
- }
-
- match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, ip, iend, maxSearches, mls, matches, minMatch);
-
- if (!last_pos && !match_num) {
- ip++;
- continue;
- }
-
- if (match_num && (matches[match_num - 1].len > sufficient_len || matches[match_num - 1].len >= ZSTD_OPT_NUM)) {
- best_mlen = matches[match_num - 1].len;
- best_off = matches[match_num - 1].off;
- cur = 0;
- last_pos = 1;
- goto _storeSequence;
- }
-
- /* set prices using matches at position = 0 */
- best_mlen = (last_pos) ? last_pos : minMatch;
- for (u = 0; u < match_num; u++) {
- mlen = (u > 0) ? matches[u - 1].len + 1 : best_mlen;
- best_mlen = matches[u].len;
- while (mlen <= best_mlen) {
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, matches[u].off - 1, mlen - MINMATCH, ultra);
- if (mlen > last_pos || price < opt[mlen].price)
- SET_PRICE(mlen, mlen, matches[u].off, litlen, price); /* note : macro modifies last_pos */
- mlen++;
- }
- }
-
- if (last_pos < minMatch) {
- ip++;
- continue;
- }
-
- /* initialize opt[0] */
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- opt[0].rep[i] = rep[i];
- }
- opt[0].mlen = 1;
- opt[0].litlen = litlen;
-
- /* check further positions */
- for (cur = 1; cur <= last_pos; cur++) {
- inr = ip + cur;
-
- if (opt[cur - 1].mlen == 1) {
- litlen = opt[cur - 1].litlen + 1;
- if (cur > litlen) {
- price = opt[cur - litlen].price + ZSTD_getLiteralPrice(seqStorePtr, litlen, inr - litlen);
- } else
- price = ZSTD_getLiteralPrice(seqStorePtr, litlen, anchor);
- } else {
- litlen = 1;
- price = opt[cur - 1].price + ZSTD_getLiteralPrice(seqStorePtr, litlen, inr - 1);
- }
-
- if (cur > last_pos || price <= opt[cur].price)
- SET_PRICE(cur, 1, 0, litlen, price);
-
- if (cur == last_pos)
- break;
-
- if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
- continue;
-
- mlen = opt[cur].mlen;
- if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
- opt[cur].rep[2] = opt[cur - mlen].rep[1];
- opt[cur].rep[1] = opt[cur - mlen].rep[0];
- opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
- } else {
- opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur - mlen].rep[1] : opt[cur - mlen].rep[2];
- opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur - mlen].rep[0] : opt[cur - mlen].rep[1];
- opt[cur].rep[0] =
- ((opt[cur].off == ZSTD_REP_MOVE_OPT) && (mlen != 1)) ? (opt[cur - mlen].rep[0] - 1) : (opt[cur - mlen].rep[opt[cur].off]);
- }
-
- best_mlen = minMatch;
- {
- U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
- for (i = (opt[cur].mlen != 1); i < last_i; i++) { /* check rep */
- const S32 repCur = (i == ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
- if ((repCur > 0) && (repCur < (S32)(inr - prefixStart)) &&
- (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(inr - repCur, minMatch))) {
- mlen = (U32)ZSTD_count(inr + minMatch, inr + minMatch - repCur, iend) + minMatch;
-
- if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
- best_mlen = mlen;
- best_off = i;
- last_pos = cur + 1;
- goto _storeSequence;
- }
-
- best_off = i - (opt[cur].mlen != 1);
- if (mlen > best_mlen)
- best_mlen = mlen;
-
- do {
- if (opt[cur].mlen == 1) {
- litlen = opt[cur].litlen;
- if (cur > litlen) {
- price = opt[cur - litlen].price + ZSTD_getPrice(seqStorePtr, litlen, inr - litlen,
- best_off, mlen - MINMATCH, ultra);
- } else
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
- } else {
- litlen = 0;
- price = opt[cur].price + ZSTD_getPrice(seqStorePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
- }
-
- if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
- SET_PRICE(cur + mlen, mlen, i, litlen, price);
- mlen--;
- } while (mlen >= minMatch);
- }
- }
- }
-
- match_num = ZSTD_BtGetAllMatches_selectMLS(ctx, inr, iend, maxSearches, mls, matches, best_mlen);
-
- if (match_num > 0 && (matches[match_num - 1].len > sufficient_len || cur + matches[match_num - 1].len >= ZSTD_OPT_NUM)) {
- best_mlen = matches[match_num - 1].len;
- best_off = matches[match_num - 1].off;
- last_pos = cur + 1;
- goto _storeSequence;
- }
-
- /* set prices using matches at position = cur */
- for (u = 0; u < match_num; u++) {
- mlen = (u > 0) ? matches[u - 1].len + 1 : best_mlen;
- best_mlen = matches[u].len;
-
- while (mlen <= best_mlen) {
- if (opt[cur].mlen == 1) {
- litlen = opt[cur].litlen;
- if (cur > litlen)
- price = opt[cur - litlen].price + ZSTD_getPrice(seqStorePtr, litlen, ip + cur - litlen,
- matches[u].off - 1, mlen - MINMATCH, ultra);
- else
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, matches[u].off - 1, mlen - MINMATCH, ultra);
- } else {
- litlen = 0;
- price = opt[cur].price + ZSTD_getPrice(seqStorePtr, 0, NULL, matches[u].off - 1, mlen - MINMATCH, ultra);
- }
-
- if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
- SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
-
- mlen++;
- }
- }
- }
-
- best_mlen = opt[last_pos].mlen;
- best_off = opt[last_pos].off;
- cur = last_pos - best_mlen;
-
- /* store sequence */
-_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
- opt[0].mlen = 1;
-
- while (1) {
- mlen = opt[cur].mlen;
- offset = opt[cur].off;
- opt[cur].mlen = best_mlen;
- opt[cur].off = best_off;
- best_mlen = mlen;
- best_off = offset;
- if (mlen > cur)
- break;
- cur -= mlen;
- }
-
- for (u = 0; u <= last_pos;) {
- u += opt[u].mlen;
- }
-
- for (cur = 0; cur < last_pos;) {
- mlen = opt[cur].mlen;
- if (mlen == 1) {
- ip++;
- cur++;
- continue;
- }
- offset = opt[cur].off;
- cur += mlen;
- litLength = (U32)(ip - anchor);
-
- if (offset > ZSTD_REP_MOVE_OPT) {
- rep[2] = rep[1];
- rep[1] = rep[0];
- rep[0] = offset - ZSTD_REP_MOVE_OPT;
- offset--;
- } else {
- if (offset != 0) {
- best_off = (offset == ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
- if (offset != 1)
- rep[2] = rep[1];
- rep[1] = rep[0];
- rep[0] = best_off;
- }
- if (litLength == 0)
- offset--;
- }
-
- ZSTD_updatePrice(seqStorePtr, litLength, anchor, offset, mlen - MINMATCH);
- ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen - MINMATCH);
- anchor = ip = ip + mlen;
- }
- } /* for (cur=0; cur < last_pos; ) */
-
- /* Save reps for next block */
- {
- int i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- ctx->repToConfirm[i] = rep[i];
- }
-
- /* Last Literals */
- {
- size_t const lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-FORCE_INLINE
-void ZSTD_compressBlock_opt_extDict_generic(ZSTD_CCtx *ctx, const void *src, size_t srcSize, const int ultra)
-{
- seqStore_t *seqStorePtr = &(ctx->seqStore);
- const BYTE *const istart = (const BYTE *)src;
- const BYTE *ip = istart;
- const BYTE *anchor = istart;
- const BYTE *const iend = istart + srcSize;
- const BYTE *const ilimit = iend - 8;
- const BYTE *const base = ctx->base;
- const U32 lowestIndex = ctx->lowLimit;
- const U32 dictLimit = ctx->dictLimit;
- const BYTE *const prefixStart = base + dictLimit;
- const BYTE *const dictBase = ctx->dictBase;
- const BYTE *const dictEnd = dictBase + dictLimit;
-
- const U32 maxSearches = 1U << ctx->params.cParams.searchLog;
- const U32 sufficient_len = ctx->params.cParams.targetLength;
- const U32 mls = ctx->params.cParams.searchLength;
- const U32 minMatch = (ctx->params.cParams.searchLength == 3) ? 3 : 4;
-
- ZSTD_optimal_t *opt = seqStorePtr->priceTable;
- ZSTD_match_t *matches = seqStorePtr->matchTable;
- const BYTE *inr;
-
- /* init */
- U32 offset, rep[ZSTD_REP_NUM];
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- rep[i] = ctx->rep[i];
- }
-
- ctx->nextToUpdate3 = ctx->nextToUpdate;
- ZSTD_rescaleFreqs(seqStorePtr, (const BYTE *)src, srcSize);
- ip += (ip == prefixStart);
-
- /* Match Loop */
- while (ip < ilimit) {
- U32 cur, match_num, last_pos, litlen, price;
- U32 u, mlen, best_mlen, best_off, litLength;
- U32 curr = (U32)(ip - base);
- memset(opt, 0, sizeof(ZSTD_optimal_t));
- last_pos = 0;
- opt[0].litlen = (U32)(ip - anchor);
-
- /* check repCode */
- {
- U32 i, last_i = ZSTD_REP_CHECK + (ip == anchor);
- for (i = (ip == anchor); i < last_i; i++) {
- const S32 repCur = (i == ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : rep[i];
- const U32 repIndex = (U32)(curr - repCur);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if ((repCur > 0 && repCur <= (S32)curr) &&
- (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch))) {
- /* repcode detected we should take it */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- mlen = (U32)ZSTD_count_2segments(ip + minMatch, repMatch + minMatch, iend, repEnd, prefixStart) + minMatch;
-
- if (mlen > sufficient_len || mlen >= ZSTD_OPT_NUM) {
- best_mlen = mlen;
- best_off = i;
- cur = 0;
- last_pos = 1;
- goto _storeSequence;
- }
-
- best_off = i - (ip == anchor);
- litlen = opt[0].litlen;
- do {
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
- if (mlen > last_pos || price < opt[mlen].price)
- SET_PRICE(mlen, mlen, i, litlen, price); /* note : macro modifies last_pos */
- mlen--;
- } while (mlen >= minMatch);
- }
- }
- }
-
- match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, ip, iend, maxSearches, mls, matches, minMatch); /* first search (depth 0) */
-
- if (!last_pos && !match_num) {
- ip++;
- continue;
- }
-
- {
- U32 i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- opt[0].rep[i] = rep[i];
- }
- opt[0].mlen = 1;
-
- if (match_num && (matches[match_num - 1].len > sufficient_len || matches[match_num - 1].len >= ZSTD_OPT_NUM)) {
- best_mlen = matches[match_num - 1].len;
- best_off = matches[match_num - 1].off;
- cur = 0;
- last_pos = 1;
- goto _storeSequence;
- }
-
- best_mlen = (last_pos) ? last_pos : minMatch;
-
- /* set prices using matches at position = 0 */
- for (u = 0; u < match_num; u++) {
- mlen = (u > 0) ? matches[u - 1].len + 1 : best_mlen;
- best_mlen = matches[u].len;
- litlen = opt[0].litlen;
- while (mlen <= best_mlen) {
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, matches[u].off - 1, mlen - MINMATCH, ultra);
- if (mlen > last_pos || price < opt[mlen].price)
- SET_PRICE(mlen, mlen, matches[u].off, litlen, price);
- mlen++;
- }
- }
-
- if (last_pos < minMatch) {
- ip++;
- continue;
- }
-
- /* check further positions */
- for (cur = 1; cur <= last_pos; cur++) {
- inr = ip + cur;
-
- if (opt[cur - 1].mlen == 1) {
- litlen = opt[cur - 1].litlen + 1;
- if (cur > litlen) {
- price = opt[cur - litlen].price + ZSTD_getLiteralPrice(seqStorePtr, litlen, inr - litlen);
- } else
- price = ZSTD_getLiteralPrice(seqStorePtr, litlen, anchor);
- } else {
- litlen = 1;
- price = opt[cur - 1].price + ZSTD_getLiteralPrice(seqStorePtr, litlen, inr - 1);
- }
-
- if (cur > last_pos || price <= opt[cur].price)
- SET_PRICE(cur, 1, 0, litlen, price);
-
- if (cur == last_pos)
- break;
-
- if (inr > ilimit) /* last match must start at a minimum distance of 8 from oend */
- continue;
-
- mlen = opt[cur].mlen;
- if (opt[cur].off > ZSTD_REP_MOVE_OPT) {
- opt[cur].rep[2] = opt[cur - mlen].rep[1];
- opt[cur].rep[1] = opt[cur - mlen].rep[0];
- opt[cur].rep[0] = opt[cur].off - ZSTD_REP_MOVE_OPT;
- } else {
- opt[cur].rep[2] = (opt[cur].off > 1) ? opt[cur - mlen].rep[1] : opt[cur - mlen].rep[2];
- opt[cur].rep[1] = (opt[cur].off > 0) ? opt[cur - mlen].rep[0] : opt[cur - mlen].rep[1];
- opt[cur].rep[0] =
- ((opt[cur].off == ZSTD_REP_MOVE_OPT) && (mlen != 1)) ? (opt[cur - mlen].rep[0] - 1) : (opt[cur - mlen].rep[opt[cur].off]);
- }
-
- best_mlen = minMatch;
- {
- U32 i, last_i = ZSTD_REP_CHECK + (mlen != 1);
- for (i = (mlen != 1); i < last_i; i++) {
- const S32 repCur = (i == ZSTD_REP_MOVE_OPT) ? (opt[cur].rep[0] - 1) : opt[cur].rep[i];
- const U32 repIndex = (U32)(curr + cur - repCur);
- const BYTE *const repBase = repIndex < dictLimit ? dictBase : base;
- const BYTE *const repMatch = repBase + repIndex;
- if ((repCur > 0 && repCur <= (S32)(curr + cur)) &&
- (((U32)((dictLimit - 1) - repIndex) >= 3) & (repIndex > lowestIndex)) /* intentional overflow */
- && (ZSTD_readMINMATCH(inr, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch))) {
- /* repcode detected */
- const BYTE *const repEnd = repIndex < dictLimit ? dictEnd : iend;
- mlen = (U32)ZSTD_count_2segments(inr + minMatch, repMatch + minMatch, iend, repEnd, prefixStart) + minMatch;
-
- if (mlen > sufficient_len || cur + mlen >= ZSTD_OPT_NUM) {
- best_mlen = mlen;
- best_off = i;
- last_pos = cur + 1;
- goto _storeSequence;
- }
-
- best_off = i - (opt[cur].mlen != 1);
- if (mlen > best_mlen)
- best_mlen = mlen;
-
- do {
- if (opt[cur].mlen == 1) {
- litlen = opt[cur].litlen;
- if (cur > litlen) {
- price = opt[cur - litlen].price + ZSTD_getPrice(seqStorePtr, litlen, inr - litlen,
- best_off, mlen - MINMATCH, ultra);
- } else
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, best_off, mlen - MINMATCH, ultra);
- } else {
- litlen = 0;
- price = opt[cur].price + ZSTD_getPrice(seqStorePtr, 0, NULL, best_off, mlen - MINMATCH, ultra);
- }
-
- if (cur + mlen > last_pos || price <= opt[cur + mlen].price)
- SET_PRICE(cur + mlen, mlen, i, litlen, price);
- mlen--;
- } while (mlen >= minMatch);
- }
- }
- }
-
- match_num = ZSTD_BtGetAllMatches_selectMLS_extDict(ctx, inr, iend, maxSearches, mls, matches, minMatch);
-
- if (match_num > 0 && (matches[match_num - 1].len > sufficient_len || cur + matches[match_num - 1].len >= ZSTD_OPT_NUM)) {
- best_mlen = matches[match_num - 1].len;
- best_off = matches[match_num - 1].off;
- last_pos = cur + 1;
- goto _storeSequence;
- }
-
- /* set prices using matches at position = cur */
- for (u = 0; u < match_num; u++) {
- mlen = (u > 0) ? matches[u - 1].len + 1 : best_mlen;
- best_mlen = matches[u].len;
-
- while (mlen <= best_mlen) {
- if (opt[cur].mlen == 1) {
- litlen = opt[cur].litlen;
- if (cur > litlen)
- price = opt[cur - litlen].price + ZSTD_getPrice(seqStorePtr, litlen, ip + cur - litlen,
- matches[u].off - 1, mlen - MINMATCH, ultra);
- else
- price = ZSTD_getPrice(seqStorePtr, litlen, anchor, matches[u].off - 1, mlen - MINMATCH, ultra);
- } else {
- litlen = 0;
- price = opt[cur].price + ZSTD_getPrice(seqStorePtr, 0, NULL, matches[u].off - 1, mlen - MINMATCH, ultra);
- }
-
- if (cur + mlen > last_pos || (price < opt[cur + mlen].price))
- SET_PRICE(cur + mlen, mlen, matches[u].off, litlen, price);
-
- mlen++;
- }
- }
- } /* for (cur = 1; cur <= last_pos; cur++) */
-
- best_mlen = opt[last_pos].mlen;
- best_off = opt[last_pos].off;
- cur = last_pos - best_mlen;
-
- /* store sequence */
-_storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
- opt[0].mlen = 1;
-
- while (1) {
- mlen = opt[cur].mlen;
- offset = opt[cur].off;
- opt[cur].mlen = best_mlen;
- opt[cur].off = best_off;
- best_mlen = mlen;
- best_off = offset;
- if (mlen > cur)
- break;
- cur -= mlen;
- }
-
- for (u = 0; u <= last_pos;) {
- u += opt[u].mlen;
- }
-
- for (cur = 0; cur < last_pos;) {
- mlen = opt[cur].mlen;
- if (mlen == 1) {
- ip++;
- cur++;
- continue;
- }
- offset = opt[cur].off;
- cur += mlen;
- litLength = (U32)(ip - anchor);
-
- if (offset > ZSTD_REP_MOVE_OPT) {
- rep[2] = rep[1];
- rep[1] = rep[0];
- rep[0] = offset - ZSTD_REP_MOVE_OPT;
- offset--;
- } else {
- if (offset != 0) {
- best_off = (offset == ZSTD_REP_MOVE_OPT) ? (rep[0] - 1) : (rep[offset]);
- if (offset != 1)
- rep[2] = rep[1];
- rep[1] = rep[0];
- rep[0] = best_off;
- }
-
- if (litLength == 0)
- offset--;
- }
-
- ZSTD_updatePrice(seqStorePtr, litLength, anchor, offset, mlen - MINMATCH);
- ZSTD_storeSeq(seqStorePtr, litLength, anchor, offset, mlen - MINMATCH);
- anchor = ip = ip + mlen;
- }
- } /* for (cur=0; cur < last_pos; ) */
-
- /* Save reps for next block */
- {
- int i;
- for (i = 0; i < ZSTD_REP_NUM; i++)
- ctx->repToConfirm[i] = rep[i];
- }
-
- /* Last Literals */
- {
- size_t lastLLSize = iend - anchor;
- memcpy(seqStorePtr->lit, anchor, lastLLSize);
- seqStorePtr->lit += lastLLSize;
- }
-}
-
-#endif /* ZSTD_OPT_H_91842398743 */
config MEMORY_ISOLATION
bool
+# IORESOURCE_SYSTEM_RAM regions in the kernel resource tree that are marked
+# IORESOURCE_EXCLUSIVE cannot be mapped to user space, for example, via
+# /dev/mem.
+config EXCLUSIVE_SYSTEM_RAM
+ def_bool y
+ depends on !DEVMEM || STRICT_DEVMEM
+
#
# Only be set on architectures that have completely implemented memory hotplug
# feature. If you are not sure, don't touch it.
config MEMORY_HOTPLUG
bool "Allow for memory hot-add"
select MEMORY_ISOLATION
- depends on SPARSEMEM || X86_64_ACPI_NUMA
+ depends on SPARSEMEM
depends on ARCH_ENABLE_MEMORY_HOTPLUG
- depends on 64BIT || BROKEN
+ depends on 64BIT
select NUMA_KEEP_MEMINFO if NUMA
-config MEMORY_HOTPLUG_SPARSE
- def_bool y
- depends on SPARSEMEM && MEMORY_HOTPLUG
-
config MEMORY_HOTPLUG_DEFAULT_ONLINE
bool "Online the newly added memory blocks by default"
depends on MEMORY_HOTPLUG
config TRANSPARENT_HUGEPAGE
bool "Transparent Hugepage Support"
- depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE
+ depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE && !PREEMPT_RT
select COMPACTION
select XARRAY_MULTI
help
config KMAP_LOCAL
bool
+config KMAP_LOCAL_NON_LINEAR_PTE_ARRAY
+ bool
+
# struct io_mapping based helper. Selected by drivers that need them
config IO_MAPPING
bool
memset(wb, 0, sizeof(*wb));
- if (wb != &bdi->wb)
- bdi_get(bdi);
wb->bdi = bdi;
wb->last_old_flush = jiffies;
INIT_LIST_HEAD(&wb->b_dirty);
err = fprop_local_init_percpu(&wb->completions, gfp);
if (err)
- goto out_put_bdi;
+ return err;
for (i = 0; i < NR_WB_STAT_ITEMS; i++) {
err = percpu_counter_init(&wb->stat[i], 0, gfp);
while (i--)
percpu_counter_destroy(&wb->stat[i]);
fprop_local_destroy_percpu(&wb->completions);
-out_put_bdi:
- if (wb != &bdi->wb)
- bdi_put(bdi);
return err;
}
percpu_counter_destroy(&wb->stat[i]);
fprop_local_destroy_percpu(&wb->completions);
- if (wb != &wb->bdi->wb)
- bdi_put(wb->bdi);
}
#ifdef CONFIG_CGROUP_WRITEBACK
struct bdi_writeback *wb = container_of(work, struct bdi_writeback,
release_work);
struct blkcg *blkcg = css_to_blkcg(wb->blkcg_css);
+ struct backing_dev_info *bdi = wb->bdi;
mutex_lock(&wb->bdi->cgwb_release_mutex);
wb_shutdown(wb);
percpu_ref_exit(&wb->refcnt);
wb_exit(wb);
+ bdi_put(bdi);
WARN_ON_ONCE(!list_empty(&wb->b_attached));
kfree_rcu(wb, rcu);
}
INIT_LIST_HEAD(&wb->b_attached);
INIT_WORK(&wb->release_work, cgwb_release_workfn);
set_bit(WB_registered, &wb->state);
+ bdi_get(bdi);
/*
* The root wb determines the registered state of the whole bdi and
goto out_put;
err_fprop_exit:
+ bdi_put(bdi);
fprop_local_destroy_percpu(&wb->memcg_completions);
err_ref_exit:
percpu_ref_exit(&wb->refcnt);
bdi->owner = NULL;
}
}
+EXPORT_SYMBOL(bdi_unregister);
static void release_bdi(struct kref *ref)
{
struct backing_dev_info *bdi =
container_of(ref, struct backing_dev_info, refcnt);
- if (test_bit(WB_registered, &bdi->wb.state))
- bdi_unregister(bdi);
+ WARN_ON_ONCE(test_bit(WB_registered, &bdi->wb.state));
WARN_ON_ONCE(bdi->dev);
wb_exit(&bdi->wb);
kfree(bdi);
return ret;
}
EXPORT_SYMBOL(congestion_wait);
-
-/**
- * wait_iff_congested - Conditionally wait for a backing_dev to become uncongested or a pgdat to complete writes
- * @sync: SYNC or ASYNC IO
- * @timeout: timeout in jiffies
- *
- * In the event of a congested backing_dev (any backing_dev) this waits
- * for up to @timeout jiffies for either a BDI to exit congestion of the
- * given @sync queue or a write to complete.
- *
- * The return value is 0 if the sleep is for the full timeout. Otherwise,
- * it is the number of jiffies that were still remaining when the function
- * returned. return_value == timeout implies the function did not sleep.
- */
-long wait_iff_congested(int sync, long timeout)
-{
- long ret;
- unsigned long start = jiffies;
- DEFINE_WAIT(wait);
- wait_queue_head_t *wqh = &congestion_wqh[sync];
-
- /*
- * If there is no congestion, yield if necessary instead
- * of sleeping on the congestion queue
- */
- if (atomic_read(&nr_wb_congested[sync]) == 0) {
- cond_resched();
-
- /* In case we scheduled, work out time remaining */
- ret = timeout - (jiffies - start);
- if (ret < 0)
- ret = 0;
-
- goto out;
- }
-
- /* Sleep until uncongested or a write happens */
- prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
- ret = io_schedule_timeout(timeout);
- finish_wait(wqh, &wait);
-
-out:
- trace_writeback_wait_iff_congested(jiffies_to_usecs(timeout),
- jiffies_to_usecs(jiffies - start));
-
- return ret;
-}
-EXPORT_SYMBOL(wait_iff_congested);
return 0;
free_mem:
- memblock_free(base, size);
+ memblock_phys_free(base, size);
err:
pr_err("Failed to reserve %ld MiB\n", (unsigned long)size / SZ_1M);
return ret;
return page;
}
+bool cma_pages_valid(struct cma *cma, const struct page *pages,
+ unsigned long count)
+{
+ unsigned long pfn;
+
+ if (!cma || !pages)
+ return false;
+
+ pfn = page_to_pfn(pages);
+
+ if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) {
+ pr_debug("%s(page %p, count %lu)\n", __func__,
+ (void *)pages, count);
+ return false;
+ }
+
+ return true;
+}
+
/**
* cma_release() - release allocated pages
* @cma: Contiguous memory region for which the allocation is performed.
{
unsigned long pfn;
- if (!cma || !pages)
+ if (!cma_pages_valid(cma, pages, count))
return false;
pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count);
pfn = page_to_pfn(pages);
- if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count)
- return false;
-
VM_BUG_ON(pfn + count > cma->base_pfn + cma->count);
free_contig_range(pfn, count);
/* Similar to reclaim, but different enough that they don't share logic */
static bool too_many_isolated(pg_data_t *pgdat)
{
+ bool too_many;
+
unsigned long active, inactive, isolated;
inactive = node_page_state(pgdat, NR_INACTIVE_FILE) +
isolated = node_page_state(pgdat, NR_ISOLATED_FILE) +
node_page_state(pgdat, NR_ISOLATED_ANON);
- return isolated > (inactive + active) / 2;
+ too_many = isolated > (inactive + active) / 2;
+ if (!too_many)
+ wake_throttle_isolated(pgdat);
+
+ return too_many;
}
/**
if (cc->mode == MIGRATE_ASYNC)
return -EAGAIN;
- congestion_wait(BLK_RW_ASYNC, HZ/10);
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
if (fatal_signal_pending(current))
return -EINTR;
select PAGE_IDLE_FLAG
help
This builds the default data access monitoring primitives for DAMON
- that works for virtual address spaces.
+ that work for virtual address spaces.
+
+config DAMON_PADDR
+ bool "Data access monitoring primitives for the physical address space"
+ depends on DAMON && MMU
+ select PAGE_IDLE_FLAG
+ help
+ This builds the default data access monitoring primitives for DAMON
+ that works for the physical address space.
config DAMON_VADDR_KUNIT_TEST
bool "Test for DAMON primitives" if !KUNIT_ALL_TESTS
config DAMON_DBGFS
bool "DAMON debugfs interface"
- depends on DAMON_VADDR && DEBUG_FS
+ depends on DAMON_VADDR && DAMON_PADDR && DEBUG_FS
help
This builds the debugfs interface for DAMON. The user space admins
can use the interface for arbitrary data access monitoring.
If unsure, say N.
+config DAMON_RECLAIM
+ bool "Build DAMON-based reclaim (DAMON_RECLAIM)"
+ depends on DAMON_PADDR
+ help
+ This builds the DAMON-based reclamation subsystem. It finds pages
+ that not accessed for a long time (cold) using DAMON and reclaim
+ those.
+
+ This is suggested to be used as a proactive and lightweight
+ reclamation under light memory pressure, while the traditional page
+ scanning-based reclamation is used for heavy pressure.
+
endmenu
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_DAMON) := core.o
-obj-$(CONFIG_DAMON_VADDR) += vaddr.o
+obj-$(CONFIG_DAMON_VADDR) += prmtv-common.o vaddr.o
+obj-$(CONFIG_DAMON_PADDR) += prmtv-common.o paddr.o
obj-$(CONFIG_DAMON_DBGFS) += dbgfs.o
+obj-$(CONFIG_DAMON_RECLAIM) += reclaim.o
#include <linux/damon.h>
#include <linux/delay.h>
#include <linux/kthread.h>
+#include <linux/mm.h>
#include <linux/random.h>
#include <linux/slab.h>
+#include <linux/string.h>
#define CREATE_TRACE_POINTS
#include <trace/events/damon.h>
region->nr_accesses = 0;
INIT_LIST_HEAD(®ion->list);
+ region->age = 0;
+ region->last_nr_accesses = 0;
+
return region;
}
damon_free_region(r);
}
+struct damos *damon_new_scheme(
+ unsigned long min_sz_region, unsigned long max_sz_region,
+ unsigned int min_nr_accesses, unsigned int max_nr_accesses,
+ unsigned int min_age_region, unsigned int max_age_region,
+ enum damos_action action, struct damos_quota *quota,
+ struct damos_watermarks *wmarks)
+{
+ struct damos *scheme;
+
+ scheme = kmalloc(sizeof(*scheme), GFP_KERNEL);
+ if (!scheme)
+ return NULL;
+ scheme->min_sz_region = min_sz_region;
+ scheme->max_sz_region = max_sz_region;
+ scheme->min_nr_accesses = min_nr_accesses;
+ scheme->max_nr_accesses = max_nr_accesses;
+ scheme->min_age_region = min_age_region;
+ scheme->max_age_region = max_age_region;
+ scheme->action = action;
+ scheme->stat_count = 0;
+ scheme->stat_sz = 0;
+ INIT_LIST_HEAD(&scheme->list);
+
+ scheme->quota.ms = quota->ms;
+ scheme->quota.sz = quota->sz;
+ scheme->quota.reset_interval = quota->reset_interval;
+ scheme->quota.weight_sz = quota->weight_sz;
+ scheme->quota.weight_nr_accesses = quota->weight_nr_accesses;
+ scheme->quota.weight_age = quota->weight_age;
+ scheme->quota.total_charged_sz = 0;
+ scheme->quota.total_charged_ns = 0;
+ scheme->quota.esz = 0;
+ scheme->quota.charged_sz = 0;
+ scheme->quota.charged_from = 0;
+ scheme->quota.charge_target_from = NULL;
+ scheme->quota.charge_addr_from = 0;
+
+ scheme->wmarks.metric = wmarks->metric;
+ scheme->wmarks.interval = wmarks->interval;
+ scheme->wmarks.high = wmarks->high;
+ scheme->wmarks.mid = wmarks->mid;
+ scheme->wmarks.low = wmarks->low;
+ scheme->wmarks.activated = true;
+
+ return scheme;
+}
+
+void damon_add_scheme(struct damon_ctx *ctx, struct damos *s)
+{
+ list_add_tail(&s->list, &ctx->schemes);
+}
+
+static void damon_del_scheme(struct damos *s)
+{
+ list_del(&s->list);
+}
+
+static void damon_free_scheme(struct damos *s)
+{
+ kfree(s);
+}
+
+void damon_destroy_scheme(struct damos *s)
+{
+ damon_del_scheme(s);
+ damon_free_scheme(s);
+}
+
/*
* Construct a damon_target struct
*
list_add_tail(&t->list, &ctx->adaptive_targets);
}
+bool damon_targets_empty(struct damon_ctx *ctx)
+{
+ return list_empty(&ctx->adaptive_targets);
+}
+
static void damon_del_target(struct damon_target *t)
{
list_del(&t->list);
ctx->max_nr_regions = 1000;
INIT_LIST_HEAD(&ctx->adaptive_targets);
+ INIT_LIST_HEAD(&ctx->schemes);
return ctx;
}
void damon_destroy_ctx(struct damon_ctx *ctx)
{
+ struct damos *s, *next_s;
+
damon_destroy_targets(ctx);
+
+ damon_for_each_scheme_safe(s, next_s, ctx)
+ damon_destroy_scheme(s);
+
kfree(ctx);
}
}
/**
+ * damon_set_schemes() - Set data access monitoring based operation schemes.
+ * @ctx: monitoring context
+ * @schemes: array of the schemes
+ * @nr_schemes: number of entries in @schemes
+ *
+ * This function should not be called while the kdamond of the context is
+ * running.
+ *
+ * Return: 0 if success, or negative error code otherwise.
+ */
+int damon_set_schemes(struct damon_ctx *ctx, struct damos **schemes,
+ ssize_t nr_schemes)
+{
+ struct damos *s, *next;
+ ssize_t i;
+
+ damon_for_each_scheme_safe(s, next, ctx)
+ damon_destroy_scheme(s);
+ for (i = 0; i < nr_schemes; i++)
+ damon_add_scheme(ctx, schemes[i]);
+ return 0;
+}
+
+/**
* damon_nr_running_ctxs() - Return number of currently running contexts.
*/
int damon_nr_running_ctxs(void)
return sz;
}
-static bool damon_kdamond_running(struct damon_ctx *ctx)
-{
- bool running;
-
- mutex_lock(&ctx->kdamond_lock);
- running = ctx->kdamond != NULL;
- mutex_unlock(&ctx->kdamond_lock);
-
- return running;
-}
-
static int kdamond_fn(void *data);
/*
mutex_lock(&ctx->kdamond_lock);
if (!ctx->kdamond) {
err = 0;
- ctx->kdamond_stop = false;
ctx->kdamond = kthread_run(kdamond_fn, ctx, "kdamond.%d",
nr_running_ctxs);
if (IS_ERR(ctx->kdamond)) {
err = PTR_ERR(ctx->kdamond);
- ctx->kdamond = 0;
+ ctx->kdamond = NULL;
}
}
mutex_unlock(&ctx->kdamond_lock);
*/
static int __damon_stop(struct damon_ctx *ctx)
{
+ struct task_struct *tsk;
+
mutex_lock(&ctx->kdamond_lock);
- if (ctx->kdamond) {
- ctx->kdamond_stop = true;
+ tsk = ctx->kdamond;
+ if (tsk) {
+ get_task_struct(tsk);
mutex_unlock(&ctx->kdamond_lock);
- while (damon_kdamond_running(ctx))
- usleep_range(ctx->sample_interval,
- ctx->sample_interval * 2);
+ kthread_stop(tsk);
+ put_task_struct(tsk);
return 0;
}
mutex_unlock(&ctx->kdamond_lock);
damon_for_each_region(r, t) {
trace_damon_aggregated(t, r, damon_nr_regions(t));
+ r->last_nr_accesses = r->nr_accesses;
r->nr_accesses = 0;
}
}
}
+static void damon_split_region_at(struct damon_ctx *ctx,
+ struct damon_target *t, struct damon_region *r,
+ unsigned long sz_r);
+
+static bool __damos_valid_target(struct damon_region *r, struct damos *s)
+{
+ unsigned long sz;
+
+ sz = r->ar.end - r->ar.start;
+ return s->min_sz_region <= sz && sz <= s->max_sz_region &&
+ s->min_nr_accesses <= r->nr_accesses &&
+ r->nr_accesses <= s->max_nr_accesses &&
+ s->min_age_region <= r->age && r->age <= s->max_age_region;
+}
+
+static bool damos_valid_target(struct damon_ctx *c, struct damon_target *t,
+ struct damon_region *r, struct damos *s)
+{
+ bool ret = __damos_valid_target(r, s);
+
+ if (!ret || !s->quota.esz || !c->primitive.get_scheme_score)
+ return ret;
+
+ return c->primitive.get_scheme_score(c, t, r, s) >= s->quota.min_score;
+}
+
+static void damon_do_apply_schemes(struct damon_ctx *c,
+ struct damon_target *t,
+ struct damon_region *r)
+{
+ struct damos *s;
+
+ damon_for_each_scheme(s, c) {
+ struct damos_quota *quota = &s->quota;
+ unsigned long sz = r->ar.end - r->ar.start;
+ struct timespec64 begin, end;
+
+ if (!s->wmarks.activated)
+ continue;
+
+ /* Check the quota */
+ if (quota->esz && quota->charged_sz >= quota->esz)
+ continue;
+
+ /* Skip previously charged regions */
+ if (quota->charge_target_from) {
+ if (t != quota->charge_target_from)
+ continue;
+ if (r == damon_last_region(t)) {
+ quota->charge_target_from = NULL;
+ quota->charge_addr_from = 0;
+ continue;
+ }
+ if (quota->charge_addr_from &&
+ r->ar.end <= quota->charge_addr_from)
+ continue;
+
+ if (quota->charge_addr_from && r->ar.start <
+ quota->charge_addr_from) {
+ sz = ALIGN_DOWN(quota->charge_addr_from -
+ r->ar.start, DAMON_MIN_REGION);
+ if (!sz) {
+ if (r->ar.end - r->ar.start <=
+ DAMON_MIN_REGION)
+ continue;
+ sz = DAMON_MIN_REGION;
+ }
+ damon_split_region_at(c, t, r, sz);
+ r = damon_next_region(r);
+ sz = r->ar.end - r->ar.start;
+ }
+ quota->charge_target_from = NULL;
+ quota->charge_addr_from = 0;
+ }
+
+ if (!damos_valid_target(c, t, r, s))
+ continue;
+
+ /* Apply the scheme */
+ if (c->primitive.apply_scheme) {
+ if (quota->esz &&
+ quota->charged_sz + sz > quota->esz) {
+ sz = ALIGN_DOWN(quota->esz - quota->charged_sz,
+ DAMON_MIN_REGION);
+ if (!sz)
+ goto update_stat;
+ damon_split_region_at(c, t, r, sz);
+ }
+ ktime_get_coarse_ts64(&begin);
+ c->primitive.apply_scheme(c, t, r, s);
+ ktime_get_coarse_ts64(&end);
+ quota->total_charged_ns += timespec64_to_ns(&end) -
+ timespec64_to_ns(&begin);
+ quota->charged_sz += sz;
+ if (quota->esz && quota->charged_sz >= quota->esz) {
+ quota->charge_target_from = t;
+ quota->charge_addr_from = r->ar.end + 1;
+ }
+ }
+ if (s->action != DAMOS_STAT)
+ r->age = 0;
+
+update_stat:
+ s->stat_count++;
+ s->stat_sz += sz;
+ }
+}
+
+/* Shouldn't be called if quota->ms and quota->sz are zero */
+static void damos_set_effective_quota(struct damos_quota *quota)
+{
+ unsigned long throughput;
+ unsigned long esz;
+
+ if (!quota->ms) {
+ quota->esz = quota->sz;
+ return;
+ }
+
+ if (quota->total_charged_ns)
+ throughput = quota->total_charged_sz * 1000000 /
+ quota->total_charged_ns;
+ else
+ throughput = PAGE_SIZE * 1024;
+ esz = throughput * quota->ms;
+
+ if (quota->sz && quota->sz < esz)
+ esz = quota->sz;
+ quota->esz = esz;
+}
+
+static void kdamond_apply_schemes(struct damon_ctx *c)
+{
+ struct damon_target *t;
+ struct damon_region *r, *next_r;
+ struct damos *s;
+
+ damon_for_each_scheme(s, c) {
+ struct damos_quota *quota = &s->quota;
+ unsigned long cumulated_sz;
+ unsigned int score, max_score = 0;
+
+ if (!s->wmarks.activated)
+ continue;
+
+ if (!quota->ms && !quota->sz)
+ continue;
+
+ /* New charge window starts */
+ if (time_after_eq(jiffies, quota->charged_from +
+ msecs_to_jiffies(
+ quota->reset_interval))) {
+ quota->total_charged_sz += quota->charged_sz;
+ quota->charged_from = jiffies;
+ quota->charged_sz = 0;
+ damos_set_effective_quota(quota);
+ }
+
+ if (!c->primitive.get_scheme_score)
+ continue;
+
+ /* Fill up the score histogram */
+ memset(quota->histogram, 0, sizeof(quota->histogram));
+ damon_for_each_target(t, c) {
+ damon_for_each_region(r, t) {
+ if (!__damos_valid_target(r, s))
+ continue;
+ score = c->primitive.get_scheme_score(
+ c, t, r, s);
+ quota->histogram[score] +=
+ r->ar.end - r->ar.start;
+ if (score > max_score)
+ max_score = score;
+ }
+ }
+
+ /* Set the min score limit */
+ for (cumulated_sz = 0, score = max_score; ; score--) {
+ cumulated_sz += quota->histogram[score];
+ if (cumulated_sz >= quota->esz || !score)
+ break;
+ }
+ quota->min_score = score;
+ }
+
+ damon_for_each_target(t, c) {
+ damon_for_each_region_safe(r, next_r, t)
+ damon_do_apply_schemes(c, t, r);
+ }
+}
+
#define sz_damon_region(r) (r->ar.end - r->ar.start)
/*
l->nr_accesses = (l->nr_accesses * sz_l + r->nr_accesses * sz_r) /
(sz_l + sz_r);
+ l->age = (l->age * sz_l + r->age * sz_r) / (sz_l + sz_r);
l->ar.end = r->ar.end;
damon_destroy_region(r, t);
}
struct damon_region *r, *prev = NULL, *next;
damon_for_each_region_safe(r, next, t) {
+ if (diff_of(r->nr_accesses, r->last_nr_accesses) > thres)
+ r->age = 0;
+ else
+ r->age++;
+
if (prev && prev->ar.end == r->ar.start &&
diff_of(prev->nr_accesses, r->nr_accesses) <= thres &&
sz_damon_region(prev) + sz_damon_region(r) <= sz_limit)
r->ar.end = new->ar.start;
+ new->age = r->age;
+ new->last_nr_accesses = r->last_nr_accesses;
+
damon_insert_region(new, r, damon_next_region(r), t);
}
static bool kdamond_need_stop(struct damon_ctx *ctx)
{
struct damon_target *t;
- bool stop;
- mutex_lock(&ctx->kdamond_lock);
- stop = ctx->kdamond_stop;
- mutex_unlock(&ctx->kdamond_lock);
- if (stop)
+ if (kthread_should_stop())
return true;
if (!ctx->primitive.target_valid)
return true;
}
-static void set_kdamond_stop(struct damon_ctx *ctx)
+static unsigned long damos_wmark_metric_value(enum damos_wmark_metric metric)
{
- mutex_lock(&ctx->kdamond_lock);
- ctx->kdamond_stop = true;
- mutex_unlock(&ctx->kdamond_lock);
+ struct sysinfo i;
+
+ switch (metric) {
+ case DAMOS_WMARK_FREE_MEM_RATE:
+ si_meminfo(&i);
+ return i.freeram * 1000 / i.totalram;
+ default:
+ break;
+ }
+ return -EINVAL;
+}
+
+/*
+ * Returns zero if the scheme is active. Else, returns time to wait for next
+ * watermark check in micro-seconds.
+ */
+static unsigned long damos_wmark_wait_us(struct damos *scheme)
+{
+ unsigned long metric;
+
+ if (scheme->wmarks.metric == DAMOS_WMARK_NONE)
+ return 0;
+
+ metric = damos_wmark_metric_value(scheme->wmarks.metric);
+ /* higher than high watermark or lower than low watermark */
+ if (metric > scheme->wmarks.high || scheme->wmarks.low > metric) {
+ if (scheme->wmarks.activated)
+ pr_debug("deactivate a scheme (%d) for %s wmark\n",
+ scheme->action,
+ metric > scheme->wmarks.high ?
+ "high" : "low");
+ scheme->wmarks.activated = false;
+ return scheme->wmarks.interval;
+ }
+
+ /* inactive and higher than middle watermark */
+ if ((scheme->wmarks.high >= metric && metric >= scheme->wmarks.mid) &&
+ !scheme->wmarks.activated)
+ return scheme->wmarks.interval;
+
+ if (!scheme->wmarks.activated)
+ pr_debug("activate a scheme (%d)\n", scheme->action);
+ scheme->wmarks.activated = true;
+ return 0;
+}
+
+static void kdamond_usleep(unsigned long usecs)
+{
+ if (usecs > 100 * 1000)
+ schedule_timeout_interruptible(usecs_to_jiffies(usecs));
+ else
+ usleep_range(usecs, usecs + 1);
+}
+
+/* Returns negative error code if it's not activated but should return */
+static int kdamond_wait_activation(struct damon_ctx *ctx)
+{
+ struct damos *s;
+ unsigned long wait_time;
+ unsigned long min_wait_time = 0;
+
+ while (!kdamond_need_stop(ctx)) {
+ damon_for_each_scheme(s, ctx) {
+ wait_time = damos_wmark_wait_us(s);
+ if (!min_wait_time || wait_time < min_wait_time)
+ min_wait_time = wait_time;
+ }
+ if (!min_wait_time)
+ return 0;
+
+ kdamond_usleep(min_wait_time);
+ }
+ return -EBUSY;
}
/*
struct damon_region *r, *next;
unsigned int max_nr_accesses = 0;
unsigned long sz_limit = 0;
+ bool done = false;
- mutex_lock(&ctx->kdamond_lock);
- pr_info("kdamond (%d) starts\n", ctx->kdamond->pid);
- mutex_unlock(&ctx->kdamond_lock);
+ pr_debug("kdamond (%d) starts\n", current->pid);
if (ctx->primitive.init)
ctx->primitive.init(ctx);
if (ctx->callback.before_start && ctx->callback.before_start(ctx))
- set_kdamond_stop(ctx);
+ done = true;
sz_limit = damon_region_sz_limit(ctx);
- while (!kdamond_need_stop(ctx)) {
+ while (!kdamond_need_stop(ctx) && !done) {
+ if (kdamond_wait_activation(ctx))
+ continue;
+
if (ctx->primitive.prepare_access_checks)
ctx->primitive.prepare_access_checks(ctx);
if (ctx->callback.after_sampling &&
ctx->callback.after_sampling(ctx))
- set_kdamond_stop(ctx);
+ done = true;
usleep_range(ctx->sample_interval, ctx->sample_interval + 1);
sz_limit);
if (ctx->callback.after_aggregation &&
ctx->callback.after_aggregation(ctx))
- set_kdamond_stop(ctx);
+ done = true;
+ kdamond_apply_schemes(ctx);
kdamond_reset_aggregated(ctx);
kdamond_split_regions(ctx);
if (ctx->primitive.reset_aggregated)
damon_destroy_region(r, t);
}
- if (ctx->callback.before_terminate &&
- ctx->callback.before_terminate(ctx))
- set_kdamond_stop(ctx);
+ if (ctx->callback.before_terminate)
+ ctx->callback.before_terminate(ctx);
if (ctx->primitive.cleanup)
ctx->primitive.cleanup(ctx);
- pr_debug("kdamond (%d) finishes\n", ctx->kdamond->pid);
+ pr_debug("kdamond (%d) finishes\n", current->pid);
mutex_lock(&ctx->kdamond_lock);
ctx->kdamond = NULL;
mutex_unlock(&ctx->kdamond_lock);
nr_running_ctxs--;
mutex_unlock(&damon_lock);
- do_exit(0);
+ return 0;
}
#include "core-test.h"
dbgfs_destroy_ctx(ctx);
}
+static void damon_dbgfs_test_set_init_regions(struct kunit *test)
+{
+ struct damon_ctx *ctx = damon_new_ctx();
+ unsigned long ids[] = {1, 2, 3};
+ /* Each line represents one region in ``<target id> <start> <end>`` */
+ char * const valid_inputs[] = {"2 10 20\n 2 20 30\n2 35 45",
+ "2 10 20\n",
+ "2 10 20\n1 39 59\n1 70 134\n 2 20 25\n",
+ ""};
+ /* Reading the file again will show sorted, clean output */
+ char * const valid_expects[] = {"2 10 20\n2 20 30\n2 35 45\n",
+ "2 10 20\n",
+ "1 39 59\n1 70 134\n2 10 20\n2 20 25\n",
+ ""};
+ char * const invalid_inputs[] = {"4 10 20\n", /* target not exists */
+ "2 10 20\n 2 14 26\n", /* regions overlap */
+ "1 10 20\n2 30 40\n 1 5 8"}; /* not sorted by address */
+ char *input, *expect;
+ int i, rc;
+ char buf[256];
+
+ damon_set_targets(ctx, ids, 3);
+
+ /* Put valid inputs and check the results */
+ for (i = 0; i < ARRAY_SIZE(valid_inputs); i++) {
+ input = valid_inputs[i];
+ expect = valid_expects[i];
+
+ rc = set_init_regions(ctx, input, strnlen(input, 256));
+ KUNIT_EXPECT_EQ(test, rc, 0);
+
+ memset(buf, 0, 256);
+ sprint_init_regions(ctx, buf, 256);
+
+ KUNIT_EXPECT_STREQ(test, (char *)buf, expect);
+ }
+ /* Put invalid inputs and check the return error code */
+ for (i = 0; i < ARRAY_SIZE(invalid_inputs); i++) {
+ input = invalid_inputs[i];
+ pr_info("input: %s\n", input);
+ rc = set_init_regions(ctx, input, strnlen(input, 256));
+ KUNIT_EXPECT_EQ(test, rc, -EINVAL);
+
+ memset(buf, 0, 256);
+ sprint_init_regions(ctx, buf, 256);
+
+ KUNIT_EXPECT_STREQ(test, (char *)buf, "");
+ }
+
+ damon_set_targets(ctx, NULL, 0);
+ damon_destroy_ctx(ctx);
+}
+
static struct kunit_case damon_test_cases[] = {
KUNIT_CASE(damon_dbgfs_test_str_to_target_ids),
KUNIT_CASE(damon_dbgfs_test_set_targets),
+ KUNIT_CASE(damon_dbgfs_test_set_init_regions),
{},
};
if (*ppos)
return ERR_PTR(-EINVAL);
- kbuf = kmalloc(count + 1, GFP_KERNEL);
+ kbuf = kmalloc(count + 1, GFP_KERNEL | __GFP_NOWARN);
if (!kbuf)
return ERR_PTR(-ENOMEM);
struct damon_ctx *ctx = file->private_data;
unsigned long s, a, r, minr, maxr;
char *kbuf;
- ssize_t ret = count;
- int err;
+ ssize_t ret;
kbuf = user_input_str(buf, count, ppos);
if (IS_ERR(kbuf))
goto unlock_out;
}
- err = damon_set_attrs(ctx, s, a, r, minr, maxr);
- if (err)
- ret = err;
+ ret = damon_set_attrs(ctx, s, a, r, minr, maxr);
+ if (!ret)
+ ret = count;
+unlock_out:
+ mutex_unlock(&ctx->kdamond_lock);
+out:
+ kfree(kbuf);
+ return ret;
+}
+
+static ssize_t sprint_schemes(struct damon_ctx *c, char *buf, ssize_t len)
+{
+ struct damos *s;
+ int written = 0;
+ int rc;
+
+ damon_for_each_scheme(s, c) {
+ rc = scnprintf(&buf[written], len - written,
+ "%lu %lu %u %u %u %u %d %lu %lu %lu %u %u %u %d %lu %lu %lu %lu %lu %lu\n",
+ s->min_sz_region, s->max_sz_region,
+ s->min_nr_accesses, s->max_nr_accesses,
+ s->min_age_region, s->max_age_region,
+ s->action,
+ s->quota.ms, s->quota.sz,
+ s->quota.reset_interval,
+ s->quota.weight_sz,
+ s->quota.weight_nr_accesses,
+ s->quota.weight_age,
+ s->wmarks.metric, s->wmarks.interval,
+ s->wmarks.high, s->wmarks.mid, s->wmarks.low,
+ s->stat_count, s->stat_sz);
+ if (!rc)
+ return -ENOMEM;
+
+ written += rc;
+ }
+ return written;
+}
+
+static ssize_t dbgfs_schemes_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ ssize_t len;
+
+ kbuf = kmalloc(count, GFP_KERNEL | __GFP_NOWARN);
+ if (!kbuf)
+ return -ENOMEM;
+
+ mutex_lock(&ctx->kdamond_lock);
+ len = sprint_schemes(ctx, kbuf, count);
+ mutex_unlock(&ctx->kdamond_lock);
+ if (len < 0)
+ goto out;
+ len = simple_read_from_buffer(buf, count, ppos, kbuf, len);
+
+out:
+ kfree(kbuf);
+ return len;
+}
+
+static void free_schemes_arr(struct damos **schemes, ssize_t nr_schemes)
+{
+ ssize_t i;
+
+ for (i = 0; i < nr_schemes; i++)
+ kfree(schemes[i]);
+ kfree(schemes);
+}
+
+static bool damos_action_valid(int action)
+{
+ switch (action) {
+ case DAMOS_WILLNEED:
+ case DAMOS_COLD:
+ case DAMOS_PAGEOUT:
+ case DAMOS_HUGEPAGE:
+ case DAMOS_NOHUGEPAGE:
+ case DAMOS_STAT:
+ return true;
+ default:
+ return false;
+ }
+}
+
+/*
+ * Converts a string into an array of struct damos pointers
+ *
+ * Returns an array of struct damos pointers that converted if the conversion
+ * success, or NULL otherwise.
+ */
+static struct damos **str_to_schemes(const char *str, ssize_t len,
+ ssize_t *nr_schemes)
+{
+ struct damos *scheme, **schemes;
+ const int max_nr_schemes = 256;
+ int pos = 0, parsed, ret;
+ unsigned long min_sz, max_sz;
+ unsigned int min_nr_a, max_nr_a, min_age, max_age;
+ unsigned int action;
+
+ schemes = kmalloc_array(max_nr_schemes, sizeof(scheme),
+ GFP_KERNEL);
+ if (!schemes)
+ return NULL;
+
+ *nr_schemes = 0;
+ while (pos < len && *nr_schemes < max_nr_schemes) {
+ struct damos_quota quota = {};
+ struct damos_watermarks wmarks;
+
+ ret = sscanf(&str[pos],
+ "%lu %lu %u %u %u %u %u %lu %lu %lu %u %u %u %u %lu %lu %lu %lu%n",
+ &min_sz, &max_sz, &min_nr_a, &max_nr_a,
+ &min_age, &max_age, &action, "a.ms,
+ "a.sz, "a.reset_interval,
+ "a.weight_sz, "a.weight_nr_accesses,
+ "a.weight_age, &wmarks.metric,
+ &wmarks.interval, &wmarks.high, &wmarks.mid,
+ &wmarks.low, &parsed);
+ if (ret != 18)
+ break;
+ if (!damos_action_valid(action)) {
+ pr_err("wrong action %d\n", action);
+ goto fail;
+ }
+
+ pos += parsed;
+ scheme = damon_new_scheme(min_sz, max_sz, min_nr_a, max_nr_a,
+ min_age, max_age, action, "a, &wmarks);
+ if (!scheme)
+ goto fail;
+
+ schemes[*nr_schemes] = scheme;
+ *nr_schemes += 1;
+ }
+ return schemes;
+fail:
+ free_schemes_arr(schemes, *nr_schemes);
+ return NULL;
+}
+
+static ssize_t dbgfs_schemes_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ struct damos **schemes;
+ ssize_t nr_schemes = 0, ret;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ schemes = str_to_schemes(kbuf, count, &nr_schemes);
+ if (!schemes) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ret = -EBUSY;
+ goto unlock_out;
+ }
+
+ ret = damon_set_schemes(ctx, schemes, nr_schemes);
+ if (!ret) {
+ ret = count;
+ nr_schemes = 0;
+ }
+
unlock_out:
mutex_unlock(&ctx->kdamond_lock);
+ free_schemes_arr(schemes, nr_schemes);
out:
kfree(kbuf);
return ret;
const char __user *buf, size_t count, loff_t *ppos)
{
struct damon_ctx *ctx = file->private_data;
+ bool id_is_pid = true;
char *kbuf, *nrs;
unsigned long *targets;
ssize_t nr_targets;
- ssize_t ret = count;
+ ssize_t ret;
int i;
- int err;
kbuf = user_input_str(buf, count, ppos);
if (IS_ERR(kbuf))
return PTR_ERR(kbuf);
nrs = kbuf;
+ if (!strncmp(kbuf, "paddr\n", count)) {
+ id_is_pid = false;
+ /* target id is meaningless here, but we set it just for fun */
+ scnprintf(kbuf, count, "42 ");
+ }
- targets = str_to_target_ids(nrs, ret, &nr_targets);
+ targets = str_to_target_ids(nrs, count, &nr_targets);
if (!targets) {
ret = -ENOMEM;
goto out;
}
- if (targetid_is_pid(ctx)) {
+ if (id_is_pid) {
for (i = 0; i < nr_targets; i++) {
targets[i] = (unsigned long)find_get_pid(
(int)targets[i]);
mutex_lock(&ctx->kdamond_lock);
if (ctx->kdamond) {
- if (targetid_is_pid(ctx))
+ if (id_is_pid)
dbgfs_put_pids(targets, nr_targets);
ret = -EBUSY;
goto unlock_out;
}
- err = damon_set_targets(ctx, targets, nr_targets);
- if (err) {
- if (targetid_is_pid(ctx))
+ /* remove targets with previously-set primitive */
+ damon_set_targets(ctx, NULL, 0);
+
+ /* Configure the context for the address space type */
+ if (id_is_pid)
+ damon_va_set_primitives(ctx);
+ else
+ damon_pa_set_primitives(ctx);
+
+ ret = damon_set_targets(ctx, targets, nr_targets);
+ if (ret) {
+ if (id_is_pid)
dbgfs_put_pids(targets, nr_targets);
- ret = err;
+ } else {
+ ret = count;
}
unlock_out:
return ret;
}
+static ssize_t sprint_init_regions(struct damon_ctx *c, char *buf, ssize_t len)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+ int written = 0;
+ int rc;
+
+ damon_for_each_target(t, c) {
+ damon_for_each_region(r, t) {
+ rc = scnprintf(&buf[written], len - written,
+ "%lu %lu %lu\n",
+ t->id, r->ar.start, r->ar.end);
+ if (!rc)
+ return -ENOMEM;
+ written += rc;
+ }
+ }
+ return written;
+}
+
+static ssize_t dbgfs_init_regions_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ ssize_t len;
+
+ kbuf = kmalloc(count, GFP_KERNEL | __GFP_NOWARN);
+ if (!kbuf)
+ return -ENOMEM;
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ mutex_unlock(&ctx->kdamond_lock);
+ len = -EBUSY;
+ goto out;
+ }
+
+ len = sprint_init_regions(ctx, kbuf, count);
+ mutex_unlock(&ctx->kdamond_lock);
+ if (len < 0)
+ goto out;
+ len = simple_read_from_buffer(buf, count, ppos, kbuf, len);
+
+out:
+ kfree(kbuf);
+ return len;
+}
+
+static int add_init_region(struct damon_ctx *c,
+ unsigned long target_id, struct damon_addr_range *ar)
+{
+ struct damon_target *t;
+ struct damon_region *r, *prev;
+ unsigned long id;
+ int rc = -EINVAL;
+
+ if (ar->start >= ar->end)
+ return -EINVAL;
+
+ damon_for_each_target(t, c) {
+ id = t->id;
+ if (targetid_is_pid(c))
+ id = (unsigned long)pid_vnr((struct pid *)id);
+ if (id == target_id) {
+ r = damon_new_region(ar->start, ar->end);
+ if (!r)
+ return -ENOMEM;
+ damon_add_region(r, t);
+ if (damon_nr_regions(t) > 1) {
+ prev = damon_prev_region(r);
+ if (prev->ar.end > r->ar.start) {
+ damon_destroy_region(r, t);
+ return -EINVAL;
+ }
+ }
+ rc = 0;
+ }
+ }
+ return rc;
+}
+
+static int set_init_regions(struct damon_ctx *c, const char *str, ssize_t len)
+{
+ struct damon_target *t;
+ struct damon_region *r, *next;
+ int pos = 0, parsed, ret;
+ unsigned long target_id;
+ struct damon_addr_range ar;
+ int err;
+
+ damon_for_each_target(t, c) {
+ damon_for_each_region_safe(r, next, t)
+ damon_destroy_region(r, t);
+ }
+
+ while (pos < len) {
+ ret = sscanf(&str[pos], "%lu %lu %lu%n",
+ &target_id, &ar.start, &ar.end, &parsed);
+ if (ret != 3)
+ break;
+ err = add_init_region(c, target_id, &ar);
+ if (err)
+ goto fail;
+ pos += parsed;
+ }
+
+ return 0;
+
+fail:
+ damon_for_each_target(t, c) {
+ damon_for_each_region_safe(r, next, t)
+ damon_destroy_region(r, t);
+ }
+ return err;
+}
+
+static ssize_t dbgfs_init_regions_write(struct file *file,
+ const char __user *buf, size_t count,
+ loff_t *ppos)
+{
+ struct damon_ctx *ctx = file->private_data;
+ char *kbuf;
+ ssize_t ret = count;
+ int err;
+
+ kbuf = user_input_str(buf, count, ppos);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
+
+ mutex_lock(&ctx->kdamond_lock);
+ if (ctx->kdamond) {
+ ret = -EBUSY;
+ goto unlock_out;
+ }
+
+ err = set_init_regions(ctx, kbuf, ret);
+ if (err)
+ ret = err;
+
+unlock_out:
+ mutex_unlock(&ctx->kdamond_lock);
+ kfree(kbuf);
+ return ret;
+}
+
static ssize_t dbgfs_kdamond_pid_read(struct file *file,
char __user *buf, size_t count, loff_t *ppos)
{
char *kbuf;
ssize_t len;
- kbuf = kmalloc(count, GFP_KERNEL);
+ kbuf = kmalloc(count, GFP_KERNEL | __GFP_NOWARN);
if (!kbuf)
return -ENOMEM;
.write = dbgfs_attrs_write,
};
+static const struct file_operations schemes_fops = {
+ .open = damon_dbgfs_open,
+ .read = dbgfs_schemes_read,
+ .write = dbgfs_schemes_write,
+};
+
static const struct file_operations target_ids_fops = {
.open = damon_dbgfs_open,
.read = dbgfs_target_ids_read,
.write = dbgfs_target_ids_write,
};
+static const struct file_operations init_regions_fops = {
+ .open = damon_dbgfs_open,
+ .read = dbgfs_init_regions_read,
+ .write = dbgfs_init_regions_write,
+};
+
static const struct file_operations kdamond_pid_fops = {
.open = damon_dbgfs_open,
.read = dbgfs_kdamond_pid_read,
static void dbgfs_fill_ctx_dir(struct dentry *dir, struct damon_ctx *ctx)
{
- const char * const file_names[] = {"attrs", "target_ids",
- "kdamond_pid"};
- const struct file_operations *fops[] = {&attrs_fops, &target_ids_fops,
- &kdamond_pid_fops};
+ const char * const file_names[] = {"attrs", "schemes", "target_ids",
+ "init_regions", "kdamond_pid"};
+ const struct file_operations *fops[] = {&attrs_fops, &schemes_fops,
+ &target_ids_fops, &init_regions_fops, &kdamond_pid_fops};
int i;
for (i = 0; i < ARRAY_SIZE(file_names); i++)
debugfs_create_file(file_names[i], 0600, dir, ctx, fops[i]);
}
-static int dbgfs_before_terminate(struct damon_ctx *ctx)
+static void dbgfs_before_terminate(struct damon_ctx *ctx)
{
struct damon_target *t, *next;
if (!targetid_is_pid(ctx))
- return 0;
+ return;
damon_for_each_target_safe(t, next, ctx) {
put_pid((struct pid *)t->id);
damon_destroy_target(t);
}
- return 0;
}
static struct damon_ctx *dbgfs_new_ctx(void)
{
char *kbuf;
char *ctx_name;
- ssize_t ret = count;
- int err;
+ ssize_t ret;
kbuf = user_input_str(buf, count, ppos);
if (IS_ERR(kbuf))
}
mutex_lock(&damon_dbgfs_lock);
- err = dbgfs_mk_context(ctx_name);
- if (err)
- ret = err;
+ ret = dbgfs_mk_context(ctx_name);
+ if (!ret)
+ ret = count;
mutex_unlock(&damon_dbgfs_lock);
out:
const char __user *buf, size_t count, loff_t *ppos)
{
char *kbuf;
- ssize_t ret = count;
- int err;
+ ssize_t ret;
char *ctx_name;
kbuf = user_input_str(buf, count, ppos);
}
mutex_lock(&damon_dbgfs_lock);
- err = dbgfs_rm_context(ctx_name);
- if (err)
- ret = err;
+ ret = dbgfs_rm_context(ctx_name);
+ if (!ret)
+ ret = count;
mutex_unlock(&damon_dbgfs_lock);
out:
static ssize_t dbgfs_monitor_on_write(struct file *file,
const char __user *buf, size_t count, loff_t *ppos)
{
- ssize_t ret = count;
+ ssize_t ret;
char *kbuf;
- int err;
kbuf = user_input_str(buf, count, ppos);
if (IS_ERR(kbuf))
return -EINVAL;
}
- if (!strncmp(kbuf, "on", count))
- err = damon_start(dbgfs_ctxs, dbgfs_nr_ctxs);
- else if (!strncmp(kbuf, "off", count))
- err = damon_stop(dbgfs_ctxs, dbgfs_nr_ctxs);
- else
- err = -EINVAL;
+ mutex_lock(&damon_dbgfs_lock);
+ if (!strncmp(kbuf, "on", count)) {
+ int i;
+
+ for (i = 0; i < dbgfs_nr_ctxs; i++) {
+ if (damon_targets_empty(dbgfs_ctxs[i])) {
+ kfree(kbuf);
+ mutex_unlock(&damon_dbgfs_lock);
+ return -EINVAL;
+ }
+ }
+ ret = damon_start(dbgfs_ctxs, dbgfs_nr_ctxs);
+ } else if (!strncmp(kbuf, "off", count)) {
+ ret = damon_stop(dbgfs_ctxs, dbgfs_nr_ctxs);
+ } else {
+ ret = -EINVAL;
+ }
+ mutex_unlock(&damon_dbgfs_lock);
- if (err)
- ret = err;
+ if (!ret)
+ ret = count;
kfree(kbuf);
return ret;
}
static int __init damon_dbgfs_init(void)
{
- int rc;
+ int rc = -ENOMEM;
+ mutex_lock(&damon_dbgfs_lock);
dbgfs_ctxs = kmalloc(sizeof(*dbgfs_ctxs), GFP_KERNEL);
if (!dbgfs_ctxs)
- return -ENOMEM;
+ goto out;
dbgfs_ctxs[0] = dbgfs_new_ctx();
if (!dbgfs_ctxs[0]) {
kfree(dbgfs_ctxs);
- return -ENOMEM;
+ goto out;
}
dbgfs_nr_ctxs = 1;
pr_err("%s: dbgfs init failed\n", __func__);
}
+out:
+ mutex_unlock(&damon_dbgfs_lock);
return rc;
}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON Primitives for The Physical Address Space
+ *
+ * Author: SeongJae Park <sj@kernel.org>
+ */
+
+#define pr_fmt(fmt) "damon-pa: " fmt
+
+#include <linux/mmu_notifier.h>
+#include <linux/page_idle.h>
+#include <linux/pagemap.h>
+#include <linux/rmap.h>
+#include <linux/swap.h>
+
+#include "../internal.h"
+#include "prmtv-common.h"
+
+static bool __damon_pa_mkold(struct page *page, struct vm_area_struct *vma,
+ unsigned long addr, void *arg)
+{
+ struct page_vma_mapped_walk pvmw = {
+ .page = page,
+ .vma = vma,
+ .address = addr,
+ };
+
+ while (page_vma_mapped_walk(&pvmw)) {
+ addr = pvmw.address;
+ if (pvmw.pte)
+ damon_ptep_mkold(pvmw.pte, vma->vm_mm, addr);
+ else
+ damon_pmdp_mkold(pvmw.pmd, vma->vm_mm, addr);
+ }
+ return true;
+}
+
+static void damon_pa_mkold(unsigned long paddr)
+{
+ struct page *page = damon_get_page(PHYS_PFN(paddr));
+ struct rmap_walk_control rwc = {
+ .rmap_one = __damon_pa_mkold,
+ .anon_lock = page_lock_anon_vma_read,
+ };
+ bool need_lock;
+
+ if (!page)
+ return;
+
+ if (!page_mapped(page) || !page_rmapping(page)) {
+ set_page_idle(page);
+ goto out;
+ }
+
+ need_lock = !PageAnon(page) || PageKsm(page);
+ if (need_lock && !trylock_page(page))
+ goto out;
+
+ rmap_walk(page, &rwc);
+
+ if (need_lock)
+ unlock_page(page);
+
+out:
+ put_page(page);
+}
+
+static void __damon_pa_prepare_access_check(struct damon_ctx *ctx,
+ struct damon_region *r)
+{
+ r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
+
+ damon_pa_mkold(r->sampling_addr);
+}
+
+void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+
+ damon_for_each_target(t, ctx) {
+ damon_for_each_region(r, t)
+ __damon_pa_prepare_access_check(ctx, r);
+ }
+}
+
+struct damon_pa_access_chk_result {
+ unsigned long page_sz;
+ bool accessed;
+};
+
+static bool __damon_pa_young(struct page *page, struct vm_area_struct *vma,
+ unsigned long addr, void *arg)
+{
+ struct damon_pa_access_chk_result *result = arg;
+ struct page_vma_mapped_walk pvmw = {
+ .page = page,
+ .vma = vma,
+ .address = addr,
+ };
+
+ result->accessed = false;
+ result->page_sz = PAGE_SIZE;
+ while (page_vma_mapped_walk(&pvmw)) {
+ addr = pvmw.address;
+ if (pvmw.pte) {
+ result->accessed = pte_young(*pvmw.pte) ||
+ !page_is_idle(page) ||
+ mmu_notifier_test_young(vma->vm_mm, addr);
+ } else {
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ result->accessed = pmd_young(*pvmw.pmd) ||
+ !page_is_idle(page) ||
+ mmu_notifier_test_young(vma->vm_mm, addr);
+ result->page_sz = ((1UL) << HPAGE_PMD_SHIFT);
+#else
+ WARN_ON_ONCE(1);
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+ }
+ if (result->accessed) {
+ page_vma_mapped_walk_done(&pvmw);
+ break;
+ }
+ }
+
+ /* If accessed, stop walking */
+ return !result->accessed;
+}
+
+static bool damon_pa_young(unsigned long paddr, unsigned long *page_sz)
+{
+ struct page *page = damon_get_page(PHYS_PFN(paddr));
+ struct damon_pa_access_chk_result result = {
+ .page_sz = PAGE_SIZE,
+ .accessed = false,
+ };
+ struct rmap_walk_control rwc = {
+ .arg = &result,
+ .rmap_one = __damon_pa_young,
+ .anon_lock = page_lock_anon_vma_read,
+ };
+ bool need_lock;
+
+ if (!page)
+ return false;
+
+ if (!page_mapped(page) || !page_rmapping(page)) {
+ if (page_is_idle(page))
+ result.accessed = false;
+ else
+ result.accessed = true;
+ put_page(page);
+ goto out;
+ }
+
+ need_lock = !PageAnon(page) || PageKsm(page);
+ if (need_lock && !trylock_page(page)) {
+ put_page(page);
+ return NULL;
+ }
+
+ rmap_walk(page, &rwc);
+
+ if (need_lock)
+ unlock_page(page);
+ put_page(page);
+
+out:
+ *page_sz = result.page_sz;
+ return result.accessed;
+}
+
+static void __damon_pa_check_access(struct damon_ctx *ctx,
+ struct damon_region *r)
+{
+ static unsigned long last_addr;
+ static unsigned long last_page_sz = PAGE_SIZE;
+ static bool last_accessed;
+
+ /* If the region is in the last checked page, reuse the result */
+ if (ALIGN_DOWN(last_addr, last_page_sz) ==
+ ALIGN_DOWN(r->sampling_addr, last_page_sz)) {
+ if (last_accessed)
+ r->nr_accesses++;
+ return;
+ }
+
+ last_accessed = damon_pa_young(r->sampling_addr, &last_page_sz);
+ if (last_accessed)
+ r->nr_accesses++;
+
+ last_addr = r->sampling_addr;
+}
+
+unsigned int damon_pa_check_accesses(struct damon_ctx *ctx)
+{
+ struct damon_target *t;
+ struct damon_region *r;
+ unsigned int max_nr_accesses = 0;
+
+ damon_for_each_target(t, ctx) {
+ damon_for_each_region(r, t) {
+ __damon_pa_check_access(ctx, r);
+ max_nr_accesses = max(r->nr_accesses, max_nr_accesses);
+ }
+ }
+
+ return max_nr_accesses;
+}
+
+bool damon_pa_target_valid(void *t)
+{
+ return true;
+}
+
+int damon_pa_apply_scheme(struct damon_ctx *ctx, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme)
+{
+ unsigned long addr;
+ LIST_HEAD(page_list);
+
+ if (scheme->action != DAMOS_PAGEOUT)
+ return -EINVAL;
+
+ for (addr = r->ar.start; addr < r->ar.end; addr += PAGE_SIZE) {
+ struct page *page = damon_get_page(PHYS_PFN(addr));
+
+ if (!page)
+ continue;
+
+ ClearPageReferenced(page);
+ test_and_clear_page_young(page);
+ if (isolate_lru_page(page)) {
+ put_page(page);
+ continue;
+ }
+ if (PageUnevictable(page)) {
+ putback_lru_page(page);
+ } else {
+ list_add(&page->lru, &page_list);
+ put_page(page);
+ }
+ }
+ reclaim_pages(&page_list);
+ cond_resched();
+ return 0;
+}
+
+int damon_pa_scheme_score(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme)
+{
+ switch (scheme->action) {
+ case DAMOS_PAGEOUT:
+ return damon_pageout_score(context, r, scheme);
+ default:
+ break;
+ }
+
+ return DAMOS_MAX_SCORE;
+}
+
+void damon_pa_set_primitives(struct damon_ctx *ctx)
+{
+ ctx->primitive.init = NULL;
+ ctx->primitive.update = NULL;
+ ctx->primitive.prepare_access_checks = damon_pa_prepare_access_checks;
+ ctx->primitive.check_accesses = damon_pa_check_accesses;
+ ctx->primitive.reset_aggregated = NULL;
+ ctx->primitive.target_valid = damon_pa_target_valid;
+ ctx->primitive.cleanup = NULL;
+ ctx->primitive.apply_scheme = damon_pa_apply_scheme;
+ ctx->primitive.get_scheme_score = damon_pa_scheme_score;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Common Primitives for Data Access Monitoring
+ *
+ * Author: SeongJae Park <sj@kernel.org>
+ */
+
+#include <linux/mmu_notifier.h>
+#include <linux/page_idle.h>
+#include <linux/pagemap.h>
+#include <linux/rmap.h>
+
+#include "prmtv-common.h"
+
+/*
+ * Get an online page for a pfn if it's in the LRU list. Otherwise, returns
+ * NULL.
+ *
+ * The body of this function is stolen from the 'page_idle_get_page()'. We
+ * steal rather than reuse it because the code is quite simple.
+ */
+struct page *damon_get_page(unsigned long pfn)
+{
+ struct page *page = pfn_to_online_page(pfn);
+
+ if (!page || !PageLRU(page) || !get_page_unless_zero(page))
+ return NULL;
+
+ if (unlikely(!PageLRU(page))) {
+ put_page(page);
+ page = NULL;
+ }
+ return page;
+}
+
+void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)
+{
+ bool referenced = false;
+ struct page *page = damon_get_page(pte_pfn(*pte));
+
+ if (!page)
+ return;
+
+ if (pte_young(*pte)) {
+ referenced = true;
+ *pte = pte_mkold(*pte);
+ }
+
+#ifdef CONFIG_MMU_NOTIFIER
+ if (mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE))
+ referenced = true;
+#endif /* CONFIG_MMU_NOTIFIER */
+
+ if (referenced)
+ set_page_young(page);
+
+ set_page_idle(page);
+ put_page(page);
+}
+
+void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+ bool referenced = false;
+ struct page *page = damon_get_page(pmd_pfn(*pmd));
+
+ if (!page)
+ return;
+
+ if (pmd_young(*pmd)) {
+ referenced = true;
+ *pmd = pmd_mkold(*pmd);
+ }
+
+#ifdef CONFIG_MMU_NOTIFIER
+ if (mmu_notifier_clear_young(mm, addr,
+ addr + ((1UL) << HPAGE_PMD_SHIFT)))
+ referenced = true;
+#endif /* CONFIG_MMU_NOTIFIER */
+
+ if (referenced)
+ set_page_young(page);
+
+ set_page_idle(page);
+ put_page(page);
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
+}
+
+#define DAMON_MAX_SUBSCORE (100)
+#define DAMON_MAX_AGE_IN_LOG (32)
+
+int damon_pageout_score(struct damon_ctx *c, struct damon_region *r,
+ struct damos *s)
+{
+ unsigned int max_nr_accesses;
+ int freq_subscore;
+ unsigned int age_in_sec;
+ int age_in_log, age_subscore;
+ unsigned int freq_weight = s->quota.weight_nr_accesses;
+ unsigned int age_weight = s->quota.weight_age;
+ int hotness;
+
+ max_nr_accesses = c->aggr_interval / c->sample_interval;
+ freq_subscore = r->nr_accesses * DAMON_MAX_SUBSCORE / max_nr_accesses;
+
+ age_in_sec = (unsigned long)r->age * c->aggr_interval / 1000000;
+ for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
+ age_in_log++, age_in_sec >>= 1)
+ ;
+
+ /* If frequency is 0, higher age means it's colder */
+ if (freq_subscore == 0)
+ age_in_log *= -1;
+
+ /*
+ * Now age_in_log is in [-DAMON_MAX_AGE_IN_LOG, DAMON_MAX_AGE_IN_LOG].
+ * Scale it to be in [0, 100] and set it as age subscore.
+ */
+ age_in_log += DAMON_MAX_AGE_IN_LOG;
+ age_subscore = age_in_log * DAMON_MAX_SUBSCORE /
+ DAMON_MAX_AGE_IN_LOG / 2;
+
+ hotness = (freq_weight * freq_subscore + age_weight * age_subscore);
+ if (freq_weight + age_weight)
+ hotness /= freq_weight + age_weight;
+ /*
+ * Transform it to fit in [0, DAMOS_MAX_SCORE]
+ */
+ hotness = hotness * DAMOS_MAX_SCORE / DAMON_MAX_SUBSCORE;
+
+ /* Return coldness of the region */
+ return DAMOS_MAX_SCORE - hotness;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Common Primitives for Data Access Monitoring
+ *
+ * Author: SeongJae Park <sj@kernel.org>
+ */
+
+#include <linux/damon.h>
+#include <linux/random.h>
+
+/* Get a random number in [l, r) */
+#define damon_rand(l, r) (l + prandom_u32_max(r - l))
+
+struct page *damon_get_page(unsigned long pfn);
+
+void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr);
+void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr);
+
+int damon_pageout_score(struct damon_ctx *c, struct damon_region *r,
+ struct damos *s);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * DAMON-based page reclamation
+ *
+ * Author: SeongJae Park <sj@kernel.org>
+ */
+
+#define pr_fmt(fmt) "damon-reclaim: " fmt
+
+#include <linux/damon.h>
+#include <linux/ioport.h>
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/workqueue.h>
+
+#ifdef MODULE_PARAM_PREFIX
+#undef MODULE_PARAM_PREFIX
+#endif
+#define MODULE_PARAM_PREFIX "damon_reclaim."
+
+/*
+ * Enable or disable DAMON_RECLAIM.
+ *
+ * You can enable DAMON_RCLAIM by setting the value of this parameter as ``Y``.
+ * Setting it as ``N`` disables DAMON_RECLAIM. Note that DAMON_RECLAIM could
+ * do no real monitoring and reclamation due to the watermarks-based activation
+ * condition. Refer to below descriptions for the watermarks parameter for
+ * this.
+ */
+static bool enabled __read_mostly;
+module_param(enabled, bool, 0600);
+
+/*
+ * Time threshold for cold memory regions identification in microseconds.
+ *
+ * If a memory region is not accessed for this or longer time, DAMON_RECLAIM
+ * identifies the region as cold, and reclaims. 120 seconds by default.
+ */
+static unsigned long min_age __read_mostly = 120000000;
+module_param(min_age, ulong, 0600);
+
+/*
+ * Limit of time for trying the reclamation in milliseconds.
+ *
+ * DAMON_RECLAIM tries to use only up to this time within a time window
+ * (quota_reset_interval_ms) for trying reclamation of cold pages. This can be
+ * used for limiting CPU consumption of DAMON_RECLAIM. If the value is zero,
+ * the limit is disabled.
+ *
+ * 10 ms by default.
+ */
+static unsigned long quota_ms __read_mostly = 10;
+module_param(quota_ms, ulong, 0600);
+
+/*
+ * Limit of size of memory for the reclamation in bytes.
+ *
+ * DAMON_RECLAIM charges amount of memory which it tried to reclaim within a
+ * time window (quota_reset_interval_ms) and makes no more than this limit is
+ * tried. This can be used for limiting consumption of CPU and IO. If this
+ * value is zero, the limit is disabled.
+ *
+ * 128 MiB by default.
+ */
+static unsigned long quota_sz __read_mostly = 128 * 1024 * 1024;
+module_param(quota_sz, ulong, 0600);
+
+/*
+ * The time/size quota charge reset interval in milliseconds.
+ *
+ * The charge reset interval for the quota of time (quota_ms) and size
+ * (quota_sz). That is, DAMON_RECLAIM does not try reclamation for more than
+ * quota_ms milliseconds or quota_sz bytes within quota_reset_interval_ms
+ * milliseconds.
+ *
+ * 1 second by default.
+ */
+static unsigned long quota_reset_interval_ms __read_mostly = 1000;
+module_param(quota_reset_interval_ms, ulong, 0600);
+
+/*
+ * The watermarks check time interval in microseconds.
+ *
+ * Minimal time to wait before checking the watermarks, when DAMON_RECLAIM is
+ * enabled but inactive due to its watermarks rule. 5 seconds by default.
+ */
+static unsigned long wmarks_interval __read_mostly = 5000000;
+module_param(wmarks_interval, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the high watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is higher than
+ * this, DAMON_RECLAIM becomes inactive, so it does nothing but periodically
+ * checks the watermarks. 500 (50%) by default.
+ */
+static unsigned long wmarks_high __read_mostly = 500;
+module_param(wmarks_high, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the middle watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is between this and
+ * the low watermark, DAMON_RECLAIM becomes active, so starts the monitoring
+ * and the reclaiming. 400 (40%) by default.
+ */
+static unsigned long wmarks_mid __read_mostly = 400;
+module_param(wmarks_mid, ulong, 0600);
+
+/*
+ * Free memory rate (per thousand) for the low watermark.
+ *
+ * If free memory of the system in bytes per thousand bytes is lower than this,
+ * DAMON_RECLAIM becomes inactive, so it does nothing but periodically checks
+ * the watermarks. In the case, the system falls back to the LRU-based page
+ * granularity reclamation logic. 200 (20%) by default.
+ */
+static unsigned long wmarks_low __read_mostly = 200;
+module_param(wmarks_low, ulong, 0600);
+
+/*
+ * Sampling interval for the monitoring in microseconds.
+ *
+ * The sampling interval of DAMON for the cold memory monitoring. Please refer
+ * to the DAMON documentation for more detail. 5 ms by default.
+ */
+static unsigned long sample_interval __read_mostly = 5000;
+module_param(sample_interval, ulong, 0600);
+
+/*
+ * Aggregation interval for the monitoring in microseconds.
+ *
+ * The aggregation interval of DAMON for the cold memory monitoring. Please
+ * refer to the DAMON documentation for more detail. 100 ms by default.
+ */
+static unsigned long aggr_interval __read_mostly = 100000;
+module_param(aggr_interval, ulong, 0600);
+
+/*
+ * Minimum number of monitoring regions.
+ *
+ * The minimal number of monitoring regions of DAMON for the cold memory
+ * monitoring. This can be used to set lower-bound of the monitoring quality.
+ * But, setting this too high could result in increased monitoring overhead.
+ * Please refer to the DAMON documentation for more detail. 10 by default.
+ */
+static unsigned long min_nr_regions __read_mostly = 10;
+module_param(min_nr_regions, ulong, 0600);
+
+/*
+ * Maximum number of monitoring regions.
+ *
+ * The maximum number of monitoring regions of DAMON for the cold memory
+ * monitoring. This can be used to set upper-bound of the monitoring overhead.
+ * However, setting this too low could result in bad monitoring quality.
+ * Please refer to the DAMON documentation for more detail. 1000 by default.
+ */
+static unsigned long max_nr_regions __read_mostly = 1000;
+module_param(max_nr_regions, ulong, 0600);
+
+/*
+ * Start of the target memory region in physical address.
+ *
+ * The start physical address of memory region that DAMON_RECLAIM will do work
+ * against. By default, biggest System RAM is used as the region.
+ */
+static unsigned long monitor_region_start __read_mostly;
+module_param(monitor_region_start, ulong, 0600);
+
+/*
+ * End of the target memory region in physical address.
+ *
+ * The end physical address of memory region that DAMON_RECLAIM will do work
+ * against. By default, biggest System RAM is used as the region.
+ */
+static unsigned long monitor_region_end __read_mostly;
+module_param(monitor_region_end, ulong, 0600);
+
+/*
+ * PID of the DAMON thread
+ *
+ * If DAMON_RECLAIM is enabled, this becomes the PID of the worker thread.
+ * Else, -1.
+ */
+static int kdamond_pid __read_mostly = -1;
+module_param(kdamond_pid, int, 0400);
+
+static struct damon_ctx *ctx;
+static struct damon_target *target;
+
+struct damon_reclaim_ram_walk_arg {
+ unsigned long start;
+ unsigned long end;
+};
+
+static int walk_system_ram(struct resource *res, void *arg)
+{
+ struct damon_reclaim_ram_walk_arg *a = arg;
+
+ if (a->end - a->start < res->end - res->start) {
+ a->start = res->start;
+ a->end = res->end;
+ }
+ return 0;
+}
+
+/*
+ * Find biggest 'System RAM' resource and store its start and end address in
+ * @start and @end, respectively. If no System RAM is found, returns false.
+ */
+static bool get_monitoring_region(unsigned long *start, unsigned long *end)
+{
+ struct damon_reclaim_ram_walk_arg arg = {};
+
+ walk_system_ram_res(0, ULONG_MAX, &arg, walk_system_ram);
+ if (arg.end <= arg.start)
+ return false;
+
+ *start = arg.start;
+ *end = arg.end;
+ return true;
+}
+
+static struct damos *damon_reclaim_new_scheme(void)
+{
+ struct damos_watermarks wmarks = {
+ .metric = DAMOS_WMARK_FREE_MEM_RATE,
+ .interval = wmarks_interval,
+ .high = wmarks_high,
+ .mid = wmarks_mid,
+ .low = wmarks_low,
+ };
+ struct damos_quota quota = {
+ /*
+ * Do not try reclamation for more than quota_ms milliseconds
+ * or quota_sz bytes within quota_reset_interval_ms.
+ */
+ .ms = quota_ms,
+ .sz = quota_sz,
+ .reset_interval = quota_reset_interval_ms,
+ /* Within the quota, page out older regions first. */
+ .weight_sz = 0,
+ .weight_nr_accesses = 0,
+ .weight_age = 1
+ };
+ struct damos *scheme = damon_new_scheme(
+ /* Find regions having PAGE_SIZE or larger size */
+ PAGE_SIZE, ULONG_MAX,
+ /* and not accessed at all */
+ 0, 0,
+ /* for min_age or more micro-seconds, and */
+ min_age / aggr_interval, UINT_MAX,
+ /* page out those, as soon as found */
+ DAMOS_PAGEOUT,
+ /* under the quota. */
+ "a,
+ /* (De)activate this according to the watermarks. */
+ &wmarks);
+
+ return scheme;
+}
+
+static int damon_reclaim_turn(bool on)
+{
+ struct damon_region *region;
+ struct damos *scheme;
+ int err;
+
+ if (!on) {
+ err = damon_stop(&ctx, 1);
+ if (!err)
+ kdamond_pid = -1;
+ return err;
+ }
+
+ err = damon_set_attrs(ctx, sample_interval, aggr_interval, 0,
+ min_nr_regions, max_nr_regions);
+ if (err)
+ return err;
+
+ if (monitor_region_start > monitor_region_end)
+ return -EINVAL;
+ if (!monitor_region_start && !monitor_region_end &&
+ !get_monitoring_region(&monitor_region_start,
+ &monitor_region_end))
+ return -EINVAL;
+ /* DAMON will free this on its own when finish monitoring */
+ region = damon_new_region(monitor_region_start, monitor_region_end);
+ if (!region)
+ return -ENOMEM;
+ damon_add_region(region, target);
+
+ /* Will be freed by 'damon_set_schemes()' below */
+ scheme = damon_reclaim_new_scheme();
+ if (!scheme) {
+ err = -ENOMEM;
+ goto free_region_out;
+ }
+ err = damon_set_schemes(ctx, &scheme, 1);
+ if (err)
+ goto free_scheme_out;
+
+ err = damon_start(&ctx, 1);
+ if (!err) {
+ kdamond_pid = ctx->kdamond->pid;
+ return 0;
+ }
+
+free_scheme_out:
+ damon_destroy_scheme(scheme);
+free_region_out:
+ damon_destroy_region(region, target);
+ return err;
+}
+
+#define ENABLE_CHECK_INTERVAL_MS 1000
+static struct delayed_work damon_reclaim_timer;
+static void damon_reclaim_timer_fn(struct work_struct *work)
+{
+ static bool last_enabled;
+ bool now_enabled;
+
+ now_enabled = enabled;
+ if (last_enabled != now_enabled) {
+ if (!damon_reclaim_turn(now_enabled))
+ last_enabled = now_enabled;
+ else
+ enabled = last_enabled;
+ }
+
+ schedule_delayed_work(&damon_reclaim_timer,
+ msecs_to_jiffies(ENABLE_CHECK_INTERVAL_MS));
+}
+static DECLARE_DELAYED_WORK(damon_reclaim_timer, damon_reclaim_timer_fn);
+
+static int __init damon_reclaim_init(void)
+{
+ ctx = damon_new_ctx();
+ if (!ctx)
+ return -ENOMEM;
+
+ damon_pa_set_primitives(ctx);
+
+ /* 4242 means nothing but fun */
+ target = damon_new_target(4242);
+ if (!target) {
+ damon_destroy_ctx(ctx);
+ return -ENOMEM;
+ }
+ damon_add_target(ctx, target);
+
+ schedule_delayed_work(&damon_reclaim_timer, 0);
+ return 0;
+}
+
+module_init(damon_reclaim_init);
* and 70-100) has totally freed and mapped to different area (30-32 and
* 65-68). The target regions which were in the old second and third big
* regions should now be removed and new target regions covering the new second
- * and third big regions should be crated.
+ * and third big regions should be created.
*/
static void damon_test_apply_three_regions4(struct kunit *test)
{
#define pr_fmt(fmt) "damon-va: " fmt
-#include <linux/damon.h>
+#include <asm-generic/mman-common.h>
+#include <linux/highmem.h>
#include <linux/hugetlb.h>
-#include <linux/mm.h>
#include <linux/mmu_notifier.h>
-#include <linux/highmem.h>
#include <linux/page_idle.h>
#include <linux/pagewalk.h>
-#include <linux/random.h>
-#include <linux/sched/mm.h>
-#include <linux/slab.h>
+
+#include "prmtv-common.h"
#ifdef CONFIG_DAMON_VADDR_KUNIT_TEST
#undef DAMON_MIN_REGION
#define DAMON_MIN_REGION 1
#endif
-/* Get a random number in [l, r) */
-#define damon_rand(l, r) (l + prandom_u32_max(r - l))
-
/*
* 't->id' should be the pointer to the relevant 'struct pid' having reference
* count. Caller must put the returned task, unless it is NULL.
struct damon_addr_range bregions[3])
{
struct damon_region *r, *next;
- unsigned int i = 0;
+ unsigned int i;
/* Remove regions which are not in the three big regions now */
damon_for_each_region_safe(r, next, t) {
}
}
-/*
- * Get an online page for a pfn if it's in the LRU list. Otherwise, returns
- * NULL.
- *
- * The body of this function is stolen from the 'page_idle_get_page()'. We
- * steal rather than reuse it because the code is quite simple.
- */
-static struct page *damon_get_page(unsigned long pfn)
-{
- struct page *page = pfn_to_online_page(pfn);
-
- if (!page || !PageLRU(page) || !get_page_unless_zero(page))
- return NULL;
-
- if (unlikely(!PageLRU(page))) {
- put_page(page);
- page = NULL;
- }
- return page;
-}
-
-static void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm,
- unsigned long addr)
-{
- bool referenced = false;
- struct page *page = damon_get_page(pte_pfn(*pte));
-
- if (!page)
- return;
-
- if (pte_young(*pte)) {
- referenced = true;
- *pte = pte_mkold(*pte);
- }
-
-#ifdef CONFIG_MMU_NOTIFIER
- if (mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE))
- referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
- if (referenced)
- set_page_young(page);
-
- set_page_idle(page);
- put_page(page);
-}
-
-static void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm,
- unsigned long addr)
-{
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- bool referenced = false;
- struct page *page = damon_get_page(pmd_pfn(*pmd));
-
- if (!page)
- return;
-
- if (pmd_young(*pmd)) {
- referenced = true;
- *pmd = pmd_mkold(*pmd);
- }
-
-#ifdef CONFIG_MMU_NOTIFIER
- if (mmu_notifier_clear_young(mm, addr,
- addr + ((1UL) << HPAGE_PMD_SHIFT)))
- referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
- if (referenced)
- set_page_young(page);
-
- set_page_idle(page);
- put_page(page);
-#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
-}
-
static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
return 0;
}
-static struct mm_walk_ops damon_mkold_ops = {
+static const struct mm_walk_ops damon_mkold_ops = {
.pmd_entry = damon_mkold_pmd_entry,
};
return 0;
}
-static struct mm_walk_ops damon_young_ops = {
+static const struct mm_walk_ops damon_young_ops = {
.pmd_entry = damon_young_pmd_entry,
};
return false;
}
+#ifndef CONFIG_ADVISE_SYSCALLS
+static int damos_madvise(struct damon_target *target, struct damon_region *r,
+ int behavior)
+{
+ return -EINVAL;
+}
+#else
+static int damos_madvise(struct damon_target *target, struct damon_region *r,
+ int behavior)
+{
+ struct mm_struct *mm;
+ int ret = -ENOMEM;
+
+ mm = damon_get_mm(target);
+ if (!mm)
+ goto out;
+
+ ret = do_madvise(mm, PAGE_ALIGN(r->ar.start),
+ PAGE_ALIGN(r->ar.end - r->ar.start), behavior);
+ mmput(mm);
+out:
+ return ret;
+}
+#endif /* CONFIG_ADVISE_SYSCALLS */
+
+int damon_va_apply_scheme(struct damon_ctx *ctx, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme)
+{
+ int madv_action;
+
+ switch (scheme->action) {
+ case DAMOS_WILLNEED:
+ madv_action = MADV_WILLNEED;
+ break;
+ case DAMOS_COLD:
+ madv_action = MADV_COLD;
+ break;
+ case DAMOS_PAGEOUT:
+ madv_action = MADV_PAGEOUT;
+ break;
+ case DAMOS_HUGEPAGE:
+ madv_action = MADV_HUGEPAGE;
+ break;
+ case DAMOS_NOHUGEPAGE:
+ madv_action = MADV_NOHUGEPAGE;
+ break;
+ case DAMOS_STAT:
+ return 0;
+ default:
+ pr_warn("Wrong action %d\n", scheme->action);
+ return -EINVAL;
+ }
+
+ return damos_madvise(t, r, madv_action);
+}
+
+int damon_va_scheme_score(struct damon_ctx *context, struct damon_target *t,
+ struct damon_region *r, struct damos *scheme)
+{
+
+ switch (scheme->action) {
+ case DAMOS_PAGEOUT:
+ return damon_pageout_score(context, r, scheme);
+ default:
+ break;
+ }
+
+ return DAMOS_MAX_SCORE;
+}
+
void damon_va_set_primitives(struct damon_ctx *ctx)
{
ctx->primitive.init = damon_va_init;
ctx->primitive.reset_aggregated = NULL;
ctx->primitive.target_valid = damon_va_target_valid;
ctx->primitive.cleanup = NULL;
+ ctx->primitive.apply_scheme = damon_va_apply_scheme;
+ ctx->primitive.get_scheme_score = damon_va_scheme_score;
}
#include "vaddr-test.h"
#include <linux/ctype.h>
#include "internal.h"
+#include <trace/events/migrate.h>
+
+/*
+ * Define EM() and EMe() so that MIGRATE_REASON from trace/events/migrate.h can
+ * be used to populate migrate_reason_names[].
+ */
+#undef EM
+#undef EMe
+#define EM(a, b) b,
+#define EMe(a, b) b
const char *migrate_reason_names[MR_TYPES] = {
- "compaction",
- "memory_failure",
- "memory_hotplug",
- "syscall_or_cpuset",
- "mempolicy_mbind",
- "numa_misplaced",
- "contig_range",
- "longterm_pin",
- "demotion",
+ MIGRATE_REASON
};
const struct trace_print_flags pageflag_names[] = {
/*
* Initialize the debugging data.
*
- * __P000 (or even __S000) will help create page table entries with
- * PROT_NONE permission as required for pxx_protnone_tests().
+ * protection_map[0] (or even protection_map[8]) will help create
+ * page table entries with PROT_NONE permission as required for
+ * pxx_protnone_tests().
*/
memset(args, 0, sizeof(*args));
args->vaddr = get_random_vaddr();
args->page_prot = vm_get_page_prot(VMFLAGS);
- args->page_prot_none = __P000;
+ args->page_prot_none = protection_map[0];
args->is_contiguous_page = false;
args->pud_pfn = ULONG_MAX;
args->pmd_pfn = ULONG_MAX;
struct address_space *mapping = page_mapping(page);
BUG_ON(!PageLocked(page));
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
__delete_from_page_cache(page, NULL);
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
page_cache_free_page(mapping, page);
}
if (!pagevec_count(pvec))
return;
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
for (i = 0; i < pagevec_count(pvec); i++) {
trace_mm_filemap_delete_from_page_cache(pvec->pages[i]);
}
page_cache_delete_batch(mapping, pvec);
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
for (i = 0; i < pagevec_count(pvec); i++)
page_cache_free_page(mapping, pvec->pages[i]);
return mapping->nrpages;
}
+static bool filemap_range_has_writeback(struct address_space *mapping,
+ loff_t start_byte, loff_t end_byte)
+{
+ XA_STATE(xas, &mapping->i_pages, start_byte >> PAGE_SHIFT);
+ pgoff_t max = end_byte >> PAGE_SHIFT;
+ struct page *page;
+
+ if (end_byte < start_byte)
+ return false;
+
+ rcu_read_lock();
+ xas_for_each(&xas, page, max) {
+ if (xas_retry(&xas, page))
+ continue;
+ if (xa_is_value(page))
+ continue;
+ if (PageDirty(page) || PageLocked(page) || PageWriteback(page))
+ break;
+ }
+ rcu_read_unlock();
+ return page != NULL;
+
+}
+
/**
* filemap_range_needs_writeback - check if range potentially needs writeback
* @mapping: address space within which to check
bool filemap_range_needs_writeback(struct address_space *mapping,
loff_t start_byte, loff_t end_byte)
{
- XA_STATE(xas, &mapping->i_pages, start_byte >> PAGE_SHIFT);
- pgoff_t max = end_byte >> PAGE_SHIFT;
- struct page *page;
-
if (!mapping_needs_writeback(mapping))
return false;
if (!mapping_tagged(mapping, PAGECACHE_TAG_DIRTY) &&
!mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK))
return false;
- if (end_byte < start_byte)
- return false;
-
- rcu_read_lock();
- xas_for_each(&xas, page, max) {
- if (xas_retry(&xas, page))
- continue;
- if (xa_is_value(page))
- continue;
- if (PageDirty(page) || PageLocked(page) || PageWriteback(page))
- break;
- }
- rcu_read_unlock();
- return page != NULL;
+ return filemap_range_has_writeback(mapping, start_byte, end_byte);
}
EXPORT_SYMBOL_GPL(filemap_range_needs_writeback);
smp_mb__after_atomic();
folio_wake(folio, PG_writeback);
+ acct_reclaim_writeback(folio);
folio_put(folio);
}
EXPORT_SYMBOL(folio_end_writeback);
if (!xa_is_value(page)) {
if (page->index < start)
goto put;
- VM_BUG_ON_PAGE(page->index != xas.xa_index, page);
if (page->index + thp_nr_pages(page) - 1 > end)
goto put;
if (!trylock_page(page))
if ((iocb->ki_flags & IOCB_WAITQ) && already_read)
iocb->ki_flags |= IOCB_NOWAIT;
+ if (unlikely(iocb->ki_pos >= i_size_read(inode)))
+ break;
+
error = filemap_get_pages(iocb, iter, &pvec);
if (error < 0)
break;
struct file *file = iocb->ki_filp;
struct address_space *mapping = file->f_mapping;
struct inode *inode = mapping->host;
- loff_t size;
- size = i_size_read(inode);
if (iocb->ki_flags & IOCB_NOWAIT) {
if (filemap_range_needs_writeback(mapping, iocb->ki_pos,
iocb->ki_pos + count - 1))
* the rest of the read. Buffered reads will not work for
* DAX files, so don't bother trying.
*/
- if (retval < 0 || !count || iocb->ki_pos >= size ||
- IS_DAX(inode))
+ if (retval < 0 || !count || IS_DAX(inode))
+ return retval;
+ if (iocb->ki_pos >= i_size_read(inode))
return retval;
}
}
if (pmd_none(*vmf->pmd) && PageTransHuge(page)) {
- vm_fault_t ret = do_set_pmd(vmf, page);
- if (!ret) {
- /* The page is mapped successfully, reference consumed. */
- unlock_page(page);
- return true;
- }
- }
-
- if (pmd_none(*vmf->pmd)) {
- vmf->ptl = pmd_lock(mm, vmf->pmd);
- if (likely(pmd_none(*vmf->pmd))) {
- mm_inc_nr_ptes(mm);
- pmd_populate(mm, vmf->pmd, vmf->prealloc_pte);
- vmf->prealloc_pte = NULL;
+ vm_fault_t ret = do_set_pmd(vmf, page);
+ if (!ret) {
+ /* The page is mapped successfully, reference consumed. */
+ unlock_page(page);
+ return true;
}
- spin_unlock(vmf->ptl);
}
+ if (pmd_none(*vmf->pmd))
+ pmd_install(mm, vmf->pmd, &vmf->prealloc_pte);
+
/* See comment in handle_pte_fault() */
if (pmd_devmap_trans_unstable(vmf->pmd)) {
unlock_page(page);
{
int nr_start = *nr;
struct dev_pagemap *pgmap = NULL;
- int ret = 1;
do {
struct page *page = pfn_to_page(pfn);
pgmap = get_dev_pagemap(pfn, pgmap);
if (unlikely(!pgmap)) {
undo_dev_pagemap(nr, nr_start, flags, pages);
- ret = 0;
break;
}
SetPageReferenced(page);
pages[*nr] = page;
if (unlikely(!try_grab_page(page, flags))) {
undo_dev_pagemap(nr, nr_start, flags, pages);
- ret = 0;
break;
}
(*nr)++;
} while (addr += PAGE_SIZE, addr != end);
put_dev_pagemap(pgmap);
- return ret;
+ return addr == end;
}
static int __gup_device_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,
unsigned this_end = min_t(unsigned, end1, PAGE_SIZE);
if (end1 > start1) {
- kaddr = kmap_atomic(page + i);
+ kaddr = kmap_local_page(page + i);
memset(kaddr + start1, 0, this_end - start1);
}
end1 -= this_end;
if (end2 > start2) {
if (!kaddr)
- kaddr = kmap_atomic(page + i);
+ kaddr = kmap_local_page(page + i);
memset(kaddr + start2, 0, this_end - start2);
}
end2 -= this_end;
}
if (kaddr) {
- kunmap_atomic(kaddr);
+ kunmap_local(kaddr);
flush_dcache_page(page + i);
}
static pte_t *__kmap_pte;
-static pte_t *kmap_get_pte(void)
+static pte_t *kmap_get_pte(unsigned long vaddr, int idx)
{
+ if (IS_ENABLED(CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY))
+ /*
+ * Set by the arch if __kmap_pte[-idx] does not produce
+ * the correct entry.
+ */
+ return virt_to_kpte(vaddr);
if (!__kmap_pte)
__kmap_pte = virt_to_kpte(__fix_to_virt(FIX_KMAP_BEGIN));
- return __kmap_pte;
+ return &__kmap_pte[-idx];
}
void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot)
{
- pte_t pteval, *kmap_pte = kmap_get_pte();
+ pte_t pteval, *kmap_pte;
unsigned long vaddr;
int idx;
preempt_disable();
idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn);
vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
- BUG_ON(!pte_none(*(kmap_pte - idx)));
+ kmap_pte = kmap_get_pte(vaddr, idx);
+ BUG_ON(!pte_none(*kmap_pte));
pteval = pfn_pte(pfn, prot);
- arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte - idx, pteval);
+ arch_kmap_local_set_pte(&init_mm, vaddr, kmap_pte, pteval);
arch_kmap_local_post_map(vaddr, pteval);
current->kmap_ctrl.pteval[kmap_local_idx()] = pteval;
preempt_enable();
void kunmap_local_indexed(void *vaddr)
{
unsigned long addr = (unsigned long) vaddr & PAGE_MASK;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int idx;
if (addr < __fix_to_virt(FIX_KMAP_END) ||
idx = arch_kmap_local_unmap_idx(kmap_local_idx(), addr);
WARN_ON_ONCE(addr != __fix_to_virt(FIX_KMAP_BEGIN + idx));
+ kmap_pte = kmap_get_pte(addr, idx);
arch_kmap_local_pre_unmap(addr);
- pte_clear(&init_mm, addr, kmap_pte - idx);
+ pte_clear(&init_mm, addr, kmap_pte);
arch_kmap_local_post_unmap(addr);
current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0);
kmap_local_idx_pop();
void __kmap_local_sched_out(void)
{
struct task_struct *tsk = current;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int i;
/* Clear kmaps */
idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
+ kmap_pte = kmap_get_pte(addr, idx);
arch_kmap_local_pre_unmap(addr);
- pte_clear(&init_mm, addr, kmap_pte - idx);
+ pte_clear(&init_mm, addr, kmap_pte);
arch_kmap_local_post_unmap(addr);
}
}
void __kmap_local_sched_in(void)
{
struct task_struct *tsk = current;
- pte_t *kmap_pte = kmap_get_pte();
+ pte_t *kmap_pte;
int i;
/* Restore kmaps */
/* See comment in __kmap_local_sched_out() */
idx = arch_kmap_local_map_idx(i, pte_pfn(pteval));
addr = __fix_to_virt(FIX_KMAP_BEGIN + idx);
- set_pte_at(&init_mm, addr, kmap_pte - idx, pteval);
+ kmap_pte = kmap_get_pte(addr, idx);
+ set_pte_at(&init_mm, addr, kmap_pte, pteval);
arch_kmap_local_post_map(addr, pteval);
}
}
#ifdef CONFIG_CMA
static struct cma *hugetlb_cma[MAX_NUMNODES];
+static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
+static bool hugetlb_cma_page(struct page *page, unsigned int order)
+{
+ return cma_pages_valid(hugetlb_cma[page_to_nid(page)], page,
+ 1 << order);
+}
+#else
+static bool hugetlb_cma_page(struct page *page, unsigned int order)
+{
+ return false;
+}
#endif
static unsigned long hugetlb_cma_size __initdata;
static unsigned long __initdata default_hstate_max_huge_pages;
static bool __initdata parsed_valid_hugepagesz = true;
static bool __initdata parsed_default_hugepagesz;
+static unsigned int default_hugepages_in_node[MAX_NUMNODES] __initdata;
/*
* Protects updates to hugepage_freelists, hugepage_activelist, nr_huge_pages,
struct file_region *org)
{
#ifdef CONFIG_CGROUP_HUGETLB
- return rg && org &&
- rg->reservation_counter == org->reservation_counter &&
+ return rg->reservation_counter == org->reservation_counter &&
rg->css == org->css;
#else
add += hugetlb_resv_map_add(resv, rg, last_accounted_offset,
t, h, h_cg, regions_needed);
- VM_BUG_ON(add < 0);
return add;
}
vma->vm_private_data = (void *)0;
}
+/*
+ * Reset and decrement one ref on hugepage private reservation.
+ * Called with mm->mmap_sem writer semaphore held.
+ * This function should be only used by move_vma() and operate on
+ * same sized vma. It should never come here with last ref on the
+ * reservation.
+ */
+void clear_vma_resv_huge_pages(struct vm_area_struct *vma)
+{
+ /*
+ * Clear the old hugetlb private page reservation.
+ * It has already been transferred to new_vma.
+ *
+ * During a mremap() operation of a hugetlb vma we call move_vma()
+ * which copies vma into new_vma and unmaps vma. After the copy
+ * operation both new_vma and vma share a reference to the resv_map
+ * struct, and at that point vma is about to be unmapped. We don't
+ * want to return the reservation to the pool at unmap of vma because
+ * the reservation still lives on in new_vma, so simply decrement the
+ * ref here and remove the resv_map reference from this vma.
+ */
+ struct resv_map *reservations = vma_resv_map(vma);
+
+ if (reservations && is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
+ resv_map_put_hugetlb_cgroup_uncharge_info(reservations);
+ kref_put(&reservations->refs, resv_map_release);
+ }
+
+ reset_vma_resv_huge_pages(vma);
+}
+
/* Returns true if the VMA has associated reserve pages */
static bool vma_has_reserves(struct vm_area_struct *vma, long chg)
{
((node = hstate_next_node_to_free(hs, mask)) || 1); \
nr_nodes--)
-#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE
-static void destroy_compound_gigantic_page(struct page *page,
- unsigned int order)
+/* used to demote non-gigantic_huge pages as well */
+static void __destroy_compound_gigantic_page(struct page *page,
+ unsigned int order, bool demote)
{
int i;
int nr_pages = 1 << order;
atomic_set(compound_pincount_ptr(page), 0);
for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) {
+ p->mapping = NULL;
clear_compound_head(p);
- set_page_refcounted(p);
+ if (!demote)
+ set_page_refcounted(p);
}
set_compound_order(page, 0);
__ClearPageHead(page);
}
+static void destroy_compound_hugetlb_page_for_demote(struct page *page,
+ unsigned int order)
+{
+ __destroy_compound_gigantic_page(page, order, true);
+}
+
+#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE
+static void destroy_compound_gigantic_page(struct page *page,
+ unsigned int order)
+{
+ __destroy_compound_gigantic_page(page, order, false);
+}
+
static void free_gigantic_page(struct page *page, unsigned int order)
{
/*
/*
* Remove hugetlb page from lists, and update dtor so that page appears
- * as just a compound page. A reference is held on the page.
+ * as just a compound page.
+ *
+ * A reference is held on the page, except in the case of demote.
*
* Must be called with hugetlb lock held.
*/
-static void remove_hugetlb_page(struct hstate *h, struct page *page,
- bool adjust_surplus)
+static void __remove_hugetlb_page(struct hstate *h, struct page *page,
+ bool adjust_surplus,
+ bool demote)
{
int nid = page_to_nid(page);
*
* This handles the case where more than one ref is held when and
* after update_and_free_page is called.
+ *
+ * In the case of demote we do not ref count the page as it will soon
+ * be turned into a page of smaller size.
*/
- set_page_refcounted(page);
+ if (!demote)
+ set_page_refcounted(page);
if (hstate_is_gigantic(h))
set_compound_page_dtor(page, NULL_COMPOUND_DTOR);
else
h->nr_huge_pages_node[nid]--;
}
+static void remove_hugetlb_page(struct hstate *h, struct page *page,
+ bool adjust_surplus)
+{
+ __remove_hugetlb_page(h, page, adjust_surplus, false);
+}
+
+static void remove_hugetlb_page_for_demote(struct hstate *h, struct page *page,
+ bool adjust_surplus)
+{
+ __remove_hugetlb_page(h, page, adjust_surplus, true);
+}
+
static void add_hugetlb_page(struct hstate *h, struct page *page,
bool adjust_surplus)
{
1 << PG_active | 1 << PG_private |
1 << PG_writeback);
}
- if (hstate_is_gigantic(h)) {
+
+ /*
+ * Non-gigantic pages demoted from CMA allocated gigantic pages
+ * need to be given back to CMA in free_gigantic_page.
+ */
+ if (hstate_is_gigantic(h) ||
+ hugetlb_cma_page(page, huge_page_order(h))) {
destroy_compound_gigantic_page(page, huge_page_order(h));
free_gigantic_page(page, huge_page_order(h));
} else {
spin_unlock_irq(&hugetlb_lock);
}
-static bool prep_compound_gigantic_page(struct page *page, unsigned int order)
+static bool __prep_compound_gigantic_page(struct page *page, unsigned int order,
+ bool demote)
{
int i, j;
int nr_pages = 1 << order;
* the set of pages can not be converted to a gigantic page.
* The caller who allocated the pages should then discard the
* pages using the appropriate free interface.
+ *
+ * In the case of demote, the ref count will be zero.
*/
- if (!page_ref_freeze(p, 1)) {
- pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n");
- goto out_error;
+ if (!demote) {
+ if (!page_ref_freeze(p, 1)) {
+ pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n");
+ goto out_error;
+ }
+ } else {
+ VM_BUG_ON_PAGE(page_count(p), p);
}
- set_page_count(p, 0);
set_compound_head(p, page);
}
atomic_set(compound_mapcount_ptr(page), -1);
return false;
}
+static bool prep_compound_gigantic_page(struct page *page, unsigned int order)
+{
+ return __prep_compound_gigantic_page(page, order, false);
+}
+
+static bool prep_compound_gigantic_page_for_demote(struct page *page,
+ unsigned int order)
+{
+ return __prep_compound_gigantic_page(page, order, true);
+}
+
/*
* PageHuge() only returns true for hugetlbfs pages, but not for normal or
* transparent huge pages. See the PageTransHuge() documentation for more
return ERR_PTR(-ENOSPC);
}
-int alloc_bootmem_huge_page(struct hstate *h)
+int alloc_bootmem_huge_page(struct hstate *h, int nid)
__attribute__ ((weak, alias("__alloc_bootmem_huge_page")));
-int __alloc_bootmem_huge_page(struct hstate *h)
+int __alloc_bootmem_huge_page(struct hstate *h, int nid)
{
- struct huge_bootmem_page *m;
+ struct huge_bootmem_page *m = NULL; /* initialize for clang */
int nr_nodes, node;
+ if (nid >= nr_online_nodes)
+ return 0;
+ /* do node specific alloc */
+ if (nid != NUMA_NO_NODE) {
+ m = memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h),
+ 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+ if (!m)
+ return 0;
+ goto found;
+ }
+ /* allocate from next node when distributing huge pages */
for_each_node_mask_to_alloc(h, nr_nodes, node, &node_states[N_MEMORY]) {
- void *addr;
-
- addr = memblock_alloc_try_nid_raw(
+ m = memblock_alloc_try_nid_raw(
huge_page_size(h), huge_page_size(h),
0, MEMBLOCK_ALLOC_ACCESSIBLE, node);
- if (addr) {
- /*
- * Use the beginning of the huge page to store the
- * huge_bootmem_page struct (until gather_bootmem
- * puts them into the mem_map).
- */
- m = addr;
- goto found;
- }
+ /*
+ * Use the beginning of the huge page to store the
+ * huge_bootmem_page struct (until gather_bootmem
+ * puts them into the mem_map).
+ */
+ if (!m)
+ return 0;
+ goto found;
}
- return 0;
found:
- BUG_ON(!IS_ALIGNED(virt_to_phys(m), huge_page_size(h)));
/* Put them into a private list first because mem_map is not up yet */
INIT_LIST_HEAD(&m->list);
list_add(&m->list, &huge_boot_pages);
cond_resched();
}
}
+static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid)
+{
+ unsigned long i;
+ char buf[32];
+
+ for (i = 0; i < h->max_huge_pages_node[nid]; ++i) {
+ if (hstate_is_gigantic(h)) {
+ if (!alloc_bootmem_huge_page(h, nid))
+ break;
+ } else {
+ struct page *page;
+ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE;
+
+ page = alloc_fresh_huge_page(h, gfp_mask, nid,
+ &node_states[N_MEMORY], NULL);
+ if (!page)
+ break;
+ put_page(page); /* free it into the hugepage allocator */
+ }
+ cond_resched();
+ }
+ if (i == h->max_huge_pages_node[nid])
+ return;
+
+ string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32);
+ pr_warn("HugeTLB: allocating %u of page size %s failed node%d. Only allocated %lu hugepages.\n",
+ h->max_huge_pages_node[nid], buf, nid, i);
+ h->max_huge_pages -= (h->max_huge_pages_node[nid] - i);
+ h->max_huge_pages_node[nid] = i;
+}
static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
{
unsigned long i;
nodemask_t *node_alloc_noretry;
+ bool node_specific_alloc = false;
+
+ /* skip gigantic hugepages allocation if hugetlb_cma enabled */
+ if (hstate_is_gigantic(h) && hugetlb_cma_size) {
+ pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n");
+ return;
+ }
+
+ /* do node specific alloc */
+ for (i = 0; i < nr_online_nodes; i++) {
+ if (h->max_huge_pages_node[i] > 0) {
+ hugetlb_hstate_alloc_pages_onenode(h, i);
+ node_specific_alloc = true;
+ }
+ }
+ if (node_specific_alloc)
+ return;
+
+ /* below will do all node balanced alloc */
if (!hstate_is_gigantic(h)) {
/*
* Bit mask controlling how hard we retry per-node allocations.
for (i = 0; i < h->max_huge_pages; ++i) {
if (hstate_is_gigantic(h)) {
- if (hugetlb_cma_size) {
- pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n");
- goto free;
- }
- if (!alloc_bootmem_huge_page(h))
+ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE))
break;
} else if (!alloc_pool_huge_page(h,
&node_states[N_MEMORY],
h->max_huge_pages, buf, i);
h->max_huge_pages = i;
}
-free:
kfree(node_alloc_noretry);
}
static void __init hugetlb_init_hstates(void)
{
- struct hstate *h;
+ struct hstate *h, *h2;
for_each_hstate(h) {
if (minimum_order > huge_page_order(h))
/* oversize hugepages were init'ed in early boot */
if (!hstate_is_gigantic(h))
hugetlb_hstate_alloc_pages(h);
+
+ /*
+ * Set demote order for each hstate. Note that
+ * h->demote_order is initially 0.
+ * - We can not demote gigantic pages if runtime freeing
+ * is not supported, so skip this.
+ * - If CMA allocation is possible, we can not demote
+ * HUGETLB_PAGE_ORDER or smaller size pages.
+ */
+ if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported())
+ continue;
+ if (hugetlb_cma_size && h->order <= HUGETLB_PAGE_ORDER)
+ continue;
+ for_each_hstate(h2) {
+ if (h2 == h)
+ continue;
+ if (h2->order < h->order &&
+ h2->order > h->demote_order)
+ h->demote_order = h2->order;
+ }
}
VM_BUG_ON(minimum_order == UINT_MAX);
}
return 0;
}
+static int demote_free_huge_page(struct hstate *h, struct page *page)
+{
+ int i, nid = page_to_nid(page);
+ struct hstate *target_hstate;
+ int rc = 0;
+
+ target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order);
+
+ remove_hugetlb_page_for_demote(h, page, false);
+ spin_unlock_irq(&hugetlb_lock);
+
+ rc = alloc_huge_page_vmemmap(h, page);
+ if (rc) {
+ /* Allocation of vmemmmap failed, we can not demote page */
+ spin_lock_irq(&hugetlb_lock);
+ set_page_refcounted(page);
+ add_hugetlb_page(h, page, false);
+ return rc;
+ }
+
+ /*
+ * Use destroy_compound_hugetlb_page_for_demote for all huge page
+ * sizes as it will not ref count pages.
+ */
+ destroy_compound_hugetlb_page_for_demote(page, huge_page_order(h));
+
+ /*
+ * Taking target hstate mutex synchronizes with set_max_huge_pages.
+ * Without the mutex, pages added to target hstate could be marked
+ * as surplus.
+ *
+ * Note that we already hold h->resize_lock. To prevent deadlock,
+ * use the convention of always taking larger size hstate mutex first.
+ */
+ mutex_lock(&target_hstate->resize_lock);
+ for (i = 0; i < pages_per_huge_page(h);
+ i += pages_per_huge_page(target_hstate)) {
+ if (hstate_is_gigantic(target_hstate))
+ prep_compound_gigantic_page_for_demote(page + i,
+ target_hstate->order);
+ else
+ prep_compound_page(page + i, target_hstate->order);
+ set_page_private(page + i, 0);
+ set_page_refcounted(page + i);
+ prep_new_huge_page(target_hstate, page + i, nid);
+ put_page(page + i);
+ }
+ mutex_unlock(&target_hstate->resize_lock);
+
+ spin_lock_irq(&hugetlb_lock);
+
+ /*
+ * Not absolutely necessary, but for consistency update max_huge_pages
+ * based on pool changes for the demoted page.
+ */
+ h->max_huge_pages--;
+ target_hstate->max_huge_pages += pages_per_huge_page(h);
+
+ return rc;
+}
+
+static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed)
+ __must_hold(&hugetlb_lock)
+{
+ int nr_nodes, node;
+ struct page *page;
+ int rc = 0;
+
+ lockdep_assert_held(&hugetlb_lock);
+
+ /* We should never get here if no demote order */
+ if (!h->demote_order) {
+ pr_warn("HugeTLB: NULL demote order passed to demote_pool_huge_page.\n");
+ return -EINVAL; /* internal error */
+ }
+
+ for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
+ if (!list_empty(&h->hugepage_freelists[node])) {
+ page = list_entry(h->hugepage_freelists[node].next,
+ struct page, lru);
+ rc = demote_free_huge_page(h, page);
+ break;
+ }
+ }
+
+ return rc;
+}
+
#define HSTATE_ATTR_RO(_name) \
static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
+#define HSTATE_ATTR_WO(_name) \
+ static struct kobj_attribute _name##_attr = __ATTR_WO(_name)
+
#define HSTATE_ATTR(_name) \
static struct kobj_attribute _name##_attr = \
__ATTR(_name, 0644, _name##_show, _name##_store)
}
HSTATE_ATTR_RO(surplus_hugepages);
+static ssize_t demote_store(struct kobject *kobj,
+ struct kobj_attribute *attr, const char *buf, size_t len)
+{
+ unsigned long nr_demote;
+ unsigned long nr_available;
+ nodemask_t nodes_allowed, *n_mask;
+ struct hstate *h;
+ int err = 0;
+ int nid;
+
+ err = kstrtoul(buf, 10, &nr_demote);
+ if (err)
+ return err;
+ h = kobj_to_hstate(kobj, &nid);
+
+ if (nid != NUMA_NO_NODE) {
+ init_nodemask_of_node(&nodes_allowed, nid);
+ n_mask = &nodes_allowed;
+ } else {
+ n_mask = &node_states[N_MEMORY];
+ }
+
+ /* Synchronize with other sysfs operations modifying huge pages */
+ mutex_lock(&h->resize_lock);
+ spin_lock_irq(&hugetlb_lock);
+
+ while (nr_demote) {
+ /*
+ * Check for available pages to demote each time thorough the
+ * loop as demote_pool_huge_page will drop hugetlb_lock.
+ */
+ if (nid != NUMA_NO_NODE)
+ nr_available = h->free_huge_pages_node[nid];
+ else
+ nr_available = h->free_huge_pages;
+ nr_available -= h->resv_huge_pages;
+ if (!nr_available)
+ break;
+
+ err = demote_pool_huge_page(h, n_mask);
+ if (err)
+ break;
+
+ nr_demote--;
+ }
+
+ spin_unlock_irq(&hugetlb_lock);
+ mutex_unlock(&h->resize_lock);
+
+ if (err)
+ return err;
+ return len;
+}
+HSTATE_ATTR_WO(demote);
+
+static ssize_t demote_size_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ int nid;
+ struct hstate *h = kobj_to_hstate(kobj, &nid);
+ unsigned long demote_size = (PAGE_SIZE << h->demote_order) / SZ_1K;
+
+ return sysfs_emit(buf, "%lukB\n", demote_size);
+}
+
+static ssize_t demote_size_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct hstate *h, *demote_hstate;
+ unsigned long demote_size;
+ unsigned int demote_order;
+ int nid;
+
+ demote_size = (unsigned long)memparse(buf, NULL);
+
+ demote_hstate = size_to_hstate(demote_size);
+ if (!demote_hstate)
+ return -EINVAL;
+ demote_order = demote_hstate->order;
+ if (demote_order < HUGETLB_PAGE_ORDER)
+ return -EINVAL;
+
+ /* demote order must be smaller than hstate order */
+ h = kobj_to_hstate(kobj, &nid);
+ if (demote_order >= h->order)
+ return -EINVAL;
+
+ /* resize_lock synchronizes access to demote size and writes */
+ mutex_lock(&h->resize_lock);
+ h->demote_order = demote_order;
+ mutex_unlock(&h->resize_lock);
+
+ return count;
+}
+HSTATE_ATTR(demote_size);
+
static struct attribute *hstate_attrs[] = {
&nr_hugepages_attr.attr,
&nr_overcommit_hugepages_attr.attr,
.attrs = hstate_attrs,
};
+static struct attribute *hstate_demote_attrs[] = {
+ &demote_size_attr.attr,
+ &demote_attr.attr,
+ NULL,
+};
+
+static const struct attribute_group hstate_demote_attr_group = {
+ .attrs = hstate_demote_attrs,
+};
+
static int hugetlb_sysfs_add_hstate(struct hstate *h, struct kobject *parent,
struct kobject **hstate_kobjs,
const struct attribute_group *hstate_attr_group)
hstate_kobjs[hi] = NULL;
}
+ if (h->demote_order) {
+ if (sysfs_create_group(hstate_kobjs[hi],
+ &hstate_demote_attr_group))
+ pr_warn("HugeTLB unable to create demote interfaces for %s\n", h->name);
+ }
+
return retval;
}
}
default_hstate.max_huge_pages =
default_hstate_max_huge_pages;
+
+ for (i = 0; i < nr_online_nodes; i++)
+ default_hstate.max_huge_pages_node[i] =
+ default_hugepages_in_node[i];
}
}
parsed_hstate = h;
}
+bool __init __weak hugetlb_node_alloc_supported(void)
+{
+ return true;
+}
/*
* hugepages command line processing
* hugepages normally follows a valid hugepagsz or default_hugepagsz
{
unsigned long *mhp;
static unsigned long *last_mhp;
+ int node = NUMA_NO_NODE;
+ int count;
+ unsigned long tmp;
+ char *p = s;
if (!parsed_valid_hugepagesz) {
pr_warn("HugeTLB: hugepages=%s does not follow a valid hugepagesz, ignoring\n", s);
return 0;
}
- if (sscanf(s, "%lu", mhp) <= 0)
- *mhp = 0;
+ while (*p) {
+ count = 0;
+ if (sscanf(p, "%lu%n", &tmp, &count) != 1)
+ goto invalid;
+ /* Parameter is node format */
+ if (p[count] == ':') {
+ if (!hugetlb_node_alloc_supported()) {
+ pr_warn("HugeTLB: architecture can't support node specific alloc, ignoring!\n");
+ return 0;
+ }
+ node = tmp;
+ p += count + 1;
+ if (node < 0 || node >= nr_online_nodes)
+ goto invalid;
+ /* Parse hugepages */
+ if (sscanf(p, "%lu%n", &tmp, &count) != 1)
+ goto invalid;
+ if (!hugetlb_max_hstate)
+ default_hugepages_in_node[node] = tmp;
+ else
+ parsed_hstate->max_huge_pages_node[node] = tmp;
+ *mhp += tmp;
+ /* Go to parse next node*/
+ if (p[count] == ',')
+ p += count + 1;
+ else
+ break;
+ } else {
+ if (p != s)
+ goto invalid;
+ *mhp = tmp;
+ break;
+ }
+ }
/*
* Global state is always initialized later in hugetlb_init.
last_mhp = mhp;
return 1;
+
+invalid:
+ pr_warn("HugeTLB: Invalid hugepages parameter %s\n", p);
+ return 0;
}
__setup("hugepages=", hugepages_setup);
static int __init default_hugepagesz_setup(char *s)
{
unsigned long size;
+ int i;
parsed_valid_hugepagesz = false;
if (parsed_default_hugepagesz) {
*/
if (default_hstate_max_huge_pages) {
default_hstate.max_huge_pages = default_hstate_max_huge_pages;
+ for (i = 0; i < nr_online_nodes; i++)
+ default_hstate.max_huge_pages_node[i] =
+ default_hugepages_in_node[i];
if (hstate_is_gigantic(&default_hstate))
hugetlb_hstate_alloc_pages(&default_hstate);
default_hstate_max_huge_pages = 0;
return ret;
}
-void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
- unsigned long start, unsigned long end,
- struct page *ref_page)
+static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
+ unsigned long new_addr, pte_t *src_pte)
+{
+ struct hstate *h = hstate_vma(vma);
+ struct mm_struct *mm = vma->vm_mm;
+ pte_t *dst_pte, pte;
+ spinlock_t *src_ptl, *dst_ptl;
+
+ dst_pte = huge_pte_offset(mm, new_addr, huge_page_size(h));
+ dst_ptl = huge_pte_lock(h, mm, dst_pte);
+ src_ptl = huge_pte_lockptr(h, mm, src_pte);
+
+ /*
+ * We don't have to worry about the ordering of src and dst ptlocks
+ * because exclusive mmap_sem (or the i_mmap_lock) prevents deadlock.
+ */
+ if (src_ptl != dst_ptl)
+ spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
+
+ pte = huge_ptep_get_and_clear(mm, old_addr, src_pte);
+ set_huge_pte_at(mm, new_addr, dst_pte, pte);
+
+ if (src_ptl != dst_ptl)
+ spin_unlock(src_ptl);
+ spin_unlock(dst_ptl);
+}
+
+int move_hugetlb_page_tables(struct vm_area_struct *vma,
+ struct vm_area_struct *new_vma,
+ unsigned long old_addr, unsigned long new_addr,
+ unsigned long len)
+{
+ struct hstate *h = hstate_vma(vma);
+ struct address_space *mapping = vma->vm_file->f_mapping;
+ unsigned long sz = huge_page_size(h);
+ struct mm_struct *mm = vma->vm_mm;
+ unsigned long old_end = old_addr + len;
+ unsigned long old_addr_copy;
+ pte_t *src_pte, *dst_pte;
+ struct mmu_notifier_range range;
+
+ mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, old_addr,
+ old_end);
+ adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
+ mmu_notifier_invalidate_range_start(&range);
+ /* Prevent race with file truncation */
+ i_mmap_lock_write(mapping);
+ for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
+ src_pte = huge_pte_offset(mm, old_addr, sz);
+ if (!src_pte)
+ continue;
+ if (huge_pte_none(huge_ptep_get(src_pte)))
+ continue;
+
+ /* old_addr arg to huge_pmd_unshare() is a pointer and so the
+ * arg may be modified. Pass a copy instead to preserve the
+ * value in old_addr.
+ */
+ old_addr_copy = old_addr;
+
+ if (huge_pmd_unshare(mm, vma, &old_addr_copy, src_pte))
+ continue;
+
+ dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz);
+ if (!dst_pte)
+ break;
+
+ move_huge_pte(vma, old_addr, new_addr, src_pte);
+ }
+ flush_tlb_range(vma, old_end - len, old_end);
+ mmu_notifier_invalidate_range_end(&range);
+ i_mmap_unlock_write(mapping);
+
+ return len + old_addr - old_end;
+}
+
+static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
+ unsigned long start, unsigned long end,
+ struct page *ref_page)
{
struct mm_struct *mm = vma->vm_mm;
unsigned long address;
struct hstate *h = hstate_vma(vma);
unsigned long sz = huge_page_size(h);
struct mmu_notifier_range range;
+ bool force_flush = false;
WARN_ON(!is_vm_hugetlb_page(vma));
BUG_ON(start & ~huge_page_mask(h));
ptl = huge_pte_lock(h, mm, ptep);
if (huge_pmd_unshare(mm, vma, &address, ptep)) {
spin_unlock(ptl);
- /*
- * We just unmapped a page of PMDs by clearing a PUD.
- * The caller's TLB flush range should cover this area.
- */
+ tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE);
+ force_flush = true;
continue;
}
}
mmu_notifier_invalidate_range_end(&range);
tlb_end_vma(tlb, vma);
+
+ /*
+ * If we unshared PMDs, the TLB flush was not recorded in mmu_gather. We
+ * could defer the flush until now, since by holding i_mmap_rwsem we
+ * guaranteed that the last refernece would not be dropped. But we must
+ * do the flushing before we return, as otherwise i_mmap_rwsem will be
+ * dropped and the last reference to the shared PMDs page might be
+ * dropped as well.
+ *
+ * In theory we could defer the freeing of the PMD pages as well, but
+ * huge_pmd_unshare() relies on the exact page_count for the PMD page to
+ * detect sharing, so we cannot defer the release of the page either.
+ * Instead, do flush now.
+ */
+ if (force_flush)
+ tlb_flush_mmu_tlbonly(tlb);
}
void __unmap_hugepage_range_final(struct mmu_gather *tlb,
/*
* Hugetlb_cow() should be called with page lock of the original hugepage held.
- * Called with hugetlb_instantiation_mutex held and pte_page locked so we
+ * Called with hugetlb_fault_mutex_table held and pte_page locked so we
* cannot race with other handlers or page migration.
* Keep the pte_same checks anyway to make transition from the mutex easier.
*/
int ret = -ENOMEM;
struct page *page;
int writable;
- bool new_pagecache_page = false;
+ bool page_in_pagecache = false;
if (is_continue) {
ret = -EFAULT;
page = find_lock_page(mapping, idx);
if (!page)
goto out;
+ page_in_pagecache = true;
} else if (!*pagep) {
/* If a page already exists, then it's UFFDIO_COPY for
* a non-missing case. Return -EEXIST.
ret = huge_add_to_page_cache(page, mapping, idx);
if (ret)
goto out_release_nounlock;
- new_pagecache_page = true;
+ page_in_pagecache = true;
}
ptl = huge_pte_lockptr(h, dst_mm, dst_pte);
if (vm_shared || is_continue)
unlock_page(page);
out_release_nounlock:
- if (!new_pagecache_page)
+ if (!page_in_pagecache)
restore_reserve_on_error(h, dst_vma, dst_addr, page);
put_page(page);
goto out;
* sharing is possible. For hugetlbfs, this prevents removal of any page
* table entries associated with the address space. This is important as we
* are setting up sharing based on existing page table entries (mappings).
- *
- * NOTE: This routine is only called from huge_pte_alloc. Some callers of
- * huge_pte_alloc know that sharing is not possible and do not take
- * i_mmap_rwsem as a performance optimization. This is handled by the
- * if !vma_shareable check at the beginning of the routine. i_mmap_rwsem is
- * only required for subsequent processing.
*/
pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
unsigned long addr, pud_t *pud)
static int __init cmdline_parse_hugetlb_cma(char *p)
{
- hugetlb_cma_size = memparse(p, &p);
+ int nid, count = 0;
+ unsigned long tmp;
+ char *s = p;
+
+ while (*s) {
+ if (sscanf(s, "%lu%n", &tmp, &count) != 1)
+ break;
+
+ if (s[count] == ':') {
+ nid = tmp;
+ if (nid < 0 || nid >= MAX_NUMNODES)
+ break;
+
+ s += count + 1;
+ tmp = memparse(s, &s);
+ hugetlb_cma_size_in_node[nid] = tmp;
+ hugetlb_cma_size += tmp;
+
+ /*
+ * Skip the separator if have one, otherwise
+ * break the parsing.
+ */
+ if (*s == ',')
+ s++;
+ else
+ break;
+ } else {
+ hugetlb_cma_size = memparse(p, &p);
+ break;
+ }
+ }
+
return 0;
}
void __init hugetlb_cma_reserve(int order)
{
unsigned long size, reserved, per_node;
+ bool node_specific_cma_alloc = false;
int nid;
cma_reserve_called = true;
if (!hugetlb_cma_size)
return;
+ for (nid = 0; nid < MAX_NUMNODES; nid++) {
+ if (hugetlb_cma_size_in_node[nid] == 0)
+ continue;
+
+ if (!node_state(nid, N_ONLINE)) {
+ pr_warn("hugetlb_cma: invalid node %d specified\n", nid);
+ hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
+ hugetlb_cma_size_in_node[nid] = 0;
+ continue;
+ }
+
+ if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) {
+ pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n",
+ nid, (PAGE_SIZE << order) / SZ_1M);
+ hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
+ hugetlb_cma_size_in_node[nid] = 0;
+ } else {
+ node_specific_cma_alloc = true;
+ }
+ }
+
+ /* Validate the CMA size again in case some invalid nodes specified. */
+ if (!hugetlb_cma_size)
+ return;
+
if (hugetlb_cma_size < (PAGE_SIZE << order)) {
pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n",
(PAGE_SIZE << order) / SZ_1M);
+ hugetlb_cma_size = 0;
return;
}
- /*
- * If 3 GB area is requested on a machine with 4 numa nodes,
- * let's allocate 1 GB on first three nodes and ignore the last one.
- */
- per_node = DIV_ROUND_UP(hugetlb_cma_size, nr_online_nodes);
- pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
- hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
+ if (!node_specific_cma_alloc) {
+ /*
+ * If 3 GB area is requested on a machine with 4 numa nodes,
+ * let's allocate 1 GB on first three nodes and ignore the last one.
+ */
+ per_node = DIV_ROUND_UP(hugetlb_cma_size, nr_online_nodes);
+ pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
+ hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
+ }
reserved = 0;
for_each_node_state(nid, N_ONLINE) {
int res;
char name[CMA_MAX_NAME];
- size = min(per_node, hugetlb_cma_size - reserved);
+ if (node_specific_cma_alloc) {
+ if (hugetlb_cma_size_in_node[nid] == 0)
+ continue;
+
+ size = hugetlb_cma_size_in_node[nid];
+ } else {
+ size = min(per_node, hugetlb_cma_size - reserved);
+ }
+
size = round_up(size, PAGE_SIZE << order);
snprintf(name, sizeof(name), "hugetlb%d", nid);
- res = cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << order,
+ /*
+ * Note that 'order per bit' is based on smallest size that
+ * may be returned to CMA allocator in the case of
+ * huge page demotion.
+ */
+ res = cma_declare_contiguous_nid(0, size, 0,
+ PAGE_SIZE << HUGETLB_PAGE_ORDER,
0, false, name,
&hugetlb_cma[nid], nid);
if (res) {
if (reserved >= hugetlb_cma_size)
break;
}
+
+ if (!reserved)
+ /*
+ * hugetlb_cma_size is used to determine if allocations from
+ * cma are possible. Set to zero if no cma regions are set up.
+ */
+ hugetlb_cma_size = 0;
}
void __init hugetlb_cma_check(void)
#define MEMFILE_IDX(val) (((val) >> 16) & 0xffff)
#define MEMFILE_ATTR(val) ((val) & 0xffff)
-#define hugetlb_cgroup_from_counter(counter, idx) \
- container_of(counter, struct hugetlb_cgroup, hugepage[idx])
-
static struct hugetlb_cgroup *root_h_cgroup __read_mostly;
static inline struct page_counter *
return (void *)(mapping & ~PAGE_MAPPING_FLAGS);
}
+void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
+ int nr_throttled);
+static inline void acct_reclaim_writeback(struct folio *folio)
+{
+ pg_data_t *pgdat = folio_pgdat(folio);
+ int nr_throttled = atomic_read(&pgdat->nr_writeback_throttled);
+
+ if (nr_throttled)
+ __acct_reclaim_writeback(pgdat, folio, nr_throttled);
+}
+
+static inline void wake_throttle_isolated(pg_data_t *pgdat)
+{
+ wait_queue_head_t *wqh;
+
+ wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_ISOLATED];
+ if (waitqueue_active(wqh))
+ wake_up(wqh);
+}
+
vm_fault_t do_swap_page(struct vm_fault *vmf);
void folio_rotate_reclaimable(struct folio *folio);
bool __folio_end_writeback(struct folio *folio);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
+void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
static inline bool can_madv_lru_vma(struct vm_area_struct *vma)
{
*/
extern int isolate_lru_page(struct page *page);
extern void putback_lru_page(struct page *page);
+extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason);
/*
* in mm/rmap.c:
#include "kasan.h"
#include "../slab.h"
-depot_stack_handle_t kasan_save_stack(gfp_t flags)
+depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc)
{
unsigned long entries[KASAN_STACK_DEPTH];
unsigned int nr_entries;
nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0);
nr_entries = filter_irq_stacks(entries, nr_entries);
- return stack_depot_save(entries, nr_entries, flags);
+ return __stack_depot_save(entries, nr_entries, flags, can_alloc);
}
void kasan_set_track(struct kasan_track *track, gfp_t flags)
{
track->pid = current->pid;
- track->stack = kasan_save_stack(flags);
+ track->stack = kasan_save_stack(flags, true);
}
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
/* For caches that either have a constructor or SLAB_TYPESAFE_BY_RCU: */
#ifdef CONFIG_SLAB
/* For SLAB assign tags based on the object index in the freelist. */
- return (u8)obj_to_index(cache, virt_to_page(object), (void *)object);
+ return (u8)obj_to_index(cache, virt_to_head_page(object), (void *)object);
#else
/*
* For SLUB assign a random tag during slab creation, otherwise reuse
DEFINE_ASAN_SET_SHADOW(f5);
DEFINE_ASAN_SET_SHADOW(f8);
-void kasan_record_aux_stack(void *addr)
+static void __kasan_record_aux_stack(void *addr, bool can_alloc)
{
struct page *page = kasan_addr_to_page(addr);
struct kmem_cache *cache;
return;
alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0];
- alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT);
+ alloc_meta->aux_stack[0] = kasan_save_stack(GFP_NOWAIT, can_alloc);
+}
+
+void kasan_record_aux_stack(void *addr)
+{
+ return __kasan_record_aux_stack(addr, true);
+}
+
+void kasan_record_aux_stack_noalloc(void *addr)
+{
+ return __kasan_record_aux_stack(addr, false);
}
void kasan_set_free_info(struct kmem_cache *cache,
}
early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
+static inline const char *kasan_mode_info(void)
+{
+ if (kasan_mode == KASAN_MODE_ASYNC)
+ return "async";
+ else if (kasan_mode == KASAN_MODE_ASYMM)
+ return "asymm";
+ else
+ return "sync";
+}
+
/* kasan_init_hw_tags_cpu() is called for each CPU. */
void kasan_init_hw_tags_cpu(void)
{
break;
}
- pr_info("KernelAddressSanitizer initialized\n");
+ pr_info("KernelAddressSanitizer initialized (hw-tags, mode=%s, stacktrace=%s)\n",
+ kasan_mode_info(),
+ kasan_stack_collection_enabled() ? "on" : "off");
}
void kasan_alloc_pages(struct page *page, unsigned int order, gfp_t flags)
struct page *kasan_addr_to_page(const void *addr);
-depot_stack_handle_t kasan_save_stack(gfp_t flags);
+depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc);
void kasan_set_track(struct kasan_track *track, gfp_t flags);
void kasan_set_free_info(struct kmem_cache *cache, void *object, u8 tag);
struct kasan_track *kasan_get_free_track(struct kmem_cache *cache,
kasan_enable_current();
}
-static void print_stack(depot_stack_handle_t stack)
-{
- unsigned long *entries;
- unsigned int nr_entries;
-
- nr_entries = stack_depot_fetch(stack, &entries);
- stack_trace_print(entries, nr_entries, 0);
-}
-
static void print_track(struct kasan_track *track, const char *prefix)
{
pr_err("%s by task %u:\n", prefix, track->pid);
if (track->stack) {
- print_stack(track->stack);
+ stack_depot_print(track->stack);
} else {
pr_err("(stack is not available)\n");
}
return;
if (alloc_meta->aux_stack[0]) {
pr_err("Last potentially related work creation:\n");
- print_stack(alloc_meta->aux_stack[0]);
+ stack_depot_print(alloc_meta->aux_stack[0]);
pr_err("\n");
}
if (alloc_meta->aux_stack[1]) {
pr_err("Second to last potentially related work creation:\n");
- print_stack(alloc_meta->aux_stack[1]);
+ stack_depot_print(alloc_meta->aux_stack[1]);
pr_err("\n");
}
#endif
static inline bool kernel_or_module_addr(const void *addr)
{
- if (addr >= (void *)_stext && addr < (void *)_end)
+ if (is_kernel((unsigned long)addr))
return true;
if (is_module_address((unsigned long)addr))
return true;
#ifdef CONFIG_KASAN_VMALLOC
+void __init __weak kasan_populate_early_vm_area_shadow(void *start,
+ unsigned long size)
+{
+}
+
static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
void *unused)
{
for_each_possible_cpu(cpu)
per_cpu(prng_state, cpu) = (u32)get_cycles();
- pr_info("KernelAddressSanitizer initialized\n");
+ pr_info("KernelAddressSanitizer initialized (sw-tags)\n");
}
/*
#include <linux/atomic.h>
#include <linux/bug.h>
#include <linux/debugfs.h>
+#include <linux/hash.h>
#include <linux/irq_work.h>
+#include <linux/jhash.h>
#include <linux/kcsan-checks.h>
#include <linux/kfence.h>
#include <linux/kmemleak.h>
#include <linux/list.h>
#include <linux/lockdep.h>
+#include <linux/log2.h>
#include <linux/memblock.h>
#include <linux/moduleparam.h>
#include <linux/random.h>
};
module_param_cb(sample_interval, &sample_interval_param_ops, &kfence_sample_interval, 0600);
+/* Pool usage% threshold when currently covered allocations are skipped. */
+static unsigned long kfence_skip_covered_thresh __read_mostly = 75;
+module_param_named(skip_covered_thresh, kfence_skip_covered_thresh, ulong, 0644);
+
/* The pool of pages used for guard pages and objects. */
char *__kfence_pool __ro_after_init;
EXPORT_SYMBOL(__kfence_pool); /* Export for test modules. */
static struct list_head kfence_freelist = LIST_HEAD_INIT(kfence_freelist);
static DEFINE_RAW_SPINLOCK(kfence_freelist_lock); /* Lock protecting freelist. */
-#ifdef CONFIG_KFENCE_STATIC_KEYS
-/* The static key to set up a KFENCE allocation. */
+/*
+ * The static key to set up a KFENCE allocation; or if static keys are not used
+ * to gate allocations, to avoid a load and compare if KFENCE is disabled.
+ */
DEFINE_STATIC_KEY_FALSE(kfence_allocation_key);
-#endif
/* Gates the allocation, ensuring only one succeeds in a given period. */
atomic_t kfence_allocation_gate = ATOMIC_INIT(1);
+/*
+ * A Counting Bloom filter of allocation coverage: limits currently covered
+ * allocations of the same source filling up the pool.
+ *
+ * Assuming a range of 15%-85% unique allocations in the pool at any point in
+ * time, the below parameters provide a probablity of 0.02-0.33 for false
+ * positive hits respectively:
+ *
+ * P(alloc_traces) = (1 - e^(-HNUM * (alloc_traces / SIZE)) ^ HNUM
+ */
+#define ALLOC_COVERED_HNUM 2
+#define ALLOC_COVERED_ORDER (const_ilog2(CONFIG_KFENCE_NUM_OBJECTS) + 2)
+#define ALLOC_COVERED_SIZE (1 << ALLOC_COVERED_ORDER)
+#define ALLOC_COVERED_HNEXT(h) hash_32(h, ALLOC_COVERED_ORDER)
+#define ALLOC_COVERED_MASK (ALLOC_COVERED_SIZE - 1)
+static atomic_t alloc_covered[ALLOC_COVERED_SIZE];
+
+/* Stack depth used to determine uniqueness of an allocation. */
+#define UNIQUE_ALLOC_STACK_DEPTH ((size_t)8)
+
+/*
+ * Randomness for stack hashes, making the same collisions across reboots and
+ * different machines less likely.
+ */
+static u32 stack_hash_seed __ro_after_init;
+
/* Statistics counters for debugfs. */
enum kfence_counter_id {
KFENCE_COUNTER_ALLOCATED,
KFENCE_COUNTER_FREES,
KFENCE_COUNTER_ZOMBIES,
KFENCE_COUNTER_BUGS,
+ KFENCE_COUNTER_SKIP_INCOMPAT,
+ KFENCE_COUNTER_SKIP_CAPACITY,
+ KFENCE_COUNTER_SKIP_COVERED,
KFENCE_COUNTER_COUNT,
};
static atomic_long_t counters[KFENCE_COUNTER_COUNT];
[KFENCE_COUNTER_FREES] = "total frees",
[KFENCE_COUNTER_ZOMBIES] = "zombie allocations",
[KFENCE_COUNTER_BUGS] = "total bugs",
+ [KFENCE_COUNTER_SKIP_INCOMPAT] = "skipped allocations (incompatible)",
+ [KFENCE_COUNTER_SKIP_CAPACITY] = "skipped allocations (capacity)",
+ [KFENCE_COUNTER_SKIP_COVERED] = "skipped allocations (covered)",
};
static_assert(ARRAY_SIZE(counter_names) == KFENCE_COUNTER_COUNT);
/* === Internals ============================================================ */
+static inline bool should_skip_covered(void)
+{
+ unsigned long thresh = (CONFIG_KFENCE_NUM_OBJECTS * kfence_skip_covered_thresh) / 100;
+
+ return atomic_long_read(&counters[KFENCE_COUNTER_ALLOCATED]) > thresh;
+}
+
+static u32 get_alloc_stack_hash(unsigned long *stack_entries, size_t num_entries)
+{
+ num_entries = min(num_entries, UNIQUE_ALLOC_STACK_DEPTH);
+ num_entries = filter_irq_stacks(stack_entries, num_entries);
+ return jhash(stack_entries, num_entries * sizeof(stack_entries[0]), stack_hash_seed);
+}
+
+/*
+ * Adds (or subtracts) count @val for allocation stack trace hash
+ * @alloc_stack_hash from Counting Bloom filter.
+ */
+static void alloc_covered_add(u32 alloc_stack_hash, int val)
+{
+ int i;
+
+ for (i = 0; i < ALLOC_COVERED_HNUM; i++) {
+ atomic_add(val, &alloc_covered[alloc_stack_hash & ALLOC_COVERED_MASK]);
+ alloc_stack_hash = ALLOC_COVERED_HNEXT(alloc_stack_hash);
+ }
+}
+
+/*
+ * Returns true if the allocation stack trace hash @alloc_stack_hash is
+ * currently contained (non-zero count) in Counting Bloom filter.
+ */
+static bool alloc_covered_contains(u32 alloc_stack_hash)
+{
+ int i;
+
+ for (i = 0; i < ALLOC_COVERED_HNUM; i++) {
+ if (!atomic_read(&alloc_covered[alloc_stack_hash & ALLOC_COVERED_MASK]))
+ return false;
+ alloc_stack_hash = ALLOC_COVERED_HNEXT(alloc_stack_hash);
+ }
+
+ return true;
+}
+
static bool kfence_protect(unsigned long addr)
{
return !KFENCE_WARN_ON(!kfence_protect_page(ALIGN_DOWN(addr, PAGE_SIZE), true));
* Update the object's metadata state, including updating the alloc/free stacks
* depending on the state transition.
*/
-static noinline void metadata_update_state(struct kfence_metadata *meta,
- enum kfence_object_state next)
+static noinline void
+metadata_update_state(struct kfence_metadata *meta, enum kfence_object_state next,
+ unsigned long *stack_entries, size_t num_stack_entries)
{
struct kfence_track *track =
next == KFENCE_OBJECT_FREED ? &meta->free_track : &meta->alloc_track;
lockdep_assert_held(&meta->lock);
- /*
- * Skip over 1 (this) functions; noinline ensures we do not accidentally
- * skip over the caller by never inlining.
- */
- track->num_stack_entries = stack_trace_save(track->stack_entries, KFENCE_STACK_DEPTH, 1);
+ if (stack_entries) {
+ memcpy(track->stack_entries, stack_entries,
+ num_stack_entries * sizeof(stack_entries[0]));
+ } else {
+ /*
+ * Skip over 1 (this) functions; noinline ensures we do not
+ * accidentally skip over the caller by never inlining.
+ */
+ num_stack_entries = stack_trace_save(track->stack_entries, KFENCE_STACK_DEPTH, 1);
+ }
+ track->num_stack_entries = num_stack_entries;
track->pid = task_pid_nr(current);
track->cpu = raw_smp_processor_id();
track->ts_nsec = local_clock(); /* Same source as printk timestamps. */
/* Check canary byte at @addr. */
static inline bool check_canary_byte(u8 *addr)
{
+ struct kfence_metadata *meta;
+ unsigned long flags;
+
if (likely(*addr == KFENCE_CANARY_PATTERN(addr)))
return true;
atomic_long_inc(&counters[KFENCE_COUNTER_BUGS]);
- kfence_report_error((unsigned long)addr, false, NULL, addr_to_metadata((unsigned long)addr),
- KFENCE_ERROR_CORRUPTION);
+
+ meta = addr_to_metadata((unsigned long)addr);
+ raw_spin_lock_irqsave(&meta->lock, flags);
+ kfence_report_error((unsigned long)addr, false, NULL, meta, KFENCE_ERROR_CORRUPTION);
+ raw_spin_unlock_irqrestore(&meta->lock, flags);
+
return false;
}
const unsigned long pageaddr = ALIGN_DOWN(meta->addr, PAGE_SIZE);
unsigned long addr;
- lockdep_assert_held(&meta->lock);
-
/*
* We'll iterate over each canary byte per-side until fn() returns
* false. However, we'll still iterate over the canary bytes to the
}
}
-static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp)
+static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size, gfp_t gfp,
+ unsigned long *stack_entries, size_t num_stack_entries,
+ u32 alloc_stack_hash)
{
struct kfence_metadata *meta = NULL;
unsigned long flags;
list_del_init(&meta->list);
}
raw_spin_unlock_irqrestore(&kfence_freelist_lock, flags);
- if (!meta)
+ if (!meta) {
+ atomic_long_inc(&counters[KFENCE_COUNTER_SKIP_CAPACITY]);
return NULL;
+ }
if (unlikely(!raw_spin_trylock_irqsave(&meta->lock, flags))) {
/*
addr = (void *)meta->addr;
/* Update remaining metadata. */
- metadata_update_state(meta, KFENCE_OBJECT_ALLOCATED);
+ metadata_update_state(meta, KFENCE_OBJECT_ALLOCATED, stack_entries, num_stack_entries);
/* Pairs with READ_ONCE() in kfence_shutdown_cache(). */
WRITE_ONCE(meta->cache, cache);
meta->size = size;
- for_each_canary(meta, set_canary_byte);
+ meta->alloc_stack_hash = alloc_stack_hash;
+ raw_spin_unlock_irqrestore(&meta->lock, flags);
+
+ alloc_covered_add(alloc_stack_hash, 1);
/* Set required struct page fields. */
page = virt_to_page(meta->addr);
if (IS_ENABLED(CONFIG_SLAB))
page->s_mem = addr;
- raw_spin_unlock_irqrestore(&meta->lock, flags);
-
/* Memory initialization. */
+ for_each_canary(meta, set_canary_byte);
/*
* We check slab_want_init_on_alloc() ourselves, rather than letting
{
struct kcsan_scoped_access assert_page_exclusive;
unsigned long flags;
+ bool init;
raw_spin_lock_irqsave(&meta->lock, flags);
meta->unprotected_page = 0;
}
+ /* Mark the object as freed. */
+ metadata_update_state(meta, KFENCE_OBJECT_FREED, NULL, 0);
+ init = slab_want_init_on_free(meta->cache);
+ raw_spin_unlock_irqrestore(&meta->lock, flags);
+
+ alloc_covered_add(meta->alloc_stack_hash, -1);
+
/* Check canary bytes for memory corruption. */
for_each_canary(meta, check_canary_byte);
* data is still there, and after a use-after-free is detected, we
* unprotect the page, so the data is still accessible.
*/
- if (!zombie && unlikely(slab_want_init_on_free(meta->cache)))
+ if (!zombie && unlikely(init))
memzero_explicit(addr, meta->size);
- /* Mark the object as freed. */
- metadata_update_state(meta, KFENCE_OBJECT_FREED);
-
- raw_spin_unlock_irqrestore(&meta->lock, flags);
-
/* Protect to detect use-after-frees. */
kfence_protect((unsigned long)addr);
if (!kfence_sample_interval)
return;
+ stack_hash_seed = (u32)random_get_entropy();
if (!kfence_init_pool()) {
pr_err("%s failed\n", __func__);
return;
}
+ if (!IS_ENABLED(CONFIG_KFENCE_STATIC_KEYS))
+ static_branch_enable(&kfence_allocation_key);
WRITE_ONCE(kfence_enabled, true);
queue_delayed_work(system_unbound_wq, &kfence_timer, 0);
pr_info("initialized - using %lu bytes for %d objects at 0x%p-0x%p\n", KFENCE_POOL_SIZE,
void *__kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags)
{
+ unsigned long stack_entries[KFENCE_STACK_DEPTH];
+ size_t num_stack_entries;
+ u32 alloc_stack_hash;
+
/*
* Perform size check before switching kfence_allocation_gate, so that
* we don't disable KFENCE without making an allocation.
*/
- if (size > PAGE_SIZE)
+ if (size > PAGE_SIZE) {
+ atomic_long_inc(&counters[KFENCE_COUNTER_SKIP_INCOMPAT]);
return NULL;
+ }
/*
* Skip allocations from non-default zones, including DMA. We cannot
* properties (e.g. reside in DMAable memory).
*/
if ((flags & GFP_ZONEMASK) ||
- (s->flags & (SLAB_CACHE_DMA | SLAB_CACHE_DMA32)))
+ (s->flags & (SLAB_CACHE_DMA | SLAB_CACHE_DMA32))) {
+ atomic_long_inc(&counters[KFENCE_COUNTER_SKIP_INCOMPAT]);
return NULL;
+ }
- /*
- * allocation_gate only needs to become non-zero, so it doesn't make
- * sense to continue writing to it and pay the associated contention
- * cost, in case we have a large number of concurrent allocations.
- */
- if (atomic_read(&kfence_allocation_gate) || atomic_inc_return(&kfence_allocation_gate) > 1)
+ if (atomic_inc_return(&kfence_allocation_gate) > 1)
return NULL;
#ifdef CONFIG_KFENCE_STATIC_KEYS
/*
if (!READ_ONCE(kfence_enabled))
return NULL;
- return kfence_guarded_alloc(s, size, flags);
+ num_stack_entries = stack_trace_save(stack_entries, KFENCE_STACK_DEPTH, 0);
+
+ /*
+ * Do expensive check for coverage of allocation in slow-path after
+ * allocation_gate has already become non-zero, even though it might
+ * mean not making any allocation within a given sample interval.
+ *
+ * This ensures reasonable allocation coverage when the pool is almost
+ * full, including avoiding long-lived allocations of the same source
+ * filling up the pool (e.g. pagecache allocations).
+ */
+ alloc_stack_hash = get_alloc_stack_hash(stack_entries, num_stack_entries);
+ if (should_skip_covered() && alloc_covered_contains(alloc_stack_hash)) {
+ atomic_long_inc(&counters[KFENCE_COUNTER_SKIP_COVERED]);
+ return NULL;
+ }
+
+ return kfence_guarded_alloc(s, size, flags, stack_entries, num_stack_entries,
+ alloc_stack_hash);
}
size_t kfence_ksize(const void *addr)
/* Allocation and free stack information. */
struct kfence_track alloc_track;
struct kfence_track free_track;
+ /* For updating alloc_covered on frees. */
+ u32 alloc_stack_hash;
};
extern struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS];
#define arch_kfence_test_address(addr) (addr)
#endif
+#define KFENCE_TEST_REQUIRES(test, cond) do { \
+ if (!(cond)) \
+ kunit_skip((test), "Test requires: " #cond); \
+} while (0)
+
/* Report as observed from console. */
static struct {
spinlock_t lock;
};
int i;
- if (!IS_ENABLED(CONFIG_INIT_ON_FREE_DEFAULT_ON))
- return;
+ KFENCE_TEST_REQUIRES(test, IS_ENABLED(CONFIG_INIT_ON_FREE_DEFAULT_ON));
/* Assume it hasn't been disabled on command line. */
setup_test_cache(test, size, 0, NULL);
char *buf1, *buf2;
int i;
- if (CONFIG_KFENCE_SAMPLE_INTERVAL > 100) {
- kunit_warn(test, "skipping ... would take too long\n");
- return;
- }
+ /* Skip if we think it'd take too long. */
+ KFENCE_TEST_REQUIRES(test, CONFIG_KFENCE_SAMPLE_INTERVAL <= 100);
setup_test_cache(test, size, 0, NULL);
buf1 = test_alloc(test, size, GFP_KERNEL, ALLOCATE_ANY);
int nr_zones = 0;
unsigned long recommended_min;
+ if (!khugepaged_enabled()) {
+ calculate_min_free_kbytes();
+ goto update_wmarks;
+ }
+
for_each_populated_zone(zone) {
/*
* We don't need to worry about fragmentation of
min_free_kbytes = recommended_min;
}
+
+update_wmarks:
setup_per_zone_wmarks();
}
if (!list_empty(&khugepaged_scan.mm_head))
wake_up_interruptible(&khugepaged_wait);
-
- set_recommended_min_free_kbytes();
} else if (khugepaged_thread) {
kthread_stop(khugepaged_thread);
khugepaged_thread = NULL;
}
+ set_recommended_min_free_kbytes();
fail:
mutex_unlock(&khugepaged_mutex);
return err;
#include "slab.h"
#ifdef CONFIG_MEMCG_KMEM
-static LIST_HEAD(list_lrus);
+static LIST_HEAD(memcg_list_lrus);
static DEFINE_MUTEX(list_lrus_mutex);
+static inline bool list_lru_memcg_aware(struct list_lru *lru)
+{
+ return lru->memcg_aware;
+}
+
static void list_lru_register(struct list_lru *lru)
{
+ if (!list_lru_memcg_aware(lru))
+ return;
+
mutex_lock(&list_lrus_mutex);
- list_add(&lru->list, &list_lrus);
+ list_add(&lru->list, &memcg_list_lrus);
mutex_unlock(&list_lrus_mutex);
}
static void list_lru_unregister(struct list_lru *lru)
{
+ if (!list_lru_memcg_aware(lru))
+ return;
+
mutex_lock(&list_lrus_mutex);
list_del(&lru->list);
mutex_unlock(&list_lrus_mutex);
return lru->shrinker_id;
}
-static inline bool list_lru_memcg_aware(struct list_lru *lru)
-{
- return lru->memcg_aware;
-}
-
static inline struct list_lru_one *
list_lru_from_memcg_idx(struct list_lru_node *nlru, int idx)
{
{
struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
- unsigned long count;
+ long count;
rcu_read_lock();
l = list_lru_from_memcg_idx(nlru, memcg_cache_id(memcg));
count = READ_ONCE(l->nr_items);
rcu_read_unlock();
+ if (unlikely(count < 0))
+ count = 0;
+
return count;
}
EXPORT_SYMBOL_GPL(list_lru_count_one);
struct list_lru_memcg *memcg_lrus;
int size = memcg_nr_cache_ids;
- memcg_lrus = kvmalloc(sizeof(*memcg_lrus) +
- size * sizeof(void *), GFP_KERNEL);
+ memcg_lrus = kvmalloc(struct_size(memcg_lrus, lru, size), GFP_KERNEL);
if (!memcg_lrus)
return -ENOMEM;
old = rcu_dereference_protected(nlru->memcg_lrus,
lockdep_is_held(&list_lrus_mutex));
- new = kvmalloc(sizeof(*new) + new_size * sizeof(void *), GFP_KERNEL);
+ new = kvmalloc(struct_size(new, lru, new_size), GFP_KERNEL);
if (!new)
return -ENOMEM;
return -ENOMEM;
}
- memcpy(&new->lru, &old->lru, old_size * sizeof(void *));
-
- /*
- * The locking below allows readers that hold nlru->lock avoid taking
- * rcu_read_lock (see list_lru_from_memcg_idx).
- *
- * Since list_lru_{add,del} may be called under an IRQ-safe lock,
- * we have to use IRQ-safe primitives here to avoid deadlock.
- */
- spin_lock_irq(&nlru->lock);
+ memcpy(&new->lru, &old->lru, flex_array_size(new, lru, old_size));
rcu_assign_pointer(nlru->memcg_lrus, new);
- spin_unlock_irq(&nlru->lock);
-
kvfree_rcu(old, rcu);
return 0;
}
{
int i;
- if (!list_lru_memcg_aware(lru))
- return 0;
-
for_each_node(i) {
if (memcg_update_list_lru_node(&lru->node[i],
old_size, new_size))
{
int i;
- if (!list_lru_memcg_aware(lru))
- return;
-
for_each_node(i)
memcg_cancel_update_list_lru_node(&lru->node[i],
old_size, new_size);
int old_size = memcg_nr_cache_ids;
mutex_lock(&list_lrus_mutex);
- list_for_each_entry(lru, &list_lrus, list) {
+ list_for_each_entry(lru, &memcg_list_lrus, list) {
ret = memcg_update_list_lru(lru, old_size, new_size);
if (ret)
goto fail;
mutex_unlock(&list_lrus_mutex);
return ret;
fail:
- list_for_each_entry_continue_reverse(lru, &list_lrus, list)
+ list_for_each_entry_continue_reverse(lru, &memcg_list_lrus, list)
memcg_cancel_update_list_lru(lru, old_size, new_size);
goto out;
}
{
int i;
- if (!list_lru_memcg_aware(lru))
- return;
-
for_each_node(i)
memcg_drain_list_lru_node(lru, i, src_idx, dst_memcg);
}
struct list_lru *lru;
mutex_lock(&list_lrus_mutex);
- list_for_each_entry(lru, &list_lrus, list)
+ list_for_each_entry(lru, &memcg_list_lrus, list)
memcg_drain_list_lru(lru, src_idx, dst_memcg);
mutex_unlock(&list_lrus_mutex);
}
struct iovec iovstack[UIO_FASTIOV], iovec;
struct iovec *iov = iovstack;
struct iov_iter iter;
- struct pid *pid;
struct task_struct *task;
struct mm_struct *mm;
size_t total_len;
if (ret < 0)
goto out;
- pid = pidfd_get_pid(pidfd, &f_flags);
- if (IS_ERR(pid)) {
- ret = PTR_ERR(pid);
+ task = pidfd_get_task(pidfd, &f_flags);
+ if (IS_ERR(task)) {
+ ret = PTR_ERR(task);
goto free_iov;
}
- task = get_pid_task(pid, PIDTYPE_PID);
- if (!task) {
- ret = -ESRCH;
- goto put_pid;
- }
-
if (!process_madvise_behavior_valid(behavior)) {
ret = -EINVAL;
goto release_task;
mmput(mm);
release_task:
put_task_struct(task);
-put_pid:
- put_pid(pid);
free_iov:
kfree(iov);
out:
{
/* pump up @end */
if (end == MEMBLOCK_ALLOC_ACCESSIBLE ||
- end == MEMBLOCK_ALLOC_KASAN)
+ end == MEMBLOCK_ALLOC_NOLEAKTRACE)
end = memblock.current_limit;
/* avoid allocating the first page */
addr = __pa(memblock.reserved.regions);
size = PAGE_ALIGN(sizeof(struct memblock_region) *
memblock.reserved.max);
- __memblock_free_late(addr, size);
+ memblock_free_late(addr, size);
}
if (memblock.memory.regions != memblock_memory_init_regions) {
addr = __pa(memblock.memory.regions);
size = PAGE_ALIGN(sizeof(struct memblock_region) *
memblock.memory.max);
- __memblock_free_late(addr, size);
+ memblock_free_late(addr, size);
}
memblock_memory = NULL;
kfree(old_array);
else if (old_array != memblock_memory_init_regions &&
old_array != memblock_reserved_init_regions)
- memblock_free_ptr(old_array, old_alloc_size);
+ memblock_free(old_array, old_alloc_size);
/*
* Reserve the new array if that comes from the memblock. Otherwise, we
* @base: base address of the new region
* @size: size of the new region
* @nid: nid of the new region
+ * @flags: flags of the new region
*
* Add new memblock region [@base, @base + @size) to the "memory"
* type. See memblock_add_range() description for mode details
* 0 on success, -errno on failure.
*/
int __init_memblock memblock_add_node(phys_addr_t base, phys_addr_t size,
- int nid)
+ int nid, enum memblock_flags flags)
{
phys_addr_t end = base + size - 1;
- memblock_dbg("%s: [%pa-%pa] nid=%d %pS\n", __func__,
- &base, &end, nid, (void *)_RET_IP_);
+ memblock_dbg("%s: [%pa-%pa] nid=%d flags=%x %pS\n", __func__,
+ &base, &end, nid, flags, (void *)_RET_IP_);
- return memblock_add_range(&memblock.memory, base, size, nid, 0);
+ return memblock_add_range(&memblock.memory, base, size, nid, flags);
}
/**
}
/**
- * memblock_free_ptr - free boot memory allocation
+ * memblock_free - free boot memory allocation
* @ptr: starting address of the boot memory allocation
* @size: size of the boot memory block in bytes
*
* Free boot memory block previously allocated by memblock_alloc_xx() API.
* The freeing memory will not be released to the buddy allocator.
*/
-void __init_memblock memblock_free_ptr(void *ptr, size_t size)
+void __init_memblock memblock_free(void *ptr, size_t size)
{
if (ptr)
- memblock_free(__pa(ptr), size);
+ memblock_phys_free(__pa(ptr), size);
}
/**
- * memblock_free - free boot memory block
+ * memblock_phys_free - free boot memory block
* @base: phys starting address of the boot memory block
* @size: size of the boot memory block in bytes
*
* Free boot memory block previously allocated by memblock_alloc_xx() API.
* The freeing memory will not be released to the buddy allocator.
*/
-int __init_memblock memblock_free(phys_addr_t base, phys_addr_t size)
+int __init_memblock memblock_phys_free(phys_addr_t base, phys_addr_t size)
{
phys_addr_t end = base + size - 1;
if (!(flags & MEMBLOCK_NOMAP) && memblock_is_nomap(m))
return true;
+ /* skip driver-managed memory unless we were asked for it explicitly */
+ if (!(flags & MEMBLOCK_DRIVER_MANAGED) && memblock_is_driver_managed(m))
+ return true;
+
return false;
}
return 0;
done:
- /* Skip kmemleak for kasan_init() due to high volume. */
- if (end != MEMBLOCK_ALLOC_KASAN)
+ /*
+ * Skip kmemleak for those places like kasan_init() and
+ * early_pgtable_alloc() due to high volume.
+ */
+ if (end != MEMBLOCK_ALLOC_NOLEAKTRACE)
/*
* The min_count is set to 0 so that memblock allocated
* blocks are never reported as leaks. This is because many
}
/**
- * __memblock_free_late - free pages directly to buddy allocator
+ * memblock_free_late - free pages directly to buddy allocator
* @base: phys starting address of the boot memory block
* @size: size of the boot memory block in bytes
*
* down, but we are still initializing the system. Pages are released directly
* to the buddy allocator.
*/
-void __init __memblock_free_late(phys_addr_t base, phys_addr_t size)
+void __init memblock_free_late(phys_addr_t base, phys_addr_t size)
{
phys_addr_t cursor, end;
* memmap array.
*/
if (pg < pgend)
- memblock_free(pg, pgend - pg);
+ memblock_phys_free(pg, pgend - pg);
}
/*
return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_noswap;
}
-/* memcg and lruvec stats flushing */
-static void flush_memcg_stats_dwork(struct work_struct *w);
-static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork);
-static DEFINE_SPINLOCK(stats_flush_lock);
-
#define THRESHOLDS_EVENTS_TARGET 128
#define SOFTLIMIT_EVENTS_TARGET 1024
iter != NULL; \
iter = mem_cgroup_iter(NULL, iter, NULL))
-static inline bool should_force_charge(void)
+static inline bool task_is_dying(void)
{
return tsk_is_oom_victim(current) || fatal_signal_pending(current) ||
(current->flags & PF_EXITING);
return mz;
}
+/*
+ * memcg and lruvec stats flushing
+ *
+ * Many codepaths leading to stats update or read are performance sensitive and
+ * adding stats flushing in such codepaths is not desirable. So, to optimize the
+ * flushing the kernel does:
+ *
+ * 1) Periodically and asynchronously flush the stats every 2 seconds to not let
+ * rstat update tree grow unbounded.
+ *
+ * 2) Flush the stats synchronously on reader side only when there are more than
+ * (MEMCG_CHARGE_BATCH * nr_cpus) update events. Though this optimization
+ * will let stats be out of sync by atmost (MEMCG_CHARGE_BATCH * nr_cpus) but
+ * only for 2 seconds due to (1).
+ */
+static void flush_memcg_stats_dwork(struct work_struct *w);
+static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork);
+static DEFINE_SPINLOCK(stats_flush_lock);
+static DEFINE_PER_CPU(unsigned int, stats_updates);
+static atomic_t stats_flush_threshold = ATOMIC_INIT(0);
+
+static inline void memcg_rstat_updated(struct mem_cgroup *memcg)
+{
+ cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id());
+ if (!(__this_cpu_inc_return(stats_updates) % MEMCG_CHARGE_BATCH))
+ atomic_inc(&stats_flush_threshold);
+}
+
+static void __mem_cgroup_flush_stats(void)
+{
+ unsigned long flag;
+
+ if (!spin_trylock_irqsave(&stats_flush_lock, flag))
+ return;
+
+ cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup);
+ atomic_set(&stats_flush_threshold, 0);
+ spin_unlock_irqrestore(&stats_flush_lock, flag);
+}
+
+void mem_cgroup_flush_stats(void)
+{
+ if (atomic_read(&stats_flush_threshold) > num_online_cpus())
+ __mem_cgroup_flush_stats();
+}
+
+static void flush_memcg_stats_dwork(struct work_struct *w)
+{
+ mem_cgroup_flush_stats();
+ queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ);
+}
+
/**
* __mod_memcg_state - update cgroup memory statistics
* @memcg: the memory cgroup
return;
__this_cpu_add(memcg->vmstats_percpu->state[idx], val);
- cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id());
+ memcg_rstat_updated(memcg);
}
/* idx can be of type enum memcg_stat_item or node_stat_item. */
memcg = pn->memcg;
/* Update memcg */
- __mod_memcg_state(memcg, idx, val);
+ __this_cpu_add(memcg->vmstats_percpu->state[idx], val);
/* Update lruvec */
__this_cpu_add(pn->lruvec_stats_percpu->state[idx], val);
+
+ memcg_rstat_updated(memcg);
}
/**
return;
__this_cpu_add(memcg->vmstats_percpu->events[idx], count);
- cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id());
+ memcg_rstat_updated(memcg);
}
static unsigned long memcg_events(struct mem_cgroup *memcg, int event)
*
* Current memory state:
*/
- cgroup_rstat_flush(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
u64 size;
* A few threads which were not waiting at mutex_lock_killable() can
* fail to bail out. Therefore, check again after holding oom_lock.
*/
- ret = should_force_charge() || out_of_memory(&oc);
+ ret = task_is_dying() || out_of_memory(&oc);
unlock:
mutex_unlock(&oom_lock);
memcg->move_lock_task = current;
memcg->move_lock_flags = flags;
}
-EXPORT_SYMBOL(folio_memcg_lock);
void lock_page_memcg(struct page *page)
{
folio_memcg_lock(page_folio(page));
}
-EXPORT_SYMBOL(lock_page_memcg);
static void __folio_memcg_unlock(struct mem_cgroup *memcg)
{
{
__folio_memcg_unlock(folio_memcg(folio));
}
-EXPORT_SYMBOL(folio_memcg_unlock);
void unlock_page_memcg(struct page *page)
{
folio_memcg_unlock(page_folio(page));
}
-EXPORT_SYMBOL(unlock_page_memcg);
struct obj_stock {
#ifdef CONFIG_MEMCG_KMEM
struct page_counter *counter;
enum oom_status oom_status;
unsigned long nr_reclaimed;
+ bool passed_oom = false;
bool may_swap = true;
bool drained = false;
unsigned long pflags;
goto force;
/*
- * Unlike in global OOM situations, memcg is not in a physical
- * memory shortage. Allow dying and OOM-killed tasks to
- * bypass the last charges so that they can exit quickly and
- * free their memory.
- */
- if (unlikely(should_force_charge()))
- goto force;
-
- /*
* Prevent unbounded recursion when reclaim operations need to
* allocate memory. This might exceed the limits temporarily,
* but we prefer facilitating memory reclaim and getting back
if (gfp_mask & __GFP_RETRY_MAYFAIL)
goto nomem;
- if (fatal_signal_pending(current))
- goto force;
+ /* Avoid endless loop for tasks bypassed by the oom killer */
+ if (passed_oom && task_is_dying())
+ goto nomem;
/*
* keep retrying as long as the memcg oom killer is able to make
*/
oom_status = mem_cgroup_oom(mem_over_limit, gfp_mask,
get_order(nr_pages * PAGE_SIZE));
- switch (oom_status) {
- case OOM_SUCCESS:
+ if (oom_status == OOM_SUCCESS) {
+ passed_oom = true;
nr_retries = MAX_RECLAIM_RETRIES;
goto retry;
- case OOM_FAILED:
- goto force;
- default:
- goto nomem;
}
nomem:
if (!(gfp_mask & __GFP_NOFAIL))
return try_charge_memcg(memcg, gfp_mask, nr_pages);
}
-#if defined(CONFIG_MEMCG_KMEM) || defined(CONFIG_MMU)
-static void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
+static inline void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
if (mem_cgroup_is_root(memcg))
return;
if (do_memsw_account())
page_counter_uncharge(&memcg->memsw, nr_pages);
}
-#endif
static void commit_charge(struct folio *folio, struct mem_cgroup *memcg)
{
static int obj_cgroup_charge_pages(struct obj_cgroup *objcg, gfp_t gfp,
unsigned int nr_pages)
{
- struct page_counter *counter;
struct mem_cgroup *memcg;
int ret;
if (ret)
goto out;
- if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) &&
- !page_counter_try_charge(&memcg->kmem, nr_pages, &counter)) {
-
- /*
- * Enforce __GFP_NOFAIL allocation because callers are not
- * prepared to see failures and likely do not have any failure
- * handling code.
- */
- if (gfp & __GFP_NOFAIL) {
- page_counter_charge(&memcg->kmem, nr_pages);
- goto out;
- }
- cancel_charge(memcg, nr_pages);
- ret = -ENOMEM;
- }
+ if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
+ page_counter_charge(&memcg->kmem, nr_pages);
out:
css_put(&memcg->css);
/* try to free all pages in this cgroup */
while (nr_retries && page_counter_read(&memcg->memory)) {
- int progress;
-
if (signal_pending(current))
return -EINTR;
- progress = try_to_free_mem_cgroup_pages(memcg, 1,
- GFP_KERNEL, true);
- if (!progress) {
+ if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, true))
nr_retries--;
- /* maybe some writeback is necessary */
- congestion_wait(BLK_RW_ASYNC, HZ/10);
- }
-
}
return 0;
unsigned long val;
if (mem_cgroup_is_root(memcg)) {
- /* mem_cgroup_threshold() calls here from irqsafe context */
- cgroup_rstat_flush_irqsafe(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
val = memcg_page_state(memcg, NR_FILE_PAGES) +
memcg_page_state(memcg, NR_ANON_MAPPED);
if (swap)
return 0;
BUG_ON(memcg->kmemcg_id >= 0);
- BUG_ON(memcg->kmem_state);
memcg_id = memcg_alloc_cache_id();
if (memcg_id < 0)
static_branch_enable(&memcg_kmem_enabled_key);
memcg->kmemcg_id = memcg_id;
- memcg->kmem_state = KMEM_ONLINE;
return 0;
}
static void memcg_offline_kmem(struct mem_cgroup *memcg)
{
- struct cgroup_subsys_state *css;
- struct mem_cgroup *parent, *child;
+ struct mem_cgroup *parent;
int kmemcg_id;
- if (memcg->kmem_state != KMEM_ONLINE)
+ if (memcg->kmemcg_id == -1)
return;
- memcg->kmem_state = KMEM_ALLOCATED;
-
parent = parent_mem_cgroup(memcg);
if (!parent)
parent = root_mem_cgroup;
BUG_ON(kmemcg_id < 0);
/*
- * Change kmemcg_id of this cgroup and all its descendants to the
- * parent's id, and then move all entries from this cgroup's list_lrus
- * to ones of the parent. After we have finished, all list_lrus
- * corresponding to this cgroup are guaranteed to remain empty. The
- * ordering is imposed by list_lru_node->lock taken by
+ * After we have finished memcg_reparent_objcgs(), all list_lrus
+ * corresponding to this cgroup are guaranteed to remain empty.
+ * The ordering is imposed by list_lru_node->lock taken by
* memcg_drain_all_list_lrus().
*/
- rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */
- css_for_each_descendant_pre(css, &memcg->css) {
- child = mem_cgroup_from_css(css);
- BUG_ON(child->kmemcg_id != kmemcg_id);
- child->kmemcg_id = parent->kmemcg_id;
- }
- rcu_read_unlock();
-
memcg_drain_all_list_lrus(kmemcg_id, parent);
memcg_free_cache_id(kmemcg_id);
-}
-
-static void memcg_free_kmem(struct mem_cgroup *memcg)
-{
- /* css_alloc() failed, offlining didn't happen */
- if (unlikely(memcg->kmem_state == KMEM_ONLINE))
- memcg_offline_kmem(memcg);
+ memcg->kmemcg_id = -1;
}
#else
static int memcg_online_kmem(struct mem_cgroup *memcg)
static void memcg_offline_kmem(struct mem_cgroup *memcg)
{
}
-static void memcg_free_kmem(struct mem_cgroup *memcg)
-{
-}
#endif /* CONFIG_MEMCG_KMEM */
-static int memcg_update_kmem_max(struct mem_cgroup *memcg,
- unsigned long max)
-{
- int ret;
-
- mutex_lock(&memcg_max_mutex);
- ret = page_counter_set_max(&memcg->kmem, max);
- mutex_unlock(&memcg_max_mutex);
- return ret;
-}
-
static int memcg_update_tcp_max(struct mem_cgroup *memcg, unsigned long max)
{
int ret;
ret = mem_cgroup_resize_max(memcg, nr_pages, true);
break;
case _KMEM:
- pr_warn_once("kmem.limit_in_bytes is deprecated and will be removed. "
- "Please report your usecase to linux-mm@kvack.org if you "
- "depend on this functionality.\n");
- ret = memcg_update_kmem_max(memcg, nr_pages);
+ /* kmem.limit_in_bytes is deprecated. */
+ ret = -EOPNOTSUPP;
break;
case _TCP:
ret = memcg_update_tcp_max(memcg, nr_pages);
int nid;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
- cgroup_rstat_flush(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) {
seq_printf(m, "%s=%lu", stat->name,
BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats));
- cgroup_rstat_flush(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) {
unsigned long nr;
struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
struct mem_cgroup *parent;
- cgroup_rstat_flush_irqsafe(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
*pdirty = memcg_page_state(memcg, NR_FILE_DIRTY);
*pwriteback = memcg_page_state(memcg, NR_WRITEBACK);
cancel_work_sync(&memcg->high_work);
mem_cgroup_remove_from_trees(memcg);
free_shrinker_info(memcg);
- memcg_free_kmem(memcg);
+
+ /* Need to offline kmem if online_css() fails */
+ memcg_offline_kmem(memcg);
mem_cgroup_free(memcg);
}
memcg_wb_domain_size_changed(memcg);
}
-void mem_cgroup_flush_stats(void)
-{
- if (!spin_trylock(&stats_flush_lock))
- return;
-
- cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup);
- spin_unlock(&stats_flush_lock);
-}
-
-static void flush_memcg_stats_dwork(struct work_struct *w)
-{
- mem_cgroup_flush_stats();
- queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ);
-}
-
static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
{
struct mem_cgroup *memcg = mem_cgroup_from_css(css);
#endif
static struct page *mc_handle_file_pte(struct vm_area_struct *vma,
- unsigned long addr, pte_t ptent, swp_entry_t *entry)
+ unsigned long addr, pte_t ptent)
{
if (!vma->vm_file) /* anonymous vma */
return NULL;
else if (is_swap_pte(ptent))
page = mc_handle_swap_pte(vma, ptent, &ent);
else if (pte_none(ptent))
- page = mc_handle_file_pte(vma, addr, ptent, &ent);
+ page = mc_handle_file_pte(vma, addr, ptent);
if (!page && !ent.val)
return ret;
int i;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
- cgroup_rstat_flush(memcg->css.cgroup);
+ mem_cgroup_flush_stats();
for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
int nid;
}
if (flags & MFD_HUGETLB) {
- struct ucounts *ucounts = NULL;
-
- file = hugetlb_file_setup(name, 0, VM_NORESERVE, &ucounts,
+ file = hugetlb_file_setup(name, 0, VM_NORESERVE,
HUGETLB_ANONHUGE_INODE,
(flags >> MFD_HUGE_SHIFT) &
MFD_HUGE_MASK);
#include <linux/kernel-page-flags.h>
#include <linux/sched/signal.h>
#include <linux/sched/task.h>
+#include <linux/dax.h>
#include <linux/ksm.h>
#include <linux/rmap.h>
#include <linux/export.h>
#define hwpoison_hugetlb_range NULL
#endif
-static struct mm_walk_ops hwp_walk_ops = {
+static const struct mm_walk_ops hwp_walk_ops = {
.pmd_entry = hwpoison_pte_range,
.hugetlb_entry = hwpoison_hugetlb_range,
};
return ret;
}
+struct page_state {
+ unsigned long mask;
+ unsigned long res;
+ enum mf_action_page_type type;
+
+ /* Callback ->action() has to unlock the relevant page inside it. */
+ int (*action)(struct page_state *ps, struct page *p);
+};
+
+/*
+ * Return true if page is still referenced by others, otherwise return
+ * false.
+ *
+ * The extra_pins is true when one extra refcount is expected.
+ */
+static bool has_extra_refcount(struct page_state *ps, struct page *p,
+ bool extra_pins)
+{
+ int count = page_count(p) - 1;
+
+ if (extra_pins)
+ count -= 1;
+
+ if (count > 0) {
+ pr_err("Memory failure: %#lx: %s still referenced by %d users\n",
+ page_to_pfn(p), action_page_types[ps->type], count);
+ return true;
+ }
+
+ return false;
+}
+
/*
* Error hit kernel page.
* Do nothing, try to be lucky and not touch this instead. For a few cases we
* could be more sophisticated.
*/
-static int me_kernel(struct page *p, unsigned long pfn)
+static int me_kernel(struct page_state *ps, struct page *p)
{
unlock_page(p);
return MF_IGNORED;
/*
* Page in unknown state. Do nothing.
*/
-static int me_unknown(struct page *p, unsigned long pfn)
+static int me_unknown(struct page_state *ps, struct page *p)
{
- pr_err("Memory failure: %#lx: Unknown page state\n", pfn);
+ pr_err("Memory failure: %#lx: Unknown page state\n", page_to_pfn(p));
unlock_page(p);
return MF_FAILED;
}
/*
* Clean (or cleaned) page cache page.
*/
-static int me_pagecache_clean(struct page *p, unsigned long pfn)
+static int me_pagecache_clean(struct page_state *ps, struct page *p)
{
int ret;
struct address_space *mapping;
*
* Open: to take i_rwsem or not for this? Right now we don't.
*/
- ret = truncate_error_page(p, pfn, mapping);
+ ret = truncate_error_page(p, page_to_pfn(p), mapping);
out:
unlock_page(p);
+
+ if (has_extra_refcount(ps, p, false))
+ ret = MF_FAILED;
+
return ret;
}
* Issues: when the error hit a hole page the error is not properly
* propagated.
*/
-static int me_pagecache_dirty(struct page *p, unsigned long pfn)
+static int me_pagecache_dirty(struct page_state *ps, struct page *p)
{
struct address_space *mapping = page_mapping(p);
mapping_set_error(mapping, -EIO);
}
- return me_pagecache_clean(p, pfn);
+ return me_pagecache_clean(ps, p);
}
/*
* Clean swap cache pages can be directly isolated. A later page fault will
* bring in the known good data from disk.
*/
-static int me_swapcache_dirty(struct page *p, unsigned long pfn)
+static int me_swapcache_dirty(struct page_state *ps, struct page *p)
{
int ret;
+ bool extra_pins = false;
ClearPageDirty(p);
/* Trigger EIO in shmem: */
ret = delete_from_lru_cache(p) ? MF_FAILED : MF_DELAYED;
unlock_page(p);
+
+ if (ret == MF_DELAYED)
+ extra_pins = true;
+
+ if (has_extra_refcount(ps, p, extra_pins))
+ ret = MF_FAILED;
+
return ret;
}
-static int me_swapcache_clean(struct page *p, unsigned long pfn)
+static int me_swapcache_clean(struct page_state *ps, struct page *p)
{
int ret;
ret = delete_from_lru_cache(p) ? MF_FAILED : MF_RECOVERED;
unlock_page(p);
+
+ if (has_extra_refcount(ps, p, false))
+ ret = MF_FAILED;
+
return ret;
}
* - Error on hugepage is contained in hugepage unit (not in raw page unit.)
* To narrow down kill region to one page, we need to break up pmd.
*/
-static int me_huge_page(struct page *p, unsigned long pfn)
+static int me_huge_page(struct page_state *ps, struct page *p)
{
int res;
struct page *hpage = compound_head(p);
mapping = page_mapping(hpage);
if (mapping) {
- res = truncate_error_page(hpage, pfn, mapping);
+ res = truncate_error_page(hpage, page_to_pfn(p), mapping);
unlock_page(hpage);
} else {
res = MF_FAILED;
}
}
+ if (has_extra_refcount(ps, p, false))
+ res = MF_FAILED;
+
return res;
}
#define slab (1UL << PG_slab)
#define reserved (1UL << PG_reserved)
-static struct page_state {
- unsigned long mask;
- unsigned long res;
- enum mf_action_page_type type;
-
- /* Callback ->action() has to unlock the relevant page inside it. */
- int (*action)(struct page *p, unsigned long pfn);
-} error_states[] = {
+static struct page_state error_states[] = {
{ reserved, reserved, MF_MSG_KERNEL, me_kernel },
/*
* free pages are specially detected outside this table:
unsigned long pfn)
{
int result;
- int count;
/* page p should be unlocked after returning from ps->action(). */
- result = ps->action(p, pfn);
+ result = ps->action(ps, p);
- count = page_count(p) - 1;
- if (ps->action == me_swapcache_dirty && result == MF_DELAYED)
- count--;
- if (count > 0) {
- pr_err("Memory failure: %#lx: %s still referenced by %d users\n",
- pfn, action_page_types[ps->type], count);
- result = MF_FAILED;
- }
action_result(pfn, ps->type, result);
/* Could do more checks here if page looks ok */
static int try_to_split_thp_page(struct page *page, const char *msg)
{
lock_page(page);
- if (!PageAnon(page) || unlikely(split_huge_page(page))) {
+ if (unlikely(split_huge_page(page))) {
unsigned long pfn = page_to_pfn(page);
unlock_page(page);
- if (!PageAnon(page))
- pr_info("%s: %#lx: non anonymous thp\n", msg, pfn);
- else
- pr_info("%s: %#lx: thp split failed\n", msg, pfn);
+ pr_info("%s: %#lx: thp split failed\n", msg, pfn);
put_page(page);
return -EBUSY;
}
}
}
-int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
+void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte)
{
- spinlock_t *ptl;
- pgtable_t new = pte_alloc_one(mm);
- if (!new)
- return -ENOMEM;
+ spinlock_t *ptl = pmd_lock(mm, pmd);
- /*
- * Ensure all pte setup (eg. pte page lock and page clearing) are
- * visible before the pte is made visible to other CPUs by being
- * put into page tables.
- *
- * The other side of the story is the pointer chasing in the page
- * table walking code (when walking the page table without locking;
- * ie. most of the time). Fortunately, these data accesses consist
- * of a chain of data-dependent loads, meaning most CPUs (alpha
- * being the notable exception) will already guarantee loads are
- * seen in-order. See the alpha page table accessors for the
- * smp_rmb() barriers in page table walking code.
- */
- smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */
-
- ptl = pmd_lock(mm, pmd);
if (likely(pmd_none(*pmd))) { /* Has another populated it ? */
mm_inc_nr_ptes(mm);
- pmd_populate(mm, pmd, new);
- new = NULL;
+ /*
+ * Ensure all pte setup (eg. pte page lock and page clearing) are
+ * visible before the pte is made visible to other CPUs by being
+ * put into page tables.
+ *
+ * The other side of the story is the pointer chasing in the page
+ * table walking code (when walking the page table without locking;
+ * ie. most of the time). Fortunately, these data accesses consist
+ * of a chain of data-dependent loads, meaning most CPUs (alpha
+ * being the notable exception) will already guarantee loads are
+ * seen in-order. See the alpha page table accessors for the
+ * smp_rmb() barriers in page table walking code.
+ */
+ smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */
+ pmd_populate(mm, pmd, *pte);
+ *pte = NULL;
}
spin_unlock(ptl);
+}
+
+int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
+{
+ pgtable_t new = pte_alloc_one(mm);
+ if (!new)
+ return -ENOMEM;
+
+ pmd_install(mm, pmd, &new);
if (new)
pte_free(mm, new);
return 0;
if (!new)
return -ENOMEM;
- smp_wmb(); /* See comment in __pte_alloc */
-
spin_lock(&init_mm.page_table_lock);
if (likely(pmd_none(*pmd))) { /* Has another populated it ? */
+ smp_wmb(); /* See comment in pmd_install() */
pmd_populate_kernel(&init_mm, pmd, new);
new = NULL;
}
struct page *page;
page = vm_normal_page(vma, addr, ptent);
- if (unlikely(details) && page) {
- /*
- * unmap_shared_mapping_pages() wants to
- * invalidate cache without truncating:
- * unmap shared but keep private pages.
- */
- if (details->check_mapping &&
- details->check_mapping != page_rmapping(page))
- continue;
- }
+ if (unlikely(zap_skip_check_mapping(details, page)))
+ continue;
ptent = ptep_get_and_clear_full(mm, addr, pte,
tlb->fullmm);
tlb_remove_tlb_entry(tlb, pte, addr);
is_device_exclusive_entry(entry)) {
struct page *page = pfn_swap_entry_to_page(entry);
- if (unlikely(details && details->check_mapping)) {
- /*
- * unmap_shared_mapping_pages() wants to
- * invalidate cache without truncating:
- * unmap shared but keep private pages.
- */
- if (details->check_mapping !=
- page_rmapping(page))
- continue;
- }
-
+ if (unlikely(zap_skip_check_mapping(details, page)))
+ continue;
pte_clear_not_present_full(mm, addr, pte, tlb->fullmm);
rss[mm_counter(page)]--;
* proceeding (but do_wp_page is only called after already making such a check;
* and do_anonymous_page can safely check later on).
*/
-static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd,
- pte_t *page_table, pte_t orig_pte)
+static inline int pte_unmap_same(struct vm_fault *vmf)
{
int same = 1;
#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION)
if (sizeof(pte_t) > sizeof(unsigned long)) {
- spinlock_t *ptl = pte_lockptr(mm, pmd);
+ spinlock_t *ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(ptl);
- same = pte_same(*page_table, orig_pte);
+ same = pte_same(*vmf->pte, vmf->orig_pte);
spin_unlock(ptl);
}
#endif
- pte_unmap(page_table);
+ pte_unmap(vmf->pte);
+ vmf->pte = NULL;
return same;
}
}
static inline void unmap_mapping_range_tree(struct rb_root_cached *root,
+ pgoff_t first_index,
+ pgoff_t last_index,
struct zap_details *details)
{
struct vm_area_struct *vma;
pgoff_t vba, vea, zba, zea;
- vma_interval_tree_foreach(vma, root,
- details->first_index, details->last_index) {
-
+ vma_interval_tree_foreach(vma, root, first_index, last_index) {
vba = vma->vm_pgoff;
vea = vba + vma_pages(vma) - 1;
- zba = details->first_index;
+ zba = first_index;
if (zba < vba)
zba = vba;
- zea = details->last_index;
+ zea = last_index;
if (zea > vea)
zea = vea;
{
struct address_space *mapping = page->mapping;
struct zap_details details = { };
+ pgoff_t first_index;
+ pgoff_t last_index;
VM_BUG_ON(!PageLocked(page));
VM_BUG_ON(PageTail(page));
- details.check_mapping = mapping;
- details.first_index = page->index;
- details.last_index = page->index + thp_nr_pages(page) - 1;
+ first_index = page->index;
+ last_index = page->index + thp_nr_pages(page) - 1;
+
+ details.zap_mapping = mapping;
details.single_page = page;
i_mmap_lock_write(mapping);
if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
- unmap_mapping_range_tree(&mapping->i_mmap, &details);
+ unmap_mapping_range_tree(&mapping->i_mmap, first_index,
+ last_index, &details);
i_mmap_unlock_write(mapping);
}
pgoff_t nr, bool even_cows)
{
struct zap_details details = { };
+ pgoff_t first_index = start;
+ pgoff_t last_index = start + nr - 1;
- details.check_mapping = even_cows ? NULL : mapping;
- details.first_index = start;
- details.last_index = start + nr - 1;
- if (details.last_index < details.first_index)
- details.last_index = ULONG_MAX;
+ details.zap_mapping = even_cows ? NULL : mapping;
+ if (last_index < first_index)
+ last_index = ULONG_MAX;
i_mmap_lock_write(mapping);
if (unlikely(!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root)))
- unmap_mapping_range_tree(&mapping->i_mmap, &details);
+ unmap_mapping_range_tree(&mapping->i_mmap, first_index,
+ last_index, &details);
i_mmap_unlock_write(mapping);
}
EXPORT_SYMBOL_GPL(unmap_mapping_pages);
vm_fault_t ret = 0;
void *shadow = NULL;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
+ if (!pte_unmap_same(vmf))
goto out;
entry = pte_to_swp_entry(vmf->orig_pte);
vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
if (!vmf->prealloc_pte)
return VM_FAULT_OOM;
- smp_wmb(); /* See comment in __pte_alloc() */
}
ret = vma->vm_ops->fault(vmf);
vmf->prealloc_pte = pte_alloc_one(vma->vm_mm);
if (!vmf->prealloc_pte)
return VM_FAULT_OOM;
- smp_wmb(); /* See comment in __pte_alloc() */
}
vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
return ret;
}
- if (vmf->prealloc_pte) {
- vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
- if (likely(pmd_none(*vmf->pmd))) {
- mm_inc_nr_ptes(vma->vm_mm);
- pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
- vmf->prealloc_pte = NULL;
- }
- spin_unlock(vmf->ptl);
- } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) {
+ if (vmf->prealloc_pte)
+ pmd_install(vma->vm_mm, vmf->pmd, &vmf->prealloc_pte);
+ else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd)))
return VM_FAULT_OOM;
- }
}
/* See comment in handle_pte_fault() */
vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
if (!vmf->prealloc_pte)
return VM_FAULT_OOM;
- smp_wmb(); /* See comment in __pte_alloc() */
}
return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
if (!new)
return -ENOMEM;
- smp_wmb(); /* See comment in __pte_alloc */
-
spin_lock(&mm->page_table_lock);
- if (pgd_present(*pgd)) /* Another has populated it */
+ if (pgd_present(*pgd)) { /* Another has populated it */
p4d_free(mm, new);
- else
+ } else {
+ smp_wmb(); /* See comment in pmd_install() */
pgd_populate(mm, pgd, new);
+ }
spin_unlock(&mm->page_table_lock);
return 0;
}
if (!new)
return -ENOMEM;
- smp_wmb(); /* See comment in __pte_alloc */
-
spin_lock(&mm->page_table_lock);
if (!p4d_present(*p4d)) {
mm_inc_nr_puds(mm);
+ smp_wmb(); /* See comment in pmd_install() */
p4d_populate(mm, p4d, new);
} else /* Another has populated it */
pud_free(mm, new);
if (!new)
return -ENOMEM;
- smp_wmb(); /* See comment in __pte_alloc */
-
ptl = pud_lock(mm, pud);
if (!pud_present(*pud)) {
mm_inc_nr_pmds(mm);
+ smp_wmb(); /* See comment in pmd_install() */
pud_populate(mm, pud, new);
- } else /* Another has populated it */
+ } else { /* Another has populated it */
pmd_free(mm, new);
+ }
spin_unlock(ptl);
return 0;
}
unsigned int pages_per_huge_page,
bool allow_pagefault)
{
- void *src = (void *)usr_src;
void *page_kaddr;
unsigned long i, rc = 0;
unsigned long ret_val = pages_per_huge_page * PAGE_SIZE;
else
page_kaddr = kmap_atomic(subpage);
rc = copy_from_user(page_kaddr,
- (const void __user *)(src + i * PAGE_SIZE),
- PAGE_SIZE);
+ usr_src + i * PAGE_SIZE, PAGE_SIZE);
if (allow_pagefault)
kunmap(subpage);
else
#include <linux/memory.h>
#include <linux/memremap.h>
#include <linux/memory_hotplug.h>
-#include <linux/highmem.h>
#include <linux/vmalloc.h>
#include <linux/ioport.h>
#include <linux/delay.h>
ONLINE_POLICY_AUTO_MOVABLE,
};
-const char *online_policy_to_str[] = {
+static const char * const online_policy_to_str[] = {
[ONLINE_POLICY_CONTIG_ZONES] = "contig-zones",
[ONLINE_POLICY_AUTO_MOVABLE] = "auto-movable",
};
kfree(res);
}
-#ifdef CONFIG_MEMORY_HOTPLUG_SPARSE
static int check_pfn_span(unsigned long pfn, unsigned long nr_pages,
const char *reason)
{
debug_pagealloc_map_pages(page, 1 << order);
__free_pages_core(page, order);
totalram_pages_add(1UL << order);
-#ifdef CONFIG_HIGHMEM
- if (PageHighMem(page))
- totalhigh_pages_add(1UL << order);
-#endif
}
EXPORT_SYMBOL_GPL(generic_online_page);
arg->status_change_nid = NUMA_NO_NODE;
arg->status_change_nid_normal = NUMA_NO_NODE;
- arg->status_change_nid_high = NUMA_NO_NODE;
if (!node_state(nid, N_MEMORY))
arg->status_change_nid = nid;
if (zone_idx(zone) <= ZONE_NORMAL && !node_state(nid, N_NORMAL_MEMORY))
arg->status_change_nid_normal = nid;
-#ifdef CONFIG_HIGHMEM
- if (zone_idx(zone) <= ZONE_HIGHMEM && !node_state(nid, N_HIGH_MEMORY))
- arg->status_change_nid_high = nid;
-#endif
}
static void node_states_set_node(int node, struct memory_notify *arg)
if (arg->status_change_nid_normal >= 0)
node_set_state(node, N_NORMAL_MEMORY);
- if (arg->status_change_nid_high >= 0)
- node_set_state(node, N_HIGH_MEMORY);
-
if (arg->status_change_nid >= 0)
node_set_state(node, N_MEMORY);
}
mem_hotplug_done();
return ret;
}
-#endif /* CONFIG_MEMORY_HOTPLUG_SPARSE */
static void reset_node_present_pages(pg_data_t *pgdat)
{
int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)
{
struct mhp_params params = { .pgprot = pgprot_mhp(PAGE_KERNEL) };
+ enum memblock_flags memblock_flags = MEMBLOCK_NONE;
struct vmem_altmap mhp_altmap = {};
struct memory_group *group = NULL;
u64 start, size;
mem_hotplug_begin();
- if (IS_ENABLED(CONFIG_ARCH_KEEP_MEMBLOCK))
- memblock_add_node(start, size, nid);
+ if (IS_ENABLED(CONFIG_ARCH_KEEP_MEMBLOCK)) {
+ if (res->flags & IORESOURCE_SYSRAM_DRIVER_MANAGED)
+ memblock_flags = MEMBLOCK_DRIVER_MANAGED;
+ ret = memblock_add_node(start, size, nid, memblock_flags);
+ if (ret)
+ goto error_mem_hotplug_end;
+ }
ret = __try_online_node(nid, false);
if (ret < 0)
rollback_node_hotadd(nid);
if (IS_ENABLED(CONFIG_ARCH_KEEP_MEMBLOCK))
memblock_remove(start, size);
+error_mem_hotplug_end:
mem_hotplug_done();
return ret;
}
arg->status_change_nid = NUMA_NO_NODE;
arg->status_change_nid_normal = NUMA_NO_NODE;
- arg->status_change_nid_high = NUMA_NO_NODE;
/*
* Check whether node_states[N_NORMAL_MEMORY] will be changed.
if (zone_idx(zone) <= ZONE_NORMAL && nr_pages >= present_pages)
arg->status_change_nid_normal = zone_to_nid(zone);
-#ifdef CONFIG_HIGHMEM
/*
- * node_states[N_HIGH_MEMORY] contains nodes which
- * have normal memory or high memory.
- * Here we add the present_pages belonging to ZONE_HIGHMEM.
- * If the zone is within the range of [0..ZONE_HIGHMEM), and
- * we determine that the zones in that range become empty,
- * we need to clear the node for N_HIGH_MEMORY.
- */
- present_pages += pgdat->node_zones[ZONE_HIGHMEM].present_pages;
- if (zone_idx(zone) <= ZONE_HIGHMEM && nr_pages >= present_pages)
- arg->status_change_nid_high = zone_to_nid(zone);
-#endif
-
- /*
- * We have accounted the pages from [0..ZONE_NORMAL), and
- * in case of CONFIG_HIGHMEM the pages from ZONE_HIGHMEM
- * as well.
+ * We have accounted the pages from [0..ZONE_NORMAL); ZONE_HIGHMEM
+ * does not apply as we don't support 32bit.
* Here we count the possible pages from ZONE_MOVABLE.
* If after having accounted all the pages, we see that the nr_pages
* to be offlined is over or equal to the accounted pages,
if (arg->status_change_nid_normal >= 0)
node_clear_state(node, N_NORMAL_MEMORY);
- if (arg->status_change_nid_high >= 0)
- node_clear_state(node, N_HIGH_MEMORY);
-
if (arg->status_change_nid >= 0)
node_clear_state(node, N_MEMORY);
}
arch_remove_memory(start, size, altmap);
if (IS_ENABLED(CONFIG_ARCH_KEEP_MEMBLOCK)) {
- memblock_free(start, size);
+ memblock_phys_free(start, size);
memblock_remove(start, size);
}
}
EXPORT_SYMBOL(folio_alloc);
+static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp,
+ struct mempolicy *pol, unsigned long nr_pages,
+ struct page **page_array)
+{
+ int nodes;
+ unsigned long nr_pages_per_node;
+ int delta;
+ int i;
+ unsigned long nr_allocated;
+ unsigned long total_allocated = 0;
+
+ nodes = nodes_weight(pol->nodes);
+ nr_pages_per_node = nr_pages / nodes;
+ delta = nr_pages - nodes * nr_pages_per_node;
+
+ for (i = 0; i < nodes; i++) {
+ if (delta) {
+ nr_allocated = __alloc_pages_bulk(gfp,
+ interleave_nodes(pol), NULL,
+ nr_pages_per_node + 1, NULL,
+ page_array);
+ delta--;
+ } else {
+ nr_allocated = __alloc_pages_bulk(gfp,
+ interleave_nodes(pol), NULL,
+ nr_pages_per_node, NULL, page_array);
+ }
+
+ page_array += nr_allocated;
+ total_allocated += nr_allocated;
+ }
+
+ return total_allocated;
+}
+
+static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
+ struct mempolicy *pol, unsigned long nr_pages,
+ struct page **page_array)
+{
+ gfp_t preferred_gfp;
+ unsigned long nr_allocated = 0;
+
+ preferred_gfp = gfp | __GFP_NOWARN;
+ preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
+
+ nr_allocated = __alloc_pages_bulk(preferred_gfp, nid, &pol->nodes,
+ nr_pages, NULL, page_array);
+
+ if (nr_allocated < nr_pages)
+ nr_allocated += __alloc_pages_bulk(gfp, numa_node_id(), NULL,
+ nr_pages - nr_allocated, NULL,
+ page_array + nr_allocated);
+ return nr_allocated;
+}
+
+/* alloc pages bulk and mempolicy should be considered at the
+ * same time in some situation such as vmalloc.
+ *
+ * It can accelerate memory allocation especially interleaving
+ * allocate memory.
+ */
+unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp,
+ unsigned long nr_pages, struct page **page_array)
+{
+ struct mempolicy *pol = &default_policy;
+
+ if (!in_interrupt() && !(gfp & __GFP_THISNODE))
+ pol = get_task_policy(current);
+
+ if (pol->mode == MPOL_INTERLEAVE)
+ return alloc_pages_bulk_array_interleave(gfp, pol,
+ nr_pages, page_array);
+
+ if (pol->mode == MPOL_PREFERRED_MANY)
+ return alloc_pages_bulk_array_preferred_many(gfp,
+ numa_node_id(), pol, nr_pages, page_array);
+
+ return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()),
+ policy_nodemask(gfp, pol), nr_pages, NULL,
+ page_array);
+}
+
int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
{
struct mempolicy *pol = mpol_dup(vma_policy(src));
p += scnprintf(p, buffer + maxlen - p, ":%*pbl",
nodemask_pr_args(&nodes));
}
-
-bool numa_demotion_enabled = false;
-
-#ifdef CONFIG_SYSFS
-static ssize_t numa_demotion_enabled_show(struct kobject *kobj,
- struct kobj_attribute *attr, char *buf)
-{
- return sysfs_emit(buf, "%s\n",
- numa_demotion_enabled? "true" : "false");
-}
-
-static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
- struct kobj_attribute *attr,
- const char *buf, size_t count)
-{
- if (!strncmp(buf, "true", 4) || !strncmp(buf, "1", 1))
- numa_demotion_enabled = true;
- else if (!strncmp(buf, "false", 5) || !strncmp(buf, "0", 1))
- numa_demotion_enabled = false;
- else
- return -EINVAL;
-
- return count;
-}
-
-static struct kobj_attribute numa_demotion_enabled_attr =
- __ATTR(demotion_enabled, 0644, numa_demotion_enabled_show,
- numa_demotion_enabled_store);
-
-static struct attribute *numa_attrs[] = {
- &numa_demotion_enabled_attr.attr,
- NULL,
-};
-
-static const struct attribute_group numa_attr_group = {
- .attrs = numa_attrs,
-};
-
-static int __init numa_init_sysfs(void)
-{
- int err;
- struct kobject *numa_kobj;
-
- numa_kobj = kobject_create_and_add("numa", mm_kobj);
- if (!numa_kobj) {
- pr_err("failed to create numa kobject\n");
- return -ENOMEM;
- }
- err = sysfs_create_group(numa_kobj, &numa_attr_group);
- if (err) {
- pr_err("failed to register numa group\n");
- goto delete_obj;
- }
- return 0;
-
-delete_obj:
- kobject_put(numa_kobj);
- return err;
-}
-subsys_initcall(numa_init_sysfs);
-#endif
newzone = folio_zone(newfolio);
xas_lock_irq(&xas);
- if (folio_ref_count(folio) != expected_count ||
- xas_load(&xas) != folio) {
- xas_unlock_irq(&xas);
- return -EAGAIN;
- }
-
if (!folio_ref_freeze(folio, expected_count)) {
xas_unlock_irq(&xas);
return -EAGAIN;
* can't be dropped from it).
*/
get_page(page);
- migrate->cpages++;
/*
* Optimize for the common case where page is only mapped once
if (trylock_page(page)) {
pte_t swp_pte;
- mpfn |= MIGRATE_PFN_LOCKED;
+ migrate->cpages++;
ptep_get_and_clear(mm, addr, ptep);
/* Setup special migration page table entry */
if (pte_present(pte))
unmapped++;
+ } else {
+ put_page(page);
+ mpfn = 0;
}
next:
}
/*
- * migrate_vma_prepare() - lock pages and isolate them from the lru
+ * migrate_vma_unmap() - replace page mapping with special migration pte entry
* @migrate: migrate struct containing all migration information
*
- * This locks pages that have been collected by migrate_vma_collect(). Once each
- * page is locked it is isolated from the lru (for non-device pages). Finally,
- * the ref taken by migrate_vma_collect() is dropped, as locked pages cannot be
- * migrated by concurrent kernel threads.
+ * Isolate pages from the LRU and replace mappings (CPU page table pte) with a
+ * special migration pte entry and check if it has been pinned. Pinned pages are
+ * restored because we cannot migrate them.
+ *
+ * This is the last step before we call the device driver callback to allocate
+ * destination memory and copy contents of original page over to new page.
*/
-static void migrate_vma_prepare(struct migrate_vma *migrate)
+static void migrate_vma_unmap(struct migrate_vma *migrate)
{
const unsigned long npages = migrate->npages;
const unsigned long start = migrate->start;
lru_add_drain();
- for (i = 0; (i < npages) && migrate->cpages; i++) {
+ for (i = 0; i < npages; i++) {
struct page *page = migrate_pfn_to_page(migrate->src[i]);
- bool remap = true;
if (!page)
continue;
- if (!(migrate->src[i] & MIGRATE_PFN_LOCKED)) {
- /*
- * Because we are migrating several pages there can be
- * a deadlock between 2 concurrent migration where each
- * are waiting on each other page lock.
- *
- * Make migrate_vma() a best effort thing and backoff
- * for any page we can not lock right away.
- */
- if (!trylock_page(page)) {
- migrate->src[i] = 0;
- migrate->cpages--;
- put_page(page);
- continue;
- }
- remap = false;
- migrate->src[i] |= MIGRATE_PFN_LOCKED;
- }
-
/* ZONE_DEVICE pages are not on LRU */
if (!is_zone_device_page(page)) {
if (!PageLRU(page) && allow_drain) {
}
if (isolate_lru_page(page)) {
- if (remap) {
- migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
- migrate->cpages--;
- restore++;
- } else {
- migrate->src[i] = 0;
- unlock_page(page);
- migrate->cpages--;
- put_page(page);
- }
+ migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
+ migrate->cpages--;
+ restore++;
continue;
}
put_page(page);
}
- if (!migrate_vma_check_page(page)) {
- if (remap) {
- migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
- migrate->cpages--;
- restore++;
-
- if (!is_zone_device_page(page)) {
- get_page(page);
- putback_lru_page(page);
- }
- } else {
- migrate->src[i] = 0;
- unlock_page(page);
- migrate->cpages--;
+ if (page_mapped(page))
+ try_to_migrate(page, 0);
- if (!is_zone_device_page(page))
- putback_lru_page(page);
- else
- put_page(page);
+ if (page_mapped(page) || !migrate_vma_check_page(page)) {
+ if (!is_zone_device_page(page)) {
+ get_page(page);
+ putback_lru_page(page);
}
- }
- }
-
- for (i = 0, addr = start; i < npages && restore; i++, addr += PAGE_SIZE) {
- struct page *page = migrate_pfn_to_page(migrate->src[i]);
-
- if (!page || (migrate->src[i] & MIGRATE_PFN_MIGRATE))
- continue;
-
- remove_migration_pte(page, migrate->vma, addr, page);
-
- migrate->src[i] = 0;
- unlock_page(page);
- put_page(page);
- restore--;
- }
-}
-
-/*
- * migrate_vma_unmap() - replace page mapping with special migration pte entry
- * @migrate: migrate struct containing all migration information
- *
- * Replace page mapping (CPU page table pte) with a special migration pte entry
- * and check again if it has been pinned. Pinned pages are restored because we
- * cannot migrate them.
- *
- * This is the last step before we call the device driver callback to allocate
- * destination memory and copy contents of original page over to new page.
- */
-static void migrate_vma_unmap(struct migrate_vma *migrate)
-{
- const unsigned long npages = migrate->npages;
- const unsigned long start = migrate->start;
- unsigned long addr, i, restore = 0;
- for (i = 0; i < npages; i++) {
- struct page *page = migrate_pfn_to_page(migrate->src[i]);
-
- if (!page || !(migrate->src[i] & MIGRATE_PFN_MIGRATE))
+ migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
+ migrate->cpages--;
+ restore++;
continue;
-
- if (page_mapped(page)) {
- try_to_migrate(page, 0);
- if (page_mapped(page))
- goto restore;
}
-
- if (migrate_vma_check_page(page))
- continue;
-
-restore:
- migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
- migrate->cpages--;
- restore++;
}
for (addr = start, i = 0; i < npages && restore; addr += PAGE_SIZE, i++) {
migrate->src[i] = 0;
unlock_page(page);
+ put_page(page);
restore--;
-
- if (is_zone_device_page(page))
- put_page(page);
- else
- putback_lru_page(page);
}
}
* it for all those entries (ie with MIGRATE_PFN_VALID and MIGRATE_PFN_MIGRATE
* flag set). Once these are allocated and copied, the caller must update each
* corresponding entry in the dst array with the pfn value of the destination
- * page and with the MIGRATE_PFN_VALID and MIGRATE_PFN_LOCKED flags set
- * (destination pages must have their struct pages locked, via lock_page()).
+ * page and with MIGRATE_PFN_VALID. Destination pages must be locked via
+ * lock_page().
*
* Note that the caller does not have to migrate all the pages that are marked
* with MIGRATE_PFN_MIGRATE flag in src array unless this is a migration from
migrate_vma_collect(args);
if (args->cpages)
- migrate_vma_prepare(args);
- if (args->cpages)
migrate_vma_unmap(args);
/*
}
late_initcall(migrate_on_reclaim_init);
#endif /* CONFIG_HOTPLUG_CPU */
+
+bool numa_demotion_enabled = false;
+
+#ifdef CONFIG_SYSFS
+static ssize_t numa_demotion_enabled_show(struct kobject *kobj,
+ struct kobj_attribute *attr, char *buf)
+{
+ return sysfs_emit(buf, "%s\n",
+ numa_demotion_enabled ? "true" : "false");
+}
+
+static ssize_t numa_demotion_enabled_store(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ const char *buf, size_t count)
+{
+ if (!strncmp(buf, "true", 4) || !strncmp(buf, "1", 1))
+ numa_demotion_enabled = true;
+ else if (!strncmp(buf, "false", 5) || !strncmp(buf, "0", 1))
+ numa_demotion_enabled = false;
+ else
+ return -EINVAL;
+
+ return count;
+}
+
+static struct kobj_attribute numa_demotion_enabled_attr =
+ __ATTR(demotion_enabled, 0644, numa_demotion_enabled_show,
+ numa_demotion_enabled_store);
+
+static struct attribute *numa_attrs[] = {
+ &numa_demotion_enabled_attr.attr,
+ NULL,
+};
+
+static const struct attribute_group numa_attr_group = {
+ .attrs = numa_attrs,
+};
+
+static int __init numa_init_sysfs(void)
+{
+ int err;
+ struct kobject *numa_kobj;
+
+ numa_kobj = kobject_create_and_add("numa", mm_kobj);
+ if (!numa_kobj) {
+ pr_err("failed to create numa kobject\n");
+ return -ENOMEM;
+ }
+ err = sysfs_create_group(numa_kobj, &numa_attr_group);
+ if (err) {
+ pr_err("failed to register numa group\n");
+ goto delete_obj;
+ }
+ return 0;
+
+delete_obj:
+ kobject_put(numa_kobj);
+ return err;
+}
+subsys_initcall(numa_init_sysfs);
+#endif
goto out_fput;
}
} else if (flags & MAP_HUGETLB) {
- struct ucounts *ucounts = NULL;
struct hstate *hs;
hs = hstate_sizelog((flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK);
*/
file = hugetlb_file_setup(HUGETLB_ANON_FILE, len,
VM_NORESERVE,
- &ucounts, HUGETLB_ANONHUGE_INODE,
+ HUGETLB_ANONHUGE_INODE,
(flags >> MAP_HUGE_SHIFT) & MAP_HUGE_MASK);
if (IS_ERR(file))
return PTR_ERR(file);
void vm_stat_account(struct mm_struct *mm, vm_flags_t flags, long npages)
{
- mm->total_vm += npages;
+ WRITE_ONCE(mm->total_vm, READ_ONCE(mm->total_vm)+npages);
if (is_exec_mapping(flags))
mm->exec_vm += npages;
error = -ENOMEM;
if (!vma)
goto out;
- prev = vma->vm_prev;
+
if (unlikely(grows & PROT_GROWSDOWN)) {
if (vma->vm_start >= end)
goto out;
goto out;
}
}
+
if (start > vma->vm_start)
prev = vma;
+ else
+ prev = vma->vm_prev;
for (nstart = start ; ; ) {
unsigned long mask_off_old_flags;
old_end = old_addr + len;
flush_cache_range(vma, old_addr, old_end);
+ if (is_vm_hugetlb_page(vma))
+ return move_hugetlb_page_tables(vma, new_vma, old_addr,
+ new_addr, len);
+
mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma, vma->vm_mm,
old_addr, old_end);
mmu_notifier_invalidate_range_start(&range);
bool *locked, unsigned long flags,
struct vm_userfaultfd_ctx *uf, struct list_head *uf_unmap)
{
+ long to_account = new_len - old_len;
struct mm_struct *mm = vma->vm_mm;
struct vm_area_struct *new_vma;
unsigned long vm_flags = vma->vm_flags;
if (mm->map_count >= sysctl_max_map_count - 3)
return -ENOMEM;
+ if (unlikely(flags & MREMAP_DONTUNMAP))
+ to_account = new_len;
+
if (vma->vm_ops && vma->vm_ops->may_split) {
if (vma->vm_start != old_addr)
err = vma->vm_ops->may_split(vma, old_addr);
if (err)
return err;
- if (unlikely(flags & MREMAP_DONTUNMAP && vm_flags & VM_ACCOUNT)) {
- if (security_vm_enough_memory_mm(mm, new_len >> PAGE_SHIFT))
+ if (vm_flags & VM_ACCOUNT) {
+ if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT))
return -ENOMEM;
}
new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff,
&need_rmap_locks);
if (!new_vma) {
- if (unlikely(flags & MREMAP_DONTUNMAP && vm_flags & VM_ACCOUNT))
- vm_unacct_memory(new_len >> PAGE_SHIFT);
+ if (vm_flags & VM_ACCOUNT)
+ vm_unacct_memory(to_account >> PAGE_SHIFT);
return -ENOMEM;
}
mremap_userfaultfd_prep(new_vma, uf);
}
+ if (is_vm_hugetlb_page(vma)) {
+ clear_vma_resv_huge_pages(vma);
+ }
+
/* Conceal VM_ACCOUNT so old reservation is not undone */
if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) {
vma->vm_flags &= ~VM_ACCOUNT;
}
static struct vm_area_struct *vma_to_resize(unsigned long addr,
- unsigned long old_len, unsigned long new_len, unsigned long flags,
- unsigned long *p)
+ unsigned long old_len, unsigned long new_len, unsigned long flags)
{
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma;
(vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP)))
return ERR_PTR(-EINVAL);
- if (is_vm_hugetlb_page(vma))
- return ERR_PTR(-EINVAL);
-
/* We can't remap across vm area boundaries */
if (old_len > vma->vm_end - addr)
return ERR_PTR(-EFAULT);
(new_len - old_len) >> PAGE_SHIFT))
return ERR_PTR(-ENOMEM);
- if (vma->vm_flags & VM_ACCOUNT) {
- unsigned long charged = (new_len - old_len) >> PAGE_SHIFT;
- if (security_vm_enough_memory_mm(mm, charged))
- return ERR_PTR(-ENOMEM);
- *p = charged;
- }
-
return vma;
}
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma;
unsigned long ret = -EINVAL;
- unsigned long charged = 0;
unsigned long map_flags = 0;
if (offset_in_page(new_addr))
old_len = new_len;
}
- vma = vma_to_resize(addr, old_len, new_len, flags, &charged);
+ vma = vma_to_resize(addr, old_len, new_len, flags);
if (IS_ERR(vma)) {
ret = PTR_ERR(vma);
goto out;
((addr - vma->vm_start) >> PAGE_SHIFT),
map_flags);
if (IS_ERR_VALUE(ret))
- goto out1;
+ goto out;
/* We got a new mapping */
if (!(flags & MREMAP_FIXED))
ret = move_vma(vma, addr, old_len, new_len, new_addr, locked, flags, uf,
uf_unmap);
- if (!(offset_in_page(ret)))
- goto out;
-
-out1:
- vm_unacct_memory(charged);
-
out:
return ret;
}
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma;
unsigned long ret = -EINVAL;
- unsigned long charged = 0;
bool locked = false;
bool downgraded = false;
struct vm_userfaultfd_ctx uf = NULL_VM_UFFD_CTX;
if (mmap_write_lock_killable(current->mm))
return -EINTR;
+ vma = find_vma(mm, addr);
+ if (!vma || vma->vm_start > addr) {
+ ret = EFAULT;
+ goto out;
+ }
+
+ if (is_vm_hugetlb_page(vma)) {
+ struct hstate *h __maybe_unused = hstate_vma(vma);
+
+ old_len = ALIGN(old_len, huge_page_size(h));
+ new_len = ALIGN(new_len, huge_page_size(h));
+
+ /* addrs must be huge page aligned */
+ if (addr & ~huge_page_mask(h))
+ goto out;
+ if (new_addr & ~huge_page_mask(h))
+ goto out;
+
+ /*
+ * Don't allow remap expansion, because the underlying hugetlb
+ * reservation is not yet capable to handle split reservation.
+ */
+ if (new_len > old_len)
+ goto out;
+ }
if (flags & (MREMAP_FIXED | MREMAP_DONTUNMAP)) {
ret = mremap_to(addr, old_len, new_addr, new_len,
/*
* Ok, we need to grow..
*/
- vma = vma_to_resize(addr, old_len, new_len, flags, &charged);
+ vma = vma_to_resize(addr, old_len, new_len, flags);
if (IS_ERR(vma)) {
ret = PTR_ERR(vma);
goto out;
if (old_len == vma->vm_end - addr) {
/* can we just expand the current mapping? */
if (vma_expandable(vma, new_len - old_len)) {
- int pages = (new_len - old_len) >> PAGE_SHIFT;
+ long pages = (new_len - old_len) >> PAGE_SHIFT;
+
+ if (vma->vm_flags & VM_ACCOUNT) {
+ if (security_vm_enough_memory_mm(mm, pages)) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
if (vma_adjust(vma, vma->vm_start, addr + new_len,
vma->vm_pgoff, NULL)) {
+ vm_unacct_memory(pages);
ret = -ENOMEM;
goto out;
}
&locked, flags, &uf, &uf_unmap);
}
out:
- if (offset_in_page(ret)) {
- vm_unacct_memory(charged);
+ if (offset_in_page(ret))
locked = false;
- }
if (downgraded)
mmap_read_unlock(current->mm);
else
}
EXPORT_SYMBOL(remap_vmalloc_range);
-unsigned long arch_get_unmapped_area(struct file *file, unsigned long addr,
- unsigned long len, unsigned long pgoff, unsigned long flags)
-{
- return -ENOMEM;
-}
-
vm_fault_t filemap_fault(struct vm_fault *vmf)
{
BUG();
static int oom_reaper(void *unused)
{
+ set_freezable();
+
while (true) {
struct task_struct *tsk = NULL;
}
/*
- * The pagefault handler calls here because it is out of memory, so kill a
- * memory-hogging task. If oom_lock is held by somebody else, a parallel oom
- * killing is already in progress so do nothing.
+ * The pagefault handler calls here because some allocation has failed. We have
+ * to take care of the memcg OOM here because this is the only safe context without
+ * any locks held but let the oom killer triggered from the allocation context care
+ * about the global OOM.
*/
void pagefault_out_of_memory(void)
{
- struct oom_control oc = {
- .zonelist = NULL,
- .nodemask = NULL,
- .memcg = NULL,
- .gfp_mask = 0,
- .order = 0,
- };
+ static DEFINE_RATELIMIT_STATE(pfoom_rs, DEFAULT_RATELIMIT_INTERVAL,
+ DEFAULT_RATELIMIT_BURST);
if (mem_cgroup_oom_synchronize(true))
return;
- if (!mutex_trylock(&oom_lock))
+ if (fatal_signal_pending(current))
return;
- out_of_memory(&oc);
- mutex_unlock(&oom_lock);
+
+ if (__ratelimit(&pfoom_rs))
+ pr_warn("Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF\n");
}
SYSCALL_DEFINE2(process_mrelease, int, pidfd, unsigned int, flags)
struct task_struct *p;
unsigned int f_flags;
bool reap = false;
- struct pid *pid;
long ret = 0;
if (flags)
return -EINVAL;
- pid = pidfd_get_pid(pidfd, &f_flags);
- if (IS_ERR(pid))
- return PTR_ERR(pid);
-
- task = get_pid_task(pid, PIDTYPE_TGID);
- if (!task) {
- ret = -ESRCH;
- goto put_pid;
- }
+ task = pidfd_get_task(pidfd, &f_flags);
+ if (IS_ERR(task))
+ return PTR_ERR(task);
/*
* Make sure to choose a thread which still has a reference to mm
mmput(mm);
put_task:
put_task_struct(task);
-put_pid:
- put_pid(pid);
return ret;
#else
return -ENOSYS;
ret = generic_writepages(mapping, wbc);
if ((ret != -ENOMEM) || (wbc->sync_mode != WB_SYNC_ALL))
break;
- cond_resched();
- congestion_wait(BLK_RW_ASYNC, HZ/50);
+
+ /*
+ * Lacking an allocation context or the locality or writeback
+ * state of any of the inode's pages, throttle based on
+ * writeback activity on the local node. It's as good a
+ * guess as any.
+ */
+ reclaim_throttle(NODE_DATA(numa_node_id()),
+ VMSCAN_THROTTLE_WRITEBACK);
}
/*
* Usually few pages are written by now from those we've just submitted
*/
void folio_wait_stable(struct folio *folio)
{
- if (folio->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES)
+ if (folio_inode(folio)->i_sb->s_iflags & SB_I_STABLE_WRITES)
folio_wait_writeback(folio);
}
EXPORT_SYMBOL_GPL(folio_wait_stable);
int order = pindex / MIGRATE_PCPTYPES;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
- if (order > PAGE_ALLOC_COSTLY_ORDER) {
+ if (order > PAGE_ALLOC_COSTLY_ORDER)
order = pageblock_order;
- VM_BUG_ON(order != pageblock_order);
- }
#else
VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
#endif
/*
* Frees a number of pages from the PCP lists
- * Assumes all pages on list are in same zone, and of same order.
+ * Assumes all pages on list are in same zone.
* count is the number of pages to free.
- *
- * If the zone was previously in an "all pages pinned" state then look to
- * see if this freeing clears that state.
- *
- * And clear the zone's pages_scanned counter, to hold off the "all pages are
- * pinned" detection logic.
*/
static void free_pcppages_bulk(struct zone *zone, int count,
struct per_cpu_pages *pcp)
for (zid = 0; zid < MAX_NR_ZONES; zid++) {
struct zone *zone = &pgdat->node_zones[zid];
- if (pfn >= zone->zone_start_pfn && pfn < zone_end_pfn(zone))
+ if (zone_spans_pfn(zone, pfn))
break;
}
__init_single_page(pfn_to_page(pfn), pfn, zid, nid);
* cpu which is alright but we also have to make sure to not move to
* a different one.
*/
- preempt_disable();
+ migrate_disable();
drain_local_pages(drain->zone);
- preempt_enable();
+ migrate_enable();
}
/*
}
#ifdef CONFIG_NUMA
+int __read_mostly node_reclaim_distance = RECLAIM_DISTANCE;
+
static bool zone_allows_reclaim(struct zone *local_zone, struct zone *zone)
{
return node_distance(zone_to_nid(local_zone), zone_to_nid(zone)) <=
trace_reclaim_retry_zone(z, order, reclaimable,
available, min_wmark, *no_progress_loops, wmark);
if (wmark) {
- /*
- * If we didn't make any progress and have a lot of
- * dirty + writeback pages then we should wait for
- * an IO to complete to slow down the reclaim and
- * prevent from pre mature OOM
- */
- if (!did_some_progress) {
- unsigned long write_pending;
-
- write_pending = zone_page_state_snapshot(zone,
- NR_ZONE_WRITE_PENDING);
-
- if (2 * write_pending > reclaimable) {
- congestion_wait(BLK_RW_ASYNC, HZ/10);
- return true;
- }
- }
-
ret = true;
- goto out;
+ break;
}
}
-out:
/*
* Memory allocation/reclaim might be called from a WQ context and the
* current implementation of the WQ concurrency control doesn't
if (!ac->preferred_zoneref->zone)
goto nopage;
+ /*
+ * Check for insane configurations where the cpuset doesn't contain
+ * any suitable zone to satisfy the request - e.g. non-movable
+ * GFP_HIGHUSER allocations from MOVABLE nodes only.
+ */
+ if (cpusets_insane_config() && (gfp_mask & __GFP_HARDWALL)) {
+ struct zoneref *z = first_zones_zonelist(ac->zonelist,
+ ac->highest_zoneidx,
+ &cpuset_current_mems_allowed);
+ if (!z->zone)
+ goto nopage;
+ }
+
if (alloc_flags & ALLOC_KSWAPD)
wake_all_kswapds(order, gfp_mask, ac);
unsigned int order = get_order(size);
unsigned long addr;
- if (WARN_ON_ONCE(gfp_mask & __GFP_COMP))
- gfp_mask &= ~__GFP_COMP;
+ if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM)))
+ gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM);
addr = __get_free_pages(gfp_mask, order);
return make_alloc_exact(addr, order, size);
unsigned int order = get_order(size);
struct page *p;
- if (WARN_ON_ONCE(gfp_mask & __GFP_COMP))
- gfp_mask &= ~__GFP_COMP;
+ if (WARN_ON_ONCE(gfp_mask & (__GFP_COMP | __GFP_HIGHMEM)))
+ gfp_mask &= ~(__GFP_COMP | __GFP_HIGHMEM);
p = alloc_pages_node(nid, gfp_mask, order);
if (!p)
printk(KERN_CONT
"%s"
" free:%lukB"
+ " boost:%lukB"
" min:%lukB"
" low:%lukB"
" high:%lukB"
"\n",
zone->name,
K(zone_page_state(zone, NR_FREE_PAGES)),
+ K(zone->watermark_boost),
K(min_wmark_pages(zone)),
K(low_wmark_pages(zone)),
K(high_wmark_pages(zone)),
*/
if (node_distance(local_node, node) !=
node_distance(local_node, prev_node))
- node_load[node] = load;
+ node_load[node] += load;
node_order[nr_nodes++] = node;
prev_node = node;
build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
build_thisnode_zonelists(pgdat);
+ pr_info("Fallback order for Node %d: ", local_node);
+ for (node = 0; node < nr_nodes; node++)
+ pr_cont("%d ", node_order[node]);
+ pr_cont("\n");
}
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
{
+ int i;
+
pgdat_resize_init(pgdat);
pgdat_init_split_queue(pgdat);
init_waitqueue_head(&pgdat->kswapd_wait);
init_waitqueue_head(&pgdat->pfmemalloc_wait);
+ for (i = 0; i < NR_VMSCAN_THROTTLE; i++)
+ init_waitqueue_head(&pgdat->reclaim_wait[i]);
+
pgdat_page_ext_init(pgdat);
lruvec_init(&pgdat->__lruvec);
}
}
if (pages && s)
- pr_info("Freeing %s memory: %ldK\n",
- s, pages << (PAGE_SHIFT - 10));
+ pr_info("Freeing %s memory: %ldK\n", s, K(pages));
return pages;
}
", %luK highmem"
#endif
")\n",
- nr_free_pages() << (PAGE_SHIFT - 10),
- physpages << (PAGE_SHIFT - 10),
+ K(nr_free_pages()), K(physpages),
codesize >> 10, datasize >> 10, rosize >> 10,
(init_data_size + init_code_size) >> 10, bss_size >> 10,
- (physpages - totalram_pages() - totalcma_pages) << (PAGE_SHIFT - 10),
- totalcma_pages << (PAGE_SHIFT - 10)
+ K(physpages - totalram_pages() - totalcma_pages),
+ K(totalcma_pages)
#ifdef CONFIG_HIGHMEM
- , totalhigh_pages() << (PAGE_SHIFT - 10)
+ , K(totalhigh_pages())
#endif
);
}
* 8192MB: 11584k
* 16384MB: 16384k
*/
-int __meminit init_per_zone_wmark_min(void)
+void calculate_min_free_kbytes(void)
{
unsigned long lowmem_kbytes;
int new_min_free_kbytes;
lowmem_kbytes = nr_free_buffer_pages() * (PAGE_SIZE >> 10);
new_min_free_kbytes = int_sqrt(lowmem_kbytes * 16);
- if (new_min_free_kbytes > user_min_free_kbytes) {
- min_free_kbytes = new_min_free_kbytes;
- if (min_free_kbytes < 128)
- min_free_kbytes = 128;
- if (min_free_kbytes > 262144)
- min_free_kbytes = 262144;
- } else {
+ if (new_min_free_kbytes > user_min_free_kbytes)
+ min_free_kbytes = clamp(new_min_free_kbytes, 128, 262144);
+ else
pr_warn("min_free_kbytes is not updated to %d because user defined value %d is preferred\n",
new_min_free_kbytes, user_min_free_kbytes);
- }
+
+}
+
+int __meminit init_per_zone_wmark_min(void)
+{
+ calculate_min_free_kbytes();
setup_per_zone_wmarks();
refresh_zone_stat_thresholds();
setup_per_zone_lowmem_reserve();
} else if (get_order(size) >= MAX_ORDER || hashdist) {
table = __vmalloc(size, gfp_flags);
virt = true;
- huge = is_vm_area_hugepages(table);
+ if (table)
+ huge = is_vm_area_hugepages(table);
} else {
/*
* If bucketsize is not a power-of-two, we may free
}
#endif
+/*
+ * This function returns a stable result only if called under zone lock.
+ */
bool is_free_buddy_page(struct page *page)
{
- struct zone *zone = page_zone(page);
unsigned long pfn = page_to_pfn(page);
- unsigned long flags;
unsigned int order;
- spin_lock_irqsave(&zone->lock, flags);
for (order = 0; order < MAX_ORDER; order++) {
struct page *page_head = page - (pfn & ((1 << order) - 1));
- if (PageBuddy(page_head) && buddy_order(page_head) >= order)
+ if (PageBuddy(page_head) &&
+ buddy_order_unsafe(page_head) >= order)
break;
}
- spin_unlock_irqrestore(&zone->lock, flags);
return order < MAX_ORDER;
}
panic("Out of memory");
}
-#else /* CONFIG_FLATMEM */
+#else /* CONFIG_SPARSEMEM */
struct page_ext *lookup_page_ext(const struct page *page)
{
buddy = page + (buddy_pfn - pfn);
if (!is_migrate_isolate_page(buddy)) {
- __isolate_free_page(page, order);
- isolated_page = true;
+ isolated_page = !!__isolate_free_page(page, order);
+ /*
+ * Isolating a free page in an isolated pageblock
+ * is expected to always work as watermarks don't
+ * apply here.
+ */
+ VM_WARN_ON(!isolated_page);
}
}
}
unsigned migratetype, int flags)
{
unsigned long pfn;
- unsigned long undo_pfn;
struct page *page;
BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages));
pfn < end_pfn;
pfn += pageblock_nr_pages) {
page = __first_valid_page(pfn, pageblock_nr_pages);
- if (page) {
- if (set_migratetype_isolate(page, migratetype, flags)) {
- undo_pfn = pfn;
- goto undo;
- }
+ if (page && set_migratetype_isolate(page, migratetype, flags)) {
+ undo_isolate_page_range(start_pfn, pfn, migratetype);
+ return -EBUSY;
}
}
return 0;
-undo:
- for (pfn = start_pfn;
- pfn < undo_pfn;
- pfn += pageblock_nr_pages) {
- struct page *page = pfn_to_online_page(pfn);
- if (!page)
- continue;
- unset_migratetype_isolate(page, migratetype);
- }
-
- return -EBUSY;
}
/*
return handle;
}
-void __reset_page_owner(struct page *page, unsigned int order)
+void __reset_page_owner(struct page *page, unsigned short order)
{
int i;
struct page_ext *page_ext;
static inline void __set_page_owner_handle(struct page_ext *page_ext,
depot_stack_handle_t handle,
- unsigned int order, gfp_t gfp_mask)
+ unsigned short order, gfp_t gfp_mask)
{
struct page_owner *page_owner;
int i;
}
}
-noinline void __set_page_owner(struct page *page, unsigned int order,
+noinline void __set_page_owner(struct page *page, unsigned short order,
gfp_t gfp_mask)
{
struct page_ext *page_ext = lookup_page_ext(page);
depot_stack_handle_t handle)
{
int ret, pageblock_mt, page_mt;
- unsigned long *entries;
- unsigned int nr_entries;
char *kbuf;
count = min_t(size_t, count, PAGE_SIZE);
if (ret >= count)
goto err;
- nr_entries = stack_depot_fetch(handle, &entries);
- ret += stack_trace_snprint(kbuf + ret, count - ret, entries, nr_entries, 0);
+ ret += stack_depot_snprint(handle, kbuf + ret, count - ret, 0);
if (ret >= count)
goto err;
struct page_ext *page_ext = lookup_page_ext(page);
struct page_owner *page_owner;
depot_stack_handle_t handle;
- unsigned long *entries;
- unsigned int nr_entries;
gfp_t gfp_mask;
int mt;
page_owner->pid, page_owner->ts_nsec, page_owner->free_ts_nsec);
handle = READ_ONCE(page_owner->handle);
- if (!handle) {
+ if (!handle)
pr_alert("page_owner allocation stack trace missing\n");
- } else {
- nr_entries = stack_depot_fetch(handle, &entries);
- stack_trace_print(entries, nr_entries, 0);
- }
+ else
+ stack_depot_print(handle);
handle = READ_ONCE(page_owner->free_handle);
if (!handle) {
pr_alert("page_owner free stack trace missing\n");
} else {
- nr_entries = stack_depot_fetch(handle, &entries);
pr_alert("page last free stack trace:\n");
- stack_trace_print(entries, nr_entries, 0);
+ stack_depot_print(handle);
}
if (page_owner->last_migrate_reason != -1)
*/
void __init pcpu_free_alloc_info(struct pcpu_alloc_info *ai)
{
- memblock_free_early(__pa(ai), ai->__ai_size);
+ memblock_free(ai, ai->__ai_size);
}
/**
out_free:
pcpu_free_alloc_info(ai);
if (areas)
- memblock_free_early(__pa(areas), areas_size);
+ memblock_free(areas, areas_size);
return rc;
}
#endif /* BUILD_EMBED_FIRST_CHUNK */
free_fn(page_address(pages[j]), PAGE_SIZE);
rc = -ENOMEM;
out_free_ar:
- memblock_free_early(__pa(pages), pages_size);
+ memblock_free(pages, pages_size);
pcpu_free_alloc_info(ai);
return rc;
}
static void __init pcpu_dfl_fc_free(void *ptr, size_t size)
{
- memblock_free_early(__pa(ptr), size);
+ memblock_free(ptr, size);
}
void __init setup_per_cpu_areas(void)
* Set the initial window size, round to next power of 2 and square
* for small size, x 4 for medium, and x 2 for large
* for 128k (32 page) max ra
- * 1-8 page = 32k initial, > 8 page = 128k initial
+ * 1-2 page = 16k, 3-4 page 32k, 5-8 page = 64k, > 8 page = 128k initial
*/
static unsigned long get_init_ra_size(unsigned long size, unsigned long max)
{
update_hiwater_rss(mm);
if (is_zone_device_page(page)) {
+ unsigned long pfn = page_to_pfn(page);
swp_entry_t entry;
pte_t swp_pte;
* pte. do_swap_page() will wait until the migration
* pte is removed and then restart fault handling.
*/
- entry = make_readable_migration_entry(
- page_to_pfn(page));
+ entry = pte_to_swp_entry(pteval);
+ if (is_writable_device_private_entry(entry))
+ entry = make_writable_migration_entry(pfn);
+ else
+ entry = make_readable_migration_entry(pfn);
swp_pte = swp_entry_to_pte(entry);
/*
return swapped << PAGE_SHIFT;
/* Here comes the more involved part */
- return shmem_partial_swap_usage(mapping,
- linear_page_index(vma, vma->vm_start),
- linear_page_index(vma, vma->vm_end));
+ return shmem_partial_swap_usage(mapping, vma->vm_pgoff,
+ vma->vm_pgoff + vma_pages(vma));
}
/*
shmem_recalc_inode(inode);
spin_unlock_irq(&info->lock);
- SetPageDirty(page);
unlock_page(page);
return 0;
out_delete_from_cache:
return shmem_unlink(dir, dentry);
}
-static int shmem_exchange(struct inode *old_dir, struct dentry *old_dentry, struct inode *new_dir, struct dentry *new_dentry)
-{
- bool old_is_dir = d_is_dir(old_dentry);
- bool new_is_dir = d_is_dir(new_dentry);
-
- if (old_dir != new_dir && old_is_dir != new_is_dir) {
- if (old_is_dir) {
- drop_nlink(old_dir);
- inc_nlink(new_dir);
- } else {
- drop_nlink(new_dir);
- inc_nlink(old_dir);
- }
- }
- old_dir->i_ctime = old_dir->i_mtime =
- new_dir->i_ctime = new_dir->i_mtime =
- d_inode(old_dentry)->i_ctime =
- d_inode(new_dentry)->i_ctime = current_time(old_dir);
-
- return 0;
-}
-
static int shmem_whiteout(struct user_namespace *mnt_userns,
struct inode *old_dir, struct dentry *old_dentry)
{
return -EINVAL;
if (flags & RENAME_EXCHANGE)
- return shmem_exchange(old_dir, old_dentry, new_dir, new_dentry);
+ return simple_rename_exchange(old_dir, old_dentry, new_dir, new_dentry);
if (!simple_empty(new_dentry))
return -ENOTEMPTY;
if (!cachep)
return;
+ trace_kmem_cache_free(_RET_IP_, objp, cachep->name);
local_irq_save(flags);
debug_check_no_locks_freed(objp, cachep->object_size);
if (!(cachep->flags & SLAB_DEBUG_OBJECTS))
debug_check_no_obj_freed(objp, cachep->object_size);
__cache_free(cachep, objp, _RET_IP_);
local_irq_restore(flags);
-
- trace_kmem_cache_free(_RET_IP_, objp, cachep->name);
}
EXPORT_SYMBOL(kmem_cache_free);
if (err)
goto end;
- if (limit && shared && batchcount)
- goto skip_setup;
/*
* The head array serves three purposes:
* - create a LIFO ordering, i.e. return objects that are cache-warm
limit = 32;
#endif
batchcount = (limit + 1) / 2;
-skip_setup:
err = do_tune_cpucache(cachep, limit, batchcount, shared, gfp);
end:
if (err)
n <= cachep->useroffset - offset + cachep->usersize)
return;
- /*
- * If the copy is still within the allocated object, produce
- * a warning instead of rejecting the copy. This is intended
- * to be a temporary method to find any missing usercopy
- * whitelists.
- */
- if (usercopy_fallback &&
- offset <= cachep->object_size &&
- n <= cachep->object_size - offset) {
- usercopy_warn("SLAB object", cachep->name, to_user, offset, n);
- return;
- }
-
usercopy_abort("SLAB object", cachep->name, to_user, offset, n);
}
#endif /* CONFIG_HARDENED_USERCOPY */
#define SLAB_CACHE_FLAGS (SLAB_NOLEAKTRACE | SLAB_RECLAIM_ACCOUNT | \
SLAB_TEMPORARY | SLAB_ACCOUNT)
#else
-#define SLAB_CACHE_FLAGS (0)
+#define SLAB_CACHE_FLAGS (SLAB_NOLEAKTRACE)
#endif
/* Common flags available with current configuration */
DEFINE_MUTEX(slab_mutex);
struct kmem_cache *kmem_cache;
-#ifdef CONFIG_HARDENED_USERCOPY
-bool usercopy_fallback __ro_after_init =
- IS_ENABLED(CONFIG_HARDENED_USERCOPY_FALLBACK);
-module_param(usercopy_fallback, bool, 0400);
-MODULE_PARM_DESC(usercopy_fallback,
- "WARN instead of reject usercopy whitelist violations");
-#endif
-
static LIST_HEAD(slab_caches_to_rcu_destroy);
static void slab_caches_to_rcu_destroy_workfn(struct work_struct *work);
static DECLARE_WORK(slab_caches_to_rcu_destroy_work,
void kmem_cache_free(struct kmem_cache *c, void *b)
{
kmemleak_free_recursive(b, c->flags);
+ trace_kmem_cache_free(_RET_IP_, b, c->name);
if (unlikely(c->flags & SLAB_TYPESAFE_BY_RCU)) {
struct slob_rcu *slob_rcu;
slob_rcu = b + (c->size - sizeof(struct slob_rcu));
} else {
__kmem_cache_free(b, c->size);
}
-
- trace_kmem_cache_free(_RET_IP_, b, c->name);
}
EXPORT_SYMBOL(kmem_cache_free);
static void prefetch_freepointer(const struct kmem_cache *s, void *object)
{
- prefetch(object + s->offset);
+ prefetchw(object + s->offset);
}
static inline void *get_freepointer_safe(struct kmem_cache *s, void *object)
return x.x & OO_MASK;
}
+#ifdef CONFIG_SLUB_CPU_PARTIAL
+static void slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects)
+{
+ unsigned int nr_pages;
+
+ s->cpu_partial = nr_objects;
+
+ /*
+ * We take the number of objects but actually limit the number of
+ * pages on the per cpu partial list, in order to limit excessive
+ * growth of the list. For simplicity we assume that the pages will
+ * be half-full.
+ */
+ nr_pages = DIV_ROUND_UP(nr_objects * 2, oo_objects(s->oo));
+ s->cpu_partial_pages = nr_pages;
+}
+#else
+static inline void
+slub_set_cpu_partial(struct kmem_cache *s, unsigned int nr_objects)
+{
+}
+#endif /* CONFIG_SLUB_CPU_PARTIAL */
+
/*
* Per slab locking using the pagelock
*/
*/
static inline void *acquire_slab(struct kmem_cache *s,
struct kmem_cache_node *n, struct page *page,
- int mode, int *objects)
+ int mode)
{
void *freelist;
unsigned long counters;
freelist = page->freelist;
counters = page->counters;
new.counters = counters;
- *objects = new.objects - new.inuse;
if (mode) {
new.inuse = page->objects;
new.freelist = NULL;
{
struct page *page, *page2;
void *object = NULL;
- unsigned int available = 0;
unsigned long flags;
- int objects;
+ unsigned int partial_pages = 0;
/*
* Racy check. If we mistakenly see no partial slabs then we
if (!pfmemalloc_match(page, gfpflags))
continue;
- t = acquire_slab(s, n, page, object == NULL, &objects);
+ t = acquire_slab(s, n, page, object == NULL);
if (!t)
break;
- available += objects;
if (!object) {
*ret_page = page;
stat(s, ALLOC_FROM_PARTIAL);
} else {
put_cpu_partial(s, page, 0);
stat(s, CPU_PARTIAL_NODE);
+ partial_pages++;
}
+#ifdef CONFIG_SLUB_CPU_PARTIAL
if (!kmem_cache_has_cpu_partial(s)
- || available > slub_cpu_partial(s) / 2)
+ || partial_pages > s->cpu_partial_pages / 2)
break;
+#else
+ break;
+#endif
}
spin_unlock_irqrestore(&n->list_lock, flags);
struct page *page_to_unfreeze = NULL;
unsigned long flags;
int pages = 0;
- int pobjects = 0;
local_lock_irqsave(&s->cpu_slab->lock, flags);
oldpage = this_cpu_read(s->cpu_slab->partial);
if (oldpage) {
- if (drain && oldpage->pobjects > slub_cpu_partial(s)) {
+ if (drain && oldpage->pages >= s->cpu_partial_pages) {
/*
* Partial array is full. Move the existing set to the
* per node partial list. Postpone the actual unfreezing
page_to_unfreeze = oldpage;
oldpage = NULL;
} else {
- pobjects = oldpage->pobjects;
pages = oldpage->pages;
}
}
pages++;
- pobjects += page->objects - page->inuse;
page->pages = pages;
- page->pobjects = pobjects;
page->next = oldpage;
this_cpu_write(s->cpu_slab->partial, page);
s = cache_from_obj(s, x);
if (!s)
return;
- slab_free(s, virt_to_head_page(x), x, NULL, 1, _RET_IP_);
trace_kmem_cache_free(_RET_IP_, x, s->name);
+ slab_free(s, virt_to_head_page(x), x, NULL, 1, _RET_IP_);
}
EXPORT_SYMBOL(kmem_cache_free);
{
unsigned int order = compound_order(page);
- VM_BUG_ON_PAGE(!PageCompound(page), page);
+ if (WARN_ON_ONCE(!PageCompound(page)))
+ pr_warn_once("object pointer: 0x%p\n", object);
+
kfree_hook(object);
mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order));
__free_pages(page, order);
static void set_cpu_partial(struct kmem_cache *s)
{
#ifdef CONFIG_SLUB_CPU_PARTIAL
+ unsigned int nr_objects;
+
/*
* cpu_partial determined the maximum number of objects kept in the
* per cpu partial lists of a processor.
* filled up again with minimal effort. The slab will never hit the
* per node partial lists and therefore no locking will be required.
*
- * This setting also determines
- *
- * A) The number of objects from per cpu partial slabs dumped to the
- * per node list when we reach the limit.
- * B) The number of objects in cpu partial slabs to extract from the
- * per node list when we run out of per cpu objects. We only fetch
- * 50% to keep some capacity around for frees.
+ * For backwards compatibility reasons, this is determined as number
+ * of objects, even though we now limit maximum number of pages, see
+ * slub_set_cpu_partial()
*/
if (!kmem_cache_has_cpu_partial(s))
- slub_set_cpu_partial(s, 0);
+ nr_objects = 0;
else if (s->size >= PAGE_SIZE)
- slub_set_cpu_partial(s, 2);
+ nr_objects = 6;
else if (s->size >= 1024)
- slub_set_cpu_partial(s, 6);
+ nr_objects = 24;
else if (s->size >= 256)
- slub_set_cpu_partial(s, 13);
+ nr_objects = 52;
else
- slub_set_cpu_partial(s, 30);
+ nr_objects = 120;
+
+ slub_set_cpu_partial(s, nr_objects);
#endif
}
{
struct kmem_cache *s;
unsigned int offset;
- size_t object_size;
bool is_kfence = is_kfence_address(ptr);
ptr = kasan_reset_tag(ptr);
n <= s->useroffset - offset + s->usersize)
return;
- /*
- * If the copy is still within the allocated object, produce
- * a warning instead of rejecting the copy. This is intended
- * to be a temporary method to find any missing usercopy
- * whitelists.
- */
- object_size = slab_ksize(s);
- if (usercopy_fallback &&
- offset <= object_size && n <= object_size - offset) {
- usercopy_warn("SLUB object", s->name, to_user, offset, n);
- return;
- }
-
usercopy_abort("SLUB object", s->name, to_user, offset, n);
}
#endif /* CONFIG_HARDENED_USERCOPY */
static ssize_t cpu_partial_show(struct kmem_cache *s, char *buf)
{
- return sysfs_emit(buf, "%u\n", slub_cpu_partial(s));
+ unsigned int nr_partial = 0;
+#ifdef CONFIG_SLUB_CPU_PARTIAL
+ nr_partial = s->cpu_partial;
+#endif
+
+ return sysfs_emit(buf, "%u\n", nr_partial);
}
static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf,
page = slub_percpu_partial(per_cpu_ptr(s->cpu_slab, cpu));
- if (page) {
+ if (page)
pages += page->pages;
- objects += page->pobjects;
- }
}
+ /* Approximate half-full pages , see slub_set_cpu_partial() */
+ objects = (pages * oo_objects(s->oo)) / 2;
len += sysfs_emit_at(buf, len, "%d(%d)", objects, pages);
#ifdef CONFIG_SMP
struct page *page;
page = slub_percpu_partial(per_cpu_ptr(s->cpu_slab, cpu));
- if (page)
+ if (page) {
+ pages = READ_ONCE(page->pages);
+ objects = (pages * oo_objects(s->oo)) / 2;
len += sysfs_emit_at(buf, len, " C%d=%d(%d)",
- cpu, page->pobjects, page->pages);
+ cpu, objects, pages);
+ }
}
#endif
len += sysfs_emit_at(buf, len, "\n");
set_pte_at(&init_mm, addr, pte, entry);
}
- /* Make pte visible before pmd. See comment in __pte_alloc(). */
+ /* Make pte visible before pmd. See comment in pmd_install(). */
smp_wmb();
pmd_populate_kernel(&init_mm, pmd, pgtable);
static inline void __meminit sparse_buffer_free(unsigned long size)
{
WARN_ON(!sparsemap_buf || size == 0);
- memblock_free_early(__pa(sparsemap_buf), size);
+ memblock_free(sparsemap_buf, size);
}
static void __init sparse_buffer_init(unsigned long size, int nid)
* put_pages_list() - release a list of pages
* @pages: list of pages threaded on page->lru
*
- * Release a list of pages which are strung together on page.lru. Currently
- * used by read_cache_pages() and related error recovery code.
+ * Release a list of pages which are strung together on page.lru.
*/
void put_pages_list(struct list_head *pages)
{
- while (!list_empty(pages)) {
- struct page *victim;
+ struct page *page, *next;
- victim = lru_to_page(pages);
- list_del(&victim->lru);
- put_page(victim);
+ list_for_each_entry_safe(page, next, pages, lru) {
+ if (!put_page_testzero(page)) {
+ list_del(&page->lru);
+ continue;
+ }
+ if (PageHead(page)) {
+ list_del(&page->lru);
+ __put_compound_page(page);
+ continue;
+ }
+ /* Cannot be PageLRU because it's passed to us using the lru */
+ __ClearPageWaiters(page);
}
+
+ free_unref_page_list(pages);
+ INIT_LIST_HEAD(pages);
}
EXPORT_SYMBOL(put_pages_list);
struct swap_info_struct *si = v;
struct file *file;
int len;
- unsigned int bytes, inuse;
+ unsigned long bytes, inuse;
if (si == SEQ_START_TOKEN) {
seq_puts(swap, "Filename\t\t\t\tType\t\tSize\t\tUsed\t\tPriority\n");
file = si->swap_file;
len = seq_file_path(swap, file, " \t\n\\");
- seq_printf(swap, "%*s%s\t%u\t%s%u\t%s%d\n",
+ seq_printf(swap, "%*s%s\t%lu\t%s%lu\t%s%d\n",
len < 40 ? 40 - len : 1, " ",
S_ISBLK(file_inode(file)->i_mode) ?
"partition" : "file\t",
{
struct request_queue *q = bdev_get_queue(si->bdev);
- if (!q || !blk_queue_discard(q))
+ if (!blk_queue_discard(q))
return false;
return true;
static void clear_shadow_entry(struct address_space *mapping, pgoff_t index,
void *entry)
{
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
__clear_shadow_entry(mapping, index, entry);
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
}
/*
return;
dax = dax_mapping(mapping);
- if (!dax)
+ if (!dax) {
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
+ }
for (i = j; i < pagevec_count(pvec); i++) {
struct page *page = pvec->pages[i];
__clear_shadow_entry(mapping, index, page);
}
- if (!dax)
+ if (!dax) {
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
+ }
pvec->nr = j;
}
if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL))
return 0;
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
if (PageDirty(page))
goto failed;
BUG_ON(page_has_private(page));
__delete_from_page_cache(page, NULL);
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
if (mapping->a_ops->freepage)
mapping->a_ops->freepage(page);
return 1;
failed:
xa_unlock_irq(&mapping->i_pages);
+ spin_unlock(&mapping->host->i_lock);
return 0;
}
pgoff_t offset, max_off;
_dst_pte = mk_pte(page, dst_vma->vm_page_prot);
+ _dst_pte = pte_mkdirty(_dst_pte);
if (page_in_cache && !vm_shared)
writable = false;
- if (writable || !page_in_cache)
- _dst_pte = pte_mkdirty(_dst_pte);
if (writable) {
if (wp_copy)
_dst_pte = pte_mkuffd_wp(_dst_pte);
{
struct vmap_area *va;
struct rb_node *node;
- unsigned long length;
/* Start from the root. */
node = free_vmap_area_root.rb_node;
- /* Adjust the search size for alignment overhead. */
- length = size + align - 1;
-
while (node) {
va = rb_entry(node, struct vmap_area, rb_node);
- if (get_subtree_max_size(node->rb_left) >= length &&
+ if (get_subtree_max_size(node->rb_left) >= size &&
vstart < va->va_start) {
node = node->rb_left;
} else {
/*
* Does not make sense to go deeper towards the right
* sub-tree if it does not have a free block that is
- * equal or bigger to the requested search length.
+ * equal or bigger to the requested search size.
*/
- if (get_subtree_max_size(node->rb_right) >= length) {
+ if (get_subtree_max_size(node->rb_right) >= size) {
node = node->rb_right;
continue;
}
/*
* OK. We roll back and find the first right sub-tree,
* that will satisfy the search criteria. It can happen
- * only once due to "vstart" restriction.
+ * due to "vstart" restriction or an alignment overhead
+ * that is bigger then PAGE_SIZE.
*/
while ((node = rb_parent(node))) {
va = rb_entry(node, struct vmap_area, rb_node);
if (is_within_this_va(va, size, align, vstart))
return va;
- if (get_subtree_max_size(node->rb_right) >= length &&
+ if (get_subtree_max_size(node->rb_right) >= size &&
vstart <= va->va_start) {
+ /*
+ * Shift the vstart forward. Please note, we update it with
+ * parent's start address adding "1" because we do not want
+ * to enter same sub-tree after it has already been checked
+ * and no suitable free block found there.
+ */
+ vstart = va->va_start + 1;
node = node->rb_right;
break;
}
}
static void
-find_vmap_lowest_match_check(unsigned long size)
+find_vmap_lowest_match_check(unsigned long size, unsigned long align)
{
struct vmap_area *va_1, *va_2;
unsigned long vstart;
get_random_bytes(&rnd, sizeof(rnd));
vstart = VMALLOC_START + rnd;
- va_1 = find_vmap_lowest_match(size, 1, vstart);
- va_2 = find_vmap_lowest_linear_match(size, 1, vstart);
+ va_1 = find_vmap_lowest_match(size, align, vstart);
+ va_2 = find_vmap_lowest_linear_match(size, align, vstart);
if (va_1 != va_2)
pr_emerg("not lowest: t: 0x%p, l: 0x%p, v: 0x%lx\n",
return vend;
#if DEBUG_AUGMENT_LOWEST_MATCH_CHECK
- find_vmap_lowest_match_check(size);
+ find_vmap_lowest_match_check(size, align);
#endif
return nva_start_addr;
*/
void __init vm_area_register_early(struct vm_struct *vm, size_t align)
{
- static size_t vm_init_off __initdata;
- unsigned long addr;
+ unsigned long addr = ALIGN(VMALLOC_START, align);
+ struct vm_struct *cur, **p;
- addr = ALIGN(VMALLOC_START + vm_init_off, align);
- vm_init_off = PFN_ALIGN(addr + vm->size) - VMALLOC_START;
+ BUG_ON(vmap_initialized);
- vm->addr = (void *)addr;
+ for (p = &vmlist; (cur = *p) != NULL; p = &cur->next) {
+ if ((unsigned long)cur->addr - addr >= vm->size)
+ break;
+ addr = ALIGN((unsigned long)cur->addr + cur->size, align);
+ }
- vm_area_add_early(vm);
+ BUG_ON(addr > VMALLOC_END - vm->size);
+ vm->addr = (void *)addr;
+ vm->next = *p;
+ *p = vm;
+ kasan_populate_early_vm_area_shadow(vm->addr, vm->size);
}
static void vmap_init_free_space(void)
might_sleep();
+ /*
+ * Your top guard is someone else's bottom guard. Not having a top
+ * guard compromises someone else's mappings too.
+ */
+ if (WARN_ON_ONCE(flags & VM_NO_GUARD))
+ flags &= ~VM_NO_GUARD;
+
if (count > totalram_pages())
return NULL;
* to fails, fallback to a single page allocator that is
* more permissive.
*/
- if (!order && nid != NUMA_NO_NODE) {
+ if (!order) {
while (nr_allocated < nr_pages) {
unsigned int nr, nr_pages_request;
*/
nr_pages_request = min(100U, nr_pages - nr_allocated);
- nr = alloc_pages_bulk_array_node(gfp, nid,
- nr_pages_request, pages + nr_allocated);
+ /* memory allocation should consider mempolicy, we can't
+ * wrongly use nearest node when nid == NUMA_NO_NODE,
+ * otherwise memory may be allocated in only one node,
+ * but mempolcy want to alloc memory by interleaving.
+ */
+ if (IS_ENABLED(CONFIG_NUMA) && nid == NUMA_NO_NODE)
+ nr = alloc_pages_bulk_array_mempolicy(gfp,
+ nr_pages_request,
+ pages + nr_allocated);
+
+ else
+ nr = alloc_pages_bulk_array_node(gfp, nid,
+ nr_pages_request,
+ pages + nr_allocated);
nr_allocated += nr;
cond_resched();
if (nr != nr_pages_request)
break;
}
- } else if (order)
+ } else
/*
* Compound pages required for remap_vmalloc_page if
* high-order pages.
/* High-order pages or fallback path if "bulk" fails. */
while (nr_allocated < nr_pages) {
+ if (fatal_signal_pending(current))
+ break;
+
if (nid == NUMA_NO_NODE)
page = alloc_pages(gfp, order);
else
int node)
{
const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO;
+ const gfp_t orig_gfp_mask = gfp_mask;
unsigned long addr = (unsigned long)area->addr;
unsigned long size = get_vm_area_size(area);
unsigned long array_size;
}
if (!area->pages) {
- warn_alloc(gfp_mask, NULL,
+ warn_alloc(orig_gfp_mask, NULL,
"vmalloc error: size %lu, failed to allocated page array size %lu",
nr_small_pages * PAGE_SIZE, array_size);
free_vm_area(area);
* allocation request, free them via __vfree() if any.
*/
if (area->nr_pages != nr_small_pages) {
- warn_alloc(gfp_mask, NULL,
+ warn_alloc(orig_gfp_mask, NULL,
"vmalloc error: size %lu, page order %u, failed to allocate pages",
area->nr_pages * PAGE_SIZE, page_order);
goto fail;
if (vmap_pages_range(addr, addr + size, prot, area->pages,
page_shift) < 0) {
- warn_alloc(gfp_mask, NULL,
+ warn_alloc(orig_gfp_mask, NULL,
"vmalloc error: size %lu, failed to map pages",
area->nr_pages * PAGE_SIZE);
goto fail;
* @caller: caller's return address
*
* Allocate enough pages to cover @size from the page level
- * allocator with @gfp_mask flags. Map them into contiguous
- * kernel virtual space, using a pagetable protection of @prot.
+ * allocator with @gfp_mask flags. Please note that the full set of gfp
+ * flags are not supported. GFP_KERNEL would be a preferred allocation mode
+ * but GFP_NOFS and GFP_NOIO are supported as well. Zone modifiers are not
+ * supported. From the reclaim modifiers__GFP_DIRECT_RECLAIM is required (aka
+ * GFP_NOWAIT is not supported) and only __GFP_NOFAIL is supported (aka
+ * __GFP_NORETRY and __GFP_RETRY_MAYFAIL are not supported).
+ * __GFP_NOWARN can be used to suppress error messages about failures.
+ *
+ * Map them into contiguous kernel virtual space, using a pagetable
+ * protection of @prot.
*
* Return: the address of the area or %NULL on failure
*/
{
if (IS_ENABLED(CONFIG_NUMA)) {
unsigned int nr, *counters = m->private;
+ unsigned int step = 1U << vm_area_page_order(v);
if (!counters)
return;
memset(counters, 0, nr_node_ids * sizeof(unsigned int));
- for (nr = 0; nr < v->nr_pages; nr++)
- counters[page_to_nid(v->pages[nr])]++;
-
+ for (nr = 0; nr < v->nr_pages; nr += step)
+ counters[page_to_nid(v->pages[nr])] += step;
for_each_node_state(nr, N_HIGH_MEMORY)
if (counters[nr])
seq_printf(m, " N%u=%u", nr, counters[nr]);
(void *)va->va_start, (void *)va->va_end,
va->va_end - va->va_start);
- return 0;
+ goto final;
}
v = va->vm;
/*
* As a final step, dump "unpurged" areas.
*/
+final:
if (list_is_last(&va->list, &vmap_area_list))
show_purge_info(m);
* asserted for a second in which subsequent
* pressure events can occur.
*/
- memcg->socket_pressure = jiffies + HZ;
+ WRITE_ONCE(memcg->socket_pressure, jiffies + HZ);
}
}
}
unlock_page(page);
}
+void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)
+{
+ wait_queue_head_t *wqh = &pgdat->reclaim_wait[reason];
+ long timeout, ret;
+ DEFINE_WAIT(wait);
+
+ /*
+ * Do not throttle IO workers, kthreads other than kswapd or
+ * workqueues. They may be required for reclaim to make
+ * forward progress (e.g. journalling workqueues or kthreads).
+ */
+ if (!current_is_kswapd() &&
+ current->flags & (PF_IO_WORKER|PF_KTHREAD))
+ return;
+
+ /*
+ * These figures are pulled out of thin air.
+ * VMSCAN_THROTTLE_ISOLATED is a transient condition based on too many
+ * parallel reclaimers which is a short-lived event so the timeout is
+ * short. Failing to make progress or waiting on writeback are
+ * potentially long-lived events so use a longer timeout. This is shaky
+ * logic as a failure to make progress could be due to anything from
+ * writeback to a slow device to excessive references pages at the tail
+ * of the inactive LRU.
+ */
+ switch(reason) {
+ case VMSCAN_THROTTLE_WRITEBACK:
+ timeout = HZ/10;
+
+ if (atomic_inc_return(&pgdat->nr_writeback_throttled) == 1) {
+ WRITE_ONCE(pgdat->nr_reclaim_start,
+ node_page_state(pgdat, NR_THROTTLED_WRITTEN));
+ }
+
+ break;
+ case VMSCAN_THROTTLE_NOPROGRESS:
+ timeout = HZ/2;
+ break;
+ case VMSCAN_THROTTLE_ISOLATED:
+ timeout = HZ/50;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ timeout = HZ;
+ break;
+ }
+
+ prepare_to_wait(wqh, &wait, TASK_UNINTERRUPTIBLE);
+ ret = schedule_timeout(timeout);
+ finish_wait(wqh, &wait);
+
+ if (reason == VMSCAN_THROTTLE_WRITEBACK)
+ atomic_dec(&pgdat->nr_writeback_throttled);
+
+ trace_mm_vmscan_throttled(pgdat->node_id, jiffies_to_usecs(timeout),
+ jiffies_to_usecs(timeout - ret),
+ reason);
+}
+
+/*
+ * Account for pages written if tasks are throttled waiting on dirty
+ * pages to clean. If enough pages have been cleaned since throttling
+ * started then wakeup the throttled tasks.
+ */
+void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,
+ int nr_throttled)
+{
+ unsigned long nr_written;
+
+ node_stat_add_folio(folio, NR_THROTTLED_WRITTEN);
+
+ /*
+ * This is an inaccurate read as the per-cpu deltas may not
+ * be synchronised. However, given that the system is
+ * writeback throttled, it is not worth taking the penalty
+ * of getting an accurate count. At worst, the throttle
+ * timeout guarantees forward progress.
+ */
+ nr_written = node_page_state(pgdat, NR_THROTTLED_WRITTEN) -
+ READ_ONCE(pgdat->nr_reclaim_start);
+
+ if (nr_written > SWAP_CLUSTER_MAX * nr_throttled)
+ wake_up(&pgdat->reclaim_wait[VMSCAN_THROTTLE_WRITEBACK]);
+}
+
/* possible outcome of pageout() */
typedef enum {
/* failed to write page out, page is locked */
BUG_ON(!PageLocked(page));
BUG_ON(mapping != page_mapping(page));
+ if (!PageSwapCache(page))
+ spin_lock(&mapping->host->i_lock);
xa_lock_irq(&mapping->i_pages);
/*
* The non racy check for a busy page.
shadow = workingset_eviction(page, target_memcg);
__delete_from_page_cache(page, shadow);
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
if (freepage != NULL)
freepage(page);
cannot_free:
xa_unlock_irq(&mapping->i_pages);
+ if (!PageSwapCache(page))
+ spin_unlock(&mapping->host->i_lock);
return 0;
}
{
int target_nid = next_demotion_node(pgdat->node_id);
unsigned int nr_succeeded;
- int err;
if (list_empty(demote_pages))
return 0;
return 0;
/* Demotion ignores all cpuset and mempolicy settings */
- err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+ migrate_pages(demote_pages, alloc_demote_page, NULL,
target_nid, MIGRATE_ASYNC, MR_DEMOTION,
&nr_succeeded);
/*
* The number of dirty pages determines if a node is marked
- * reclaim_congested which affects wait_iff_congested. kswapd
- * will stall and start writing pages if the tail of the LRU
- * is all dirty unqueued pages.
+ * reclaim_congested. kswapd will stall and start writing
+ * pages if the tail of the LRU is all dirty unqueued pages.
*/
page_check_dirty_writeback(page, &dirty, &writeback);
if (dirty || writeback)
struct scan_control *sc)
{
unsigned long inactive, isolated;
+ bool too_many;
if (current_is_kswapd())
return 0;
if ((sc->gfp_mask & (__GFP_IO | __GFP_FS)) == (__GFP_IO | __GFP_FS))
inactive >>= 3;
- return isolated > inactive;
+ too_many = isolated > inactive;
+
+ /* Wake up tasks throttled due to too_many_isolated. */
+ if (!too_many)
+ wake_throttle_isolated(pgdat);
+
+ return too_many;
}
/*
return 0;
/* wait a bit for the reclaimer. */
- msleep(100);
stalled = true;
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
/* We are about to die and free our memory. Return now. */
if (fatal_signal_pending(current))
* If kswapd scans pages marked for immediate
* reclaim and under writeback (nr_immediate), it
* implies that pages are cycling through the LRU
- * faster than they are written so also forcibly stall.
+ * faster than they are written so forcibly stall
+ * until some pages complete writeback.
*/
if (sc->nr.immediate)
- congestion_wait(BLK_RW_ASYNC, HZ/10);
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
}
/*
- * Tag a node/memcg as congested if all the dirty pages
- * scanned were backed by a congested BDI and
- * wait_iff_congested will stall.
+ * Tag a node/memcg as congested if all the dirty pages were marked
+ * for writeback and immediate reclaim (counted in nr.congested).
*
* Legacy memcg will stall in page writeback so avoid forcibly
- * stalling in wait_iff_congested().
+ * stalling in reclaim_throttle().
*/
if ((current_is_kswapd() ||
(cgroup_reclaim(sc) && writeback_throttling_sane(sc))) &&
set_bit(LRUVEC_CONGESTED, &target_lruvec->flags);
/*
- * Stall direct reclaim for IO completions if underlying BDIs
- * and node is congested. Allow kswapd to continue until it
+ * Stall direct reclaim for IO completions if the lruvec is
+ * node is congested. Allow kswapd to continue until it
* starts encountering unqueued dirty pages or cycling through
* the LRU too quickly.
*/
if (!current_is_kswapd() && current_may_throttle() &&
!sc->hibernation_mode &&
test_bit(LRUVEC_CONGESTED, &target_lruvec->flags))
- wait_iff_congested(BLK_RW_ASYNC, HZ/10);
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_WRITEBACK);
if (should_continue_reclaim(pgdat, sc->nr_reclaimed - nr_reclaimed,
sc))
return zone_watermark_ok_safe(zone, 0, watermark, sc->reclaim_idx);
}
+static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
+{
+ /*
+ * If reclaim is making progress greater than 12% efficiency then
+ * wake all the NOPROGRESS throttled tasks.
+ */
+ if (sc->nr_reclaimed > (sc->nr_scanned >> 3)) {
+ wait_queue_head_t *wqh;
+
+ wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS];
+ if (waitqueue_active(wqh))
+ wake_up(wqh);
+
+ return;
+ }
+
+ /*
+ * Do not throttle kswapd on NOPROGRESS as it will throttle on
+ * VMSCAN_THROTTLE_WRITEBACK if there are too many pages under
+ * writeback and marked for immediate reclaim at the tail of
+ * the LRU.
+ */
+ if (current_is_kswapd())
+ return;
+
+ /* Throttle if making no progress at high prioities. */
+ if (sc->priority < DEF_PRIORITY - 2)
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS);
+}
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
continue;
last_pgdat = zone->zone_pgdat;
shrink_node(zone->zone_pgdat, sc);
+ consider_reclaim_throttle(zone->zone_pgdat, sc);
}
/*
WRITE_ONCE(pgdat->kswapd_order, 0);
WRITE_ONCE(pgdat->kswapd_highest_zoneidx, MAX_NR_ZONES);
+ atomic_set(&pgdat->nr_writeback_throttled, 0);
for ( ; ; ) {
bool ret;
EXPORT_SYMBOL(vm_zone_stat);
EXPORT_SYMBOL(vm_node_stat);
+#ifdef CONFIG_NUMA
+static void fold_vm_zone_numa_events(struct zone *zone)
+{
+ unsigned long zone_numa_events[NR_VM_NUMA_EVENT_ITEMS] = { 0, };
+ int cpu;
+ enum numa_stat_item item;
+
+ for_each_online_cpu(cpu) {
+ struct per_cpu_zonestat *pzstats;
+
+ pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu);
+ for (item = 0; item < NR_VM_NUMA_EVENT_ITEMS; item++)
+ zone_numa_events[item] += xchg(&pzstats->vm_numa_event[item], 0);
+ }
+
+ for (item = 0; item < NR_VM_NUMA_EVENT_ITEMS; item++)
+ zone_numa_event_add(zone_numa_events[item], zone, item);
+}
+
+void fold_vm_numa_events(void)
+{
+ struct zone *zone;
+
+ for_each_populated_zone(zone)
+ fold_vm_zone_numa_events(zone);
+}
+#endif
+
#ifdef CONFIG_SMP
int calculate_pressure_threshold(struct zone *zone)
return changes;
}
-#ifdef CONFIG_NUMA
-static void fold_vm_zone_numa_events(struct zone *zone)
-{
- unsigned long zone_numa_events[NR_VM_NUMA_EVENT_ITEMS] = { 0, };
- int cpu;
- enum numa_stat_item item;
-
- for_each_online_cpu(cpu) {
- struct per_cpu_zonestat *pzstats;
-
- pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu);
- for (item = 0; item < NR_VM_NUMA_EVENT_ITEMS; item++)
- zone_numa_events[item] += xchg(&pzstats->vm_numa_event[item], 0);
- }
-
- for (item = 0; item < NR_VM_NUMA_EVENT_ITEMS; item++)
- zone_numa_event_add(zone_numa_events[item], zone, item);
-}
-
-void fold_vm_numa_events(void)
-{
- struct zone *zone;
-
- for_each_populated_zone(zone)
- fold_vm_zone_numa_events(zone);
-}
-#endif
-
/*
* Update the zone counters for the current cpu.
*
for (order = 0; order < MAX_ORDER; order++) {
unsigned long blocks;
- /* Count number of free blocks */
- blocks = zone->free_area[order].nr_free;
+ /*
+ * Count number of free blocks.
+ *
+ * Access to nr_free is lockless as nr_free is used only for
+ * diagnostic purposes. Use data_race to avoid KCSAN warning.
+ */
+ blocks = data_race(zone->free_area[order].nr_free);
info->free_blocks_total += blocks;
/* Count free base pages */
"nr_vmscan_immediate_reclaim",
"nr_dirtied",
"nr_written",
+ "nr_throttled_written",
"nr_kernel_misc_reclaimable",
"nr_foll_pin_acquired",
"nr_foll_pin_released",
seq_printf(m, "Node %d, zone %8s ", pgdat->node_id, zone->name);
for (order = 0; order < MAX_ORDER; ++order)
- seq_printf(m, "%6lu ", zone->free_area[order].nr_free);
+ /*
+ * Access to nr_free is lockless as nr_free is used only for
+ * printing purposes. Use data_race to avoid KCSAN warning.
+ */
+ seq_printf(m, "%6lu ", data_race(zone->free_area[order].nr_free));
seq_putc(m, '\n');
}
}
seq_printf(m,
"\n pages free %lu"
+ "\n boost %lu"
"\n min %lu"
"\n low %lu"
"\n high %lu"
"\n managed %lu"
"\n cma %lu",
zone_page_state(zone, NR_FREE_PAGES),
+ zone->watermark_boost,
min_wmark_pages(zone),
low_wmark_pages(zone),
high_wmark_pages(zone),
for (order = 0; order < MAX_ORDER; ++order) {
fill_contig_page_info(zone, order, &info);
index = __fragmentation_index(order, &info);
- seq_printf(m, "%d.%03d ", index / 1000, index % 1000);
+ seq_printf(m, "%2d.%03d ", index / 1000, index % 1000);
}
seq_putc(m, '\n');
goto out;
}
+ if (!spin_trylock(&mapping->host->i_lock)) {
+ xa_unlock(&mapping->i_pages);
+ spin_unlock_irq(lru_lock);
+ ret = LRU_RETRY;
+ goto out;
+ }
+
list_lru_isolate(lru, item);
__dec_lruvec_kmem_state(node, WORKINGSET_NODES);
out_invalid:
xa_unlock_irq(&mapping->i_pages);
+ if (mapping_shrinkable(mapping))
+ inode_add_lru(mapping->host);
+ spin_unlock(&mapping->host->i_lock);
ret = LRU_REMOVED_RETRY;
out:
cond_resched();
VM_BUG_ON(atomic_long_read(&pool->isolated_pages) <= 0);
atomic_long_dec(&pool->isolated_pages);
/*
- * There's no possibility of racing, since wait_for_isolated_drain()
- * checks the isolated count under &class->lock after enqueuing
- * on migration_wait.
+ * Checking pool->destroying must happen after atomic_long_dec()
+ * for pool->isolated_pages above. Paired with the smp_mb() in
+ * zs_unregister_migration().
*/
+ smp_mb__after_atomic();
if (atomic_long_read(&pool->isolated_pages) == 0 && pool->destroying)
wake_up_all(&pool->migration_wait);
}
}
vlan_vid_del(real_dev, vlan->vlan_proto, vlan_id);
-
- /* Get rid of the vlan's reference to real_dev */
- dev_put(real_dev);
}
int vlan_check_real_dev(struct net_device *real_dev,
free_percpu(vlan->vlan_pcpu_stats);
vlan->vlan_pcpu_stats = NULL;
+
+ /* Get rid of the vlan's reference to real_dev */
+ dev_put(vlan->real_dev);
}
void vlan_setup(struct net_device *dev)
// SPDX-License-Identifier: GPL-2.0-only
/*
- * net/9p/clnt.c
- *
* 9P Client
*
* Copyright (C) 2008 by Eric Van Hensbergen <ericvh@gmail.com>
#define DEFAULT_MSIZE (128 * 1024)
-/*
- * Client Option Parsing (code inspired by NFS code)
- * - a little lazy - parse all client options
- */
+/* Client Option Parsing (code inspired by NFS code)
+ * - a little lazy - parse all client options
+ */
enum {
Opt_msize,
}
EXPORT_SYMBOL(p9_show_client_options);
-/*
- * Some error codes are taken directly from the server replies,
+/* Some error codes are taken directly from the server replies,
* make sure they are valid.
*/
static int safe_errno(int err)
{
- if ((err > 0) || (err < -MAX_ERRNO)) {
+ if (err > 0 || err < -MAX_ERRNO) {
p9_debug(P9_DEBUG_ERROR, "Invalid error code %d\n", err);
return -EPROTO;
}
return err;
}
-
/* Interpret mount option for protocol version */
static int get_protocol_version(char *s)
{
} else if (!strcmp(s, "9p2000.L")) {
version = p9_proto_2000L;
p9_debug(P9_DEBUG_9P, "Protocol version: 9P2000.L\n");
- } else
+ } else {
pr_info("Unknown protocol version %s\n", s);
+ }
return version;
}
return 0;
tmp_options = kstrdup(opts, GFP_KERNEL);
- if (!tmp_options) {
- p9_debug(P9_DEBUG_ERROR,
- "failed to allocate copy of option string\n");
+ if (!tmp_options)
return -ENOMEM;
- }
options = tmp_options;
while ((p = strsep(&options, ",")) != NULL) {
int token, r;
+
if (!*p)
continue;
token = match_token(p, tokens, args);
v9fs_put_trans(clnt->trans_mod);
clnt->trans_mod = v9fs_get_trans_by_name(s);
- if (clnt->trans_mod == NULL) {
+ if (!clnt->trans_mod) {
pr_info("Could not find request transport: %s\n",
s);
ret = -EINVAL;
static void p9_req_free(struct kref *ref)
{
struct p9_req_t *r = container_of(ref, struct p9_req_t, refcount);
+
p9_fcall_fini(&r->tc);
p9_fcall_fini(&r->rc);
kmem_cache_free(p9_req_cache, r);
{
p9_debug(P9_DEBUG_MUX, " tag %d\n", req->tc.tag);
- /*
- * This barrier is needed to make sure any change made to req before
+ /* This barrier is needed to make sure any change made to req before
* the status change is visible to another thread
*/
smp_wmb();
*/
int
-p9_parse_header(struct p9_fcall *pdu, int32_t *size, int8_t *type, int16_t *tag,
- int rewind)
+p9_parse_header(struct p9_fcall *pdu, int32_t *size, int8_t *type,
+ int16_t *tag, int rewind)
{
- int8_t r_type;
- int16_t r_tag;
- int32_t r_size;
+ s8 r_type;
+ s16 r_tag;
+ s32 r_size;
int offset = pdu->offset;
int err;
static int p9_check_errors(struct p9_client *c, struct p9_req_t *req)
{
- int8_t type;
+ s8 type;
int err;
int ecode;
req->rc.size);
return -EIO;
}
- /*
- * dump the response from server
+ /* dump the response from server
* This should be after check errors which poplulate pdu_fcall.
*/
trace_9p_protocol_dump(c, &req->rc);
if (!p9_is_proto_dotl(c)) {
char *ename;
+
err = p9pdu_readf(&req->rc, c->proto_version, "s?d",
&ename, &ecode);
if (err)
kfree(ename);
} else {
err = p9pdu_readf(&req->rc, c->proto_version, "d", &ecode);
+ if (err)
+ goto out_err;
err = -ecode;
p9_debug(P9_DEBUG_9P, "<<< RLERROR (%d)\n", -ecode);
{
int err;
int ecode;
- int8_t type;
+ s8 type;
char *ename = NULL;
err = p9_parse_header(&req->rc, NULL, &type, NULL, 0);
- /*
- * dump the response from server
+ /* dump the response from server
* This should be after parse_header which poplulate pdu_fcall.
*/
trace_9p_protocol_dump(c, &req->rc);
if (len > inline_len) {
/* We have error in external buffer */
if (!copy_from_iter_full(ename + inline_len,
- len - inline_len, uidata)) {
+ len - inline_len, uidata)) {
err = -EFAULT;
goto out_err;
}
static int p9_client_flush(struct p9_client *c, struct p9_req_t *oldreq)
{
struct p9_req_t *req;
- int16_t oldtag;
+ s16 oldtag;
int err;
err = p9_parse_header(&oldreq->tc, NULL, NULL, &oldtag, 1);
if (IS_ERR(req))
return PTR_ERR(req);
- /*
- * if we haven't received a response for oldreq,
+ /* if we haven't received a response for oldreq,
* remove it from the list
*/
if (oldreq->status == REQ_STATUS_SENT) {
return ERR_PTR(-EIO);
/* if status is begin_disconnected we allow only clunk request */
- if ((c->status == BeginDisconnect) && (type != P9_TCLUNK))
+ if (c->status == BeginDisconnect && type != P9_TCLUNK)
return ERR_PTR(-EIO);
req = p9_tag_alloc(c, type, req_size);
if (signal_pending(current)) {
sigpending = 1;
clear_thread_flag(TIF_SIGPENDING);
- } else
+ } else {
sigpending = 0;
+ }
err = c->trans_mod->request(c, req);
if (err < 0) {
/* Wait for the response */
err = wait_event_killable(req->wq, req->status >= REQ_STATUS_RCVD);
- /*
- * Make sure our req is coherent with regard to updates in other
+ /* Make sure our req is coherent with regard to updates in other
* threads - echoes to wmb() in the callback
*/
smp_rmb();
- if ((err == -ERESTARTSYS) && (c->status == Connected)
- && (type == P9_TFLUSH)) {
+ if (err == -ERESTARTSYS && c->status == Connected &&
+ type == P9_TFLUSH) {
sigpending = 1;
clear_thread_flag(TIF_SIGPENDING);
goto again;
p9_debug(P9_DEBUG_ERROR, "req_status error %d\n", req->t_err);
err = req->t_err;
}
- if ((err == -ERESTARTSYS) && (c->status == Connected)) {
+ if (err == -ERESTARTSYS && c->status == Connected) {
p9_debug(P9_DEBUG_MUX, "flushing\n");
sigpending = 1;
clear_thread_flag(TIF_SIGPENDING);
struct p9_req_t *req;
va_start(ap, fmt);
- /*
- * We allocate a inline protocol data of only 4k bytes.
+ /* We allocate a inline protocol data of only 4k bytes.
* The actual content is passed in zero-copy fashion.
*/
req = p9_client_prepare_req(c, type, P9_ZC_HDR_SZ, fmt, ap);
if (signal_pending(current)) {
sigpending = 1;
clear_thread_flag(TIF_SIGPENDING);
- } else
+ } else {
sigpending = 0;
+ }
err = c->trans_mod->zc_request(c, req, uidata, uodata,
inlen, olen, in_hdrlen);
p9_debug(P9_DEBUG_ERROR, "req_status error %d\n", req->t_err);
err = req->t_err;
}
- if ((err == -ERESTARTSYS) && (c->status == Connected)) {
+ if (err == -ERESTARTSYS && c->status == Connected) {
p9_debug(P9_DEBUG_MUX, "flushing\n");
sigpending = 1;
clear_thread_flag(TIF_SIGPENDING);
struct p9_fid *fid;
p9_debug(P9_DEBUG_FID, "clnt %p\n", clnt);
- fid = kmalloc(sizeof(struct p9_fid), GFP_KERNEL);
+ fid = kmalloc(sizeof(*fid), GFP_KERNEL);
if (!fid)
return NULL;
- memset(&fid->qid, 0, sizeof(struct p9_qid));
+ memset(&fid->qid, 0, sizeof(fid->qid));
fid->mode = -1;
fid->uid = current_fsuid();
fid->clnt = clnt;
switch (c->proto_version) {
case p9_proto_2000L:
req = p9_client_rpc(c, P9_TVERSION, "ds",
- c->msize, "9P2000.L");
+ c->msize, "9P2000.L");
break;
case p9_proto_2000u:
req = p9_client_rpc(c, P9_TVERSION, "ds",
- c->msize, "9P2000.u");
+ c->msize, "9P2000.u");
break;
case p9_proto_legacy:
req = p9_client_rpc(c, P9_TVERSION, "ds",
- c->msize, "9P2000");
+ c->msize, "9P2000");
break;
default:
return -EINVAL;
}
p9_debug(P9_DEBUG_9P, "<<< RVERSION msize %d %s\n", msize, version);
- if (!strncmp(version, "9P2000.L", 8))
+ if (!strncmp(version, "9P2000.L", 8)) {
c->proto_version = p9_proto_2000L;
- else if (!strncmp(version, "9P2000.u", 8))
+ } else if (!strncmp(version, "9P2000.u", 8)) {
c->proto_version = p9_proto_2000u;
- else if (!strncmp(version, "9P2000", 6))
+ } else if (!strncmp(version, "9P2000", 6)) {
c->proto_version = p9_proto_legacy;
- else {
+ } else {
p9_debug(P9_DEBUG_ERROR,
"server returned an unknown version: %s\n", version);
err = -EREMOTEIO;
char *client_id;
err = 0;
- clnt = kmalloc(sizeof(struct p9_client), GFP_KERNEL);
+ clnt = kmalloc(sizeof(*clnt), GFP_KERNEL);
if (!clnt)
return ERR_PTR(-ENOMEM);
if (!clnt->trans_mod)
clnt->trans_mod = v9fs_get_default_trans();
- if (clnt->trans_mod == NULL) {
+ if (!clnt->trans_mod) {
err = -EPROTONOSUPPORT;
p9_debug(P9_DEBUG_ERROR,
"No transport defined or default transport\n");
EXPORT_SYMBOL(p9_client_begin_disconnect);
struct p9_fid *p9_client_attach(struct p9_client *clnt, struct p9_fid *afid,
- const char *uname, kuid_t n_uname, const char *aname)
+ const char *uname, kuid_t n_uname,
+ const char *aname)
{
int err = 0;
struct p9_req_t *req;
struct p9_fid *fid;
struct p9_qid qid;
-
p9_debug(P9_DEBUG_9P, ">>> TATTACH afid %d uname %s aname %s\n",
afid ? afid->fid : -1, uname, aname);
fid = p9_fid_create(clnt);
fid->uid = n_uname;
req = p9_client_rpc(clnt, P9_TATTACH, "ddss?u", fid->fid,
- afid ? afid->fid : P9_NOFID, uname, aname, n_uname);
+ afid ? afid->fid : P9_NOFID, uname, aname, n_uname);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
}
p9_debug(P9_DEBUG_9P, "<<< RATTACH qid %x.%llx.%x\n",
- qid.type, (unsigned long long)qid.path, qid.version);
+ qid.type, qid.path, qid.version);
memmove(&fid->qid, &qid, sizeof(struct p9_qid));
EXPORT_SYMBOL(p9_client_attach);
struct p9_fid *p9_client_walk(struct p9_fid *oldfid, uint16_t nwname,
- const unsigned char * const *wnames, int clone)
+ const unsigned char * const *wnames, int clone)
{
int err;
struct p9_client *clnt;
struct p9_fid *fid;
struct p9_qid *wqids;
struct p9_req_t *req;
- uint16_t nwqids, count;
+ u16 nwqids, count;
err = 0;
wqids = NULL;
}
fid->uid = oldfid->uid;
- } else
+ } else {
fid = oldfid;
-
+ }
p9_debug(P9_DEBUG_9P, ">>> TWALK fids %d,%d nwname %ud wname[0] %s\n",
oldfid->fid, fid->fid, nwname, wnames ? wnames[0] : NULL);
req = p9_client_rpc(clnt, P9_TWALK, "ddT", oldfid->fid, fid->fid,
- nwname, wnames);
+ nwname, wnames);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
for (count = 0; count < nwqids; count++)
p9_debug(P9_DEBUG_9P, "<<< [%d] %x.%llx.%x\n",
- count, wqids[count].type,
- (unsigned long long)wqids[count].path,
- wqids[count].version);
+ count, wqids[count].type,
+ wqids[count].path,
+ wqids[count].version);
if (nwname)
memmove(&fid->qid, &wqids[nwqids - 1], sizeof(struct p9_qid));
fid = NULL;
error:
- if (fid && (fid != oldfid))
+ if (fid && fid != oldfid)
p9_fid_destroy(fid);
return ERR_PTR(err);
clnt = fid->clnt;
p9_debug(P9_DEBUG_9P, ">>> %s fid %d mode %d\n",
- p9_is_proto_dotl(clnt) ? "TLOPEN" : "TOPEN", fid->fid, mode);
+ p9_is_proto_dotl(clnt) ? "TLOPEN" : "TOPEN", fid->fid, mode);
err = 0;
if (fid->mode != -1)
}
p9_debug(P9_DEBUG_9P, "<<< %s qid %x.%llx.%x iounit %x\n",
- p9_is_proto_dotl(clnt) ? "RLOPEN" : "ROPEN", qid.type,
- (unsigned long long)qid.path, qid.version, iounit);
+ p9_is_proto_dotl(clnt) ? "RLOPEN" : "ROPEN", qid.type,
+ qid.path, qid.version, iounit);
memmove(&fid->qid, &qid, sizeof(struct p9_qid));
fid->mode = mode;
}
EXPORT_SYMBOL(p9_client_open);
-int p9_client_create_dotl(struct p9_fid *ofid, const char *name, u32 flags, u32 mode,
- kgid_t gid, struct p9_qid *qid)
+int p9_client_create_dotl(struct p9_fid *ofid, const char *name, u32 flags,
+ u32 mode, kgid_t gid, struct p9_qid *qid)
{
int err = 0;
struct p9_client *clnt;
int iounit;
p9_debug(P9_DEBUG_9P,
- ">>> TLCREATE fid %d name %s flags %d mode %d gid %d\n",
- ofid->fid, name, flags, mode,
- from_kgid(&init_user_ns, gid));
+ ">>> TLCREATE fid %d name %s flags %d mode %d gid %d\n",
+ ofid->fid, name, flags, mode,
+ from_kgid(&init_user_ns, gid));
clnt = ofid->clnt;
if (ofid->mode != -1)
return -EINVAL;
req = p9_client_rpc(clnt, P9_TLCREATE, "dsddg", ofid->fid, name, flags,
- mode, gid);
+ mode, gid);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
}
p9_debug(P9_DEBUG_9P, "<<< RLCREATE qid %x.%llx.%x iounit %x\n",
- qid->type,
- (unsigned long long)qid->path,
- qid->version, iounit);
+ qid->type, qid->path, qid->version, iounit);
memmove(&ofid->qid, qid, sizeof(struct p9_qid));
ofid->mode = mode;
int iounit;
p9_debug(P9_DEBUG_9P, ">>> TCREATE fid %d name %s perm %d mode %d\n",
- fid->fid, name, perm, mode);
+ fid->fid, name, perm, mode);
err = 0;
clnt = fid->clnt;
return -EINVAL;
req = p9_client_rpc(clnt, P9_TCREATE, "dsdb?s", fid->fid, name, perm,
- mode, extension);
+ mode, extension);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
}
p9_debug(P9_DEBUG_9P, "<<< RCREATE qid %x.%llx.%x iounit %x\n",
- qid.type,
- (unsigned long long)qid.path,
- qid.version, iounit);
+ qid.type, qid.path, qid.version, iounit);
memmove(&fid->qid, &qid, sizeof(struct p9_qid));
fid->mode = mode;
EXPORT_SYMBOL(p9_client_fcreate);
int p9_client_symlink(struct p9_fid *dfid, const char *name,
- const char *symtgt, kgid_t gid, struct p9_qid *qid)
+ const char *symtgt, kgid_t gid, struct p9_qid *qid)
{
int err = 0;
struct p9_client *clnt;
struct p9_req_t *req;
p9_debug(P9_DEBUG_9P, ">>> TSYMLINK dfid %d name %s symtgt %s\n",
- dfid->fid, name, symtgt);
+ dfid->fid, name, symtgt);
clnt = dfid->clnt;
req = p9_client_rpc(clnt, P9_TSYMLINK, "dssg", dfid->fid, name, symtgt,
- gid);
+ gid);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
}
p9_debug(P9_DEBUG_9P, "<<< RSYMLINK qid %x.%llx.%x\n",
- qid->type, (unsigned long long)qid->path, qid->version);
+ qid->type, qid->path, qid->version);
free_and_error:
p9_tag_remove(clnt, req);
struct p9_req_t *req;
p9_debug(P9_DEBUG_9P, ">>> TLINK dfid %d oldfid %d newname %s\n",
- dfid->fid, oldfid->fid, newname);
+ dfid->fid, oldfid->fid, newname);
clnt = dfid->clnt;
req = p9_client_rpc(clnt, P9_TLINK, "dds", dfid->fid, oldfid->fid,
- newname);
+ newname);
if (IS_ERR(req))
return PTR_ERR(req);
struct p9_req_t *req;
p9_debug(P9_DEBUG_9P, ">>> TFSYNC fid %d datasync:%d\n",
- fid->fid, datasync);
+ fid->fid, datasync);
err = 0;
clnt = fid->clnt;
return 0;
again:
- p9_debug(P9_DEBUG_9P, ">>> TCLUNK fid %d (try %d)\n", fid->fid,
- retries);
+ p9_debug(P9_DEBUG_9P, ">>> TCLUNK fid %d (try %d)\n",
+ fid->fid, retries);
err = 0;
clnt = fid->clnt;
p9_tag_remove(clnt, req);
error:
- /*
- * Fid is not valid even after a failed clunk
+ /* Fid is not valid even after a failed clunk
* If interrupted, retry once then give up and
* leak fid until umount.
*/
if (err == -ERESTARTSYS) {
if (retries++ == 0)
goto again;
- } else
+ } else {
p9_fid_destroy(fid);
+ }
return err;
}
EXPORT_SYMBOL(p9_client_clunk);
struct p9_client *clnt;
p9_debug(P9_DEBUG_9P, ">>> TUNLINKAT fid %d %s %d\n",
- dfid->fid, name, flags);
+ dfid->fid, name, flags);
clnt = dfid->clnt;
req = p9_client_rpc(clnt, P9_TUNLINKAT, "dsd", dfid->fid, name, flags);
char *dataptr;
*err = 0;
- p9_debug(P9_DEBUG_9P, ">>> TREAD fid %d offset %llu %d\n",
- fid->fid, (unsigned long long) offset, (int)iov_iter_count(to));
+ p9_debug(P9_DEBUG_9P, ">>> TREAD fid %d offset %llu %zu\n",
+ fid->fid, offset, iov_iter_count(to));
rsize = fid->iounit;
if (!rsize || rsize > clnt->msize - P9_IOHDRSZ)
*err = 0;
p9_debug(P9_DEBUG_9P, ">>> TWRITE fid %d offset %llu count %zd\n",
- fid->fid, (unsigned long long) offset,
- iov_iter_count(from));
+ fid->fid, offset, iov_iter_count(from));
while (iov_iter_count(from)) {
int count = iov_iter_count(from);
int rsize = fid->iounit;
- if (!rsize || rsize > clnt->msize-P9_IOHDRSZ)
+
+ if (!rsize || rsize > clnt->msize - P9_IOHDRSZ)
rsize = clnt->msize - P9_IOHDRSZ;
if (count < rsize)
fid->fid, offset, rsize);
} else {
req = p9_client_rpc(clnt, P9_TWRITE, "dqV", fid->fid,
- offset, rsize, from);
+ offset, rsize, from);
}
if (IS_ERR(req)) {
*err = PTR_ERR(req);
{
int err;
struct p9_client *clnt;
- struct p9_wstat *ret = kmalloc(sizeof(struct p9_wstat), GFP_KERNEL);
+ struct p9_wstat *ret;
struct p9_req_t *req;
u16 ignored;
p9_debug(P9_DEBUG_9P, ">>> TSTAT fid %d\n", fid->fid);
+ ret = kmalloc(sizeof(*ret), GFP_KERNEL);
if (!ret)
return ERR_PTR(-ENOMEM);
}
p9_debug(P9_DEBUG_9P,
- "<<< RSTAT sz=%x type=%x dev=%x qid=%x.%llx.%x\n"
- "<<< mode=%8.8x atime=%8.8x mtime=%8.8x length=%llx\n"
- "<<< name=%s uid=%s gid=%s muid=%s extension=(%s)\n"
- "<<< uid=%d gid=%d n_muid=%d\n",
- ret->size, ret->type, ret->dev, ret->qid.type,
- (unsigned long long)ret->qid.path, ret->qid.version, ret->mode,
- ret->atime, ret->mtime, (unsigned long long)ret->length,
- ret->name, ret->uid, ret->gid, ret->muid, ret->extension,
- from_kuid(&init_user_ns, ret->n_uid),
- from_kgid(&init_user_ns, ret->n_gid),
- from_kuid(&init_user_ns, ret->n_muid));
+ "<<< RSTAT sz=%x type=%x dev=%x qid=%x.%llx.%x\n"
+ "<<< mode=%8.8x atime=%8.8x mtime=%8.8x length=%llx\n"
+ "<<< name=%s uid=%s gid=%s muid=%s extension=(%s)\n"
+ "<<< uid=%d gid=%d n_muid=%d\n",
+ ret->size, ret->type, ret->dev, ret->qid.type, ret->qid.path,
+ ret->qid.version, ret->mode,
+ ret->atime, ret->mtime, ret->length,
+ ret->name, ret->uid, ret->gid, ret->muid, ret->extension,
+ from_kuid(&init_user_ns, ret->n_uid),
+ from_kgid(&init_user_ns, ret->n_gid),
+ from_kuid(&init_user_ns, ret->n_muid));
p9_tag_remove(clnt, req);
return ret;
EXPORT_SYMBOL(p9_client_stat);
struct p9_stat_dotl *p9_client_getattr_dotl(struct p9_fid *fid,
- u64 request_mask)
+ u64 request_mask)
{
int err;
struct p9_client *clnt;
- struct p9_stat_dotl *ret = kmalloc(sizeof(struct p9_stat_dotl),
- GFP_KERNEL);
+ struct p9_stat_dotl *ret;
struct p9_req_t *req;
p9_debug(P9_DEBUG_9P, ">>> TGETATTR fid %d, request_mask %lld\n",
- fid->fid, request_mask);
+ fid->fid, request_mask);
+ ret = kmalloc(sizeof(*ret), GFP_KERNEL);
if (!ret)
return ERR_PTR(-ENOMEM);
goto error;
}
- p9_debug(P9_DEBUG_9P,
- "<<< RGETATTR st_result_mask=%lld\n"
- "<<< qid=%x.%llx.%x\n"
- "<<< st_mode=%8.8x st_nlink=%llu\n"
- "<<< st_uid=%d st_gid=%d\n"
- "<<< st_rdev=%llx st_size=%llx st_blksize=%llu st_blocks=%llu\n"
- "<<< st_atime_sec=%lld st_atime_nsec=%lld\n"
- "<<< st_mtime_sec=%lld st_mtime_nsec=%lld\n"
- "<<< st_ctime_sec=%lld st_ctime_nsec=%lld\n"
- "<<< st_btime_sec=%lld st_btime_nsec=%lld\n"
- "<<< st_gen=%lld st_data_version=%lld\n",
- ret->st_result_mask, ret->qid.type, ret->qid.path,
- ret->qid.version, ret->st_mode, ret->st_nlink,
- from_kuid(&init_user_ns, ret->st_uid),
- from_kgid(&init_user_ns, ret->st_gid),
- ret->st_rdev, ret->st_size, ret->st_blksize,
- ret->st_blocks, ret->st_atime_sec, ret->st_atime_nsec,
- ret->st_mtime_sec, ret->st_mtime_nsec, ret->st_ctime_sec,
- ret->st_ctime_nsec, ret->st_btime_sec, ret->st_btime_nsec,
- ret->st_gen, ret->st_data_version);
+ p9_debug(P9_DEBUG_9P, "<<< RGETATTR st_result_mask=%lld\n"
+ "<<< qid=%x.%llx.%x\n"
+ "<<< st_mode=%8.8x st_nlink=%llu\n"
+ "<<< st_uid=%d st_gid=%d\n"
+ "<<< st_rdev=%llx st_size=%llx st_blksize=%llu st_blocks=%llu\n"
+ "<<< st_atime_sec=%lld st_atime_nsec=%lld\n"
+ "<<< st_mtime_sec=%lld st_mtime_nsec=%lld\n"
+ "<<< st_ctime_sec=%lld st_ctime_nsec=%lld\n"
+ "<<< st_btime_sec=%lld st_btime_nsec=%lld\n"
+ "<<< st_gen=%lld st_data_version=%lld\n",
+ ret->st_result_mask,
+ ret->qid.type, ret->qid.path, ret->qid.version,
+ ret->st_mode, ret->st_nlink,
+ from_kuid(&init_user_ns, ret->st_uid),
+ from_kgid(&init_user_ns, ret->st_gid),
+ ret->st_rdev, ret->st_size, ret->st_blksize, ret->st_blocks,
+ ret->st_atime_sec, ret->st_atime_nsec,
+ ret->st_mtime_sec, ret->st_mtime_nsec,
+ ret->st_ctime_sec, ret->st_ctime_nsec,
+ ret->st_btime_sec, ret->st_btime_nsec,
+ ret->st_gen, ret->st_data_version);
p9_tag_remove(clnt, req);
return ret;
/* size[2] type[2] dev[4] qid[13] */
/* mode[4] atime[4] mtime[4] length[8]*/
/* name[s] uid[s] gid[s] muid[s] */
- ret = 2+4+13+4+4+4+8+2+2+2+2;
+ ret = 2 + 4 + 13 + 4 + 4 + 4 + 8 + 2 + 2 + 2 + 2;
if (wst->name)
ret += strlen(wst->name);
if (wst->muid)
ret += strlen(wst->muid);
- if ((proto_version == p9_proto_2000u) ||
- (proto_version == p9_proto_2000L)) {
- ret += 2+4+4+4; /* extension[s] n_uid[4] n_gid[4] n_muid[4] */
+ if (proto_version == p9_proto_2000u ||
+ proto_version == p9_proto_2000L) {
+ /* extension[s] n_uid[4] n_gid[4] n_muid[4] */
+ ret += 2 + 4 + 4 + 4;
if (wst->extension)
ret += strlen(wst->extension);
}
err = 0;
clnt = fid->clnt;
wst->size = p9_client_statsize(wst, clnt->proto_version);
- p9_debug(P9_DEBUG_9P, ">>> TWSTAT fid %d\n", fid->fid);
+ p9_debug(P9_DEBUG_9P, ">>> TWSTAT fid %d\n",
+ fid->fid);
p9_debug(P9_DEBUG_9P,
- " sz=%x type=%x dev=%x qid=%x.%llx.%x\n"
- " mode=%8.8x atime=%8.8x mtime=%8.8x length=%llx\n"
- " name=%s uid=%s gid=%s muid=%s extension=(%s)\n"
- " uid=%d gid=%d n_muid=%d\n",
- wst->size, wst->type, wst->dev, wst->qid.type,
- (unsigned long long)wst->qid.path, wst->qid.version, wst->mode,
- wst->atime, wst->mtime, (unsigned long long)wst->length,
- wst->name, wst->uid, wst->gid, wst->muid, wst->extension,
- from_kuid(&init_user_ns, wst->n_uid),
- from_kgid(&init_user_ns, wst->n_gid),
- from_kuid(&init_user_ns, wst->n_muid));
-
- req = p9_client_rpc(clnt, P9_TWSTAT, "dwS", fid->fid, wst->size+2, wst);
+ " sz=%x type=%x dev=%x qid=%x.%llx.%x\n"
+ " mode=%8.8x atime=%8.8x mtime=%8.8x length=%llx\n"
+ " name=%s uid=%s gid=%s muid=%s extension=(%s)\n"
+ " uid=%d gid=%d n_muid=%d\n",
+ wst->size, wst->type, wst->dev, wst->qid.type,
+ wst->qid.path, wst->qid.version,
+ wst->mode, wst->atime, wst->mtime, wst->length,
+ wst->name, wst->uid, wst->gid, wst->muid, wst->extension,
+ from_kuid(&init_user_ns, wst->n_uid),
+ from_kgid(&init_user_ns, wst->n_gid),
+ from_kuid(&init_user_ns, wst->n_muid));
+
+ req = p9_client_rpc(clnt, P9_TWSTAT, "dwS",
+ fid->fid, wst->size + 2, wst);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
err = 0;
clnt = fid->clnt;
p9_debug(P9_DEBUG_9P, ">>> TSETATTR fid %d\n", fid->fid);
- p9_debug(P9_DEBUG_9P,
- " valid=%x mode=%x uid=%d gid=%d size=%lld\n"
- " atime_sec=%lld atime_nsec=%lld\n"
- " mtime_sec=%lld mtime_nsec=%lld\n",
- p9attr->valid, p9attr->mode,
- from_kuid(&init_user_ns, p9attr->uid),
- from_kgid(&init_user_ns, p9attr->gid),
- p9attr->size, p9attr->atime_sec, p9attr->atime_nsec,
- p9attr->mtime_sec, p9attr->mtime_nsec);
+ p9_debug(P9_DEBUG_9P, " valid=%x mode=%x uid=%d gid=%d size=%lld\n",
+ p9attr->valid, p9attr->mode,
+ from_kuid(&init_user_ns, p9attr->uid),
+ from_kgid(&init_user_ns, p9attr->gid),
+ p9attr->size);
+ p9_debug(P9_DEBUG_9P, " atime_sec=%lld atime_nsec=%lld\n",
+ p9attr->atime_sec, p9attr->atime_nsec);
+ p9_debug(P9_DEBUG_9P, " mtime_sec=%lld mtime_nsec=%lld\n",
+ p9attr->mtime_sec, p9attr->mtime_nsec);
req = p9_client_rpc(clnt, P9_TSETATTR, "dI", fid->fid, p9attr);
goto error;
}
- p9_debug(P9_DEBUG_9P, "<<< RSTATFS fid %d type 0x%lx bsize %ld "
- "blocks %llu bfree %llu bavail %llu files %llu ffree %llu "
- "fsid %llu namelen %ld\n",
- fid->fid, (long unsigned int)sb->type, (long int)sb->bsize,
- sb->blocks, sb->bfree, sb->bavail, sb->files, sb->ffree,
- sb->fsid, (long int)sb->namelen);
+ p9_debug(P9_DEBUG_9P,
+ "<<< RSTATFS fid %d type 0x%x bsize %u blocks %llu bfree %llu bavail %llu files %llu ffree %llu fsid %llu namelen %u\n",
+ fid->fid, sb->type, sb->bsize, sb->blocks, sb->bfree,
+ sb->bavail, sb->files, sb->ffree, sb->fsid, sb->namelen);
p9_tag_remove(clnt, req);
error:
clnt = fid->clnt;
p9_debug(P9_DEBUG_9P, ">>> TRENAME fid %d newdirfid %d name %s\n",
- fid->fid, newdirfid->fid, name);
+ fid->fid, newdirfid->fid, name);
req = p9_client_rpc(clnt, P9_TRENAME, "dds", fid->fid,
- newdirfid->fid, name);
+ newdirfid->fid, name);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
err = 0;
clnt = olddirfid->clnt;
- p9_debug(P9_DEBUG_9P, ">>> TRENAMEAT olddirfid %d old name %s"
- " newdirfid %d new name %s\n", olddirfid->fid, old_name,
- newdirfid->fid, new_name);
+ p9_debug(P9_DEBUG_9P,
+ ">>> TRENAMEAT olddirfid %d old name %s newdirfid %d new name %s\n",
+ olddirfid->fid, old_name, newdirfid->fid, new_name);
req = p9_client_rpc(clnt, P9_TRENAMEAT, "dsds", olddirfid->fid,
old_name, newdirfid->fid, new_name);
}
p9_debug(P9_DEBUG_9P, "<<< RRENAMEAT newdirfid %d new name %s\n",
- newdirfid->fid, new_name);
+ newdirfid->fid, new_name);
p9_tag_remove(clnt, req);
error:
}
EXPORT_SYMBOL(p9_client_renameat);
-/*
- * An xattrwalk without @attr_name gives the fid for the lisxattr namespace
+/* An xattrwalk without @attr_name gives the fid for the lisxattr namespace
*/
struct p9_fid *p9_client_xattrwalk(struct p9_fid *file_fid,
- const char *attr_name, u64 *attr_size)
+ const char *attr_name, u64 *attr_size)
{
int err;
struct p9_req_t *req;
goto error;
}
p9_debug(P9_DEBUG_9P,
- ">>> TXATTRWALK file_fid %d, attr_fid %d name %s\n",
- file_fid->fid, attr_fid->fid, attr_name);
+ ">>> TXATTRWALK file_fid %d, attr_fid %d name %s\n",
+ file_fid->fid, attr_fid->fid, attr_name);
req = p9_client_rpc(clnt, P9_TXATTRWALK, "dds",
- file_fid->fid, attr_fid->fid, attr_name);
+ file_fid->fid, attr_fid->fid, attr_name);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
}
p9_tag_remove(clnt, req);
p9_debug(P9_DEBUG_9P, "<<< RXATTRWALK fid %d size %llu\n",
- attr_fid->fid, *attr_size);
+ attr_fid->fid, *attr_size);
return attr_fid;
clunk_fid:
p9_client_clunk(attr_fid);
attr_fid = NULL;
error:
- if (attr_fid && (attr_fid != file_fid))
+ if (attr_fid && attr_fid != file_fid)
p9_fid_destroy(attr_fid);
return ERR_PTR(err);
EXPORT_SYMBOL_GPL(p9_client_xattrwalk);
int p9_client_xattrcreate(struct p9_fid *fid, const char *name,
- u64 attr_size, int flags)
+ u64 attr_size, int flags)
{
int err;
struct p9_req_t *req;
struct p9_client *clnt;
p9_debug(P9_DEBUG_9P,
- ">>> TXATTRCREATE fid %d name %s size %lld flag %d\n",
- fid->fid, name, (long long)attr_size, flags);
+ ">>> TXATTRCREATE fid %d name %s size %llu flag %d\n",
+ fid->fid, name, attr_size, flags);
err = 0;
clnt = fid->clnt;
req = p9_client_rpc(clnt, P9_TXATTRCREATE, "dsqd",
- fid->fid, name, attr_size, flags);
+ fid->fid, name, attr_size, flags);
if (IS_ERR(req)) {
err = PTR_ERR(req);
goto error;
iov_iter_kvec(&to, READ, &kv, 1, count);
p9_debug(P9_DEBUG_9P, ">>> TREADDIR fid %d offset %llu count %d\n",
- fid->fid, (unsigned long long) offset, count);
+ fid->fid, offset, count);
err = 0;
clnt = fid->clnt;
rsize = fid->iounit;
- if (!rsize || rsize > clnt->msize-P9_READDIRHDRSZ)
+ if (!rsize || rsize > clnt->msize - P9_READDIRHDRSZ)
rsize = clnt->msize - P9_READDIRHDRSZ;
if (count < rsize)
/* Don't bother zerocopy for small IO (< 1024) */
if (clnt->trans_mod->zc_request && rsize > 1024) {
- /*
- * response header len is 11
+ /* response header len is 11
* PDU Header(7) + IO Size (4)
*/
req = p9_client_zc_rpc(clnt, P9_TREADDIR, &to, NULL, rsize, 0,
EXPORT_SYMBOL(p9_client_readdir);
int p9_client_mknod_dotl(struct p9_fid *fid, const char *name, int mode,
- dev_t rdev, kgid_t gid, struct p9_qid *qid)
+ dev_t rdev, kgid_t gid, struct p9_qid *qid)
{
int err;
struct p9_client *clnt;
err = 0;
clnt = fid->clnt;
- p9_debug(P9_DEBUG_9P, ">>> TMKNOD fid %d name %s mode %d major %d "
- "minor %d\n", fid->fid, name, mode, MAJOR(rdev), MINOR(rdev));
+ p9_debug(P9_DEBUG_9P,
+ ">>> TMKNOD fid %d name %s mode %d major %d minor %d\n",
+ fid->fid, name, mode, MAJOR(rdev), MINOR(rdev));
req = p9_client_rpc(clnt, P9_TMKNOD, "dsdddg", fid->fid, name, mode,
- MAJOR(rdev), MINOR(rdev), gid);
+ MAJOR(rdev), MINOR(rdev), gid);
if (IS_ERR(req))
return PTR_ERR(req);
trace_9p_protocol_dump(clnt, &req->rc);
goto error;
}
- p9_debug(P9_DEBUG_9P, "<<< RMKNOD qid %x.%llx.%x\n", qid->type,
- (unsigned long long)qid->path, qid->version);
+ p9_debug(P9_DEBUG_9P, "<<< RMKNOD qid %x.%llx.%x\n",
+ qid->type, qid->path, qid->version);
error:
p9_tag_remove(clnt, req);
return err;
-
}
EXPORT_SYMBOL(p9_client_mknod_dotl);
int p9_client_mkdir_dotl(struct p9_fid *fid, const char *name, int mode,
- kgid_t gid, struct p9_qid *qid)
+ kgid_t gid, struct p9_qid *qid)
{
int err;
struct p9_client *clnt;
clnt = fid->clnt;
p9_debug(P9_DEBUG_9P, ">>> TMKDIR fid %d name %s mode %d gid %d\n",
fid->fid, name, mode, from_kgid(&init_user_ns, gid));
- req = p9_client_rpc(clnt, P9_TMKDIR, "dsdg", fid->fid, name, mode,
- gid);
+ req = p9_client_rpc(clnt, P9_TMKDIR, "dsdg",
+ fid->fid, name, mode, gid);
if (IS_ERR(req))
return PTR_ERR(req);
goto error;
}
p9_debug(P9_DEBUG_9P, "<<< RMKDIR qid %x.%llx.%x\n", qid->type,
- (unsigned long long)qid->path, qid->version);
+ qid->path, qid->version);
error:
p9_tag_remove(clnt, req);
return err;
-
}
EXPORT_SYMBOL(p9_client_mkdir_dotl);
err = 0;
clnt = fid->clnt;
- p9_debug(P9_DEBUG_9P, ">>> TLOCK fid %d type %i flags %d "
- "start %lld length %lld proc_id %d client_id %s\n",
- fid->fid, flock->type, flock->flags, flock->start,
- flock->length, flock->proc_id, flock->client_id);
+ p9_debug(P9_DEBUG_9P,
+ ">>> TLOCK fid %d type %i flags %d start %lld length %lld proc_id %d client_id %s\n",
+ fid->fid, flock->type, flock->flags, flock->start,
+ flock->length, flock->proc_id, flock->client_id);
req = p9_client_rpc(clnt, P9_TLOCK, "dbdqqds", fid->fid, flock->type,
- flock->flags, flock->start, flock->length,
- flock->proc_id, flock->client_id);
+ flock->flags, flock->start, flock->length,
+ flock->proc_id, flock->client_id);
if (IS_ERR(req))
return PTR_ERR(req);
error:
p9_tag_remove(clnt, req);
return err;
-
}
EXPORT_SYMBOL(p9_client_lock_dotl);
err = 0;
clnt = fid->clnt;
- p9_debug(P9_DEBUG_9P, ">>> TGETLOCK fid %d, type %i start %lld "
- "length %lld proc_id %d client_id %s\n", fid->fid, glock->type,
- glock->start, glock->length, glock->proc_id, glock->client_id);
+ p9_debug(P9_DEBUG_9P,
+ ">>> TGETLOCK fid %d, type %i start %lld length %lld proc_id %d client_id %s\n",
+ fid->fid, glock->type, glock->start, glock->length,
+ glock->proc_id, glock->client_id);
- req = p9_client_rpc(clnt, P9_TGETLOCK, "dbqqds", fid->fid, glock->type,
- glock->start, glock->length, glock->proc_id, glock->client_id);
+ req = p9_client_rpc(clnt, P9_TGETLOCK, "dbqqds", fid->fid,
+ glock->type, glock->start, glock->length,
+ glock->proc_id, glock->client_id);
if (IS_ERR(req))
return PTR_ERR(req);
trace_9p_protocol_dump(clnt, &req->rc);
goto error;
}
- p9_debug(P9_DEBUG_9P, "<<< RGETLOCK type %i start %lld length %lld "
- "proc_id %d client_id %s\n", glock->type, glock->start,
- glock->length, glock->proc_id, glock->client_id);
+ p9_debug(P9_DEBUG_9P,
+ "<<< RGETLOCK type %i start %lld length %lld proc_id %d client_id %s\n",
+ glock->type, glock->start, glock->length,
+ glock->proc_id, glock->client_id);
error:
p9_tag_remove(clnt, req);
return err;
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/error.c
- *
* Error string handling
*
* Plan 9 uses error strings, Unix uses error numbers. These functions
INIT_HLIST_HEAD(&hash_errmap[bucket]);
/* load initial error map into hash table */
- for (c = errmap; c->name != NULL; c++) {
+ for (c = errmap; c->name; c++) {
c->namelen = strlen(c->name);
bucket = jhash(c->name, c->namelen, 0) % ERRHASHSZ;
INIT_HLIST_NODE(&c->list);
// SPDX-License-Identifier: GPL-2.0-only
/*
- * net/9p/9p.c
- *
* 9P entry point
*
* Copyright (C) 2007 by Latchesar Ionkov <lucho@ionkov.net>
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>
+#include <linux/kmod.h>
#include <linux/errno.h>
#include <linux/sched.h>
#include <linux/moduleparam.h>
#include <linux/spinlock.h>
#ifdef CONFIG_NET_9P_DEBUG
-unsigned int p9_debug_level = 0; /* feature-rific global debug level */
+unsigned int p9_debug_level; /* feature-rific global debug level */
EXPORT_SYMBOL(p9_debug_level);
module_param_named(debug, p9_debug_level, uint, 0);
MODULE_PARM_DESC(debug, "9P debugging level");
void _p9_debug(enum p9_debug_flags level, const char *func,
- const char *fmt, ...)
+ const char *fmt, ...)
{
struct va_format vaf;
va_list args;
EXPORT_SYMBOL(_p9_debug);
#endif
-/*
- * Dynamic Transport Registration Routines
- *
- */
+/* Dynamic Transport Registration Routines */
static DEFINE_SPINLOCK(v9fs_trans_lock);
static LIST_HEAD(v9fs_trans_list);
}
EXPORT_SYMBOL(v9fs_unregister_trans);
-/**
- * v9fs_get_trans_by_name - get transport with the matching name
- * @s: string identifying transport
- *
- */
-struct p9_trans_module *v9fs_get_trans_by_name(char *s)
+static struct p9_trans_module *_p9_get_trans_by_name(char *s)
{
struct p9_trans_module *t, *found = NULL;
}
spin_unlock(&v9fs_trans_lock);
+
+ return found;
+}
+
+/**
+ * v9fs_get_trans_by_name - get transport with the matching name
+ * @s: string identifying transport
+ *
+ */
+struct p9_trans_module *v9fs_get_trans_by_name(char *s)
+{
+ struct p9_trans_module *found = NULL;
+
+ found = _p9_get_trans_by_name(s);
+
+#ifdef CONFIG_MODULES
+ if (!found) {
+ request_module("9p-%s", s);
+ found = _p9_get_trans_by_name(s);
+ }
+#endif
+
return found;
}
EXPORT_SYMBOL(v9fs_get_trans_by_name);
// SPDX-License-Identifier: GPL-2.0-only
/*
- * net/9p/protocol.c
- *
* 9P Protocol Support Code
*
* Copyright (C) 2008 by Eric Van Hensbergen <ericvh@gmail.com>
size_t pdu_read(struct p9_fcall *pdu, void *data, size_t size)
{
size_t len = min(pdu->size - pdu->offset, size);
+
memcpy(data, &pdu->sdata[pdu->offset], len);
pdu->offset += len;
return size - len;
static size_t pdu_write(struct p9_fcall *pdu, const void *data, size_t size)
{
size_t len = min(pdu->capacity - pdu->size, size);
+
memcpy(&pdu->sdata[pdu->size], data, len);
pdu->size += len;
return size - len;
{
size_t len = min(pdu->capacity - pdu->size, size);
struct iov_iter i = *from;
+
if (!copy_from_iter_full(&pdu->sdata[pdu->size], len, &i))
len = 0;
return size - len;
}
-/*
- b - int8_t
- w - int16_t
- d - int32_t
- q - int64_t
- s - string
- u - numeric uid
- g - numeric gid
- S - stat
- Q - qid
- D - data blob (int32_t size followed by void *, results are not freed)
- T - array of strings (int16_t count, followed by strings)
- R - array of qids (int16_t count, followed by qids)
- A - stat for 9p2000.L (p9_stat_dotl)
- ? - if optional = 1, continue parsing
-*/
+/* b - int8_t
+ * w - int16_t
+ * d - int32_t
+ * q - int64_t
+ * s - string
+ * u - numeric uid
+ * g - numeric gid
+ * S - stat
+ * Q - qid
+ * D - data blob (int32_t size followed by void *, results are not freed)
+ * T - array of strings (int16_t count, followed by strings)
+ * R - array of qids (int16_t count, followed by qids)
+ * A - stat for 9p2000.L (p9_stat_dotl)
+ * ? - if optional = 1, continue parsing
+ */
static int
p9pdu_vreadf(struct p9_fcall *pdu, int proto_version, const char *fmt,
- va_list ap)
+ va_list ap)
{
const char *ptr;
int errcode = 0;
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * net/9p/protocol.h
- *
* 9P Protocol Support Code
*
* Copyright (C) 2008 by Eric Van Hensbergen <ericvh@gmail.com>
*/
int p9pdu_vwritef(struct p9_fcall *pdu, int proto_version, const char *fmt,
- va_list ap);
+ va_list ap);
int p9pdu_readf(struct p9_fcall *pdu, int proto_version, const char *fmt, ...);
int p9pdu_prepare(struct p9_fcall *pdu, int16_t tag, int8_t type);
int p9pdu_finalize(struct p9_client *clnt, struct p9_fcall *pdu);
+// SPDX-License-Identifier: LGPL-2.1
/*
* Copyright IBM Corporation, 2010
* Author Venkateswararao Jujjuri <jvrao@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
#include <linux/mm.h>
+/* SPDX-License-Identifier: LGPL-2.1 */
/*
* Copyright IBM Corporation, 2010
* Author Venkateswararao Jujjuri <jvrao@linux.vnet.ibm.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of version 2.1 of the GNU Lesser General Public License
- * as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it would be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
- *
*/
-void p9_release_pages(struct page **, int);
+void p9_release_pages(struct page **pages, int nr_pages);
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/trans_fd.c
- *
* Fd transport layer. Includes deprecated socket layer.
*
* Copyright (C) 2006 by Russ Cox <rsc@swtch.com>
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/fs/9p/trans_rdma.c
- *
* RDMA transport layer based on the trans_fd.c implementation.
*
* Copyright (C) 2008 by Tom Tucker <tom@opengridcomputing.com>
module_init(p9_trans_rdma_init);
module_exit(p9_trans_rdma_exit);
+MODULE_ALIAS_9P("rdma");
MODULE_AUTHOR("Tom Tucker <tom@opengridcomputing.com>");
MODULE_DESCRIPTION("RDMA Transport for 9P");
module_init(p9_virtio_init);
module_exit(p9_virtio_cleanup);
+MODULE_ALIAS_9P("virtio");
MODULE_DEVICE_TABLE(virtio, id_table);
MODULE_AUTHOR("Eric Van Hensbergen <ericvh@gmail.com>");
+// SPDX-License-Identifier: GPL-2.0-only
/*
* linux/fs/9p/trans_xen
*
* Xen transport layer.
*
* Copyright (C) 2017 by Stefano Stabellini <stefano@aporeto.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License version 2
- * as published by the Free Software Foundation; or, when distributed
- * separately from the Linux kernel or incorporated into other
- * software packages, subject to the following license:
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this source file (the "Software"), to deal in the Software without
- * restriction, including without limitation the rights to use, copy, modify,
- * merge, publish, distribute, sublicense, and/or sell copies of the Software,
- * and to permit persons to whom the Software is furnished to do so, subject to
- * the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
- * IN THE SOFTWARE.
*/
#include <xen/events.h>
return rc;
}
module_init(p9_trans_xen_init);
+MODULE_ALIAS_9P("xen");
static void p9_trans_xen_exit(void)
{
batadv_tp_vars_put(tp_vars);
- do_exit(0);
+ return 0;
}
/**
skcb->addr.pgn = (cf->can_id >> 8) & J1939_PGN_MAX;
/* set default message type */
skcb->addr.type = J1939_TP;
+
+ if (!j1939_address_is_valid(skcb->addr.sa)) {
+ netdev_err_once(priv->ndev, "%s: sa is broadcast address, ignoring!\n",
+ __func__);
+ goto done;
+ }
+
if (j1939_pgn_is_pdu1(skcb->addr.pgn)) {
/* Type 1: with destination address */
skcb->addr.da = skcb->addr.pgn;
extd = J1939_ETP;
fallthrough;
case J1939_TP_CMD_BAM:
+ if (cmd == J1939_TP_CMD_BAM && !j1939_cb_is_broadcast(skcb)) {
+ netdev_err_once(priv->ndev, "%s: BAM to unicast (%02x), ignoring!\n",
+ __func__, skcb->addr.sa);
+ return;
+ }
fallthrough;
case J1939_TP_CMD_RTS:
if (skcb->addr.type != extd)
break;
case J1939_ETP_CMD_ABORT: /* && J1939_TP_CMD_ABORT */
+ if (j1939_cb_is_broadcast(skcb)) {
+ netdev_err_once(priv->ndev, "%s: abort to broadcast (%02x), ignoring!\n",
+ __func__, skcb->addr.sa);
+ return;
+ }
+
if (j1939_tp_im_transmitter(skcb))
j1939_xtp_rx_abort(priv, skb, true);
int ceph_monc_init(struct ceph_mon_client *monc, struct ceph_client *cl)
{
- int err = 0;
+ int err;
dout("init\n");
memset(monc, 0, sizeof(*monc));
monc->client = cl;
- monc->monmap = NULL;
mutex_init(&monc->mutex);
err = build_initial_monmap(monc);
ceph_msgpool_destroy(&osdc->msgpool_op_reply);
}
-static int osd_req_op_copy_from_init(struct ceph_osd_request *req,
- u64 src_snapid, u64 src_version,
- struct ceph_object_id *src_oid,
- struct ceph_object_locator *src_oloc,
- u32 src_fadvise_flags,
- u32 dst_fadvise_flags,
- u32 truncate_seq, u64 truncate_size,
- u8 copy_from_flags)
+int osd_req_op_copy_from_init(struct ceph_osd_request *req,
+ u64 src_snapid, u64 src_version,
+ struct ceph_object_id *src_oid,
+ struct ceph_object_locator *src_oloc,
+ u32 src_fadvise_flags,
+ u32 dst_fadvise_flags,
+ u32 truncate_seq, u64 truncate_size,
+ u8 copy_from_flags)
{
struct ceph_osd_req_op *op;
struct page **pages;
op->indata_len, 0, false, true);
return 0;
}
-
-int ceph_osdc_copy_from(struct ceph_osd_client *osdc,
- u64 src_snapid, u64 src_version,
- struct ceph_object_id *src_oid,
- struct ceph_object_locator *src_oloc,
- u32 src_fadvise_flags,
- struct ceph_object_id *dst_oid,
- struct ceph_object_locator *dst_oloc,
- u32 dst_fadvise_flags,
- u32 truncate_seq, u64 truncate_size,
- u8 copy_from_flags)
-{
- struct ceph_osd_request *req;
- int ret;
-
- req = ceph_osdc_alloc_request(osdc, NULL, 1, false, GFP_KERNEL);
- if (!req)
- return -ENOMEM;
-
- req->r_flags = CEPH_OSD_FLAG_WRITE;
-
- ceph_oloc_copy(&req->r_t.base_oloc, dst_oloc);
- ceph_oid_copy(&req->r_t.base_oid, dst_oid);
-
- ret = osd_req_op_copy_from_init(req, src_snapid, src_version, src_oid,
- src_oloc, src_fadvise_flags,
- dst_fadvise_flags, truncate_seq,
- truncate_size, copy_from_flags);
- if (ret)
- goto out;
-
- ret = ceph_osdc_alloc_messages(req, GFP_KERNEL);
- if (ret)
- goto out;
-
- ceph_osdc_start_request(osdc, req, false);
- ret = ceph_osdc_wait_request(osdc, req);
-
-out:
- ceph_osdc_put_request(req);
- return ret;
-}
-EXPORT_SYMBOL(ceph_osdc_copy_from);
+EXPORT_SYMBOL(osd_req_op_copy_from_init);
int __init ceph_osdc_setup(void)
{
skb->truesize += truesize;
if (sk && sk->sk_type == SOCK_STREAM) {
sk_wmem_queued_add(sk, truesize);
- sk_mem_charge(sk, truesize);
+ if (!skb_zcopy_pure(skb))
+ sk_mem_charge(sk, truesize);
} else {
refcount_add(truesize, &skb->sk->sk_wmem_alloc);
}
might_sleep();
set_bit(NAPI_STATE_DISABLE, &n->state);
- do {
+ for ( ; ; ) {
val = READ_ONCE(n->state);
if (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) {
usleep_range(20, 200);
new = val | NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC;
new &= ~(NAPIF_STATE_THREADED | NAPIF_STATE_PREFER_BUSY_POLL);
- } while (cmpxchg(&n->state, val, new) != val);
+
+ if (cmpxchg(&n->state, val, new) == val)
+ break;
+ }
hrtimer_cancel(&n->timer);
u8 reload_failed:1;
refcount_t refcount;
struct completion comp;
- char priv[0] __aligned(NETDEV_ALIGN);
+ char priv[] __aligned(NETDEV_ALIGN);
};
void *devlink_priv(struct devlink *devlink)
WARN_ON(cmd != DEVLINK_CMD_FLASH_UPDATE &&
cmd != DEVLINK_CMD_FLASH_UPDATE_END &&
cmd != DEVLINK_CMD_FLASH_UPDATE_STATUS);
- WARN_ON(!xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED));
+
+ if (!xa_get_mark(&devlinks, devlink->index, DEVLINK_REGISTERED))
+ return;
msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL);
if (!msg)
#endif
case BPF_FUNC_sk_storage_get:
return &bpf_sk_storage_get_cg_sock_proto;
+ case BPF_FUNC_ktime_get_coarse_ns:
+ return &bpf_ktime_get_coarse_ns_proto;
default:
return bpf_base_func_proto(func_id);
}
static struct bpf_insn *bpf_convert_data_end_access(const struct bpf_insn *si,
struct bpf_insn *insn)
{
- /* si->dst_reg = skb->data */
+ int reg;
+ int temp_reg_off = offsetof(struct sk_buff, cb) +
+ offsetof(struct sk_skb_cb, temp_reg);
+
+ if (si->src_reg == si->dst_reg) {
+ /* We need an extra register, choose and save a register. */
+ reg = BPF_REG_9;
+ if (si->src_reg == reg || si->dst_reg == reg)
+ reg--;
+ if (si->src_reg == reg || si->dst_reg == reg)
+ reg--;
+ *insn++ = BPF_STX_MEM(BPF_DW, si->src_reg, reg, temp_reg_off);
+ } else {
+ reg = si->dst_reg;
+ }
+
+ /* reg = skb->data */
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, data),
- si->dst_reg, si->src_reg,
+ reg, si->src_reg,
offsetof(struct sk_buff, data));
/* AX = skb->len */
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, len),
BPF_REG_AX, si->src_reg,
offsetof(struct sk_buff, len));
- /* si->dst_reg = skb->data + skb->len */
- *insn++ = BPF_ALU64_REG(BPF_ADD, si->dst_reg, BPF_REG_AX);
+ /* reg = skb->data + skb->len */
+ *insn++ = BPF_ALU64_REG(BPF_ADD, reg, BPF_REG_AX);
/* AX = skb->data_len */
*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct sk_buff, data_len),
BPF_REG_AX, si->src_reg,
offsetof(struct sk_buff, data_len));
- /* si->dst_reg = skb->data + skb->len - skb->data_len */
- *insn++ = BPF_ALU64_REG(BPF_SUB, si->dst_reg, BPF_REG_AX);
+
+ /* reg = skb->data + skb->len - skb->data_len */
+ *insn++ = BPF_ALU64_REG(BPF_SUB, reg, BPF_REG_AX);
+
+ if (si->src_reg == si->dst_reg) {
+ /* Restore the saved register */
+ *insn++ = BPF_MOV64_REG(BPF_REG_AX, si->src_reg);
+ *insn++ = BPF_MOV64_REG(si->dst_reg, reg);
+ *insn++ = BPF_LDX_MEM(BPF_DW, reg, BPF_REG_AX, temp_reg_off);
+ }
return insn;
}
struct bpf_prog *prog, u32 *target_size)
{
struct bpf_insn *insn = insn_buf;
+ int off;
switch (si->off) {
case offsetof(struct __sk_buff, data_end):
insn = bpf_convert_data_end_access(si, insn);
break;
+ case offsetof(struct __sk_buff, cb[0]) ...
+ offsetofend(struct __sk_buff, cb[4]) - 1:
+ BUILD_BUG_ON(sizeof_field(struct sk_skb_cb, data) < 20);
+ BUILD_BUG_ON((offsetof(struct sk_buff, cb) +
+ offsetof(struct sk_skb_cb, data)) %
+ sizeof(__u64));
+
+ prog->cb_access = 1;
+ off = si->off;
+ off -= offsetof(struct __sk_buff, cb[0]);
+ off += offsetof(struct sk_buff, cb);
+ off += offsetof(struct sk_skb_cb, data);
+ if (type == BPF_WRITE)
+ *insn++ = BPF_STX_MEM(BPF_SIZE(si->code), si->dst_reg,
+ si->src_reg, off);
+ else
+ *insn++ = BPF_LDX_MEM(BPF_SIZE(si->code), si->dst_reg,
+ si->src_reg, off);
+ break;
+
+
default:
return bpf_convert_ctx_access(type, si, insn_buf, prog,
target_size);
return &sk_reuseport_load_bytes_relative_proto;
case BPF_FUNC_get_socket_cookie:
return &bpf_get_socket_ptr_cookie_proto;
+ case BPF_FUNC_ktime_get_coarse_ns:
+ return &bpf_ktime_get_coarse_ns_proto;
default:
return bpf_base_func_proto(func_id);
}
return -EINVAL;
if (unlikely(sk && sk_is_refcounted(sk)))
return -ESOCKTNOSUPPORT; /* reject non-RCU freed sockets */
- if (unlikely(sk && sk->sk_state == TCP_ESTABLISHED))
- return -ESOCKTNOSUPPORT; /* reject connected sockets */
+ if (unlikely(sk && sk_is_tcp(sk) && sk->sk_state != TCP_LISTEN))
+ return -ESOCKTNOSUPPORT; /* only accept TCP socket in LISTEN */
+ if (unlikely(sk && sk_is_udp(sk) && sk->sk_state != TCP_CLOSE))
+ return -ESOCKTNOSUPPORT; /* only accept UDP socket in CLOSE */
/* Check if socket is suitable for packet L3/L4 protocol */
if (sk && sk->sk_protocol != ctx->protocol)
case BPF_FUNC_skc_to_unix_sock:
func = &bpf_skc_to_unix_sock_proto;
break;
+ case BPF_FUNC_ktime_get_coarse_ns:
+ return &bpf_ktime_get_coarse_ns_proto;
default:
return bpf_base_func_proto(func_id);
}
* which is the XDP_TX use-case.
*/
if (pool->p.flags & PP_FLAG_DMA_MAP) {
- /* DMA-mapping is not supported on 32-bit systems with
- * 64-bit DMA mapping.
- */
- if (sizeof(dma_addr_t) > sizeof(unsigned long))
- return -EOPNOTSUPP;
-
if ((pool->p.dma_dir != DMA_FROM_DEVICE) &&
(pool->p.dma_dir != DMA_BIDIRECTIONAL))
return -EINVAL;
*/
}
+ if (PAGE_POOL_DMA_USE_PP_FRAG_COUNT &&
+ pool->p.flags & PP_FLAG_PAGE_FRAG)
+ return -EINVAL;
+
if (ptr_ring_init(&pool->ring, ring_qsize, GFP_KERNEL) < 0)
return -ENOMEM;
void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)
{
int pos = skb_headlen(skb);
+ const int zc_flags = SKBFL_SHARED_FRAG | SKBFL_PURE_ZEROCOPY;
- skb_shinfo(skb1)->flags |= skb_shinfo(skb)->flags & SKBFL_SHARED_FRAG;
+ skb_shinfo(skb1)->flags |= skb_shinfo(skb)->flags & zc_flags;
skb_zerocopy_clone(skb1, skb, 0);
if (len < pos) /* Split line is inside header. */
skb_split_inside_header(skb, skb1, len, pos);
*/
static int skb_prepare_for_shift(struct sk_buff *skb)
{
- int ret = 0;
-
- if (skb_cloned(skb)) {
- /* Save and restore truesize: pskb_expand_head() may reallocate
- * memory where ksize(kmalloc(S)) != ksize(kmalloc(S)), but we
- * cannot change truesize at this point.
- */
- unsigned int save_truesize = skb->truesize;
-
- ret = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
- skb->truesize = save_truesize;
- }
- return ret;
+ return skb_unclone_keeptruesize(skb, GFP_ATOMIC);
}
/**
bool charged;
int pages;
- if (!mem_cgroup_sockets_enabled || !sk->sk_memcg)
+ if (!mem_cgroup_sockets_enabled || !sk->sk_memcg || !sk_has_account(sk))
return -EOPNOTSUPP;
if (!bytes)
newsk->sk_prot_creator = prot;
/* SANITY */
- if (likely(newsk->sk_net_refcnt))
+ if (likely(newsk->sk_net_refcnt)) {
get_net(sock_net(newsk));
+ sock_inuse_add(sock_net(newsk), 1);
+ }
sk_node_init(&newsk->sk_node);
sock_lock_init(newsk);
bh_lock_sock(newsk);
newsk->sk_err_soft = 0;
newsk->sk_priority = 0;
newsk->sk_incoming_cpu = raw_smp_processor_id();
- if (likely(newsk->sk_net_refcnt))
- sock_inuse_add(sock_net(newsk), 1);
/* Before updating sk_refcnt, we must commit prior changes to memory
* (Documentation/RCU/rculist_nulls.rst for details)
ops->op == BPF_SOCK_OPS_TCP_LISTEN_CB;
}
-static bool sk_is_tcp(const struct sock *sk)
-{
- return sk->sk_type == SOCK_STREAM &&
- sk->sk_protocol == IPPROTO_TCP;
-}
-
static bool sock_map_redirect_allowed(const struct sock *sk)
{
if (sk_is_tcp(sk))
struct dsa_port *dp;
u8 *extraction;
u16 vlan_tpid;
+ u64 rew_val;
/* Revert skb->data by the amount consumed by the DSA master,
* so it points to the beginning of the frame.
ocelot_xfh_get_qos_class(extraction, &qos_class);
ocelot_xfh_get_tag_type(extraction, &tag_type);
ocelot_xfh_get_vlan_tci(extraction, &vlan_tci);
+ ocelot_xfh_get_rew_val(extraction, &rew_val);
skb->dev = dsa_master_find_slave(netdev, 0, src_port);
if (!skb->dev)
dsa_default_offload_fwd_mark(skb);
skb->priority = qos_class;
+ OCELOT_SKB_CB(skb)->tstamp_lo = rew_val;
/* Ocelot switches copy frames unmodified to the CPU. However, it is
* possible for the user to request a VLAN modification through
if (req_base->flags & ETHTOOL_FLAG_STATS)
n += nla_total_size(0) + /* _PAUSE_STATS */
- nla_total_size_64bit(sizeof(u64)) *
- (ETHTOOL_A_PAUSE_STAT_MAX - 2);
+ nla_total_size_64bit(sizeof(u64)) * ETHTOOL_PAUSE_STAT_CNT;
return n;
}
offsetof(struct tcp_congestion_ops, release))
return &bpf_sk_getsockopt_proto;
return NULL;
+ case BPF_FUNC_ktime_get_coarse_ns:
+ return &bpf_ktime_get_coarse_ns_proto;
default:
return bpf_base_func_proto(func_id);
}
free:
kfree(t);
out:
- return -ENOBUFS;
+ return -ENOMEM;
}
static void __devinet_sysctl_unregister(struct net *net,
#include <linux/random.h>
#include <linux/memblock.h>
#include <linux/highmem.h>
-#include <linux/swap.h>
#include <linux/cache.h>
#include <linux/err.h>
#include <linux/time.h>
if (likely(skb)) {
bool mem_scheduled;
+ skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
if (force_schedule) {
mem_scheduled = true;
sk_forced_mem_schedule(sk, skb->truesize);
copy = min_t(int, copy, pfrag->size - pfrag->offset);
+ /* skb changing from pure zc to mixed, must charge zc */
+ if (unlikely(skb_zcopy_pure(skb))) {
+ if (!sk_wmem_schedule(sk, skb->data_len))
+ goto wait_for_space;
+
+ sk_mem_charge(sk, skb->data_len);
+ skb_shinfo(skb)->flags &= ~SKBFL_PURE_ZEROCOPY;
+ }
+
if (!sk_wmem_schedule(sk, copy))
goto wait_for_space;
}
pfrag->offset += copy;
} else {
- if (!sk_wmem_schedule(sk, copy))
- goto wait_for_space;
+ /* First append to a fragless skb builds initial
+ * pure zerocopy skb
+ */
+ if (!skb->len)
+ skb_shinfo(skb)->flags |= SKBFL_PURE_ZEROCOPY;
+
+ if (!skb_zcopy_pure(skb)) {
+ if (!sk_wmem_schedule(sk, copy))
+ goto wait_for_space;
+ }
err = skb_zerocopy_iter_stream(sk, skb, msg, copy, uarg);
if (err == -EMSGSIZE || err == -EEXIST) {
{
skb_frag_t *frag;
+ if (unlikely(offset_skb >= skb->len))
+ return NULL;
+
offset_skb -= skb_headlen(skb);
if ((int)offset_skb < 0 || skb_has_frag_list(skb))
return NULL;
return ret;
}
+static int tcp_bpf_recvmsg_parser(struct sock *sk,
+ struct msghdr *msg,
+ size_t len,
+ int nonblock,
+ int flags,
+ int *addr_len)
+{
+ struct sk_psock *psock;
+ int copied;
+
+ if (unlikely(flags & MSG_ERRQUEUE))
+ return inet_recv_error(sk, msg, len, addr_len);
+
+ psock = sk_psock_get(sk);
+ if (unlikely(!psock))
+ return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
+
+ lock_sock(sk);
+msg_bytes_ready:
+ copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
+ if (!copied) {
+ long timeo;
+ int data;
+
+ timeo = sock_rcvtimeo(sk, nonblock);
+ data = tcp_msg_wait_data(sk, psock, timeo);
+ if (data && !sk_psock_queue_empty(psock))
+ goto msg_bytes_ready;
+ copied = -EAGAIN;
+ }
+ release_sock(sk);
+ sk_psock_put(sk, psock);
+ return copied;
+}
+
static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
int nonblock, int flags, int *addr_len)
{
enum {
TCP_BPF_BASE,
TCP_BPF_TX,
+ TCP_BPF_RX,
+ TCP_BPF_TXRX,
TCP_BPF_NUM_CFGS,
};
struct proto *base)
{
prot[TCP_BPF_BASE] = *base;
- prot[TCP_BPF_BASE].unhash = sock_map_unhash;
prot[TCP_BPF_BASE].close = sock_map_close;
prot[TCP_BPF_BASE].recvmsg = tcp_bpf_recvmsg;
prot[TCP_BPF_BASE].sock_is_readable = sk_msg_is_readable;
prot[TCP_BPF_TX] = prot[TCP_BPF_BASE];
prot[TCP_BPF_TX].sendmsg = tcp_bpf_sendmsg;
prot[TCP_BPF_TX].sendpage = tcp_bpf_sendpage;
+
+ prot[TCP_BPF_RX] = prot[TCP_BPF_BASE];
+ prot[TCP_BPF_RX].recvmsg = tcp_bpf_recvmsg_parser;
+
+ prot[TCP_BPF_TXRX] = prot[TCP_BPF_TX];
+ prot[TCP_BPF_TXRX].recvmsg = tcp_bpf_recvmsg_parser;
}
static void tcp_bpf_check_v6_needs_rebuild(struct proto *ops)
int family = sk->sk_family == AF_INET6 ? TCP_BPF_IPV6 : TCP_BPF_IPV4;
int config = psock->progs.msg_parser ? TCP_BPF_TX : TCP_BPF_BASE;
+ if (psock->progs.stream_verdict || psock->progs.skb_verdict) {
+ config = (config == TCP_BPF_TX) ? TCP_BPF_TXRX : TCP_BPF_RX;
+ }
+
if (restore) {
if (inet_csk_has_ulp(sk)) {
/* TLS does not have an unhash proto in SW cases,
return tp->snd_una != tp->snd_up;
}
-#define OPTION_SACK_ADVERTISE (1 << 0)
-#define OPTION_TS (1 << 1)
-#define OPTION_MD5 (1 << 2)
-#define OPTION_WSCALE (1 << 3)
-#define OPTION_FAST_OPEN_COOKIE (1 << 8)
-#define OPTION_SMC (1 << 9)
-#define OPTION_MPTCP (1 << 10)
+#define OPTION_SACK_ADVERTISE BIT(0)
+#define OPTION_TS BIT(1)
+#define OPTION_MD5 BIT(2)
+#define OPTION_WSCALE BIT(3)
+#define OPTION_FAST_OPEN_COOKIE BIT(8)
+#define OPTION_SMC BIT(9)
+#define OPTION_MPTCP BIT(10)
static void smc_options_write(__be32 *ptr, u16 *options)
{
return -ENOMEM;
}
- if (skb_unclone(skb, gfp))
+ if (skb_unclone_keeptruesize(skb, gfp))
return -ENOMEM;
/* Get a new skb... force flag on. */
{
u32 delta_truesize;
- if (skb_unclone(skb, GFP_ATOMIC))
+ if (skb_unclone_keeptruesize(skb, GFP_ATOMIC))
return -ENOMEM;
delta_truesize = __pskb_trim_head(skb, len);
if (delta_truesize) {
skb->truesize -= delta_truesize;
sk_wmem_queued_add(sk, -delta_truesize);
- sk_mem_uncharge(sk, delta_truesize);
+ if (!skb_zcopy_pure(skb))
+ sk_mem_uncharge(sk, delta_truesize);
}
/* Any change of skb->len requires recalculation of tso factor. */
if (len <= skb->len)
break;
- if (unlikely(TCP_SKB_CB(skb)->eor) || tcp_has_tx_tstamp(skb))
+ if (unlikely(TCP_SKB_CB(skb)->eor) ||
+ tcp_has_tx_tstamp(skb) ||
+ !skb_pure_zcopy_same(skb, next))
return false;
len -= skb->len;
cur_mss, GFP_ATOMIC))
return -ENOMEM; /* We'll try again later. */
} else {
- if (skb_unclone(skb, GFP_ATOMIC))
+ if (skb_unclone_keeptruesize(skb, GFP_ATOMIC))
return -ENOMEM;
diff = tcp_skb_pcount(skb);
#include <asm/ioctls.h>
#include <linux/memblock.h>
#include <linux/highmem.h>
-#include <linux/swap.h>
#include <linux/types.h>
#include <linux/fcntl.h>
#include <linux/module.h>
skb = skb_recv_udp(sk, 0, 1, &err);
if (!skb)
return err;
+
+ if (udp_lib_checksum_complete(skb)) {
+ __UDP_INC_STATS(sock_net(sk), UDP_MIB_CSUMERRORS,
+ IS_UDPLITE(sk));
+ __UDP_INC_STATS(sock_net(sk), UDP_MIB_INERRORS,
+ IS_UDPLITE(sk));
+ atomic_inc(&sk->sk_drops);
+ kfree_skb(skb);
+ continue;
+ }
+
used = recv_actor(desc, skb, 0, skb->len);
if (used <= 0) {
if (!copied)
struct tcphdr *th;
offset = ipv6_skip_exthdr(skb, offset, &nexthdr, &frag_off);
+
+ if (offset < 0) {
+ err = -EINVAL;
+ goto out;
+ }
+
uh = (void *)(skb->data + offset);
th = (void *)(skb->data + offset);
hdr_len += offset;
kfree(rcu_dereference_raw(sdata->tun_src));
kfree(sdata);
return -ENOMEM;
- };
+ }
#endif
return 0;
inet_sk(newsk)->pinet6 = tcp_inet6_sk(newsk);
- newinet = inet_sk(newsk);
newnp = tcp_inet6_sk(newsk);
newtp = tcp_sk(newsk);
ret = encap_rcv(sk, skb);
if (ret <= 0) {
- __UDP_INC_STATS(sock_net(sk),
- UDP_MIB_INDATAGRAMS,
- is_udplite);
+ __UDP6_INC_STATS(sock_net(sk),
+ UDP_MIB_INDATAGRAMS,
+ is_udplite);
return -ret;
}
}
}
/* also validate MU-MIMO change */
- monitor_sdata = rtnl_dereference(local->monitor_sdata);
+ monitor_sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
if (!monitor_sdata &&
(params->vht_mumimo_groups || params->vht_mumimo_follow_addr))
mutex_lock(&local->mtx);
if (local->use_chanctx) {
- sdata = rtnl_dereference(local->monitor_sdata);
+ sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
if (sdata) {
ieee80211_vif_release_channel(sdata);
ret = ieee80211_vif_use_channel(sdata, chandef,
sdata = IEEE80211_WDEV_TO_SUB_IF(wdev);
if (sdata->vif.type == NL80211_IFTYPE_MONITOR) {
- sdata = rtnl_dereference(local->monitor_sdata);
+ sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
if (!sdata)
return -EOPNOTSUPP;
}
mutex_unlock(&local->iflist_mtx);
if (has_monitor) {
- sdata = rtnl_dereference(local->monitor_sdata);
+ sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
if (sdata) {
sdata->user_power_level = local->user_power_level;
if (txp_type != sdata->vif.bss_conf.txpower_type)
*/
if (local->suspended) {
WARN_ON(local->wowlan);
- WARN_ON(rtnl_dereference(local->monitor_sdata));
+ WARN_ON(rcu_access_pointer(local->monitor_sdata));
return;
}
return 0;
ASSERT_RTNL();
+ lockdep_assert_wiphy(local->hw.wiphy);
if (local->monitor_sdata)
return 0;
return;
ASSERT_RTNL();
+ lockdep_assert_wiphy(local->hw.wiphy);
mutex_lock(&local->iflist_mtx);
#endif
static inline void
-ieee80211_tpt_led_trig_tx(struct ieee80211_local *local, __le16 fc, int bytes)
+ieee80211_tpt_led_trig_tx(struct ieee80211_local *local, int bytes)
{
#ifdef CONFIG_MAC80211_LEDS
- if (ieee80211_is_data(fc) && atomic_read(&local->tpt_led_active))
+ if (atomic_read(&local->tpt_led_active))
local->tpt_led_trigger->tx_bytes += bytes;
#endif
}
static inline void
-ieee80211_tpt_led_trig_rx(struct ieee80211_local *local, __le16 fc, int bytes)
+ieee80211_tpt_led_trig_rx(struct ieee80211_local *local, int bytes)
{
#ifdef CONFIG_MAC80211_LEDS
- if (ieee80211_is_data(fc) && atomic_read(&local->tpt_led_active))
+ if (atomic_read(&local->tpt_led_active))
local->tpt_led_trigger->rx_bytes += bytes;
#endif
}
* the compiler to think we have walked past the end of the
* struct member.
*/
- pos = (void *)&rthdr->it_optional[it_present - rthdr->it_optional];
+ pos = (void *)&rthdr->it_optional[it_present + 1 - rthdr->it_optional];
/* the order of the following fields is important */
int keyid = rx->sta->ptk_idx;
sta_ptk = rcu_dereference(rx->sta->ptk[keyid]);
- if (ieee80211_has_protected(fc)) {
+ if (ieee80211_has_protected(fc) &&
+ !(status->flag & RX_FLAG_IV_STRIPPED)) {
cs = rx->sta->cipher_scheme;
keyid = ieee80211_get_keyid(rx->skb, cs);
struct ieee80211_rate *rate = NULL;
struct ieee80211_supported_band *sband;
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
+ struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
WARN_ON_ONCE(softirq_count() == 0);
if (!(status->flag & RX_FLAG_8023))
skb = ieee80211_rx_monitor(local, skb, rate);
if (skb) {
- ieee80211_tpt_led_trig_rx(local,
- ((struct ieee80211_hdr *)skb->data)->frame_control,
- skb->len);
+ if ((status->flag & RX_FLAG_8023) ||
+ ieee80211_is_data_present(hdr->frame_control))
+ ieee80211_tpt_led_trig_rx(local, skb->len);
if (status->flag & RX_FLAG_8023)
__ieee80211_rx_handle_8023(hw, pubsta, skb, list);
* Returns false if the frame couldn't be transmitted but was queued instead.
*/
static bool __ieee80211_tx(struct ieee80211_local *local,
- struct sk_buff_head *skbs, int led_len,
- struct sta_info *sta, bool txpending)
+ struct sk_buff_head *skbs, struct sta_info *sta,
+ bool txpending)
{
struct ieee80211_tx_info *info;
struct ieee80211_sub_if_data *sdata;
struct ieee80211_vif *vif;
struct sk_buff *skb;
bool result;
- __le16 fc;
if (WARN_ON(skb_queue_empty(skbs)))
return true;
skb = skb_peek(skbs);
- fc = ((struct ieee80211_hdr *)skb->data)->frame_control;
info = IEEE80211_SKB_CB(skb);
sdata = vif_to_sdata(info->control.vif);
if (sta && !sta->uploaded)
result = ieee80211_tx_frags(local, vif, sta, skbs, txpending);
- ieee80211_tpt_led_trig_tx(local, fc, led_len);
-
WARN_ON_ONCE(!skb_queue_empty(skbs));
return result;
ieee80211_tx_result res_prepare;
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
bool result = true;
- int led_len;
if (unlikely(skb->len < 10)) {
dev_kfree_skb(skb);
}
/* initialises tx */
- led_len = skb->len;
res_prepare = ieee80211_tx_prepare(sdata, &tx, sta, skb);
if (unlikely(res_prepare == TX_DROP)) {
return true;
if (!invoke_tx_handlers_late(&tx))
- result = __ieee80211_tx(local, &tx.skbs, led_len,
- tx.sta, txpending);
+ result = __ieee80211_tx(local, &tx.skbs, tx.sta, txpending);
return result;
}
struct ieee80211_local *local = sdata->local;
struct sta_info *sta;
struct sk_buff *next;
+ int len = skb->len;
if (unlikely(skb->len < ETH_HLEN)) {
kfree_skb(skb);
}
} else {
/* we cannot process non-linear frames on this path */
- if (skb_linearize(skb)) {
- kfree_skb(skb);
- goto out;
- }
+ if (skb_linearize(skb))
+ goto out_free;
/* the frame could be fragmented, software-encrypted, and other
* things so we cannot really handle checksum offload with it -
goto out;
out_free:
kfree_skb(skb);
+ len = 0;
out:
+ if (len)
+ ieee80211_tpt_led_trig_tx(local, len);
rcu_read_unlock();
}
}
static bool ieee80211_tx_8023(struct ieee80211_sub_if_data *sdata,
- struct sk_buff *skb, int led_len,
- struct sta_info *sta,
+ struct sk_buff *skb, struct sta_info *sta,
bool txpending)
{
struct ieee80211_local *local = sdata->local;
if (sta)
sk_pacing_shift_update(skb->sk, local->hw.tx_sk_pacing_shift);
+ ieee80211_tpt_led_trig_tx(local, skb->len);
+
if (ieee80211_queue_skb(local, sdata, sta, skb))
return true;
if (key)
info->control.hw_key = &key->conf;
- ieee80211_tx_8023(sdata, skb, skb->len, sta, false);
+ ieee80211_tx_8023(sdata, skb, sta, false);
return;
if (IS_ERR(sta) || (sta && !sta->uploaded))
sta = NULL;
- result = ieee80211_tx_8023(sdata, skb, skb->len, sta, true);
+ result = ieee80211_tx_8023(sdata, skb, sta, true);
} else {
struct sk_buff_head skbs;
hdr = (struct ieee80211_hdr *)skb->data;
sta = sta_info_get(sdata, hdr->addr1);
- result = __ieee80211_tx(local, &skbs, skb->len, sta, true);
+ result = __ieee80211_tx(local, &skbs, sta, true);
}
return result;
sdata = rcu_dereference_check(local->monitor_sdata,
lockdep_is_held(&local->iflist_mtx) ||
- lockdep_rtnl_is_held());
+ lockdep_is_held(&local->hw.wiphy->mtx));
if (sdata &&
(iter_flags & IEEE80211_IFACE_ITER_RESUME_ALL || !active_only ||
sdata->flags & IEEE80211_SDATA_IN_DRIVER))
IEEE80211_TPT_LEDTRIG_FL_RADIO, 0);
/* add interfaces */
- sdata = rtnl_dereference(local->monitor_sdata);
+ sdata = wiphy_dereference(local->hw.wiphy, local->monitor_sdata);
if (sdata) {
/* in HW restart it exists already */
WARN_ON(local->resuming);
WARN_ON(drv_add_chanctx(local, ctx));
mutex_unlock(&local->chanctx_mtx);
- sdata = rtnl_dereference(local->monitor_sdata);
+ sdata = wiphy_dereference(local->hw.wiphy,
+ local->monitor_sdata);
if (sdata && ieee80211_sdata_running(sdata))
ieee80211_assign_chanctx(local, sdata);
}
u16 __ieee80211_select_queue(struct ieee80211_sub_if_data *sdata,
struct sta_info *sta, struct sk_buff *skb)
{
- struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
struct mac80211_qos_map *qos_map;
bool qos;
else
qos = false;
- if (!qos || (info->control.flags & IEEE80211_TX_CTRL_DONT_REORDER)) {
+ if (!qos) {
skb->priority = 0; /* required for correct WPA/11i MIC */
return IEEE80211_AC_BE;
}
return 0;
}
+/* Generic sockaddr checks, padding checks only so far */
+static bool mctp_sockaddr_is_ok(const struct sockaddr_mctp *addr)
+{
+ return !addr->__smctp_pad0 && !addr->__smctp_pad1;
+}
+
+static bool mctp_sockaddr_ext_is_ok(const struct sockaddr_mctp_ext *addr)
+{
+ return !addr->__smctp_pad0[0] &&
+ !addr->__smctp_pad0[1] &&
+ !addr->__smctp_pad0[2];
+}
+
static int mctp_bind(struct socket *sock, struct sockaddr *addr, int addrlen)
{
struct sock *sk = sock->sk;
/* it's a valid sockaddr for MCTP, cast and do protocol checks */
smctp = (struct sockaddr_mctp *)addr;
+ if (!mctp_sockaddr_is_ok(smctp))
+ return -EINVAL;
+
lock_sock(sk);
/* TODO: allow rebind */
return -EINVAL;
if (addr->smctp_family != AF_MCTP)
return -EINVAL;
+ if (!mctp_sockaddr_is_ok(addr))
+ return -EINVAL;
if (addr->smctp_tag & ~(MCTP_TAG_MASK | MCTP_TAG_OWNER))
return -EINVAL;
DECLARE_SOCKADDR(struct sockaddr_mctp_ext *,
extaddr, msg->msg_name);
- if (extaddr->smctp_halen > sizeof(cb->haddr)) {
+ if (!mctp_sockaddr_ext_is_ok(extaddr) ||
+ extaddr->smctp_halen > sizeof(cb->haddr)) {
rc = -EINVAL;
goto err_free;
}
addr = msg->msg_name;
addr->smctp_family = AF_MCTP;
+ addr->__smctp_pad0 = 0;
addr->smctp_network = cb->net;
addr->smctp_addr.s_addr = hdr->src;
addr->smctp_type = type;
addr->smctp_tag = hdr->flags_seq_tag &
(MCTP_HDR_TAG_MASK | MCTP_HDR_FLAG_TO);
+ addr->__smctp_pad1 = 0;
msg->msg_namelen = sizeof(*addr);
if (msk->addr_ext) {
msg->msg_namelen = sizeof(*ae);
ae->smctp_ifindex = cb->ifindex;
ae->smctp_halen = cb->halen;
+ memset(ae->__smctp_pad0, 0x0, sizeof(ae->__smctp_pad0));
memset(ae->smctp_haddr, 0x0, sizeof(ae->smctp_haddr));
memcpy(ae->smctp_haddr, cb->haddr, cb->halen);
}
#include <linux/sysctl.h>
#include <linux/proc_fs.h>
#include <linux/workqueue.h>
-#include <linux/swap.h>
#include <linux/seq_file.h>
#include <linux/slab.h>
#include <net/ip_vs.h>
+MODULE_ALIAS_GENL_FAMILY(IPVS_GENL_NAME);
+
/* semaphore for IPVS sockopts. And, [gs]etsockopt may sleep. */
static DEFINE_MUTEX(__ip_vs_mutex);
goto nla_put_failure;
if (indev && entskb->dev &&
- entskb->mac_header != entskb->network_header) {
+ skb_mac_header_was_set(entskb)) {
struct nfqnl_msg_packet_hw phw;
int len;
device_lock(&dev->dev);
- if (dev->rfkill && rfkill_blocked(dev->rfkill)) {
- rc = -ERFKILL;
+ if (!device_is_registered(&dev->dev)) {
+ rc = -ENODEV;
goto error;
}
- if (!device_is_registered(&dev->dev)) {
- rc = -ENODEV;
+ if (dev->rfkill && rfkill_blocked(dev->rfkill)) {
+ rc = -ERFKILL;
goto error;
}
if (rc)
pr_err("Could not register llcp device\n");
- rc = nfc_genl_device_added(dev);
- if (rc)
- pr_debug("The userspace won't be notified that the device %s was added\n",
- dev_name(&dev->dev));
-
+ device_lock(&dev->dev);
dev->rfkill = rfkill_alloc(dev_name(&dev->dev), &dev->dev,
RFKILL_TYPE_NFC, &nfc_rfkill_ops, dev);
if (dev->rfkill) {
dev->rfkill = NULL;
}
}
+ device_unlock(&dev->dev);
+
+ rc = nfc_genl_device_added(dev);
+ if (rc)
+ pr_debug("The userspace won't be notified that the device %s was added\n",
+ dev_name(&dev->dev));
return 0;
}
pr_debug("dev_name=%s\n", dev_name(&dev->dev));
+ rc = nfc_genl_device_removed(dev);
+ if (rc)
+ pr_debug("The userspace won't be notified that the device %s "
+ "was removed\n", dev_name(&dev->dev));
+
+ device_lock(&dev->dev);
if (dev->rfkill) {
rfkill_unregister(dev->rfkill);
rfkill_destroy(dev->rfkill);
}
+ device_unlock(&dev->dev);
if (dev->ops->check_presence) {
device_lock(&dev->dev);
cancel_work_sync(&dev->check_pres_work);
}
- rc = nfc_genl_device_removed(dev);
- if (rc)
- pr_debug("The userspace won't be notified that the device %s "
- "was removed\n", dev_name(&dev->dev));
-
nfc_llcp_unregister_device(dev);
mutex_lock(&nfc_devlist_mutex);
{
int rc;
- if (!test_bit(NCI_UP, &ndev->flags))
- return -ENETDOWN;
-
/* Serialize all requests */
mutex_lock(&ndev->req_lock);
- rc = __nci_request(ndev, req, opt, timeout);
+ /* check the state after obtaing the lock against any races
+ * from nci_close_device when the device gets removed.
+ */
+ if (test_bit(NCI_UP, &ndev->flags))
+ rc = __nci_request(ndev, req, opt, timeout);
+ else
+ rc = -ENETDOWN;
mutex_unlock(&ndev->req_lock);
return rc;
mutex_lock(&ndev->req_lock);
+ if (test_bit(NCI_UNREG, &ndev->flags)) {
+ rc = -ENODEV;
+ goto done;
+ }
+
if (test_bit(NCI_UP, &ndev->flags)) {
rc = -EALREADY;
goto done;
static int nci_close_device(struct nci_dev *ndev)
{
nci_req_cancel(ndev, ENODEV);
+
+ /* This mutex needs to be held as a barrier for
+ * caller nci_unregister_device
+ */
mutex_lock(&ndev->req_lock);
if (!test_and_clear_bit(NCI_UP, &ndev->flags)) {
del_timer_sync(&ndev->cmd_timer);
- /* Clear flags */
- ndev->flags = 0;
+ /* Clear flags except NCI_UNREG */
+ ndev->flags &= BIT(NCI_UNREG);
mutex_unlock(&ndev->req_lock);
{
struct nci_conn_info *conn_info, *n;
+ /* This set_bit is not protected with specialized barrier,
+ * However, it is fine because the mutex_lock(&ndev->req_lock);
+ * in nci_close_device() will help to emit one.
+ */
+ set_bit(NCI_UNREG, &ndev->flags);
+
nci_close_device(ndev);
destroy_workqueue(ndev->cmd_wq);
.cmd = NFC_CMD_DEV_UP,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_dev_up,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_DEV_DOWN,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_dev_down,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_START_POLL,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_start_poll,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_STOP_POLL,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_stop_poll,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_DEP_LINK_UP,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_dep_link_up,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_DEP_LINK_DOWN,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_dep_link_down,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_GET_TARGET,
.cmd = NFC_CMD_LLC_SET_PARAMS,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_llc_set_params,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_LLC_SDREQ,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_llc_sdreq,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_FW_DOWNLOAD,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_fw_download,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_ENABLE_SE,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_enable_se,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_DISABLE_SE,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_disable_se,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_GET_SE,
.cmd = NFC_CMD_SE_IO,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_se_io,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_ACTIVATE_TARGET,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_activate_target,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_VENDOR,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_vendor_cmd,
+ .flags = GENL_ADMIN_PERM,
},
{
.cmd = NFC_CMD_DEACTIVATE_TARGET,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.doit = nfc_genl_deactivate_target,
+ .flags = GENL_ADMIN_PERM,
},
};
#include <linux/openvswitch.h>
#include <linux/netlink.h>
#include <linux/rculist.h>
-#include <linux/swap.h>
#include <net/netlink.h>
#include <net/genetlink.h>
#include <linux/if_arp.h>
#include <net/net_namespace.h>
#include <net/netlink.h>
+#include <net/dst.h>
#include <net/pkt_sched.h>
#include <net/pkt_cls.h>
#include <linux/tc_act/tc_mirred.h>
bool want_ingress;
bool is_redirect;
bool expects_nh;
+ bool at_ingress;
int m_eaction;
int mac_len;
bool at_nh;
* ingress - that covers the TC S/W datapath.
*/
is_redirect = tcf_mirred_is_act_redirect(m_eaction);
- use_reinsert = skb_at_tc_ingress(skb) && is_redirect &&
+ at_ingress = skb_at_tc_ingress(skb);
+ use_reinsert = at_ingress && is_redirect &&
tcf_mirred_can_reinsert(retval);
if (!use_reinsert) {
skb2 = skb_clone(skb, GFP_ATOMIC);
goto out;
}
+ want_ingress = tcf_mirred_act_wants_ingress(m_eaction);
+
/* All mirred/redirected skbs should clear previous ct info */
nf_reset_ct(skb2);
-
- want_ingress = tcf_mirred_act_wants_ingress(m_eaction);
+ if (want_ingress && !at_ingress) /* drop dst for egress -> ingress */
+ skb_dst_drop(skb2);
expects_nh = want_ingress || !m_mac_header_xmit;
at_nh = skb->data == skb_network_header(skb);
return ns_to_ktime(sched->base_time);
}
-static ktime_t taprio_get_time(struct taprio_sched *q)
+static ktime_t taprio_mono_to_any(const struct taprio_sched *q, ktime_t mono)
{
- ktime_t mono = ktime_get();
+ /* This pairs with WRITE_ONCE() in taprio_parse_clockid() */
+ enum tk_offsets tk_offset = READ_ONCE(q->tk_offset);
- switch (q->tk_offset) {
+ switch (tk_offset) {
case TK_OFFS_MAX:
return mono;
default:
- return ktime_mono_to_any(mono, q->tk_offset);
+ return ktime_mono_to_any(mono, tk_offset);
}
+}
- return KTIME_MAX;
+static ktime_t taprio_get_time(const struct taprio_sched *q)
+{
+ return taprio_mono_to_any(q, ktime_get());
}
static void taprio_free_sched_cb(struct rcu_head *head)
return 0;
}
- return ktime_mono_to_any(skb->skb_mstamp_ns, q->tk_offset);
+ return taprio_mono_to_any(q, skb->skb_mstamp_ns);
}
/* There are a few scenarios where we will have to modify the txtime from
}
} else if (tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]) {
int clockid = nla_get_s32(tb[TCA_TAPRIO_ATTR_SCHED_CLOCKID]);
+ enum tk_offsets tk_offset;
/* We only support static clockids and we don't allow
* for it to be modified after the first init.
switch (clockid) {
case CLOCK_REALTIME:
- q->tk_offset = TK_OFFS_REAL;
+ tk_offset = TK_OFFS_REAL;
break;
case CLOCK_MONOTONIC:
- q->tk_offset = TK_OFFS_MAX;
+ tk_offset = TK_OFFS_MAX;
break;
case CLOCK_BOOTTIME:
- q->tk_offset = TK_OFFS_BOOT;
+ tk_offset = TK_OFFS_BOOT;
break;
case CLOCK_TAI:
- q->tk_offset = TK_OFFS_TAI;
+ tk_offset = TK_OFFS_TAI;
break;
default:
NL_SET_ERR_MSG(extack, "Invalid 'clockid'");
err = -EINVAL;
goto out;
}
+ /* This pairs with READ_ONCE() in taprio_mono_to_any */
+ WRITE_ONCE(q->tk_offset, tk_offset);
q->clockid = clockid;
} else {
#include <linux/seq_file.h>
#include <linux/memblock.h>
#include <linux/highmem.h>
-#include <linux/swap.h>
#include <linux/slab.h>
#include <net/net_namespace.h>
#include <net/protocol.h>
struct sctp_packet *packet;
int len;
- /* Update socket peer label if first association. */
- if (security_sctp_assoc_request((struct sctp_endpoint *)ep,
- chunk->skb))
- return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
-
/* 6.10 Bundling
* An endpoint MUST NOT bundle INIT, INIT ACK or
* SHUTDOWN COMPLETE with any other chunks.
if (!new_asoc)
goto nomem;
+ /* Update socket peer label if first association. */
+ if (security_sctp_assoc_request(new_asoc, chunk->skb)) {
+ sctp_association_free(new_asoc);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
+ }
+
if (sctp_assoc_set_bind_addr_from_ep(new_asoc,
sctp_scope(sctp_source(chunk)),
GFP_ATOMIC) < 0)
}
}
+ if (security_sctp_assoc_request(new_asoc, chunk->skb)) {
+ sctp_association_free(new_asoc);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
+ }
/* Delay state machine commands until later.
*
struct sctp_packet *packet;
int len;
- /* Update socket peer label if first association. */
- if (security_sctp_assoc_request((struct sctp_endpoint *)ep,
- chunk->skb))
- return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
-
/* 6.10 Bundling
* An endpoint MUST NOT bundle INIT, INIT ACK or
* SHUTDOWN COMPLETE with any other chunks.
if (!new_asoc)
goto nomem;
+ /* Update socket peer label if first association. */
+ if (security_sctp_assoc_request(new_asoc, chunk->skb)) {
+ sctp_association_free(new_asoc);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
+ }
+
if (sctp_assoc_set_bind_addr_from_ep(new_asoc,
sctp_scope(sctp_source(chunk)), GFP_ATOMIC) < 0)
goto nomem;
}
/* Update socket peer label if first association. */
- if (security_sctp_assoc_request((struct sctp_endpoint *)ep,
- chunk->skb)) {
+ if (security_sctp_assoc_request(new_asoc, chunk->skb)) {
sctp_association_free(new_asoc);
return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
}
{
static const char err_str[] = "The following chunk violates protocol:";
- if (!asoc)
- return sctp_sf_violation(net, ep, asoc, type, arg, commands);
-
return sctp_sf_abort_violation(net, ep, asoc, arg, commands, err_str,
sizeof(err_str));
}
struct inet_sock *inet = inet_sk(sk);
struct inet_sock *newinet;
struct sctp_sock *sp = sctp_sk(sk);
- struct sctp_endpoint *ep = sp->ep;
newsk->sk_type = sk->sk_type;
newsk->sk_bound_dev_if = sk->sk_bound_dev_if;
net_enable_timestamp();
/* Set newsk security attributes from original sk and connection
- * security attribute from ep.
+ * security attribute from asoc.
*/
- security_sctp_sk_clone(ep, sk, newsk);
+ security_sctp_sk_clone(asoc, sk, newsk);
}
static inline void sctp_copy_descendant(struct sock *sk_to,
sock_set_flag(sk, SOCK_DEAD);
sk->sk_shutdown |= SHUTDOWN_MASK;
} else {
- if (sk->sk_state != SMC_LISTEN && sk->sk_state != SMC_INIT)
- sock_put(sk); /* passive closing */
- if (sk->sk_state == SMC_LISTEN) {
- /* wake up clcsock accept */
- rc = kernel_sock_shutdown(smc->clcsock, SHUT_RDWR);
+ if (sk->sk_state != SMC_CLOSED) {
+ if (sk->sk_state != SMC_LISTEN &&
+ sk->sk_state != SMC_INIT)
+ sock_put(sk); /* passive closing */
+ if (sk->sk_state == SMC_LISTEN) {
+ /* wake up clcsock accept */
+ rc = kernel_sock_shutdown(smc->clcsock,
+ SHUT_RDWR);
+ }
+ sk->sk_state = SMC_CLOSED;
+ sk->sk_state_change(sk);
}
- sk->sk_state = SMC_CLOSED;
- sk->sk_state_change(sk);
smc_restore_fallback_changes(smc);
}
static void smc_switch_to_fallback(struct smc_sock *smc, int reason_code)
{
+ wait_queue_head_t *smc_wait = sk_sleep(&smc->sk);
+ wait_queue_head_t *clc_wait = sk_sleep(smc->clcsock->sk);
+ unsigned long flags;
+
smc->use_fallback = true;
smc->fallback_rsn = reason_code;
smc_stat_fallback(smc);
smc->clcsock->file->private_data = smc->clcsock;
smc->clcsock->wq.fasync_list =
smc->sk.sk_socket->wq.fasync_list;
+
+ /* There may be some entries remaining in
+ * smc socket->wq, which should be removed
+ * to clcsocket->wq during the fallback.
+ */
+ spin_lock_irqsave(&smc_wait->lock, flags);
+ spin_lock(&clc_wait->lock);
+ list_splice_init(&smc_wait->head, &clc_wait->head);
+ spin_unlock(&clc_wait->lock);
+ spin_unlock_irqrestore(&smc_wait->lock, flags);
}
}
int i;
while (1) {
+again:
link_id = ++lgr->next_link_id;
if (!link_id) /* skip zero as link_id */
link_id = ++lgr->next_link_id;
for (i = 0; i < SMC_LINKS_PER_LGR_MAX; i++) {
if (smc_link_usable(&lgr->lnk[i]) &&
lgr->lnk[i].link_id == link_id)
- continue;
+ goto again;
}
break;
}
__entry->location = location;
),
- TP_printk("lnk=%p lgr=%p state=%d dev=%s location=%p",
+ TP_printk("lnk=%p lgr=%p state=%d dev=%s location=%pS",
__entry->lnk, __entry->lgr,
__entry->state, __get_str(name),
__entry->location)
static struct workqueue_struct *strp_wq;
-struct _strp_msg {
- /* Internal cb structure. struct strp_msg must be first for passing
- * to upper layer.
- */
- struct strp_msg strp;
- int accum_len;
-};
-
static inline struct _strp_msg *_strp_msg(struct sk_buff *skb)
{
return (struct _strp_msg *)((void *)skb->cb +
- offsetof(struct qdisc_skb_cb, data));
+ offsetof(struct sk_skb_cb, strp));
}
/* Lower lock held */
const size_t buflen, const char *delim,
struct sockaddr_in6 *sin6)
{
- char *p;
+ char p[IPV6_SCOPE_ID_LEN + 1];
size_t len;
+ u32 scope_id = 0;
+ struct net_device *dev;
if ((buf + buflen) == delim)
return 1;
return 0;
len = (buf + buflen) - delim - 1;
- p = kmemdup_nul(delim + 1, len, GFP_KERNEL);
- if (p) {
- u32 scope_id = 0;
- struct net_device *dev;
-
- dev = dev_get_by_name(net, p);
- if (dev != NULL) {
- scope_id = dev->ifindex;
- dev_put(dev);
- } else {
- if (kstrtou32(p, 10, &scope_id) != 0) {
- kfree(p);
- return 0;
- }
- }
-
- kfree(p);
-
- sin6->sin6_scope_id = scope_id;
- return 1;
+ if (len > IPV6_SCOPE_ID_LEN)
+ return 0;
+
+ memcpy(p, delim + 1, len);
+ p[len] = 0;
+
+ dev = dev_get_by_name(net, p);
+ if (dev != NULL) {
+ scope_id = dev->ifindex;
+ dev_put(dev);
+ } else {
+ if (kstrtou32(p, 10, &scope_id) != 0)
+ return 0;
}
- return 0;
+ sin6->sin6_scope_id = scope_id;
+ return 1;
}
static size_t rpc_pton6(struct net *net, const char *buf, const size_t buflen,
svc_putnl(rqstp->rq_res.head, RPC_AUTH_GSS);
xdr_seq = kmalloc(4, GFP_KERNEL);
if (!xdr_seq)
- return -1;
+ return -ENOMEM;
*xdr_seq = htonl(seq);
iov.iov_base = xdr_seq;
static
void rpc_task_set_client(struct rpc_task *task, struct rpc_clnt *clnt)
{
-
- if (clnt != NULL) {
- rpc_task_set_transport(task, clnt);
- task->tk_client = clnt;
- refcount_inc(&clnt->cl_count);
- if (clnt->cl_softrtry)
- task->tk_flags |= RPC_TASK_SOFT;
- if (clnt->cl_softerr)
- task->tk_flags |= RPC_TASK_TIMEOUT;
- if (clnt->cl_noretranstimeo)
- task->tk_flags |= RPC_TASK_NO_RETRANS_TIMEOUT;
- if (atomic_read(&clnt->cl_swapper))
- task->tk_flags |= RPC_TASK_SWAPPER;
- /* Add to the client's list of all tasks */
- spin_lock(&clnt->cl_lock);
- list_add_tail(&task->tk_task, &clnt->cl_tasks);
- spin_unlock(&clnt->cl_lock);
- }
+ rpc_task_set_transport(task, clnt);
+ task->tk_client = clnt;
+ refcount_inc(&clnt->cl_count);
+ if (clnt->cl_softrtry)
+ task->tk_flags |= RPC_TASK_SOFT;
+ if (clnt->cl_softerr)
+ task->tk_flags |= RPC_TASK_TIMEOUT;
+ if (clnt->cl_noretranstimeo)
+ task->tk_flags |= RPC_TASK_NO_RETRANS_TIMEOUT;
+ if (atomic_read(&clnt->cl_swapper))
+ task->tk_flags |= RPC_TASK_SWAPPER;
+ /* Add to the client's list of all tasks */
+ spin_lock(&clnt->cl_lock);
+ list_add_tail(&task->tk_task, &clnt->cl_tasks);
+ spin_unlock(&clnt->cl_lock);
}
static void
#if IS_ENABLED(CONFIG_SUNRPC_DEBUG) || IS_ENABLED(CONFIG_TRACEPOINTS)
static void rpc_task_set_debuginfo(struct rpc_task *task)
{
- static atomic_t rpc_pid;
+ struct rpc_clnt *clnt = task->tk_client;
- task->tk_pid = atomic_inc_return(&rpc_pid);
+ /* Might be a task carrying a reverse-direction operation */
+ if (!clnt) {
+ static atomic_t rpc_pid;
+
+ task->tk_pid = atomic_inc_return(&rpc_pid);
+ return;
+ }
+
+ task->tk_pid = atomic_inc_return(&clnt->cl_pid);
}
#else
static inline void rpc_task_set_debuginfo(struct rpc_task *task)
else if (task->tk_client)
rpc_count_iostats(task, task->tk_client->cl_metrics);
if (task->tk_ops->rpc_call_done != NULL) {
+ trace_rpc_task_call_done(task, task->tk_ops->rpc_call_done);
task->tk_ops->rpc_call_done(task, task->tk_calldata);
if (task->tk_action != NULL) {
/* Always release the RPC slot and buffer memory */
/*
* Lockless check for whether task is sleeping or not.
*/
- if (!RPC_IS_QUEUED(task))
+ if (!RPC_IS_QUEUED(task)) {
+ cond_resched();
continue;
+ }
/*
* Signalled tasks should exit rather than sleep.
if (!wq)
goto out_failed;
rpciod_workqueue = wq;
- /* Note: highpri because network receive is latency sensitive */
- wq = alloc_workqueue("xprtiod", WQ_UNBOUND|WQ_MEM_RECLAIM|WQ_HIGHPRI, 0);
+ wq = alloc_workqueue("xprtiod", WQ_UNBOUND | WQ_MEM_RECLAIM, 0);
if (!wq)
goto free_rpciod;
xprtiod_workqueue = wq;
static __printf(2,3) void svc_printk(struct svc_rqst *rqstp, const char *fmt, ...) {}
#endif
-static int
-svc_generic_dispatch(struct svc_rqst *rqstp, __be32 *statp)
-{
- struct kvec *argv = &rqstp->rq_arg.head[0];
- struct kvec *resv = &rqstp->rq_res.head[0];
- const struct svc_procedure *procp = rqstp->rq_procinfo;
-
- /*
- * Decode arguments
- * XXX: why do we ignore the return value?
- */
- if (procp->pc_decode &&
- !procp->pc_decode(rqstp, argv->iov_base)) {
- *statp = rpc_garbage_args;
- return 1;
- }
-
- *statp = procp->pc_func(rqstp);
-
- if (*statp == rpc_drop_reply ||
- test_bit(RQ_DROPME, &rqstp->rq_flags))
- return 0;
-
- if (rqstp->rq_auth_stat != rpc_auth_ok)
- return 1;
-
- if (*statp != rpc_success)
- return 1;
-
- /* Encode reply */
- if (procp->pc_encode &&
- !procp->pc_encode(rqstp, resv->iov_base + resv->iov_len)) {
- dprintk("svc: failed to encode reply\n");
- /* serv->sv_stats->rpcsystemerr++; */
- *statp = rpc_system_err;
- }
- return 1;
-}
-
__be32
svc_generic_init_request(struct svc_rqst *rqstp,
const struct svc_program *progp,
__be32 *statp;
u32 prog, vers;
__be32 rpc_stat;
- int auth_res;
+ int auth_res, rc;
__be32 *reply_statp;
rpc_stat = rpc_success;
svc_reserve_auth(rqstp, procp->pc_xdrressize<<2);
/* Call the function that processes the request. */
- if (!process.dispatch) {
- if (!svc_generic_dispatch(rqstp, statp))
- goto release_dropit;
- if (*statp == rpc_garbage_args)
- goto err_garbage;
- } else {
- dprintk("svc: calling dispatcher\n");
- if (!process.dispatch(rqstp, statp))
- goto release_dropit; /* Release reply info */
- }
-
+ rc = process.dispatch(rqstp, statp);
+ if (procp->pc_release)
+ procp->pc_release(rqstp);
+ if (!rc)
+ goto dropit;
if (rqstp->rq_auth_stat != rpc_auth_ok)
- goto err_release_bad_auth;
+ goto err_bad_auth;
/* Check RPC status result */
if (*statp != rpc_success)
resv->iov_len = ((void*)statp) - resv->iov_base + 4;
- /* Release reply info */
- if (procp->pc_release)
- procp->pc_release(rqstp);
-
if (procp->pc_encode == NULL)
goto dropit;
goto close_xprt;
return 1; /* Caller can now send it */
-release_dropit:
- if (procp->pc_release)
- procp->pc_release(rqstp);
dropit:
svc_authorise(rqstp); /* doesn't hurt to call this twice */
dprintk("svc: svc_process dropit\n");
svc_putnl(resv, 2);
goto sendit;
-err_release_bad_auth:
- if (procp->pc_release)
- procp->pc_release(rqstp);
err_bad_auth:
dprintk("svc: authentication failed (%d)\n",
be32_to_cpu(rqstp->rq_auth_stat));
/**
* svc_fill_write_vector - Construct data argument for VFS write call
* @rqstp: svc_rqst to operate on
- * @pages: list of pages containing data payload
- * @first: buffer containing first section of write payload
- * @total: total number of bytes of write payload
+ * @payload: xdr_buf containing only the write data payload
*
* Fills in rqstp::rq_vec, and returns the number of elements.
*/
-unsigned int svc_fill_write_vector(struct svc_rqst *rqstp, struct page **pages,
- struct kvec *first, size_t total)
+unsigned int svc_fill_write_vector(struct svc_rqst *rqstp,
+ struct xdr_buf *payload)
{
+ struct page **pages = payload->pages;
+ struct kvec *first = payload->head;
struct kvec *vec = rqstp->rq_vec;
+ size_t total = payload->len;
unsigned int i;
/* Some types of transport can present the write payload
set_current_state(TASK_RUNNING);
return -EINTR;
}
+ trace_svc_alloc_arg_err(pages);
schedule_timeout(msecs_to_jiffies(500));
}
rqstp->rq_page_end = &rqstp->rq_pages[pages];
struct sock_xprt *sock;
ssize_t ret = -1;
- if (!xprt)
- return 0;
+ if (!xprt || !xprt_connected(xprt)) {
+ xprt_put(xprt);
+ return -ENOTCONN;
+ }
sock = container_of(xprt, struct sock_xprt, xprt);
if (kernel_getsockname(sock->sock, (struct sockaddr *)&saddr) < 0)
struct rpc_xprt *xprt = rpc_sysfs_xprt_kobj_get_xprt(kobj);
ssize_t ret;
- if (!xprt)
- return 0;
+ if (!xprt || !xprt_connected(xprt)) {
+ xprt_put(xprt);
+ return -ENOTCONN;
+ }
ret = sprintf(buf, "last_used=%lu\ncur_cong=%lu\ncong_win=%lu\n"
"max_num_slots=%u\nmin_num_slots=%u\nnum_reqs=%u\n"
* Sets up @subbuf to represent a portion of @xdr. The portion
* starts at the current offset in @xdr, and extends for a length
* of @nbytes. If this is successful, @xdr is advanced to the next
- * position following that portion.
+ * XDR data item following that portion.
*
* Return values:
* %true: @subbuf has been initialized, and @xdr has been advanced.
bool xdr_stream_subsegment(struct xdr_stream *xdr, struct xdr_buf *subbuf,
unsigned int nbytes)
{
- unsigned int remaining, offset, len;
+ unsigned int start = xdr_stream_pos(xdr);
+ unsigned int remaining, len;
- if (xdr_buf_subsegment(xdr->buf, subbuf, xdr_stream_pos(xdr), nbytes))
+ /* Extract @subbuf and bounds-check the fn arguments */
+ if (xdr_buf_subsegment(xdr->buf, subbuf, start, nbytes))
return false;
- if (subbuf->head[0].iov_len)
- if (!__xdr_inline_decode(xdr, subbuf->head[0].iov_len))
- return false;
-
- remaining = subbuf->page_len;
- offset = subbuf->page_base;
- while (remaining) {
- len = min_t(unsigned int, remaining, PAGE_SIZE) - offset;
-
+ /* Advance @xdr by @nbytes */
+ for (remaining = nbytes; remaining;) {
if (xdr->p == xdr->end && !xdr_set_next_buffer(xdr))
return false;
- if (!__xdr_inline_decode(xdr, len))
- return false;
+ len = (char *)xdr->end - (char *)xdr->p;
+ if (remaining <= len) {
+ xdr->p = (__be32 *)((char *)xdr->p +
+ (remaining + xdr_pad_size(nbytes)));
+ break;
+ }
+
+ xdr->p = (__be32 *)((char *)xdr->p + len);
+ xdr->end = xdr->p;
remaining -= len;
- offset = 0;
}
+ xdr_stream_set_pos(xdr, start + nbytes);
return true;
}
EXPORT_SYMBOL_GPL(xdr_stream_subsegment);
static void xprt_clear_locked(struct rpc_xprt *xprt)
{
xprt->snd_task = NULL;
- if (!test_bit(XPRT_CLOSE_WAIT, &xprt->state)) {
- smp_mb__before_atomic();
- clear_bit(XPRT_LOCKED, &xprt->state);
- smp_mb__after_atomic();
- } else
+ if (!test_bit(XPRT_CLOSE_WAIT, &xprt->state))
+ clear_bit_unlock(XPRT_LOCKED, &xprt->state);
+ else
queue_work(xprtiod_workqueue, &xprt->task_cleanup);
}
unsigned int pflags = memalloc_nofs_save();
trace_xprt_disconnect_auto(xprt);
+ xprt->connect_cookie++;
+ smp_mb__before_atomic();
clear_bit(XPRT_CLOSE_WAIT, &xprt->state);
xprt->ops->close(xprt);
xprt_release_write(xprt, NULL);
*/
static void xprt_schedule_autoclose_locked(struct rpc_xprt *xprt)
{
- set_bit(XPRT_CLOSE_WAIT, &xprt->state);
+ if (test_and_set_bit(XPRT_CLOSE_WAIT, &xprt->state))
+ return;
if (test_and_set_bit(XPRT_LOCKED, &xprt->state) == 0)
queue_work(xprtiod_workqueue, &xprt->task_cleanup);
else if (xprt->snd_task && !test_bit(XPRT_SND_IS_COOKIE, &xprt->state))
{
struct rpc_rqst *next, *req = task->tk_rqstp;
struct rpc_xprt *xprt = req->rq_xprt;
- int counter, status;
+ int status;
spin_lock(&xprt->queue_lock);
- counter = 0;
- while (!list_empty(&xprt->xmit_queue)) {
- if (++counter == 20)
+ for (;;) {
+ next = list_first_entry_or_null(&xprt->xmit_queue,
+ struct rpc_rqst, rq_xmit);
+ if (!next)
break;
- next = list_first_entry(&xprt->xmit_queue,
- struct rpc_rqst, rq_xmit);
xprt_pin_rqst(next);
spin_unlock(&xprt->queue_lock);
status = xprt_request_transmit(next, task);
status = 0;
spin_lock(&xprt->queue_lock);
xprt_unpin_rqst(next);
- if (status == 0) {
- if (!xprt_request_data_received(task) ||
- test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
- continue;
- } else if (test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
- task->tk_status = status;
- break;
+ if (status < 0) {
+ if (test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+ task->tk_status = status;
+ break;
+ }
+ /* Was @task transmitted, and has it received a reply? */
+ if (xprt_request_data_received(task) &&
+ !test_bit(RPC_TASK_NEED_XMIT, &task->tk_runstate))
+ break;
+ cond_resched_lock(&xprt->queue_lock);
}
spin_unlock(&xprt->queue_lock);
}
* a single ib_post_send() call.
*/
prev = &first;
- while ((mr = rpcrdma_mr_pop(&req->rl_registered))) {
-
+ mr = rpcrdma_mr_pop(&req->rl_registered);
+ do {
trace_xprtrdma_mr_localinv(mr);
r_xprt->rx_stats.local_inv_needed++;
*prev = last;
prev = &last->next;
- }
+ } while ((mr = rpcrdma_mr_pop(&req->rl_registered)));
+
mr = container_of(last, struct rpcrdma_mr, mr_invwr);
/* Strong send queue ordering guarantees that when the
* a single ib_post_send() call.
*/
prev = &first;
- while ((mr = rpcrdma_mr_pop(&req->rl_registered))) {
-
+ mr = rpcrdma_mr_pop(&req->rl_registered);
+ do {
trace_xprtrdma_mr_localinv(mr);
r_xprt->rx_stats.local_inv_needed++;
*prev = last;
prev = &last->next;
- }
+ } while ((mr = rpcrdma_mr_pop(&req->rl_registered)));
/* Strong send queue ordering guarantees that when the
* last WR in the chain completes, all WRs in the chain
*/
rpcrdma_force_disconnect(ep);
}
+
+/**
+ * frwr_wp_create - Create an MR for padding Write chunks
+ * @r_xprt: transport resources to use
+ *
+ * Return 0 on success, negative errno on failure.
+ */
+int frwr_wp_create(struct rpcrdma_xprt *r_xprt)
+{
+ struct rpcrdma_ep *ep = r_xprt->rx_ep;
+ struct rpcrdma_mr_seg seg;
+ struct rpcrdma_mr *mr;
+
+ mr = rpcrdma_mr_get(r_xprt);
+ if (!mr)
+ return -EAGAIN;
+ mr->mr_req = NULL;
+ ep->re_write_pad_mr = mr;
+
+ seg.mr_len = XDR_UNIT;
+ seg.mr_page = virt_to_page(ep->re_write_pad);
+ seg.mr_offset = offset_in_page(ep->re_write_pad);
+ if (IS_ERR(frwr_map(r_xprt, &seg, 1, true, xdr_zero, mr)))
+ return -EIO;
+ trace_xprtrdma_mr_fastreg(mr);
+
+ mr->mr_cqe.done = frwr_wc_fastreg;
+ mr->mr_regwr.wr.next = NULL;
+ mr->mr_regwr.wr.wr_cqe = &mr->mr_cqe;
+ mr->mr_regwr.wr.num_sge = 0;
+ mr->mr_regwr.wr.opcode = IB_WR_REG_MR;
+ mr->mr_regwr.wr.send_flags = 0;
+
+ return ib_post_send(ep->re_id->qp, &mr->mr_regwr.wr, NULL);
+}
page_base = 0;
}
- if (type == rpcrdma_readch)
- goto out;
-
- /* When encoding a Write chunk, some servers need to see an
- * extra segment for non-XDR-aligned Write chunks. The upper
- * layer provides space in the tail iovec that may be used
- * for this purpose.
- */
- if (type == rpcrdma_writech && r_xprt->rx_ep->re_implicit_roundup)
+ if (type == rpcrdma_readch || type == rpcrdma_writech)
goto out;
if (xdrbuf->tail[0].iov_len)
enum rpcrdma_chunktype wtype)
{
struct xdr_stream *xdr = &req->rl_stream;
+ struct rpcrdma_ep *ep = r_xprt->rx_ep;
struct rpcrdma_mr_seg *seg;
struct rpcrdma_mr *mr;
int nsegs, nchunks;
nsegs -= mr->mr_nents;
} while (nsegs);
+ if (xdr_pad_size(rqst->rq_rcv_buf.page_len)) {
+ if (encode_rdma_segment(xdr, ep->re_write_pad_mr) < 0)
+ return -EMSGSIZE;
+
+ trace_xprtrdma_chunk_wp(rqst->rq_task, ep->re_write_pad_mr,
+ nsegs);
+ r_xprt->rx_stats.write_chunk_count++;
+ r_xprt->rx_stats.total_rdma_request += mr->mr_length;
+ nchunks++;
+ nsegs -= mr->mr_nents;
+ }
+
/* Update count of segments in this Write chunk */
*segcount = cpu_to_be32(nchunks);
/* WARNING: Only wc->wr_cqe and wc->status are reliable */
ctxt = container_of(cqe, struct svc_rdma_recv_ctxt, rc_cqe);
- trace_svcrdma_wc_receive(wc, &ctxt->rc_cid);
if (wc->status != IB_WC_SUCCESS)
goto flushed;
+ trace_svcrdma_wc_recv(wc, &ctxt->rc_cid);
/* If receive posting fails, the connection is about to be
* lost anyway. The server will not be able to send a reply
*/
if (rdma->sc_pending_recvs < rdma->sc_max_requests)
if (!svc_rdma_refresh_recvs(rdma, rdma->sc_recv_batch, false))
- goto flushed;
+ goto dropped;
/* All wc fields are now known to be valid */
ctxt->rc_byte_len = wc->byte_len;
return;
flushed:
+ if (wc->status == IB_WC_WR_FLUSH_ERR)
+ trace_svcrdma_wc_recv_flush(wc, &ctxt->rc_cid);
+ else
+ trace_svcrdma_wc_recv_err(wc, &ctxt->rc_cid);
+dropped:
svc_rdma_recv_ctxt_put(rdma, ctxt);
svc_xprt_deferred_close(&rdma->sc_xprt);
}
struct ib_cqe cc_cqe;
struct svcxprt_rdma *cc_rdma;
struct list_head cc_rwctxts;
+ ktime_t cc_posttime;
int cc_sqecount;
enum ib_wc_status cc_status;
struct completion cc_done;
struct svc_rdma_write_info *info =
container_of(cc, struct svc_rdma_write_info, wi_cc);
- trace_svcrdma_wc_write(wc, &cc->cc_cid);
+ switch (wc->status) {
+ case IB_WC_SUCCESS:
+ trace_svcrdma_wc_write(wc, &cc->cc_cid);
+ break;
+ case IB_WC_WR_FLUSH_ERR:
+ trace_svcrdma_wc_write_flush(wc, &cc->cc_cid);
+ break;
+ default:
+ trace_svcrdma_wc_write_err(wc, &cc->cc_cid);
+ }
svc_rdma_wake_send_waiters(rdma, cc->cc_sqecount);
struct ib_cqe *cqe = wc->wr_cqe;
struct svc_rdma_chunk_ctxt *cc =
container_of(cqe, struct svc_rdma_chunk_ctxt, cc_cqe);
- struct svcxprt_rdma *rdma = cc->cc_rdma;
+ struct svc_rdma_read_info *info;
- trace_svcrdma_wc_read(wc, &cc->cc_cid);
+ switch (wc->status) {
+ case IB_WC_SUCCESS:
+ info = container_of(cc, struct svc_rdma_read_info, ri_cc);
+ trace_svcrdma_wc_read(wc, &cc->cc_cid, info->ri_totalbytes,
+ cc->cc_posttime);
+ break;
+ case IB_WC_WR_FLUSH_ERR:
+ trace_svcrdma_wc_read_flush(wc, &cc->cc_cid);
+ break;
+ default:
+ trace_svcrdma_wc_read_err(wc, &cc->cc_cid);
+ }
- svc_rdma_wake_send_waiters(rdma, cc->cc_sqecount);
+ svc_rdma_wake_send_waiters(cc->cc_rdma, cc->cc_sqecount);
cc->cc_status = wc->status;
complete(&cc->cc_done);
return;
do {
if (atomic_sub_return(cc->cc_sqecount,
&rdma->sc_sq_avail) > 0) {
+ cc->cc_posttime = ktime_get();
ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
if (ret)
break;
struct svc_rdma_send_ctxt *ctxt =
container_of(cqe, struct svc_rdma_send_ctxt, sc_cqe);
- trace_svcrdma_wc_send(wc, &ctxt->sc_cid);
-
svc_rdma_wake_send_waiters(rdma, 1);
complete(&ctxt->sc_done);
if (unlikely(wc->status != IB_WC_SUCCESS))
- svc_xprt_deferred_close(&rdma->sc_xprt);
+ goto flushed;
+
+ trace_svcrdma_wc_send(wc, &ctxt->sc_cid);
+ return;
+
+flushed:
+ if (wc->status != IB_WC_WR_FLUSH_ERR)
+ trace_svcrdma_wc_send_err(wc, &ctxt->sc_cid);
+ else
+ trace_svcrdma_wc_send_flush(wc, &ctxt->sc_cid);
+ svc_xprt_deferred_close(&rdma->sc_xprt);
}
/**
unsigned int rsize, wsize;
/* Default settings for RPC-over-RDMA Version One */
- ep->re_implicit_roundup = xprt_rdma_pad_optimize;
rsize = RPCRDMA_V1_DEF_INLINE_SIZE;
wsize = RPCRDMA_V1_DEF_INLINE_SIZE;
if (pmsg &&
pmsg->cp_magic == rpcrdma_cmp_magic &&
pmsg->cp_version == RPCRDMA_CMP_VERSION) {
- ep->re_implicit_roundup = true;
rsize = rpcrdma_decode_buffer_size(pmsg->cp_send_size);
wsize = rpcrdma_decode_buffer_size(pmsg->cp_recv_size);
}
goto out;
}
rpcrdma_mrs_create(r_xprt);
+ frwr_wp_create(r_xprt);
out:
trace_xprtrdma_connect(r_xprt, rc);
/*
* RDMA Endpoint -- connection endpoint details
*/
+struct rpcrdma_mr;
struct rpcrdma_ep {
struct kref re_kref;
struct rdma_cm_id *re_id;
struct ib_pd *re_pd;
unsigned int re_max_rdma_segs;
unsigned int re_max_fr_depth;
- bool re_implicit_roundup;
+ struct rpcrdma_mr *re_write_pad_mr;
enum ib_mr_type re_mrtype;
struct completion re_done;
unsigned int re_send_count;
unsigned int re_inline_recv; /* negotiated */
atomic_t re_completion_ids;
+
+ char re_write_pad[XDR_UNIT];
};
/* Pre-allocate extra Work Requests for handling reverse-direction
void frwr_reminv(struct rpcrdma_rep *rep, struct list_head *mrs);
void frwr_unmap_sync(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
void frwr_unmap_async(struct rpcrdma_xprt *r_xprt, struct rpcrdma_req *req);
+int frwr_wp_create(struct rpcrdma_xprt *r_xprt);
/*
* RPC/RDMA protocol calls - xprtrdma/rpc_rdma.c
static void xs_sock_reset_connection_flags(struct rpc_xprt *xprt)
{
+ xprt->connect_cookie++;
smp_mb__before_atomic();
clear_bit(XPRT_CLOSE_WAIT, &xprt->state);
clear_bit(XPRT_CLOSING, &xprt->state);
struct sock_xprt *transport;
struct rpc_xprt *xprt;
- read_lock_bh(&sk->sk_callback_lock);
if (!(xprt = xprt_from_sock(sk)))
- goto out;
+ return;
transport = container_of(xprt, struct sock_xprt, xprt);
transport->xprt_err = -sk->sk_err;
if (transport->xprt_err == 0)
- goto out;
+ return;
dprintk("RPC: xs_error_report client %p, error=%d...\n",
xprt, -transport->xprt_err);
trace_rpc_socket_error(xprt, sk->sk_socket, transport->xprt_err);
/* barrier ensures xprt_err is set before XPRT_SOCK_WAKE_ERROR */
smp_mb__before_atomic();
xs_run_error_worker(transport, XPRT_SOCK_WAKE_ERROR);
- out:
- read_unlock_bh(&sk->sk_callback_lock);
}
static void xs_reset_transport(struct sock_xprt *transport)
kernel_sock_shutdown(sock, SHUT_RDWR);
mutex_lock(&transport->recv_mutex);
- write_lock_bh(&sk->sk_callback_lock);
+ lock_sock(sk);
transport->inet = NULL;
transport->sock = NULL;
transport->file = NULL;
xs_restore_old_callbacks(transport, sk);
xprt_clear_connected(xprt);
- write_unlock_bh(&sk->sk_callback_lock);
xs_sock_reset_connection_flags(xprt);
/* Reset stream record info */
xs_stream_reset_connect(transport);
+ release_sock(sk);
mutex_unlock(&transport->recv_mutex);
trace_rpc_socket_close(xprt, sock);
{
struct rpc_xprt *xprt;
- read_lock_bh(&sk->sk_callback_lock);
dprintk("RPC: xs_data_ready...\n");
xprt = xprt_from_sock(sk);
if (xprt != NULL) {
if (!test_and_set_bit(XPRT_SOCK_DATA_READY, &transport->sock_state))
queue_work(xprtiod_workqueue, &transport->recv_worker);
}
- read_unlock_bh(&sk->sk_callback_lock);
}
/*
struct rpc_xprt *xprt;
struct sock_xprt *transport;
- read_lock_bh(&sk->sk_callback_lock);
if (!(xprt = xprt_from_sock(sk)))
- goto out;
+ return;
dprintk("RPC: xs_tcp_state_change client %p...\n", xprt);
dprintk("RPC: state %x conn %d dead %d zapped %d sk_shutdown %d\n",
sk->sk_state, xprt_connected(xprt),
/* Trigger the socket release */
xs_run_error_worker(transport, XPRT_SOCK_WAKE_DISCONNECT);
}
- out:
- read_unlock_bh(&sk->sk_callback_lock);
}
static void xs_write_space(struct sock *sk)
*/
static void xs_udp_write_space(struct sock *sk)
{
- read_lock_bh(&sk->sk_callback_lock);
-
/* from net/core/sock.c:sock_def_write_space */
if (sock_writeable(sk))
xs_write_space(sk);
-
- read_unlock_bh(&sk->sk_callback_lock);
}
/**
*/
static void xs_tcp_write_space(struct sock *sk)
{
- read_lock_bh(&sk->sk_callback_lock);
-
/* from net/core/stream.c:sk_stream_write_space */
if (sk_stream_is_writeable(sk))
xs_write_space(sk);
-
- read_unlock_bh(&sk->sk_callback_lock);
}
static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt)
if (!transport->inet) {
struct sock *sk = sock->sk;
- write_lock_bh(&sk->sk_callback_lock);
+ lock_sock(sk);
xs_save_old_callbacks(transport, sk);
transport->sock = sock;
transport->inet = sk;
- write_unlock_bh(&sk->sk_callback_lock);
+ release_sock(sk);
}
xs_stream_start_connect(transport);
if (!transport->inet) {
struct sock *sk = sock->sk;
- write_lock_bh(&sk->sk_callback_lock);
+ lock_sock(sk);
xs_save_old_callbacks(transport, sk);
xs_set_memalloc(xprt);
- write_unlock_bh(&sk->sk_callback_lock);
+ release_sock(sk);
}
xs_udp_do_set_buffer_size(xprt);
static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
{
struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
- int ret = -ENOTCONN;
if (!transport->inet) {
struct sock *sk = sock->sk;
xs_tcp_set_socket_timeouts(xprt, sock);
tcp_sock_set_nodelay(sk);
- write_lock_bh(&sk->sk_callback_lock);
+ lock_sock(sk);
xs_save_old_callbacks(transport, sk);
transport->sock = sock;
transport->inet = sk;
- write_unlock_bh(&sk->sk_callback_lock);
+ release_sock(sk);
}
if (!xprt_bound(xprt))
- goto out;
+ return -ENOTCONN;
xs_set_memalloc(xprt);
/* Tell the socket layer to start connecting... */
set_bit(XPRT_SOCK_CONNECTING, &transport->sock_state);
- ret = kernel_connect(sock, xs_addr(xprt), xprt->addrlen, O_NONBLOCK);
- switch (ret) {
- case 0:
- xs_set_srcport(transport, sock);
- fallthrough;
- case -EINPROGRESS:
- /* SYN_SENT! */
- if (xprt->reestablish_timeout < XS_TCP_INIT_REEST_TO)
- xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
- break;
- case -EADDRNOTAVAIL:
- /* Source port number is unavailable. Try a new one! */
- transport->srcport = 0;
- }
-out:
- return ret;
+ return kernel_connect(sock, xs_addr(xprt), xprt->addrlen, O_NONBLOCK);
}
/**
container_of(work, struct sock_xprt, connect_worker.work);
struct socket *sock = transport->sock;
struct rpc_xprt *xprt = &transport->xprt;
- int status = -EIO;
+ int status;
if (!sock) {
sock = xs_create_sock(xprt, transport,
xs_addr(xprt)->sa_family, SOCK_STREAM,
IPPROTO_TCP, true);
if (IS_ERR(sock)) {
- status = PTR_ERR(sock);
+ xprt_wake_pending_tasks(xprt, PTR_ERR(sock));
goto out;
}
}
xprt, -status, xprt_connected(xprt),
sock->sk->sk_state);
switch (status) {
- default:
- printk("%s: connect returned unhandled error %d\n",
- __func__, status);
- fallthrough;
- case -EADDRNOTAVAIL:
- /* We're probably in TIME_WAIT. Get rid of existing socket,
- * and retry
- */
- xs_tcp_force_close(xprt);
- break;
case 0:
+ xs_set_srcport(transport, sock);
+ fallthrough;
case -EINPROGRESS:
+ /* SYN_SENT! */
+ if (xprt->reestablish_timeout < XS_TCP_INIT_REEST_TO)
+ xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+ fallthrough;
case -EALREADY:
- xprt_unlock_connect(xprt, transport);
- return;
+ goto out_unlock;
+ case -EADDRNOTAVAIL:
+ /* Source port number is unavailable. Try a new one! */
+ transport->srcport = 0;
+ status = -EAGAIN;
+ break;
case -EINVAL:
/* Happens, for instance, if the user specified a link
* local IPv6 address without a scope-id.
case -EHOSTUNREACH:
case -EADDRINUSE:
case -ENOBUFS:
- /* xs_tcp_force_close() wakes tasks with a fixed error code.
- * We need to wake them first to ensure the correct error code.
- */
- xprt_wake_pending_tasks(xprt, status);
- xs_tcp_force_close(xprt);
- goto out;
+ break;
+ default:
+ printk("%s: connect returned unhandled error %d\n",
+ __func__, status);
+ status = -EAGAIN;
}
- status = -EAGAIN;
+
+ /* xs_tcp_force_close() wakes tasks with a fixed error code.
+ * We need to wake them first to ensure the correct error code.
+ */
+ xprt_wake_pending_tasks(xprt, status);
+ xs_tcp_force_close(xprt);
out:
xprt_clear_connecting(xprt);
+out_unlock:
xprt_unlock_connect(xprt, transport);
- xprt_wake_pending_tasks(xprt, status);
}
/**
WARN_ON_ONCE(!xprt_lock_connect(xprt, task, transport));
- if (transport->sock != NULL) {
+ if (transport->sock != NULL && !xprt_connecting(xprt)) {
dprintk("RPC: xs_connect delayed xprt %p for %lu "
"seconds\n",
xprt, xprt->reestablish_timeout / HZ);
addr = (unsigned long)ent->data;
if (is_module_address(addr))
where = "module";
- else if (core_kernel_data(addr))
+ else if (is_kernel_core_data(addr))
where = "kernel";
else
continue;
return -EEXIST;
/* Allocate a new AEAD */
- tmp = kzalloc(sizeof(*tmp), GFP_ATOMIC);
+ tmp = kzalloc(sizeof(*tmp), GFP_KERNEL);
if (unlikely(!tmp))
return -ENOMEM;
tmp->cloned = NULL;
tmp->authsize = TIPC_AES_GCM_TAG_SIZE;
tmp->key = kmemdup(ukey, tipc_aead_key_size(ukey), GFP_KERNEL);
+ if (!tmp->key) {
+ tipc_aead_free(&tmp->rcu);
+ return -ENOMEM;
+ }
memcpy(&tmp->salt, ukey->key + keylen, TIPC_AES_GCM_SALT_SIZE);
atomic_set(&tmp->users, 0);
atomic64_set(&tmp->seqno, 0);
return -EEXIST;
/* Allocate crypto */
- c = kzalloc(sizeof(*c), GFP_ATOMIC);
+ c = kzalloc(sizeof(*c), GFP_KERNEL);
if (!c)
return -ENOMEM;
}
/* Allocate statistic structure */
- c->stats = alloc_percpu_gfp(struct tipc_crypto_stats, GFP_ATOMIC);
+ c->stats = alloc_percpu(struct tipc_crypto_stats);
if (!c->stats) {
if (c->wq)
destroy_workqueue(c->wq);
}
/* Lets duplicate it first */
- skey = kmemdup(aead->key, tipc_aead_key_size(aead->key), GFP_ATOMIC);
+ skey = kmemdup(aead->key, tipc_aead_key_size(aead->key), GFP_KERNEL);
rcu_read_unlock();
/* Now, generate new key, initiate & distribute it */
return false;
#ifdef CONFIG_TIPC_CRYPTO
case MSG_CRYPTO:
- tipc_crypto_msg_rcv(l->net, skb);
- return true;
+ if (TIPC_SKB_CB(skb)->decrypted) {
+ tipc_crypto_msg_rcv(l->net, skb);
+ return true;
+ }
+ fallthrough;
#endif
default:
pr_warn("Dropping received illegal msg type\n");
* non-blocking call.
*/
err = -EALREADY;
+ if (flags & O_NONBLOCK)
+ goto out;
break;
default:
if ((sk->sk_state == TCP_LISTEN) ||
[NL80211_PKTPAT_OFFSET] = { .type = NLA_U32 },
};
-int nl80211_prepare_wdev_dump(struct netlink_callback *cb,
- struct cfg80211_registered_device **rdev,
- struct wireless_dev **wdev)
+static int nl80211_prepare_wdev_dump(struct netlink_callback *cb,
+ struct cfg80211_registered_device **rdev,
+ struct wireless_dev **wdev,
+ struct nlattr **attrbuf)
{
int err;
if (!cb->args[0]) {
- struct nlattr **attrbuf;
+ struct nlattr **attrbuf_free = NULL;
- attrbuf = kcalloc(NUM_NL80211_ATTR, sizeof(*attrbuf),
- GFP_KERNEL);
- if (!attrbuf)
- return -ENOMEM;
+ if (!attrbuf) {
+ attrbuf = kcalloc(NUM_NL80211_ATTR, sizeof(*attrbuf),
+ GFP_KERNEL);
+ if (!attrbuf)
+ return -ENOMEM;
+ attrbuf_free = attrbuf;
+ }
err = nlmsg_parse_deprecated(cb->nlh,
GENL_HDRLEN + nl80211_fam.hdrsize,
attrbuf, nl80211_fam.maxattr,
nl80211_policy, NULL);
if (err) {
- kfree(attrbuf);
+ kfree(attrbuf_free);
return err;
}
rtnl_lock();
*wdev = __cfg80211_wdev_from_attrs(NULL, sock_net(cb->skb->sk),
attrbuf);
- kfree(attrbuf);
+ kfree(attrbuf_free);
if (IS_ERR(*wdev)) {
rtnl_unlock();
return PTR_ERR(*wdev);
int sta_idx = cb->args[2];
int err;
- err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev);
+ err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev, NULL);
if (err)
return err;
/* nl80211_prepare_wdev_dump acquired it in the successful case */
int path_idx = cb->args[2];
int err;
- err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev);
+ err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev, NULL);
if (err)
return err;
/* nl80211_prepare_wdev_dump acquired it in the successful case */
int path_idx = cb->args[2];
int err;
- err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev);
+ err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev, NULL);
if (err)
return err;
/* nl80211_prepare_wdev_dump acquired it in the successful case */
int start = cb->args[2], idx = 0;
int err;
- err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev);
+ err = nl80211_prepare_wdev_dump(cb, &rdev, &wdev, NULL);
if (err)
return err;
/* nl80211_prepare_wdev_dump acquired it in the successful case */
if (!attrbuf)
return -ENOMEM;
- res = nl80211_prepare_wdev_dump(cb, &rdev, &wdev);
+ res = nl80211_prepare_wdev_dump(cb, &rdev, &wdev, attrbuf);
if (res) {
kfree(attrbuf);
return res;
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Portions of this file
- * Copyright (C) 2018, 2020 Intel Corporation
+ * Copyright (C) 2018, 2020-2021 Intel Corporation
*/
#ifndef __NET_WIRELESS_NL80211_H
#define __NET_WIRELESS_NL80211_H
((u64)wiphy_to_rdev(wdev->wiphy)->wiphy_idx << 32);
}
-int nl80211_prepare_wdev_dump(struct netlink_callback *cb,
- struct cfg80211_registered_device **rdev,
- struct wireless_dev **wdev);
-
int nl80211_parse_chandef(struct cfg80211_registered_device *rdev,
struct genl_info *info,
struct cfg80211_chan_def *chandef);
switch (otype) {
case NL80211_IFTYPE_AP:
+ case NL80211_IFTYPE_P2P_GO:
cfg80211_stop_ap(rdev, dev, true);
break;
case NL80211_IFTYPE_ADHOC:
pool->free_list_cnt--;
xskb = list_first_entry(&pool->free_list, struct xdp_buff_xsk,
free_list_node);
- list_del(&xskb->free_list_node);
+ list_del_init(&xskb->free_list_node);
}
xskb->xdp.data = xskb->xdp.data_hard_start + XDP_PACKET_HEADROOM;
i = nb_entries;
while (i--) {
xskb = list_first_entry(&pool->free_list, struct xdp_buff_xsk, free_list_node);
- list_del(&xskb->free_list_node);
+ list_del_init(&xskb->free_list_node);
*xdp = &xskb->xdp;
xdp++;
void xp_free(struct xdp_buff_xsk *xskb)
{
+ if (!list_empty(&xskb->free_list_node))
+ return;
+
xskb->pool->free_list_cnt++;
list_add(&xskb->free_list_node, &xskb->pool->free_list);
}
config SAMPLE_FTRACE_DIRECT
tristate "Build register_ftrace_direct() example"
depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS && m
- depends on X86_64 # has x86_64 inlined asm
+ depends on HAVE_SAMPLE_FTRACE_DIRECT
help
This builds an ftrace direct function example
that hooks to wake_up_process and prints the parameters.
+config SAMPLE_FTRACE_DIRECT_MULTI
+ tristate "Build register_ftrace_direct_multi() example"
+ depends on DYNAMIC_FTRACE_WITH_DIRECT_CALLS && m
+ depends on HAVE_SAMPLE_FTRACE_DIRECT_MULTI
+ help
+ This builds an ftrace direct function example
+ that hooks to wake_up_process and schedule, and prints
+ the function addresses.
+
config SAMPLE_TRACE_ARRAY
tristate "Build sample module for kernel access to Ftrace instancess"
depends on EVENT_TRACING && m
with it.
See also Documentation/driver-api/connector.rst
+config SAMPLE_FANOTIFY_ERROR
+ bool "Build fanotify error monitoring sample"
+ depends on FANOTIFY && CC_CAN_LINK && HEADERS_INSTALL
+ help
+ When enabled, this builds an example code that uses the
+ FAN_FS_ERROR fanotify mechanism to monitor filesystem
+ errors.
+ See also Documentation/admin-guide/filesystem-monitoring.rst.
+
config SAMPLE_HIDRAW
bool "hidraw sample"
depends on CC_CAN_LINK && HEADERS_INSTALL
sb_notify() syscalls and the KEYCTL_WATCH_KEY keyctl() function.
endif # SAMPLES
+
+config HAVE_SAMPLE_FTRACE_DIRECT
+ bool
+
+config HAVE_SAMPLE_FTRACE_DIRECT_MULTI
+ bool
subdir-$(CONFIG_SAMPLE_ANDROID_BINDERFS) += binderfs
obj-$(CONFIG_SAMPLE_CONFIGFS) += configfs/
obj-$(CONFIG_SAMPLE_CONNECTOR) += connector/
+obj-$(CONFIG_SAMPLE_FANOTIFY_ERROR) += fanotify/
subdir-$(CONFIG_SAMPLE_HIDRAW) += hidraw
obj-$(CONFIG_SAMPLE_HW_BREAKPOINT) += hw_breakpoint/
obj-$(CONFIG_SAMPLE_KDB) += kdb/
obj-$(CONFIG_SAMPLE_TRACE_EVENTS) += trace_events/
obj-$(CONFIG_SAMPLE_TRACE_PRINTK) += trace_printk/
obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace/
+obj-$(CONFIG_SAMPLE_FTRACE_DIRECT_MULTI) += ftrace/
obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += ftrace/
subdir-$(CONFIG_SAMPLE_UHID) += uhid
obj-$(CONFIG_VIDEO_PCI_SKELETON) += v4l/
* Include file for sample Host Bandwidth Manager (HBM) BPF programs
*/
#define KBUILD_MODNAME "foo"
-#include <stddef.h>
-#include <stdbool.h>
#include <uapi/linux/bpf.h>
#include <uapi/linux/if_ether.h>
#include <uapi/linux/if_packet.h>
const char *mprog_filename = NULL, *mprog_name = NULL;
struct xdp_redirect_cpu *skel;
struct bpf_map_info info = {};
- char ifname_buf[IF_NAMESIZE];
struct bpf_cpumap_val value;
__u32 infosz = sizeof(info);
int ret = EXIT_FAIL_OPTION;
case 'd':
if (strlen(optarg) >= IF_NAMESIZE) {
fprintf(stderr, "-d/--dev name too long\n");
+ usage(argv, long_options, __doc__, mask, true, skel->obj);
goto end_cpu;
}
- safe_strncpy(ifname_buf, optarg, strlen(ifname_buf));
- ifindex = if_nametoindex(ifname_buf);
+ ifindex = if_nametoindex(optarg);
if (!ifindex)
ifindex = strtoul(optarg, NULL, 0);
if (!ifindex) {
__u64 xmit;
} totals;
struct {
- __u64 pps;
+ union {
+ __u64 pps;
+ __u64 num;
+ };
__u64 drop;
__u64 err;
} rx_cnt;
static void sample_summary_print(void)
{
- double period = sample_out.rx_cnt.pps;
+ double num = sample_out.rx_cnt.num;
if (sample_out.totals.rx) {
double pkts = sample_out.totals.rx;
print_always(" Packets received : %'-10llu\n",
sample_out.totals.rx);
print_always(" Average packets/s : %'-10.0f\n",
- sample_round(pkts / period));
+ sample_round(pkts / num));
}
if (sample_out.totals.redir) {
double pkts = sample_out.totals.redir;
print_always(" Packets redirected : %'-10llu\n",
sample_out.totals.redir);
print_always(" Average redir/s : %'-10.0f\n",
- sample_round(pkts / period));
+ sample_round(pkts / num));
}
if (sample_out.totals.drop)
print_always(" Rx dropped : %'-10llu\n",
print_always(" Packets transmitted : %'-10llu\n",
sample_out.totals.xmit);
print_always(" Average transmit/s : %'-10.0f\n",
- sample_round(pkts / period));
+ sample_round(pkts / num));
}
}
return 0;
}
-static void sample_summary_update(struct sample_output *out, int interval)
+static void sample_summary_update(struct sample_output *out)
{
sample_out.totals.rx += out->totals.rx;
sample_out.totals.redir += out->totals.redir;
sample_out.totals.drop_xmit += out->totals.drop_xmit;
sample_out.totals.err += out->totals.err;
sample_out.totals.xmit += out->totals.xmit;
- sample_out.rx_cnt.pps += interval;
+ sample_out.rx_cnt.num++;
}
static void sample_stats_print(int mask, struct stats_record *cur,
- struct stats_record *prev, char *prog_name,
- int interval)
+ struct stats_record *prev, char *prog_name)
{
struct sample_output out = {};
else if (mask & SAMPLE_DEVMAP_XMIT_CNT_MULTI)
stats_get_devmap_xmit_multi(cur, prev, 0, &out,
mask & SAMPLE_DEVMAP_XMIT_CNT);
- sample_summary_update(&out, interval);
+ sample_summary_update(&out);
stats_print(prog_name, mask, cur, prev, &out);
}
}
static int sample_timer_cb(int timerfd, struct stats_record **rec,
- struct stats_record **prev, int interval)
+ struct stats_record **prev)
{
char line[64] = "Summary";
int ret;
snprintf(line, sizeof(line), "%s->%s", f ?: "?", t ?: "?");
}
- sample_stats_print(sample_mask, *rec, *prev, line, interval);
+ sample_stats_print(sample_mask, *rec, *prev, line);
return 0;
}
if (pfd[0].revents & POLLIN)
ret = sample_signal_cb();
else if (pfd[1].revents & POLLIN)
- ret = sample_timer_cb(timerfd, &rec, &prev, interval);
+ ret = sample_timer_cb(timerfd, &rec, &prev);
if (ret)
break;
--- /dev/null
+fs-monitor
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0-only
+userprogs-always-y += fs-monitor
+
+userccflags += -I usr/include -Wall
+
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2021, Collabora Ltd.
+ */
+
+#define _GNU_SOURCE
+#include <errno.h>
+#include <err.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <fcntl.h>
+#include <sys/fanotify.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#ifndef FAN_FS_ERROR
+#define FAN_FS_ERROR 0x00008000
+#define FAN_EVENT_INFO_TYPE_ERROR 5
+
+struct fanotify_event_info_error {
+ struct fanotify_event_info_header hdr;
+ __s32 error;
+ __u32 error_count;
+};
+#endif
+
+#ifndef FILEID_INO32_GEN
+#define FILEID_INO32_GEN 1
+#endif
+
+#ifndef FILEID_INVALID
+#define FILEID_INVALID 0xff
+#endif
+
+static void print_fh(struct file_handle *fh)
+{
+ int i;
+ uint32_t *h = (uint32_t *) fh->f_handle;
+
+ printf("\tfh: ");
+ for (i = 0; i < fh->handle_bytes; i++)
+ printf("%hhx", fh->f_handle[i]);
+ printf("\n");
+
+ printf("\tdecoded fh: ");
+ if (fh->handle_type == FILEID_INO32_GEN)
+ printf("inode=%u gen=%u\n", h[0], h[1]);
+ else if (fh->handle_type == FILEID_INVALID && !fh->handle_bytes)
+ printf("Type %d (Superblock error)\n", fh->handle_type);
+ else
+ printf("Type %d (Unknown)\n", fh->handle_type);
+
+}
+
+static void handle_notifications(char *buffer, int len)
+{
+ struct fanotify_event_metadata *event =
+ (struct fanotify_event_metadata *) buffer;
+ struct fanotify_event_info_header *info;
+ struct fanotify_event_info_error *err;
+ struct fanotify_event_info_fid *fid;
+ int off;
+
+ for (; FAN_EVENT_OK(event, len); event = FAN_EVENT_NEXT(event, len)) {
+
+ if (event->mask != FAN_FS_ERROR) {
+ printf("unexpected FAN MARK: %llx\n",
+ (unsigned long long)event->mask);
+ goto next_event;
+ }
+
+ if (event->fd != FAN_NOFD) {
+ printf("Unexpected fd (!= FAN_NOFD)\n");
+ goto next_event;
+ }
+
+ printf("FAN_FS_ERROR (len=%d)\n", event->event_len);
+
+ for (off = sizeof(*event) ; off < event->event_len;
+ off += info->len) {
+ info = (struct fanotify_event_info_header *)
+ ((char *) event + off);
+
+ switch (info->info_type) {
+ case FAN_EVENT_INFO_TYPE_ERROR:
+ err = (struct fanotify_event_info_error *) info;
+
+ printf("\tGeneric Error Record: len=%d\n",
+ err->hdr.len);
+ printf("\terror: %d\n", err->error);
+ printf("\terror_count: %d\n", err->error_count);
+ break;
+
+ case FAN_EVENT_INFO_TYPE_FID:
+ fid = (struct fanotify_event_info_fid *) info;
+
+ printf("\tfsid: %x%x\n",
+ fid->fsid.val[0], fid->fsid.val[1]);
+ print_fh((struct file_handle *) &fid->handle);
+ break;
+
+ default:
+ printf("\tUnknown info type=%d len=%d:\n",
+ info->info_type, info->len);
+ }
+ }
+next_event:
+ printf("---\n\n");
+ }
+}
+
+int main(int argc, char **argv)
+{
+ int fd;
+
+ char buffer[BUFSIZ];
+
+ if (argc < 2) {
+ printf("Missing path argument\n");
+ return 1;
+ }
+
+ fd = fanotify_init(FAN_CLASS_NOTIF|FAN_REPORT_FID, O_RDONLY);
+ if (fd < 0)
+ errx(1, "fanotify_init");
+
+ if (fanotify_mark(fd, FAN_MARK_ADD|FAN_MARK_FILESYSTEM,
+ FAN_FS_ERROR, AT_FDCWD, argv[1])) {
+ errx(1, "fanotify_mark");
+ }
+
+ while (1) {
+ int n = read(fd, buffer, BUFSIZ);
+
+ if (n < 0)
+ errx(1, "read");
+
+ handle_notifications(buffer, n);
+ }
+
+ return 0;
+}
obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct.o
obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o
obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o
-obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-multi.o
+obj-$(CONFIG_SAMPLE_FTRACE_DIRECT_MULTI) += ftrace-direct-multi.o
CFLAGS_sample-trace-array.o := -I$(src)
obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += sample-trace-array.o
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/ftrace.h>
+#include <asm/asm-offsets.h>
void my_direct_func1(void)
{
static unsigned long my_ip = (unsigned long)schedule;
+#ifdef CONFIG_X86_64
+
asm (
" .pushsection .text, \"ax\", @progbits\n"
" .type my_tramp1, @function\n"
" .popsection\n"
);
+#endif /* CONFIG_X86_64 */
+
+#ifdef CONFIG_S390
+
+asm (
+" .pushsection .text, \"ax\", @progbits\n"
+" .type my_tramp1, @function\n"
+" .globl my_tramp1\n"
+" my_tramp1:"
+" lgr %r1,%r15\n"
+" stmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" stg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" aghi %r15,"__stringify(-STACK_FRAME_OVERHEAD)"\n"
+" stg %r1,"__stringify(__SF_BACKCHAIN)"(%r15)\n"
+" brasl %r14,my_direct_func1\n"
+" aghi %r15,"__stringify(STACK_FRAME_OVERHEAD)"\n"
+" lmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" lg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" lgr %r1,%r0\n"
+" br %r1\n"
+" .size my_tramp1, .-my_tramp1\n"
+" .type my_tramp2, @function\n"
+" .globl my_tramp2\n"
+" my_tramp2:"
+" lgr %r1,%r15\n"
+" stmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" stg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" aghi %r15,"__stringify(-STACK_FRAME_OVERHEAD)"\n"
+" stg %r1,"__stringify(__SF_BACKCHAIN)"(%r15)\n"
+" brasl %r14,my_direct_func2\n"
+" aghi %r15,"__stringify(STACK_FRAME_OVERHEAD)"\n"
+" lmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" lg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" lgr %r1,%r0\n"
+" br %r1\n"
+" .size my_tramp2, .-my_tramp2\n"
+" .popsection\n"
+);
+
+#endif /* CONFIG_S390 */
+
static unsigned long my_tramp = (unsigned long)my_tramp1;
static unsigned long tramps[2] = {
(unsigned long)my_tramp1,
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
#include <linux/sched/stat.h>
+#include <asm/asm-offsets.h>
extern void my_direct_func(unsigned long ip);
extern void my_tramp(void *);
+#ifdef CONFIG_X86_64
+
asm (
" .pushsection .text, \"ax\", @progbits\n"
" .type my_tramp, @function\n"
" .popsection\n"
);
+#endif /* CONFIG_X86_64 */
+
+#ifdef CONFIG_S390
+
+asm (
+" .pushsection .text, \"ax\", @progbits\n"
+" .type my_tramp, @function\n"
+" .globl my_tramp\n"
+" my_tramp:"
+" lgr %r1,%r15\n"
+" stmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" stg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" aghi %r15,"__stringify(-STACK_FRAME_OVERHEAD)"\n"
+" stg %r1,"__stringify(__SF_BACKCHAIN)"(%r15)\n"
+" lgr %r2,%r0\n"
+" brasl %r14,my_direct_func\n"
+" aghi %r15,"__stringify(STACK_FRAME_OVERHEAD)"\n"
+" lmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" lg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" lgr %r1,%r0\n"
+" br %r1\n"
+" .size my_tramp, .-my_tramp\n"
+" .popsection\n"
+);
+
+#endif /* CONFIG_S390 */
+
static struct ftrace_ops direct;
static int __init ftrace_direct_multi_init(void)
#include <linux/mm.h> /* for handle_mm_fault() */
#include <linux/ftrace.h>
+#include <asm/asm-offsets.h>
void my_direct_func(struct vm_area_struct *vma,
unsigned long address, unsigned int flags)
extern void my_tramp(void *);
+#ifdef CONFIG_X86_64
+
asm (
" .pushsection .text, \"ax\", @progbits\n"
" .type my_tramp, @function\n"
" .popsection\n"
);
+#endif /* CONFIG_X86_64 */
+
+#ifdef CONFIG_S390
+
+asm (
+" .pushsection .text, \"ax\", @progbits\n"
+" .type my_tramp, @function\n"
+" .globl my_tramp\n"
+" my_tramp:"
+" lgr %r1,%r15\n"
+" stmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" stg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" aghi %r15,"__stringify(-STACK_FRAME_OVERHEAD)"\n"
+" stg %r1,"__stringify(__SF_BACKCHAIN)"(%r15)\n"
+" brasl %r14,my_direct_func\n"
+" aghi %r15,"__stringify(STACK_FRAME_OVERHEAD)"\n"
+" lmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" lg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" lgr %r1,%r0\n"
+" br %r1\n"
+" .size my_tramp, .-my_tramp\n"
+" .popsection\n"
+);
+
+#endif /* CONFIG_S390 */
static int __init ftrace_direct_init(void)
{
#include <linux/sched.h> /* for wake_up_process() */
#include <linux/ftrace.h>
+#include <asm/asm-offsets.h>
void my_direct_func(struct task_struct *p)
{
extern void my_tramp(void *);
+#ifdef CONFIG_X86_64
+
asm (
" .pushsection .text, \"ax\", @progbits\n"
" .type my_tramp, @function\n"
" .popsection\n"
);
+#endif /* CONFIG_X86_64 */
+
+#ifdef CONFIG_S390
+
+asm (
+" .pushsection .text, \"ax\", @progbits\n"
+" .type my_tramp, @function\n"
+" .globl my_tramp\n"
+" my_tramp:"
+" lgr %r1,%r15\n"
+" stmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" stg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" aghi %r15,"__stringify(-STACK_FRAME_OVERHEAD)"\n"
+" stg %r1,"__stringify(__SF_BACKCHAIN)"(%r15)\n"
+" brasl %r14,my_direct_func\n"
+" aghi %r15,"__stringify(STACK_FRAME_OVERHEAD)"\n"
+" lmg %r0,%r5,"__stringify(__SF_GPRS)"(%r15)\n"
+" lg %r14,"__stringify(__SF_GPRS+8*8)"(%r15)\n"
+" lgr %r1,%r0\n"
+" br %r1\n"
+" .size my_tramp, .-my_tramp\n"
+" .popsection\n"
+);
+
+#endif /* CONFIG_S390 */
static int __init ftrace_direct_init(void)
{
# (See cmd_cc_o_c + relevant part of rule_cc_o_c)
quiet_cmd_cc_o_c = CC $(quiet_modtag) $@
- cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
+ cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< $(cmd_objtool)
ifdef CONFIG_MODVERSIONS
# When module versioning is enabled the following steps are executed:
endif # CONFIG_FTRACE_MCOUNT_USE_RECORDMCOUNT
ifdef CONFIG_STACK_VALIDATION
-ifndef CONFIG_LTO_CLANG
-__objtool_obj := $(objtree)/tools/objtool/objtool
+objtool := $(objtree)/tools/objtool/objtool
+
+objtool_args = \
+ $(if $(CONFIG_UNWINDER_ORC),orc generate,check) \
+ $(if $(part-of-module), --module) \
+ $(if $(CONFIG_FRAME_POINTER),, --no-fp) \
+ $(if $(CONFIG_GCOV_KERNEL)$(CONFIG_LTO_CLANG), --no-unreachable)\
+ $(if $(CONFIG_RETPOLINE), --retpoline) \
+ $(if $(CONFIG_X86_SMAP), --uaccess) \
+ $(if $(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL), --mcount)
+
+cmd_objtool = $(if $(objtool-enabled), ; $(objtool) $(objtool_args) $@)
+cmd_gen_objtooldep = $(if $(objtool-enabled), { echo ; echo '$@: $$(wildcard $(objtool))' ; } >> $(dot-target).cmd)
+
+endif # CONFIG_STACK_VALIDATION
+
+ifdef CONFIG_LTO_CLANG
+
+# Skip objtool for LLVM bitcode
+$(obj)/%.o: objtool-enabled :=
+
+else
# 'OBJECT_FILES_NON_STANDARD := y': skip objtool checking for a directory
# 'OBJECT_FILES_NON_STANDARD_foo.o := 'y': skip objtool checking for a file
# 'OBJECT_FILES_NON_STANDARD_foo.o := 'n': override directory skip for a file
-cmd_objtool = $(if $(patsubst y%,, \
- $(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n), \
- $(__objtool_obj) $(objtool_args) $@)
-objtool_obj = $(if $(patsubst y%,, \
- $(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n), \
- $(__objtool_obj))
-
-endif # CONFIG_LTO_CLANG
-endif # CONFIG_STACK_VALIDATION
-# Rebuild all objects when objtool changes, or is enabled/disabled.
-objtool_dep = $(objtool_obj) \
- $(wildcard include/config/ORC_UNWINDER \
- include/config/STACK_VALIDATION)
+$(obj)/%.o: objtool-enabled = $(if $(filter-out y%, \
+ $(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n),y)
+
+endif
ifdef CONFIG_TRIM_UNUSED_KSYMS
cmd_gen_ksymdeps = \
$(call cmd,gen_ksymdeps)
$(call cmd,checksrc)
$(call cmd,checkdoc)
- $(call cmd,objtool)
+ $(call cmd,gen_objtooldep)
$(call cmd,modversions_c)
$(call cmd,record_mcount)
endef
define rule_as_o_S
$(call cmd_and_fixdep,as_o_S)
$(call cmd,gen_ksymdeps)
- $(call cmd,objtool)
+ $(call cmd,gen_objtooldep)
$(call cmd,modversions_S)
endef
# Built-in and composite module parts
-.SECONDEXPANSION:
-$(obj)/%.o: $(src)/%.c $(recordmcount_source) $$(objtool_dep) FORCE
+$(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
$(call if_changed_rule,cc_o_c)
$(call cmd,force_checksrc)
$(LD) $(ld_flags) -r -o $@ \
$(shell [ -s $(@:.lto.o=.o.symversions) ] && \
echo -T $(@:.lto.o=.o.symversions)) \
- --whole-archive $(filter-out FORCE,$^)
+ --whole-archive $(filter-out FORCE,$^) \
+ $(cmd_objtool)
-ifdef CONFIG_STACK_VALIDATION
# objtool was skipped for LLVM bitcode, run it now that we have compiled
# modules into native code
-cmd_cc_lto_link_modules += ; \
- $(objtree)/tools/objtool/objtool $(objtool_args) --module $@
-endif
+$(obj)/%.lto.o: objtool-enabled = y
+$(obj)/%.lto.o: part-of-module := y
$(obj)/%.lto.o: $(obj)/%.o FORCE
$(call if_changed,cc_lto_link_modules)
$(call if_changed_dep,cpp_s_S)
quiet_cmd_as_o_S = AS $(quiet_modtag) $@
- cmd_as_o_S = $(CC) $(a_flags) -c -o $@ $<
+ cmd_as_o_S = $(CC) $(a_flags) -c -o $@ $< $(cmd_objtool)
ifdef CONFIG_ASM_MODVERSIONS
fi
endif
-$(obj)/%.o: $(src)/%.S $$(objtool_dep) FORCE
+$(obj)/%.o: $(src)/%.S FORCE
$(call if_changed_rule,as_o_S)
targets += $(filter-out $(subdir-builtin), $(real-obj-y))
--- /dev/null
+DEBUG_CFLAGS :=
+
+ifdef CONFIG_DEBUG_INFO_SPLIT
+DEBUG_CFLAGS += -gsplit-dwarf
+else
+DEBUG_CFLAGS += -g
+endif
+
+ifndef CONFIG_AS_IS_LLVM
+KBUILD_AFLAGS += -Wa,-gdwarf-2
+endif
+
+ifndef CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT
+dwarf-version-$(CONFIG_DEBUG_INFO_DWARF4) := 4
+dwarf-version-$(CONFIG_DEBUG_INFO_DWARF5) := 5
+DEBUG_CFLAGS += -gdwarf-$(dwarf-version-y)
+endif
+
+ifdef CONFIG_DEBUG_INFO_REDUCED
+DEBUG_CFLAGS += -fno-var-tracking
+ifdef CONFIG_CC_IS_GCC
+DEBUG_CFLAGS += -femit-struct-debug-baseonly
+endif
+endif
+
+ifdef CONFIG_DEBUG_INFO_COMPRESSED
+DEBUG_CFLAGS += -gz=zlib
+KBUILD_AFLAGS += -gz=zlib
+KBUILD_LDFLAGS += --compress-debug-sections=zlib
+endif
+
+KBUILD_CFLAGS += $(DEBUG_CFLAGS)
+export DEBUG_CFLAGS
mod-prelink-ext := .lto
endif
-# Objtool arguments are also needed for modfinal with LTO, so we define
-# then here to avoid duplication.
-objtool_args = \
- $(if $(CONFIG_UNWINDER_ORC),orc generate,check) \
- $(if $(part-of-module), --module) \
- $(if $(CONFIG_FRAME_POINTER),, --no-fp) \
- $(if $(CONFIG_GCOV_KERNEL)$(CONFIG_LTO_CLANG), --no-unreachable)\
- $(if $(CONFIG_RETPOLINE), --retpoline) \
- $(if $(CONFIG_X86_SMAP), --uaccess) \
- $(if $(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL), --mcount)
-
# Useful for describing the dependency of composite objects
# Usage:
# $(call multi_depend, multi_used_targets, suffix_to_remove, suffix_to_add)
# tarball targets
# ---------------------------------------------------------------------------
-tar-pkgs := dir-pkg tar-pkg targz-pkg tarbz2-pkg tarxz-pkg
+tar-pkgs := dir-pkg tar-pkg targz-pkg tarbz2-pkg tarxz-pkg tarzst-pkg
PHONY += $(tar-pkgs)
$(tar-pkgs):
$(MAKE) -f $(srctree)/Makefile
$(if $(findstring bz2,$@),$(KBZIP2), \
$(if $(findstring gz,$@),$(KGZIP), \
$(if $(findstring xz,$@),$(XZ), \
-$(error unknown target $@)))) \
+$(if $(findstring zst,$@),$(ZSTD), \
+$(error unknown target $@))))) \
-f -9 $(perf-tar).tar)
-perf-tar-pkgs := perf-tar-src-pkg perf-targz-src-pkg perf-tarbz2-src-pkg perf-tarxz-src-pkg
+perf-tar-pkgs := perf-tar-src-pkg perf-targz-src-pkg perf-tarbz2-src-pkg \
+ perf-tarxz-src-pkg perf-tarzst-src-pkg
PHONY += $(perf-tar-pkgs)
$(perf-tar-pkgs):
$(call cmd,perf_tar)
@echo ' targz-pkg - Build the kernel as a gzip compressed tarball'
@echo ' tarbz2-pkg - Build the kernel as a bzip2 compressed tarball'
@echo ' tarxz-pkg - Build the kernel as a xz compressed tarball'
+ @echo ' tarzst-pkg - Build the kernel as a zstd compressed tarball'
@echo ' perf-tar-src-pkg - Build $(perf-tar).tar source tarball'
@echo ' perf-targz-src-pkg - Build $(perf-tar).tar.gz source tarball'
@echo ' perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball'
@echo ' perf-tarxz-src-pkg - Build $(perf-tar).tar.xz source tarball'
+ @echo ' perf-tarzst-src-pkg - Build $(perf-tar).tar.zst source tarball'
.PHONY: $(PHONY)
my $spelling_file = "$D/spelling.txt";
my $codespell = 0;
my $codespellfile = "/usr/share/codespell/dictionary.txt";
+my $user_codespellfile = "";
my $conststructsfile = "$D/const_structs.checkpatch";
my $docsfile = "$D/../Documentation/dev-tools/checkpatch.rst";
my $typedefsfile;
--ignore-perl-version override checking of perl version. expect
runtime errors.
--codespell Use the codespell dictionary for spelling/typos
- (default:/usr/share/codespell/dictionary.txt)
+ (default:$codespellfile)
--codespellfile Use this codespell dictionary
--typedefsfile Read additional types from this file
--color[=WHEN] Use colors 'always', 'never', or only when output
'debug=s' => \%debug,
'test-only=s' => \$tst_only,
'codespell!' => \$codespell,
- 'codespellfile=s' => \$codespellfile,
+ 'codespellfile=s' => \$user_codespellfile,
'typedefsfile=s' => \$typedefsfile,
'color=s' => \$color,
'no-color' => \$color, #keep old behaviors of -nocolor
'kconfig-prefix=s' => \${CONFIG_},
'h|help' => \$help,
'version' => \$help
-) or help(1);
+) or $help = 2;
+
+if ($user_codespellfile) {
+ # Use the user provided codespell file unconditionally
+ $codespellfile = $user_codespellfile;
+} elsif (!(-f $codespellfile)) {
+ # If /usr/share/codespell/dictionary.txt is not present, try to find it
+ # under codespell's install directory: <codespell_root>/data/dictionary.txt
+ if (($codespell || $help) && which("codespell") ne "" && which("python") ne "") {
+ my $python_codespell_dict = << "EOF";
+
+import os.path as op
+import codespell_lib
+codespell_dir = op.dirname(codespell_lib.__file__)
+codespell_file = op.join(codespell_dir, 'data', 'dictionary.txt')
+print(codespell_file, end='')
+EOF
+
+ my $codespell_dict = `python -c "$python_codespell_dict" 2> /dev/null`;
+ $codespellfile = $codespell_dict if (-f $codespell_dict);
+ }
+}
-help(0) if ($help);
+# $help is 1 if either -h, --help or --version is passed as option - exitcode: 0
+# $help is 2 if invalid option is passed - exitcode: 1
+help($help - 1) if ($help);
die "$P: --git cannot be used with --file or --fix\n" if ($git && ($file || $fix));
die "$P: --verbose cannot be used with --terse\n" if ($verbose && $terse);
____cacheline_aligned|
____cacheline_aligned_in_smp|
____cacheline_internodealigned_in_smp|
- __weak
+ __weak|
+ __alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\)
}x;
our $Modifier;
our $Inline = qr{inline|__always_inline|noinline|__inline|__inline__};
# XXX(foo);
# EXPORT_SYMBOL(something_foo);
my $name = $1;
+ $name =~ s/^\s*($Ident).*/$1/;
if ($stat =~ /^(?:.\s*}\s*\n)?.([A-Z_]+)\s*\(\s*($Ident)/ &&
$name =~ /^${Ident}_$2/) {
#print "FOO C name<$name>\n";
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/// do_div() does a 64-by-32 division.
+/// When the divisor is long, unsigned long, u64, or s64,
+/// do_div() truncates it to 32 bits, this means it can test
+/// non-zero and be truncated to 0 for division on 64bit platforms.
+///
+//# This makes an effort to find those inappropriate do_div() calls.
+//
+// Confidence: Moderate
+// Copyright: (C) 2020 Wen Yang, Alibaba.
+// Comments:
+// Options: --no-includes --include-headers
+
+virtual context
+virtual org
+virtual report
+
+@initialize:python@
+@@
+
+def get_digit_type_and_value(str):
+ is_digit = False
+ value = 0
+
+ try:
+ if (str.isdigit()):
+ is_digit = True
+ value = int(str, 0)
+ elif (str.upper().endswith('ULL')):
+ is_digit = True
+ value = int(str[:-3], 0)
+ elif (str.upper().endswith('LL')):
+ is_digit = True
+ value = int(str[:-2], 0)
+ elif (str.upper().endswith('UL')):
+ is_digit = True
+ value = int(str[:-2], 0)
+ elif (str.upper().endswith('L')):
+ is_digit = True
+ value = int(str[:-1], 0)
+ elif (str.upper().endswith('U')):
+ is_digit = True
+ value = int(str[:-1], 0)
+ except Exception as e:
+ print('Error:',e)
+ is_digit = False
+ value = 0
+ finally:
+ return is_digit, value
+
+def filter_out_safe_constants(str):
+ is_digit, value = get_digit_type_and_value(str)
+ if (is_digit):
+ if (value >= 0x100000000):
+ return True
+ else:
+ return False
+ else:
+ return True
+
+def construct_warnings(suggested_fun):
+ msg="WARNING: do_div() does a 64-by-32 division, please consider using %s instead."
+ return msg % suggested_fun
+
+@depends on context@
+expression f;
+long l: script:python() { filter_out_safe_constants(l) };
+unsigned long ul : script:python() { filter_out_safe_constants(ul) };
+u64 ul64 : script:python() { filter_out_safe_constants(ul64) };
+s64 sl64 : script:python() { filter_out_safe_constants(sl64) };
+
+@@
+(
+* do_div(f, l);
+|
+* do_div(f, ul);
+|
+* do_div(f, ul64);
+|
+* do_div(f, sl64);
+)
+
+@r depends on (org || report)@
+expression f;
+position p;
+long l: script:python() { filter_out_safe_constants(l) };
+unsigned long ul : script:python() { filter_out_safe_constants(ul) };
+u64 ul64 : script:python() { filter_out_safe_constants(ul64) };
+s64 sl64 : script:python() { filter_out_safe_constants(sl64) };
+@@
+(
+do_div@p(f, l);
+|
+do_div@p(f, ul);
+|
+do_div@p(f, ul64);
+|
+do_div@p(f, sl64);
+)
+
+@script:python depends on org@
+p << r.p;
+ul << r.ul;
+@@
+
+coccilib.org.print_todo(p[0], construct_warnings("div64_ul"))
+
+@script:python depends on org@
+p << r.p;
+l << r.l;
+@@
+
+coccilib.org.print_todo(p[0], construct_warnings("div64_long"))
+
+@script:python depends on org@
+p << r.p;
+ul64 << r.ul64;
+@@
+
+coccilib.org.print_todo(p[0], construct_warnings("div64_u64"))
+
+@script:python depends on org@
+p << r.p;
+sl64 << r.sl64;
+@@
+
+coccilib.org.print_todo(p[0], construct_warnings("div64_s64"))
+
+@script:python depends on report@
+p << r.p;
+ul << r.ul;
+@@
+
+coccilib.report.print_report(p[0], construct_warnings("div64_ul"))
+
+@script:python depends on report@
+p << r.p;
+l << r.l;
+@@
+
+coccilib.report.print_report(p[0], construct_warnings("div64_long"))
+
+@script:python depends on report@
+p << r.p;
+sl64 << r.sl64;
+@@
+
+coccilib.report.print_report(p[0], construct_warnings("div64_s64"))
+
+@script:python depends on report@
+p << r.p;
+ul64 << r.ul64;
+@@
+
+coccilib.report.print_report(p[0], construct_warnings("div64_u64"))
seq_operations
sirfsoc_padmux
snd_ac97_build_ops
+snd_pcm_ops
+snd_rawmidi_ops
snd_soc_component_driver
+snd_soc_dai_ops
+snd_soc_ops
soc_pcmcia_socket_ops
stacktrace_ops
sysfs_ops
fi
echo Code starting with the faulting instruction > $T.aa
echo =========================================== >> $T.aa
-code=`echo $code | sed -e 's/ [<(]/ /;s/[>)] / /;s/ /,0x/g; s/[>)]$//'`
+code=`echo $code | sed -e 's/\r//;s/ [<(]/ /;s/[>)] / /;s/ /,0x/g; s/[>)]$//'`
echo -n " .$type 0x" > $T.s
echo $code >> $T.s
disas $T 0
# drop all current symbols and reload vmlinux
orig_vmlinux = 'vmlinux'
for obj in gdb.objfiles():
- if obj.filename.endswith('vmlinux'):
+ if (obj.filename.endswith('vmlinux') or
+ obj.filename.endswith('vmlinux.debug')):
orig_vmlinux = obj.filename
gdb.execute("symbol-file", to_string=True)
gdb.execute("symbol-file {0}".format(orig_vmlinux))
switch (input_mode) {
case listnewconfig:
- if (sym->name) {
- const char *str;
-
- if (sym->type == S_STRING) {
- str = sym_get_string_value(sym);
- str = sym_escape_string_value(str);
- printf("%s%s=%s\n", CONFIG_, sym->name, str);
- free((void *)str);
- } else {
- str = sym_get_string_value(sym);
- printf("%s%s=%s\n", CONFIG_, sym->name, str);
- }
- }
+ if (sym->name)
+ print_symbol_for_listconfig(sym);
break;
case helpnewconfig:
printf("-----\n");
#include <fcntl.h>
#include <limits.h>
#include <stdarg.h>
+#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* touch depfile for symbol 'name' */
static int conf_touch_dep(const char *name)
{
- int fd, ret;
- char *d;
+ int fd;
/* check overflow: prefix + name + '\0' must fit in buffer. */
if (depfile_prefix_len + strlen(name) + 1 > sizeof(depfile_path))
return -1;
- d = depfile_path + depfile_prefix_len;
- strcpy(d, name);
+ strcpy(depfile_path + depfile_prefix_len, name);
- /* Assume directory path already exists. */
fd = open(depfile_path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
- if (fd == -1) {
- if (errno != ENOENT)
- return -1;
-
- ret = make_parent_dir(depfile_path);
- if (ret)
- return ret;
-
- /* Try it again. */
- fd = open(depfile_path, O_WRONLY | O_CREAT | O_TRUNC, 0644);
- if (fd == -1)
- return -1;
- }
+ if (fd == -1)
+ return -1;
close(fd);
return 0;
}
-struct conf_printer {
- void (*print_symbol)(FILE *, struct symbol *, const char *, void *);
- void (*print_comment)(FILE *, const char *, void *);
-};
-
static void conf_warning(const char *fmt, ...)
__attribute__ ((format (printf, 1, 2)));
return name ? name : "include/config/auto.conf";
}
+static const char *conf_get_autoheader_name(void)
+{
+ char *name = getenv("KCONFIG_AUTOHEADER");
+
+ return name ? name : "include/generated/autoconf.h";
+}
+
static int conf_set_sym_val(struct symbol *sym, int def, int def_flags, char *p)
{
char *p2;
return 0;
}
-/*
- * Kconfig configuration printer
- *
- * This printer is used when generating the resulting configuration after
- * kconfig invocation and `defconfig' files. Unset symbol might be omitted by
- * passing a non-NULL argument to the printer.
- *
- */
-static void
-kconfig_print_symbol(FILE *fp, struct symbol *sym, const char *value, void *arg)
+struct comment_style {
+ const char *decoration;
+ const char *prefix;
+ const char *postfix;
+};
+
+static const struct comment_style comment_style_pound = {
+ .decoration = "#",
+ .prefix = "#",
+ .postfix = "#",
+};
+
+static const struct comment_style comment_style_c = {
+ .decoration = " *",
+ .prefix = "/*",
+ .postfix = " */",
+};
+
+static void conf_write_heading(FILE *fp, const struct comment_style *cs)
{
+ fprintf(fp, "%s\n", cs->prefix);
- switch (sym->type) {
- case S_BOOLEAN:
- case S_TRISTATE:
- if (*value == 'n') {
- bool skip_unset = (arg != NULL);
+ fprintf(fp, "%s Automatically generated file; DO NOT EDIT.\n",
+ cs->decoration);
- if (!skip_unset)
- fprintf(fp, "# %s%s is not set\n",
- CONFIG_, sym->name);
- return;
- }
- break;
- default:
- break;
- }
+ fprintf(fp, "%s %s\n", cs->decoration, rootmenu.prompt->text);
- fprintf(fp, "%s%s=%s\n", CONFIG_, sym->name, value);
+ fprintf(fp, "%s\n", cs->postfix);
}
-static void
-kconfig_print_comment(FILE *fp, const char *value, void *arg)
+/* The returned pointer must be freed on the caller side */
+static char *escape_string_value(const char *in)
{
- const char *p = value;
- size_t l;
+ const char *p;
+ char *out;
+ size_t len;
- for (;;) {
- l = strcspn(p, "\n");
- fprintf(fp, "#");
- if (l) {
- fprintf(fp, " ");
- xfwrite(p, l, 1, fp);
- p += l;
- }
- fprintf(fp, "\n");
- if (*p++ == '\0')
+ len = strlen(in) + strlen("\"\"") + 1;
+
+ p = in;
+ while (1) {
+ p += strcspn(p, "\"\\");
+
+ if (p[0] == '\0')
break;
+
+ len++;
+ p++;
}
-}
-static struct conf_printer kconfig_printer_cb =
-{
- .print_symbol = kconfig_print_symbol,
- .print_comment = kconfig_print_comment,
-};
+ out = xmalloc(len);
+ out[0] = '\0';
+
+ strcat(out, "\"");
+
+ p = in;
+ while (1) {
+ len = strcspn(p, "\"\\");
+ strncat(out, p, len);
+ p += len;
+
+ if (p[0] == '\0')
+ break;
+
+ strcat(out, "\\");
+ strncat(out, p++, 1);
+ }
+
+ strcat(out, "\"");
+
+ return out;
+}
/*
- * Header printer
+ * Kconfig configuration printer
*
- * This printer is used when generating the `include/generated/autoconf.h' file.
+ * This printer is used when generating the resulting configuration after
+ * kconfig invocation and `defconfig' files. Unset symbol might be omitted by
+ * passing a non-NULL argument to the printer.
*/
-static void
-header_print_symbol(FILE *fp, struct symbol *sym, const char *value, void *arg)
+enum output_n { OUTPUT_N, OUTPUT_N_AS_UNSET, OUTPUT_N_NONE };
+
+static void __print_symbol(FILE *fp, struct symbol *sym, enum output_n output_n,
+ bool escape_string)
{
+ const char *val;
+ char *escaped = NULL;
- switch (sym->type) {
- case S_BOOLEAN:
- case S_TRISTATE: {
- const char *suffix = "";
+ if (sym->type == S_UNKNOWN)
+ return;
- switch (*value) {
- case 'n':
- break;
- case 'm':
- suffix = "_MODULE";
- /* fall through */
- default:
- fprintf(fp, "#define %s%s%s 1\n",
- CONFIG_, sym->name, suffix);
- }
- break;
- }
- case S_HEX: {
- const char *prefix = "";
+ val = sym_get_string_value(sym);
- if (value[0] != '0' || (value[1] != 'x' && value[1] != 'X'))
- prefix = "0x";
- fprintf(fp, "#define %s%s %s%s\n",
- CONFIG_, sym->name, prefix, value);
- break;
+ if ((sym->type == S_BOOLEAN || sym->type == S_TRISTATE) &&
+ output_n != OUTPUT_N && *val == 'n') {
+ if (output_n == OUTPUT_N_AS_UNSET)
+ fprintf(fp, "# %s%s is not set\n", CONFIG_, sym->name);
+ return;
}
- case S_STRING:
- case S_INT:
- fprintf(fp, "#define %s%s %s\n",
- CONFIG_, sym->name, value);
- break;
- default:
- break;
+
+ if (sym->type == S_STRING && escape_string) {
+ escaped = escape_string_value(val);
+ val = escaped;
}
+ fprintf(fp, "%s%s=%s\n", CONFIG_, sym->name, val);
+
+ free(escaped);
}
-static void
-header_print_comment(FILE *fp, const char *value, void *arg)
+static void print_symbol_for_dotconfig(FILE *fp, struct symbol *sym)
{
- const char *p = value;
- size_t l;
+ __print_symbol(fp, sym, OUTPUT_N_AS_UNSET, true);
+}
- fprintf(fp, "/*\n");
- for (;;) {
- l = strcspn(p, "\n");
- fprintf(fp, " *");
- if (l) {
- fprintf(fp, " ");
- xfwrite(p, l, 1, fp);
- p += l;
- }
- fprintf(fp, "\n");
- if (*p++ == '\0')
- break;
- }
- fprintf(fp, " */\n");
+static void print_symbol_for_autoconf(FILE *fp, struct symbol *sym)
+{
+ __print_symbol(fp, sym, OUTPUT_N_NONE, true);
}
-static struct conf_printer header_printer_cb =
+void print_symbol_for_listconfig(struct symbol *sym)
{
- .print_symbol = header_print_symbol,
- .print_comment = header_print_comment,
-};
+ __print_symbol(stdout, sym, OUTPUT_N, true);
+}
-static void conf_write_symbol(FILE *fp, struct symbol *sym,
- struct conf_printer *printer, void *printer_arg)
+static void print_symbol_for_c(FILE *fp, struct symbol *sym)
{
- const char *str;
+ const char *val;
+ const char *sym_suffix = "";
+ const char *val_prefix = "";
+ char *escaped = NULL;
+
+ if (sym->type == S_UNKNOWN)
+ return;
+
+ val = sym_get_string_value(sym);
switch (sym->type) {
- case S_UNKNOWN:
+ case S_BOOLEAN:
+ case S_TRISTATE:
+ switch (*val) {
+ case 'n':
+ return;
+ case 'm':
+ sym_suffix = "_MODULE";
+ /* fall through */
+ default:
+ val = "1";
+ }
break;
- case S_STRING:
- str = sym_get_string_value(sym);
- str = sym_escape_string_value(str);
- printer->print_symbol(fp, sym, str, printer_arg);
- free((void *)str);
+ case S_HEX:
+ if (val[0] != '0' || (val[1] != 'x' && val[1] != 'X'))
+ val_prefix = "0x";
break;
+ case S_STRING:
+ escaped = escape_string_value(val);
+ val = escaped;
default:
- str = sym_get_string_value(sym);
- printer->print_symbol(fp, sym, str, printer_arg);
+ break;
}
-}
-
-static void
-conf_write_heading(FILE *fp, struct conf_printer *printer, void *printer_arg)
-{
- char buf[256];
- snprintf(buf, sizeof(buf),
- "\n"
- "Automatically generated file; DO NOT EDIT.\n"
- "%s\n",
- rootmenu.prompt->text);
+ fprintf(fp, "#define %s%s%s %s%s\n", CONFIG_, sym->name, sym_suffix,
+ val_prefix, val);
- printer->print_comment(fp, buf, printer_arg);
+ free(escaped);
}
/*
goto next_menu;
}
}
- conf_write_symbol(out, sym, &kconfig_printer_cb, NULL);
+ print_symbol_for_dotconfig(out, sym);
}
next_menu:
if (menu->list != NULL) {
if (!out)
return 1;
- conf_write_heading(out, &kconfig_printer_cb, NULL);
+ conf_write_heading(out, &comment_style_pound);
if (!conf_get_changed())
sym_clear_all_valid();
need_newline = false;
}
sym->flags |= SYMBOL_WRITTEN;
- conf_write_symbol(out, sym, &kconfig_printer_cb, NULL);
+ print_symbol_for_dotconfig(out, sym);
}
next:
}
/* write a dependency file as used by kbuild to track dependencies */
-static int conf_write_dep(const char *name)
+static int conf_write_autoconf_cmd(const char *autoconf_name)
{
+ char name[PATH_MAX], tmp[PATH_MAX];
struct file *file;
FILE *out;
+ int ret;
- out = fopen("..config.tmp", "w");
- if (!out)
- return 1;
- fprintf(out, "deps_config := \\\n");
- for (file = file_list; file; file = file->next) {
- if (file->next)
- fprintf(out, "\t%s \\\n", file->name);
- else
- fprintf(out, "\t%s\n", file->name);
+ ret = snprintf(name, sizeof(name), "%s.cmd", autoconf_name);
+ if (ret >= sizeof(name)) /* check truncation */
+ return -1;
+
+ if (make_parent_dir(name))
+ return -1;
+
+ ret = snprintf(tmp, sizeof(tmp), "%s.cmd.tmp", autoconf_name);
+ if (ret >= sizeof(tmp)) /* check truncation */
+ return -1;
+
+ out = fopen(tmp, "w");
+ if (!out) {
+ perror("fopen");
+ return -1;
}
- fprintf(out, "\n%s: \\\n"
- "\t$(deps_config)\n\n", conf_get_autoconfig_name());
- env_write_dep(out, conf_get_autoconfig_name());
+ fprintf(out, "deps_config := \\\n");
+ for (file = file_list; file; file = file->next)
+ fprintf(out, "\t%s \\\n", file->name);
+
+ fprintf(out, "\n%s: $(deps_config)\n\n", autoconf_name);
+
+ env_write_dep(out, autoconf_name);
fprintf(out, "\n$(deps_config): ;\n");
+
+ if (ferror(out)) /* error check for all fprintf() calls */
+ return -1;
+
fclose(out);
- if (make_parent_dir(name))
- return 1;
- rename("..config.tmp", name);
+ if (rename(tmp, name)) {
+ perror("rename");
+ return -1;
+ }
+
return 0;
}
return 0;
}
+static int __conf_write_autoconf(const char *filename,
+ void (*print_symbol)(FILE *, struct symbol *),
+ const struct comment_style *comment_style)
+{
+ char tmp[PATH_MAX];
+ FILE *file;
+ struct symbol *sym;
+ int ret, i;
+
+ if (make_parent_dir(filename))
+ return -1;
+
+ ret = snprintf(tmp, sizeof(tmp), "%s.tmp", filename);
+ if (ret >= sizeof(tmp)) /* check truncation */
+ return -1;
+
+ file = fopen(tmp, "w");
+ if (!file) {
+ perror("fopen");
+ return -1;
+ }
+
+ conf_write_heading(file, comment_style);
+
+ for_all_symbols(i, sym)
+ if ((sym->flags & SYMBOL_WRITE) && sym->name)
+ print_symbol(file, sym);
+
+ /* check possible errors in conf_write_heading() and print_symbol() */
+ if (ferror(file))
+ return -1;
+
+ fclose(file);
+
+ if (rename(tmp, filename)) {
+ perror("rename");
+ return -1;
+ }
+
+ return 0;
+}
+
int conf_write_autoconf(int overwrite)
{
struct symbol *sym;
- const char *name;
const char *autoconf_name = conf_get_autoconfig_name();
- FILE *out, *out_h;
- int i;
+ int ret, i;
if (!overwrite && is_present(autoconf_name))
return 0;
- conf_write_dep("include/config/auto.conf.cmd");
+ ret = conf_write_autoconf_cmd(autoconf_name);
+ if (ret)
+ return -1;
if (conf_touch_deps())
return 1;
- out = fopen(".tmpconfig", "w");
- if (!out)
- return 1;
-
- out_h = fopen(".tmpconfig.h", "w");
- if (!out_h) {
- fclose(out);
- return 1;
- }
-
- conf_write_heading(out, &kconfig_printer_cb, NULL);
- conf_write_heading(out_h, &header_printer_cb, NULL);
-
- for_all_symbols(i, sym) {
+ for_all_symbols(i, sym)
sym_calc_value(sym);
- if (!(sym->flags & SYMBOL_WRITE) || !sym->name)
- continue;
-
- /* write symbols to auto.conf and autoconf.h */
- conf_write_symbol(out, sym, &kconfig_printer_cb, (void *)1);
- conf_write_symbol(out_h, sym, &header_printer_cb, NULL);
- }
- fclose(out);
- fclose(out_h);
- name = getenv("KCONFIG_AUTOHEADER");
- if (!name)
- name = "include/generated/autoconf.h";
- if (make_parent_dir(name))
- return 1;
- if (rename(".tmpconfig.h", name))
- return 1;
+ ret = __conf_write_autoconf(conf_get_autoheader_name(),
+ print_symbol_for_c,
+ &comment_style_c);
+ if (ret)
+ return ret;
- if (make_parent_dir(autoconf_name))
- return 1;
/*
- * This must be the last step, kbuild has a dependency on auto.conf
- * and this marks the successful completion of the previous steps.
+ * Create include/config/auto.conf. This must be the last step because
+ * Kbuild has a dependency on auto.conf and this marks the successful
+ * completion of the previous steps.
*/
- if (rename(".tmpconfig", autoconf_name))
- return 1;
+ ret = __conf_write_autoconf(conf_get_autoconfig_name(),
+ print_symbol_for_autoconf,
+ &comment_style_pound);
+ if (ret)
+ return ret;
return 0;
}
n [A-Za-z0-9_-]
%%
- int str = 0;
- int ts, i;
+ char open_quote = 0;
#.* /* ignore comment */
[ \t]* /* whitespaces */
":=" return T_COLON_EQUAL;
"+=" return T_PLUS_EQUAL;
\"|\' {
- str = yytext[0];
+ open_quote = yytext[0];
new_string();
BEGIN(STRING);
}
append_string(yytext + 1, yyleng - 1);
}
\'|\" {
- if (str == yytext[0]) {
+ if (open_quote == yytext[0]) {
BEGIN(INITIAL);
yylval.string = text;
return T_WORD_QUOTE;
<HELP>{
[ \t]+ {
+ int ts, i;
+
ts = 0;
for (i = 0; i < yyleng; i++) {
if (yytext[i] == '\t')
struct symbol * sym_lookup(const char *name, int flags);
struct symbol * sym_find(const char *name);
-const char * sym_escape_string_value(const char *in);
+void print_symbol_for_listconfig(struct symbol *sym);
struct symbol ** sym_re_search(const char *pattern);
const char * sym_type_name(enum symbol_type type);
void sym_calc_value(struct symbol *sym);
get_dep_str(r, prop->visible.expr, " Visible if: ");
menu = prop->menu->parent;
- for (i = 0; menu != &rootmenu && i < 8; menu = menu->parent) {
+ for (i = 0; menu && i < 8; menu = menu->parent) {
bool accessible = menu_is_visible(menu);
submenu[i++] = menu;
list_add_tail(&jump->entries, head);
}
- if (i > 0) {
- str_printf(r, " Location:\n");
- for (j = 4; --i >= 0; j += 2) {
- menu = submenu[i];
- if (jump && menu == location)
- jump->offset = strlen(r->s);
- str_printf(r, "%*c-> %s", j, ' ',
- menu_get_prompt(menu));
- if (menu->sym) {
- str_printf(r, " (%s [=%s])", menu->sym->name ?
- menu->sym->name : "<choice>",
- sym_get_string_value(menu->sym));
- }
- str_append(r, "\n");
+ str_printf(r, " Location:\n");
+ for (j = 4; --i >= 0; j += 2) {
+ menu = submenu[i];
+ if (jump && menu == location)
+ jump->offset = strlen(r->s);
+
+ if (menu == &rootmenu)
+ /* The real rootmenu prompt is ugly */
+ str_printf(r, "%*cMain menu", j, ' ');
+ else
+ str_printf(r, "%*c-> %s", j, ' ', menu_get_prompt(menu));
+
+ if (menu->sym) {
+ str_printf(r, " (%s [=%s])", menu->sym->name ?
+ menu->sym->name : "<choice>",
+ sym_get_string_value(menu->sym));
}
+ str_append(r, "\n");
}
}
return symbol;
}
-const char *sym_escape_string_value(const char *in)
-{
- const char *p;
- size_t reslen;
- char *res;
- size_t l;
-
- reslen = strlen(in) + strlen("\"\"") + 1;
-
- p = in;
- for (;;) {
- l = strcspn(p, "\"\\");
- p += l;
-
- if (p[0] == '\0')
- break;
-
- reslen++;
- p++;
- }
-
- res = xmalloc(reslen);
- res[0] = '\0';
-
- strcat(res, "\"");
-
- p = in;
- for (;;) {
- l = strcspn(p, "\"\\");
- strncat(res, p, l);
- p += l;
-
- if (p[0] == '\0')
- break;
-
- strcat(res, "\\");
- strncat(res, p++, 1);
- }
-
- strcat(res, "\"");
- return res;
-}
-
struct sym_match {
struct symbol *sym;
off_t so, eo;
# kallsyms support
# Generate section listing all symbols and add it into vmlinux
# It's a three step process:
- # 1) Link .tmp_vmlinux1 so it has all symbols and sections,
+ # 1) Link .tmp_vmlinux.kallsyms1 so it has all symbols and sections,
# but __kallsyms is empty.
# Running kallsyms on that gives us .tmp_kallsyms1.o with
# the right size
- # 2) Link .tmp_vmlinux2 so it now has a __kallsyms section of
+ # 2) Link .tmp_vmlinux.kallsyms2 so it now has a __kallsyms section of
# the right size, but due to the added section, some
# addresses have shifted.
- # From here, we generate a correct .tmp_kallsyms2.o
+ # From here, we generate a correct .tmp_vmlinux.kallsyms2.o
# 3) That link may have expanded the kernel image enough that
# more linker branch stubs / trampolines had to be added, which
# introduces new names, which further expands kallsyms. Do another
DEVID_FIELD(dfl_device_id, type);
DEVID_FIELD(dfl_device_id, feature_id);
+ DEVID(ishtp_device_id);
+ DEVID_FIELD(ishtp_device_id, guid);
+
return 0;
}
uuid.b[12], uuid.b[13], uuid.b[14], uuid.b[15]);
}
+static inline void add_guid(char *str, guid_t guid)
+{
+ int len = strlen(str);
+
+ sprintf(str + len, "%02X%02X%02X%02X-%02X%02X-%02X%02X-%02X%02X-%02X%02X%02X%02X%02X%02X",
+ guid.b[3], guid.b[2], guid.b[1], guid.b[0],
+ guid.b[5], guid.b[4], guid.b[7], guid.b[6],
+ guid.b[8], guid.b[9], guid.b[10], guid.b[11],
+ guid.b[12], guid.b[13], guid.b[14], guid.b[15]);
+}
+
/**
* Check that sizeof(device_id type) are consistent with size of section
* in .o file. If in-consistent then userspace and kernel does not agree
return 1;
}
+/* Looks like: ishtp:{guid} */
+static int do_ishtp_entry(const char *filename, void *symval, char *alias)
+{
+ DEF_FIELD(symval, ishtp_device_id, guid);
+
+ strcpy(alias, ISHTP_MODULE_PREFIX "{");
+ add_guid(alias, guid);
+ strcat(alias, "}");
+
+ return 1;
+}
+
static int do_auxiliary_entry(const char *filename, void *symval, char *alias)
{
DEF_FIELD_ADDR(symval, auxiliary_device_id, name);
{"auxiliary", SIZE_auxiliary_device_id, do_auxiliary_entry},
{"ssam", SIZE_ssam_device_id, do_ssam_entry},
{"dfl", SIZE_dfl_device_id, do_dfl_entry},
+ {"ishtp", SIZE_ishtp_device_id, do_ishtp_entry},
};
/* Create MODULE_ALIAS() statements.
opts="-I ${XZ}"
tarball=${tarball}.xz
;;
+ tarzst-pkg)
+ opts="-I ${ZSTD}"
+ tarball=${tarball}.zst
+ ;;
*)
echo "Unknown tarball target \"${1}\" requested, please add it to ${0}." >&2
exit 1
do
rm -f arch/arm/boot/compressed/${f}
done
+
+ for f in uart-ath79.c ashldi3.c bswapdi.c bswapsi.c
+ do
+ rm -f arch/mips/boot/compressed/${f}
+ done
fi
assumtpion||assumption
asuming||assuming
asycronous||asynchronous
+asychronous||asynchronous
asynchnous||asynchronous
asynchromous||asynchronous
asymetric||asymmetric
betweeen||between
bianries||binaries
bitmast||bitmask
+bitwiedh||bitwidth
boardcast||broadcast
borad||board
boundry||boundary
calulate||calculate
cancelation||cancellation
cancle||cancel
+cant||can't
+cant'||can't
canot||cannot
+cann't||can't
capabilites||capabilities
capabilties||capabilities
capabilty||capability
disgest||digest
disired||desired
dispalying||displaying
+dissable||disable
diplay||display
directon||direction
direcly||directly
exceds||exceeds
exceeed||exceed
excellant||excellent
+exchnage||exchange
execeeded||exceeded
execeeds||exceeds
exeed||exceed
milliseonds||milliseconds
minium||minimum
minimam||minimum
+minimun||minimum
miniumum||minimum
minumum||minimum
misalinged||misaligned
mnay||many
modfiy||modify
modifer||modifier
+modul||module
modulues||modules
momery||memory
memomry||memory
programers||programmers
programm||program
programms||programs
+progres||progress
progresss||progress
prohibitted||prohibited
prohibitting||prohibiting
setts||sets
settting||setting
shapshot||snapshot
+shoft||shift
shotdown||shutdown
shoud||should
shouldnt||shouldn't
symetric||symmetric
synax||syntax
synchonized||synchronized
+sychronization||synchronization
synchronuously||synchronously
syncronize||synchronize
syncronized||synchronized
unfortunatelly||unfortunately
unifiy||unify
uniterrupted||uninterrupted
+uninterruptable||uninterruptible
unintialized||uninitialized
unitialized||uninitialized
unkmown||unknown
unvalid||invalid
upate||update
upsupported||unsupported
+useable||usable
usefule||useful
usefull||useful
usege||usage
vaule||value
verbse||verbose
veify||verify
+verfication||verification
veriosn||version
verisons||versions
verison||version
vitual||virtual
vunerable||vulnerable
wakeus||wakeups
+was't||wasn't
wathdog||watchdog
wating||waiting
wiat||wait
or are part of the kernel text. This kills entire classes
of heap overflow exploits and similar kernel memory exposures.
-config HARDENED_USERCOPY_FALLBACK
- bool "Allow usercopy whitelist violations to fallback to object size"
- depends on HARDENED_USERCOPY
- default y
- help
- This is a temporary option that allows missing usercopy whitelists
- to be discovered via a WARN() to the kernel log, instead of
- rejecting the copy, falling back to non-whitelisted hardened
- usercopy that checks the slab allocation size instead of the
- whitelist size. This option will be removed once it seems like
- all missing usercopy whitelists have been identified and fixed.
- Booting with "slab_common.usercopy_fallback=Y/N" can change
- this setting.
-
config HARDENED_USERCOPY_PAGESPAN
bool "Refuse to copy allocations that span multiple pages"
depends on HARDENED_USERCOPY
};
#define MULTI_TRANSACTION_LIMIT (PAGE_SIZE - sizeof(struct multi_transaction))
-/* TODO: replace with per file lock */
-static DEFINE_SPINLOCK(multi_transaction_lock);
static void multi_transaction_kref(struct kref *kref)
{
AA_BUG(n > MULTI_TRANSACTION_LIMIT);
new->size = n;
- spin_lock(&multi_transaction_lock);
+ spin_lock(&file->f_lock);
old = (struct multi_transaction *) file->private_data;
file->private_data = new;
- spin_unlock(&multi_transaction_lock);
+ spin_unlock(&file->f_lock);
put_multi_transaction(old);
}
struct multi_transaction *t;
ssize_t ret;
- spin_lock(&multi_transaction_lock);
+ spin_lock(&file->f_lock);
t = get_multi_transaction(file->private_data);
- spin_unlock(&multi_transaction_lock);
+ spin_unlock(&file->f_lock);
+
if (!t)
return 0;
struct aa_loaddata *loaddata;
struct rawdata_f_data *private;
- if (!policy_view_capable(NULL))
+ if (!aa_current_policy_view_capable(NULL))
return -EACCES;
loaddata = __aa_get_loaddata(inode->i_private);
/**
* __next_profile - step to the next profile in a profile tree
- * @profile: current profile in tree (NOT NULL)
+ * @p: current profile in tree (NOT NULL)
*
* Perform a depth first traversal on the profile tree in a namespace
*
static int profiles_open(struct inode *inode, struct file *file)
{
- if (!policy_view_capable(NULL))
+ if (!aa_current_policy_view_capable(NULL))
return -EACCES;
return seq_open(file, &aa_sfs_profiles_op);
* @perms: permission table indexed by the matched state accept entry of @dfa
* @trans: transition table for indexed by named x transitions
*
- * File permission are determined by matching a path against @dfa and then
+ * File permission are determined by matching a path against @dfa and
* then using the value of the accept entry for the matching state as
* an index into @perms. If a named exec transition is required it is
* looked up in the transition table.
#define __labelset_for_each(LS, N) \
for ((N) = rb_first(&(LS)->root); (N); (N) = rb_next(N))
-void aa_labelset_destroy(struct aa_labelset *ls);
-void aa_labelset_init(struct aa_labelset *ls);
-
-
enum label_flags {
FLAG_HAT = 1, /* profile is a hat */
FLAG_UNCONFINED = 2, /* label unconfined only if all */
#define __label_make_stale(X) ((X)->flags |= FLAG_STALE)
#define labels_ns(X) (vec_ns(&((X)->vec[0]), (X)->size))
#define labels_set(X) (&labels_ns(X)->labels)
+#define labels_view(X) labels_ns(X)
#define labels_profile(X) ((X)->vec[(X)->size - 1])
#define AA_WARN(X) WARN((X), "APPARMOR WARN %s: %s\n", __func__, #X)
-#define AA_BUG(X, args...) AA_BUG_FMT((X), "" args)
+#define AA_BUG(X, args...) \
+ do { \
+ _Pragma("GCC diagnostic ignored \"-Wformat-zero-length\""); \
+ AA_BUG_FMT((X), "" args); \
+ _Pragma("GCC diagnostic warning \"-Wformat-zero-length\""); \
+ } while (0)
#ifdef CONFIG_SECURITY_APPARMOR_DEBUG_ASSERTS
#define AA_BUG_FMT(X, fmt, args...) \
WARN((X), "AppArmor WARN %s: (" #X "): " fmt, __func__, ##args)
#else
-#define AA_BUG_FMT(X, fmt, args...)
+#define AA_BUG_FMT(X, fmt, args...) no_printk(fmt, ##args)
#endif
#define AA_ERROR(fmt, args...) \
return profile->audit;
}
-bool policy_view_capable(struct aa_ns *ns);
-bool policy_admin_capable(struct aa_ns *ns);
+bool aa_policy_view_capable(struct aa_label *label, struct aa_ns *ns);
+bool aa_policy_admin_capable(struct aa_label *label, struct aa_ns *ns);
int aa_may_manage_policy(struct aa_label *label, struct aa_ns *ns,
u32 mask);
+bool aa_current_policy_view_capable(struct aa_ns *ns);
+bool aa_current_policy_admin_capable(struct aa_ns *ns);
#endif /* __AA_POLICY_H */
AA_BUG(size < 1);
/* + 1 for null terminator entry on vec */
- new = kzalloc(sizeof(*new) + sizeof(struct aa_profile *) * (size + 1),
- gfp);
+ new = kzalloc(struct_size(new, vec, size + 1), gfp);
AA_DEBUG("%s (%p)\n", __func__, new);
if (!new)
goto fail;
if (label->hname || labels_ns(label) != ns)
return res;
- if (aa_label_acntsxprint(&name, ns, label, FLAGS_NONE, gfp) == -1)
+ if (aa_label_acntsxprint(&name, ns, label, FLAGS_NONE, gfp) < 0)
return res;
ls = labels_set(label);
/**
* aa_label_acntsxprint - allocate a __counted string buffer and print label
- * @strp: buffer to write to. (MAY BE NULL if @size == 0)
+ * @strp: buffer to write to.
* @ns: namespace profile is being viewed from
* @label: label to view (NOT NULL)
* @flags: flags controlling what label info is printed
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_admin_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_admin_capable(NULL))
return -EPERM;
return param_set_bool(val, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return param_get_bool(buffer, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_admin_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_admin_capable(NULL))
return -EPERM;
return param_set_bool(val, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return param_get_bool(buffer, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return param_get_uint(buffer, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return param_get_int(buffer, kp);
}
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return sprintf(buffer, "%s", audit_mode_names[aa_g_audit]);
}
return -EINVAL;
if (!val)
return -EINVAL;
- if (apparmor_initialized && !policy_admin_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_admin_capable(NULL))
return -EPERM;
i = match_string(audit_mode_names, AUDIT_MAX_INDEX, val);
{
if (!apparmor_enabled)
return -EINVAL;
- if (apparmor_initialized && !policy_view_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_view_capable(NULL))
return -EPERM;
return sprintf(buffer, "%s", aa_profile_mode_names[aa_g_profile_mode]);
return -EINVAL;
if (!val)
return -EINVAL;
- if (apparmor_initialized && !policy_admin_capable(NULL))
+ if (apparmor_initialized && !aa_current_policy_admin_capable(NULL))
return -EPERM;
i = match_string(aa_profile_mode_names, APPARMOR_MODE_NAMES_MAX_INDEX,
static int apparmor_dointvec(struct ctl_table *table, int write,
void *buffer, size_t *lenp, loff_t *ppos)
{
- if (!policy_admin_capable(NULL))
+ if (!aa_current_policy_admin_capable(NULL))
return -EPERM;
if (!apparmor_enabled)
return -EINVAL;
}
-static unsigned int apparmor_ipv4_postroute(void *priv,
- struct sk_buff *skb,
- const struct nf_hook_state *state)
-{
- return apparmor_ip_postroute(priv, skb, state);
-}
-
-#if IS_ENABLED(CONFIG_IPV6)
-static unsigned int apparmor_ipv6_postroute(void *priv,
- struct sk_buff *skb,
- const struct nf_hook_state *state)
-{
- return apparmor_ip_postroute(priv, skb, state);
-}
-#endif
-
static const struct nf_hook_ops apparmor_nf_ops[] = {
{
- .hook = apparmor_ipv4_postroute,
+ .hook = apparmor_ip_postroute,
.pf = NFPROTO_IPV4,
.hooknum = NF_INET_POST_ROUTING,
.priority = NF_IP_PRI_SELINUX_FIRST,
},
#if IS_ENABLED(CONFIG_IPV6)
{
- .hook = apparmor_ipv6_postroute,
+ .hook = apparmor_ip_postroute,
.pf = NFPROTO_IPV6,
.hooknum = NF_INET_POST_ROUTING,
.priority = NF_IP6_PRI_SELINUX_FIRST,
*
* Returns: %0 else error code if path lookup fails
* When no error the path name is returned in @name which points to
- * to a position in @buf
+ * a position in @buf
*/
static int d_namespace_path(const struct path *path, char *buf, char **name,
int flags, const char *disconnected)
struct aa_profile *profile;
/* freed by free_profile - usually through aa_put_profile */
- profile = kzalloc(sizeof(*profile) + sizeof(struct aa_profile *) * 2,
- gfp);
+ profile = kzalloc(struct_size(profile, label.vec, 2), gfp);
if (!profile)
return NULL;
return error;
}
+/* don't call out to other LSMs in the stack for apparmor policy admin
+ * permissions
+ */
+static int policy_ns_capable(struct aa_label *label,
+ struct user_namespace *userns, int cap)
+{
+ int err;
+
+ /* check for MAC_ADMIN cap in cred */
+ err = cap_capable(current_cred(), userns, cap, CAP_OPT_NONE);
+ if (!err)
+ err = aa_capable(label, cap, CAP_OPT_NONE);
+
+ return err;
+}
+
/**
- * policy_view_capable - check if viewing policy in at @ns is allowed
- * ns: namespace being viewed by current task (may be NULL)
+ * aa_policy_view_capable - check if viewing policy in at @ns is allowed
+ * label: label that is trying to view policy in ns
+ * ns: namespace being viewed by @label (may be NULL if @label's ns)
* Returns: true if viewing policy is allowed
*
* If @ns is NULL then the namespace being viewed is assumed to be the
* tasks current namespace.
*/
-bool policy_view_capable(struct aa_ns *ns)
+bool aa_policy_view_capable(struct aa_label *label, struct aa_ns *ns)
{
struct user_namespace *user_ns = current_user_ns();
- struct aa_ns *view_ns = aa_get_current_ns();
+ struct aa_ns *view_ns = labels_view(label);
bool root_in_user_ns = uid_eq(current_euid(), make_kuid(user_ns, 0)) ||
in_egroup_p(make_kgid(user_ns, 0));
bool response = false;
(unprivileged_userns_apparmor_policy != 0 &&
user_ns->level == view_ns->level)))
response = true;
- aa_put_ns(view_ns);
return response;
}
-bool policy_admin_capable(struct aa_ns *ns)
+bool aa_policy_admin_capable(struct aa_label *label, struct aa_ns *ns)
{
struct user_namespace *user_ns = current_user_ns();
- bool capable = ns_capable(user_ns, CAP_MAC_ADMIN);
+ bool capable = policy_ns_capable(label, user_ns, CAP_MAC_ADMIN) == 0;
AA_DEBUG("cap_mac_admin? %d\n", capable);
AA_DEBUG("policy locked? %d\n", aa_g_lock_policy);
- return policy_view_capable(ns) && capable && !aa_g_lock_policy;
+ return aa_policy_view_capable(label, ns) && capable &&
+ !aa_g_lock_policy;
+}
+
+bool aa_current_policy_view_capable(struct aa_ns *ns)
+{
+ struct aa_label *label;
+ bool res;
+
+ label = __begin_current_label_crit_section();
+ res = aa_policy_view_capable(label, ns);
+ __end_current_label_crit_section(label);
+
+ return res;
+}
+
+bool aa_current_policy_admin_capable(struct aa_ns *ns)
+{
+ struct aa_label *label;
+ bool res;
+
+ label = __begin_current_label_crit_section();
+ res = aa_policy_admin_capable(label, ns);
+ __end_current_label_crit_section(label);
+
+ return res;
}
/**
return audit_policy(label, op, NULL, NULL, "policy_locked",
-EACCES);
- if (!policy_admin_capable(ns))
+ if (!aa_policy_admin_capable(label, ns))
return audit_policy(label, op, NULL, NULL, "not policy admin",
-EACCES);
/*
* The AppArmor interface treats data as a type byte followed by the
- * actual data. The interface has the notion of a a named entry
+ * actual data. The interface has the notion of a named entry
* which has a name (AA_NAME typecode followed by name string) followed by
* the entries typecode and data. Named types allow for optional
* elements and extensions to be added and tested for without breaking
* @profile: the profile to print profile info about (NOT NULL)
* @string: Returns - string containing the profile info (NOT NULL)
*
- * Returns: length of @string on success else error on failure
- *
* Requires: profile != NULL
*
* Creates a string containing the namespace_name://profile_name for
}
EXPORT_SYMBOL(security_tun_dev_open);
-int security_sctp_assoc_request(struct sctp_endpoint *ep, struct sk_buff *skb)
+int security_sctp_assoc_request(struct sctp_association *asoc, struct sk_buff *skb)
{
- return call_int_hook(sctp_assoc_request, 0, ep, skb);
+ return call_int_hook(sctp_assoc_request, 0, asoc, skb);
}
EXPORT_SYMBOL(security_sctp_assoc_request);
}
EXPORT_SYMBOL(security_sctp_bind_connect);
-void security_sctp_sk_clone(struct sctp_endpoint *ep, struct sock *sk,
+void security_sctp_sk_clone(struct sctp_association *asoc, struct sock *sk,
struct sock *newsk)
{
- call_void_hook(sctp_sk_clone, ep, sk, newsk);
+ call_void_hook(sctp_sk_clone, asoc, sk, newsk);
}
EXPORT_SYMBOL(security_sctp_sk_clone);
* connect(2), sctp_connectx(3) or sctp_sendmsg(3) (with no association
* already present).
*/
-static int selinux_sctp_assoc_request(struct sctp_endpoint *ep,
+static int selinux_sctp_assoc_request(struct sctp_association *asoc,
struct sk_buff *skb)
{
- struct sk_security_struct *sksec = ep->base.sk->sk_security;
+ struct sk_security_struct *sksec = asoc->base.sk->sk_security;
struct common_audit_data ad;
struct lsm_network_audit net = {0,};
u8 peerlbl_active;
/* This will return peer_sid = SECSID_NULL if there are
* no peer labels, see security_net_peersid_resolve().
*/
- err = selinux_skb_peerlbl_sid(skb, ep->base.sk->sk_family,
+ err = selinux_skb_peerlbl_sid(skb, asoc->base.sk->sk_family,
&peer_sid);
if (err)
return err;
*/
ad.type = LSM_AUDIT_DATA_NET;
ad.u.net = &net;
- ad.u.net->sk = ep->base.sk;
+ ad.u.net->sk = asoc->base.sk;
err = avc_has_perm(&selinux_state,
sksec->peer_sid, peer_sid, sksec->sclass,
SCTP_SOCKET__ASSOCIATION, &ad);
}
/* Compute the MLS component for the connection and store
- * the information in ep. This will be used by SCTP TCP type
+ * the information in asoc. This will be used by SCTP TCP type
* sockets and peeled off connections as they cause a new
* socket to be generated. selinux_sctp_sk_clone() will then
* plug this into the new socket.
if (err)
return err;
- ep->secid = conn_sid;
- ep->peer_secid = peer_sid;
+ asoc->secid = conn_sid;
+ asoc->peer_secid = peer_sid;
/* Set any NetLabel labels including CIPSO/CALIPSO options. */
- return selinux_netlbl_sctp_assoc_request(ep, skb);
+ return selinux_netlbl_sctp_assoc_request(asoc, skb);
}
/* Check if sctp IPv4/IPv6 addresses are valid for binding or connecting
}
/* Called whenever a new socket is created by accept(2) or sctp_peeloff(3). */
-static void selinux_sctp_sk_clone(struct sctp_endpoint *ep, struct sock *sk,
+static void selinux_sctp_sk_clone(struct sctp_association *asoc, struct sock *sk,
struct sock *newsk)
{
struct sk_security_struct *sksec = sk->sk_security;
if (!selinux_policycap_extsockclass())
return selinux_sk_clone_security(sk, newsk);
- newsksec->sid = ep->secid;
- newsksec->peer_sid = ep->peer_secid;
+ newsksec->sid = asoc->secid;
+ newsksec->peer_sid = asoc->peer_secid;
newsksec->sclass = sksec->sclass;
selinux_netlbl_sctp_sk_clone(sk, newsk);
}
int selinux_netlbl_skbuff_setsid(struct sk_buff *skb,
u16 family,
u32 sid);
-int selinux_netlbl_sctp_assoc_request(struct sctp_endpoint *ep,
+int selinux_netlbl_sctp_assoc_request(struct sctp_association *asoc,
struct sk_buff *skb);
int selinux_netlbl_inet_conn_request(struct request_sock *req, u16 family);
void selinux_netlbl_inet_csk_clone(struct sock *sk, u16 family);
return 0;
}
-static inline int selinux_netlbl_sctp_assoc_request(struct sctp_endpoint *ep,
+static inline int selinux_netlbl_sctp_assoc_request(struct sctp_association *asoc,
struct sk_buff *skb)
{
return 0;
/**
* selinux_netlbl_sctp_assoc_request - Label an incoming sctp association.
- * @ep: incoming association endpoint.
+ * @asoc: incoming association.
* @skb: the packet.
*
* Description:
- * A new incoming connection is represented by @ep, ......
+ * A new incoming connection is represented by @asoc, ......
* Returns zero on success, negative values on failure.
*
*/
-int selinux_netlbl_sctp_assoc_request(struct sctp_endpoint *ep,
+int selinux_netlbl_sctp_assoc_request(struct sctp_association *asoc,
struct sk_buff *skb)
{
int rc;
struct netlbl_lsm_secattr secattr;
- struct sk_security_struct *sksec = ep->base.sk->sk_security;
+ struct sk_security_struct *sksec = asoc->base.sk->sk_security;
struct sockaddr_in addr4;
struct sockaddr_in6 addr6;
- if (ep->base.sk->sk_family != PF_INET &&
- ep->base.sk->sk_family != PF_INET6)
+ if (asoc->base.sk->sk_family != PF_INET &&
+ asoc->base.sk->sk_family != PF_INET6)
return 0;
netlbl_secattr_init(&secattr);
rc = security_netlbl_sid_to_secattr(&selinux_state,
- ep->secid, &secattr);
+ asoc->secid, &secattr);
if (rc != 0)
goto assoc_request_return;
if (ip_hdr(skb)->version == 4) {
addr4.sin_family = AF_INET;
addr4.sin_addr.s_addr = ip_hdr(skb)->saddr;
- rc = netlbl_conn_setattr(ep->base.sk, (void *)&addr4, &secattr);
+ rc = netlbl_conn_setattr(asoc->base.sk, (void *)&addr4, &secattr);
} else if (IS_ENABLED(CONFIG_IPV6) && ip_hdr(skb)->version == 6) {
addr6.sin6_family = AF_INET6;
addr6.sin6_addr = ipv6_hdr(skb)->saddr;
- rc = netlbl_conn_setattr(ep->base.sk, (void *)&addr6, &secattr);
+ rc = netlbl_conn_setattr(asoc->base.sk, (void *)&addr6, &secattr);
} else {
rc = -EAFNOSUPPORT;
}
int hashtab_init(struct hashtab *h, u32 nel_hint)
{
- h->size = hashtab_compute_size(nel_hint);
+ u32 size = hashtab_compute_size(nel_hint);
+
+ /* should already be zeroed, but better be safe */
h->nel = 0;
- if (!h->size)
- return 0;
+ h->size = 0;
+ h->htable = NULL;
- h->htable = kcalloc(h->size, sizeof(*h->htable), GFP_KERNEL);
- return h->htable ? 0 : -ENOMEM;
+ if (size) {
+ h->htable = kcalloc(size, sizeof(*h->htable), GFP_KERNEL);
+ if (!h->htable)
+ return -ENOMEM;
+ h->size = size;
+ }
+ return 0;
}
int __hashtab_insert(struct hashtab *h, struct hashtab_node **dst,
snd-pcm-y := pcm.o pcm_native.o pcm_lib.o pcm_misc.o \
pcm_memory.o memalloc.o
snd-pcm-$(CONFIG_SND_PCM_TIMER) += pcm_timer.o
+snd-pcm-$(CONFIG_SND_DMA_SGBUF) += sgbuf.o
snd-pcm-$(CONFIG_SND_PCM_ELD) += pcm_drm_eld.o
snd-pcm-$(CONFIG_SND_PCM_IEC958) += pcm_iec958.o
int snd_dma_buffer_mmap(struct snd_dma_buffer *dmab,
struct vm_area_struct *area)
{
- const struct snd_malloc_ops *ops = snd_dma_get_ops(dmab);
+ const struct snd_malloc_ops *ops;
+ if (!dmab)
+ return -ENOENT;
+ ops = snd_dma_get_ops(dmab);
if (ops && ops->mmap)
return ops->mmap(dmab, area);
else
}
}
-static const struct snd_malloc_ops snd_dma_noncontig_ops = {
- .alloc = snd_dma_noncontig_alloc,
- .free = snd_dma_noncontig_free,
- .mmap = snd_dma_noncontig_mmap,
- .sync = snd_dma_noncontig_sync,
- /* re-use vmalloc helpers for get_* ops */
- .get_addr = snd_dma_vmalloc_get_addr,
- .get_page = snd_dma_vmalloc_get_page,
- .get_chunk_size = snd_dma_vmalloc_get_chunk_size,
-};
+static inline void snd_dma_noncontig_iter_set(struct snd_dma_buffer *dmab,
+ struct sg_page_iter *piter,
+ size_t offset)
+{
+ struct sg_table *sgt = dmab->private_data;
-/* x86-specific SG-buffer with WC pages */
-#ifdef CONFIG_SND_DMA_SGBUF
-#define vmalloc_to_virt(v) (unsigned long)page_to_virt(vmalloc_to_page(v))
+ __sg_page_iter_start(piter, sgt->sgl, sgt->orig_nents,
+ offset >> PAGE_SHIFT);
+}
-static void *snd_dma_sg_wc_alloc(struct snd_dma_buffer *dmab, size_t size)
+static dma_addr_t snd_dma_noncontig_get_addr(struct snd_dma_buffer *dmab,
+ size_t offset)
{
- void *p = snd_dma_noncontig_alloc(dmab, size);
- size_t ofs;
+ struct sg_dma_page_iter iter;
- if (!p)
- return NULL;
- for (ofs = 0; ofs < size; ofs += PAGE_SIZE)
- set_memory_uc(vmalloc_to_virt(p + ofs), 1);
- return p;
+ snd_dma_noncontig_iter_set(dmab, &iter.base, offset);
+ __sg_page_iter_dma_next(&iter);
+ return sg_page_iter_dma_address(&iter) + offset % PAGE_SIZE;
}
-static void snd_dma_sg_wc_free(struct snd_dma_buffer *dmab)
+static struct page *snd_dma_noncontig_get_page(struct snd_dma_buffer *dmab,
+ size_t offset)
{
- size_t ofs;
+ struct sg_page_iter iter;
- for (ofs = 0; ofs < dmab->bytes; ofs += PAGE_SIZE)
- set_memory_wb(vmalloc_to_virt(dmab->area + ofs), 1);
- snd_dma_noncontig_free(dmab);
+ snd_dma_noncontig_iter_set(dmab, &iter, offset);
+ __sg_page_iter_next(&iter);
+ return sg_page_iter_page(&iter);
}
-static int snd_dma_sg_wc_mmap(struct snd_dma_buffer *dmab,
- struct vm_area_struct *area)
+static unsigned int
+snd_dma_noncontig_get_chunk_size(struct snd_dma_buffer *dmab,
+ unsigned int ofs, unsigned int size)
{
- area->vm_page_prot = pgprot_writecombine(area->vm_page_prot);
- /* FIXME: dma_mmap_noncontiguous() works? */
- return -ENOENT; /* continue with the default mmap handler */
+ struct sg_dma_page_iter iter;
+ unsigned int start, end;
+ unsigned long addr;
+
+ start = ALIGN_DOWN(ofs, PAGE_SIZE);
+ end = ofs + size - 1; /* the last byte address */
+ snd_dma_noncontig_iter_set(dmab, &iter.base, start);
+ if (!__sg_page_iter_dma_next(&iter))
+ return 0;
+ /* check page continuity */
+ addr = sg_page_iter_dma_address(&iter);
+ for (;;) {
+ start += PAGE_SIZE;
+ if (start > end)
+ break;
+ addr += PAGE_SIZE;
+ if (!__sg_page_iter_dma_next(&iter) ||
+ sg_page_iter_dma_address(&iter) != addr)
+ return start - ofs;
+ }
+ /* ok, all on continuous pages */
+ return size;
}
-const struct snd_malloc_ops snd_dma_sg_wc_ops = {
- .alloc = snd_dma_sg_wc_alloc,
- .free = snd_dma_sg_wc_free,
- .mmap = snd_dma_sg_wc_mmap,
+static const struct snd_malloc_ops snd_dma_noncontig_ops = {
+ .alloc = snd_dma_noncontig_alloc,
+ .free = snd_dma_noncontig_free,
+ .mmap = snd_dma_noncontig_mmap,
.sync = snd_dma_noncontig_sync,
- .get_addr = snd_dma_vmalloc_get_addr,
- .get_page = snd_dma_vmalloc_get_page,
- .get_chunk_size = snd_dma_vmalloc_get_chunk_size,
+ .get_addr = snd_dma_noncontig_get_addr,
+ .get_page = snd_dma_noncontig_get_page,
+ .get_chunk_size = snd_dma_noncontig_get_chunk_size,
};
-#endif /* CONFIG_SND_DMA_SGBUF */
/*
* Non-coherent pages allocator
[SNDRV_DMA_TYPE_DEV_WC] = &snd_dma_wc_ops,
[SNDRV_DMA_TYPE_NONCONTIG] = &snd_dma_noncontig_ops,
[SNDRV_DMA_TYPE_NONCOHERENT] = &snd_dma_noncoherent_ops,
-#ifdef CONFIG_SND_DMA_SGBUF
- [SNDRV_DMA_TYPE_DEV_WC_SG] = &snd_dma_sg_wc_ops,
-#endif
#ifdef CONFIG_GENERIC_ALLOCATOR
[SNDRV_DMA_TYPE_DEV_IRAM] = &snd_dma_iram_ops,
#endif /* CONFIG_GENERIC_ALLOCATOR */
#endif /* CONFIG_HAS_DMA */
+#ifdef CONFIG_SND_DMA_SGBUF
+ [SNDRV_DMA_TYPE_DEV_SG] = &snd_dma_sg_ops,
+ [SNDRV_DMA_TYPE_DEV_WC_SG] = &snd_dma_sg_ops,
+#endif
};
static const struct snd_malloc_ops *snd_dma_get_ops(struct snd_dma_buffer *dmab)
{
+ if (WARN_ON_ONCE(!dmab))
+ return NULL;
if (WARN_ON_ONCE(dmab->dev.type <= SNDRV_DMA_TYPE_UNKNOWN ||
dmab->dev.type >= ARRAY_SIZE(dma_ops)))
return NULL;
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Scatter-Gather buffer
+ *
+ * Copyright (c) by Takashi Iwai <tiwai@suse.de>
+ */
+
+#include <linux/slab.h>
+#include <linux/mm.h>
+#include <linux/vmalloc.h>
+#include <linux/export.h>
+#include <sound/memalloc.h>
+#include "memalloc_local.h"
+
+struct snd_sg_page {
+ void *buf;
+ dma_addr_t addr;
+};
+
+struct snd_sg_buf {
+ int size; /* allocated byte size */
+ int pages; /* allocated pages */
+ int tblsize; /* allocated table size */
+ struct snd_sg_page *table; /* address table */
+ struct page **page_table; /* page table (for vmap/vunmap) */
+ struct device *dev;
+};
+
+/* table entries are align to 32 */
+#define SGBUF_TBL_ALIGN 32
+#define sgbuf_align_table(tbl) ALIGN((tbl), SGBUF_TBL_ALIGN)
+
+static void snd_dma_sg_free(struct snd_dma_buffer *dmab)
+{
+ struct snd_sg_buf *sgbuf = dmab->private_data;
+ struct snd_dma_buffer tmpb;
+ int i;
+
+ if (!sgbuf)
+ return;
+
+ vunmap(dmab->area);
+ dmab->area = NULL;
+
+ tmpb.dev.type = SNDRV_DMA_TYPE_DEV;
+ if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_WC_SG)
+ tmpb.dev.type = SNDRV_DMA_TYPE_DEV_WC;
+ tmpb.dev.dev = sgbuf->dev;
+ for (i = 0; i < sgbuf->pages; i++) {
+ if (!(sgbuf->table[i].addr & ~PAGE_MASK))
+ continue; /* continuous pages */
+ tmpb.area = sgbuf->table[i].buf;
+ tmpb.addr = sgbuf->table[i].addr & PAGE_MASK;
+ tmpb.bytes = (sgbuf->table[i].addr & ~PAGE_MASK) << PAGE_SHIFT;
+ snd_dma_free_pages(&tmpb);
+ }
+
+ kfree(sgbuf->table);
+ kfree(sgbuf->page_table);
+ kfree(sgbuf);
+ dmab->private_data = NULL;
+}
+
+#define MAX_ALLOC_PAGES 32
+
+static void *snd_dma_sg_alloc(struct snd_dma_buffer *dmab, size_t size)
+{
+ struct snd_sg_buf *sgbuf;
+ unsigned int i, pages, chunk, maxpages;
+ struct snd_dma_buffer tmpb;
+ struct snd_sg_page *table;
+ struct page **pgtable;
+ int type = SNDRV_DMA_TYPE_DEV;
+ pgprot_t prot = PAGE_KERNEL;
+ void *area;
+
+ dmab->private_data = sgbuf = kzalloc(sizeof(*sgbuf), GFP_KERNEL);
+ if (!sgbuf)
+ return NULL;
+ if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_WC_SG) {
+ type = SNDRV_DMA_TYPE_DEV_WC;
+#ifdef pgprot_noncached
+ prot = pgprot_noncached(PAGE_KERNEL);
+#endif
+ }
+ sgbuf->dev = dmab->dev.dev;
+ pages = snd_sgbuf_aligned_pages(size);
+ sgbuf->tblsize = sgbuf_align_table(pages);
+ table = kcalloc(sgbuf->tblsize, sizeof(*table), GFP_KERNEL);
+ if (!table)
+ goto _failed;
+ sgbuf->table = table;
+ pgtable = kcalloc(sgbuf->tblsize, sizeof(*pgtable), GFP_KERNEL);
+ if (!pgtable)
+ goto _failed;
+ sgbuf->page_table = pgtable;
+
+ /* allocate pages */
+ maxpages = MAX_ALLOC_PAGES;
+ while (pages > 0) {
+ chunk = pages;
+ /* don't be too eager to take a huge chunk */
+ if (chunk > maxpages)
+ chunk = maxpages;
+ chunk <<= PAGE_SHIFT;
+ if (snd_dma_alloc_pages_fallback(type, dmab->dev.dev,
+ chunk, &tmpb) < 0) {
+ if (!sgbuf->pages)
+ goto _failed;
+ size = sgbuf->pages * PAGE_SIZE;
+ break;
+ }
+ chunk = tmpb.bytes >> PAGE_SHIFT;
+ for (i = 0; i < chunk; i++) {
+ table->buf = tmpb.area;
+ table->addr = tmpb.addr;
+ if (!i)
+ table->addr |= chunk; /* mark head */
+ table++;
+ *pgtable++ = virt_to_page(tmpb.area);
+ tmpb.area += PAGE_SIZE;
+ tmpb.addr += PAGE_SIZE;
+ }
+ sgbuf->pages += chunk;
+ pages -= chunk;
+ if (chunk < maxpages)
+ maxpages = chunk;
+ }
+
+ sgbuf->size = size;
+ area = vmap(sgbuf->page_table, sgbuf->pages, VM_MAP, prot);
+ if (!area)
+ goto _failed;
+ return area;
+
+ _failed:
+ snd_dma_sg_free(dmab); /* free the table */
+ return NULL;
+}
+
+static dma_addr_t snd_dma_sg_get_addr(struct snd_dma_buffer *dmab,
+ size_t offset)
+{
+ struct snd_sg_buf *sgbuf = dmab->private_data;
+ dma_addr_t addr;
+
+ addr = sgbuf->table[offset >> PAGE_SHIFT].addr;
+ addr &= ~((dma_addr_t)PAGE_SIZE - 1);
+ return addr + offset % PAGE_SIZE;
+}
+
+static struct page *snd_dma_sg_get_page(struct snd_dma_buffer *dmab,
+ size_t offset)
+{
+ struct snd_sg_buf *sgbuf = dmab->private_data;
+ unsigned int idx = offset >> PAGE_SHIFT;
+
+ if (idx >= (unsigned int)sgbuf->pages)
+ return NULL;
+ return sgbuf->page_table[idx];
+}
+
+static unsigned int snd_dma_sg_get_chunk_size(struct snd_dma_buffer *dmab,
+ unsigned int ofs,
+ unsigned int size)
+{
+ struct snd_sg_buf *sg = dmab->private_data;
+ unsigned int start, end, pg;
+
+ start = ofs >> PAGE_SHIFT;
+ end = (ofs + size - 1) >> PAGE_SHIFT;
+ /* check page continuity */
+ pg = sg->table[start].addr >> PAGE_SHIFT;
+ for (;;) {
+ start++;
+ if (start > end)
+ break;
+ pg++;
+ if ((sg->table[start].addr >> PAGE_SHIFT) != pg)
+ return (start << PAGE_SHIFT) - ofs;
+ }
+ /* ok, all on continuous pages */
+ return size;
+}
+
+static int snd_dma_sg_mmap(struct snd_dma_buffer *dmab,
+ struct vm_area_struct *area)
+{
+ if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_WC_SG)
+ area->vm_page_prot = pgprot_writecombine(area->vm_page_prot);
+ return -ENOENT; /* continue with the default mmap handler */
+}
+
+const struct snd_malloc_ops snd_dma_sg_ops = {
+ .alloc = snd_dma_sg_alloc,
+ .free = snd_dma_sg_free,
+ .get_addr = snd_dma_sg_get_addr,
+ .get_page = snd_dma_sg_get_page,
+ .get_chunk_size = snd_dma_sg_get_chunk_size,
+ .mmap = snd_dma_sg_mmap,
+};
if (!timer)
return -EINVAL;
spin_lock_irqsave(&timer->lock, flags);
+ list_del_init(&timeri->ack_list);
+ list_del_init(&timeri->active_list);
if (!(timeri->flags & (SNDRV_TIMER_IFLG_RUNNING |
SNDRV_TIMER_IFLG_START))) {
result = -EBUSY;
goto unlock;
}
- list_del_init(&timeri->ack_list);
- list_del_init(&timeri->active_list);
if (timer->card && timer->card->shutdown)
goto unlock;
if (stop) {
static int snd_timer_stop_slave(struct snd_timer_instance *timeri, bool stop)
{
unsigned long flags;
+ bool running;
spin_lock_irqsave(&slave_active_lock, flags);
- if (!(timeri->flags & SNDRV_TIMER_IFLG_RUNNING)) {
- spin_unlock_irqrestore(&slave_active_lock, flags);
- return -EBUSY;
- }
+ running = timeri->flags & SNDRV_TIMER_IFLG_RUNNING;
timeri->flags &= ~SNDRV_TIMER_IFLG_RUNNING;
if (timeri->timer) {
spin_lock(&timeri->timer->lock);
list_del_init(&timeri->ack_list);
list_del_init(&timeri->active_list);
- snd_timer_notify1(timeri, stop ? SNDRV_TIMER_EVENT_STOP :
- SNDRV_TIMER_EVENT_PAUSE);
+ if (running)
+ snd_timer_notify1(timeri, stop ? SNDRV_TIMER_EVENT_STOP :
+ SNDRV_TIMER_EVENT_PAUSE);
spin_unlock(&timeri->timer->lock);
}
spin_unlock_irqrestore(&slave_active_lock, flags);
- return 0;
+ return running ? 0 : -EBUSY;
}
/*
* 828
* 896
* 828mk2
+ * 896hd
* Traveler
* Ultralite
* 8pre
* 828mk3 (Hybrid)
* Ultralite mk3 (FireWire only)
* Ultralite mk3 (Hybrid)
+ * Traveler mk3
* Audio Express
+ * Track 16
* 4pre
To compile this driver as a module, choose M here: the module
efw->firmware_version == 0x5070300 ||
efw->firmware_version == 0x5080000))
efw->tx_stream.flags |= CIP_UNALIGHED_DBC;
- // AudioFire9 always reports wrong dbs.
- if (efw->is_af9)
+ // AudioFire9 always reports wrong dbs. Onyx 1200F with the latest firmware (v4.6.0)
+ // also report wrong dbs at 88.2 kHz or greater.
+ if (efw->is_af9 || efw->firmware_version == 0x4060000)
efw->tx_stream.flags |= CIP_WRONG_DBS;
// Firmware version 5.5 reports fixed interval for dbc.
if (efw->firmware_version == 0x5050000)
#define V3_CLOCK_SRC_INTERNAL 0x00
#define V3_CLOCK_SRC_WORD_ON_BNC 0x01
#define V3_CLOCK_SRC_SPH 0x02
+#define V3_CLOCK_SRC_AESEBU_ON_XLR 0x08
#define V3_CLOCK_SRC_SPDIF_ON_COAX 0x10
#define V3_CLOCK_SRC_OPT_IFACE_A 0x18
#define V3_CLOCK_SRC_OPT_IFACE_B 0x19
case V3_CLOCK_SRC_SPH:
*src = SND_MOTU_CLOCK_SOURCE_SPH;
break;
+ case V3_CLOCK_SRC_AESEBU_ON_XLR:
+ *src = SND_MOTU_CLOCK_SOURCE_AESEBU_ON_XLR;
+ break;
case V3_CLOCK_SRC_SPDIF_ON_COAX:
*src = SND_MOTU_CLOCK_SOURCE_SPDIF_ON_COAX;
break;
sizeof(reg));
}
-static int detect_packet_formats_828mk3(struct snd_motu *motu, u32 data)
+static int detect_packet_formats_with_opt_ifaces(struct snd_motu *motu, u32 data)
{
if (data & V3_ENABLE_OPT_IN_IFACE_A) {
if (data & V3_NO_ADAT_OPT_IN_IFACE_A) {
motu->spec->rx_fixed_pcm_chunks,
sizeof(motu->rx_packet_formats.pcm_chunks));
- if (motu->spec == &snd_motu_spec_828mk3_fw || motu->spec == &snd_motu_spec_828mk3_hybrid)
- return detect_packet_formats_828mk3(motu, data);
+ if (motu->spec == &snd_motu_spec_828mk3_fw ||
+ motu->spec == &snd_motu_spec_828mk3_hybrid ||
+ motu->spec == &snd_motu_spec_traveler_mk3 ||
+ motu->spec == &snd_motu_spec_track16)
+ return detect_packet_formats_with_opt_ifaces(motu, data);
else
return 0;
}
.rx_fixed_pcm_chunks = {14, 14, 14}, // Additional 4 dummy chunks at higher rate.
};
+const struct snd_motu_spec snd_motu_spec_traveler_mk3 = {
+ .name = "TravelerMk3",
+ .protocol_version = SND_MOTU_PROTOCOL_V3,
+ .flags = SND_MOTU_SPEC_RX_MIDI_3RD_Q |
+ SND_MOTU_SPEC_TX_MIDI_3RD_Q |
+ SND_MOTU_SPEC_COMMAND_DSP,
+ .tx_fixed_pcm_chunks = {18, 14, 10},
+ .rx_fixed_pcm_chunks = {14, 14, 10},
+};
+
const struct snd_motu_spec snd_motu_spec_ultralite_mk3 = {
.name = "UltraLiteMk3",
.protocol_version = SND_MOTU_PROTOCOL_V3,
.rx_fixed_pcm_chunks = {10, 10, 0},
};
+const struct snd_motu_spec snd_motu_spec_track16 = {
+ .name = "Track16",
+ .protocol_version = SND_MOTU_PROTOCOL_V3,
+ .flags = SND_MOTU_SPEC_RX_MIDI_3RD_Q |
+ SND_MOTU_SPEC_TX_MIDI_3RD_Q |
+ SND_MOTU_SPEC_COMMAND_DSP,
+ .tx_fixed_pcm_chunks = {14, 14, 14},
+ .rx_fixed_pcm_chunks = {6, 6, 6},
+};
+
const struct snd_motu_spec snd_motu_spec_4pre = {
.name = "4pre",
.protocol_version = SND_MOTU_PROTOCOL_V3,
SND_MOTU_DEV_ENTRY(0x00000f, &snd_motu_spec_8pre),
SND_MOTU_DEV_ENTRY(0x000015, &snd_motu_spec_828mk3_fw), // FireWire only.
SND_MOTU_DEV_ENTRY(0x000019, &snd_motu_spec_ultralite_mk3), // FireWire only.
+ SND_MOTU_DEV_ENTRY(0x00001b, &snd_motu_spec_traveler_mk3),
SND_MOTU_DEV_ENTRY(0x000030, &snd_motu_spec_ultralite_mk3), // Hybrid.
SND_MOTU_DEV_ENTRY(0x000035, &snd_motu_spec_828mk3_hybrid), // Hybrid.
SND_MOTU_DEV_ENTRY(0x000033, &snd_motu_spec_audio_express),
+ SND_MOTU_DEV_ENTRY(0x000039, &snd_motu_spec_track16),
SND_MOTU_DEV_ENTRY(0x000045, &snd_motu_spec_4pre),
{ }
};
extern const struct snd_motu_spec snd_motu_spec_828mk3_fw;
extern const struct snd_motu_spec snd_motu_spec_828mk3_hybrid;
+extern const struct snd_motu_spec snd_motu_spec_traveler_mk3;
extern const struct snd_motu_spec snd_motu_spec_ultralite_mk3;
extern const struct snd_motu_spec snd_motu_spec_audio_express;
+extern const struct snd_motu_spec snd_motu_spec_track16;
extern const struct snd_motu_spec snd_motu_spec_4pre;
int amdtp_motu_init(struct amdtp_stream *s, struct fw_unit *unit,
},
#endif
+/* JasperLake */
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_JASPERLAKE)
+ {
+ .flags = FLAG_SOF,
+ .device = 0x4dc8,
+ .codec_hid = "ESSX8336",
+ },
+#endif
+
/* Tigerlake */
#if IS_ENABLED(CONFIG_SND_SOC_SOF_TIGERLAKE)
{
{
static int dev;
struct snd_card *card;
- struct cmipci *cm;
int err;
if (dev >= SNDRV_CARDS)
}
err = snd_devm_card_new(&pci->dev, index[dev], id[dev], THIS_MODULE,
- sizeof(*cm), &card);
+ sizeof(struct cmipci), &card);
if (err < 0)
return err;
- cm = card->private_data;
switch (pci->device) {
case PCI_DEVICE_ID_CMEDIA_CM8738:
#define BLANK_SLOT 4094
-static int amixer_master(struct rsc *rsc)
+static void amixer_master(struct rsc *rsc)
{
rsc->conj = 0;
- return rsc->idx = container_of(rsc, struct amixer, rsc)->idx[0];
+ rsc->idx = container_of(rsc, struct amixer, rsc)->idx[0];
}
-static int amixer_next_conj(struct rsc *rsc)
+static void amixer_next_conj(struct rsc *rsc)
{
rsc->conj++;
- return container_of(rsc, struct amixer, rsc)->idx[rsc->conj];
}
static int amixer_index(const struct rsc *rsc)
/* SUM resource management */
-static int sum_master(struct rsc *rsc)
+static void sum_master(struct rsc *rsc)
{
rsc->conj = 0;
- return rsc->idx = container_of(rsc, struct sum, rsc)->idx[0];
+ rsc->idx = container_of(rsc, struct sum, rsc)->idx[0];
}
-static int sum_next_conj(struct rsc *rsc)
+static void sum_next_conj(struct rsc *rsc)
{
rsc->conj++;
- return container_of(rsc, struct sum, rsc)->idx[rsc->conj];
}
static int sum_index(const struct rsc *rsc)
[SPDIFIO] = {.left = 0x05, .right = 0x85},
};
-static int daio_master(struct rsc *rsc)
+static void daio_master(struct rsc *rsc)
{
/* Actually, this is not the resource index of DAIO.
* For DAO, it is the input mapper index. And, for DAI,
* it is the output time-slot index. */
- return rsc->conj = rsc->idx;
+ rsc->conj = rsc->idx;
}
static int daio_index(const struct rsc *rsc)
return rsc->conj;
}
-static int daio_out_next_conj(struct rsc *rsc)
+static void daio_out_next_conj(struct rsc *rsc)
{
- return rsc->conj += 2;
+ rsc->conj += 2;
}
-static int daio_in_next_conj_20k1(struct rsc *rsc)
+static void daio_in_next_conj_20k1(struct rsc *rsc)
{
- return rsc->conj += 0x200;
+ rsc->conj += 0x200;
}
-static int daio_in_next_conj_20k2(struct rsc *rsc)
+static void daio_in_next_conj_20k2(struct rsc *rsc)
{
- return rsc->conj += 0x100;
+ rsc->conj += 0x100;
}
static const struct rsc_ops daio_out_rsc_ops = {
return (rsc->conj << 4) + offset_in_audio_slot_block[rsc->type];
}
-static int rsc_next_conj(struct rsc *rsc)
+static void rsc_next_conj(struct rsc *rsc)
{
unsigned int i;
for (i = 0; (i < 8) && (!(rsc->msr & (0x1 << i))); )
i++;
rsc->conj += (AUDIO_SLOT_BLOCK_NUM >> i);
- return rsc->conj;
}
-static int rsc_master(struct rsc *rsc)
+static void rsc_master(struct rsc *rsc)
{
- return rsc->conj = rsc->idx;
+ rsc->conj = rsc->idx;
}
static const struct rsc_ops rsc_generic_ops = {
};
struct rsc_ops {
- int (*master)(struct rsc *rsc); /* Move to master resource */
- int (*next_conj)(struct rsc *rsc); /* Move to next conjugate resource */
+ void (*master)(struct rsc *rsc); /* Move to master resource */
+ void (*next_conj)(struct rsc *rsc); /* Move to next conjugate resource */
int (*index)(const struct rsc *rsc); /* Return the index of resource */
/* Return the output slot number */
int (*output_slot)(const struct rsc *rsc);
/* SRCIMP resource manager operations */
-static int srcimp_master(struct rsc *rsc)
+static void srcimp_master(struct rsc *rsc)
{
rsc->conj = 0;
- return rsc->idx = container_of(rsc, struct srcimp, rsc)->idx[0];
+ rsc->idx = container_of(rsc, struct srcimp, rsc)->idx[0];
}
-static int srcimp_next_conj(struct rsc *rsc)
+static void srcimp_next_conj(struct rsc *rsc)
{
rsc->conj++;
- return container_of(rsc, struct srcimp, rsc)->idx[rsc->conj];
}
static int srcimp_index(const struct rsc *rsc)
out_free:
if (err < 0) {
- azx_free(chip);
+ pci_set_drvdata(pci, NULL);
+ snd_card_free(chip->card);
return err;
}
cancel_delayed_work_sync(&hda->probe_work);
device_lock(&pci->dev);
+ pci_set_drvdata(pci, NULL);
snd_card_free(card);
}
}
alc_write_coef_idx(codec, 0x45, 0x5089);
}
+static const struct coef_fw alc233_fixup_no_audio_jack_coefs[] = {
+ WRITE_COEF(0x1a, 0x9003), WRITE_COEF(0x1b, 0x0e2b), WRITE_COEF(0x37, 0xfe06),
+ WRITE_COEF(0x38, 0x4981), WRITE_COEF(0x45, 0xd489), WRITE_COEF(0x46, 0x0074),
+ WRITE_COEF(0x49, 0x0149),
+ {}
+};
+
+static void alc233_fixup_no_audio_jack(struct hda_codec *codec,
+ const struct hda_fixup *fix,
+ int action)
+{
+ /*
+ * The audio jack input and output is not detected on the ASRock NUC Box
+ * 1100 series when cold booting without this fix. Warm rebooting from a
+ * certain other OS makes the audio functional, as COEF settings are
+ * preserved in this case. This fix sets these altered COEF values as
+ * the default.
+ */
+ alc_process_coef_fw(codec, alc233_fixup_no_audio_jack_coefs);
+}
+
enum {
ALC269_FIXUP_GPIO2,
ALC269_FIXUP_SONY_VAIO,
ALC287_FIXUP_YOGA7_14ITL_SPEAKERS,
ALC287_FIXUP_13S_GEN2_SPEAKERS,
ALC256_FIXUP_TONGFANG_RESET_PERSISTENT_SETTINGS,
+ ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE,
+ ALC233_FIXUP_NO_AUDIO_JACK,
};
static const struct hda_fixup alc269_fixups[] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc245_fixup_hp_gpio_led,
},
+ [ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x19, 0x03a11120 }, /* use as headset mic, without its own jack detect */
+ { }
+ },
+ .chained = true,
+ .chain_id = ALC269_FIXUP_HEADSET_MODE_NO_HP_MIC,
+ },
+ [ALC233_FIXUP_NO_AUDIO_JACK] = {
+ .type = HDA_FIXUP_FUNC,
+ .v.func = alc233_fixup_no_audio_jack,
+ },
};
static const struct snd_pci_quirk alc269_fixup_tbl[] = {
SND_PCI_QUIRK(0x1025, 0x1308, "Acer Aspire Z24-890", ALC286_FIXUP_ACER_AIO_HEADSET_MIC),
SND_PCI_QUIRK(0x1025, 0x132a, "Acer TravelMate B114-21", ALC233_FIXUP_ACER_HEADSET_MIC),
SND_PCI_QUIRK(0x1025, 0x1330, "Acer TravelMate X514-51T", ALC255_FIXUP_ACER_HEADSET_MIC),
+ SND_PCI_QUIRK(0x1025, 0x141f, "Acer Spin SP513-54N", ALC255_FIXUP_ACER_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1025, 0x142b, "Acer Swift SF314-42", ALC255_FIXUP_ACER_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1025, 0x1430, "Acer TravelMate B311R-31", ALC256_FIXUP_ACER_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1025, 0x1466, "Acer Aspire A515-56", ALC255_FIXUP_ACER_HEADPHONE_AND_MIC),
SND_PCI_QUIRK(0x103c, 0x8716, "HP Elite Dragonfly G2 Notebook PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x8720, "HP EliteBook x360 1040 G8 Notebook PC", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x8724, "HP EliteBook 850 G7", ALC285_FIXUP_HP_GPIO_LED),
+ SND_PCI_QUIRK(0x103c, 0x8728, "HP EliteBook 840 G7", ALC285_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8729, "HP", ALC285_FIXUP_HP_GPIO_LED),
SND_PCI_QUIRK(0x103c, 0x8730, "HP ProBook 445 G7", ALC236_FIXUP_HP_MUTE_LED_MICMUTE_VREF),
+ SND_PCI_QUIRK(0x103c, 0x8735, "HP ProBook 435 G7", ALC236_FIXUP_HP_MUTE_LED_MICMUTE_VREF),
SND_PCI_QUIRK(0x103c, 0x8736, "HP", ALC285_FIXUP_HP_GPIO_AMP_INIT),
SND_PCI_QUIRK(0x103c, 0x8760, "HP", ALC285_FIXUP_HP_MUTE_LED),
SND_PCI_QUIRK(0x103c, 0x877a, "HP", ALC285_FIXUP_HP_MUTE_LED),
SND_PCI_QUIRK(0x1043, 0x18b1, "Asus MJ401TA", ALC256_FIXUP_ASUS_HEADSET_MIC),
SND_PCI_QUIRK(0x1043, 0x18f1, "Asus FX505DT", ALC256_FIXUP_ASUS_HEADSET_MIC),
SND_PCI_QUIRK(0x1043, 0x194e, "ASUS UX563FD", ALC294_FIXUP_ASUS_HPE),
+ SND_PCI_QUIRK(0x1043, 0x1970, "ASUS UX550VE", ALC289_FIXUP_ASUS_GA401),
SND_PCI_QUIRK(0x1043, 0x1982, "ASUS B1400CEPE", ALC256_FIXUP_ASUS_HPE),
SND_PCI_QUIRK(0x1043, 0x19ce, "ASUS B9450FA", ALC294_FIXUP_ASUS_HPE),
SND_PCI_QUIRK(0x1043, 0x19e1, "ASUS UX581LV", ALC295_FIXUP_ASUS_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x40a1, "Clevo NL40GU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x40c1, "Clevo NL40[CZ]U", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x40d1, "Clevo NL41DU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x5015, "Clevo NH5[58]H[HJK]Q", ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x5017, "Clevo NH7[79]H[HJK]Q", ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50a3, "Clevo NJ51GU", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50b3, "Clevo NK50S[BEZ]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50b6, "Clevo NK50S5", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50b8, "Clevo NK50SZ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50d5, "Clevo NP50D5", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x50e1, "Clevo NH5[58]HPQ", ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
+ SND_PCI_QUIRK(0x1558, 0x50e2, "Clevo NH7[79]HPQ", ALC256_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50f0, "Clevo NH50A[CDF]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50f2, "Clevo NH50E[PR]", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x1558, 0x50f3, "Clevo NH58DPQ", ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE),
SND_PCI_QUIRK(0x17aa, 0x511e, "Thinkpad", ALC298_FIXUP_TPT470_DOCK),
SND_PCI_QUIRK(0x17aa, 0x511f, "Thinkpad", ALC298_FIXUP_TPT470_DOCK),
SND_PCI_QUIRK(0x17aa, 0x9e54, "LENOVO NB", ALC269_FIXUP_LENOVO_EAPD),
+ SND_PCI_QUIRK(0x1849, 0x1233, "ASRock NUC Box 1100", ALC233_FIXUP_NO_AUDIO_JACK),
SND_PCI_QUIRK(0x19e5, 0x3204, "Huawei MACH-WX9", ALC256_FIXUP_HUAWEI_MACH_WX9_PINS),
SND_PCI_QUIRK(0x1b35, 0x1235, "CZC B20", ALC269_FIXUP_CZC_B20),
SND_PCI_QUIRK(0x1b35, 0x1236, "CZC TMI", ALC269_FIXUP_CZC_TMI),
unsigned char ss_out_channels;
u32 io_loopback; /* output loopback channel states*/
- struct snd_dma_buffer *capture_dma_buf;
- struct snd_dma_buffer *playback_dma_buf;
+ /* DMA buffers; those are copied instances from the original snd_dma_buf
+ * objects (which are managed via devres) for the address alignments
+ */
+ struct snd_dma_buffer capture_dma_buf;
+ struct snd_dma_buffer playback_dma_buf;
unsigned char *capture_buffer; /* suitably aligned address */
unsigned char *playback_buffer; /* suitably aligned address */
static int snd_hdsp_initialize_memory(struct hdsp *hdsp)
{
- unsigned long pb_bus, cb_bus;
+ struct snd_dma_buffer *capture_dma, *playback_dma;
- hdsp->capture_dma_buf =
- snd_hammerfall_get_buffer(hdsp->pci, HDSP_DMA_AREA_BYTES);
- hdsp->playback_dma_buf =
- snd_hammerfall_get_buffer(hdsp->pci, HDSP_DMA_AREA_BYTES);
- if (!hdsp->capture_dma_buf || !hdsp->playback_dma_buf) {
+ capture_dma = snd_hammerfall_get_buffer(hdsp->pci, HDSP_DMA_AREA_BYTES);
+ playback_dma = snd_hammerfall_get_buffer(hdsp->pci, HDSP_DMA_AREA_BYTES);
+ if (!capture_dma || !playback_dma) {
dev_err(hdsp->card->dev,
"%s: no buffers available\n", hdsp->card_name);
return -ENOMEM;
}
- /* Align to bus-space 64K boundary */
+ /* copy to the own data for alignment */
+ hdsp->capture_dma_buf = *capture_dma;
+ hdsp->playback_dma_buf = *playback_dma;
- cb_bus = ALIGN(hdsp->capture_dma_buf->addr, 0x10000ul);
- pb_bus = ALIGN(hdsp->playback_dma_buf->addr, 0x10000ul);
+ /* Align to bus-space 64K boundary */
+ hdsp->capture_dma_buf.addr = ALIGN(capture_dma->addr, 0x10000ul);
+ hdsp->playback_dma_buf.addr = ALIGN(playback_dma->addr, 0x10000ul);
/* Tell the card where it is */
+ hdsp_write(hdsp, HDSP_inputBufferAddress, hdsp->capture_dma_buf.addr);
+ hdsp_write(hdsp, HDSP_outputBufferAddress, hdsp->playback_dma_buf.addr);
- hdsp_write(hdsp, HDSP_inputBufferAddress, cb_bus);
- hdsp_write(hdsp, HDSP_outputBufferAddress, pb_bus);
-
- hdsp->capture_buffer = hdsp->capture_dma_buf->area + (cb_bus - hdsp->capture_dma_buf->addr);
- hdsp->playback_buffer = hdsp->playback_dma_buf->area + (pb_bus - hdsp->playback_dma_buf->addr);
+ hdsp->capture_dma_buf.area += hdsp->capture_dma_buf.addr - capture_dma->addr;
+ hdsp->playback_dma_buf.area += hdsp->playback_dma_buf.addr - playback_dma->addr;
+ hdsp->capture_buffer = hdsp->capture_dma_buf.area;
+ hdsp->playback_buffer = hdsp->playback_dma_buf.area;
return 0;
}
snd_pcm_set_sync(substream);
runtime->hw = snd_hdsp_playback_subinfo;
- snd_pcm_set_runtime_buffer(substream, hdsp->playback_dma_buf);
+ snd_pcm_set_runtime_buffer(substream, &hdsp->playback_dma_buf);
hdsp->playback_pid = current->pid;
hdsp->playback_substream = substream;
snd_pcm_set_sync(substream);
runtime->hw = snd_hdsp_capture_subinfo;
- snd_pcm_set_runtime_buffer(substream, hdsp->capture_dma_buf);
+ snd_pcm_set_runtime_buffer(substream, &hdsp->capture_dma_buf);
hdsp->capture_pid = current->pid;
hdsp->capture_substream = substream;
unsigned char ds_channels;
unsigned char ss_channels; /* different for hammerfall/hammerfall-light */
- struct snd_dma_buffer *playback_dma_buf;
- struct snd_dma_buffer *capture_dma_buf;
+ /* DMA buffers; those are copied instances from the original snd_dma_buf
+ * objects (which are managed via devres) for the address alignments
+ */
+ struct snd_dma_buffer playback_dma_buf;
+ struct snd_dma_buffer capture_dma_buf;
unsigned char *capture_buffer; /* suitably aligned address */
unsigned char *playback_buffer; /* suitably aligned address */
static int snd_rme9652_initialize_memory(struct snd_rme9652 *rme9652)
{
- unsigned long pb_bus, cb_bus;
+ struct snd_dma_buffer *capture_dma, *playback_dma;
- rme9652->capture_dma_buf =
- snd_hammerfall_get_buffer(rme9652->pci, RME9652_DMA_AREA_BYTES);
- rme9652->playback_dma_buf =
- snd_hammerfall_get_buffer(rme9652->pci, RME9652_DMA_AREA_BYTES);
- if (!rme9652->capture_dma_buf || !rme9652->playback_dma_buf) {
+ capture_dma = snd_hammerfall_get_buffer(rme9652->pci, RME9652_DMA_AREA_BYTES);
+ playback_dma = snd_hammerfall_get_buffer(rme9652->pci, RME9652_DMA_AREA_BYTES);
+ if (!capture_dma || !playback_dma) {
dev_err(rme9652->card->dev,
"%s: no buffers available\n", rme9652->card_name);
return -ENOMEM;
}
- /* Align to bus-space 64K boundary */
+ /* copy to the own data for alignment */
+ rme9652->capture_dma_buf = *capture_dma;
+ rme9652->playback_dma_buf = *playback_dma;
- cb_bus = ALIGN(rme9652->capture_dma_buf->addr, 0x10000ul);
- pb_bus = ALIGN(rme9652->playback_dma_buf->addr, 0x10000ul);
+ /* Align to bus-space 64K boundary */
+ rme9652->capture_dma_buf.addr = ALIGN(capture_dma->addr, 0x10000ul);
+ rme9652->playback_dma_buf.addr = ALIGN(playback_dma->addr, 0x10000ul);
/* Tell the card where it is */
+ rme9652_write(rme9652, RME9652_rec_buffer, rme9652->capture_dma_buf.addr);
+ rme9652_write(rme9652, RME9652_play_buffer, rme9652->playback_dma_buf.addr);
- rme9652_write(rme9652, RME9652_rec_buffer, cb_bus);
- rme9652_write(rme9652, RME9652_play_buffer, pb_bus);
-
- rme9652->capture_buffer = rme9652->capture_dma_buf->area + (cb_bus - rme9652->capture_dma_buf->addr);
- rme9652->playback_buffer = rme9652->playback_dma_buf->area + (pb_bus - rme9652->playback_dma_buf->addr);
+ rme9652->capture_dma_buf.area += rme9652->capture_dma_buf.addr - capture_dma->addr;
+ rme9652->playback_dma_buf.area += rme9652->playback_dma_buf.addr - playback_dma->addr;
+ rme9652->capture_buffer = rme9652->capture_dma_buf.area;
+ rme9652->playback_buffer = rme9652->playback_dma_buf.area;
return 0;
}
snd_pcm_set_sync(substream);
runtime->hw = snd_rme9652_playback_subinfo;
- snd_pcm_set_runtime_buffer(substream, rme9652->playback_dma_buf);
+ snd_pcm_set_runtime_buffer(substream, &rme9652->playback_dma_buf);
if (rme9652->capture_substream == NULL) {
rme9652_stop(rme9652);
snd_pcm_set_sync(substream);
runtime->hw = snd_rme9652_capture_subinfo;
- snd_pcm_set_runtime_buffer(substream, rme9652->capture_dma_buf);
+ snd_pcm_set_runtime_buffer(substream, &rme9652->capture_dma_buf);
if (rme9652->playback_substream == NULL) {
rme9652_stop(rme9652);
SND_SOC_DAPM_AIF_OUT("ASPTX3", NULL, 0, CS35L41_SP_ENABLES, 2, 0),
SND_SOC_DAPM_AIF_OUT("ASPTX4", NULL, 0, CS35L41_SP_ENABLES, 3, 0),
+ SND_SOC_DAPM_SIGGEN("VSENSE"),
+ SND_SOC_DAPM_SIGGEN("ISENSE"),
+ SND_SOC_DAPM_SIGGEN("VP"),
+ SND_SOC_DAPM_SIGGEN("VBST"),
+ SND_SOC_DAPM_SIGGEN("TEMP"),
+
SND_SOC_DAPM_ADC("VMON ADC", NULL, CS35L41_PWR_CTRL2, 12, 0),
SND_SOC_DAPM_ADC("IMON ADC", NULL, CS35L41_PWR_CTRL2, 13, 0),
SND_SOC_DAPM_ADC("VPMON ADC", NULL, CS35L41_PWR_CTRL2, 8, 0),
cs35l41_main_amp_event,
SND_SOC_DAPM_POST_PMD | SND_SOC_DAPM_POST_PMU),
- SND_SOC_DAPM_INPUT("VP"),
- SND_SOC_DAPM_INPUT("VBST"),
- SND_SOC_DAPM_INPUT("ISENSE"),
- SND_SOC_DAPM_INPUT("VSENSE"),
- SND_SOC_DAPM_INPUT("TEMP"),
-
SND_SOC_DAPM_MUX("ASP TX1 Source", SND_SOC_NOPM, 0, 0, &asp_tx1_mux),
SND_SOC_DAPM_MUX("ASP TX2 Source", SND_SOC_NOPM, 0, 0, &asp_tx2_mux),
SND_SOC_DAPM_MUX("ASP TX3 Source", SND_SOC_NOPM, 0, 0, &asp_tx3_mux),
{"VMON ADC", NULL, "VSENSE"},
{"IMON ADC", NULL, "ISENSE"},
{"VPMON ADC", NULL, "VP"},
- {"TEMPMON ADC", NULL, "TEMP"},
{"VBSTMON ADC", NULL, "VBST"},
+ {"TEMPMON ADC", NULL, "TEMP"},
{"ASPRX1", NULL, "AMP Playback"},
{"ASPRX2", NULL, "AMP Playback"},
snd_soc_component_update_bits(component,
CDC_RX_CLSH_DECAY_CTRL,
CDC_RX_CLSH_DECAY_RATE_MASK, 0x0);
- snd_soc_component_update_bits(component,
+ snd_soc_component_write_field(component,
CDC_RX_RX1_RX_PATH_CFG0,
CDC_RX_RXn_CLSH_EN_MASK, 0x1);
break;
.put = rt1011_r0_load_mode_put \
}
-static const char * const rt1011_i2s_ref_texts[] = {
- "Left Channel", "Right Channel"
+static const char * const rt1011_i2s_ref[] = {
+ "None", "Left Channel", "Right Channel"
};
-static SOC_ENUM_SINGLE_DECL(rt1011_i2s_ref_enum,
- RT1011_TDM1_SET_1, 7,
- rt1011_i2s_ref_texts);
+static SOC_ENUM_SINGLE_DECL(rt1011_i2s_ref_enum, 0, 0,
+ rt1011_i2s_ref);
+
+static int rt1011_i2s_ref_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_component *component =
+ snd_soc_kcontrol_component(kcontrol);
+ struct rt1011_priv *rt1011 =
+ snd_soc_component_get_drvdata(component);
+
+ rt1011->i2s_ref = ucontrol->value.enumerated.item[0];
+ switch (rt1011->i2s_ref) {
+ case RT1011_I2S_REF_LEFT_CH:
+ regmap_write(rt1011->regmap, RT1011_TDM_TOTAL_SET, 0x0240);
+ regmap_write(rt1011->regmap, RT1011_TDM1_SET_2, 0x8);
+ regmap_write(rt1011->regmap, RT1011_TDM1_SET_1, 0x1022);
+ regmap_write(rt1011->regmap, RT1011_ADCDAT_OUT_SOURCE, 0x4);
+ break;
+ case RT1011_I2S_REF_RIGHT_CH:
+ regmap_write(rt1011->regmap, RT1011_TDM_TOTAL_SET, 0x0240);
+ regmap_write(rt1011->regmap, RT1011_TDM1_SET_2, 0x8);
+ regmap_write(rt1011->regmap, RT1011_TDM1_SET_1, 0x10a2);
+ regmap_write(rt1011->regmap, RT1011_ADCDAT_OUT_SOURCE, 0x4);
+ break;
+ default:
+ dev_info(component->dev, "I2S Reference: Do nothing\n");
+ }
+
+ return 0;
+}
+
+static int rt1011_i2s_ref_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_component *component =
+ snd_soc_kcontrol_component(kcontrol);
+ struct rt1011_priv *rt1011 =
+ snd_soc_component_get_drvdata(component);
+
+ ucontrol->value.enumerated.item[0] = rt1011->i2s_ref;
+
+ return 0;
+}
static const struct snd_kcontrol_new rt1011_snd_controls[] = {
/* I2S Data In Selection */
SOC_SINGLE("R0 Temperature", RT1011_STP_INITIAL_RESISTANCE_TEMP,
2, 255, 0),
/* I2S Reference */
- SOC_ENUM("I2S Reference", rt1011_i2s_ref_enum),
+ SOC_ENUM_EXT("I2S Reference", rt1011_i2s_ref_enum,
+ rt1011_i2s_ref_get, rt1011_i2s_ref_put),
};
static int rt1011_is_sys_clk_from_pll(struct snd_soc_dapm_widget *source,
schedule_work(&rt1011->cali_work);
+ rt1011->i2s_ref = 0;
rt1011->bq_drc_params = devm_kcalloc(component->dev,
RT1011_ADVMODE_NUM, sizeof(struct rt1011_bq_drc_params *),
GFP_KERNEL);
RT1011_AIFS
};
+enum {
+ RT1011_I2S_REF_NONE,
+ RT1011_I2S_REF_LEFT_CH,
+ RT1011_I2S_REF_RIGHT_CH,
+};
+
/* BiQual & DRC related settings */
#define RT1011_BQ_DRC_NUM 128
struct rt1011_bq_drc_params {
unsigned int r0_reg, cali_done;
unsigned int r0_calib, temperature_calib;
int recv_spk_mode;
+ int i2s_ref;
};
#endif /* end of _RT1011_H_ */
}
mutex_init(&rt5682->calibrate_mutex);
+ mutex_init(&rt5682->jdet_mutex);
rt5682_calibrate(rt5682);
rt5682_apply_patch_list(rt5682, &i2c->dev);
{RT5682_SAR_IL_CMD_6, 0x0110},
{RT5682_CHARGE_PUMP_1, 0x0210},
{RT5682_HP_LOGIC_CTRL_2, 0x0007},
+ {RT5682_SAR_IL_CMD_2, 0xac00},
+ {RT5682_CBJ_CTRL_7, 0x0104},
};
void rt5682_apply_patch_list(struct rt5682_priv *rt5682, struct device *dev)
snd_soc_component_update_bits(component,
RT5682_HP_CHARGE_PUMP_1,
RT5682_OSW_L_MASK | RT5682_OSW_R_MASK, 0);
+ rt5682_enable_push_button_irq(component, false);
+ snd_soc_component_update_bits(component, RT5682_CBJ_CTRL_1,
+ RT5682_TRIG_JD_MASK, RT5682_TRIG_JD_LOW);
+ usleep_range(55000, 60000);
snd_soc_component_update_bits(component, RT5682_CBJ_CTRL_1,
RT5682_TRIG_JD_MASK, RT5682_TRIG_JD_HIGH);
while (!rt5682->component->card->instantiated)
usleep_range(10000, 15000);
+ mutex_lock(&rt5682->jdet_mutex);
mutex_lock(&rt5682->calibrate_mutex);
val = snd_soc_component_read(rt5682->component, RT5682_AJD1_CTRL)
}
mutex_unlock(&rt5682->calibrate_mutex);
+ mutex_unlock(&rt5682->jdet_mutex);
}
EXPORT_SYMBOL_GPL(rt5682_jack_detect_handler);
{
struct snd_soc_component *component =
snd_soc_dapm_to_component(w->dapm);
+ struct rt5682_priv *rt5682 = snd_soc_component_get_drvdata(component);
switch (event) {
case SND_SOC_DAPM_PRE_PMU:
RT5682_DEPOP_1, 0x60, 0x60);
snd_soc_component_update_bits(component,
RT5682_DAC_ADC_DIG_VOL1, 0x00c0, 0x0080);
+
+ mutex_lock(&rt5682->jdet_mutex);
+
snd_soc_component_update_bits(component, RT5682_HP_CTRL_2,
RT5682_HP_C2_DAC_L_EN | RT5682_HP_C2_DAC_R_EN,
RT5682_HP_C2_DAC_L_EN | RT5682_HP_C2_DAC_R_EN);
usleep_range(5000, 10000);
snd_soc_component_update_bits(component, RT5682_CHARGE_PUMP_1,
RT5682_CP_SW_SIZE_MASK, RT5682_CP_SW_SIZE_L);
+
+ mutex_unlock(&rt5682->jdet_mutex);
break;
case SND_SOC_DAPM_POST_PMD:
cancel_delayed_work_sync(&rt5682->jack_detect_work);
cancel_delayed_work_sync(&rt5682->jd_check_work);
- if (rt5682->hs_jack && rt5682->jack_type == SND_JACK_HEADSET) {
- snd_soc_component_update_bits(component, RT5682_CBJ_CTRL_1,
- RT5682_MB1_PATH_MASK | RT5682_MB2_PATH_MASK,
- RT5682_CTRL_MB1_REG | RT5682_CTRL_MB2_REG);
+ if (rt5682->hs_jack && (rt5682->jack_type & SND_JACK_HEADSET) == SND_JACK_HEADSET) {
val = snd_soc_component_read(component,
RT5682_CBJ_CTRL_2) & RT5682_JACK_TYPE_MASK;
/* enter SAR ADC power saving mode */
snd_soc_component_update_bits(component, RT5682_SAR_IL_CMD_1,
RT5682_SAR_BUTT_DET_MASK | RT5682_SAR_BUTDET_MODE_MASK |
- RT5682_SAR_BUTDET_RST_MASK | RT5682_SAR_SEL_MB1_MB2_MASK, 0);
+ RT5682_SAR_SEL_MB1_MB2_MASK, 0);
+ usleep_range(5000, 6000);
+ snd_soc_component_update_bits(component, RT5682_CBJ_CTRL_1,
+ RT5682_MB1_PATH_MASK | RT5682_MB2_PATH_MASK,
+ RT5682_CTRL_MB1_REG | RT5682_CTRL_MB2_REG);
+ usleep_range(10000, 12000);
snd_soc_component_update_bits(component, RT5682_SAR_IL_CMD_1,
- RT5682_SAR_BUTT_DET_MASK | RT5682_SAR_BUTDET_MODE_MASK | RT5682_SAR_BUTDET_RST_MASK,
- RT5682_SAR_BUTT_DET_EN | RT5682_SAR_BUTDET_POW_SAV | RT5682_SAR_BUTDET_RST_NORMAL);
+ RT5682_SAR_BUTT_DET_MASK | RT5682_SAR_BUTDET_MODE_MASK,
+ RT5682_SAR_BUTT_DET_EN | RT5682_SAR_BUTDET_POW_SAV);
+ snd_soc_component_update_bits(component, RT5682_HP_CHARGE_PUMP_1,
+ RT5682_OSW_L_MASK | RT5682_OSW_R_MASK, 0);
}
regcache_cache_only(rt5682->regmap, true);
regcache_cache_only(rt5682->regmap, false);
regcache_sync(rt5682->regmap);
- if (rt5682->hs_jack && rt5682->jack_type == SND_JACK_HEADSET) {
+ if (rt5682->hs_jack && (rt5682->jack_type & SND_JACK_HEADSET) == SND_JACK_HEADSET) {
snd_soc_component_update_bits(component, RT5682_SAR_IL_CMD_1,
RT5682_SAR_BUTDET_MODE_MASK | RT5682_SAR_SEL_MB1_MB2_MASK,
RT5682_SAR_BUTDET_POW_NORM | RT5682_SAR_SEL_MB1_MB2_AUTO);
+ usleep_range(5000, 6000);
snd_soc_component_update_bits(component, RT5682_CBJ_CTRL_1,
RT5682_MB1_PATH_MASK | RT5682_MB2_PATH_MASK,
RT5682_CTRL_MB1_FSM | RT5682_CTRL_MB2_FSM);
RT5682_PWR_CBJ, RT5682_PWR_CBJ);
}
+ rt5682->jack_type = 0;
mod_delayed_work(system_power_efficient_wq,
- &rt5682->jack_detect_work, msecs_to_jiffies(250));
+ &rt5682->jack_detect_work, msecs_to_jiffies(0));
return 0;
}
int jack_type;
int irq_work_delay_time;
+ struct mutex jdet_mutex;
};
extern const char *rt5682_supply_names[RT5682_NUM_SUPPLIES];
// SPDX-License-Identifier: GPL-2.0
#include <linux/bits.h>
+#include <linux/bitfield.h>
#include <linux/delay.h>
#include <linux/gpio/consumer.h>
#include <linux/i2c.h>
#define RT9120_REG_ERRRPT 0x10
#define RT9120_REG_MSVOL 0x20
#define RT9120_REG_SWRESET 0x40
+#define RT9120_REG_INTERCFG 0x63
#define RT9120_REG_INTERNAL0 0x65
#define RT9120_REG_INTERNAL1 0x69
#define RT9120_REG_UVPOPT 0x6C
+#define RT9120_REG_DIGCFG 0xF8
#define RT9120_VID_MASK GENMASK(15, 8)
#define RT9120_SWRST_MASK BIT(7)
#define RT9120_CFG_WORDLEN_24 24
#define RT9120_CFG_WORDLEN_32 32
#define RT9120_DVDD_UVSEL_MASK GENMASK(5, 4)
+#define RT9120_AUTOSYNC_MASK BIT(6)
-#define RT9120_VENDOR_ID 0x4200
+#define RT9120_VENDOR_ID 0x42
+#define RT9120S_VENDOR_ID 0x43
#define RT9120_RESET_WAITMS 20
#define RT9120_CHIPON_WAITMS 20
#define RT9120_AMPON_WAITMS 50
SNDRV_PCM_FMTBIT_S24_LE |\
SNDRV_PCM_FMTBIT_S32_LE)
+enum {
+ CHIP_IDX_RT9120 = 0,
+ CHIP_IDX_RT9120S,
+ CHIP_IDX_MAX
+};
+
struct rt9120_data {
struct device *dev;
struct regmap *regmap;
+ int chip_idx;
};
/* 11bit [min,max,step] = [-103.9375dB, 24dB, 0.0625dB] */
snd_soc_component_init_regmap(comp, data->regmap);
/* Internal setting */
- snd_soc_component_write(comp, RT9120_REG_INTERNAL1, 0x03);
- snd_soc_component_write(comp, RT9120_REG_INTERNAL0, 0x69);
+ if (data->chip_idx == CHIP_IDX_RT9120S) {
+ snd_soc_component_write(comp, RT9120_REG_INTERCFG, 0xde);
+ snd_soc_component_write(comp, RT9120_REG_INTERNAL0, 0x66);
+ } else
+ snd_soc_component_write(comp, RT9120_REG_INTERNAL0, 0x04);
+
return 0;
}
struct snd_soc_dai *dai)
{
struct snd_soc_component *comp = dai->component;
- unsigned int param_width, param_slot_width;
- int width;
+ unsigned int param_width, param_slot_width, auto_sync;
+ int width, fs;
switch (width = params_width(param)) {
case 16:
snd_soc_component_update_bits(comp, RT9120_REG_I2SWL,
RT9120_AUDWL_MASK, param_slot_width);
+
+ fs = width * params_channels(param);
+ /* If fs is divided by 48, disable auto sync */
+ if (fs % 48 == 0)
+ auto_sync = 0;
+ else
+ auto_sync = RT9120_AUTOSYNC_MASK;
+
+ snd_soc_component_update_bits(comp, RT9120_REG_DIGCFG,
+ RT9120_AUTOSYNC_MASK, auto_sync);
return 0;
}
regmap_reg_range(0x20, 0x27),
regmap_reg_range(0x30, 0x38),
regmap_reg_range(0x3A, 0x40),
+ regmap_reg_range(0x63, 0x63),
regmap_reg_range(0x65, 0x65),
regmap_reg_range(0x69, 0x69),
- regmap_reg_range(0x6C, 0x6C)
+ regmap_reg_range(0x6C, 0x6C),
+ regmap_reg_range(0xF8, 0xF8)
};
static const struct regmap_access_table rt9120_rd_table = {
regmap_reg_range(0x30, 0x38),
regmap_reg_range(0x3A, 0x3D),
regmap_reg_range(0x40, 0x40),
+ regmap_reg_range(0x63, 0x63),
regmap_reg_range(0x65, 0x65),
regmap_reg_range(0x69, 0x69),
- regmap_reg_range(0x6C, 0x6C)
+ regmap_reg_range(0x6C, 0x6C),
+ regmap_reg_range(0xF8, 0xF8)
};
static const struct regmap_access_table rt9120_wr_table = {
static const struct regmap_config rt9120_regmap_config = {
.reg_bits = 8,
.val_bits = 32,
- .max_register = RT9120_REG_UVPOPT,
+ .max_register = RT9120_REG_DIGCFG,
.reg_read = rt9120_reg_read,
.reg_write = rt9120_reg_write,
if (ret)
return ret;
- if ((devid & RT9120_VID_MASK) != RT9120_VENDOR_ID) {
- dev_err(data->dev, "DEVID not correct [0x%04x]\n", devid);
+ devid = FIELD_GET(RT9120_VID_MASK, devid);
+ switch (devid) {
+ case RT9120_VENDOR_ID:
+ data->chip_idx = CHIP_IDX_RT9120;
+ break;
+ case RT9120S_VENDOR_ID:
+ data->chip_idx = CHIP_IDX_RT9120S;
+ break;
+ default:
+ dev_err(data->dev, "DEVID not correct [0x%0x]\n", devid);
return -ENODEV;
}
}
wcd->dai[dai->id].sconfig.rate = params_rate(params);
- wcd934x_slim_set_hw_params(wcd, &wcd->dai[dai->id], substream->stream);
- return 0;
+ return wcd934x_slim_set_hw_params(wcd, &wcd->dai[dai->id], substream->stream);
}
static int wcd934x_hw_free(struct snd_pcm_substream *substream,
case WCD938X_DIGITAL_INTR_STATUS_0:
case WCD938X_DIGITAL_INTR_STATUS_1:
case WCD938X_DIGITAL_INTR_STATUS_2:
+ case WCD938X_DIGITAL_INTR_CLEAR_0:
+ case WCD938X_DIGITAL_INTR_CLEAR_1:
+ case WCD938X_DIGITAL_INTR_CLEAR_2:
case WCD938X_DIGITAL_SWR_HM_TEST_0:
case WCD938X_DIGITAL_SWR_HM_TEST_1:
case WCD938X_DIGITAL_EFUSE_T_DATA_0:
switch (cs_dsp->fw_ver) {
case 0:
case 1:
- snprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN, "%s %s %x",
- cs_dsp->name, region_name, cs_ctl->alg_region.alg);
+ ret = scnprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN,
+ "%s %s %x", cs_dsp->name, region_name,
+ cs_ctl->alg_region.alg);
break;
case 2:
ret = scnprintf(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN,
SOF_BT_OFFLOAD_SSP(2) |
SOF_SSP_BT_OFFLOAD_PRESENT),
},
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0AF3"),
+ },
+ /* No Jack */
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ SOF_SDW_FOUR_SPK),
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B00")
+ },
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ RT711_JD2 |
+ SOF_SDW_FOUR_SPK),
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B01")
+ },
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ RT711_JD2 |
+ SOF_SDW_FOUR_SPK),
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B11")
+ },
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ RT711_JD2 |
+ SOF_SDW_FOUR_SPK),
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B12")
+ },
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ RT711_JD2 |
+ SOF_SDW_FOUR_SPK),
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B13"),
+ },
+ /* No Jack */
+ .driver_data = (void *)SOF_SDW_TGL_HDMI,
+ },
+ {
+ .callback = sof_sdw_quirk_cb,
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
+ DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0B29"),
+ },
+ .driver_data = (void *)(SOF_SDW_TGL_HDMI |
+ RT711_JD2 |
+ SOF_SDW_FOUR_SPK),
+ },
{}
};
}
};
+static const struct snd_soc_acpi_adr_device rt711_sdca_2_adr[] = {
+ {
+ .adr = 0x000230025D071101ull,
+ .num_endpoints = 1,
+ .endpoints = &single_endpoint,
+ .name_prefix = "rt711"
+ }
+};
+
static const struct snd_soc_acpi_adr_device rt1316_1_group1_adr[] = {
{
.adr = 0x000131025D131601ull, /* unique ID is set for some reason */
}
};
+static const struct snd_soc_acpi_adr_device rt1316_0_group2_adr[] = {
+ {
+ .adr = 0x000031025D131601ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_l_endpoint,
+ .name_prefix = "rt1316-1"
+ }
+};
+
+static const struct snd_soc_acpi_adr_device rt1316_1_group2_adr[] = {
+ {
+ .adr = 0x000130025D131601ull,
+ .num_endpoints = 1,
+ .endpoints = &spk_r_endpoint,
+ .name_prefix = "rt1316-2"
+ }
+};
+
static const struct snd_soc_acpi_adr_device rt1316_2_single_adr[] = {
{
.adr = 0x000230025D131601ull,
{}
};
+static const struct snd_soc_acpi_link_adr adl_sdw_rt711_link2_rt1316_link01_rt714_link3[] = {
+ {
+ .mask = BIT(2),
+ .num_adr = ARRAY_SIZE(rt711_sdca_2_adr),
+ .adr_d = rt711_sdca_2_adr,
+ },
+ {
+ .mask = BIT(0),
+ .num_adr = ARRAY_SIZE(rt1316_0_group2_adr),
+ .adr_d = rt1316_0_group2_adr,
+ },
+ {
+ .mask = BIT(1),
+ .num_adr = ARRAY_SIZE(rt1316_1_group2_adr),
+ .adr_d = rt1316_1_group2_adr,
+ },
+ {
+ .mask = BIT(3),
+ .num_adr = ARRAY_SIZE(rt714_3_adr),
+ .adr_d = rt714_3_adr,
+ },
+ {}
+};
+
+static const struct snd_soc_acpi_link_adr adl_sdw_rt1316_link12_rt714_link0[] = {
+ {
+ .mask = BIT(1),
+ .num_adr = ARRAY_SIZE(rt1316_1_group1_adr),
+ .adr_d = rt1316_1_group1_adr,
+ },
+ {
+ .mask = BIT(2),
+ .num_adr = ARRAY_SIZE(rt1316_2_group1_adr),
+ .adr_d = rt1316_2_group1_adr,
+ },
+ {
+ .mask = BIT(0),
+ .num_adr = ARRAY_SIZE(rt714_0_adr),
+ .adr_d = rt714_0_adr,
+ },
+ {}
+};
+
+static const struct snd_soc_acpi_link_adr adl_sdw_rt1316_link2_rt714_link3[] = {
+ {
+ .mask = BIT(2),
+ .num_adr = ARRAY_SIZE(rt1316_2_single_adr),
+ .adr_d = rt1316_2_single_adr,
+ },
+ {
+ .mask = BIT(3),
+ .num_adr = ARRAY_SIZE(rt714_3_adr),
+ .adr_d = rt714_3_adr,
+ },
+ {}
+};
+
static const struct snd_soc_acpi_link_adr adl_sdw_rt1316_link2_rt714_link0[] = {
{
.mask = BIT(2),
.sof_tplg_filename = "sof-adl-rt711-l0-rt1316-l13-rt714-l2.tplg",
},
{
+ .link_mask = 0xF, /* 4 active links required */
+ .links = adl_sdw_rt711_link2_rt1316_link01_rt714_link3,
+ .drv_name = "sof_sdw",
+ .sof_fw_filename = "sof-adl.ri",
+ .sof_tplg_filename = "sof-adl-rt711-l2-rt1316-l01-rt714-l3.tplg",
+ },
+ {
+ .link_mask = 0xC, /* rt1316 on link2 & rt714 on link3 */
+ .links = adl_sdw_rt1316_link2_rt714_link3,
+ .drv_name = "sof_sdw",
+ .sof_fw_filename = "sof-adl.ri",
+ .sof_tplg_filename = "sof-adl-rt1316-l2-mono-rt714-l3.tplg",
+ },
+ {
+ .link_mask = 0x7, /* rt714 on link0 & two rt1316s on link1 and link2 */
+ .links = adl_sdw_rt1316_link12_rt714_link0,
+ .drv_name = "sof_sdw",
+ .sof_fw_filename = "sof-adl.ri",
+ .sof_tplg_filename = "sof-adl-rt1316-l12-rt714-l0.tplg",
+ },
+ {
.link_mask = 0x5, /* 2 active links required */
.links = adl_sdw_rt1316_link2_rt714_link0,
.drv_name = "sof_sdw",
int irq_id;
struct mtk_base_afe *afe;
struct mt8173_afe_private *afe_priv;
+ struct snd_soc_component *comp_pcm, *comp_hdmi;
ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(33));
if (ret)
if (ret)
goto err_pm_disable;
- ret = devm_snd_soc_register_component(&pdev->dev,
- &mt8173_afe_pcm_dai_component,
- mt8173_afe_pcm_dais,
- ARRAY_SIZE(mt8173_afe_pcm_dais));
+ comp_pcm = devm_kzalloc(&pdev->dev, sizeof(*comp_pcm), GFP_KERNEL);
+ if (!comp_pcm) {
+ ret = -ENOMEM;
+ goto err_pm_disable;
+ }
+
+ ret = snd_soc_component_initialize(comp_pcm,
+ &mt8173_afe_pcm_dai_component,
+ &pdev->dev);
if (ret)
goto err_pm_disable;
- ret = devm_snd_soc_register_component(&pdev->dev,
- &mt8173_afe_hdmi_dai_component,
- mt8173_afe_hdmi_dais,
- ARRAY_SIZE(mt8173_afe_hdmi_dais));
+#ifdef CONFIG_DEBUG_FS
+ comp_pcm->debugfs_prefix = "pcm";
+#endif
+
+ ret = snd_soc_add_component(comp_pcm,
+ mt8173_afe_pcm_dais,
+ ARRAY_SIZE(mt8173_afe_pcm_dais));
+ if (ret)
+ goto err_pm_disable;
+
+ comp_hdmi = devm_kzalloc(&pdev->dev, sizeof(*comp_hdmi), GFP_KERNEL);
+ if (!comp_hdmi) {
+ ret = -ENOMEM;
+ goto err_pm_disable;
+ }
+
+ ret = snd_soc_component_initialize(comp_hdmi,
+ &mt8173_afe_hdmi_dai_component,
+ &pdev->dev);
if (ret)
goto err_pm_disable;
+#ifdef CONFIG_DEBUG_FS
+ comp_hdmi->debugfs_prefix = "hdmi";
+#endif
+
+ ret = snd_soc_add_component(comp_hdmi,
+ mt8173_afe_hdmi_dais,
+ ARRAY_SIZE(mt8173_afe_hdmi_dais));
+ if (ret)
+ goto err_cleanup_components;
+
dev_info(&pdev->dev, "MT8173 AFE driver initialized.\n");
return 0;
+err_cleanup_components:
+ snd_soc_unregister_component(&pdev->dev);
err_pm_disable:
pm_runtime_disable(&pdev->dev);
return ret;
static int mt8173_afe_pcm_dev_remove(struct platform_device *pdev)
{
+ snd_soc_unregister_component(&pdev->dev);
+
pm_runtime_disable(&pdev->dev);
if (!pm_runtime_status_suspended(&pdev->dev))
mt8173_afe_runtime_suspend(&pdev->dev);
};
static const struct snd_soc_dapm_widget mt8173_rt5650_widgets[] = {
- SND_SOC_DAPM_SPK("Speaker", NULL),
+ SND_SOC_DAPM_SPK("Ext Spk", NULL),
SND_SOC_DAPM_MIC("Int Mic", NULL),
SND_SOC_DAPM_HP("Headphone", NULL),
SND_SOC_DAPM_MIC("Headset Mic", NULL),
};
static const struct snd_soc_dapm_route mt8173_rt5650_routes[] = {
- {"Speaker", NULL, "SPOL"},
- {"Speaker", NULL, "SPOR"},
+ {"Ext Spk", NULL, "SPOL"},
+ {"Ext Spk", NULL, "SPOR"},
{"DMIC L1", NULL, "Int Mic"},
{"DMIC R1", NULL, "Int Mic"},
{"Headphone", NULL, "HPOL"},
};
static const struct snd_kcontrol_new mt8173_rt5650_controls[] = {
- SOC_DAPM_PIN_SWITCH("Speaker"),
+ SOC_DAPM_PIN_SWITCH("Ext Spk"),
SOC_DAPM_PIN_SWITCH("Int Mic"),
SOC_DAPM_PIN_SWITCH("Headphone"),
SOC_DAPM_PIN_SWITCH("Headset Mic"),
uint32_t clock_root;
} __packed;
+struct audio_hw_clk_rel_cfg {
+ uint32_t clock_id;
+} __packed;
+
#define PARAM_ID_HW_EP_POWER_MODE_CFG 0x8001176
#define AR_HW_EP_POWER_MODE_0 0 /* default */
#define AR_HW_EP_POWER_MODE_1 1 /* XO Shutdown allowed */
int ret = 0;
if (port_id < 0) {
- dev_err(dev, "Invalid port_id 0x%x\n", port_id);
+ dev_err(dev, "Invalid port_id %d\n", port_id);
return ERR_PTR(-EINVAL);
}
int port_idx = payload_map.port_id[i];
if (port_idx < 0) {
- dev_err(dev, "Invalid port_id 0x%x\n",
+ dev_err(dev, "Invalid port_id %d\n",
payload_map.port_id[i]);
kfree(pkt);
return -EINVAL;
if (ret < 0) {
dev_err(dev, "%s: q6asm_open_write failed\n", __func__);
- q6asm_audio_client_free(prtd->audio_client);
- prtd->audio_client = NULL;
- return -ENOMEM;
+ goto open_err;
}
prtd->session_id = q6asm_get_session_id(prtd->audio_client);
prtd->session_id, substream->stream);
if (ret) {
dev_err(dev, "%s: stream reg failed ret:%d\n", __func__, ret);
- return ret;
+ goto routing_err;
}
if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
}
if (ret < 0)
dev_info(dev, "%s: CMD Format block failed\n", __func__);
+ else
+ prtd->state = Q6ASM_STREAM_RUNNING;
- prtd->state = Q6ASM_STREAM_RUNNING;
+ return ret;
- return 0;
+routing_err:
+ q6asm_cmd(prtd->audio_client, prtd->stream_id, CMD_CLOSE);
+open_err:
+ q6asm_unmap_memory_regions(substream->stream, prtd->audio_client);
+ q6asm_audio_client_free(prtd->audio_client);
+ prtd->audio_client = NULL;
+
+ return ret;
}
static int q6asm_dai_trigger(struct snd_soc_component *component,
struct audio_hw_clk_cfg clock_id;
} __packed;
+struct prm_cmd_release_rsc {
+ struct apm_module_param_data param_data;
+ uint32_t num_clk_id;
+ struct audio_hw_clk_rel_cfg clock_id;
+} __packed;
+
static int q6prm_send_cmd_sync(struct q6prm *prm, struct gpr_pkt *pkt, uint32_t rsp_opcode)
{
return audioreach_send_cmd_sync(prm->dev, prm->gdev, &prm->result, &prm->lock,
}
EXPORT_SYMBOL_GPL(q6prm_unvote_lpass_core_hw);
-int q6prm_set_lpass_clock(struct device *dev, int clk_id, int clk_attr, int clk_root,
- unsigned int freq)
+static int q6prm_request_lpass_clock(struct device *dev, int clk_id, int clk_attr, int clk_root,
+ unsigned int freq)
{
struct q6prm *prm = dev_get_drvdata(dev->parent);
struct apm_module_param_data *param_data;
return rc;
}
+
+static int q6prm_release_lpass_clock(struct device *dev, int clk_id, int clk_attr, int clk_root,
+ unsigned int freq)
+{
+ struct q6prm *prm = dev_get_drvdata(dev->parent);
+ struct apm_module_param_data *param_data;
+ struct prm_cmd_release_rsc *rel;
+ gpr_device_t *gdev = prm->gdev;
+ struct gpr_pkt *pkt;
+ int rc;
+
+ pkt = audioreach_alloc_cmd_pkt(sizeof(*rel), PRM_CMD_RELEASE_HW_RSC, 0, gdev->svc.id,
+ GPR_PRM_MODULE_IID);
+ if (IS_ERR(pkt))
+ return PTR_ERR(pkt);
+
+ rel = (void *)pkt + GPR_HDR_SIZE + APM_CMD_HDR_SIZE;
+
+ param_data = &rel->param_data;
+
+ param_data->module_instance_id = GPR_PRM_MODULE_IID;
+ param_data->error_code = 0;
+ param_data->param_id = PARAM_ID_RSC_AUDIO_HW_CLK;
+ param_data->param_size = sizeof(*rel) - APM_MODULE_PARAM_DATA_SIZE;
+
+ rel->num_clk_id = 1;
+ rel->clock_id.clock_id = clk_id;
+
+ rc = q6prm_send_cmd_sync(prm, pkt, PRM_CMD_RSP_RELEASE_HW_RSC);
+
+ kfree(pkt);
+
+ return rc;
+}
+
+int q6prm_set_lpass_clock(struct device *dev, int clk_id, int clk_attr, int clk_root,
+ unsigned int freq)
+{
+ if (freq)
+ return q6prm_request_lpass_clock(dev, clk_id, clk_attr, clk_attr, freq);
+
+ return q6prm_release_lpass_clock(dev, clk_id, clk_attr, clk_attr, freq);
+}
EXPORT_SYMBOL_GPL(q6prm_set_lpass_clock);
static int prm_callback(struct gpr_resp_pkt *data, void *priv, int op)
}
session = &routing_data->sessions[stream_id - 1];
+ if (session->port_id < 0) {
+ dev_err(routing_data->dev, "Routing not setup for MultiMedia%d Session\n",
+ session->fedai_id);
+ return -EINVAL;
+ }
+
pdata = &routing_data->port_data[session->port_id];
mutex_lock(&routing_data->lock);
session->port_id = be_id;
snd_soc_dapm_mixer_update_power(dapm, kcontrol, 1, update);
} else {
- session->port_id = -1;
+ if (session->port_id == be_id) {
+ session->port_id = -1;
+ return 0;
+ }
+
snd_soc_dapm_mixer_update_power(dapm, kcontrol, 0, update);
}
struct rsnd_dmaen *dmaen = rsnd_dma_to_dmaen(dma);
if (dmaen->chan)
- dmaengine_terminate_sync(dmaen->chan);
+ dmaengine_terminate_async(dmaen->chan);
return 0;
}
return NULL;
}
-static int snd_soc_dapm_set_pin(struct snd_soc_dapm_context *dapm,
- const char *pin, int status)
+/*
+ * set the DAPM pin status:
+ * returns 1 when the value has been updated, 0 when unchanged, or a negative
+ * error code; called from kcontrol put callback
+ */
+static int __snd_soc_dapm_set_pin(struct snd_soc_dapm_context *dapm,
+ const char *pin, int status)
{
struct snd_soc_dapm_widget *w = dapm_find_widget(dapm, pin, true);
int ret = 0;
return ret;
}
+/*
+ * similar as __snd_soc_dapm_set_pin(), but returns 0 when successful;
+ * called from several API functions below
+ */
+static int snd_soc_dapm_set_pin(struct snd_soc_dapm_context *dapm,
+ const char *pin, int status)
+{
+ int ret = __snd_soc_dapm_set_pin(dapm, pin, status);
+
+ return ret < 0 ? ret : 0;
+}
+
/**
* snd_soc_dapm_sync_unlocked - scan and power dapm paths
* @dapm: DAPM context
const char *pin = (const char *)kcontrol->private_value;
int ret;
- if (ucontrol->value.integer.value[0])
- ret = snd_soc_dapm_enable_pin(&card->dapm, pin);
- else
- ret = snd_soc_dapm_disable_pin(&card->dapm, pin);
+ mutex_lock_nested(&card->dapm_mutex, SND_SOC_DAPM_CLASS_RUNTIME);
+ ret = __snd_soc_dapm_set_pin(&card->dapm, pin,
+ !!ucontrol->value.integer.value[0]);
+ mutex_unlock(&card->dapm_mutex);
snd_soc_dapm_sync(&card->dapm);
return ret;
/* remove dynamic controls from the component driver */
int snd_soc_tplg_component_remove(struct snd_soc_component *comp)
{
+ struct snd_card *card = comp->card->snd_card;
struct snd_soc_dobj *dobj, *next_dobj;
int pass = SOC_TPLG_PASS_END;
while (pass >= SOC_TPLG_PASS_START) {
/* remove mixer controls */
+ down_write(&card->controls_rwsem);
list_for_each_entry_safe(dobj, next_dobj, &comp->dobj_list,
list) {
break;
}
}
+ up_write(&card->controls_rwsem);
pass--;
}
Say Y if you need this option. If unsure select "N".
config SND_SOC_SOF_COMPRESS
- tristate
+ bool
select SND_SOC_COMPRESS
config SND_SOC_SOF_DEBUG_PROBES
{
struct sof_ipc_ctrl_data *cdata = scontrol->control_data;
struct snd_soc_component *scomp = scontrol->scomp;
- enum sof_ipc_ctrl_type ctrl_type;
+ u32 ipc_cmd;
int ret;
if (!scontrol->comp_data_dirty)
return;
if (scontrol->cmd == SOF_CTRL_CMD_BINARY)
- ctrl_type = SOF_IPC_COMP_GET_DATA;
+ ipc_cmd = SOF_IPC_COMP_GET_DATA;
else
- ctrl_type = SOF_IPC_COMP_GET_VALUE;
+ ipc_cmd = SOF_IPC_COMP_GET_VALUE;
/* set the ABI header values */
cdata->data->magic = SOF_ABI_MAGIC;
/* refresh the component data from DSP */
scontrol->comp_data_dirty = false;
- ret = snd_sof_ipc_set_get_comp_data(scontrol, ctrl_type,
+ ret = snd_sof_ipc_set_get_comp_data(scontrol, ipc_cmd,
SOF_CTRL_TYPE_VALUE_CHAN_GET,
scontrol->cmd, false);
if (ret < 0) {
#include <linux/io.h>
#include <sound/hdaudio.h>
#include <sound/hda_i915.h>
+#include <sound/hda_codec.h>
+#include <sound/hda_register.h>
#include "../sof-priv.h"
#include "hda.h"
#endif
#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+static void update_codec_wake_enable(struct hdac_bus *bus, unsigned int addr, bool link_power)
+{
+ unsigned int mask = snd_hdac_chip_readw(bus, WAKEEN);
+
+ if (link_power)
+ mask &= ~BIT(addr);
+ else
+ mask |= BIT(addr);
+
+ snd_hdac_chip_updatew(bus, WAKEEN, STATESTS_INT_MASK, mask);
+}
+
static void sof_hda_bus_link_power(struct hdac_device *codec, bool enable)
{
struct hdac_bus *bus = codec->bus;
*/
if (codec->addr == HDA_IDISP_ADDR && !enable)
snd_hdac_display_power(bus, HDA_CODEC_IDX_CONTROLLER, false);
+
+ /* WAKEEN needs to be set for disabled links */
+ update_codec_wake_enable(bus, codec->addr, enable);
}
static const struct hdac_bus_ops bus_core_ops = {
hda_dsp_ipc_int_disable(sdev);
#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
- if (runtime_suspend)
- hda_codec_jack_wake_enable(sdev, true);
+ hda_codec_jack_wake_enable(sdev, runtime_suspend);
/* power down all hda link */
snd_hdac_ext_bus_link_power_down_all(bus);
return 0;
}
+static void hda_check_for_state_change(struct snd_sof_dev *sdev)
+{
+#if IS_ENABLED(CONFIG_SND_SOC_SOF_HDA)
+ struct hdac_bus *bus = sof_to_bus(sdev);
+ unsigned int codec_mask;
+
+ codec_mask = snd_hdac_chip_readw(bus, STATESTS);
+ if (codec_mask) {
+ hda_codec_jack_check(sdev);
+ snd_hdac_chip_writew(bus, STATESTS, codec_mask);
+ }
+#endif
+}
+
static irqreturn_t hda_dsp_interrupt_handler(int irq, void *context)
{
struct snd_sof_dev *sdev = context;
if (hda_sdw_check_wakeen_irq(sdev))
hda_sdw_process_wakeen(sdev);
+ hda_check_for_state_change(sdev);
+
/* enable GIE interrupt */
snd_sof_dsp_update_bits(sdev, HDA_DSP_HDA_BAR,
SOF_HDA_INTCTL,
if (ret < 0)
return ret;
- nb_bits = frame_len * ((cgfr & I2S_CGFR_CHLEN) + 1);
+ nb_bits = frame_len * (FIELD_GET(I2S_CGFR_CHLEN, cgfr) + 1);
ret = stm32_i2s_calc_clk_div(i2s, i2s_clock_rate,
(nb_bits * rate));
if (ret)
emu->name = kstrdup(name, GFP_KERNEL);
emu->voices = kcalloc(emu->max_voices, sizeof(struct snd_emux_voice),
GFP_KERNEL);
- if (emu->voices == NULL)
+ if (emu->name == NULL || emu->voices == NULL)
return -ENOMEM;
/* create soundfont list */
return 0;
}
+/* free-wheeling mode? (e.g. dmix) */
+static int in_free_wheeling_mode(struct snd_pcm_runtime *runtime)
+{
+ return runtime->stop_threshold > runtime->buffer_size;
+}
+
/* check whether early start is needed for playback stream */
static int lowlatency_playback_available(struct snd_pcm_runtime *runtime,
struct snd_usb_substream *subs)
/* disabled via module option? */
if (!chip->lowlatency)
return false;
- /* free-wheeling mode? (e.g. dmix) */
- if (runtime->stop_threshold > runtime->buffer_size)
+ if (in_free_wheeling_mode(runtime))
return false;
/* implicit feedback mode has own operation mode */
if (snd_usb_endpoint_implicit_feedback_sink(subs->data_endpoint))
runtime->delay = 0;
subs->lowlatency_playback = lowlatency_playback_available(runtime, subs);
- if (!subs->lowlatency_playback)
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK &&
+ !subs->lowlatency_playback)
ret = start_endpoints(subs);
unlock:
subs);
if (subs->lowlatency_playback &&
cmd == SNDRV_PCM_TRIGGER_START) {
+ if (in_free_wheeling_mode(substream->runtime))
+ subs->lowlatency_playback = false;
err = start_endpoints(subs);
if (err < 0) {
snd_usb_endpoint_set_callback(subs->data_endpoint,
@echo ' intel-speed-select - Intel Speed Select tool'
@echo ' kvm_stat - top-like utility for displaying kvm statistics'
@echo ' leds - LEDs tools'
- @echo ' liblockdep - user-space wrapper for kernel locking-validator'
@echo ' objtool - an ELF object analysis tool'
@echo ' pci - PCI tools'
@echo ' perf - Linux performance measurement and analysis tool'
bpf/%: FORCE
$(call descend,$@)
-liblockdep: FORCE
- $(call descend,lib/lockdep)
-
libapi: FORCE
$(call descend,lib/api)
kvm_stat: FORCE
$(call descend,kvm/$@)
-all: acpi cgroup counter cpupower gpio hv firewire liblockdep \
+all: acpi cgroup counter cpupower gpio hv firewire \
perf selftests bootconfig spi turbostat usb \
virtio vm bpf x86_energy_perf_policy \
tmon freefall iio objtool kvm_stat wmi \
cgroup_install counter_install firewire_install gpio_install hv_install iio_install perf_install bootconfig_install spi_install usb_install virtio_install vm_install bpf_install objtool_install wmi_install pci_install debugging_install tracing_install:
$(call descend,$(@:_install=),install)
-liblockdep_install:
- $(call descend,lib/lockdep,install)
-
selftests_install:
$(call descend,testing/$(@:_install=),install)
$(call descend,kvm/$(@:_install=),install)
install: acpi_install cgroup_install counter_install cpupower_install gpio_install \
- hv_install firewire_install iio_install liblockdep_install \
+ hv_install firewire_install iio_install \
perf_install selftests_install turbostat_install usb_install \
virtio_install vm_install bpf_install x86_energy_perf_policy_install \
tmon_install freefall_install objtool_install kvm_stat_install \
cgroup_clean counter_clean hv_clean firewire_clean bootconfig_clean spi_clean usb_clean virtio_clean vm_clean wmi_clean bpf_clean iio_clean gpio_clean objtool_clean leds_clean pci_clean firmware_clean debugging_clean tracing_clean:
$(call descend,$(@:_clean=),clean)
-liblockdep_clean:
- $(call descend,lib/lockdep,clean)
-
libapi_clean:
$(call descend,lib/api,clean)
clean: acpi_clean cgroup_clean counter_clean cpupower_clean hv_clean firewire_clean \
perf_clean selftests_clean turbostat_clean bootconfig_clean spi_clean usb_clean virtio_clean \
vm_clean bpf_clean iio_clean x86_energy_perf_policy_clean tmon_clean \
- freefall_clean build_clean libbpf_clean libsubcmd_clean liblockdep_clean \
+ freefall_clean build_clean libbpf_clean libsubcmd_clean \
gpio_clean objtool_clean leds_clean wmi_clean pci_clean firmware_clean debugging_clean \
intel-speed-select_clean tracing_clean
PERF_REG_POWERPC_PMC4,
PERF_REG_POWERPC_PMC5,
PERF_REG_POWERPC_PMC6,
- /* Max regs without the extended regs */
+ PERF_REG_POWERPC_SDAR,
+ PERF_REG_POWERPC_SIAR,
+ /* Max mask value for interrupt regs w/o extended regs */
PERF_REG_POWERPC_MAX = PERF_REG_POWERPC_MMCRA + 1,
+ /* Max mask value for interrupt regs including extended regs */
+ PERF_REG_EXTENDED_MAX = PERF_REG_POWERPC_SIAR + 1,
};
#define PERF_REG_PMU_MASK ((1ULL << PERF_REG_POWERPC_MAX) - 1)
-/* Exclude MMCR3, SIER2, SIER3 for CPU_FTR_ARCH_300 */
-#define PERF_EXCLUDE_REG_EXT_300 (7ULL << PERF_REG_POWERPC_MMCR3)
-
/*
* PERF_REG_EXTENDED_MASK value for CPU_FTR_ARCH_300
- * includes 9 SPRS from MMCR0 to PMC6 excluding the
- * unsupported SPRS in PERF_EXCLUDE_REG_EXT_300.
+ * includes 11 SPRS from MMCR0 to SIAR excluding the
+ * unsupported SPRS MMCR3, SIER2 and SIER3.
*/
-#define PERF_REG_PMU_MASK_300 ((0xfffULL << PERF_REG_POWERPC_MMCR0) - PERF_EXCLUDE_REG_EXT_300)
+#define PERF_REG_PMU_MASK_300 \
+ ((1ULL << PERF_REG_POWERPC_MMCR0) | (1ULL << PERF_REG_POWERPC_MMCR1) | \
+ (1ULL << PERF_REG_POWERPC_MMCR2) | (1ULL << PERF_REG_POWERPC_PMC1) | \
+ (1ULL << PERF_REG_POWERPC_PMC2) | (1ULL << PERF_REG_POWERPC_PMC3) | \
+ (1ULL << PERF_REG_POWERPC_PMC4) | (1ULL << PERF_REG_POWERPC_PMC5) | \
+ (1ULL << PERF_REG_POWERPC_PMC6) | (1ULL << PERF_REG_POWERPC_SDAR) | \
+ (1ULL << PERF_REG_POWERPC_SIAR))
/*
* PERF_REG_EXTENDED_MASK value for CPU_FTR_ARCH_31
- * includes 12 SPRs from MMCR0 to PMC6.
+ * includes 14 SPRs from MMCR0 to SIAR.
*/
-#define PERF_REG_PMU_MASK_31 (0xfffULL << PERF_REG_POWERPC_MMCR0)
+#define PERF_REG_PMU_MASK_31 \
+ (PERF_REG_PMU_MASK_300 | (1ULL << PERF_REG_POWERPC_MMCR3) | \
+ (1ULL << PERF_REG_POWERPC_SIER2) | (1ULL << PERF_REG_POWERPC_SIER3))
-#define PERF_REG_EXTENDED_MAX (PERF_REG_POWERPC_PMC6 + 1)
#endif /* _UAPI_ASM_POWERPC_PERF_REGS_H */
#define X86_FEATURE_XSAVEC (10*32+ 1) /* XSAVEC instruction */
#define X86_FEATURE_XGETBV1 (10*32+ 2) /* XGETBV with ECX = 1 instruction */
#define X86_FEATURE_XSAVES (10*32+ 3) /* XSAVES/XRSTORS instructions */
+#define X86_FEATURE_XFD (10*32+ 4) /* "" eXtended Feature Disabling */
/*
* Extended auxiliary flags: Linux defined - for features scattered in various
/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
#define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */
#define X86_FEATURE_AVX512_BF16 (12*32+ 5) /* AVX512 BFLOAT16 instructions */
+#define X86_FEATURE_AMX_TILE (18*32+24) /* AMX tile Support */
/* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */
#define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */
#define MSR_IA32_BNDCFGS_RSVD 0x00000ffc
+#define MSR_IA32_XFD 0x000001c4
+#define MSR_IA32_XFD_ERR 0x000001c5
#define MSR_IA32_XSS 0x00000da0
#define MSR_IA32_APICBASE 0x0000001b
#define KVM_PMU_EVENT_ALLOW 0
#define KVM_PMU_EVENT_DENY 1
+/* for KVM_{GET,SET,HAS}_DEVICE_ATTR */
+#define KVM_VCPU_TSC_CTRL 0 /* control group for the timestamp counter (TSC) */
+#define KVM_VCPU_TSC_OFFSET 0 /* attribute for the TSC offset */
+
#endif /* _ASM_X86_KVM_H */
#define ARCH_GET_CPUID 0x1011
#define ARCH_SET_CPUID 0x1012
+#define ARCH_GET_XCOMP_SUPP 0x1021
+#define ARCH_GET_XCOMP_PERM 0x1022
+#define ARCH_REQ_XCOMP_PERM 0x1023
+
#define ARCH_MAP_VDSO_X32 0x2001
#define ARCH_MAP_VDSO_32 0x2002
#define ARCH_MAP_VDSO_64 0x2003
_OUTPUT := $(CURDIR)
endif
BOOTSTRAP_OUTPUT := $(_OUTPUT)/bootstrap/
+
LIBBPF_OUTPUT := $(_OUTPUT)/libbpf/
LIBBPF_DESTDIR := $(LIBBPF_OUTPUT)
LIBBPF_INCLUDE := $(LIBBPF_DESTDIR)/include
LIBBPF_HDRS_DIR := $(LIBBPF_INCLUDE)/bpf
+LIBBPF := $(LIBBPF_OUTPUT)libbpf.a
-LIBBPF = $(LIBBPF_OUTPUT)libbpf.a
-LIBBPF_BOOTSTRAP_OUTPUT = $(BOOTSTRAP_OUTPUT)libbpf/
-LIBBPF_BOOTSTRAP = $(LIBBPF_BOOTSTRAP_OUTPUT)libbpf.a
+LIBBPF_BOOTSTRAP_OUTPUT := $(BOOTSTRAP_OUTPUT)libbpf/
+LIBBPF_BOOTSTRAP_DESTDIR := $(LIBBPF_BOOTSTRAP_OUTPUT)
+LIBBPF_BOOTSTRAP_INCLUDE := $(LIBBPF_BOOTSTRAP_DESTDIR)/include
+LIBBPF_BOOTSTRAP_HDRS_DIR := $(LIBBPF_BOOTSTRAP_INCLUDE)/bpf
+LIBBPF_BOOTSTRAP := $(LIBBPF_BOOTSTRAP_OUTPUT)libbpf.a
# We need to copy hashmap.h and nlattr.h which is not otherwise exported by
# libbpf, but still required by bpftool.
LIBBPF_INTERNAL_HDRS := $(addprefix $(LIBBPF_HDRS_DIR)/,hashmap.h nlattr.h)
+LIBBPF_BOOTSTRAP_INTERNAL_HDRS := $(addprefix $(LIBBPF_BOOTSTRAP_HDRS_DIR)/,hashmap.h)
ifeq ($(BPFTOOL_VERSION),)
BPFTOOL_VERSION := $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
endif
-$(LIBBPF_OUTPUT) $(BOOTSTRAP_OUTPUT) $(LIBBPF_BOOTSTRAP_OUTPUT) $(LIBBPF_HDRS_DIR):
+$(LIBBPF_OUTPUT) $(BOOTSTRAP_OUTPUT) $(LIBBPF_BOOTSTRAP_OUTPUT) $(LIBBPF_HDRS_DIR) $(LIBBPF_BOOTSTRAP_HDRS_DIR):
$(QUIET_MKDIR)mkdir -p $@
$(LIBBPF): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_OUTPUT)
$(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT)
$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
- ARCH= CC=$(HOSTCC) LD=$(HOSTLD) $@
+ DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR) prefix= \
+ ARCH= CC=$(HOSTCC) LD=$(HOSTLD) $@ install_headers
+
+$(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR)
+ $(call QUIET_INSTALL, $@)
+ $(Q)install -m 644 -t $(LIBBPF_BOOTSTRAP_HDRS_DIR) $<
$(LIBBPF)-clean: FORCE | $(LIBBPF_OUTPUT)
$(call QUIET_CLEAN, libbpf)
$(Q)cp "$(VMLINUX_H)" $@
endif
-$(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF)
+$(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF_BOOTSTRAP)
$(QUIET_CLANG)$(CLANG) \
-I$(if $(OUTPUT),$(OUTPUT),.) \
-I$(srctree)/tools/include/uapi/ \
- -I$(LIBBPF_INCLUDE) \
+ -I$(LIBBPF_BOOTSTRAP_INCLUDE) \
-g -O2 -Wall -target bpf -c $< -o $@ && $(LLVM_STRIP) -g $@
$(OUTPUT)%.skel.h: $(OUTPUT)%.bpf.o $(BPFTOOL_BOOTSTRAP)
$(OUTPUT)bpftool: $(OBJS) $(LIBBPF)
$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS)
-$(BOOTSTRAP_OUTPUT)%.o: %.c $(LIBBPF_INTERNAL_HDRS) | $(BOOTSTRAP_OUTPUT)
- $(QUIET_CC)$(HOSTCC) $(CFLAGS) -c -MMD -o $@ $<
+$(BOOTSTRAP_OUTPUT)%.o: %.c $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS) | $(BOOTSTRAP_OUTPUT)
+ $(QUIET_CC)$(HOSTCC) \
+ $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),$(CFLAGS)) \
+ -c -MMD -o $@ $<
$(OUTPUT)%.o: %.c
$(QUIET_CC)$(CC) $(CFLAGS) -c -MMD -o $@ $<
FORCE:
.SECONDARY:
-.PHONY: all FORCE clean install-bin install uninstall
+.PHONY: all FORCE bootstrap clean install-bin install uninstall
.PHONY: doc doc-clean doc-install doc-uninstall
.DEFAULT_GOAL := all
$(DEFAULT_BPFTOOL): $(BPFOBJ) | $(BPFTOOL_OUTPUT)
$(Q)$(MAKE) $(submake_extras) -C ../bpftool OUTPUT=$(BPFTOOL_OUTPUT) \
- LIBBPF_OUTPUT=$(BPFOBJ_OUTPUT) \
- LIBBPF_DESTDIR=$(BPF_DESTDIR) CC=$(HOSTCC) LD=$(HOSTLD)
+ CC=$(HOSTCC) LD=$(HOSTLD)
libslang \
libslang-include-subdir \
libtraceevent \
+ libtracefs \
libcrypto \
libunwind \
pthread-attr-setaffinity-np \
test-libslang.bin \
test-libslang-include-subdir.bin \
test-libtraceevent.bin \
+ test-libtracefs.bin \
test-libcrypto.bin \
test-libunwind.bin \
test-libunwind-debug-frame.bin \
###############################
$(OUTPUT)test-all.bin:
- $(BUILD) -fstack-protector-all -O2 -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -I/usr/include/slang -lslang $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -llzma -lzstd -lcap
+ $(BUILD) -fstack-protector-all -O2 -D_FORTIFY_SOURCE=2 -ldw -lelf -lnuma -lelf -lslang $(FLAGS_PERL_EMBED) $(FLAGS_PYTHON_EMBED) -DPACKAGE='"perf"' -lbfd -ldl -lz -llzma -lzstd -lcap
$(OUTPUT)test-hello.bin:
$(BUILD)
$(OUTPUT)test-libtraceevent.bin:
$(BUILD) -ltraceevent
+$(OUTPUT)test-libtracefs.bin:
+ $(BUILD) -ltracefs
+
$(OUTPUT)test-libcrypto.bin:
$(BUILD) -lcrypto
$(BUILD)
$(OUTPUT)test-llvm.bin:
- $(BUILDXX) -std=gnu++11 \
+ $(BUILDXX) -std=gnu++14 \
-I$(shell $(LLVM_CONFIG) --includedir) \
-L$(shell $(LLVM_CONFIG) --libdir) \
$(shell $(LLVM_CONFIG) --libs Core BPF) \
> $(@:.bin=.make.output) 2>&1
$(OUTPUT)test-llvm-version.bin:
- $(BUILDXX) -std=gnu++11 \
+ $(BUILDXX) -std=gnu++14 \
-I$(shell $(LLVM_CONFIG) --includedir) \
> $(@:.bin=.make.output) 2>&1
$(OUTPUT)test-clang.bin:
- $(BUILDXX) -std=gnu++11 \
+ $(BUILDXX) -std=gnu++14 \
-I$(shell $(LLVM_CONFIG) --includedir) \
-L$(shell $(LLVM_CONFIG) --libdir) \
-Wl,--start-group -lclangBasic -lclangDriver \
main_test_timerfd();
main_test_stackprotector_all();
main_test_libdw_dwarf_unwind();
- main_test_sync_compare_and_swap(argc, argv);
main_test_zlib();
main_test_pthread_attr_setaffinity_np();
main_test_pthread_barrier();
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <tracefs/tracefs.h>
+
+int main(void)
+{
+ struct tracefs_instance *inst = tracefs_instance_create("dummy");
+
+ tracefs_instance_destroy(inst);
+ return 0;
+}
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_LIST_SORT_H
+#define _LINUX_LIST_SORT_H
+
+#include <linux/types.h>
+
+struct list_head;
+
+typedef int __attribute__((nonnull(2,3))) (*list_cmp_func_t)(void *,
+ const struct list_head *, const struct list_head *);
+
+__attribute__((nonnull(2,3)))
+void list_sort(void *priv, struct list_head *head, list_cmp_func_t cmp);
+#endif
#define __NR_process_mrelease 448
__SYSCALL(__NR_process_mrelease, sys_process_mrelease)
+#define __NR_futex_waitv 449
+__SYSCALL(__NR_futex_waitv, sys_futex_waitv)
+
#undef __NR_syscalls
-#define __NR_syscalls 449
+#define __NR_syscalls 450
/*
* 32 bit systems traditionally used different
#define I915_TILING_NONE 0
#define I915_TILING_X 1
#define I915_TILING_Y 2
+/*
+ * Do not add new tiling types here. The I915_TILING_* values are for
+ * de-tiling fence registers that no longer exist on modern platforms. Although
+ * the hardware may support new types of tiling in general (e.g., Tile4), we
+ * do not need to add them to the uapi that is specific to now-defunct ioctls.
+ */
#define I915_TILING_LAST I915_TILING_Y
#define I915_BIT_6_SWIZZLE_NONE 0
* Extensions:
* i915_context_engines_load_balance (I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE)
* i915_context_engines_bond (I915_CONTEXT_ENGINES_EXT_BOND)
+ * i915_context_engines_parallel_submit (I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT)
*/
#define I915_CONTEXT_PARAM_ENGINES 0xa
* attempted to use it, never re-use this context param number.
*/
#define I915_CONTEXT_PARAM_RINGSIZE 0xc
+
+/*
+ * I915_CONTEXT_PARAM_PROTECTED_CONTENT:
+ *
+ * Mark that the context makes use of protected content, which will result
+ * in the context being invalidated when the protected content session is.
+ * Given that the protected content session is killed on suspend, the device
+ * is kept awake for the lifetime of a protected context, so the user should
+ * make sure to dispose of them once done.
+ * This flag can only be set at context creation time and, when set to true,
+ * must be preceded by an explicit setting of I915_CONTEXT_PARAM_RECOVERABLE
+ * to false. This flag can't be set to true in conjunction with setting the
+ * I915_CONTEXT_PARAM_BANNABLE flag to false. Creation example:
+ *
+ * .. code-block:: C
+ *
+ * struct drm_i915_gem_context_create_ext_setparam p_protected = {
+ * .base = {
+ * .name = I915_CONTEXT_CREATE_EXT_SETPARAM,
+ * },
+ * .param = {
+ * .param = I915_CONTEXT_PARAM_PROTECTED_CONTENT,
+ * .value = 1,
+ * }
+ * };
+ * struct drm_i915_gem_context_create_ext_setparam p_norecover = {
+ * .base = {
+ * .name = I915_CONTEXT_CREATE_EXT_SETPARAM,
+ * .next_extension = to_user_pointer(&p_protected),
+ * },
+ * .param = {
+ * .param = I915_CONTEXT_PARAM_RECOVERABLE,
+ * .value = 0,
+ * }
+ * };
+ * struct drm_i915_gem_context_create_ext create = {
+ * .flags = I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS,
+ * .extensions = to_user_pointer(&p_norecover);
+ * };
+ *
+ * ctx_id = gem_context_create_ext(drm_fd, &create);
+ *
+ * In addition to the normal failure cases, setting this flag during context
+ * creation can result in the following errors:
+ *
+ * -ENODEV: feature not available
+ * -EPERM: trying to mark a recoverable or not bannable context as protected
+ */
+#define I915_CONTEXT_PARAM_PROTECTED_CONTENT 0xd
/* Must be kept compact -- no holes and well documented */
__u64 value;
} __attribute__((packed)) name__
/**
+ * struct i915_context_engines_parallel_submit - Configure engine for
+ * parallel submission.
+ *
+ * Setup a slot in the context engine map to allow multiple BBs to be submitted
+ * in a single execbuf IOCTL. Those BBs will then be scheduled to run on the GPU
+ * in parallel. Multiple hardware contexts are created internally in the i915 to
+ * run these BBs. Once a slot is configured for N BBs only N BBs can be
+ * submitted in each execbuf IOCTL and this is implicit behavior e.g. The user
+ * doesn't tell the execbuf IOCTL there are N BBs, the execbuf IOCTL knows how
+ * many BBs there are based on the slot's configuration. The N BBs are the last
+ * N buffer objects or first N if I915_EXEC_BATCH_FIRST is set.
+ *
+ * The default placement behavior is to create implicit bonds between each
+ * context if each context maps to more than 1 physical engine (e.g. context is
+ * a virtual engine). Also we only allow contexts of same engine class and these
+ * contexts must be in logically contiguous order. Examples of the placement
+ * behavior are described below. Lastly, the default is to not allow BBs to be
+ * preempted mid-batch. Rather insert coordinated preemption points on all
+ * hardware contexts between each set of BBs. Flags could be added in the future
+ * to change both of these default behaviors.
+ *
+ * Returns -EINVAL if hardware context placement configuration is invalid or if
+ * the placement configuration isn't supported on the platform / submission
+ * interface.
+ * Returns -ENODEV if extension isn't supported on the platform / submission
+ * interface.
+ *
+ * .. code-block:: none
+ *
+ * Examples syntax:
+ * CS[X] = generic engine of same class, logical instance X
+ * INVALID = I915_ENGINE_CLASS_INVALID, I915_ENGINE_CLASS_INVALID_NONE
+ *
+ * Example 1 pseudo code:
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=1,
+ * engines=CS[0],CS[1])
+ *
+ * Results in the following valid placement:
+ * CS[0], CS[1]
+ *
+ * Example 2 pseudo code:
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CS[0],CS[2],CS[1],CS[3])
+ *
+ * Results in the following valid placements:
+ * CS[0], CS[1]
+ * CS[2], CS[3]
+ *
+ * This can be thought of as two virtual engines, each containing two
+ * engines thereby making a 2D array. However, there are bonds tying the
+ * entries together and placing restrictions on how they can be scheduled.
+ * Specifically, the scheduler can choose only vertical columns from the 2D
+ * array. That is, CS[0] is bonded to CS[1] and CS[2] to CS[3]. So if the
+ * scheduler wants to submit to CS[0], it must also choose CS[1] and vice
+ * versa. Same for CS[2] requires also using CS[3].
+ * VE[0] = CS[0], CS[2]
+ * VE[1] = CS[1], CS[3]
+ *
+ * Example 3 pseudo code:
+ * set_engines(INVALID)
+ * set_parallel(engine_index=0, width=2, num_siblings=2,
+ * engines=CS[0],CS[1],CS[1],CS[3])
+ *
+ * Results in the following valid and invalid placements:
+ * CS[0], CS[1]
+ * CS[1], CS[3] - Not logically contiguous, return -EINVAL
+ */
+struct i915_context_engines_parallel_submit {
+ /**
+ * @base: base user extension.
+ */
+ struct i915_user_extension base;
+
+ /**
+ * @engine_index: slot for parallel engine
+ */
+ __u16 engine_index;
+
+ /**
+ * @width: number of contexts per parallel engine or in other words the
+ * number of batches in each submission
+ */
+ __u16 width;
+
+ /**
+ * @num_siblings: number of siblings per context or in other words the
+ * number of possible placements for each submission
+ */
+ __u16 num_siblings;
+
+ /**
+ * @mbz16: reserved for future use; must be zero
+ */
+ __u16 mbz16;
+
+ /**
+ * @flags: all undefined flags must be zero, currently not defined flags
+ */
+ __u64 flags;
+
+ /**
+ * @mbz64: reserved for future use; must be zero
+ */
+ __u64 mbz64[3];
+
+ /**
+ * @engines: 2-d array of engine instances to configure parallel engine
+ *
+ * length = width (i) * num_siblings (j)
+ * index = j + i * num_siblings
+ */
+ struct i915_engine_class_instance engines[0];
+
+} __packed;
+
+#define I915_DEFINE_CONTEXT_ENGINES_PARALLEL_SUBMIT(name__, N__) struct { \
+ struct i915_user_extension base; \
+ __u16 engine_index; \
+ __u16 width; \
+ __u16 num_siblings; \
+ __u16 mbz16; \
+ __u64 flags; \
+ __u64 mbz64[3]; \
+ struct i915_engine_class_instance engines[N__]; \
+} __attribute__((packed)) name__
+
+/**
* DOC: Context Engine Map uAPI
*
* Context engine map is a new way of addressing engines when submitting batch-
__u64 extensions; /* linked chain of extension blocks, 0 terminates */
#define I915_CONTEXT_ENGINES_EXT_LOAD_BALANCE 0 /* see i915_context_engines_load_balance */
#define I915_CONTEXT_ENGINES_EXT_BOND 1 /* see i915_context_engines_bond */
+#define I915_CONTEXT_ENGINES_EXT_PARALLEL_SUBMIT 2 /* see i915_context_engines_parallel_submit */
struct i915_engine_class_instance engines[0];
} __attribute__((packed));
/** @flags: Engine flags. */
__u64 flags;
+#define I915_ENGINE_INFO_HAS_LOGICAL_INSTANCE (1 << 0)
/** @capabilities: Capabilities of this engine. */
__u64 capabilities;
#define I915_VIDEO_CLASS_CAPABILITY_HEVC (1 << 0)
#define I915_VIDEO_AND_ENHANCE_CLASS_CAPABILITY_SFC (1 << 1)
+ /** @logical_instance: Logical instance of engine */
+ __u16 logical_instance;
+
/** @rsvd1: Reserved fields. */
- __u64 rsvd1[4];
+ __u16 rsvd1[3];
+ /** @rsvd2: Reserved fields. */
+ __u64 rsvd2[3];
};
/**
*
* For I915_GEM_CREATE_EXT_MEMORY_REGIONS usage see
* struct drm_i915_gem_create_ext_memory_regions.
+ *
+ * For I915_GEM_CREATE_EXT_PROTECTED_CONTENT usage see
+ * struct drm_i915_gem_create_ext_protected_content.
*/
#define I915_GEM_CREATE_EXT_MEMORY_REGIONS 0
+#define I915_GEM_CREATE_EXT_PROTECTED_CONTENT 1
__u64 extensions;
};
__u64 regions;
};
+/**
+ * struct drm_i915_gem_create_ext_protected_content - The
+ * I915_OBJECT_PARAM_PROTECTED_CONTENT extension.
+ *
+ * If this extension is provided, buffer contents are expected to be protected
+ * by PXP encryption and require decryption for scan out and processing. This
+ * is only possible on platforms that have PXP enabled, on all other scenarios
+ * using this extension will cause the ioctl to fail and return -ENODEV. The
+ * flags parameter is reserved for future expansion and must currently be set
+ * to zero.
+ *
+ * The buffer contents are considered invalid after a PXP session teardown.
+ *
+ * The encryption is guaranteed to be processed correctly only if the object
+ * is submitted with a context created using the
+ * I915_CONTEXT_PARAM_PROTECTED_CONTENT flag. This will also enable extra checks
+ * at submission time on the validity of the objects involved.
+ *
+ * Below is an example on how to create a protected object:
+ *
+ * .. code-block:: C
+ *
+ * struct drm_i915_gem_create_ext_protected_content protected_ext = {
+ * .base = { .name = I915_GEM_CREATE_EXT_PROTECTED_CONTENT },
+ * .flags = 0,
+ * };
+ * struct drm_i915_gem_create_ext create_ext = {
+ * .size = PAGE_SIZE,
+ * .extensions = (uintptr_t)&protected_ext,
+ * };
+ *
+ * int err = ioctl(fd, DRM_IOCTL_I915_GEM_CREATE_EXT, &create_ext);
+ * if (err) ...
+ */
+struct drm_i915_gem_create_ext_protected_content {
+ /** @base: Extension link. See struct i915_user_extension. */
+ struct i915_user_extension base;
+ /** @flags: reserved for future usage, currently MBZ */
+ __u32 flags;
+};
+
+/* ID of the protected content session managed by i915 when PXP is active */
+#define I915_PROTECTED_CONTENT_DEFAULT_SESSION 0xf
+
#if defined(__cplusplus)
}
#endif
#define KVM_EXIT_AP_RESET_HOLD 32
#define KVM_EXIT_X86_BUS_LOCK 33
#define KVM_EXIT_XEN 34
+#define KVM_EXIT_RISCV_SBI 35
/* For KVM_EXIT_INTERNAL_ERROR */
/* Emulate instruction failed. */
* "ndata" is correct, that new fields are enumerated in "flags",
* and that each flag enumerates fields that are 64-bit aligned
* and sized (so that ndata+internal.data[] is valid/accurate).
+ *
+ * Space beyond the defined fields may be used to store arbitrary
+ * debug information relating to the emulation failure. It is
+ * accounted for in "ndata" but the format is unspecified and is
+ * not represented in "flags". Any such information is *not* ABI!
*/
struct {
__u32 suberror;
__u32 ndata;
__u64 flags;
- __u8 insn_size;
- __u8 insn_bytes[15];
+ union {
+ struct {
+ __u8 insn_size;
+ __u8 insn_bytes[15];
+ };
+ };
+ /* Arbitrary debug data may follow. */
} emulation_failure;
/* KVM_EXIT_OSI */
struct {
} msr;
/* KVM_EXIT_XEN */
struct kvm_xen_exit xen;
+ /* KVM_EXIT_RISCV_SBI */
+ struct {
+ unsigned long extension_id;
+ unsigned long function_id;
+ unsigned long args[6];
+ unsigned long ret[2];
+ } riscv_sbi;
/* Fix the size of the union. */
char padding[256];
};
#define KVM_CAP_BINARY_STATS_FD 203
#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204
#define KVM_CAP_ARM_MTE 205
+#define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206
#ifdef KVM_CAP_IRQ_ROUTING
/* Do not use 1, KVM_CHECK_EXTENSION returned it before we had flags. */
#define KVM_CLOCK_TSC_STABLE 2
+#define KVM_CLOCK_REALTIME (1 << 2)
+#define KVM_CLOCK_HOST_TSC (1 << 3)
struct kvm_clock_data {
__u64 clock;
__u32 flags;
- __u32 pad[9];
+ __u32 pad0;
+ __u64 realtime;
+ __u64 host_tsc;
+ __u32 pad[4];
};
/* For KVM_CAP_SW_TLB */
*/
PERF_RECORD_TEXT_POKE = 20,
+ /*
+ * Data written to the AUX area by hardware due to aux_output, may need
+ * to be matched to the event by an architecture-specific hardware ID.
+ * This records the hardware ID, but requires sample_id to provide the
+ * event ID. e.g. Intel PT uses this record to disambiguate PEBS-via-PT
+ * records from multiple events.
+ *
+ * struct {
+ * struct perf_event_header header;
+ * u64 hw_id;
+ * struct sample_id sample_id;
+ * };
+ */
+ PERF_RECORD_AUX_OUTPUT_HW_ID = 21,
+
PERF_RECORD_MAX, /* non-ABI */
};
#define PR_GET_TAGGED_ADDR_CTRL 56
# define PR_TAGGED_ADDR_ENABLE (1UL << 0)
/* MTE tag check fault modes */
-# define PR_MTE_TCF_NONE 0
+# define PR_MTE_TCF_NONE 0UL
# define PR_MTE_TCF_SYNC (1UL << 1)
# define PR_MTE_TCF_ASYNC (1UL << 2)
# define PR_MTE_TCF_MASK (PR_MTE_TCF_SYNC | PR_MTE_TCF_ASYNC)
# define PR_SCHED_CORE_SHARE_TO 2 /* push core_sched cookie to pid */
# define PR_SCHED_CORE_SHARE_FROM 3 /* pull core_sched cookie to pid */
# define PR_SCHED_CORE_MAX 4
+# define PR_SCHED_CORE_SCOPE_THREAD 0
+# define PR_SCHED_CORE_SCOPE_THREAD_GROUP 1
+# define PR_SCHED_CORE_SCOPE_PROCESS_GROUP 2
#endif /* _LINUX_PRCTL_H */
#define SNDRV_CTL_ELEM_ACCESS_WRITE (1<<1)
#define SNDRV_CTL_ELEM_ACCESS_READWRITE (SNDRV_CTL_ELEM_ACCESS_READ|SNDRV_CTL_ELEM_ACCESS_WRITE)
#define SNDRV_CTL_ELEM_ACCESS_VOLATILE (1<<2) /* control value may be changed without a notification */
-// (1 << 3) is unused.
+/* (1 << 3) is unused. */
#define SNDRV_CTL_ELEM_ACCESS_TLV_READ (1<<4) /* TLV read is possible */
#define SNDRV_CTL_ELEM_ACCESS_TLV_WRITE (1<<5) /* TLV write is possible */
#define SNDRV_CTL_ELEM_ACCESS_TLV_READWRITE (SNDRV_CTL_ELEM_ACCESS_TLV_READ|SNDRV_CTL_ELEM_ACCESS_TLV_WRITE)
int bpf_map_lookup_and_delete_elem_flags(int fd, const void *key, void *value, __u64 flags)
{
union bpf_attr attr;
+ int ret;
memset(&attr, 0, sizeof(attr));
attr.map_fd = fd;
attr.value = ptr_to_u64(value);
attr.flags = flags;
- return sys_bpf(BPF_MAP_LOOKUP_AND_DELETE_ELEM, &attr, sizeof(attr));
+ ret = sys_bpf(BPF_MAP_LOOKUP_AND_DELETE_ELEM, &attr, sizeof(attr));
+ return libbpf_err_errno(ret);
}
int bpf_map_delete_elem(int fd, const void *key)
int nr_fd_array;
};
-void bpf_gen__init(struct bpf_gen *gen, int log_level);
-int bpf_gen__finish(struct bpf_gen *gen);
+void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps);
+int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps);
void bpf_gen__free(struct bpf_gen *gen);
void bpf_gen__load_btf(struct bpf_gen *gen, const void *raw_data, __u32 raw_size);
void bpf_gen__map_create(struct bpf_gen *gen, struct bpf_create_map_params *map_attr, int map_idx);
#define MAX_USED_MAPS 64
#define MAX_USED_PROGS 32
#define MAX_KFUNC_DESCS 256
-#define MAX_FD_ARRAY_SZ (MAX_USED_PROGS + MAX_KFUNC_DESCS)
+#define MAX_FD_ARRAY_SZ (MAX_USED_MAPS + MAX_KFUNC_DESCS)
/* The following structure describes the stack layout of the loader program.
* In addition R6 contains the pointer to context.
*/
struct loader_stack {
__u32 btf_fd;
- __u32 prog_fd[MAX_USED_PROGS];
__u32 inner_map_fd;
+ __u32 prog_fd[MAX_USED_PROGS];
};
#define stack_off(field) \
#define attr_field(attr, field) (attr + offsetof(union bpf_attr, field))
+static int blob_fd_array_off(struct bpf_gen *gen, int index)
+{
+ return gen->fd_array + index * sizeof(int);
+}
+
static int realloc_insn_buf(struct bpf_gen *gen, __u32 size)
{
size_t off = gen->insn_cur - gen->insn_start;
emit(gen, insn2);
}
-void bpf_gen__init(struct bpf_gen *gen, int log_level)
+static int add_data(struct bpf_gen *gen, const void *data, __u32 size);
+static void emit_sys_close_blob(struct bpf_gen *gen, int blob_off);
+
+void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps)
{
- size_t stack_sz = sizeof(struct loader_stack);
+ size_t stack_sz = sizeof(struct loader_stack), nr_progs_sz;
int i;
+ gen->fd_array = add_data(gen, NULL, MAX_FD_ARRAY_SZ * sizeof(int));
gen->log_level = log_level;
/* save ctx pointer into R6 */
emit(gen, BPF_MOV64_REG(BPF_REG_6, BPF_REG_1));
emit(gen, BPF_MOV64_IMM(BPF_REG_3, 0));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel));
+ /* amount of stack actually used, only used to calculate iterations, not stack offset */
+ nr_progs_sz = offsetof(struct loader_stack, prog_fd[nr_progs]);
/* jump over cleanup code */
emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0,
- /* size of cleanup code below */
- (stack_sz / 4) * 3 + 2));
+ /* size of cleanup code below (including map fd cleanup) */
+ (nr_progs_sz / 4) * 3 + 2 +
+ /* 6 insns for emit_sys_close_blob,
+ * 6 insns for debug_regs in emit_sys_close_blob
+ */
+ nr_maps * (6 + (gen->log_level ? 6 : 0))));
/* remember the label where all error branches will jump to */
gen->cleanup_label = gen->insn_cur - gen->insn_start;
/* emit cleanup code: close all temp FDs */
- for (i = 0; i < stack_sz; i += 4) {
+ for (i = 0; i < nr_progs_sz; i += 4) {
emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_1, BPF_REG_10, -stack_sz + i));
emit(gen, BPF_JMP_IMM(BPF_JSLE, BPF_REG_1, 0, 1));
emit(gen, BPF_EMIT_CALL(BPF_FUNC_sys_close));
}
+ for (i = 0; i < nr_maps; i++)
+ emit_sys_close_blob(gen, blob_fd_array_off(gen, i));
/* R7 contains the error code from sys_bpf. Copy it into R0 and exit. */
emit(gen, BPF_MOV64_REG(BPF_REG_0, BPF_REG_7));
emit(gen, BPF_EXIT_INSN());
*/
static int add_map_fd(struct bpf_gen *gen)
{
- if (!gen->fd_array)
- gen->fd_array = add_data(gen, NULL, MAX_FD_ARRAY_SZ * sizeof(int));
if (gen->nr_maps == MAX_USED_MAPS) {
pr_warn("Total maps exceeds %d\n", MAX_USED_MAPS);
gen->error = -E2BIG;
{
int cur;
- if (!gen->fd_array)
- gen->fd_array = add_data(gen, NULL, MAX_FD_ARRAY_SZ * sizeof(int));
if (gen->nr_fd_array == MAX_KFUNC_DESCS) {
cur = add_data(gen, NULL, sizeof(int));
return (cur - gen->fd_array) / sizeof(int);
return MAX_USED_MAPS + gen->nr_fd_array++;
}
-static int blob_fd_array_off(struct bpf_gen *gen, int index)
-{
- return gen->fd_array + index * sizeof(int);
-}
-
static int insn_bytes_to_bpf_size(__u32 sz)
{
switch (sz) {
__emit_sys_close(gen);
}
-int bpf_gen__finish(struct bpf_gen *gen)
+int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps)
{
int i;
+ if (nr_progs != gen->nr_progs || nr_maps != gen->nr_maps) {
+ pr_warn("progs/maps mismatch\n");
+ gen->error = -EFAULT;
+ return gen->error;
+ }
emit_sys_close_stack(gen, stack_off(btf_fd));
for (i = 0; i < gen->nr_progs; i++)
move_stack2ctx(gen,
}
if (obj->gen_loader)
- bpf_gen__init(obj->gen_loader, attr->log_level);
+ bpf_gen__init(obj->gen_loader, attr->log_level, obj->nr_programs, obj->nr_maps);
err = bpf_object__probe_loading(obj);
err = err ? : bpf_object__load_vmlinux_btf(obj, false);
for (i = 0; i < obj->nr_maps; i++)
obj->maps[i].fd = -1;
if (!err)
- err = bpf_gen__finish(obj->gen_loader);
+ err = bpf_gen__finish(obj->gen_loader, obj->nr_programs, obj->nr_maps);
}
/* clean up fd_array */
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/kernel.h>
+#include <linux/compiler.h>
+#include <linux/export.h>
+#include <linux/string.h>
+#include <linux/list_sort.h>
+#include <linux/list.h>
+
+/*
+ * Returns a list organized in an intermediate format suited
+ * to chaining of merge() calls: null-terminated, no reserved or
+ * sentinel head node, "prev" links not maintained.
+ */
+__attribute__((nonnull(2,3,4)))
+static struct list_head *merge(void *priv, list_cmp_func_t cmp,
+ struct list_head *a, struct list_head *b)
+{
+ struct list_head *head, **tail = &head;
+
+ for (;;) {
+ /* if equal, take 'a' -- important for sort stability */
+ if (cmp(priv, a, b) <= 0) {
+ *tail = a;
+ tail = &a->next;
+ a = a->next;
+ if (!a) {
+ *tail = b;
+ break;
+ }
+ } else {
+ *tail = b;
+ tail = &b->next;
+ b = b->next;
+ if (!b) {
+ *tail = a;
+ break;
+ }
+ }
+ }
+ return head;
+}
+
+/*
+ * Combine final list merge with restoration of standard doubly-linked
+ * list structure. This approach duplicates code from merge(), but
+ * runs faster than the tidier alternatives of either a separate final
+ * prev-link restoration pass, or maintaining the prev links
+ * throughout.
+ */
+__attribute__((nonnull(2,3,4,5)))
+static void merge_final(void *priv, list_cmp_func_t cmp, struct list_head *head,
+ struct list_head *a, struct list_head *b)
+{
+ struct list_head *tail = head;
+ u8 count = 0;
+
+ for (;;) {
+ /* if equal, take 'a' -- important for sort stability */
+ if (cmp(priv, a, b) <= 0) {
+ tail->next = a;
+ a->prev = tail;
+ tail = a;
+ a = a->next;
+ if (!a)
+ break;
+ } else {
+ tail->next = b;
+ b->prev = tail;
+ tail = b;
+ b = b->next;
+ if (!b) {
+ b = a;
+ break;
+ }
+ }
+ }
+
+ /* Finish linking remainder of list b on to tail */
+ tail->next = b;
+ do {
+ /*
+ * If the merge is highly unbalanced (e.g. the input is
+ * already sorted), this loop may run many iterations.
+ * Continue callbacks to the client even though no
+ * element comparison is needed, so the client's cmp()
+ * routine can invoke cond_resched() periodically.
+ */
+ if (unlikely(!++count))
+ cmp(priv, b, b);
+ b->prev = tail;
+ tail = b;
+ b = b->next;
+ } while (b);
+
+ /* And the final links to make a circular doubly-linked list */
+ tail->next = head;
+ head->prev = tail;
+}
+
+/**
+ * list_sort - sort a list
+ * @priv: private data, opaque to list_sort(), passed to @cmp
+ * @head: the list to sort
+ * @cmp: the elements comparison function
+ *
+ * The comparison function @cmp must return > 0 if @a should sort after
+ * @b ("@a > @b" if you want an ascending sort), and <= 0 if @a should
+ * sort before @b *or* their original order should be preserved. It is
+ * always called with the element that came first in the input in @a,
+ * and list_sort is a stable sort, so it is not necessary to distinguish
+ * the @a < @b and @a == @b cases.
+ *
+ * This is compatible with two styles of @cmp function:
+ * - The traditional style which returns <0 / =0 / >0, or
+ * - Returning a boolean 0/1.
+ * The latter offers a chance to save a few cycles in the comparison
+ * (which is used by e.g. plug_ctx_cmp() in block/blk-mq.c).
+ *
+ * A good way to write a multi-word comparison is::
+ *
+ * if (a->high != b->high)
+ * return a->high > b->high;
+ * if (a->middle != b->middle)
+ * return a->middle > b->middle;
+ * return a->low > b->low;
+ *
+ *
+ * This mergesort is as eager as possible while always performing at least
+ * 2:1 balanced merges. Given two pending sublists of size 2^k, they are
+ * merged to a size-2^(k+1) list as soon as we have 2^k following elements.
+ *
+ * Thus, it will avoid cache thrashing as long as 3*2^k elements can
+ * fit into the cache. Not quite as good as a fully-eager bottom-up
+ * mergesort, but it does use 0.2*n fewer comparisons, so is faster in
+ * the common case that everything fits into L1.
+ *
+ *
+ * The merging is controlled by "count", the number of elements in the
+ * pending lists. This is beautifully simple code, but rather subtle.
+ *
+ * Each time we increment "count", we set one bit (bit k) and clear
+ * bits k-1 .. 0. Each time this happens (except the very first time
+ * for each bit, when count increments to 2^k), we merge two lists of
+ * size 2^k into one list of size 2^(k+1).
+ *
+ * This merge happens exactly when the count reaches an odd multiple of
+ * 2^k, which is when we have 2^k elements pending in smaller lists,
+ * so it's safe to merge away two lists of size 2^k.
+ *
+ * After this happens twice, we have created two lists of size 2^(k+1),
+ * which will be merged into a list of size 2^(k+2) before we create
+ * a third list of size 2^(k+1), so there are never more than two pending.
+ *
+ * The number of pending lists of size 2^k is determined by the
+ * state of bit k of "count" plus two extra pieces of information:
+ *
+ * - The state of bit k-1 (when k == 0, consider bit -1 always set), and
+ * - Whether the higher-order bits are zero or non-zero (i.e.
+ * is count >= 2^(k+1)).
+ *
+ * There are six states we distinguish. "x" represents some arbitrary
+ * bits, and "y" represents some arbitrary non-zero bits:
+ * 0: 00x: 0 pending of size 2^k; x pending of sizes < 2^k
+ * 1: 01x: 0 pending of size 2^k; 2^(k-1) + x pending of sizes < 2^k
+ * 2: x10x: 0 pending of size 2^k; 2^k + x pending of sizes < 2^k
+ * 3: x11x: 1 pending of size 2^k; 2^(k-1) + x pending of sizes < 2^k
+ * 4: y00x: 1 pending of size 2^k; 2^k + x pending of sizes < 2^k
+ * 5: y01x: 2 pending of size 2^k; 2^(k-1) + x pending of sizes < 2^k
+ * (merge and loop back to state 2)
+ *
+ * We gain lists of size 2^k in the 2->3 and 4->5 transitions (because
+ * bit k-1 is set while the more significant bits are non-zero) and
+ * merge them away in the 5->2 transition. Note in particular that just
+ * before the 5->2 transition, all lower-order bits are 11 (state 3),
+ * so there is one list of each smaller size.
+ *
+ * When we reach the end of the input, we merge all the pending
+ * lists, from smallest to largest. If you work through cases 2 to
+ * 5 above, you can see that the number of elements we merge with a list
+ * of size 2^k varies from 2^(k-1) (cases 3 and 5 when x == 0) to
+ * 2^(k+1) - 1 (second merge of case 5 when x == 2^(k-1) - 1).
+ */
+__attribute__((nonnull(2,3)))
+void list_sort(void *priv, struct list_head *head, list_cmp_func_t cmp)
+{
+ struct list_head *list = head->next, *pending = NULL;
+ size_t count = 0; /* Count of pending */
+
+ if (list == head->prev) /* Zero or one elements */
+ return;
+
+ /* Convert to a null-terminated singly-linked list. */
+ head->prev->next = NULL;
+
+ /*
+ * Data structure invariants:
+ * - All lists are singly linked and null-terminated; prev
+ * pointers are not maintained.
+ * - pending is a prev-linked "list of lists" of sorted
+ * sublists awaiting further merging.
+ * - Each of the sorted sublists is power-of-two in size.
+ * - Sublists are sorted by size and age, smallest & newest at front.
+ * - There are zero to two sublists of each size.
+ * - A pair of pending sublists are merged as soon as the number
+ * of following pending elements equals their size (i.e.
+ * each time count reaches an odd multiple of that size).
+ * That ensures each later final merge will be at worst 2:1.
+ * - Each round consists of:
+ * - Merging the two sublists selected by the highest bit
+ * which flips when count is incremented, and
+ * - Adding an element from the input as a size-1 sublist.
+ */
+ do {
+ size_t bits;
+ struct list_head **tail = &pending;
+
+ /* Find the least-significant clear bit in count */
+ for (bits = count; bits & 1; bits >>= 1)
+ tail = &(*tail)->prev;
+ /* Do the indicated merge */
+ if (likely(bits)) {
+ struct list_head *a = *tail, *b = a->prev;
+
+ a = merge(priv, cmp, b, a);
+ /* Install the merged result in place of the inputs */
+ a->prev = b->prev;
+ *tail = a;
+ }
+
+ /* Move one element from input list to pending */
+ list->prev = pending;
+ pending = list;
+ list = list->next;
+ pending->next = NULL;
+ count++;
+ } while (list);
+
+ /* End of input; merge together all the pending lists. */
+ list = pending;
+ pending = pending->prev;
+ for (;;) {
+ struct list_head *next = pending->prev;
+
+ if (!next)
+ break;
+ list = merge(priv, cmp, pending, list);
+ pending = next;
+ }
+ /* The final merge, rebuilding prev links */
+ merge_final(priv, cmp, head, pending, list);
+}
+EXPORT_SYMBOL(list_sort);
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0-only
-liblockdep.so.*
+++ /dev/null
-liblockdep-y += common.o lockdep.o preload.o rbtree.o
+++ /dev/null
-# SPDX-License-Identifier: GPL-2.0
-# file format version
-FILE_VERSION = 1
-
-LIBLOCKDEP_VERSION=$(shell make --no-print-directory -sC ../../.. kernelversion)
-
-# Makefiles suck: This macro sets a default value of $(2) for the
-# variable named by $(1), unless the variable has been set by
-# environment or command line. This is necessary for CC and AR
-# because make sets default values, so the simpler ?= approach
-# won't work as expected.
-define allow-override
- $(if $(or $(findstring environment,$(origin $(1))),\
- $(findstring command line,$(origin $(1)))),,\
- $(eval $(1) = $(2)))
-endef
-
-# Allow setting CC and AR and LD, or setting CROSS_COMPILE as a prefix.
-$(call allow-override,CC,$(CROSS_COMPILE)gcc)
-$(call allow-override,AR,$(CROSS_COMPILE)ar)
-$(call allow-override,LD,$(CROSS_COMPILE)ld)
-
-INSTALL = install
-
-# Use DESTDIR for installing into a different root directory.
-# This is useful for building a package. The program will be
-# installed in this directory as if it was the root directory.
-# Then the build tool can move it later.
-DESTDIR ?=
-DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
-
-prefix ?= /usr/local
-libdir_relative = lib
-libdir = $(prefix)/$(libdir_relative)
-bindir_relative = bin
-bindir = $(prefix)/$(bindir_relative)
-
-export DESTDIR DESTDIR_SQ INSTALL
-
-MAKEFLAGS += --no-print-directory
-
-include ../../scripts/Makefile.include
-
-# copy a bit from Linux kbuild
-
-ifeq ("$(origin V)", "command line")
- VERBOSE = $(V)
-endif
-ifndef VERBOSE
- VERBOSE = 0
-endif
-
-ifeq ($(srctree),)
-srctree := $(patsubst %/,%,$(dir $(CURDIR)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-#$(info Determined 'srctree' to be $(srctree))
-endif
-
-# Shell quotes
-libdir_SQ = $(subst ','\'',$(libdir))
-bindir_SQ = $(subst ','\'',$(bindir))
-
-LIB_IN := $(OUTPUT)liblockdep-in.o
-
-BIN_FILE = lockdep
-LIB_FILE = $(OUTPUT)liblockdep.a $(OUTPUT)liblockdep.so.$(LIBLOCKDEP_VERSION)
-
-CONFIG_INCLUDES =
-CONFIG_LIBS =
-CONFIG_FLAGS =
-
-OBJ = $@
-N =
-
-export Q VERBOSE
-
-INCLUDES = -I. -I./uinclude -I./include -I../../include $(CONFIG_INCLUDES)
-
-# Set compile option CFLAGS if not set elsewhere
-CFLAGS ?= -g -DCONFIG_LOCKDEP -DCONFIG_STACKTRACE -DCONFIG_PROVE_LOCKING -DBITS_PER_LONG=__WORDSIZE -DLIBLOCKDEP_VERSION='"$(LIBLOCKDEP_VERSION)"' -rdynamic -O0 -g
-CFLAGS += -fPIC
-CFLAGS += -Wall
-
-override CFLAGS += $(CONFIG_FLAGS) $(INCLUDES) $(PLUGIN_DIR_SQ)
-
-ifeq ($(VERBOSE),1)
- Q =
- print_shared_lib_compile =
- print_install =
-else
- Q = @
- print_shared_lib_compile = echo ' LD '$(OBJ);
- print_static_lib_build = echo ' LD '$(OBJ);
- print_install = echo ' INSTALL '$1' to $(DESTDIR_SQ)$2';
-endif
-
-all:
-
-export srctree OUTPUT CC LD CFLAGS V
-include $(srctree)/tools/build/Makefile.include
-
-do_compile_shared_library = \
- ($(print_shared_lib_compile) \
- $(CC) $(LDFLAGS) --shared $^ -o $@ -lpthread -ldl -Wl,-soname='$(@F)';$(shell ln -sf $(@F) $(@D)/liblockdep.so))
-
-do_build_static_lib = \
- ($(print_static_lib_build) \
- $(RM) $@; $(AR) rcs $@ $^)
-
-CMD_TARGETS = $(LIB_FILE)
-
-TARGETS = $(CMD_TARGETS)
-
-
-all: fixdep all_cmd
-
-all_cmd: $(CMD_TARGETS)
-
-$(LIB_IN): force
- $(Q)$(MAKE) $(build)=liblockdep
-
-$(OUTPUT)liblockdep.so.$(LIBLOCKDEP_VERSION): $(LIB_IN)
- $(Q)$(do_compile_shared_library)
-
-$(OUTPUT)liblockdep.a: $(LIB_IN)
- $(Q)$(do_build_static_lib)
-
-tags: force
- $(RM) tags
- find . -name '*.[ch]' | xargs ctags --extra=+f --c-kinds=+px \
- --regex-c++='/_PE\(([^,)]*).*/TEP_ERRNO__\1/'
-
-TAGS: force
- $(RM) TAGS
- find . -name '*.[ch]' | xargs etags \
- --regex='/_PE(\([^,)]*\).*/TEP_ERRNO__\1/'
-
-define do_install
- $(print_install) \
- if [ ! -d '$(DESTDIR_SQ)$2' ]; then \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$2'; \
- fi; \
- $(INSTALL) $1 '$(DESTDIR_SQ)$2'
-endef
-
-install_lib: all_cmd
- $(Q)$(call do_install,$(LIB_FILE),$(libdir_SQ))
- $(Q)$(call do_install,$(BIN_FILE),$(bindir_SQ))
-
-install: install_lib
-
-clean:
- $(RM) $(OUTPUT)*.o *~ $(TARGETS) $(OUTPUT)*.a $(OUTPUT)*liblockdep*.so* $(VERSION_FILES) $(OUTPUT).*.d $(OUTPUT).*.cmd
- $(RM) tags TAGS
-
-PHONY += force
-force:
-
-# Declare the contents of the .PHONY variable as phony. We keep that
-# information in a variable so we can use it in if_changed and friends.
-.PHONY: $(PHONY)
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <stddef.h>
-#include <stdbool.h>
-#include <linux/compiler.h>
-#include <linux/lockdep.h>
-#include <unistd.h>
-#include <sys/syscall.h>
-
-static __thread struct task_struct current_obj;
-
-/* lockdep wants these */
-bool debug_locks = true;
-bool debug_locks_silent;
-
-__attribute__((destructor)) static void liblockdep_exit(void)
-{
- debug_check_no_locks_held();
-}
-
-struct task_struct *__curr(void)
-{
- if (current_obj.pid == 0) {
- /* Makes lockdep output pretty */
- prctl(PR_GET_NAME, current_obj.comm);
- current_obj.pid = syscall(__NR_gettid);
- }
-
- return ¤t_obj;
-}
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LIBLOCKDEP_COMMON_H
-#define _LIBLOCKDEP_COMMON_H
-
-#include <pthread.h>
-
-#define NR_LOCKDEP_CACHING_CLASSES 2
-#define MAX_LOCKDEP_SUBCLASSES 8UL
-
-#ifndef CALLER_ADDR0
-#define CALLER_ADDR0 ((unsigned long)__builtin_return_address(0))
-#endif
-
-#ifndef _RET_IP_
-#define _RET_IP_ CALLER_ADDR0
-#endif
-
-#ifndef _THIS_IP_
-#define _THIS_IP_ ({ __label__ __here; __here: (unsigned long)&&__here; })
-#endif
-
-struct lockdep_subclass_key {
- char __one_byte;
-};
-
-struct lock_class_key {
- struct lockdep_subclass_key subkeys[MAX_LOCKDEP_SUBCLASSES];
-};
-
-struct lockdep_map {
- struct lock_class_key *key;
- struct lock_class *class_cache[NR_LOCKDEP_CACHING_CLASSES];
- const char *name;
-#ifdef CONFIG_LOCK_STAT
- int cpu;
- unsigned long ip;
-#endif
-};
-
-void lockdep_init_map(struct lockdep_map *lock, const char *name,
- struct lock_class_key *key, int subclass);
-void lock_acquire(struct lockdep_map *lock, unsigned int subclass,
- int trylock, int read, int check,
- struct lockdep_map *nest_lock, unsigned long ip);
-void lock_release(struct lockdep_map *lock, unsigned long ip);
-void lockdep_reset_lock(struct lockdep_map *lock);
-void lockdep_register_key(struct lock_class_key *key);
-void lockdep_unregister_key(struct lock_class_key *key);
-extern void debug_check_no_locks_freed(const void *from, unsigned long len);
-
-#define STATIC_LOCKDEP_MAP_INIT(_name, _key) \
- { .name = (_name), .key = (void *)(_key), }
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LIBLOCKDEP_MUTEX_H
-#define _LIBLOCKDEP_MUTEX_H
-
-#include <pthread.h>
-#include "common.h"
-
-struct liblockdep_pthread_mutex {
- pthread_mutex_t mutex;
- struct lock_class_key key;
- struct lockdep_map dep_map;
-};
-
-typedef struct liblockdep_pthread_mutex liblockdep_pthread_mutex_t;
-
-#define LIBLOCKDEP_PTHREAD_MUTEX_INITIALIZER(mtx) \
- (const struct liblockdep_pthread_mutex) { \
- .mutex = PTHREAD_MUTEX_INITIALIZER, \
- .dep_map = STATIC_LOCKDEP_MAP_INIT(#mtx, &((&(mtx))->dep_map)), \
-}
-
-static inline int __mutex_init(liblockdep_pthread_mutex_t *lock,
- const char *name,
- struct lock_class_key *key,
- const pthread_mutexattr_t *__mutexattr)
-{
- lockdep_init_map(&lock->dep_map, name, key, 0);
- return pthread_mutex_init(&lock->mutex, __mutexattr);
-}
-
-#define liblockdep_pthread_mutex_init(mutex, mutexattr) \
-({ \
- lockdep_register_key(&(mutex)->key); \
- __mutex_init((mutex), #mutex, &(mutex)->key, (mutexattr)); \
-})
-
-static inline int liblockdep_pthread_mutex_lock(liblockdep_pthread_mutex_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 0, 0, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_mutex_lock(&lock->mutex);
-}
-
-static inline int liblockdep_pthread_mutex_unlock(liblockdep_pthread_mutex_t *lock)
-{
- lock_release(&lock->dep_map, (unsigned long)_RET_IP_);
- return pthread_mutex_unlock(&lock->mutex);
-}
-
-static inline int liblockdep_pthread_mutex_trylock(liblockdep_pthread_mutex_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 1, 0, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_mutex_trylock(&lock->mutex) == 0 ? 1 : 0;
-}
-
-static inline int liblockdep_pthread_mutex_destroy(liblockdep_pthread_mutex_t *lock)
-{
- lockdep_reset_lock(&lock->dep_map);
- lockdep_unregister_key(&lock->key);
- return pthread_mutex_destroy(&lock->mutex);
-}
-
-#ifdef __USE_LIBLOCKDEP
-
-#define pthread_mutex_t liblockdep_pthread_mutex_t
-#define pthread_mutex_init liblockdep_pthread_mutex_init
-#define pthread_mutex_lock liblockdep_pthread_mutex_lock
-#define pthread_mutex_unlock liblockdep_pthread_mutex_unlock
-#define pthread_mutex_trylock liblockdep_pthread_mutex_trylock
-#define pthread_mutex_destroy liblockdep_pthread_mutex_destroy
-
-#endif
-
-#endif
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LIBLOCKDEP_RWLOCK_H
-#define _LIBLOCKDEP_RWLOCK_H
-
-#include <pthread.h>
-#include "common.h"
-
-struct liblockdep_pthread_rwlock {
- pthread_rwlock_t rwlock;
- struct lockdep_map dep_map;
-};
-
-typedef struct liblockdep_pthread_rwlock liblockdep_pthread_rwlock_t;
-
-#define LIBLOCKDEP_PTHREAD_RWLOCK_INITIALIZER(rwl) \
- (struct liblockdep_pthread_rwlock) { \
- .rwlock = PTHREAD_RWLOCK_INITIALIZER, \
- .dep_map = STATIC_LOCKDEP_MAP_INIT(#rwl, &((&(rwl))->dep_map)), \
-}
-
-static inline int __rwlock_init(liblockdep_pthread_rwlock_t *lock,
- const char *name,
- struct lock_class_key *key,
- const pthread_rwlockattr_t *attr)
-{
- lockdep_init_map(&lock->dep_map, name, key, 0);
-
- return pthread_rwlock_init(&lock->rwlock, attr);
-}
-
-#define liblockdep_pthread_rwlock_init(lock, attr) \
-({ \
- static struct lock_class_key __key; \
- \
- __rwlock_init((lock), #lock, &__key, (attr)); \
-})
-
-static inline int liblockdep_pthread_rwlock_rdlock(liblockdep_pthread_rwlock_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 0, 2, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_rwlock_rdlock(&lock->rwlock);
-
-}
-
-static inline int liblockdep_pthread_rwlock_unlock(liblockdep_pthread_rwlock_t *lock)
-{
- lock_release(&lock->dep_map, (unsigned long)_RET_IP_);
- return pthread_rwlock_unlock(&lock->rwlock);
-}
-
-static inline int liblockdep_pthread_rwlock_wrlock(liblockdep_pthread_rwlock_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 0, 0, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_rwlock_wrlock(&lock->rwlock);
-}
-
-static inline int liblockdep_pthread_rwlock_tryrdlock(liblockdep_pthread_rwlock_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 1, 2, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_rwlock_tryrdlock(&lock->rwlock) == 0 ? 1 : 0;
-}
-
-static inline int liblockdep_pthread_rwlock_trywrlock(liblockdep_pthread_rwlock_t *lock)
-{
- lock_acquire(&lock->dep_map, 0, 1, 0, 1, NULL, (unsigned long)_RET_IP_);
- return pthread_rwlock_trywrlock(&lock->rwlock) == 0 ? 1 : 0;
-}
-
-static inline int liblockdep_rwlock_destroy(liblockdep_pthread_rwlock_t *lock)
-{
- return pthread_rwlock_destroy(&lock->rwlock);
-}
-
-#ifdef __USE_LIBLOCKDEP
-
-#define pthread_rwlock_t liblockdep_pthread_rwlock_t
-#define pthread_rwlock_init liblockdep_pthread_rwlock_init
-#define pthread_rwlock_rdlock liblockdep_pthread_rwlock_rdlock
-#define pthread_rwlock_unlock liblockdep_pthread_rwlock_unlock
-#define pthread_rwlock_wrlock liblockdep_pthread_rwlock_wrlock
-#define pthread_rwlock_tryrdlock liblockdep_pthread_rwlock_tryrdlock
-#define pthread_rwlock_trywrlock liblockdep_pthread_rwlock_trywrlock
-#define pthread_rwlock_destroy liblockdep_rwlock_destroy
-
-#endif
-
-#endif
+++ /dev/null
-#!/bin/bash
-
-LD_PRELOAD="./liblockdep.so $LD_PRELOAD" "$@"
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <linux/lockdep.h>
-#include <stdlib.h>
-
-/* Trivial API wrappers, we don't (yet) have RCU in user-space: */
-#define hlist_for_each_entry_rcu hlist_for_each_entry
-#define hlist_add_head_rcu hlist_add_head
-#define hlist_del_rcu hlist_del
-#define list_for_each_entry_rcu list_for_each_entry
-#define list_add_tail_rcu list_add_tail
-
-u32 prandom_u32(void)
-{
- /* Used only by lock_pin_lock() which is dead code */
- abort();
-}
-
-void print_irqtrace_events(struct task_struct *curr)
-{
- abort();
-}
-
-static struct new_utsname *init_utsname(void)
-{
- static struct new_utsname n = (struct new_utsname) {
- .release = "liblockdep",
- .version = LIBLOCKDEP_VERSION,
- };
-
- return &n;
-}
-
-#include "../../../kernel/locking/lockdep.c"
+++ /dev/null
-#include "../../../kernel/locking/lockdep_internals.h"
+++ /dev/null
-#include "../../../kernel/locking/lockdep_states.h"
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#define _GNU_SOURCE
-#include <pthread.h>
-#include <stdio.h>
-#include <dlfcn.h>
-#include <stdlib.h>
-#include <sysexits.h>
-#include <unistd.h>
-#include "include/liblockdep/mutex.h"
-#include "../../include/linux/rbtree.h"
-
-/**
- * struct lock_lookup - liblockdep's view of a single unique lock
- * @orig: pointer to the original pthread lock, used for lookups
- * @dep_map: lockdep's dep_map structure
- * @key: lockdep's key structure
- * @node: rb-tree node used to store the lock in a global tree
- * @name: a unique name for the lock
- */
-struct lock_lookup {
- void *orig; /* Original pthread lock, used for lookups */
- struct lockdep_map dep_map; /* Since all locks are dynamic, we need
- * a dep_map and a key for each lock */
- /*
- * Wait, there's no support for key classes? Yup :(
- * Most big projects wrap the pthread api with their own calls to
- * be compatible with different locking methods. This means that
- * "classes" will be brokes since the function that creates all
- * locks will point to a generic locking function instead of the
- * actual code that wants to do the locking.
- */
- struct lock_class_key key;
- struct rb_node node;
-#define LIBLOCKDEP_MAX_LOCK_NAME 22
- char name[LIBLOCKDEP_MAX_LOCK_NAME];
-};
-
-/* This is where we store our locks */
-static struct rb_root locks = RB_ROOT;
-static pthread_rwlock_t locks_rwlock = PTHREAD_RWLOCK_INITIALIZER;
-
-/* pthread mutex API */
-
-#ifdef __GLIBC__
-extern int __pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *attr);
-extern int __pthread_mutex_lock(pthread_mutex_t *mutex);
-extern int __pthread_mutex_trylock(pthread_mutex_t *mutex);
-extern int __pthread_mutex_unlock(pthread_mutex_t *mutex);
-extern int __pthread_mutex_destroy(pthread_mutex_t *mutex);
-#else
-#define __pthread_mutex_init NULL
-#define __pthread_mutex_lock NULL
-#define __pthread_mutex_trylock NULL
-#define __pthread_mutex_unlock NULL
-#define __pthread_mutex_destroy NULL
-#endif
-static int (*ll_pthread_mutex_init)(pthread_mutex_t *mutex,
- const pthread_mutexattr_t *attr) = __pthread_mutex_init;
-static int (*ll_pthread_mutex_lock)(pthread_mutex_t *mutex) = __pthread_mutex_lock;
-static int (*ll_pthread_mutex_trylock)(pthread_mutex_t *mutex) = __pthread_mutex_trylock;
-static int (*ll_pthread_mutex_unlock)(pthread_mutex_t *mutex) = __pthread_mutex_unlock;
-static int (*ll_pthread_mutex_destroy)(pthread_mutex_t *mutex) = __pthread_mutex_destroy;
-
-/* pthread rwlock API */
-
-#ifdef __GLIBC__
-extern int __pthread_rwlock_init(pthread_rwlock_t *rwlock, const pthread_rwlockattr_t *attr);
-extern int __pthread_rwlock_destroy(pthread_rwlock_t *rwlock);
-extern int __pthread_rwlock_wrlock(pthread_rwlock_t *rwlock);
-extern int __pthread_rwlock_trywrlock(pthread_rwlock_t *rwlock);
-extern int __pthread_rwlock_rdlock(pthread_rwlock_t *rwlock);
-extern int __pthread_rwlock_tryrdlock(pthread_rwlock_t *rwlock);
-extern int __pthread_rwlock_unlock(pthread_rwlock_t *rwlock);
-#else
-#define __pthread_rwlock_init NULL
-#define __pthread_rwlock_destroy NULL
-#define __pthread_rwlock_wrlock NULL
-#define __pthread_rwlock_trywrlock NULL
-#define __pthread_rwlock_rdlock NULL
-#define __pthread_rwlock_tryrdlock NULL
-#define __pthread_rwlock_unlock NULL
-#endif
-
-static int (*ll_pthread_rwlock_init)(pthread_rwlock_t *rwlock,
- const pthread_rwlockattr_t *attr) = __pthread_rwlock_init;
-static int (*ll_pthread_rwlock_destroy)(pthread_rwlock_t *rwlock) = __pthread_rwlock_destroy;
-static int (*ll_pthread_rwlock_rdlock)(pthread_rwlock_t *rwlock) = __pthread_rwlock_rdlock;
-static int (*ll_pthread_rwlock_tryrdlock)(pthread_rwlock_t *rwlock) = __pthread_rwlock_tryrdlock;
-static int (*ll_pthread_rwlock_trywrlock)(pthread_rwlock_t *rwlock) = __pthread_rwlock_trywrlock;
-static int (*ll_pthread_rwlock_wrlock)(pthread_rwlock_t *rwlock) = __pthread_rwlock_wrlock;
-static int (*ll_pthread_rwlock_unlock)(pthread_rwlock_t *rwlock) = __pthread_rwlock_unlock;
-
-enum { none, prepare, done, } __init_state;
-static void init_preload(void);
-static void try_init_preload(void)
-{
- if (__init_state != done)
- init_preload();
-}
-
-static struct rb_node **__get_lock_node(void *lock, struct rb_node **parent)
-{
- struct rb_node **node = &locks.rb_node;
- struct lock_lookup *l;
-
- *parent = NULL;
-
- while (*node) {
- l = rb_entry(*node, struct lock_lookup, node);
-
- *parent = *node;
- if (lock < l->orig)
- node = &l->node.rb_left;
- else if (lock > l->orig)
- node = &l->node.rb_right;
- else
- return node;
- }
-
- return node;
-}
-
-#ifndef LIBLOCKDEP_STATIC_ENTRIES
-#define LIBLOCKDEP_STATIC_ENTRIES 1024
-#endif
-
-static struct lock_lookup __locks[LIBLOCKDEP_STATIC_ENTRIES];
-static int __locks_nr;
-
-static inline bool is_static_lock(struct lock_lookup *lock)
-{
- return lock >= __locks && lock < __locks + ARRAY_SIZE(__locks);
-}
-
-static struct lock_lookup *alloc_lock(void)
-{
- if (__init_state != done) {
- /*
- * Some programs attempt to initialize and use locks in their
- * allocation path. This means that a call to malloc() would
- * result in locks being initialized and locked.
- *
- * Why is it an issue for us? dlsym() below will try allocating
- * to give us the original function. Since this allocation will
- * result in a locking operations, we have to let pthread deal
- * with it, but we can't! we don't have the pointer to the
- * original API since we're inside dlsym() trying to get it
- */
-
- int idx = __locks_nr++;
- if (idx >= ARRAY_SIZE(__locks)) {
- dprintf(STDERR_FILENO,
- "LOCKDEP error: insufficient LIBLOCKDEP_STATIC_ENTRIES\n");
- exit(EX_UNAVAILABLE);
- }
- return __locks + idx;
- }
-
- return malloc(sizeof(struct lock_lookup));
-}
-
-static inline void free_lock(struct lock_lookup *lock)
-{
- if (likely(!is_static_lock(lock)))
- free(lock);
-}
-
-/**
- * __get_lock - find or create a lock instance
- * @lock: pointer to a pthread lock function
- *
- * Try to find an existing lock in the rbtree using the provided pointer. If
- * one wasn't found - create it.
- */
-static struct lock_lookup *__get_lock(void *lock)
-{
- struct rb_node **node, *parent;
- struct lock_lookup *l;
-
- ll_pthread_rwlock_rdlock(&locks_rwlock);
- node = __get_lock_node(lock, &parent);
- ll_pthread_rwlock_unlock(&locks_rwlock);
- if (*node) {
- return rb_entry(*node, struct lock_lookup, node);
- }
-
- /* We didn't find the lock, let's create it */
- l = alloc_lock();
- if (l == NULL)
- return NULL;
-
- l->orig = lock;
- /*
- * Currently the name of the lock is the ptr value of the pthread lock,
- * while not optimal, it makes debugging a bit easier.
- *
- * TODO: Get the real name of the lock using libdwarf
- */
- sprintf(l->name, "%p", lock);
- lockdep_init_map(&l->dep_map, l->name, &l->key, 0);
-
- ll_pthread_rwlock_wrlock(&locks_rwlock);
- /* This might have changed since the last time we fetched it */
- node = __get_lock_node(lock, &parent);
- rb_link_node(&l->node, parent, node);
- rb_insert_color(&l->node, &locks);
- ll_pthread_rwlock_unlock(&locks_rwlock);
-
- return l;
-}
-
-static void __del_lock(struct lock_lookup *lock)
-{
- ll_pthread_rwlock_wrlock(&locks_rwlock);
- rb_erase(&lock->node, &locks);
- ll_pthread_rwlock_unlock(&locks_rwlock);
- free_lock(lock);
-}
-
-int pthread_mutex_init(pthread_mutex_t *mutex,
- const pthread_mutexattr_t *attr)
-{
- int r;
-
- /*
- * We keep trying to init our preload module because there might be
- * code in init sections that tries to touch locks before we are
- * initialized, in that case we'll need to manually call preload
- * to get us going.
- *
- * Funny enough, kernel's lockdep had the same issue, and used
- * (almost) the same solution. See look_up_lock_class() in
- * kernel/locking/lockdep.c for details.
- */
- try_init_preload();
-
- r = ll_pthread_mutex_init(mutex, attr);
- if (r == 0)
- /*
- * We do a dummy initialization here so that lockdep could
- * warn us if something fishy is going on - such as
- * initializing a held lock.
- */
- __get_lock(mutex);
-
- return r;
-}
-
-int pthread_mutex_lock(pthread_mutex_t *mutex)
-{
- int r;
-
- try_init_preload();
-
- lock_acquire(&__get_lock(mutex)->dep_map, 0, 0, 0, 1, NULL,
- (unsigned long)_RET_IP_);
- /*
- * Here's the thing with pthread mutexes: unlike the kernel variant,
- * they can fail.
- *
- * This means that the behaviour here is a bit different from what's
- * going on in the kernel: there we just tell lockdep that we took the
- * lock before actually taking it, but here we must deal with the case
- * that locking failed.
- *
- * To do that we'll "release" the lock if locking failed - this way
- * we'll get lockdep doing the correct checks when we try to take
- * the lock, and if that fails - we'll be back to the correct
- * state by releasing it.
- */
- r = ll_pthread_mutex_lock(mutex);
- if (r)
- lock_release(&__get_lock(mutex)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_mutex_trylock(pthread_mutex_t *mutex)
-{
- int r;
-
- try_init_preload();
-
- lock_acquire(&__get_lock(mutex)->dep_map, 0, 1, 0, 1, NULL, (unsigned long)_RET_IP_);
- r = ll_pthread_mutex_trylock(mutex);
- if (r)
- lock_release(&__get_lock(mutex)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_mutex_unlock(pthread_mutex_t *mutex)
-{
- int r;
-
- try_init_preload();
-
- lock_release(&__get_lock(mutex)->dep_map, (unsigned long)_RET_IP_);
- /*
- * Just like taking a lock, only in reverse!
- *
- * If we fail releasing the lock, tell lockdep we're holding it again.
- */
- r = ll_pthread_mutex_unlock(mutex);
- if (r)
- lock_acquire(&__get_lock(mutex)->dep_map, 0, 0, 0, 1, NULL, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_mutex_destroy(pthread_mutex_t *mutex)
-{
- try_init_preload();
-
- /*
- * Let's see if we're releasing a lock that's held.
- *
- * TODO: Hook into free() and add that check there as well.
- */
- debug_check_no_locks_freed(mutex, sizeof(*mutex));
- __del_lock(__get_lock(mutex));
- return ll_pthread_mutex_destroy(mutex);
-}
-
-/* This is the rwlock part, very similar to what happened with mutex above */
-int pthread_rwlock_init(pthread_rwlock_t *rwlock,
- const pthread_rwlockattr_t *attr)
-{
- int r;
-
- try_init_preload();
-
- r = ll_pthread_rwlock_init(rwlock, attr);
- if (r == 0)
- __get_lock(rwlock);
-
- return r;
-}
-
-int pthread_rwlock_destroy(pthread_rwlock_t *rwlock)
-{
- try_init_preload();
-
- debug_check_no_locks_freed(rwlock, sizeof(*rwlock));
- __del_lock(__get_lock(rwlock));
- return ll_pthread_rwlock_destroy(rwlock);
-}
-
-int pthread_rwlock_rdlock(pthread_rwlock_t *rwlock)
-{
- int r;
-
- init_preload();
-
- lock_acquire(&__get_lock(rwlock)->dep_map, 0, 0, 2, 1, NULL, (unsigned long)_RET_IP_);
- r = ll_pthread_rwlock_rdlock(rwlock);
- if (r)
- lock_release(&__get_lock(rwlock)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_rwlock_tryrdlock(pthread_rwlock_t *rwlock)
-{
- int r;
-
- init_preload();
-
- lock_acquire(&__get_lock(rwlock)->dep_map, 0, 1, 2, 1, NULL, (unsigned long)_RET_IP_);
- r = ll_pthread_rwlock_tryrdlock(rwlock);
- if (r)
- lock_release(&__get_lock(rwlock)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_rwlock_trywrlock(pthread_rwlock_t *rwlock)
-{
- int r;
-
- init_preload();
-
- lock_acquire(&__get_lock(rwlock)->dep_map, 0, 1, 0, 1, NULL, (unsigned long)_RET_IP_);
- r = ll_pthread_rwlock_trywrlock(rwlock);
- if (r)
- lock_release(&__get_lock(rwlock)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_rwlock_wrlock(pthread_rwlock_t *rwlock)
-{
- int r;
-
- init_preload();
-
- lock_acquire(&__get_lock(rwlock)->dep_map, 0, 0, 0, 1, NULL, (unsigned long)_RET_IP_);
- r = ll_pthread_rwlock_wrlock(rwlock);
- if (r)
- lock_release(&__get_lock(rwlock)->dep_map, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-int pthread_rwlock_unlock(pthread_rwlock_t *rwlock)
-{
- int r;
-
- init_preload();
-
- lock_release(&__get_lock(rwlock)->dep_map, (unsigned long)_RET_IP_);
- r = ll_pthread_rwlock_unlock(rwlock);
- if (r)
- lock_acquire(&__get_lock(rwlock)->dep_map, 0, 0, 0, 1, NULL, (unsigned long)_RET_IP_);
-
- return r;
-}
-
-__attribute__((constructor)) static void init_preload(void)
-{
- if (__init_state == done)
- return;
-
-#ifndef __GLIBC__
- __init_state = prepare;
-
- ll_pthread_mutex_init = dlsym(RTLD_NEXT, "pthread_mutex_init");
- ll_pthread_mutex_lock = dlsym(RTLD_NEXT, "pthread_mutex_lock");
- ll_pthread_mutex_trylock = dlsym(RTLD_NEXT, "pthread_mutex_trylock");
- ll_pthread_mutex_unlock = dlsym(RTLD_NEXT, "pthread_mutex_unlock");
- ll_pthread_mutex_destroy = dlsym(RTLD_NEXT, "pthread_mutex_destroy");
-
- ll_pthread_rwlock_init = dlsym(RTLD_NEXT, "pthread_rwlock_init");
- ll_pthread_rwlock_destroy = dlsym(RTLD_NEXT, "pthread_rwlock_destroy");
- ll_pthread_rwlock_rdlock = dlsym(RTLD_NEXT, "pthread_rwlock_rdlock");
- ll_pthread_rwlock_tryrdlock = dlsym(RTLD_NEXT, "pthread_rwlock_tryrdlock");
- ll_pthread_rwlock_wrlock = dlsym(RTLD_NEXT, "pthread_rwlock_wrlock");
- ll_pthread_rwlock_trywrlock = dlsym(RTLD_NEXT, "pthread_rwlock_trywrlock");
- ll_pthread_rwlock_unlock = dlsym(RTLD_NEXT, "pthread_rwlock_unlock");
-#endif
-
- __init_state = done;
-}
+++ /dev/null
-#include "../../lib/rbtree.c"
+++ /dev/null
-#! /bin/bash
-# SPDX-License-Identifier: GPL-2.0
-
-if ! make >/dev/null; then
- echo "Building liblockdep failed."
- echo "FAILED!"
- exit 1
-fi
-
-find tests -name '*.c' | sort | while read -r i; do
- testname=$(basename "$i" .c)
- echo -ne "$testname... "
- if gcc -o "tests/$testname" -pthread "$i" liblockdep.a -Iinclude -D__USE_LIBLOCKDEP &&
- timeout 1 "tests/$testname" 2>&1 | /bin/bash "tests/${testname}.sh"; then
- echo "PASSED!"
- else
- echo "FAILED!"
- fi
- rm -f "tests/$testname"
-done
-
-find tests -name '*.c' | sort | while read -r i; do
- testname=$(basename "$i" .c)
- echo -ne "(PRELOAD) $testname... "
- if gcc -o "tests/$testname" -pthread -Iinclude "$i" &&
- timeout 1 ./lockdep "tests/$testname" 2>&1 |
- /bin/bash "tests/${testname}.sh"; then
- echo "PASSED!"
- else
- echo "FAILED!"
- fi
- rm -f "tests/$testname"
-done
-
-find tests -name '*.c' | sort | while read -r i; do
- testname=$(basename "$i" .c)
- echo -ne "(PRELOAD + Valgrind) $testname... "
- if gcc -o "tests/$testname" -pthread -Iinclude "$i" &&
- { timeout 10 valgrind --read-var-info=yes ./lockdep "./tests/$testname" >& "tests/${testname}.vg.out"; true; } &&
- /bin/bash "tests/${testname}.sh" < "tests/${testname}.vg.out" &&
- ! grep -Eq '(^==[0-9]*== (Invalid |Uninitialised ))|Mismatched free|Source and destination overlap| UME ' "tests/${testname}.vg.out"; then
- echo "PASSED!"
- else
- echo "FAILED!"
- fi
- rm -f "tests/$testname"
-done
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-
-int main(void)
-{
- pthread_mutex_t a;
-
- pthread_mutex_init(&a, NULL);
-
- pthread_mutex_lock(&a);
- pthread_mutex_lock(&a);
-
- return 0;
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible recursive locking detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-
-void main(void)
-{
- pthread_mutex_t a, b;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
-
- pthread_mutex_lock(&a);
- pthread_mutex_lock(&b);
- pthread_mutex_lock(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible recursive locking detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(b, a);
-
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(b, a);
-
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <stdio.h>
-#include <pthread.h>
-
-pthread_mutex_t a = PTHREAD_MUTEX_INITIALIZER;
-pthread_mutex_t b = PTHREAD_MUTEX_INITIALIZER;
-pthread_barrier_t bar;
-
-void *ba_lock(void *arg)
-{
- int ret, i;
-
- pthread_mutex_lock(&b);
-
- if (pthread_barrier_wait(&bar) == PTHREAD_BARRIER_SERIAL_THREAD)
- pthread_barrier_destroy(&bar);
-
- pthread_mutex_lock(&a);
-
- pthread_mutex_unlock(&a);
- pthread_mutex_unlock(&b);
-}
-
-int main(void)
-{
- pthread_t t;
-
- pthread_barrier_init(&bar, NULL, 2);
-
- if (pthread_create(&t, NULL, ba_lock, NULL)) {
- fprintf(stderr, "pthread_create() failed\n");
- return 1;
- }
- pthread_mutex_lock(&a);
-
- if (pthread_barrier_wait(&bar) == PTHREAD_BARRIER_SERIAL_THREAD)
- pthread_barrier_destroy(&bar);
-
- pthread_mutex_lock(&b);
-
- pthread_mutex_unlock(&b);
- pthread_mutex_unlock(&a);
-
- pthread_join(t, NULL);
-
- return 0;
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b, c;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
- pthread_mutex_init(&c, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(b, c);
- LOCK_UNLOCK_2(c, a);
-
- pthread_mutex_destroy(&c);
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b, c, d;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
- pthread_mutex_init(&c, NULL);
- pthread_mutex_init(&d, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(b, c);
- LOCK_UNLOCK_2(c, d);
- LOCK_UNLOCK_2(d, a);
-
- pthread_mutex_destroy(&d);
- pthread_mutex_destroy(&c);
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b, c;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
- pthread_mutex_init(&c, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(c, a);
- LOCK_UNLOCK_2(b, c);
-
- pthread_mutex_destroy(&c);
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b, c, d;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
- pthread_mutex_init(&c, NULL);
- pthread_mutex_init(&d, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(c, d);
- LOCK_UNLOCK_2(b, c);
- LOCK_UNLOCK_2(d, a);
-
- pthread_mutex_destroy(&d);
- pthread_mutex_destroy(&c);
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-#include "common.h"
-
-void main(void)
-{
- pthread_mutex_t a, b, c, d;
-
- pthread_mutex_init(&a, NULL);
- pthread_mutex_init(&b, NULL);
- pthread_mutex_init(&c, NULL);
- pthread_mutex_init(&d, NULL);
-
- LOCK_UNLOCK_2(a, b);
- LOCK_UNLOCK_2(c, d);
- LOCK_UNLOCK_2(b, d);
- LOCK_UNLOCK_2(d, a);
-
- pthread_mutex_destroy(&d);
- pthread_mutex_destroy(&c);
- pthread_mutex_destroy(&b);
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible circular locking dependency detected'
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/rwlock.h>
-
-void main(void)
-{
- pthread_rwlock_t a, b;
-
- pthread_rwlock_init(&a, NULL);
- pthread_rwlock_init(&b, NULL);
-
- pthread_rwlock_wrlock(&a);
- pthread_rwlock_rdlock(&b);
- pthread_rwlock_wrlock(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: possible recursive locking detected'
+++ /dev/null
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _LIBLOCKDEP_TEST_COMMON_H
-#define _LIBLOCKDEP_TEST_COMMON_H
-
-#define LOCK_UNLOCK_2(a, b) \
- do { \
- pthread_mutex_lock(&(a)); \
- pthread_mutex_lock(&(b)); \
- pthread_mutex_unlock(&(b)); \
- pthread_mutex_unlock(&(a)); \
- } while(0)
-
-#endif
+++ /dev/null
-// SPDX-License-Identifier: GPL-2.0
-#include <liblockdep/mutex.h>
-
-void main(void)
-{
- pthread_mutex_t a;
-
- pthread_mutex_init(&a, NULL);
-
- pthread_mutex_lock(&a);
- pthread_mutex_unlock(&a);
- pthread_mutex_unlock(&a);
-
- pthread_mutex_destroy(&a);
-}
+++ /dev/null
-#!/bin/bash
-grep -q 'WARNING: bad unlock balance detected'
int perf_cpu_map__idx(struct perf_cpu_map *cpus, int cpu)
{
- int i;
+ int low = 0, high = cpus->nr;
- for (i = 0; i < cpus->nr; ++i) {
- if (cpus->map[i] == cpu)
- return i;
+ while (low < high) {
+ int idx = (low + high) / 2,
+ cpu_at_idx = cpus->map[idx];
+
+ if (cpu_at_idx == cpu)
+ return idx;
+
+ if (cpu_at_idx > cpu)
+ high = idx;
+ else
+ low = idx + 1;
}
return -1;
__u32 tid;
};
+struct perf_record_aux_output_hw_id {
+ struct perf_event_header header;
+ __u64 hw_id;
+};
+
struct perf_record_thread_map_entry {
__u64 pid;
char comm[16];
struct perf_record_auxtrace_error auxtrace_error;
struct perf_record_aux aux;
struct perf_record_itrace_start itrace_start;
+ struct perf_record_aux_output_hw_id aux_output_hw_id;
struct perf_record_switch context_switch;
struct perf_record_thread_map thread_map;
struct perf_record_cpu_map cpu_map;
if (!insn->func)
return false;
+ if (insn->func->static_call_tramp)
+ return true;
+
/*
* CONFIG_UBSAN_TRAP inserts a UD2 when it sees
* __builtin_unreachable(). The BUG() macro has an unreachable() after
pmu-events/pmu-events.c
pmu-events/jevents
feature/
+libbpf/
fixdep
libtraceevent-dynamic-list
Documentation/doc.dep
L synthesize last branch entries on existing event records
s skip initial number of events
q quicker (less detailed) decoding
+ A approximate IPC
Z prefer to ignore timestamps (so-called "timeless" decoding)
The default is all events i.e. the same as --itrace=ibxwpe,
debug messages will or will not be logged. Each flag must be preceded
by either '+' or '-'. The flags are:
a all perf events
+ o output to stdout
If supported, the 'q' option may be repeated to increase the effect.
tasks slept. sched_switch contains a callchain where a task slept and
sched_stat contains a timeslice how long a task slept.
+-k::
+--vmlinux=<file>::
+ vmlinux pathname
+
+--ignore-vmlinux::
+ Ignore vmlinux files.
+
--kallsyms=<file>::
kallsyms pathname
the average IPC since the last IPC for that event type. Note IPC for "branches"
events is calculated separately from IPC for "instructions" events.
+Even with the 'cyc' config term, it is possible to produce IPC information for
+every change of timestamp, but at the expense of accuracy. That is selected by
+specifying the itrace 'A' option. Due to the granularity of timestamps, the
+actual number of cycles increases even though the cycles reported does not.
+The number of instructions is known, but if IPC is reported, cycles can be too
+low and so IPC is too high. Note that inaccuracy decreases as the period of
+sampling increases i.e. if the number of cycles is too low by a small amount,
+that becomes less significant if the number of cycles is large. It may also be
+useful to use the 'A' option in conjunction with dlfilter-show-cycles.so to
+provide higher granularity cycle information.
+
Also note that the IPC instruction count may or may not include the current
instruction. If the cycle count is associated with an asynchronous branch
(e.g. page fault or interrupt), then the instruction count does not include the
L synthesize last branch entries on existing event records
s skip initial number of events
q quicker (less detailed) decoding
+ A approximate IPC
Z prefer to ignore timestamps (so-called "timeless" decoding)
"Instructions" events look like they were recorded by "perf record -e
must be preceded by either '+' or '-'. The flags support by Intel PT are:
-a Suppress logging of perf events
+a Log all perf events
+ +o Output to stdout instead of "intel_pt.log"
By default, logged perf events are filtered by any specified time ranges, but
flag +a overrides that.
decoding a trace of a virtual machine.
+dlfilter-show-cycles.so
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Cycles can be displayed using dlfilter-show-cycles.so in which case the itrace A
+option can be useful to provide higher granularity cycle information:
+
+ perf script --itrace=A --call-trace --dlfilter dlfilter-show-cycles.so
+
+To see a list of dlfilters:
+
+ perf script -v --list-dlfilters
+
+See also linkperf:perf-dlfilters[1]
+
+
dump option
~~~~~~~~~~~
perf record -c 10000 -e '{intel_pt/branch=0/,cycles/aux-output/ppp}' uname
-Note that currently, software only supports redirecting at most one PEBS event.
+Originally, software only supported redirecting at most one PEBS event because it
+was not able to differentiate one event from another. To overcome that, more recent
+kernels and perf tools add support for the PERF_RECORD_AUX_OUTPUT_HW_ID side-band event.
+To check for the presence of that event in a PEBS-via-PT trace:
+
+ perf script -D --no-itrace | grep PERF_RECORD_AUX_OUTPUT_HW_ID
To display PEBS events from the Intel PT trace, use the itrace 'o' option e.g.
SYNOPSIS
--------
[verse]
-'perf kmem' {record|stat} [<options>]
+'perf kmem' [<options>] {record|stat}
DESCRIPTION
-----------
There are two variants of perf kmem:
- 'perf kmem record <command>' to record the kmem events
- of an arbitrary workload.
+ 'perf kmem [<options>] record [<perf-record-options>] <command>' to
+ record the kmem events of an arbitrary workload. Additional 'perf
+ record' options may be specified after record, such as '-o' to
+ change the output file name.
- 'perf kmem stat' to report kernel memory statistics.
+ 'perf kmem [<options>] stat' to report kernel memory statistics.
OPTIONS
-------
-i <file>::
--input=<file>::
- Select the input file (default: perf.data unless stdin is a fifo)
+ For stat, select the input file (default: perf.data unless stdin is a
+ fifo)
-f::
--force::
--deprecated::
Print deprecated events. By default the deprecated events are hidden.
+--cputype::
+Print events applying cpu with this type for hybrid platform
+(e.g. --cputype core or --cputype atom)
+
[[EVENT_MODIFIERS]]
EVENT MODIFIERS
---------------
--switch-events::
Record context switch events i.e. events of type PERF_RECORD_SWITCH or
-PERF_RECORD_SWITCH_CPU_WIDE. In some cases (e.g. Intel PT or CoreSight)
+PERF_RECORD_SWITCH_CPU_WIDE. In some cases (e.g. Intel PT, CoreSight or Arm SPE)
switch events will be enabled automatically, which can be suppressed by
by the option --no-switch-events.
'perf record --dry-run -e' can act as a BPF script compiler if llvm.dump-obj
in config file is set to true.
+--synth=TYPE::
+Collect and synthesize given type of events (comma separated). Note that
+this option controls the synthesis from the /proc filesystem which represent
+task status for pre-existing threads.
+
+Kernel (and some other) events are recorded regardless of the
+choice in this option. For example, --synth=no would have MMAP events for
+kernel and modules.
+
+Available types are:
+ 'task' - synthesize FORK and COMM events for each task
+ 'mmap' - synthesize MMAP events for each process (implies 'task')
+ 'cgroup' - synthesize CGROUP events for each cgroup
+ 'all' - synthesize all events (default)
+ 'no' - do not synthesize any of the above events
+
--tail-synthesize::
Instead of collecting non-sample events (for example, fork, comm, mmap) at
the beginning of record, collect them during finalizing an output file.
comm, tid, pid, time, cpu, event, trace, ip, sym, dso, addr, symoff,
srcline, period, iregs, uregs, brstack, brstacksym, flags, bpf-output,
brstackinsn, brstackoff, callindent, insn, insnlen, synth, phys_addr,
- metric, misc, srccode, ipc, data_page_size, code_page_size.
+ metric, misc, srccode, ipc, data_page_size, code_page_size, ins_lat.
Field list can be prepended with the type, trace, sw or hw,
to indicate to which event type the field list applies.
e.g., -F sw:comm,tid,time,ip,sym and -F trace:time,cpu,trace
HEADER_BPF_PROG_INFO = 25,
-struct bpf_prog_info_linear, which contains detailed information about
+struct perf_bpil, which contains detailed information about
a BPF program, including type, id, tag, jited/xlated instructions, etc.
HEADER_BPF_BTF = 26,
tools/lib/symbol/kallsyms.h
tools/lib/find_bit.c
tools/lib/bitmap.c
+tools/lib/list_sort.c
tools/lib/str_error_r.c
tools/lib/vsprintf.c
tools/lib/zalloc.c
scripts/bpf_doc.py
+tools/bpf/bpftool
+kernel/bpf/disasm.c
+kernel/bpf/disasm.h
CORE_CFLAGS += -Wextra
CORE_CFLAGS += -std=gnu99
-CXXFLAGS += -std=gnu++11 -fno-exceptions -fno-rtti
+CXXFLAGS += -std=gnu++14 -fno-exceptions -fno-rtti
CXXFLAGS += -Wall
CXXFLAGS += -fno-omit-frame-pointer
CXXFLAGS += -ggdb3
ifndef NO_AUXTRACE
$(call detected,CONFIG_AUXTRACE)
CFLAGS += -DHAVE_AUXTRACE_SUPPORT
+ ifeq ($(feature-reallocarray), 0)
+ CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
+ endif
endif
endif
$(call feature_check,libtraceevent)
ifeq ($(feature-libtraceevent), 1)
EXTLIBS += -ltraceevent
+ LIBTRACEEVENT_VERSION := $(shell $(PKG_CONFIG) --modversion libtraceevent)
+ LIBTRACEEVENT_VERSION_1 := $(word 1, $(subst ., ,$(LIBTRACEEVENT_VERSION)))
+ LIBTRACEEVENT_VERSION_2 := $(word 2, $(subst ., ,$(LIBTRACEEVENT_VERSION)))
+ LIBTRACEEVENT_VERSION_3 := $(word 3, $(subst ., ,$(LIBTRACEEVENT_VERSION)))
+ LIBTRACEEVENT_VERSION_CPP := $(shell expr $(LIBTRACEEVENT_VERSION_1) \* 255 \* 255 + $(LIBTRACEEVENT_VERSION_2) \* 255 + $(LIBTRACEEVENT_VERSION_3))
+ CFLAGS += -DLIBTRACEEVENT_VERSION=$(LIBTRACEEVENT_VERSION_CPP)
else
dummy := $(error Error: No libtraceevent devel library found, please install libtraceevent-devel);
endif
endif
+ifdef LIBTRACEFS_DYNAMIC
+ $(call feature_check,libtracefs)
+ ifeq ($(feature-libtracefs), 1)
+ EXTLIBS += -ltracefs
+ LIBTRACEFS_VERSION := $(shell $(PKG_CONFIG) --modversion libtracefs)
+ LIBTRACEFS_VERSION_1 := $(word 1, $(subst ., ,$(LIBTRACEFS_VERSION)))
+ LIBTRACEFS_VERSION_2 := $(word 2, $(subst ., ,$(LIBTRACEFS_VERSION)))
+ LIBTRACEFS_VERSION_3 := $(word 3, $(subst ., ,$(LIBTRACEFS_VERSION)))
+ LIBTRACEFS_VERSION_CPP := $(shell expr $(LIBTRACEFS_VERSION_1) \* 255 \* 255 + $(LIBTRACEFS_VERSION_2) \* 255 + $(LIBTRACEFS_VERSION_3))
+ CFLAGS += -DLIBTRACEFS_VERSION=$(LIBTRACEFS_VERSION_CPP)
+ else
+ dummy := $(error Error: No libtracefs devel library found, please install libtracefs-dev);
+ endif
+endif
+
# Among the variables below, these:
# perfexecdir
# perf_include_dir
#
# Define LIBTRACEEVENT_DYNAMIC to enable libtraceevent dynamic linking
#
+# Define LIBTRACEFS_DYNAMIC to enable libtracefs dynamic linking
+#
# As per kernel Makefile, avoid funny character set dependencies
unexport LC_ALL
LIB_DIR = $(srctree)/tools/lib/api/
TRACE_EVENT_DIR = $(srctree)/tools/lib/traceevent/
-BPF_DIR = $(srctree)/tools/lib/bpf/
+LIBBPF_DIR = $(srctree)/tools/lib/bpf/
SUBCMD_DIR = $(srctree)/tools/lib/subcmd/
LIBPERF_DIR = $(srctree)/tools/lib/perf/
DOC_DIR = $(srctree)/tools/perf/Documentation/
ifneq ($(OUTPUT),)
TE_PATH=$(OUTPUT)
PLUGINS_PATH=$(OUTPUT)
- BPF_PATH=$(OUTPUT)
SUBCMD_PATH=$(OUTPUT)
LIBPERF_PATH=$(OUTPUT)
ifneq ($(subdir),)
TE_PATH=$(TRACE_EVENT_DIR)
PLUGINS_PATH=$(TRACE_EVENT_DIR)plugins/
API_PATH=$(LIB_DIR)
- BPF_PATH=$(BPF_DIR)
SUBCMD_PATH=$(SUBCMD_DIR)
LIBPERF_PATH=$(LIBPERF_DIR)
endif
LIBAPI = $(API_PATH)libapi.a
export LIBAPI
-LIBBPF = $(BPF_PATH)libbpf.a
+ifneq ($(OUTPUT),)
+ LIBBPF_OUTPUT = $(abspath $(OUTPUT))/libbpf
+else
+ LIBBPF_OUTPUT = $(CURDIR)/libbpf
+endif
+LIBBPF_DESTDIR = $(LIBBPF_OUTPUT)
+LIBBPF_INCLUDE = $(LIBBPF_DESTDIR)/include
+LIBBPF = $(LIBBPF_OUTPUT)/libbpf.a
LIBSUBCMD = $(SUBCMD_PATH)libsubcmd.a
PROGRAMS += $(OUTPUT)$(LIBJVMTI)
endif
-DLFILTERS := dlfilter-test-api-v0.so
+DLFILTERS := dlfilter-test-api-v0.so dlfilter-show-cycles.so
DLFILTERS := $(patsubst %,$(OUTPUT)dlfilters/%,$(DLFILTERS))
# what 'all' will build and 'install' will install, in perfexecdir
$(kvm_ioctl_array): $(kvm_hdr_dir)/kvm.h $(kvm_ioctl_tbl)
$(Q)$(SHELL) '$(kvm_ioctl_tbl)' $(kvm_hdr_dir) > $@
-socket_ipproto_array := $(beauty_outdir)/socket_ipproto_array.c
-socket_ipproto_tbl := $(srctree)/tools/perf/trace/beauty/socket_ipproto.sh
+socket_arrays := $(beauty_outdir)/socket.c
+socket_tbl := $(srctree)/tools/perf/trace/beauty/socket.sh
-$(socket_ipproto_array): $(linux_uapi_dir)/in.h $(socket_ipproto_tbl)
- $(Q)$(SHELL) '$(socket_ipproto_tbl)' $(linux_uapi_dir) > $@
+$(socket_arrays): $(linux_uapi_dir)/in.h $(beauty_linux_dir)/socket.h $(socket_tbl)
+ $(Q)$(SHELL) '$(socket_tbl)' $(linux_uapi_dir) $(beauty_linux_dir) > $@
-socket_arrays := $(beauty_outdir)/socket_arrays.c
-socket_tbl := $(srctree)/tools/perf/trace/beauty/socket.sh
+sockaddr_arrays := $(beauty_outdir)/sockaddr.c
+sockaddr_tbl := $(srctree)/tools/perf/trace/beauty/sockaddr.sh
-$(socket_arrays): $(beauty_linux_dir)/socket.h $(socket_tbl)
- $(Q)$(SHELL) '$(socket_tbl)' $(beauty_linux_dir) > $@
+$(sockaddr_arrays): $(beauty_linux_dir)/socket.h $(sockaddr_tbl)
+ $(Q)$(SHELL) '$(sockaddr_tbl)' $(beauty_linux_dir) > $@
vhost_virtio_ioctl_array := $(beauty_ioctl_outdir)/vhost_virtio_ioctl_array.c
vhost_virtio_hdr_dir := $(srctree)/tools/include/uapi/linux
$(sndrv_ctl_ioctl_array) \
$(kcmp_type_array) \
$(kvm_ioctl_array) \
- $(socket_ipproto_array) \
$(socket_arrays) \
+ $(sockaddr_arrays) \
$(vhost_virtio_ioctl_array) \
$(madvise_behavior_array) \
$(mmap_flags_array) \
$(call QUIET_CLEAN, libapi)
$(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) clean >/dev/null
-$(LIBBPF): FORCE
- $(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) $(OUTPUT)libbpf.a FEATURES_DUMP=$(FEATURE_DUMP_EXPORT)
+$(LIBBPF): FORCE | $(LIBBPF_OUTPUT)
+ $(Q)$(MAKE) -C $(LIBBPF_DIR) FEATURES_DUMP=$(FEATURE_DUMP_EXPORT) \
+ O= OUTPUT=$(LIBBPF_OUTPUT)/ DESTDIR=$(LIBBPF_DESTDIR) prefix= \
+ $@ install_headers
$(LIBBPF)-clean:
$(call QUIET_CLEAN, libbpf)
- $(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) clean >/dev/null
+ $(Q)$(RM) -r -- $(LIBBPF_OUTPUT)
$(LIBPERF): FORCE
$(Q)$(MAKE) -C $(LIBPERF_DIR) EXTRA_CFLAGS="$(LIBPERF_CFLAGS)" O=$(OUTPUT) $(OUTPUT)libperf.a
SKELETONS += $(SKEL_OUT)/bperf_leader.skel.h $(SKEL_OUT)/bperf_follower.skel.h
SKELETONS += $(SKEL_OUT)/bperf_cgroup.skel.h
+$(SKEL_TMP_OUT) $(LIBBPF_OUTPUT):
+ $(Q)$(MKDIR) -p $@
+
ifdef BUILD_BPF_SKEL
BPFTOOL := $(SKEL_TMP_OUT)/bootstrap/bpftool
-LIBBPF_SRC := $(abspath ../lib/bpf)
-BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(BPF_PATH) -I$(LIBBPF_SRC)/..
-
-$(SKEL_TMP_OUT):
- $(Q)$(MKDIR) -p $@
+BPF_INCLUDE := -I$(SKEL_TMP_OUT)/.. -I$(LIBBPF_INCLUDE)
$(BPFTOOL): | $(SKEL_TMP_OUT)
- CFLAGS= $(MAKE) -C ../bpf/bpftool \
+ $(Q)CFLAGS= $(MAKE) -C ../bpf/bpftool \
OUTPUT=$(SKEL_TMP_OUT)/ bootstrap
VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux) \
$(OUTPUT)$(sndrv_pcm_ioctl_array) \
$(OUTPUT)$(kvm_ioctl_array) \
$(OUTPUT)$(kcmp_type_array) \
- $(OUTPUT)$(socket_ipproto_array) \
$(OUTPUT)$(socket_arrays) \
+ $(OUTPUT)$(sockaddr_arrays) \
$(OUTPUT)$(vhost_virtio_ioctl_array) \
$(OUTPUT)$(perf_ioctl_array) \
$(OUTPUT)$(prctl_option_array) \
#ifndef ARCH_TESTS_H
#define ARCH_TESTS_H
-extern struct test arch_tests[];
+extern struct test_suite *arch_tests[];
#endif
#include "tests/tests.h"
#include "arch-tests.h"
-struct test arch_tests[] = {
+struct test_suite *arch_tests[] = {
#ifdef HAVE_DWARF_UNWIND_SUPPORT
- {
- .desc = "DWARF unwind",
- .func = test__dwarf_unwind,
- },
+ &suite__dwarf_unwind,
#endif
- {
- .desc = "Vectors page",
- .func = test__vectors_page,
- },
- {
- .func = NULL,
- },
+ &suite__vectors_page,
+ NULL,
};
#define VECTORS__MAP_NAME "[vectors]"
-int test__vectors_page(struct test *test __maybe_unused,
- int subtest __maybe_unused)
+static int test__vectors_page(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
void *start, *end;
return TEST_OK;
}
+
+DEFINE_SUITE("Vectors page", vectors_page);
#ifndef ARCH_TESTS_H
#define ARCH_TESTS_H
-extern struct test arch_tests[];
+extern struct test_suite *arch_tests[];
#endif
#include "tests/tests.h"
#include "arch-tests.h"
-struct test arch_tests[] = {
+struct test_suite *arch_tests[] = {
#ifdef HAVE_DWARF_UNWIND_SUPPORT
- {
- .desc = "DWARF unwind",
- .func = test__dwarf_unwind,
- },
+ &suite__dwarf_unwind,
#endif
- {
- .func = NULL,
- },
+ NULL,
};
#include "../../../util/auxtrace.h"
#include "../../../util/record.h"
#include "../../../util/arm-spe.h"
+#include <tools/libc_compat.h> // reallocarray
#define KiB(x) ((x) * 1024)
#define MiB(x) ((x) * 1024 * 1024)
struct auxtrace_record itr;
struct perf_pmu *arm_spe_pmu;
struct evlist *evlist;
+ int wrapped_cnt;
+ bool *wrapped;
};
static void arm_spe_set_timestamp(struct auxtrace_record *itr,
return 0;
}
+static void
+arm_spe_snapshot_resolve_auxtrace_defaults(struct record_opts *opts,
+ bool privileged)
+{
+ /*
+ * The default snapshot size is the auxtrace mmap size. If neither auxtrace mmap size nor
+ * snapshot size is specified, then the default is 4MiB for privileged users, 128KiB for
+ * unprivileged users.
+ *
+ * The default auxtrace mmap size is 4MiB/page_size for privileged users, 128KiB for
+ * unprivileged users. If an unprivileged user does not specify mmap pages, the mmap pages
+ * will be reduced from the default 512KiB/page_size to 256KiB/page_size, otherwise the
+ * user is likely to get an error as they exceed their mlock limmit.
+ */
+
+ /*
+ * No size were given to '-S' or '-m,', so go with the default
+ */
+ if (!opts->auxtrace_snapshot_size && !opts->auxtrace_mmap_pages) {
+ if (privileged) {
+ opts->auxtrace_mmap_pages = MiB(4) / page_size;
+ } else {
+ opts->auxtrace_mmap_pages = KiB(128) / page_size;
+ if (opts->mmap_pages == UINT_MAX)
+ opts->mmap_pages = KiB(256) / page_size;
+ }
+ } else if (!opts->auxtrace_mmap_pages && !privileged && opts->mmap_pages == UINT_MAX) {
+ opts->mmap_pages = KiB(256) / page_size;
+ }
+
+ /*
+ * '-m,xyz' was specified but no snapshot size, so make the snapshot size as big as the
+ * auxtrace mmap area.
+ */
+ if (!opts->auxtrace_snapshot_size)
+ opts->auxtrace_snapshot_size = opts->auxtrace_mmap_pages * (size_t)page_size;
+
+ /*
+ * '-Sxyz' was specified but no auxtrace mmap area, so make the auxtrace mmap area big
+ * enough to fit the requested snapshot size.
+ */
+ if (!opts->auxtrace_mmap_pages) {
+ size_t sz = opts->auxtrace_snapshot_size;
+
+ sz = round_up(sz, page_size) / page_size;
+ opts->auxtrace_mmap_pages = roundup_pow_of_two(sz);
+ }
+}
+
static int arm_spe_recording_options(struct auxtrace_record *itr,
struct evlist *evlist,
struct record_opts *opts)
if (!opts->full_auxtrace)
return 0;
+ /*
+ * we are in snapshot mode.
+ */
+ if (opts->auxtrace_snapshot_mode) {
+ /*
+ * Command arguments '-Sxyz' and/or '-m,xyz' are missing, so fill those in with
+ * default values.
+ */
+ if (!opts->auxtrace_snapshot_size || !opts->auxtrace_mmap_pages)
+ arm_spe_snapshot_resolve_auxtrace_defaults(opts, privileged);
+
+ /*
+ * Snapshot size can't be bigger than the auxtrace area.
+ */
+ if (opts->auxtrace_snapshot_size > opts->auxtrace_mmap_pages * (size_t)page_size) {
+ pr_err("Snapshot size %zu must not be greater than AUX area tracing mmap size %zu\n",
+ opts->auxtrace_snapshot_size,
+ opts->auxtrace_mmap_pages * (size_t)page_size);
+ return -EINVAL;
+ }
+
+ /*
+ * Something went wrong somewhere - this shouldn't happen.
+ */
+ if (!opts->auxtrace_snapshot_size || !opts->auxtrace_mmap_pages) {
+ pr_err("Failed to calculate default snapshot size and/or AUX area tracing mmap pages\n");
+ return -EINVAL;
+ }
+ }
+
/* We are in full trace mode but '-m,xyz' wasn't specified */
if (!opts->auxtrace_mmap_pages) {
if (privileged) {
}
}
+ if (opts->auxtrace_snapshot_mode)
+ pr_debug2("%sx snapshot size: %zu\n", ARM_SPE_PMU_NAME,
+ opts->auxtrace_snapshot_size);
/*
* To obtain the auxtrace buffer file descriptor, the auxtrace event
tracking_evsel->core.attr.sample_period = 1;
/* In per-cpu case, always need the time of mmap events etc */
- if (!perf_cpu_map__empty(cpus))
+ if (!perf_cpu_map__empty(cpus)) {
evsel__set_sample_bit(tracking_evsel, TIME);
+ evsel__set_sample_bit(tracking_evsel, CPU);
+
+ /* also track task context switch */
+ if (!record_opts__no_switch_events(opts))
+ tracking_evsel->core.attr.context_switch = 1;
+ }
+
+ return 0;
+}
+
+static int arm_spe_parse_snapshot_options(struct auxtrace_record *itr __maybe_unused,
+ struct record_opts *opts,
+ const char *str)
+{
+ unsigned long long snapshot_size = 0;
+ char *endptr;
+
+ if (str) {
+ snapshot_size = strtoull(str, &endptr, 0);
+ if (*endptr || snapshot_size > SIZE_MAX)
+ return -1;
+ }
+
+ opts->auxtrace_snapshot_mode = true;
+ opts->auxtrace_snapshot_size = snapshot_size;
+
+ return 0;
+}
+
+static int arm_spe_snapshot_start(struct auxtrace_record *itr)
+{
+ struct arm_spe_recording *ptr =
+ container_of(itr, struct arm_spe_recording, itr);
+ struct evsel *evsel;
+
+ evlist__for_each_entry(ptr->evlist, evsel) {
+ if (evsel->core.attr.type == ptr->arm_spe_pmu->type)
+ return evsel__disable(evsel);
+ }
+ return -EINVAL;
+}
+
+static int arm_spe_snapshot_finish(struct auxtrace_record *itr)
+{
+ struct arm_spe_recording *ptr =
+ container_of(itr, struct arm_spe_recording, itr);
+ struct evsel *evsel;
+
+ evlist__for_each_entry(ptr->evlist, evsel) {
+ if (evsel->core.attr.type == ptr->arm_spe_pmu->type)
+ return evsel__enable(evsel);
+ }
+ return -EINVAL;
+}
+
+static int arm_spe_alloc_wrapped_array(struct arm_spe_recording *ptr, int idx)
+{
+ bool *wrapped;
+ int cnt = ptr->wrapped_cnt, new_cnt, i;
+
+ /*
+ * No need to allocate, so return early.
+ */
+ if (idx < cnt)
+ return 0;
+
+ /*
+ * Make ptr->wrapped as big as idx.
+ */
+ new_cnt = idx + 1;
+
+ /*
+ * Free'ed in arm_spe_recording_free().
+ */
+ wrapped = reallocarray(ptr->wrapped, new_cnt, sizeof(bool));
+ if (!wrapped)
+ return -ENOMEM;
+
+ /*
+ * init new allocated values.
+ */
+ for (i = cnt; i < new_cnt; i++)
+ wrapped[i] = false;
+
+ ptr->wrapped_cnt = new_cnt;
+ ptr->wrapped = wrapped;
+
+ return 0;
+}
+
+static bool arm_spe_buffer_has_wrapped(unsigned char *buffer,
+ size_t buffer_size, u64 head)
+{
+ u64 i, watermark;
+ u64 *buf = (u64 *)buffer;
+ size_t buf_size = buffer_size;
+
+ /*
+ * Defensively handle the case where head might be continually increasing - if its value is
+ * equal or greater than the size of the ring buffer, then we can safely determine it has
+ * wrapped around. Otherwise, continue to detect if head might have wrapped.
+ */
+ if (head >= buffer_size)
+ return true;
+
+ /*
+ * We want to look the very last 512 byte (chosen arbitrarily) in the ring buffer.
+ */
+ watermark = buf_size - 512;
+
+ /*
+ * The value of head is somewhere within the size of the ring buffer. This can be that there
+ * hasn't been enough data to fill the ring buffer yet or the trace time was so long that
+ * head has numerically wrapped around. To find we need to check if we have data at the
+ * very end of the ring buffer. We can reliably do this because mmap'ed pages are zeroed
+ * out and there is a fresh mapping with every new session.
+ */
+
+ /*
+ * head is less than 512 byte from the end of the ring buffer.
+ */
+ if (head > watermark)
+ watermark = head;
+
+ /*
+ * Speed things up by using 64 bit transactions (see "u64 *buf" above)
+ */
+ watermark /= sizeof(u64);
+ buf_size /= sizeof(u64);
+
+ /*
+ * If we find trace data at the end of the ring buffer, head has been there and has
+ * numerically wrapped around at least once.
+ */
+ for (i = watermark; i < buf_size; i++)
+ if (buf[i])
+ return true;
+
+ return false;
+}
+
+static int arm_spe_find_snapshot(struct auxtrace_record *itr, int idx,
+ struct auxtrace_mmap *mm, unsigned char *data,
+ u64 *head, u64 *old)
+{
+ int err;
+ bool wrapped;
+ struct arm_spe_recording *ptr =
+ container_of(itr, struct arm_spe_recording, itr);
+
+ /*
+ * Allocate memory to keep track of wrapping if this is the first
+ * time we deal with this *mm.
+ */
+ if (idx >= ptr->wrapped_cnt) {
+ err = arm_spe_alloc_wrapped_array(ptr, idx);
+ if (err)
+ return err;
+ }
+
+ /*
+ * Check to see if *head has wrapped around. If it hasn't only the
+ * amount of data between *head and *old is snapshot'ed to avoid
+ * bloating the perf.data file with zeros. But as soon as *head has
+ * wrapped around the entire size of the AUX ring buffer it taken.
+ */
+ wrapped = ptr->wrapped[idx];
+ if (!wrapped && arm_spe_buffer_has_wrapped(data, mm->len, *head)) {
+ wrapped = true;
+ ptr->wrapped[idx] = true;
+ }
+
+ pr_debug3("%s: mmap index %d old head %zu new head %zu size %zu\n",
+ __func__, idx, (size_t)*old, (size_t)*head, mm->len);
+
+ /*
+ * No wrap has occurred, we can just use *head and *old.
+ */
+ if (!wrapped)
+ return 0;
+
+ /*
+ * *head has wrapped around - adjust *head and *old to pickup the
+ * entire content of the AUX buffer.
+ */
+ if (*head >= mm->len) {
+ *old = *head - mm->len;
+ } else {
+ *head += mm->len;
+ *old = *head - mm->len;
+ }
return 0;
}
struct arm_spe_recording *sper =
container_of(itr, struct arm_spe_recording, itr);
+ free(sper->wrapped);
free(sper);
}
sper->arm_spe_pmu = arm_spe_pmu;
sper->itr.pmu = arm_spe_pmu;
+ sper->itr.snapshot_start = arm_spe_snapshot_start;
+ sper->itr.snapshot_finish = arm_spe_snapshot_finish;
+ sper->itr.find_snapshot = arm_spe_find_snapshot;
+ sper->itr.parse_snapshot_options = arm_spe_parse_snapshot_options;
sper->itr.recording_options = arm_spe_recording_options;
sper->itr.info_priv_size = arm_spe_info_priv_size;
sper->itr.info_fill = arm_spe_info_fill;
#include "../../../util/cpumap.h"
#include "../../../util/pmu.h"
-struct pmu_events_map *pmu_events_map__find(void)
+const struct pmu_events_map *pmu_events_map__find(void)
{
struct perf_pmu *pmu = NULL;
446 n64 landlock_restrict_self sys_landlock_restrict_self
# 447 reserved for memfd_secret
448 n64 process_mrelease sys_process_mrelease
+449 n64 futex_waitv sys_futex_waitv
#ifndef ARCH_TESTS_H
#define ARCH_TESTS_H
-extern struct test arch_tests[];
+extern struct test_suite *arch_tests[];
#endif
[PERF_REG_POWERPC_PMC4] = "pmc4",
[PERF_REG_POWERPC_PMC5] = "pmc5",
[PERF_REG_POWERPC_PMC6] = "pmc6",
+ [PERF_REG_POWERPC_SDAR] = "sdar",
+ [PERF_REG_POWERPC_SIAR] = "siar",
};
static inline const char *__perf_reg_name(int id)
#include "tests/tests.h"
#include "arch-tests.h"
-struct test arch_tests[] = {
+
+struct test_suite *arch_tests[] = {
#ifdef HAVE_DWARF_UNWIND_SUPPORT
- {
- .desc = "Test dwarf unwind",
- .func = test__dwarf_unwind,
- },
+ &suite__dwarf_unwind,
#endif
- {
- .func = NULL,
- },
+ NULL,
};
return bufp;
}
-int arch_get_runtimeparam(struct pmu_event *pe)
+int arch_get_runtimeparam(const struct pmu_event *pe)
{
int count;
char path[PATH_MAX] = "/devices/hv_24x7/interface/";
struct parse_events_error err;
int ret;
- bzero(&err, sizeof(err));
+ parse_events_error__init(&err);
ret = parse_events(evlist, str, &err);
if (err.str)
- parse_events_print_error(&err, "tracepoint");
+ parse_events_error__print(&err, "tracepoint");
+ parse_events_error__exit(&err);
return ret;
}
SMPL_REG(pmc4, PERF_REG_POWERPC_PMC4),
SMPL_REG(pmc5, PERF_REG_POWERPC_PMC5),
SMPL_REG(pmc6, PERF_REG_POWERPC_PMC6),
+ SMPL_REG(sdar, PERF_REG_POWERPC_SDAR),
+ SMPL_REG(siar, PERF_REG_POWERPC_SIAR),
SMPL_REG_END
};
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+static
+struct ins_ops *riscv64__associate_ins_ops(struct arch *arch, const char *name)
+{
+ struct ins_ops *ops = NULL;
+
+ if (!strncmp(name, "jal", 3) ||
+ !strncmp(name, "jr", 2) ||
+ !strncmp(name, "call", 4))
+ ops = &call_ops;
+ else if (!strncmp(name, "ret", 3))
+ ops = &ret_ops;
+ else if (name[0] == 'j' || name[0] == 'b')
+ ops = &jump_ops;
+ else
+ return NULL;
+
+ arch__associate_ins_ops(arch, name, ops);
+
+ return ops;
+}
+
+static
+int riscv64__annotate_init(struct arch *arch, char *cpuid __maybe_unused)
+{
+ if (!arch->initialized) {
+ arch->associate_instruction_ops = riscv64__associate_ins_ops;
+ arch->initialized = true;
+ arch->objdump.comment_char = '#';
+ }
+
+ return 0;
+}
{ .name = "xorps", .ops = &mov_ops, },
};
-static bool x86__ins_is_fused(struct arch *arch, const char *ins1,
+static bool amd__ins_is_fused(struct arch *arch, const char *ins1,
const char *ins2)
{
+ if (strstr(ins2, "jmp"))
+ return false;
+
+ /* Family >= 15h supports cmp/test + branch fusion */
+ if (arch->family >= 0x15 && (strstarts(ins1, "test") ||
+ (strstarts(ins1, "cmp") && !strstr(ins1, "xchg")))) {
+ return true;
+ }
+
+ /* Family >= 19h supports some ALU + branch fusion */
+ if (arch->family >= 0x19 && (strstarts(ins1, "add") ||
+ strstarts(ins1, "sub") || strstarts(ins1, "and") ||
+ strstarts(ins1, "inc") || strstarts(ins1, "dec") ||
+ strstarts(ins1, "or") || strstarts(ins1, "xor"))) {
+ return true;
+ }
+
+ return false;
+}
+
+static bool intel__ins_is_fused(struct arch *arch, const char *ins1,
+ const char *ins2)
+{
if (arch->family != 6 || arch->model < 0x1e || strstr(ins2, "jmp"))
return false;
if (ret == 3) {
arch->family = family;
arch->model = model;
+ arch->ins_is_fused = strstarts(cpuid, "AuthenticAMD") ?
+ amd__ins_is_fused :
+ intel__ins_is_fused;
return 0;
}
446 common landlock_restrict_self sys_landlock_restrict_self
447 common memfd_secret sys_memfd_secret
448 common process_mrelease sys_process_mrelease
+449 common futex_waitv sys_futex_waitv
#
# Due to a historical design error, certain syscalls are numbered differently
#ifndef ARCH_TESTS_H
#define ARCH_TESTS_H
-struct test;
+struct test_suite;
/* Tests */
-int test__rdpmc(struct test *test, int subtest);
-int test__insn_x86(struct test *test, int subtest);
-int test__intel_pt_pkt_decoder(struct test *test, int subtest);
-int test__bp_modify(struct test *test, int subtest);
-int test__x86_sample_parsing(struct test *test, int subtest);
+int test__rdpmc(struct test_suite *test, int subtest);
+int test__insn_x86(struct test_suite *test, int subtest);
+int test__intel_pt_pkt_decoder(struct test_suite *test, int subtest);
+int test__bp_modify(struct test_suite *test, int subtest);
+int test__x86_sample_parsing(struct test_suite *test, int subtest);
-extern struct test arch_tests[];
+extern struct test_suite *arch_tests[];
#endif
#include "tests/tests.h"
#include "arch-tests.h"
-struct test arch_tests[] = {
- {
- .desc = "x86 rdpmc",
- .func = test__rdpmc,
- },
+DEFINE_SUITE("x86 rdpmc", rdpmc);
+#ifdef HAVE_AUXTRACE_SUPPORT
+DEFINE_SUITE("x86 instruction decoder - new instructions", insn_x86);
+DEFINE_SUITE("Intel PT packet decoder", intel_pt_pkt_decoder);
+#endif
+#if defined(__x86_64__)
+DEFINE_SUITE("x86 bp modify", bp_modify);
+#endif
+DEFINE_SUITE("x86 Sample parsing", x86_sample_parsing);
+
+struct test_suite *arch_tests[] = {
+ &suite__rdpmc,
#ifdef HAVE_DWARF_UNWIND_SUPPORT
- {
- .desc = "DWARF unwind",
- .func = test__dwarf_unwind,
- },
+ &suite__dwarf_unwind,
#endif
#ifdef HAVE_AUXTRACE_SUPPORT
- {
- .desc = "x86 instruction decoder - new instructions",
- .func = test__insn_x86,
- },
- {
- .desc = "Intel PT packet decoder",
- .func = test__intel_pt_pkt_decoder,
- },
+ &suite__insn_x86,
+ &suite__intel_pt_pkt_decoder,
#endif
#if defined(__x86_64__)
- {
- .desc = "x86 bp modify",
- .func = test__bp_modify,
- },
+ &suite__bp_modify,
#endif
- {
- .desc = "x86 Sample parsing",
- .func = test__x86_sample_parsing,
- },
- {
- .func = NULL,
- },
-
+ &suite__x86_sample_parsing,
+ NULL,
};
return rip == (unsigned long) bp_1 ? TEST_OK : TEST_FAIL;
}
-int test__bp_modify(struct test *test __maybe_unused,
+int test__bp_modify(struct test_suite *test __maybe_unused,
int subtest __maybe_unused)
{
TEST_ASSERT_VAL("modify test 1 failed\n", !bp_modify1());
* verbose (-v) option to see all the instructions and whether or not they
* decoded successfully.
*/
-int test__insn_x86(struct test *test __maybe_unused, int subtest __maybe_unused)
+int test__insn_x86(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret = 0;
* the last read counter value to avoid triggering a WARN_ON_ONCE() in
* smp_call_function_many() caused by sending IPIs from NMI context.
*/
-int test__intel_cqm_count_nmi_context(struct test *test __maybe_unused, int subtest __maybe_unused)
+int test__intel_cqm_count_nmi_context(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct evlist *evlist = NULL;
struct evsel *evsel = NULL;
* This test feeds byte sequences to the Intel PT packet decoder and checks the
* results. Changes to the packet context are also checked.
*/
-int test__intel_pt_pkt_decoder(struct test *test __maybe_unused, int subtest __maybe_unused)
+int test__intel_pt_pkt_decoder(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct test_data *d = data;
int ret;
return 0;
}
-int test__rdpmc(struct test *test __maybe_unused, int subtest __maybe_unused)
+int test__rdpmc(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int status = 0;
int wret = 0;
* For now, the PERF_SAMPLE_WEIGHT_STRUCT is the only X86 specific sample type.
* The test only checks the PERF_SAMPLE_WEIGHT_STRUCT type.
*/
-int test__x86_sample_parsing(struct test *test __maybe_unused, int subtest __maybe_unused)
+int test__x86_sample_parsing(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
return do_test(PERF_SAMPLE_WEIGHT_STRUCT);
}
// SPDX-License-Identifier: GPL-2.0
#include <stdio.h>
+#include <stdlib.h>
#include "util/evsel.h"
+#include "util/env.h"
+#include "linux/string.h"
void arch_evsel__set_sample_weight(struct evsel *evsel)
{
evsel__set_sample_bit(evsel, WEIGHT_STRUCT);
}
+
+void arch_evsel__fixup_new_cycles(struct perf_event_attr *attr)
+{
+ struct perf_env env = { .total_mem = 0, } ;
+
+ if (!perf_env__cpuid(&env))
+ return;
+
+ /*
+ * On AMD, precise cycles event sampling internally uses IBS pmu.
+ * But IBS does not have filtering capabilities and perf by default
+ * sets exclude_guest = 1. This makes IBS pmu event init fail and
+ * thus perf ends up doing non-precise sampling. Avoid it by clearing
+ * exclude_guest.
+ */
+ if (env.cpuid && strstarts(env.cpuid, "AuthenticAMD"))
+ attr->exclude_guest = 0;
+
+ free(env.cpuid);
+}
static int nr_events = 1;
static const char *event_string = "dummy";
+static inline u64 timeval2usec(struct timeval *tv)
+{
+ return tv->tv_sec * USEC_PER_SEC + tv->tv_usec;
+}
+
static struct record_opts opts = {
.sample_time = true,
.mmap_pages = UINT_MAX,
static struct evlist *bench__create_evlist(char *evstr)
{
- struct parse_events_error err = { .idx = 0, };
+ struct parse_events_error err;
struct evlist *evlist = evlist__new();
int ret;
return NULL;
}
+ parse_events_error__init(&err);
ret = parse_events(evlist, evstr, &err);
if (ret) {
- parse_events_print_error(&err, evstr);
+ parse_events_error__print(&err, evstr);
+ parse_events_error__exit(&err);
pr_err("Run 'perf list' for a list of valid events\n");
ret = 1;
goto out_delete_evlist;
}
-
+ parse_events_error__exit(&err);
ret = evlist__create_maps(evlist, &opts.target);
if (ret < 0) {
pr_err("Not enough memory to create thread/cpu maps\n");
gettimeofday(&end, NULL);
timersub(&end, &start, &diff);
- runtime_us = diff.tv_sec * USEC_PER_SEC + diff.tv_usec;
+ runtime_us = timeval2usec(&diff);
update_stats(&time_stats, runtime_us);
evlist__delete(evlist);
print_summary();
free(worker);
+ perf_cpu_map__put(cpu);
return ret;
err:
usage_with_options(bench_futex_lock_pi_usage, options);
print_summary();
free(worker);
+ perf_cpu_map__put(cpu);
return ret;
err:
usage_with_options(bench_futex_requeue_usage, options);
print_summary();
free(blocked_worker);
+ perf_cpu_map__put(cpu);
return ret;
}
#endif /* HAVE_PTHREAD_BARRIER */
print_summary();
free(worker);
+ perf_cpu_map__put(cpu);
return ret;
}
};
/**
- * futex() - SYS_futex syscall wrapper
+ * futex_syscall() - SYS_futex syscall wrapper
* @uaddr: address of first futex
* @op: futex op code
* @val: typically expected value of uaddr, but varies by op
* @val3: varies by op
* @opflags: flags to be bitwise OR'd with op, such as FUTEX_PRIVATE_FLAG
*
- * futex() is used by all the following futex op wrappers. It can also be
+ * futex_syscall() is used by all the following futex op wrappers. It can also be
* used for misuse and abuse testing. Generally, the specific op wrappers
- * should be used instead. It is a macro instead of an static inline function as
- * some of the types over overloaded (timeout is used for nr_requeue for
- * example).
+ * should be used instead.
*
* These argument descriptions are the defaults for all
* like-named arguments in the following wrappers except where noted below.
*/
-#define futex(uaddr, op, val, timeout, uaddr2, val3, opflags) \
- syscall(SYS_futex, uaddr, op | opflags, val, timeout, uaddr2, val3)
+static inline int
+futex_syscall(volatile u_int32_t *uaddr, int op, u_int32_t val, struct timespec *timeout,
+ volatile u_int32_t *uaddr2, int val3, int opflags)
+{
+ return syscall(SYS_futex, uaddr, op | opflags, val, timeout, uaddr2, val3);
+}
+
+static inline int
+futex_syscall_nr_requeue(volatile u_int32_t *uaddr, int op, u_int32_t val, int nr_requeue,
+ volatile u_int32_t *uaddr2, int val3, int opflags)
+{
+ return syscall(SYS_futex, uaddr, op | opflags, val, nr_requeue, uaddr2, val3);
+}
/**
* futex_wait() - block on uaddr with optional timeout
static inline int
futex_wait(u_int32_t *uaddr, u_int32_t val, struct timespec *timeout, int opflags)
{
- return futex(uaddr, FUTEX_WAIT, val, timeout, NULL, 0, opflags);
+ return futex_syscall(uaddr, FUTEX_WAIT, val, timeout, NULL, 0, opflags);
}
/**
static inline int
futex_wake(u_int32_t *uaddr, int nr_wake, int opflags)
{
- return futex(uaddr, FUTEX_WAKE, nr_wake, NULL, NULL, 0, opflags);
+ return futex_syscall(uaddr, FUTEX_WAKE, nr_wake, NULL, NULL, 0, opflags);
}
/**
static inline int
futex_lock_pi(u_int32_t *uaddr, struct timespec *timeout, int opflags)
{
- return futex(uaddr, FUTEX_LOCK_PI, 0, timeout, NULL, 0, opflags);
+ return futex_syscall(uaddr, FUTEX_LOCK_PI, 0, timeout, NULL, 0, opflags);
}
/**
static inline int
futex_unlock_pi(u_int32_t *uaddr, int opflags)
{
- return futex(uaddr, FUTEX_UNLOCK_PI, 0, NULL, NULL, 0, opflags);
+ return futex_syscall(uaddr, FUTEX_UNLOCK_PI, 0, NULL, NULL, 0, opflags);
}
/**
futex_cmp_requeue(u_int32_t *uaddr, u_int32_t val, u_int32_t *uaddr2, int nr_wake,
int nr_requeue, int opflags)
{
- return futex(uaddr, FUTEX_CMP_REQUEUE, nr_wake, nr_requeue, uaddr2,
- val, opflags);
+ return futex_syscall_nr_requeue(uaddr, FUTEX_CMP_REQUEUE, nr_wake, nr_requeue, uaddr2,
+ val, opflags);
}
/**
futex_wait_requeue_pi(u_int32_t *uaddr, u_int32_t val, u_int32_t *uaddr2,
struct timespec *timeout, int opflags)
{
- return futex(uaddr, FUTEX_WAIT_REQUEUE_PI, val, timeout, uaddr2, 0,
- opflags);
+ return futex_syscall(uaddr, FUTEX_WAIT_REQUEUE_PI, val, timeout, uaddr2, 0,
+ opflags);
}
/**
futex_cmp_requeue_pi(u_int32_t *uaddr, u_int32_t val, u_int32_t *uaddr2,
int nr_requeue, int opflags)
{
- return futex(uaddr, FUTEX_CMP_REQUEUE_PI, 1, nr_requeue, uaddr2,
- val, opflags);
+ return futex_syscall_nr_requeue(uaddr, FUTEX_CMP_REQUEUE_PI, 1, nr_requeue, uaddr2,
+ val, opflags);
}
#endif /* _FUTEX_H */
snd_ctx->out_fds[i] = fds[1];
if (!thread_mode)
close(fds[0]);
+
+ free(ctx);
}
/* Now we have all the fds, fork the senders */
for (i = 0; i < num_fds; i++)
close(snd_ctx->out_fds[i]);
+ free(snd_ctx);
+
/* Return number of children to reap */
return num_fds * 2;
}
NULL,
target, threads,
process_synthesized_event,
- data_mmap,
+ true, data_mmap,
nr_threads_synthesize);
if (err)
return err;
NULL,
target, NULL,
process_synthesized_event,
- false,
+ true, false,
nr_threads_synthesize);
if (err) {
perf_session__delete(session);
return ret;
}
+ ret = symbol__validate_sym_arguments();
+ if (ret)
+ return ret;
+
if (quiet)
perf_quiet_option();
if (c2c.stats_only)
c2c.use_stdio = true;
+ err = symbol__validate_sym_arguments();
+ if (err)
+ goto out;
+
if (!input_name || !strlen(input_name))
input_name = "perf.data";
#ifndef F_TLOCK
#define F_TLOCK 2
-#include <sys/file.h>
-
static int lockf(int fd, int cmd, off_t len)
{
if (cmd != F_TLOCK || len != 0)
static int send_cmd_list(struct daemon *daemon)
{
- union cmd cmd = { .cmd = CMD_LIST, };
+ union cmd cmd;
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.list.cmd = CMD_LIST;
cmd.list.verbose = verbose;
cmd.list.csv_sep = daemon->csv_sep ? *daemon->csv_sep : 0;
return -1;
}
+ memset(&cmd, 0, sizeof(cmd));
cmd.signal.cmd = CMD_SIGNAL,
cmd.signal.sig = SIGUSR2;
strncpy(cmd.signal.name, name, sizeof(cmd.signal.name) - 1);
OPT_PARENT(parent_options),
OPT_END()
};
- union cmd cmd = { .cmd = CMD_STOP, };
+ union cmd cmd;
argc = parse_options(argc, argv, start_options, daemon_usage, 0);
if (argc)
return -1;
}
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.cmd = CMD_STOP;
return send_cmd(daemon, &cmd);
}
OPT_PARENT(parent_options),
OPT_END()
};
- union cmd cmd = { .cmd = CMD_PING, };
+ union cmd cmd;
argc = parse_options(argc, argv, ping_options, daemon_usage, 0);
if (argc)
return -1;
}
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.cmd = CMD_PING;
scnprintf(cmd.ping.name, sizeof(cmd.ping.name), "%s", name);
return send_cmd(daemon, &cmd);
}
inject->tool.auxtrace_info = perf_event__process_auxtrace_info;
inject->tool.auxtrace = perf_event__process_auxtrace;
inject->tool.aux = perf_event__drop_aux;
- inject->tool.itrace_start = perf_event__drop_aux,
+ inject->tool.itrace_start = perf_event__drop_aux;
+ inject->tool.aux_output_hw_id = perf_event__drop_aux;
inject->tool.ordered_events = true;
inject->tool.ordering_requires_timestamps = true;
/* Allow space in the header for new attributes */
.lost_samples = perf_event__repipe,
.aux = perf_event__repipe,
.itrace_start = perf_event__repipe,
+ .aux_output_hw_id = perf_event__repipe,
.context_switch = perf_event__repipe,
.throttle = perf_event__repipe,
.unthrottle = perf_event__repipe,
#endif
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show build ids, etc)"),
+ OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name,
+ "file", "vmlinux pathname"),
+ OPT_BOOLEAN(0, "ignore-vmlinux", &symbol_conf.ignore_vmlinux,
+ "don't load vmlinux even if found"),
OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name, "file",
"kallsyms pathname"),
OPT_BOOLEAN('f', "force", &data.force, "don't complain, do it"),
return -1;
}
+ if (symbol__validate_sym_arguments())
+ return -1;
+
if (inject.in_place_update) {
if (!strcmp(inject.input_name, "-")) {
pr_err("Input file name required for in-place updating\n");
perf_session__set_id_hdr_size(kvm->session);
ordered_events__set_copy_on_queue(&kvm->session->ordered_events, true);
machine__synthesize_threads(&kvm->session->machines.host, &kvm->opts.target,
- kvm->evlist->core.threads, false, 1);
+ kvm->evlist->core.threads, true, false, 1);
err = kvm_live_open_events(kvm);
if (err)
goto out;
#include "util/parse-events.h"
#include "util/pmu.h"
+#include "util/pmu-hybrid.h"
#include "util/debug.h"
#include "util/metricgroup.h"
#include <subcmd/pager.h>
static bool desc_flag = true;
static bool details_flag;
+static const char *hybrid_type;
int cmd_list(int argc, const char **argv)
{
- int i;
+ int i, ret = 0;
bool raw_dump = false;
bool long_desc_flag = false;
bool deprecated = false;
+ char *pmu_name = NULL;
struct option list_options[] = {
OPT_BOOLEAN(0, "raw-dump", &raw_dump, "Dump raw events"),
OPT_BOOLEAN('d', "desc", &desc_flag,
"Print information on the perf event names and expressions used internally by events."),
OPT_BOOLEAN(0, "deprecated", &deprecated,
"Print deprecated events."),
+ OPT_STRING(0, "cputype", &hybrid_type, "hybrid cpu type",
+ "Print events applying cpu with this type for hybrid platform "
+ "(e.g. core or atom)"),
OPT_INCR(0, "debug", &verbose,
"Enable debugging output"),
OPT_END()
if (!raw_dump && pager_in_use())
printf("\nList of pre-defined events (to be used in -e):\n\n");
+ if (hybrid_type) {
+ pmu_name = perf_pmu__hybrid_type_to_pmu(hybrid_type);
+ if (!pmu_name)
+ pr_warning("WARNING: hybrid cputype is not supported!\n");
+ }
+
if (argc == 0) {
print_events(NULL, raw_dump, !desc_flag, long_desc_flag,
- details_flag, deprecated);
- return 0;
+ details_flag, deprecated, pmu_name);
+ goto out;
}
for (i = 0; i < argc; ++i) {
else if (strcmp(argv[i], "pmu") == 0)
print_pmu_events(NULL, raw_dump, !desc_flag,
long_desc_flag, details_flag,
- deprecated);
+ deprecated, pmu_name);
else if (strcmp(argv[i], "sdt") == 0)
print_sdt_events(NULL, NULL, raw_dump);
else if (strcmp(argv[i], "metric") == 0 || strcmp(argv[i], "metrics") == 0)
- metricgroup__print(true, false, NULL, raw_dump, details_flag);
+ metricgroup__print(true, false, NULL, raw_dump, details_flag, pmu_name);
else if (strcmp(argv[i], "metricgroup") == 0 || strcmp(argv[i], "metricgroups") == 0)
- metricgroup__print(false, true, NULL, raw_dump, details_flag);
+ metricgroup__print(false, true, NULL, raw_dump, details_flag, pmu_name);
else if ((sep = strchr(argv[i], ':')) != NULL) {
int sep_idx;
sep_idx = sep - argv[i];
s = strdup(argv[i]);
- if (s == NULL)
- return -1;
+ if (s == NULL) {
+ ret = -1;
+ goto out;
+ }
s[sep_idx] = '\0';
print_tracepoint_events(s, s + sep_idx + 1, raw_dump);
print_sdt_events(s, s + sep_idx + 1, raw_dump);
- metricgroup__print(true, true, s, raw_dump, details_flag);
+ metricgroup__print(true, true, s, raw_dump, details_flag, pmu_name);
free(s);
} else {
if (asprintf(&s, "*%s*", argv[i]) < 0) {
print_pmu_events(s, raw_dump, !desc_flag,
long_desc_flag,
details_flag,
- deprecated);
+ deprecated,
+ pmu_name);
print_tracepoint_events(NULL, s, raw_dump);
print_sdt_events(NULL, s, raw_dump);
- metricgroup__print(true, true, s, raw_dump, details_flag);
+ metricgroup__print(true, true, s, raw_dump, details_flag, pmu_name);
free(s);
}
}
- return 0;
+
+out:
+ free(pmu_name);
+ return ret;
}
#include "util/build-id.h"
#include "util/strlist.h"
#include "util/strfilter.h"
+#include "util/symbol.h"
#include "util/symbol_conf.h"
#include "util/debug.h"
#include <subcmd/parse-options.h>
params.command = 'a';
}
+ ret = symbol__validate_sym_arguments();
+ if (ret)
+ return ret;
+
if (params.quiet) {
if (verbose != 0) {
pr_err(" Error: -v and -q are exclusive.\n");
{
int err;
struct perf_thread_map *thread_map;
+ bool needs_mmap = rec->opts.synth & PERF_SYNTH_MMAP;
if (rec->opts.tail_synthesize != tail)
return 0;
err = perf_event__synthesize_thread_map(&rec->tool, thread_map,
process_synthesized_event,
&rec->session->machines.host,
+ needs_mmap,
rec->opts.sample_address);
perf_thread_map__put(thread_map);
return err;
goto out;
/* Synthesize id_index before auxtrace_info */
- if (rec->opts.auxtrace_sample_mode) {
+ if (rec->opts.auxtrace_sample_mode || rec->opts.full_auxtrace) {
err = perf_event__synthesize_id_index(tool,
process_synthesized_event,
session->evlist, machine);
if (err < 0)
pr_warning("Couldn't synthesize bpf events.\n");
- err = perf_event__synthesize_cgroups(tool, process_synthesized_event,
- machine);
- if (err < 0)
- pr_warning("Couldn't synthesize cgroup events.\n");
+ if (rec->opts.synth & PERF_SYNTH_CGROUP) {
+ err = perf_event__synthesize_cgroups(tool, process_synthesized_event,
+ machine);
+ if (err < 0)
+ pr_warning("Couldn't synthesize cgroup events.\n");
+ }
if (rec->opts.nr_threads_synthesize > 1) {
perf_set_multithreaded();
f = process_locked_synthesized_event;
}
- err = __machine__synthesize_threads(machine, tool, &opts->target, rec->evlist->core.threads,
- f, opts->sample_address,
- rec->opts.nr_threads_synthesize);
+ if (rec->opts.synth & PERF_SYNTH_TASK) {
+ bool needs_mmap = rec->opts.synth & PERF_SYNTH_MMAP;
+
+ err = __machine__synthesize_threads(machine, tool, &opts->target,
+ rec->evlist->core.threads,
+ f, needs_mmap, opts->sample_address,
+ rec->opts.nr_threads_synthesize);
+ }
if (rec->opts.nr_threads_synthesize > 1)
perf_set_singlethreaded();
return 0;
}
+static int parse_record_synth_option(const struct option *opt,
+ const char *str,
+ int unset __maybe_unused)
+{
+ struct record_opts *opts = opt->value;
+ char *p = strdup(str);
+
+ if (p == NULL)
+ return -1;
+
+ opts->synth = parse_synth_opt(p);
+ free(p);
+
+ if (opts->synth < 0) {
+ pr_err("Invalid synth option: %s\n", str);
+ return -1;
+ }
+ return 0;
+}
+
/*
* XXX Ideally would be local to cmd_record() and passed to a record__new
* because we need to have access to it in record__exit, that is called
.nr_threads_synthesize = 1,
.ctl_fd = -1,
.ctl_fd_ack = -1,
+ .synth = PERF_SYNTH_ALL,
},
.tool = {
.sample = process_sample_event,
"\t\t\t Optionally send control command completion ('ack\\n') to ack-fd descriptor.\n"
"\t\t\t Alternatively, ctl-fifo / ack-fifo will be opened and used as ctl-fd / ack-fd.",
parse_control_option),
+ OPT_CALLBACK(0, "synth", &record.opts, "no|all|task|mmap|cgroup",
+ "Fine-tune event synthesis: default=all", parse_record_synth_option),
OPT_END()
};
if (quiet)
perf_quiet_option();
+ err = symbol__validate_sym_arguments();
+ if (err)
+ return err;
+
/* Make system wide (-a) the default target. */
if (!argc && target__none(&rec->opts.target))
rec->opts.target.system_wide = true;
int ret;
struct perf_session *session = rep->session;
struct evlist *evlist = session->evlist;
- const char *help = perf_tip(system_path(TIPDIR));
+ char *help = NULL, *path = NULL;
- if (help == NULL) {
+ path = system_path(TIPDIR);
+ if (perf_tip(&help, path) || help == NULL) {
/* fallback for people who don't install perf ;-) */
- help = perf_tip(DOCDIR);
- if (help == NULL)
- help = "Cannot load tips.txt file, please install perf!";
+ free(path);
+ path = system_path(DOCDIR);
+ if (perf_tip(&help, path) || help == NULL)
+ help = strdup("Cannot load tips.txt file, please install perf!");
}
+ free(path);
switch (use_browser) {
case 1:
ret = evlist__tty_browse_hists(evlist, rep, help);
break;
}
-
+ free(help);
return ret;
}
if (quiet)
perf_quiet_option();
- if (symbol_conf.vmlinux_name &&
- access(symbol_conf.vmlinux_name, R_OK)) {
- pr_err("Invalid file: %s\n", symbol_conf.vmlinux_name);
- ret = -EINVAL;
- goto exit;
- }
- if (symbol_conf.kallsyms_name &&
- access(symbol_conf.kallsyms_name, R_OK)) {
- pr_err("Invalid file: %s\n", symbol_conf.kallsyms_name);
- ret = -EINVAL;
+ ret = symbol__validate_sym_arguments();
+ if (ret)
goto exit;
- }
if (report.inverted_callchain)
callchain_param.order = ORDER_CALLER;
.fork_event = replay_fork_event,
};
unsigned int i;
+ int ret;
for (i = 0; i < ARRAY_SIZE(sched.curr_pid); i++)
sched.curr_pid[i] = -1;
parse_options_usage(NULL, timehist_options, "n", true);
return -EINVAL;
}
+ ret = symbol__validate_sym_arguments();
+ if (ret)
+ return ret;
return perf_sched__timehist(&sched);
} else {
PERF_OUTPUT_TOD = 1ULL << 32,
PERF_OUTPUT_DATA_PAGE_SIZE = 1ULL << 33,
PERF_OUTPUT_CODE_PAGE_SIZE = 1ULL << 34,
+ PERF_OUTPUT_INS_LAT = 1ULL << 35,
};
struct perf_script {
{.str = "tod", .field = PERF_OUTPUT_TOD},
{.str = "data_page_size", .field = PERF_OUTPUT_DATA_PAGE_SIZE},
{.str = "code_page_size", .field = PERF_OUTPUT_CODE_PAGE_SIZE},
+ {.str = "ins_lat", .field = PERF_OUTPUT_INS_LAT},
};
enum {
PERF_OUTPUT_DSO | PERF_OUTPUT_PERIOD |
PERF_OUTPUT_ADDR | PERF_OUTPUT_DATA_SRC |
PERF_OUTPUT_WEIGHT | PERF_OUTPUT_PHYS_ADDR |
- PERF_OUTPUT_DATA_PAGE_SIZE | PERF_OUTPUT_CODE_PAGE_SIZE,
+ PERF_OUTPUT_DATA_PAGE_SIZE | PERF_OUTPUT_CODE_PAGE_SIZE |
+ PERF_OUTPUT_INS_LAT,
.invalid_fields = PERF_OUTPUT_TRACE | PERF_OUTPUT_BPF_OUTPUT,
},
evsel__check_stype(evsel, PERF_SAMPLE_CODE_PAGE_SIZE, "CODE_PAGE_SIZE", PERF_OUTPUT_CODE_PAGE_SIZE))
return -EINVAL;
+ if (PRINT_FIELD(INS_LAT) &&
+ evsel__check_stype(evsel, PERF_SAMPLE_WEIGHT_STRUCT, "WEIGHT_STRUCT", PERF_OUTPUT_INS_LAT))
+ return -EINVAL;
+
return 0;
}
if (PRINT_FIELD(WEIGHT))
fprintf(fp, "%16" PRIu64, sample->weight);
+ if (PRINT_FIELD(INS_LAT))
+ fprintf(fp, "%16" PRIu16, sample->ins_lat);
+
if (PRINT_FIELD(IP)) {
struct callchain_cursor *cursor = NULL;
"addr,symoff,srcline,period,iregs,uregs,brstack,"
"brstacksym,flags,bpf-output,brstackinsn,brstackoff,"
"callindent,insn,insnlen,synth,phys_addr,metric,misc,ipc,tod,"
- "data_page_size,code_page_size",
+ "data_page_size,code_page_size,ins_lat",
parse_output_fields),
OPT_BOOLEAN('a', "all-cpus", &system_wide,
"system-wide collection from all CPUs"),
data.path = input_name;
data.force = symbol_conf.force;
+ if (symbol__validate_sym_arguments())
+ return -1;
+
if (argc > 1 && !strncmp(argv[0], "rec", strlen("rec"))) {
rec_script_path = get_script_path(argv[1], RECORD_SUFFIX);
if (!rec_script_path)
(PERF_COUNT_HW_CACHE_OP_PREFETCH << 8) |
(PERF_COUNT_HW_CACHE_RESULT_MISS << 16) },
};
- struct parse_events_error errinfo;
-
/* Set attrs if no event is selected and !null_run: */
if (stat_config.null_run)
return 0;
- bzero(&errinfo, sizeof(errinfo));
if (transaction_run) {
+ struct parse_events_error errinfo;
/* Handle -T as -M transaction. Once platform specific metrics
* support has been added to the json files, all architectures
* will use this approach. To determine transaction support
&stat_config.metric_events);
}
+ parse_events_error__init(&errinfo);
if (pmu_have_event("cpu", "cycles-ct") &&
pmu_have_event("cpu", "el-start"))
err = parse_events(evsel_list, transaction_attrs,
&errinfo);
if (err) {
fprintf(stderr, "Cannot set up transaction events\n");
- parse_events_print_error(&errinfo, transaction_attrs);
- return -1;
+ parse_events_error__print(&errinfo, transaction_attrs);
}
- return 0;
+ parse_events_error__exit(&errinfo);
+ return err ? -1 : 0;
}
if (smi_cost) {
+ struct parse_events_error errinfo;
int smi;
if (sysfs__read_int(FREEZE_ON_SMI_PATH, &smi) < 0) {
smi_reset = true;
}
- if (pmu_have_event("msr", "aperf") &&
- pmu_have_event("msr", "smi")) {
- if (!force_metric_only)
- stat_config.metric_only = true;
- err = parse_events(evsel_list, smi_cost_attrs, &errinfo);
- } else {
+ if (!pmu_have_event("msr", "aperf") ||
+ !pmu_have_event("msr", "smi")) {
fprintf(stderr, "To measure SMI cost, it needs "
"msr/aperf/, msr/smi/ and cpu/cycles/ support\n");
- parse_events_print_error(&errinfo, smi_cost_attrs);
return -1;
}
+ if (!force_metric_only)
+ stat_config.metric_only = true;
+
+ parse_events_error__init(&errinfo);
+ err = parse_events(evsel_list, smi_cost_attrs, &errinfo);
if (err) {
- parse_events_print_error(&errinfo, smi_cost_attrs);
+ parse_events_error__print(&errinfo, smi_cost_attrs);
fprintf(stderr, "Cannot set up SMI cost events\n");
- return -1;
}
- return 0;
+ parse_events_error__exit(&errinfo);
+ return err ? -1 : 0;
}
if (topdown_run) {
return -1;
}
if (topdown_attrs[0] && str) {
+ struct parse_events_error errinfo;
if (warn)
arch_topdown_group_warn();
setup_metrics:
+ parse_events_error__init(&errinfo);
err = parse_events(evsel_list, str, &errinfo);
if (err) {
fprintf(stderr,
"Cannot set up top down events %s: %d\n",
str, err);
- parse_events_print_error(&errinfo, str);
+ parse_events_error__print(&errinfo, str);
+ parse_events_error__exit(&errinfo);
free(str);
return -1;
}
+ parse_events_error__exit(&errinfo);
} else {
fprintf(stderr, "System does not support topdown\n");
return -1;
if (!evsel_list->core.nr_entries) {
if (perf_pmu__has_hybrid()) {
+ struct parse_events_error errinfo;
const char *hybrid_str = "cycles,instructions,branches,branch-misses";
if (target__has_cpu(&target))
return -1;
}
+ parse_events_error__init(&errinfo);
err = parse_events(evsel_list, hybrid_str, &errinfo);
if (err) {
fprintf(stderr,
"Cannot set up hybrid events %s: %d\n",
hybrid_str, err);
- parse_events_print_error(&errinfo, hybrid_str);
- return -1;
+ parse_events_error__print(&errinfo, hybrid_str);
}
- return err;
+ parse_events_error__exit(&errinfo);
+ return err ? -1 : 0;
}
if (target__has_cpu(&target))
pr_debug("Couldn't synthesize cgroup events.\n");
machine__synthesize_threads(&top->session->machines.host, &opts->target,
- top->evlist->core.threads, false,
+ top->evlist->core.threads, true, false,
top->nr_threads_synthesize);
if (top->nr_threads_synthesize > 1)
if (argc)
usage_with_options(top_usage, options);
+ status = symbol__validate_sym_arguments();
+ if (status)
+ goto out_delete_evlist;
+
if (annotate_check_args(&top.annotation_opts) < 0)
goto out_delete_evlist;
.arg = { [2] = { .scnprintf = SCA_GETRANDOM_FLAGS, /* flags */ }, }, },
{ .name = "getrlimit",
.arg = { [0] = STRARRAY(resource, rlimit_resources), }, },
+ { .name = "getsockopt",
+ .arg = { [1] = STRARRAY(level, socket_level), }, },
{ .name = "gettid", .errpid = true, },
{ .name = "ioctl",
.arg = {
.arg = { [0] = STRARRAY(which, itimers), }, },
{ .name = "setrlimit",
.arg = { [0] = STRARRAY(resource, rlimit_resources), }, },
+ { .name = "setsockopt",
+ .arg = { [1] = STRARRAY(level, socket_level), }, },
{ .name = "socket",
.arg = { [0] = STRARRAY(family, socket_families),
[1] = { .scnprintf = SCA_SK_TYPE, /* type */ },
goto out;
err = __machine__synthesize_threads(trace->host, &trace->tool, &trace->opts.target,
- evlist->core.threads, trace__tool_process, false,
- 1);
+ evlist->core.threads, trace__tool_process,
+ true, false, 1);
out:
if (err)
symbol__exit();
struct parse_events_error err;
int ret;
- bzero(&err, sizeof(err));
+ parse_events_error__init(&err);
ret = parse_events(evlist, "probe:vfs_getname*", &err);
- if (ret) {
- free(err.str);
- free(err.help);
- free(err.first_str);
- free(err.first_help);
+ parse_events_error__exit(&err);
+ if (ret)
return false;
- }
evlist__for_each_entry_safe(evlist, evsel, tmp) {
if (!strstarts(evsel__name(evsel), "probe:vfs_getname"))
if (trace.perfconfig_events != NULL) {
struct parse_events_error parse_err;
- bzero(&parse_err, sizeof(parse_err));
+ parse_events_error__init(&parse_err);
err = parse_events(trace.evlist, trace.perfconfig_events, &parse_err);
- if (err) {
- parse_events_print_error(&parse_err, trace.perfconfig_events);
+ if (err)
+ parse_events_error__print(&parse_err, trace.perfconfig_events);
+ parse_events_error__exit(&parse_err);
+ if (err)
goto out;
- }
}
if ((nr_cgroups || trace.cgroup) && !trace.opts.target.system_wide) {
include/linux/const.h
include/vdso/const.h
include/linux/hash.h
+include/linux/list-sort.h
include/uapi/linux/hw_breakpoint.h
arch/x86/include/asm/disabled-features.h
arch/x86/include/asm/required-features.h
check include/linux/build_bug.h '-I "^#\(ifndef\|endif\)\( \/\/\)* static_assert$"'
check include/linux/ctype.h '-I "isdigit("'
check lib/ctype.c '-I "^EXPORT_SYMBOL" -I "^#include <linux/export.h>" -B'
+check lib/list_sort.c '-I "^#include <linux/bug.h>"'
# diff non-symmetric files
check_2 tools/perf/arch/x86/entry/syscalls/syscall_64.tbl arch/x86/entry/syscalls/syscall_64.tbl
PERF_COUNT_HW_BRANCH_INSTRUCTIONS = 4,
PERF_COUNT_HW_BRANCH_MISSES = 5,
PERF_COUNT_HW_BUS_CYCLES = 6,
+ PERF_COUNT_HW_STALLED_CYCLES_FRONTEND = 7,
+ PERF_COUNT_HW_STALLED_CYCLES_BACKEND = 8,
+ PERF_COUNT_HW_REF_CPU_CYCLES = 9,
};
These are standardized types of events that work relatively uniformly
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * dlfilter-show-cycles.c: Print the number of cycles at the start of each line
+ * Copyright (c) 2021, Intel Corporation.
+ */
+#include <perf/perf_dlfilter.h>
+#include <string.h>
+#include <stdio.h>
+
+#define MAX_CPU 4096
+
+enum {
+ INSTR_CYC,
+ BRNCH_CYC,
+ OTHER_CYC,
+ MAX_ENTRY
+};
+
+static __u64 cycles[MAX_CPU][MAX_ENTRY];
+static __u64 cycles_rpt[MAX_CPU][MAX_ENTRY];
+
+#define BITS 16
+#define TABLESZ (1 << BITS)
+#define TABLEMAX (TABLESZ / 2)
+#define MASK (TABLESZ - 1)
+
+static struct entry {
+ __u32 used;
+ __s32 tid;
+ __u64 cycles[MAX_ENTRY];
+ __u64 cycles_rpt[MAX_ENTRY];
+} table[TABLESZ];
+
+static int tid_cnt;
+
+static int event_entry(const char *event)
+{
+ if (!event)
+ return OTHER_CYC;
+ if (!strncmp(event, "instructions", 12))
+ return INSTR_CYC;
+ if (!strncmp(event, "branches", 8))
+ return BRNCH_CYC;
+ return OTHER_CYC;
+}
+
+static struct entry *find_entry(__s32 tid)
+{
+ __u32 pos = tid & MASK;
+ struct entry *e;
+
+ e = &table[pos];
+ while (e->used) {
+ if (e->tid == tid)
+ return e;
+ if (++pos == TABLESZ)
+ pos = 0;
+ e = &table[pos];
+ }
+
+ if (tid_cnt >= TABLEMAX) {
+ fprintf(stderr, "Too many threads\n");
+ return NULL;
+ }
+
+ tid_cnt += 1;
+ e->used = 1;
+ e->tid = tid;
+ return e;
+}
+
+static void add_entry(__s32 tid, int pos, __u64 cnt)
+{
+ struct entry *e = find_entry(tid);
+
+ if (e)
+ e->cycles[pos] += cnt;
+}
+
+int filter_event_early(void *data, const struct perf_dlfilter_sample *sample, void *ctx)
+{
+ __s32 cpu = sample->cpu;
+ __s32 tid = sample->tid;
+ int pos;
+
+ if (!sample->cyc_cnt)
+ return 0;
+
+ pos = event_entry(sample->event);
+
+ if (cpu >= 0 && cpu < MAX_CPU)
+ cycles[cpu][pos] += sample->cyc_cnt;
+ else if (tid != -1)
+ add_entry(tid, pos, sample->cyc_cnt);
+ return 0;
+}
+
+static void print_vals(__u64 cycles, __u64 delta)
+{
+ if (delta)
+ printf("%10llu %10llu ", cycles, delta);
+ else
+ printf("%10llu %10s ", cycles, "");
+}
+
+int filter_event(void *data, const struct perf_dlfilter_sample *sample, void *ctx)
+{
+ __s32 cpu = sample->cpu;
+ __s32 tid = sample->tid;
+ int pos;
+
+ pos = event_entry(sample->event);
+
+ if (cpu >= 0 && cpu < MAX_CPU) {
+ print_vals(cycles[cpu][pos], cycles[cpu][pos] - cycles_rpt[cpu][pos]);
+ cycles_rpt[cpu][pos] = cycles[cpu][pos];
+ return 0;
+ }
+
+ if (tid != -1) {
+ struct entry *e = find_entry(tid);
+
+ if (e) {
+ print_vals(e->cycles[pos], e->cycles[pos] - e->cycles_rpt[pos]);
+ e->cycles_rpt[pos] = e->cycles[pos];
+ return 0;
+ }
+ }
+
+ printf("%22s", "");
+ return 0;
+}
+
+const char *filter_description(const char **long_description)
+{
+ static char *long_desc = "Cycle counts are accumulated per CPU (or "
+ "per thread if CPU is not recorded) from IPC information, and "
+ "printed together with the change since the last print, at the "
+ "start of each line. Separate counts are kept for branches, "
+ "instructions or other events.";
+
+ *long_description = long_desc;
+ return "Print the number of cycles at the start of each line";
+}
"ArchStdEvent": "BUS_ACCESS_PERIPH"
},
{
- "ArchStdEvent": "BUS_ACCESS",
+ "ArchStdEvent": "BUS_ACCESS"
}
]
"ArchStdEvent": "L2D_CACHE_INVAL"
},
{
- "ArchStdEvent": "L1I_CACHE_REFILL",
+ "ArchStdEvent": "L1I_CACHE_REFILL"
},
{
- "ArchStdEvent": "L1I_TLB_REFILL",
+ "ArchStdEvent": "L1I_TLB_REFILL"
},
{
- "ArchStdEvent": "L1D_CACHE_REFILL",
+ "ArchStdEvent": "L1D_CACHE_REFILL"
},
{
- "ArchStdEvent": "L1D_CACHE",
+ "ArchStdEvent": "L1D_CACHE"
},
{
- "ArchStdEvent": "L1D_TLB_REFILL",
+ "ArchStdEvent": "L1D_TLB_REFILL"
},
{
- "ArchStdEvent": "L1I_CACHE",
+ "ArchStdEvent": "L1I_CACHE"
},
{
- "ArchStdEvent": "L2D_CACHE",
+ "ArchStdEvent": "L2D_CACHE"
},
{
- "ArchStdEvent": "L2D_CACHE_REFILL",
+ "ArchStdEvent": "L2D_CACHE_REFILL"
},
{
- "ArchStdEvent": "L2D_CACHE_WB",
+ "ArchStdEvent": "L2D_CACHE_WB"
},
{
"PublicDescription": "This event counts any load or store operation which accesses the data L1 TLB",
},
{
"PublicDescription": "This event counts any instruction fetch which accesses the instruction L1 TLB",
- "ArchStdEvent": "L1I_TLB",
+ "ArchStdEvent": "L1I_TLB"
},
{
"PublicDescription": "Level 2 access to data TLB that caused a page table walk. This event counts on any data access which causes L2D_TLB_REFILL to count",
[
{
"PublicDescription": "The number of core clock cycles",
- "ArchStdEvent": "CPU_CYCLES",
+ "ArchStdEvent": "CPU_CYCLES"
},
{
"PublicDescription": "FSU clocking gated off cycle",
"ArchStdEvent": "EXC_TRAP_FIQ"
},
{
- "ArchStdEvent": "EXC_TAKEN",
+ "ArchStdEvent": "EXC_TAKEN"
},
{
- "ArchStdEvent": "EXC_RETURN",
+ "ArchStdEvent": "EXC_RETURN"
}
]
"BriefDescription": "Software increment"
},
{
- "ArchStdEvent": "INST_RETIRED",
+ "ArchStdEvent": "INST_RETIRED"
},
{
"ArchStdEvent": "CID_WRITE_RETIRED",
"BriefDescription": "Write to CONTEXTIDR"
},
{
- "ArchStdEvent": "INST_SPEC",
+ "ArchStdEvent": "INST_SPEC"
},
{
- "ArchStdEvent": "TTBR_WRITE_RETIRED",
+ "ArchStdEvent": "TTBR_WRITE_RETIRED"
},
{
"PublicDescription": "This event counts all branches, taken or not. This excludes exception entries, debug entries and CCFAIL branches",
- "ArchStdEvent": "BR_RETIRED",
+ "ArchStdEvent": "BR_RETIRED"
},
{
"PublicDescription": "This event counts any branch counted by BR_RETIRED which is not correctly predicted and causes a pipeline flush",
- "ArchStdEvent": "BR_MIS_PRED_RETIRED",
+ "ArchStdEvent": "BR_MIS_PRED_RETIRED"
},
{
"PublicDescription": "Operation speculatively executed, NOP",
"ArchStdEvent": "UNALIGNED_LDST_SPEC"
},
{
- "ArchStdEvent": "MEM_ACCESS",
+ "ArchStdEvent": "MEM_ACCESS"
},
{
"PublicDescription": "This event counts any correctable or uncorrectable memory error (ECC or parity) in the protected core RAMs",
- "ArchStdEvent": "MEMORY_ERROR",
+ "ArchStdEvent": "MEMORY_ERROR"
}
]
[
{
"PublicDescription": "This event counts any predictable branch instruction which is mispredicted either due to dynamic misprediction or because the MMU is off and the branches are statically predicted not taken",
- "ArchStdEvent": "BR_MIS_PRED",
+ "ArchStdEvent": "BR_MIS_PRED"
},
{
"PublicDescription": "This event counts all predictable branches.",
- "ArchStdEvent": "BR_PRED",
+ "ArchStdEvent": "BR_PRED"
}
]
[
{
- "PublicDescription": "The number of core clock cycles"
+ "PublicDescription": "The number of core clock cycles",
"ArchStdEvent": "CPU_CYCLES",
"BriefDescription": "The number of core clock cycles."
},
{
"PublicDescription": "This event counts for every beat of data transferred over the data channels between the core and the SCU. If both read and write data beats are transferred on a given cycle, this event is counted twice on that cycle. This event counts the sum of BUS_ACCESS_RD and BUS_ACCESS_WR.",
- "ArchStdEvent": "BUS_ACCESS",
+ "ArchStdEvent": "BUS_ACCESS"
},
{
- "PublicDescription": "This event duplicates CPU_CYCLES."
- "ArchStdEvent": "BUS_CYCLES",
+ "PublicDescription": "This event duplicates CPU_CYCLES.",
+ "ArchStdEvent": "BUS_CYCLES"
},
{
- "ArchStdEvent": "BUS_ACCESS_RD",
+ "ArchStdEvent": "BUS_ACCESS_RD"
},
{
- "ArchStdEvent": "BUS_ACCESS_WR",
+ "ArchStdEvent": "BUS_ACCESS_WR"
}
]
[
{
"PublicDescription": "This event counts any instruction fetch which misses in the cache.",
- "ArchStdEvent": "L1I_CACHE_REFILL",
+ "ArchStdEvent": "L1I_CACHE_REFILL"
},
{
"PublicDescription": "This event counts any refill of the instruction L1 TLB from the L2 TLB. This includes refills that result in a translation fault.",
- "ArchStdEvent": "L1I_TLB_REFILL",
+ "ArchStdEvent": "L1I_TLB_REFILL"
},
{
"PublicDescription": "This event counts any load or store operation or page table walk access which causes data to be read from outside the L1, including accesses which do not allocate into L1.",
- "ArchStdEvent": "L1D_CACHE_REFILL",
+ "ArchStdEvent": "L1D_CACHE_REFILL"
},
{
"PublicDescription": "This event counts any load or store operation or page table walk access which looks up in the L1 data cache. In particular, any access which could count the L1D_CACHE_REFILL event causes this event to count.",
- "ArchStdEvent": "L1D_CACHE",
+ "ArchStdEvent": "L1D_CACHE"
},
{
"PublicDescription": "This event counts any refill of the data L1 TLB from the L2 TLB. This includes refills that result in a translation fault.",
- "ArchStdEvent": "L1D_TLB_REFILL",
+ "ArchStdEvent": "L1D_TLB_REFILL"
},
- {,
+ {
"PublicDescription": "Level 1 instruction cache access or Level 0 Macro-op cache access. This event counts any instruction fetch which accesses the L1 instruction cache or L0 Macro-op cache.",
- "ArchStdEvent": "L1I_CACHE",
+ "ArchStdEvent": "L1I_CACHE"
},
{
"PublicDescription": "This event counts any write-back of data from the L1 data cache to L2 or L3. This counts both victim line evictions and snoops, including cache maintenance operations.",
- "ArchStdEvent": "L1D_CACHE_WB",
+ "ArchStdEvent": "L1D_CACHE_WB"
},
{
"PublicDescription": "This event counts any transaction from L1 which looks up in the L2 cache, and any write-back from the L1 to the L2. Snoops from outside the core and cache maintenance operations are not counted.",
- "ArchStdEvent": "L2D_CACHE",
+ "ArchStdEvent": "L2D_CACHE"
},
{
"PublicDescription": "L2 data cache refill. This event counts any cacheable transaction from L1 which causes data to be read from outside the core. L2 refills caused by stashes into L2 should not be counted",
- "ArchStdEvent": "L2D_CACHE_REFILL",
+ "ArchStdEvent": "L2D_CACHE_REFILL"
},
{
"PublicDescription": "This event counts any write-back of data from the L2 cache to outside the core. This includes snoops to the L2 which return data, regardless of whether they cause an invalidation. Invalidations from the L2 which do not write data outside of the core and snoops which return data from the L1 are not counted",
- "ArchStdEvent": "L2D_CACHE_WB",
+ "ArchStdEvent": "L2D_CACHE_WB"
},
{
"PublicDescription": "This event counts any full cache line write into the L2 cache which does not cause a linefill, including write-backs from L1 to L2 and full-line writes which do not allocate into L1.",
- "ArchStdEvent": "L2D_CACHE_ALLOCATE",
+ "ArchStdEvent": "L2D_CACHE_ALLOCATE"
},
{
"PublicDescription": "This event counts any load or store operation which accesses the data L1 TLB. If both a load and a store are executed on a cycle, this event counts twice. This event counts regardless of whether the MMU is enabled.",
},
{
"PublicDescription": "This event counts on any access to the L2 TLB (caused by a refill of any of the L1 TLBs). This event does not count if the MMU is disabled.",
- "ArchStdEvent": "L2D_TLB",
+ "ArchStdEvent": "L2D_TLB"
},
{
"PublicDescription": "This event counts on any data access which causes L2D_TLB_REFILL to count.",
- "ArchStdEvent": "DTLB_WALK",
+ "ArchStdEvent": "DTLB_WALK"
},
{
"PublicDescription": "This event counts on any instruction access which causes L2D_TLB_REFILL to count.",
- "ArchStdEvent": "ITLB_WALK",
+ "ArchStdEvent": "ITLB_WALK"
},
{
- "ArchStdEvent": "LL_CACHE_RD",
+ "ArchStdEvent": "LL_CACHE_RD"
},
{
- "ArchStdEvent": "LL_CACHE_MISS_RD",
+ "ArchStdEvent": "LL_CACHE_MISS_RD"
},
{
"ArchStdEvent": "L1D_CACHE_INVAL"
[
{
- "ArchStdEvent": "EXC_TAKEN",
+ "ArchStdEvent": "EXC_TAKEN"
},
{
"PublicDescription": "This event counts any correctable or uncorrectable memory error (ECC or parity) in the protected core RAMs",
- "ArchStdEvent": "MEMORY_ERROR",
+ "ArchStdEvent": "MEMORY_ERROR"
},
{
"ArchStdEvent": "EXC_DABORT"
[
{
- "ArchStdEvent": "SW_INCR",
+ "ArchStdEvent": "SW_INCR"
},
{
"PublicDescription": "This event counts all retired instructions, including those that fail their condition check.",
- "ArchStdEvent": "INST_RETIRED",
+ "ArchStdEvent": "INST_RETIRED"
},
{
- "ArchStdEvent": "EXC_RETURN",
+ "ArchStdEvent": "EXC_RETURN"
},
{
"PublicDescription": "This event only counts writes to CONTEXTIDR in AArch32 state, and via the CONTEXTIDR_EL1 mnemonic in AArch64 state.",
- "ArchStdEvent": "CID_WRITE_RETIRED",
+ "ArchStdEvent": "CID_WRITE_RETIRED"
},
{
- "ArchStdEvent": "INST_SPEC",
+ "ArchStdEvent": "INST_SPEC"
},
{
"PublicDescription": "This event only counts writes to TTBR0/TTBR1 in AArch32 state and TTBR0_EL1/TTBR1_EL1 in AArch64 state.",
- "ArchStdEvent": "TTBR_WRITE_RETIRED",
+ "ArchStdEvent": "TTBR_WRITE_RETIRED"
},
- {,
+ {
"PublicDescription": "This event counts all branches, taken or not. This excludes exception entries, debug entries and CCFAIL branches.",
- "ArchStdEvent": "BR_RETIRED",
+ "ArchStdEvent": "BR_RETIRED"
},
{
"PublicDescription": "This event counts any branch counted by BR_RETIRED which is not correctly predicted and causes a pipeline flush.",
- "ArchStdEvent": "BR_MIS_PRED_RETIRED",
+ "ArchStdEvent": "BR_MIS_PRED_RETIRED"
},
{
"ArchStdEvent": "ASE_SPEC"
[
{
"PublicDescription": "This event counts memory accesses due to load or store instructions. This event counts the sum of MEM_ACCESS_RD and MEM_ACCESS_WR.",
- "ArchStdEvent": "MEM_ACCESS",
+ "ArchStdEvent": "MEM_ACCESS"
},
{
"ArchStdEvent": "MEM_ACCESS_RD"
[
{
- "ArchStdEvent": "REMOTE_ACCESS",
+ "ArchStdEvent": "REMOTE_ACCESS"
}
]
[
{
"PublicDescription": "The counter counts on any cycle when there are no fetched instructions available to dispatch.",
- "ArchStdEvent": "STALL_FRONTEND",
+ "ArchStdEvent": "STALL_FRONTEND"
},
{
"PublicDescription": "The counter counts on any cycle fetched instructions are not dispatched due to resource constraints.",
- "ArchStdEvent": "STALL_BACKEND",
+ "ArchStdEvent": "STALL_BACKEND"
}
]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "BR_MIS_PRED"
+ },
+ {
+ "ArchStdEvent": "BR_PRED"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "CPU_CYCLES"
+ },
+ {
+ "ArchStdEvent": "BUS_ACCESS"
+ },
+ {
+ "ArchStdEvent": "BUS_CYCLES"
+ },
+ {
+ "ArchStdEvent": "BUS_ACCESS_RD"
+ },
+ {
+ "ArchStdEvent": "BUS_ACCESS_WR"
+ },
+ {
+ "ArchStdEvent": "CNT_CYCLES"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "L1I_CACHE_REFILL"
+ },
+ {
+ "ArchStdEvent": "L1I_TLB_REFILL"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_REFILL"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB_REFILL"
+ },
+ {
+ "ArchStdEvent": "L1I_CACHE"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_WB"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_REFILL"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_WB"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_ALLOCATE"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB"
+ },
+ {
+ "ArchStdEvent": "L1I_TLB"
+ },
+ {
+ "ArchStdEvent": "L3D_CACHE_ALLOCATE"
+ },
+ {
+ "ArchStdEvent": "L3D_CACHE_REFILL"
+ },
+ {
+ "ArchStdEvent": "L3D_CACHE"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB_REFILL"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB"
+ },
+ {
+ "ArchStdEvent": "DTLB_WALK"
+ },
+ {
+ "ArchStdEvent": "ITLB_WALK"
+ },
+ {
+ "ArchStdEvent": "LL_CACHE_RD"
+ },
+ {
+ "ArchStdEvent": "LL_CACHE_MISS_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_LMISS_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_WR"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_REFILL_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_REFILL_WR"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_REFILL_INNER"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_REFILL_OUTER"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_WB_VICTIM"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_WB_CLEAN"
+ },
+ {
+ "ArchStdEvent": "L1D_CACHE_INVAL"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB_REFILL_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB_REFILL_WR"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB_RD"
+ },
+ {
+ "ArchStdEvent": "L1D_TLB_WR"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_RD"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_WR"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_REFILL_RD"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_REFILL_WR"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_WB_VICTIM"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_WB_CLEAN"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_INVAL"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB_REFILL_RD"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB_REFILL_WR"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB_RD"
+ },
+ {
+ "ArchStdEvent": "L2D_TLB_WR"
+ },
+ {
+ "ArchStdEvent": "L3D_CACHE_RD"
+ },
+ {
+ "ArchStdEvent": "L1I_CACHE_LMISS"
+ },
+ {
+ "ArchStdEvent": "L2D_CACHE_LMISS_RD"
+ },
+ {
+ "ArchStdEvent": "L3D_CACHE_LMISS_RD"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "EXC_TAKEN"
+ },
+ {
+ "ArchStdEvent": "MEMORY_ERROR"
+ },
+ {
+ "ArchStdEvent": "EXC_UNDEF"
+ },
+ {
+ "ArchStdEvent": "EXC_SVC"
+ },
+ {
+ "ArchStdEvent": "EXC_PABORT"
+ },
+ {
+ "ArchStdEvent": "EXC_DABORT"
+ },
+ {
+ "ArchStdEvent": "EXC_IRQ"
+ },
+ {
+ "ArchStdEvent": "EXC_FIQ"
+ },
+ {
+ "ArchStdEvent": "EXC_SMC"
+ },
+ {
+ "ArchStdEvent": "EXC_HVC"
+ },
+ {
+ "ArchStdEvent": "EXC_TRAP_PABORT"
+ },
+ {
+ "ArchStdEvent": "EXC_TRAP_DABORT"
+ },
+ {
+ "ArchStdEvent": "EXC_TRAP_OTHER"
+ },
+ {
+ "ArchStdEvent": "EXC_TRAP_IRQ"
+ },
+ {
+ "ArchStdEvent": "EXC_TRAP_FIQ"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "SW_INCR"
+ },
+ {
+ "ArchStdEvent": "INST_RETIRED"
+ },
+ {
+ "ArchStdEvent": "EXC_RETURN"
+ },
+ {
+ "ArchStdEvent": "CID_WRITE_RETIRED"
+ },
+ {
+ "ArchStdEvent": "INST_SPEC"
+ },
+ {
+ "ArchStdEvent": "TTBR_WRITE_RETIRED"
+ },
+ {
+ "ArchStdEvent": "BR_RETIRED"
+ },
+ {
+ "ArchStdEvent": "BR_MIS_PRED_RETIRED"
+ },
+ {
+ "ArchStdEvent": "OP_RETIRED"
+ },
+ {
+ "ArchStdEvent": "OP_SPEC"
+ },
+ {
+ "ArchStdEvent": "LDREX_SPEC"
+ },
+ {
+ "ArchStdEvent": "STREX_PASS_SPEC"
+ },
+ {
+ "ArchStdEvent": "STREX_FAIL_SPEC"
+ },
+ {
+ "ArchStdEvent": "STREX_SPEC"
+ },
+ {
+ "ArchStdEvent": "LD_SPEC"
+ },
+ {
+ "ArchStdEvent": "ST_SPEC"
+ },
+ {
+ "ArchStdEvent": "DP_SPEC"
+ },
+ {
+ "ArchStdEvent": "ASE_SPEC"
+ },
+ {
+ "ArchStdEvent": "VFP_SPEC"
+ },
+ {
+ "ArchStdEvent": "PC_WRITE_SPEC"
+ },
+ {
+ "ArchStdEvent": "CRYPTO_SPEC"
+ },
+ {
+ "ArchStdEvent": "BR_IMMED_SPEC"
+ },
+ {
+ "ArchStdEvent": "BR_RETURN_SPEC"
+ },
+ {
+ "ArchStdEvent": "BR_INDIRECT_SPEC"
+ },
+ {
+ "ArchStdEvent": "ISB_SPEC"
+ },
+ {
+ "ArchStdEvent": "DSB_SPEC"
+ },
+ {
+ "ArchStdEvent": "DMB_SPEC"
+ },
+ {
+ "ArchStdEvent": "RC_LD_SPEC"
+ },
+ {
+ "ArchStdEvent": "RC_ST_SPEC"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "MEM_ACCESS"
+ },
+ {
+ "ArchStdEvent": "MEM_ACCESS_RD"
+ },
+ {
+ "ArchStdEvent": "MEM_ACCESS_WR"
+ },
+ {
+ "ArchStdEvent": "UNALIGNED_LD_SPEC"
+ },
+ {
+ "ArchStdEvent": "UNALIGNED_ST_SPEC"
+ },
+ {
+ "ArchStdEvent": "UNALIGNED_LDST_SPEC"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "REMOTE_ACCESS"
+ }
+]
--- /dev/null
+[
+ {
+ "ArchStdEvent": "STALL_FRONTEND"
+ },
+ {
+ "ArchStdEvent": "STALL_BACKEND"
+ },
+ {
+ "ArchStdEvent": "STALL"
+ },
+ {
+ "ArchStdEvent": "STALL_SLOT_BACKEND"
+ },
+ {
+ "ArchStdEvent": "STALL_SLOT_FRONTEND"
+ },
+ {
+ "ArchStdEvent": "STALL_SLOT"
+ },
+ {
+ "ArchStdEvent": "STALL_BACKEND_MEM"
+ }
+]
"BriefDescription": "Last level cache miss, read"
},
{
+ "PublicDescription": "Level 1 data cache long-latency read miss. The counter counts each memory read access counted by L1D_CACHE that incurs additional latency because it returns data from outside the Level 1 data or unified cache of this processing element.",
+ "EventCode": "0x39",
+ "EventName": "L1D_CACHE_LMISS_RD",
+ "BriefDescription": "Level 1 data cache long-latency read miss"
+ },
+ {
+ "PublicDescription": "Micro-operation architecturally executed. The counter counts each operation counted by OP_SPEC that would be executed in a simple sequential execution of the program.",
+ "EventCode": "0x3A",
+ "EventName": "OP_RETIRED",
+ "BriefDescription": "Micro-operation architecturally executed"
+ },
+ {
+ "PublicDescription": "Micro-operation speculatively executed. The counter counts the number of operations executed by the processing element, including those that are executed speculatively and would not be executed in a simple sequential execution of the program.",
+ "EventCode": "0x3B",
+ "EventName": "OP_SPEC",
+ "BriefDescription": "Micro-operation speculatively executed"
+ },
+ {
+ "PublicDescription": "No operation sent for execution. The counter counts every attributable cycle on which no attributable instruction or operation was sent for execution on this processing element.",
+ "EventCode": "0x3C",
+ "EventName": "STALL",
+ "BriefDescription": "No operation sent for execution"
+ },
+ {
+ "PublicDescription": "No operation sent for execution on a slot due to the backend. Counts each slot counted by STALL_SLOT where no attributable instruction or operation was sent for execution because the backend is unable to accept it.",
+ "EventCode": "0x3D",
+ "EventName": "STALL_SLOT_BACKEND",
+ "BriefDescription": "No operation sent for execution on a slot due to the backend"
+ },
+ {
+ "PublicDescription": "No operation sent for execution on a slot due to the frontend. Counts each slot counted by STALL_SLOT where no attributable instruction or operation was sent for execution because there was no attributable instruction or operation available to issue from the processing element from the frontend for the slot.",
+ "EventCode": "0x3E",
+ "EventName": "STALL_SLOT_FRONTEND",
+ "BriefDescription": "No operation sent for execution on a slot due to the frontend"
+ },
+ {
+ "PublicDescription": "No operation sent for execution on a slot. The counter counts on each attributable cycle the number of instruction or operation slots that were not occupied by an instruction or operation attributable to the processing element.",
+ "EventCode": "0x3F",
+ "EventName": "STALL_SLOT",
+ "BriefDescription": "No operation sent for execution on a slot"
+ },
+ {
+ "PublicDescription": "Constant frequency cycles. The counter increments at a constant frequency equal to the rate of increment of the system counter, CNTPCT_EL0.",
+ "EventCode": "0x4004",
+ "EventName": "CNT_CYCLES",
+ "BriefDescription": "Constant frequency cycles"
+ },
+ {
+ "PublicDescription": "Memory stall cycles. The counter counts each cycle counted by STALL_BACKEND where there is a cache miss in the last level of cache within the processing element clock domain",
+ "EventCode": "0x4005",
+ "EventName": "STALL_BACKEND_MEM",
+ "BriefDescription": "Memory stall cycles"
+ },
+ {
+ "PublicDescription": "Level 1 instruction cache long-latency read miss. If the L1I_CACHE_RD event is implemented, the counter counts each access counted by L1I_CACHE_RD that incurs additional latency because it returns instructions from outside of the Level 1 instruction cache of this PE. If the L1I_CACHE_RD event is not implemented, the counter counts each access counted by L1I_CACHE that incurs additional latency because it returns instructions from outside the Level 1 instruction cache of this PE. The event indicates to software that the access missed in the Level 1 instruction cache and might have a significant performance impact due to the additional latency, compared to the latency of an access that hits in the Level 1 instruction cache.",
+ "EventCode": "0x4006",
+ "EventName": "L1I_CACHE_LMISS",
+ "BriefDescription": "Level 1 instruction cache long-latency read miss"
+ },
+ {
+ "PublicDescription": "Level 2 data cache long-latency read miss. The counter counts each memory read access counted by L2D_CACHE that incurs additional latency because it returns data from outside the Level 2 data or unified cache of this processing element. The event indicates to software that the access missed in the Level 2 data or unified cache and might have a significant performance impact compared to the latency of an access that hits in the Level 2 data or unified cache.",
+ "EventCode": "0x4009",
+ "EventName": "L2D_CACHE_LMISS_RD",
+ "BriefDescription": "Level 2 data cache long-latency read miss"
+ },
+ {
+ "PublicDescription": "Level 3 data cache long-latency read miss. The counter counts each memory read access counted by L3D_CACHE that incurs additional latency because it returns data from outside the Level 3 data or unified cache of this processing element. The event indicates to software that the access missed in the Level 3 data or unified cache and might have a significant performance impact compared to the latency of an access that hits in the Level 3 data or unified cache.",
+ "EventCode": "0x400B",
+ "EventName": "L3D_CACHE_LMISS_RD",
+ "BriefDescription": "Level 3 data cache long-latency read miss"
+ },
+ {
"PublicDescription": "SIMD Instruction architecturally executed.",
"EventCode": "0x8000",
"EventName": "SIMD_INST_RETIRED",
"BriefDescription": "Store bound L3 topdown metric",
"MetricGroup": "TopDownL3",
"MetricName": "store_bound"
- },
+ }
]
[
{
- "EventCode": "0x00",
- "EventName": "uncore_hisi_ddrc.flux_wr",
+ "ConfigCode": "0x00",
+ "EventName": "flux_wr",
"BriefDescription": "DDRC total write operations",
"PublicDescription": "DDRC total write operations",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x01",
- "EventName": "uncore_hisi_ddrc.flux_rd",
+ "ConfigCode": "0x01",
+ "EventName": "flux_rd",
"BriefDescription": "DDRC total read operations",
"PublicDescription": "DDRC total read operations",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x02",
- "EventName": "uncore_hisi_ddrc.flux_wcmd",
+ "ConfigCode": "0x02",
+ "EventName": "flux_wcmd",
"BriefDescription": "DDRC write commands",
"PublicDescription": "DDRC write commands",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x03",
- "EventName": "uncore_hisi_ddrc.flux_rcmd",
+ "ConfigCode": "0x03",
+ "EventName": "flux_rcmd",
"BriefDescription": "DDRC read commands",
"PublicDescription": "DDRC read commands",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x04",
- "EventName": "uncore_hisi_ddrc.pre_cmd",
+ "ConfigCode": "0x04",
+ "EventName": "pre_cmd",
"BriefDescription": "DDRC precharge commands",
"PublicDescription": "DDRC precharge commands",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x05",
- "EventName": "uncore_hisi_ddrc.act_cmd",
+ "ConfigCode": "0x05",
+ "EventName": "act_cmd",
"BriefDescription": "DDRC active commands",
"PublicDescription": "DDRC active commands",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x06",
- "EventName": "uncore_hisi_ddrc.rnk_chg",
+ "ConfigCode": "0x06",
+ "EventName": "rnk_chg",
"BriefDescription": "DDRC rank commands",
"PublicDescription": "DDRC rank commands",
"Unit": "hisi_sccl,ddrc"
},
{
- "EventCode": "0x07",
- "EventName": "uncore_hisi_ddrc.rw_chg",
+ "ConfigCode": "0x07",
+ "EventName": "rw_chg",
"BriefDescription": "DDRC read and write changes",
"PublicDescription": "DDRC read and write changes",
"Unit": "hisi_sccl,ddrc"
[
{
- "EventCode": "0x00",
- "EventName": "uncore_hisi_hha.rx_ops_num",
+ "ConfigCode": "0x00",
+ "EventName": "rx_ops_num",
"BriefDescription": "The number of all operations received by the HHA",
"PublicDescription": "The number of all operations received by the HHA",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x01",
- "EventName": "uncore_hisi_hha.rx_outer",
+ "ConfigCode": "0x01",
+ "EventName": "rx_outer",
"BriefDescription": "The number of all operations received by the HHA from another socket",
"PublicDescription": "The number of all operations received by the HHA from another socket",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x02",
- "EventName": "uncore_hisi_hha.rx_sccl",
+ "ConfigCode": "0x02",
+ "EventName": "rx_sccl",
"BriefDescription": "The number of all operations received by the HHA from another SCCL in this socket",
"PublicDescription": "The number of all operations received by the HHA from another SCCL in this socket",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x03",
- "EventName": "uncore_hisi_hha.rx_ccix",
+ "ConfigCode": "0x03",
+ "EventName": "rx_ccix",
"BriefDescription": "Count of the number of operations that HHA has received from CCIX",
"PublicDescription": "Count of the number of operations that HHA has received from CCIX",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x1c",
- "EventName": "uncore_hisi_hha.rd_ddr_64b",
+ "ConfigCode": "0x4",
+ "EventName": "rx_wbi",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x5",
+ "EventName": "rx_wbip",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x11",
+ "EventName": "rx_wtistash",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x1c",
+ "EventName": "rd_ddr_64b",
"BriefDescription": "The number of read operations sent by HHA to DDRC which size is 64 bytes",
"PublicDescription": "The number of read operations sent by HHA to DDRC which size is 64bytes",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x1d",
- "EventName": "uncore_hisi_hha.wr_ddr_64b",
+ "ConfigCode": "0x1d",
+ "EventName": "wr_ddr_64b",
"BriefDescription": "The number of write operations sent by HHA to DDRC which size is 64 bytes",
"PublicDescription": "The number of write operations sent by HHA to DDRC which size is 64 bytes",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x1e",
- "EventName": "uncore_hisi_hha.rd_ddr_128b",
+ "ConfigCode": "0x1e",
+ "EventName": "rd_ddr_128b",
"BriefDescription": "The number of read operations sent by HHA to DDRC which size is 128 bytes",
"PublicDescription": "The number of read operations sent by HHA to DDRC which size is 128 bytes",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x1f",
- "EventName": "uncore_hisi_hha.wr_ddr_128b",
+ "ConfigCode": "0x1f",
+ "EventName": "wr_ddr_128b",
"BriefDescription": "The number of write operations sent by HHA to DDRC which size is 128 bytes",
"PublicDescription": "The number of write operations sent by HHA to DDRC which size is 128 bytes",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x20",
- "EventName": "uncore_hisi_hha.spill_num",
+ "ConfigCode": "0x20",
+ "EventName": "spill_num",
"BriefDescription": "Count of the number of spill operations that the HHA has sent",
"PublicDescription": "Count of the number of spill operations that the HHA has sent",
"Unit": "hisi_sccl,hha"
},
{
- "EventCode": "0x21",
- "EventName": "uncore_hisi_hha.spill_success",
+ "ConfigCode": "0x21",
+ "EventName": "spill_success",
"BriefDescription": "Count of the number of successful spill operations that the HHA has sent",
"PublicDescription": "Count of the number of successful spill operations that the HHA has sent",
"Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x23",
+ "EventName": "bi_num",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x32",
+ "EventName": "mediated_num",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x33",
+ "EventName": "tx_snp_num",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x34",
+ "EventName": "tx_snp_outer",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x35",
+ "EventName": "tx_snp_ccix",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x38",
+ "EventName": "rx_snprspdata",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x3c",
+ "EventName": "rx_snprsp_outer",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x40",
+ "EventName": "sdir-lookup",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x41",
+ "EventName": "edir-lookup",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x42",
+ "EventName": "sdir-hit",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x43",
+ "EventName": "edir-hit",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x4c",
+ "EventName": "sdir-home-migrate",
+ "Unit": "hisi_sccl,hha"
+ },
+ {
+ "ConfigCode": "0x4d",
+ "EventName": "edir-home-migrate",
+ "Unit": "hisi_sccl,hha"
}
]
[
{
- "EventCode": "0x00",
- "EventName": "uncore_hisi_l3c.rd_cpipe",
+ "ConfigCode": "0x00",
+ "EventName": "rd_cpipe",
"BriefDescription": "Total read accesses",
"PublicDescription": "Total read accesses",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x01",
- "EventName": "uncore_hisi_l3c.wr_cpipe",
+ "ConfigCode": "0x01",
+ "EventName": "wr_cpipe",
"BriefDescription": "Total write accesses",
"PublicDescription": "Total write accesses",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x02",
- "EventName": "uncore_hisi_l3c.rd_hit_cpipe",
+ "ConfigCode": "0x02",
+ "EventName": "rd_hit_cpipe",
"BriefDescription": "Total read hits",
"PublicDescription": "Total read hits",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x03",
- "EventName": "uncore_hisi_l3c.wr_hit_cpipe",
+ "ConfigCode": "0x03",
+ "EventName": "wr_hit_cpipe",
"BriefDescription": "Total write hits",
"PublicDescription": "Total write hits",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x04",
- "EventName": "uncore_hisi_l3c.victim_num",
+ "ConfigCode": "0x04",
+ "EventName": "victim_num",
"BriefDescription": "l3c precharge commands",
"PublicDescription": "l3c precharge commands",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x20",
- "EventName": "uncore_hisi_l3c.rd_spipe",
+ "ConfigCode": "0x20",
+ "EventName": "rd_spipe",
"BriefDescription": "Count of the number of read lines that come from this cluster of CPU core in spipe",
"PublicDescription": "Count of the number of read lines that come from this cluster of CPU core in spipe",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x21",
- "EventName": "uncore_hisi_l3c.wr_spipe",
+ "ConfigCode": "0x21",
+ "EventName": "wr_spipe",
"BriefDescription": "Count of the number of write lines that come from this cluster of CPU core in spipe",
"PublicDescription": "Count of the number of write lines that come from this cluster of CPU core in spipe",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x22",
- "EventName": "uncore_hisi_l3c.rd_hit_spipe",
+ "ConfigCode": "0x22",
+ "EventName": "rd_hit_spipe",
"BriefDescription": "Count of the number of read lines that hits in spipe of this L3C",
"PublicDescription": "Count of the number of read lines that hits in spipe of this L3C",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x23",
- "EventName": "uncore_hisi_l3c.wr_hit_spipe",
+ "ConfigCode": "0x23",
+ "EventName": "wr_hit_spipe",
"BriefDescription": "Count of the number of write lines that hits in spipe of this L3C",
"PublicDescription": "Count of the number of write lines that hits in spipe of this L3C",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x29",
- "EventName": "uncore_hisi_l3c.back_invalid",
+ "ConfigCode": "0x29",
+ "EventName": "back_invalid",
"BriefDescription": "Count of the number of L3C back invalid operations",
"PublicDescription": "Count of the number of L3C back invalid operations",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x40",
- "EventName": "uncore_hisi_l3c.retry_cpu",
+ "ConfigCode": "0x40",
+ "EventName": "retry_cpu",
"BriefDescription": "Count of the number of retry that L3C suppresses the CPU operations",
"PublicDescription": "Count of the number of retry that L3C suppresses the CPU operations",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x41",
- "EventName": "uncore_hisi_l3c.retry_ring",
+ "ConfigCode": "0x41",
+ "EventName": "retry_ring",
"BriefDescription": "Count of the number of retry that L3C suppresses the ring operations",
"PublicDescription": "Count of the number of retry that L3C suppresses the ring operations",
"Unit": "hisi_sccl,l3c"
},
{
- "EventCode": "0x42",
- "EventName": "uncore_hisi_l3c.prefetch_drop",
+ "ConfigCode": "0x42",
+ "EventName": "prefetch_drop",
"BriefDescription": "Count of the number of prefetch drops from this L3C",
"PublicDescription": "Count of the number of prefetch drops from this L3C",
"Unit": "hisi_sccl,l3c"
0x00000000410fd080,v1,arm/cortex-a57-a72,core
0x00000000410fd0b0,v1,arm/cortex-a76-n1,core
0x00000000410fd0c0,v1,arm/cortex-a76-n1,core
+0x00000000410fd400,v1,arm/neoverse-v1,core
0x00000000420f5160,v1,cavium/thunderx2,core
0x00000000430f0af0,v1,cavium/thunderx2,core
0x00000000460f0010,v1,fujitsu/a64fx,core
"EventCode": "0x21e",
"EventName": "pop25_inst",
"BriefDescription": "V3 POP25 instructions"
- },
+ }
]
--- /dev/null
+[
+ {
+ "BriefDescription": "Percentage of cycles that are run cycles",
+ "MetricExpr": "PM_RUN_CYC / PM_CYC * 100",
+ "MetricGroup": "General",
+ "MetricName": "RUN_CYCLES_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction",
+ "MetricExpr": "PM_CYC / PM_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "CYCLES_PER_INSTRUCTION"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled for any reason",
+ "MetricExpr": "PM_DISP_STALL_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled because there was a flush",
+ "MetricExpr": "PM_DISP_STALL_FLUSH / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_FLUSH_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled because the MMU was handling a translation miss",
+ "MetricExpr": "PM_DISP_STALL_TRANSLATION / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_TRANSLATION_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled waiting to resolve an instruction ERAT miss",
+ "MetricExpr": "PM_DISP_STALL_IERAT_ONLY_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_IERAT_ONLY_MISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled waiting to resolve an instruction TLB miss",
+ "MetricExpr": "PM_DISP_STALL_ITLB_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_ITLB_MISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled due to an icache miss",
+ "MetricExpr": "PM_DISP_STALL_IC_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_IC_MISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while the instruction was fetched from the local L2",
+ "MetricExpr": "PM_DISP_STALL_IC_L2 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_IC_L2_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while the instruction was fetched from the local L3",
+ "MetricExpr": "PM_DISP_STALL_IC_L3 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_IC_L3_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while the instruction was fetched from any source beyond the local L3",
+ "MetricExpr": "PM_DISP_STALL_IC_L3MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_IC_L3MISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled due to an icache miss after a branch mispredict",
+ "MetricExpr": "PM_DISP_STALL_BR_MPRED_ICMISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_BR_MPRED_ICMISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while instruction was fetched from the local L2 after suffering a branch mispredict",
+ "MetricExpr": "PM_DISP_STALL_BR_MPRED_IC_L2 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_BR_MPRED_IC_L2_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while instruction was fetched from the local L3 after suffering a branch mispredict",
+ "MetricExpr": "PM_DISP_STALL_BR_MPRED_IC_L3 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_BR_MPRED_IC_L3_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled while instruction was fetched from any source beyond the local L3 after suffering a branch mispredict",
+ "MetricExpr": "PM_DISP_STALL_BR_MPRED_IC_L3MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_BR_MPRED_IC_L3MISS_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled due to a branch mispredict",
+ "MetricExpr": "PM_DISP_STALL_BR_MPRED / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_BR_MPRED_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch for any reason",
+ "MetricExpr": "PM_DISP_STALL_HELD_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch because of a synchronizing instruction that requires the ICT to be empty before dispatch",
+ "MetricExpr": "PM_DISP_STALL_HELD_SYNC_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISP_HELD_STALL_SYNC_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch while waiting on the scoreboard",
+ "MetricExpr": "PM_DISP_STALL_HELD_SCOREBOARD_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISP_HELD_STALL_SCOREBOARD_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch due to issue queue full",
+ "MetricExpr": "PM_DISP_STALL_HELD_ISSQ_FULL_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISP_HELD_STALL_ISSQ_FULL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch because the mapper/SRB was full",
+ "MetricExpr": "PM_DISP_STALL_HELD_RENAME_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_RENAME_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch because the STF mapper/SRB was full",
+ "MetricExpr": "PM_DISP_STALL_HELD_STF_MAPPER_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_STF_MAPPER_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch because the XVFC mapper/SRB was full",
+ "MetricExpr": "PM_DISP_STALL_HELD_XVFC_MAPPER_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_XVFC_MAPPER_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch for any other reason",
+ "MetricExpr": "PM_DISP_STALL_HELD_OTHER_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_OTHER_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction has been dispatched but not issued for any reason",
+ "MetricExpr": "PM_ISSUE_STALL / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "ISSUE_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting to be finished in one of the execution units",
+ "MetricExpr": "PM_EXEC_STALL / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "EXECUTION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction spent executing an NTC instruction that gets flushed some time after dispatch",
+ "MetricExpr": "PM_EXEC_STALL_NTC_FLUSH / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "NTC_FLUSH_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTF instruction finishes at dispatch",
+ "MetricExpr": "PM_EXEC_STALL_FIN_AT_DISP / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "FIN_AT_DISP_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is executing in the branch unit",
+ "MetricExpr": "PM_EXEC_STALL_BRU / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "BRU_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a simple fixed point instruction that is executing in the LSU",
+ "MetricExpr": "PM_EXEC_STALL_SIMPLE_FX / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "SIMPLE_FX_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is executing in the VSU",
+ "MetricExpr": "PM_EXEC_STALL_VSU / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "VSU_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting to be finished in one of the execution units",
+ "MetricExpr": "PM_EXEC_STALL_TRANSLATION / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "TRANSLATION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a load or store that suffered a translation miss",
+ "MetricExpr": "PM_EXEC_STALL_DERAT_ONLY_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DERAT_ONLY_MISS_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is recovering from a TLB miss",
+ "MetricExpr": "PM_EXEC_STALL_DERAT_DTLB_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DERAT_DTLB_MISS_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is executing in the LSU",
+ "MetricExpr": "PM_EXEC_STALL_LSU / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "LSU_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a load that is executing in the LSU",
+ "MetricExpr": "PM_EXEC_STALL_LOAD / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "LOAD_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from either the local L2 or local L3",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_L2L3 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_L2L3_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from either the local L2 or local L3, with an RC dispatch conflict",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_L2L3_CONFLICT / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_L2L3_CONFLICT_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from either the local L2 or local L3, without an RC dispatch conflict",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_L2L3_NOCONFLICT / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_L2L3_NOCONFLICT_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from a source beyond the local L2 and local L3",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_L3MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_L3MISS_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from a neighbor chiplet's L2 or L3 in the same chip",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_L21_L31 / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_L21_L31_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from L4, local memory or OpenCAPI chip",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_LMEM / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_LMEM_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from a remote chip (cache, L4, memory or OpenCAPI) in the same group",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_OFF_CHIP / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_OFF_CHIP_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is waiting for a load miss to resolve from a distant chip (cache, L4, memory or OpenCAPI chip)",
+ "MetricExpr": "PM_EXEC_STALL_DMISS_OFF_NODE / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DMISS_OFF_NODE_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is executing a TLBIEL instruction",
+ "MetricExpr": "PM_EXEC_STALL_TLBIEL / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "TLBIEL_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is finishing a load after its data has been reloaded from a data source beyond the local L1, OR when the LSU is processing an L1-hit, OR when the NTF instruction merged with another load in the LMQ",
+ "MetricExpr": "PM_EXEC_STALL_LOAD_FINISH / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "LOAD_FINISH_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a store that is executing in the LSU",
+ "MetricExpr": "PM_EXEC_STALL_STORE / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "STORE_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is in the store unit outside of handling store misses or other special store operations",
+ "MetricExpr": "PM_EXEC_STALL_STORE_PIPE / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "STORE_PIPE_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a store whose cache line was not resident in the L1 and had to wait for allocation of the missing line into the L1",
+ "MetricExpr": "PM_EXEC_STALL_STORE_MISS / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "STORE_MISS_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a TLBIE instruction waiting for a response from the L2",
+ "MetricExpr": "PM_EXEC_STALL_TLBIE / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "TLBIE_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is executing a PTESYNC instruction",
+ "MetricExpr": "PM_EXEC_STALL_PTESYNC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "PTESYNC_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction cannot complete because the thread was blocked",
+ "MetricExpr": "PM_CMPL_STALL / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction cannot complete because it was interrupted by ANY exception",
+ "MetricExpr": "PM_CMPL_STALL_EXCEPTION / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "EXCEPTION_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is stuck at finish waiting for the non-speculative finish of either a STCX instruction waiting for its result or a load waiting for non-critical sectors of data and ECC",
+ "MetricExpr": "PM_CMPL_STALL_MEM_ECC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "MEM_ECC_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a STCX instruction waiting for resolution from the nest",
+ "MetricExpr": "PM_CMPL_STALL_STCX / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "STCX_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a LWSYNC instruction waiting to complete",
+ "MetricExpr": "PM_CMPL_STALL_LWSYNC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "LWSYNC_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction is a HWSYNC instruction stuck at finish waiting for a response from the L2",
+ "MetricExpr": "PM_CMPL_STALL_HWSYNC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "HWSYNC_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction required special handling before completion",
+ "MetricExpr": "PM_CMPL_STALL_SPECIAL / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "SPECIAL_COMPLETION_STALL_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when dispatch was stalled because fetch was being held, so there was nothing in the pipeline for this thread",
+ "MetricExpr": "PM_DISP_STALL_FETCH / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_FETCH_CPI"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTC instruction was held at dispatch because of power management",
+ "MetricExpr": "PM_DISP_STALL_HELD_HALT_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "CPI",
+ "MetricName": "DISPATCHED_HELD_HALT_CPI"
+ },
+ {
+ "BriefDescription": "Percentage of flushes per completed instruction",
+ "MetricExpr": "PM_FLUSH / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Others",
+ "MetricName": "FLUSH_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of flushes due to a branch mispredict per completed instruction",
+ "MetricExpr": "PM_FLUSH_MPRED / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Others",
+ "MetricName": "BR_MPRED_FLUSH_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of branch mispredictions per completed instruction",
+ "MetricExpr": "PM_BR_MPRED_CMPL / PM_RUN_INST_CMPL",
+ "MetricGroup": "Others",
+ "MetricName": "BRANCH_MISPREDICTION_RATE"
+ },
+ {
+ "BriefDescription": "Percentage of finished loads that missed in the L1",
+ "MetricExpr": "PM_LD_MISS_L1 / PM_LD_REF_L1 * 100",
+ "MetricGroup": "Others",
+ "MetricName": "L1_LD_MISS_RATIO",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions that were loads that missed the L1",
+ "MetricExpr": "PM_LD_MISS_L1 / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Others",
+ "MetricName": "L1_LD_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions when the DPTEG required for the load/store instruction in execution was missing from the TLB",
+ "MetricExpr": "PM_DTLB_MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Others",
+ "MetricName": "DTLB_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of completed instructions dispatched per instruction completed",
+ "MetricExpr": "PM_INST_DISP / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "DISPATCH_PER_INST_CMPL"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions that were a demand load that did not hit in the L1 or L2",
+ "MetricExpr": "PM_DATA_FROM_L2MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "General",
+ "MetricName": "L2_LD_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions that were demand fetches that missed the L1 icache",
+ "MetricExpr": "PM_L1_ICACHE_MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Instruction_Misses",
+ "MetricName": "L1_INST_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions that were demand fetches that reloaded from beyond the L3 icache",
+ "MetricExpr": "PM_INST_FROM_L3MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "General",
+ "MetricName": "L3_INST_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of completed instructions per cycle",
+ "MetricExpr": "PM_INST_CMPL / PM_CYC",
+ "MetricGroup": "General",
+ "MetricName": "IPC"
+ },
+ {
+ "BriefDescription": "Average number of cycles per completed instruction group",
+ "MetricExpr": "PM_CYC / PM_1PLUS_PPC_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "CYCLES_PER_COMPLETED_INSTRUCTIONS_SET"
+ },
+ {
+ "BriefDescription": "Percentage of cycles when at least 1 instruction dispatched",
+ "MetricExpr": "PM_1PLUS_PPC_DISP / PM_RUN_CYC * 100",
+ "MetricGroup": "General",
+ "MetricName": "CYCLES_ATLEAST_ONE_INST_DISPATCHED",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of finished loads per completed instruction",
+ "MetricExpr": "PM_LD_REF_L1 / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "LOADS_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of finished stores per completed instruction",
+ "MetricExpr": "PM_ST_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "STORES_PER_INST"
+ },
+ {
+ "BriefDescription": "Percentage of demand loads that reloaded from beyond the L2 per completed instruction",
+ "MetricExpr": "PM_DATA_FROM_L2MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "dL1_Reloads",
+ "MetricName": "DL1_RELOAD_FROM_L2_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of demand loads that reloaded from beyond the L3 per completed instruction",
+ "MetricExpr": "PM_DATA_FROM_L3MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "dL1_Reloads",
+ "MetricName": "DL1_RELOAD_FROM_L3_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses with 4k page size per completed instruction",
+ "MetricExpr": "PM_DERAT_MISS_4K / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_4K_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses with 64k page size per completed instruction",
+ "MetricExpr": "PM_DERAT_MISS_64K / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_64K_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of run cycles per completed instruction",
+ "MetricExpr": "PM_RUN_CYC / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "RUN_CPI"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses per completed instruction",
+ "MetricExpr": "PM_DERAT_MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of completed instructions per run cycle",
+ "MetricExpr": "PM_RUN_INST_CMPL / PM_RUN_CYC",
+ "MetricGroup": "General",
+ "MetricName": "RUN_IPC"
+ },
+ {
+ "BriefDescription": "Average number of completed instructions per instruction group",
+ "MetricExpr": "PM_RUN_INST_CMPL / PM_1PLUS_PPC_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "AVERAGE_COMPLETED_INSTRUCTION_SET_SIZE"
+ },
+ {
+ "BriefDescription": "Average number of finished instructions per completed instructions",
+ "MetricExpr": "PM_INST_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "INST_FIN_PER_CMPL"
+ },
+ {
+ "BriefDescription": "Average cycles per completed instruction when the NTF instruction is completing and the finish was overlooked",
+ "MetricExpr": "PM_EXEC_STALL_UNKNOWN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "EXEC_STALL_UNKOWN_CPI"
+ },
+ {
+ "BriefDescription": "Percentage of finished branches that were taken",
+ "MetricExpr": "PM_BR_TAKEN_CMPL / PM_BR_FIN * 100",
+ "MetricGroup": "General",
+ "MetricName": "TAKEN_BRANCHES",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of completed instructions that were a demand load that did not hit in the L1, L2, or the L3",
+ "MetricExpr": "PM_DATA_FROM_L3MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "General",
+ "MetricName": "L3_LD_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Average number of finished branches per completed instruction",
+ "MetricExpr": "PM_BR_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "BRANCHES_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of instructions finished in the LSU per completed instruction",
+ "MetricExpr": "PM_LSU_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "LSU_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of instructions finished in the VSU per completed instruction",
+ "MetricExpr": "PM_VSU_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "VSU_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of TLBIE instructions finished in the LSU per completed instruction",
+ "MetricExpr": "PM_TLBIE_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "TLBIE_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of STCX instructions finshed per completed instruction",
+ "MetricExpr": "PM_STCX_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "STXC_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of LARX instructions finshed per completed instruction",
+ "MetricExpr": "PM_LARX_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "LARX_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of PTESYNC instructions finshed per completed instruction",
+ "MetricExpr": "PM_PTESYNC_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "PTESYNC_PER_INST"
+ },
+ {
+ "BriefDescription": "Average number of simple fixed-point instructions finshed in the store unit per completed instruction",
+ "MetricExpr": "PM_FX_LSU_FIN / PM_RUN_INST_CMPL",
+ "MetricGroup": "General",
+ "MetricName": "FX_PER_INST"
+ },
+ {
+ "BriefDescription": "Percentage of demand load misses that reloaded the L1 cache",
+ "MetricExpr": "PM_LD_DEMAND_MISS_L1 / PM_LD_MISS_L1 * 100",
+ "MetricGroup": "General",
+ "MetricName": "DL1_MISS_RELOADS",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of demand load misses that reloaded from beyond the local L2",
+ "MetricExpr": "PM_DATA_FROM_L2MISS / PM_LD_DEMAND_MISS_L1 * 100",
+ "MetricGroup": "dL1_Reloads",
+ "MetricName": "DL1_RELOAD_FROM_L2_MISS",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of demand load misses that reloaded from beyond the local L3",
+ "MetricExpr": "PM_DATA_FROM_L3MISS / PM_LD_DEMAND_MISS_L1 * 100",
+ "MetricGroup": "dL1_Reloads",
+ "MetricName": "DL1_RELOAD_FROM_L3_MISS",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of cycles stalled due to the NTC instruction waiting for a load miss to resolve from a source beyond the local L2 and local L3",
+ "MetricExpr": "DMISS_L3MISS_STALL_CPI / RUN_CPI * 100",
+ "MetricGroup": "General",
+ "MetricName": "DCACHE_MISS_CPI",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses with 2M page size per completed instruction",
+ "MetricExpr": "PM_DERAT_MISS_2M / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_2M_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses with 16M page size per completed instruction",
+ "MetricExpr": "PM_DERAT_MISS_16M / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_16M_MISS_RATE",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "DERAT miss ratio for 4K page size",
+ "MetricExpr": "PM_DERAT_MISS_4K / PM_DERAT_MISS",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_4K_MISS_RATIO"
+ },
+ {
+ "BriefDescription": "DERAT miss ratio for 2M page size",
+ "MetricExpr": "PM_DERAT_MISS_2M / PM_DERAT_MISS",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_2M_MISS_RATIO"
+ },
+ {
+ "BriefDescription": "DERAT miss ratio for 16M page size",
+ "MetricExpr": "PM_DERAT_MISS_16M / PM_DERAT_MISS",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_16M_MISS_RATIO"
+ },
+ {
+ "BriefDescription": "DERAT miss ratio for 64K page size",
+ "MetricExpr": "PM_DERAT_MISS_64K / PM_DERAT_MISS",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_64K_MISS_RATIO"
+ },
+ {
+ "BriefDescription": "Percentage of DERAT misses that resulted in TLB reloads",
+ "MetricExpr": "PM_DTLB_MISS / PM_DERAT_MISS * 100",
+ "MetricGroup": "Translation",
+ "MetricName": "DERAT_MISS_RELOAD",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of icache misses that were reloaded from beyond the local L3",
+ "MetricExpr": "PM_INST_FROM_L3MISS / PM_L1_ICACHE_MISS * 100",
+ "MetricGroup": "Instruction_Misses",
+ "MetricName": "INST_FROM_L3_MISS",
+ "ScaleUnit": "1%"
+ },
+ {
+ "BriefDescription": "Percentage of icache reloads from the beyond the L3 per completed instruction",
+ "MetricExpr": "PM_INST_FROM_L3MISS / PM_RUN_INST_CMPL * 100",
+ "MetricGroup": "Instruction_Misses",
+ "MetricName": "INST_FROM_L3_MISS_RATE",
+ "ScaleUnit": "1%"
+ }
+]
"EventName": "PROBLEM_STATE_L1D_PENALTY_CYCLES",
"BriefDescription": "Problem-State L1D Penalty Cycles",
"PublicDescription": "Problem-State Level-1 D-Cache Penalty Cycle Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "L2C_STORES_SENT",
"BriefDescription": "L2C Stores Sent",
"PublicDescription": "Incremented by one for every store sent to Level-2 (L1.5) cache"
- },
+ }
]
"EventName": "PROBLEM_STATE_L1D_PENALTY_CYCLES",
"BriefDescription": "Problem-State L1D Penalty Cycles",
"PublicDescription": "Problem-State Level-1 D-Cache Penalty Cycle Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "MT_DIAG_CYCLES_TWO_THR_ACTIVE",
"BriefDescription": "Cycle count with two threads active",
"PublicDescription": "Cycle count with two threads active"
- },
+ }
]
"EventName": "PROBLEM_STATE_INSTRUCTIONS",
"BriefDescription": "Problem-State Instructions",
"PublicDescription": "Problem-State Instruction Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "MT_DIAG_CYCLES_TWO_THR_ACTIVE",
"BriefDescription": "Cycle count with two threads active",
"PublicDescription": "Cycle count with two threads active"
- },
+ }
]
"EventName": "PROBLEM_STATE_INSTRUCTIONS",
"BriefDescription": "Problem-State Instructions",
"PublicDescription": "Problem-State Instruction Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "ECC_BLOCKED_CYCLES_COUNT",
"BriefDescription": "ECC Blocked Cycles Count",
"PublicDescription": "This counter counts the total number of CPU cycles blocked for the elliptic-curve cryptography (ECC) functions issued by the CPU because the ECC coprocessor is busy performing a function issued by another CPU."
- },
+ }
]
"EventName": "MT_DIAG_CYCLES_TWO_THR_ACTIVE",
"BriefDescription": "Cycle count with two threads active",
"PublicDescription": "Cycle count with two threads active"
- },
+ }
]
"EventName": "PROBLEM_STATE_L1D_PENALTY_CYCLES",
"BriefDescription": "Problem-State L1D Penalty Cycles",
"PublicDescription": "Problem-State Level-1 D-Cache Penalty Cycle Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "L1I_OFFCHIP_L3_SOURCED_WRITES",
"BriefDescription": "L1I Off-Chip L3 Sourced Writes",
"PublicDescription": "A directory write to the Level-1 I-Cache directory where the returned cache line was sourced from an Off Chip/On Book Level-3 cache"
- },
+ }
]
"EventName": "PROBLEM_STATE_L1D_PENALTY_CYCLES",
"BriefDescription": "Problem-State L1D Penalty Cycles",
"PublicDescription": "Problem-State Level-1 D-Cache Penalty Cycle Count"
- },
+ }
]
"EventName": "AES_BLOCKED_CYCLES",
"BriefDescription": "AES Blocked Cycles",
"PublicDescription": "Total number of CPU cycles blocked for the AES functions issued by the CPU because the DEA/AES coprocessor is busy performing a function issued by another CPU"
- },
+ }
]
"EventName": "TX_C_TABORT_SPECIAL",
"BriefDescription": "Aborted transactions in constrained TX mode using special completion logic",
"PublicDescription": "A transaction abort has occurred in a constrained transactional-execution mode and the CPU is using special logic to allow the transaction to complete"
- },
+ }
]
"BriefDescription": "Total cache hits",
"PublicDescription": "Total cache hits",
"Unit": "imc"
- },
+ }
]
"Unit": "sys_ddr_pmu",
"Compat": "v8"
},
+ {
+ "BriefDescription": "ccn read-cycles event",
+ "ConfigCode": "0x2c",
+ "EventName": "sys_ccn_pmu.read_cycles",
+ "Unit": "sys_ccn_pmu",
+ "Compat": "0x01"
+ }
]
"MetricExpr": "(cstate_pkg@c6\\-residency@ / msr@tsc@) * 100",
"MetricGroup": "Power",
"MetricName": "C6_Pkg_Residency"
- },
+ }
]
#include <sys/resource.h> /* getrlimit */
#include <ftw.h>
#include <sys/stat.h>
+#include <linux/compiler.h>
#include <linux/list.h>
#include "jsmn.h"
#include "json.h"
char *metric_constraint;
};
-enum aggr_mode_class convert(const char *aggr_mode)
+static enum aggr_mode_class convert(const char *aggr_mode)
{
if (!strcmp(aggr_mode, "PerCore"))
return PerCore;
return -1;
}
-typedef int (*func)(void *data, struct json_event *je);
-
static LIST_HEAD(sys_event_tables);
struct sys_event_table {
static void print_events_table_prefix(FILE *fp, const char *tblname)
{
- fprintf(fp, "struct pmu_event %s[] = {\n", tblname);
+ fprintf(fp, "static const struct pmu_event %s[] = {\n", tblname);
close_table = 1;
}
{
struct perf_entry_data *pd = data;
FILE *outfp = pd->outfp;
- char *topic = pd->topic;
+ char *topic_local = pd->topic;
/*
* TODO: Remove formatting chars after debugging to reduce
fprintf(outfp, "\t.desc = \"%s\",\n", je->desc);
if (je->compat)
fprintf(outfp, "\t.compat = \"%s\",\n", je->compat);
- fprintf(outfp, "\t.topic = \"%s\",\n", topic);
+ fprintf(outfp, "\t.topic = \"%s\",\n", topic_local);
if (je->long_desc && je->long_desc[0])
fprintf(outfp, "\t.long_desc = \"%s\",\n", je->long_desc);
if (je->pmu)
}
}
-static int save_arch_std_events(void *data, struct json_event *je)
+static int save_arch_std_events(void *data __maybe_unused, struct json_event *je)
{
struct event_struct *es;
struct json_event je = {};
char *arch_std = NULL;
unsigned long long eventcode = 0;
+ unsigned long long configcode = 0;
struct msrmap *msr = NULL;
jsmntok_t *msrval = NULL;
jsmntok_t *precise = NULL;
jsmntok_t *obj = tok++;
+ bool configcode_present = false;
EXPECT(obj->type == JSMN_OBJECT, obj, "expected object");
for (j = 0; j < obj->size; j += 2) {
addfield(map, &code, "", "", val);
eventcode |= strtoul(code, NULL, 0);
free(code);
+ } else if (json_streq(map, field, "ConfigCode")) {
+ char *code = NULL;
+ addfield(map, &code, "", "", val);
+ configcode |= strtoul(code, NULL, 0);
+ free(code);
+ configcode_present = true;
} else if (json_streq(map, field, "ExtSel")) {
char *code = NULL;
addfield(map, &code, "", "", val);
addfield(map, &extra_desc, " ",
"(Precise event)", NULL);
}
- snprintf(buf, sizeof buf, "event=%#llx", eventcode);
+ if (configcode_present)
+ snprintf(buf, sizeof buf, "config=%#llx", configcode);
+ else
+ snprintf(buf, sizeof buf, "event=%#llx", eventcode);
addfield(map, &event, ",", buf, NULL);
if (je.desc && extra_desc)
addfield(map, &je.desc, " ", extra_desc, NULL);
static void print_mapping_table_prefix(FILE *outfp)
{
- fprintf(outfp, "struct pmu_events_map pmu_events_map[] = {\n");
+ fprintf(outfp, "const struct pmu_events_map pmu_events_map[] = {\n");
}
static void print_mapping_table_suffix(FILE *outfp)
static void print_system_event_mapping_table_prefix(FILE *outfp)
{
- fprintf(outfp, "\nstruct pmu_sys_events pmu_sys_event_tables[] = {");
+ fprintf(outfp, "\nconst struct pmu_sys_events pmu_sys_event_tables[] = {");
}
static void print_system_event_mapping_table_suffix(FILE *outfp)
const char *arch;
const char *output_file;
const char *start_dirname;
- char *err_string_ext = "";
+ const char *err_string_ext = "";
struct stat stbuf;
prog = basename(argv[0]);
#include <stdlib.h>
#include "jsmn.h"
+#define JSMN_STRICT
/*
* Allocates a fresh unused token from the token pool.
jsmnerr_t r;
int i;
jsmntok_t *token;
+#ifdef JSMN_STRICT
+ /*
+ * Keeps track of whether a new object/list/primitive is expected. New items are only
+ * allowed after an opening brace, comma or colon. A closing brace after a comma is not
+ * valid JSON.
+ */
+ int expecting_item = 1;
+#endif
for (; parser->pos < len; parser->pos++) {
char c;
switch (c) {
case '{':
case '[':
+#ifdef JSMN_STRICT
+ if (!expecting_item)
+ return JSMN_ERROR_INVAL;
+#endif
token = jsmn_alloc_token(parser, tokens, num_tokens);
if (token == NULL)
return JSMN_ERROR_NOMEM;
break;
case '}':
case ']':
+#ifdef JSMN_STRICT
+ if (expecting_item)
+ return JSMN_ERROR_INVAL;
+#endif
type = (c == '}' ? JSMN_OBJECT : JSMN_ARRAY);
for (i = parser->toknext - 1; i >= 0; i--) {
token = &tokens[i];
}
break;
case '\"':
+#ifdef JSMN_STRICT
+ if (!expecting_item)
+ return JSMN_ERROR_INVAL;
+ expecting_item = 0;
+#endif
r = jsmn_parse_string(parser, js, len, tokens,
num_tokens);
if (r < 0)
case '\t':
case '\r':
case '\n':
- case ':':
- case ',':
case ' ':
break;
#ifdef JSMN_STRICT
+ case ':':
+ case ',':
+ if (expecting_item)
+ return JSMN_ERROR_INVAL;
+ expecting_item = 1;
+ break;
/*
* In strict mode primitives are:
* numbers and booleans.
case 'f':
case 'n':
#else
+ case ':':
+ case ',':
+ break;
/*
* In non-strict mode every unquoted value
* is a primitive.
/*FALL THROUGH */
default:
#endif
+
+#ifdef JSMN_STRICT
+ if (!expecting_item)
+ return JSMN_ERROR_INVAL;
+ expecting_item = 0;
+#endif
r = jsmn_parse_primitive(parser, js, len, tokens,
num_tokens);
if (r < 0)
return JSMN_ERROR_PART;
}
+#ifdef JSMN_STRICT
+ return expecting_item ? JSMN_ERROR_INVAL : JSMN_SUCCESS;
+#else
return JSMN_SUCCESS;
+#endif
}
/*
const char *cpuid;
const char *version;
const char *type; /* core, uncore etc */
- struct pmu_event *table;
+ const struct pmu_event *table;
};
struct pmu_sys_events {
const char *name;
- struct pmu_event *table;
+ const struct pmu_event *table;
};
/*
* Global table mapping each known CPU for the architecture to its
* table of PMU events.
*/
-extern struct pmu_events_map pmu_events_map[];
-extern struct pmu_sys_events pmu_sys_event_tables[];
+extern const struct pmu_events_map pmu_events_map[];
+extern const struct pmu_sys_events pmu_sys_event_tables[];
#endif
return ret;
}
-int test__api_io(struct test *test __maybe_unused,
- int subtest __maybe_unused)
+static int test__api_io(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int ret = 0;
ret = TEST_FAIL;
return ret;
}
+
+DEFINE_SUITE("Test api io", api_io);
return system(cmd) ? TEST_FAIL : TEST_OK;
}
-int test__attr(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__attr(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct stat st;
char path_perf[PATH_MAX];
return TEST_SKIP;
}
+
+DEFINE_SUITE("Setup struct perf_event_attr", attr);
}
-int test__backward_ring_buffer(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__backward_ring_buffer(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret = TEST_SKIP, err, sample_count = 0, comm_count = 0;
char pid[16], sbuf[STRERR_BUFSIZE];
goto out_delete_evlist;
}
- bzero(&parse_error, sizeof(parse_error));
+ parse_events_error__init(&parse_error);
/*
* Set backward bit, ring buffer should be writing from end. Record
* it in aux evlist
*/
err = parse_events(evlist, "syscalls:sys_enter_prctl/overwrite/", &parse_error);
+ parse_events_error__exit(&parse_error);
if (err) {
pr_debug("Failed to parse tracepoint event, try use root\n");
ret = TEST_SKIP;
evlist__delete(evlist);
return ret;
}
+
+DEFINE_SUITE("Read backward ring buffer", backward_ring_buffer);
return ret;
}
-int test__bitmap_print(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__bitmap_print(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
TEST_ASSERT_VAL("failed to convert map", test_bitmap("1"));
TEST_ASSERT_VAL("failed to convert map", test_bitmap("1,5"));
TEST_ASSERT_VAL("failed to convert map", test_bitmap("1-10,12-20,22-30,32-40"));
return 0;
}
+
+DEFINE_SUITE("Print bitmap", bitmap_print);
#include "../perf-sys.h"
#include "cloexec.h"
+/*
+ * PowerPC and S390 do not support creation of instruction breakpoints using the
+ * perf_event interface.
+ *
+ * Just disable the test for these architectures until these issues are
+ * resolved.
+ */
+#if defined(__powerpc__) || defined(__s390x__)
+#define BP_ACCOUNT_IS_SUPPORTED 0
+#else
+#define BP_ACCOUNT_IS_SUPPORTED 1
+#endif
+
static volatile long the_var;
static noinline int test_function(void)
* we create another watchpoint to ensure
* the slot accounting is correct
*/
-int test__bp_accounting(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__bp_accounting(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int has_ioctl = detect_ioctl();
int wp_cnt = detect_cnt(false);
int bp_cnt = detect_cnt(true);
int share = detect_share(wp_cnt, bp_cnt);
+ if (!BP_ACCOUNT_IS_SUPPORTED) {
+ pr_debug("Test not supported on this architecture");
+ return TEST_SKIP;
+ }
+
pr_debug("watchpoints count %d, breakpoints count %d, has_ioctl %d, share %d\n",
wp_cnt, bp_cnt, has_ioctl, share);
return bp_accounting(wp_cnt, share);
}
-bool test__bp_account_is_supported(void)
-{
- /*
- * PowerPC and S390 do not support creation of instruction
- * breakpoints using the perf_event interface.
- *
- * Just disable the test for these architectures until these
- * issues are resolved.
- */
-#if defined(__powerpc__) || defined(__s390x__)
- return false;
-#else
- return true;
-#endif
-}
+DEFINE_SUITE("Breakpoint accounting", bp_accounting);
return count;
}
-int test__bp_signal(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__bp_signal(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct sigaction sa;
long long count1, count2, count3;
+ if (!BP_SIGNAL_IS_SUPPORTED) {
+ pr_debug("Test not supported on this architecture");
+ return TEST_SKIP;
+ }
+
/* setup SIGIO signal handler */
memset(&sa, 0, sizeof(struct sigaction));
sa.sa_sigaction = (void *) sig_handler;
TEST_OK : TEST_FAIL;
}
-bool test__bp_signal_is_supported(void)
-{
- /*
- * PowerPC and S390 do not support creation of instruction
- * breakpoints using the perf_event interface.
- *
- * ARM requires explicit rounding down of the instruction
- * pointer in Thumb mode, and then requires the single-step
- * to be handled explicitly in the overflow handler to avoid
- * stepping into the SIGIO handler and getting stuck on the
- * breakpointed instruction.
- *
- * Since arm64 has the same issue with arm for the single-step
- * handling, this case also gets stuck on the breakpointed
- * instruction.
- *
- * Just disable the test for these architectures until these
- * issues are resolved.
- */
-#if defined(__powerpc__) || defined(__s390x__) || defined(__arm__) || \
- defined(__aarch64__)
- return false;
-#else
- return true;
-#endif
-}
+DEFINE_SUITE("Breakpoint overflow signal handler", bp_signal);
#define EXECUTIONS 10000
#define THRESHOLD 100
-int test__bp_signal_overflow(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__bp_signal_overflow(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_event_attr pe;
struct sigaction sa;
long long count;
int fd, i, fails = 0;
+ if (!BP_SIGNAL_IS_SUPPORTED) {
+ pr_debug("Test not supported on this architecture");
+ return TEST_SKIP;
+ }
+
/* setup SIGIO signal handler */
memset(&sa, 0, sizeof(struct sigaction));
sa.sa_sigaction = (void *) sig_handler;
return fails ? TEST_FAIL : TEST_OK;
}
+
+DEFINE_SUITE("Breakpoint overflow sampling", bp_signal_overflow);
static struct {
enum test_llvm__testcase prog_id;
- const char *desc;
const char *name;
const char *msg_compile_fail;
const char *msg_load_fail;
} bpf_testcase_table[] = {
{
.prog_id = LLVM_TESTCASE_BASE,
- .desc = "Basic BPF filtering",
.name = "[basic_bpf_test]",
.msg_compile_fail = "fix 'perf test LLVM' first",
.msg_load_fail = "load bpf object failed",
},
{
.prog_id = LLVM_TESTCASE_BASE,
- .desc = "BPF pinning",
.name = "[bpf_pinning]",
.msg_compile_fail = "fix kbuild first",
.msg_load_fail = "check your vmlinux setting?",
#ifdef HAVE_BPF_PROLOGUE
{
.prog_id = LLVM_TESTCASE_BPF_PROLOGUE,
- .desc = "BPF prologue generation",
.name = "[bpf_prologue_test]",
.msg_compile_fail = "fix kbuild first",
.msg_load_fail = "check your vmlinux setting?",
struct parse_events_state parse_state;
struct parse_events_error parse_error;
- bzero(&parse_error, sizeof(parse_error));
+ parse_events_error__init(&parse_error);
bzero(&parse_state, sizeof(parse_state));
parse_state.error = &parse_error;
INIT_LIST_HEAD(&parse_state.list);
err = parse_events_load_bpf_obj(&parse_state, &parse_state.list, obj, NULL);
+ parse_events_error__exit(&parse_error);
if (err || list_empty(&parse_state.list)) {
pr_debug("Failed to add events selected by BPF\n");
return TEST_FAIL;
return ret;
}
-int test__bpf_subtest_get_nr(void)
-{
- return (int)ARRAY_SIZE(bpf_testcase_table);
-}
-
-const char *test__bpf_subtest_get_desc(int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(bpf_testcase_table))
- return NULL;
- return bpf_testcase_table[i].desc;
-}
-
static int check_env(void)
{
int err;
}
err = bpf_load_program(BPF_PROG_TYPE_KPROBE, insns,
- sizeof(insns) / sizeof(insns[0]),
+ ARRAY_SIZE(insns),
license, kver_int, NULL, 0);
if (err < 0) {
pr_err("Missing basic BPF support, skip this test: %s\n",
return 0;
}
-int test__bpf(struct test *test __maybe_unused, int i)
+static int test__bpf(int i)
{
int err;
err = __test__bpf(i);
return err;
}
+#endif
-#else
-int test__bpf_subtest_get_nr(void)
+static int test__basic_bpf_test(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return 0;
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__bpf(0);
+#else
+ pr_debug("Skip BPF test because BPF support is not compiled\n");
+ return TEST_SKIP;
+#endif
}
-const char *test__bpf_subtest_get_desc(int i __maybe_unused)
+static int test__bpf_pinning(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return NULL;
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__bpf(1);
+#else
+ pr_debug("Skip BPF test because BPF support is not compiled\n");
+ return TEST_SKIP;
+#endif
}
-int test__bpf(struct test *test __maybe_unused, int i __maybe_unused)
+static int test__bpf_prologue_test(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#if defined(HAVE_LIBBPF_SUPPORT) && defined(HAVE_BPF_PROLOGUE)
+ return test__bpf(2);
+#else
pr_debug("Skip BPF test because BPF support is not compiled\n");
return TEST_SKIP;
+#endif
}
+
+
+static struct test_case bpf_tests[] = {
+#ifdef HAVE_LIBBPF_SUPPORT
+ TEST_CASE("Basic BPF filtering", basic_bpf_test),
+ TEST_CASE("BPF pinning", bpf_pinning),
+#ifdef HAVE_BPF_PROLOGUE
+ TEST_CASE("BPF prologue generation", bpf_prologue_test),
+#else
+ TEST_CASE_REASON("BPF prologue generation", bpf_prologue_test, "not compiled in"),
+#endif
+#else
+ TEST_CASE_REASON("Basic BPF filtering", basic_bpf_test, "not compiled in"),
+ TEST_CASE_REASON("BPF pinning", bpf_pinning, "not compiled in"),
+ TEST_CASE_REASON("BPF prologue generation", bpf_prologue_test, "not compiled in"),
#endif
+ { .name = NULL, }
+};
+
+struct test_suite suite__bpf = {
+ .desc = "BPF filter",
+ .test_cases = bpf_tests,
+};
static bool dont_fork;
-struct test __weak arch_tests[] = {
- {
- .func = NULL,
- },
+struct test_suite *__weak arch_tests[] = {
+ NULL,
};
-static struct test generic_tests[] = {
- {
- .desc = "vmlinux symtab matches kallsyms",
- .func = test__vmlinux_matches_kallsyms,
- },
- {
- .desc = "Detect openat syscall event",
- .func = test__openat_syscall_event,
- },
- {
- .desc = "Detect openat syscall event on all cpus",
- .func = test__openat_syscall_event_on_all_cpus,
- },
- {
- .desc = "Read samples using the mmap interface",
- .func = test__basic_mmap,
- },
- {
- .desc = "Test data source output",
- .func = test__mem,
- },
- {
- .desc = "Parse event definition strings",
- .func = test__parse_events,
- },
- {
- .desc = "Simple expression parser",
- .func = test__expr,
- },
- {
- .desc = "PERF_RECORD_* events & perf_sample fields",
- .func = test__PERF_RECORD,
- },
- {
- .desc = "Parse perf pmu format",
- .func = test__pmu,
- },
- {
- .desc = "PMU events",
- .func = test__pmu_events,
- .subtest = {
- .skip_if_fail = false,
- .get_nr = test__pmu_events_subtest_get_nr,
- .get_desc = test__pmu_events_subtest_get_desc,
- .skip_reason = test__pmu_events_subtest_skip_reason,
- },
-
- },
- {
- .desc = "DSO data read",
- .func = test__dso_data,
- },
- {
- .desc = "DSO data cache",
- .func = test__dso_data_cache,
- },
- {
- .desc = "DSO data reopen",
- .func = test__dso_data_reopen,
- },
- {
- .desc = "Roundtrip evsel->name",
- .func = test__perf_evsel__roundtrip_name_test,
- },
- {
- .desc = "Parse sched tracepoints fields",
- .func = test__perf_evsel__tp_sched_test,
- },
- {
- .desc = "syscalls:sys_enter_openat event fields",
- .func = test__syscall_openat_tp_fields,
- },
- {
- .desc = "Setup struct perf_event_attr",
- .func = test__attr,
- },
- {
- .desc = "Match and link multiple hists",
- .func = test__hists_link,
- },
- {
- .desc = "'import perf' in python",
- .func = test__python_use,
- },
- {
- .desc = "Breakpoint overflow signal handler",
- .func = test__bp_signal,
- .is_supported = test__bp_signal_is_supported,
- },
- {
- .desc = "Breakpoint overflow sampling",
- .func = test__bp_signal_overflow,
- .is_supported = test__bp_signal_is_supported,
- },
- {
- .desc = "Breakpoint accounting",
- .func = test__bp_accounting,
- .is_supported = test__bp_account_is_supported,
- },
- {
- .desc = "Watchpoint",
- .func = test__wp,
- .is_supported = test__wp_is_supported,
- .subtest = {
- .skip_if_fail = false,
- .get_nr = test__wp_subtest_get_nr,
- .get_desc = test__wp_subtest_get_desc,
- .skip_reason = test__wp_subtest_skip_reason,
- },
- },
- {
- .desc = "Number of exit events of a simple workload",
- .func = test__task_exit,
- },
- {
- .desc = "Software clock events period values",
- .func = test__sw_clock_freq,
- },
- {
- .desc = "Object code reading",
- .func = test__code_reading,
- },
- {
- .desc = "Sample parsing",
- .func = test__sample_parsing,
- },
- {
- .desc = "Use a dummy software event to keep tracking",
- .func = test__keep_tracking,
- },
- {
- .desc = "Parse with no sample_id_all bit set",
- .func = test__parse_no_sample_id_all,
- },
- {
- .desc = "Filter hist entries",
- .func = test__hists_filter,
- },
- {
- .desc = "Lookup mmap thread",
- .func = test__mmap_thread_lookup,
- },
- {
- .desc = "Share thread maps",
- .func = test__thread_maps_share,
- },
- {
- .desc = "Sort output of hist entries",
- .func = test__hists_output,
- },
- {
- .desc = "Cumulate child hist entries",
- .func = test__hists_cumulate,
- },
- {
- .desc = "Track with sched_switch",
- .func = test__switch_tracking,
- },
- {
- .desc = "Filter fds with revents mask in a fdarray",
- .func = test__fdarray__filter,
- },
- {
- .desc = "Add fd to a fdarray, making it autogrow",
- .func = test__fdarray__add,
- },
- {
- .desc = "kmod_path__parse",
- .func = test__kmod_path__parse,
- },
- {
- .desc = "Thread map",
- .func = test__thread_map,
- },
- {
- .desc = "LLVM search and compile",
- .func = test__llvm,
- .subtest = {
- .skip_if_fail = true,
- .get_nr = test__llvm_subtest_get_nr,
- .get_desc = test__llvm_subtest_get_desc,
- },
- },
- {
- .desc = "Session topology",
- .func = test__session_topology,
- },
- {
- .desc = "BPF filter",
- .func = test__bpf,
- .subtest = {
- .skip_if_fail = true,
- .get_nr = test__bpf_subtest_get_nr,
- .get_desc = test__bpf_subtest_get_desc,
- },
- },
- {
- .desc = "Synthesize thread map",
- .func = test__thread_map_synthesize,
- },
- {
- .desc = "Remove thread map",
- .func = test__thread_map_remove,
- },
- {
- .desc = "Synthesize cpu map",
- .func = test__cpu_map_synthesize,
- },
- {
- .desc = "Synthesize stat config",
- .func = test__synthesize_stat_config,
- },
- {
- .desc = "Synthesize stat",
- .func = test__synthesize_stat,
- },
- {
- .desc = "Synthesize stat round",
- .func = test__synthesize_stat_round,
- },
- {
- .desc = "Synthesize attr update",
- .func = test__event_update,
- },
- {
- .desc = "Event times",
- .func = test__event_times,
- },
- {
- .desc = "Read backward ring buffer",
- .func = test__backward_ring_buffer,
- },
- {
- .desc = "Print cpu map",
- .func = test__cpu_map_print,
- },
- {
- .desc = "Merge cpu map",
- .func = test__cpu_map_merge,
- },
-
- {
- .desc = "Probe SDT events",
- .func = test__sdt_event,
- },
- {
- .desc = "is_printable_array",
- .func = test__is_printable_array,
- },
- {
- .desc = "Print bitmap",
- .func = test__bitmap_print,
- },
- {
- .desc = "perf hooks",
- .func = test__perf_hooks,
- },
- {
- .desc = "builtin clang support",
- .func = test__clang,
- .subtest = {
- .skip_if_fail = true,
- .get_nr = test__clang_subtest_get_nr,
- .get_desc = test__clang_subtest_get_desc,
- }
- },
- {
- .desc = "unit_number__scnprintf",
- .func = test__unit_number__scnprint,
- },
- {
- .desc = "mem2node",
- .func = test__mem2node,
- },
- {
- .desc = "time utils",
- .func = test__time_utils,
- },
- {
- .desc = "Test jit_write_elf",
- .func = test__jit_write_elf,
- },
- {
- .desc = "Test libpfm4 support",
- .func = test__pfm,
- .subtest = {
- .skip_if_fail = true,
- .get_nr = test__pfm_subtest_get_nr,
- .get_desc = test__pfm_subtest_get_desc,
- }
- },
- {
- .desc = "Test api io",
- .func = test__api_io,
- },
- {
- .desc = "maps__merge_in",
- .func = test__maps__merge_in,
- },
- {
- .desc = "Demangle Java",
- .func = test__demangle_java,
- },
- {
- .desc = "Demangle OCaml",
- .func = test__demangle_ocaml,
- },
- {
- .desc = "Parse and process metrics",
- .func = test__parse_metric,
- },
- {
- .desc = "PE file support",
- .func = test__pe_file_parsing,
- },
- {
- .desc = "Event expansion for cgroups",
- .func = test__expand_cgroup_events,
- },
- {
- .desc = "Convert perf time to TSC",
- .func = test__perf_time_to_tsc,
- .is_supported = test__tsc_is_supported,
- },
- {
- .desc = "dlfilter C API",
- .func = test__dlfilter,
- },
- {
- .func = NULL,
- },
+static struct test_suite *generic_tests[] = {
+ &suite__vmlinux_matches_kallsyms,
+ &suite__openat_syscall_event,
+ &suite__openat_syscall_event_on_all_cpus,
+ &suite__basic_mmap,
+ &suite__mem,
+ &suite__parse_events,
+ &suite__expr,
+ &suite__PERF_RECORD,
+ &suite__pmu,
+ &suite__pmu_events,
+ &suite__dso_data,
+ &suite__dso_data_cache,
+ &suite__dso_data_reopen,
+ &suite__perf_evsel__roundtrip_name_test,
+ &suite__perf_evsel__tp_sched_test,
+ &suite__syscall_openat_tp_fields,
+ &suite__attr,
+ &suite__hists_link,
+ &suite__python_use,
+ &suite__bp_signal,
+ &suite__bp_signal_overflow,
+ &suite__bp_accounting,
+ &suite__wp,
+ &suite__task_exit,
+ &suite__sw_clock_freq,
+ &suite__code_reading,
+ &suite__sample_parsing,
+ &suite__keep_tracking,
+ &suite__parse_no_sample_id_all,
+ &suite__hists_filter,
+ &suite__mmap_thread_lookup,
+ &suite__thread_maps_share,
+ &suite__hists_output,
+ &suite__hists_cumulate,
+ &suite__switch_tracking,
+ &suite__fdarray__filter,
+ &suite__fdarray__add,
+ &suite__kmod_path__parse,
+ &suite__thread_map,
+ &suite__llvm,
+ &suite__session_topology,
+ &suite__bpf,
+ &suite__thread_map_synthesize,
+ &suite__thread_map_remove,
+ &suite__cpu_map_synthesize,
+ &suite__synthesize_stat_config,
+ &suite__synthesize_stat,
+ &suite__synthesize_stat_round,
+ &suite__event_update,
+ &suite__event_times,
+ &suite__backward_ring_buffer,
+ &suite__cpu_map_print,
+ &suite__cpu_map_merge,
+ &suite__sdt_event,
+ &suite__is_printable_array,
+ &suite__bitmap_print,
+ &suite__perf_hooks,
+ &suite__clang,
+ &suite__unit_number__scnprint,
+ &suite__mem2node,
+ &suite__time_utils,
+ &suite__jit_write_elf,
+ &suite__pfm,
+ &suite__api_io,
+ &suite__maps__merge_in,
+ &suite__demangle_java,
+ &suite__demangle_ocaml,
+ &suite__parse_metric,
+ &suite__pe_file_parsing,
+ &suite__expand_cgroup_events,
+ &suite__perf_time_to_tsc,
+ &suite__dlfilter,
+ NULL,
};
-static struct test *tests[] = {
+static struct test_suite **tests[] = {
generic_tests,
arch_tests,
};
+static int num_subtests(const struct test_suite *t)
+{
+ int num;
+
+ if (!t->test_cases)
+ return 0;
+
+ num = 0;
+ while (t->test_cases[num].name)
+ num++;
+
+ return num;
+}
+
+static bool has_subtests(const struct test_suite *t)
+{
+ return num_subtests(t) > 1;
+}
+
+static const char *skip_reason(const struct test_suite *t, int subtest)
+{
+ if (t->test_cases && subtest >= 0)
+ return t->test_cases[subtest].skip_reason;
+
+ return NULL;
+}
+
+static const char *test_description(const struct test_suite *t, int subtest)
+{
+ if (t->test_cases && subtest >= 0)
+ return t->test_cases[subtest].desc;
+
+ return t->desc;
+}
+
+static test_fnptr test_function(const struct test_suite *t, int subtest)
+{
+ if (subtest <= 0)
+ return t->test_cases[0].run_case;
+
+ return t->test_cases[subtest].run_case;
+}
+
static bool perf_test__matches(const char *desc, int curr, int argc, const char *argv[])
{
int i;
return false;
}
-static int run_test(struct test *test, int subtest)
+static int run_test(struct test_suite *test, int subtest)
{
int status, err = -1, child = dont_fork ? 0 : fork();
char sbuf[STRERR_BUFSIZE];
}
}
- err = test->func(test, subtest);
+ err = test_function(test, subtest)(test, subtest);
if (!dont_fork)
exit(err);
}
return err;
}
-#define for_each_test(j, t) \
+#define for_each_test(j, k, t) \
for (j = 0; j < ARRAY_SIZE(tests); j++) \
- for (t = &tests[j][0]; t->func; t++)
+ for (k = 0, t = tests[j][k]; tests[j][k]; k++, t = tests[j][k])
-static int test_and_print(struct test *t, bool force_skip, int subtest)
+static int test_and_print(struct test_suite *t, int subtest)
{
int err;
- if (!force_skip) {
- pr_debug("\n--- start ---\n");
- err = run_test(t, subtest);
- pr_debug("---- end ----\n");
- } else {
- pr_debug("\n--- force skipped ---\n");
- err = TEST_SKIP;
- }
+ pr_debug("\n--- start ---\n");
+ err = run_test(t, subtest);
+ pr_debug("---- end ----\n");
- if (!t->subtest.get_nr)
+ if (!has_subtests(t))
pr_debug("%s:", t->desc);
else
pr_debug("%s subtest %d:", t->desc, subtest + 1);
pr_info(" Ok\n");
break;
case TEST_SKIP: {
- const char *skip_reason = NULL;
- if (t->subtest.skip_reason)
- skip_reason = t->subtest.skip_reason(subtest);
- if (skip_reason)
- color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (%s)\n", skip_reason);
+ const char *reason = skip_reason(t, subtest);
+
+ if (reason)
+ color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (%s)\n", reason);
else
color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip\n");
}
const char *file;
};
-static int shell_test__run(struct test *test, int subdir __maybe_unused)
+static int shell_test__run(struct test_suite *test, int subdir __maybe_unused)
{
int err;
char script[PATH_MAX];
for_each_shell_test(entlist, n_dirs, st.dir, ent) {
int curr = i++;
char desc[256];
- struct test test = {
- .desc = shell_test__description(desc, sizeof(desc), st.dir, ent->d_name),
- .func = shell_test__run,
+ struct test_case test_cases[] = {
+ {
+ .desc = shell_test__description(desc,
+ sizeof(desc),
+ st.dir,
+ ent->d_name),
+ .run_case = shell_test__run,
+ },
+ { .name = NULL, }
+ };
+ struct test_suite test_suite = {
+ .desc = test_cases[0].desc,
+ .test_cases = test_cases,
.priv = &st,
};
- if (!perf_test__matches(test.desc, curr, argc, argv))
+ if (!perf_test__matches(test_suite.desc, curr, argc, argv))
continue;
st.file = ent->d_name;
- pr_info("%2d: %-*s:", i, width, test.desc);
+ pr_info("%2d: %-*s:", i, width, test_suite.desc);
if (intlist__find(skiplist, i)) {
color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (user override)\n");
continue;
}
- test_and_print(&test, false, -1);
+ test_and_print(&test_suite, 0);
}
for (e = 0; e < n_dirs; e++)
static int __cmd_test(int argc, const char *argv[], struct intlist *skiplist)
{
- struct test *t;
- unsigned int j;
+ struct test_suite *t;
+ unsigned int j, k;
int i = 0;
int width = shell_tests__max_desc_width();
- for_each_test(j, t) {
- int len = strlen(t->desc);
+ for_each_test(j, k, t) {
+ int len = strlen(test_description(t, -1));
if (width < len)
width = len;
}
- for_each_test(j, t) {
- int curr = i++, err;
+ for_each_test(j, k, t) {
+ int curr = i++;
int subi;
- if (!perf_test__matches(t->desc, curr, argc, argv)) {
+ if (!perf_test__matches(test_description(t, -1), curr, argc, argv)) {
bool skip = true;
int subn;
- if (!t->subtest.get_nr)
- continue;
-
- subn = t->subtest.get_nr();
+ subn = num_subtests(t);
for (subi = 0; subi < subn; subi++) {
- if (perf_test__matches(t->subtest.get_desc(subi), curr, argc, argv))
+ if (perf_test__matches(test_description(t, subi),
+ curr, argc, argv))
skip = false;
}
continue;
}
- if (t->is_supported && !t->is_supported()) {
- pr_debug("%2d: %-*s: Disabled\n", i, width, t->desc);
- continue;
- }
-
- pr_info("%2d: %-*s:", i, width, t->desc);
+ pr_info("%2d: %-*s:", i, width, test_description(t, -1));
if (intlist__find(skiplist, i)) {
color_fprintf(stderr, PERF_COLOR_YELLOW, " Skip (user override)\n");
continue;
}
- if (!t->subtest.get_nr) {
- test_and_print(t, false, -1);
+ if (!has_subtests(t)) {
+ test_and_print(t, -1);
} else {
- int subn = t->subtest.get_nr();
+ int subn = num_subtests(t);
/*
* minus 2 to align with normal testcases.
* For subtest we print additional '.x' in number.
* 35.1: Basic BPF llvm compiling test : Ok
*/
int subw = width > 2 ? width - 2 : width;
- bool skip = false;
if (subn <= 0) {
color_fprintf(stderr, PERF_COLOR_YELLOW,
pr_info("\n");
for (subi = 0; subi < subn; subi++) {
- int len = strlen(t->subtest.get_desc(subi));
+ int len = strlen(test_description(t, subi));
if (subw < len)
subw = len;
}
for (subi = 0; subi < subn; subi++) {
- if (!perf_test__matches(t->subtest.get_desc(subi), curr, argc, argv))
+ if (!perf_test__matches(test_description(t, subi),
+ curr, argc, argv))
continue;
pr_info("%2d.%1d: %-*s:", i, subi + 1, subw,
- t->subtest.get_desc(subi));
- err = test_and_print(t, skip, subi);
- if (err != TEST_OK && t->subtest.skip_if_fail)
- skip = true;
+ test_description(t, subi));
+ test_and_print(t, subi);
}
}
}
for_each_shell_test(entlist, n_dirs, path, ent) {
int curr = i++;
char bf[256];
- struct test t = {
+ struct test_suite t = {
.desc = shell_test__description(bf, sizeof(bf), path, ent->d_name),
};
static int perf_test__list(int argc, const char **argv)
{
- unsigned int j;
- struct test *t;
+ unsigned int j, k;
+ struct test_suite *t;
int i = 0;
- for_each_test(j, t) {
+ for_each_test(j, k, t) {
int curr = i++;
- if (!perf_test__matches(t->desc, curr, argc, argv) ||
- (t->is_supported && !t->is_supported()))
+ if (!perf_test__matches(test_description(t, -1), curr, argc, argv))
continue;
- pr_info("%2d: %s\n", i, t->desc);
+ pr_info("%2d: %s\n", i, test_description(t, -1));
- if (t->subtest.get_nr) {
- int subn = t->subtest.get_nr();
+ if (has_subtests(t)) {
+ int subn = num_subtests(t);
int subi;
for (subi = 0; subi < subn; subi++)
pr_info("%2d:%1d: %s\n", i, subi + 1,
- t->subtest.get_desc(subi));
+ test_description(t, subi));
}
}
#include "c++/clang-c.h"
#include <linux/kernel.h>
-static struct {
- int (*func)(void);
- const char *desc;
-} clang_testcase_table[] = {
-#ifdef HAVE_LIBCLANGLLVM_SUPPORT
- {
- .func = test__clang_to_IR,
- .desc = "builtin clang compile C source to IR",
- },
- {
- .func = test__clang_to_obj,
- .desc = "builtin clang compile C source to ELF object",
- },
-#endif
-};
-
-int test__clang_subtest_get_nr(void)
-{
- return (int)ARRAY_SIZE(clang_testcase_table);
-}
-
-const char *test__clang_subtest_get_desc(int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(clang_testcase_table))
- return NULL;
- return clang_testcase_table[i].desc;
-}
-
#ifndef HAVE_LIBCLANGLLVM_SUPPORT
-int test__clang(struct test *test __maybe_unused, int i __maybe_unused)
+static int test__clang_to_IR(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
return TEST_SKIP;
}
-#else
-int test__clang(struct test *test __maybe_unused, int i)
+
+static int test__clang_to_obj(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- if (i < 0 || i >= (int)ARRAY_SIZE(clang_testcase_table))
- return TEST_FAIL;
- return clang_testcase_table[i].func();
+ return TEST_SKIP;
}
#endif
+
+static struct test_case clang_tests[] = {
+ TEST_CASE_REASON("builtin clang compile C source to IR", clang_to_IR,
+ "not compiled in"),
+ TEST_CASE_REASON("builtin clang compile C source to ELF object",
+ clang_to_obj,
+ "not compiled in"),
+ { .name = NULL, }
+};
+
+struct test_suite suite__clang = {
+ .desc = "builtin clang support",
+ .test_cases = clang_tests,
+};
}
ret = perf_event__synthesize_thread_map(NULL, threads,
- perf_event__process, machine, false);
+ perf_event__process, machine,
+ true, false);
if (ret < 0) {
pr_debug("perf_event__synthesize_thread_map failed\n");
goto out_err;
return err;
}
-int test__code_reading(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__code_reading(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret;
return -1;
};
}
+
+DEFINE_SUITE("Object code reading", code_reading);
}
-int test__cpu_map_synthesize(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__cpu_map_synthesize(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_cpu_map *cpus;
return !strcmp(buf, str);
}
-int test__cpu_map_print(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__cpu_map_print(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
TEST_ASSERT_VAL("failed to convert map", cpu_map_print("1"));
TEST_ASSERT_VAL("failed to convert map", cpu_map_print("1,5"));
return 0;
}
-int test__cpu_map_merge(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__cpu_map_merge(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_cpu_map *a = perf_cpu_map__new("4,2,1");
struct perf_cpu_map *b = perf_cpu_map__new("4,5,7");
perf_cpu_map__put(c);
return 0;
}
+
+DEFINE_SUITE("Synthesize cpu map", cpu_map_synthesize);
+DEFINE_SUITE("Print cpu map", cpu_map_print);
+DEFINE_SUITE("Merge cpu map", cpu_map_merge);
#include "debug.h"
#include "demangle-java.h"
-int test__demangle_java(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__demangle_java(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret = TEST_OK;
char *buf = NULL;
return ret;
}
+
+DEFINE_SUITE("Demangle Java", demangle_java);
#include "debug.h"
#include "demangle-ocaml.h"
-int test__demangle_ocaml(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__demangle_ocaml(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret = TEST_OK;
char *buf = NULL;
return ret;
}
+
+DEFINE_SUITE("Demangle OCaml", demangle_ocaml);
}
}
-int test__dlfilter(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__dlfilter(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct test_data td = {.fd = -1};
int pid = getpid();
test_data__free(&td);
return err;
}
+
+DEFINE_SUITE("dlfilter C API", dlfilter);
return fd;
}
-int test__dso_data(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__dso_data(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct machine machine;
struct dso *dso;
return setrlimit(RLIMIT_NOFILE, &rlim);
}
-int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__dso_data_cache(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct machine machine;
long nr_end, nr = open_files_cnt();
return ret;
}
-int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__dso_data_reopen(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct machine machine;
long nr_end, nr = open_files_cnt(), lim = new_limit(3);
TEST_ASSERT_VAL("failed leaking files", nr == nr_end);
return 0;
}
+
+DEFINE_SUITE("DSO data read", dso_data);
+DEFINE_SUITE("DSO data cache", dso_data_cache);
+DEFINE_SUITE("DSO data reopen", dso_data_reopen);
return ret;
}
-int test__dwarf_unwind(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__dwarf_unwind(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct machine *machine;
struct thread *thread;
machine__delete(machine);
return err;
}
+
+DEFINE_SUITE("Test dwarf unwind", dwarf_unwind);
* and checks that enabled and running times
* match.
*/
-int test__event_times(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__event_times(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err, ret = 0;
#undef _T
return ret;
}
+
+DEFINE_SUITE("Event times", event_times);
return 0;
}
-int test__event_update(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__event_update(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct evsel *evsel;
struct event_name tmp;
struct evlist *evlist = evlist__new_default();
- char *unit = strdup("KRAVA");
TEST_ASSERT_VAL("failed to get evlist", evlist);
perf_evlist__id_add(&evlist->core, &evsel->core, 0, 0, 123);
- evsel->unit = unit;
+ free((char *)evsel->unit);
+ evsel->unit = strdup("KRAVA");
TEST_ASSERT_VAL("failed to synthesize attr update unit",
!perf_event__synthesize_event_update_unit(NULL, evsel, process_event_unit));
TEST_ASSERT_VAL("failed to synthesize attr update cpus",
!perf_event__synthesize_event_update_cpus(&tmp.tool, evsel, process_event_cpus));
- free(unit);
evlist__delete(evlist);
return 0;
}
+
+DEFINE_SUITE("Synthesize attr update", event_update);
#define perf_evsel__name_array_test(names, distance) \
__perf_evsel__name_array_test(names, ARRAY_SIZE(names), distance)
-int test__perf_evsel__roundtrip_name_test(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__perf_evsel__roundtrip_name_test(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int err = 0, ret = 0;
return ret;
}
+
+DEFINE_SUITE("Roundtrip evsel->name", perf_evsel__roundtrip_name_test);
return ret;
}
-int test__perf_evsel__tp_sched_test(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__perf_evsel__tp_sched_test(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct evsel *evsel = evsel__newtp("sched", "sched_switch");
int ret = 0;
evsel__delete(evsel);
return ret;
}
+
+DEFINE_SUITE("Parse sched tracepoints fields", perf_evsel__tp_sched_test);
evlist = evlist__new();
TEST_ASSERT_VAL("failed to get evlist", evlist);
+ parse_events_error__init(&err);
ret = parse_events(evlist, event_str, &err);
if (ret < 0) {
pr_debug("failed to parse event '%s', err %d, str '%s'\n",
event_str, ret, err.str);
- parse_events_print_error(&err, event_str);
+ parse_events_error__print(&err, event_str);
goto out;
}
rblist__init(&metric_events);
ret = test_expand_events(evlist, &metric_events);
out:
+ parse_events_error__exit(&err);
evlist__delete(evlist);
return ret;
}
.metric_name = NULL,
},
};
- struct pmu_events_map ev_map = {
+ const struct pmu_events_map ev_map = {
.cpuid = "test",
.version = "1",
.type = "core",
return ret;
}
-int test__expand_cgroup_events(struct test *test __maybe_unused,
- int subtest __maybe_unused)
+static int test__expand_cgroup_events(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int ret;
return ret;
}
+
+DEFINE_SUITE("Event expansion for cgroups", expand_cgroup_events);
// SPDX-License-Identifier: GPL-2.0
#include "util/debug.h"
#include "util/expr.h"
+#include "util/smt.h"
#include "tests.h"
#include <stdlib.h>
#include <string.h>
#include <linux/zalloc.h>
+static int test_ids_union(void)
+{
+ struct hashmap *ids1, *ids2;
+
+ /* Empty union. */
+ ids1 = ids__new();
+ TEST_ASSERT_VAL("ids__new", ids1);
+ ids2 = ids__new();
+ TEST_ASSERT_VAL("ids__new", ids2);
+
+ ids1 = ids__union(ids1, ids2);
+ TEST_ASSERT_EQUAL("union", (int)hashmap__size(ids1), 0);
+
+ /* Union {foo, bar} against {}. */
+ ids2 = ids__new();
+ TEST_ASSERT_VAL("ids__new", ids2);
+
+ TEST_ASSERT_EQUAL("ids__insert", ids__insert(ids1, strdup("foo")), 0);
+ TEST_ASSERT_EQUAL("ids__insert", ids__insert(ids1, strdup("bar")), 0);
+
+ ids1 = ids__union(ids1, ids2);
+ TEST_ASSERT_EQUAL("union", (int)hashmap__size(ids1), 2);
+
+ /* Union {foo, bar} against {foo}. */
+ ids2 = ids__new();
+ TEST_ASSERT_VAL("ids__new", ids2);
+ TEST_ASSERT_EQUAL("ids__insert", ids__insert(ids2, strdup("foo")), 0);
+
+ ids1 = ids__union(ids1, ids2);
+ TEST_ASSERT_EQUAL("union", (int)hashmap__size(ids1), 2);
+
+ /* Union {foo, bar} against {bar,baz}. */
+ ids2 = ids__new();
+ TEST_ASSERT_VAL("ids__new", ids2);
+ TEST_ASSERT_EQUAL("ids__insert", ids__insert(ids2, strdup("bar")), 0);
+ TEST_ASSERT_EQUAL("ids__insert", ids__insert(ids2, strdup("baz")), 0);
+
+ ids1 = ids__union(ids1, ids2);
+ TEST_ASSERT_EQUAL("union", (int)hashmap__size(ids1), 3);
+
+ ids__free(ids1);
+
+ return 0;
+}
+
static int test(struct expr_parse_ctx *ctx, const char *e, double val2)
{
double val;
- if (expr__parse(&val, ctx, e, 1))
+ if (expr__parse(&val, ctx, e))
TEST_ASSERT_VAL("parse test failed", 0);
TEST_ASSERT_VAL("unexpected value", val == val2);
return 0;
}
-int test__expr(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__expr(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
struct expr_id_data *val_ptr;
const char *p;
- double val;
+ double val, num_cpus, num_cores, num_dies, num_packages;
int ret;
- struct expr_parse_ctx ctx;
-
- expr__ctx_init(&ctx);
- expr__add_id_val(&ctx, strdup("FOO"), 1);
- expr__add_id_val(&ctx, strdup("BAR"), 2);
-
- ret = test(&ctx, "1+1", 2);
- ret |= test(&ctx, "FOO+BAR", 3);
- ret |= test(&ctx, "(BAR/2)%2", 1);
- ret |= test(&ctx, "1 - -4", 5);
- ret |= test(&ctx, "(FOO-1)*2 + (BAR/2)%2 - -4", 5);
- ret |= test(&ctx, "1-1 | 1", 1);
- ret |= test(&ctx, "1-1 & 1", 0);
- ret |= test(&ctx, "min(1,2) + 1", 2);
- ret |= test(&ctx, "max(1,2) + 1", 3);
- ret |= test(&ctx, "1+1 if 3*4 else 0", 2);
- ret |= test(&ctx, "1.1 + 2.1", 3.2);
- ret |= test(&ctx, ".1 + 2.", 2.1);
- ret |= test(&ctx, "d_ratio(1, 2)", 0.5);
- ret |= test(&ctx, "d_ratio(2.5, 0)", 0);
- ret |= test(&ctx, "1.1 < 2.2", 1);
- ret |= test(&ctx, "2.2 > 1.1", 1);
- ret |= test(&ctx, "1.1 < 1.1", 0);
- ret |= test(&ctx, "2.2 > 2.2", 0);
- ret |= test(&ctx, "2.2 < 1.1", 0);
- ret |= test(&ctx, "1.1 > 2.2", 0);
-
- if (ret)
+ struct expr_parse_ctx *ctx;
+
+ TEST_ASSERT_EQUAL("ids_union", test_ids_union(), 0);
+
+ ctx = expr__ctx_new();
+ TEST_ASSERT_VAL("expr__ctx_new", ctx);
+ expr__add_id_val(ctx, strdup("FOO"), 1);
+ expr__add_id_val(ctx, strdup("BAR"), 2);
+
+ ret = test(ctx, "1+1", 2);
+ ret |= test(ctx, "FOO+BAR", 3);
+ ret |= test(ctx, "(BAR/2)%2", 1);
+ ret |= test(ctx, "1 - -4", 5);
+ ret |= test(ctx, "(FOO-1)*2 + (BAR/2)%2 - -4", 5);
+ ret |= test(ctx, "1-1 | 1", 1);
+ ret |= test(ctx, "1-1 & 1", 0);
+ ret |= test(ctx, "min(1,2) + 1", 2);
+ ret |= test(ctx, "max(1,2) + 1", 3);
+ ret |= test(ctx, "1+1 if 3*4 else 0", 2);
+ ret |= test(ctx, "1.1 + 2.1", 3.2);
+ ret |= test(ctx, ".1 + 2.", 2.1);
+ ret |= test(ctx, "d_ratio(1, 2)", 0.5);
+ ret |= test(ctx, "d_ratio(2.5, 0)", 0);
+ ret |= test(ctx, "1.1 < 2.2", 1);
+ ret |= test(ctx, "2.2 > 1.1", 1);
+ ret |= test(ctx, "1.1 < 1.1", 0);
+ ret |= test(ctx, "2.2 > 2.2", 0);
+ ret |= test(ctx, "2.2 < 1.1", 0);
+ ret |= test(ctx, "1.1 > 2.2", 0);
+
+ if (ret) {
+ expr__ctx_free(ctx);
return ret;
+ }
p = "FOO/0";
- ret = expr__parse(&val, &ctx, p, 1);
+ ret = expr__parse(&val, ctx, p);
TEST_ASSERT_VAL("division by zero", ret == -1);
p = "BAR/";
- ret = expr__parse(&val, &ctx, p, 1);
+ ret = expr__parse(&val, ctx, p);
TEST_ASSERT_VAL("missing operand", ret == -1);
- expr__ctx_clear(&ctx);
- TEST_ASSERT_VAL("find other",
- expr__find_other("FOO + BAR + BAZ + BOZO", "FOO",
- &ctx, 1) == 0);
- TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 3);
- TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAR",
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("find ids",
+ expr__find_ids("FOO + BAR + BAZ + BOZO", "FOO",
+ ctx) == 0);
+ TEST_ASSERT_VAL("find ids", hashmap__size(ctx->ids) == 3);
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "BAR",
+ (void **)&val_ptr));
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "BAZ",
(void **)&val_ptr));
- TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BAZ",
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "BOZO",
(void **)&val_ptr));
- TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "BOZO",
+
+ expr__ctx_clear(ctx);
+ ctx->runtime = 3;
+ TEST_ASSERT_VAL("find ids",
+ expr__find_ids("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@",
+ NULL, ctx) == 0);
+ TEST_ASSERT_VAL("find ids", hashmap__size(ctx->ids) == 2);
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "EVENT1,param=3@",
+ (void **)&val_ptr));
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "EVENT2,param=3@",
(void **)&val_ptr));
- expr__ctx_clear(&ctx);
- TEST_ASSERT_VAL("find other",
- expr__find_other("EVENT1\\,param\\=?@ + EVENT2\\,param\\=?@",
- NULL, &ctx, 3) == 0);
- TEST_ASSERT_VAL("find other", hashmap__size(&ctx.ids) == 2);
- TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT1,param=3/",
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("find ids",
+ expr__find_ids("dash\\-event1 - dash\\-event2",
+ NULL, ctx) == 0);
+ TEST_ASSERT_VAL("find ids", hashmap__size(ctx->ids) == 2);
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "dash-event1",
(void **)&val_ptr));
- TEST_ASSERT_VAL("find other", hashmap__find(&ctx.ids, "EVENT2,param=3/",
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids, "dash-event2",
(void **)&val_ptr));
- expr__ctx_clear(&ctx);
+ /* Only EVENT1 or EVENT2 need be measured depending on the value of smt_on. */
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("find ids",
+ expr__find_ids("EVENT1 if #smt_on else EVENT2",
+ NULL, ctx) == 0);
+ TEST_ASSERT_VAL("find ids", hashmap__size(ctx->ids) == 1);
+ TEST_ASSERT_VAL("find ids", hashmap__find(ctx->ids,
+ smt_on() ? "EVENT1" : "EVENT2",
+ (void **)&val_ptr));
+
+ /* The expression is a constant 1.0 without needing to evaluate EVENT1. */
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("find ids",
+ expr__find_ids("1.0 if EVENT1 > 100.0 else 1.0",
+ NULL, ctx) == 0);
+ TEST_ASSERT_VAL("find ids", hashmap__size(ctx->ids) == 0);
+
+ /* Test toplogy constants appear well ordered. */
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("#num_cpus", expr__parse(&num_cpus, ctx, "#num_cpus") == 0);
+ TEST_ASSERT_VAL("#num_cores", expr__parse(&num_cores, ctx, "#num_cores") == 0);
+ TEST_ASSERT_VAL("#num_cpus >= #num_cores", num_cpus >= num_cores);
+ TEST_ASSERT_VAL("#num_dies", expr__parse(&num_dies, ctx, "#num_dies") == 0);
+ TEST_ASSERT_VAL("#num_cores >= #num_dies", num_cores >= num_dies);
+ TEST_ASSERT_VAL("#num_packages", expr__parse(&num_packages, ctx, "#num_packages") == 0);
+ TEST_ASSERT_VAL("#num_dies >= #num_packages", num_dies >= num_packages);
+
+ /*
+ * Source count returns the number of events aggregating in a leader
+ * event including the leader. Check parsing yields an id.
+ */
+ expr__ctx_clear(ctx);
+ TEST_ASSERT_VAL("source count",
+ expr__find_ids("source_count(EVENT1)",
+ NULL, ctx) == 0);
+ TEST_ASSERT_VAL("source count", hashmap__size(ctx->ids) == 1);
+ TEST_ASSERT_VAL("source count", hashmap__find(ctx->ids, "EVENT1",
+ (void **)&val_ptr));
+
+ expr__ctx_free(ctx);
return 0;
}
+
+DEFINE_SUITE("Simple expression parser", expr);
return printed + fdarray__fprintf(fda, fp);
}
-int test__fdarray__filter(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__fdarray__filter(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int nr_fds, err = TEST_FAIL;
struct fdarray *fda = fdarray__new(5, 5);
return err;
}
-int test__fdarray__add(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__fdarray__add(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = TEST_FAIL;
struct fdarray *fda = fdarray__new(2, 2);
out:
return err;
}
+
+DEFINE_SUITE("Filter fds with revents mask in a fdarray", fdarray__filter);
+DEFINE_SUITE("Add fd to a fdarray, making it autogrow", fdarray__add);
#define TEMPL "/tmp/perf-test-XXXXXX"
-int test__jit_write_elf(struct test *test __maybe_unused,
- int subtest __maybe_unused)
+static int test__jit_write_elf(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
#ifdef HAVE_JITDUMP
static unsigned char x86_code[] = {
return TEST_SKIP;
#endif
}
+
+DEFINE_SUITE("Test jit_write_elf", jit_write_elf);
return err;
}
-int test__hists_cumulate(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__hists_cumulate(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = TEST_FAIL;
struct machines machines;
return err;
}
+
+DEFINE_SUITE("Cumulate child hist entries", hists_cumulate);
return TEST_FAIL;
}
-int test__hists_filter(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__hists_filter(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = TEST_FAIL;
struct machines machines;
return err;
}
+
+DEFINE_SUITE("Filter hist entries", hists_filter);
return __validate_link(leader, 0) || __validate_link(other, 1);
}
-int test__hists_link(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__hists_link(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = -1;
struct hists *hists, *first_hists;
return err;
}
+
+DEFINE_SUITE("Match and link multiple hists", hists_link);
return err;
}
-int test__hists_output(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__hists_output(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = TEST_FAIL;
struct machines machines;
return err;
}
+
+DEFINE_SUITE("Sort output of hist entries", hists_output);
#include "debug.h"
#include "print_binary.h"
-int test__is_printable_array(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__is_printable_array(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
char buf1[] = { 'k', 'r', 4, 'v', 'a', 0 };
char buf2[] = { 'k', 'r', 'a', 'v', 4, 0 };
return TEST_OK;
}
+
+DEFINE_SUITE("is_printable_array", is_printable_array);
* when an event is disabled but a dummy software event is not disabled. If the
* test passes %0 is returned, otherwise %-1 is returned.
*/
-int test__keep_tracking(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__keep_tracking(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct record_opts opts = {
.mmap_pages = UINT_MAX,
return err;
}
+
+DEFINE_SUITE("Use a dummy software event to keep tracking", keep_tracking);
#define M(path, c, e) \
TEST_ASSERT_VAL("failed", !test_is_kernel_module(path, c, e))
-int test__kmod_path__parse(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__kmod_path__parse(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
/* path alloc_name kmod comp name */
T("/xxxx/xxxx/x-x.ko", true , true, 0 , "[x_x]");
return 0;
}
+
+DEFINE_SUITE("kmod_path__parse", kmod_path__parse);
return ret;
}
-int test__llvm(struct test *test __maybe_unused, int subtest)
+static int test__llvm(int subtest)
{
int ret;
void *obj_buf = NULL;
return ret;
}
+#endif //HAVE_LIBBPF_SUPPORT
-int test__llvm_subtest_get_nr(void)
+static int test__llvm__bpf_base_prog(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return __LLVM_TESTCASE_MAX;
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__llvm(LLVM_TESTCASE_BASE);
+#else
+ pr_debug("Skip LLVM test because BPF support is not compiled\n");
+ return TEST_SKIP;
+#endif
}
-const char *test__llvm_subtest_get_desc(int subtest)
-{
- if ((subtest < 0) || (subtest >= __LLVM_TESTCASE_MAX))
- return NULL;
-
- return bpf_source_table[subtest].desc;
-}
-#else //HAVE_LIBBPF_SUPPORT
-int test__llvm(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__llvm__bpf_test_kbuild_prog(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__llvm(LLVM_TESTCASE_KBUILD);
+#else
+ pr_debug("Skip LLVM test because BPF support is not compiled\n");
return TEST_SKIP;
+#endif
}
-int test__llvm_subtest_get_nr(void)
+static int test__llvm__bpf_test_prologue_prog(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return 0;
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__llvm(LLVM_TESTCASE_BPF_PROLOGUE);
+#else
+ pr_debug("Skip LLVM test because BPF support is not compiled\n");
+ return TEST_SKIP;
+#endif
}
-const char *test__llvm_subtest_get_desc(int subtest __maybe_unused)
+static int test__llvm__bpf_test_relocation(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return NULL;
+#ifdef HAVE_LIBBPF_SUPPORT
+ return test__llvm(LLVM_TESTCASE_BPF_RELOCATION);
+#else
+ pr_debug("Skip LLVM test because BPF support is not compiled\n");
+ return TEST_SKIP;
+#endif
}
-#endif // HAVE_LIBBPF_SUPPORT
+
+
+static struct test_case llvm_tests[] = {
+#ifdef HAVE_LIBBPF_SUPPORT
+ TEST_CASE("Basic BPF llvm compile", llvm__bpf_base_prog),
+ TEST_CASE("kbuild searching", llvm__bpf_test_kbuild_prog),
+ TEST_CASE("Compile source for BPF prologue generation",
+ llvm__bpf_test_prologue_prog),
+ TEST_CASE("Compile source for BPF relocation", llvm__bpf_test_relocation),
+#else
+ TEST_CASE_REASON("Basic BPF llvm compile", llvm__bpf_base_prog, "not compiled in"),
+ TEST_CASE_REASON("kbuild searching", llvm__bpf_test_kbuild_prog, "not compiled in"),
+ TEST_CASE_REASON("Compile source for BPF prologue generation",
+ llvm__bpf_test_prologue_prog, "not compiled in"),
+ TEST_CASE_REASON("Compile source for BPF relocation",
+ llvm__bpf_test_relocation, "not compiled in"),
+#endif
+ { .name = NULL, }
+};
+
+struct test_suite suite__llvm = {
+ .desc = "LLVM search and compile",
+ .test_cases = llvm_tests,
+};
return TEST_OK;
}
-int test__maps__merge_in(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__maps__merge_in(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
struct maps maps;
unsigned int i;
maps__exit(&maps);
return TEST_OK;
}
+
+DEFINE_SUITE("maps__merge_in", maps__merge_in);
return 0;
}
-int test__mem(struct test *text __maybe_unused, int subtest __maybe_unused)
+static int test__mem(struct test_suite *text __maybe_unused, int subtest __maybe_unused)
{
int ret = 0;
union perf_mem_data_src src;
return ret;
}
+
+DEFINE_SUITE("Test data source output", mem);
return bm && map ? bm : NULL;
}
-int test__mem2node(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__mem2node(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
struct mem2node map;
struct memory_node nodes[3];
mem2node__exit(&map);
return 0;
}
+
+DEFINE_SUITE("mem2node", mem2node);
* Then it checks if the number of syscalls reported as perf events by
* the kernel corresponds to the number of syscalls made.
*/
-int test__basic_mmap(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__basic_mmap(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = -1;
union perf_event *event;
perf_thread_map__put(threads);
return err;
}
+
+DEFINE_SUITE("Read samples using the mmap interface", basic_mmap);
{
return perf_event__synthesize_threads(NULL,
perf_event__process,
- machine, 0, 1);
+ machine, 1, 0, 1);
}
static int synth_process(struct machine *machine)
err = perf_event__synthesize_thread_map(NULL, map,
perf_event__process,
- machine, 0);
+ machine, 1, 0);
perf_thread_map__put(map);
return err;
*
* by using all thread objects.
*/
-int test__mmap_thread_lookup(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__mmap_thread_lookup(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
/* perf_event__synthesize_threads synthesize */
TEST_ASSERT_VAL("failed with sythesizing all",
return 0;
}
+
+DEFINE_SUITE("Lookup mmap thread", mmap_thread_lookup);
#include "stat.h"
#include "util/counts.h"
-int test__openat_syscall_event_on_all_cpus(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__openat_syscall_event_on_all_cpus(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int err = -1, fd, cpu;
struct perf_cpu_map *cpus;
perf_thread_map__put(threads);
return err;
}
+
+DEFINE_SUITE("Detect openat syscall event on all cpus", openat_syscall_event_on_all_cpus);
#define AT_FDCWD -100
#endif
-int test__syscall_openat_tp_fields(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__syscall_openat_tp_fields(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct record_opts opts = {
.target = {
out:
return err;
}
+
+DEFINE_SUITE("syscalls:sys_enter_openat event fields", syscall_openat_tp_fields);
#include "tests.h"
#include "util/counts.h"
-int test__openat_syscall_event(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__openat_syscall_event(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int err = -1, fd;
struct evsel *evsel;
perf_thread_map__put(threads);
return err;
}
+
+DEFINE_SUITE("Detect openat syscall event", openat_syscall_event);
TEST_ASSERT_VAL("wrong type val",
term->type_val == PARSE_EVENTS__TERM_TYPE_NUM);
TEST_ASSERT_VAL("wrong val", term->val.num == 10);
- TEST_ASSERT_VAL("wrong config", !term->config);
+ TEST_ASSERT_VAL("wrong config", !strcmp(term->config, "config"));
/* config1 */
term = list_entry(term->list.next, struct parse_events_term, list);
TEST_ASSERT_VAL("wrong type val",
term->type_val == PARSE_EVENTS__TERM_TYPE_NUM);
TEST_ASSERT_VAL("wrong val", term->val.num == 1);
- TEST_ASSERT_VAL("wrong config", !term->config);
+ TEST_ASSERT_VAL("wrong config", !strcmp(term->config, "config1"));
/* config2=3 */
term = list_entry(term->list.next, struct parse_events_term, list);
TEST_ASSERT_VAL("wrong type val",
term->type_val == PARSE_EVENTS__TERM_TYPE_NUM);
TEST_ASSERT_VAL("wrong val", term->val.num == 3);
- TEST_ASSERT_VAL("wrong config", !term->config);
+ TEST_ASSERT_VAL("wrong config", !strcmp(term->config, "config2"));
/* umask=1*/
term = list_entry(term->list.next, struct parse_events_term, list);
TEST_ASSERT_VAL("wrong type val",
term->type_val == PARSE_EVENTS__TERM_TYPE_NUM);
TEST_ASSERT_VAL("wrong val", term->val.num == 0xead);
- TEST_ASSERT_VAL("wrong config", !term->config);
+ TEST_ASSERT_VAL("wrong config", !strcmp(term->config, "config"));
return 0;
}
struct evlist *evlist;
int ret;
- bzero(&err, sizeof(err));
if (e->valid && !e->valid()) {
pr_debug("... SKIP");
return 0;
if (evlist == NULL)
return -ENOMEM;
+ parse_events_error__init(&err);
ret = parse_events(evlist, e->name, &err);
if (ret) {
pr_debug("failed to parse event '%s', err %d, str '%s'\n",
e->name, ret, err.str);
- parse_events_print_error(&err, e->name);
+ parse_events_error__print(&err, e->name);
} else {
ret = e->check(evlist);
}
-
+ parse_events_error__exit(&err);
evlist__delete(evlist);
return ret;
return test_event(&e);
}
-int test__parse_events(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__parse_events(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret1, ret2 = 0;
char *event, *alias;
return ret2;
}
+
+DEFINE_SUITE("Parse event definition strings", parse_events);
}
};
-static struct pmu_events_map map = {
+static const struct pmu_events_map map = {
.cpuid = "test",
.version = "1",
.type = "core",
return 0;
}
-int test__parse_metric(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__parse_metric(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
TEST_ASSERT_VAL("IPC failed", test_ipc() == 0);
TEST_ASSERT_VAL("frontend failed", test_frontend() == 0);
}
return 0;
}
+
+DEFINE_SUITE("Parse and process metrics", parse_metric);
*
* Return: %0 on success, %-1 if the test fails.
*/
-int test__parse_no_sample_id_all(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__parse_no_sample_id_all(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int err;
return 0;
}
+
+DEFINE_SUITE("Parse with no sample_id_all bit set", parse_no_sample_id_all);
return TEST_OK;
}
-int test__pe_file_parsing(struct test *test __maybe_unused,
+static int test__pe_file_parsing(struct test_suite *test __maybe_unused,
int subtest __maybe_unused)
{
struct stat st;
#else
-int test__pe_file_parsing(struct test *test __maybe_unused,
+static int test__pe_file_parsing(struct test_suite *test __maybe_unused,
int subtest __maybe_unused)
{
return TEST_SKIP;
}
#endif
+
+DEFINE_SUITE("PE file support", pe_file_parsing);
raise(SIGSEGV);
}
-int test__perf_hooks(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__perf_hooks(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int hook_flags = 0;
return TEST_FAIL;
return TEST_OK;
}
+
+DEFINE_SUITE("perf hooks", perf_hooks);
return cpu;
}
-int test__PERF_RECORD(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__PERF_RECORD(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct record_opts opts = {
.target = {
out:
return (err < 0 || errs > 0) ? -1 : 0;
}
+
+DEFINE_SUITE("PERF_RECORD_* events & perf_sample fields", PERF_RECORD);
#include "pmu.h"
#include "pmu-hybrid.h"
+/*
+ * Except x86_64/i386 and Arm64, other archs don't support TSC in perf. Just
+ * enable the test for x86_64/i386 and Arm64 archs.
+ */
+#if defined(__x86_64__) || defined(__i386__) || defined(__aarch64__)
+#define TSC_IS_SUPPORTED 1
+#else
+#define TSC_IS_SUPPORTED 0
+#endif
+
#define CHECK__(x) { \
while ((x) < 0) { \
pr_debug(#x " failed!\n"); \
* %0 is returned, otherwise %-1 is returned. If TSC conversion is not
* supported then then the test passes but " (not supported)" is printed.
*/
-int test__perf_time_to_tsc(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__perf_time_to_tsc(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct record_opts opts = {
.mmap_pages = UINT_MAX,
u64 test_time, comm1_time = 0, comm2_time = 0;
struct mmap *md;
+ if (!TSC_IS_SUPPORTED) {
+ pr_debug("Test not supported on this architecture");
+ return TEST_SKIP;
+ }
+
threads = thread_map__new(-1, getpid(), UINT_MAX);
CHECK_NOT_NULL__(threads);
return err;
}
-bool test__tsc_is_supported(void)
-{
- /*
- * Except x86_64/i386 and Arm64, other archs don't support TSC in perf.
- * Just enable the test for x86_64/i386 and Arm64 archs.
- */
-#if defined(__x86_64__) || defined(__i386__) || defined(__aarch64__)
- return true;
-#else
- return false;
-#endif
-}
+DEFINE_SUITE("Convert perf time to TSC", perf_time_to_tsc);
#include <linux/kernel.h>
#ifdef HAVE_LIBPFM
-static int test__pfm_events(void);
-static int test__pfm_group(void);
-#endif
-
-static const struct {
- int (*func)(void);
- const char *desc;
-} pfm_testcase_table[] = {
-#ifdef HAVE_LIBPFM
- {
- .func = test__pfm_events,
- .desc = "test of individual --pfm-events",
- },
- {
- .func = test__pfm_group,
- .desc = "test groups of --pfm-events",
- },
-#endif
-};
-
-#ifdef HAVE_LIBPFM
static int count_pfm_events(struct perf_evlist *evlist)
{
struct perf_evsel *evsel;
return count;
}
-static int test__pfm_events(void)
+static int test__pfm_events(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct evlist *evlist;
struct option opt;
return 0;
}
-static int test__pfm_group(void)
+static int test__pfm_group(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct evlist *evlist;
struct option opt;
}
return 0;
}
-#endif
-
-const char *test__pfm_subtest_get_desc(int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(pfm_testcase_table))
- return NULL;
- return pfm_testcase_table[i].desc;
-}
-
-int test__pfm_subtest_get_nr(void)
+#else
+static int test__pfm_events(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- return (int)ARRAY_SIZE(pfm_testcase_table);
+ return TEST_SKIP;
}
-int test__pfm(struct test *test __maybe_unused, int i __maybe_unused)
+static int test__pfm_group(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
-#ifdef HAVE_LIBPFM
- if (i < 0 || i >= (int)ARRAY_SIZE(pfm_testcase_table))
- return TEST_FAIL;
- return pfm_testcase_table[i].func();
-#else
return TEST_SKIP;
-#endif
}
+#endif
+
+static struct test_case pfm_tests[] = {
+ TEST_CASE_REASON("test of individual --pfm-events", pfm_events, "not compiled in"),
+ TEST_CASE_REASON("test groups of --pfm-events", pfm_group, "not compiled in"),
+ { .name = NULL, }
+};
+
+struct test_suite suite__pfm = {
+ .desc = "Test libpfm4 support",
+ .test_cases = pfm_tests,
+};
.desc = "Number of segment register loads",
.topic = "other",
},
- .alias_str = "umask=0x80,(null)=0x30d40,event=0x6",
+ .alias_str = "umask=0x80,period=0x30d40,event=0x6",
.alias_long_desc = "Number of segment register loads",
};
.desc = "Memory cluster signals to block micro-op dispatch for any reason",
.topic = "other",
},
- .alias_str = "umask=0x20,(null)=0x30d40,event=0x9",
+ .alias_str = "umask=0x20,period=0x30d40,event=0x9",
.alias_long_desc = "Memory cluster signals to block micro-op dispatch for any reason",
};
.desc = "Number of Enhanced Intel SpeedStep(R) Technology (EIST) transitions",
.topic = "other",
},
- .alias_str = "umask=0,(null)=0x30d40,event=0x3a",
+ .alias_str = "umask=0,period=0x30d40,event=0x3a",
.alias_long_desc = "Number of Enhanced Intel SpeedStep(R) Technology (EIST) transitions",
};
static const struct perf_pmu_test_event uncore_hisi_l3c_rd_hit_cpipe = {
.event = {
.name = "uncore_hisi_l3c.rd_hit_cpipe",
- .event = "event=0x2",
+ .event = "event=0x7",
.desc = "Total read hits. Unit: hisi_sccl,l3c ",
.topic = "uncore",
.long_desc = "Total read hits",
.matching_pmu = "uncore_sys_ddr_pmu",
};
+static const struct perf_pmu_test_event sys_ccn_pmu_read_cycles = {
+ .event = {
+ .name = "sys_ccn_pmu.read_cycles",
+ .event = "config=0x2c",
+ .desc = "ccn read-cycles event. Unit: uncore_sys_ccn_pmu ",
+ .topic = "uncore",
+ .pmu = "uncore_sys_ccn_pmu",
+ .compat = "0x01",
+ },
+ .alias_str = "config=0x2c",
+ .alias_long_desc = "ccn read-cycles event. Unit: uncore_sys_ccn_pmu ",
+ .matching_pmu = "uncore_sys_ccn_pmu",
+};
+
static const struct perf_pmu_test_event *sys_events[] = {
&sys_ddr_pmu_write_cycles,
+ &sys_ccn_pmu_read_cycles,
NULL
};
return !strcmp(reference, test);
}
-static struct pmu_events_map *__test_pmu_get_events_map(void)
+static const struct pmu_events_map *__test_pmu_get_events_map(void)
{
- struct pmu_events_map *map;
+ const struct pmu_events_map *map;
for (map = &pmu_events_map[0]; map->cpuid; map++) {
if (!strcmp(map->cpuid, "testcpu"))
return NULL;
}
-static struct pmu_event *__test_pmu_get_sys_events_table(void)
+static const struct pmu_event *__test_pmu_get_sys_events_table(void)
{
- struct pmu_sys_events *tables = &pmu_sys_event_tables[0];
+ const struct pmu_sys_events *tables = &pmu_sys_event_tables[0];
for ( ; tables->name; tables++) {
if (!strcmp("pme_test_soc_sys", tables->name))
return NULL;
}
-static int compare_pmu_events(struct pmu_event *e1, const struct pmu_event *e2)
+static int compare_pmu_events(const struct pmu_event *e1, const struct pmu_event *e2)
{
+ if (!is_same(e1->name, e2->name)) {
+ pr_debug2("testing event e1 %s: mismatched name string, %s vs %s\n",
+ e1->name, e1->name, e2->name);
+ return -1;
+ }
+
+ if (!is_same(e1->compat, e2->compat)) {
+ pr_debug2("testing event e1 %s: mismatched compat string, %s vs %s\n",
+ e1->name, e1->compat, e2->compat);
+ return -1;
+ }
+
+ if (!is_same(e1->event, e2->event)) {
+ pr_debug2("testing event e1 %s: mismatched event, %s vs %s\n",
+ e1->name, e1->event, e2->event);
+ return -1;
+ }
+
if (!is_same(e1->desc, e2->desc)) {
pr_debug2("testing event e1 %s: mismatched desc, %s vs %s\n",
e1->name, e1->desc, e2->desc);
return -1;
}
+ if (!is_same(e1->pmu, e2->pmu)) {
+ pr_debug2("testing event e1 %s: mismatched pmu string, %s vs %s\n",
+ e1->name, e1->pmu, e2->pmu);
+ return -1;
+ }
+
if (!is_same(e1->unit, e2->unit)) {
pr_debug2("testing event e1 %s: mismatched unit, %s vs %s\n",
e1->name, e1->unit, e2->unit);
return -1;
}
+ if (!is_same(e1->aggr_mode, e2->aggr_mode)) {
+ pr_debug2("testing event e1 %s: mismatched aggr_mode, %s vs %s\n",
+ e1->name, e1->aggr_mode, e2->aggr_mode);
+ return -1;
+ }
+
if (!is_same(e1->metric_expr, e2->metric_expr)) {
pr_debug2("testing event e1 %s: mismatched metric_expr, %s vs %s\n",
e1->name, e1->metric_expr, e2->metric_expr);
return -1;
}
- if (!is_same(e1->deprecated, e2->deprecated)) {
- pr_debug2("testing event e1 %s: mismatched deprecated, %s vs %s\n",
- e1->name, e1->deprecated, e2->deprecated);
+ if (!is_same(e1->metric_group, e2->metric_group)) {
+ pr_debug2("testing event e1 %s: mismatched metric_group, %s vs %s\n",
+ e1->name, e1->metric_group, e2->metric_group);
return -1;
}
- if (!is_same(e1->pmu, e2->pmu)) {
- pr_debug2("testing event e1 %s: mismatched pmu string, %s vs %s\n",
- e1->name, e1->pmu, e2->pmu);
+ if (!is_same(e1->deprecated, e2->deprecated)) {
+ pr_debug2("testing event e1 %s: mismatched deprecated, %s vs %s\n",
+ e1->name, e1->deprecated, e2->deprecated);
return -1;
}
- if (!is_same(e1->compat, e2->compat)) {
- pr_debug2("testing event e1 %s: mismatched compat string, %s vs %s\n",
- e1->name, e1->compat, e2->compat);
+ if (!is_same(e1->metric_constraint, e2->metric_constraint)) {
+ pr_debug2("testing event e1 %s: mismatched metric_constant, %s vs %s\n",
+ e1->name, e1->metric_constraint, e2->metric_constraint);
return -1;
}
}
/* Verify generated events from pmu-events.c are as expected */
-static int test_pmu_event_table(void)
+static int test__pmu_event_table(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- struct pmu_event *sys_event_tables = __test_pmu_get_sys_events_table();
- struct pmu_events_map *map = __test_pmu_get_events_map();
- struct pmu_event *table;
+ const struct pmu_event *sys_event_tables = __test_pmu_get_sys_events_table();
+ const struct pmu_events_map *map = __test_pmu_get_events_map();
+ const struct pmu_event *table;
int map_events = 0, expected_events;
/* ignore 3x sentinels */
struct perf_pmu *pmu;
LIST_HEAD(aliases);
int res = 0;
- struct pmu_events_map *map = __test_pmu_get_events_map();
+ const struct pmu_events_map *map = __test_pmu_get_events_map();
struct perf_pmu_alias *a, *tmp;
if (!map)
struct perf_pmu *pmu = &test_pmu->pmu;
const char *pmu_name = pmu->name;
struct perf_pmu_alias *a, *tmp, *alias;
- struct pmu_events_map *map;
+ const struct pmu_events_map *map;
LIST_HEAD(aliases);
int res = 0;
&sys_ddr_pmu_write_cycles,
},
},
+ {
+ .pmu = {
+ .name = (char *)"uncore_sys_ccn_pmu4",
+ .is_uncore = 1,
+ .id = (char *)"0x01",
+ },
+ .aliases = {
+ &sys_ccn_pmu_read_cycles,
+ },
+ },
};
/* Test that aliases generated are as expected */
-static int test_aliases(void)
+static int test__aliases(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct perf_pmu *pmu = NULL;
unsigned long i;
{
struct evlist *evlist;
int ret;
+ char *dup, *cur;
/* Numbers are always valid. */
if (is_number(id))
evlist = evlist__new();
if (!evlist)
return -ENOMEM;
- ret = __parse_events(evlist, id, error, fake_pmu);
+
+ dup = strdup(id);
+ if (!dup)
+ return -ENOMEM;
+
+ for (cur = strchr(dup, '@') ; cur; cur = strchr(++cur, '@'))
+ *cur = '/';
+
+ ret = __parse_events(evlist, dup, error, fake_pmu);
+ free(dup);
+
evlist__delete(evlist);
return ret;
}
-static int check_parse_cpu(const char *id, bool same_cpu, struct pmu_event *pe)
+static int check_parse_cpu(const char *id, bool same_cpu, const struct pmu_event *pe)
{
- struct parse_events_error error = { .idx = 0, };
+ struct parse_events_error error;
+ int ret;
- int ret = check_parse_id(id, &error, NULL);
+ parse_events_error__init(&error);
+ ret = check_parse_id(id, &error, NULL);
if (ret && same_cpu) {
pr_warning("Parse event failed metric '%s' id '%s' expr '%s'\n",
pe->metric_name, id, pe->metric_expr);
id, pe->metric_name, pe->metric_expr);
ret = 0;
}
- free(error.str);
- free(error.help);
- free(error.first_str);
- free(error.first_help);
+ parse_events_error__exit(&error);
return ret;
}
static int check_parse_fake(const char *id)
{
- struct parse_events_error error = { .idx = 0, };
- int ret = check_parse_id(id, &error, &perf_pmu__fake);
+ struct parse_events_error error;
+ int ret;
- free(error.str);
- free(error.help);
- free(error.first_str);
- free(error.first_help);
+ parse_events_error__init(&error);
+ ret = check_parse_id(id, &error, &perf_pmu__fake);
+ parse_events_error__exit(&error);
return ret;
}
static int resolve_metric_simple(struct expr_parse_ctx *pctx,
struct list_head *compound_list,
- struct pmu_events_map *map,
+ const struct pmu_events_map *map,
const char *metric_name)
{
struct hashmap_entry *cur, *cur_tmp;
do {
all = true;
- hashmap__for_each_entry_safe((&pctx->ids), cur, cur_tmp, bkt) {
+ hashmap__for_each_entry_safe(pctx->ids, cur, cur_tmp, bkt) {
struct metric_ref *ref;
- struct pmu_event *pe;
+ const struct pmu_event *pe;
pe = metricgroup__find_metric(cur->key, map);
if (!pe)
ref->metric_expr = pe->metric_expr;
list_add_tail(&metric->list, compound_list);
- rc = expr__find_other(pe->metric_expr, NULL, pctx, 0);
+ rc = expr__find_ids(pe->metric_expr, NULL, pctx);
if (rc)
goto out_err;
break; /* The hashmap has been modified, so restart */
}
-static int test_parsing(void)
+static int test__parsing(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- struct pmu_events_map *cpus_map = pmu_events_map__find();
- struct pmu_events_map *map;
- struct pmu_event *pe;
+ const struct pmu_events_map *cpus_map = pmu_events_map__find();
+ const struct pmu_events_map *map;
+ const struct pmu_event *pe;
int i, j, k;
int ret = 0;
- struct expr_parse_ctx ctx;
+ struct expr_parse_ctx *ctx;
double result;
+ ctx = expr__ctx_new();
+ if (!ctx) {
+ pr_debug("expr__ctx_new failed");
+ return TEST_FAIL;
+ }
i = 0;
for (;;) {
map = &pmu_events_map[i++];
break;
if (!pe->metric_expr)
continue;
- expr__ctx_init(&ctx);
- if (expr__find_other(pe->metric_expr, NULL, &ctx, 0)
- < 0) {
- expr_failure("Parse other failed", map, pe);
+ expr__ctx_clear(ctx);
+ if (expr__find_ids(pe->metric_expr, NULL, ctx) < 0) {
+ expr_failure("Parse find ids failed", map, pe);
ret++;
continue;
}
- if (resolve_metric_simple(&ctx, &compound_list, map,
+ if (resolve_metric_simple(ctx, &compound_list, map,
pe->metric_name)) {
expr_failure("Could not resolve metrics", map, pe);
ret++;
* make them unique.
*/
k = 1;
- hashmap__for_each_entry((&ctx.ids), cur, bkt)
- expr__add_id_val(&ctx, strdup(cur->key), k++);
+ hashmap__for_each_entry(ctx->ids, cur, bkt)
+ expr__add_id_val(ctx, strdup(cur->key), k++);
- hashmap__for_each_entry((&ctx.ids), cur, bkt) {
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
if (check_parse_cpu(cur->key, map == cpus_map,
pe))
ret++;
}
list_for_each_entry_safe(metric, tmp, &compound_list, list) {
- expr__add_ref(&ctx, &metric->metric_ref);
+ expr__add_ref(ctx, &metric->metric_ref);
free(metric);
}
- if (expr__parse(&result, &ctx, pe->metric_expr, 0)) {
+ if (expr__parse(&result, ctx, pe->metric_expr)) {
expr_failure("Parse failed", map, pe);
ret++;
}
- expr__ctx_clear(&ctx);
}
}
+ expr__ctx_free(ctx);
/* TODO: fail when not ok */
exit:
return ret == 0 ? TEST_OK : TEST_SKIP;
static int metric_parse_fake(const char *str)
{
- struct expr_parse_ctx ctx;
+ struct expr_parse_ctx *ctx;
struct hashmap_entry *cur;
double result;
int ret = -1;
pr_debug("parsing '%s'\n", str);
- expr__ctx_init(&ctx);
- if (expr__find_other(str, NULL, &ctx, 0) < 0) {
- pr_err("expr__find_other failed\n");
+ ctx = expr__ctx_new();
+ if (!ctx) {
+ pr_debug("expr__ctx_new failed");
+ return TEST_FAIL;
+ }
+ if (expr__find_ids(str, NULL, ctx) < 0) {
+ pr_err("expr__find_ids failed\n");
return -1;
}
* make them unique.
*/
i = 1;
- hashmap__for_each_entry((&ctx.ids), cur, bkt)
- expr__add_id_val(&ctx, strdup(cur->key), i++);
+ hashmap__for_each_entry(ctx->ids, cur, bkt)
+ expr__add_id_val(ctx, strdup(cur->key), i++);
- hashmap__for_each_entry((&ctx.ids), cur, bkt) {
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
if (check_parse_fake(cur->key)) {
pr_err("check_parse_fake failed\n");
goto out;
}
}
- if (expr__parse(&result, &ctx, str, 0))
+ if (expr__parse(&result, ctx, str))
pr_err("expr__parse failed\n");
else
ret = 0;
out:
- expr__ctx_clear(&ctx);
+ expr__ctx_free(ctx);
return ret;
}
* or all defined cpus via the 'fake_pmu'
* in parse_events.
*/
-static int test_parsing_fake(void)
+static int test__parsing_fake(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
- struct pmu_events_map *map;
- struct pmu_event *pe;
+ const struct pmu_events_map *map;
+ const struct pmu_event *pe;
unsigned int i, j;
int err = 0;
return 0;
}
-static const struct {
- int (*func)(void);
- const char *desc;
-} pmu_events_testcase_table[] = {
- {
- .func = test_pmu_event_table,
- .desc = "PMU event table sanity",
- },
- {
- .func = test_aliases,
- .desc = "PMU event map aliases",
- },
- {
- .func = test_parsing,
- .desc = "Parsing of PMU event table metrics",
- },
- {
- .func = test_parsing_fake,
- .desc = "Parsing of PMU event table metrics with fake PMUs",
- },
+static struct test_case pmu_events_tests[] = {
+ TEST_CASE("PMU event table sanity", pmu_event_table),
+ TEST_CASE("PMU event map aliases", aliases),
+ TEST_CASE_REASON("Parsing of PMU event table metrics", parsing,
+ "some metrics failed"),
+ TEST_CASE("Parsing of PMU event table metrics with fake PMUs", parsing_fake),
+ { .name = NULL, }
};
-const char *test__pmu_events_subtest_get_desc(int subtest)
-{
- if (subtest < 0 ||
- subtest >= (int)ARRAY_SIZE(pmu_events_testcase_table))
- return NULL;
- return pmu_events_testcase_table[subtest].desc;
-}
-
-const char *test__pmu_events_subtest_skip_reason(int subtest)
-{
- if (subtest < 0 ||
- subtest >= (int)ARRAY_SIZE(pmu_events_testcase_table))
- return NULL;
- if (pmu_events_testcase_table[subtest].func != test_parsing)
- return NULL;
- return "some metrics failed";
-}
-
-int test__pmu_events_subtest_get_nr(void)
-{
- return (int)ARRAY_SIZE(pmu_events_testcase_table);
-}
-
-int test__pmu_events(struct test *test __maybe_unused, int subtest)
-{
- if (subtest < 0 ||
- subtest >= (int)ARRAY_SIZE(pmu_events_testcase_table))
- return TEST_FAIL;
- return pmu_events_testcase_table[subtest].func();
-}
+struct test_suite suite__pmu_events = {
+ .desc = "PMU events",
+ .test_cases = pmu_events_tests,
+};
return &terms;
}
-int test__pmu(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__pmu(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
char *format = test_format_dir_get();
LIST_HEAD(formats);
test_format_dir_put(format);
return ret;
}
+
+DEFINE_SUITE("Parse perf pmu format", pmu);
#include "tests.h"
#include "util/debug.h"
-int test__python_use(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__python_use(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
char *cmd;
int ret;
free(cmd);
return ret;
}
+
+DEFINE_SUITE("'import perf' in python", python_use);
#include "evsel.h"
#include "debug.h"
#include "util/synthetic-events.h"
+#include "util/trace-event.h"
#include "tests.h"
} \
} while (0)
+/*
+ * Hardcode the expected values for branch_entry flags.
+ * These are based on the input value (213) specified
+ * in branch_stack variable.
+ */
+#define BS_EXPECTED_BE 0xa000d00000000000
+#define BS_EXPECTED_LE 0xd5000000
+#define FLAG(s) s->branch_stack->entries[i].flags
+
static bool samples_same(const struct perf_sample *s1,
const struct perf_sample *s2,
- u64 type, u64 read_format)
+ u64 type, u64 read_format, bool needs_swap)
{
size_t i;
if (type & PERF_SAMPLE_BRANCH_STACK) {
COMP(branch_stack->nr);
COMP(branch_stack->hw_idx);
- for (i = 0; i < s1->branch_stack->nr; i++)
- MCOMP(branch_stack->entries[i]);
+ for (i = 0; i < s1->branch_stack->nr; i++) {
+ if (needs_swap)
+ return ((tep_is_bigendian()) ?
+ (FLAG(s2).value == BS_EXPECTED_BE) :
+ (FLAG(s2).value == BS_EXPECTED_LE));
+ else
+ MCOMP(branch_stack->entries[i]);
+ }
}
if (type & PERF_SAMPLE_REGS_USER) {
},
};
struct sample_read_value values[] = {{1, 5}, {9, 3}, {2, 7}, {6, 4},};
- struct perf_sample sample_out;
+ struct perf_sample sample_out, sample_out_endian;
size_t i, sz, bufsz;
int err, ret = -1;
goto out_free;
}
- if (!samples_same(&sample, &sample_out, sample_type, read_format)) {
+ if (!samples_same(&sample, &sample_out, sample_type, read_format, evsel.needs_swap)) {
pr_debug("parsing failed for sample_type %#"PRIx64"\n",
sample_type);
goto out_free;
}
+ if (sample_type == PERF_SAMPLE_BRANCH_STACK) {
+ evsel.needs_swap = true;
+ evsel.sample_size = __evsel__sample_size(sample_type);
+ err = evsel__parse_sample(&evsel, event, &sample_out_endian);
+ if (err) {
+ pr_debug("%s failed for sample_type %#"PRIx64", error %d\n",
+ "evsel__parse_sample", sample_type, err);
+ goto out_free;
+ }
+
+ if (!samples_same(&sample, &sample_out_endian, sample_type, read_format, evsel.needs_swap)) {
+ pr_debug("parsing failed for sample_type %#"PRIx64"\n",
+ sample_type);
+ goto out_free;
+ }
+ }
+
ret = 0;
out_free:
free(event);
* checks sample format bits separately and together. If the test passes %0 is
* returned, otherwise %-1 is returned.
*/
-int test__sample_parsing(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__sample_parsing(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
const u64 rf[] = {4, 5, 6, 7, 12, 13, 14, 15};
u64 sample_type;
return 0;
}
+
+DEFINE_SUITE("Sample parsing", sample_parsing);
return ret;
}
-int test__sdt_event(struct test *test __maybe_unused, int subtests __maybe_unused)
+static int test__sdt_event(struct test_suite *test __maybe_unused, int subtests __maybe_unused)
{
int ret = TEST_FAIL;
char __tempdir[] = "./test-buildid-XXXXXX";
return ret;
}
#else
-int test__sdt_event(struct test *test __maybe_unused, int subtests __maybe_unused)
+static int test__sdt_event(struct test_suite *test __maybe_unused, int subtests __maybe_unused)
{
pr_debug("Skip SDT event test because SDT support is not compiled\n");
return TEST_SKIP;
}
#endif
+
+DEFINE_SUITE("Probe SDT events", sdt_event);
. $(dirname $0)/lib/probe_vfs_getname.sh
-perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
-file=$(mktemp /tmp/temporary_file.XXXXX)
-
record_open_file() {
echo "Recording open file:"
perf record -o ${perfdata} -e probe:vfs_getname\* touch $file
exit $err
fi
+perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+file=$(mktemp /tmp/temporary_file.XXXXX)
+
record_open_file && perf_script_filenames
err=$?
rm -f ${perfdata}
collect_z_record() {
echo "Collecting compressed record file:"
- [[ "$(uname -m)" != s390x ]] && gflag='-g'
+ [ "$(uname -m)" != s390x ] && gflag='-g'
$perf_tool record -o $trace_file $gflag -z -F 5000 -- \
dd count=500 if=/dev/urandom of=/dev/null
}
--- /dev/null
+#!/bin/sh
+# perf all metricgroups test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metricgroups); do
+ echo "Testing $m"
+ perf stat -M "$m" true
+done
+
+exit 0
--- /dev/null
+#!/bin/sh
+# perf all metrics test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for m in $(perf list --raw-dump metrics); do
+ echo "Testing $m"
+ result=$(perf stat -M "$m" true 2>&1)
+ if [[ ! "$result" =~ "$m" ]] && [[ ! "$result" =~ "<not supported>" ]]; then
+ # We failed to see the metric and the events are support. Possibly the
+ # workload was too small so retry with something longer.
+ result=$(perf stat -M "$m" perf bench internals synthesize 2>&1)
+ if [[ ! "$result" =~ "$m" ]]; then
+ echo "Metric '$m' not printed in:"
+ echo "$result"
+ exit 1
+ fi
+ fi
+done
+
+exit 0
--- /dev/null
+#!/bin/sh
+# perf all PMU test
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+for p in $(perf list --raw-dump pmu); do
+ echo "Testing $p"
+ result=$(perf stat -e "$p" true 2>&1)
+ if ! echo "$result" | grep -q "$p" && ! echo "$result" | grep -q "<not supported>" ; then
+ # We failed to see the event and it is supported. Possibly the workload was
+ # too small so retry with something longer.
+ result=$(perf stat -e "$p" perf bench internals synthesize 2>&1)
+ if ! echo "$result" | grep -q "$p" ; then
+ echo "Event '$p' not printed in:"
+ echo "$result"
+ exit 1
+ fi
+ fi
+done
+
+exit 0
# skip if --bpf-counters is not supported
if ! perf stat --bpf-counters true > /dev/null 2>&1; then
- if [ "$1" == "-v" ]; then
+ if [ "$1" = "-v" ]; then
echo "Skipping: --bpf-counters not supported"
perf --no-pager stat --bpf-counters true || true
fi
# SPDX-License-Identifier: GPL-2.0
# Leo Yan <leo.yan@linaro.org>, 2020
-perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
-file=$(mktemp /tmp/temporary_file.XXXXX)
glb_err=0
skip_if_no_cs_etm_event() {
skip_if_no_cs_etm_event || exit 2
+perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+file=$(mktemp /tmp/temporary_file.XXXXX)
+
cleanup_files()
{
rm -f ${perfdata}
rm -f ${file}
+ rm -f "${perfdata}.old"
+ trap - exit term int
+ kill -2 $$
+ exit $glb_err
}
-trap cleanup_files exit
+trap cleanup_files exit term int
record_touch_file() {
echo "Recording trace (only user mode) with path: CPU$2 => $1"
--- /dev/null
+#!/bin/sh
+# Check Arm SPE trace data recording and synthesized samples
+
+# Uses the 'perf record' to record trace data of Arm SPE events;
+# then verify if any SPE event samples are generated by SPE with
+# 'perf script' and 'perf report' commands.
+
+# SPDX-License-Identifier: GPL-2.0
+# German Gomez <german.gomez@arm.com>, 2021
+
+skip_if_no_arm_spe_event() {
+ perf list | egrep -q 'arm_spe_[0-9]+//' && return 0
+
+ # arm_spe event doesn't exist
+ return 2
+}
+
+skip_if_no_arm_spe_event || exit 2
+
+perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+glb_err=0
+
+cleanup_files()
+{
+ rm -f ${perfdata}
+ exit $glb_err
+}
+
+trap cleanup_files exit term int
+
+arm_spe_report() {
+ if [ $2 != 0 ]; then
+ echo "$1: FAIL"
+ glb_err=$2
+ else
+ echo "$1: PASS"
+ fi
+}
+
+perf_script_samples() {
+ echo "Looking at perf.data file for dumping samples:"
+
+ # from arm-spe.c/arm_spe_synth_events()
+ events="(ld1-miss|ld1-access|llc-miss|lld-access|tlb-miss|tlb-access|branch-miss|remote-access|memory)"
+
+ # Below is an example of the samples dumping:
+ # dd 3048 [002] 1 l1d-access: ffffaa64999c __GI___libc_write+0x3c (/lib/aarch64-linux-gnu/libc-2.27.so)
+ # dd 3048 [002] 1 tlb-access: ffffaa64999c __GI___libc_write+0x3c (/lib/aarch64-linux-gnu/libc-2.27.so)
+ # dd 3048 [002] 1 memory: ffffaa64999c __GI___libc_write+0x3c (/lib/aarch64-linux-gnu/libc-2.27.so)
+ perf script -F,-time -i ${perfdata} 2>&1 | \
+ egrep " +$1 +[0-9]+ .* +${events}:(.*:)? +" > /dev/null 2>&1
+}
+
+perf_report_samples() {
+ echo "Looking at perf.data file for reporting samples:"
+
+ # Below is an example of the samples reporting:
+ # 73.04% 73.04% dd libc-2.27.so [.] _dl_addr
+ # 7.71% 7.71% dd libc-2.27.so [.] getenv
+ # 2.59% 2.59% dd ld-2.27.so [.] strcmp
+ perf report --stdio -i ${perfdata} 2>&1 | \
+ egrep " +[0-9]+\.[0-9]+% +[0-9]+\.[0-9]+% +$1 " > /dev/null 2>&1
+}
+
+arm_spe_snapshot_test() {
+ echo "Recording trace with snapshot mode $perfdata"
+ perf record -o ${perfdata} -e arm_spe// -S \
+ -- dd if=/dev/zero of=/dev/null > /dev/null 2>&1 &
+ PERFPID=$!
+
+ # Wait for perf program
+ sleep 1
+
+ # Send signal to snapshot trace data
+ kill -USR2 $PERFPID
+
+ # Stop perf program
+ kill $PERFPID
+ wait $PERFPID
+
+ perf_script_samples dd &&
+ perf_report_samples dd
+
+ err=$?
+ arm_spe_report "SPE snapshot testing" $err
+}
+
+arm_spe_snapshot_test
+exit $glb_err
. $(dirname $0)/lib/probe_vfs_getname.sh
-file=$(mktemp /tmp/temporary_file.XXXXX)
-
trace_open_vfs_getname() {
evts=$(echo $(perf list syscalls:sys_enter_open* 2>/dev/null | egrep 'open(at)? ' | sed -r 's/.*sys_enter_([a-z]+) +\[.*$/\1/') | sed 's/ /,/')
perf trace -e $evts touch $file 2>&1 | \
exit $err
fi
+file=$(mktemp /tmp/temporary_file.XXXXX)
+
# Do not use whatever ~/.perfconfig file, it may change the output
# via trace.{show_timestamp,show_prefix,etc}
export PERF_CONFIG=/dev/null
return 0;
}
-int test__synthesize_stat_config(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__synthesize_stat_config(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
struct perf_stat_config stat_config = {
.aggr_mode = AGGR_CORE,
return 0;
}
-int test__synthesize_stat(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__synthesize_stat(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_counts_values count;
return 0;
}
-int test__synthesize_stat_round(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__synthesize_stat_round(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
TEST_ASSERT_VAL("failed to synthesize stat_config",
!perf_event__synthesize_stat_round(NULL, 0xdeadbeef, PERF_STAT_ROUND_TYPE__INTERVAL,
return 0;
}
+
+DEFINE_SUITE("Synthesize stat config", synthesize_stat_config);
+DEFINE_SUITE("Synthesize stat", synthesize_stat);
+DEFINE_SUITE("Synthesize stat round", synthesize_stat_round);
return err;
}
-int test__sw_clock_freq(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__sw_clock_freq(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret;
return ret;
}
+
+DEFINE_SUITE("Software clock events period values", sw_clock_freq);
* evsel->core.system_wide and evsel->tracking flags (respectively) with other events
* sometimes enabled or disabled.
*/
-int test__switch_tracking(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__switch_tracking(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
const char *sched_switch = "sched:sched_switch";
struct switch_tracking switch_tracking = { .tids = NULL, };
err = -1;
goto out;
}
+
+DEFINE_SUITE("Track with sched_switch", switch_tracking);
* if the number of exit event reported by the kernel is 1 or not
* in order to check the kernel returns correct number of event.
*/
-int test__task_exit(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__task_exit(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int err = -1;
union perf_event *event;
evlist__delete(evlist);
return err;
}
+
+DEFINE_SUITE("Number of exit events of a simple workload", task_exit);
TEST_SKIP = -2,
};
-struct test {
+struct test_suite;
+
+typedef int (*test_fnptr)(struct test_suite *, int);
+
+struct test_case {
+ const char *name;
+ const char *desc;
+ const char *skip_reason;
+ test_fnptr run_case;
+};
+
+struct test_suite {
const char *desc;
- int (*func)(struct test *test, int subtest);
- struct {
- bool skip_if_fail;
- int (*get_nr)(void);
- const char *(*get_desc)(int subtest);
- const char *(*skip_reason)(int subtest);
- } subtest;
- bool (*is_supported)(void);
+ struct test_case *test_cases;
void *priv;
};
+#define DECLARE_SUITE(name) \
+ extern struct test_suite suite__##name;
+
+#define TEST_CASE(description, _name) \
+ { \
+ .name = #_name, \
+ .desc = description, \
+ .run_case = test__##_name, \
+ }
+
+#define TEST_CASE_REASON(description, _name, _reason) \
+ { \
+ .name = #_name, \
+ .desc = description, \
+ .run_case = test__##_name, \
+ .skip_reason = _reason, \
+ }
+
+#define DEFINE_SUITE(description, _name) \
+ struct test_case tests__##_name[] = { \
+ TEST_CASE(description, _name), \
+ { .name = NULL, } \
+ }; \
+ struct test_suite suite__##_name = { \
+ .desc = description, \
+ .test_cases = tests__##_name, \
+ }
+
/* Tests */
-int test__vmlinux_matches_kallsyms(struct test *test, int subtest);
-int test__openat_syscall_event(struct test *test, int subtest);
-int test__openat_syscall_event_on_all_cpus(struct test *test, int subtest);
-int test__basic_mmap(struct test *test, int subtest);
-int test__PERF_RECORD(struct test *test, int subtest);
-int test__perf_evsel__roundtrip_name_test(struct test *test, int subtest);
-int test__perf_evsel__tp_sched_test(struct test *test, int subtest);
-int test__syscall_openat_tp_fields(struct test *test, int subtest);
-int test__pmu(struct test *test, int subtest);
-int test__pmu_events(struct test *test, int subtest);
-const char *test__pmu_events_subtest_get_desc(int subtest);
-const char *test__pmu_events_subtest_skip_reason(int subtest);
-int test__pmu_events_subtest_get_nr(void);
-int test__attr(struct test *test, int subtest);
-int test__dso_data(struct test *test, int subtest);
-int test__dso_data_cache(struct test *test, int subtest);
-int test__dso_data_reopen(struct test *test, int subtest);
-int test__parse_events(struct test *test, int subtest);
-int test__hists_link(struct test *test, int subtest);
-int test__python_use(struct test *test, int subtest);
-int test__bp_signal(struct test *test, int subtest);
-int test__bp_signal_overflow(struct test *test, int subtest);
-int test__bp_accounting(struct test *test, int subtest);
-int test__wp(struct test *test, int subtest);
-const char *test__wp_subtest_get_desc(int subtest);
-const char *test__wp_subtest_skip_reason(int subtest);
-int test__wp_subtest_get_nr(void);
-int test__task_exit(struct test *test, int subtest);
-int test__mem(struct test *test, int subtest);
-int test__sw_clock_freq(struct test *test, int subtest);
-int test__code_reading(struct test *test, int subtest);
-int test__sample_parsing(struct test *test, int subtest);
-int test__keep_tracking(struct test *test, int subtest);
-int test__parse_no_sample_id_all(struct test *test, int subtest);
-int test__dwarf_unwind(struct test *test, int subtest);
-int test__expr(struct test *test, int subtest);
-int test__hists_filter(struct test *test, int subtest);
-int test__mmap_thread_lookup(struct test *test, int subtest);
-int test__thread_maps_share(struct test *test, int subtest);
-int test__hists_output(struct test *test, int subtest);
-int test__hists_cumulate(struct test *test, int subtest);
-int test__switch_tracking(struct test *test, int subtest);
-int test__fdarray__filter(struct test *test, int subtest);
-int test__fdarray__add(struct test *test, int subtest);
-int test__kmod_path__parse(struct test *test, int subtest);
-int test__thread_map(struct test *test, int subtest);
-int test__llvm(struct test *test, int subtest);
-const char *test__llvm_subtest_get_desc(int subtest);
-int test__llvm_subtest_get_nr(void);
-int test__bpf(struct test *test, int subtest);
-const char *test__bpf_subtest_get_desc(int subtest);
-int test__bpf_subtest_get_nr(void);
-int test__session_topology(struct test *test, int subtest);
-int test__thread_map_synthesize(struct test *test, int subtest);
-int test__thread_map_remove(struct test *test, int subtest);
-int test__cpu_map_synthesize(struct test *test, int subtest);
-int test__synthesize_stat_config(struct test *test, int subtest);
-int test__synthesize_stat(struct test *test, int subtest);
-int test__synthesize_stat_round(struct test *test, int subtest);
-int test__event_update(struct test *test, int subtest);
-int test__event_times(struct test *test, int subtest);
-int test__backward_ring_buffer(struct test *test, int subtest);
-int test__cpu_map_print(struct test *test, int subtest);
-int test__cpu_map_merge(struct test *test, int subtest);
-int test__sdt_event(struct test *test, int subtest);
-int test__is_printable_array(struct test *test, int subtest);
-int test__bitmap_print(struct test *test, int subtest);
-int test__perf_hooks(struct test *test, int subtest);
-int test__clang(struct test *test, int subtest);
-const char *test__clang_subtest_get_desc(int subtest);
-int test__clang_subtest_get_nr(void);
-int test__unit_number__scnprint(struct test *test, int subtest);
-int test__mem2node(struct test *t, int subtest);
-int test__maps__merge_in(struct test *t, int subtest);
-int test__time_utils(struct test *t, int subtest);
-int test__jit_write_elf(struct test *test, int subtest);
-int test__api_io(struct test *test, int subtest);
-int test__demangle_java(struct test *test, int subtest);
-int test__demangle_ocaml(struct test *test, int subtest);
-int test__pfm(struct test *test, int subtest);
-const char *test__pfm_subtest_get_desc(int subtest);
-int test__pfm_subtest_get_nr(void);
-int test__parse_metric(struct test *test, int subtest);
-int test__pe_file_parsing(struct test *test, int subtest);
-int test__expand_cgroup_events(struct test *test, int subtest);
-int test__perf_time_to_tsc(struct test *test, int subtest);
-int test__dlfilter(struct test *test, int subtest);
-
-bool test__bp_signal_is_supported(void);
-bool test__bp_account_is_supported(void);
-bool test__wp_is_supported(void);
-bool test__tsc_is_supported(void);
+DECLARE_SUITE(vmlinux_matches_kallsyms);
+DECLARE_SUITE(openat_syscall_event);
+DECLARE_SUITE(openat_syscall_event_on_all_cpus);
+DECLARE_SUITE(basic_mmap);
+DECLARE_SUITE(PERF_RECORD);
+DECLARE_SUITE(perf_evsel__roundtrip_name_test);
+DECLARE_SUITE(perf_evsel__tp_sched_test);
+DECLARE_SUITE(syscall_openat_tp_fields);
+DECLARE_SUITE(pmu);
+DECLARE_SUITE(pmu_events);
+DECLARE_SUITE(attr);
+DECLARE_SUITE(dso_data);
+DECLARE_SUITE(dso_data_cache);
+DECLARE_SUITE(dso_data_reopen);
+DECLARE_SUITE(parse_events);
+DECLARE_SUITE(hists_link);
+DECLARE_SUITE(python_use);
+DECLARE_SUITE(bp_signal);
+DECLARE_SUITE(bp_signal_overflow);
+DECLARE_SUITE(bp_accounting);
+DECLARE_SUITE(wp);
+DECLARE_SUITE(task_exit);
+DECLARE_SUITE(mem);
+DECLARE_SUITE(sw_clock_freq);
+DECLARE_SUITE(code_reading);
+DECLARE_SUITE(sample_parsing);
+DECLARE_SUITE(keep_tracking);
+DECLARE_SUITE(parse_no_sample_id_all);
+DECLARE_SUITE(dwarf_unwind);
+DECLARE_SUITE(expr);
+DECLARE_SUITE(hists_filter);
+DECLARE_SUITE(mmap_thread_lookup);
+DECLARE_SUITE(thread_maps_share);
+DECLARE_SUITE(hists_output);
+DECLARE_SUITE(hists_cumulate);
+DECLARE_SUITE(switch_tracking);
+DECLARE_SUITE(fdarray__filter);
+DECLARE_SUITE(fdarray__add);
+DECLARE_SUITE(kmod_path__parse);
+DECLARE_SUITE(thread_map);
+DECLARE_SUITE(llvm);
+DECLARE_SUITE(bpf);
+DECLARE_SUITE(session_topology);
+DECLARE_SUITE(thread_map_synthesize);
+DECLARE_SUITE(thread_map_remove);
+DECLARE_SUITE(cpu_map_synthesize);
+DECLARE_SUITE(synthesize_stat_config);
+DECLARE_SUITE(synthesize_stat);
+DECLARE_SUITE(synthesize_stat_round);
+DECLARE_SUITE(event_update);
+DECLARE_SUITE(event_times);
+DECLARE_SUITE(backward_ring_buffer);
+DECLARE_SUITE(cpu_map_print);
+DECLARE_SUITE(cpu_map_merge);
+DECLARE_SUITE(sdt_event);
+DECLARE_SUITE(is_printable_array);
+DECLARE_SUITE(bitmap_print);
+DECLARE_SUITE(perf_hooks);
+DECLARE_SUITE(clang);
+DECLARE_SUITE(unit_number__scnprint);
+DECLARE_SUITE(mem2node);
+DECLARE_SUITE(maps__merge_in);
+DECLARE_SUITE(time_utils);
+DECLARE_SUITE(jit_write_elf);
+DECLARE_SUITE(api_io);
+DECLARE_SUITE(demangle_java);
+DECLARE_SUITE(demangle_ocaml);
+DECLARE_SUITE(pfm);
+DECLARE_SUITE(parse_metric);
+DECLARE_SUITE(pe_file_parsing);
+DECLARE_SUITE(expand_cgroup_events);
+DECLARE_SUITE(perf_time_to_tsc);
+DECLARE_SUITE(dlfilter);
+
+/*
+ * PowerPC and S390 do not support creation of instruction breakpoints using the
+ * perf_event interface.
+ *
+ * ARM requires explicit rounding down of the instruction pointer in Thumb mode,
+ * and then requires the single-step to be handled explicitly in the overflow
+ * handler to avoid stepping into the SIGIO handler and getting stuck on the
+ * breakpointed instruction.
+ *
+ * Since arm64 has the same issue with arm for the single-step handling, this
+ * case also gets stuck on the breakpointed instruction.
+ *
+ * Just disable the test for these architectures until these issues are
+ * resolved.
+ */
+#if defined(__powerpc__) || defined(__s390x__) || defined(__arm__) || defined(__aarch64__)
+#define BP_SIGNAL_IS_SUPPORTED 0
+#else
+#define BP_SIGNAL_IS_SUPPORTED 1
+#endif
#ifdef HAVE_DWARF_UNWIND_SUPPORT
struct thread;
#endif
#if defined(__arm__)
-int test__vectors_page(struct test *test, int subtest);
+DECLARE_SUITE(vectors_page);
#endif
#endif /* TESTS_H */
#define NAME (const char *) "perf"
#define NAMEUL (unsigned long) NAME
-int test__thread_map(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__thread_map(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_thread_map *map;
return 0;
}
-int test__thread_map_synthesize(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__thread_map_synthesize(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_thread_map *threads;
return 0;
}
-int test__thread_map_remove(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__thread_map_remove(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct perf_thread_map *threads;
char *str;
perf_thread_map__put(threads);
return 0;
}
+
+DEFINE_SUITE("Thread map", thread_map);
+DEFINE_SUITE("Synthesize thread map", thread_map_synthesize);
+DEFINE_SUITE("Remove thread map", thread_map_remove);
#include "thread.h"
#include "debug.h"
-int test__thread_maps_share(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__thread_maps_share(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
struct machines machines;
struct machine *machine;
machines__exit(&machines);
return 0;
}
+
+DEFINE_SUITE("Share thread maps", thread_maps_share);
return pass;
}
-int test__time_utils(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__time_utils(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
bool pass = true;
return pass ? 0 : TEST_FAIL;
}
+
+DEFINE_SUITE("time utils", time_utils);
session->evlist = evlist__new();
TEST_ASSERT_VAL("can't get evlist", session->evlist);
+ parse_events_error__init(&err);
parse_events(session->evlist, "cpu_core/cycles/", &err);
+ parse_events_error__exit(&err);
}
perf_header__set_feat(&session->header, HEADER_CPU_TOPOLOGY);
return 0;
}
-int test__session_topology(struct test *test __maybe_unused, int subtest __maybe_unused)
+static int test__session_topology(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
char path[PATH_MAX];
struct perf_cpu_map *map;
unlink(path);
return ret;
}
+
+DEFINE_SUITE("Session topology", session_topology);
#include "units.h"
#include "debug.h"
-int test__unit_number__scnprint(struct test *t __maybe_unused, int subtest __maybe_unused)
+static int test__unit_number__scnprint(struct test_suite *t __maybe_unused, int subtest __maybe_unused)
{
struct {
u64 n;
return TEST_OK;
}
+
+DEFINE_SUITE("unit_number__scnprintf", unit_number__scnprint);
#include <linux/rbtree.h>
#include <inttypes.h>
#include <string.h>
+#include <ctype.h>
#include <stdlib.h>
#include "dso.h"
#include "map.h"
#define UM(x) kallsyms_map->unmap_ip(kallsyms_map, (x))
-int test__vmlinux_matches_kallsyms(struct test *test __maybe_unused, int subtest __maybe_unused)
+static bool is_ignored_symbol(const char *name, char type)
+{
+ /* Symbol names that exactly match to the following are ignored.*/
+ static const char * const ignored_symbols[] = {
+ /*
+ * Symbols which vary between passes. Passes 1 and 2 must have
+ * identical symbol lists. The kallsyms_* symbols below are
+ * only added after pass 1, they would be included in pass 2
+ * when --all-symbols is specified so exclude them to get a
+ * stable symbol list.
+ */
+ "kallsyms_addresses",
+ "kallsyms_offsets",
+ "kallsyms_relative_base",
+ "kallsyms_num_syms",
+ "kallsyms_names",
+ "kallsyms_markers",
+ "kallsyms_token_table",
+ "kallsyms_token_index",
+ /* Exclude linker generated symbols which vary between passes */
+ "_SDA_BASE_", /* ppc */
+ "_SDA2_BASE_", /* ppc */
+ NULL
+ };
+
+ /* Symbol names that begin with the following are ignored.*/
+ static const char * const ignored_prefixes[] = {
+ "$", /* local symbols for ARM, MIPS, etc. */
+ ".LASANPC", /* s390 kasan local symbols */
+ "__crc_", /* modversions */
+ "__efistub_", /* arm64 EFI stub namespace */
+ "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */
+ "__AArch64ADRPThunk_", /* arm64 lld */
+ "__ARMV5PILongThunk_", /* arm lld */
+ "__ARMV7PILongThunk_",
+ "__ThumbV7PILongThunk_",
+ "__LA25Thunk_", /* mips lld */
+ "__microLA25Thunk_",
+ NULL
+ };
+
+ /* Symbol names that end with the following are ignored.*/
+ static const char * const ignored_suffixes[] = {
+ "_from_arm", /* arm */
+ "_from_thumb", /* arm */
+ "_veneer", /* arm */
+ NULL
+ };
+
+ /* Symbol names that contain the following are ignored.*/
+ static const char * const ignored_matches[] = {
+ ".long_branch.", /* ppc stub */
+ ".plt_branch.", /* ppc stub */
+ NULL
+ };
+
+ const char * const *p;
+
+ for (p = ignored_symbols; *p; p++)
+ if (!strcmp(name, *p))
+ return true;
+
+ for (p = ignored_prefixes; *p; p++)
+ if (!strncmp(name, *p, strlen(*p)))
+ return true;
+
+ for (p = ignored_suffixes; *p; p++) {
+ int l = strlen(name) - strlen(*p);
+
+ if (l >= 0 && !strcmp(name + l, *p))
+ return true;
+ }
+
+ for (p = ignored_matches; *p; p++) {
+ if (strstr(name, *p))
+ return true;
+ }
+
+ if (type == 'U' || type == 'u')
+ return true;
+ /* exclude debugging symbols */
+ if (type == 'N' || type == 'n')
+ return true;
+
+ if (toupper(type) == 'A') {
+ /* Keep these useful absolute symbols */
+ if (strcmp(name, "__kernel_syscall_via_break") &&
+ strcmp(name, "__kernel_syscall_via_epc") &&
+ strcmp(name, "__kernel_sigtramp") &&
+ strcmp(name, "__gp"))
+ return true;
+ }
+
+ return false;
+}
+
+static int test__vmlinux_matches_kallsyms(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
int err = -1;
struct rb_node *nd;
* such as __indirect_thunk_end.
*/
continue;
+ } else if (is_ignored_symbol(sym->name, sym->type)) {
+ /*
+ * Ignore hidden symbols, see scripts/kallsyms.c for the details
+ */
+ continue;
} else {
pr_debug("ERR : %#" PRIx64 ": %s not on kallsyms\n",
mem_start, sym->name);
machine__exit(&vmlinux);
return err;
}
+
+DEFINE_SUITE("vmlinux symtab matches kallsyms", vmlinux_matches_kallsyms);
volatile u64 data1;
volatile u8 data2[3];
+#ifndef __s390x__
static int wp_read(int fd, long long *count, int size)
{
int ret = read(fd, count, size);
return fd;
}
+#endif
-static int wp_ro_test(void)
+static int test__wp_ro(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#if defined(__s390x__) || defined(__x86_64__) || defined(__i386__)
+ return TEST_SKIP;
+#else
int fd;
unsigned long tmp, tmp1 = rand();
close(fd);
return 0;
+#endif
}
-static int wp_wo_test(void)
+static int test__wp_wo(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#if defined(__s390x__)
+ return TEST_SKIP;
+#else
int fd;
unsigned long tmp, tmp1 = rand();
close(fd);
return 0;
+#endif
}
-static int wp_rw_test(void)
+static int test__wp_rw(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#if defined(__s390x__)
+ return TEST_SKIP;
+#else
int fd;
unsigned long tmp, tmp1 = rand();
close(fd);
return 0;
+#endif
}
-static int wp_modify_test(void)
+static int test__wp_modify(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
+#if defined(__s390x__)
+ return TEST_SKIP;
+#else
int fd, ret;
unsigned long tmp = rand();
struct perf_event_attr new_attr;
close(fd);
return 0;
-}
-
-static bool wp_ro_supported(void)
-{
-#if defined (__x86_64__) || defined (__i386__)
- return false;
-#else
- return true;
-#endif
-}
-
-static const char *wp_ro_skip_msg(void)
-{
-#if defined (__x86_64__) || defined (__i386__)
- return "missing hardware support";
-#else
- return NULL;
#endif
}
-static struct {
- const char *desc;
- int (*target_func)(void);
- bool (*is_supported)(void);
- const char *(*skip_msg)(void);
-} wp_testcase_table[] = {
- {
- .desc = "Read Only Watchpoint",
- .target_func = &wp_ro_test,
- .is_supported = &wp_ro_supported,
- .skip_msg = &wp_ro_skip_msg,
- },
- {
- .desc = "Write Only Watchpoint",
- .target_func = &wp_wo_test,
- },
- {
- .desc = "Read / Write Watchpoint",
- .target_func = &wp_rw_test,
- },
- {
- .desc = "Modify Watchpoint",
- .target_func = &wp_modify_test,
- },
+static struct test_case wp_tests[] = {
+ TEST_CASE_REASON("Read Only Watchpoint", wp_ro, "missing hardware support"),
+ TEST_CASE_REASON("Write Only Watchpoint", wp_wo, "missing hardware support"),
+ TEST_CASE_REASON("Read / Write Watchpoint", wp_rw, "missing hardware support"),
+ TEST_CASE_REASON("Modify Watchpoint", wp_modify, "missing hardware support"),
+ { .name = NULL, }
};
-int test__wp_subtest_get_nr(void)
-{
- return (int)ARRAY_SIZE(wp_testcase_table);
-}
-
-const char *test__wp_subtest_get_desc(int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(wp_testcase_table))
- return NULL;
- return wp_testcase_table[i].desc;
-}
-
-const char *test__wp_subtest_skip_reason(int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(wp_testcase_table))
- return NULL;
- if (!wp_testcase_table[i].skip_msg)
- return NULL;
- return wp_testcase_table[i].skip_msg();
-}
-
-int test__wp(struct test *test __maybe_unused, int i)
-{
- if (i < 0 || i >= (int)ARRAY_SIZE(wp_testcase_table))
- return TEST_FAIL;
-
- if (wp_testcase_table[i].is_supported &&
- !wp_testcase_table[i].is_supported())
- return TEST_SKIP;
-
- return !wp_testcase_table[i].target_func() ? TEST_OK : TEST_FAIL;
-}
-
-/* The s390 so far does not have support for
- * instruction breakpoint using the perf_event_open() system call.
- */
-bool test__wp_is_supported(void)
-{
-#if defined(__s390x__)
- return false;
-#else
- return true;
-#endif
-}
+struct test_suite suite__wp = {
+ .desc = "Watchpoint",
+ .test_cases = wp_tests,
+};
extern struct strarray strarray__socket_families;
+extern struct strarray strarray__socket_level;
+
/**
* augmented_arg: extra payload for syscall pointer arguments
size_t syscall_arg__scnprintf_socket_protocol(char *bf, size_t size, struct syscall_arg *arg);
#define SCA_SK_PROTO syscall_arg__scnprintf_socket_protocol
+size_t syscall_arg__scnprintf_socket_level(char *bf, size_t size, struct syscall_arg *arg);
+#define SCA_SK_LEVEL syscall_arg__scnprintf_socket_level
+
size_t syscall_arg__scnprintf_statx_flags(char *bf, size_t size, struct syscall_arg *arg);
#define SCA_STATX_FLAGS syscall_arg__scnprintf_statx_flags
#define SOL_KCM 281
#define SOL_TLS 282
#define SOL_XDP 283
+#define SOL_MPTCP 284
+#define SOL_MCTP 285
/* IPX options */
#define IPX_TYPE 1
#include <sys/un.h>
#include <arpa/inet.h>
-#include "trace/beauty/generated/socket_arrays.c"
+#include "trace/beauty/generated/sockaddr.c"
DEFINE_STRARRAY(socket_families, "PF_");
static size_t af_inet__scnprintf(struct sockaddr *sa, char *bf, size_t size)
--- /dev/null
+#!/bin/sh
+# SPDX-License-Identifier: LGPL-2.1
+
+# This one uses a copy from the kernel sources headers that is in a
+# place used just for these tools/perf/beauty/ usage, we shouldn't not
+# put it in tools/include/linux otherwise they would be used in the
+# normal compiler building process and would drag needless stuff from the
+# kernel.
+
+# When what these scripts need is already in tools/include/ then use it,
+# otherwise grab and check the copy from the kernel sources just for these
+# string table building scripts.
+
+[ $# -eq 1 ] && header_dir=$1 || header_dir=tools/perf/trace/beauty/include/linux/
+
+printf "static const char *socket_families[] = {\n"
+# #define AF_LOCAL 1 /* POSIX name for AF_UNIX */
+regex='^#define[[:space:]]+AF_(\w+)[[:space:]]+([[:digit:]]+).*'
+
+egrep $regex ${header_dir}/socket.h | \
+ sed -r "s/$regex/\2 \1/g" | \
+ xargs printf "\t[%s] = \"%s\",\n" | \
+ egrep -v "\"(UNIX|MAX)\""
+printf "};\n"
#include <sys/types.h>
#include <sys/socket.h>
+#include "trace/beauty/generated/socket.c"
+
static size_t socket__scnprintf_ipproto(int protocol, char *bf, size_t size, bool show_prefix)
{
-#include "trace/beauty/generated/socket_ipproto_array.c"
static DEFINE_STRARRAY(socket_ipproto, "IPPROTO_");
return strarray__scnprintf(&strarray__socket_ipproto, bf, size, "%d", show_prefix, protocol);
return syscall_arg__scnprintf_int(bf, size, arg);
}
+
+static size_t socket__scnprintf_level(int level, char *bf, size_t size, bool show_prefix)
+{
+#if defined(__alpha__) || defined(__hppa__) || defined(__mips__) || defined(__sparc__)
+ const int sol_socket = 0xffff;
+#else
+ const int sol_socket = 1;
+#endif
+ if (level == sol_socket)
+ return scnprintf(bf, size, "%sSOCKET", show_prefix ? "SOL_" : "");
+
+ return strarray__scnprintf(&strarray__socket_level, bf, size, "%d", show_prefix, level);
+}
+
+size_t syscall_arg__scnprintf_socket_level(char *bf, size_t size, struct syscall_arg *arg)
+{
+ return socket__scnprintf_level(arg->val, bf, size, arg->show_string_prefix);
+}
#!/bin/sh
# SPDX-License-Identifier: LGPL-2.1
-# This one uses a copy from the kernel sources headers that is in a
-# place used just for these tools/perf/beauty/ usage, we shouldn't not
-# put it in tools/include/linux otherwise they would be used in the
-# normal compiler building process and would drag needless stuff from the
-# kernel.
+if [ $# -gt 0 ] ; then
+ uapi_header_dir=$1
+ beauty_header_dir=$2
+else
+ uapi_header_dir=tools/include/uapi/linux/
+ beauty_header_dir=tools/perf/trace/beauty/include/linux/
+fi
-# When what these scripts need is already in tools/include/ then use it,
-# otherwise grab and check the copy from the kernel sources just for these
-# string table building scripts.
+printf "static const char *socket_ipproto[] = {\n"
+ipproto_regex='^[[:space:]]+IPPROTO_(\w+)[[:space:]]+=[[:space:]]+([[:digit:]]+),.*'
-[ $# -eq 1 ] && header_dir=$1 || header_dir=tools/perf/trace/beauty/include/linux/
+egrep $ipproto_regex ${uapi_header_dir}/in.h | \
+ sed -r "s/$ipproto_regex/\2 \1/g" | \
+ sort -n | xargs printf "\t[%s] = \"%s\",\n"
+printf "};\n\n"
-printf "static const char *socket_families[] = {\n"
-# #define AF_LOCAL 1 /* POSIX name for AF_UNIX */
-regex='^#define[[:space:]]+AF_(\w+)[[:space:]]+([[:digit:]]+).*'
+printf "static const char *socket_level[] = {\n"
+socket_level_regex='^#define[[:space:]]+SOL_(\w+)[[:space:]]+([[:digit:]]+)([[:space:]]+\/.*)?'
-egrep $regex ${header_dir}/socket.h | \
- sed -r "s/$regex/\2 \1/g" | \
- xargs printf "\t[%s] = \"%s\",\n" | \
- egrep -v "\"(UNIX|MAX)\""
-printf "};\n"
+egrep $socket_level_regex ${beauty_header_dir}/socket.h | \
+ sed -r "s/$socket_level_regex/\2 \1/g" | \
+ sort -n | xargs printf "\t[%s] = \"%s\",\n"
+printf "};\n\n"
+
+printf 'DEFINE_STRARRAY(socket_level, "SOL_");\n'
+++ /dev/null
-#!/bin/sh
-# SPDX-License-Identifier: LGPL-2.1
-
-[ $# -eq 1 ] && header_dir=$1 || header_dir=tools/include/uapi/linux/
-
-printf "static const char *socket_ipproto[] = {\n"
-regex='^[[:space:]]+IPPROTO_(\w+)[[:space:]]+=[[:space:]]+([[:digit:]]+),.*'
-
-egrep $regex ${header_dir}/in.h | \
- sed -r "s/$regex/\2 \1/g" | \
- sort | xargs printf "\t[%s] = \"%s\",\n"
-printf "};\n"
#undef __HPP_SORT_ACC_FN
#undef __HPP_SORT_RAW_FN
+static void fmt_free(struct perf_hpp_fmt *fmt)
+{
+ /*
+ * At this point fmt should be completely
+ * unhooked, if not it's a bug.
+ */
+ BUG_ON(!list_empty(&fmt->list));
+ BUG_ON(!list_empty(&fmt->sort_list));
+
+ if (fmt->free)
+ fmt->free(fmt);
+}
void perf_hpp__init(void)
{
list_add(&format->sort_list, &list->sorts);
}
-void perf_hpp__column_unregister(struct perf_hpp_fmt *format)
+static void perf_hpp__column_unregister(struct perf_hpp_fmt *format)
{
list_del_init(&format->list);
+ fmt_free(format);
}
void perf_hpp__cancel_cumulate(void)
}
-static void fmt_free(struct perf_hpp_fmt *fmt)
-{
- /*
- * At this point fmt should be completely
- * unhooked, if not it's a bug.
- */
- BUG_ON(!list_empty(&fmt->list));
- BUG_ON(!list_empty(&fmt->sort_list));
-
- if (fmt->free)
- fmt->free(fmt);
-}
-
void perf_hpp__reset_output_field(struct perf_hpp_list *list)
{
struct perf_hpp_fmt *fmt, *tmp;
perf-y += branch.o
perf-y += mem2node.o
perf-y += clockid.o
+perf-y += list_sort.o
perf-$(CONFIG_LIBBPF) += bpf-loader.o
perf-$(CONFIG_LIBBPF) += bpf_map.o
perf-y += perf-hooks.o
perf-$(CONFIG_LIBBPF) += bpf-event.o
+perf-$(CONFIG_LIBBPF) += bpf-utils.o
perf-$(CONFIG_CXX) += c++/
$(OUTPUT)util/vsprintf.o: ../lib/vsprintf.c FORCE
$(call rule_mkdir)
$(call if_changed_dep,cc_o_c)
+
+$(OUTPUT)util/list_sort.o: ../lib/list_sort.c FORCE
+ $(call rule_mkdir)
+ $(call if_changed_dep,cc_o_c)
#include "evsel.h"
#include "evlist.h"
#include "bpf-event.h"
+#include "bpf-utils.h"
#include "block-range.h"
#include "string2.h"
#include "util/event.h"
#include "arch/mips/annotate/instructions.c"
#include "arch/x86/annotate/instructions.c"
#include "arch/powerpc/annotate/instructions.c"
+#include "arch/riscv64/annotate/instructions.c"
#include "arch/s390/annotate/instructions.c"
#include "arch/sparc/annotate/instructions.c"
.init = x86__annotate_init,
.instructions = x86__instructions,
.nr_instructions = ARRAY_SIZE(x86__instructions),
- .ins_is_fused = x86__ins_is_fused,
.objdump = {
.comment_char = '#',
},
.init = powerpc__annotate_init,
},
{
+ .name = "riscv64",
+ .init = riscv64__annotate_init,
+ },
+ {
.name = "s390",
.init = s390__annotate_init,
.objdump = {
return ins__scnprintf(&dl->ins, bf, size, &dl->ops, max_ins_name);
}
+void annotation__init(struct annotation *notes)
+{
+ pthread_mutex_init(¬es->lock, NULL);
+}
+
+void annotation__exit(struct annotation *notes)
+{
+ annotated_source__delete(notes->src);
+ pthread_mutex_destroy(¬es->lock);
+}
+
static void annotation_line__add(struct annotation_line *al, struct list_head *head)
{
list_add_tail(&al->node, head);
{
struct annotation *notes = symbol__annotation(sym);
struct annotation_options *opts = args->options;
- struct bpf_prog_info_linear *info_linear;
struct bpf_prog_linfo *prog_linfo = NULL;
struct bpf_prog_info_node *info_node;
int len = sym->end - sym->start;
disassembler_ftype disassemble;
struct map *map = args->ms.map;
+ struct perf_bpil *info_linear;
struct disassemble_info info;
struct dso *dso = map->dso;
int pc = 0, count, sub_id;
notes->nr_events = nr_pcnt;
annotation__update_column_widths(notes);
- sym->annotate2 = true;
+ sym->annotate2 = 1;
return 0;
struct annotated_source *src;
};
+void annotation__init(struct annotation *notes);
+void annotation__exit(struct annotation *notes);
+
static inline int annotation__cycles_width(struct annotation *notes)
{
if (notes->have_cycles && notes->options->show_minmax_cycle)
u64 payload, ip;
memset(&decoder->record, 0x0, sizeof(decoder->record));
+ decoder->record.context_id = (u64)-1;
while (1) {
err = arm_spe_get_next_packet(decoder);
case ARM_SPE_COUNTER:
break;
case ARM_SPE_CONTEXT:
+ decoder->record.context_id = payload;
break;
case ARM_SPE_OP_TYPE:
if (idx == SPE_OP_PKT_HDR_CLASS_LD_ST_ATOMIC) {
u64 timestamp;
u64 virt_addr;
u64 phys_addr;
+ u64 context_id;
};
struct arm_spe_insn;
#include "arm-spe-pkt-decoder.h"
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define le16_to_cpu bswap_16
#define le32_to_cpu bswap_32
#define le64_to_cpu bswap_64
u8 timeless_decoding;
u8 data_queued;
+ u64 sample_type;
u8 sample_flc;
u8 sample_llc;
u8 sample_tlb;
u64 kernel_start;
unsigned long num_events;
+ u8 use_ctx_pkt_for_pid;
};
struct arm_spe_queue {
const char *color = PERF_COLOR_BLUE;
color_fprintf(stdout, color,
- ". ... ARM SPE data: size %zu bytes\n",
+ ". ... ARM SPE data: size %#zx bytes\n",
len);
while (len) {
PERF_RECORD_MISC_USER;
}
+static void arm_spe_set_pid_tid_cpu(struct arm_spe *spe,
+ struct auxtrace_queue *queue)
+{
+ struct arm_spe_queue *speq = queue->priv;
+ pid_t tid;
+
+ tid = machine__get_current_tid(spe->machine, speq->cpu);
+ if (tid != -1) {
+ speq->tid = tid;
+ thread__zput(speq->thread);
+ } else
+ speq->tid = queue->tid;
+
+ if ((!speq->thread) && (speq->tid != -1)) {
+ speq->thread = machine__find_thread(spe->machine, -1,
+ speq->tid);
+ }
+
+ if (speq->thread) {
+ speq->pid = speq->thread->pid_;
+ if (queue->cpu == -1)
+ speq->cpu = speq->thread->cpu;
+ }
+}
+
+static int arm_spe_set_tid(struct arm_spe_queue *speq, pid_t tid)
+{
+ struct arm_spe *spe = speq->spe;
+ int err = machine__set_current_tid(spe->machine, speq->cpu, -1, tid);
+
+ if (err)
+ return err;
+
+ arm_spe_set_pid_tid_cpu(spe, &spe->queues.queue_array[speq->queue_nr]);
+
+ return 0;
+}
+
static void arm_spe_prep_sample(struct arm_spe *spe,
struct arm_spe_queue *speq,
union perf_event *event,
event->sample.header.size = sizeof(struct perf_event_header);
}
+static int arm_spe__inject_event(union perf_event *event, struct perf_sample *sample, u64 type)
+{
+ event->header.size = perf_event__sample_event_size(sample, type, 0);
+ return perf_event__synthesize_sample(event, type, 0, sample);
+}
+
static inline int
arm_spe_deliver_synth_event(struct arm_spe *spe,
struct arm_spe_queue *speq __maybe_unused,
{
int ret;
+ if (spe->synth_opts.inject) {
+ ret = arm_spe__inject_event(event, sample, spe->sample_type);
+ if (ret)
+ return ret;
+ }
+
ret = perf_session__deliver_synth_event(spe->session, event, sample);
if (ret)
pr_err("ARM SPE: failed to deliver event, error %d\n", ret);
* can correlate samples between Arm SPE trace data and other
* perf events with correct time ordering.
*/
+
+ /*
+ * Update pid/tid info.
+ */
+ record = &speq->decoder->record;
+ if (!spe->timeless_decoding && record->context_id != (u64)-1) {
+ ret = arm_spe_set_tid(speq, record->context_id);
+ if (ret)
+ return ret;
+
+ spe->use_ctx_pkt_for_pid = true;
+ }
+
ret = arm_spe_sample(speq);
if (ret)
return ret;
return timeless_decoding;
}
-static void arm_spe_set_pid_tid_cpu(struct arm_spe *spe,
- struct auxtrace_queue *queue)
-{
- struct arm_spe_queue *speq = queue->priv;
- pid_t tid;
-
- tid = machine__get_current_tid(spe->machine, speq->cpu);
- if (tid != -1) {
- speq->tid = tid;
- thread__zput(speq->thread);
- } else
- speq->tid = queue->tid;
-
- if ((!speq->thread) && (speq->tid != -1)) {
- speq->thread = machine__find_thread(spe->machine, -1,
- speq->tid);
- }
-
- if (speq->thread) {
- speq->pid = speq->thread->pid_;
- if (queue->cpu == -1)
- speq->cpu = speq->thread->cpu;
- }
-}
-
static int arm_spe_process_queues(struct arm_spe *spe, u64 timestamp)
{
unsigned int queue_nr;
ts = timestamp;
}
- arm_spe_set_pid_tid_cpu(spe, queue);
+ /*
+ * A previous context-switch event has set pid/tid in the machine's context, so
+ * here we need to update the pid/tid in the thread and SPE queue.
+ */
+ if (!spe->use_ctx_pkt_for_pid)
+ arm_spe_set_pid_tid_cpu(spe, queue);
ret = arm_spe_run_decoder(speq, &ts);
if (ret < 0) {
return 0;
}
+static int arm_spe_context_switch(struct arm_spe *spe, union perf_event *event,
+ struct perf_sample *sample)
+{
+ pid_t pid, tid;
+ int cpu;
+
+ if (!(event->header.misc & PERF_RECORD_MISC_SWITCH_OUT))
+ return 0;
+
+ pid = event->context_switch.next_prev_pid;
+ tid = event->context_switch.next_prev_tid;
+ cpu = sample->cpu;
+
+ if (tid == -1)
+ pr_warning("context_switch event has no tid\n");
+
+ return machine__set_current_tid(spe->machine, cpu, pid, tid);
+}
+
static int arm_spe_process_event(struct perf_session *session,
union perf_event *event,
struct perf_sample *sample,
}
} else if (timestamp) {
err = arm_spe_process_queues(spe, timestamp);
+ if (err)
+ return err;
+
+ if (!spe->use_ctx_pkt_for_pid &&
+ (event->header.type == PERF_RECORD_SWITCH_CPU_WIDE ||
+ event->header.type == PERF_RECORD_SWITCH))
+ err = arm_spe_context_switch(spe, event, sample);
}
return err;
return arm_spe_process_timeless_queues(spe, -1,
MAX_TIMESTAMP - 1);
- return arm_spe_process_queues(spe, MAX_TIMESTAMP);
+ ret = arm_spe_process_queues(spe, MAX_TIMESTAMP);
+ if (ret)
+ return ret;
+
+ if (!spe->use_ctx_pkt_for_pid)
+ ui__warning("Arm SPE CONTEXT packets not found in the traces.\n"
+ "Matching of TIDs to SPE events could be inaccurate.\n");
+
+ return 0;
}
static void arm_spe_free_queue(void *priv)
else
attr.sample_type |= PERF_SAMPLE_TIME;
+ spe->sample_type = attr.sample_type;
+
attr.exclude_user = evsel->core.attr.exclude_user;
attr.exclude_kernel = evsel->core.attr.exclude_kernel;
attr.exclude_hv = evsel->core.attr.exclude_hv;
case 'q':
synth_opts->quick += 1;
break;
+ case 'A':
+ synth_opts->approx_ipc = true;
+ break;
case 'Z':
synth_opts->timeless_decoding = true;
break;
#define AUXTRACE_ERR_FLG_DATA_LOST (1 << ('l' - 'a'))
#define AUXTRACE_LOG_FLG_ALL_PERF_EVTS (1 << ('a' - 'a'))
+#define AUXTRACE_LOG_FLG_USE_STDOUT (1 << ('o' - 'a'))
/**
* struct itrace_synth_opts - AUX area tracing synthesis options.
* @thread_stack: feed branches to the thread_stack
* @last_branch: add branch context to 'instruction' events
* @add_last_branch: add branch context to existing event records
+ * @approx_ipc: approximate IPC
* @flc: whether to synthesize first level cache events
* @llc: whether to synthesize last level cache events
* @tlb: whether to synthesize TLB events
bool thread_stack;
bool last_branch;
bool add_last_branch;
+ bool approx_ipc;
bool flc;
bool llc;
bool tlb;
" d[flags]: create a debug log\n" \
" each flag must be preceded by + or -\n" \
" log flags are: a (all perf events)\n" \
+" o (output to stdout)\n" \
" f: synthesize first level cache events\n" \
" m: synthesize last level cache events\n" \
" t: synthesize TLB events\n" \
" L[len]: synthesize last branch entries on existing event records\n" \
" sNUMBER: skip initial number of events\n" \
" q: quicker (less detailed) decoding\n" \
+" A: approximate IPC\n" \
+" Z: prefer to ignore timestamps (so-called \"timeless\" decoding)\n" \
" PERIOD[ns|us|ms|i|t]: specify period to sample stream\n" \
" concatenate multiple options. Default is ibxwpe or cewp\n"
#include <internal/lib.h>
#include <symbol/kallsyms.h>
#include "bpf-event.h"
+#include "bpf-utils.h"
#include "debug.h"
#include "dso.h"
#include "symbol.h"
return err ? ERR_PTR(err) : btf;
}
-#define ptr_to_u64(ptr) ((__u64)(unsigned long)(ptr))
+struct bpf_program * __weak
+bpf_object__next_program(const struct bpf_object *obj, struct bpf_program *prev)
+{
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+ return bpf_program__next(prev, obj);
+#pragma GCC diagnostic pop
+}
+
+struct bpf_map * __weak
+bpf_object__next_map(const struct bpf_object *obj, const struct bpf_map *prev)
+{
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+ return bpf_map__next(prev, obj);
+#pragma GCC diagnostic pop
+}
+
+const void * __weak
+btf__raw_data(const struct btf *btf_ro, __u32 *size)
+{
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+ return btf__get_raw_data(btf_ro, size);
+#pragma GCC diagnostic pop
+}
static int snprintf_hex(char *buf, size_t size, unsigned char *data, size_t len)
{
union perf_event *event,
struct perf_sample *sample __maybe_unused)
{
- struct bpf_prog_info_linear *info_linear;
struct bpf_prog_info_node *info_node;
struct perf_env *env = machine->env;
+ struct perf_bpil *info_linear;
int id = event->bpf.id;
unsigned int i;
node->data_size = data_size;
memcpy(node->data, data, data_size);
- perf_env__insert_btf(env, node);
+ if (!perf_env__insert_btf(env, node)) {
+ /* Insertion failed because of a duplicate. */
+ free(node);
+ return -1;
+ }
return 0;
}
{
struct perf_record_ksymbol *ksymbol_event = &event->ksymbol;
struct perf_record_bpf_event *bpf_event = &event->bpf;
- struct bpf_prog_info_linear *info_linear;
struct perf_tool *tool = session->tool;
struct bpf_prog_info_node *info_node;
+ struct perf_bpil *info_linear;
struct bpf_prog_info *info;
struct btf *btf = NULL;
struct perf_env *env;
*/
env = session->data ? &session->header.env : &perf_env;
- arrays = 1UL << BPF_PROG_INFO_JITED_KSYMS;
- arrays |= 1UL << BPF_PROG_INFO_JITED_FUNC_LENS;
- arrays |= 1UL << BPF_PROG_INFO_FUNC_INFO;
- arrays |= 1UL << BPF_PROG_INFO_PROG_TAGS;
- arrays |= 1UL << BPF_PROG_INFO_JITED_INSNS;
- arrays |= 1UL << BPF_PROG_INFO_LINE_INFO;
- arrays |= 1UL << BPF_PROG_INFO_JITED_LINE_INFO;
+ arrays = 1UL << PERF_BPIL_JITED_KSYMS;
+ arrays |= 1UL << PERF_BPIL_JITED_FUNC_LENS;
+ arrays |= 1UL << PERF_BPIL_FUNC_INFO;
+ arrays |= 1UL << PERF_BPIL_PROG_TAGS;
+ arrays |= 1UL << PERF_BPIL_JITED_INSNS;
+ arrays |= 1UL << PERF_BPIL_LINE_INFO;
+ arrays |= 1UL << PERF_BPIL_JITED_LINE_INFO;
- info_linear = bpf_program__get_prog_info_linear(fd, arrays);
+ info_linear = get_bpf_prog_info_linear(fd, arrays);
if (IS_ERR_OR_NULL(info_linear)) {
info_linear = NULL;
pr_debug("%s: failed to get BPF program info. aborting\n", __func__);
static void perf_env__add_bpf_info(struct perf_env *env, u32 id)
{
- struct bpf_prog_info_linear *info_linear;
struct bpf_prog_info_node *info_node;
+ struct perf_bpil *info_linear;
struct btf *btf = NULL;
u64 arrays;
u32 btf_id;
if (fd < 0)
return;
- arrays = 1UL << BPF_PROG_INFO_JITED_KSYMS;
- arrays |= 1UL << BPF_PROG_INFO_JITED_FUNC_LENS;
- arrays |= 1UL << BPF_PROG_INFO_FUNC_INFO;
- arrays |= 1UL << BPF_PROG_INFO_PROG_TAGS;
- arrays |= 1UL << BPF_PROG_INFO_JITED_INSNS;
- arrays |= 1UL << BPF_PROG_INFO_LINE_INFO;
- arrays |= 1UL << BPF_PROG_INFO_JITED_LINE_INFO;
+ arrays = 1UL << PERF_BPIL_JITED_KSYMS;
+ arrays |= 1UL << PERF_BPIL_JITED_FUNC_LENS;
+ arrays |= 1UL << PERF_BPIL_FUNC_INFO;
+ arrays |= 1UL << PERF_BPIL_PROG_TAGS;
+ arrays |= 1UL << PERF_BPIL_JITED_INSNS;
+ arrays |= 1UL << PERF_BPIL_LINE_INFO;
+ arrays |= 1UL << PERF_BPIL_JITED_LINE_INFO;
- info_linear = bpf_program__get_prog_info_linear(fd, arrays);
+ info_linear = get_bpf_prog_info_linear(fd, arrays);
if (IS_ERR_OR_NULL(info_linear)) {
pr_debug("%s: failed to get BPF program info. aborting\n", __func__);
goto out;
synthesize_bpf_prog_name(name, KSYM_NAME_LEN, info, btf, 0);
fprintf(fp, "# bpf_prog_info %u: %s addr 0x%llx size %u\n",
info->id, name, prog_addrs[0], prog_lens[0]);
- return;
+ goto out;
}
fprintf(fp, "# bpf_prog_info %u:\n", info->id);
fprintf(fp, "# \tsub_prog %u: %s addr 0x%llx size %u\n",
i, name, prog_addrs[i], prog_lens[i]);
}
+out:
+ btf__free(btf);
}
struct target;
struct bpf_prog_info_node {
- struct bpf_prog_info_linear *info_linear;
+ struct perf_bpil *info_linear;
struct rb_node rb_node;
};
--- /dev/null
+// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+#ifndef _GNU_SOURCE
+#define _GNU_SOURCE
+#endif
+
+#include <errno.h>
+#include <stdlib.h>
+#include <linux/err.h>
+#include <linux/kernel.h>
+#include <bpf/bpf.h>
+#include "bpf-utils.h"
+#include "debug.h"
+
+struct bpil_array_desc {
+ int array_offset; /* e.g. offset of jited_prog_insns */
+ int count_offset; /* e.g. offset of jited_prog_len */
+ int size_offset; /* > 0: offset of rec size,
+ * < 0: fix size of -size_offset
+ */
+};
+
+static struct bpil_array_desc bpil_array_desc[] = {
+ [PERF_BPIL_JITED_INSNS] = {
+ offsetof(struct bpf_prog_info, jited_prog_insns),
+ offsetof(struct bpf_prog_info, jited_prog_len),
+ -1,
+ },
+ [PERF_BPIL_XLATED_INSNS] = {
+ offsetof(struct bpf_prog_info, xlated_prog_insns),
+ offsetof(struct bpf_prog_info, xlated_prog_len),
+ -1,
+ },
+ [PERF_BPIL_MAP_IDS] = {
+ offsetof(struct bpf_prog_info, map_ids),
+ offsetof(struct bpf_prog_info, nr_map_ids),
+ -(int)sizeof(__u32),
+ },
+ [PERF_BPIL_JITED_KSYMS] = {
+ offsetof(struct bpf_prog_info, jited_ksyms),
+ offsetof(struct bpf_prog_info, nr_jited_ksyms),
+ -(int)sizeof(__u64),
+ },
+ [PERF_BPIL_JITED_FUNC_LENS] = {
+ offsetof(struct bpf_prog_info, jited_func_lens),
+ offsetof(struct bpf_prog_info, nr_jited_func_lens),
+ -(int)sizeof(__u32),
+ },
+ [PERF_BPIL_FUNC_INFO] = {
+ offsetof(struct bpf_prog_info, func_info),
+ offsetof(struct bpf_prog_info, nr_func_info),
+ offsetof(struct bpf_prog_info, func_info_rec_size),
+ },
+ [PERF_BPIL_LINE_INFO] = {
+ offsetof(struct bpf_prog_info, line_info),
+ offsetof(struct bpf_prog_info, nr_line_info),
+ offsetof(struct bpf_prog_info, line_info_rec_size),
+ },
+ [PERF_BPIL_JITED_LINE_INFO] = {
+ offsetof(struct bpf_prog_info, jited_line_info),
+ offsetof(struct bpf_prog_info, nr_jited_line_info),
+ offsetof(struct bpf_prog_info, jited_line_info_rec_size),
+ },
+ [PERF_BPIL_PROG_TAGS] = {
+ offsetof(struct bpf_prog_info, prog_tags),
+ offsetof(struct bpf_prog_info, nr_prog_tags),
+ -(int)sizeof(__u8) * BPF_TAG_SIZE,
+ },
+
+};
+
+static __u32 bpf_prog_info_read_offset_u32(struct bpf_prog_info *info,
+ int offset)
+{
+ __u32 *array = (__u32 *)info;
+
+ if (offset >= 0)
+ return array[offset / sizeof(__u32)];
+ return -(int)offset;
+}
+
+static __u64 bpf_prog_info_read_offset_u64(struct bpf_prog_info *info,
+ int offset)
+{
+ __u64 *array = (__u64 *)info;
+
+ if (offset >= 0)
+ return array[offset / sizeof(__u64)];
+ return -(int)offset;
+}
+
+static void bpf_prog_info_set_offset_u32(struct bpf_prog_info *info, int offset,
+ __u32 val)
+{
+ __u32 *array = (__u32 *)info;
+
+ if (offset >= 0)
+ array[offset / sizeof(__u32)] = val;
+}
+
+static void bpf_prog_info_set_offset_u64(struct bpf_prog_info *info, int offset,
+ __u64 val)
+{
+ __u64 *array = (__u64 *)info;
+
+ if (offset >= 0)
+ array[offset / sizeof(__u64)] = val;
+}
+
+struct perf_bpil *
+get_bpf_prog_info_linear(int fd, __u64 arrays)
+{
+ struct bpf_prog_info info = {};
+ struct perf_bpil *info_linear;
+ __u32 info_len = sizeof(info);
+ __u32 data_len = 0;
+ int i, err;
+ void *ptr;
+
+ if (arrays >> PERF_BPIL_LAST_ARRAY)
+ return ERR_PTR(-EINVAL);
+
+ /* step 1: get array dimensions */
+ err = bpf_obj_get_info_by_fd(fd, &info, &info_len);
+ if (err) {
+ pr_debug("can't get prog info: %s", strerror(errno));
+ return ERR_PTR(-EFAULT);
+ }
+
+ /* step 2: calculate total size of all arrays */
+ for (i = PERF_BPIL_FIRST_ARRAY; i < PERF_BPIL_LAST_ARRAY; ++i) {
+ bool include_array = (arrays & (1UL << i)) > 0;
+ struct bpil_array_desc *desc;
+ __u32 count, size;
+
+ desc = bpil_array_desc + i;
+
+ /* kernel is too old to support this field */
+ if (info_len < desc->array_offset + sizeof(__u32) ||
+ info_len < desc->count_offset + sizeof(__u32) ||
+ (desc->size_offset > 0 && info_len < (__u32)desc->size_offset))
+ include_array = false;
+
+ if (!include_array) {
+ arrays &= ~(1UL << i); /* clear the bit */
+ continue;
+ }
+
+ count = bpf_prog_info_read_offset_u32(&info, desc->count_offset);
+ size = bpf_prog_info_read_offset_u32(&info, desc->size_offset);
+
+ data_len += count * size;
+ }
+
+ /* step 3: allocate continuous memory */
+ data_len = roundup(data_len, sizeof(__u64));
+ info_linear = malloc(sizeof(struct perf_bpil) + data_len);
+ if (!info_linear)
+ return ERR_PTR(-ENOMEM);
+
+ /* step 4: fill data to info_linear->info */
+ info_linear->arrays = arrays;
+ memset(&info_linear->info, 0, sizeof(info));
+ ptr = info_linear->data;
+
+ for (i = PERF_BPIL_FIRST_ARRAY; i < PERF_BPIL_LAST_ARRAY; ++i) {
+ struct bpil_array_desc *desc;
+ __u32 count, size;
+
+ if ((arrays & (1UL << i)) == 0)
+ continue;
+
+ desc = bpil_array_desc + i;
+ count = bpf_prog_info_read_offset_u32(&info, desc->count_offset);
+ size = bpf_prog_info_read_offset_u32(&info, desc->size_offset);
+ bpf_prog_info_set_offset_u32(&info_linear->info,
+ desc->count_offset, count);
+ bpf_prog_info_set_offset_u32(&info_linear->info,
+ desc->size_offset, size);
+ bpf_prog_info_set_offset_u64(&info_linear->info,
+ desc->array_offset,
+ ptr_to_u64(ptr));
+ ptr += count * size;
+ }
+
+ /* step 5: call syscall again to get required arrays */
+ err = bpf_obj_get_info_by_fd(fd, &info_linear->info, &info_len);
+ if (err) {
+ pr_debug("can't get prog info: %s", strerror(errno));
+ free(info_linear);
+ return ERR_PTR(-EFAULT);
+ }
+
+ /* step 6: verify the data */
+ for (i = PERF_BPIL_FIRST_ARRAY; i < PERF_BPIL_LAST_ARRAY; ++i) {
+ struct bpil_array_desc *desc;
+ __u32 v1, v2;
+
+ if ((arrays & (1UL << i)) == 0)
+ continue;
+
+ desc = bpil_array_desc + i;
+ v1 = bpf_prog_info_read_offset_u32(&info, desc->count_offset);
+ v2 = bpf_prog_info_read_offset_u32(&info_linear->info,
+ desc->count_offset);
+ if (v1 != v2)
+ pr_warning("%s: mismatch in element count\n", __func__);
+
+ v1 = bpf_prog_info_read_offset_u32(&info, desc->size_offset);
+ v2 = bpf_prog_info_read_offset_u32(&info_linear->info,
+ desc->size_offset);
+ if (v1 != v2)
+ pr_warning("%s: mismatch in rec size\n", __func__);
+ }
+
+ /* step 7: update info_len and data_len */
+ info_linear->info_len = sizeof(struct bpf_prog_info);
+ info_linear->data_len = data_len;
+
+ return info_linear;
+}
+
+void bpil_addr_to_offs(struct perf_bpil *info_linear)
+{
+ int i;
+
+ for (i = PERF_BPIL_FIRST_ARRAY; i < PERF_BPIL_LAST_ARRAY; ++i) {
+ struct bpil_array_desc *desc;
+ __u64 addr, offs;
+
+ if ((info_linear->arrays & (1UL << i)) == 0)
+ continue;
+
+ desc = bpil_array_desc + i;
+ addr = bpf_prog_info_read_offset_u64(&info_linear->info,
+ desc->array_offset);
+ offs = addr - ptr_to_u64(info_linear->data);
+ bpf_prog_info_set_offset_u64(&info_linear->info,
+ desc->array_offset, offs);
+ }
+}
+
+void bpil_offs_to_addr(struct perf_bpil *info_linear)
+{
+ int i;
+
+ for (i = PERF_BPIL_FIRST_ARRAY; i < PERF_BPIL_LAST_ARRAY; ++i) {
+ struct bpil_array_desc *desc;
+ __u64 addr, offs;
+
+ if ((info_linear->arrays & (1UL << i)) == 0)
+ continue;
+
+ desc = bpil_array_desc + i;
+ offs = bpf_prog_info_read_offset_u64(&info_linear->info,
+ desc->array_offset);
+ addr = offs + ptr_to_u64(info_linear->data);
+ bpf_prog_info_set_offset_u64(&info_linear->info,
+ desc->array_offset, addr);
+ }
+}
--- /dev/null
+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
+
+#ifndef __PERF_BPF_UTILS_H
+#define __PERF_BPF_UTILS_H
+
+#define ptr_to_u64(ptr) ((__u64)(unsigned long)(ptr))
+
+#ifdef HAVE_LIBBPF_SUPPORT
+
+#include <bpf/libbpf.h>
+
+/*
+ * Get bpf_prog_info in continuous memory
+ *
+ * struct bpf_prog_info has multiple arrays. The user has option to choose
+ * arrays to fetch from kernel. The following APIs provide an uniform way to
+ * fetch these data. All arrays in bpf_prog_info are stored in a single
+ * continuous memory region. This makes it easy to store the info in a
+ * file.
+ *
+ * Before writing perf_bpil to files, it is necessary to
+ * translate pointers in bpf_prog_info to offsets. Helper functions
+ * bpil_addr_to_offs() and bpil_offs_to_addr()
+ * are introduced to switch between pointers and offsets.
+ *
+ * Examples:
+ * # To fetch map_ids and prog_tags:
+ * __u64 arrays = (1UL << PERF_BPIL_MAP_IDS) |
+ * (1UL << PERF_BPIL_PROG_TAGS);
+ * struct perf_bpil *info_linear =
+ * get_bpf_prog_info_linear(fd, arrays);
+ *
+ * # To save data in file
+ * bpil_addr_to_offs(info_linear);
+ * write(f, info_linear, sizeof(*info_linear) + info_linear->data_len);
+ *
+ * # To read data from file
+ * read(f, info_linear, <proper_size>);
+ * bpil_offs_to_addr(info_linear);
+ */
+enum perf_bpil_array_types {
+ PERF_BPIL_FIRST_ARRAY = 0,
+ PERF_BPIL_JITED_INSNS = 0,
+ PERF_BPIL_XLATED_INSNS,
+ PERF_BPIL_MAP_IDS,
+ PERF_BPIL_JITED_KSYMS,
+ PERF_BPIL_JITED_FUNC_LENS,
+ PERF_BPIL_FUNC_INFO,
+ PERF_BPIL_LINE_INFO,
+ PERF_BPIL_JITED_LINE_INFO,
+ PERF_BPIL_PROG_TAGS,
+ PERF_BPIL_LAST_ARRAY,
+};
+
+struct perf_bpil {
+ /* size of struct bpf_prog_info, when the tool is compiled */
+ __u32 info_len;
+ /* total bytes allocated for data, round up to 8 bytes */
+ __u32 data_len;
+ /* which arrays are included in data */
+ __u64 arrays;
+ struct bpf_prog_info info;
+ __u8 data[];
+};
+
+struct perf_bpil *
+get_bpf_prog_info_linear(int fd, __u64 arrays);
+
+void
+bpil_addr_to_offs(struct perf_bpil *info_linear);
+
+void
+bpil_offs_to_addr(struct perf_bpil *info_linear);
+
+#endif /* HAVE_LIBBPF_SUPPORT */
+#endif /* __PERF_BPF_UTILS_H */
#include <perf/bpf_perf.h>
#include "bpf_counter.h"
+#include "bpf-utils.h"
#include "counts.h"
#include "debug.h"
#include "evsel.h"
static char *bpf_target_prog_name(int tgt_fd)
{
- struct bpf_prog_info_linear *info_linear;
struct bpf_func_info *func_info;
+ struct perf_bpil *info_linear;
const struct btf_type *t;
struct btf *btf = NULL;
char *name = NULL;
- info_linear = bpf_program__get_prog_info_linear(
- tgt_fd, 1UL << BPF_PROG_INFO_FUNC_INFO);
+ info_linear = get_bpf_prog_info_linear(tgt_fd, 1UL << PERF_BPIL_FUNC_INFO);
if (IS_ERR_OR_NULL(info_linear)) {
pr_debug("failed to get info_linear for prog FD %d\n", tgt_fd);
return NULL;
skel->rodata->num_cpu = evsel__nr_cpus(evsel);
- bpf_map__resize(skel->maps.events, evsel__nr_cpus(evsel));
- bpf_map__resize(skel->maps.fentry_readings, 1);
- bpf_map__resize(skel->maps.accum_readings, 1);
+ bpf_map__set_max_entries(skel->maps.events, evsel__nr_cpus(evsel));
+ bpf_map__set_max_entries(skel->maps.fentry_readings, 1);
+ bpf_map__set_max_entries(skel->maps.accum_readings, 1);
prog_name = bpf_target_prog_name(prog_fd);
if (!prog_name) {
return -1;
}
- bpf_map__resize(skel->maps.events, libbpf_num_possible_cpus());
+ bpf_map__set_max_entries(skel->maps.events, libbpf_num_possible_cpus());
err = bperf_leader_bpf__load(skel);
if (err) {
pr_err("Failed to load leader skeleton\n");
/* we need one copy of events per cpu for reading */
map_size = total_cpus * evlist->core.nr_entries / nr_cgroups;
- bpf_map__resize(skel->maps.events, map_size);
- bpf_map__resize(skel->maps.cgrp_idx, nr_cgroups);
+ bpf_map__set_max_entries(skel->maps.events, map_size);
+ bpf_map__set_max_entries(skel->maps.cgrp_idx, nr_cgroups);
/* previous result is saved in a per-cpu array */
map_size = evlist->core.nr_entries / nr_cgroups;
- bpf_map__resize(skel->maps.prev_readings, map_size);
+ bpf_map__set_max_entries(skel->maps.prev_readings, map_size);
/* cgroup result needs all events (per-cpu) */
map_size = evlist->core.nr_entries;
- bpf_map__resize(skel->maps.cgrp_readings, map_size);
+ bpf_map__set_max_entries(skel->maps.cgrp_readings, map_size);
set_max_rlimit();
extern void perf_clang__init(void);
extern void perf_clang__cleanup(void);
-extern int test__clang_to_IR(void);
-extern int test__clang_to_obj(void);
+struct test_suite;
+extern int test__clang_to_IR(struct test_suite *test, int subtest);
+extern int test__clang_to_obj(struct test_suite *test, int subtest);
extern int perf_clang__compile_bpf(const char *filename,
void **p_obj_buf,
static inline void perf_clang__init(void) { }
static inline void perf_clang__cleanup(void) { }
-static inline int test__clang_to_IR(void) { return -1; }
-static inline int test__clang_to_obj(void) { return -1;}
-
static inline int
perf_clang__compile_bpf(const char *filename __maybe_unused,
void **p_obj_buf __maybe_unused,
}
extern "C" {
-int test__clang_to_IR(void)
+int test__clang_to_IR(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
perf_clang_scope _scope;
return -1;
}
-int test__clang_to_obj(void)
+int test__clang_to_obj(struct test_suite *test __maybe_unused,
+ int subtest __maybe_unused)
{
perf_clang_scope _scope;
"-cc1",
"-triple", "bpf-pc-linux",
"-fsyntax-only",
- "-ferror-limit", "19",
- "-fmessage-length", "127",
"-O2",
"-nostdsysteminc",
"-nobuiltininc",
"-x", "c"};
CCArgs.append(CFlags.begin(), CFlags.end());
- CompilerInvocation *CI = tooling::newInvocation(&Diags, CCArgs);
+ CompilerInvocation *CI = tooling::newInvocation(&Diags, CCArgs
+#if CLANG_VERSION_MAJOR >= 11
+ ,/*BinaryName=*/nullptr
+#endif
+ );
FrontendOptions& Opts = CI->getFrontendOpts();
Opts.Inputs.clear();
legacy::PassManager PM;
bool NotAdded;
-#if CLANG_VERSION_MAJOR < 7
- NotAdded = TargetMachine->addPassesToEmitFile(PM, ostream,
- TargetMachine::CGFT_ObjectFile);
+ NotAdded = TargetMachine->addPassesToEmitFile(PM, ostream
+#if CLANG_VERSION_MAJOR >= 7
+ , /*DwoOut=*/nullptr
+#endif
+#if CLANG_VERSION_MAJOR < 10
+ , TargetMachine::CGFT_ObjectFile
#else
- NotAdded = TargetMachine->addPassesToEmitFile(PM, ostream, nullptr,
- TargetMachine::CGFT_ObjectFile);
+ , llvm::CGFT_ObjectFile
#endif
+ );
if (NotAdded) {
llvm::errs() << "TargetMachine can't emit a file of this type\n";
return std::unique_ptr<llvm::SmallVectorImpl<char>>(nullptr);
#include "env.h"
#include "pmu-hybrid.h"
-#define CORE_SIB_FMT \
+#define PACKAGE_CPUS_FMT \
+ "%s/devices/system/cpu/cpu%d/topology/package_cpus_list"
+#define PACKAGE_CPUS_FMT_OLD \
"%s/devices/system/cpu/cpu%d/topology/core_siblings_list"
-#define DIE_SIB_FMT \
+#define DIE_CPUS_FMT \
"%s/devices/system/cpu/cpu%d/topology/die_cpus_list"
-#define THRD_SIB_FMT \
- "%s/devices/system/cpu/cpu%d/topology/thread_siblings_list"
-#define THRD_SIB_FMT_NEW \
+#define CORE_CPUS_FMT \
"%s/devices/system/cpu/cpu%d/topology/core_cpus_list"
+#define CORE_CPUS_FMT_OLD \
+ "%s/devices/system/cpu/cpu%d/topology/thread_siblings_list"
#define NODE_ONLINE_FMT \
"%s/devices/system/node/online"
#define NODE_MEMINFO_FMT \
u32 i = 0;
int ret = -1;
- scnprintf(filename, MAXPATHLEN, CORE_SIB_FMT,
+ scnprintf(filename, MAXPATHLEN, PACKAGE_CPUS_FMT,
sysfs__mountpoint(), cpu);
+ if (access(filename, F_OK) == -1) {
+ scnprintf(filename, MAXPATHLEN, PACKAGE_CPUS_FMT_OLD,
+ sysfs__mountpoint(), cpu);
+ }
fp = fopen(filename, "r");
if (!fp)
goto try_dies;
if (p)
*p = '\0';
- for (i = 0; i < tp->core_sib; i++) {
- if (!strcmp(buf, tp->core_siblings[i]))
+ for (i = 0; i < tp->package_cpus_lists; i++) {
+ if (!strcmp(buf, tp->package_cpus_list[i]))
break;
}
- if (i == tp->core_sib) {
- tp->core_siblings[i] = buf;
- tp->core_sib++;
+ if (i == tp->package_cpus_lists) {
+ tp->package_cpus_list[i] = buf;
+ tp->package_cpus_lists++;
buf = NULL;
len = 0;
}
ret = 0;
try_dies:
- if (!tp->die_siblings)
+ if (!tp->die_cpus_list)
goto try_threads;
- scnprintf(filename, MAXPATHLEN, DIE_SIB_FMT,
+ scnprintf(filename, MAXPATHLEN, DIE_CPUS_FMT,
sysfs__mountpoint(), cpu);
fp = fopen(filename, "r");
if (!fp)
if (p)
*p = '\0';
- for (i = 0; i < tp->die_sib; i++) {
- if (!strcmp(buf, tp->die_siblings[i]))
+ for (i = 0; i < tp->die_cpus_lists; i++) {
+ if (!strcmp(buf, tp->die_cpus_list[i]))
break;
}
- if (i == tp->die_sib) {
- tp->die_siblings[i] = buf;
- tp->die_sib++;
+ if (i == tp->die_cpus_lists) {
+ tp->die_cpus_list[i] = buf;
+ tp->die_cpus_lists++;
buf = NULL;
len = 0;
}
ret = 0;
try_threads:
- scnprintf(filename, MAXPATHLEN, THRD_SIB_FMT_NEW,
+ scnprintf(filename, MAXPATHLEN, CORE_CPUS_FMT,
sysfs__mountpoint(), cpu);
if (access(filename, F_OK) == -1) {
- scnprintf(filename, MAXPATHLEN, THRD_SIB_FMT,
+ scnprintf(filename, MAXPATHLEN, CORE_CPUS_FMT_OLD,
sysfs__mountpoint(), cpu);
}
fp = fopen(filename, "r");
if (p)
*p = '\0';
- for (i = 0; i < tp->thread_sib; i++) {
- if (!strcmp(buf, tp->thread_siblings[i]))
+ for (i = 0; i < tp->core_cpus_lists; i++) {
+ if (!strcmp(buf, tp->core_cpus_list[i]))
break;
}
- if (i == tp->thread_sib) {
- tp->thread_siblings[i] = buf;
- tp->thread_sib++;
+ if (i == tp->core_cpus_lists) {
+ tp->core_cpus_list[i] = buf;
+ tp->core_cpus_lists++;
buf = NULL;
}
ret = 0;
if (!tp)
return;
- for (i = 0 ; i < tp->core_sib; i++)
- zfree(&tp->core_siblings[i]);
+ for (i = 0 ; i < tp->package_cpus_lists; i++)
+ zfree(&tp->package_cpus_list[i]);
- if (tp->die_sib) {
- for (i = 0 ; i < tp->die_sib; i++)
- zfree(&tp->die_siblings[i]);
- }
+ for (i = 0 ; i < tp->die_cpus_lists; i++)
+ zfree(&tp->die_cpus_list[i]);
- for (i = 0 ; i < tp->thread_sib; i++)
- zfree(&tp->thread_siblings[i]);
+ for (i = 0 ; i < tp->core_cpus_lists; i++)
+ zfree(&tp->core_cpus_list[i]);
free(tp);
}
if (strncmp(uts.machine, "x86_64", 6))
return false;
- scnprintf(filename, MAXPATHLEN, DIE_SIB_FMT,
+ scnprintf(filename, MAXPATHLEN, DIE_CPUS_FMT,
sysfs__mountpoint(), 0);
if (access(filename, F_OK) == -1)
return false;
tp = addr;
addr += sizeof(*tp);
- tp->core_siblings = addr;
+ tp->package_cpus_list = addr;
addr += sz;
if (has_die) {
- tp->die_siblings = addr;
+ tp->die_cpus_list = addr;
addr += sz;
}
- tp->thread_siblings = addr;
+ tp->core_cpus_list = addr;
for (i = 0; i < nr; i++) {
if (!cpu_map__has(map, i))
#include <linux/types.h>
struct cpu_topology {
- u32 core_sib;
- u32 die_sib;
- u32 thread_sib;
- char **core_siblings;
- char **die_siblings;
- char **thread_siblings;
+ /* The number of unique package_cpus_lists below. */
+ u32 package_cpus_lists;
+ /* The number of unique die_cpu_lists below. */
+ u32 die_cpus_lists;
+ /* The number of unique core_cpu_lists below. */
+ u32 core_cpus_lists;
+ /*
+ * An array of strings where each string is unique and read from
+ * /sys/devices/system/cpu/cpuX/topology/package_cpus_list. From the ABI
+ * each of these is a human-readable list of CPUs sharing the same
+ * physical_package_id. The format is like 0-3, 8-11, 14,17.
+ */
+ const char **package_cpus_list;
+ /*
+ * An array of string where each string is unique and from
+ * /sys/devices/system/cpu/cpuX/topology/die_cpus_list. From the ABI
+ * each of these is a human-readable list of CPUs within the same die.
+ * The format is like 0-3, 8-11, 14,17.
+ */
+ const char **die_cpus_list;
+ /*
+ * An array of string where each string is unique and from
+ * /sys/devices/system/cpu/cpuX/topology/core_cpus_list. From the ABI
+ * each of these is a human-readable list of CPUs within the same
+ * core. The format is like 0-3, 8-11, 14,17.
+ */
+ const char **core_cpus_list;
};
struct numa_topology_node {
fprintf(stdout, "\n");
color_fprintf(stdout, color,
- ". ... CoreSight %s Trace data: size %zu bytes\n",
+ ". ... CoreSight %s Trace data: size %#zx bytes\n",
cs_etm_decoder__get_name(etmq->decoder), buffer->size);
do {
bt_ctf_field_type_integer_set_base(type, BT_CTF_INTEGER_BASE_HEXADECIMAL))
goto err;
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
bt_ctf_field_type_set_byte_order(type, BT_CTF_BYTE_ORDER_BIG_ENDIAN);
#else
bt_ctf_field_type_set_byte_order(type, BT_CTF_BYTE_ORDER_LITTLE_ENDIAN);
#include "util/parse-sublevel-options.h"
#include <linux/ctype.h>
+#include <traceevent/event-parse.h>
+
+#define MAKE_LIBTRACEEVENT_VERSION(a, b, c) ((a)*255*255+(b)*255+(c))
+#ifndef LIBTRACEEVENT_VERSION
+/*
+ * If LIBTRACEEVENT_VERSION wasn't computed then set to version 1.1.0 that ships
+ * with the Linux kernel tools.
+ */
+#define LIBTRACEEVENT_VERSION MAKE_LIBTRACEEVENT_VERSION(1, 1, 0)
+#endif
int verbose;
int debug_peo_args;
/* Allow only verbose value in range (0, 10), otherwise set 0. */
verbose = (verbose < 0) || (verbose > 10) ? 0 : verbose;
+#if MAKE_LIBTRACEEVENT_VERSION(1, 3, 0) <= LIBTRACEEVENT_VERSION
+ if (verbose == 1)
+ tep_set_loglevel(TEP_LOG_INFO);
+ else if (verbose == 2)
+ tep_set_loglevel(TEP_LOG_DEBUG);
+ else if (verbose >= 3)
+ tep_set_loglevel(TEP_LOG_ALL);
+#endif
+
return 0;
}
#ifdef HAVE_LIBBPF_SUPPORT
#include <bpf/libbpf.h>
#include "bpf-event.h"
+#include "bpf-utils.h"
#endif
#include "compress.h"
#include "env.h"
int fd;
int status;
u32 status_seen;
- size_t file_size;
+ u64 file_size;
struct list_head open_entry;
u64 debug_frame_offset;
u64 eh_frame_hdr_offset;
#ifdef HAVE_LIBBPF_SUPPORT
#include "bpf-event.h"
+#include "bpf-utils.h"
#include <bpf/libbpf.h>
void perf_env__insert_bpf_prog_info(struct perf_env *env,
return node;
}
-void perf_env__insert_btf(struct perf_env *env, struct btf_node *btf_node)
+bool perf_env__insert_btf(struct perf_env *env, struct btf_node *btf_node)
{
struct rb_node *parent = NULL;
__u32 btf_id = btf_node->id;
struct btf_node *node;
struct rb_node **p;
+ bool ret = true;
down_write(&env->bpf_progs.lock);
p = &env->bpf_progs.btfs.rb_node;
p = &(*p)->rb_right;
} else {
pr_debug("duplicated btf %u\n", btf_id);
+ ret = false;
goto out;
}
}
env->bpf_progs.btfs_cnt++;
out:
up_write(&env->bpf_progs.lock);
+ return ret;
}
struct btf_node *perf_env__find_btf(struct perf_env *env, __u32 btf_id)
struct bpf_prog_info_node *info_node);
struct bpf_prog_info_node *perf_env__find_bpf_prog_info(struct perf_env *env,
__u32 prog_id);
-void perf_env__insert_btf(struct perf_env *env, struct btf_node *btf_node);
+bool perf_env__insert_btf(struct perf_env *env, struct btf_node *btf_node);
struct btf_node *perf_env__find_btf(struct perf_env *env, __u32 btf_id);
int perf_env__numa_node(struct perf_env *env, int cpu);
[PERF_RECORD_BPF_EVENT] = "BPF_EVENT",
[PERF_RECORD_CGROUP] = "CGROUP",
[PERF_RECORD_TEXT_POKE] = "TEXT_POKE",
+ [PERF_RECORD_AUX_OUTPUT_HW_ID] = "AUX_OUTPUT_HW_ID",
[PERF_RECORD_HEADER_ATTR] = "ATTR",
[PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE",
[PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA",
return machine__process_itrace_start_event(machine, event);
}
+int perf_event__process_aux_output_hw_id(struct perf_tool *tool __maybe_unused,
+ union perf_event *event,
+ struct perf_sample *sample __maybe_unused,
+ struct machine *machine)
+{
+ return machine__process_aux_output_hw_id_event(machine, event);
+}
+
int perf_event__process_lost_samples(struct perf_tool *tool __maybe_unused,
union perf_event *event,
struct perf_sample *sample,
event->itrace_start.pid, event->itrace_start.tid);
}
+size_t perf_event__fprintf_aux_output_hw_id(union perf_event *event, FILE *fp)
+{
+ return fprintf(fp, " hw_id: %#"PRI_lx64"\n",
+ event->aux_output_hw_id.hw_id);
+}
+
size_t perf_event__fprintf_switch(union perf_event *event, FILE *fp)
{
bool out = event->header.misc & PERF_RECORD_MISC_SWITCH_OUT;
case PERF_RECORD_TEXT_POKE:
ret += perf_event__fprintf_text_poke(event, machine, fp);
break;
+ case PERF_RECORD_AUX_OUTPUT_HW_ID:
+ ret += perf_event__fprintf_aux_output_hw_id(event, fp);
+ break;
default:
ret += fprintf(fp, "\n");
}
union perf_event *event,
struct perf_sample *sample,
struct machine *machine);
+int perf_event__process_aux_output_hw_id(struct perf_tool *tool,
+ union perf_event *event,
+ struct perf_sample *sample,
+ struct machine *machine);
int perf_event__process_switch(struct perf_tool *tool,
union perf_event *event,
struct perf_sample *sample,
size_t perf_event__fprintf_task(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_aux(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_itrace_start(union perf_event *event, FILE *fp);
+size_t perf_event__fprintf_aux_output_hw_id(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_switch(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_thread_map(union perf_event *event, FILE *fp);
size_t perf_event__fprintf_cpu_map(union perf_event *event, FILE *fp);
{
perf_evsel__init(&evsel->core, attr, idx);
evsel->tracking = !idx;
- evsel->unit = "";
+ evsel->unit = strdup("");
evsel->scale = 1.0;
evsel->max_events = ULONG_MAX;
evsel->evlist = NULL;
}
if (evsel__is_clock(evsel)) {
- /*
- * The evsel->unit points to static alias->unit
- * so it's ok to use static string in here.
- */
- static const char *unit = "msec";
-
- evsel->unit = unit;
+ free((char *)evsel->unit);
+ evsel->unit = strdup("msec");
evsel->scale = 1e-6;
}
return perf_event_paranoid_check(1);
}
-struct evsel *evsel__new_cycles(bool precise, __u32 type, __u64 config)
+struct evsel *evsel__new_cycles(bool precise __maybe_unused, __u32 type, __u64 config)
{
struct perf_event_attr attr = {
.type = type,
event_attr_init(&attr);
- if (!precise)
- goto new_event;
-
/*
* Now let the usual logic to set up the perf_event_attr defaults
* to kick in when we return and before perf_evsel__open() is called.
*/
-new_event:
evsel = evsel__new(&attr);
if (evsel == NULL)
goto out;
+ arch_evsel__fixup_new_cycles(&evsel->core.attr);
+
evsel->precise_max = true;
/* use asprintf() because free(evsel) assumes name is allocated */
if (evsel->filter == NULL)
goto out_err;
}
+ if (orig->metric_id) {
+ evsel->metric_id = strdup(orig->metric_id);
+ if (evsel->metric_id == NULL)
+ goto out_err;
+ }
evsel->cgrp = cgroup__get(orig->cgrp);
evsel->tp_format = orig->tp_format;
evsel->handler = orig->handler;
evsel->max_events = orig->max_events;
evsel->tool_event = orig->tool_event;
- evsel->unit = orig->unit;
+ free((char *)evsel->unit);
+ evsel->unit = strdup(orig->unit);
+ if (evsel->unit == NULL)
+ goto out_err;
+
evsel->scale = orig->scale;
evsel->snapshot = orig->snapshot;
evsel->per_pkg = orig->per_pkg;
return "unknown";
}
+const char *evsel__metric_id(const struct evsel *evsel)
+{
+ if (evsel->metric_id)
+ return evsel->metric_id;
+
+ if (evsel->core.attr.type == PERF_TYPE_SOFTWARE && evsel->tool_event)
+ return "duration_time";
+
+ return "unknown";
+}
+
const char *evsel__group_name(struct evsel *evsel)
{
return evsel->group_name ?: "anon group";
evsel__set_sample_bit(evsel, WEIGHT);
}
+void __weak arch_evsel__fixup_new_cycles(struct perf_event_attr *attr __maybe_unused)
+{
+}
+
/*
* The enable_on_exec/disabled value strategy:
*
zfree(&evsel->group_name);
zfree(&evsel->name);
zfree(&evsel->pmu_name);
+ zfree(&evsel->unit);
+ zfree(&evsel->metric_id);
evsel__zero_per_pkg(evsel);
hashmap__free(evsel->per_pkg_mask);
evsel->per_pkg_mask = NULL;
evsel->open_flags &= ~(unsigned long)PERF_FLAG_FD_CLOEXEC;
if (perf_missing_features.mmap2)
evsel->core.attr.mmap2 = 0;
- if (perf_missing_features.exclude_guest)
+ if (evsel->pmu && evsel->pmu->missing_features.exclude_guest)
evsel->core.attr.exclude_guest = evsel->core.attr.exclude_host = 0;
if (perf_missing_features.lbr_flags)
evsel->core.attr.branch_sample_type &= ~(PERF_SAMPLE_BRANCH_NO_FLAGS |
perf_missing_features.mmap2 = true;
pr_debug2_peo("switching off mmap2\n");
return true;
- } else if (!perf_missing_features.exclude_guest &&
- (evsel->core.attr.exclude_guest || evsel->core.attr.exclude_host)) {
- perf_missing_features.exclude_guest = true;
- pr_debug2_peo("switching off exclude_guest, exclude_host\n");
+ } else if ((evsel->core.attr.exclude_guest || evsel->core.attr.exclude_host) &&
+ (evsel->pmu == NULL || evsel->pmu->missing_features.exclude_guest)) {
+ if (evsel->pmu == NULL) {
+ evsel->pmu = evsel__find_pmu(evsel);
+ if (evsel->pmu)
+ evsel->pmu->missing_features.exclude_guest = true;
+ else {
+ /* we cannot find PMU, disable attrs now */
+ evsel->core.attr.exclude_host = false;
+ evsel->core.attr.exclude_guest = false;
+ }
+ }
+
+ if (evsel->exclude_GH) {
+ pr_debug2_peo("PMU has no exclude_host/guest support, bailing out\n");
+ return false;
+ }
+ if (!perf_missing_features.exclude_guest) {
+ perf_missing_features.exclude_guest = true;
+ pr_debug2_peo("switching off exclude_guest, exclude_host\n");
+ }
return true;
} else if (!perf_missing_features.sample_id_all) {
perf_missing_features.sample_id_all = true;
data->weight = *array;
}
+u64 evsel__bitfield_swap_branch_flags(u64 value)
+{
+ u64 new_val = 0;
+
+ /*
+ * branch_flags
+ * union {
+ * u64 values;
+ * struct {
+ * mispred:1 //target mispredicted
+ * predicted:1 //target predicted
+ * in_tx:1 //in transaction
+ * abort:1 //transaction abort
+ * cycles:16 //cycle count to last branch
+ * type:4 //branch type
+ * reserved:40
+ * }
+ * }
+ *
+ * Avoid bswap64() the entire branch_flag.value,
+ * as it has variable bit-field sizes. Instead the
+ * macro takes the bit-field position/size,
+ * swaps it based on the host endianness.
+ *
+ * tep_is_bigendian() is used here instead of
+ * bigendian() to avoid python test fails.
+ */
+ if (tep_is_bigendian()) {
+ new_val = bitfield_swap(value, 0, 1);
+ new_val |= bitfield_swap(value, 1, 1);
+ new_val |= bitfield_swap(value, 2, 1);
+ new_val |= bitfield_swap(value, 3, 1);
+ new_val |= bitfield_swap(value, 4, 16);
+ new_val |= bitfield_swap(value, 20, 4);
+ new_val |= bitfield_swap(value, 24, 40);
+ } else {
+ new_val = bitfield_swap(value, 63, 1);
+ new_val |= bitfield_swap(value, 62, 1);
+ new_val |= bitfield_swap(value, 61, 1);
+ new_val |= bitfield_swap(value, 60, 1);
+ new_val |= bitfield_swap(value, 44, 16);
+ new_val |= bitfield_swap(value, 40, 4);
+ new_val |= bitfield_swap(value, 0, 40);
+ }
+
+ return new_val;
+}
+
int evsel__parse_sample(struct evsel *evsel, union perf_event *event,
struct perf_sample *data)
{
if (type & PERF_SAMPLE_BRANCH_STACK) {
const u64 max_branch_nr = UINT64_MAX /
sizeof(struct branch_entry);
+ struct branch_entry *e;
+ unsigned int i;
OVERFLOW_CHECK_u64(array);
data->branch_stack = (struct branch_stack *)array++;
return -EFAULT;
sz = data->branch_stack->nr * sizeof(struct branch_entry);
- if (evsel__has_branch_hw_idx(evsel))
+ if (evsel__has_branch_hw_idx(evsel)) {
sz += sizeof(u64);
- else
+ e = &data->branch_stack->entries[0];
+ } else {
data->no_hw_idx = true;
+ /*
+ * if the PERF_SAMPLE_BRANCH_HW_INDEX is not applied,
+ * only nr and entries[] will be output by kernel.
+ */
+ e = (struct branch_entry *)&data->branch_stack->hw_idx;
+ }
+
+ if (swapped) {
+ /*
+ * struct branch_flag does not have endian
+ * specific bit field definition. And bswap
+ * will not resolve the issue, since these
+ * are bit fields.
+ *
+ * evsel__bitfield_swap_branch_flags() uses a
+ * bitfield_swap macro to swap the bit position
+ * based on the host endians.
+ */
+ for (i = 0; i < data->branch_stack->nr; i++, e++)
+ e->flags.value = evsel__bitfield_swap_branch_flags(e->flags.value);
+ }
+
OVERFLOW_CHECK(array, sz, max_size);
array = (void *)array + sz;
}
{
evsel->core.leader = &leader->core;
}
+
+int evsel__source_count(const struct evsel *evsel)
+{
+ struct evsel *pos;
+ int count = 0;
+
+ evlist__for_each_entry(evsel->evlist, pos) {
+ if (pos->metric_leader == evsel)
+ count++;
+ }
+ return count;
+}
struct hashmap;
struct bperf_leader_bpf;
struct bperf_follower_bpf;
+struct perf_pmu;
typedef int (evsel__sb_cb_t)(union perf_event *event, void *data);
double scale;
const char *unit;
struct cgroup *cgrp;
+ const char *metric_id;
enum perf_tool_event tool_event;
/* parse modifier helper */
int exclude_GH;
};
unsigned long open_flags;
int precise_ip_original;
+
+ /* for missing_features */
+ struct perf_pmu *pmu;
};
struct perf_missing_features {
int __evsel__hw_cache_type_op_res_name(u8 type, u8 op, u8 result, char *bf, size_t size);
const char *evsel__name(struct evsel *evsel);
+const char *evsel__metric_id(const struct evsel *evsel);
const char *evsel__group_name(struct evsel *evsel);
int evsel__group_desc(struct evsel *evsel, char *buf, size_t size);
void evsel__set_sample_id(struct evsel *evsel, bool use_sample_identifier);
void arch_evsel__set_sample_weight(struct evsel *evsel);
+void arch_evsel__fixup_new_cycles(struct perf_event_attr *attr);
int evsel__set_filter(struct evsel *evsel, const char *filter);
int evsel__append_tp_filter(struct evsel *evsel, const char *filter);
bool evsel__has_leader(struct evsel *evsel, struct evsel *leader);
bool evsel__is_leader(struct evsel *evsel);
void evsel__set_leader(struct evsel *evsel, struct evsel *leader);
+int evsel__source_count(const struct evsel *evsel);
+
+/*
+ * Macro to swap the bit-field postition and size.
+ * Used when,
+ * - dont need to swap the entire u64 &&
+ * - when u64 has variable bit-field sizes &&
+ * - when presented in a host endian which is different
+ * than the source endian of the perf.data file
+ */
+#define bitfield_swap(src, pos, size) \
+ ((((src) >> (pos)) & ((1ull << (size)) - 1)) << (63 - ((pos) + (size) - 1)))
+
+u64 evsel__bitfield_swap_branch_flags(u64 value);
#endif /* __PERF_EVSEL_H */
#include "strlist.h"
#include "symbol.h"
#include "srcline.h"
+#include "dso.h"
static int comma_fprintf(FILE *fp, bool *first, const char *fmt, ...)
{
if (print_arrow && !first)
printed += fprintf(fp, " <-");
- if (print_ip)
- printed += fprintf(fp, "%c%16" PRIx64, s, node->ip);
-
if (map)
addr = map->map_ip(map, node->ip);
+ if (print_ip) {
+ /* Show binary offset for userspace addr */
+ if (map && !map->dso->kernel)
+ printed += fprintf(fp, "%c%16" PRIx64, s, addr);
+ else
+ printed += fprintf(fp, "%c%16" PRIx64, s, node->ip);
+ }
+
if (print_sym) {
printed += fprintf(fp, " ");
node_al.addr = addr;
#include <stdlib.h>
#include <string.h>
#include "metricgroup.h"
+#include "cpumap.h"
+#include "cputopo.h"
#include "debug.h"
#include "expr.h"
#include "expr-bison.h"
#include "expr-flex.h"
+#include "smt.h"
#include <linux/kernel.h>
#include <linux/zalloc.h>
#include <ctype.h>
+#include <math.h>
#ifdef PARSER_DEBUG
extern int expr_debug;
struct expr_id_data {
union {
- double val;
+ struct {
+ double val;
+ int source_count;
+ } val;
struct {
double val;
const char *metric_name;
const char *metric_expr;
} ref;
- struct expr_id *parent;
};
enum {
EXPR_ID_DATA__REF,
/* A reference but the value has been computed. */
EXPR_ID_DATA__REF_VALUE,
- /* A parent is remembered for the recursion check. */
- EXPR_ID_DATA__PARENT,
} kind;
};
return !strcmp((const char *)key1, (const char *)key2);
}
-/* Caller must make sure id is allocated */
-int expr__add_id(struct expr_parse_ctx *ctx, const char *id)
+struct hashmap *ids__new(void)
+{
+ return hashmap__new(key_hash, key_equal, NULL);
+}
+
+void ids__free(struct hashmap *ids)
+{
+ struct hashmap_entry *cur;
+ size_t bkt;
+
+ if (ids == NULL)
+ return;
+
+ hashmap__for_each_entry(ids, cur, bkt) {
+ free((char *)cur->key);
+ free(cur->value);
+ }
+
+ hashmap__free(ids);
+}
+
+int ids__insert(struct hashmap *ids, const char *id)
{
struct expr_id_data *data_ptr = NULL, *old_data = NULL;
char *old_key = NULL;
int ret;
- data_ptr = malloc(sizeof(*data_ptr));
- if (!data_ptr)
- return -ENOMEM;
-
- data_ptr->parent = ctx->parent;
- data_ptr->kind = EXPR_ID_DATA__PARENT;
-
- ret = hashmap__set(&ctx->ids, id, data_ptr,
+ ret = hashmap__set(ids, id, data_ptr,
(const void **)&old_key, (void **)&old_data);
if (ret)
free(data_ptr);
return ret;
}
+struct hashmap *ids__union(struct hashmap *ids1, struct hashmap *ids2)
+{
+ size_t bkt;
+ struct hashmap_entry *cur;
+ int ret;
+ struct expr_id_data *old_data = NULL;
+ char *old_key = NULL;
+
+ if (!ids1)
+ return ids2;
+
+ if (!ids2)
+ return ids1;
+
+ if (hashmap__size(ids1) < hashmap__size(ids2)) {
+ struct hashmap *tmp = ids1;
+
+ ids1 = ids2;
+ ids2 = tmp;
+ }
+ hashmap__for_each_entry(ids2, cur, bkt) {
+ ret = hashmap__set(ids1, cur->key, cur->value,
+ (const void **)&old_key, (void **)&old_data);
+ free(old_key);
+ free(old_data);
+
+ if (ret) {
+ hashmap__free(ids1);
+ hashmap__free(ids2);
+ return NULL;
+ }
+ }
+ hashmap__free(ids2);
+ return ids1;
+}
+
+/* Caller must make sure id is allocated */
+int expr__add_id(struct expr_parse_ctx *ctx, const char *id)
+{
+ return ids__insert(ctx->ids, id);
+}
+
/* Caller must make sure id is allocated */
int expr__add_id_val(struct expr_parse_ctx *ctx, const char *id, double val)
{
+ return expr__add_id_val_source_count(ctx, id, val, /*source_count=*/1);
+}
+
+/* Caller must make sure id is allocated */
+int expr__add_id_val_source_count(struct expr_parse_ctx *ctx, const char *id,
+ double val, int source_count)
+{
struct expr_id_data *data_ptr = NULL, *old_data = NULL;
char *old_key = NULL;
int ret;
data_ptr = malloc(sizeof(*data_ptr));
if (!data_ptr)
return -ENOMEM;
- data_ptr->val = val;
+ data_ptr->val.val = val;
+ data_ptr->val.source_count = source_count;
data_ptr->kind = EXPR_ID_DATA__VALUE;
- ret = hashmap__set(&ctx->ids, id, data_ptr,
+ ret = hashmap__set(ctx->ids, id, data_ptr,
(const void **)&old_key, (void **)&old_data);
if (ret)
free(data_ptr);
data_ptr->ref.metric_expr = ref->metric_expr;
data_ptr->kind = EXPR_ID_DATA__REF;
- ret = hashmap__set(&ctx->ids, name, data_ptr,
+ ret = hashmap__set(ctx->ids, name, data_ptr,
(const void **)&old_key, (void **)&old_data);
if (ret)
free(data_ptr);
int expr__get_id(struct expr_parse_ctx *ctx, const char *id,
struct expr_id_data **data)
{
- return hashmap__find(&ctx->ids, id, (void **)data) ? 0 : -1;
+ return hashmap__find(ctx->ids, id, (void **)data) ? 0 : -1;
+}
+
+bool expr__subset_of_ids(struct expr_parse_ctx *haystack,
+ struct expr_parse_ctx *needles)
+{
+ struct hashmap_entry *cur;
+ size_t bkt;
+ struct expr_id_data *data;
+
+ hashmap__for_each_entry(needles->ids, cur, bkt) {
+ if (expr__get_id(haystack, cur->key, &data))
+ return false;
+ }
+ return true;
}
+
int expr__resolve_id(struct expr_parse_ctx *ctx, const char *id,
struct expr_id_data **datap)
{
switch (data->kind) {
case EXPR_ID_DATA__VALUE:
- pr_debug2("lookup(%s): val %f\n", id, data->val);
- break;
- case EXPR_ID_DATA__PARENT:
- pr_debug2("lookup(%s): parent %s\n", id, data->parent->id);
+ pr_debug2("lookup(%s): val %f\n", id, data->val.val);
break;
case EXPR_ID_DATA__REF:
pr_debug2("lookup(%s): ref metric name %s\n", id,
data->ref.metric_name);
pr_debug("processing metric: %s ENTRY\n", id);
data->kind = EXPR_ID_DATA__REF_VALUE;
- if (expr__parse(&data->ref.val, ctx, data->ref.metric_expr, 1)) {
+ if (expr__parse(&data->ref.val, ctx, data->ref.metric_expr)) {
pr_debug("%s failed to count\n", id);
return -1;
}
- pr_debug("processing metric: %s EXIT: %f\n", id, data->val);
+ pr_debug("processing metric: %s EXIT: %f\n", id, data->ref.val);
break;
case EXPR_ID_DATA__REF_VALUE:
pr_debug2("lookup(%s): ref val %f metric name %s\n", id,
struct expr_id_data *old_val = NULL;
char *old_key = NULL;
- hashmap__delete(&ctx->ids, id,
+ hashmap__delete(ctx->ids, id,
(const void **)&old_key, (void **)&old_val);
free(old_key);
free(old_val);
}
-void expr__ctx_init(struct expr_parse_ctx *ctx)
+struct expr_parse_ctx *expr__ctx_new(void)
{
- hashmap__init(&ctx->ids, key_hash, key_equal, NULL);
+ struct expr_parse_ctx *ctx;
+
+ ctx = malloc(sizeof(struct expr_parse_ctx));
+ if (!ctx)
+ return NULL;
+
+ ctx->ids = hashmap__new(key_hash, key_equal, NULL);
+ ctx->runtime = 0;
+
+ return ctx;
}
void expr__ctx_clear(struct expr_parse_ctx *ctx)
struct hashmap_entry *cur;
size_t bkt;
- hashmap__for_each_entry((&ctx->ids), cur, bkt) {
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
free((char *)cur->key);
free(cur->value);
}
- hashmap__clear(&ctx->ids);
+ hashmap__clear(ctx->ids);
+}
+
+void expr__ctx_free(struct expr_parse_ctx *ctx)
+{
+ struct hashmap_entry *cur;
+ size_t bkt;
+
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
+ free((char *)cur->key);
+ free(cur->value);
+ }
+ hashmap__free(ctx->ids);
+ free(ctx);
}
static int
__expr__parse(double *val, struct expr_parse_ctx *ctx, const char *expr,
- int start, int runtime)
+ bool compute_ids)
{
struct expr_scanner_ctx scanner_ctx = {
- .start_token = start,
- .runtime = runtime,
+ .runtime = ctx->runtime,
};
YY_BUFFER_STATE buffer;
void *scanner;
expr_set_debug(1, scanner);
#endif
- ret = expr_parse(val, ctx, scanner);
+ ret = expr_parse(val, ctx, compute_ids, scanner);
expr__flush_buffer(buffer, scanner);
expr__delete_buffer(buffer, scanner);
}
int expr__parse(double *final_val, struct expr_parse_ctx *ctx,
- const char *expr, int runtime)
+ const char *expr)
{
- return __expr__parse(final_val, ctx, expr, EXPR_PARSE, runtime) ? -1 : 0;
+ return __expr__parse(final_val, ctx, expr, /*compute_ids=*/false) ? -1 : 0;
}
-int expr__find_other(const char *expr, const char *one,
- struct expr_parse_ctx *ctx, int runtime)
+int expr__find_ids(const char *expr, const char *one,
+ struct expr_parse_ctx *ctx)
{
- int ret = __expr__parse(NULL, ctx, expr, EXPR_OTHER, runtime);
+ int ret = __expr__parse(NULL, ctx, expr, /*compute_ids=*/true);
if (one)
expr__del_id(ctx, one);
double expr_id_data__value(const struct expr_id_data *data)
{
if (data->kind == EXPR_ID_DATA__VALUE)
- return data->val;
+ return data->val.val;
assert(data->kind == EXPR_ID_DATA__REF_VALUE);
return data->ref.val;
}
-struct expr_id *expr_id_data__parent(struct expr_id_data *data)
+double expr_id_data__source_count(const struct expr_id_data *data)
{
- assert(data->kind == EXPR_ID_DATA__PARENT);
- return data->parent;
+ assert(data->kind == EXPR_ID_DATA__VALUE);
+ return data->val.source_count;
+}
+
+double expr__get_literal(const char *literal)
+{
+ static struct cpu_topology *topology;
+
+ if (!strcmp("#smt_on", literal))
+ return smt_on() > 0 ? 1.0 : 0.0;
+
+ if (!strcmp("#num_cpus", literal))
+ return cpu__max_present_cpu();
+
+ /*
+ * Assume that topology strings are consistent, such as CPUs "0-1"
+ * wouldn't be listed as "0,1", and so after deduplication the number of
+ * these strings gives an indication of the number of packages, dies,
+ * etc.
+ */
+ if (!topology) {
+ topology = cpu_topology__new();
+ if (!topology) {
+ pr_err("Error creating CPU topology");
+ return NAN;
+ }
+ }
+ if (!strcmp("#num_packages", literal))
+ return topology->package_cpus_lists;
+ if (!strcmp("#num_dies", literal))
+ return topology->die_cpus_lists;
+ if (!strcmp("#num_cores", literal))
+ return topology->core_cpus_lists;
+
+ pr_err("Unrecognized literal '%s'", literal);
+ return NAN;
}
struct metric_ref;
-struct expr_id {
- char *id;
- struct expr_id *parent;
-};
-
struct expr_parse_ctx {
- struct hashmap ids;
- struct expr_id *parent;
+ struct hashmap *ids;
+ int runtime;
};
struct expr_id_data;
struct expr_scanner_ctx {
- int start_token;
int runtime;
};
-void expr__ctx_init(struct expr_parse_ctx *ctx);
+struct hashmap *ids__new(void);
+void ids__free(struct hashmap *ids);
+int ids__insert(struct hashmap *ids, const char *id);
+/*
+ * Union two sets of ids (hashmaps) and construct a third, freeing ids1 and
+ * ids2.
+ */
+struct hashmap *ids__union(struct hashmap *ids1, struct hashmap *ids2);
+
+struct expr_parse_ctx *expr__ctx_new(void);
void expr__ctx_clear(struct expr_parse_ctx *ctx);
+void expr__ctx_free(struct expr_parse_ctx *ctx);
+
void expr__del_id(struct expr_parse_ctx *ctx, const char *id);
int expr__add_id(struct expr_parse_ctx *ctx, const char *id);
int expr__add_id_val(struct expr_parse_ctx *ctx, const char *id, double val);
+int expr__add_id_val_source_count(struct expr_parse_ctx *ctx, const char *id,
+ double val, int source_count);
int expr__add_ref(struct expr_parse_ctx *ctx, struct metric_ref *ref);
int expr__get_id(struct expr_parse_ctx *ctx, const char *id,
struct expr_id_data **data);
+bool expr__subset_of_ids(struct expr_parse_ctx *haystack,
+ struct expr_parse_ctx *needles);
int expr__resolve_id(struct expr_parse_ctx *ctx, const char *id,
struct expr_id_data **datap);
+
int expr__parse(double *final_val, struct expr_parse_ctx *ctx,
- const char *expr, int runtime);
-int expr__find_other(const char *expr, const char *one,
- struct expr_parse_ctx *ids, int runtime);
+ const char *expr);
+
+int expr__find_ids(const char *expr, const char *one,
+ struct expr_parse_ctx *ids);
double expr_id_data__value(const struct expr_id_data *data);
-struct expr_id *expr_id_data__parent(struct expr_id_data *data);
+double expr_id_data__source_count(const struct expr_id_data *data);
+double expr__get_literal(const char *literal);
#endif
#include <linux/compiler.h>
#include "expr.h"
#include "expr-bison.h"
+#include <math.h>
char *expr_get_text(yyscan_t yyscanner);
YYSTYPE *expr_get_lval(yyscan_t yyscanner);
char *dst = str;
while (*str) {
- if (*str == '@')
- *dst++ = '/';
- else if (*str == '\\')
+ if (*str == '\\')
*dst++ = *++str;
- else if (*str == '?') {
+ else if (*str == '?') {
char *paramval;
int i = 0;
int size = asprintf(¶mval, "%d", runtime);
yylval->str = normalize(yylval->str, runtime);
return token;
}
+
+static int literal(yyscan_t scanner)
+{
+ YYSTYPE *yylval = expr_get_lval(scanner);
+
+ yylval->num = expr__get_literal(expr_get_text(scanner));
+ if (isnan(yylval->num))
+ return EXPR_ERROR;
+
+ return LITERAL;
+}
%}
number ([0-9]+\.?[0-9]*|[0-9]*\.?[0-9]+)
spec \\{sch}
sym [0-9a-zA-Z_\.:@?]+
symbol ({spec}|{sym})+
+literal #[0-9a-zA-Z_\.\-]+
%%
struct expr_scanner_ctx *sctx = expr_get_extra(yyscanner);
- {
- int start_token = sctx->start_token;
-
- if (sctx->start_token) {
- sctx->start_token = 0;
- return start_token;
- }
- }
-
d_ratio { return D_RATIO; }
max { return MAX; }
min { return MIN; }
if { return IF; }
else { return ELSE; }
-#smt_on { return SMT_ON; }
+source_count { return SOURCE_COUNT; }
+{literal} { return literal(yyscanner); }
{number} { return value(yyscanner); }
{symbol} { return str(yyscanner, ID, sctx->runtime); }
"|" { return '|'; }
/* Simple expression parser */
%{
#define YYDEBUG 1
-#include <stdio.h>
-#include "util.h"
+#include <assert.h>
+#include <math.h>
+#include <stdlib.h>
#include "util/debug.h"
-#include <stdlib.h> // strtod()
#define IN_EXPR_Y 1
#include "expr.h"
-#include "smt.h"
-#include <string.h>
-
-static double d_ratio(double val0, double val1)
-{
- if (val1 == 0) {
- return 0;
- }
- return val0 / val1;
-}
-
%}
%define api.pure full
%parse-param { double *final_val }
%parse-param { struct expr_parse_ctx *ctx }
+%parse-param { bool compute_ids }
%parse-param {void *scanner}
%lex-param {void* scanner}
%union {
double num;
char *str;
+ struct ids {
+ /*
+ * When creating ids, holds the working set of event ids. NULL
+ * implies the set is empty.
+ */
+ struct hashmap *ids;
+ /*
+ * The metric value. When not creating ids this is the value
+ * read from a counter, a constant or some computed value. When
+ * creating ids the value is either a constant or BOTTOM. NAN is
+ * used as the special BOTTOM value, representing a "set of all
+ * values" case.
+ */
+ double val;
+ } ids;
}
-%token EXPR_PARSE EXPR_OTHER EXPR_ERROR
-%token <num> NUMBER
-%token <str> ID
-%destructor { free ($$); } <str>
-%token MIN MAX IF ELSE SMT_ON D_RATIO
+%token ID NUMBER MIN MAX IF ELSE LITERAL D_RATIO SOURCE_COUNT EXPR_ERROR
%left MIN MAX IF
%left '|'
%left '^'
%left '-' '+'
%left '*' '/' '%'
%left NEG NOT
-%type <num> expr if_expr
+%type <num> NUMBER LITERAL
+%type <str> ID
+%destructor { free ($$); } <str>
+%type <ids> expr if_expr
+%destructor { ids__free($$.ids); } <ids>
%{
static void expr_error(double *final_val __maybe_unused,
struct expr_parse_ctx *ctx __maybe_unused,
+ bool compute_ids __maybe_unused,
void *scanner,
const char *s)
{
pr_debug("%s\n", s);
}
+/*
+ * During compute ids, the special "bottom" value uses NAN to represent the set
+ * of all values. NAN is selected as it isn't a useful constant value.
+ */
+#define BOTTOM NAN
+
+/* During computing ids, does val represent a constant (non-BOTTOM) value? */
+static bool is_const(double val)
+{
+ return isfinite(val);
+}
+
+static struct ids union_expr(struct ids ids1, struct ids ids2)
+{
+ struct ids result = {
+ .val = BOTTOM,
+ .ids = ids__union(ids1.ids, ids2.ids),
+ };
+ return result;
+}
+
+static struct ids handle_id(struct expr_parse_ctx *ctx, char *id,
+ bool compute_ids, bool source_count)
+{
+ struct ids result;
+
+ if (!compute_ids) {
+ /*
+ * Compute the event's value from ID. If the ID isn't known then
+ * it isn't used to compute the formula so set to NAN.
+ */
+ struct expr_id_data *data;
+
+ result.val = NAN;
+ if (expr__resolve_id(ctx, id, &data) == 0) {
+ result.val = source_count
+ ? expr_id_data__source_count(data)
+ : expr_id_data__value(data);
+ }
+ result.ids = NULL;
+ free(id);
+ } else {
+ /*
+ * Set the value to BOTTOM to show that any value is possible
+ * when the event is computed. Create a set of just the ID.
+ */
+ result.val = BOTTOM;
+ result.ids = ids__new();
+ if (!result.ids || ids__insert(result.ids, id)) {
+ pr_err("Error creating IDs for '%s'", id);
+ free(id);
+ }
+ }
+ return result;
+}
+
+/*
+ * If we're not computing ids or $1 and $3 are constants, compute the new
+ * constant value using OP. Its invariant that there are no ids. If computing
+ * ids for non-constants union the set of IDs that must be computed.
+ */
+#define BINARY_LONG_OP(RESULT, OP, LHS, RHS) \
+ if (!compute_ids || (is_const(LHS.val) && is_const(RHS.val))) { \
+ assert(LHS.ids == NULL); \
+ assert(RHS.ids == NULL); \
+ RESULT.val = (long)LHS.val OP (long)RHS.val; \
+ RESULT.ids = NULL; \
+ } else { \
+ RESULT = union_expr(LHS, RHS); \
+ }
+
+#define BINARY_OP(RESULT, OP, LHS, RHS) \
+ if (!compute_ids || (is_const(LHS.val) && is_const(RHS.val))) { \
+ assert(LHS.ids == NULL); \
+ assert(RHS.ids == NULL); \
+ RESULT.val = LHS.val OP RHS.val; \
+ RESULT.ids = NULL; \
+ } else { \
+ RESULT = union_expr(LHS, RHS); \
+ }
+
%}
%%
-start:
-EXPR_PARSE all_expr
-|
-EXPR_OTHER all_other
+start: if_expr
+{
+ if (compute_ids)
+ ctx->ids = ids__union($1.ids, ctx->ids);
+
+ if (final_val)
+ *final_val = $1.val;
+}
+;
-all_other: all_other other
-|
+if_expr: expr IF expr ELSE expr
+{
+ if (fpclassify($3.val) == FP_ZERO) {
+ /*
+ * The IF expression evaluated to 0 so treat as false, take the
+ * ELSE and discard everything else.
+ */
+ $$.val = $5.val;
+ $$.ids = $5.ids;
+ ids__free($1.ids);
+ ids__free($3.ids);
+ } else if (!compute_ids || is_const($3.val)) {
+ /*
+ * If ids aren't computed then treat the expression as true. If
+ * ids are being computed and the IF expr is a non-zero
+ * constant, then also evaluate the true case.
+ */
+ $$.val = $1.val;
+ $$.ids = $1.ids;
+ ids__free($3.ids);
+ ids__free($5.ids);
+ } else if ($1.val == $5.val) {
+ /*
+ * LHS == RHS, so both are an identical constant. No need to
+ * evaluate any events.
+ */
+ $$.val = $1.val;
+ $$.ids = NULL;
+ ids__free($1.ids);
+ ids__free($3.ids);
+ ids__free($5.ids);
+ } else {
+ /*
+ * Value is either the LHS or RHS and we need the IF expression
+ * to compute it.
+ */
+ $$ = union_expr($1, union_expr($3, $5));
+ }
+}
+| expr
+;
-other: ID
+expr: NUMBER
+{
+ $$.val = $1;
+ $$.ids = NULL;
+}
+| ID { $$ = handle_id(ctx, $1, compute_ids, /*source_count=*/false); }
+| SOURCE_COUNT '(' ID ')' { $$ = handle_id(ctx, $3, compute_ids, /*source_count=*/true); }
+| expr '|' expr { BINARY_LONG_OP($$, |, $1, $3); }
+| expr '&' expr { BINARY_LONG_OP($$, &, $1, $3); }
+| expr '^' expr { BINARY_LONG_OP($$, ^, $1, $3); }
+| expr '<' expr { BINARY_OP($$, <, $1, $3); }
+| expr '>' expr { BINARY_OP($$, >, $1, $3); }
+| expr '+' expr { BINARY_OP($$, +, $1, $3); }
+| expr '-' expr { BINARY_OP($$, -, $1, $3); }
+| expr '*' expr { BINARY_OP($$, *, $1, $3); }
+| expr '/' expr
+{
+ if (fpclassify($3.val) == FP_ZERO) {
+ pr_debug("division by zero\n");
+ YYABORT;
+ } else if (!compute_ids || (is_const($1.val) && is_const($3.val))) {
+ assert($1.ids == NULL);
+ assert($3.ids == NULL);
+ $$.val = $1.val / $3.val;
+ $$.ids = NULL;
+ } else {
+ /* LHS and/or RHS need computing from event IDs so union. */
+ $$ = union_expr($1, $3);
+ }
+}
+| expr '%' expr
+{
+ if (fpclassify($3.val) == FP_ZERO) {
+ pr_debug("division by zero\n");
+ YYABORT;
+ } else if (!compute_ids || (is_const($1.val) && is_const($3.val))) {
+ assert($1.ids == NULL);
+ assert($3.ids == NULL);
+ $$.val = (long)$1.val % (long)$3.val;
+ $$.ids = NULL;
+ } else {
+ /* LHS and/or RHS need computing from event IDs so union. */
+ $$ = union_expr($1, $3);
+ }
+}
+| D_RATIO '(' expr ',' expr ')'
+{
+ if (fpclassify($5.val) == FP_ZERO) {
+ /*
+ * Division by constant zero always yields zero and no events
+ * are necessary.
+ */
+ assert($5.ids == NULL);
+ $$.val = 0.0;
+ $$.ids = NULL;
+ ids__free($3.ids);
+ } else if (!compute_ids || (is_const($3.val) && is_const($5.val))) {
+ assert($3.ids == NULL);
+ assert($5.ids == NULL);
+ $$.val = $3.val / $5.val;
+ $$.ids = NULL;
+ } else {
+ /* LHS and/or RHS need computing from event IDs so union. */
+ $$ = union_expr($3, $5);
+ }
+}
+| '-' expr %prec NEG
+{
+ $$.val = -$2.val;
+ $$.ids = $2.ids;
+}
+| '(' if_expr ')'
{
- expr__add_id(ctx, $1);
-}
-|
-MIN | MAX | IF | ELSE | SMT_ON | NUMBER | '|' | '^' | '&' | '-' | '+' | '*' | '/' | '%' | '(' | ')' | ','
-|
-'<' | '>' | D_RATIO
-
-all_expr: if_expr { *final_val = $1; }
- ;
-
-if_expr:
- expr IF expr ELSE expr { $$ = $3 ? $1 : $5; }
- | expr
- ;
-
-expr: NUMBER
- | ID {
- struct expr_id_data *data;
-
- if (expr__resolve_id(ctx, $1, &data)) {
- free($1);
- YYABORT;
- }
-
- $$ = expr_id_data__value(data);
- free($1);
- }
- | expr '|' expr { $$ = (long)$1 | (long)$3; }
- | expr '&' expr { $$ = (long)$1 & (long)$3; }
- | expr '^' expr { $$ = (long)$1 ^ (long)$3; }
- | expr '<' expr { $$ = $1 < $3; }
- | expr '>' expr { $$ = $1 > $3; }
- | expr '+' expr { $$ = $1 + $3; }
- | expr '-' expr { $$ = $1 - $3; }
- | expr '*' expr { $$ = $1 * $3; }
- | expr '/' expr { if ($3 == 0) {
- pr_debug("division by zero\n");
- YYABORT;
- }
- $$ = $1 / $3;
- }
- | expr '%' expr { if ((long)$3 == 0) {
- pr_debug("division by zero\n");
- YYABORT;
- }
- $$ = (long)$1 % (long)$3;
- }
- | '-' expr %prec NEG { $$ = -$2; }
- | '(' if_expr ')' { $$ = $2; }
- | MIN '(' expr ',' expr ')' { $$ = $3 < $5 ? $3 : $5; }
- | MAX '(' expr ',' expr ')' { $$ = $3 > $5 ? $3 : $5; }
- | SMT_ON { $$ = smt_on() > 0; }
- | D_RATIO '(' expr ',' expr ')' { $$ = d_ratio($3,$5); }
- ;
+ $$ = $2;
+}
+| MIN '(' expr ',' expr ')'
+{
+ if (!compute_ids) {
+ $$.val = $3.val < $5.val ? $3.val : $5.val;
+ $$.ids = NULL;
+ } else {
+ $$ = union_expr($3, $5);
+ }
+}
+| MAX '(' expr ',' expr ')'
+{
+ if (!compute_ids) {
+ $$.val = $3.val > $5.val ? $3.val : $5.val;
+ $$.ids = NULL;
+ } else {
+ $$ = union_expr($3, $5);
+ }
+}
+| LITERAL
+{
+ $$.val = $1;
+ $$.ids = NULL;
+}
+;
%%
#error "unsupported architecture"
#endif
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define GEN_ELF_ENDIAN ELFDATA2MSB
#else
#define GEN_ELF_ENDIAN ELFDATA2LSB
#include "util/util.h" // perf_exe()
#include "cputopo.h"
#include "bpf-event.h"
+#include "bpf-utils.h"
#include "clockid.h"
#include "pmu-hybrid.h"
if (!tp)
return -1;
- ret = do_write(ff, &tp->core_sib, sizeof(tp->core_sib));
+ ret = do_write(ff, &tp->package_cpus_lists, sizeof(tp->package_cpus_lists));
if (ret < 0)
goto done;
- for (i = 0; i < tp->core_sib; i++) {
- ret = do_write_string(ff, tp->core_siblings[i]);
+ for (i = 0; i < tp->package_cpus_lists; i++) {
+ ret = do_write_string(ff, tp->package_cpus_list[i]);
if (ret < 0)
goto done;
}
- ret = do_write(ff, &tp->thread_sib, sizeof(tp->thread_sib));
+ ret = do_write(ff, &tp->core_cpus_lists, sizeof(tp->core_cpus_lists));
if (ret < 0)
goto done;
- for (i = 0; i < tp->thread_sib; i++) {
- ret = do_write_string(ff, tp->thread_siblings[i]);
+ for (i = 0; i < tp->core_cpus_lists; i++) {
+ ret = do_write_string(ff, tp->core_cpus_list[i]);
if (ret < 0)
break;
}
return ret;
}
- if (!tp->die_sib)
+ if (!tp->die_cpus_lists)
goto done;
- ret = do_write(ff, &tp->die_sib, sizeof(tp->die_sib));
+ ret = do_write(ff, &tp->die_cpus_lists, sizeof(tp->die_cpus_lists));
if (ret < 0)
goto done;
- for (i = 0; i < tp->die_sib; i++) {
- ret = do_write_string(ff, tp->die_siblings[i]);
+ for (i = 0; i < tp->die_cpus_lists; i++) {
+ ret = do_write_string(ff, tp->die_cpus_list[i]);
if (ret < 0)
goto done;
}
node = rb_entry(next, struct bpf_prog_info_node, rb_node);
next = rb_next(&node->rb_node);
- len = sizeof(struct bpf_prog_info_linear) +
+ len = sizeof(struct perf_bpil) +
node->info_linear->data_len;
/* before writing to file, translate address to offset */
- bpf_program__bpil_addr_to_offs(node->info_linear);
+ bpil_addr_to_offs(node->info_linear);
ret = do_write(ff, node->info_linear, len);
/*
* translate back to address even when do_write() fails,
* so that this function never changes the data.
*/
- bpf_program__bpil_offs_to_addr(node->info_linear);
+ bpil_offs_to_addr(node->info_linear);
if (ret < 0)
goto out;
}
#ifdef HAVE_LIBBPF_SUPPORT
static int process_bpf_prog_info(struct feat_fd *ff, void *data __maybe_unused)
{
- struct bpf_prog_info_linear *info_linear;
struct bpf_prog_info_node *info_node;
struct perf_env *env = &ff->ph->env;
+ struct perf_bpil *info_linear;
u32 count, i;
int err = -1;
goto out;
}
- info_linear = malloc(sizeof(struct bpf_prog_info_linear) +
+ info_linear = malloc(sizeof(struct perf_bpil) +
data_len);
if (!info_linear)
goto out;
goto out;
/* after reading from file, translate offset to address */
- bpf_program__bpil_offs_to_addr(info_linear);
+ bpil_offs_to_addr(info_linear);
info_node->info_linear = info_linear;
perf_env__insert_bpf_prog_info(env, info_node);
}
switch (ev->type) {
case PERF_EVENT_UPDATE__UNIT:
+ free((char *)evsel->unit);
evsel->unit = strdup(ev->data);
break;
case PERF_EVENT_UPDATE__NAME:
+ free(evsel->name);
evsel->name = strdup(ev->data);
break;
case PERF_EVENT_UPDATE__SCALE:
break;
case PERF_EVENT_UPDATE__CPUS:
ev_cpus = (struct perf_record_event_update_cpus *)ev->data;
-
map = cpu_map__new_data(&ev_cpus->cpus);
- if (map)
+ if (map) {
+ perf_cpu_map__put(evsel->core.own_cpus);
evsel->core.own_cpus = map;
- else
+ } else
pr_err("failed to get event_update cpus\n");
default:
break;
return htime;
}
-static void he_stat__add_period(struct he_stat *he_stat, u64 period,
- u64 weight, u64 ins_lat, u64 p_stage_cyc)
+static void he_stat__add_period(struct he_stat *he_stat, u64 period)
{
-
he_stat->period += period;
- he_stat->weight += weight;
he_stat->nr_events += 1;
- he_stat->ins_lat += ins_lat;
- he_stat->p_stage_cyc += p_stage_cyc;
}
static void he_stat__add_stat(struct he_stat *dest, struct he_stat *src)
dest->period_guest_sys += src->period_guest_sys;
dest->period_guest_us += src->period_guest_us;
dest->nr_events += src->nr_events;
- dest->weight += src->weight;
- dest->ins_lat += src->ins_lat;
- dest->p_stage_cyc += src->p_stage_cyc;
}
static void he_stat__decay(struct he_stat *he_stat)
struct hist_entry *he;
int64_t cmp;
u64 period = entry->stat.period;
- u64 weight = entry->stat.weight;
- u64 ins_lat = entry->stat.ins_lat;
- u64 p_stage_cyc = entry->stat.p_stage_cyc;
bool leftmost = true;
p = &hists->entries_in->rb_root.rb_node;
if (!cmp) {
if (sample_self) {
- he_stat__add_period(&he->stat, period, weight, ins_lat, p_stage_cyc);
+ he_stat__add_period(&he->stat, period);
hist_entry__add_callchain_period(he, period);
}
if (symbol_conf.cumulate_callchain)
- he_stat__add_period(he->stat_acc, period, weight, ins_lat, p_stage_cyc);
+ he_stat__add_period(he->stat_acc, period);
/*
* This mem info was allocated from sample__resolve_mem
.stat = {
.nr_events = 1,
.period = sample->period,
- .weight = sample->weight,
- .ins_lat = sample->ins_lat,
- .p_stage_cyc = sample->p_stage_cyc,
},
.parent = sym_parent,
.filtered = symbol__parent_filter(sym_parent) | al->filtered,
.raw_size = sample->raw_size,
.ops = ops,
.time = hist_time(sample->time),
+ .weight = sample->weight,
+ .ins_lat = sample->ins_lat,
+ .p_stage_cyc = sample->p_stage_cyc,
}, *he = hists__findnew_entry(hists, &entry, al, sample_self);
if (!hists->has_callchains && he && he->callchain_size != 0)
};
void perf_hpp__init(void);
-void perf_hpp__column_unregister(struct perf_hpp_fmt *format);
void perf_hpp__cancel_cumulate(void);
void perf_hpp__setup_output_field(struct perf_hpp_list *list);
void perf_hpp__reset_output_field(struct perf_hpp_list *list);
#define INTEL_BTS_ERR_NOINSN 5
#define INTEL_BTS_ERR_LOST 9
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define le64_to_cpu bswap_64
#else
#define le64_to_cpu
{
decoder->sample_timestamp = decoder->timestamp;
decoder->sample_insn_cnt = decoder->timestamp_insn_cnt;
+ decoder->state.cycles = decoder->tot_cyc_cnt;
}
static void intel_pt_reposition(struct intel_pt_decoder *decoder)
uint64_t to_ip;
uint64_t tot_insn_cnt;
uint64_t tot_cyc_cnt;
+ uint64_t cycles;
uint64_t timestamp;
uint64_t est_timestamp;
uint64_t trace_nr;
if (branch == INTEL_PT_BR_CONDITIONAL ||
branch == INTEL_PT_BR_UNCONDITIONAL) {
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
switch (insn->immediate.nbytes) {
case 1:
intel_pt_insn->rel = insn->immediate.value;
if (f)
return 0;
- if (!log_name[0])
- return -1;
-
- f = fopen(log_name, "w+");
+ if (log_name[0])
+ f = fopen(log_name, "w+");
+ else
+ f = stdout;
if (!f) {
intel_pt_enable_logging = false;
return -1;
#define BIT63 ((uint64_t)1 << 63)
-#if __BYTE_ORDER == __BIG_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define le16_to_cpu bswap_16
#define le32_to_cpu bswap_32
#define le64_to_cpu bswap_64
u64 cbr_id;
u64 psb_id;
+ bool single_pebs;
bool sample_pebs;
struct evsel *pebs_evsel;
INTEL_PT_SS_EXPECTING_SWITCH_IP,
};
+/* applicable_counters is 64-bits */
+#define INTEL_PT_MAX_PEBS 64
+
+struct intel_pt_pebs_event {
+ struct evsel *evsel;
+ u64 id;
+};
+
struct intel_pt_queue {
struct intel_pt *pt;
unsigned int queue_nr;
bool step_through_buffers;
bool use_buffer_pid_tid;
bool sync_switch;
+ bool sample_ipc;
pid_t pid, tid;
int cpu;
int switch_state;
u64 last_br_cyc_cnt;
unsigned int cbr_seen;
char insn[INTEL_PT_INSN_BUF_SZ];
+ struct intel_pt_pebs_event pebs[INTEL_PT_MAX_PEBS];
};
static void intel_pt_dump(struct intel_pt *pt __maybe_unused,
sample.branch_stack = (struct branch_stack *)&dummy_bs;
}
- if (ptq->state->flags & INTEL_PT_SAMPLE_IPC)
+ if (ptq->sample_ipc)
sample.cyc_cnt = ptq->ipc_cyc_cnt - ptq->last_br_cyc_cnt;
if (sample.cyc_cnt) {
sample.insn_cnt = ptq->ipc_insn_cnt - ptq->last_br_insn_cnt;
else
sample.period = ptq->state->tot_insn_cnt - ptq->last_insn_cnt;
- if (ptq->state->flags & INTEL_PT_SAMPLE_IPC)
+ if (ptq->sample_ipc)
sample.cyc_cnt = ptq->ipc_cyc_cnt - ptq->last_in_cyc_cnt;
if (sample.cyc_cnt) {
sample.insn_cnt = ptq->ipc_insn_cnt - ptq->last_in_insn_cnt;
}
}
-static int intel_pt_synth_pebs_sample(struct intel_pt_queue *ptq)
+static int intel_pt_do_synth_pebs_sample(struct intel_pt_queue *ptq, struct evsel *evsel, u64 id)
{
const struct intel_pt_blk_items *items = &ptq->state->items;
struct perf_sample sample = { .ip = 0, };
union perf_event *event = ptq->event_buf;
struct intel_pt *pt = ptq->pt;
- struct evsel *evsel = pt->pebs_evsel;
u64 sample_type = evsel->core.attr.sample_type;
- u64 id = evsel->core.id[0];
u8 cpumode;
u64 regs[8 * sizeof(sample.intr_regs.mask)];
return intel_pt_deliver_synth_event(pt, event, &sample, sample_type);
}
+static int intel_pt_synth_single_pebs_sample(struct intel_pt_queue *ptq)
+{
+ struct intel_pt *pt = ptq->pt;
+ struct evsel *evsel = pt->pebs_evsel;
+ u64 id = evsel->core.id[0];
+
+ return intel_pt_do_synth_pebs_sample(ptq, evsel, id);
+}
+
+static int intel_pt_synth_pebs_sample(struct intel_pt_queue *ptq)
+{
+ const struct intel_pt_blk_items *items = &ptq->state->items;
+ struct intel_pt_pebs_event *pe;
+ struct intel_pt *pt = ptq->pt;
+ int err = -EINVAL;
+ int hw_id;
+
+ if (!items->has_applicable_counters || !items->applicable_counters) {
+ if (!pt->single_pebs)
+ pr_err("PEBS-via-PT record with no applicable_counters\n");
+ return intel_pt_synth_single_pebs_sample(ptq);
+ }
+
+ for_each_set_bit(hw_id, (unsigned long *)&items->applicable_counters, INTEL_PT_MAX_PEBS) {
+ pe = &ptq->pebs[hw_id];
+ if (!pe->evsel) {
+ if (!pt->single_pebs)
+ pr_err("PEBS-via-PT record with no matching event, hw_id %d\n",
+ hw_id);
+ return intel_pt_synth_single_pebs_sample(ptq);
+ }
+ err = intel_pt_do_synth_pebs_sample(ptq, pe->evsel, pe->id);
+ if (err)
+ return err;
+ }
+
+ return err;
+}
+
static int intel_pt_synth_error(struct intel_pt *pt, int code, int cpu,
pid_t pid, pid_t tid, u64 ip, u64 timestamp)
{
ptq->have_sample = false;
- ptq->ipc_insn_cnt = ptq->state->tot_insn_cnt;
- ptq->ipc_cyc_cnt = ptq->state->tot_cyc_cnt;
+ if (pt->synth_opts.approx_ipc) {
+ ptq->ipc_insn_cnt = ptq->state->tot_insn_cnt;
+ ptq->ipc_cyc_cnt = ptq->state->cycles;
+ ptq->sample_ipc = true;
+ } else {
+ ptq->ipc_insn_cnt = ptq->state->tot_insn_cnt;
+ ptq->ipc_cyc_cnt = ptq->state->tot_cyc_cnt;
+ ptq->sample_ipc = ptq->state->flags & INTEL_PT_SAMPLE_IPC;
+ }
/*
* Do PEBS first to allow for the possibility that the PEBS timestamp
event->itrace_start.tid);
}
+static int intel_pt_process_aux_output_hw_id(struct intel_pt *pt,
+ union perf_event *event,
+ struct perf_sample *sample)
+{
+ u64 hw_id = event->aux_output_hw_id.hw_id;
+ struct auxtrace_queue *queue;
+ struct intel_pt_queue *ptq;
+ struct evsel *evsel;
+
+ queue = auxtrace_queues__sample_queue(&pt->queues, sample, pt->session);
+ evsel = evlist__id2evsel_strict(pt->session->evlist, sample->id);
+ if (!queue || !queue->priv || !evsel || hw_id > INTEL_PT_MAX_PEBS) {
+ pr_err("Bad AUX output hardware ID\n");
+ return -EINVAL;
+ }
+
+ ptq = queue->priv;
+
+ ptq->pebs[hw_id].evsel = evsel;
+ ptq->pebs[hw_id].id = sample->id;
+
+ return 0;
+}
+
static int intel_pt_find_map(struct thread *thread, u8 cpumode, u64 addr,
struct addr_location *al)
{
err = intel_pt_process_switch(pt, sample);
else if (event->header.type == PERF_RECORD_ITRACE_START)
err = intel_pt_process_itrace_start(pt, event, sample);
+ else if (event->header.type == PERF_RECORD_AUX_OUTPUT_HW_ID)
+ err = intel_pt_process_aux_output_hw_id(pt, event, sample);
else if (event->header.type == PERF_RECORD_SWITCH ||
event->header.type == PERF_RECORD_SWITCH_CPU_WIDE)
err = intel_pt_context_switch(pt, event, sample);
evlist__for_each_entry(pt->session->evlist, evsel) {
if (evsel->core.attr.aux_output && evsel->core.id) {
+ if (pt->single_pebs) {
+ pt->single_pebs = false;
+ return;
+ }
+ pt->single_pebs = true;
pt->sample_pebs = true;
pt->pebs_evsel = evsel;
- return;
}
}
}
if (err)
goto err_free;
- intel_pt_log_set_name(INTEL_PT_PMU_NAME);
-
if (session->itrace_synth_opts->set) {
pt->synth_opts = *session->itrace_synth_opts;
} else {
pt->synth_opts.thread_stack = opts->thread_stack;
}
+ if (!(pt->synth_opts.log_plus_flags & AUXTRACE_LOG_FLG_USE_STDOUT))
+ intel_pt_log_set_name(INTEL_PT_PMU_NAME);
+
pt->session = session;
pt->machine = &session->machines.host; /* No kvm support */
pt->auxtrace_type = auxtrace_info->type;
return 0;
}
+int machine__process_aux_output_hw_id_event(struct machine *machine __maybe_unused,
+ union perf_event *event)
+{
+ if (dump_trace)
+ perf_event__fprintf_aux_output_hw_id(event, stdout);
+ return 0;
+}
+
int machine__process_switch_event(struct machine *machine __maybe_unused,
union perf_event *event)
{
ret = machine__process_bpf(machine, event, sample); break;
case PERF_RECORD_TEXT_POKE:
ret = machine__process_text_poke(machine, event, sample); break;
+ case PERF_RECORD_AUX_OUTPUT_HW_ID:
+ ret = machine__process_aux_output_hw_id_event(machine, event); break;
default:
ret = -1;
break;
union perf_event *event);
int machine__process_itrace_start_event(struct machine *machine,
union perf_event *event);
+int machine__process_aux_output_hw_id_event(struct machine *machine,
+ union perf_event *event);
int machine__process_switch_event(struct machine *machine,
union perf_event *event);
int machine__process_namespaces_event(struct machine *machine,
#include "evsel.h"
#include "strbuf.h"
#include "pmu.h"
+#include "pmu-hybrid.h"
#include "expr.h"
#include "rblist.h"
#include <string.h>
#include "strlist.h"
#include <assert.h>
#include <linux/ctype.h>
+#include <linux/list_sort.h>
#include <linux/string.h>
#include <linux/zalloc.h>
#include <subcmd/parse-options.h>
struct metric_expr *expr, *tmp;
list_for_each_entry_safe(expr, tmp, &me->head, nd) {
+ free((char *)expr->metric_name);
free(expr->metric_refs);
free(expr->metric_events);
free(expr);
struct list_head list;
};
+/**
+ * The metric under construction. The data held here will be placed in a
+ * metric_expr.
+ */
struct metric {
struct list_head nd;
- struct expr_parse_ctx pctx;
+ /**
+ * The expression parse context importantly holding the IDs contained
+ * within the expression.
+ */
+ struct expr_parse_ctx *pctx;
+ /** The name of the metric such as "IPC". */
const char *metric_name;
+ /** Modifier on the metric such as "u" or NULL for none. */
+ const char *modifier;
+ /** The expression to parse, for example, "instructions/cycles". */
const char *metric_expr;
+ /**
+ * The "ScaleUnit" that scales and adds a unit to the metric during
+ * output.
+ */
const char *metric_unit;
- struct list_head metric_refs;
- int metric_refs_cnt;
- int runtime;
+ /** Optional null terminated array of referenced metrics. */
+ struct metric_ref *metric_refs;
+ /**
+ * Is there a constraint on the group of events? In which case the
+ * events won't be grouped.
+ */
bool has_constraint;
+ /**
+ * Parsed events for the metric. Optional as events may be taken from a
+ * different metric whose group contains all the IDs necessary for this
+ * one.
+ */
+ struct evlist *evlist;
};
-#define RECURSION_ID_MAX 1000
+static void metricgroup___watchdog_constraint_hint(const char *name, bool foot)
+{
+ static bool violate_nmi_constraint;
-struct expr_ids {
- struct expr_id id[RECURSION_ID_MAX];
- int cnt;
-};
+ if (!foot) {
+ pr_warning("Splitting metric group %s into standalone metrics.\n", name);
+ violate_nmi_constraint = true;
+ return;
+ }
-static struct expr_id *expr_ids__alloc(struct expr_ids *ids)
+ if (!violate_nmi_constraint)
+ return;
+
+ pr_warning("Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:\n"
+ " echo 0 > /proc/sys/kernel/nmi_watchdog\n"
+ " perf stat ...\n"
+ " echo 1 > /proc/sys/kernel/nmi_watchdog\n");
+}
+
+static bool metricgroup__has_constraint(const struct pmu_event *pe)
{
- if (ids->cnt >= RECURSION_ID_MAX)
- return NULL;
- return &ids->id[ids->cnt++];
+ if (!pe->metric_constraint)
+ return false;
+
+ if (!strcmp(pe->metric_constraint, "NO_NMI_WATCHDOG") &&
+ sysctl__nmi_watchdog_enabled()) {
+ metricgroup___watchdog_constraint_hint(pe->metric_name, false);
+ return true;
+ }
+
+ return false;
}
-static void expr_ids__exit(struct expr_ids *ids)
+static struct metric *metric__new(const struct pmu_event *pe,
+ const char *modifier,
+ bool metric_no_group,
+ int runtime)
{
- int i;
+ struct metric *m;
+
+ m = zalloc(sizeof(*m));
+ if (!m)
+ return NULL;
+
+ m->pctx = expr__ctx_new();
+ if (!m->pctx) {
+ free(m);
+ return NULL;
+ }
- for (i = 0; i < ids->cnt; i++)
- free(ids->id[i].id);
+ m->metric_name = pe->metric_name;
+ m->modifier = modifier ? strdup(modifier) : NULL;
+ if (modifier && !m->modifier) {
+ free(m);
+ expr__ctx_free(m->pctx);
+ return NULL;
+ }
+ m->metric_expr = pe->metric_expr;
+ m->metric_unit = pe->unit;
+ m->pctx->runtime = runtime;
+ m->has_constraint = metric_no_group || metricgroup__has_constraint(pe);
+ m->metric_refs = NULL;
+ m->evlist = NULL;
+
+ return m;
}
-static bool contains_event(struct evsel **metric_events, int num_events,
- const char *event_name)
+static void metric__free(struct metric *m)
+{
+ free(m->metric_refs);
+ expr__ctx_free(m->pctx);
+ free((char *)m->modifier);
+ evlist__delete(m->evlist);
+ free(m);
+}
+
+static bool contains_metric_id(struct evsel **metric_events, int num_events,
+ const char *metric_id)
{
int i;
for (i = 0; i < num_events; i++) {
- if (!strcmp(metric_events[i]->name, event_name))
+ if (!strcmp(evsel__metric_id(metric_events[i]), metric_id))
return true;
}
return false;
}
-static bool evsel_same_pmu_or_none(struct evsel *ev1, struct evsel *ev2)
-{
- if (!ev1->pmu_name || !ev2->pmu_name)
- return true;
-
- return !strcmp(ev1->pmu_name, ev2->pmu_name);
-}
-
/**
- * Find a group of events in perf_evlist that correspond to those from a parsed
- * metric expression. Note, as find_evsel_group is called in the same order as
- * perf_evlist was constructed, metric_no_merge doesn't need to test for
- * underfilling a group.
- * @perf_evlist: a list of events something like: {metric1 leader, metric1
- * sibling, metric1 sibling}:W,duration_time,{metric2 leader, metric2 sibling,
- * metric2 sibling}:W,duration_time
- * @pctx: the parse context for the metric expression.
- * @metric_no_merge: don't attempt to share events for the metric with other
- * metrics.
- * @has_constraint: is there a constraint on the group of events? In which case
- * the events won't be grouped.
- * @metric_events: out argument, null terminated array of evsel's associated
- * with the metric.
- * @evlist_used: in/out argument, bitmap tracking which evlist events are used.
- * @return the first metric event or NULL on failure.
+ * setup_metric_events - Find a group of events in metric_evlist that correspond
+ * to the IDs from a parsed metric expression.
+ * @ids: the metric IDs to match.
+ * @metric_evlist: the list of perf events.
+ * @out_metric_events: holds the created metric events array.
*/
-static struct evsel *find_evsel_group(struct evlist *perf_evlist,
- struct expr_parse_ctx *pctx,
- bool metric_no_merge,
- bool has_constraint,
- struct evsel **metric_events,
- unsigned long *evlist_used)
+static int setup_metric_events(struct hashmap *ids,
+ struct evlist *metric_evlist,
+ struct evsel ***out_metric_events)
{
- struct evsel *ev, *current_leader = NULL;
- struct expr_id_data *val_ptr;
- int i = 0, matched_events = 0, events_to_match;
- const int idnum = (int)hashmap__size(&pctx->ids);
+ struct evsel **metric_events;
+ const char *metric_id;
+ struct evsel *ev;
+ size_t ids_size, matched_events, i;
- /*
- * duration_time is always grouped separately, when events are grouped
- * (ie has_constraint is false) then ignore it in the matching loop and
- * add it to metric_events at the end.
- */
- if (!has_constraint &&
- hashmap__find(&pctx->ids, "duration_time", (void **)&val_ptr))
- events_to_match = idnum - 1;
- else
- events_to_match = idnum;
+ *out_metric_events = NULL;
+ ids_size = hashmap__size(ids);
+
+ metric_events = calloc(sizeof(void *), ids_size + 1);
+ if (!metric_events)
+ return -ENOMEM;
+
+ matched_events = 0;
+ evlist__for_each_entry(metric_evlist, ev) {
+ struct expr_id_data *val_ptr;
- evlist__for_each_entry (perf_evlist, ev) {
/*
- * Events with a constraint aren't grouped and match the first
- * events available.
+ * Check for duplicate events with the same name. For
+ * example, uncore_imc/cas_count_read/ will turn into 6
+ * events per socket on skylakex. Only the first such
+ * event is placed in metric_events.
*/
- if (has_constraint && ev->weak_group)
- continue;
- /* Ignore event if already used and merging is disabled. */
- if (metric_no_merge && test_bit(ev->core.idx, evlist_used))
+ metric_id = evsel__metric_id(ev);
+ if (contains_metric_id(metric_events, matched_events, metric_id))
continue;
- if (!has_constraint && !evsel__has_leader(ev, current_leader)) {
- /*
- * Start of a new group, discard the whole match and
- * start again.
- */
- matched_events = 0;
- memset(metric_events, 0,
- sizeof(struct evsel *) * idnum);
- current_leader = evsel__leader(ev);
- }
/*
- * Check for duplicate events with the same name. For example,
- * uncore_imc/cas_count_read/ will turn into 6 events per socket
- * on skylakex. Only the first such event is placed in
- * metric_events. If events aren't grouped then this also
- * ensures that the same event in different sibling groups
- * aren't both added to metric_events.
+ * Does this event belong to the parse context? For
+ * combined or shared groups, this metric may not care
+ * about this event.
*/
- if (contains_event(metric_events, matched_events, ev->name))
- continue;
- /* Does this event belong to the parse context? */
- if (hashmap__find(&pctx->ids, ev->name, (void **)&val_ptr))
+ if (hashmap__find(ids, metric_id, (void **)&val_ptr)) {
metric_events[matched_events++] = ev;
- if (matched_events == events_to_match)
- break;
- }
-
- if (events_to_match != idnum) {
- /* Add the first duration_time. */
- evlist__for_each_entry(perf_evlist, ev) {
- if (!strcmp(ev->name, "duration_time")) {
- metric_events[matched_events++] = ev;
+ if (matched_events >= ids_size)
break;
- }
}
}
-
- if (matched_events != idnum) {
- /* Not a whole match */
- return NULL;
+ if (matched_events < ids_size) {
+ free(metric_events);
+ return -EINVAL;
}
-
- metric_events[idnum] = NULL;
-
- for (i = 0; i < idnum; i++) {
+ for (i = 0; i < ids_size; i++) {
ev = metric_events[i];
- /* Don't free the used events. */
- set_bit(ev->core.idx, evlist_used);
+ ev->collect_stat = true;
+
/*
- * The metric leader points to the identically named event in
- * metric_events.
+ * The metric leader points to the identically named
+ * event in metric_events.
*/
ev->metric_leader = ev;
/*
- * Mark two events with identical names in the same group (or
- * globally) as being in use as uncore events may be duplicated
- * for each pmu. Set the metric leader of such events to be the
- * event that appears in metric_events.
+ * Mark two events with identical names in the same
+ * group (or globally) as being in use as uncore events
+ * may be duplicated for each pmu. Set the metric leader
+ * of such events to be the event that appears in
+ * metric_events.
*/
- evlist__for_each_entry_continue(perf_evlist, ev) {
- /*
- * If events are grouped then the search can terminate
- * when then group is left.
- */
- if (!has_constraint &&
- ev->core.leader != metric_events[i]->core.leader &&
- evsel_same_pmu_or_none(evsel__leader(ev), evsel__leader(metric_events[i])))
- break;
- if (!strcmp(metric_events[i]->name, ev->name)) {
- set_bit(ev->core.idx, evlist_used);
+ metric_id = evsel__metric_id(ev);
+ evlist__for_each_entry_continue(metric_evlist, ev) {
+ if (!strcmp(evsel__metric_id(metric_events[i]), metric_id))
ev->metric_leader = metric_events[i];
- }
}
}
-
- return metric_events[0];
-}
-
-static int metricgroup__setup_events(struct list_head *groups,
- bool metric_no_merge,
- struct evlist *perf_evlist,
- struct rblist *metric_events_list)
-{
- struct metric_event *me;
- struct metric_expr *expr;
- int i = 0;
- int ret = 0;
- struct metric *m;
- struct evsel *evsel, *tmp;
- unsigned long *evlist_used;
-
- evlist_used = bitmap_zalloc(perf_evlist->core.nr_entries);
- if (!evlist_used)
- return -ENOMEM;
-
- list_for_each_entry (m, groups, nd) {
- struct evsel **metric_events;
- struct metric_ref *metric_refs = NULL;
-
- metric_events = calloc(sizeof(void *),
- hashmap__size(&m->pctx.ids) + 1);
- if (!metric_events) {
- ret = -ENOMEM;
- break;
- }
- evsel = find_evsel_group(perf_evlist, &m->pctx,
- metric_no_merge,
- m->has_constraint, metric_events,
- evlist_used);
- if (!evsel) {
- pr_debug("Cannot resolve %s: %s\n",
- m->metric_name, m->metric_expr);
- free(metric_events);
- continue;
- }
- for (i = 0; metric_events[i]; i++)
- metric_events[i]->collect_stat = true;
- me = metricgroup__lookup(metric_events_list, evsel, true);
- if (!me) {
- ret = -ENOMEM;
- free(metric_events);
- break;
- }
- expr = malloc(sizeof(struct metric_expr));
- if (!expr) {
- ret = -ENOMEM;
- free(metric_events);
- break;
- }
-
- /*
- * Collect and store collected nested expressions
- * for metric processing.
- */
- if (m->metric_refs_cnt) {
- struct metric_ref_node *ref;
-
- metric_refs = zalloc(sizeof(struct metric_ref) * (m->metric_refs_cnt + 1));
- if (!metric_refs) {
- ret = -ENOMEM;
- free(metric_events);
- free(expr);
- break;
- }
-
- i = 0;
- list_for_each_entry(ref, &m->metric_refs, list) {
- /*
- * Intentionally passing just const char pointers,
- * originally from 'struct pmu_event' object.
- * We don't need to change them, so there's no
- * need to create our own copy.
- */
- metric_refs[i].metric_name = ref->metric_name;
- metric_refs[i].metric_expr = ref->metric_expr;
- i++;
- }
- }
-
- expr->metric_refs = metric_refs;
- expr->metric_expr = m->metric_expr;
- expr->metric_name = m->metric_name;
- expr->metric_unit = m->metric_unit;
- expr->metric_events = metric_events;
- expr->runtime = m->runtime;
- list_add(&expr->nd, &me->head);
- }
-
- evlist__for_each_entry_safe(perf_evlist, tmp, evsel) {
- if (!test_bit(evsel->core.idx, evlist_used)) {
- evlist__remove(perf_evlist, evsel);
- evsel__delete(evsel);
- }
- }
- bitmap_free(evlist_used);
-
- return ret;
+ *out_metric_events = metric_events;
+ return 0;
}
static bool match_metric(const char *n, const char *list)
return false;
}
-static bool match_pe_metric(struct pmu_event *pe, const char *metric)
+static bool match_pe_metric(const struct pmu_event *pe, const char *metric)
{
return match_metric(pe->metric_group, metric) ||
match_metric(pe->metric_name, metric);
putchar('\n');
}
-static int metricgroup__print_pmu_event(struct pmu_event *pe,
+static int metricgroup__print_pmu_event(const struct pmu_event *pe,
bool metricgroups, char *filter,
bool raw, bool details,
struct rblist *groups,
bool details;
};
-typedef int (*metricgroup_sys_event_iter_fn)(struct pmu_event *pe, void *);
+typedef int (*metricgroup_sys_event_iter_fn)(const struct pmu_event *pe, void *);
struct metricgroup_iter_data {
metricgroup_sys_event_iter_fn fn;
void *data;
};
-static int metricgroup__sys_event_iter(struct pmu_event *pe, void *data)
+static int metricgroup__sys_event_iter(const struct pmu_event *pe, void *data)
{
struct metricgroup_iter_data *d = data;
struct perf_pmu *pmu = NULL;
return 0;
}
-static int metricgroup__print_sys_event_iter(struct pmu_event *pe, void *data)
+static int metricgroup__print_sys_event_iter(const struct pmu_event *pe, void *data)
{
struct metricgroup_print_sys_idata *d = data;
}
void metricgroup__print(bool metrics, bool metricgroups, char *filter,
- bool raw, bool details)
+ bool raw, bool details, const char *pmu_name)
{
- struct pmu_events_map *map = pmu_events_map__find();
- struct pmu_event *pe;
+ const struct pmu_events_map *map = pmu_events_map__find();
+ const struct pmu_event *pe;
int i;
struct rblist groups;
struct rb_node *node, *next;
break;
if (!pe->metric_expr)
continue;
+ if (pmu_name && perf_pmu__is_hybrid(pe->pmu) &&
+ strcmp(pmu_name, pe->pmu)) {
+ continue;
+ }
if (metricgroup__print_pmu_event(pe, metricgroups, filter,
raw, details, &groups,
metriclist) < 0)
strlist__delete(metriclist);
}
-static void metricgroup__add_metric_weak_group(struct strbuf *events,
- struct expr_parse_ctx *ctx)
+static const char *code_characters = ",-=@";
+
+static int encode_metric_id(struct strbuf *sb, const char *x)
{
- struct hashmap_entry *cur;
- size_t bkt;
- bool no_group = true, has_duration = false;
+ char *c;
+ int ret = 0;
- hashmap__for_each_entry((&ctx->ids), cur, bkt) {
- pr_debug("found event %s\n", (const char *)cur->key);
- /*
- * Duration time maps to a software event and can make
- * groups not count. Always use it outside a
- * group.
- */
- if (!strcmp(cur->key, "duration_time")) {
- has_duration = true;
- continue;
+ for (; *x; x++) {
+ c = strchr(code_characters, *x);
+ if (c) {
+ ret = strbuf_addch(sb, '!');
+ if (ret)
+ break;
+
+ ret = strbuf_addch(sb, '0' + (c - code_characters));
+ if (ret)
+ break;
+ } else {
+ ret = strbuf_addch(sb, *x);
+ if (ret)
+ break;
}
- strbuf_addf(events, "%s%s",
- no_group ? "{" : ",",
- (const char *)cur->key);
- no_group = false;
}
- if (!no_group) {
- strbuf_addf(events, "}:W");
- if (has_duration)
- strbuf_addf(events, ",duration_time");
- } else if (has_duration)
- strbuf_addf(events, "duration_time");
+ return ret;
}
-static void metricgroup__add_metric_non_group(struct strbuf *events,
- struct expr_parse_ctx *ctx)
+static int decode_metric_id(struct strbuf *sb, const char *x)
{
- struct hashmap_entry *cur;
- size_t bkt;
- bool first = true;
+ const char *orig = x;
+ size_t i;
+ char c;
+ int ret;
- hashmap__for_each_entry((&ctx->ids), cur, bkt) {
- if (!first)
- strbuf_addf(events, ",");
- strbuf_addf(events, "%s", (const char *)cur->key);
- first = false;
+ for (; *x; x++) {
+ c = *x;
+ if (*x == '!') {
+ x++;
+ i = *x - '0';
+ if (i > strlen(code_characters)) {
+ pr_err("Bad metric-id encoding in: '%s'", orig);
+ return -1;
+ }
+ c = code_characters[i];
+ }
+ ret = strbuf_addch(sb, c);
+ if (ret)
+ return ret;
}
+ return 0;
}
-static void metricgroup___watchdog_constraint_hint(const char *name, bool foot)
+static int decode_all_metric_ids(struct evlist *perf_evlist, const char *modifier)
{
- static bool violate_nmi_constraint;
+ struct evsel *ev;
+ struct strbuf sb = STRBUF_INIT;
+ char *cur;
+ int ret = 0;
- if (!foot) {
- pr_warning("Splitting metric group %s into standalone metrics.\n", name);
- violate_nmi_constraint = true;
- return;
- }
+ evlist__for_each_entry(perf_evlist, ev) {
+ if (!ev->metric_id)
+ continue;
- if (!violate_nmi_constraint)
- return;
+ ret = strbuf_setlen(&sb, 0);
+ if (ret)
+ break;
- pr_warning("Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:\n"
- " echo 0 > /proc/sys/kernel/nmi_watchdog\n"
- " perf stat ...\n"
- " echo 1 > /proc/sys/kernel/nmi_watchdog\n");
+ ret = decode_metric_id(&sb, ev->metric_id);
+ if (ret)
+ break;
+
+ free((char *)ev->metric_id);
+ ev->metric_id = strdup(sb.buf);
+ if (!ev->metric_id) {
+ ret = -ENOMEM;
+ break;
+ }
+ /*
+ * If the name is just the parsed event, use the metric-id to
+ * give a more friendly display version.
+ */
+ if (strstr(ev->name, "metric-id=")) {
+ bool has_slash = false;
+
+ free(ev->name);
+ for (cur = strchr(sb.buf, '@') ; cur; cur = strchr(++cur, '@')) {
+ *cur = '/';
+ has_slash = true;
+ }
+
+ if (modifier) {
+ if (!has_slash && !strchr(sb.buf, ':')) {
+ ret = strbuf_addch(&sb, ':');
+ if (ret)
+ break;
+ }
+ ret = strbuf_addstr(&sb, modifier);
+ if (ret)
+ break;
+ }
+ ev->name = strdup(sb.buf);
+ if (!ev->name) {
+ ret = -ENOMEM;
+ break;
+ }
+ }
+ }
+ strbuf_release(&sb);
+ return ret;
}
-static bool metricgroup__has_constraint(struct pmu_event *pe)
+static int metricgroup__build_event_string(struct strbuf *events,
+ const struct expr_parse_ctx *ctx,
+ const char *modifier,
+ bool has_constraint)
{
- if (!pe->metric_constraint)
- return false;
+ struct hashmap_entry *cur;
+ size_t bkt;
+ bool no_group = true, has_duration = false;
+ int ret = 0;
- if (!strcmp(pe->metric_constraint, "NO_NMI_WATCHDOG") &&
- sysctl__nmi_watchdog_enabled()) {
- metricgroup___watchdog_constraint_hint(pe->metric_name, false);
- return true;
+#define RETURN_IF_NON_ZERO(x) do { if (x) return x; } while (0)
+
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
+ const char *sep, *rsep, *id = cur->key;
+
+ pr_debug("found event %s\n", id);
+ /*
+ * Duration time maps to a software event and can make
+ * groups not count. Always use it outside a
+ * group.
+ */
+ if (!strcmp(id, "duration_time")) {
+ has_duration = true;
+ continue;
+ }
+ /* Separate events with commas and open the group if necessary. */
+ if (no_group) {
+ if (!has_constraint) {
+ ret = strbuf_addch(events, '{');
+ RETURN_IF_NON_ZERO(ret);
+ }
+
+ no_group = false;
+ } else {
+ ret = strbuf_addch(events, ',');
+ RETURN_IF_NON_ZERO(ret);
+ }
+ /*
+ * Encode the ID as an event string. Add a qualifier for
+ * metric_id that is the original name except with characters
+ * that parse-events can't parse replaced. For example,
+ * 'msr@tsc@' gets added as msr/tsc,metric-id=msr!3tsc!3/
+ */
+ sep = strchr(id, '@');
+ if (sep != NULL) {
+ ret = strbuf_add(events, id, sep - id);
+ RETURN_IF_NON_ZERO(ret);
+ ret = strbuf_addch(events, '/');
+ RETURN_IF_NON_ZERO(ret);
+ rsep = strrchr(sep, '@');
+ ret = strbuf_add(events, sep + 1, rsep - sep - 1);
+ RETURN_IF_NON_ZERO(ret);
+ ret = strbuf_addstr(events, ",metric-id=");
+ RETURN_IF_NON_ZERO(ret);
+ sep = rsep;
+ } else {
+ sep = strchr(id, ':');
+ if (sep != NULL) {
+ ret = strbuf_add(events, id, sep - id);
+ RETURN_IF_NON_ZERO(ret);
+ } else {
+ ret = strbuf_addstr(events, id);
+ RETURN_IF_NON_ZERO(ret);
+ }
+ ret = strbuf_addstr(events, "/metric-id=");
+ RETURN_IF_NON_ZERO(ret);
+ }
+ ret = encode_metric_id(events, id);
+ RETURN_IF_NON_ZERO(ret);
+ ret = strbuf_addstr(events, "/");
+ RETURN_IF_NON_ZERO(ret);
+
+ if (sep != NULL) {
+ ret = strbuf_addstr(events, sep + 1);
+ RETURN_IF_NON_ZERO(ret);
+ }
+ if (modifier) {
+ ret = strbuf_addstr(events, modifier);
+ RETURN_IF_NON_ZERO(ret);
+ }
}
+ if (has_duration) {
+ if (no_group) {
+ /* Strange case of a metric of just duration_time. */
+ ret = strbuf_addf(events, "duration_time");
+ } else if (!has_constraint)
+ ret = strbuf_addf(events, "}:W,duration_time");
+ else
+ ret = strbuf_addf(events, ",duration_time");
+ } else if (!no_group && !has_constraint)
+ ret = strbuf_addf(events, "}:W");
- return false;
+ return ret;
+#undef RETURN_IF_NON_ZERO
}
-int __weak arch_get_runtimeparam(struct pmu_event *pe __maybe_unused)
+int __weak arch_get_runtimeparam(const struct pmu_event *pe __maybe_unused)
{
return 1;
}
+/*
+ * A singly linked list on the stack of the names of metrics being
+ * processed. Used to identify recursion.
+ */
+struct visited_metric {
+ const char *name;
+ const struct visited_metric *parent;
+};
+
struct metricgroup_add_iter_data {
struct list_head *metric_list;
- const char *metric;
- struct expr_ids *ids;
+ const char *metric_name;
+ const char *modifier;
int *ret;
bool *has_match;
bool metric_no_group;
+ struct metric *root_metric;
+ const struct visited_metric *visited;
+ const struct pmu_events_map *map;
};
+static int add_metric(struct list_head *metric_list,
+ const struct pmu_event *pe,
+ const char *modifier,
+ bool metric_no_group,
+ struct metric *root_metric,
+ const struct visited_metric *visited,
+ const struct pmu_events_map *map);
+
+/**
+ * resolve_metric - Locate metrics within the root metric and recursively add
+ * references to them.
+ * @metric_list: The list the metric is added to.
+ * @modifier: if non-null event modifiers like "u".
+ * @metric_no_group: Should events written to events be grouped "{}" or
+ * global. Grouping is the default but due to multiplexing the
+ * user may override.
+ * @root_metric: Metrics may reference other metrics to form a tree. In this
+ * case the root_metric holds all the IDs and a list of referenced
+ * metrics. When adding a root this argument is NULL.
+ * @visited: A singly linked list of metric names being added that is used to
+ * detect recursion.
+ * @map: The map that is searched for metrics, most commonly the table for the
+ * architecture perf is running upon.
+ */
+static int resolve_metric(struct list_head *metric_list,
+ const char *modifier,
+ bool metric_no_group,
+ struct metric *root_metric,
+ const struct visited_metric *visited,
+ const struct pmu_events_map *map)
+{
+ struct hashmap_entry *cur;
+ size_t bkt;
+ struct to_resolve {
+ /* The metric to resolve. */
+ const struct pmu_event *pe;
+ /*
+ * The key in the IDs map, this may differ from in case,
+ * etc. from pe->metric_name.
+ */
+ const char *key;
+ } *pending = NULL;
+ int i, ret = 0, pending_cnt = 0;
+
+ /*
+ * Iterate all the parsed IDs and if there's a matching metric and it to
+ * the pending array.
+ */
+ hashmap__for_each_entry(root_metric->pctx->ids, cur, bkt) {
+ const struct pmu_event *pe;
+
+ pe = metricgroup__find_metric(cur->key, map);
+ if (pe) {
+ pending = realloc(pending,
+ (pending_cnt + 1) * sizeof(struct to_resolve));
+ if (!pending)
+ return -ENOMEM;
+
+ pending[pending_cnt].pe = pe;
+ pending[pending_cnt].key = cur->key;
+ pending_cnt++;
+ }
+ }
+
+ /* Remove the metric IDs from the context. */
+ for (i = 0; i < pending_cnt; i++)
+ expr__del_id(root_metric->pctx, pending[i].key);
+
+ /*
+ * Recursively add all the metrics, IDs are added to the root metric's
+ * context.
+ */
+ for (i = 0; i < pending_cnt; i++) {
+ ret = add_metric(metric_list, pending[i].pe, modifier, metric_no_group,
+ root_metric, visited, map);
+ if (ret)
+ break;
+ }
+
+ free(pending);
+ return ret;
+}
+
+/**
+ * __add_metric - Add a metric to metric_list.
+ * @metric_list: The list the metric is added to.
+ * @pe: The pmu_event containing the metric to be added.
+ * @modifier: if non-null event modifiers like "u".
+ * @metric_no_group: Should events written to events be grouped "{}" or
+ * global. Grouping is the default but due to multiplexing the
+ * user may override.
+ * @runtime: A special argument for the parser only known at runtime.
+ * @root_metric: Metrics may reference other metrics to form a tree. In this
+ * case the root_metric holds all the IDs and a list of referenced
+ * metrics. When adding a root this argument is NULL.
+ * @visited: A singly linked list of metric names being added that is used to
+ * detect recursion.
+ * @map: The map that is searched for metrics, most commonly the table for the
+ * architecture perf is running upon.
+ */
static int __add_metric(struct list_head *metric_list,
- struct pmu_event *pe,
+ const struct pmu_event *pe,
+ const char *modifier,
bool metric_no_group,
int runtime,
- struct metric **mp,
- struct expr_id *parent,
- struct expr_ids *ids)
+ struct metric *root_metric,
+ const struct visited_metric *visited,
+ const struct pmu_events_map *map)
{
- struct metric_ref_node *ref;
- struct metric *m;
+ const struct visited_metric *vm;
+ int ret;
+ bool is_root = !root_metric;
+ struct visited_metric visited_node = {
+ .name = pe->metric_name,
+ .parent = visited,
+ };
- if (*mp == NULL) {
+ for (vm = visited; vm; vm = vm->parent) {
+ if (!strcmp(pe->metric_name, vm->name)) {
+ pr_err("failed: recursion detected for %s\n", pe->metric_name);
+ return -1;
+ }
+ }
+
+ if (is_root) {
/*
- * We got in here for the parent group,
- * allocate it and put it on the list.
+ * This metric is the root of a tree and may reference other
+ * metrics that are added recursively.
*/
- m = zalloc(sizeof(*m));
- if (!m)
+ root_metric = metric__new(pe, modifier, metric_no_group, runtime);
+ if (!root_metric)
return -ENOMEM;
- expr__ctx_init(&m->pctx);
- m->metric_name = pe->metric_name;
- m->metric_expr = pe->metric_expr;
- m->metric_unit = pe->unit;
- m->runtime = runtime;
- m->has_constraint = metric_no_group || metricgroup__has_constraint(pe);
- INIT_LIST_HEAD(&m->metric_refs);
- m->metric_refs_cnt = 0;
-
- parent = expr_ids__alloc(ids);
- if (!parent) {
- free(m);
- return -EINVAL;
- }
-
- parent->id = strdup(pe->metric_name);
- if (!parent->id) {
- free(m);
- return -ENOMEM;
- }
- *mp = m;
} else {
+ int cnt = 0;
+
/*
- * We got here for the referenced metric, via the
- * recursive metricgroup__add_metric call, add
- * it to the parent group.
+ * This metric was referenced in a metric higher in the
+ * tree. Check if the same metric is already resolved in the
+ * metric_refs list.
*/
- m = *mp;
+ if (root_metric->metric_refs) {
+ for (; root_metric->metric_refs[cnt].metric_name; cnt++) {
+ if (!strcmp(pe->metric_name,
+ root_metric->metric_refs[cnt].metric_name))
+ return 0;
+ }
+ }
- ref = malloc(sizeof(*ref));
- if (!ref)
+ /* Create reference. Need space for the entry and the terminator. */
+ root_metric->metric_refs = realloc(root_metric->metric_refs,
+ (cnt + 2) * sizeof(struct metric_ref));
+ if (!root_metric->metric_refs)
return -ENOMEM;
/*
* need to change them, so there's no need to create
* our own copy.
*/
- ref->metric_name = pe->metric_name;
- ref->metric_expr = pe->metric_expr;
+ root_metric->metric_refs[cnt].metric_name = pe->metric_name;
+ root_metric->metric_refs[cnt].metric_expr = pe->metric_expr;
- list_add(&ref->list, &m->metric_refs);
- m->metric_refs_cnt++;
+ /* Null terminate array. */
+ root_metric->metric_refs[cnt+1].metric_name = NULL;
+ root_metric->metric_refs[cnt+1].metric_expr = NULL;
}
- /* Force all found IDs in metric to have us as parent ID. */
- WARN_ON_ONCE(!parent);
- m->pctx.parent = parent;
-
/*
* For both the parent and referenced metrics, we parse
- * all the metric's IDs and add it to the parent context.
+ * all the metric's IDs and add it to the root context.
*/
- if (expr__find_other(pe->metric_expr, NULL, &m->pctx, runtime) < 0) {
- if (m->metric_refs_cnt == 0) {
- expr__ctx_clear(&m->pctx);
- free(m);
- *mp = NULL;
- }
- return -EINVAL;
+ if (expr__find_ids(pe->metric_expr, NULL, root_metric->pctx) < 0) {
+ /* Broken metric. */
+ ret = -EINVAL;
+ } else {
+ /* Resolve referenced metrics. */
+ ret = resolve_metric(metric_list, modifier, metric_no_group, root_metric,
+ &visited_node, map);
}
- /*
- * We add new group only in the 'parent' call,
- * so bail out for referenced metric case.
- */
- if (m->metric_refs_cnt)
- return 0;
-
- if (list_empty(metric_list))
- list_add(&m->nd, metric_list);
- else {
- struct list_head *pos;
-
- /* Place the largest groups at the front. */
- list_for_each_prev(pos, metric_list) {
- struct metric *old = list_entry(pos, struct metric, nd);
+ if (ret) {
+ if (is_root)
+ metric__free(root_metric);
- if (hashmap__size(&m->pctx.ids) <=
- hashmap__size(&old->pctx.ids))
- break;
- }
- list_add(&m->nd, pos);
- }
+ } else if (is_root)
+ list_add(&root_metric->nd, metric_list);
- return 0;
+ return ret;
}
#define map_for_each_event(__pe, __idx, __map) \
(match_metric(__pe->metric_group, __metric) || \
match_metric(__pe->metric_name, __metric)))
-struct pmu_event *metricgroup__find_metric(const char *metric,
- struct pmu_events_map *map)
+const struct pmu_event *metricgroup__find_metric(const char *metric,
+ const struct pmu_events_map *map)
{
- struct pmu_event *pe;
+ const struct pmu_event *pe;
int i;
map_for_each_event(pe, i, map) {
return NULL;
}
-static int recursion_check(struct metric *m, const char *id, struct expr_id **parent,
- struct expr_ids *ids)
-{
- struct expr_id_data *data;
- struct expr_id *p;
- int ret;
-
- /*
- * We get the parent referenced by 'id' argument and
- * traverse through all the parent object IDs to check
- * if we already processed 'id', if we did, it's recursion
- * and we fail.
- */
- ret = expr__get_id(&m->pctx, id, &data);
- if (ret)
- return ret;
-
- p = expr_id_data__parent(data);
-
- while (p->parent) {
- if (!strcmp(p->id, id)) {
- pr_err("failed: recursion detected for %s\n", id);
- return -1;
- }
- p = p->parent;
- }
-
- /*
- * If we are over the limit of static entris, the metric
- * is too difficult/nested to process, fail as well.
- */
- p = expr_ids__alloc(ids);
- if (!p) {
- pr_err("failed: too many nested metrics\n");
- return -EINVAL;
- }
-
- p->id = strdup(id);
- p->parent = expr_id_data__parent(data);
- *parent = p;
-
- return p->id ? 0 : -ENOMEM;
-}
-
static int add_metric(struct list_head *metric_list,
- struct pmu_event *pe,
+ const struct pmu_event *pe,
+ const char *modifier,
bool metric_no_group,
- struct metric **mp,
- struct expr_id *parent,
- struct expr_ids *ids);
-
-static int __resolve_metric(struct metric *m,
- bool metric_no_group,
- struct list_head *metric_list,
- struct pmu_events_map *map,
- struct expr_ids *ids)
+ struct metric *root_metric,
+ const struct visited_metric *visited,
+ const struct pmu_events_map *map)
{
- struct hashmap_entry *cur;
- size_t bkt;
- bool all;
- int ret;
-
- /*
- * Iterate all the parsed IDs and if there's metric,
- * add it to the context.
- */
- do {
- all = true;
- hashmap__for_each_entry((&m->pctx.ids), cur, bkt) {
- struct expr_id *parent;
- struct pmu_event *pe;
-
- pe = metricgroup__find_metric(cur->key, map);
- if (!pe)
- continue;
-
- ret = recursion_check(m, cur->key, &parent, ids);
- if (ret)
- return ret;
-
- all = false;
- /* The metric key itself needs to go out.. */
- expr__del_id(&m->pctx, cur->key);
-
- /* ... and it gets resolved to the parent context. */
- ret = add_metric(metric_list, pe, metric_no_group, &m, parent, ids);
- if (ret)
- return ret;
-
- /*
- * We added new metric to hashmap, so we need
- * to break the iteration and start over.
- */
- break;
- }
- } while (!all);
-
- return 0;
-}
-
-static int resolve_metric(bool metric_no_group,
- struct list_head *metric_list,
- struct pmu_events_map *map,
- struct expr_ids *ids)
-{
- struct metric *m;
- int err;
-
- list_for_each_entry(m, metric_list, nd) {
- err = __resolve_metric(m, metric_no_group, metric_list, map, ids);
- if (err)
- return err;
- }
- return 0;
-}
-
-static int add_metric(struct list_head *metric_list,
- struct pmu_event *pe,
- bool metric_no_group,
- struct metric **m,
- struct expr_id *parent,
- struct expr_ids *ids)
-{
- struct metric *orig = *m;
int ret = 0;
pr_debug("metric expr %s for %s\n", pe->metric_expr, pe->metric_name);
if (!strstr(pe->metric_expr, "?")) {
- ret = __add_metric(metric_list, pe, metric_no_group, 1, m, parent, ids);
+ ret = __add_metric(metric_list, pe, modifier, metric_no_group, 0,
+ root_metric, visited, map);
} else {
int j, count;
* those events to metric_list.
*/
- for (j = 0; j < count && !ret; j++, *m = orig)
- ret = __add_metric(metric_list, pe, metric_no_group, j, m, parent, ids);
+ for (j = 0; j < count && !ret; j++)
+ ret = __add_metric(metric_list, pe, modifier, metric_no_group, j,
+ root_metric, visited, map);
}
return ret;
}
-static int metricgroup__add_metric_sys_event_iter(struct pmu_event *pe,
+static int metricgroup__add_metric_sys_event_iter(const struct pmu_event *pe,
void *data)
{
struct metricgroup_add_iter_data *d = data;
- struct metric *m = NULL;
int ret;
- if (!match_pe_metric(pe, d->metric))
+ if (!match_pe_metric(pe, d->metric_name))
return 0;
- ret = add_metric(d->metric_list, pe, d->metric_no_group, &m, NULL, d->ids);
- if (ret)
- goto out;
-
- ret = resolve_metric(d->metric_no_group,
- d->metric_list, NULL, d->ids);
+ ret = add_metric(d->metric_list, pe, d->modifier, d->metric_no_group,
+ d->root_metric, d->visited, d->map);
if (ret)
goto out;
return ret;
}
-static int metricgroup__add_metric(const char *metric, bool metric_no_group,
- struct strbuf *events,
+static int metric_list_cmp(void *priv __maybe_unused, const struct list_head *l,
+ const struct list_head *r)
+{
+ const struct metric *left = container_of(l, struct metric, nd);
+ const struct metric *right = container_of(r, struct metric, nd);
+
+ return hashmap__size(right->pctx->ids) - hashmap__size(left->pctx->ids);
+}
+
+/**
+ * metricgroup__add_metric - Find and add a metric, or a metric group.
+ * @metric_name: The name of the metric or metric group. For example, "IPC"
+ * could be the name of a metric and "TopDownL1" the name of a
+ * metric group.
+ * @modifier: if non-null event modifiers like "u".
+ * @metric_no_group: Should events written to events be grouped "{}" or
+ * global. Grouping is the default but due to multiplexing the
+ * user may override.
+ * @metric_list: The list that the metric or metric group are added to.
+ * @map: The map that is searched for metrics, most commonly the table for the
+ * architecture perf is running upon.
+ */
+static int metricgroup__add_metric(const char *metric_name, const char *modifier,
+ bool metric_no_group,
struct list_head *metric_list,
- struct pmu_events_map *map)
+ const struct pmu_events_map *map)
{
- struct expr_ids ids = { .cnt = 0, };
- struct pmu_event *pe;
- struct metric *m;
+ const struct pmu_event *pe;
LIST_HEAD(list);
int i, ret;
bool has_match = false;
- map_for_each_metric(pe, i, map, metric) {
+ /*
+ * Iterate over all metrics seeing if metric matches either the name or
+ * group. When it does add the metric to the list.
+ */
+ map_for_each_metric(pe, i, map, metric_name) {
has_match = true;
- m = NULL;
-
- ret = add_metric(&list, pe, metric_no_group, &m, NULL, &ids);
- if (ret)
- goto out;
-
- /*
- * Process any possible referenced metrics
- * included in the expression.
- */
- ret = resolve_metric(metric_no_group,
- &list, map, &ids);
+ ret = add_metric(&list, pe, modifier, metric_no_group,
+ /*root_metric=*/NULL,
+ /*visited_metrics=*/NULL, map);
if (ret)
goto out;
}
.fn = metricgroup__add_metric_sys_event_iter,
.data = (void *) &(struct metricgroup_add_iter_data) {
.metric_list = &list,
- .metric = metric,
+ .metric_name = metric_name,
+ .modifier = modifier,
.metric_no_group = metric_no_group,
- .ids = &ids,
.has_match = &has_match,
.ret = &ret,
+ .map = map,
},
};
pmu_for_each_sys_event(metricgroup__sys_event_iter, &data);
}
/* End of pmu events. */
- if (!has_match) {
+ if (!has_match)
ret = -EINVAL;
- goto out;
- }
-
- list_for_each_entry(m, &list, nd) {
- if (events->len > 0)
- strbuf_addf(events, ",");
-
- if (m->has_constraint) {
- metricgroup__add_metric_non_group(events,
- &m->pctx);
- } else {
- metricgroup__add_metric_weak_group(events,
- &m->pctx);
- }
- }
out:
/*
* even if it's failed
*/
list_splice(&list, metric_list);
- expr_ids__exit(&ids);
return ret;
}
+/**
+ * metricgroup__add_metric_list - Find and add metrics, or metric groups,
+ * specified in a list.
+ * @list: the list of metrics or metric groups. For example, "IPC,CPI,TopDownL1"
+ * would match the IPC and CPI metrics, and TopDownL1 would match all
+ * the metrics in the TopDownL1 group.
+ * @metric_no_group: Should events written to events be grouped "{}" or
+ * global. Grouping is the default but due to multiplexing the
+ * user may override.
+ * @metric_list: The list that metrics are added to.
+ * @map: The map that is searched for metrics, most commonly the table for the
+ * architecture perf is running upon.
+ */
static int metricgroup__add_metric_list(const char *list, bool metric_no_group,
- struct strbuf *events,
struct list_head *metric_list,
- struct pmu_events_map *map)
+ const struct pmu_events_map *map)
{
- char *llist, *nlist, *p;
- int ret = -EINVAL;
+ char *list_itr, *list_copy, *metric_name, *modifier;
+ int ret, count = 0;
- nlist = strdup(list);
- if (!nlist)
+ list_copy = strdup(list);
+ if (!list_copy)
return -ENOMEM;
- llist = nlist;
+ list_itr = list_copy;
+
+ while ((metric_name = strsep(&list_itr, ",")) != NULL) {
+ modifier = strchr(metric_name, ':');
+ if (modifier)
+ *modifier++ = '\0';
- strbuf_init(events, 100);
- strbuf_addf(events, "%s", "");
+ ret = metricgroup__add_metric(metric_name, modifier,
+ metric_no_group, metric_list,
+ map);
+ if (ret == -EINVAL)
+ pr_err("Cannot find metric or group `%s'\n", metric_name);
- while ((p = strsep(&llist, ",")) != NULL) {
- ret = metricgroup__add_metric(p, metric_no_group, events,
- metric_list, map);
- if (ret == -EINVAL) {
- fprintf(stderr, "Cannot find metric or group `%s'\n",
- p);
+ if (ret)
break;
- }
+
+ count++;
}
- free(nlist);
+ free(list_copy);
- if (!ret)
+ if (!ret) {
+ /*
+ * Warn about nmi_watchdog if any parsed metrics had the
+ * NO_NMI_WATCHDOG constraint.
+ */
metricgroup___watchdog_constraint_hint(NULL, true);
-
+ /* No metrics. */
+ if (count == 0)
+ return -EINVAL;
+ }
return ret;
}
-static void metric__free_refs(struct metric *metric)
+static void metricgroup__free_metrics(struct list_head *metric_list)
{
- struct metric_ref_node *ref, *tmp;
+ struct metric *m, *tmp;
- list_for_each_entry_safe(ref, tmp, &metric->metric_refs, list) {
- list_del(&ref->list);
- free(ref);
+ list_for_each_entry_safe (m, tmp, metric_list, nd) {
+ list_del_init(&m->nd);
+ metric__free(m);
}
}
-static void metricgroup__free_metrics(struct list_head *metric_list)
+/**
+ * build_combined_expr_ctx - Make an expr_parse_ctx with all has_constraint
+ * metric IDs, as the IDs are held in a set,
+ * duplicates will be removed.
+ * @metric_list: List to take metrics from.
+ * @combined: Out argument for result.
+ */
+static int build_combined_expr_ctx(const struct list_head *metric_list,
+ struct expr_parse_ctx **combined)
{
- struct metric *m, *tmp;
+ struct hashmap_entry *cur;
+ size_t bkt;
+ struct metric *m;
+ char *dup;
+ int ret;
- list_for_each_entry_safe (m, tmp, metric_list, nd) {
- metric__free_refs(m);
- expr__ctx_clear(&m->pctx);
- list_del_init(&m->nd);
- free(m);
+ *combined = expr__ctx_new();
+ if (!*combined)
+ return -ENOMEM;
+
+ list_for_each_entry(m, metric_list, nd) {
+ if (m->has_constraint && !m->modifier) {
+ hashmap__for_each_entry(m->pctx->ids, cur, bkt) {
+ dup = strdup(cur->key);
+ if (!dup) {
+ ret = -ENOMEM;
+ goto err_out;
+ }
+ ret = expr__add_id(*combined, dup);
+ if (ret)
+ goto err_out;
+ }
+ }
}
+ return 0;
+err_out:
+ expr__ctx_free(*combined);
+ *combined = NULL;
+ return ret;
+}
+
+/**
+ * parse_ids - Build the event string for the ids and parse them creating an
+ * evlist. The encoded metric_ids are decoded.
+ * @fake_pmu: used when testing metrics not supported by the current CPU.
+ * @ids: the event identifiers parsed from a metric.
+ * @modifier: any modifiers added to the events.
+ * @has_constraint: false if events should be placed in a weak group.
+ * @out_evlist: the created list of events.
+ */
+static int parse_ids(struct perf_pmu *fake_pmu, struct expr_parse_ctx *ids,
+ const char *modifier, bool has_constraint, struct evlist **out_evlist)
+{
+ struct parse_events_error parse_error;
+ struct evlist *parsed_evlist;
+ struct strbuf events = STRBUF_INIT;
+ int ret;
+
+ *out_evlist = NULL;
+ if (hashmap__size(ids->ids) == 0) {
+ char *tmp;
+ /*
+ * No ids/events in the expression parsing context. Events may
+ * have been removed because of constant evaluation, e.g.:
+ * event1 if #smt_on else 0
+ * Add a duration_time event to avoid a parse error on an empty
+ * string.
+ */
+ tmp = strdup("duration_time");
+ if (!tmp)
+ return -ENOMEM;
+
+ ids__insert(ids->ids, tmp);
+ }
+ ret = metricgroup__build_event_string(&events, ids, modifier,
+ has_constraint);
+ if (ret)
+ return ret;
+
+ parsed_evlist = evlist__new();
+ if (!parsed_evlist) {
+ ret = -ENOMEM;
+ goto err_out;
+ }
+ pr_debug("Parsing metric events '%s'\n", events.buf);
+ parse_events_error__init(&parse_error);
+ ret = __parse_events(parsed_evlist, events.buf, &parse_error, fake_pmu);
+ if (ret) {
+ parse_events_error__print(&parse_error, events.buf);
+ goto err_out;
+ }
+ ret = decode_all_metric_ids(parsed_evlist, modifier);
+ if (ret)
+ goto err_out;
+
+ *out_evlist = parsed_evlist;
+ parsed_evlist = NULL;
+err_out:
+ parse_events_error__exit(&parse_error);
+ evlist__delete(parsed_evlist);
+ strbuf_release(&events);
+ return ret;
}
static int parse_groups(struct evlist *perf_evlist, const char *str,
bool metric_no_group,
bool metric_no_merge,
struct perf_pmu *fake_pmu,
- struct rblist *metric_events,
- struct pmu_events_map *map)
+ struct rblist *metric_events_list,
+ const struct pmu_events_map *map)
{
- struct parse_events_error parse_error;
- struct strbuf extra_events;
+ struct evlist *combined_evlist = NULL;
LIST_HEAD(metric_list);
+ struct metric *m;
int ret;
- if (metric_events->nr_entries == 0)
- metricgroup__rblist_init(metric_events);
+ if (metric_events_list->nr_entries == 0)
+ metricgroup__rblist_init(metric_events_list);
ret = metricgroup__add_metric_list(str, metric_no_group,
- &extra_events, &metric_list, map);
+ &metric_list, map);
if (ret)
goto out;
- pr_debug("adding %s\n", extra_events.buf);
- bzero(&parse_error, sizeof(parse_error));
- ret = __parse_events(perf_evlist, extra_events.buf, &parse_error, fake_pmu);
- if (ret) {
- parse_events_print_error(&parse_error, extra_events.buf);
- goto out;
+
+ /* Sort metrics from largest to smallest. */
+ list_sort(NULL, &metric_list, metric_list_cmp);
+
+ if (!metric_no_merge) {
+ struct expr_parse_ctx *combined = NULL;
+
+ ret = build_combined_expr_ctx(&metric_list, &combined);
+
+ if (!ret && combined && hashmap__size(combined->ids)) {
+ ret = parse_ids(fake_pmu, combined, /*modifier=*/NULL,
+ /*has_constraint=*/true,
+ &combined_evlist);
+ }
+ if (combined)
+ expr__ctx_free(combined);
+
+ if (ret)
+ goto out;
}
- ret = metricgroup__setup_events(&metric_list, metric_no_merge,
- perf_evlist, metric_events);
+
+ list_for_each_entry(m, &metric_list, nd) {
+ struct metric_event *me;
+ struct evsel **metric_events;
+ struct evlist *metric_evlist = NULL;
+ struct metric *n;
+ struct metric_expr *expr;
+
+ if (combined_evlist && m->has_constraint) {
+ metric_evlist = combined_evlist;
+ } else if (!metric_no_merge) {
+ /*
+ * See if the IDs for this metric are a subset of an
+ * earlier metric.
+ */
+ list_for_each_entry(n, &metric_list, nd) {
+ if (m == n)
+ break;
+
+ if (n->evlist == NULL)
+ continue;
+
+ if ((!m->modifier && n->modifier) ||
+ (m->modifier && !n->modifier) ||
+ (m->modifier && n->modifier &&
+ strcmp(m->modifier, n->modifier)))
+ continue;
+
+ if (expr__subset_of_ids(n->pctx, m->pctx)) {
+ pr_debug("Events in '%s' fully contained within '%s'\n",
+ m->metric_name, n->metric_name);
+ metric_evlist = n->evlist;
+ break;
+ }
+
+ }
+ }
+ if (!metric_evlist) {
+ ret = parse_ids(fake_pmu, m->pctx, m->modifier,
+ m->has_constraint, &m->evlist);
+ if (ret)
+ goto out;
+
+ metric_evlist = m->evlist;
+ }
+ ret = setup_metric_events(m->pctx->ids, metric_evlist, &metric_events);
+ if (ret) {
+ pr_debug("Cannot resolve IDs for %s: %s\n",
+ m->metric_name, m->metric_expr);
+ goto out;
+ }
+
+ me = metricgroup__lookup(metric_events_list, metric_events[0], true);
+
+ expr = malloc(sizeof(struct metric_expr));
+ if (!expr) {
+ ret = -ENOMEM;
+ free(metric_events);
+ goto out;
+ }
+
+ expr->metric_refs = m->metric_refs;
+ m->metric_refs = NULL;
+ expr->metric_expr = m->metric_expr;
+ if (m->modifier) {
+ char *tmp;
+
+ if (asprintf(&tmp, "%s:%s", m->metric_name, m->modifier) < 0)
+ expr->metric_name = NULL;
+ else
+ expr->metric_name = tmp;
+ } else
+ expr->metric_name = strdup(m->metric_name);
+
+ if (!expr->metric_name) {
+ ret = -ENOMEM;
+ free(metric_events);
+ goto out;
+ }
+ expr->metric_unit = m->metric_unit;
+ expr->metric_events = metric_events;
+ expr->runtime = m->pctx->runtime;
+ list_add(&expr->nd, &me->head);
+ }
+
+
+ if (combined_evlist) {
+ evlist__splice_list_tail(perf_evlist, &combined_evlist->core.entries);
+ evlist__delete(combined_evlist);
+ }
+
+ list_for_each_entry(m, &metric_list, nd) {
+ if (m->evlist)
+ evlist__splice_list_tail(perf_evlist, &m->evlist->core.entries);
+ }
+
out:
metricgroup__free_metrics(&metric_list);
- strbuf_release(&extra_events);
return ret;
}
struct rblist *metric_events)
{
struct evlist *perf_evlist = *(struct evlist **)opt->value;
- struct pmu_events_map *map = pmu_events_map__find();
+ const struct pmu_events_map *map = pmu_events_map__find();
return parse_groups(perf_evlist, str, metric_no_group,
metric_no_merge, NULL, metric_events, map);
}
int metricgroup__parse_groups_test(struct evlist *evlist,
- struct pmu_events_map *map,
+ const struct pmu_events_map *map,
const char *str,
bool metric_no_group,
bool metric_no_merge,
bool metricgroup__has_metric(const char *metric)
{
- struct pmu_events_map *map = pmu_events_map__find();
- struct pmu_event *pe;
+ const struct pmu_events_map *map = pmu_events_map__find();
+ const struct pmu_event *pe;
int i;
if (!map)
return -ENOMEM;
new_expr->metric_expr = old_expr->metric_expr;
- new_expr->metric_name = old_expr->metric_name;
+ new_expr->metric_name = strdup(old_expr->metric_name);
+ if (!new_expr->metric_name)
+ return -ENOMEM;
+
new_expr->metric_unit = old_expr->metric_unit;
new_expr->runtime = old_expr->runtime;
struct pmu_events_map;
struct cgroup;
+/**
+ * A node in a rblist keyed by the evsel. The global rblist of metric events
+ * generally exists in perf_stat_config. The evsel is looked up in the rblist
+ * yielding a list of metric_expr.
+ */
struct metric_event {
struct rb_node nd;
struct evsel *evsel;
struct list_head head; /* list of metric_expr */
};
+/**
+ * A metric referenced by a metric_expr. When parsing a metric expression IDs
+ * will be looked up, matching either a value (from metric_events) or a
+ * metric_ref. A metric_ref will then be parsed recursively. The metric_refs and
+ * metric_events need to be known before parsing so that their values may be
+ * placed in the parse context for lookup.
+ */
struct metric_ref {
const char *metric_name;
const char *metric_expr;
};
+/**
+ * One in a list of metric_expr associated with an evsel. The data is used to
+ * generate a metric value during stat output.
+ */
struct metric_expr {
struct list_head nd;
+ /** The expression to parse, for example, "instructions/cycles". */
const char *metric_expr;
+ /** The name of the meric such as "IPC". */
const char *metric_name;
+ /**
+ * The "ScaleUnit" that scales and adds a unit to the metric during
+ * output. For example, "6.4e-05MiB" means to scale the resulting metric
+ * by 6.4e-05 (typically converting a unit like cache lines to something
+ * more human intelligible) and then add "MiB" afterward when displayed.
+ */
const char *metric_unit;
+ /** Null terminated array of events used by the metric. */
struct evsel **metric_events;
+ /** Null terminated array of referenced metrics. */
struct metric_ref *metric_refs;
+ /** A value substituted for '?' during parsing. */
int runtime;
};
bool metric_no_group,
bool metric_no_merge,
struct rblist *metric_events);
-struct pmu_event *metricgroup__find_metric(const char *metric,
- struct pmu_events_map *map);
+const struct pmu_event *metricgroup__find_metric(const char *metric,
+ const struct pmu_events_map *map);
int metricgroup__parse_groups_test(struct evlist *evlist,
- struct pmu_events_map *map,
+ const struct pmu_events_map *map,
const char *str,
bool metric_no_group,
bool metric_no_merge,
struct rblist *metric_events);
void metricgroup__print(bool metrics, bool groups, char *filter,
- bool raw, bool details);
+ bool raw, bool details, const char *pmu_name);
bool metricgroup__has_metric(const char *metric);
-int arch_get_runtimeparam(struct pmu_event *pe __maybe_unused);
+int arch_get_runtimeparam(const struct pmu_event *pe __maybe_unused);
void metricgroup__rblist_exit(struct rblist *metric_events);
int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp,
out:
return rc;
}
+
+int mmap_cpu_mask__duplicate(struct mmap_cpu_mask *original, struct mmap_cpu_mask *clone)
+{
+ clone->nbits = original->nbits;
+ clone->bits = bitmap_zalloc(original->nbits);
+ if (!clone->bits)
+ return -ENOMEM;
+
+ memcpy(clone->bits, original->bits, MMAP_CPU_MASK_BYTES(original));
+ return 0;
+}
void mmap_cpu_mask__scnprintf(struct mmap_cpu_mask *mask, const char *tag);
+int mmap_cpu_mask__duplicate(struct mmap_cpu_mask *original,
+ struct mmap_cpu_mask *clone);
+
#endif /*__PERF_MMAP_H */
static int create_event_hybrid(__u32 config_type, int *idx,
struct list_head *list,
- struct perf_event_attr *attr, char *name,
+ struct perf_event_attr *attr, const char *name,
+ const char *metric_id,
struct list_head *config_terms,
struct perf_pmu *pmu)
{
__u64 config = attr->config;
config_hybrid_attr(attr, config_type, pmu->type);
- evsel = parse_events__add_event_hybrid(list, idx, attr, name,
+ evsel = parse_events__add_event_hybrid(list, idx, attr, name, metric_id,
pmu, config_terms);
if (evsel)
evsel->pmu_name = strdup(pmu->name);
static int add_hw_hybrid(struct parse_events_state *parse_state,
struct list_head *list, struct perf_event_attr *attr,
- char *name, struct list_head *config_terms)
+ const char *name, const char *metric_id,
+ struct list_head *config_terms)
{
struct perf_pmu *pmu;
int ret;
copy_config_terms(&terms, config_terms);
ret = create_event_hybrid(PERF_TYPE_HARDWARE,
&parse_state->idx, list, attr, name,
- &terms, pmu);
+ metric_id, &terms, pmu);
free_config_terms(&terms);
if (ret)
return ret;
}
static int create_raw_event_hybrid(int *idx, struct list_head *list,
- struct perf_event_attr *attr, char *name,
+ struct perf_event_attr *attr,
+ const char *name,
+ const char *metric_id,
struct list_head *config_terms,
struct perf_pmu *pmu)
{
struct evsel *evsel;
attr->type = pmu->type;
- evsel = parse_events__add_event_hybrid(list, idx, attr, name,
+ evsel = parse_events__add_event_hybrid(list, idx, attr, name, metric_id,
pmu, config_terms);
if (evsel)
evsel->pmu_name = strdup(pmu->name);
static int add_raw_hybrid(struct parse_events_state *parse_state,
struct list_head *list, struct perf_event_attr *attr,
- char *name, struct list_head *config_terms)
+ const char *name, const char *metric_id,
+ struct list_head *config_terms)
{
struct perf_pmu *pmu;
int ret;
copy_config_terms(&terms, config_terms);
ret = create_raw_event_hybrid(&parse_state->idx, list, attr,
- name, &terms, pmu);
+ name, metric_id, &terms, pmu);
free_config_terms(&terms);
if (ret)
return ret;
int parse_events__add_numeric_hybrid(struct parse_events_state *parse_state,
struct list_head *list,
struct perf_event_attr *attr,
- char *name, struct list_head *config_terms,
+ const char *name, const char *metric_id,
+ struct list_head *config_terms,
bool *hybrid)
{
*hybrid = false;
*hybrid = true;
if (attr->type != PERF_TYPE_RAW) {
- return add_hw_hybrid(parse_state, list, attr, name,
+ return add_hw_hybrid(parse_state, list, attr, name, metric_id,
config_terms);
}
- return add_raw_hybrid(parse_state, list, attr, name,
+ return add_raw_hybrid(parse_state, list, attr, name, metric_id,
config_terms);
}
int parse_events__add_cache_hybrid(struct list_head *list, int *idx,
- struct perf_event_attr *attr, char *name,
+ struct perf_event_attr *attr,
+ const char *name,
+ const char *metric_id,
struct list_head *config_terms,
bool *hybrid,
struct parse_events_state *parse_state)
copy_config_terms(&terms, config_terms);
ret = create_event_hybrid(PERF_TYPE_HW_CACHE, idx, list,
- attr, name, &terms, pmu);
+ attr, name, metric_id, &terms, pmu);
free_config_terms(&terms);
if (ret)
return ret;
int parse_events__add_numeric_hybrid(struct parse_events_state *parse_state,
struct list_head *list,
struct perf_event_attr *attr,
- char *name, struct list_head *config_terms,
+ const char *name, const char *metric_id,
+ struct list_head *config_terms,
bool *hybrid);
int parse_events__add_cache_hybrid(struct list_head *list, int *idx,
- struct perf_event_attr *attr, char *name,
+ struct perf_event_attr *attr,
+ const char *name, const char *metric_id,
struct list_head *config_terms,
bool *hybrid,
struct parse_events_state *parse_state);
#include <subcmd/exec-cmd.h>
#include "string2.h"
#include "strlist.h"
-#include "symbol.h"
-#include "header.h"
#include "bpf-loader.h"
#include "debug.h"
#include <api/fs/tracing_path.h>
#define MAX_EVENT_LENGTH 512
-void parse_events__handle_error(struct parse_events_error *err, int idx,
- char *str, char *help)
-{
- if (WARN(!str, "WARNING: failed to provide error string\n")) {
- free(help);
- return;
- }
- switch (err->num_errors) {
- case 0:
- err->idx = idx;
- err->str = str;
- err->help = help;
- break;
- case 1:
- err->first_idx = err->idx;
- err->idx = idx;
- err->first_str = err->str;
- err->str = str;
- err->first_help = err->help;
- err->help = help;
- break;
- default:
- pr_debug("Multiple errors dropping message: %s (%s)\n",
- err->str, err->help);
- free(err->str);
- err->str = str;
- free(err->help);
- err->help = help;
- break;
- }
- err->num_errors++;
-}
-
struct tracepoint_path *tracepoint_id_to_path(u64 config)
{
struct tracepoint_path *path = NULL;
return "unknown";
}
-static int parse_events__is_name_term(struct parse_events_term *term)
-{
- return term->type_term == PARSE_EVENTS__TERM_TYPE_NAME;
-}
-
-static char *get_config_name(struct list_head *head_terms)
+static char *get_config_str(struct list_head *head_terms, int type_term)
{
struct parse_events_term *term;
return NULL;
list_for_each_entry(term, head_terms, list)
- if (parse_events__is_name_term(term))
+ if (term->type_term == type_term)
return term->val.str;
return NULL;
}
+static char *get_config_metric_id(struct list_head *head_terms)
+{
+ return get_config_str(head_terms, PARSE_EVENTS__TERM_TYPE_METRIC_ID);
+}
+
+static char *get_config_name(struct list_head *head_terms)
+{
+ return get_config_str(head_terms, PARSE_EVENTS__TERM_TYPE_NAME);
+}
+
static struct evsel *
__add_event(struct list_head *list, int *idx,
struct perf_event_attr *attr,
bool init_attr,
- char *name, struct perf_pmu *pmu,
+ const char *name, const char *metric_id, struct perf_pmu *pmu,
struct list_head *config_terms, bool auto_merge_stats,
const char *cpu_list)
{
if (name)
evsel->name = strdup(name);
+ if (metric_id)
+ evsel->metric_id = strdup(metric_id);
+
if (config_terms)
list_splice_init(config_terms, &evsel->config_terms);
}
struct evsel *parse_events__add_event(int idx, struct perf_event_attr *attr,
- char *name, struct perf_pmu *pmu)
+ const char *name, const char *metric_id,
+ struct perf_pmu *pmu)
{
- return __add_event(NULL, &idx, attr, false, name, pmu, NULL, false,
- NULL);
+ return __add_event(/*list=*/NULL, &idx, attr, /*init_attr=*/false, name,
+ metric_id, pmu, /*config_terms=*/NULL,
+ /*auto_merge_stats=*/false, /*cpu_list=*/NULL);
}
static int add_event(struct list_head *list, int *idx,
- struct perf_event_attr *attr, char *name,
- struct list_head *config_terms)
+ struct perf_event_attr *attr, const char *name,
+ const char *metric_id, struct list_head *config_terms)
{
- return __add_event(list, idx, attr, true, name, NULL, config_terms,
- false, NULL) ? 0 : -ENOMEM;
+ return __add_event(list, idx, attr, /*init_attr*/true, name, metric_id,
+ /*pmu=*/NULL, config_terms,
+ /*auto_merge_stats=*/false, /*cpu_list=*/NULL) ? 0 : -ENOMEM;
}
static int add_event_tool(struct list_head *list, int *idx,
.config = PERF_COUNT_SW_DUMMY,
};
- evsel = __add_event(list, idx, &attr, true, NULL, NULL, NULL, false,
- "0");
+ evsel = __add_event(list, idx, &attr, /*init_attr=*/true, /*name=*/NULL,
+ /*metric_id=*/NULL, /*pmu=*/NULL,
+ /*config_terms=*/NULL, /*auto_merge_stats=*/false,
+ /*cpu_list=*/"0");
if (!evsel)
return -ENOMEM;
evsel->tool_event = tool_event;
- if (tool_event == PERF_TOOL_DURATION_TIME)
- evsel->unit = "ns";
+ if (tool_event == PERF_TOOL_DURATION_TIME) {
+ free((char *)evsel->unit);
+ evsel->unit = strdup("ns");
+ }
return 0;
}
{
struct perf_event_attr attr;
LIST_HEAD(config_terms);
- char name[MAX_NAME_LEN], *config_name;
+ char name[MAX_NAME_LEN];
+ const char *config_name, *metric_id;
int cache_type = -1, cache_op = -1, cache_result = -1;
char *op_result[2] = { op_result1, op_result2 };
int i, n, ret;
return -ENOMEM;
}
+ metric_id = get_config_metric_id(head_config);
ret = parse_events__add_cache_hybrid(list, idx, &attr,
- config_name ? : name, &config_terms,
+ config_name ? : name,
+ metric_id,
+ &config_terms,
&hybrid, parse_state);
if (hybrid)
goto out_free_terms;
- ret = add_event(list, idx, &attr, config_name ? : name, &config_terms);
+ ret = add_event(list, idx, &attr, config_name ? : name, metric_id,
+ &config_terms);
out_free_terms:
free_config_terms(&config_terms);
return ret;
}
tracing_path__strerror_open_tp(err, help, sizeof(help), sys, name);
- parse_events__handle_error(e, 0, strdup(str), strdup(help));
+ parse_events_error__handle(e, 0, strdup(str), strdup(help));
}
static int add_tracepoint(struct list_head *list, int *idx,
return 0;
errout:
- parse_events__handle_error(parse_state->error, 0,
+ parse_events_error__handle(parse_state->error, 0,
strdup(errbuf), strdup("(add -v to see detail)"));
return err;
}
int err;
if (term->type_term != PARSE_EVENTS__TERM_TYPE_USER) {
- parse_events__handle_error(parse_state->error, term->err_term,
+ parse_events_error__handle(parse_state->error, term->err_term,
strdup("Invalid config term for BPF object"),
NULL);
return -EINVAL;
else
idx = term->err_term + error_pos;
- parse_events__handle_error(parse_state->error, idx,
+ parse_events_error__handle(parse_state->error, idx,
strdup(errbuf),
strdup(
"Hint:\tValid config terms:\n"
-err, errbuf,
sizeof(errbuf));
- parse_events__handle_error(parse_state->error, 0,
+ parse_events_error__handle(parse_state->error, 0,
strdup(errbuf), strdup("(add -v to see detail)"));
return err;
}
struct bpf_object *obj __maybe_unused,
struct list_head *head_config __maybe_unused)
{
- parse_events__handle_error(parse_state->error, 0,
+ parse_events_error__handle(parse_state->error, 0,
strdup("BPF support is not compiled"),
strdup("Make sure libbpf-devel is available at build time."));
return -ENOTSUP;
bool source __maybe_unused,
struct list_head *head_config __maybe_unused)
{
- parse_events__handle_error(parse_state->error, 0,
+ parse_events_error__handle(parse_state->error, 0,
strdup("BPF support is not compiled"),
strdup("Make sure libbpf-devel is available at build time."));
return -ENOTSUP;
attr.type = PERF_TYPE_BREAKPOINT;
attr.sample_period = 1;
- return add_event(list, idx, &attr, NULL, NULL);
+ return add_event(list, idx, &attr, /*name=*/NULL, /*mertic_id=*/NULL,
+ /*config_terms=*/NULL);
}
static int check_type_val(struct parse_events_term *term,
return 0;
if (err) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
type == PARSE_EVENTS__TERM_TYPE_NUM
? strdup("expected numeric value")
: strdup("expected string value"),
[PARSE_EVENTS__TERM_TYPE_PERCORE] = "percore",
[PARSE_EVENTS__TERM_TYPE_AUX_OUTPUT] = "aux-output",
[PARSE_EVENTS__TERM_TYPE_AUX_SAMPLE_SIZE] = "aux-sample-size",
+ [PARSE_EVENTS__TERM_TYPE_METRIC_ID] = "metric-id",
};
static bool config_term_shrinked;
char *err_str;
if (term_type < 0 || term_type >= __PARSE_EVENTS__TERM_TYPE_NR) {
- parse_events__handle_error(err, -1,
+ parse_events_error__handle(err, -1,
strdup("Invalid term_type"), NULL);
return false;
}
case PARSE_EVENTS__TERM_TYPE_CONFIG1:
case PARSE_EVENTS__TERM_TYPE_CONFIG2:
case PARSE_EVENTS__TERM_TYPE_NAME:
+ case PARSE_EVENTS__TERM_TYPE_METRIC_ID:
case PARSE_EVENTS__TERM_TYPE_SAMPLE_PERIOD:
case PARSE_EVENTS__TERM_TYPE_PERCORE:
return true;
/* term_type is validated so indexing is safe */
if (asprintf(&err_str, "'%s' is not usable in 'perf stat'",
config_term_names[term_type]) >= 0)
- parse_events__handle_error(err, -1, err_str, NULL);
+ parse_events_error__handle(err, -1, err_str, NULL);
return false;
}
}
if (strcmp(term->val.str, "no") &&
parse_branch_str(term->val.str,
&attr->branch_sample_type)) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("invalid branch sample type"),
NULL);
return -EINVAL;
case PARSE_EVENTS__TERM_TYPE_TIME:
CHECK_TYPE_VAL(NUM);
if (term->val.num > 1) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("expected 0 or 1"),
NULL);
return -EINVAL;
case PARSE_EVENTS__TERM_TYPE_NAME:
CHECK_TYPE_VAL(STR);
break;
+ case PARSE_EVENTS__TERM_TYPE_METRIC_ID:
+ CHECK_TYPE_VAL(STR);
+ break;
case PARSE_EVENTS__TERM_TYPE_MAX_STACK:
CHECK_TYPE_VAL(NUM);
break;
case PARSE_EVENTS__TERM_TYPE_PERCORE:
CHECK_TYPE_VAL(NUM);
if ((unsigned int)term->val.num > 1) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("expected 0 or 1"),
NULL);
return -EINVAL;
case PARSE_EVENTS__TERM_TYPE_AUX_SAMPLE_SIZE:
CHECK_TYPE_VAL(NUM);
if (term->val.num > UINT_MAX) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("too big"),
NULL);
return -EINVAL;
}
break;
default:
- parse_events__handle_error(err, term->err_term,
+ parse_events_error__handle(err, term->err_term,
strdup("unknown term"),
parse_events_formats_error_string(NULL));
return -EINVAL;
return config_term_common(attr, term, err);
default:
if (err) {
- parse_events__handle_error(err, term->err_term,
+ parse_events_error__handle(err, term->err_term,
strdup("unknown term"),
strdup("valid terms: call-graph,stack-size\n"));
}
{
struct perf_event_attr attr;
LIST_HEAD(config_terms);
+ const char *name, *metric_id;
bool hybrid;
int ret;
return -ENOMEM;
}
+ name = get_config_name(head_config);
+ metric_id = get_config_metric_id(head_config);
ret = parse_events__add_numeric_hybrid(parse_state, list, &attr,
- get_config_name(head_config),
+ name, metric_id,
&config_terms, &hybrid);
if (hybrid)
goto out_free_terms;
- ret = add_event(list, &parse_state->idx, &attr,
- get_config_name(head_config), &config_terms);
+ ret = add_event(list, &parse_state->idx, &attr, name, metric_id,
+ &config_terms);
out_free_terms:
free_config_terms(&config_terms);
return ret;
int parse_events_add_tool(struct parse_events_state *parse_state,
struct list_head *list,
- enum perf_tool_event tool_event)
+ int tool_event)
{
return add_event_tool(list, &parse_state->idx, tool_event);
}
if (asprintf(&err_str,
"Cannot find PMU `%s'. Missing kernel support?",
name) >= 0)
- parse_events__handle_error(err, 0, err_str, NULL);
+ parse_events_error__handle(err, 0, err_str, NULL);
return -EINVAL;
}
if (!head_config) {
attr.type = pmu->type;
- evsel = __add_event(list, &parse_state->idx, &attr, true, NULL,
- pmu, NULL, auto_merge_stats, NULL);
+ evsel = __add_event(list, &parse_state->idx, &attr,
+ /*init_attr=*/true, /*name=*/NULL,
+ /*metric_id=*/NULL, pmu,
+ /*config_terms=*/NULL, auto_merge_stats,
+ /*cpu_list=*/NULL);
if (evsel) {
evsel->pmu_name = name ? strdup(name) : NULL;
evsel->use_uncore_alias = use_uncore_alias;
return -EINVAL;
}
- evsel = __add_event(list, &parse_state->idx, &attr, true,
- get_config_name(head_config), pmu,
- &config_terms, auto_merge_stats, NULL);
+ evsel = __add_event(list, &parse_state->idx, &attr, /*init_attr=*/true,
+ get_config_name(head_config),
+ get_config_metric_id(head_config), pmu,
+ &config_terms, auto_merge_stats, /*cpu_list=*/NULL);
if (!evsel)
return -ENOMEM;
if (parse_state->fake_pmu)
return 0;
- evsel->unit = info.unit;
+ free((char *)evsel->unit);
+ evsel->unit = strdup(info.unit);
evsel->scale = info.scale;
evsel->per_pkg = info.per_pkg;
evsel->snapshot = info.snapshot;
}
int parse_events_multi_pmu_add(struct parse_events_state *parse_state,
- char *str, struct list_head **listp)
+ char *str, struct list_head *head,
+ struct list_head **listp)
{
struct parse_events_term *term;
- struct list_head *list;
+ struct list_head *list = NULL;
struct perf_pmu *pmu = NULL;
int ok = 0;
+ char *config;
*listp = NULL;
+
+ if (!head) {
+ head = malloc(sizeof(struct list_head));
+ if (!head)
+ goto out_err;
+
+ INIT_LIST_HEAD(head);
+ }
+ config = strdup(str);
+ if (!config)
+ goto out_err;
+
+ if (parse_events_term__num(&term,
+ PARSE_EVENTS__TERM_TYPE_USER,
+ config, 1, false, &config,
+ NULL) < 0) {
+ free(config);
+ goto out_err;
+ }
+ list_add_tail(&term->list, head);
+
+
/* Add it for all PMUs that support the alias */
list = malloc(sizeof(struct list_head));
if (!list)
- return -1;
+ goto out_err;
+
INIT_LIST_HEAD(list);
+
while ((pmu = perf_pmu__scan(pmu)) != NULL) {
struct perf_pmu_alias *alias;
list_for_each_entry(alias, &pmu->aliases, list) {
if (!strcasecmp(alias->name, str)) {
- struct list_head *head;
- char *config;
-
- head = malloc(sizeof(struct list_head));
- if (!head)
- return -1;
- INIT_LIST_HEAD(head);
- config = strdup(str);
- if (!config)
- return -1;
- if (parse_events_term__num(&term,
- PARSE_EVENTS__TERM_TYPE_USER,
- config, 1, false, &config,
- NULL) < 0) {
- free(list);
- free(config);
- return -1;
- }
- list_add_tail(&term->list, head);
-
if (!parse_events_add_pmu(parse_state, list,
pmu->name, head,
true, true)) {
pmu->name, alias->str);
ok++;
}
-
- parse_events_terms__delete(head);
}
}
}
- if (!ok) {
+out_err:
+ if (ok)
+ *listp = list;
+ else
free(list);
- return -1;
- }
- *listp = list;
- return 0;
+
+ parse_events_terms__delete(head);
+ return ok ? 0 : -1;
}
int parse_events__modifier_group(struct list_head *list,
return 0;
}
-int parse_events_name(struct list_head *list, char *name)
+int parse_events_name(struct list_head *list, const char *name)
{
struct evsel *evsel;
return ret;
}
+void parse_events_error__init(struct parse_events_error *err)
+{
+ bzero(err, sizeof(*err));
+}
+
+void parse_events_error__exit(struct parse_events_error *err)
+{
+ zfree(&err->str);
+ zfree(&err->help);
+ zfree(&err->first_str);
+ zfree(&err->first_help);
+}
+
+void parse_events_error__handle(struct parse_events_error *err, int idx,
+ char *str, char *help)
+{
+ if (WARN(!str, "WARNING: failed to provide error string\n")) {
+ free(help);
+ return;
+ }
+ switch (err->num_errors) {
+ case 0:
+ err->idx = idx;
+ err->str = str;
+ err->help = help;
+ break;
+ case 1:
+ err->first_idx = err->idx;
+ err->idx = idx;
+ err->first_str = err->str;
+ err->str = str;
+ err->first_help = err->help;
+ err->help = help;
+ break;
+ default:
+ pr_debug("Multiple errors dropping message: %s (%s)\n",
+ err->str, err->help);
+ free(err->str);
+ err->str = str;
+ free(err->help);
+ err->help = help;
+ break;
+ }
+ err->num_errors++;
+}
+
#define MAX_WIDTH 1000
static int get_term_width(void)
{
return ws.ws_col > MAX_WIDTH ? MAX_WIDTH : ws.ws_col;
}
-static void __parse_events_print_error(int err_idx, const char *err_str,
- const char *err_help, const char *event)
+static void __parse_events_error__print(int err_idx, const char *err_str,
+ const char *err_help, const char *event)
{
const char *str = "invalid or unsupported event: ";
char _buf[MAX_WIDTH];
}
}
-void parse_events_print_error(struct parse_events_error *err,
- const char *event)
+void parse_events_error__print(struct parse_events_error *err,
+ const char *event)
{
if (!err->num_errors)
return;
- __parse_events_print_error(err->idx, err->str, err->help, event);
- zfree(&err->str);
- zfree(&err->help);
+ __parse_events_error__print(err->idx, err->str, err->help, event);
if (err->num_errors > 1) {
fputs("\nInitial error:\n", stderr);
- __parse_events_print_error(err->first_idx, err->first_str,
+ __parse_events_error__print(err->first_idx, err->first_str,
err->first_help, event);
- zfree(&err->first_str);
- zfree(&err->first_help);
}
}
struct parse_events_error err;
int ret;
- bzero(&err, sizeof(err));
+ parse_events_error__init(&err);
ret = parse_events(evlist, str, &err);
if (ret) {
- parse_events_print_error(&err, str);
+ parse_events_error__print(&err, str);
fprintf(stderr, "Run 'perf list' for a list of valid events\n");
}
+ parse_events_error__exit(&err);
return ret;
}
return 0;
}
-static bool is_event_supported(u8 type, unsigned config)
+static bool is_event_supported(u8 type, u64 config)
{
bool ret = true;
int open_return;
int print_hwcache_events(const char *event_glob, bool name_only)
{
- unsigned int type, op, i, evt_i = 0, evt_num = 0;
- char name[64];
- char **evt_list = NULL;
+ unsigned int type, op, i, evt_i = 0, evt_num = 0, npmus = 0;
+ char name[64], new_name[128];
+ char **evt_list = NULL, **evt_pmus = NULL;
bool evt_num_known = false;
+ struct perf_pmu *pmu = NULL;
+
+ if (perf_pmu__has_hybrid()) {
+ npmus = perf_pmu__hybrid_pmu_num();
+ evt_pmus = zalloc(sizeof(char *) * npmus);
+ if (!evt_pmus)
+ goto out_enomem;
+ }
restart:
if (evt_num_known) {
continue;
for (i = 0; i < PERF_COUNT_HW_CACHE_RESULT_MAX; i++) {
+ unsigned int hybrid_supported = 0, j;
+ bool supported;
+
__evsel__hw_cache_type_op_res_name(type, op, i, name, sizeof(name));
if (event_glob != NULL && !strglobmatch(name, event_glob))
continue;
- if (!is_event_supported(PERF_TYPE_HW_CACHE,
- type | (op << 8) | (i << 16)))
- continue;
+ if (!perf_pmu__has_hybrid()) {
+ if (!is_event_supported(PERF_TYPE_HW_CACHE,
+ type | (op << 8) | (i << 16))) {
+ continue;
+ }
+ } else {
+ perf_pmu__for_each_hybrid_pmu(pmu) {
+ if (!evt_num_known) {
+ evt_num++;
+ continue;
+ }
+
+ supported = is_event_supported(
+ PERF_TYPE_HW_CACHE,
+ type | (op << 8) | (i << 16) |
+ ((__u64)pmu->type << PERF_PMU_TYPE_SHIFT));
+ if (supported) {
+ snprintf(new_name, sizeof(new_name), "%s/%s/",
+ pmu->name, name);
+ evt_pmus[hybrid_supported] = strdup(new_name);
+ hybrid_supported++;
+ }
+ }
+
+ if (hybrid_supported == 0)
+ continue;
+ }
if (!evt_num_known) {
evt_num++;
continue;
}
- evt_list[evt_i] = strdup(name);
+ if ((hybrid_supported == 0) ||
+ (hybrid_supported == npmus)) {
+ evt_list[evt_i] = strdup(name);
+ if (npmus > 0) {
+ for (j = 0; j < npmus; j++)
+ zfree(&evt_pmus[j]);
+ }
+ } else {
+ for (j = 0; j < hybrid_supported; j++) {
+ evt_list[evt_i++] = evt_pmus[j];
+ evt_pmus[j] = NULL;
+ }
+ continue;
+ }
+
if (evt_list[evt_i] == NULL)
goto out_enomem;
evt_i++;
evt_num_known = true;
goto restart;
}
+
+ for (evt_i = 0; evt_i < evt_num; evt_i++) {
+ if (!evt_list[evt_i])
+ break;
+ }
+
+ evt_num = evt_i;
qsort(evt_list, evt_num, sizeof(char *), cmp_string);
evt_i = 0;
while (evt_i < evt_num) {
for (evt_i = 0; evt_i < evt_num; evt_i++)
zfree(&evt_list[evt_i]);
zfree(&evt_list);
+
+ for (evt_i = 0; evt_i < npmus; evt_i++)
+ zfree(&evt_pmus[evt_i]);
+ zfree(&evt_pmus);
return evt_num;
out_enomem:
* Print the help text for the event symbols:
*/
void print_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc, bool details_flag, bool deprecated)
+ bool long_desc, bool details_flag, bool deprecated,
+ const char *pmu_name)
{
print_symbol_events(event_glob, PERF_TYPE_HARDWARE,
event_symbols_hw, PERF_COUNT_HW_MAX, name_only);
print_hwcache_events(event_glob, name_only);
print_pmu_events(event_glob, name_only, quiet_flag, long_desc,
- details_flag, deprecated);
+ details_flag, deprecated, pmu_name);
if (event_glob != NULL)
return;
print_sdt_events(NULL, NULL, name_only);
- metricgroup__print(true, true, NULL, name_only, details_flag);
+ metricgroup__print(true, true, NULL, name_only, details_flag,
+ pmu_name);
print_libpfm_events(name_only, long_desc);
}
struct parse_events_term temp = {
.type_val = PARSE_EVENTS__TERM_TYPE_NUM,
.type_term = type_term,
- .config = config,
+ .config = config ? : strdup(config_term_names[type_term]),
.no_value = no_value,
.err_term = loc_term ? loc_term->first_column : 0,
.err_val = loc_val ? loc_val->first_column : 0,
if (!parse_state->error)
return;
- parse_events__handle_error(parse_state->error, idx, strdup(str), NULL);
+ parse_events_error__handle(parse_state->error, idx, strdup(str), NULL);
}
static void config_terms_list(char *buf, size_t buf_sz)
struct evsel *parse_events__add_event_hybrid(struct list_head *list, int *idx,
struct perf_event_attr *attr,
- char *name, struct perf_pmu *pmu,
+ const char *name,
+ const char *metric_id,
+ struct perf_pmu *pmu,
struct list_head *config_terms)
{
- return __add_event(list, idx, attr, true, name, pmu,
- config_terms, false, NULL);
+ return __add_event(list, idx, attr, /*init_attr=*/true, name, metric_id,
+ pmu, config_terms, /*auto_merge_stats=*/false,
+ /*cpu_list=*/NULL);
}
PARSE_EVENTS__TERM_TYPE_PERCORE,
PARSE_EVENTS__TERM_TYPE_AUX_OUTPUT,
PARSE_EVENTS__TERM_TYPE_AUX_SAMPLE_SIZE,
+ PARSE_EVENTS__TERM_TYPE_METRIC_ID,
__PARSE_EVENTS__TERM_TYPE_NR,
};
char *hybrid_pmu_name;
};
-void parse_events__handle_error(struct parse_events_error *err, int idx,
- char *str, char *help);
void parse_events__shrink_config_terms(void);
int parse_events__is_hardcoded_term(struct parse_events_term *term);
int parse_events_term__num(struct parse_events_term **term,
void parse_events__clear_array(struct parse_events_array *a);
int parse_events__modifier_event(struct list_head *list, char *str, bool add);
int parse_events__modifier_group(struct list_head *list, char *event_mod);
-int parse_events_name(struct list_head *list, char *name);
+int parse_events_name(struct list_head *list, const char *name);
int parse_events_add_tracepoint(struct list_head *list, int *idx,
const char *sys, const char *event,
struct parse_events_error *error,
struct list_head *list,
u32 type, u64 config,
struct list_head *head_config);
-enum perf_tool_event;
int parse_events_add_tool(struct parse_events_state *parse_state,
struct list_head *list,
- enum perf_tool_event tool_event);
+ int tool_event);
int parse_events_add_cache(struct list_head *list, int *idx,
char *type, char *op_result1, char *op_result2,
struct parse_events_error *error,
bool use_alias);
struct evsel *parse_events__add_event(int idx, struct perf_event_attr *attr,
- char *name, struct perf_pmu *pmu);
+ const char *name, const char *metric_id,
+ struct perf_pmu *pmu);
int parse_events_multi_pmu_add(struct parse_events_state *parse_state,
char *str,
+ struct list_head *head_config,
struct list_head **listp);
int parse_events_copy_term_list(struct list_head *old,
int idx, const char *str);
void print_events(const char *event_glob, bool name_only, bool quiet,
- bool long_desc, bool details_flag, bool deprecated);
+ bool long_desc, bool details_flag, bool deprecated,
+ const char *pmu_name);
struct event_symbol {
const char *symbol;
int valid_event_mount(const char *eventfs);
char *parse_events_formats_error_string(char *additional_terms);
-void parse_events_print_error(struct parse_events_error *err,
- const char *event);
+void parse_events_error__init(struct parse_events_error *err);
+void parse_events_error__exit(struct parse_events_error *err);
+void parse_events_error__handle(struct parse_events_error *err, int idx,
+ char *str, char *help);
+void parse_events_error__print(struct parse_events_error *err,
+ const char *event);
#ifdef HAVE_LIBELF_SUPPORT
/*
struct evsel *parse_events__add_event_hybrid(struct list_head *list, int *idx,
struct perf_event_attr *attr,
- char *name, struct perf_pmu *pmu,
+ const char *name,
+ const char *metric_id,
+ struct perf_pmu *pmu,
struct list_head *config_terms);
#endif /* __PERF_PARSE_EVENTS_H */
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
-#include "../perf.h"
#include "parse-events.h"
#include "parse-events-bison.h"
#include "evsel.h"
yylval->str = strdup(text);
- if (parse_state->fake_pmu)
- return PE_PMU_EVENT_FAKE;
-
+ /*
+ * If we're not testing then parse check determines the PMU event type
+ * which if it isn't a PMU returns PE_NAME. When testing the result of
+ * parse check can't be trusted so we return PE_PMU_EVENT_FAKE unless
+ * an '!' is present in which case the text can't be a PMU name.
+ */
switch (perf_pmu__parse_check(text)) {
case PMU_EVENT_SYMBOL_PREFIX:
return PE_PMU_EVENT_PRE;
case PMU_EVENT_SYMBOL_SUFFIX:
return PE_PMU_EVENT_SUF;
case PMU_EVENT_SYMBOL:
- return PE_KERNEL_PMU_EVENT;
+ return parse_state->fake_pmu
+ ? PE_PMU_EVENT_FAKE : PE_KERNEL_PMU_EVENT;
default:
- return PE_NAME;
+ return parse_state->fake_pmu && !strchr(text,'!')
+ ? PE_PMU_EVENT_FAKE : PE_NAME;
}
}
num_dec [0-9]+
num_hex 0x[a-fA-F0-9]+
num_raw_hex [a-fA-F0-9]+
-name [a-zA-Z_*?\[\]][a-zA-Z0-9_*?.\[\]]*
+name [a-zA-Z_*?\[\]][a-zA-Z0-9_*?.\[\]!]*
name_tag [\'][a-zA-Z_*?\[\]][a-zA-Z0-9_*?\-,\.\[\]:=]*[\']
name_minus [a-zA-Z_*?][a-zA-Z0-9\-_*?.:]*
drv_cfg_term [a-zA-Z0-9_\.]+(=[a-zA-Z0-9_*?\.:]+)?
percore { return term(yyscanner, PARSE_EVENTS__TERM_TYPE_PERCORE); }
aux-output { return term(yyscanner, PARSE_EVENTS__TERM_TYPE_AUX_OUTPUT); }
aux-sample-size { return term(yyscanner, PARSE_EVENTS__TERM_TYPE_AUX_SAMPLE_SIZE); }
+metric-id { return term(yyscanner, PARSE_EVENTS__TERM_TYPE_METRIC_ID); }
r{num_raw_hex} { return raw(yyscanner); }
r0x{num_raw_hex} { return raw(yyscanner); }
, { return ','; }
err = parse_events__modifier_group(list, $3);
free($3);
if (err) {
+ struct parse_events_state *parse_state = _parse_state;
+ struct parse_events_error *error = parse_state->error;
+
+ parse_events_error__handle(error, @3.first_column,
+ strdup("Bad modifier"), NULL);
free_list_evsel(list);
YYABORT;
}
err = parse_events__modifier_event(list, $2, false);
free($2);
if (err) {
+ struct parse_events_state *parse_state = _parse_state;
+ struct parse_events_error *error = parse_state->error;
+
+ parse_events_error__handle(error, @2.first_column,
+ strdup("Bad modifier"), NULL);
free_list_evsel(list);
YYABORT;
}
struct list_head *list;
int err;
- err = parse_events_multi_pmu_add(_parse_state, $1, &list);
+ err = parse_events_multi_pmu_add(_parse_state, $1, NULL, &list);
+ free($1);
+ if (err < 0)
+ YYABORT;
+ $$ = list;
+}
+|
+PE_KERNEL_PMU_EVENT opt_pmu_config
+{
+ struct list_head *list;
+ int err;
+
+ /* frees $2 */
+ err = parse_events_multi_pmu_add(_parse_state, $1, $2, &list);
free($1);
if (err < 0)
YYABORT;
snprintf(pmu_name, sizeof(pmu_name), "%s-%s", $1, $3);
free($1);
free($3);
- if (parse_events_multi_pmu_add(_parse_state, pmu_name, &list) < 0)
+ if (parse_events_multi_pmu_add(_parse_state, pmu_name, NULL, &list) < 0)
YYABORT;
$$ = list;
}
pmu = perf_pmu__find_by_type((unsigned int)attr.type);
evsel = parse_events__add_event(evlist->core.nr_entries,
- &attr, q, pmu);
+ &attr, q, /*metric_id=*/NULL,
+ pmu);
if (evsel == NULL)
goto error;
}
static int __perf_pmu__new_alias(struct list_head *list, char *dir, char *name,
- char *desc, char *val, struct pmu_event *pe)
+ char *desc, char *val, const struct pmu_event *pe)
{
struct parse_events_term *term;
struct perf_pmu_alias *alias;
return cpuid;
}
-struct pmu_events_map *perf_pmu__find_map(struct perf_pmu *pmu)
+const struct pmu_events_map *perf_pmu__find_map(struct perf_pmu *pmu)
{
- struct pmu_events_map *map;
+ const struct pmu_events_map *map;
char *cpuid = perf_pmu__getcpuid(pmu);
int i;
return map;
}
-struct pmu_events_map *__weak pmu_events_map__find(void)
+const struct pmu_events_map *__weak pmu_events_map__find(void)
{
return perf_pmu__find_map(NULL);
}
* as aliases.
*/
void pmu_add_cpu_aliases_map(struct list_head *head, struct perf_pmu *pmu,
- struct pmu_events_map *map)
+ const struct pmu_events_map *map)
{
int i;
const char *name = pmu->name;
i = 0;
while (1) {
const char *cpu_name = is_arm_pmu_core(name) ? name : "cpu";
- struct pmu_event *pe = &map->table[i++];
+ const struct pmu_event *pe = &map->table[i++];
const char *pname = pe->pmu ? pe->pmu : cpu_name;
if (!pe->name) {
static void pmu_add_cpu_aliases(struct list_head *head, struct perf_pmu *pmu)
{
- struct pmu_events_map *map;
+ const struct pmu_events_map *map;
map = perf_pmu__find_map(pmu);
if (!map)
int i = 0;
while (1) {
- struct pmu_sys_events *event_table;
+ const struct pmu_sys_events *event_table;
int j = 0;
event_table = &pmu_sys_event_tables[i++];
break;
while (1) {
- struct pmu_event *pe = &event_table->table[j++];
+ const struct pmu_event *pe = &event_table->table[j++];
int ret;
if (!pe->name && !pe->metric_group && !pe->metric_name)
struct perf_pmu *pmu;
};
-static int pmu_add_sys_aliases_iter_fn(struct pmu_event *pe, void *data)
+static int pmu_add_sys_aliases_iter_fn(const struct pmu_event *pe, void *data)
{
struct pmu_sys_event_iter_data *idata = data;
struct perf_pmu *pmu = idata->pmu;
unknown_term = NULL;
help_msg = parse_events_formats_error_string(pmu_term);
if (err) {
- parse_events__handle_error(err, term->err_term,
+ parse_events_error__handle(err, term->err_term,
unknown_term,
help_msg);
} else {
if (term->no_value &&
bitmap_weight(format->bits, PERF_PMU_FORMAT_BITS) > 1) {
if (err) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("no value assigned for term"),
NULL);
}
term->config, term->val.str);
}
if (err) {
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
strdup("expected numeric value"),
NULL);
}
if (err) {
char *err_str;
- parse_events__handle_error(err, term->err_val,
+ parse_events_error__handle(err, term->err_val,
asprintf(&err_str,
"value too big for format, maximum is %llu",
(unsigned long long)max_val) < 0
{
const struct sevent *as = a;
const struct sevent *bs = b;
+ int ret;
/* Put extra events last */
if (!!as->desc != !!bs->desc)
if (as->is_cpu != bs->is_cpu)
return bs->is_cpu - as->is_cpu;
- return strcmp(as->name, bs->name);
+ ret = strcmp(as->name, bs->name);
+ if (!ret) {
+ if (as->pmu && bs->pmu)
+ return strcmp(as->pmu, bs->pmu);
+ }
+
+ return ret;
}
static void wordwrap(char *s, int start, int max, int corr)
}
void print_pmu_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc, bool details_flag, bool deprecated)
+ bool long_desc, bool details_flag, bool deprecated,
+ const char *pmu_name)
{
struct perf_pmu *pmu;
struct perf_pmu_alias *alias;
pmu = NULL;
j = 0;
while ((pmu = perf_pmu__scan(pmu)) != NULL) {
+ if (pmu_name && perf_pmu__is_hybrid(pmu->name) &&
+ strcmp(pmu_name, pmu->name)) {
+ continue;
+ }
+
list_for_each_entry(alias, &pmu->aliases, list) {
char *name = alias->desc ? alias->name :
format_alias(buf, sizeof(buf), pmu, alias);
- bool is_cpu = is_pmu_core(pmu->name);
+ bool is_cpu = is_pmu_core(pmu->name) ||
+ perf_pmu__is_hybrid(pmu->name);
if (alias->deprecated && !deprecated)
continue;
qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
for (j = 0; j < len; j++) {
/* Skip duplicates */
- if (j > 0 && !strcmp(aliases[j].name, aliases[j - 1].name))
- continue;
+ if (j > 0 && !strcmp(aliases[j].name, aliases[j - 1].name)) {
+ if (!aliases[j].pmu || !aliases[j - 1].pmu ||
+ !strcmp(aliases[j].pmu, aliases[j - 1].pmu)) {
+ continue;
+ }
+ }
+
if (name_only) {
printf("%s ", aliases[j].name);
continue;
}
void perf_pmu__warn_invalid_config(struct perf_pmu *pmu, __u64 config,
- char *name)
+ const char *name)
{
struct perf_pmu_format *format;
__u64 masks = 0, bits;
struct list_head caps; /* HEAD struct perf_pmu_caps -> list */
struct list_head list; /* ELEM */
struct list_head hybrid_list;
+
+ struct {
+ bool exclude_guest;
+ } missing_features;
};
extern struct perf_pmu perf_pmu__fake;
bool is_pmu_core(const char *name);
void print_pmu_events(const char *event_glob, bool name_only, bool quiet,
bool long_desc, bool details_flag,
- bool deprecated);
+ bool deprecated, const char *pmu_name);
bool pmu_have_event(const char *pname, const char *name);
int perf_pmu__scan_file(struct perf_pmu *pmu, const char *name, const char *fmt, ...) __scanf(3, 4);
struct perf_event_attr *perf_pmu__get_default_config(struct perf_pmu *pmu);
void pmu_add_cpu_aliases_map(struct list_head *head, struct perf_pmu *pmu,
- struct pmu_events_map *map);
+ const struct pmu_events_map *map);
-struct pmu_events_map *perf_pmu__find_map(struct perf_pmu *pmu);
-struct pmu_events_map *pmu_events_map__find(void);
+const struct pmu_events_map *perf_pmu__find_map(struct perf_pmu *pmu);
+const struct pmu_events_map *pmu_events_map__find(void);
bool pmu_uncore_alias_match(const char *pmu_name, const char *name);
void perf_pmu_free_alias(struct perf_pmu_alias *alias);
-typedef int (*pmu_sys_event_iter_fn)(struct pmu_event *pe, void *data);
+typedef int (*pmu_sys_event_iter_fn)(const struct pmu_event *pe, void *data);
void pmu_for_each_sys_event(pmu_sys_event_iter_fn fn, void *data);
int perf_pmu__convert_scale(const char *scale, char **end, double *sval);
int perf_pmu__caps_parse(struct perf_pmu *pmu);
void perf_pmu__warn_invalid_config(struct perf_pmu *pmu, __u64 config,
- char *name);
+ const char *name);
bool perf_pmu__has_hybrid(void);
int perf_pmu__match(char *pattern, char *name, char *tok);
util/namespaces.c
../lib/bitmap.c
../lib/find_bit.c
+../lib/list_sort.c
../lib/hweight.c
../lib/string.c
../lib/vsprintf.c
}
/*
+ * This one is needed not to drag the PMU bandwagon, jevents generated
+ * pmu_sys_event_tables, etc and evsel__find_pmu() is used so far just for
+ * doing per PMU perf_event_attr.exclude_guest handling, not really needed, so
+ * far, for the perf python binding known usecases, revisit if this become
+ * necessary.
+ */
+struct perf_pmu *evsel__find_pmu(struct evsel *evsel __maybe_unused)
+{
+ return NULL;
+}
+
+/*
* Add this one here not to drag util/metricgroup.c
*/
int metricgroup__copy_metric_events(struct evlist *evlist, struct cgroup *cgrp,
int ctl_fd;
int ctl_fd_ack;
bool ctl_fd_close;
+ int synth;
};
extern const char * const *record_usage;
struct hws_basic_entry *basicp)
{
struct hws_basic_entry *basic = basicp;
-#if __BYTE_ORDER == __LITTLE_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
struct hws_basic_entry local;
unsigned long long word = be64toh(*(unsigned long long *)basicp);
struct hws_diag_entry *diagp)
{
struct hws_diag_entry *diag = diagp;
-#if __BYTE_ORDER == __LITTLE_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
struct hws_diag_entry local;
unsigned long long word = be64toh(*(unsigned long long *)diagp);
static bool s390_cpumsf_trailer_show(const char *color, size_t pos,
struct hws_trailer_entry *te)
{
-#if __BYTE_ORDER == __LITTLE_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
struct hws_trailer_entry local;
const unsigned long long flags = be64toh(te->flags);
te = (struct hws_trailer_entry *)(buf + S390_CPUMSF_PAGESZ
- sizeof(*te));
-#if __BYTE_ORDER == __LITTLE_ENDIAN
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
clock_base = be64toh(te->progusage[0]) >> 63 & 0x1;
progusage2 = be64toh(te->progusage[1]);
#else
* the name of this counter.
* If no match is found a NULL pointer is returned.
*/
-static const char *get_counter_name(int set, int nr, struct pmu_events_map *map)
+static const char *get_counter_name(int set, int nr, const struct pmu_events_map *map)
{
int rc, event_nr, wanted = get_counterset_start(set) + nr;
if (map) {
- struct pmu_event *evp = map->table;
+ const struct pmu_event *evp = map->table;
for (; evp->name || evp->event || evp->desc; ++evp) {
if (evp->name == NULL || evp->event == NULL)
unsigned char *buf = sample->raw_data;
const char *color = PERF_COLOR_BLUE;
struct cf_ctrset_entry *cep, ce;
- struct pmu_events_map *map;
+ const struct pmu_events_map *map;
u64 *p;
map = pmu_events_map__find();
size_t decomp_size, src_size;
u64 decomp_last_rem = 0;
size_t mmap_len, decomp_len = session->header.env.comp_mmap_len;
- struct decomp *decomp, *decomp_last = session->decomp_last;
+ struct decomp *decomp, *decomp_last = session->active_decomp->decomp_last;
if (decomp_last) {
decomp_last_rem = decomp_last->size - decomp_last->head;
src = (void *)event + sizeof(struct perf_record_compressed);
src_size = event->pack.header.size - sizeof(struct perf_record_compressed);
- decomp_size = zstd_decompress_stream(&(session->zstd_data), src, src_size,
+ decomp_size = zstd_decompress_stream(session->active_decomp->zstd_decomp, src, src_size,
&(decomp->data[decomp_last_rem]), decomp_len - decomp_last_rem);
if (!decomp_size) {
munmap(decomp, mmap_len);
decomp->size += decomp_size;
- if (session->decomp == NULL) {
- session->decomp = decomp;
- session->decomp_last = decomp;
- } else {
- session->decomp_last->next = decomp;
- session->decomp_last = decomp;
- }
+ if (session->active_decomp->decomp == NULL)
+ session->active_decomp->decomp = decomp;
+ else
+ session->active_decomp->decomp_last->next = decomp;
+
+ session->active_decomp->decomp_last = decomp;
pr_debug("decomp (B): %zd to %zd\n", src_size, decomp_size);
session->repipe = repipe;
session->tool = tool;
+ session->decomp_data.zstd_decomp = &session->zstd_data;
+ session->active_decomp = &session->decomp_data;
INIT_LIST_HEAD(&session->auxtrace_index);
machines__init(&session->machines);
ordered_events__init(&session->ordered_events,
machine__delete_threads(&session->machines.host);
}
-static void perf_session__release_decomp_events(struct perf_session *session)
+static void perf_decomp__release_events(struct decomp *next)
{
- struct decomp *next, *decomp;
+ struct decomp *decomp;
size_t mmap_len;
- next = session->decomp;
+
do {
decomp = next;
if (decomp == NULL)
auxtrace_index__free(&session->auxtrace_index);
perf_session__destroy_kernel_maps(session);
perf_session__delete_threads(session);
- perf_session__release_decomp_events(session);
+ perf_decomp__release_events(session->decomp_data.decomp);
perf_env__exit(&session->header.env);
machines__exit(&session->machines);
if (session->data) {
tool->bpf = perf_event__process_bpf;
if (tool->text_poke == NULL)
tool->text_poke = perf_event__process_text_poke;
+ if (tool->aux_output_hw_id == NULL)
+ tool->aux_output_hw_id = perf_event__process_aux_output_hw_id;
if (tool->read == NULL)
tool->read = process_event_sample_stub;
if (tool->throttle == NULL)
[PERF_RECORD_NAMESPACES] = perf_event__namespaces_swap,
[PERF_RECORD_CGROUP] = perf_event__cgroup_swap,
[PERF_RECORD_TEXT_POKE] = perf_event__text_poke_swap,
+ [PERF_RECORD_AUX_OUTPUT_HW_ID] = perf_event__all64_swap,
[PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap,
[PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap,
[PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
return tool->bpf(tool, event, sample, machine);
case PERF_RECORD_TEXT_POKE:
return tool->text_poke(tool, event, sample, machine);
+ case PERF_RECORD_AUX_OUTPUT_HW_ID:
+ return tool->aux_output_hw_id(tool, event, sample, machine);
default:
++evlist->stats.nr_unknown_events;
return -1;
{
s64 skip;
u64 size;
- struct decomp *decomp = session->decomp_last;
+ struct decomp *decomp = session->active_decomp->decomp_last;
if (!decomp)
return 0;
u64 data_offset;
reader_cb_t process;
bool in_place_update;
+ char *mmaps[NUM_MMAPS];
+ size_t mmap_size;
+ int mmap_idx;
+ char *mmap_cur;
+ u64 file_pos;
+ u64 file_offset;
+ u64 head;
+ struct zstd_data zstd_data;
+ struct decomp_data decomp_data;
};
static int
-reader__process_events(struct reader *rd, struct perf_session *session,
- struct ui_progress *prog)
+reader__init(struct reader *rd, bool *one_mmap)
{
u64 data_size = rd->data_size;
- u64 head, page_offset, file_offset, file_pos, size;
- int err = 0, mmap_prot, mmap_flags, map_idx = 0;
- size_t mmap_size;
- char *buf, *mmaps[NUM_MMAPS];
- union perf_event *event;
- s64 skip;
-
- page_offset = page_size * (rd->data_offset / page_size);
- file_offset = page_offset;
- head = rd->data_offset - page_offset;
-
- ui_progress__init_size(prog, data_size, "Processing events...");
+ char **mmaps = rd->mmaps;
+ rd->head = rd->data_offset;
data_size += rd->data_offset;
- mmap_size = MMAP_SIZE;
- if (mmap_size > data_size) {
- mmap_size = data_size;
- session->one_mmap = true;
+ rd->mmap_size = MMAP_SIZE;
+ if (rd->mmap_size > data_size) {
+ rd->mmap_size = data_size;
+ if (one_mmap)
+ *one_mmap = true;
}
- memset(mmaps, 0, sizeof(mmaps));
+ memset(mmaps, 0, sizeof(rd->mmaps));
+
+ if (zstd_init(&rd->zstd_data, 0))
+ return -1;
+ rd->decomp_data.zstd_decomp = &rd->zstd_data;
+
+ return 0;
+}
+
+static void
+reader__release_decomp(struct reader *rd)
+{
+ perf_decomp__release_events(rd->decomp_data.decomp);
+ zstd_fini(&rd->zstd_data);
+}
+
+static int
+reader__mmap(struct reader *rd, struct perf_session *session)
+{
+ int mmap_prot, mmap_flags;
+ char *buf, **mmaps = rd->mmaps;
+ u64 page_offset;
mmap_prot = PROT_READ;
mmap_flags = MAP_SHARED;
mmap_prot |= PROT_WRITE;
mmap_flags = MAP_PRIVATE;
}
-remap:
- buf = mmap(NULL, mmap_size, mmap_prot, mmap_flags, rd->fd,
- file_offset);
+
+ if (mmaps[rd->mmap_idx]) {
+ munmap(mmaps[rd->mmap_idx], rd->mmap_size);
+ mmaps[rd->mmap_idx] = NULL;
+ }
+
+ page_offset = page_size * (rd->head / page_size);
+ rd->file_offset += page_offset;
+ rd->head -= page_offset;
+
+ buf = mmap(NULL, rd->mmap_size, mmap_prot, mmap_flags, rd->fd,
+ rd->file_offset);
if (buf == MAP_FAILED) {
pr_err("failed to mmap file\n");
- err = -errno;
- goto out;
+ return -errno;
}
- mmaps[map_idx] = buf;
- map_idx = (map_idx + 1) & (ARRAY_SIZE(mmaps) - 1);
- file_pos = file_offset + head;
+ mmaps[rd->mmap_idx] = rd->mmap_cur = buf;
+ rd->mmap_idx = (rd->mmap_idx + 1) & (ARRAY_SIZE(rd->mmaps) - 1);
+ rd->file_pos = rd->file_offset + rd->head;
if (session->one_mmap) {
session->one_mmap_addr = buf;
- session->one_mmap_offset = file_offset;
+ session->one_mmap_offset = rd->file_offset;
}
-more:
- event = fetch_mmaped_event(head, mmap_size, buf, session->header.needs_swap);
+ return 0;
+}
+
+enum {
+ READER_OK,
+ READER_NODATA,
+};
+
+static int
+reader__read_event(struct reader *rd, struct perf_session *session,
+ struct ui_progress *prog)
+{
+ u64 size;
+ int err = READER_OK;
+ union perf_event *event;
+ s64 skip;
+
+ event = fetch_mmaped_event(rd->head, rd->mmap_size, rd->mmap_cur,
+ session->header.needs_swap);
if (IS_ERR(event))
return PTR_ERR(event);
- if (!event) {
- if (mmaps[map_idx]) {
- munmap(mmaps[map_idx], mmap_size);
- mmaps[map_idx] = NULL;
- }
-
- page_offset = page_size * (head / page_size);
- file_offset += page_offset;
- head -= page_offset;
- goto remap;
- }
+ if (!event)
+ return READER_NODATA;
size = event->header.size;
skip = -EINVAL;
if (size < sizeof(struct perf_event_header) ||
- (skip = rd->process(session, event, file_pos)) < 0) {
+ (skip = rd->process(session, event, rd->file_pos)) < 0) {
pr_err("%#" PRIx64 " [%#x]: failed to process type: %d [%s]\n",
- file_offset + head, event->header.size,
+ rd->file_offset + rd->head, event->header.size,
event->header.type, strerror(-skip));
err = skip;
goto out;
if (skip)
size += skip;
- head += size;
- file_pos += size;
+ rd->head += size;
+ rd->file_pos += size;
err = __perf_session__process_decomp_events(session);
if (err)
ui_progress__update(prog, size);
+out:
+ return err;
+}
+
+static inline bool
+reader__eof(struct reader *rd)
+{
+ return (rd->file_pos >= rd->data_size + rd->data_offset);
+}
+
+static int
+reader__process_events(struct reader *rd, struct perf_session *session,
+ struct ui_progress *prog)
+{
+ int err;
+
+ err = reader__init(rd, &session->one_mmap);
+ if (err)
+ goto out;
+
+ session->active_decomp = &rd->decomp_data;
+
+remap:
+ err = reader__mmap(rd, session);
+ if (err)
+ goto out;
+
+more:
+ err = reader__read_event(rd, session, prog);
+ if (err < 0)
+ goto out;
+ else if (err == READER_NODATA)
+ goto remap;
+
if (session_done())
goto out;
- if (file_pos < data_size)
+ if (!reader__eof(rd))
goto more;
out:
+ session->active_decomp = &session->decomp_data;
return err;
}
*/
ordered_events__reinit(&session->ordered_events);
auxtrace__free_events(session);
+ reader__release_decomp(&rd);
session->one_mmap = false;
return err;
}
struct auxtrace;
struct itrace_synth_opts;
+struct decomp_data {
+ struct decomp *decomp;
+ struct decomp *decomp_last;
+ struct zstd_data *zstd_decomp;
+};
+
struct perf_session {
struct perf_header header;
struct machines machines;
u64 bytes_transferred;
u64 bytes_compressed;
struct zstd_data zstd_data;
- struct decomp *decomp;
- struct decomp *decomp_last;
+ struct decomp_data decomp_data;
+ struct decomp_data *active_decomp;
};
struct decomp {
.se_width_idx = HISTC_MISPREDICT,
};
-static u64 he_weight(struct hist_entry *he)
-{
- return he->stat.nr_events ? he->stat.weight / he->stat.nr_events : 0;
-}
-
static int64_t
-sort__local_weight_cmp(struct hist_entry *left, struct hist_entry *right)
+sort__weight_cmp(struct hist_entry *left, struct hist_entry *right)
{
- return he_weight(left) - he_weight(right);
+ return left->weight - right->weight;
}
static int hist_entry__local_weight_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
- return repsep_snprintf(bf, size, "%-*llu", width, he_weight(he));
+ return repsep_snprintf(bf, size, "%-*llu", width, he->weight);
}
struct sort_entry sort_local_weight = {
.se_header = "Local Weight",
- .se_cmp = sort__local_weight_cmp,
+ .se_cmp = sort__weight_cmp,
.se_snprintf = hist_entry__local_weight_snprintf,
.se_width_idx = HISTC_LOCAL_WEIGHT,
};
-static int64_t
-sort__global_weight_cmp(struct hist_entry *left, struct hist_entry *right)
-{
- return left->stat.weight - right->stat.weight;
-}
-
static int hist_entry__global_weight_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
- return repsep_snprintf(bf, size, "%-*llu", width, he->stat.weight);
+ return repsep_snprintf(bf, size, "%-*llu", width,
+ he->weight * he->stat.nr_events);
}
struct sort_entry sort_global_weight = {
.se_header = "Weight",
- .se_cmp = sort__global_weight_cmp,
+ .se_cmp = sort__weight_cmp,
.se_snprintf = hist_entry__global_weight_snprintf,
.se_width_idx = HISTC_GLOBAL_WEIGHT,
};
-static u64 he_ins_lat(struct hist_entry *he)
-{
- return he->stat.nr_events ? he->stat.ins_lat / he->stat.nr_events : 0;
-}
-
static int64_t
-sort__local_ins_lat_cmp(struct hist_entry *left, struct hist_entry *right)
+sort__ins_lat_cmp(struct hist_entry *left, struct hist_entry *right)
{
- return he_ins_lat(left) - he_ins_lat(right);
+ return left->ins_lat - right->ins_lat;
}
static int hist_entry__local_ins_lat_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
- return repsep_snprintf(bf, size, "%-*u", width, he_ins_lat(he));
+ return repsep_snprintf(bf, size, "%-*u", width, he->ins_lat);
}
struct sort_entry sort_local_ins_lat = {
.se_header = "Local INSTR Latency",
- .se_cmp = sort__local_ins_lat_cmp,
+ .se_cmp = sort__ins_lat_cmp,
.se_snprintf = hist_entry__local_ins_lat_snprintf,
.se_width_idx = HISTC_LOCAL_INS_LAT,
};
-static int64_t
-sort__global_ins_lat_cmp(struct hist_entry *left, struct hist_entry *right)
-{
- return left->stat.ins_lat - right->stat.ins_lat;
-}
-
static int hist_entry__global_ins_lat_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
- return repsep_snprintf(bf, size, "%-*u", width, he->stat.ins_lat);
+ return repsep_snprintf(bf, size, "%-*u", width,
+ he->ins_lat * he->stat.nr_events);
}
struct sort_entry sort_global_ins_lat = {
.se_header = "INSTR Latency",
- .se_cmp = sort__global_ins_lat_cmp,
+ .se_cmp = sort__ins_lat_cmp,
.se_snprintf = hist_entry__global_ins_lat_snprintf,
.se_width_idx = HISTC_GLOBAL_INS_LAT,
};
static int64_t
sort__global_p_stage_cyc_cmp(struct hist_entry *left, struct hist_entry *right)
{
- return left->stat.p_stage_cyc - right->stat.p_stage_cyc;
+ return left->p_stage_cyc - right->p_stage_cyc;
}
static int hist_entry__p_stage_cyc_snprintf(struct hist_entry *he, char *bf,
size_t size, unsigned int width)
{
- return repsep_snprintf(bf, size, "%-*u", width, he->stat.p_stage_cyc);
+ return repsep_snprintf(bf, size, "%-*u", width, he->p_stage_cyc);
}
struct sort_entry sort_p_stage_cyc = {
u64 period_us;
u64 period_guest_sys;
u64 period_guest_us;
- u64 weight;
- u64 ins_lat;
- u64 p_stage_cyc;
u32 nr_events;
};
s32 socket;
s32 cpu;
u64 code_page_size;
+ u64 weight;
+ u64 ins_lat;
+ u64 p_stage_cyc;
u8 cpumode;
u8 depth;
// SPDX-License-Identifier: GPL-2.0
#include <inttypes.h>
+#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#include <sys/types.h>
#include <linux/kernel.h>
#include <linux/string.h>
#include "srcline.h"
#include "string2.h"
#include "symbol.h"
+#include "subcmd/run-command.h"
bool srcline_full_filename;
return inline_sym;
}
+#define MAX_INLINE_NEST 1024
+
#ifdef HAVE_LIBBFD_SUPPORT
/*
free(a2l);
}
-#define MAX_INLINE_NEST 1024
-
static int inline_list__append_dso_a2l(struct dso *dso,
struct inline_node *node,
struct symbol *sym)
dso->a2l = NULL;
}
-static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
- struct dso *dso, struct symbol *sym)
-{
- struct inline_node *node;
-
- node = zalloc(sizeof(*node));
- if (node == NULL) {
- perror("not enough memory for the inline node");
- return NULL;
- }
-
- INIT_LIST_HEAD(&node->val);
- node->addr = addr;
-
- addr2line(dso_name, addr, NULL, NULL, dso, true, node, sym);
- return node;
-}
-
#else /* HAVE_LIBBFD_SUPPORT */
+struct a2l_subprocess {
+ struct child_process addr2line;
+ FILE *to_child;
+ FILE *from_child;
+};
+
static int filename_split(char *filename, unsigned int *line_nr)
{
char *sep;
return 0;
}
-static int addr2line(const char *dso_name, u64 addr,
- char **file, unsigned int *line_nr,
- struct dso *dso __maybe_unused,
- bool unwind_inlines __maybe_unused,
- struct inline_node *node __maybe_unused,
- struct symbol *sym __maybe_unused)
+static void addr2line_subprocess_cleanup(struct a2l_subprocess *a2l)
{
- FILE *fp;
- char cmd[PATH_MAX];
- char *filename = NULL;
- size_t len;
- int ret = 0;
+ if (a2l->addr2line.pid != -1) {
+ kill(a2l->addr2line.pid, SIGKILL);
+ finish_command(&a2l->addr2line); /* ignore result, we don't care */
+ a2l->addr2line.pid = -1;
+ }
- scnprintf(cmd, sizeof(cmd), "addr2line -e %s %016"PRIx64,
- dso_name, addr);
+ if (a2l->to_child != NULL) {
+ fclose(a2l->to_child);
+ a2l->to_child = NULL;
+ }
- fp = popen(cmd, "r");
- if (fp == NULL) {
- pr_warning("popen failed for %s\n", dso_name);
- return 0;
+ if (a2l->from_child != NULL) {
+ fclose(a2l->from_child);
+ a2l->from_child = NULL;
+ }
+
+ free(a2l);
+}
+
+static struct a2l_subprocess *addr2line_subprocess_init(const char *path)
+{
+ const char *argv[] = { "addr2line", "-e", path, "-i", "-f", NULL };
+ struct a2l_subprocess *a2l = zalloc(sizeof(*a2l));
+ int start_command_status = 0;
+
+ if (a2l == NULL)
+ goto out;
+
+ a2l->to_child = NULL;
+ a2l->from_child = NULL;
+
+ a2l->addr2line.pid = -1;
+ a2l->addr2line.in = -1;
+ a2l->addr2line.out = -1;
+ a2l->addr2line.no_stderr = 1;
+
+ a2l->addr2line.argv = argv;
+ start_command_status = start_command(&a2l->addr2line);
+ a2l->addr2line.argv = NULL; /* it's not used after start_command; avoid dangling pointers */
+
+ if (start_command_status != 0) {
+ pr_warning("could not start addr2line for %s: start_command return code %d\n",
+ path,
+ start_command_status);
+ goto out;
}
- if (getline(&filename, &len, fp) < 0 || !len) {
- pr_warning("addr2line has no output for %s\n", dso_name);
+ a2l->to_child = fdopen(a2l->addr2line.in, "w");
+ if (a2l->to_child == NULL) {
+ pr_warning("could not open write-stream to addr2line of %s\n", path);
goto out;
}
- ret = filename_split(filename, line_nr);
- if (ret != 1) {
- free(filename);
+ a2l->from_child = fdopen(a2l->addr2line.out, "r");
+ if (a2l->from_child == NULL) {
+ pr_warning("could not open read-stream from addr2line of %s\n", path);
goto out;
}
- *file = filename;
+ return a2l;
out:
- pclose(fp);
- return ret;
+ if (a2l)
+ addr2line_subprocess_cleanup(a2l);
+
+ return NULL;
}
-void dso__free_a2l(struct dso *dso __maybe_unused)
+static int read_addr2line_record(struct a2l_subprocess *a2l,
+ char **function,
+ char **filename,
+ unsigned int *line_nr)
{
+ /*
+ * Returns:
+ * -1 ==> error
+ * 0 ==> sentinel (or other ill-formed) record read
+ * 1 ==> a genuine record read
+ */
+ char *line = NULL;
+ size_t line_len = 0;
+ unsigned int dummy_line_nr = 0;
+ int ret = -1;
+
+ if (function != NULL)
+ zfree(function);
+
+ if (filename != NULL)
+ zfree(filename);
+
+ if (line_nr != NULL)
+ *line_nr = 0;
+
+ if (getline(&line, &line_len, a2l->from_child) < 0 || !line_len)
+ goto error;
+
+ if (function != NULL)
+ *function = strdup(strim(line));
+
+ zfree(&line);
+ line_len = 0;
+
+ if (getline(&line, &line_len, a2l->from_child) < 0 || !line_len)
+ goto error;
+
+ if (filename_split(line, line_nr == NULL ? &dummy_line_nr : line_nr) == 0) {
+ ret = 0;
+ goto error;
+ }
+
+ if (filename != NULL)
+ *filename = strdup(line);
+
+ zfree(&line);
+ line_len = 0;
+
+ return 1;
+
+error:
+ free(line);
+ if (function != NULL)
+ zfree(function);
+ if (filename != NULL)
+ zfree(filename);
+ return ret;
}
-static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
- struct dso *dso __maybe_unused,
- struct symbol *sym)
+static int inline_list__append_record(struct dso *dso,
+ struct inline_node *node,
+ struct symbol *sym,
+ const char *function,
+ const char *filename,
+ unsigned int line_nr)
{
- FILE *fp;
- char cmd[PATH_MAX];
- struct inline_node *node;
- char *filename = NULL;
- char *funcname = NULL;
- size_t filelen, funclen;
- unsigned int line_nr = 0;
+ struct symbol *inline_sym = new_inline_sym(dso, sym, function);
- scnprintf(cmd, sizeof(cmd), "addr2line -e %s -i -f %016"PRIx64,
- dso_name, addr);
+ return inline_list__append(inline_sym, srcline_from_fileline(filename, line_nr), node);
+}
- fp = popen(cmd, "r");
- if (fp == NULL) {
- pr_err("popen failed for %s\n", dso_name);
- return NULL;
+static int addr2line(const char *dso_name, u64 addr,
+ char **file, unsigned int *line_nr,
+ struct dso *dso,
+ bool unwind_inlines,
+ struct inline_node *node,
+ struct symbol *sym __maybe_unused)
+{
+ struct a2l_subprocess *a2l = dso->a2l;
+ char *record_function = NULL;
+ char *record_filename = NULL;
+ unsigned int record_line_nr = 0;
+ int record_status = -1;
+ int ret = 0;
+ size_t inline_count = 0;
+
+ if (!a2l) {
+ dso->a2l = addr2line_subprocess_init(dso_name);
+ a2l = dso->a2l;
}
- node = zalloc(sizeof(*node));
- if (node == NULL) {
- perror("not enough memory for the inline node");
+ if (a2l == NULL) {
+ if (!symbol_conf.disable_add2line_warn)
+ pr_warning("%s %s: addr2line_subprocess_init failed\n", __func__, dso_name);
goto out;
}
- INIT_LIST_HEAD(&node->val);
- node->addr = addr;
-
- /* addr2line -f generates two lines for each inlined functions */
- while (getline(&funcname, &funclen, fp) != -1) {
- char *srcline;
- struct symbol *inline_sym;
+ /*
+ * Send our request and then *deliberately* send something that can't be interpreted as
+ * a valid address to ask addr2line about (namely, ","). This causes addr2line to first
+ * write out the answer to our request, in an unbounded/unknown number of records, and
+ * then to write out the lines "??" and "??:0", so that we can detect when it has
+ * finished giving us anything useful. We have to be careful about the first record,
+ * though, because it may be genuinely unknown, in which case we'll get two sets of
+ * "??"/"??:0" lines.
+ */
+ if (fprintf(a2l->to_child, "%016"PRIx64"\n,\n", addr) < 0 || fflush(a2l->to_child) != 0) {
+ pr_warning("%s %s: could not send request\n", __func__, dso_name);
+ goto out;
+ }
- strim(funcname);
+ switch (read_addr2line_record(a2l, &record_function, &record_filename, &record_line_nr)) {
+ case -1:
+ pr_warning("%s %s: could not read first record\n", __func__, dso_name);
+ goto out;
+ case 0:
+ /*
+ * The first record was invalid, so return failure, but first read another
+ * record, since we asked a junk question and have to clear the answer out.
+ */
+ switch (read_addr2line_record(a2l, NULL, NULL, NULL)) {
+ case -1:
+ pr_warning("%s %s: could not read delimiter record\n", __func__, dso_name);
+ break;
+ case 0:
+ /* As expected. */
+ break;
+ default:
+ pr_warning("%s %s: unexpected record instead of sentinel",
+ __func__, dso_name);
+ break;
+ }
+ goto out;
+ default:
+ break;
+ }
- if (getline(&filename, &filelen, fp) == -1)
- goto out;
+ if (file) {
+ *file = strdup(record_filename);
+ ret = 1;
+ }
+ if (line_nr)
+ *line_nr = record_line_nr;
- if (filename_split(filename, &line_nr) != 1)
+ if (unwind_inlines) {
+ if (node && inline_list__append_record(dso, node, sym,
+ record_function,
+ record_filename,
+ record_line_nr)) {
+ ret = 0;
goto out;
+ }
+ }
- srcline = srcline_from_fileline(filename, line_nr);
- inline_sym = new_inline_sym(dso, sym, funcname);
-
- if (inline_list__append(inline_sym, srcline, node) != 0) {
- free(srcline);
- if (inline_sym && inline_sym->inlined)
- symbol__delete(inline_sym);
- goto out;
+ /* We have to read the records even if we don't care about the inline info. */
+ while ((record_status = read_addr2line_record(a2l,
+ &record_function,
+ &record_filename,
+ &record_line_nr)) == 1) {
+ if (unwind_inlines && node && inline_count++ < MAX_INLINE_NEST) {
+ if (inline_list__append_record(dso, node, sym,
+ record_function,
+ record_filename,
+ record_line_nr)) {
+ ret = 0;
+ goto out;
+ }
+ ret = 1; /* found at least one inline frame */
}
}
out:
- pclose(fp);
- free(filename);
- free(funcname);
+ free(record_function);
+ free(record_filename);
+ return ret;
+}
- return node;
+void dso__free_a2l(struct dso *dso)
+{
+ struct a2l_subprocess *a2l = dso->a2l;
+
+ if (!a2l)
+ return;
+
+ addr2line_subprocess_cleanup(a2l);
+
+ dso->a2l = NULL;
}
#endif /* HAVE_LIBBFD_SUPPORT */
+static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
+ struct dso *dso, struct symbol *sym)
+{
+ struct inline_node *node;
+
+ node = zalloc(sizeof(*node));
+ if (node == NULL) {
+ perror("not enough memory for the inline node");
+ return NULL;
+ }
+
+ INIT_LIST_HEAD(&node->val);
+ node->addr = addr;
+
+ addr2line(dso_name, addr, NULL, NULL, dso, true, node, sym);
+ return node;
+}
+
/*
* Number of addr2line failures (without success) before disabling it for that
* dso.
// SPDX-License-Identifier: GPL-2.0
+#include <math.h>
#include <stdio.h>
#include "evsel.h"
#include "stat.h"
#include "color.h"
+#include "debug.h"
#include "pmu.h"
#include "rblist.h"
#include "evlist.h"
{
struct evsel *counter, *leader, **metric_events, *oc;
bool found;
- struct expr_parse_ctx ctx;
+ struct expr_parse_ctx *ctx;
struct hashmap_entry *cur;
size_t bkt;
int i;
- expr__ctx_init(&ctx);
+ ctx = expr__ctx_new();
+ if (!ctx) {
+ pr_debug("expr__ctx_new failed");
+ return;
+ }
evlist__for_each_entry(evsel_list, counter) {
bool invalid = false;
if (!counter->metric_expr)
continue;
- expr__ctx_clear(&ctx);
+ expr__ctx_clear(ctx);
metric_events = counter->metric_events;
if (!metric_events) {
- if (expr__find_other(counter->metric_expr,
- counter->name,
- &ctx, 1) < 0)
+ if (expr__find_ids(counter->metric_expr,
+ counter->name,
+ ctx) < 0)
continue;
metric_events = calloc(sizeof(struct evsel *),
- hashmap__size(&ctx.ids) + 1);
+ hashmap__size(ctx->ids) + 1);
if (!metric_events) {
- expr__ctx_clear(&ctx);
+ expr__ctx_free(ctx);
return;
}
counter->metric_events = metric_events;
}
i = 0;
- hashmap__for_each_entry((&ctx.ids), cur, bkt) {
+ hashmap__for_each_entry(ctx->ids, cur, bkt) {
const char *metric_name = (const char *)cur->key;
found = false;
"Add %s event to groups to get metric expression for %s\n",
metric_name,
counter->name);
+ free(printed);
printed = strdup(metric_name);
}
invalid = true;
counter->metric_expr = NULL;
}
}
- expr__ctx_clear(&ctx);
+ expr__ctx_free(ctx);
}
static double runtime_stat_avg(struct runtime_stat *st,
struct runtime_stat *st)
{
double scale;
- char *n, *pn;
+ char *n;
int i, j, ret;
- expr__ctx_init(pctx);
for (i = 0; metric_events[i]; i++) {
struct saved_value *v;
struct stats *stats;
u64 metric_total = 0;
+ int source_count;
if (!strcmp(metric_events[i]->name, "duration_time")) {
stats = &walltime_nsecs_stats;
scale = 1e-9;
+ source_count = 1;
} else {
v = saved_value_lookup(metric_events[i], cpu, false,
STAT_NONE, 0, st,
break;
stats = &v->stats;
scale = 1.0;
+ source_count = evsel__source_count(metric_events[i]);
if (v->metric_other)
metric_total = v->metric_total;
}
-
- n = strdup(metric_events[i]->name);
+ n = strdup(evsel__metric_id(metric_events[i]));
if (!n)
return -ENOMEM;
- /*
- * This display code with --no-merge adds [cpu] postfixes.
- * These are not supported by the parser. Remove everything
- * after the space.
- */
- pn = strchr(n, ' ');
- if (pn)
- *pn = 0;
-
- if (metric_total)
- expr__add_id_val(pctx, n, metric_total);
- else
- expr__add_id_val(pctx, n, avg_stats(stats)*scale);
+
+ expr__add_id_val_source_count(pctx, n,
+ metric_total ? : avg_stats(stats) * scale,
+ source_count);
}
for (j = 0; metric_refs && metric_refs[j].metric_name; j++) {
struct runtime_stat *st)
{
print_metric_t print_metric = out->print_metric;
- struct expr_parse_ctx pctx;
+ struct expr_parse_ctx *pctx;
double ratio, scale;
int i;
void *ctxp = out->ctx;
- i = prepare_metric(metric_events, metric_refs, &pctx, cpu, st);
- if (i < 0)
+ pctx = expr__ctx_new();
+ if (!pctx)
return;
+ pctx->runtime = runtime;
+ i = prepare_metric(metric_events, metric_refs, pctx, cpu, st);
+ if (i < 0) {
+ expr__ctx_free(pctx);
+ return;
+ }
if (!metric_events[i]) {
- if (expr__parse(&ratio, &pctx, metric_expr, runtime) == 0) {
+ if (expr__parse(&ratio, pctx, metric_expr) == 0) {
char *unit;
char metric_bf[64];
(metric_name ? metric_name : name) : "", 0);
}
- expr__ctx_clear(&pctx);
+ expr__ctx_free(pctx);
}
double test_generic_metric(struct metric_expr *mexp, int cpu, struct runtime_stat *st)
{
- struct expr_parse_ctx pctx;
+ struct expr_parse_ctx *pctx;
double ratio = 0.0;
- if (prepare_metric(mexp->metric_events, mexp->metric_refs, &pctx, cpu, st) < 0)
+ pctx = expr__ctx_new();
+ if (!pctx)
+ return NAN;
+
+ if (prepare_metric(mexp->metric_events, mexp->metric_refs, pctx, cpu, st) < 0)
goto out;
- if (expr__parse(&ratio, &pctx, mexp->metric_expr, 1))
+ if (expr__parse(&ratio, pctx, mexp->metric_expr))
ratio = 0.0;
out:
- expr__ctx_clear(&pctx);
+ expr__ctx_free(pctx);
return ratio;
}
if (symbol_conf.priv_size) {
if (symbol_conf.init_annotation) {
struct annotation *notes = (void *)sym;
- pthread_mutex_init(¬es->lock, NULL);
+ annotation__init(notes);
}
sym = ((void *)sym) + symbol_conf.priv_size;
}
void symbol__delete(struct symbol *sym)
{
+ if (symbol_conf.priv_size) {
+ if (symbol_conf.init_annotation) {
+ struct annotation *notes = symbol__annotation(sym);
+
+ annotation__exit(notes);
+ }
+ }
free(((void *)sym) - symbol_conf.priv_size);
}
if (!symbol_type__filter(type))
return 0;
+ /* Ignore local symbols for ARM modules */
+ if (name[0] == '$')
+ return 0;
+
/*
* module symbols are not sorted so we add all
* symbols, setting length to 0, and rely on
refcount_set(&mi->refcnt, 1);
return mi;
}
+
+/*
+ * Checks that user supplied symbol kernel files are accessible because
+ * the default mechanism for accessing elf files fails silently. i.e. if
+ * debug syms for a build ID aren't found perf carries on normally. When
+ * they are user supplied we should assume that the user doesn't want to
+ * silently fail.
+ */
+int symbol__validate_sym_arguments(void)
+{
+ if (symbol_conf.vmlinux_name &&
+ access(symbol_conf.vmlinux_name, R_OK)) {
+ pr_err("Invalid file: %s\n", symbol_conf.vmlinux_name);
+ return -EINVAL;
+ }
+ if (symbol_conf.kallsyms_name &&
+ access(symbol_conf.kallsyms_name, R_OK)) {
+ pr_err("Invalid file: %s\n", symbol_conf.kallsyms_name);
+ return -EINVAL;
+ }
+ return 0;
+}
GElf_Shdr *shp, const char *name, size_t *idx);
#endif
-/** struct symbol - symtab entry
- *
- * @ignore - resolvable but tools ignore it (e.g. idle routines)
+/**
+ * A symtab entry. When allocated this may be preceded by an annotation (see
+ * symbol__annotation), a browser_index (see symbol__browser_index) and rb_node
+ * to sort by name (see struct symbol_name_rb_node).
*/
struct symbol {
struct rb_node rb_node;
+ /** Range of symbol [start, end). */
u64 start;
u64 end;
+ /** Length of the string name. */
u16 namelen;
+ /** ELF symbol type as defined for st_info. E.g STT_OBJECT or STT_FUNC. */
u8 type:4;
+ /** ELF binding type as defined for st_info. E.g. STB_WEAK or STB_GLOBAL. */
u8 binding:4;
+ /** Set true for kernel symbols of idle routines. */
u8 idle:1;
+ /** Resolvable but tools ignore it (e.g. idle routines). */
u8 ignore:1;
+ /** Symbol for an inlined function. */
u8 inlined:1;
+ /** Has symbol__annotate2 been performed. */
+ u8 annotate2:1;
+ /** Architecture specific. Unused except on PPC where it holds st_other. */
u8 arch_sym;
- bool annotate2;
+ /** The name of length namelen associated with the symbol. */
char name[];
};
#define mem_info__zput(mi) __mem_info__zput(&mi)
+int symbol__validate_sym_arguments(void);
+
#endif /* __PERF_SYMBOL */
union perf_event *fork_event,
union perf_event *namespaces_event,
pid_t pid, int full, perf_event__handler_t process,
- struct perf_tool *tool, struct machine *machine, bool mmap_data)
+ struct perf_tool *tool, struct machine *machine,
+ bool needs_mmap, bool mmap_data)
{
char filename[PATH_MAX];
struct dirent **dirent;
* send mmap only for thread group leader
* see thread__init_maps()
*/
- if (pid == tgid &&
+ if (pid == tgid && needs_mmap &&
perf_event__synthesize_mmap_events(tool, mmap_event, pid, tgid,
process, machine, mmap_data))
return -1;
break;
rc = 0;
- if (_pid == pid && !kernel_thread) {
+ if (_pid == pid && !kernel_thread && needs_mmap) {
/* process the parent's maps too */
rc = perf_event__synthesize_mmap_events(tool, mmap_event, pid, tgid,
process, machine, mmap_data);
struct perf_thread_map *threads,
perf_event__handler_t process,
struct machine *machine,
- bool mmap_data)
+ bool needs_mmap, bool mmap_data)
{
union perf_event *comm_event, *mmap_event, *fork_event;
union perf_event *namespaces_event;
fork_event, namespaces_event,
perf_thread_map__pid(threads, thread), 0,
process, tool, machine,
- mmap_data)) {
+ needs_mmap, mmap_data)) {
err = -1;
break;
}
fork_event, namespaces_event,
comm_event->comm.pid, 0,
process, tool, machine,
- mmap_data)) {
+ needs_mmap, mmap_data)) {
err = -1;
break;
}
static int __perf_event__synthesize_threads(struct perf_tool *tool,
perf_event__handler_t process,
struct machine *machine,
+ bool needs_mmap,
bool mmap_data,
struct dirent **dirent,
int start,
*/
__event__synthesize_thread(comm_event, mmap_event, fork_event,
namespaces_event, pid, 1, process,
- tool, machine, mmap_data);
+ tool, machine, needs_mmap, mmap_data);
}
err = 0;
struct perf_tool *tool;
perf_event__handler_t process;
struct machine *machine;
+ bool needs_mmap;
bool mmap_data;
struct dirent **dirent;
int num;
struct synthesize_threads_arg *args = arg;
__perf_event__synthesize_threads(args->tool, args->process,
- args->machine, args->mmap_data,
+ args->machine,
+ args->needs_mmap, args->mmap_data,
args->dirent,
args->start, args->num);
return NULL;
int perf_event__synthesize_threads(struct perf_tool *tool,
perf_event__handler_t process,
struct machine *machine,
- bool mmap_data,
+ bool needs_mmap, bool mmap_data,
unsigned int nr_threads_synthesize)
{
struct synthesize_threads_arg *args = NULL;
if (thread_nr <= 1) {
err = __perf_event__synthesize_threads(tool, process,
- machine, mmap_data,
+ machine,
+ needs_mmap, mmap_data,
dirent, base, n);
goto free_dirent;
}
args[i].tool = tool;
args[i].process = process;
args[i].machine = machine;
+ args[i].needs_mmap = needs_mmap;
args[i].mmap_data = mmap_data;
args[i].dirent = dirent;
}
int __machine__synthesize_threads(struct machine *machine, struct perf_tool *tool,
struct target *target, struct perf_thread_map *threads,
- perf_event__handler_t process, bool data_mmap,
- unsigned int nr_threads_synthesize)
+ perf_event__handler_t process, bool needs_mmap,
+ bool data_mmap, unsigned int nr_threads_synthesize)
{
if (target__has_task(target))
- return perf_event__synthesize_thread_map(tool, threads, process, machine, data_mmap);
+ return perf_event__synthesize_thread_map(tool, threads, process, machine,
+ needs_mmap, data_mmap);
else if (target__has_cpu(target))
- return perf_event__synthesize_threads(tool, process,
- machine, data_mmap,
+ return perf_event__synthesize_threads(tool, process, machine,
+ needs_mmap, data_mmap,
nr_threads_synthesize);
/* command specified */
return 0;
}
int machine__synthesize_threads(struct machine *machine, struct target *target,
- struct perf_thread_map *threads, bool data_mmap,
- unsigned int nr_threads_synthesize)
+ struct perf_thread_map *threads, bool needs_mmap,
+ bool data_mmap, unsigned int nr_threads_synthesize)
{
return __machine__synthesize_threads(machine, NULL, target, threads,
- perf_event__process, data_mmap,
- nr_threads_synthesize);
+ perf_event__process, needs_mmap,
+ data_mmap, nr_threads_synthesize);
}
static struct perf_record_event_update *event_update_event__new(size_t size, u64 type, u64 id)
return ret;
}
+
+int parse_synth_opt(char *synth)
+{
+ char *p, *q;
+ int ret = 0;
+
+ if (synth == NULL)
+ return -1;
+
+ for (q = synth; (p = strsep(&q, ",")); p = q) {
+ if (!strcasecmp(p, "no") || !strcasecmp(p, "none"))
+ return 0;
+
+ if (!strcasecmp(p, "all"))
+ return PERF_SYNTH_ALL;
+
+ if (!strcasecmp(p, "task"))
+ ret |= PERF_SYNTH_TASK;
+ else if (!strcasecmp(p, "mmap"))
+ ret |= PERF_SYNTH_TASK | PERF_SYNTH_MMAP;
+ else if (!strcasecmp(p, "cgroup"))
+ ret |= PERF_SYNTH_CGROUP;
+ else
+ return -1;
+ }
+
+ return ret;
+}
union perf_event;
+enum perf_record_synth {
+ PERF_SYNTH_TASK = 1 << 0,
+ PERF_SYNTH_MMAP = 1 << 1,
+ PERF_SYNTH_CGROUP = 1 << 2,
+
+ /* last element */
+ PERF_SYNTH_MAX = 1 << 3,
+};
+#define PERF_SYNTH_ALL (PERF_SYNTH_MAX - 1)
+
+int parse_synth_opt(char *str);
+
typedef int (*perf_event__handler_t)(struct perf_tool *tool, union perf_event *event,
struct perf_sample *sample, struct machine *machine);
int perf_event__synthesize_stat_round(struct perf_tool *tool, u64 time, u64 type, perf_event__handler_t process, struct machine *machine);
int perf_event__synthesize_stat(struct perf_tool *tool, u32 cpu, u32 thread, u64 id, struct perf_counts_values *count, perf_event__handler_t process, struct machine *machine);
int perf_event__synthesize_thread_map2(struct perf_tool *tool, struct perf_thread_map *threads, perf_event__handler_t process, struct machine *machine);
-int perf_event__synthesize_thread_map(struct perf_tool *tool, struct perf_thread_map *threads, perf_event__handler_t process, struct machine *machine, bool mmap_data);
-int perf_event__synthesize_threads(struct perf_tool *tool, perf_event__handler_t process, struct machine *machine, bool mmap_data, unsigned int nr_threads_synthesize);
+int perf_event__synthesize_thread_map(struct perf_tool *tool, struct perf_thread_map *threads, perf_event__handler_t process, struct machine *machine, bool needs_mmap, bool mmap_data);
+int perf_event__synthesize_threads(struct perf_tool *tool, perf_event__handler_t process, struct machine *machine, bool needs_mmap, bool mmap_data, unsigned int nr_threads_synthesize);
int perf_event__synthesize_tracing_data(struct perf_tool *tool, int fd, struct evlist *evlist, perf_event__handler_t process);
int perf_event__synth_time_conv(const struct perf_event_mmap_page *pc, struct perf_tool *tool, perf_event__handler_t process, struct machine *machine);
pid_t perf_event__synthesize_comm(struct perf_tool *tool, union perf_event *event, pid_t pid, perf_event__handler_t process, struct machine *machine);
int __machine__synthesize_threads(struct machine *machine, struct perf_tool *tool,
struct target *target, struct perf_thread_map *threads,
- perf_event__handler_t process, bool data_mmap,
+ perf_event__handler_t process, bool needs_mmap, bool data_mmap,
unsigned int nr_threads_synthesize);
int machine__synthesize_threads(struct machine *machine, struct target *target,
- struct perf_thread_map *threads, bool data_mmap,
+ struct perf_thread_map *threads, bool needs_mmap, bool data_mmap,
unsigned int nr_threads_synthesize);
#ifdef HAVE_AUXTRACE_SUPPORT
lost_samples,
aux,
itrace_start,
+ aux_output_hw_id,
context_switch,
throttle,
unthrottle,
return 0;
}
-const char *perf_tip(const char *dirpath)
+int perf_tip(char **strp, const char *dirpath)
{
struct strlist *tips;
struct str_node *node;
- char *tip = NULL;
struct strlist_config conf = {
.dirname = dirpath,
.file_only = true,
};
+ int ret = 0;
+ *strp = NULL;
tips = strlist__new("tips.txt", &conf);
if (tips == NULL)
- return errno == ENOENT ? NULL :
- "Tip: check path of tips.txt or get more memory! ;-p";
+ return -errno;
if (strlist__nr_entries(tips) == 0)
goto out;
node = strlist__entry(tips, random() % strlist__nr_entries(tips));
- if (asprintf(&tip, "Tip: %s", node->s) < 0)
- tip = (char *)"Tip: get more memory! ;-)";
+ if (asprintf(strp, "Tip: %s", node->s) < 0)
+ ret = -ENOMEM;
out:
strlist__delete(tips);
- return tip;
+ return ret;
}
char *perf_exe(char *buf, int len)
#define KVER_FMT "%d.%d.%d"
#define KVER_PARAM(x) KVER_VERSION(x), KVER_PATCHLEVEL(x), KVER_SUBLEVEL(x)
-const char *perf_tip(const char *dirpath);
+int perf_tip(char **strp, const char *dirpath);
#ifndef HAVE_SCHED_GETCPU_SUPPORT
int sched_getcpu(void);
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+ldflags-y += --wrap=is_acpi_device_node
+ldflags-y += --wrap=acpi_get_table
+ldflags-y += --wrap=acpi_put_table
+ldflags-y += --wrap=acpi_evaluate_integer
+ldflags-y += --wrap=acpi_pci_find_root
+ldflags-y += --wrap=pci_walk_bus
+ldflags-y += --wrap=nvdimm_bus_register
+
+DRIVERS := ../../../drivers
+CXL_SRC := $(DRIVERS)/cxl
+CXL_CORE_SRC := $(DRIVERS)/cxl/core
+ccflags-y := -I$(srctree)/drivers/cxl/
+ccflags-y += -D__mock=__weak
+
+obj-m += cxl_acpi.o
+
+cxl_acpi-y := $(CXL_SRC)/acpi.o
+cxl_acpi-y += mock_acpi.o
+cxl_acpi-y += config_check.o
+
+obj-m += cxl_pmem.o
+
+cxl_pmem-y := $(CXL_SRC)/pmem.o
+cxl_pmem-y += config_check.o
+
+obj-m += cxl_core.o
+
+cxl_core-y := $(CXL_CORE_SRC)/bus.o
+cxl_core-y += $(CXL_CORE_SRC)/pmem.o
+cxl_core-y += $(CXL_CORE_SRC)/regs.o
+cxl_core-y += $(CXL_CORE_SRC)/memdev.o
+cxl_core-y += $(CXL_CORE_SRC)/mbox.o
+cxl_core-y += config_check.o
+
+cxl_core-y += mock_pmem.o
+
+obj-m += test/
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bug.h>
+
+void check(void)
+{
+ /*
+ * These kconfig symbols must be set to "m" for cxl_test to load
+ * and operate.
+ */
+ BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_BUS));
+ BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_ACPI));
+ BUILD_BUG_ON(!IS_MODULE(CONFIG_CXL_PMEM));
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
+
+#include <linux/platform_device.h>
+#include <linux/device.h>
+#include <linux/acpi.h>
+#include <linux/pci.h>
+#include <cxl.h>
+#include "test/mock.h"
+
+struct acpi_device *to_cxl_host_bridge(struct device *host, struct device *dev)
+{
+ int index;
+ struct acpi_device *adev, *found = NULL;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+
+ if (ops && ops->is_mock_bridge(dev)) {
+ found = ACPI_COMPANION(dev);
+ goto out;
+ }
+
+ if (dev->bus == &platform_bus_type)
+ goto out;
+
+ adev = to_acpi_device(dev);
+ if (!acpi_pci_find_root(adev->handle))
+ goto out;
+
+ if (strcmp(acpi_device_hid(adev), "ACPI0016") == 0) {
+ found = adev;
+ dev_dbg(host, "found host bridge %s\n", dev_name(&adev->dev));
+ }
+out:
+ put_cxl_mock_ops(index);
+ return found;
+}
+
+static int match_add_root_port(struct pci_dev *pdev, void *data)
+{
+ struct cxl_walk_context *ctx = data;
+ struct pci_bus *root_bus = ctx->root;
+ struct cxl_port *port = ctx->port;
+ int type = pci_pcie_type(pdev);
+ struct device *dev = ctx->dev;
+ u32 lnkcap, port_num;
+ int rc;
+
+ if (pdev->bus != root_bus)
+ return 0;
+ if (!pci_is_pcie(pdev))
+ return 0;
+ if (type != PCI_EXP_TYPE_ROOT_PORT)
+ return 0;
+ if (pci_read_config_dword(pdev, pci_pcie_cap(pdev) + PCI_EXP_LNKCAP,
+ &lnkcap) != PCIBIOS_SUCCESSFUL)
+ return 0;
+
+ /* TODO walk DVSEC to find component register base */
+ port_num = FIELD_GET(PCI_EXP_LNKCAP_PN, lnkcap);
+ rc = cxl_add_dport(port, &pdev->dev, port_num, CXL_RESOURCE_NONE);
+ if (rc) {
+ dev_err(dev, "failed to add dport: %s (%d)\n",
+ dev_name(&pdev->dev), rc);
+ ctx->error = rc;
+ return rc;
+ }
+ ctx->count++;
+
+ dev_dbg(dev, "add dport%d: %s\n", port_num, dev_name(&pdev->dev));
+
+ return 0;
+}
+
+static int mock_add_root_port(struct platform_device *pdev, void *data)
+{
+ struct cxl_walk_context *ctx = data;
+ struct cxl_port *port = ctx->port;
+ struct device *dev = ctx->dev;
+ int rc;
+
+ rc = cxl_add_dport(port, &pdev->dev, pdev->id, CXL_RESOURCE_NONE);
+ if (rc) {
+ dev_err(dev, "failed to add dport: %s (%d)\n",
+ dev_name(&pdev->dev), rc);
+ ctx->error = rc;
+ return rc;
+ }
+ ctx->count++;
+
+ dev_dbg(dev, "add dport%d: %s\n", pdev->id, dev_name(&pdev->dev));
+
+ return 0;
+}
+
+int match_add_root_ports(struct pci_dev *dev, void *data)
+{
+ int index, rc;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+ struct platform_device *pdev = (struct platform_device *) dev;
+
+ if (ops && ops->is_mock_port(pdev))
+ rc = mock_add_root_port(pdev, data);
+ else
+ rc = match_add_root_port(dev, data);
+
+ put_cxl_mock_ops(index);
+
+ return rc;
+}
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
+#include <cxl.h>
+#include "test/mock.h"
+#include <core/core.h>
+
+int match_nvdimm_bridge(struct device *dev, const void *data)
+{
+ int index, rc = 0;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+ const struct cxl_nvdimm *cxl_nvd = data;
+
+ if (ops) {
+ if (dev->type == &cxl_nvdimm_bridge_type &&
+ (ops->is_mock_dev(dev->parent->parent) ==
+ ops->is_mock_dev(cxl_nvd->dev.parent->parent)))
+ rc = 1;
+ } else
+ rc = dev->type == &cxl_nvdimm_bridge_type;
+
+ put_cxl_mock_ops(index);
+
+ return rc;
+}
--- /dev/null
+# SPDX-License-Identifier: GPL-2.0
+ccflags-y := -I$(srctree)/drivers/cxl/
+
+obj-m += cxl_test.o
+obj-m += cxl_mock.o
+obj-m += cxl_mock_mem.o
+
+cxl_test-y := cxl.o
+cxl_mock-y := mock.o
+cxl_mock_mem-y := mem.o
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2021 Intel Corporation. All rights reserved.
+
+#include <linux/platform_device.h>
+#include <linux/genalloc.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/acpi.h>
+#include <linux/pci.h>
+#include <linux/mm.h>
+#include "mock.h"
+
+#define NR_CXL_HOST_BRIDGES 4
+#define NR_CXL_ROOT_PORTS 2
+
+static struct platform_device *cxl_acpi;
+static struct platform_device *cxl_host_bridge[NR_CXL_HOST_BRIDGES];
+static struct platform_device
+ *cxl_root_port[NR_CXL_HOST_BRIDGES * NR_CXL_ROOT_PORTS];
+struct platform_device *cxl_mem[NR_CXL_HOST_BRIDGES * NR_CXL_ROOT_PORTS];
+
+static struct acpi_device acpi0017_mock;
+static struct acpi_device host_bridge[NR_CXL_HOST_BRIDGES] = {
+ [0] = {
+ .handle = &host_bridge[0],
+ },
+ [1] = {
+ .handle = &host_bridge[1],
+ },
+ [2] = {
+ .handle = &host_bridge[2],
+ },
+ [3] = {
+ .handle = &host_bridge[3],
+ },
+};
+
+static bool is_mock_dev(struct device *dev)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(cxl_mem); i++)
+ if (dev == &cxl_mem[i]->dev)
+ return true;
+ if (dev == &cxl_acpi->dev)
+ return true;
+ return false;
+}
+
+static bool is_mock_adev(struct acpi_device *adev)
+{
+ int i;
+
+ if (adev == &acpi0017_mock)
+ return true;
+
+ for (i = 0; i < ARRAY_SIZE(host_bridge); i++)
+ if (adev == &host_bridge[i])
+ return true;
+
+ return false;
+}
+
+static struct {
+ struct acpi_table_cedt cedt;
+ struct acpi_cedt_chbs chbs[NR_CXL_HOST_BRIDGES];
+ struct {
+ struct acpi_cedt_cfmws cfmws;
+ u32 target[1];
+ } cfmws0;
+ struct {
+ struct acpi_cedt_cfmws cfmws;
+ u32 target[4];
+ } cfmws1;
+ struct {
+ struct acpi_cedt_cfmws cfmws;
+ u32 target[1];
+ } cfmws2;
+ struct {
+ struct acpi_cedt_cfmws cfmws;
+ u32 target[4];
+ } cfmws3;
+} __packed mock_cedt = {
+ .cedt = {
+ .header = {
+ .signature = "CEDT",
+ .length = sizeof(mock_cedt),
+ .revision = 1,
+ },
+ },
+ .chbs[0] = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CHBS,
+ .length = sizeof(mock_cedt.chbs[0]),
+ },
+ .uid = 0,
+ .cxl_version = ACPI_CEDT_CHBS_VERSION_CXL20,
+ },
+ .chbs[1] = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CHBS,
+ .length = sizeof(mock_cedt.chbs[0]),
+ },
+ .uid = 1,
+ .cxl_version = ACPI_CEDT_CHBS_VERSION_CXL20,
+ },
+ .chbs[2] = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CHBS,
+ .length = sizeof(mock_cedt.chbs[0]),
+ },
+ .uid = 2,
+ .cxl_version = ACPI_CEDT_CHBS_VERSION_CXL20,
+ },
+ .chbs[3] = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CHBS,
+ .length = sizeof(mock_cedt.chbs[0]),
+ },
+ .uid = 3,
+ .cxl_version = ACPI_CEDT_CHBS_VERSION_CXL20,
+ },
+ .cfmws0 = {
+ .cfmws = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CFMWS,
+ .length = sizeof(mock_cedt.cfmws0),
+ },
+ .interleave_ways = 0,
+ .granularity = 4,
+ .restrictions = ACPI_CEDT_CFMWS_RESTRICT_TYPE3 |
+ ACPI_CEDT_CFMWS_RESTRICT_VOLATILE,
+ .qtg_id = 0,
+ .window_size = SZ_256M,
+ },
+ .target = { 0 },
+ },
+ .cfmws1 = {
+ .cfmws = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CFMWS,
+ .length = sizeof(mock_cedt.cfmws1),
+ },
+ .interleave_ways = 2,
+ .granularity = 4,
+ .restrictions = ACPI_CEDT_CFMWS_RESTRICT_TYPE3 |
+ ACPI_CEDT_CFMWS_RESTRICT_VOLATILE,
+ .qtg_id = 1,
+ .window_size = SZ_256M * 4,
+ },
+ .target = { 0, 1, 2, 3 },
+ },
+ .cfmws2 = {
+ .cfmws = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CFMWS,
+ .length = sizeof(mock_cedt.cfmws2),
+ },
+ .interleave_ways = 0,
+ .granularity = 4,
+ .restrictions = ACPI_CEDT_CFMWS_RESTRICT_TYPE3 |
+ ACPI_CEDT_CFMWS_RESTRICT_PMEM,
+ .qtg_id = 2,
+ .window_size = SZ_256M,
+ },
+ .target = { 0 },
+ },
+ .cfmws3 = {
+ .cfmws = {
+ .header = {
+ .type = ACPI_CEDT_TYPE_CFMWS,
+ .length = sizeof(mock_cedt.cfmws3),
+ },
+ .interleave_ways = 2,
+ .granularity = 4,
+ .restrictions = ACPI_CEDT_CFMWS_RESTRICT_TYPE3 |
+ ACPI_CEDT_CFMWS_RESTRICT_PMEM,
+ .qtg_id = 3,
+ .window_size = SZ_256M * 4,
+ },
+ .target = { 0, 1, 2, 3 },
+ },
+};
+
+struct cxl_mock_res {
+ struct list_head list;
+ struct range range;
+};
+
+static LIST_HEAD(mock_res);
+static DEFINE_MUTEX(mock_res_lock);
+static struct gen_pool *cxl_mock_pool;
+
+static void depopulate_all_mock_resources(void)
+{
+ struct cxl_mock_res *res, *_res;
+
+ mutex_lock(&mock_res_lock);
+ list_for_each_entry_safe(res, _res, &mock_res, list) {
+ gen_pool_free(cxl_mock_pool, res->range.start,
+ range_len(&res->range));
+ list_del(&res->list);
+ kfree(res);
+ }
+ mutex_unlock(&mock_res_lock);
+}
+
+static struct cxl_mock_res *alloc_mock_res(resource_size_t size)
+{
+ struct cxl_mock_res *res = kzalloc(sizeof(*res), GFP_KERNEL);
+ struct genpool_data_align data = {
+ .align = SZ_256M,
+ };
+ unsigned long phys;
+
+ INIT_LIST_HEAD(&res->list);
+ phys = gen_pool_alloc_algo(cxl_mock_pool, size,
+ gen_pool_first_fit_align, &data);
+ if (!phys)
+ return NULL;
+
+ res->range = (struct range) {
+ .start = phys,
+ .end = phys + size - 1,
+ };
+ mutex_lock(&mock_res_lock);
+ list_add(&res->list, &mock_res);
+ mutex_unlock(&mock_res_lock);
+
+ return res;
+}
+
+static int populate_cedt(void)
+{
+ struct acpi_cedt_cfmws *cfmws[4] = {
+ [0] = &mock_cedt.cfmws0.cfmws,
+ [1] = &mock_cedt.cfmws1.cfmws,
+ [2] = &mock_cedt.cfmws2.cfmws,
+ [3] = &mock_cedt.cfmws3.cfmws,
+ };
+ struct cxl_mock_res *res;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mock_cedt.chbs); i++) {
+ struct acpi_cedt_chbs *chbs = &mock_cedt.chbs[i];
+ resource_size_t size;
+
+ if (chbs->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL20)
+ size = ACPI_CEDT_CHBS_LENGTH_CXL20;
+ else
+ size = ACPI_CEDT_CHBS_LENGTH_CXL11;
+
+ res = alloc_mock_res(size);
+ if (!res)
+ return -ENOMEM;
+ chbs->base = res->range.start;
+ chbs->length = size;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(cfmws); i++) {
+ struct acpi_cedt_cfmws *window = cfmws[i];
+
+ res = alloc_mock_res(window->window_size);
+ if (!res)
+ return -ENOMEM;
+ window->base_hpa = res->range.start;
+ }
+
+ return 0;
+}
+
+static acpi_status mock_acpi_get_table(char *signature, u32 instance,
+ struct acpi_table_header **out_table)
+{
+ if (instance < U32_MAX || strcmp(signature, ACPI_SIG_CEDT) != 0)
+ return acpi_get_table(signature, instance, out_table);
+
+ *out_table = (struct acpi_table_header *) &mock_cedt;
+ return AE_OK;
+}
+
+static void mock_acpi_put_table(struct acpi_table_header *table)
+{
+ if (table == (struct acpi_table_header *) &mock_cedt)
+ return;
+ acpi_put_table(table);
+}
+
+static bool is_mock_bridge(struct device *dev)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(cxl_host_bridge); i++)
+ if (dev == &cxl_host_bridge[i]->dev)
+ return true;
+
+ return false;
+}
+
+static int host_bridge_index(struct acpi_device *adev)
+{
+ return adev - host_bridge;
+}
+
+static struct acpi_device *find_host_bridge(acpi_handle handle)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(host_bridge); i++)
+ if (handle == host_bridge[i].handle)
+ return &host_bridge[i];
+ return NULL;
+}
+
+static acpi_status
+mock_acpi_evaluate_integer(acpi_handle handle, acpi_string pathname,
+ struct acpi_object_list *arguments,
+ unsigned long long *data)
+{
+ struct acpi_device *adev = find_host_bridge(handle);
+
+ if (!adev || strcmp(pathname, METHOD_NAME__UID) != 0)
+ return acpi_evaluate_integer(handle, pathname, arguments, data);
+
+ *data = host_bridge_index(adev);
+ return AE_OK;
+}
+
+static struct pci_bus mock_pci_bus[NR_CXL_HOST_BRIDGES];
+static struct acpi_pci_root mock_pci_root[NR_CXL_HOST_BRIDGES] = {
+ [0] = {
+ .bus = &mock_pci_bus[0],
+ },
+ [1] = {
+ .bus = &mock_pci_bus[1],
+ },
+ [2] = {
+ .bus = &mock_pci_bus[2],
+ },
+ [3] = {
+ .bus = &mock_pci_bus[3],
+ },
+};
+
+static struct platform_device *mock_cxl_root_port(struct pci_bus *bus, int index)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mock_pci_bus); i++)
+ if (bus == &mock_pci_bus[i])
+ return cxl_root_port[index + i * NR_CXL_ROOT_PORTS];
+ return NULL;
+}
+
+static bool is_mock_port(struct platform_device *pdev)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(cxl_root_port); i++)
+ if (pdev == cxl_root_port[i])
+ return true;
+ return false;
+}
+
+static bool is_mock_bus(struct pci_bus *bus)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(mock_pci_bus); i++)
+ if (bus == &mock_pci_bus[i])
+ return true;
+ return false;
+}
+
+static struct acpi_pci_root *mock_acpi_pci_find_root(acpi_handle handle)
+{
+ struct acpi_device *adev = find_host_bridge(handle);
+
+ if (!adev)
+ return acpi_pci_find_root(handle);
+ return &mock_pci_root[host_bridge_index(adev)];
+}
+
+static struct cxl_mock_ops cxl_mock_ops = {
+ .is_mock_adev = is_mock_adev,
+ .is_mock_bridge = is_mock_bridge,
+ .is_mock_bus = is_mock_bus,
+ .is_mock_port = is_mock_port,
+ .is_mock_dev = is_mock_dev,
+ .mock_port = mock_cxl_root_port,
+ .acpi_get_table = mock_acpi_get_table,
+ .acpi_put_table = mock_acpi_put_table,
+ .acpi_evaluate_integer = mock_acpi_evaluate_integer,
+ .acpi_pci_find_root = mock_acpi_pci_find_root,
+ .list = LIST_HEAD_INIT(cxl_mock_ops.list),
+};
+
+static void mock_companion(struct acpi_device *adev, struct device *dev)
+{
+ device_initialize(&adev->dev);
+ fwnode_init(&adev->fwnode, NULL);
+ dev->fwnode = &adev->fwnode;
+ adev->fwnode.dev = dev;
+}
+
+#ifndef SZ_64G
+#define SZ_64G (SZ_32G * 2)
+#endif
+
+#ifndef SZ_512G
+#define SZ_512G (SZ_64G * 8)
+#endif
+
+static struct platform_device *alloc_memdev(int id)
+{
+ struct resource res[] = {
+ [0] = {
+ .flags = IORESOURCE_MEM,
+ },
+ [1] = {
+ .flags = IORESOURCE_MEM,
+ .desc = IORES_DESC_PERSISTENT_MEMORY,
+ },
+ };
+ struct platform_device *pdev;
+ int i, rc;
+
+ for (i = 0; i < ARRAY_SIZE(res); i++) {
+ struct cxl_mock_res *r = alloc_mock_res(SZ_256M);
+
+ if (!r)
+ return NULL;
+ res[i].start = r->range.start;
+ res[i].end = r->range.end;
+ }
+
+ pdev = platform_device_alloc("cxl_mem", id);
+ if (!pdev)
+ return NULL;
+
+ rc = platform_device_add_resources(pdev, res, ARRAY_SIZE(res));
+ if (rc)
+ goto err;
+
+ return pdev;
+
+err:
+ platform_device_put(pdev);
+ return NULL;
+}
+
+static __init int cxl_test_init(void)
+{
+ int rc, i;
+
+ register_cxl_mock_ops(&cxl_mock_ops);
+
+ cxl_mock_pool = gen_pool_create(ilog2(SZ_2M), NUMA_NO_NODE);
+ if (!cxl_mock_pool) {
+ rc = -ENOMEM;
+ goto err_gen_pool_create;
+ }
+
+ rc = gen_pool_add(cxl_mock_pool, SZ_512G, SZ_64G, NUMA_NO_NODE);
+ if (rc)
+ goto err_gen_pool_add;
+
+ rc = populate_cedt();
+ if (rc)
+ goto err_populate;
+
+ for (i = 0; i < ARRAY_SIZE(cxl_host_bridge); i++) {
+ struct acpi_device *adev = &host_bridge[i];
+ struct platform_device *pdev;
+
+ pdev = platform_device_alloc("cxl_host_bridge", i);
+ if (!pdev)
+ goto err_bridge;
+
+ mock_companion(adev, &pdev->dev);
+ rc = platform_device_add(pdev);
+ if (rc) {
+ platform_device_put(pdev);
+ goto err_bridge;
+ }
+ cxl_host_bridge[i] = pdev;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(cxl_root_port); i++) {
+ struct platform_device *bridge =
+ cxl_host_bridge[i / NR_CXL_ROOT_PORTS];
+ struct platform_device *pdev;
+
+ pdev = platform_device_alloc("cxl_root_port", i);
+ if (!pdev)
+ goto err_port;
+ pdev->dev.parent = &bridge->dev;
+
+ rc = platform_device_add(pdev);
+ if (rc) {
+ platform_device_put(pdev);
+ goto err_port;
+ }
+ cxl_root_port[i] = pdev;
+ }
+
+ BUILD_BUG_ON(ARRAY_SIZE(cxl_mem) != ARRAY_SIZE(cxl_root_port));
+ for (i = 0; i < ARRAY_SIZE(cxl_mem); i++) {
+ struct platform_device *port = cxl_root_port[i];
+ struct platform_device *pdev;
+
+ pdev = alloc_memdev(i);
+ if (!pdev)
+ goto err_mem;
+ pdev->dev.parent = &port->dev;
+
+ rc = platform_device_add(pdev);
+ if (rc) {
+ platform_device_put(pdev);
+ goto err_mem;
+ }
+ cxl_mem[i] = pdev;
+ }
+
+ cxl_acpi = platform_device_alloc("cxl_acpi", 0);
+ if (!cxl_acpi)
+ goto err_mem;
+
+ mock_companion(&acpi0017_mock, &cxl_acpi->dev);
+ acpi0017_mock.dev.bus = &platform_bus_type;
+
+ rc = platform_device_add(cxl_acpi);
+ if (rc)
+ goto err_add;
+
+ return 0;
+
+err_add:
+ platform_device_put(cxl_acpi);
+err_mem:
+ for (i = ARRAY_SIZE(cxl_mem) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_mem[i]);
+err_port:
+ for (i = ARRAY_SIZE(cxl_root_port) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_root_port[i]);
+err_bridge:
+ for (i = ARRAY_SIZE(cxl_host_bridge) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_host_bridge[i]);
+err_populate:
+ depopulate_all_mock_resources();
+err_gen_pool_add:
+ gen_pool_destroy(cxl_mock_pool);
+err_gen_pool_create:
+ unregister_cxl_mock_ops(&cxl_mock_ops);
+ return rc;
+}
+
+static __exit void cxl_test_exit(void)
+{
+ int i;
+
+ platform_device_unregister(cxl_acpi);
+ for (i = ARRAY_SIZE(cxl_mem) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_mem[i]);
+ for (i = ARRAY_SIZE(cxl_root_port) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_root_port[i]);
+ for (i = ARRAY_SIZE(cxl_host_bridge) - 1; i >= 0; i--)
+ platform_device_unregister(cxl_host_bridge[i]);
+ depopulate_all_mock_resources();
+ gen_pool_destroy(cxl_mock_pool);
+ unregister_cxl_mock_ops(&cxl_mock_ops);
+}
+
+module_init(cxl_test_init);
+module_exit(cxl_test_exit);
+MODULE_LICENSE("GPL v2");
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2021 Intel Corporation. All rights reserved.
+
+#include <linux/platform_device.h>
+#include <linux/mod_devicetable.h>
+#include <linux/module.h>
+#include <linux/sizes.h>
+#include <linux/bits.h>
+#include <cxlmem.h>
+
+#define LSA_SIZE SZ_128K
+#define EFFECT(x) (1U << x)
+
+static struct cxl_cel_entry mock_cel[] = {
+ {
+ .opcode = cpu_to_le16(CXL_MBOX_OP_GET_SUPPORTED_LOGS),
+ .effect = cpu_to_le16(0),
+ },
+ {
+ .opcode = cpu_to_le16(CXL_MBOX_OP_IDENTIFY),
+ .effect = cpu_to_le16(0),
+ },
+ {
+ .opcode = cpu_to_le16(CXL_MBOX_OP_GET_LSA),
+ .effect = cpu_to_le16(0),
+ },
+ {
+ .opcode = cpu_to_le16(CXL_MBOX_OP_SET_LSA),
+ .effect = cpu_to_le16(EFFECT(1) | EFFECT(2)),
+ },
+};
+
+static struct {
+ struct cxl_mbox_get_supported_logs gsl;
+ struct cxl_gsl_entry entry;
+} mock_gsl_payload = {
+ .gsl = {
+ .entries = cpu_to_le16(1),
+ },
+ .entry = {
+ .uuid = DEFINE_CXL_CEL_UUID,
+ .size = cpu_to_le32(sizeof(mock_cel)),
+ },
+};
+
+static int mock_gsl(struct cxl_mbox_cmd *cmd)
+{
+ if (cmd->size_out < sizeof(mock_gsl_payload))
+ return -EINVAL;
+
+ memcpy(cmd->payload_out, &mock_gsl_payload, sizeof(mock_gsl_payload));
+ cmd->size_out = sizeof(mock_gsl_payload);
+
+ return 0;
+}
+
+static int mock_get_log(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
+{
+ struct cxl_mbox_get_log *gl = cmd->payload_in;
+ u32 offset = le32_to_cpu(gl->offset);
+ u32 length = le32_to_cpu(gl->length);
+ uuid_t uuid = DEFINE_CXL_CEL_UUID;
+ void *data = &mock_cel;
+
+ if (cmd->size_in < sizeof(*gl))
+ return -EINVAL;
+ if (length > cxlm->payload_size)
+ return -EINVAL;
+ if (offset + length > sizeof(mock_cel))
+ return -EINVAL;
+ if (!uuid_equal(&gl->uuid, &uuid))
+ return -EINVAL;
+ if (length > cmd->size_out)
+ return -EINVAL;
+
+ memcpy(cmd->payload_out, data + offset, length);
+
+ return 0;
+}
+
+static int mock_id(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
+{
+ struct platform_device *pdev = to_platform_device(cxlm->dev);
+ struct cxl_mbox_identify id = {
+ .fw_revision = { "mock fw v1 " },
+ .lsa_size = cpu_to_le32(LSA_SIZE),
+ /* FIXME: Add partition support */
+ .partition_align = cpu_to_le64(0),
+ };
+ u64 capacity = 0;
+ int i;
+
+ if (cmd->size_out < sizeof(id))
+ return -EINVAL;
+
+ for (i = 0; i < 2; i++) {
+ struct resource *res;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+ if (!res)
+ break;
+
+ capacity += resource_size(res) / CXL_CAPACITY_MULTIPLIER;
+
+ if (le64_to_cpu(id.partition_align))
+ continue;
+
+ if (res->desc == IORES_DESC_PERSISTENT_MEMORY)
+ id.persistent_capacity = cpu_to_le64(
+ resource_size(res) / CXL_CAPACITY_MULTIPLIER);
+ else
+ id.volatile_capacity = cpu_to_le64(
+ resource_size(res) / CXL_CAPACITY_MULTIPLIER);
+ }
+
+ id.total_capacity = cpu_to_le64(capacity);
+
+ memcpy(cmd->payload_out, &id, sizeof(id));
+
+ return 0;
+}
+
+static int mock_get_lsa(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
+{
+ struct cxl_mbox_get_lsa *get_lsa = cmd->payload_in;
+ void *lsa = dev_get_drvdata(cxlm->dev);
+ u32 offset, length;
+
+ if (sizeof(*get_lsa) > cmd->size_in)
+ return -EINVAL;
+ offset = le32_to_cpu(get_lsa->offset);
+ length = le32_to_cpu(get_lsa->length);
+ if (offset + length > LSA_SIZE)
+ return -EINVAL;
+ if (length > cmd->size_out)
+ return -EINVAL;
+
+ memcpy(cmd->payload_out, lsa + offset, length);
+ return 0;
+}
+
+static int mock_set_lsa(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
+{
+ struct cxl_mbox_set_lsa *set_lsa = cmd->payload_in;
+ void *lsa = dev_get_drvdata(cxlm->dev);
+ u32 offset, length;
+
+ if (sizeof(*set_lsa) > cmd->size_in)
+ return -EINVAL;
+ offset = le32_to_cpu(set_lsa->offset);
+ length = cmd->size_in - sizeof(*set_lsa);
+ if (offset + length > LSA_SIZE)
+ return -EINVAL;
+
+ memcpy(lsa + offset, &set_lsa->data[0], length);
+ return 0;
+}
+
+static int cxl_mock_mbox_send(struct cxl_mem *cxlm, struct cxl_mbox_cmd *cmd)
+{
+ struct device *dev = cxlm->dev;
+ int rc = -EIO;
+
+ switch (cmd->opcode) {
+ case CXL_MBOX_OP_GET_SUPPORTED_LOGS:
+ rc = mock_gsl(cmd);
+ break;
+ case CXL_MBOX_OP_GET_LOG:
+ rc = mock_get_log(cxlm, cmd);
+ break;
+ case CXL_MBOX_OP_IDENTIFY:
+ rc = mock_id(cxlm, cmd);
+ break;
+ case CXL_MBOX_OP_GET_LSA:
+ rc = mock_get_lsa(cxlm, cmd);
+ break;
+ case CXL_MBOX_OP_SET_LSA:
+ rc = mock_set_lsa(cxlm, cmd);
+ break;
+ default:
+ break;
+ }
+
+ dev_dbg(dev, "opcode: %#x sz_in: %zd sz_out: %zd rc: %d\n", cmd->opcode,
+ cmd->size_in, cmd->size_out, rc);
+
+ return rc;
+}
+
+static void label_area_release(void *lsa)
+{
+ vfree(lsa);
+}
+
+static int cxl_mock_mem_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct cxl_memdev *cxlmd;
+ struct cxl_mem *cxlm;
+ void *lsa;
+ int rc;
+
+ lsa = vmalloc(LSA_SIZE);
+ if (!lsa)
+ return -ENOMEM;
+ rc = devm_add_action_or_reset(dev, label_area_release, lsa);
+ if (rc)
+ return rc;
+ dev_set_drvdata(dev, lsa);
+
+ cxlm = cxl_mem_create(dev);
+ if (IS_ERR(cxlm))
+ return PTR_ERR(cxlm);
+
+ cxlm->mbox_send = cxl_mock_mbox_send;
+ cxlm->payload_size = SZ_4K;
+
+ rc = cxl_mem_enumerate_cmds(cxlm);
+ if (rc)
+ return rc;
+
+ rc = cxl_mem_identify(cxlm);
+ if (rc)
+ return rc;
+
+ rc = cxl_mem_create_range_info(cxlm);
+ if (rc)
+ return rc;
+
+ cxlmd = devm_cxl_add_memdev(cxlm);
+ if (IS_ERR(cxlmd))
+ return PTR_ERR(cxlmd);
+
+ if (range_len(&cxlm->pmem_range) && IS_ENABLED(CONFIG_CXL_PMEM))
+ rc = devm_cxl_add_nvdimm(dev, cxlmd);
+
+ return 0;
+}
+
+static const struct platform_device_id cxl_mock_mem_ids[] = {
+ { .name = "cxl_mem", },
+ { },
+};
+MODULE_DEVICE_TABLE(platform, cxl_mock_mem_ids);
+
+static struct platform_driver cxl_mock_mem_driver = {
+ .probe = cxl_mock_mem_probe,
+ .id_table = cxl_mock_mem_ids,
+ .driver = {
+ .name = KBUILD_MODNAME,
+ },
+};
+
+module_platform_driver(cxl_mock_mem_driver);
+MODULE_LICENSE("GPL v2");
+MODULE_IMPORT_NS(CXL);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+//Copyright(c) 2021 Intel Corporation. All rights reserved.
+
+#include <linux/libnvdimm.h>
+#include <linux/rculist.h>
+#include <linux/device.h>
+#include <linux/export.h>
+#include <linux/acpi.h>
+#include <linux/pci.h>
+#include "mock.h"
+
+static LIST_HEAD(mock);
+
+void register_cxl_mock_ops(struct cxl_mock_ops *ops)
+{
+ list_add_rcu(&ops->list, &mock);
+}
+EXPORT_SYMBOL_GPL(register_cxl_mock_ops);
+
+static DEFINE_SRCU(cxl_mock_srcu);
+
+void unregister_cxl_mock_ops(struct cxl_mock_ops *ops)
+{
+ list_del_rcu(&ops->list);
+ synchronize_srcu(&cxl_mock_srcu);
+}
+EXPORT_SYMBOL_GPL(unregister_cxl_mock_ops);
+
+struct cxl_mock_ops *get_cxl_mock_ops(int *index)
+{
+ *index = srcu_read_lock(&cxl_mock_srcu);
+ return list_first_or_null_rcu(&mock, struct cxl_mock_ops, list);
+}
+EXPORT_SYMBOL_GPL(get_cxl_mock_ops);
+
+void put_cxl_mock_ops(int index)
+{
+ srcu_read_unlock(&cxl_mock_srcu, index);
+}
+EXPORT_SYMBOL_GPL(put_cxl_mock_ops);
+
+bool __wrap_is_acpi_device_node(const struct fwnode_handle *fwnode)
+{
+ struct acpi_device *adev =
+ container_of(fwnode, struct acpi_device, fwnode);
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+ bool retval = false;
+
+ if (ops)
+ retval = ops->is_mock_adev(adev);
+
+ if (!retval)
+ retval = is_acpi_device_node(fwnode);
+
+ put_cxl_mock_ops(index);
+ return retval;
+}
+EXPORT_SYMBOL(__wrap_is_acpi_device_node);
+
+acpi_status __wrap_acpi_get_table(char *signature, u32 instance,
+ struct acpi_table_header **out_table)
+{
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+ acpi_status status;
+
+ if (ops)
+ status = ops->acpi_get_table(signature, instance, out_table);
+ else
+ status = acpi_get_table(signature, instance, out_table);
+
+ put_cxl_mock_ops(index);
+
+ return status;
+}
+EXPORT_SYMBOL(__wrap_acpi_get_table);
+
+void __wrap_acpi_put_table(struct acpi_table_header *table)
+{
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+
+ if (ops)
+ ops->acpi_put_table(table);
+ else
+ acpi_put_table(table);
+ put_cxl_mock_ops(index);
+}
+EXPORT_SYMBOL(__wrap_acpi_put_table);
+
+acpi_status __wrap_acpi_evaluate_integer(acpi_handle handle,
+ acpi_string pathname,
+ struct acpi_object_list *arguments,
+ unsigned long long *data)
+{
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+ acpi_status status;
+
+ if (ops)
+ status = ops->acpi_evaluate_integer(handle, pathname, arguments,
+ data);
+ else
+ status = acpi_evaluate_integer(handle, pathname, arguments,
+ data);
+ put_cxl_mock_ops(index);
+
+ return status;
+}
+EXPORT_SYMBOL(__wrap_acpi_evaluate_integer);
+
+struct acpi_pci_root *__wrap_acpi_pci_find_root(acpi_handle handle)
+{
+ int index;
+ struct acpi_pci_root *root;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+
+ if (ops)
+ root = ops->acpi_pci_find_root(handle);
+ else
+ root = acpi_pci_find_root(handle);
+
+ put_cxl_mock_ops(index);
+
+ return root;
+}
+EXPORT_SYMBOL_GPL(__wrap_acpi_pci_find_root);
+
+void __wrap_pci_walk_bus(struct pci_bus *bus,
+ int (*cb)(struct pci_dev *, void *), void *userdata)
+{
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+
+ if (ops && ops->is_mock_bus(bus)) {
+ int rc, i;
+
+ /*
+ * Simulate 2 root ports per host-bridge and no
+ * depth recursion.
+ */
+ for (i = 0; i < 2; i++) {
+ rc = cb((struct pci_dev *) ops->mock_port(bus, i),
+ userdata);
+ if (rc)
+ break;
+ }
+ } else
+ pci_walk_bus(bus, cb, userdata);
+
+ put_cxl_mock_ops(index);
+}
+EXPORT_SYMBOL_GPL(__wrap_pci_walk_bus);
+
+struct nvdimm_bus *
+__wrap_nvdimm_bus_register(struct device *dev,
+ struct nvdimm_bus_descriptor *nd_desc)
+{
+ int index;
+ struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
+
+ if (ops && ops->is_mock_dev(dev->parent->parent))
+ nd_desc->provider_name = "cxl_test";
+ put_cxl_mock_ops(index);
+
+ return nvdimm_bus_register(dev, nd_desc);
+}
+EXPORT_SYMBOL_GPL(__wrap_nvdimm_bus_register);
+
+MODULE_LICENSE("GPL v2");
--- /dev/null
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#include <linux/list.h>
+#include <linux/acpi.h>
+
+struct cxl_mock_ops {
+ struct list_head list;
+ bool (*is_mock_adev)(struct acpi_device *dev);
+ acpi_status (*acpi_get_table)(char *signature, u32 instance,
+ struct acpi_table_header **out_table);
+ void (*acpi_put_table)(struct acpi_table_header *table);
+ bool (*is_mock_bridge)(struct device *dev);
+ acpi_status (*acpi_evaluate_integer)(acpi_handle handle,
+ acpi_string pathname,
+ struct acpi_object_list *arguments,
+ unsigned long long *data);
+ struct acpi_pci_root *(*acpi_pci_find_root)(acpi_handle handle);
+ struct platform_device *(*mock_port)(struct pci_bus *bus, int index);
+ bool (*is_mock_bus)(struct pci_bus *bus);
+ bool (*is_mock_port)(struct platform_device *pdev);
+ bool (*is_mock_dev)(struct device *dev);
+};
+
+void register_cxl_mock_ops(struct cxl_mock_ops *ops);
+void unregister_cxl_mock_ops(struct cxl_mock_ops *ops);
+struct cxl_mock_ops *get_cxl_mock_ops(int *index);
+void put_cxl_mock_ops(int index);
$(OUTPUT)/runqslower: $(BPFOBJ) | $(DEFAULT_BPFTOOL) $(RUNQSLOWER_OUTPUT)
$(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower \
OUTPUT=$(RUNQSLOWER_OUTPUT) VMLINUX_BTF=$(VMLINUX_BTF) \
- BPFTOOL_OUTPUT=$(BUILD_DIR)/bpftool/ \
+ BPFTOOL_OUTPUT=$(HOST_BUILD_DIR)/bpftool/ \
BPFOBJ_OUTPUT=$(BUILD_DIR)/libbpf \
BPFOBJ=$(BPFOBJ) BPF_INCLUDE=$(INCLUDE_DIR) && \
cp $(RUNQSLOWER_OUTPUT)runqslower $@
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include "test_helper_restricted.skel.h"
+
+void test_helper_restricted(void)
+{
+ int prog_i = 0, prog_cnt;
+ int duration = 0;
+
+ do {
+ struct test_helper_restricted *test;
+ int maybeOK;
+
+ test = test_helper_restricted__open();
+ if (!ASSERT_OK_PTR(test, "open"))
+ return;
+
+ prog_cnt = test->skeleton->prog_cnt;
+
+ for (int j = 0; j < prog_cnt; ++j) {
+ struct bpf_program *prog = *test->skeleton->progs[j].prog;
+
+ maybeOK = bpf_program__set_autoload(prog, prog_i == j);
+ ASSERT_OK(maybeOK, "set autoload");
+ }
+
+ maybeOK = test_helper_restricted__load(test);
+ CHECK(!maybeOK, test->skeleton->progs[prog_i].name, "helper isn't restricted");
+
+ test_helper_restricted__destroy(test);
+ } while (++prog_i < prog_cnt);
+}
#define CG_NAME "/netcnt"
-void test_netcnt(void)
+void serial_test_netcnt(void)
{
union percpu_net_cnt *percpu_netcnt = NULL;
struct bpf_cgroup_storage_key key;
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2020 Facebook */
#define _GNU_SOURCE
+#include <stdio.h>
#include <sched.h>
#include <sys/mount.h>
#include <sys/stat.h>
static int fn(void)
{
- int err, duration = 0;
+ struct stat a, b, c;
+ int err, map;
err = unshare(CLONE_NEWNS);
- if (CHECK(err, "unshare", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "unshare"))
goto out;
err = mount("", "/", "", MS_REC | MS_PRIVATE, NULL);
- if (CHECK(err, "mount /", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mount /"))
goto out;
err = umount(TDIR);
- if (CHECK(err, "umount " TDIR, "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "umount " TDIR))
goto out;
err = mount("none", TDIR, "tmpfs", 0, NULL);
- if (CHECK(err, "mount", "mount root failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mount tmpfs"))
goto out;
err = mkdir(TDIR "/fs1", 0777);
- if (CHECK(err, "mkdir "TDIR"/fs1", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mkdir " TDIR "/fs1"))
goto out;
err = mkdir(TDIR "/fs2", 0777);
- if (CHECK(err, "mkdir "TDIR"/fs2", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mkdir " TDIR "/fs2"))
goto out;
err = mount("bpf", TDIR "/fs1", "bpf", 0, NULL);
- if (CHECK(err, "mount bpffs "TDIR"/fs1", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mount bpffs " TDIR "/fs1"))
goto out;
err = mount("bpf", TDIR "/fs2", "bpf", 0, NULL);
- if (CHECK(err, "mount bpffs " TDIR "/fs2", "failed: %d\n", errno))
+ if (!ASSERT_OK(err, "mount bpffs " TDIR "/fs2"))
goto out;
err = read_iter(TDIR "/fs1/maps.debug");
- if (CHECK(err, "reading " TDIR "/fs1/maps.debug", "failed\n"))
+ if (!ASSERT_OK(err, "reading " TDIR "/fs1/maps.debug"))
goto out;
err = read_iter(TDIR "/fs2/progs.debug");
- if (CHECK(err, "reading " TDIR "/fs2/progs.debug", "failed\n"))
+ if (!ASSERT_OK(err, "reading " TDIR "/fs2/progs.debug"))
goto out;
+
+ err = mkdir(TDIR "/fs1/a", 0777);
+ if (!ASSERT_OK(err, "creating " TDIR "/fs1/a"))
+ goto out;
+ err = mkdir(TDIR "/fs1/a/1", 0777);
+ if (!ASSERT_OK(err, "creating " TDIR "/fs1/a/1"))
+ goto out;
+ err = mkdir(TDIR "/fs1/b", 0777);
+ if (!ASSERT_OK(err, "creating " TDIR "/fs1/b"))
+ goto out;
+
+ map = bpf_create_map(BPF_MAP_TYPE_ARRAY, 4, 4, 1, 0);
+ if (!ASSERT_GT(map, 0, "create_map(ARRAY)"))
+ goto out;
+ err = bpf_obj_pin(map, TDIR "/fs1/c");
+ if (!ASSERT_OK(err, "pin map"))
+ goto out;
+ close(map);
+
+ /* Check that RENAME_EXCHANGE works for directories. */
+ err = stat(TDIR "/fs1/a", &a);
+ if (!ASSERT_OK(err, "stat(" TDIR "/fs1/a)"))
+ goto out;
+ err = renameat2(0, TDIR "/fs1/a", 0, TDIR "/fs1/b", RENAME_EXCHANGE);
+ if (!ASSERT_OK(err, "renameat2(/fs1/a, /fs1/b, RENAME_EXCHANGE)"))
+ goto out;
+ err = stat(TDIR "/fs1/b", &b);
+ if (!ASSERT_OK(err, "stat(" TDIR "/fs1/b)"))
+ goto out;
+ if (!ASSERT_EQ(a.st_ino, b.st_ino, "b should have a's inode"))
+ goto out;
+ err = access(TDIR "/fs1/b/1", F_OK);
+ if (!ASSERT_OK(err, "access(" TDIR "/fs1/b/1)"))
+ goto out;
+
+ /* Check that RENAME_EXCHANGE works for mixed file types. */
+ err = stat(TDIR "/fs1/c", &c);
+ if (!ASSERT_OK(err, "stat(" TDIR "/fs1/map)"))
+ goto out;
+ err = renameat2(0, TDIR "/fs1/c", 0, TDIR "/fs1/b", RENAME_EXCHANGE);
+ if (!ASSERT_OK(err, "renameat2(/fs1/c, /fs1/b, RENAME_EXCHANGE)"))
+ goto out;
+ err = stat(TDIR "/fs1/b", &b);
+ if (!ASSERT_OK(err, "stat(" TDIR "/fs1/b)"))
+ goto out;
+ if (!ASSERT_EQ(c.st_ino, b.st_ino, "b should have c's inode"))
+ goto out;
+ err = access(TDIR "/fs1/c/1", F_OK);
+ if (!ASSERT_OK(err, "access(" TDIR "/fs1/c/1)"))
+ goto out;
+
+ /* Check that RENAME_NOREPLACE works. */
+ err = renameat2(0, TDIR "/fs1/b", 0, TDIR "/fs1/a", RENAME_NOREPLACE);
+ if (!ASSERT_ERR(err, "renameat2(RENAME_NOREPLACE)")) {
+ err = -EINVAL;
+ goto out;
+ }
+ err = access(TDIR "/fs1/b", F_OK);
+ if (!ASSERT_OK(err, "access(" TDIR "/fs1/b)"))
+ goto out;
+
out:
umount(TDIR "/fs1");
umount(TDIR "/fs2");
int output;
};
+const volatile int bypass_unused = 1;
+
+static __u64
+unused_subprog(struct bpf_map *map, __u32 *key, __u64 *val,
+ struct callback_ctx *data)
+{
+ data->output = 0;
+ return 1;
+}
+
static __u64
check_array_elem(struct bpf_map *map, __u32 *key, __u64 *val,
struct callback_ctx *data)
data.output = 0;
bpf_for_each_map_elem(&arraymap, check_array_elem, &data, 0);
+ if (!bypass_unused)
+ bpf_for_each_map_elem(&arraymap, unused_subprog, &data, 0);
arraymap_output = data.output;
bpf_for_each_map_elem(&percpu_map, check_percpu_elem, (void *)0, 0);
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include <time.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+struct timer {
+ struct bpf_timer t;
+};
+
+struct lock {
+ struct bpf_spin_lock l;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, struct timer);
+} timers SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, __u32);
+ __type(value, struct lock);
+} locks SEC(".maps");
+
+static int timer_cb(void *map, int *key, struct timer *timer)
+{
+ return 0;
+}
+
+static void timer_work(void)
+{
+ struct timer *timer;
+ const int key = 0;
+
+ timer = bpf_map_lookup_elem(&timers, &key);
+ if (timer) {
+ bpf_timer_init(&timer->t, &timers, CLOCK_MONOTONIC);
+ bpf_timer_set_callback(&timer->t, timer_cb);
+ bpf_timer_start(&timer->t, 10E9, 0);
+ bpf_timer_cancel(&timer->t);
+ }
+}
+
+static void spin_lock_work(void)
+{
+ const int key = 0;
+ struct lock *lock;
+
+ lock = bpf_map_lookup_elem(&locks, &key);
+ if (lock) {
+ bpf_spin_lock(&lock->l);
+ bpf_spin_unlock(&lock->l);
+ }
+}
+
+SEC("raw_tp/sys_enter")
+int raw_tp_timer(void *ctx)
+{
+ timer_work();
+
+ return 0;
+}
+
+SEC("tp/syscalls/sys_enter_nanosleep")
+int tp_timer(void *ctx)
+{
+ timer_work();
+
+ return 0;
+}
+
+SEC("kprobe/sys_nanosleep")
+int kprobe_timer(void *ctx)
+{
+ timer_work();
+
+ return 0;
+}
+
+SEC("perf_event")
+int perf_event_timer(void *ctx)
+{
+ timer_work();
+
+ return 0;
+}
+
+SEC("raw_tp/sys_enter")
+int raw_tp_spin_lock(void *ctx)
+{
+ spin_lock_work();
+
+ return 0;
+}
+
+SEC("tp/syscalls/sys_enter_nanosleep")
+int tp_spin_lock(void *ctx)
+{
+ spin_lock_work();
+
+ return 0;
+}
+
+SEC("kprobe/sys_nanosleep")
+int kprobe_spin_lock(void *ctx)
+{
+ spin_lock_work();
+
+ return 0;
+}
+
+SEC("perf_event")
+int perf_event_spin_lock(void *ctx)
+{
+ spin_lock_work();
+
+ return 0;
+}
+
+const char LICENSE[] SEC("license") = "GPL";
int fixup_map_event_output[MAX_FIXUPS];
int fixup_map_reuseport_array[MAX_FIXUPS];
int fixup_map_ringbuf[MAX_FIXUPS];
+ int fixup_map_timer[MAX_FIXUPS];
/* Expected verifier log output for result REJECT or VERBOSE_ACCEPT.
* Can be a tab-separated sequence of expected strings. An empty string
* means no log verification.
* int cnt;
* struct bpf_spin_lock l;
* };
+ * struct bpf_timer {
+ * __u64 :64;
+ * __u64 :64;
+ * } __attribute__((aligned(8)));
+ * struct timer {
+ * struct bpf_timer t;
+ * };
*/
-static const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l";
+static const char btf_str_sec[] = "\0bpf_spin_lock\0val\0cnt\0l\0bpf_timer\0timer\0t";
static __u32 btf_raw_types[] = {
/* int */
BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [1] */
BTF_TYPE_ENC(15, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 2), 8),
BTF_MEMBER_ENC(19, 1, 0), /* int cnt; */
BTF_MEMBER_ENC(23, 2, 32),/* struct bpf_spin_lock l; */
+ /* struct bpf_timer */ /* [4] */
+ BTF_TYPE_ENC(25, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 0), 16),
+ /* struct timer */ /* [5] */
+ BTF_TYPE_ENC(35, BTF_INFO_ENC(BTF_KIND_STRUCT, 0, 1), 16),
+ BTF_MEMBER_ENC(41, 4, 0), /* struct bpf_timer t; */
};
static int load_btf(void)
return fd;
}
+static int create_map_timer(void)
+{
+ struct bpf_create_map_attr attr = {
+ .name = "test_map",
+ .map_type = BPF_MAP_TYPE_ARRAY,
+ .key_size = 4,
+ .value_size = 16,
+ .max_entries = 1,
+ .btf_key_type_id = 1,
+ .btf_value_type_id = 5,
+ };
+ int fd, btf_fd;
+
+ btf_fd = load_btf();
+ if (btf_fd < 0)
+ return -1;
+ attr.btf_fd = btf_fd;
+ fd = bpf_create_map_xattr(&attr);
+ if (fd < 0)
+ printf("Failed to create map with timer\n");
+ return fd;
+}
+
static char bpf_vlog[UINT_MAX >> 8];
static void do_test_fixup(struct bpf_test *test, enum bpf_prog_type prog_type,
int *fixup_map_event_output = test->fixup_map_event_output;
int *fixup_map_reuseport_array = test->fixup_map_reuseport_array;
int *fixup_map_ringbuf = test->fixup_map_ringbuf;
+ int *fixup_map_timer = test->fixup_map_timer;
if (test->fill_helper) {
test->fill_insns = calloc(MAX_TEST_INSNS, sizeof(struct bpf_insn));
fixup_map_ringbuf++;
} while (*fixup_map_ringbuf);
}
+ if (*fixup_map_timer) {
+ map_fds[21] = create_map_timer();
+ do {
+ prog[*fixup_map_timer].imm = map_fds[21];
+ fixup_map_timer++;
+ } while (*fixup_map_timer);
+ }
}
struct libcap {
# SPDX-License-Identifier: GPL-2.0
#
# Test topology:
-# - - - - - - - - - - - - - - - - - - - - - - - - -
-# | veth1 veth2 veth3 | ... init net
+# - - - - - - - - - - - - - - - - - - -
+# | veth1 veth2 veth3 | ns0
# - -| - - - - - - | - - - - - - | - -
# --------- --------- ---------
-# | veth0 | | veth0 | | veth0 | ...
+# | veth0 | | veth0 | | veth0 |
# --------- --------- ---------
# ns1 ns2 ns3
#
DRV_MODE="xdpgeneric xdpdrv xdpegress"
PASS=0
FAIL=0
+LOG_DIR=$(mktemp -d)
test_pass()
{
ip link del veth$i 2> /dev/null
ip netns del ns$i 2> /dev/null
done
+ ip netns del ns0 2> /dev/null
}
# Kselftest framework requirement - SKIP code is 4.
mode="xdpdrv"
fi
+ ip netns add ns0
for i in $(seq $NUM); do
ip netns add ns$i
- ip link add veth$i type veth peer name veth0 netns ns$i
- ip link set veth$i up
+ ip -n ns$i link add veth0 index 2 type veth \
+ peer name veth$i netns ns0 index $((1 + $i))
+ ip -n ns0 link set veth$i up
ip -n ns$i link set veth0 up
ip -n ns$i addr add 192.0.2.$i/24 dev veth0
xdp_dummy.o sec xdp &> /dev/null || \
{ test_fail "Unable to load dummy xdp" && exit 1; }
IFACES="$IFACES veth$i"
- veth_mac[$i]=$(ip link show veth$i | awk '/link\/ether/ {print $2}')
+ veth_mac[$i]=$(ip -n ns0 link show veth$i | awk '/link\/ether/ {print $2}')
done
}
local mode=$1
# mac test
- ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-2_${mode}.log &
- ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> mac_ns1-3_${mode}.log &
+ ip netns exec ns2 tcpdump -e -i veth0 -nn -l -e &> ${LOG_DIR}/mac_ns1-2_${mode}.log &
+ ip netns exec ns3 tcpdump -e -i veth0 -nn -l -e &> ${LOG_DIR}/mac_ns1-3_${mode}.log &
sleep 0.5
ip netns exec ns1 ping 192.0.2.254 -i 0.1 -c 4 &> /dev/null
sleep 0.5
- pkill -9 tcpdump
+ pkill tcpdump
# mac check
- grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" mac_ns1-2_${mode}.log && \
+ grep -q "${veth_mac[2]} > ff:ff:ff:ff:ff:ff" ${LOG_DIR}/mac_ns1-2_${mode}.log && \
test_pass "$mode mac ns1-2" || test_fail "$mode mac ns1-2"
- grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" mac_ns1-3_${mode}.log && \
+ grep -q "${veth_mac[3]} > ff:ff:ff:ff:ff:ff" ${LOG_DIR}/mac_ns1-3_${mode}.log && \
test_pass "$mode mac ns1-3" || test_fail "$mode mac ns1-3"
}
# ping6 test: echo request should be redirect back to itself, not others
ip netns exec ns1 ip neigh add 2001:db8::2 dev veth0 lladdr 00:00:00:00:00:02
- ip netns exec ns1 tcpdump -i veth0 -nn -l -e &> ns1-1_${mode}.log &
- ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> ns1-2_${mode}.log &
- ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> ns1-3_${mode}.log &
+ ip netns exec ns1 tcpdump -i veth0 -nn -l -e &> ${LOG_DIR}/ns1-1_${mode}.log &
+ ip netns exec ns2 tcpdump -i veth0 -nn -l -e &> ${LOG_DIR}/ns1-2_${mode}.log &
+ ip netns exec ns3 tcpdump -i veth0 -nn -l -e &> ${LOG_DIR}/ns1-3_${mode}.log &
sleep 0.5
# ARP test
- ip netns exec ns1 ping 192.0.2.254 -i 0.1 -c 4 &> /dev/null
+ ip netns exec ns1 arping -q -c 2 -I veth0 192.0.2.254
# IPv4 test
ip netns exec ns1 ping 192.0.2.253 -i 0.1 -c 4 &> /dev/null
# IPv6 test
ip netns exec ns1 ping6 2001:db8::2 -i 0.1 -c 2 &> /dev/null
sleep 0.5
- pkill -9 tcpdump
+ pkill tcpdump
# All netns should receive the redirect arp requests
- [ $(grep -c "who-has 192.0.2.254" ns1-1_${mode}.log) -gt 4 ] && \
+ [ $(grep -cF "who-has 192.0.2.254" ${LOG_DIR}/ns1-1_${mode}.log) -eq 4 ] && \
test_pass "$mode arp(F_BROADCAST) ns1-1" || \
test_fail "$mode arp(F_BROADCAST) ns1-1"
- [ $(grep -c "who-has 192.0.2.254" ns1-2_${mode}.log) -le 4 ] && \
+ [ $(grep -cF "who-has 192.0.2.254" ${LOG_DIR}/ns1-2_${mode}.log) -eq 2 ] && \
test_pass "$mode arp(F_BROADCAST) ns1-2" || \
test_fail "$mode arp(F_BROADCAST) ns1-2"
- [ $(grep -c "who-has 192.0.2.254" ns1-3_${mode}.log) -le 4 ] && \
+ [ $(grep -cF "who-has 192.0.2.254" ${LOG_DIR}/ns1-3_${mode}.log) -eq 2 ] && \
test_pass "$mode arp(F_BROADCAST) ns1-3" || \
test_fail "$mode arp(F_BROADCAST) ns1-3"
# ns1 should not receive the redirect echo request, others should
- [ $(grep -c "ICMP echo request" ns1-1_${mode}.log) -eq 4 ] && \
+ [ $(grep -c "ICMP echo request" ${LOG_DIR}/ns1-1_${mode}.log) -eq 4 ] && \
test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1" || \
test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-1"
- [ $(grep -c "ICMP echo request" ns1-2_${mode}.log) -eq 4 ] && \
+ [ $(grep -c "ICMP echo request" ${LOG_DIR}/ns1-2_${mode}.log) -eq 4 ] && \
test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2" || \
test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-2"
- [ $(grep -c "ICMP echo request" ns1-3_${mode}.log) -eq 4 ] && \
+ [ $(grep -c "ICMP echo request" ${LOG_DIR}/ns1-3_${mode}.log) -eq 4 ] && \
test_pass "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3" || \
test_fail "$mode IPv4 (F_BROADCAST|F_EXCLUDE_INGRESS) ns1-3"
# ns1 should receive the echo request, ns2 should not
- [ $(grep -c "ICMP6, echo request" ns1-1_${mode}.log) -eq 4 ] && \
+ [ $(grep -c "ICMP6, echo request" ${LOG_DIR}/ns1-1_${mode}.log) -eq 4 ] && \
test_pass "$mode IPv6 (no flags) ns1-1" || \
test_fail "$mode IPv6 (no flags) ns1-1"
- [ $(grep -c "ICMP6, echo request" ns1-2_${mode}.log) -eq 0 ] && \
+ [ $(grep -c "ICMP6, echo request" ${LOG_DIR}/ns1-2_${mode}.log) -eq 0 ] && \
test_pass "$mode IPv6 (no flags) ns1-2" || \
test_fail "$mode IPv6 (no flags) ns1-2"
}
xdpgeneric) drv_p="-S";;
esac
- ./xdp_redirect_multi $drv_p $IFACES &> xdp_redirect_${mode}.log &
+ ip netns exec ns0 ./xdp_redirect_multi $drv_p $IFACES &> ${LOG_DIR}/xdp_redirect_${mode}.log &
xdp_pid=$!
sleep 1
+ if ! ps -p $xdp_pid > /dev/null; then
+ test_fail "$mode xdp_redirect_multi start failed"
+ return 1
+ fi
if [ "$mode" = "xdpegress" ]; then
do_egress_tests $mode
kill $xdp_pid
}
-trap clean_up 0 2 3 6 9
+trap clean_up EXIT
check_env
-rm -f xdp_redirect_*.log ns*.log mac_ns*.log
for mode in ${DRV_MODE}; do
setup_ns $mode
do_tests $mode
clean_up
done
+rm -rf ${LOG_DIR}
echo "Summary: PASS $PASS, FAIL $FAIL"
[ $FAIL -eq 0 ] && exit 0 || exit 1
--- /dev/null
+{
+ "bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_KPROBE",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_ktime_get_coarse_ns),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unknown func bpf_ktime_get_coarse_ns",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_KPROBE,
+},
+{
+ "bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_TRACEPOINT",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_ktime_get_coarse_ns),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unknown func bpf_ktime_get_coarse_ns",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_TRACEPOINT,
+},
+{
+ "bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_PERF_EVENT",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_ktime_get_coarse_ns),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unknown func bpf_ktime_get_coarse_ns",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_PERF_EVENT,
+},
+{
+ "bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_RAW_TRACEPOINT",
+ .insns = {
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_ktime_get_coarse_ns),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "unknown func bpf_ktime_get_coarse_ns",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT,
+},
+{
+ "bpf_timer_init isn restricted in BPF_PROG_TYPE_KPROBE",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_EMIT_CALL(BPF_FUNC_timer_init),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_timer = { 3, 8 },
+ .errstr = "tracing progs cannot use bpf_timer yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_KPROBE,
+},
+{
+ "bpf_timer_init is forbidden in BPF_PROG_TYPE_PERF_EVENT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_EMIT_CALL(BPF_FUNC_timer_init),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_timer = { 3, 8 },
+ .errstr = "tracing progs cannot use bpf_timer yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_PERF_EVENT,
+},
+{
+ "bpf_timer_init is forbidden in BPF_PROG_TYPE_TRACEPOINT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_EMIT_CALL(BPF_FUNC_timer_init),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_timer = { 3, 8 },
+ .errstr = "tracing progs cannot use bpf_timer yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_TRACEPOINT,
+},
+{
+ "bpf_timer_init is forbidden in BPF_PROG_TYPE_RAW_TRACEPOINT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 4),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_LD_MAP_FD(BPF_REG_2, 0),
+ BPF_MOV64_IMM(BPF_REG_3, 1),
+ BPF_EMIT_CALL(BPF_FUNC_timer_init),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_timer = { 3, 8 },
+ .errstr = "tracing progs cannot use bpf_timer yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT,
+},
+{
+ "bpf_spin_lock is forbidden in BPF_PROG_TYPE_KPROBE",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_EMIT_CALL(BPF_FUNC_spin_lock),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .errstr = "tracing progs cannot use bpf_spin_lock yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_KPROBE,
+},
+{
+ "bpf_spin_lock is forbidden in BPF_PROG_TYPE_TRACEPOINT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_EMIT_CALL(BPF_FUNC_spin_lock),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .errstr = "tracing progs cannot use bpf_spin_lock yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_TRACEPOINT,
+},
+{
+ "bpf_spin_lock is forbidden in BPF_PROG_TYPE_PERF_EVENT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_EMIT_CALL(BPF_FUNC_spin_lock),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .errstr = "tracing progs cannot use bpf_spin_lock yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_PERF_EVENT,
+},
+{
+ "bpf_spin_lock is forbidden in BPF_PROG_TYPE_RAW_TRACEPOINT",
+ .insns = {
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_EMIT_CALL(BPF_FUNC_spin_lock),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_spin_lock = { 3 },
+ .errstr = "tracing progs cannot use bpf_spin_lock yet",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_RAW_TRACEPOINT,
+},
.result = ACCEPT,
},
{
+ "map in map state pruning",
+ .insns = {
+ BPF_ST_MEM(0, BPF_REG_10, -4, 0),
+ BPF_MOV64_REG(BPF_REG_6, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_6, -4),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_6),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_6),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 11),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_6),
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_6),
+ BPF_MOV64_REG(BPF_REG_1, BPF_REG_0),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, 1),
+ BPF_EXIT_INSN(),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .fixup_map_in_map = { 4, 14 },
+ .flags = BPF_F_TEST_STATE_FREQ,
+ .result = VERBOSE_ACCEPT,
+ .errstr = "processed 25 insns",
+ .prog_type = BPF_PROG_TYPE_XDP,
+},
+{
"invalid inner map pointer",
.insns = {
BPF_ST_MEM(0, BPF_REG_10, -4, 0),
.result = ACCEPT,
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
},
+{
+ "Spill a u32 scalar at fp-4 and then at fp-8",
+ .insns = {
+ /* r4 = 4321 */
+ BPF_MOV32_IMM(BPF_REG_4, 4321),
+ /* *(u32 *)(r10 -4) = r4 */
+ BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_4, -4),
+ /* *(u32 *)(r10 -8) = r4 */
+ BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_4, -8),
+ /* r4 = *(u64 *)(r10 -8) */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_4, BPF_REG_10, -8),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+},
goto err_out;
}
- printf("Get interfaces");
+ printf("Get interfaces:");
for (i = 0; i < MAX_IFACE_NUM && argv[optind + i]; i++) {
ifaces[i] = if_nametoindex(argv[optind + i]);
if (!ifaces[i])
goto err_out;
}
if (ifaces[i] > MAX_INDEX_NUM) {
- printf("Interface index to large\n");
+ printf(" interface index too large\n");
goto err_out;
}
printf(" %d", ifaces[i]);
test_content "$file" "$orig_content" "1 2 3 4 5" "successfully written"
echo "$orig_content" > "$file"
+# Test schemes file
+# =================
+
+file="$DBGFS/schemes"
+orig_content=$(cat "$file")
+
+test_write_succ "$file" "1 2 3 4 5 6 4 0 0 0 1 2 3 1 100 3 2 1" \
+ "$orig_content" "valid input"
+test_write_fail "$file" "1 2
+3 4 5 6 3 0 0 0 1 2 3 1 100 3 2 1" "$orig_content" "multi lines"
+test_write_succ "$file" "" "$orig_content" "disabling"
+echo "$orig_content" > "$file"
+
# Test target_ids file
# ====================
ppc*)
ARG1=%r3
;;
+s390*)
+ ARG1=%r2
+;;
*)
echo "Please implement other architecture here"
exit_untested
GOODREG=%r3
BADREG=%msr
;;
+s390*)
+ GOODREG=%r2
+ BADREG=%s2
+;;
*)
echo "Please implement other architecture here"
exit_untested
TEST_PROGS := gpio-mockup.sh
TEST_FILES := gpio-mockup-sysfs.sh
TEST_GEN_PROGS_EXTENDED := gpio-mockup-cdev
+CFLAGS += -O2 -g -Wall -I../../../../usr/include/
include ../lib.mk
{
char *chip;
int opt, ret, cfd, lfd;
- unsigned int offset, val, abiv;
+ unsigned int offset, val = 0, abiv;
uint32_t flags_v1;
uint64_t flags_v2;
{
# Make sure tests will time out if utility is available.
if [ -x /usr/bin/timeout ] ; then
- /usr/bin/timeout --foreground "$kselftest_timeout" "$1"
+ /usr/bin/timeout --foreground "$kselftest_timeout" $1
else
- "$1"
+ $1
fi
}
TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
echo "# $TEST_HDR_MSG"
- if [ ! -x "$TEST" ]; then
- echo -n "# Warning: file $TEST is "
- if [ ! -e "$TEST" ]; then
- echo "missing!"
- else
- echo "not executable, correct this."
- fi
+ if [ ! -e "$TEST" ]; then
+ echo "# Warning: file $TEST is missing!"
echo "not ok $test_num $TEST_HDR_MSG"
else
+ cmd="./$BASENAME_TEST"
+ if [ ! -x "$TEST" ]; then
+ echo "# Warning: file $TEST is not executable"
+
+ if [ $(head -n 1 "$TEST" | cut -c -2) = "#!" ]
+ then
+ interpreter=$(head -n 1 "$TEST" | cut -c 3-)
+ cmd="$interpreter ./$BASENAME_TEST"
+ else
+ echo "not ok $test_num $TEST_HDR_MSG"
+ return
+ fi
+ fi
cd `dirname $TEST` > /dev/null
- ((((( tap_timeout ./$BASENAME_TEST 2>&1; echo $? >&3) |
+ ((((( tap_timeout "$cmd" 2>&1; echo $? >&3) |
tap_prefix >&4) 3>&1) |
(read xs; exit $xs)) 4>>"$logfile" &&
echo "ok $test_num $TEST_HDR_MSG") ||
/x86_64/platform_info_test
/x86_64/set_boot_cpu_id
/x86_64/set_sregs_test
+/x86_64/sev_migrate_tests
/x86_64/smm_test
/x86_64/state_test
/x86_64/svm_vmcall_test
TEST_GEN_PROGS_x86_64 += x86_64/vmx_pmu_msrs_test
TEST_GEN_PROGS_x86_64 += x86_64/xen_shinfo_test
TEST_GEN_PROGS_x86_64 += x86_64/xen_vmcall_test
-TEST_GEN_PROGS_x86_64 += access_tracking_perf_test
+TEST_GEN_PROGS_x86_64 += x86_64/vmx_pi_mmio_test
+TEST_GEN_PROGS_x86_64 += x86_64/sev_migrate_tests
TEST_GEN_PROGS_x86_64 += demand_paging_test
TEST_GEN_PROGS_x86_64 += dirty_log_test
TEST_GEN_PROGS_x86_64 += dirty_log_perf_test
#include "guest_modes.h"
/* Global variable used to synchronize all of the vCPU threads. */
-static int iteration = -1;
+static int iteration;
/* Defines what vCPU threads should do during a given iteration. */
static enum {
return true;
}
-static void *vcpu_thread_main(void *arg)
+static void vcpu_thread_main(struct perf_test_vcpu_args *vcpu_args)
{
- struct perf_test_vcpu_args *vcpu_args = arg;
struct kvm_vm *vm = perf_test_args.vm;
int vcpu_id = vcpu_args->vcpu_id;
- int current_iteration = -1;
+ int current_iteration = 0;
while (spin_wait_for_next_iteration(¤t_iteration)) {
switch (READ_ONCE(iteration_work)) {
vcpu_last_completed_iteration[vcpu_id] = current_iteration;
}
-
- return NULL;
}
static void spin_wait_for_vcpu(int vcpu_id, int target_iteration)
static void access_memory(struct kvm_vm *vm, int vcpus, enum access_type access,
const char *description)
{
- perf_test_args.wr_fract = (access == ACCESS_READ) ? INT_MAX : 1;
- sync_global_to_guest(vm, perf_test_args);
+ perf_test_set_wr_fract(vm, (access == ACCESS_READ) ? INT_MAX : 1);
iteration_work = ITERATION_ACCESS_MEMORY;
run_iteration(vm, vcpus, description);
}
run_iteration(vm, vcpus, "Mark memory idle");
}
-static pthread_t *create_vcpu_threads(int vcpus)
-{
- pthread_t *vcpu_threads;
- int i;
-
- vcpu_threads = malloc(vcpus * sizeof(vcpu_threads[0]));
- TEST_ASSERT(vcpu_threads, "Failed to allocate vcpu_threads.");
-
- for (i = 0; i < vcpus; i++) {
- vcpu_last_completed_iteration[i] = iteration;
- pthread_create(&vcpu_threads[i], NULL, vcpu_thread_main,
- &perf_test_args.vcpu_args[i]);
- }
-
- return vcpu_threads;
-}
-
-static void terminate_vcpu_threads(pthread_t *vcpu_threads, int vcpus)
-{
- int i;
-
- /* Set done to signal the vCPU threads to exit */
- done = true;
-
- for (i = 0; i < vcpus; i++)
- pthread_join(vcpu_threads[i], NULL);
-}
-
static void run_test(enum vm_guest_mode mode, void *arg)
{
struct test_params *params = arg;
struct kvm_vm *vm;
- pthread_t *vcpu_threads;
int vcpus = params->vcpus;
vm = perf_test_create_vm(mode, vcpus, params->vcpu_memory_bytes, 1,
- params->backing_src);
+ params->backing_src, !overlap_memory_access);
- perf_test_setup_vcpus(vm, vcpus, params->vcpu_memory_bytes,
- !overlap_memory_access);
-
- vcpu_threads = create_vcpu_threads(vcpus);
+ perf_test_start_vcpu_threads(vcpus, vcpu_thread_main);
pr_info("\n");
access_memory(vm, vcpus, ACCESS_WRITE, "Populating memory");
mark_memory_idle(vm, vcpus);
access_memory(vm, vcpus, ACCESS_READ, "Reading from idle memory");
- terminate_vcpu_threads(vcpu_threads, vcpus);
- free(vcpu_threads);
+ /* Set done to signal the vCPU threads to exit */
+ done = true;
+
+ perf_test_join_vcpu_threads(vcpus);
perf_test_destroy_vm(vm);
}
static size_t demand_paging_size;
static char *guest_data_prototype;
-static void *vcpu_worker(void *data)
+static void vcpu_worker(struct perf_test_vcpu_args *vcpu_args)
{
int ret;
- struct perf_test_vcpu_args *vcpu_args = (struct perf_test_vcpu_args *)data;
int vcpu_id = vcpu_args->vcpu_id;
struct kvm_vm *vm = perf_test_args.vm;
struct kvm_run *run;
ts_diff = timespec_elapsed(start);
PER_VCPU_DEBUG("vCPU %d execution time: %ld.%.9lds\n", vcpu_id,
ts_diff.tv_sec, ts_diff.tv_nsec);
-
- return NULL;
}
static int handle_uffd_page_request(int uffd_mode, int uffd, uint64_t addr)
static void run_test(enum vm_guest_mode mode, void *arg)
{
struct test_params *p = arg;
- pthread_t *vcpu_threads;
pthread_t *uffd_handler_threads = NULL;
struct uffd_handler_args *uffd_args = NULL;
struct timespec start;
int r;
vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, 1,
- p->src_type);
-
- perf_test_args.wr_fract = 1;
+ p->src_type, p->partition_vcpu_memory_access);
demand_paging_size = get_backing_src_pagesz(p->src_type);
"Failed to allocate buffer for guest data pattern");
memset(guest_data_prototype, 0xAB, demand_paging_size);
- vcpu_threads = malloc(nr_vcpus * sizeof(*vcpu_threads));
- TEST_ASSERT(vcpu_threads, "Memory allocation failed");
-
- perf_test_setup_vcpus(vm, nr_vcpus, guest_percpu_mem_size,
- p->partition_vcpu_memory_access);
-
if (p->uffd_mode) {
uffd_handler_threads =
malloc(nr_vcpus * sizeof(*uffd_handler_threads));
TEST_ASSERT(pipefds, "Unable to allocate memory for pipefd");
for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++) {
- vm_paddr_t vcpu_gpa;
+ struct perf_test_vcpu_args *vcpu_args;
void *vcpu_hva;
void *vcpu_alias;
- uint64_t vcpu_mem_size;
-
- if (p->partition_vcpu_memory_access) {
- vcpu_gpa = guest_test_phys_mem +
- (vcpu_id * guest_percpu_mem_size);
- vcpu_mem_size = guest_percpu_mem_size;
- } else {
- vcpu_gpa = guest_test_phys_mem;
- vcpu_mem_size = guest_percpu_mem_size * nr_vcpus;
- }
- PER_VCPU_DEBUG("Added VCPU %d with test mem gpa [%lx, %lx)\n",
- vcpu_id, vcpu_gpa, vcpu_gpa + vcpu_mem_size);
+ vcpu_args = &perf_test_args.vcpu_args[vcpu_id];
/* Cache the host addresses of the region */
- vcpu_hva = addr_gpa2hva(vm, vcpu_gpa);
- vcpu_alias = addr_gpa2alias(vm, vcpu_gpa);
+ vcpu_hva = addr_gpa2hva(vm, vcpu_args->gpa);
+ vcpu_alias = addr_gpa2alias(vm, vcpu_args->gpa);
/*
* Set up user fault fd to handle demand paging
pipefds[vcpu_id * 2], p->uffd_mode,
p->uffd_delay, &uffd_args[vcpu_id],
vcpu_hva, vcpu_alias,
- vcpu_mem_size);
+ vcpu_args->pages * perf_test_args.guest_page_size);
}
}
- /* Export the shared variables to the guest */
- sync_global_to_guest(vm, perf_test_args);
-
pr_info("Finished creating vCPUs and starting uffd threads\n");
clock_gettime(CLOCK_MONOTONIC, &start);
-
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++) {
- pthread_create(&vcpu_threads[vcpu_id], NULL, vcpu_worker,
- &perf_test_args.vcpu_args[vcpu_id]);
- }
-
+ perf_test_start_vcpu_threads(nr_vcpus, vcpu_worker);
pr_info("Started all vCPUs\n");
- /* Wait for the vcpu threads to quit */
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++) {
- pthread_join(vcpu_threads[vcpu_id], NULL);
- PER_VCPU_DEBUG("Joined thread for vCPU %d\n", vcpu_id);
- }
-
+ perf_test_join_vcpu_threads(nr_vcpus);
ts_diff = timespec_elapsed(start);
-
pr_info("All vCPU threads joined\n");
if (p->uffd_mode) {
perf_test_destroy_vm(vm);
free(guest_data_prototype);
- free(vcpu_threads);
if (p->uffd_mode) {
free(uffd_handler_threads);
free(uffd_args);
static int iteration;
static int vcpu_last_completed_iteration[KVM_MAX_VCPUS];
-static void *vcpu_worker(void *data)
+static void vcpu_worker(struct perf_test_vcpu_args *vcpu_args)
{
int ret;
struct kvm_vm *vm = perf_test_args.vm;
struct timespec ts_diff;
struct timespec total = (struct timespec){0};
struct timespec avg;
- struct perf_test_vcpu_args *vcpu_args = (struct perf_test_vcpu_args *)data;
int vcpu_id = vcpu_args->vcpu_id;
run = vcpu_state(vm, vcpu_id);
pr_debug("\nvCPU %d dirtied 0x%lx pages over %d iterations in %ld.%.9lds. (Avg %ld.%.9lds/iteration)\n",
vcpu_id, pages_count, vcpu_last_completed_iteration[vcpu_id],
total.tv_sec, total.tv_nsec, avg.tv_sec, avg.tv_nsec);
-
- return NULL;
}
struct test_params {
static void run_test(enum vm_guest_mode mode, void *arg)
{
struct test_params *p = arg;
- pthread_t *vcpu_threads;
struct kvm_vm *vm;
unsigned long **bitmaps;
uint64_t guest_num_pages;
struct timespec clear_dirty_log_total = (struct timespec){0};
vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size,
- p->slots, p->backing_src);
+ p->slots, p->backing_src,
+ p->partition_vcpu_memory_access);
- perf_test_args.wr_fract = p->wr_fract;
+ perf_test_set_wr_fract(vm, p->wr_fract);
guest_num_pages = (nr_vcpus * guest_percpu_mem_size) >> vm_get_page_shift(vm);
guest_num_pages = vm_adjust_num_guest_pages(mode, guest_num_pages);
vm_enable_cap(vm, &cap);
}
- vcpu_threads = malloc(nr_vcpus * sizeof(*vcpu_threads));
- TEST_ASSERT(vcpu_threads, "Memory allocation failed");
-
- perf_test_setup_vcpus(vm, nr_vcpus, guest_percpu_mem_size,
- p->partition_vcpu_memory_access);
-
- sync_global_to_guest(vm, perf_test_args);
-
/* Start the iterations */
iteration = 0;
host_quit = false;
clock_gettime(CLOCK_MONOTONIC, &start);
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++) {
+ for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++)
vcpu_last_completed_iteration[vcpu_id] = -1;
- pthread_create(&vcpu_threads[vcpu_id], NULL, vcpu_worker,
- &perf_test_args.vcpu_args[vcpu_id]);
- }
+ perf_test_start_vcpu_threads(nr_vcpus, vcpu_worker);
/* Allow the vCPUs to populate memory */
pr_debug("Starting iteration %d - Populating\n", iteration);
/* Tell the vcpu thread to quit */
host_quit = true;
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++)
- pthread_join(vcpu_threads[vcpu_id], NULL);
+ perf_test_join_vcpu_threads(nr_vcpus);
avg = timespec_div(get_dirty_log_total, p->iterations);
pr_info("Get dirty log over %lu iterations took %ld.%.9lds. (Avg %ld.%.9lds/iteration)\n",
}
free_bitmaps(bitmaps, p->slots);
- free(vcpu_threads);
perf_test_destroy_vm(vm);
}
addr = guest_test_virt_mem;
addr += (READ_ONCE(random_array[i]) % guest_num_pages)
* guest_page_size;
- addr &= ~(host_page_size - 1);
+ addr = align_down(addr, host_page_size);
*(uint64_t *)addr = READ_ONCE(iteration);
}
if (!p->phys_offset) {
guest_test_phys_mem = (vm_get_max_gfn(vm) -
guest_num_pages) * guest_page_size;
- guest_test_phys_mem &= ~(host_page_size - 1);
+ guest_test_phys_mem = align_down(guest_test_phys_mem, host_page_size);
} else {
guest_test_phys_mem = p->phys_offset;
}
#ifdef __s390x__
/* Align to 1M (segment size) */
- guest_test_phys_mem &= ~((1 << 20) - 1);
+ guest_test_phys_mem = align_down(guest_test_phys_mem, 1 << 20);
#endif
pr_info("guest physical test memory offset: 0x%lx\n", guest_test_phys_mem);
};
extern const struct vm_guest_mode_params vm_guest_mode_params[];
+int open_path_or_exit(const char *path, int flags);
int open_kvm_dev_path_or_exit(void);
int kvm_check_cap(long cap);
int vm_enable_cap(struct kvm_vm *vm, struct kvm_enable_cap *cap);
#ifndef SELFTEST_KVM_PERF_TEST_UTIL_H
#define SELFTEST_KVM_PERF_TEST_UTIL_H
+#include <pthread.h>
+
#include "kvm_util.h"
/* Default guest test virtual memory offset */
#define PERF_TEST_MEM_SLOT_INDEX 1
struct perf_test_vcpu_args {
+ uint64_t gpa;
uint64_t gva;
uint64_t pages;
struct perf_test_args {
struct kvm_vm *vm;
- uint64_t host_page_size;
+ uint64_t gpa;
uint64_t guest_page_size;
int wr_fract;
extern struct perf_test_args perf_test_args;
-/*
- * Guest physical memory offset of the testing memory slot.
- * This will be set to the topmost valid physical address minus
- * the test memory size.
- */
-extern uint64_t guest_test_phys_mem;
-
struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus,
uint64_t vcpu_memory_bytes, int slots,
- enum vm_mem_backing_src_type backing_src);
+ enum vm_mem_backing_src_type backing_src,
+ bool partition_vcpu_memory_access);
void perf_test_destroy_vm(struct kvm_vm *vm);
-void perf_test_setup_vcpus(struct kvm_vm *vm, int vcpus,
- uint64_t vcpu_memory_bytes,
- bool partition_vcpu_memory_access);
+
+void perf_test_set_wr_fract(struct kvm_vm *vm, int wr_fract);
+
+void perf_test_start_vcpu_threads(int vcpus, void (*vcpu_fn)(struct perf_test_vcpu_args *));
+void perf_test_join_vcpu_threads(int vcpus);
#endif /* SELFTEST_KVM_PERF_TEST_UTIL_H */
size_t get_def_hugetlb_pagesz(void);
const struct vm_mem_backing_src_alias *vm_mem_backing_src_alias(uint32_t i);
size_t get_backing_src_pagesz(uint32_t i);
+bool is_backing_src_hugetlb(uint32_t i);
void backing_src_help(const char *flag);
enum vm_mem_backing_src_type parse_backing_src_type(const char *type_name);
long get_run_delay(void);
return vm_mem_backing_src_alias(t)->flag & MAP_SHARED;
}
+/* Aligns x up to the next multiple of size. Size must be a power of 2. */
+static inline uint64_t align_up(uint64_t x, uint64_t size)
+{
+ uint64_t mask = size - 1;
+
+ TEST_ASSERT(size != 0 && !(size & (size - 1)),
+ "size not a power of 2: %lu", size);
+ return ((x + mask) & ~mask);
+}
+
+static inline uint64_t align_down(uint64_t x, uint64_t size)
+{
+ uint64_t x_aligned_up = align_up(x, size);
+
+ if (x == x_aligned_up)
+ return x;
+ else
+ return x_aligned_up - size;
+}
+
+static inline void *align_ptr_up(void *x, size_t size)
+{
+ return (void *)align_up((unsigned long)x, size);
+}
+
#endif /* SELFTEST_KVM_TEST_UTIL_H */
return ecx & CPUID_SVM;
}
+int open_sev_dev_path_or_exit(void);
+
#endif /* SELFTEST_KVM_SVM_UTILS_H */
#ifdef __s390x__
alignment = max(0x100000, alignment);
#endif
- guest_test_phys_mem &= ~(alignment - 1);
+ guest_test_phys_mem = align_down(guest_test_virt_mem, alignment);
/* Set up the shared data structure test_args */
test_args.vm = vm;
"memsize of 0,\n"
" phdr index: %u p_memsz: 0x%" PRIx64,
n1, (uint64_t) phdr.p_memsz);
- vm_vaddr_t seg_vstart = phdr.p_vaddr;
- seg_vstart &= ~(vm_vaddr_t)(vm->page_size - 1);
+ vm_vaddr_t seg_vstart = align_down(phdr.p_vaddr, vm->page_size);
vm_vaddr_t seg_vend = phdr.p_vaddr + phdr.p_memsz - 1;
seg_vend |= vm->page_size - 1;
size_t seg_size = seg_vend - seg_vstart + 1;
static int vcpu_mmap_sz(void);
-/* Aligns x up to the next multiple of size. Size must be a power of 2. */
-static void *align(void *x, size_t size)
+int open_path_or_exit(const char *path, int flags)
{
- size_t mask = size - 1;
- TEST_ASSERT(size != 0 && !(size & (size - 1)),
- "size not a power of 2: %lu", size);
- return (void *) (((size_t) x + mask) & ~mask);
+ int fd;
+
+ fd = open(path, flags);
+ if (fd < 0) {
+ print_skip("%s not available (errno: %d)", path, errno);
+ exit(KSFT_SKIP);
+ }
+
+ return fd;
}
/*
*/
static int _open_kvm_dev_path_or_exit(int flags)
{
- int fd;
-
- fd = open(KVM_DEV_PATH, flags);
- if (fd < 0) {
- print_skip("%s not available, is KVM loaded? (errno: %d)",
- KVM_DEV_PATH, errno);
- exit(KSFT_SKIP);
- }
-
- return fd;
+ return open_path_or_exit(KVM_DEV_PATH, flags);
}
int open_kvm_dev_path_or_exit(void)
}
const struct vm_guest_mode_params vm_guest_mode_params[] = {
- { 52, 48, 0x1000, 12 },
- { 52, 48, 0x10000, 16 },
- { 48, 48, 0x1000, 12 },
- { 48, 48, 0x10000, 16 },
- { 40, 48, 0x1000, 12 },
- { 40, 48, 0x10000, 16 },
- { 0, 0, 0x1000, 12 },
- { 47, 64, 0x1000, 12 },
- { 44, 64, 0x1000, 12 },
+ [VM_MODE_P52V48_4K] = { 52, 48, 0x1000, 12 },
+ [VM_MODE_P52V48_64K] = { 52, 48, 0x10000, 16 },
+ [VM_MODE_P48V48_4K] = { 48, 48, 0x1000, 12 },
+ [VM_MODE_P48V48_64K] = { 48, 48, 0x10000, 16 },
+ [VM_MODE_P40V48_4K] = { 40, 48, 0x1000, 12 },
+ [VM_MODE_P40V48_64K] = { 40, 48, 0x10000, 16 },
+ [VM_MODE_PXXV48_4K] = { 0, 0, 0x1000, 12 },
+ [VM_MODE_P47V64_4K] = { 47, 64, 0x1000, 12 },
+ [VM_MODE_P44V64_4K] = { 44, 64, 0x1000, 12 },
};
_Static_assert(sizeof(vm_guest_mode_params)/sizeof(struct vm_guest_mode_params) == NUM_VM_MODES,
"Missing new mode params?");
alignment = 1;
#endif
+ /*
+ * When using THP mmap is not guaranteed to returned a hugepage aligned
+ * address so we have to pad the mmap. Padding is not needed for HugeTLB
+ * because mmap will always return an address aligned to the HugeTLB
+ * page size.
+ */
if (src_type == VM_MEM_SRC_ANONYMOUS_THP)
alignment = max(backing_src_pagesz, alignment);
+ ASSERT_EQ(guest_paddr, align_up(guest_paddr, backing_src_pagesz));
+
/* Add enough memory to align up if necessary */
if (alignment > 1)
region->mmap_size += alignment;
"test_malloc failed, mmap_start: %p errno: %i",
region->mmap_start, errno);
+ TEST_ASSERT(!is_backing_src_hugetlb(src_type) ||
+ region->mmap_start == align_ptr_up(region->mmap_start, backing_src_pagesz),
+ "mmap_start %p is not aligned to HugeTLB page size 0x%lx",
+ region->mmap_start, backing_src_pagesz);
+
/* Align host address */
- region->host_mem = align(region->mmap_start, alignment);
+ region->host_mem = align_ptr_up(region->mmap_start, alignment);
/* As needed perform madvise */
if ((src_type == VM_MEM_SRC_ANONYMOUS ||
"mmap of alias failed, errno: %i", errno);
/* Align host alias address */
- region->host_alias = align(region->mmap_alias, alignment);
+ region->host_alias = align_ptr_up(region->mmap_alias, alignment);
}
}
struct perf_test_args perf_test_args;
-uint64_t guest_test_phys_mem;
-
/*
* Guest virtual memory offset of the testing memory slot.
* Must not conflict with identity mapped test code.
*/
static uint64_t guest_test_virt_mem = DEFAULT_GUEST_TEST_MEM;
+struct vcpu_thread {
+ /* The id of the vCPU. */
+ int vcpu_id;
+
+ /* The pthread backing the vCPU. */
+ pthread_t thread;
+
+ /* Set to true once the vCPU thread is up and running. */
+ bool running;
+};
+
+/* The vCPU threads involved in this test. */
+static struct vcpu_thread vcpu_threads[KVM_MAX_VCPUS];
+
+/* The function run by each vCPU thread, as provided by the test. */
+static void (*vcpu_thread_fn)(struct perf_test_vcpu_args *);
+
+/* Set to true once all vCPU threads are up and running. */
+static bool all_vcpu_threads_running;
+
/*
* Continuously write to the first 8 bytes of each page in the
* specified region.
*/
static void guest_code(uint32_t vcpu_id)
{
- struct perf_test_vcpu_args *vcpu_args = &perf_test_args.vcpu_args[vcpu_id];
+ struct perf_test_args *pta = &perf_test_args;
+ struct perf_test_vcpu_args *vcpu_args = &pta->vcpu_args[vcpu_id];
uint64_t gva;
uint64_t pages;
int i;
while (true) {
for (i = 0; i < pages; i++) {
- uint64_t addr = gva + (i * perf_test_args.guest_page_size);
+ uint64_t addr = gva + (i * pta->guest_page_size);
- if (i % perf_test_args.wr_fract == 0)
+ if (i % pta->wr_fract == 0)
*(uint64_t *)addr = 0x0123456789ABCDEF;
else
READ_ONCE(*(uint64_t *)addr);
}
}
+void perf_test_setup_vcpus(struct kvm_vm *vm, int vcpus,
+ uint64_t vcpu_memory_bytes,
+ bool partition_vcpu_memory_access)
+{
+ struct perf_test_args *pta = &perf_test_args;
+ struct perf_test_vcpu_args *vcpu_args;
+ int vcpu_id;
+
+ for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++) {
+ vcpu_args = &pta->vcpu_args[vcpu_id];
+
+ vcpu_args->vcpu_id = vcpu_id;
+ if (partition_vcpu_memory_access) {
+ vcpu_args->gva = guest_test_virt_mem +
+ (vcpu_id * vcpu_memory_bytes);
+ vcpu_args->pages = vcpu_memory_bytes /
+ pta->guest_page_size;
+ vcpu_args->gpa = pta->gpa + (vcpu_id * vcpu_memory_bytes);
+ } else {
+ vcpu_args->gva = guest_test_virt_mem;
+ vcpu_args->pages = (vcpus * vcpu_memory_bytes) /
+ pta->guest_page_size;
+ vcpu_args->gpa = pta->gpa;
+ }
+
+ vcpu_args_set(vm, vcpu_id, 1, vcpu_id);
+
+ pr_debug("Added VCPU %d with test mem gpa [%lx, %lx)\n",
+ vcpu_id, vcpu_args->gpa, vcpu_args->gpa +
+ (vcpu_args->pages * pta->guest_page_size));
+ }
+}
+
struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus,
uint64_t vcpu_memory_bytes, int slots,
- enum vm_mem_backing_src_type backing_src)
+ enum vm_mem_backing_src_type backing_src,
+ bool partition_vcpu_memory_access)
{
+ struct perf_test_args *pta = &perf_test_args;
struct kvm_vm *vm;
uint64_t guest_num_pages;
+ uint64_t backing_src_pagesz = get_backing_src_pagesz(backing_src);
int i;
pr_info("Testing guest mode: %s\n", vm_guest_mode_string(mode));
- perf_test_args.host_page_size = getpagesize();
- perf_test_args.guest_page_size = vm_guest_mode_params[mode].page_size;
+ /* By default vCPUs will write to memory. */
+ pta->wr_fract = 1;
+
+ /*
+ * Snapshot the non-huge page size. This is used by the guest code to
+ * access/dirty pages at the logging granularity.
+ */
+ pta->guest_page_size = vm_guest_mode_params[mode].page_size;
guest_num_pages = vm_adjust_num_guest_pages(mode,
- (vcpus * vcpu_memory_bytes) / perf_test_args.guest_page_size);
+ (vcpus * vcpu_memory_bytes) / pta->guest_page_size);
- TEST_ASSERT(vcpu_memory_bytes % perf_test_args.host_page_size == 0,
+ TEST_ASSERT(vcpu_memory_bytes % getpagesize() == 0,
"Guest memory size is not host page size aligned.");
- TEST_ASSERT(vcpu_memory_bytes % perf_test_args.guest_page_size == 0,
+ TEST_ASSERT(vcpu_memory_bytes % pta->guest_page_size == 0,
"Guest memory size is not guest page size aligned.");
TEST_ASSERT(guest_num_pages % slots == 0,
"Guest memory cannot be evenly divided into %d slots.",
slots);
+ /*
+ * Pass guest_num_pages to populate the page tables for test memory.
+ * The memory is also added to memslot 0, but that's a benign side
+ * effect as KVM allows aliasing HVAs in meslots.
+ */
vm = vm_create_with_vcpus(mode, vcpus, DEFAULT_GUEST_PHY_PAGES,
- (vcpus * vcpu_memory_bytes) / perf_test_args.guest_page_size,
- 0, guest_code, NULL);
+ guest_num_pages, 0, guest_code, NULL);
- perf_test_args.vm = vm;
+ pta->vm = vm;
/*
* If there should be more memory in the guest test region than there
guest_num_pages, vm_get_max_gfn(vm), vcpus,
vcpu_memory_bytes);
- guest_test_phys_mem = (vm_get_max_gfn(vm) - guest_num_pages) *
- perf_test_args.guest_page_size;
- guest_test_phys_mem &= ~(perf_test_args.host_page_size - 1);
+ pta->gpa = (vm_get_max_gfn(vm) - guest_num_pages) * pta->guest_page_size;
+ pta->gpa = align_down(pta->gpa, backing_src_pagesz);
#ifdef __s390x__
/* Align to 1M (segment size) */
- guest_test_phys_mem &= ~((1 << 20) - 1);
+ pta->gpa = align_down(pta->gpa, 1 << 20);
#endif
- pr_info("guest physical test memory offset: 0x%lx\n", guest_test_phys_mem);
+ pr_info("guest physical test memory offset: 0x%lx\n", pta->gpa);
/* Add extra memory slots for testing */
for (i = 0; i < slots; i++) {
uint64_t region_pages = guest_num_pages / slots;
- vm_paddr_t region_start = guest_test_phys_mem +
- region_pages * perf_test_args.guest_page_size * i;
+ vm_paddr_t region_start = pta->gpa + region_pages * pta->guest_page_size * i;
vm_userspace_mem_region_add(vm, backing_src, region_start,
PERF_TEST_MEM_SLOT_INDEX + i,
}
/* Do mapping for the demand paging memory slot */
- virt_map(vm, guest_test_virt_mem, guest_test_phys_mem, guest_num_pages);
+ virt_map(vm, guest_test_virt_mem, pta->gpa, guest_num_pages);
+
+ perf_test_setup_vcpus(vm, vcpus, vcpu_memory_bytes, partition_vcpu_memory_access);
ucall_init(vm, NULL);
+ /* Export the shared variables to the guest. */
+ sync_global_to_guest(vm, perf_test_args);
+
return vm;
}
kvm_vm_free(vm);
}
-void perf_test_setup_vcpus(struct kvm_vm *vm, int vcpus,
- uint64_t vcpu_memory_bytes,
- bool partition_vcpu_memory_access)
+void perf_test_set_wr_fract(struct kvm_vm *vm, int wr_fract)
+{
+ perf_test_args.wr_fract = wr_fract;
+ sync_global_to_guest(vm, perf_test_args);
+}
+
+static void *vcpu_thread_main(void *data)
+{
+ struct vcpu_thread *vcpu = data;
+
+ WRITE_ONCE(vcpu->running, true);
+
+ /*
+ * Wait for all vCPU threads to be up and running before calling the test-
+ * provided vCPU thread function. This prevents thread creation (which
+ * requires taking the mmap_sem in write mode) from interfering with the
+ * guest faulting in its memory.
+ */
+ while (!READ_ONCE(all_vcpu_threads_running))
+ ;
+
+ vcpu_thread_fn(&perf_test_args.vcpu_args[vcpu->vcpu_id]);
+
+ return NULL;
+}
+
+void perf_test_start_vcpu_threads(int vcpus, void (*vcpu_fn)(struct perf_test_vcpu_args *))
{
- vm_paddr_t vcpu_gpa;
- struct perf_test_vcpu_args *vcpu_args;
int vcpu_id;
+ vcpu_thread_fn = vcpu_fn;
+ WRITE_ONCE(all_vcpu_threads_running, false);
+
for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++) {
- vcpu_args = &perf_test_args.vcpu_args[vcpu_id];
+ struct vcpu_thread *vcpu = &vcpu_threads[vcpu_id];
- vcpu_args->vcpu_id = vcpu_id;
- if (partition_vcpu_memory_access) {
- vcpu_args->gva = guest_test_virt_mem +
- (vcpu_id * vcpu_memory_bytes);
- vcpu_args->pages = vcpu_memory_bytes /
- perf_test_args.guest_page_size;
- vcpu_gpa = guest_test_phys_mem +
- (vcpu_id * vcpu_memory_bytes);
- } else {
- vcpu_args->gva = guest_test_virt_mem;
- vcpu_args->pages = (vcpus * vcpu_memory_bytes) /
- perf_test_args.guest_page_size;
- vcpu_gpa = guest_test_phys_mem;
- }
+ vcpu->vcpu_id = vcpu_id;
+ WRITE_ONCE(vcpu->running, false);
- vcpu_args_set(vm, vcpu_id, 1, vcpu_id);
+ pthread_create(&vcpu->thread, NULL, vcpu_thread_main, vcpu);
+ }
- pr_debug("Added VCPU %d with test mem gpa [%lx, %lx)\n",
- vcpu_id, vcpu_gpa, vcpu_gpa +
- (vcpu_args->pages * perf_test_args.guest_page_size));
+ for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++) {
+ while (!READ_ONCE(vcpu_threads[vcpu_id].running))
+ ;
}
+
+ WRITE_ONCE(all_vcpu_threads_running, true);
+}
+
+void perf_test_join_vcpu_threads(int vcpus)
+{
+ int vcpu_id;
+
+ for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++)
+ pthread_join(vcpu_threads[vcpu_id].thread, NULL);
}
}
}
+bool is_backing_src_hugetlb(uint32_t i)
+{
+ return !!(vm_mem_backing_src_alias(i)->flag & MAP_HUGETLB);
+}
+
static void print_available_backing_src_types(const char *prefix)
{
int i;
#include "processor.h"
#include "svm_util.h"
+#define SEV_DEV_PATH "/dev/sev"
+
struct gpr64_regs guest_regs;
u64 rflags;
exit(KSFT_SKIP);
}
}
+
+/*
+ * Open SEV_DEV_PATH if available, otherwise exit the entire program.
+ *
+ * Return:
+ * The opened file descriptor of /dev/sev.
+ */
+int open_sev_dev_path_or_exit(void)
+{
+ return open_path_or_exit(SEV_DEV_PATH, 0);
+}
static bool run_vcpus = true;
-static void *vcpu_worker(void *data)
+static void vcpu_worker(struct perf_test_vcpu_args *vcpu_args)
{
int ret;
- struct perf_test_vcpu_args *vcpu_args =
- (struct perf_test_vcpu_args *)data;
int vcpu_id = vcpu_args->vcpu_id;
struct kvm_vm *vm = perf_test_args.vm;
struct kvm_run *run;
"Invalid guest sync status: exit_reason=%s\n",
exit_reason_str(run->exit_reason));
}
-
- return NULL;
}
struct memslot_antagonist_args {
* Add the dummy memslot just below the perf_test_util memslot, which is
* at the top of the guest physical address space.
*/
- gpa = guest_test_phys_mem - pages * vm_get_page_size(vm);
+ gpa = perf_test_args.gpa - pages * vm_get_page_size(vm);
for (i = 0; i < nr_modifications; i++) {
usleep(delay);
static void run_test(enum vm_guest_mode mode, void *arg)
{
struct test_params *p = arg;
- pthread_t *vcpu_threads;
struct kvm_vm *vm;
- int vcpu_id;
vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, 1,
- VM_MEM_SRC_ANONYMOUS);
-
- perf_test_args.wr_fract = 1;
-
- vcpu_threads = malloc(nr_vcpus * sizeof(*vcpu_threads));
- TEST_ASSERT(vcpu_threads, "Memory allocation failed");
-
- perf_test_setup_vcpus(vm, nr_vcpus, guest_percpu_mem_size,
- p->partition_vcpu_memory_access);
-
- /* Export the shared variables to the guest */
- sync_global_to_guest(vm, perf_test_args);
+ VM_MEM_SRC_ANONYMOUS,
+ p->partition_vcpu_memory_access);
pr_info("Finished creating vCPUs\n");
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++)
- pthread_create(&vcpu_threads[vcpu_id], NULL, vcpu_worker,
- &perf_test_args.vcpu_args[vcpu_id]);
+ perf_test_start_vcpu_threads(nr_vcpus, vcpu_worker);
pr_info("Started all vCPUs\n");
run_vcpus = false;
- /* Wait for the vcpu threads to quit */
- for (vcpu_id = 0; vcpu_id < nr_vcpus; vcpu_id++)
- pthread_join(vcpu_threads[vcpu_id], NULL);
-
+ perf_test_join_vcpu_threads(nr_vcpus);
pr_info("All vCPU threads joined\n");
- ucall_uninit(vm);
- kvm_vm_free(vm);
-
- free(vcpu_threads);
+ perf_test_destroy_vm(vm);
}
static void help(char *name)
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/kvm.h>
+#include <linux/psp-sev.h>
+#include <stdio.h>
+#include <sys/ioctl.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <pthread.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+#include "kselftest.h"
+#include "../lib/kvm_util_internal.h"
+
+#define SEV_POLICY_ES 0b100
+
+#define NR_MIGRATE_TEST_VCPUS 4
+#define NR_MIGRATE_TEST_VMS 3
+#define NR_LOCK_TESTING_THREADS 3
+#define NR_LOCK_TESTING_ITERATIONS 10000
+
+static void sev_ioctl(int vm_fd, int cmd_id, void *data)
+{
+ struct kvm_sev_cmd cmd = {
+ .id = cmd_id,
+ .data = (uint64_t)data,
+ .sev_fd = open_sev_dev_path_or_exit(),
+ };
+ int ret;
+
+ ret = ioctl(vm_fd, KVM_MEMORY_ENCRYPT_OP, &cmd);
+ TEST_ASSERT((ret == 0 || cmd.error == SEV_RET_SUCCESS),
+ "%d failed: return code: %d, errno: %d, fw error: %d",
+ cmd_id, ret, errno, cmd.error);
+}
+
+static struct kvm_vm *sev_vm_create(bool es)
+{
+ struct kvm_vm *vm;
+ struct kvm_sev_launch_start start = { 0 };
+ int i;
+
+ vm = vm_create(VM_MODE_DEFAULT, 0, O_RDWR);
+ sev_ioctl(vm->fd, es ? KVM_SEV_ES_INIT : KVM_SEV_INIT, NULL);
+ for (i = 0; i < NR_MIGRATE_TEST_VCPUS; ++i)
+ vm_vcpu_add(vm, i);
+ if (es)
+ start.policy |= SEV_POLICY_ES;
+ sev_ioctl(vm->fd, KVM_SEV_LAUNCH_START, &start);
+ if (es)
+ sev_ioctl(vm->fd, KVM_SEV_LAUNCH_UPDATE_VMSA, NULL);
+ return vm;
+}
+
+static struct kvm_vm *__vm_create(void)
+{
+ struct kvm_vm *vm;
+ int i;
+
+ vm = vm_create(VM_MODE_DEFAULT, 0, O_RDWR);
+ for (i = 0; i < NR_MIGRATE_TEST_VCPUS; ++i)
+ vm_vcpu_add(vm, i);
+
+ return vm;
+}
+
+static int __sev_migrate_from(int dst_fd, int src_fd)
+{
+ struct kvm_enable_cap cap = {
+ .cap = KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM,
+ .args = { src_fd }
+ };
+
+ return ioctl(dst_fd, KVM_ENABLE_CAP, &cap);
+}
+
+
+static void sev_migrate_from(int dst_fd, int src_fd)
+{
+ int ret;
+
+ ret = __sev_migrate_from(dst_fd, src_fd);
+ TEST_ASSERT(!ret, "Migration failed, ret: %d, errno: %d\n", ret, errno);
+}
+
+static void test_sev_migrate_from(bool es)
+{
+ struct kvm_vm *src_vm;
+ struct kvm_vm *dst_vms[NR_MIGRATE_TEST_VMS];
+ int i;
+
+ src_vm = sev_vm_create(es);
+ for (i = 0; i < NR_MIGRATE_TEST_VMS; ++i)
+ dst_vms[i] = __vm_create();
+
+ /* Initial migration from the src to the first dst. */
+ sev_migrate_from(dst_vms[0]->fd, src_vm->fd);
+
+ for (i = 1; i < NR_MIGRATE_TEST_VMS; i++)
+ sev_migrate_from(dst_vms[i]->fd, dst_vms[i - 1]->fd);
+
+ /* Migrate the guest back to the original VM. */
+ sev_migrate_from(src_vm->fd, dst_vms[NR_MIGRATE_TEST_VMS - 1]->fd);
+
+ kvm_vm_free(src_vm);
+ for (i = 0; i < NR_MIGRATE_TEST_VMS; ++i)
+ kvm_vm_free(dst_vms[i]);
+}
+
+struct locking_thread_input {
+ struct kvm_vm *vm;
+ int source_fds[NR_LOCK_TESTING_THREADS];
+};
+
+static void *locking_test_thread(void *arg)
+{
+ int i, j;
+ struct locking_thread_input *input = (struct locking_thread_input *)arg;
+
+ for (i = 0; i < NR_LOCK_TESTING_ITERATIONS; ++i) {
+ j = i % NR_LOCK_TESTING_THREADS;
+ __sev_migrate_from(input->vm->fd, input->source_fds[j]);
+ }
+
+ return NULL;
+}
+
+static void test_sev_migrate_locking(void)
+{
+ struct locking_thread_input input[NR_LOCK_TESTING_THREADS];
+ pthread_t pt[NR_LOCK_TESTING_THREADS];
+ int i;
+
+ for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i) {
+ input[i].vm = sev_vm_create(/* es= */ false);
+ input[0].source_fds[i] = input[i].vm->fd;
+ }
+ for (i = 1; i < NR_LOCK_TESTING_THREADS; ++i)
+ memcpy(input[i].source_fds, input[0].source_fds,
+ sizeof(input[i].source_fds));
+
+ for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i)
+ pthread_create(&pt[i], NULL, locking_test_thread, &input[i]);
+
+ for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i)
+ pthread_join(pt[i], NULL);
+}
+
+static void test_sev_migrate_parameters(void)
+{
+ struct kvm_vm *sev_vm, *sev_es_vm, *vm_no_vcpu, *vm_no_sev,
+ *sev_es_vm_no_vmsa;
+ int ret;
+
+ sev_vm = sev_vm_create(/* es= */ false);
+ sev_es_vm = sev_vm_create(/* es= */ true);
+ vm_no_vcpu = vm_create(VM_MODE_DEFAULT, 0, O_RDWR);
+ vm_no_sev = __vm_create();
+ sev_es_vm_no_vmsa = vm_create(VM_MODE_DEFAULT, 0, O_RDWR);
+ sev_ioctl(sev_es_vm_no_vmsa->fd, KVM_SEV_ES_INIT, NULL);
+ vm_vcpu_add(sev_es_vm_no_vmsa, 1);
+
+
+ ret = __sev_migrate_from(sev_vm->fd, sev_es_vm->fd);
+ TEST_ASSERT(
+ ret == -1 && errno == EINVAL,
+ "Should not be able migrate to SEV enabled VM. ret: %d, errno: %d\n",
+ ret, errno);
+
+ ret = __sev_migrate_from(sev_es_vm->fd, sev_vm->fd);
+ TEST_ASSERT(
+ ret == -1 && errno == EINVAL,
+ "Should not be able migrate to SEV-ES enabled VM. ret: %d, errno: %d\n",
+ ret, errno);
+
+ ret = __sev_migrate_from(vm_no_vcpu->fd, sev_es_vm->fd);
+ TEST_ASSERT(
+ ret == -1 && errno == EINVAL,
+ "SEV-ES migrations require same number of vCPUS. ret: %d, errno: %d\n",
+ ret, errno);
+
+ ret = __sev_migrate_from(vm_no_vcpu->fd, sev_es_vm_no_vmsa->fd);
+ TEST_ASSERT(
+ ret == -1 && errno == EINVAL,
+ "SEV-ES migrations require UPDATE_VMSA. ret %d, errno: %d\n",
+ ret, errno);
+
+ ret = __sev_migrate_from(vm_no_vcpu->fd, vm_no_sev->fd);
+ TEST_ASSERT(ret == -1 && errno == EINVAL,
+ "Migrations require SEV enabled. ret %d, errno: %d\n", ret,
+ errno);
+}
+
+int main(int argc, char *argv[])
+{
+ test_sev_migrate_from(/* es= */ false);
+ test_sev_migrate_from(/* es= */ true);
+ test_sev_migrate_locking();
+ test_sev_migrate_parameters();
+ return 0;
+}
#define PVTIME_ADDR (SHINFO_REGION_GPA + PAGE_SIZE)
#define RUNSTATE_ADDR (SHINFO_REGION_GPA + PAGE_SIZE + 0x20)
+#define VCPU_INFO_ADDR (SHINFO_REGION_GPA + 0x40)
#define RUNSTATE_VADDR (SHINFO_REGION_GVA + PAGE_SIZE + 0x20)
+#define VCPU_INFO_VADDR (SHINFO_REGION_GVA + 0x40)
+
+#define EVTCHN_VECTOR 0x10
static struct kvm_vm *vm;
uint64_t time[4];
};
+struct arch_vcpu_info {
+ unsigned long cr2;
+ unsigned long pad; /* sizeof(vcpu_info_t) == 64 */
+};
+
+struct vcpu_info {
+ uint8_t evtchn_upcall_pending;
+ uint8_t evtchn_upcall_mask;
+ unsigned long evtchn_pending_sel;
+ struct arch_vcpu_info arch;
+ struct pvclock_vcpu_time_info time;
+}; /* 64 bytes (x86) */
+
#define RUNSTATE_running 0
#define RUNSTATE_runnable 1
#define RUNSTATE_blocked 2
#define RUNSTATE_offline 3
+static void evtchn_handler(struct ex_regs *regs)
+{
+ struct vcpu_info *vi = (void *)VCPU_INFO_VADDR;
+ vi->evtchn_upcall_pending = 0;
+
+ GUEST_SYNC(0x20);
+}
+
static void guest_code(void)
{
struct vcpu_runstate_info *rs = (void *)RUNSTATE_VADDR;
+ __asm__ __volatile__(
+ "sti\n"
+ "nop\n"
+ );
+
+ /* Trigger an interrupt injection */
+ GUEST_SYNC(0);
+
/* Test having the host set runstates manually */
GUEST_SYNC(RUNSTATE_runnable);
GUEST_ASSERT(rs->time[RUNSTATE_runnable] != 0);
struct kvm_xen_vcpu_attr vi = {
.type = KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO,
- .u.gpa = SHINFO_REGION_GPA + 0x40,
+ .u.gpa = VCPU_INFO_ADDR,
};
vcpu_ioctl(vm, VCPU_ID, KVM_XEN_VCPU_SET_ATTR, &vi);
};
vcpu_ioctl(vm, VCPU_ID, KVM_XEN_VCPU_SET_ATTR, &pvclock);
+ struct kvm_xen_hvm_attr vec = {
+ .type = KVM_XEN_ATTR_TYPE_UPCALL_VECTOR,
+ .u.vector = EVTCHN_VECTOR,
+ };
+ vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &vec);
+
+ vm_init_descriptor_tables(vm);
+ vcpu_init_descriptor_tables(vm, VCPU_ID);
+ vm_install_exception_handler(vm, EVTCHN_VECTOR, evtchn_handler);
+
if (do_runstate_tests) {
struct kvm_xen_vcpu_attr st = {
.type = KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADDR,
vcpu_ioctl(vm, VCPU_ID, KVM_XEN_VCPU_SET_ATTR, &st);
}
+ struct vcpu_info *vinfo = addr_gpa2hva(vm, VCPU_INFO_VADDR);
+ vinfo->evtchn_upcall_pending = 0;
+
struct vcpu_runstate_info *rs = addr_gpa2hva(vm, RUNSTATE_ADDR);
rs->state = 0x5a;
+ bool evtchn_irq_expected = false;
+
for (;;) {
volatile struct kvm_run *run = vcpu_state(vm, VCPU_ID);
struct ucall uc;
struct kvm_xen_vcpu_attr rst;
long rundelay;
- /* If no runstate support, bail out early */
- if (!do_runstate_tests)
- goto done;
-
- TEST_ASSERT(rs->state_entry_time == rs->time[0] +
- rs->time[1] + rs->time[2] + rs->time[3],
- "runstate times don't add up");
+ if (do_runstate_tests)
+ TEST_ASSERT(rs->state_entry_time == rs->time[0] +
+ rs->time[1] + rs->time[2] + rs->time[3],
+ "runstate times don't add up");
switch (uc.args[1]) {
- case RUNSTATE_running...RUNSTATE_offline:
+ case 0:
+ evtchn_irq_expected = true;
+ vinfo->evtchn_upcall_pending = 1;
+ break;
+
+ case RUNSTATE_runnable...RUNSTATE_offline:
+ TEST_ASSERT(!evtchn_irq_expected, "Event channel IRQ not seen");
+ if (!do_runstate_tests)
+ goto done;
rst.type = KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_CURRENT;
rst.u.runstate.state = uc.args[1];
vcpu_ioctl(vm, VCPU_ID, KVM_XEN_VCPU_SET_ATTR, &rst);
sched_yield();
} while (get_run_delay() < rundelay);
break;
+ case 0x20:
+ TEST_ASSERT(evtchn_irq_expected, "Unexpected event channel IRQ");
+ evtchn_irq_expected = false;
+ break;
}
break;
}
CONFIG_MEMORY_HOTPLUG=y
-CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_NOTIFIER_ERROR_INJECTION=y
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
CONFIG_MEMORY_HOTREMOVE=y
TEST_PROGS += test_vxlan_fdb_changelink.sh so_txtime.sh ipv6_flowlabel.sh
TEST_PROGS += tcp_fastopen_backup_key.sh fcnal-test.sh l2tp.sh traceroute.sh
TEST_PROGS += fin_ack_lat.sh fib_nexthop_multiprefix.sh fib_nexthops.sh
-TEST_PROGS += altnames.sh icmp_redirect.sh ip6_gre_headroom.sh
+TEST_PROGS += altnames.sh icmp.sh icmp_redirect.sh ip6_gre_headroom.sh
TEST_PROGS += route_localnet.sh
TEST_PROGS += reuseaddr_ports_exhausted.sh
TEST_PROGS += txtimestamp.sh
TEST_PROGS += gro.sh
TEST_PROGS += gre_gso.sh
TEST_PROGS += cmsg_so_mark.sh
-TEST_PROGS_EXTENDED := in_netns.sh
+TEST_PROGS += srv6_end_dt46_l3vpn_test.sh
+TEST_PROGS += srv6_end_dt4_l3vpn_test.sh
+TEST_PROGS += srv6_end_dt6_l3vpn_test.sh
+TEST_PROGS += vrf_strict_mode_test.sh
+TEST_PROGS_EXTENDED := in_netns.sh setup_loopback.sh setup_veth.sh
+TEST_PROGS_EXTENDED += toeplitz_client.sh toeplitz.sh
TEST_GEN_FILES = socket nettest
TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy reuseport_addr_any
TEST_GEN_FILES += tcp_mmap tcp_inq psock_snd txring_overwrite
CONFIG_NET_VRF=m
CONFIG_BPF_SYSCALL=y
CONFIG_CGROUP_BPF=y
+CONFIG_NET_ACT_CT=m
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_MPLS=m
CONFIG_NET_ACT_VLAN=m
test_ip6gretap()
{
- test_vlan_match gt6 'skip_hw vlan_id 555 vlan_ethtype ip' \
+ test_vlan_match gt6 'skip_hw vlan_id 555 vlan_ethtype ipv6' \
"mirror to ip6gretap"
}
mirror_install $swp1 ingress $tundev "matchall $tcflags"
tc filter add dev $h3 ingress pref 77 prot $prot \
- flower ip_ttl 50 action pass
+ flower skip_hw ip_ttl 50 action pass
mirror_test v$h1 192.0.2.1 192.0.2.2 $h3 77 0
test_ip6gretap()
{
- test_vlan_match gt6 'skip_hw vlan_id 555 vlan_ethtype ip' \
+ test_vlan_match gt6 'skip_hw vlan_id 555 vlan_ethtype ipv6' \
"mirror to ip6gretap"
}
test_span_gre_untagged_egress()
{
local tundev=$1; shift
+ local ul_proto=$1; shift
local what=$1; shift
RET=0
mirror_install $swp1 ingress $tundev "matchall $tcflags"
quick_test_span_gre_dir $tundev ingress
- quick_test_span_vlan_dir $h3 555 ingress
+ quick_test_span_vlan_dir $h3 555 ingress "$ul_proto"
h3_addr_add_del del $h3.555
bridge vlan add dev $swp3 vid 555 pvid untagged
sleep 5
quick_test_span_gre_dir $tundev ingress
- fail_test_span_vlan_dir $h3 555 ingress
+ fail_test_span_vlan_dir $h3 555 ingress "$ul_proto"
h3_addr_add_del del $h3
bridge vlan add dev $swp3 vid 555
sleep 5
quick_test_span_gre_dir $tundev ingress
- quick_test_span_vlan_dir $h3 555 ingress
+ quick_test_span_vlan_dir $h3 555 ingress "$ul_proto"
mirror_uninstall $swp1 ingress
test_gretap_untagged_egress()
{
- test_span_gre_untagged_egress gt4 "mirror to gretap"
+ test_span_gre_untagged_egress gt4 ip "mirror to gretap"
}
test_ip6gretap_untagged_egress()
{
- test_span_gre_untagged_egress gt6 "mirror to ip6gretap"
+ test_span_gre_untagged_egress gt6 ipv6 "mirror to ip6gretap"
}
test_span_gre_fdb_roaming()
local dev=$1; shift
local vid=$1; shift
local direction=$1; shift
+ local ul_proto=$1; shift
local ip1=$1; shift
local ip2=$1; shift
# Install the capture as skip_hw to avoid double-counting of packets.
# The traffic is meant for local box anyway, so will be trapped to
# kernel.
- vlan_capture_install $dev "skip_hw vlan_id $vid vlan_ethtype ip"
+ vlan_capture_install $dev "skip_hw vlan_id $vid vlan_ethtype $ul_proto"
mirror_test v$h1 $ip1 $ip2 $dev 100 $expect
mirror_test v$h2 $ip2 $ip1 $dev 100 $expect
vlan_capture_uninstall $dev
RET=0
mirror_install $swp1 $direction $swp3.555 "matchall $tcflags"
- do_test_span_vlan_dir_ips 10 "$h3.555" 111 "$direction" \
+ do_test_span_vlan_dir_ips 10 "$h3.555" 111 "$direction" ip \
192.0.2.17 192.0.2.18
- do_test_span_vlan_dir_ips 0 "$h3.555" 555 "$direction" \
+ do_test_span_vlan_dir_ips 0 "$h3.555" 555 "$direction" ip \
192.0.2.17 192.0.2.18
mirror_uninstall $swp1 $direction
ALL_TESTS="gact_drop_and_ok_test mirred_egress_redirect_test \
mirred_egress_mirror_test matchall_mirred_egress_mirror_test \
- gact_trap_test"
+ gact_trap_test mirred_egress_to_ingress_test"
NUM_NETIFS=4
source tc_common.sh
source lib.sh
h1_create()
{
simple_if_init $h1 192.0.2.1/24
+ tc qdisc add dev $h1 clsact
}
h1_destroy()
{
+ tc qdisc del dev $h1 clsact
simple_if_fini $h1 192.0.2.1/24
}
log_test "trap ($tcflags)"
}
+mirred_egress_to_ingress_test()
+{
+ RET=0
+
+ tc filter add dev $h1 protocol ip pref 100 handle 100 egress flower \
+ ip_proto icmp src_ip 192.0.2.1 dst_ip 192.0.2.2 type 8 action \
+ ct commit nat src addr 192.0.2.2 pipe \
+ ct clear pipe \
+ ct commit nat dst addr 192.0.2.1 pipe \
+ mirred ingress redirect dev $h1
+
+ tc filter add dev $swp1 protocol ip pref 11 handle 111 ingress flower \
+ ip_proto icmp src_ip 192.0.2.1 dst_ip 192.0.2.2 type 8 action drop
+ tc filter add dev $swp1 protocol ip pref 12 handle 112 ingress flower \
+ ip_proto icmp src_ip 192.0.2.1 dst_ip 192.0.2.2 type 0 action pass
+
+ $MZ $h1 -c 1 -p 64 -a $h1mac -b $h2mac -A 192.0.2.1 -B 192.0.2.2 \
+ -t icmp "ping,id=42,seq=10" -q
+
+ tc_check_packets "dev $h1 egress" 100 1
+ check_err $? "didn't mirror first packet"
+
+ tc_check_packets "dev $swp1 ingress" 111 1
+ check_fail $? "didn't redirect first packet"
+ tc_check_packets "dev $swp1 ingress" 112 1
+ check_err $? "didn't receive reply to first packet"
+
+ ping 192.0.2.2 -I$h1 -c1 -w1 -q 1>/dev/null 2>&1
+
+ tc_check_packets "dev $h1 egress" 100 2
+ check_err $? "didn't mirror second packet"
+ tc_check_packets "dev $swp1 ingress" 111 1
+ check_fail $? "didn't redirect second packet"
+ tc_check_packets "dev $swp1 ingress" 112 2
+ check_err $? "didn't receive reply to second packet"
+
+ tc filter del dev $h1 egress protocol ip pref 100 handle 100 flower
+ tc filter del dev $swp1 ingress protocol ip pref 11 handle 111 flower
+ tc filter del dev $swp1 ingress protocol ip pref 12 handle 112 flower
+
+ log_test "mirred_egress_to_ingress ($tcflags)"
+}
+
setup_prepare()
{
h1=${NETIFS[p1]}
{
local name=$1
local addr=$2
+ local proto=$3
- $NS_EXEC nc -kl $port >/dev/null &
+ [ "$proto" == 6 ] && addr="[$addr]"
+
+ $NS_EXEC socat - tcp${proto}-listen:$port,reuseaddr,fork >/dev/null &
PID=$!
while ! $NS_EXEC ss -ltn | grep -q $port; do ((i++)); sleep 0.01; done
- cat $TMPFILE | timeout 1 nc $addr $port
+ cat $TMPFILE | timeout 1 socat -u STDIN TCP:$addr:$port
log_test $? 0 "$name - copy file w/ TSO"
ethtool -K veth0 tso off
- cat $TMPFILE | timeout 1 nc $addr $port
+ cat $TMPFILE | timeout 1 socat -u STDIN TCP:$addr:$port
log_test $? 0 "$name - copy file w/ GSO"
ethtool -K veth0 tso on
sleep 2
- gre_gst_test_checks GREv6/v4 172.16.2.2
- gre_gst_test_checks GREv6/v6 2001:db8:1::2
+ gre_gst_test_checks GREv6/v4 172.16.2.2 4
+ gre_gst_test_checks GREv6/v6 2001:db8:1::2 6
cleanup
}
exit $ksft_skip
fi
-if [ ! -x "$(command -v nc)" ]; then
- echo "SKIP: Could not run test without nc tool"
+if [ ! -x "$(command -v socat)" ]; then
+ echo "SKIP: Could not run test without socat tool"
exit $ksft_skip
fi
/* Forward iterate */
for (node = 0; node < len; ++node) {
+ if (!numa_bitmask_isbitset(numa_nodes_ptr, node))
+ continue;
send_from_node(node, family, proto);
receive_on_node(rcv_fd, len, epfd, node, proto);
}
/* Reverse iterate */
for (node = len - 1; node >= 0; --node) {
+ if (!numa_bitmask_isbitset(numa_nodes_ptr, node))
+ continue;
send_from_node(node, family, proto);
receive_on_node(rcv_fd, len, epfd, node, proto);
}
ip -netns hv-$id link set veth-tap master br0
ip -netns hv-$id link set veth-tap up
+ ip link set veth-hv address 02:1d:8d:dd:0c:6$id
+
ip link set veth-hv netns vm-$id
ip -netns vm-$id addr add 10.0.0.$id/24 dev veth-hv
ip -netns vm-$id link set veth-hv up
TEST_F(tls, recvmsg_multiple)
{
unsigned int msg_iovlen = 1024;
- unsigned int len_compared = 0;
struct iovec vec[1024];
char *iov_base[1024];
unsigned int iov_len = 16;
hdr.msg_iovlen = msg_iovlen;
hdr.msg_iov = vec;
EXPECT_NE(recvmsg(self->cfd, &hdr, 0), -1);
- for (i = 0; i < msg_iovlen; i++)
- len_compared += iov_len;
for (i = 0; i < msg_iovlen; i++)
free(iov_base[i]);
static void parse_opts(int argc, char **argv)
{
+ const char *bind_addr = NULL;
int c;
- /* bind to any by default */
- setup_sockaddr(PF_INET6, "::", &cfg_bind_addr);
while ((c = getopt(argc, argv, "4b:C:Gl:n:p:rR:S:tv")) != -1) {
switch (c) {
case '4':
cfg_family = PF_INET;
cfg_alen = sizeof(struct sockaddr_in);
- setup_sockaddr(PF_INET, "0.0.0.0", &cfg_bind_addr);
break;
case 'b':
- setup_sockaddr(cfg_family, optarg, &cfg_bind_addr);
+ bind_addr = optarg;
break;
case 'C':
cfg_connect_timeout_ms = strtoul(optarg, NULL, 0);
}
}
+ if (!bind_addr)
+ bind_addr = cfg_family == PF_INET6 ? "::" : "0.0.0.0";
+
+ setup_sockaddr(cfg_family, bind_addr, &cfg_bind_addr);
+
if (optind != argc)
usage(argv[0]);
/proc-self-syscall
/proc-self-wchan
/proc-subset-pid
+/proc-tid0
/proc-uptime-001
/proc-uptime-002
/read
# SPDX-License-Identifier: GPL-2.0-only
CFLAGS += -Wall -O2 -Wno-unused-function
CFLAGS += -D_GNU_SOURCE
+LDFLAGS += -pthread
TEST_GEN_PROGS :=
TEST_GEN_PROGS += fd-001-lookup
TEST_GEN_PROGS += proc-self-syscall
TEST_GEN_PROGS += proc-self-wchan
TEST_GEN_PROGS += proc-subset-pid
+TEST_GEN_PROGS += proc-tid0
TEST_GEN_PROGS += proc-uptime-001
TEST_GEN_PROGS += proc-uptime-002
TEST_GEN_PROGS += read
--- /dev/null
+/*
+ * Copyright (c) 2021 Alexey Dobriyan <adobriyan@gmail.com>
+ *
+ * Permission to use, copy, modify, and distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+ * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+ * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+ * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+ * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+ */
+// Test that /proc/*/task never contains "0".
+#include <sys/types.h>
+#include <dirent.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <pthread.h>
+
+static pid_t pid = -1;
+
+static void atexit_hook(void)
+{
+ if (pid > 0) {
+ kill(pid, SIGKILL);
+ }
+}
+
+static void *f(void *_)
+{
+ return NULL;
+}
+
+static void sigalrm(int _)
+{
+ exit(0);
+}
+
+int main(void)
+{
+ pid = fork();
+ if (pid == 0) {
+ /* child */
+ while (1) {
+ pthread_t pth;
+ pthread_create(&pth, NULL, f, NULL);
+ pthread_join(pth, NULL);
+ }
+ } else if (pid > 0) {
+ /* parent */
+ atexit(atexit_hook);
+
+ char buf[64];
+ snprintf(buf, sizeof(buf), "/proc/%u/task", pid);
+
+ signal(SIGALRM, sigalrm);
+ alarm(1);
+
+ while (1) {
+ DIR *d = opendir(buf);
+ struct dirent *de;
+ while ((de = readdir(d))) {
+ if (strcmp(de->d_name, "0") == 0) {
+ exit(1);
+ }
+ }
+ closedir(d);
+ }
+
+ return 0;
+ } else {
+ perror("fork");
+ return 1;
+ }
+}
# SPDX-License-Identifier: GPL-2.0-only
hugepage-mmap
+hugepage-mremap
hugepage-shm
khugepaged
map_hugetlb
TEST_GEN_FILES += gup_test
TEST_GEN_FILES += hmm-tests
TEST_GEN_FILES += hugepage-mmap
+TEST_GEN_FILES += hugepage-mremap
TEST_GEN_FILES += hugepage-shm
TEST_GEN_FILES += khugepaged
TEST_GEN_FILES += madv_populate
--- /dev/null
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * hugepage-mremap:
+ *
+ * Example of remapping huge page memory in a user application using the
+ * mremap system call. Code assumes a hugetlbfs filesystem is mounted
+ * at './huge'. The code will use 10MB worth of huge pages.
+ */
+
+#define _GNU_SOURCE
+#include <stdlib.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <fcntl.h> /* Definition of O_* constants */
+#include <sys/syscall.h> /* Definition of SYS_* constants */
+#include <linux/userfaultfd.h>
+#include <sys/ioctl.h>
+
+#define LENGTH (1UL * 1024 * 1024 * 1024)
+
+#define PROTECTION (PROT_READ | PROT_WRITE | PROT_EXEC)
+#define FLAGS (MAP_SHARED | MAP_ANONYMOUS)
+
+static void check_bytes(char *addr)
+{
+ printf("First hex is %x\n", *((unsigned int *)addr));
+}
+
+static void write_bytes(char *addr)
+{
+ unsigned long i;
+
+ for (i = 0; i < LENGTH; i++)
+ *(addr + i) = (char)i;
+}
+
+static int read_bytes(char *addr)
+{
+ unsigned long i;
+
+ check_bytes(addr);
+ for (i = 0; i < LENGTH; i++)
+ if (*(addr + i) != (char)i) {
+ printf("Mismatch at %lu\n", i);
+ return 1;
+ }
+ return 0;
+}
+
+static void register_region_with_uffd(char *addr, size_t len)
+{
+ long uffd; /* userfaultfd file descriptor */
+ struct uffdio_api uffdio_api;
+ struct uffdio_register uffdio_register;
+
+ /* Create and enable userfaultfd object. */
+
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd == -1) {
+ perror("userfaultfd");
+ exit(1);
+ }
+
+ uffdio_api.api = UFFD_API;
+ uffdio_api.features = 0;
+ if (ioctl(uffd, UFFDIO_API, &uffdio_api) == -1) {
+ perror("ioctl-UFFDIO_API");
+ exit(1);
+ }
+
+ /* Create a private anonymous mapping. The memory will be
+ * demand-zero paged--that is, not yet allocated. When we
+ * actually touch the memory, it will be allocated via
+ * the userfaultfd.
+ */
+
+ addr = mmap(NULL, len, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (addr == MAP_FAILED) {
+ perror("mmap");
+ exit(1);
+ }
+
+ printf("Address returned by mmap() = %p\n", addr);
+
+ /* Register the memory range of the mapping we just created for
+ * handling by the userfaultfd object. In mode, we request to track
+ * missing pages (i.e., pages that have not yet been faulted in).
+ */
+
+ uffdio_register.range.start = (unsigned long)addr;
+ uffdio_register.range.len = len;
+ uffdio_register.mode = UFFDIO_REGISTER_MODE_MISSING;
+ if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register) == -1) {
+ perror("ioctl-UFFDIO_REGISTER");
+ exit(1);
+ }
+}
+
+int main(void)
+{
+ int ret = 0;
+
+ int fd = open("/huge/test", O_CREAT | O_RDWR, 0755);
+
+ if (fd < 0) {
+ perror("Open failed");
+ exit(1);
+ }
+
+ /* mmap to a PUD aligned address to hopefully trigger pmd sharing. */
+ unsigned long suggested_addr = 0x7eaa40000000;
+ void *haddr = mmap((void *)suggested_addr, LENGTH, PROTECTION,
+ MAP_HUGETLB | MAP_SHARED | MAP_POPULATE, fd, 0);
+ printf("Map haddr: Returned address is %p\n", haddr);
+ if (haddr == MAP_FAILED) {
+ perror("mmap1");
+ exit(1);
+ }
+
+ /* mmap again to a dummy address to hopefully trigger pmd sharing. */
+ suggested_addr = 0x7daa40000000;
+ void *daddr = mmap((void *)suggested_addr, LENGTH, PROTECTION,
+ MAP_HUGETLB | MAP_SHARED | MAP_POPULATE, fd, 0);
+ printf("Map daddr: Returned address is %p\n", daddr);
+ if (daddr == MAP_FAILED) {
+ perror("mmap3");
+ exit(1);
+ }
+
+ suggested_addr = 0x7faa40000000;
+ void *vaddr =
+ mmap((void *)suggested_addr, LENGTH, PROTECTION, FLAGS, -1, 0);
+ printf("Map vaddr: Returned address is %p\n", vaddr);
+ if (vaddr == MAP_FAILED) {
+ perror("mmap2");
+ exit(1);
+ }
+
+ register_region_with_uffd(haddr, LENGTH);
+
+ void *addr = mremap(haddr, LENGTH, LENGTH,
+ MREMAP_MAYMOVE | MREMAP_FIXED, vaddr);
+ if (addr == MAP_FAILED) {
+ perror("mremap");
+ exit(1);
+ }
+
+ printf("Mremap: Returned address is %p\n", addr);
+ check_bytes(addr);
+ write_bytes(addr);
+ ret = read_bytes(addr);
+
+ munmap(addr, LENGTH);
+
+ return ret;
+}
#include <time.h>
#include <string.h>
#include <numa.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <err.h>
#include "../kselftest.h"
#include "../../../../include/vdso/time64.h"
#define KSM_MERGE_ACROSS_NODES_DEFAULT true
#define MB (1ul << 20)
+#define PAGE_SHIFT 12
+#define HPAGE_SHIFT 21
+
+#define PAGE_SIZE (1 << PAGE_SHIFT)
+#define HPAGE_SIZE (1 << HPAGE_SHIFT)
+
+#define PAGEMAP_PRESENT(ent) (((ent) & (1ull << 63)) != 0)
+#define PAGEMAP_PFN(ent) ((ent) & ((1ull << 55) - 1))
+
struct ksm_sysfs {
unsigned long max_page_sharing;
unsigned long merge_across_nodes;
CHECK_KSM_ZERO_PAGE_MERGE,
CHECK_KSM_NUMA_MERGE,
KSM_MERGE_TIME,
+ KSM_MERGE_TIME_HUGE_PAGES,
KSM_COW_TIME
};
" -P evaluate merging time and speed.\n"
" For this test, the size of duplicated memory area (in MiB)\n"
" must be provided using -s option\n"
+ " -H evaluate merging time and speed of area allocated mostly with huge pages\n"
+ " For this test, the size of duplicated memory area (in MiB)\n"
+ " must be provided using -s option\n"
" -C evaluate the time required to break COW of merged pages.\n\n");
printf(" -a: specify the access protections of pages.\n"
return KSFT_FAIL;
}
+static int get_next_mem_node(int node)
+{
+
+ long node_size;
+ int mem_node = 0;
+ int i, max_node = numa_max_node();
+
+ for (i = node + 1; i <= max_node + node; i++) {
+ mem_node = i % (max_node + 1);
+ node_size = numa_node_size(mem_node, NULL);
+ if (node_size > 0)
+ break;
+ }
+ return mem_node;
+}
+
+static int get_first_mem_node(void)
+{
+ return get_next_mem_node(numa_max_node());
+}
+
static int check_ksm_numa_merge(int mapping, int prot, int timeout, bool merge_across_nodes,
size_t page_size)
{
void *numa1_map_ptr, *numa2_map_ptr;
struct timespec start_time;
int page_count = 2;
+ int first_node;
if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
perror("clock_gettime");
perror("NUMA support not enabled");
return KSFT_SKIP;
}
- if (numa_max_node() < 1) {
+ if (numa_num_configured_nodes() <= 1) {
printf("At least 2 NUMA nodes must be available\n");
return KSFT_SKIP;
}
return KSFT_FAIL;
/* allocate 2 pages in 2 different NUMA nodes and fill them with the same data */
- numa1_map_ptr = numa_alloc_onnode(page_size, 0);
- numa2_map_ptr = numa_alloc_onnode(page_size, 1);
+ first_node = get_first_mem_node();
+ numa1_map_ptr = numa_alloc_onnode(page_size, first_node);
+ numa2_map_ptr = numa_alloc_onnode(page_size, get_next_mem_node(first_node));
if (!numa1_map_ptr || !numa2_map_ptr) {
perror("numa_alloc_onnode");
return KSFT_FAIL;
return KSFT_FAIL;
}
+int64_t allocate_transhuge(void *ptr, int pagemap_fd)
+{
+ uint64_t ent[2];
+
+ /* drop pmd */
+ if (mmap(ptr, HPAGE_SIZE, PROT_READ | PROT_WRITE,
+ MAP_FIXED | MAP_ANONYMOUS |
+ MAP_NORESERVE | MAP_PRIVATE, -1, 0) != ptr)
+ errx(2, "mmap transhuge");
+
+ if (madvise(ptr, HPAGE_SIZE, MADV_HUGEPAGE))
+ err(2, "MADV_HUGEPAGE");
+
+ /* allocate transparent huge page */
+ *(volatile void **)ptr = ptr;
+
+ if (pread(pagemap_fd, ent, sizeof(ent),
+ (uintptr_t)ptr >> (PAGE_SHIFT - 3)) != sizeof(ent))
+ err(2, "read pagemap");
+
+ if (PAGEMAP_PRESENT(ent[0]) && PAGEMAP_PRESENT(ent[1]) &&
+ PAGEMAP_PFN(ent[0]) + 1 == PAGEMAP_PFN(ent[1]) &&
+ !(PAGEMAP_PFN(ent[0]) & ((1 << (HPAGE_SHIFT - PAGE_SHIFT)) - 1)))
+ return PAGEMAP_PFN(ent[0]);
+
+ return -1;
+}
+
+static int ksm_merge_hugepages_time(int mapping, int prot, int timeout, size_t map_size)
+{
+ void *map_ptr, *map_ptr_orig;
+ struct timespec start_time, end_time;
+ unsigned long scan_time_ns;
+ int pagemap_fd, n_normal_pages, n_huge_pages;
+
+ map_size *= MB;
+ size_t len = map_size;
+
+ len -= len % HPAGE_SIZE;
+ map_ptr_orig = mmap(NULL, len + HPAGE_SIZE, PROT_READ | PROT_WRITE,
+ MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, -1, 0);
+ map_ptr = map_ptr_orig + HPAGE_SIZE - (uintptr_t)map_ptr_orig % HPAGE_SIZE;
+
+ if (map_ptr_orig == MAP_FAILED)
+ err(2, "initial mmap");
+
+ if (madvise(map_ptr, len + HPAGE_SIZE, MADV_HUGEPAGE))
+ err(2, "MADV_HUGEPAGE");
+
+ pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+ if (pagemap_fd < 0)
+ err(2, "open pagemap");
+
+ n_normal_pages = 0;
+ n_huge_pages = 0;
+ for (void *p = map_ptr; p < map_ptr + len; p += HPAGE_SIZE) {
+ if (allocate_transhuge(p, pagemap_fd) < 0)
+ n_normal_pages++;
+ else
+ n_huge_pages++;
+ }
+ printf("Number of normal pages: %d\n", n_normal_pages);
+ printf("Number of huge pages: %d\n", n_huge_pages);
+
+ memset(map_ptr, '*', len);
+
+ if (clock_gettime(CLOCK_MONOTONIC_RAW, &start_time)) {
+ perror("clock_gettime");
+ goto err_out;
+ }
+ if (ksm_merge_pages(map_ptr, map_size, start_time, timeout))
+ goto err_out;
+ if (clock_gettime(CLOCK_MONOTONIC_RAW, &end_time)) {
+ perror("clock_gettime");
+ goto err_out;
+ }
+
+ scan_time_ns = (end_time.tv_sec - start_time.tv_sec) * NSEC_PER_SEC +
+ (end_time.tv_nsec - start_time.tv_nsec);
+
+ printf("Total size: %lu MiB\n", map_size / MB);
+ printf("Total time: %ld.%09ld s\n", scan_time_ns / NSEC_PER_SEC,
+ scan_time_ns % NSEC_PER_SEC);
+ printf("Average speed: %.3f MiB/s\n", (map_size / MB) /
+ ((double)scan_time_ns / NSEC_PER_SEC));
+
+ munmap(map_ptr_orig, len + HPAGE_SIZE);
+ return KSFT_PASS;
+
+err_out:
+ printf("Not OK\n");
+ munmap(map_ptr_orig, len + HPAGE_SIZE);
+ return KSFT_FAIL;
+}
+
static int ksm_merge_time(int mapping, int prot, int timeout, size_t map_size)
{
void *map_ptr;
bool merge_across_nodes = KSM_MERGE_ACROSS_NODES_DEFAULT;
long size_MB = 0;
- while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPC")) != -1) {
+ while ((opt = getopt(argc, argv, "ha:p:l:z:m:s:MUZNPCH")) != -1) {
switch (opt) {
case 'a':
prot = str_to_prot(optarg);
case 'P':
test_name = KSM_MERGE_TIME;
break;
+ case 'H':
+ test_name = KSM_MERGE_TIME_HUGE_PAGES;
+ break;
case 'C':
test_name = KSM_COW_TIME;
break;
ret = ksm_merge_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec,
size_MB);
break;
+ case KSM_MERGE_TIME_HUGE_PAGES:
+ if (size_MB == 0) {
+ printf("Option '-s' is required.\n");
+ return KSFT_FAIL;
+ }
+ ret = ksm_merge_hugepages_time(MAP_PRIVATE | MAP_ANONYMOUS, prot,
+ ksm_scan_limit_sec, size_MB);
+ break;
case KSM_COW_TIME:
ret = ksm_cow_time(MAP_PRIVATE | MAP_ANONYMOUS, prot, ksm_scan_limit_sec,
page_size);
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
+#include <linux/mman.h>
#include <sys/mman.h>
#include "../kselftest.h"
-#if defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE)
-
/*
* For now, we're using 2 MiB of private anonymous memory for all tests.
*/
err, ksft_test_num());
return ksft_exit_pass();
}
-
-#else /* defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) */
-
-#warning "missing MADV_POPULATE_READ or MADV_POPULATE_WRITE definition"
-
-int main(int argc, char **argv)
-{
- ksft_print_header();
- ksft_exit_skip("MADV_POPULATE_READ or MADV_POPULATE_WRITE not defined\n");
-}
-
-#endif /* defined(MADV_POPULATE_READ) && defined(MADV_POPULATE_WRITE) */
echo "[PASS]"
fi
+echo "-----------------------"
+echo "running hugepage-mremap"
+echo "-----------------------"
+./hugepage-mremap
+if [ $? -ne 0 ]; then
+ echo "[FAIL]"
+ exitcode=1
+else
+ echo "[PASS]"
+fi
+
echo "NOTE: The above hugetlb tests provide minimal coverage. Use"
echo " https://github.com/libhugetlbfs/libhugetlbfs.git for"
echo " hugetlb regression testing."
warnx("allocate %zd transhuge pages, using %zd MiB virtual memory"
" and %zd MiB of ram", len >> HPAGE_SHIFT, len >> 20,
- len >> (20 + HPAGE_SHIFT - PAGE_SHIFT - 1));
+ ram >> (20 + HPAGE_SHIFT - PAGE_SHIFT - 1));
pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
if (pagemap_fd < 0)
#include <assert.h>
#include <inttypes.h>
#include <stdint.h>
+#include <sys/random.h>
#include "../kselftest.h"
}
struct uffd_test_ops {
- unsigned long expected_ioctls;
void (*allocate_area)(void **alloc_area);
void (*release_pages)(char *rel_area);
void (*alias_mapping)(__u64 *start, size_t len, unsigned long offset);
};
-#define SHMEM_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \
- (1 << _UFFDIO_COPY) | \
- (1 << _UFFDIO_ZEROPAGE))
-
-#define ANON_EXPECTED_IOCTLS ((1 << _UFFDIO_WAKE) | \
- (1 << _UFFDIO_COPY) | \
- (1 << _UFFDIO_ZEROPAGE) | \
- (1 << _UFFDIO_WRITEPROTECT))
-
static struct uffd_test_ops anon_uffd_test_ops = {
- .expected_ioctls = ANON_EXPECTED_IOCTLS,
.allocate_area = anon_allocate_area,
.release_pages = anon_release_pages,
.alias_mapping = noop_alias_mapping,
};
static struct uffd_test_ops shmem_uffd_test_ops = {
- .expected_ioctls = SHMEM_EXPECTED_IOCTLS,
.allocate_area = shmem_allocate_area,
.release_pages = shmem_release_pages,
.alias_mapping = shmem_alias_mapping,
};
static struct uffd_test_ops hugetlb_uffd_test_ops = {
- .expected_ioctls = UFFD_API_RANGE_IOCTLS_BASIC & ~(1 << _UFFDIO_CONTINUE),
.allocate_area = hugetlb_allocate_area,
.release_pages = hugetlb_release_pages,
.alias_mapping = hugetlb_alias_mapping,
static struct uffd_test_ops *uffd_test_ops;
+static inline uint64_t uffd_minor_feature(void)
+{
+ if (test_type == TEST_HUGETLB && map_shared)
+ return UFFD_FEATURE_MINOR_HUGETLBFS;
+ else if (test_type == TEST_SHMEM)
+ return UFFD_FEATURE_MINOR_SHMEM;
+ else
+ return 0;
+}
+
+static uint64_t get_expected_ioctls(uint64_t mode)
+{
+ uint64_t ioctls = UFFD_API_RANGE_IOCTLS;
+
+ if (test_type == TEST_HUGETLB)
+ ioctls &= ~(1 << _UFFDIO_ZEROPAGE);
+
+ if (!((mode & UFFDIO_REGISTER_MODE_WP) && test_uffdio_wp))
+ ioctls &= ~(1 << _UFFDIO_WRITEPROTECT);
+
+ if (!((mode & UFFDIO_REGISTER_MODE_MINOR) && test_uffdio_minor))
+ ioctls &= ~(1 << _UFFDIO_CONTINUE);
+
+ return ioctls;
+}
+
+static void assert_expected_ioctls_present(uint64_t mode, uint64_t ioctls)
+{
+ uint64_t expected = get_expected_ioctls(mode);
+ uint64_t actual = ioctls & expected;
+
+ if (actual != expected) {
+ err("missing ioctl(s): expected %"PRIx64" actual: %"PRIx64,
+ expected, actual);
+ }
+}
+
static void userfaultfd_open(uint64_t *features)
{
struct uffdio_api uffdio_api;
munmap_area((void **)&area_dst_alias);
}
-static void uffd_test_ctx_init_ext(uint64_t *features)
+static void uffd_test_ctx_init(uint64_t features)
{
unsigned long nr, cpu;
uffd_test_ops->allocate_area((void **)&area_src);
uffd_test_ops->allocate_area((void **)&area_dst);
- userfaultfd_open(features);
+ userfaultfd_open(&features);
count_verify = malloc(nr_pages * sizeof(unsigned long long));
if (!count_verify)
err("pipe");
}
-static inline void uffd_test_ctx_init(uint64_t features)
-{
- uffd_test_ctx_init_ext(&features);
-}
-
static int my_bcmp(char *str1, char *str2, size_t n)
{
unsigned long i;
static void *locking_thread(void *arg)
{
unsigned long cpu = (unsigned long) arg;
- struct random_data rand;
unsigned long page_nr = *(&(page_nr)); /* uninitialized warning */
- int32_t rand_nr;
unsigned long long count;
- char randstate[64];
- unsigned int seed;
- if (bounces & BOUNCE_RANDOM) {
- seed = (unsigned int) time(NULL) - bounces;
- if (!(bounces & BOUNCE_RACINGFAULTS))
- seed += cpu;
- bzero(&rand, sizeof(rand));
- bzero(&randstate, sizeof(randstate));
- if (initstate_r(seed, randstate, sizeof(randstate), &rand))
- err("initstate_r failed");
- } else {
+ if (!(bounces & BOUNCE_RANDOM)) {
page_nr = -bounces;
if (!(bounces & BOUNCE_RACINGFAULTS))
page_nr += cpu * nr_pages_per_cpu;
while (!finished) {
if (bounces & BOUNCE_RANDOM) {
- if (random_r(&rand, &rand_nr))
- err("random_r failed");
- page_nr = rand_nr;
- if (sizeof(page_nr) > sizeof(rand_nr)) {
- if (random_r(&rand, &rand_nr))
- err("random_r failed");
- page_nr |= (((unsigned long) rand_nr) << 16) <<
- 16;
- }
+ if (getrandom(&page_nr, sizeof(page_nr), 0) != sizeof(page_nr))
+ err("getrandom failed");
} else
page_nr += 1;
page_nr %= nr_pages;
{
struct uffdio_zeropage uffdio_zeropage;
int ret;
- unsigned long has_zeropage;
+ bool has_zeropage = get_expected_ioctls(0) & (1 << _UFFDIO_ZEROPAGE);
__s64 res;
- has_zeropage = uffd_test_ops->expected_ioctls & (1 << _UFFDIO_ZEROPAGE);
-
if (offset >= nr_pages * page_size)
err("unexpected offset %lu", offset);
uffdio_zeropage.range.start = (unsigned long) area_dst + offset;
static int userfaultfd_zeropage_test(void)
{
struct uffdio_register uffdio_register;
- unsigned long expected_ioctls;
printf("testing UFFDIO_ZEROPAGE: ");
fflush(stdout);
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
err("register failure");
- expected_ioctls = uffd_test_ops->expected_ioctls;
- if ((uffdio_register.ioctls & expected_ioctls) != expected_ioctls)
- err("unexpected missing ioctl for anon memory");
+ assert_expected_ioctls_present(
+ uffdio_register.mode, uffdio_register.ioctls);
if (uffdio_zeropage(uffd, 0))
if (my_bcmp(area_dst, zeropage, page_size))
static int userfaultfd_events_test(void)
{
struct uffdio_register uffdio_register;
- unsigned long expected_ioctls;
pthread_t uffd_mon;
int err, features;
pid_t pid;
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
err("register failure");
- expected_ioctls = uffd_test_ops->expected_ioctls;
- if ((uffdio_register.ioctls & expected_ioctls) != expected_ioctls)
- err("unexpected missing ioctl for anon memory");
+ assert_expected_ioctls_present(
+ uffdio_register.mode, uffdio_register.ioctls);
if (pthread_create(&uffd_mon, &attr, uffd_poll_thread, &stats))
err("uffd_poll_thread create");
static int userfaultfd_sig_test(void)
{
struct uffdio_register uffdio_register;
- unsigned long expected_ioctls;
unsigned long userfaults;
pthread_t uffd_mon;
int err, features;
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
err("register failure");
- expected_ioctls = uffd_test_ops->expected_ioctls;
- if ((uffdio_register.ioctls & expected_ioctls) != expected_ioctls)
- err("unexpected missing ioctl for anon memory");
+ assert_expected_ioctls_present(
+ uffdio_register.mode, uffdio_register.ioctls);
if (faulting_process(1))
err("faulting process failed");
static int userfaultfd_minor_test(void)
{
struct uffdio_register uffdio_register;
- unsigned long expected_ioctls;
unsigned long p;
pthread_t uffd_mon;
uint8_t expected_byte;
void *expected_page;
char c;
struct uffd_stats stats = { 0 };
- uint64_t req_features, features_out;
if (!test_uffdio_minor)
return 0;
printf("testing minor faults: ");
fflush(stdout);
- if (test_type == TEST_HUGETLB)
- req_features = UFFD_FEATURE_MINOR_HUGETLBFS;
- else if (test_type == TEST_SHMEM)
- req_features = UFFD_FEATURE_MINOR_SHMEM;
- else
- return 1;
-
- features_out = req_features;
- uffd_test_ctx_init_ext(&features_out);
- /* If kernel reports required features aren't supported, skip test. */
- if ((features_out & req_features) != req_features) {
- printf("skipping test due to lack of feature support\n");
- fflush(stdout);
- return 0;
- }
+ uffd_test_ctx_init(uffd_minor_feature());
uffdio_register.range.start = (unsigned long)area_dst_alias;
uffdio_register.range.len = nr_pages * page_size;
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
err("register failure");
- expected_ioctls = uffd_test_ops->expected_ioctls;
- expected_ioctls |= 1 << _UFFDIO_CONTINUE;
- if ((uffdio_register.ioctls & expected_ioctls) != expected_ioctls)
- err("unexpected missing ioctl(s)");
+ assert_expected_ioctls_present(
+ uffdio_register.mode, uffdio_register.ioctls);
/*
* After registering with UFFD, populate the non-UFFD-registered side of
pthread_attr_setstacksize(&attr, 16*1024*1024);
while (bounces--) {
- unsigned long expected_ioctls;
-
printf("bounces: %d, mode:", bounces);
if (bounces & BOUNCE_RANDOM)
printf(" rnd");
uffdio_register.mode |= UFFDIO_REGISTER_MODE_WP;
if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register))
err("register failure");
- expected_ioctls = uffd_test_ops->expected_ioctls;
- if ((uffdio_register.ioctls & expected_ioctls) !=
- expected_ioctls)
- err("unexpected missing ioctl for anon memory");
+ assert_expected_ioctls_present(
+ uffdio_register.mode, uffdio_register.ioctls);
if (area_dst_alias) {
uffdio_register.range.start = (unsigned long)
static void set_test_type(const char *type)
{
+ uint64_t features = UFFD_API_FEATURES;
+
if (!strcmp(type, "anon")) {
test_type = TEST_ANON;
uffd_test_ops = &anon_uffd_test_ops;
if ((unsigned long) area_count(NULL, 0) + sizeof(unsigned long long) * 2
> page_size)
err("Impossible to run this test");
+
+ /*
+ * Whether we can test certain features depends not just on test type,
+ * but also on whether or not this particular kernel supports the
+ * feature.
+ */
+
+ userfaultfd_open(&features);
+
+ test_uffdio_wp = test_uffdio_wp &&
+ (features & UFFD_FEATURE_PAGEFAULT_FLAG_WP);
+ test_uffdio_minor = test_uffdio_minor &&
+ (features & uffd_minor_feature());
+
+ close(uffd);
+ uffd = -1;
}
static void sigalrm(int sig)
if (opt_pid)
printf("%lx\t", voff);
if (opt_file)
- printf("%lu\t", voff);
+ printf("%lx\t", voff);
if (opt_list_cgroup)
printf("@%llu\t", (unsigned long long)cgroup0);
if (opt_list_mapcnt)
if (opt_pid)
printf("%lx\t", voffset);
if (opt_file)
- printf("%lu\t", voffset);
+ printf("%lx\t", voffset);
if (opt_list_cgroup)
printf("@%llu\t", (unsigned long long)cgroup);
if (opt_list_mapcnt)
.sa_flags = SA_SIGINFO,
};
-static void walk_file(const char *name, const struct stat *st)
+static void walk_file_range(const char *name, int fd,
+ unsigned long off, unsigned long end)
{
uint8_t vec[PAGEMAP_BATCH];
uint64_t buf[PAGEMAP_BATCH], flags;
uint64_t cgroup = 0;
uint64_t mapcnt = 0;
unsigned long nr_pages, pfn, i;
- off_t off, end = st->st_size;
- int fd;
ssize_t len;
void *ptr;
int first = 1;
- fd = checked_open(name, O_RDONLY|O_NOATIME|O_NOFOLLOW);
-
- for (off = 0; off < end; off += len) {
+ for (; off < end; off += len) {
nr_pages = (end - off + page_size - 1) / page_size;
if (nr_pages > PAGEMAP_BATCH)
nr_pages = PAGEMAP_BATCH;
if (first && opt_list) {
first = 0;
flush_page_range();
- show_file(name, st);
}
add_page(off / page_size + i, pfn,
flags, cgroup, mapcnt, buf[i]);
}
}
+}
+
+static void walk_file(const char *name, const struct stat *st)
+{
+ int i;
+ int fd;
+
+ fd = checked_open(name, O_RDONLY|O_NOATIME|O_NOFOLLOW);
+
+ if (!nr_addr_ranges)
+ add_addr_range(0, st->st_size / page_size);
+
+ for (i = 0; i < nr_addr_ranges; i++)
+ walk_file_range(name, fd, opt_offset[i] * page_size,
+ (opt_offset[i] + opt_size[i]) * page_size);
close(fd);
}
return 0;
}
+struct stat st;
+
static void walk_page_cache(void)
{
- struct stat st;
-
kpageflags_fd = checked_open(opt_kpageflags, O_RDONLY);
pagemap_fd = checked_open("/proc/self/pagemap", O_RDONLY);
sigaction(SIGBUS, &sigbus_action, NULL);
if (opt_list)
printf("\n\n");
+ if (opt_file) {
+ show_file(opt_file, &st);
+ printf("\n");
+ }
+
show_summary();
if (opt_list_mapcnt)
* Example use:
* cat /sys/kernel/debug/page_owner > page_owner_full.txt
* ./page_owner_sort page_owner_full.txt sorted_page_owner.txt
+ * Or sort by total memory:
+ * ./page_owner_sort -m page_owner_full.txt sorted_page_owner.txt
*
* See Documentation/vm/page_owner.rst
*/
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
+#include <regex.h>
+#include <errno.h>
struct block_list {
char *txt;
int len;
int num;
+ int page_num;
};
-
+static int sort_by_memory;
+static regex_t order_pattern;
static struct block_list *list;
static int list_size;
static int max_size;
return l2->num - l1->num;
}
+static int compare_page_num(const void *p1, const void *p2)
+{
+ const struct block_list *l1 = p1, *l2 = p2;
+
+ return l2->page_num - l1->page_num;
+}
+
+static int get_page_num(char *buf)
+{
+ int err, val_len, order_val;
+ char order_str[4] = {0};
+ char *endptr;
+ regmatch_t pmatch[2];
+
+ err = regexec(&order_pattern, buf, 2, pmatch, REG_NOTBOL);
+ if (err != 0 || pmatch[1].rm_so == -1) {
+ printf("no order pattern in %s\n", buf);
+ return 0;
+ }
+ val_len = pmatch[1].rm_eo - pmatch[1].rm_so;
+ if (val_len > 2) /* max_order should not exceed 2 digits */
+ goto wrong_order;
+
+ memcpy(order_str, buf + pmatch[1].rm_so, val_len);
+
+ errno = 0;
+ order_val = strtol(order_str, &endptr, 10);
+ if (errno != 0 || endptr == order_str || *endptr != '\0')
+ goto wrong_order;
+
+ return 1 << order_val;
+
+wrong_order:
+ printf("wrong order in follow buf:\n%s\n", buf);
+ return 0;
+}
+
static void add_list(char *buf, int len)
{
if (list_size != 0 &&
len == list[list_size-1].len &&
memcmp(buf, list[list_size-1].txt, len) == 0) {
list[list_size-1].num++;
+ list[list_size-1].page_num += get_page_num(buf);
return;
}
if (list_size == max_size) {
list[list_size].txt = malloc(len+1);
list[list_size].len = len;
list[list_size].num = 1;
+ list[list_size].page_num = get_page_num(buf);
memcpy(list[list_size].txt, buf, len);
list[list_size].txt[len] = 0;
list_size++;
#define BUF_SIZE (128 * 1024)
+static void usage(void)
+{
+ printf("Usage: ./page_owner_sort [-m] <input> <output>\n"
+ "-m Sort by total memory. If this option is unset, sort by times\n"
+ );
+}
+
int main(int argc, char **argv)
{
FILE *fin, *fout;
int ret, i, count;
struct block_list *list2;
struct stat st;
+ int err;
+ int opt;
- if (argc < 3) {
- printf("Usage: ./program <input> <output>\n");
- perror("open: ");
+ while ((opt = getopt(argc, argv, "m")) != -1)
+ switch (opt) {
+ case 'm':
+ sort_by_memory = 1;
+ break;
+ default:
+ usage();
+ exit(1);
+ }
+
+ if (optind >= (argc - 1)) {
+ usage();
exit(1);
}
- fin = fopen(argv[1], "r");
- fout = fopen(argv[2], "w");
+ fin = fopen(argv[optind], "r");
+ fout = fopen(argv[optind + 1], "w");
if (!fin || !fout) {
- printf("Usage: ./program <input> <output>\n");
+ usage();
perror("open: ");
exit(1);
}
+ err = regcomp(&order_pattern, "order\\s*([0-9]*),", REG_EXTENDED|REG_NEWLINE);
+ if (err != 0 || order_pattern.re_nsub != 1) {
+ printf("%s: Invalid pattern 'order\\s*([0-9]*),' code %d\n",
+ argv[0], err);
+ exit(1);
+ }
+
fstat(fileno(fin), &st);
max_size = st.st_size / 100; /* hack ... */
list2[count++] = list[i];
} else {
list2[count-1].num += list[i].num;
+ list2[count-1].page_num += list[i].page_num;
}
}
- qsort(list2, count, sizeof(list[0]), compare_num);
+ if (sort_by_memory)
+ qsort(list2, count, sizeof(list[0]), compare_page_num);
+ else
+ qsort(list2, count, sizeof(list[0]), compare_num);
for (i = 0; i < count; i++)
- fprintf(fout, "%d times:\n%s\n", list2[i].num, list2[i].txt);
+ fprintf(fout, "%d times, %d pages:\n%s\n",
+ list2[i].num, list2[i].page_num, list2[i].txt);
+ regfree(&order_pattern);
return 0;
}
mode_t mode;
};
-static struct generic_type generic_type_table[] = {
+static const struct generic_type generic_type_table[] = {
[GT_DIR] = {
.type = "dir",
.mode = S_IFDIR
goto error;
}
+ if (buf.st_mtime > 0xffffffff) {
+ fprintf(stderr, "%s: Timestamp exceeds maximum cpio timestamp, clipping.\n",
+ location);
+ buf.st_mtime = 0xffffffff;
+ }
+
filebuf = malloc(buf.st_size);
if (!filebuf) {
fprintf (stderr, "out of memory\n");
prog);
}
-struct file_handler file_handler_table[] = {
+static const struct file_handler file_handler_table[] = {
{
.type = "file",
.handler = cpio_mkfile_line,
}
}
+ /*
+ * Timestamps after 2106-02-07 06:28:15 UTC have an ascii hex time_t
+ * representation that exceeds 8 chars and breaks the cpio header
+ * specification.
+ */
+ if (default_mtime > 0xffffffff) {
+ fprintf(stderr, "ERROR: Timestamp too large for cpio format\n");
+ exit(1);
+ }
+
if (argc - optind != 1) {
usage(argv[0]);
exit(1);
}
EXPORT_SYMBOL_GPL(gfn_to_page);
-void kvm_release_pfn(kvm_pfn_t pfn, bool dirty, struct gfn_to_pfn_cache *cache)
+void kvm_release_pfn(kvm_pfn_t pfn, bool dirty)
{
if (pfn == 0)
return;
- if (cache)
- cache->pfn = cache->gfn = 0;
-
if (dirty)
kvm_release_pfn_dirty(pfn);
else
kvm_release_pfn_clean(pfn);
}
-static void kvm_cache_gfn_to_pfn(struct kvm_memory_slot *slot, gfn_t gfn,
- struct gfn_to_pfn_cache *cache, u64 gen)
-{
- kvm_release_pfn(cache->pfn, cache->dirty, cache);
-
- cache->pfn = gfn_to_pfn_memslot(slot, gfn);
- cache->gfn = gfn;
- cache->dirty = false;
- cache->generation = gen;
-}
-
-static int __kvm_map_gfn(struct kvm_memslots *slots, gfn_t gfn,
- struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache,
- bool atomic)
+int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map)
{
kvm_pfn_t pfn;
void *hva = NULL;
struct page *page = KVM_UNMAPPED_PAGE;
- struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
- u64 gen = slots->generation;
if (!map)
return -EINVAL;
- if (cache) {
- if (!cache->pfn || cache->gfn != gfn ||
- cache->generation != gen) {
- if (atomic)
- return -EAGAIN;
- kvm_cache_gfn_to_pfn(slot, gfn, cache, gen);
- }
- pfn = cache->pfn;
- } else {
- if (atomic)
- return -EAGAIN;
- pfn = gfn_to_pfn_memslot(slot, gfn);
- }
+ pfn = gfn_to_pfn(vcpu->kvm, gfn);
if (is_error_noslot_pfn(pfn))
return -EINVAL;
if (pfn_valid(pfn)) {
page = pfn_to_page(pfn);
- if (atomic)
- hva = kmap_atomic(page);
- else
- hva = kmap(page);
+ hva = kmap(page);
#ifdef CONFIG_HAS_IOMEM
- } else if (!atomic) {
- hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
} else {
- return -EINVAL;
+ hva = memremap(pfn_to_hpa(pfn), PAGE_SIZE, MEMREMAP_WB);
#endif
}
return 0;
}
-
-int kvm_map_gfn(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache, bool atomic)
-{
- return __kvm_map_gfn(kvm_memslots(vcpu->kvm), gfn, map,
- cache, atomic);
-}
-EXPORT_SYMBOL_GPL(kvm_map_gfn);
-
-int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map)
-{
- return __kvm_map_gfn(kvm_vcpu_memslots(vcpu), gfn, map,
- NULL, false);
-}
EXPORT_SYMBOL_GPL(kvm_vcpu_map);
-static void __kvm_unmap_gfn(struct kvm *kvm,
- struct kvm_memory_slot *memslot,
- struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache,
- bool dirty, bool atomic)
+void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map, bool dirty)
{
if (!map)
return;
if (!map->hva)
return;
- if (map->page != KVM_UNMAPPED_PAGE) {
- if (atomic)
- kunmap_atomic(map->hva);
- else
- kunmap(map->page);
- }
+ if (map->page != KVM_UNMAPPED_PAGE)
+ kunmap(map->page);
#ifdef CONFIG_HAS_IOMEM
- else if (!atomic)
- memunmap(map->hva);
else
- WARN_ONCE(1, "Unexpected unmapping in atomic context");
+ memunmap(map->hva);
#endif
if (dirty)
- mark_page_dirty_in_slot(kvm, memslot, map->gfn);
+ kvm_vcpu_mark_page_dirty(vcpu, map->gfn);
- if (cache)
- cache->dirty |= dirty;
- else
- kvm_release_pfn(map->pfn, dirty, NULL);
+ kvm_release_pfn(map->pfn, dirty);
map->hva = NULL;
map->page = NULL;
}
-
-int kvm_unmap_gfn(struct kvm_vcpu *vcpu, struct kvm_host_map *map,
- struct gfn_to_pfn_cache *cache, bool dirty, bool atomic)
-{
- __kvm_unmap_gfn(vcpu->kvm, gfn_to_memslot(vcpu->kvm, map->gfn), map,
- cache, dirty, atomic);
- return 0;
-}
-EXPORT_SYMBOL_GPL(kvm_unmap_gfn);
-
-void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map, bool dirty)
-{
- __kvm_unmap_gfn(vcpu->kvm, kvm_vcpu_gfn_to_memslot(vcpu, map->gfn),
- map, NULL, dirty, false);
-}
EXPORT_SYMBOL_GPL(kvm_vcpu_unmap);
struct page *kvm_vcpu_gfn_to_page(struct kvm_vcpu *vcpu, gfn_t gfn)
struct kvm_fpu *fpu = NULL;
struct kvm_sregs *kvm_sregs = NULL;
- if (vcpu->kvm->mm != current->mm || vcpu->kvm->vm_bugged)
+ if (vcpu->kvm->mm != current->mm || vcpu->kvm->vm_dead)
return -EIO;
if (unlikely(_IOC_TYPE(ioctl) != KVMIO))
void __user *argp = compat_ptr(arg);
int r;
- if (vcpu->kvm->mm != current->mm || vcpu->kvm->vm_bugged)
+ if (vcpu->kvm->mm != current->mm || vcpu->kvm->vm_dead)
return -EIO;
switch (ioctl) {
{
struct kvm_device *dev = filp->private_data;
- if (dev->kvm->mm != current->mm || dev->kvm->vm_bugged)
+ if (dev->kvm->mm != current->mm || dev->kvm->vm_dead)
return -EIO;
switch (ioctl) {
void __user *argp = (void __user *)arg;
int r;
- if (kvm->mm != current->mm || kvm->vm_bugged)
+ if (kvm->mm != current->mm || kvm->vm_dead)
return -EIO;
switch (ioctl) {
case KVM_CREATE_VCPU:
struct kvm *kvm = filp->private_data;
int r;
- if (kvm->mm != current->mm || kvm->vm_bugged)
+ if (kvm->mm != current->mm || kvm->vm_dead)
return -EIO;
switch (ioctl) {
#ifdef CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT
projects / platform / kernel / linux-rpi.git / commitdiff